Fulcrum Microsystems Blog

Recently, BusinessWeek ran an article on Arista Networks and its founder Andy Bechtolsheim, identifying the company as a legitimate challenger to Cisco’s position as the “dominant provider of the networking equipment at the heart of the Internet.” While the article takes a high level view of the competitive landscape, it points out that Arista can challenge Cisco in the “fastest-growing part of the market: blazing-fast gear for the data centers where more and more computing is done.”

An interesting element to this article, though, is the discussion that takes place in the comments section in which one reader notes:

“The key to not becoming a dinosaur is to always be in the business of putting your legacy behind you. Keep your brand on the legacy, but at the same time drive the revolution that will unseat that legacy, under its own brand.”

The elephant of the internet is struggling to reposition its enterprise products in the data center.

Cisco clearly has the most market share outside the data center, but it comes at the data center with a router-centric enterprise LAN mentality and significant incentive to maintain that architecture, which limits data center performance and scale.

It’s a stark contrast to Arista and other data center network innovators, who have eliminated the latency that had impeded Ethernet in the data center.  By offering low-latency switches with high port densities and non-blocking scalability, Arista is flattening network architectures and offering a performance-optimized alternative to the traditional router-centric model.

Comments welcome: feedback@fulcrummicro.com

So how important is low latency on Wall Street?  In September I attended the HPC on Wall Street conference, and low latency was simply everywhere.  If you look at the program, you could have easily spent the entire day listening to presentations just on low latency. Even Cisco was promoting low latency-  despite their new Nexus switch has 5x higher latency than a similar Fulcrum-based switch.  In their pitch, they bring up a good point in that the biggest feature in the data center being “trust.”  “If you can’t trust your network, you need a new one.” I definitely agree with that, but the assertion that you can only trust Cisco is wrong.

Cisco has traditionally developed the ASICs for its switches (or bought a company that has done this) such that they are the only customer for these ASICs.  This is a great way to control quality, while providing a level of trust that is critical for production deployment in some of the biggest networks in the world.  Now compare this to commodity switch ASICs like FocalPoint.  We provide the same ASICs to Cisco’s competitors as well as customers in telecommunications, storage, IPTV, broadband services, wireless, military, aerospace, etc.

The fact that our switches will find literally hundreds of different customers and applications that need to work on a battlefield or in a 10-15 year lifecycle telecom access box or even be sent into space, means that it needs to attain a high level of trust across multiple markets and applications—not just in the data center.

The only way to get this right in so many different environments is to start with rock solid building blocks—our proven high performance architecture, tape outs in the most stable of process nodes, silicon proven high speed I/Os, and software compatibility across the FocalPoint family.  It is hard to measure trustworthiness—you could try to measure it in dollars spent in testing—but this doesn’t get to the engineering and design that the silicon is built and tested on.  If you use the market itself as a gauge, you will see FocalPoint deployed in some of the most demanding environments, especially the datacenter.

Comments welcome: feedback@fulcrummicro.com

Recently, I attended the Hot Interconnects conference, which was hosted by Credit-Suisse in New York City. This was an especially apt location because of the interesting things that Wall Street is doing with its networks.  And on Wall Street, interconnect performance, including low latency, is critical.

An important topic at the conference was multicast.  I presented a paper co-authored by IBM that presented test results with an emphasis on full line-rate multicast.  We stressed the switch with lots of 10GbE traffic (more traffic than most real networks could ever possibly throw at it) and the switch performance, of course, was awesome.  When I presented the paper, I talked up the idea of parallel multicast that falls out from these results.

At the conference, the issue of fairness of information came up quite a bit.  For example, if two traders are subscribing to the same market ticker feed (IP multicast), and one trader receives the updates 1 millisecond before the other, this trader has a distinct trading advantage over the other.  As a result, Wall Street brokerages subscribing to this feed are even moving their servers in with the exchanges that generate the ticker information, to save the latency incurred from the data having to physically leave the exchange and reach the trader.

So then, the question is, how fair does this need to be?  A difference of a millisecond today is described as a “lifetime,” while saving 100 microseconds can directly affect the bottom line of a trading operation.  For this reason, you need a network that can deliver updates to each trader at essentially the same time, by taking the message sent from the exchange and copying it in parallel to each trader connected to the network.  This is a key feature provided by our FocalPoint switch.

When a frame is multicasted out multiple ports, a copy of the frame will be read out of each egress port (after being scheduled) at essentially the same time assuming those egress ports are not congested.  As a result of our shared memory architecture, the maximum amount of skew between these copies is only 67.2 nanoseconds.  This latency skew is so small that it is difficult to measure.  This minimal amount of skew allows these multicast frames to be delivered through a large-scale data center network with a “skew budget” of just a few hundred nanoseconds or less, providing a high degree of fairness to the traders.

While all the talks at the conference were excellent, I’ve included links to a couple below I thought were closely related to the types of things we are working on:

The other face of on-chip interconnect. Probably the best way to utilize a 10GbE or 40GbE link is to put thousands of cores on one chip, which is why Tilera [http://www.tilera.com] is pretty exciting.

Data Center Switch Architecture. A paper on how to build a monstrous 64,000-port switch using a three-tier Clos architecture.  The key here is to build it using off-the-shelf Ethernet switches, making network bandwidth just as commoditized and freely available as computing power has recently become.  Pretty cool stuff.

Designing Next-Generation Clusters. This paper compared Nehalem processors with the previous Intel generation (Cloverton) and found that there wasn’t much of a performance leap within a small HPC cluster.  Interestingly, there was very little difference when double data rate (DDR 20G) InfiniBand vs. quad data rate (QDR 40G) InfiniBand was used.  So even a Nehalem can’t really utilize more than a DDR IB connection.  If you can’t really utilize the faster data rate of IB, what is the point of using it?

Now that Ethernet is viewed as the unified data center fabric, one of the major questions remaining is, what is the best data center switch architecture.  An article in Network World by Jim Duffy describes Cisco’s data center networking worldview, which, coincidentally, mirrors their traditional LAN worldview.

The argument: If you can reduce a three-tier network design to two tiers by eliminating the aggregation layer, you can reduce the latency to an acceptable level for sensitive data center applications.

Let’s examine the notion.

The Cisco multi-tier architecture is fundamentally a router-centric model — a large, complex, latency-sucking, and prohibitively-expensive router is at the core of their networks.  Multiple tiers of aggregation switches are connected to the central router in an effort to gather traffic and better utilize the expensive resource.  It happens to have three tiers in it—two of which are aggregating to the central router. But what’s more important is that the scalability of the system is limited to the capabilities of the core router and a company’s appetite for oversubscribing the network on the way to that central router.

Let’s contrast that with a proven scalable model for data center networking—Clos (or fat tree) network architectures, which is the model employed by low-latency InfiniBand networks.

Fat trees are logically-flat peer-to-peer architectures that can be scaled by adding additional layers of switching in the system — without adding logical hierarchy.  In this design, each of the switching elements feature the intelligence, low latency and high performance to contribute to data throughput and not just serve to groom traffic toward some almighty router to preserve and protect the precious capabilities of that box.  This design is about linking common switching elements together to create a larger, distributed version of the same switch, which acts as a single logical switch entity—even though it is sometimes composed of multiple layers of physical switching. With this flatter logical architecture and the low latency of each switching element, built-in aggregation enables you to boost scalability.

So…two tiers or three? Which is better for the data center?  The answer lies in whether the tiers are logical or physical.  Cisco’s proposal of removing an aggregation layer but maintaining a multi-tier logical hierarchy does very little to improve latency, and reduces scale and overall network performance.  Doesn’t sound like much of a solution for the data center.

Read the Network World article…

Comments are welcome: feedback@fulcrummicro.com