Fulcrum Microsystems Blog

By: Gary Lee

As former employees of Fulcrum Microsystems, we are very pleased to now be part of Intel Corporation. Our group has remained largely intact within Intel. Our 10GbE and 40GbE switch silicon provides an ideal complement to Intel Ethernet controllers and embedded processors used in applications such as large data center networks, broadband access and network security.

Fulcrum switches are now a key part of Intel’s data center strategy. Large data centers require efficient scalability that cannot be provided with traditional enterprise networking gear. There has been a lot of press recently about the importance of flat networks as more and more east-west traffic starts to dominate the data center. In addition, a single client transaction can spawn multiple server-to-server workflows making latency important in networking equipment purchasing decisions. New data center networking requirements such as virtualization and data center bridging (DCB) elevate the need for coordination between Ethernet controllers and network switches.

By adding Fulcrum switch silicon to its data center product portfolio, Intel owns a complete family of silicon components to enable these new large and efficient data centers. Fulcrum switches have been designed from the ground up to provide low latency data center fabrics that can scale up to tens of thousands of 10GbE server port. In addition, Fulcrum switches can be optimized with Intel Ethernet controllers to provide superior virtualization and DCB features than can be found in competing solutions. As part of Intel, we will see our vision of the future data center networks become a reality.

To see my future blogs, go to: http://communities.intel.com/community/wired?view=blog

By Gary Lee

Many of you have already seen the announcement of Intel’s agreement to acquire Fulcrum Microsystems. We have been very pleased at the positive response of our customers, partners, press and analysts to this news. Here are some examples from the press:

• Fulcrum buy could signal shift for Intel
• Buying Fulcrum Could Deepen Intel’s Data-center Role
• Intel Fulcrum Acquisition: Good News for HP (and Dell), Not So Much for Cisco
• Small Intel Deal Points to Bigger Chip Trend
• Arista Blog: The March to Merchant Silicon in 10Gbe Cloud Networking
• Intel Seeks Ethernet Leverage With Fulcrum
• Intel snaps up network chipper Fulcrum
• Dell buys Force10, Intel snags Fulcrum and Cisco gets heartburn

If you didn’t see the original press release, you can find it both on the Fulcrum site (pdf download) and the Intel site.

By Gary Lee

I’ve discussed several times in recent blogs the hot button industry issue of questionable latency measurement practices. I’m pleased to see that this is an issue of interest and importance worldwide, as articles on the subject have recently been published in Chinese media outlets, including EE Times China, C114.net, EEFocus.com and Electronic Fans.

In February of this year, Fulcrum announced the expansion of its team in China to address demand for advanced 10GbE switch silicon. Since then we’ve been working to broaden awareness in the emerging China networking equipment market for our FocalPoint FM6000 switch family. We’re pleased to see that our efforts to meet the growing demand for networking equipment in China with the best switch solutions on the market today are paying off, and that there is awareness and interest in the issues surrounding that market.

It can only help our cause when the media outlets in emerging markets provide the latest information on industry issues that can educate customers and guide them in making informed purchasing decisions.

By Gary Lee

As we saw in the OpenFlow Lab at Interop in May, OpenFlow is starting to take hold in the industry, potentially offering a better way to manage and operate large data center networks. By providing a standard API and centralized controller functions, it can potentially reduce both CAPEX and OPEX due to lower-cost switch hardware and simplified network operation and maintenance. A recent Network Computing article highlighted the potential value provided by OpenFlow, and included the following statement:

There are already companies taking merchant silicon from Broadcom and Fulcrum Micro to produce extremely low-cost Ethernet switches that have a forwarding table and an OpenFlow client support. This is opening a whole new market segment for “whitebox” switches to compete against the established vendors.

Fulcrum Microsystems has been supporting the OpenFlow standard in our ToR switch reference platforms for some time now. We are also working with our OEM partners to bring low-cost data center network solutions to the marketplace. But with a common network operating system like OpenFlow running on low cost “whitebox” switches, what value can our system partners bring to the equation?

In some ways, this can be compared to the early PC market, where system vendors not only developed applications, but also the underlying operating systems. Just as the PC market evolved into Windows and Mac OS, OpenFlow has the potential to become the standard network operating system, allowing vendors to differentiate themselves by focusing their efforts on advanced applications instead of expending resources by creating yet another network operating system. This should accelerate the pace of innovation in our industry.

by Gary Lee

Much of the recent technology innovation in cloud data centers has centered on improving server utilization (and therefore data center efficiency) by partitioning physical servers into smaller virtual machines. This has allowed big businesses access to very granular and scalable computing services for a range of business applications.

But why not look at these resources in another way? With large pools of interconnected servers in a cloud data center, applications that require compute clustering could also be run in the cloud.

Many small to medium size companies may need intensive compute resources during certain periods within their product development cycles. Why should they pay for in-house clustering resources that are under-utilized most of the time?

The Register has picked up on this theme with a new post on “High Performance Computing for you?” Here, three industry experts discuss how in-house HPC applications can migrate to the cloud.

… HPC isn’t just on-premise, isn’t just for experts, and isn’t just for finance and scientific apps  …  combining HPC with the cloud will put you in control of how you use it, and how much you pay for it.

Our FocalPoint switch IC product line has been designed from the ground up for use in scalable cloud data center fabrics. This includes extensive support for DCB, TRILL and server virtualization. But a key aspect of compute clustering is switch fabric latency. FocalPoint provides the lowest latency 10/40GbE L2/L3 switch fabric chips in the industry, making it a top choice for the interconnect fabric used in these new HPC applications.

by Gary Lee

Several years ago, Cisco developed a new product line called Nexus targeted at the data center networking market. Although this product family contained several advanced features for the data center, such as FCoE, companies like Arista have been beating them in several data center market segments that require ultra low latency, including financial trading. To address these markets, Cisco has been forced to move away from its internal ASICs and use merchant silicon in its new Nexus 3064 top of rack switch. In a recent Network Asia article, Cisco states:

Cisco also commissioned tests of the fabric, conducted by Miercom, that it claims show it outperforming Arista and BLADE switches in latency and jitter. Cisco also claims the Miercom tests highlight differences in software features, such as Layer 2/3 switching and routing, and management, required for high-performance trading, and that many of these features are only available on the Cisco fabric.

The Miercom report shows latency numbers ranging from 1.0uS-1.4uS for the Cisco 64-port switch, which is a big improvement compared to its previous Nexus 3000 products. This performance, though, doesn’t quite compare to the Arista 7100 series with latencies as low as 0.5uS for a 24-port switch.  Similarly, in a recent Lippis report test, Blade Network Technologies (IBM) demonstrated latencies in the range of 1.0uS-1.4uS for its 64-port switch and as low as 0.6uS for its 24-port switch.

As the switching engine that produced the results of the 24-port switch highlighted above, Fulcrum leads the industry with the lowest latency L2/L3 data center switch silicon, and has continued to do so with our FM6000 product family with densities up to 72 10GbE ports or 18 40GbE ports. You will soon see this silicon in ToR switch products, which will easily beat the 64-port latency numbers listed above by a factor of 2-3.

By Gary Lee

I noticed this article yesterday referencing a paper from Blade Network Technologies (now part of IBM) that estimates the market for 10GbE in high performance computing (HPC) at more than $15B in 2012.

But I think the article confuses HPC with high performance data centers.

In my mind, high performance computing is used in research labs and financial markets where low latency 10GbE switches and network adapters are used. The article only mentions latency in the context of bandwidth and not in terms of switch and NIC forwarding latency. It also fails to mention that low latency SFP+ PHYs are required in HPC applications and that you won’t see 10GBase-T PHYs in these high performance data paths.

The article goes on to talk about virtual machines, which are a corner stone of the new flexible data center, but are not as relevant to high performance compute clusters that have well understood workloads and server allocations. This is in contrast to the data center where workloads are dynamic requiring efficient server reallocation methods.  In any case, our FocalPoint product line has been architected around the concept of virtual ports from day one. Our latest FM6000 family can support up to 64,000 virtual ports across a multi-stage data center network and include VEPA or VN-Tag features.

What ever you call it, the market for 10GbE in the data center is growing rapidly, and Fulcrum expects to continue to be a dominant player in the 10GbE data center networking market along with providing the lowest latency 10GbE and 40GbE solutions for HPC applications.

By Gary Lee

It took almost 10 years for the 10GbE market to finally take off, but now the top of rack (ToR) switch segment of that market is showing 50% year-over-year growth and should reach $1B this year according to a recent Dell’Oro Report. What are some of the key factors driving this growth?

Ethernet NIC vendors have been offering 10GbE solutions for some time, but only recently have multi-core servers needed performance levels above one or two 1GbE links. Also, 10GBase-T PHY power levels have come down to the point where servers with built-in 10GBase-T ports are expected to start shipping in volume this year. Data center network administrators are seeing this coming wave of 10GbE enabled servers and are starting to purchase 10GbE ToR switches in volume.

New technology trends also play a role in this upsurge. Cisco and Brocade have been driving the adoption of FCoE in the data center. This requires a 10GbE network infrastructure in order to keep up with the performance levels currently provided by 4Gb and 8Gb Fibre Channel networks. Although most data center networks are not supporting FCoE today, network administrators are betting they will need this capability (and therefore bandwidth) in the near future.

Finally, in the financial services industry there is a huge incentive to drive down network latency. One component of latency is packet transfer in and out of the ToR switch. At higher port bandwidth, faster packet transfers and therefore lower latency can be achieved. This is why our customers like Arista, who is second only to Cisco in ToR switch sales according to Dell’Oro, use our 10GbE FocalPoint switch silicon to provide advanced features for the data center including the lowest latency in the industry.

By Gary Lee

Some of you may have been following a thread on this blog about questionable latency measurement practices used by some vendors.  This has been one of our most well read topics.

I’ve touched on this subject a couple of times, but didn’t feel like a blog post was a place to go into depth.  So we decided to submit an article in E.E. Times, which is an excellent place to go in–depth, and they have just published our article entitled: “LIFO or FIFO? How to accurately measure data center Ethernet latency.”

In this earlier blog post, I touched on the issue of store-and-forward switches taking advantage of a loophole in the measurement guidelines that make their latency measurements look better than is technically possible.  We joked that if that same loophole was applied to our FocalPoint switches, we could report negative latency – that is packets that arrive at their destination before they even leave the ingress port.

In the EE Times article, I discuss various switch architectures and their impact on latency, along with several ways to measure latency:

There are several ways to measure latency through a switch: first-bit-in to last-bit-out (FILO), last-bit-in to first-bit-out (LIFO), first-bit-in to first-bit-out (FIFO) and last-bit-in to last-bit-out (LILO). In each case, latency is measured at the switch ingress and egress ports.

Finally, the article concludes with test results of a FocalPoint reference design applying both the FIFO and LIFO measurement techniques to really drive home the impact that misusing latency standard can have.

Latency is one of the key issues in data center Ethernet, so it’s not enough to leave it to vendors to accurately report their latency measurements without knowledge of their testing methodology.  I hope you enjoy the article.

By Gary Lee

Most of what I write about on this blog is related to low-latency data center Ethernet, but in real life I’m also involved in showing customers how to best use high-speed Ethernet in telecom chassis backplane applications.  As today’s data traffic is increasingly packet-based, it only makes sense to send packets across the backplanes of these networking systems.

But as speeds reach 40Gbps and higher on network interfaces, one of the challenges engineers face is to ratchet up the performance and efficiency when using MCUs, NPUs and other processors to provide vital routing, security and other functions for the network.

One option is to use switch-based load distribution to spread high-bandwidth traffic across multiple processors.  To this end, Advanced and MicroTCA Systems Magazine just published an article I wrote on how to use 40G Ethernet load distribution in ATCA backplanes.  The article’s lead gives you the premise:

As network bandwidth capacity increases along with network services and security requirements, single processor subsystems cannot keep up with the load. In many of these applications, service blades utilize multiple CPUs or NPUs each with a 40G link to the backplane. ATCA systems are one example where backplanes are moving toward 40G per blade in response to these new requirements. These systems use 4 x 10G KR lanes as described by the IEEE in 802.3ap.

One key technology for distributing data loads across multiple processing blades is a flow-based hashing algorithm. In the article, I describe what that is and how it can be used to improve performance.  I was also able to illustrate this using a real-life example based on the FocalPoint product family.  Adding hash functionality to a chip also adds processing overhead, so the trick is to maximize that trade off.  In the FocalPoint family, we use a modified Pearson’s (PDF download) hash that offers high load distribution efficiency with a modest implementation overhead.

The rest of the article shows how this hash algorithm works across a 288-port fat tree network. With network bandwidth increasing dramatically over time, the challenges of processing every single packet for security, L4 services and other functions will only increase.  Load distribution is one way to tap the increasing capabilities built into today’s switch chips to help ensure the highest performance for these applications.  I hope you enjoy the article.