Interview: Greg Tiedemann on RapidIO

By: dl
Dec. 7, 2009

Greg Tiedemann, a product line director at Mercury Computer Systems, has been heavily involved with RapidIO since 2001. In this interview, he brings Open Architecture Review up to date on the standard, considers its place in the evolution of system interfaces and describes RapidIO’s relationship with various processor architectures.

OAR: Greg, the last time I took a serious look at RapidIO, the parallel version was happening, and the serial version was expected soon; StarFabric was a viable contender; HyperTransport made noises like a possible competitor; InfiniBand looked to have a future, at least box to box; Ethernet was going to take over everything; and the PCI Express interests were all rallying around Advanced Switching. Bring me up to date.

Tiedemann: Well, the dust has settled and Serial RapidIO is the fabric of choice for embedded fabric applications, the technology all the [telecommunications] systems companies and defense contractors went with. Compared to Parallel RapidIO, Serial RapidIO has a better power signature, delivers a lot more bandwidth and takes up less silicon, so the silicon manufacturers found it more efficient and easier to work with. 

OAR: So is parallel RapidIO dead?

Tiedemann: Pretty much. We have a legacy system we’re still shipping to one customer.

OAR: And the other fabric contenders?

Tiedemann: Advanced Switching was shut down; I’m not aware of any real market share for StarFabric; we don’t see any HyperTransport in our markets; InfiniBand is widely deployed in the IT computing infrastructure, blade to blade and box to box, mostly in the high-end server space; and PCI Express has emerged as a complementary interface to RapidIO, as well as a master–slave fabric in some of the embedded computing systems.

OAR: That’s ironic, given the historical Battle of the Fabrics.

Tiedemann: Yes, in the early days of the fabric wars, the PCI Express community was trying to take over the backplane and displace RapidIO. They looked at it as an all-or-nothing competition and wanted to dominate all of the interfaces—-board, backplane and intersystem. It really was a waste of energy because PCI Express is a very effective chip-to-chip technology that has evolved into a very effective switch fabric for some smaller-scale applications.

OAR: So you’re using both RapidIO and PCI Express?

Tiedemann: Yes. They’re complementary technologies. PCI Express is a good point-to-point solution, so we use it a great deal on the board and between adjacent boards; but whenever we need to move a great deal of data across a number of boards, we generally like to use RapidIO. It’s more scalable and easier to deploy.

OAR: Does PCI Express ever make sense board to board in the applications you serve?

Tiedemann: Yes, we are using it in our co-processing plane. Where there is a compute node in slot 2 and a co-processor, like an FPGA or GPU in slot 3, we use PCI Express to extend the computing to adjacent slots. We are also using it when you have an Intel host in a master-slave arrangement, but when you start to scale up to multiple computer nodes, that’s where RapidIO excels.

The DSP guys like PCI Express because it’s very efficient and doesn’t take up a lot of gates within their processors. It’s designed for packet switching, which the signal processing guys, especially in the base station market, really like. And it’s an easy programming interface. All those things make RapidIO an ideal solution in larger systems and PCI Express an ideal solution in other areas.

OAR: So the two interfaces coexist in a hierarchical arrangement with a division of labor. Don’t you also support Ethernet?

Tiedemann: Yes, exactly, a division of labor. You could also look at it as a “multi-plane” architecture. And by the way, the Ethernet guys also thought they were going to take over everything in the box, but what really evolved is coexistence.

All our systems will generally support Gigabit Ethernet as the control plane interface; above that is the data plane interface, generally RapidIO, which provides a very deterministic data plane that doesn’t interfere with the control plane traffic; and then we have the concept of a co-processing or expansion plane interface, which is usually PCI Express. And they all scale nicely together. 

OAR: What do you mean by coprocessing plane?

Tiedemann: There are architectures where you would like to couple compute resources together between boards. That’s the coprocessing plane, where a PCI Express chip-to-chip interface may extend between boards, say, from a PowerPC on one board to an FPGA on an adjacent board. In this scenario, PCI Express can provide a very low-latency, deterministic interface.

OAR: So what does RapidIO have that PCI Express doesn’t? 

Tiedemann: I don’t think of it anymore as a one or the other sort of comparison. RapidIO was designed from the ground up to scale to hundreds of nodes. PCI Express has evolved as a very effective chip-to-chip interconnect with a tremendous amount of bandwidth. They have evolved from different places and are good at specific things. There are some cases where there is an overlap in functionality and you could use one or the other. You have to look at it on a case-by-case basis.

OAR: Why not use RapidIO chip-to-chip, too? Wouldn’t that simplify things?

Tiedemann: Good question. We will use RapidIO between processors on a board and then extend that interface board to board through switches and in meshes. And where it’s appropriate, we would use PCI Express. You try to use the right solution for the right application, rather than force-fit an interface.

OAR: I assume RapidIO has upped the performance ante over time. If I remember right, RapidIO 1 ran at roughly 1, 2 and 3 Gbits/sec, and x1 (single-lane) and x4 (quad-lane) links were defined.

Tiedemann: Yes. RapidIO 2 brought that up to 5 and 6.25 Gbaud, and it has added x2, x8 and x16 links. The RapidIO Trade Association is now working on Serial RapidIO 3. [Click here to take a look at the RapidIO Trade Association’s roadmap.]

OAR: To what do you attribute the success of RapidIO?

Tiedemann: This is an interesting area to discuss. There were several events for RapidIO that made it successful. The first, I believe, was establishing an open standards group and soliciting the end customers [Ericsson, Nortel, EMC, Lucent and others] to drive the requirements. The second was having a variety of silicon suppliers actively involved to work collaboratively with the end customer to debate what was feasible, both technically and from a business perspective.

Then, it turned into a race to get the silicon to market. Freescale initially only put Parallel RapidIO into one version of its PowerQUICC, then moved everything over to Serial RapidIO once it became available. Then TI and Freescale adopted it for their DSPs; Xilinx and Altera had RapidIO end points available in their FPGAs; and Tundra and IDT put it in their switches.

In the end, getting the right silicon into solutions that the end customers could deploy and demonstrating a high level of interoperability — that’s what really drove the adoption and success of RapidIO. And for that matter, it’s a common model for any open technology standard to be successful. That’s where some of the other technologies we were talking about failed.

OAR: A few weeks ago, you introduced an AMC that couples an Intel processor with RapidIO, which I think is the first time someone’s done that. With its native PCI Express interface, why would you want to use Intel in a RapidIO environment instead a processor with a native RapidIO interface?

Tiedemann: From a technology perspective, Intel processors are attractive to us for a number of different reasons, they have a level of performance we’d like to leverage for various signal processing applications and Intel software support is tremendous. However, as we discussed, the native PCI Express from an Intel chip can’t really scale up in a system architecture. We could use 10 Gigabit Ethernet, but that’s not going to support some of our embedded performance requirements. So in order to provide a more deterministic fabric, we bridged the [processors’ native] PCI Express interface to RapidIO. That will let us use Intel in some higher-end multicomputer applications. 

OAR: You do the bridging in an FPGA?

Tiedemann: Yes.

OAR: So you’re putting x86 processors up against TI DSPs and Freescale PowerPCs?

Tiedemann: Not really TI DSPs; it’s more applicable to PPCs. The DSPs are very low power, very efficient processors, designed for very specific application areas, not general-purpose processing, while Intel chips are heavier-weight, general-purpose processors that could be used in a number of applications. Their power is certainly higher and the footprint of the chip is certainly much bigger, but there are some signal processing applications that can afford the power where the customer wants the flexibility of working with an x86 architecture versus a fixed-point DSP programming model.

The AltiVec engine of the PowerPC products, in turn, really set the bar for multicomputing performance, and we’ve leveraged it for a lot of applications. But if the customer doesn’t want to do that special coding [to take advantage of AltiVec], the x86 might be more suitable.

Intel can scale up to some signal processing applications that we have traditionally used PowerPCs or FPGAs for, but those are mostly single-processor applications. It makes the most sense when you have an Intel host in a master-slave arrangement.

OAR: Can I assume that the Intel approach is less expensive?

Tiedemann: That may be the perception, but there are some challenges to consider when deploying Intel. The chips are designed for commercial applications, so you have to invest in technologies to cool them properly, and then spend some time to make them rugged enough from some of the defense applications. These are just a couple of examples of perception versus reality. In the end, the embedded computing companies have to show a cost-per-performance improvement to stay competitive. So we will always be looking for areas to add value.

OAR: What do you see coming for RapidIO in the future?

Tiedemann: More of the same. It’s widely deployed throughout the defense and commercial markets, and it will continue to grow in market share. We also talked about the improvement in performance with RapidIO that we are tracking and will support. RapidIO clearly has become the fabric of choice for all our high-performance multicomputers.

*************************************

For an entirely different view (Ray Alderman’s) on PCI Express, click here to read Why PCI Express stinks (for multiprocessing).