John Spiers |
I don’t see much difference with between this and FCIP (tunneling Fibre Channel data over IP networks.) The FCIP Spec. was ratified in 2004 (IETF RFC 3821.) I guess this is the T11’s cut at the same thing. FCIP never got any traction, what makes them think FCoE will? This still requires expensive HBAs for protocol encapsulation into TCP/IP packets and “DMA like” data movement.
When you look at the economics of deploying FCoE it may be less expensive to move to 10 Gig—E infrastructure, especially when 10GBASET hits the market. Why pay $2,000 (estimate) for a FCoE adapter when a 10 GBaseT adapter will cost $500 by the time these products hit the market in 2009. Bottomed line: Fibre Channel requires hardware for DMA data movement, whereas Microsoft and others are incorporating DMA functionality into the O/S for iSCSI, and you can always run with just a standard software initiator and NIC. There are no software initiators for FCoE. If there was, it would be iSCSI, because Fibre Channel uses the same SCSI protocol that iSCSI uses.
What customer is going to replace all his Fibre Channel HBAs, storage target HW and replace their FC switches with 10 Gig Ethernet/FC hybrid switches or blades when they can implement end-to-end 10 Gig iSCSI for less? I don’t see the ROI or benefit. This protocol will be relegated to remote replication like FCIP.
Karl Lewis |
I think FCoE makes good sense for array-to-array replication. On a campus backbone with 10GbE interconnects, it’s considerably cheaper than dedicated fiber, just to connect SAN islands. FCoE can leverage an existing (presumably underutilized) 10GbE infrastructure and add significant value.
FCoE between the server and storage, however, makes little or no sense, except for specific DR concerns. If a storage array is in a hardened bunker and its server is separated by distance, that could work. 10GbE, however, would clearly be cheaper way to go, however.
FCoE is not FC over IP. It requires an enhancement to Ethernet to provide reliable transport at Layer 2. This enhancement solves the determinism problem.
As there is no TCP/IP stack required for storage traffic, CPU overhead is not a factor, like with iSCSI. A TOE is not required to move FCoE at high rates.
FCoE can exist in a single subnet, and is not routable. It is a short-range protocol, and not an alternative to FC over IP over extended distances, or iSCSI across subnets. FCoE is not suitable for replication.
As for FCoE at 10Gb being overkill for servers, the concept is there will be a single (or single pair) of 10Gb Ethernet connections to a server, and all IP and FC traffic will flow over this converged network. Think of a pair of 10Gb Ethernet ports as being an alternative to four 1000BaseT ports and two 4Gb FC ports on a server.
Server virtualization is driving up the utilization rates and I/O requirements on servers, so 20Gb of connectivity vs 12Gb (4 x 1000BaseT, plus 2 x 4 Gb FC) seems reasonable.
Julian Satran |
What a piece of nostalgia
Around 1997 when a team at IBM Research (Haifa and Almaden) started looking at connecting storage to servers using the “regular network” (the ubiquitous LAN) we considered many alternatives (another team even had a look at ATM – still a computer network candidate at the time). I won’t get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple:
• Fiber Channel Protocol (SCSI over Fiber Channel Link) is “mildly” effective because:
• it implements endpoints in a dedicated engine (Offload)
• it has no transport layer (recovery is done at the application layer under the assumption that the error rate will be very low)
• the network is limited in physical span and logical span (number of switches)
• flow-control/congestion control is achieved with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer
• FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechanism)
• However FCP endpoints are inherently costlier than simple NICs – the cost argument (initiators are more expensive)
• The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) – the scaling argument
• The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s – the scaling argument
• Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has always been the preferred choice of the networking community – the community argument
• The “performance penalty” of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computing scene have enough compute cycles available to make any “offloading” possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.)
• Building on a complete stack makes available a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service location etc.) – the community argument
• Higher level storage access over an IP network is widely available and having both block and file served over the same connection with the same support and management structure is compelling – the community argument
• Highly efficient networks are easy to build over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don’t know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited – the scaling argument
As a side argument – a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI – and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows).
The networking and storage community acknowledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc.
The community also acknowledged the need to support existing infrastructure and extend it ina reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been
implemented and their foundation is solid.
The current attempt of developing a “new-age” FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc.
It ignores the networking layering practice, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applications more expensive and leaves aside the whole “ecosystem” that accompanies TCP/IP (and not Ethernet).
In some related effort (and at a point also when developing iSCSI) we considered also moving away from SCSI (like some “no standardized” but popular in some circles software did – e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time.
Graham Husk |
As and Administrator of storage and network services in a small corporation this proliferation of flavours of “SAN” is becoming a hinderance to our objectives, to move into virtualization. I am getting more and more nervous with the proliferation of remote disk technologies such as iSCSI or FC or FCoE . . .”. This as has been stated appears to be a marketing ploy, but also brings in doubt the longevity of any technology we choose. This is in my view criminal as what most businesses turn over in 4 years we normally turn over in 10. Our size and budget still points at iSCSI but we have to be right the first time to allow expansion of storage access as virtualization grows. Forums like this will help, but competing technologies only serve to frustrate the decision making process.
Jamie Borst |
As someone that recommends SAN’s to customers on a daily basis, I think that fiber is slowly but surely being edged out all but the most demanding environments by iSCSI.
Fiber’s got a nasty history and anyone thats had to go thru that nightmare will most likely not want to repeat history if given a suitable alternative.
iSCSI’s purity of transport and protocols have been a welcome relief to us since 2003. No compatibility issues, high per-port costs, host licensing fees, steep learning curves etc….
And some vendors like EqualLogic make a damn fast iSCSI array.