2.7.5 IP Storage (ips)

NOTE: This charter is a snapshot of the 49th IETF Meeting in San Diego, California. It may now be out-of-date. Last Modified: 23-Oct-00


Steve Bellovin <smb@research.att.com>
Elizabeth Rodriguez <egrodriguez@lucent.com>
David Black <black_david@emc.com>

Transport Area Director(s):

Scott Bradner <sob@harvard.edu>
Allison Mankin <mankin@east.isi.edu>

Transport Area Advisor:

Allison Mankin <mankin@east.isi.edu>

Mailing Lists:

General Discussion:ips@ece.cmu.edu
To Subscribe: ips-request@ece.cmu.edu
In Body: subscribe ips
Archive: http://ips.pdl.cs.cmu.edu/mail/maillist.html

Description of Working Group:

There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing protocols, such as SCSI and Fibre Channel, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing protocols. Standards for the protocols to be encapsulated are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable. In that case the WG will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The WG will consider whether a layered architecture providing common transport, security, and/or other functionality for its encapsulations is the best technical approach.

The protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery at higher levels is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). The WG's encapsulations may require quality of service assurances (e.g., bounded latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide them in shared traffic environments (e.g., the Internet) based on existing IETF QoS mechanisms such as Differentiated Services.

Use of IP-based transports raises issues that do not occur in the existing transports for the protocols to be encapsulated. The WG will address at least the following:

- Congestion control suitable for shared traffic network environments such as the Internet.

- Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network.

- Naming and discovery mechanisms for the encapsulated protocols on IP-based networks, including both discovery of resources (e.g., storage) for access by the discovering entity, and discovery for management.

- Management, including appropriate MIB definition(s).

The WG will address security and congestion control as an integral part of bits protocol encapsulation(s); naming, discovery, and management are important related issues, but may be addressed in companion documents.

The WG specifications will provide support for bridges and gateways that connect to existing implementations of the encapsulated protocols. The WG will preserve the approaches to discovery, multi-pathing, booting, and similar issues taken by the protocols it encapsulates to the extent feasible.

It may be necessary for traffic utilizing the WG's encapsulations to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways.

Effective implementations of some IP transports for the encapsulated protocols are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware.

The standard internet checksum is weaker than the checksums use by other implementations of the protocols to be encapsulated. The WG will consider what levels of data integrity assurance are required and how they should be achieved.

The WG will produce a framework document that provides an overview of the environments in which its encapsulated protocols and related protocols are expected to operate. The WG will produce requirements and specification documents for each protocol encapsulation, and may produce applicability statements. The requirements and specification documents will consider both disk and tape devices, taking note of the variation in scale from single drives to large disk arrays and tape libraries, although the requirements and specifications need not encompass all such devices.

The WG will not work on:

- Extensions to existing protocols such as SCSI and Fibre Channel beyond those strictly necessary for the use of IP-based transports.

- Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic.

- Support for environments in which significant data loss or data corruption is acceptable.

- File system protocols.

Operational Structure:

Due to the scope of the task and the need for parallel progress on multiple work items, the WG effort is organized as follows:

A technical coordinator will be identified and selected for each protocol encapsulation adopted as a work item by the group. This person will be responsible for coordinating the technical efforts of the group with respect to that encapsulation, working with and motivating the document editors, and evangelizing the group's work within both the community and relevant external organizations such as T10 and T11.

In addition to the normal responsibilities of IETF working group chairs, the IPS chairs hold primary responsibility for selection of coordinators, identifying areas of technical commonality and building cross-technology efforts within the group.

Coordinators for initially important encapsulations:

SCSI over IP (aka iSCSI): John Hufferd (hufferd@us.ibm.com)

Fibre Channel (FC-2) over IP: Murali Rajagopal (muralir@lightsand.com)

Goals and Milestones:

Oct 00


Submit the initial protocol encapsulations as working group Internet-Drafts.

Nov 00


Submit initial version of framework document as an Internet-Draft.

Dec 00


Discuss drafts and issues at the IETF meeting in San Diego.

Feb 01


Submit final versions of requirements drafts to the IESG for consideration as Informational RFCs.

Mar 01


Discuss framework, specification and related drafts (e.g., MIBs, discovery) for the protocol encapsulations at IETF meeting in Minneapolis.

May 01


Submit protocol specification drafts to the IESG for consideration as Proposed Standard RFCs.

Jun 01


Begin revision of WG charter in consultation with the Area Directors.

Aug 01


Meet at IETF meeting to close any open issues and finish any outstanding work items, including MIB, discovery, and framework drafts.

Sep 01


Submit MIB, discovery, framework, and any other WG drafts to the IESG for consideration as appropriate to each draft.


No Request For Comments

Current Meeting Report

IETF IP Storage (ips) Working Group Meeting Minutes
San Diego IETF Meeting, December 11-12, 2000

---------- Monday December 11, 2000

EMC will be sending out an IPR notice regarding a patent related to iSCSI and FCIP to the mailing list.

Interim meeting being scheduled for week of January 15, to coincide with T10 in Orlando - Grosvenor resort.

-- Framework document - Mark Carlson (Sun)
Describes environments for IP Storage. Includes terms, background on various protocols. This is a living document.
Currently more of a survey.
This document will coordinate with Naming and Discovery.
Looking for more co-authors, please contact Mark if you are interested.

-- Framing discussion -- Randy Haagens (HP) and Allyn Romanow (Cisco)
- Allyn and Randy were asked to compose this presentation by the ADs.
Purpose was to try to clarify the problem and present a range of solutions.
- Framing is a common challenge with for both iSCSI, FCIP as well as non IPS documents. While framing is not explicitly required, a solution for a more effective iSCSI specification is highly desirable. The focus of the presentation was understanding the requirements of framing (i.e. the problem). Reaching consensus on a solution was not one the goals of the presentation. Allyn started the presentation by pointing out that this topic will also be discussed on Monday night in the TSVWG.
- The problem: TCP reassembly can be costly, and in some instances not feasible.
Also, there is limited host memory and host bus bandwidth, so one wants to avoid manipulating the data more than once. Best would be one use of the bus and memory - zero copy. Note: This is not the same as TCP zero copy. TCP typically waits for all the data to arrive, and then copies the data to host.
- In outbound direction, data can be transferred directly from memory to the protocol controller and out onto the wire. In the inbound direction, when received out of order, data has to be put in a reassembly buffer until all data is received.
- One solution: Direct Memory Placement (Payload steering; data steering; RDMA) --

- In this solution, must address loss of ULP sync - when a segment containing a ULP header is dropped or delayed, ULP sync is lost. Direct data placement cannot continue; data must be diverted to a reassembly buffer. Goal is to recover ULP sync at the next ULP header. There are both TCP aware and TCP unaware solutions to recovering ULP sync.
- TCP unaware approaches:
a) SCTP - issues with this include lack of widespread deployment
b) Special Characters - requires byte by bytes processing
c) Fixed length ULP messages - Inefficient for short ULP messages
d) Periodic Marker - Best solution for this class of approaches
Sublayer of a framing protocol. Manageable; relatively easy to implement in hardware Marker 4 byte field number of ULP bytes remaining in current PDU. Marker inserted and removed by framing protocol; e.g. iSCSI. After loss of sync, locate next marker; use to locate the next ULP PDU. Markers are transmitted twice in a row; ensures markers cannot be split by stream fragmentation/segmentation.
- TCP aware Approaches
a) URGent pointer - disallowed
b) PSH bit - disallowed
- Another TCP aware approach can be considered by the TSV working group.
The TSV working group works on small items in the transport area that do not need a full working group as well as TCP/UDP transport issues.
- Allyn Romanow presented a technique for demarcating message boundaries using a TCP option. This consists of using one of the reserved bits in the TCP header to extend TCP to support this type of framing. Then can add up to 40 bytes before the TCP payload. Problem is that these reserved bits are a scarce resource; need to evaluate the need for the change. Also any time a change to TCP is proposed, there is tension, e.g. tension between the need to update TCP and stability of TCP.
- Procedure for standardizing a TCP option consists of
a) The IESG has to approve new work items for the TSV wg.
b) Ask the Transport Services (TSV) working group to adopt this as a WG item
c) Pros-and cons will be discussed on the TSV wg mailing list. If it supported,
hopefully the spec will be wrapped at the next IETF (roughly 3 month time frame). If no support, it's dead. The advantage of the TSV wg is that transport experts will be able to contribute feedback.
d) If supported, will be adopted at next IETF meeting.

Advantage is that people who are experts in transport will be able to contribute, and that this will not be an iSCSI specific solution. IPS should follow this process and contribute. Make sure that the solution (since not iSCSI specific) meets the needs of this group.

This is a very common problem, that is worthy of consideration at the transport layer. Addresses areas beyond IPS. The TCP option is not the only approach. TCP header bits could potentially be used for framing.

The flag approach may send many packets that are less than MSS. This is potentially a risky change to TCP.

Message Boundary Option
Two approaches. Not in drafts yet:
- Flag approach -- Costa has written up; will post as draft.
The flag approach may send many packets that are less than MSS.
This is potentially a risky change to TCP. ULP header is aligned with first byte of TCP payload.
- Offset Approach -- 4 bytes. 2 byte offset indicates offset into TCP payload of first ULP header in the segment. Write-up forthcoming.

Discussion - Lead by Steve Bellovin
Steve Requested the group concentrate on Requirements. The discussion raised the following points:
- Another option for alignment - periodic alignment instead of periodic marker. There could be a requirement in iSCSI that an upper-layer header appear every n kbytes in the TCP stream. Padding could be used to make sure this happens.
- This is not the first time that this issue has arisen, and there is value in a general solution that is applicable to other protocols, even though this may take longer to deploy. The consensus in the room was that a general approach is preferable to one specific to iSCSI.
- Multiple message boundaries in a single TCP segment are not a problem. Once the first boundary is found, the rest are found by examining the iSCSI (or other ULP) headers.
- If there is a large gap between message boundaries, the data in the gap will need buffering. Implementations may wish to consider this in setting maximum data size for a PDU.
- RDMA is different but related to this topic. Any RDMA protocol will either incorporate or assume framing. It may make sense to spec a generalized RDMA protocol on top of this framing mechanism.
- Implementation of this sort of framing would be optional.
- A generic data framing protocol may also be a good place to put in a stronger CRC than the 16-bit Internet checksum. Drafts making specific proposals are welcome.

Steve Bellovin asked for a hum of the room on whether to solve the "framing problem" in an iSCSI-specific way or whether to pursue a mechanism to add to TCP. The hum in the room was to do it in TCP.

-- ISCSI document review - presented by Julian Satran.

- Rough consensus has been reached on the session model - Symmetric with optional multiple connections.
- Login Session context - good understanding.
- Login Security context - more work needed.
- Commands, messages, tasks, and tags almost complete. Items open - coding, some layout.
- Response numbering scheme is well understood; complete.
- The data numbering scheme has received no consensus. It may be removed. Julian's personal opinion is that it's optional and low cost with advantages.
- For recovery, command restart and status well understood. No consensus on data recovery. Digest not well understood; needs to be readdressed.
- Text commands - negotiation mechanisms done.
- Mapping moved to T10 (aliasing). Dropped from iSCSI.
- RDMA/Sync, Security/Authentication - all are still open issues.
- Authentication - login phase must provide authentication. This was the consensus at the last meeting. Every iSCSI PDU must provide data integrity and authentication.
- A mechanism should enable optional end2end data protection/authentication.
Would like to use TCP recovery in presence of error. Digests can be activated at a higher level. Need a mechanism that can be activated on demand, ideally at login.
- The current digest scheme needs to be changed. Julian suggested using IPSec for data integrity, since all the above mechanisms are provided by IPSec, it is a best fit for what is needed and very cheap if use only what is needed. Can insert own policies, including policies that will verify integrity verses provide security but use same mechanisms. Policies will be addressed in next two weeks.
- David: IPSec does negotiation securely. What is currently in the draft is most likely vulnerable to man-in-the-middle attack.
- Steve Bellovin indicated that the IPSec WG would be extremely opposed to any insecure non-cryptographic algorithm being defined for IPSec. Silicon must support SHA-1 or MD5 in order to do key negotiation. There are active discussions/proposals on how to do high speed encryption/negotiation. Early in process; drafts not yet standards, but worth looking at this.
- Mark Bakke really wants to maintain the separate iSCSI header/iSCSI payload digests. This separation is lost by moving to IPSec. Gained data integrity is only as good as the group is willing to pay. Good integration with encryption.
- Can use IPSec in transport mode, which will provide end2end protection. Integrity is required end2end, but security may not be. Security may need to be removed at the firewall/gateway, but need to still be able to verify integrity at the endpoints. Can have multiple layers of IPSec if needed. Comment from audience
- not recommended.
- David Peterson of Cisco asked whether ACA will be mandated by the draft.
The consensus, after the discussion, is that iSCSI must support ACA but that a device need not support ACA (Ralph Weber pointed out that few initiator use ACA today). There was some grumbling because ACA is needed for reliable pipelining of ordered commands in the face of errors.

- There was a question on whether asynchronous event notification (AEN) was mandatory to implement in iSCSI. Again, iSCSI transports must support asynchronous events but iSCSI devices need not. Somebody pointed out that SCSI mode pages can be used to regulate whether a device generate AENs.

- Ralph Weber (T10 secretary) praised iSCSI for trying to advance the state of the art in SCSI.

-- iSCSI requirements --- presented by Marjorie Krueger (HP)

T10 work on authorization will not be integrated into iSCSI; to the extent that SCSI provides authorization, that's T10's domain. The fact that IP networks are less secure than typical SCSI environments have been in the past introduces additional issues that need to be addressed here in iSCSI. T10 work will be used and referenced where applicable.

It was noted that the point of iSCSI authentication and authorization was to control who was able to get to a target.

-- Bootstrapping -- presented by Prasenjit Sarkar (IBM)

This document contains guidelines for how iSCSI boot clients connect to iSCSI boot server. Included description of how to use existing techniques. iSCSI boot clients need IP address, iSCSI boot server service delivery port name, default; LUN= 0; iSCSI initiator software.

Boot process steps:
Client software stage
Use PXE or related bootp/tftp protocol to get iSCSI initiator software
DHCP stage
Use DHCP to configure client IP address
Use new DHCP option to configure iSCSI boot server service delivery port name
Discovery server stage
Use "to be defined" iSCSI delivery service to get iSCSI

There was a question on whether the boot client had to have IPsec, in light of the integrity proposal by Julian and security proposals by others. It is not required; bootp is sufficient.

The absence of security requirements for boot was pointed out. The current goal of the boot document is to remain neutral on security (neither mandate nor disallow).

There was some question on what to do with the iSCSI session once a bootstrap program was done with it. It was noted that it was probably simplest to close it and have the loaded program establish a new iSCSI session, but this is up to implementations.

-- MIB presentation - Mark Bakke (Cisco)

A group is forming to work on iSCSI MIB. The scope is management of iSCSI as opposed to SCSI. If necessary, a separate SCSI MIB (if one does not already exist) would be addressed separately. The original MIB structure in the current draft is not adequate, and is being redone. These revisions will also bring the MIB up to date with the current iSCSI draft.

A question was raised about how FC-style zoning works with the MIB. It's not clear how zoning fits into the iSCSI architecture.

The MIB could be implemented on anything running iSCSI including initiator, target, and gateway.

The FC HBA API available from SNIA might be of interest to this group. It has a complete list of things management tools want to be able to see out of an initiator.

----- Tuesday, December 12, 2000

-- Naming and Discovery Requirements - Mark Bakke (Cisco)

Naming and discovery will specify target discovery but would leave LUN discovery to SCSI mechanisms, such as REPORT LUNs. There was a bit of debate on this; why not go all the way and support LUN discovery in the naming system? The counter-argument is based on layering: "Leave unto SCSI that which is SCSI's".

Scaling requirements include both small and large environments.
Find targets by querying SNS. Small environments do not require SNS.
Hierarchical format, with Naming Authority.

World Wide Unique Identifier
Address composed of IP addr+TCP port+Target Name, URL like.
Plan to apply for well known port for TCP. In such a case, an address w/o TCP specified would default to this well known port.

Format includes info on naming authority, including support for 'local' naming authority.

Character set to be allowed? Unicode?
Recommend UI schemes for naming authority.
Need to look at security issues.

T10 issues - reservations, reset, LUN naming. Target reset discussion. Noted that T10 is thinking of making target reset optional.

Is breaking of a connection in iSCSI equivalent to a target reset? Consensus is no: the end of a session was equivalent to a target reset and would also cause any persistent reservations to be released.

Naming scheme will allow multiple port and multiple initiator/target discovery. Will give list of targets + all paths to that target.

Draft currently an individual submission - consensus (hum) taken, to be adopted as working group document. No opposition hums.

-- iSNS document presented by Josh Tseng, Nishan

ISNS describes a scalable information facility for registration, discovery and management of networked facilities.

ISNS follows a client/server architecture. If client registers with name server, allows itself to be managed by the name server.

Why needed? Simplifies storage management implementations. Allows greater scalability over broadcast/multicast discovery methods. Supports zoning.

Next step - incorporate requirements/suggestions from IPS working group. Extend document for FCIP

Access control - what is name server role? Targets upload public key to name server. Enforced at the end node/target. Supports both soft and hard zoning.

How does it fit into discovery? Naming and discovery team will look at this to see how well it fits. Should this be maintained as a separate document vs incorporated into naming/discovery team? Yes, this is a separate document because it supports more than just iSCSI.

In reading the draft, reliance on WWN. This draft would need to be redone to support WWUI of n&d requirements.

Direction is one in which naming and discovery team approves of? Yes, close.

Is there working group consensus as a base document; working w/ NDT group to produce a revised document, aligned with N&D, which would then be adopted as an official wg document. Rough consensus - next revised version of document will become an official working group document. Not unanimous.

-- FCIP - Status and progress of FCIP. - Raj Bhagwat (LightSand)

Current status - difference from previous presentation Solution for bridging remote FC SAN islands. From FC point of view, appears to be entirely an FC network. Initially did not have congestion management (previous presentation).

Draft overhauled to incorporate TCP as transport in order to address congestion management and recovery mechanisms. In rev -00, PSH flag incorporated. Based on feedback from mailing list, this was eliminated and in -01, a new frame boundary mechanism introduced. Topics under discussion -- QOS, security, MTU/MSS, Framing/synchronization, order of delivery, discovery, error recovery.

Alignment with new project in T11 - FC-BB2. FC-BB2 focused on issues outside the scope of the IETF, including link level issues. Target date for completion - June 2001.

Much FC/IP work is being done on conference calls. Conference calls are design team calls open to design team members and authors. Public review is on the mailing list.

An FCIP device is a gateway between an FC SAN and IP network. Discovery of FCIP gateway (device) and other FCIP gateways is currently via static configuration.
Dynamic configuration support is envisioned, perhaps using iSNS.

David Black will work with authors on revising the QoS text.

-- iFCP - presented by Charles Monia, Nishan

What is the difference between iFCP and FCIP?
- FCIP is a tunneling model between FC SANs. A conduit for FC frames to flow transparently to FC network over IP backbone.
- iFCP network model extends up to the FC storage device itself. Uses a session model. Consolidates FC storage switching and routing functions in the IP fabric. Reduces total cost of ownership, unifies network and storage management domains and exploits IP technology investment. Extend SAN over lan/man/wan distances.

Next step -- complete the n_port session model. Encapsulation changes for additional end-to-end error detection.

The authors of iFCP would like to see it considered for adoption as a work group item. Adoption of iFCP as a work group item requires modification to the WG charter. David Black requested input on this be set to the WG chairs. Revising of the charter requires consultation of the area directors and working group chairs.

After the presentation, Suggestions were made that iFCP and FC/IP should merge since they are so similar. It was pointed out that the two protocols take different approaches. iFCP works by intercepting FC logins (connection requests) and modifying FC frames. In addition, it doesn't run FC routing protocols between FC SANs.

Clarification of FCIP and iFCP - the latter is for FCP protocol mapping only, whereas FCIP can transport any FC upper level protocol.

FC/IP works at a lower level than iFCP. It doesn't modify FC frames.

FC/IP requires running FC routing/switching protocols between FC domains.

Some thought that iFCP was a superset of FC/IP.

There was a concern that the iFCP gateway would need to run IP routing protocols. It was eventually decided the iFCP gateway was just an IP host and didn't have to run IP routing protocols.

Other comments need to be sent to mailing list or chairs directly.

-- Adaptation Layer presentation -- Randall Stewart, Cisco

Randall Stewart's presentation introduced how the IPS protocols could be architected with an adaptation layer independent of the underlying transport (i.e. at least both SCTP and TCP).

To do this, a uniform API boundary between the ULP and transport would need to be defined. This would require many changes to all existing drafts. APIs would need to be a message oriented type of mechanism. Critical path would need to be done so that they would be protocol agnostic.

Transport interface would need to provide methods for passing buffers to/from control of transport, e.g. for zero copy.

Adaptation layer would need to worry about
Zero copy
Parallel paths
Message retrieval

Must be very careful that this API would not make assumptions about the transport being used. In adaptation model, would need to figure out how to overcome the issues.

Randall would be more than glad to help by contributing both advice and/or drafts to bring about this sort of adaptation layer.

A concern was expressed that the adaptation layer would add too many layers between iSCSI and TCP and that separate protocol should be done for SCTP.

It was suggested that the CAM may be an inspiration for the adaptation layer.
Others responded that the CAM is at the wrong layer, above iSCSI.


IP Storage Framework Doc
TCP ULP Message Framing / iSCSI Framing
iSCSI Requirements
iSCSI - a SCSI over TCP Mapping
iSCSI Bootstrappimg Draft
iSCSI MIB Structure
iSCSI Naming & Discovery Team Status
iSNS Internet Storage Name Service
FC over TCP/IP
iFCP - A Protocol for Internet Fibre Channel Storage Networking
Abstracting Transport for IPS