< draft-ietf-storm-rdmap-ext-06.txt   draft-ietf-storm-rdmap-ext-07.txt >
Storage Maintenance (storm) Working Group Hemal Shah Storage Maintenance (storm) Working Group Hemal Shah
Internet Draft Broadcom Corporation Internet Draft Broadcom Corporation
Intended status: Standards Track Felix Marti Intended status: Standards Track Felix Marti
Expires: March 2014 Wael Noureddine Expires: March 2014 Wael Noureddine
Asgeir Eiriksson Asgeir Eiriksson
Chelsio Communications, Inc. Chelsio Communications, Inc.
Robert Sharp Robert Sharp
Intel Corporation Intel Corporation
September 10, 2013 September 16, 2013
RDMA Protocol Extensions RDMA Protocol Extensions
draft-ietf-storm-rdmap-ext-06.txt draft-ietf-storm-rdmap-ext-07.txt
Status of this Memo Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with This Internet-Draft is submitted to IETF in full conformance with
the provisions of BCP 78 and BCP 79. the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current. Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 10, 2014. This Internet-Draft will expire on March 16, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 10, line 17 skipping to change at page 10, line 17
Atomic operations as specified in this document execute a 64-bit Atomic operations as specified in this document execute a 64-bit
memory operation at a specified destination ULP Buffer address on a memory operation at a specified destination ULP Buffer address on a
Responder node using the Tagged Buffer data transfer model. The Responder node using the Tagged Buffer data transfer model. The
operations atomically read, modify and write back the contents of operations atomically read, modify and write back the contents of
the destination ULP Buffer address and guarantee that Atomic the destination ULP Buffer address and guarantee that Atomic
Operations on this ULP Buffer address by other RDMAP Streams on the Operations on this ULP Buffer address by other RDMAP Streams on the
same RNIC do not occur between the read and the write caused by the same RNIC do not occur between the read and the write caused by the
Atomic Operation. Therefore, the Responder RNIC MUST implement Atomic Operation. Therefore, the Responder RNIC MUST implement
mechanisms to prevent Atomic Operations to a memory registered for mechanisms to prevent Atomic Operations to a memory registered for
Atomic Operations while an Atomic Operation targeting the memory is Atomic Operations while an Atomic Operation targeting the memory is
in progress. Atomicity guarantees between multiple RNICs or between in progress. The Requester of an atomic operation cannot rely on
RNICs and software running independent of the RNIC is outside the atomic operation behavior at the Responder across multiple RNICs or
scope of this specification. An RNIC that supports Atomic with respect to other applications/ULPs running at the Responder
Operations as specified in this document MUST implement all Atomic that can access the ULP Buffer. Some RNIC implementations may
Operation Codes defined in Figure 5. The advertisement of Tagged provide such atomic behavior, but it is OPTIONAL for the atomic
Buffer information for Atomic Operations is outside the scope of operations specified in this document.. An RNIC that supports
this specification and must be handled by the ULPs. Atomic Operations as specified in this document MUST implement all
Atomic Operation Codes defined in Figure 5. The advertisement of
Tagged Buffer information for Atomic Operations is outside the scope
of this specification and must be handled by the ULPs.
Implementation note: It is recommended that the applications do not Implementation note: It is recommended that the applications do not
use the ULP Buffer addresses used for Atomic Operations for other use the ULP Buffer addresses used for Atomic Operations for other
RDMA operations. RDMA operations.
Implementation note: Errors related to the alignment in the Implementation note: Errors related to the alignment in the
following sections cover Atomic Operations targeted at a ULP Buffer following sections cover Atomic Operations targeted at a ULP Buffer
address that is not aligned to a 64-bit boundary. address that is not aligned to a 64-bit boundary.
Atomic Operation Request Messages use the same remote addressing Atomic Operation Request Messages use the same remote addressing
skipping to change at page 14, line 5 skipping to change at page 14, line 5
else else
Remote Data Value = Original Remote Data Value Remote Data Value = Original Remote Data Value
After the operation, the remote data buffer MUST contain the After the operation, the remote data buffer MUST contain the
"Original Remote Data Value" (if comparison did not match) or the "Original Remote Data Value" (if comparison did not match) or the
masked "Swap Data" (if the comparison did match). CmpSwap Atomic masked "Swap Data" (if the comparison did match). CmpSwap Atomic
Operations MUST target buffer addresses that are 64-bit aligned. If Operations MUST target buffer addresses that are 64-bit aligned. If
a CmpSwap Atomic Operation is attempted on a target ULP Buffer a CmpSwap Atomic Operation is attempted on a target ULP Buffer
addresse that is not 64-bit aligned: address that is not 64-bit aligned:
o The operation MUST NOT be performed, o The operation MUST NOT be performed,
o The Responder's memory MUST NOT be modified, o The Responder's memory MUST NOT be modified,
o The result MUST be surfaced as an error, and o The result MUST be surfaced as an error, and
o A terminate message MUST be generated (see Section 8.2. for the o A terminate message MUST be generated (see Section 8.2. for the
terminate message contents) terminate message contents)
skipping to change at page 15, line 8 skipping to change at page 15, line 8
The Atomic Operation Request Message carries an Atomic Operation The Atomic Operation Request Message carries an Atomic Operation
Header that describes the ULP Buffer address in the Responder's Header that describes the ULP Buffer address in the Responder's
memory. The Atomic Operation Request header immediately follows the memory. The Atomic Operation Request header immediately follows the
DDP header. The RDMAP layer passes to the DDP layer a RDMAP Control DDP header. The RDMAP layer passes to the DDP layer a RDMAP Control
Field. The following figure depicts the Atomic Operation Request Field. The following figure depicts the Atomic Operation Request
Header that MUST be used for all Atomic Operation Request Messages: Header that MUST be used for all Atomic Operation Request Messages:
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Un-used (Not Used) |AOpCode| | Reserved (Not Used) |AOpCode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Request Identifier | | Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote STag | | Remote STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote Tagged Offset | | Remote Tagged Offset |
+ + + +
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Data | | Add or Swap Data |
skipping to change at page 15, line 37 skipping to change at page 15, line 37
+ + + +
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compare Mask | | Compare Mask |
+ + + +
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4 Atomic Operation Request Header Figure 4 Atomic Operation Request Header
Un-used (Not Used): 28 bits Reserved (Not Used): 28 bits
This field MUST be set to zero on transmit, ignored on This field MUST be set to zero on transmit, ignored on
receive. receive.
Atomic Operation Code (AOpCode): 4 bits. Atomic Operation Code (AOpCode): 4 bits.
See Figure 5. All Atomic Operation Codes from Figure 5 MUST See Figure 5. All Atomic Operation Codes from Figure 5 MUST
be implemented by an RNIC that support Atomic Operations. be implemented by an RNIC that support Atomic Operations.
Request Identifier: 32 bits. Request Identifier: 32 bits.
skipping to change at page 22, line 25 skipping to change at page 22, line 25
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data | | Immediate Data |
+ + + +
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7 Immediate Data or Immediate Data with SE Message Header Figure 7 Immediate Data or Immediate Data with SE Message Header
Immediate Data: 64 bits. Immediate Data: 64 bits.
Eight bytes of data transferred from the Requester to an Eight bytes of data transferred from the Data Source to an
untagged buffer at the Responder. untagged buffer at the Data Sink.
6.3. Immediate Data or Immediate Data with SE Message 6.3. Immediate Data or Immediate Data with SE Message
The Immediate Data or Immediate Data with SE Message uses the DDP The Immediate Data or Immediate Data with SE Message uses the DDP
Untagged Buffer Model to transfer Immediate Data from the Data Untagged Buffer Model to transfer Immediate Data from the Data
Source to the Data Sink. Source to the Data Sink.
. An Immediate Data or Immediate Data with SE Message MUST . An Immediate Data or Immediate Data with SE Message MUST
reference an Untagged Buffer. That is, the Local Peer's RDMAP reference an Untagged Buffer. That is, the Local Peer's RDMAP
Layer MUST request that the DDP layer mark the Message as Layer MUST request that the DDP layer mark the Message as
Untagged. Untagged.
skipping to change at page 30, line 27 skipping to change at page 30, line 27
[RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol [RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007. Specification", RFC 5040, October 2007.
[RFC5041] Shah, H. et al., "Direct Data Placement over Reliable [RFC5041] Shah, H. et al., "Direct Data Placement over Reliable
Transports", RFC 5041, October 2007. Transports", RFC 5041, October 2007.
[RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement
Protocol (DDP) / Remote Direct Memory Access Protocol Protocol (DDP) / Remote Direct Memory Access Protocol
(RDMAP) Security", October 2007. (RDMAP) Security", October 2007.
[RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", May 2008.
11.2. Informative References 11.2. Informative References
[IB] Infiniband Trade Association, "Infiniband Architecture [IB] Infiniband Trade Association, "Infiniband Architecture
Specification Volumes 1 and 2", Release 1.1, November Specification Volumes 1 and 2", Release 1.1, November
2002, available from http://www.infinibandta.org/specs. 2002, available from http://www.infinibandta.org/specs.
[RSOCKETS] RSockets, RDMA enabled Sockets library for Open Fabrics, [RSOCKETS] RSockets, RDMA enabled Sockets library for Open Fabrics,
available from available from
http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=b http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=b
lob;f=src/rsocket.c;h=d544dd097cda228de114173c8fe569dc1881 lob;f=src/rsocket.c;h=d544dd097cda228de114173c8fe569dc1881
skipping to change at page 33, line 10 skipping to change at page 33, line 10
A.1. DDP Segment for Atomic Operation Request A.1. DDP Segment for Atomic Operation Request
The following figure depicts an Atomic Operation Request, DDP The following figure depicts an Atomic Operation Request, DDP
Segment: Segment:
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control | | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Un-used (Not Used) | | Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Queue Number | | DDP (Atomic Operation Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Sequence Number | | DDP (Atomic Operation Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Offset | | DDP (Atomic Operation Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Un-used (Not Used) |AOpCode| | Reserved (Not Used) |AOpCode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Request Identifier | | Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote STag | | Remote STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote Tagged Offset | | Remote Tagged Offset |
+ + + +
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Data | | Add or Swap Data |
skipping to change at page 34, line 15 skipping to change at page 34, line 15
A.2. DDP Segment for Atomic Response A.2. DDP Segment for Atomic Response
The following figure depicts an Atomic Operation Response, DDP The following figure depicts an Atomic Operation Response, DDP
Segment: Segment:
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control | | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Un-used (Not Used) | | Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Queue Number | | DDP (Atomic Operation Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Sequence Number | | DDP (Atomic Operation Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Offset | | DDP (Atomic Operation Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Request Identifier | | Original Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Remote Value | | Original Remote Value |
skipping to change at page 35, line 10 skipping to change at page 35, line 10
A.3. DDP Segment for Immediate Data and Immediate Data with SE A.3. DDP Segment for Immediate Data and Immediate Data with SE
The following figure depicts an Immediate Data or Immediate data The following figure depicts an Immediate Data or Immediate data
with SE, DDP Segment: with SE, DDP Segment:
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control | | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Un-used (Not Used) | | Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Send) Queue Number | | DDP (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Send) Message Sequence Number | | DDP (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Message Offset | | DDP Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data | | Immediate Data |
+ + + +
| | | |
 End of changes. 13 change blocks. 
19 lines changed or deleted 25 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/