< draft-ietf-storm-rdmap-ext-07.txt		draft-ietf-storm-rdmap-ext-08.txt >
	Storage Maintenance (storm) Working Group Hemal Shah		Storage Maintenance (storm) Working Group Hemal Shah
	Internet Draft Broadcom Corporation		Internet Draft Broadcom Corporation
	Intended status: Standards Track Felix Marti		Intended status: Standards Track Felix Marti
	Expires: March 2014 Wael Noureddine		Expires: April 2014 Wael Noureddine
	Asgeir Eiriksson		Asgeir Eiriksson
	Chelsio Communications, Inc.		Chelsio Communications, Inc.
	Robert Sharp		Robert Sharp
	Intel Corporation		Intel Corporation
	September 16, 2013		October 18, 2013
	RDMA Protocol Extensions		RDMA Protocol Extensions
	draft-ietf-storm-rdmap-ext-07.txt		draft-ietf-storm-rdmap-ext-08.txt
	Status of this Memo		Status of this Memo
	This Internet-Draft is submitted to IETF in full conformance with		This Internet-Draft is submitted to IETF in full conformance with
	the provisions of BCP 78 and BCP 79.		the provisions of BCP 78 and BCP 79.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at http://datatracker.ietf.org/drafts/current.		Drafts is at http://datatracker.ietf.org/drafts/current.
	Internet-Drafts are draft documents valid for a maximum of six		Internet-Drafts are draft documents valid for a maximum of six
	months and may be updated, replaced, or obsoleted by other documents		months and may be updated, replaced, or obsoleted by other documents
	at any time. It is inappropriate to use Internet-Drafts as		at any time. It is inappropriate to use Internet-Drafts as
	reference material or to cite them other than as "work in progress."		reference material or to cite them other than as "work in progress."
	This Internet-Draft will expire on March 16, 2014.		This Internet-Draft will expire on April 18, 2014.
	Copyright Notice		Copyright Notice
	Copyright (c) 2013 IETF Trust and the persons identified as the		Copyright (c) 2013 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(http://trustee.ietf.org/license-info) in effect on the date of		(http://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	skipping to change at page 3, line 41 ¶		skipping to change at page 3, line 41 ¶
	following an RDMA Write Message, the combination of the two		following an RDMA Write Message, the combination of the two
	messages is an implementation of RDMA Write with Immediate		messages is an implementation of RDMA Write with Immediate
	message that is found in other RDMA transport protocols.		message that is found in other RDMA transport protocols.
	Other RDMA transport protocols define the functionality added by		Other RDMA transport protocols define the functionality added by
	these extensions leading to differences in RDMA applications and/or		these extensions leading to differences in RDMA applications and/or
	Upper Layer Protocols. Removing these differences in the transport		Upper Layer Protocols. Removing these differences in the transport
	protocols simplifies these applications and ULPs and that is the		protocols simplifies these applications and ULPs and that is the
	main motivation for the extensions specified in this document.		main motivation for the extensions specified in this document.
	[RSOCKETS] is an example of RDMA enabled middleware that provides a		RSockets [RSOCKETS] is an example of RDMA enabled middleware that
	socket interface as the upper edge interface and utilizes RDMA to		provides a socket interface as the upper edge interface and utilizes
	provide more efficient networking for sockets based applications.		RDMA to provide more efficient networking for sockets based
	[RSOCKETS] is aware of Immediate Data support in [IB]. [RSOCKETS]		applications. RSockets [RSOCKETS] is aware of Immediate Data
	cannot utilize the RDMA Write with Immediate Data operation from		support in [IB]. [RSOCKETS] cannot utilize the RDMA Write with
	[IB] on iWARP. The addition of the Immediate Data operation		Immediate Data operation from [IB] on iWARP. The addition of the
	specified in this draft will alleviate this difference in [RSOCKETS]		Immediate Data operation specified in this draft will alleviate this
	when running on [IB] and iWARP.		difference in [RSOCKETS] when running on [IB] and iWARP.
	[DAT_ATOMICS] is an example of RDMA enable middleware that provides		DAT Atomics [DAT_ATOMICS] is an example of RDMA enable middleware
	a portable RDMA programming interface for various RDMA transport		that provides a portable RDMA programming interface for various RDMA
	protocols. [DAT_ATOMICS] includes a primitive for [IB] that is not		transport protocols. [DAT_ATOMICS] includes a primitive for [IB]
	supported by iWARP RNICs. The addition of Atomic Operations as		that is not supported by iWARP RNICs. The addition of Atomic
	specified in this draft will allow atomic operations in		Operations as specified in this draft will allow atomic operations
	[DAT_ATOMICS] to work for both [IB] and iWARP interchangeably.		in [DAT_ATOMICS] to work for both [IB] and iWARP interchangeably.
	1.1. Discovery of RDMAP Extensions		1.1. Discovery of RDMAP Extensions
	Today there are RDMA applications and/or ULPs that are aware of the		Today there are RDMA applications and/or ULPs that are aware of the
	existence of Atomic and Immediate data operations for RDMA		existence of Atomic and Immediate data operations for RDMA
	transports such as [IB] and application programming interfaces such		transports such as [IB] and application programming interfaces such
	as [OFA Verbs]. Today, these applications need to be aware that		as [OFAVERBS]. Today, these applications need to be aware that
	iWARP RNICs do not support these operations. Typically the		iWARP RNICs do not support these operations. Typically the
	availability of these capabilities are exposed to the applications		availability of these capabilities are exposed to the applications
	through adapter query interfaces in software. Applications then		through adapter query interfaces in software. Applications then
	have to decide to use or not to use Immediate Data or Atomic		have to decide to use or not to use Immediate Data or Atomic
	Operations based on the results of the query interfaces.		Operations based on the results of the query interfaces.
	Negotiation of Atomic Operations typically are to determine the		Negotiation of Atomic Operations typically are to determine the
	scope of atomicity guarantees, not down to the individual Atomic		scope of atomicity guarantees, not down to the individual Atomic
	Operations supported. Therefore, this specification requires all		Operations supported. Therefore, this specification requires all
	Atomic Operations defined within to be supported if an RNIC supports		Atomic Operations defined within to be supported if an RNIC supports
	any Atomic Operations.		any Atomic Operations.
	skipping to change at page 6, line 9 ¶		skipping to change at page 6, line 9 ¶
	send Immediate Data to the Data Sink		send Immediate Data to the Data Sink
	Immediate Data with Solicited Event (SE) Message - An RDMA Message		Immediate Data with Solicited Event (SE) Message - An RDMA Message
	used by the Data Source to send Immediate Data with Solicited Event		used by the Data Source to send Immediate Data with Solicited Event
	to the Data Sink		to the Data Sink
	Requester - the sender of an RDMA Atomic Operation request.		Requester - the sender of an RDMA Atomic Operation request.
	Responder - the receiver of an RDMA Atomic Operation request.		Responder - the receiver of an RDMA Atomic Operation request.
	ULP - Upper Layer Protocol. The protocol layer above the one		ULP - Upper Layer Protocol. The protocol layer above the one
	currently being referenced. The ULP for RDMAP[RFC5040]/DDP[RFC5041]		currently being referenced. The ULP for RDMAP [RFC5040] / DDP
	is expected to be an OS, Application, adaptation layer, or		[RFC5041] is expected to be an OS, Application, adaptation layer, or
	proprietary device. The RDMAP[RFC5040]/DDP[RFC5041] documents do not		proprietary device. The RDMAP [RFC5040] / DDP [RFC5041] documents do
	specify a ULP -- they provide a set of semantics that allow a ULP to		not specify a ULP -- they provide a set of semantics that allow a
	be designed to utilize RDMAP[RFC5040]/DDP[RFC5041].		ULP to be designed to utilize RDMAP [RFC5040] / DDP [RFC5041].
	4. Header Format Extensions		4. Header Format Extensions
	The control information of RDMA Messages is included in DDP protocol		The control information of RDMA Messages is included in DDP protocol
	[RFC5041] defined header fields. [RFC5040] defines the RDMAP header		[RFC5041] defined header fields. [RFC5040] defines the RDMAP header
	formats layered on the [RFC5041] DDP header definition. This		formats layered on the [RFC5041] DDP header definition. This
	specification extends [RFC5040] with the following new formats:		specification extends [RFC5040] with the following new formats:
	. Four new RDMA Messages carry additional RDMAP headers. The		. Four new RDMA Messages carry additional RDMAP headers. The
	Immediate Data operation and Immediate Data with Solicited Event		Immediate Data operation and Immediate Data with Solicited Event
	operation include 8 bytes of data following the RDMAP header.		operation include 8 bytes of data following the RDMAP header.
	skipping to change at page 8, line 18 ¶		skipping to change at page 8, line 18 ¶
	All RDMA Messages defined in this specification MUST have:		All RDMA Messages defined in this specification MUST have:
	The RDMA Version (RV) field: 01b.		The RDMA Version (RV) field: 01b.
	Opcode field: See Figure 2.		Opcode field: See Figure 2.
	Invalidate STag: MUST be set to zero by the sender, ignored by the		Invalidate STag: MUST be set to zero by the sender, ignored by the
	receiver.		receiver.
	RDMA | Message | Tagged| STag | Queue | Invalidate| Message		-------+-----------+-------+------+-------+---------+-------------
	Opcode | Type | Flag | and | Number| STag | Length		RDMA | Message | Tagged| STag | Queue | In- | Message
	| | | TO | | | Communicated		Opcode | Type | Flag | and | Number| validate| Length
	| | | | | | between DDP		| | | TO | | STag | Communicated
	| | | | | | and RDMAP		| | | | | | between DDP
	1000b | Immediate | 0 | N/A | 0 | N/A | Yes		| | | | | | and RDMAP
	| Data | | | | |		-------+-----------+-------+------+-------+---------+-------------
	1001b | Immediate | 0 | N/A | 0 | N/A | Yes		1000b | Immediate | 0 | N/A | 0 | N/A | Yes
	| Data with | | | | |		| Data | | | | |
	| SE | | | | |		-------+-----------+----------------------------------------------
	1010b | Atomic | 0 | N/A | 1 | N/A | Yes		1001b | Immediate | 0 | N/A | 0 | N/A | Yes
	| Request | | | | |		| Data with | | | | |
	1011b | Atomic | 0 | N/A | 3 | N/A | Yes		| SE | | | | |
	| Response | | | | |		-------+-----------+----------------------------------------------
			1010b | Atomic | 0 | N/A | 1 | N/A | Yes
			| Request | | | | |
			-------+-----------+----------------------------------------------
			1011b | Atomic | 0 | N/A | 3 | N/A | Yes
			| Response | | | | |
			-------+-----------+----------------------------------------------
	Figure 2 Additional RDMA Usage of DDP Fields		Figure 2 Additional RDMA Usage of DDP Fields
	Note: N/A means Not Applicable.		Note: N/A means Not Applicable.
	This extension defines RDMAP use of Queue Number 3 for Untagged		This extension defines RDMAP use of Queue Number 3 for Untagged
	Buffers for Atomic Responses. This queue is used for tracking		Buffers for Atomic Responses. This queue is used for tracking
	outstanding Atomic Requests.		outstanding Atomic Requests.
	All other DDP and RDMAP control fields MUST be set as described in		All other DDP and RDMAP control fields MUST be set as described in
	[RFC5040].		[RFC5040].
	4.2. RDMA Message Definitions		4.2. RDMA Message Definitions
	The following figure defines which RDMA Headers MUST be used on each		The following figure defines which RDMA Headers MUST be used on each
	new RDMA Message and which new RDMA Messages are allowed to carry		new RDMA Message and which new RDMA Messages are allowed to carry
	ULP payload:		ULP payload:
	RDMA | Message | RDMA Header Used | ULP Message allowed in		-------+-----------+-------------------+-------------------------
	Message| Type | | the RDMA Message		RDMA | Message | RDMA Header Used | ULP Message allowed in
	OpCode | | |		Message| Type | | the RDMA Message
	| | |		OpCode | | |
	1000b | Immediate | Immediate Data | No		| | |
	| Data | Header |		-------+-----------+-------------------+-------------------------
	1001b | Immediate | Immediate Data | No		1000b | Immediate | Immediate Data | No
	| Data with | Header |		| Data | Header |
	| SE | |		-------+-----------+-------------------+-------------------------
	1010b | Atomic | Atomic Request | No		1001b | Immediate | Immediate Data | No
	| Request | Header |		| Data with | Header |
	1011b | Atomic | Atomic Response | No		| SE | |
	| Response | Header |		-------+-----------+-------------------+-------------------------
			1010b | Atomic | Atomic Request | No
			| Request | Header |
			-------+-----------+-------------------+-------------------------
			1011b | Atomic | Atomic Response | No
			| Response | Header |
			-------+-----------+-------------------+-------------------------
	Figure 3 RDMA Message Definitions		Figure 3 RDMA Message Definitions
	5. Atomic Operations		5. Atomic Operations
	The RDMA Protocol Specification in [RFC5040] does not include		The RDMA Protocol Specification in [RFC5040] does not include
	support for Atomic Operations which are an important building block		support for Atomic Operations which are an important building block
	for implementing distributed shared memory.		for implementing distributed shared memory.
	This document extends the RDMA Protocol specification with a set of		This document extends the RDMA Protocol specification with a set of
	basic Atomic Operations, and specifies their resource and ordering		basic Atomic Operations, and specifies their resource and ordering
	rules. The Atomic Operations specified in this document provide		rules. The Atomic Operations specified in this document provide
	equivalent functionality to the [IB] RDMA transport as well as		equivalent functionality to the [IB] RDMA transport as well as
	extended Atomic Operations defined in [OFA Verbs], to allow		extended Atomic Operations defined in [OFAVERBS], to allow
	applications that use these primitives to work interchangeably over		applications that use these primitives to work interchangeably over
	iWARP. Other operations are left for future consideration.		iWARP. Other operations are left for future consideration.
	Atomic operations as specified in this document execute a 64-bit		Atomic operations as specified in this document execute a 64-bit
	memory operation at a specified destination ULP Buffer address on a		memory operation at a specified destination ULP Buffer address on a
	Responder node using the Tagged Buffer data transfer model. The		Responder node using the Tagged Buffer data transfer model. The
	operations atomically read, modify and write back the contents of		operations atomically read, modify and write back the contents of
	the destination ULP Buffer address and guarantee that Atomic		the destination ULP Buffer address and guarantee that Atomic
	Operations on this ULP Buffer address by other RDMAP Streams on the		Operations on this ULP Buffer address by other RDMAP Streams on the
	same RNIC do not occur between the read and the write caused by the		same RNIC do not occur between the read and the write caused by the
	Atomic Operation. Therefore, the Responder RNIC MUST implement		Atomic Operation. Therefore, the Responder RNIC MUST implement
	mechanisms to prevent Atomic Operations to a memory registered for		mechanisms to prevent Atomic Operations to a memory registered for
	Atomic Operations while an Atomic Operation targeting the memory is		Atomic Operations while an Atomic Operation targeting the memory is
	in progress. The Requester of an atomic operation cannot rely on		in progress. The Requester of an atomic operation cannot rely on
	atomic operation behavior at the Responder across multiple RNICs or		atomic operation behavior at the Responder across multiple RNICs or
	with respect to other applications/ULPs running at the Responder		with respect to other applications/ULPs running at the Responder
	that can access the ULP Buffer. Some RNIC implementations may		that can access the ULP Buffer. Some RNIC implementations may
	provide such atomic behavior, but it is OPTIONAL for the atomic		provide such atomic behavior, but it is OPTIONAL for the atomic
	operations specified in this document.. An RNIC that supports		operations specified in this document. An RNIC that supports Atomic
	Atomic Operations as specified in this document MUST implement all		Operations as specified in this document MUST implement all Atomic
	Atomic Operation Codes defined in Figure 5. The advertisement of		Operation Codes defined in Figure 5. The advertisement of Tagged
	Tagged Buffer information for Atomic Operations is outside the scope		Buffer information for Atomic Operations is outside the scope of
	of this specification and must be handled by the ULPs.		this specification and must be handled by the ULPs.
	Implementation note: It is recommended that the applications do not		Implementation note: It is recommended that the applications do not
	use the ULP Buffer addresses used for Atomic Operations for other		use the ULP Buffer addresses used for Atomic Operations for other
	RDMA operations.		RDMA operations.
	Implementation note: Errors related to the alignment in the		Implementation note: Errors related to the alignment in the
	following sections cover Atomic Operations targeted at a ULP Buffer		following sections cover Atomic Operations targeted at a ULP Buffer
	address that is not aligned to a 64-bit boundary.		address that is not aligned to a 64-bit boundary.
	Atomic Operation Request Messages use the same remote addressing		Atomic Operation Request Messages use the same remote addressing
	skipping to change at page 12, line 6 ¶		skipping to change at page 12, line 6 ¶
	{		{
	if (bit != 0 ) bit_location = bit_location << 1		if (bit != 0 ) bit_location = bit_location << 1
	val1 = (Original Remote Data Value & bit_location) >> bit		val1 = (Original Remote Data Value & bit_location) >> bit
	val2 = (Add Data & bit_location) >> bit		val2 = (Add Data & bit_location) >> bit
	sum = carry + val1 + val2		sum = carry + val1 + val2
	carry = !(!(sum & 2))		carry = (sum & 2) >> 1
	sum = sum & 1		sum = sum & 1
	if (sum)		if (sum)
	Remote Data Value |= bit_location		Remote Data Value |= bit_location
	carry = ((carry) && (!(Add Mask & bit_location)))		carry = ((carry) && (!(Add Mask & bit_location)))
	}		}
	skipping to change at page 21, line 7 ¶		skipping to change at page 21, line 7 ¶
	8. If more than one outstanding Atomic Request Messages are		8. If more than one outstanding Atomic Request Messages are
	supported by both peers, the Atomic Operation Request Messages		supported by both peers, the Atomic Operation Request Messages
	MUST be processed in the order they were delivered by the DDP		MUST be processed in the order they were delivered by the DDP
	layer on the Responder. Atomic Operation Response Messages MUST		layer on the Responder. Atomic Operation Response Messages MUST
	be submitted to the DDP layer on the Responder in the order the		be submitted to the DDP layer on the Responder in the order the
	Atomic Operation Request Messages were Delivered by DDP.		Atomic Operation Request Messages were Delivered by DDP.
	6. Immediate Data		6. Immediate Data
	The Immediate Data operation is typically used in conjunction with an		The Immediate Data operation is typically used in conjunction with
	RDMA Write Operation to improve ULP processing efficiency. The		an RDMA Write Operation to improve ULP processing efficiency. The
	efficiency is gained by causing an RDMA Completion to be generated		efficiency is gained by causing an RDMA Completion to be generated
	immediately following the RDMA Write operation. This RDMA Completion		immediately following the RDMA Write operation. This RDMA Completion
	delivers 8 bytes of immediate data at the Remote Peer. The		delivers 8 bytes of immediate data at the Remote Peer. The
	combination of an RDMA Write Message followed by an Immediate Data		combination of an RDMA Write Message followed by an Immediate Data
	Operation has the same behavior as the RDMA Write with Immediate Data		Operation has the same behavior as the RDMA Write with Immediate Data
	operation found in [IB]. An Immediate Data operation that is not		operation found in [IB]. An Immediate Data operation that is not
	preceded by an RDMA Write operation causes an RDMA Completion.		preceded by an RDMA Write operation causes an RDMA Completion.
	6.1. RDMAP Interactions with ULP for Immediate Data		6.1. RDMAP Interactions with ULP for Immediate Data
	skipping to change at page 23, line 28 ¶		skipping to change at page 23, line 28 ¶
	layer. The DDP header carries the length field that is reported		layer. The DDP header carries the length field that is reported
	by the DDP layer.		by the DDP layer.
	6.4. Ordering and Completions		6.4. Ordering and Completions
	Ordering and completion rules for Immediate Data are the same as		Ordering and completion rules for Immediate Data are the same as
	those for a Send operation as described in section 5.5 of RFC 5040.		those for a Send operation as described in section 5.5 of RFC 5040.
	7. Ordering and Completions Table		7. Ordering and Completions Table
	The following table summarizes the ordering relationships for Atomic		The following table summarizes the ordering relationships for Atomic
	and Immediate Data operations from the standpoint of Local Peer issuing		and Immediate Data operations from the standpoint of Local Peer
	the Operations. Note that in the table that follows, Send includes		issuing the Operations. Note that in the table that follows, Send
	Send, Send with Invalidate, Send with Solicited Event, and Send with		includes Send, Send with Invalidate, Send with Solicited Event, and
	Solicited Event and Invalidate. Also note that in the table below,		Send with Solicited Event and Invalidate. Also note that in the
	Immediate Data includes Immediate Data and Immediate Data with		table below, Immediate Data includes Immediate Data and Immediate
	Solicited Event.		Data with Solicited Event.
	First | Second | Placement | Placement | Ordering		---------+----------+-------------+-------------+------------------
	Operation | Operation | Guarantee at| Guarantee at| Guarantee at		First | Second | Placement | Placement | Ordering
	| | Remote Peer | Local Peer | Remote Peer		Operation| Operation| Guarantee at| Guarantee at| Guarantee at
	Immediate | Send | No Placement| Not | Completed in		| | Remote Peer | Local Peer | Remote Peer
	Data | | Guarantee | Applicable | Order		---------+----------+-------------+-------------+------------------
	| | between Send| |		Immediate| Send | No Placement| Not | Completed in
	| | Payload and | |		Data | | Guarantee | Applicable | Order
	| | Immediate | |		| | between Send| |
	| | Data | |		| | Payload and | |
	Immediate | RDMA | No Placement| Not | Not		| | Immediate | |
	Data | Write | Guarantee | Applicable | Applicable		| | Data | |
	| | between RDMA| |		---------+----------+-------------+-------------+------------------
	| | Write | |		Immediate| RDMA | No Placement| Not | Not
	| | Payload and | |		Data | Write | Guarantee | Applicable | Applicable
	| | Immediate | |		| | between RDMA| |
	| | Data | |		| | Write | |
	Immediate | RDMA | No Placement| RDMA Read | RDMA Read		| | Payload and | |
	Data | Read | Guarantee | Response | Response		| | Immediate | |
	| | between | will not be | Message will		| | Data | |
	| | Immediate | Placed until| not be		---------+----------+-------------+-------------+------------------
	| | Data and | Immediate | generated		Immediate| RDMA | No Placement| RDMA Read | RDMA Read
	| | RDMA Read | Data is | until		Data | Read | Guarantee | Response | Response
	| | Request | Placed at | Immediate Data		| | between | will not be | Message will
	| | | Remote Peer | has been		| | Immediate | Placed until| not be
	| | | | Completed		| | Data and | Immediate | generated
	Immediate | Atomic | No Placement| Atomic | Atomic		| | RDMA Read | Data is | until
	Data | | Guarantee | Response | Response		| | Request | Placed at | Immediate Data
	| | between | will not be | Message will		| | | Remote Peer | has been
	| | Immediate | Placed until| not be		| | | | Completed
	| | Data and | Immediate | generated		---------+----------+-------------+-------------+------------------
	| | Atomic | Data is | until		Immediate| Atomic | No Placement| Atomic | Atomic
	| | Request | Placed at | Immediate Data		Data | | Guarantee | Response | Response
	| | | Remote Peer | has been		| | between | will not be | Message will
	| | | | Completed		| | Immediate | Placed until| not be
	Immediate | Immediate | No Placement| Not | Completed in		| | Data and | Immediate | generated
	Data or | Data | Guarantee | Applicable | Order		| | Atomic | Data is | until
	Send | | | |		| | Request | Placed at | Immediate Data
	RDMA Write| Immediate | No Placement| Not | Immediate Data		| | | Remote Peer | has been
	| Data | Guarantee | Applicable | is Completed		| | | | Completed
	| | | | after RDMA		---------+----------+-------------+-------------+------------------
	| | | | Write is Placed		Immediate| Immediate| No Placement| Not | Completed in
	| | | | and Delivered		Data or | Data | Guarantee | Applicable | Order
	RDMA Read | Immediate | No Placement| Immediate | Not Applicable		Send | | | |
	| Data | Guarantee | Data MAY be |		---------+----------+-------------+-------------+------------------
	| | between | Placed |		RDMA | Immediate| No Placement| Not | Immediate Data
	| | Immediate | before |		Write | Data | Guarantee | Applicable | is Completed
	| | Data and | RDMA Read |		| | | | after RDMA
	| | RDMA Read | Response is |		| | | | Write is Placed
	| | Request | generated |		| | | | and Delivered
			---------+----------+-------------+-------------+------------------
			RDMA Read| Immediate| No Placement| Immediate | Not Applicable
			| Data | Guarantee | Data MAY be |
			| | between | Placed |
			| | Immediate | before |
			| | Data and | RDMA Read |
			| | RDMA Read | Response is |
			| | Request | generated |
	Atomic | Immediate | No Placement| Immediate | Not Applicable		---------+----------+-------------+-------------+------------------
	| Data | Guarantee | Data MAY be |		Atomic | Immediate| No Placement| Immediate | Not Applicable
	| | between | Placed |		| Data | Guarantee | Data MAY be |
	| | Immediate | before |		| | between | Placed |
	| | Data and | Atomic |		| | Immediate | before |
	| | Atomic | Response is |		| | Data and | Atomic |
	| | Request | generated |		| | Atomic | Response is |
	Atomic | Send | No Placement| Send Payload| Not Applicable		| | Request | generated |
	| | Guarantee | MAY be |		---------+----------+-------------+-------------+------------------
	| | between Send| Placed |		Atomic | Send | No Placement| Send Payload| Not Applicable
	| | Payload and | before |		| | Guarantee | MAY be |
	| | Atomic | Atomic |		| | between Send| Placed |
	| | Request | Response is |		| | Payload and | before |
	| | | generated |		| | Atomic | Atomic |
	Atomic | RDMA | No Placement| RDMA Write | Not		| | Request | Response is |
	| Write | Guarantee | Payload MAY | Applicable		| | | generated |
	| | between RDMA| be Placed |		---------+----------+-------------+-------------+------------------
	| | Write | before |		Atomic | RDMA | No Placement| RDMA Write | Not
	| | Payload and | Atomic |		| Write | Guarantee | Payload MAY | Applicable
	| | Atomic | Response is |		| | between RDMA| be Placed |
	| | Request | generated |		| | Write | before |
	Atomic | RDMA | No Placement| No Placement| RDMA Read		| | Payload and | Atomic |
	| Read | Guarantee | Guarantee | Response		| | Atomic | Response is |
	| | between | between | Message will		| | Request | generated |
	| | Atomic | Atomic | not be		---------+----------+-------------+-------------+------------------
	| | Request and | Response | generated		Atomic | RDMA | No Placement| No Placement| RDMA Read
	| | RDMA Read | and RDMA | until Atomic		| Read | Guarantee | Guarantee | Response
	| | Request | Read | Response Message		| | between | between | Message will
	| | | Response | has been		| | Atomic | Atomic | not be
	| | | | generated		| | Request and | Response | generated
	Atomic | Atomic | Placed in | No Placement| Second Atomic		| | RDMA Read | and RDMA | until Atomic
	| | order | Guarantee | Request		| | Request | Read | Response Message
	| | | between two | Message will		| | | Response | has been
	| | | Atomic | not be		| | | | generated
	| | | Responses | processed		---------+----------+-------------+-------------+------------------
	| | | | until first		Atomic | Atomic | Placed in | No Placement| Second Atomic
	| | | | Atomic Response		| | order | Guarantee | Request
	| | | | has been		| | | between two | Message will
	| | | | generated		| | | Atomic | not be
	Send | Atomic | No Placement| Atomic | Atomic Response		| | | Responses | processed
	| | Guarantee | Response | Message will not		| | | | until first
	| | between Send| will not be | be generated until		| | | | Atomic Response
	| | Payload and | Placed at | Send has been		| | | | has been
	| | Atomic | the Local | Completed		| | | | generated
	| | Request | Peer Until |		---------+----------+-------------+-------------+------------------
	| | | Send Payload|		Send | Atomic | No Placement| Atomic | Atomic Response
	| | | is Placed |		| | Guarantee | Response | Message will not
	| | | at the |		| | between Send| will not be | be generated
	| | | Remote Peer |		| | Payload and | Placed at | until Send has
	RDMA | Atomic | No Placement| Atomic | Not		| | Atomic | the Local | been Completed
	Write | | Guarantee | Response | Applicable		| | Request | Peer Until |
	| | between RDMA| will not be |		| | | Send Payload|
	| | Write | Placed at |		| | | is Placed |
	| | Payload and | the Local |		| | | at the |
	| | Atomic | Peer Until |		| | | Remote Peer |
	| | Request | RDMA Write |		---------+----------+-------------+-------------+------------------
	| | | Payload |		RDMA | Atomic | No Placement| Atomic | Not
	| | | is Placed |		Write | | Guarantee | Response | Applicable
	| | | at the |		| | between RDMA| will not be |
	| | | Remote Peer |		| | Write | Placed at |
	RDMA | Atomic | No Placement| No Placement| Atomic Response		| | Payload and | the Local |
	Read | | Guarantee | Guarantee | Message will		| | Atomic | Peer Until |
	| | between | between | not be generated		| | Request | RDMA Write |
	| | Atomic | Atomic | until RDMA		| | | Payload |
	| | Request and | Response | Read Response		| | | is Placed |
	| | RDMA Read | and RDMA | has been		| | | at the |
	| | Request | Read | generated		| | | Remote Peer |
	| | | Response |		---------+----------+-------------+-------------+------------------
			RDMA | Atomic | No Placement| No Placement| Atomic Response
			Read | | Guarantee | Guarantee | Message will
			| | between | between | not be generated
			| | Atomic | Atomic | until RDMA
			| | Request and | Response | Read Response
			| | RDMA Read | and RDMA | has been
			| | Request | Read | generated
			| | | Response |
			---------+----------+-------------+-------------+------------------
	8. Error Processing		8. Error Processing
	In addition to error processing described in section 7 of [RFC5040],		In addition to error processing described in section 7 of [RFC5040],
	the following rules apply for the new RDMA Messages defined in this		the following rules apply for the new RDMA Messages defined in this
	specification.		specification.
	8.1. Errors Detected at the Local Peer		8.1. Errors Detected at the Local Peer
	The Local Peer MUST send a Terminate Message for each of the		The Local Peer MUST send a Terminate Message for each of the
	skipping to change at page 28, line 32 ¶		skipping to change at page 28, line 32 ¶
	0x9, Immediate Data with Solicited Event, [RFCXXXX]		0x9, Immediate Data with Solicited Event, [RFCXXXX]
	0xA, Atomic Request, [RFCXXXX]		0xA, Atomic Request, [RFCXXXX]
	0xB, Atomic Response, [RFCXXXX]		0xB, Atomic Response, [RFCXXXX]
	In addition, the following registry is requested to be added to		In addition, the following registry is requested to be added to
	"RDDP Registries". The following section specifies the registry, its		"RDDP Registries". The following section specifies the registry, its
	initial contents and the administration policy in more detail.		initial contents and the administration policy in more detail.
			RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above
			with the RFC number of this document and remove this note.
	10.1. RDMAP Message Atomic Operation Subcodes		10.1. RDMAP Message Atomic Operation Subcodes
	Name of the registry: "RDMAP Message Atomic Operation Subcodes"		Name of the registry: "RDMAP Message Atomic Operation Subcodes"
	Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit		Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit
	values [RFCXXXX].		values [RFCXXXX].
	Information that must be provided to assign a new value: An IESG-		Information that must be provided to assign a new value: An IESG-
	approved standards-track specification defining the semantics and		approved standards-track specification defining the semantics and
	interoperability requirements of the proposed new value and the		interoperability requirements of the proposed new value and the
	skipping to change at page 30, line 30 ¶		skipping to change at page 30, line 36 ¶
	[RFC5041] Shah, H. et al., "Direct Data Placement over Reliable		[RFC5041] Shah, H. et al., "Direct Data Placement over Reliable
	Transports", RFC 5041, October 2007.		Transports", RFC 5041, October 2007.
	[RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement		[RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement
	Protocol (DDP) / Remote Direct Memory Access Protocol		Protocol (DDP) / Remote Direct Memory Access Protocol
	(RDMAP) Security", October 2007.		(RDMAP) Security", October 2007.
	[RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an		[RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an
	IANA Considerations Section in RFCs", May 2008.		IANA Considerations Section in RFCs", May 2008.
			RFC Editor: Please remove reference to RFC5226 if the associated
			IANA Considerations reference is also removed before publication.
	11.2. Informative References		11.2. Informative References
	[IB] Infiniband Trade Association, "Infiniband Architecture		[IB] Infiniband Trade Association, "Infiniband Architecture
	Specification Volumes 1 and 2", Release 1.1, November		Specification Volumes 1 and 2", Release 1.1, November
	2002, available from http://www.infinibandta.org/specs.		2002, available from http://www.infinibandta.org/specs.
	[RSOCKETS] RSockets, RDMA enabled Sockets library for Open Fabrics,		[RSOCKETS] RSockets, RDMA enabled Sockets library for Open Fabrics,
	available from		available from
	http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=b		http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=b
	lob;f=src/rsocket.c;h=d544dd097cda228de114173c8fe569dc1881		lob;f=src/rsocket.c;h=d544dd097cda228de114173c8fe569dc1881
	f057;hb=HEAD.		f057;hb=HEAD.
	[OFA Verbs] Open Fabrics Alliance Verbs Enhanced Atomic Operations,		[OFAVERBS] Open Fabrics Alliance Verbs Enhanced Atomic Operations,
	"[PATCH 0/2] Add support for enhanced atomic operations",		"[PATCH 0/2] Add support for enhanced atomic operations",
	available from		available from
	http://comments.gmane.org/gmane.linux.drivers.rdma/2397.		http://comments.gmane.org/gmane.linux.drivers.rdma/2397.
	[DAT_ATOMICS] DAT Collaborative, User Direct Access Programming		[DAT_ATOMICS] DAT Collaborative, User Direct Access Programming
	Library, "Ratified DAT IB extension spec", available from		Library, "Ratified DAT IB extension spec", available from
	http://www.datcollaborative.org/DAT_IB_Extensions.pdf.		http://www.datcollaborative.org/DAT_IB_Extensions.pdf.
	12. Acknowledgments		12. Acknowledgments
End of changes. 20 change blocks.
	185 lines changed or deleted		220 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/