From nobody Sun Sep 6 02:59:05 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 49D901B589E for ; Sun, 6 Sep 2015 02:59:04 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=5 tests=[BAYES_20=-0.001, HTML_MESSAGE=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BY38X0MwmPWO for ; Sun, 6 Sep 2015 02:59:01 -0700 (PDT) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0604.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe04::604]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 68B5E1B589C for ; Sun, 6 Sep 2015 02:59:00 -0700 (PDT) Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0651.eurprd02.prod.outlook.com (10.161.118.11) with Microsoft SMTP Server (TLS) id 15.1.262.15; Sun, 6 Sep 2015 09:58:42 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Sun, 6 Sep 2015 09:58:42 +0000 From: Elena Gurevich To: "storm@ietf.org" Thread-Topic: Send vs. Immediate Thread-Index: AdDoie0+xkvsjYh7T92FPZdqNGMGkA== Date: Sun, 6 Sep 2015 09:58:42 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0651; 5:Gzypdr0zBp6N9Q4eYbvG2YH89S5sxVT4EBflDXapNKmJsKnQmdmJ+pfyCOC9Vcw3VlwMD+1QJUoMPrLtUq3Tz9MD9i2WVGn8qAD+zVlUQX71bl4TLGm04UVUEB+2Q9IdJ5G6vC/aZjmg3Lg9syXVQg==; 24:SDFaAHzR1CQpJd/G5QBXz52zRsfhDsnc/rlf9g4ghV5XbGHes/b8x+diRXUg9qJs5FwYmSBjB0i/F7Kyp11PCvroVoxBBO584X53GYc6F0w=; 20:fb+iWTkc+om752dSWUAX2tHLd4fKi+oK5rT21DT/XDbEbz2iPdMOpurioQko5W4BRbDVquyJULya2+SRosGLHQ== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0651; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(108003899814671); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(8121501046)(3002001); SRVR:HE1PR02MB0651; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0651; x-forefront-prvs: 06911FE69E x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(252514010)(189002)(34854003)(42154003)(199003)(33656002)(2900100001)(19580405001)(122556002)(68736005)(86362001)(107886002)(62966003)(5001830100001)(54356999)(5001960100002)(81156007)(4001540100001)(66066001)(229853001)(10400500002)(16236675004)(15975445007)(97736004)(110136002)(77096005)(76576001)(5002640100001)(46102003)(189998001)(64706001)(101416001)(102836002)(40100003)(2351001)(5001860100001)(74316001)(50986999)(92566002)(19300405004)(5007970100001)(87936001)(19625215002)(19580395003)(106356001)(77156002)(2501003)(5890100001)(105586002)(450100001)(5003600100002)(5004730100002)(14773001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0651; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_HE1PR02MB0652B5507B4434A631F6C70CF9550HE1PR02MB0652eurp_" MIME-Version: 1.0 X-OriginatorOrg: toganetworks.com X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2015 09:58:42.3170 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0651 Archived-At: Subject: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 09:59:04 -0000 --_000_HE1PR02MB0652B5507B4434A631F6C70CF9550HE1PR02MB0652eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello dear authors, I am new in iWARP and need your help to clear my misunderstanding of RFCs. RFC5041 Section 5.4 stands that: "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all of the= following are true: * the last DDP Segment of the DDP Message had its Last flag set, * all of the DDP Segments of the DDP Message have been Placed, * all preceding DDP Messages have been Placed, and * each preceding DDP Message has been Delivered to the ULP." Let's assume that data sink receives sequence of "rdma_write" and send" mes= sages. According to RFC even if miss ordering happens "send" can be delivered to R= DMAP only after last segment of rdma_write arrives and is placed. So RDMAP layer completes "send" only after full rdma_write assembly. If so why we need to generate new "Immediate request type" and cannot reuse= 8 bytes length "send" ? Best regards, Elena Gurevich --------------------------------------------------------- Software Architector at Toga Networks Ltd. Email: Elena.Gurevich@toganetwors.com --------------------------------------------------------- ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- --_000_HE1PR02MB0652B5507B4434A631F6C70CF9550HE1PR02MB0652eurp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello dear authors,

 

I am new in iWARP and need your help to clear my &nb= sp;misunderstanding of RFCs.

RFC5041 Section 5.4 stands that:

 

At the Data Sink, DDP MUST Deliver = a DDP Message if and only if all of the following are true:

* the last DDP Segment of the DDP Message = had its Last flag set,

* all of the DDP Segments of the DDP Messa= ge have been Placed,

* all preceding DDP Messages have been Pla= ced, and

* each preceding DDP Message has been Delivered to the ULP.”= ;

 

Let’s assume that data sink receives sequence = of “rdma_write” and send” messages.

 

According to RFC even if miss ordering happens ̶= 0;send” can be delivered to RDMAP only after last segment of rdma_wri= te arrives and is placed.

So RDMAP layer completes “send” only aft= er full rdma_write assembly.

If so why we need to generate new “Immediate r= equest type” and cannot reuse 8 bytes length “send” ?

 

 

Best regards,

Elena Gurevich

----------------------------------------------------= -----

Software Architector at Toga Networks Ltd.

 

Email: Elena.Gurevich@toganetwors.com

----------------------------------------------------= -----

 

 

 

 

---------------------------------------------= ---------------------------------------------------------------------------= -------------------------

This email and any files transmitted and/or attachment= s with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed.
If you have received this email in error please notify the system manager. = This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not
the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on
the contents of this information is strictly prohibited.

---------------------------------------------= ---------------------------------------------------------------------------= ------------------------


--_000_HE1PR02MB0652B5507B4434A631F6C70CF9550HE1PR02MB0652eurp_-- From nobody Sun Sep 6 06:33:40 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0686B1B57A0 for ; Sun, 6 Sep 2015 06:33:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sdLsSgRJdSDH for ; Sun, 6 Sep 2015 06:33:38 -0700 (PDT) Received: from p3plsmtpa12-04.prod.phx3.secureserver.net (p3plsmtpa12-04.prod.phx3.secureserver.net [68.178.252.233]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B9F8D1B5818 for ; Sun, 6 Sep 2015 06:33:38 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa12-04.prod.phx3.secureserver.net with id DpZd1r0081n35Pc01pZdoB; Sun, 06 Sep 2015 06:33:38 -0700 To: storm@ietf.org References: From: Tom Talpey Message-ID: <55EC40AD.4080005@talpey.com> Date: Sun, 6 Sep 2015 09:33:33 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Subject: Re: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 13:33:40 -0000 On 9/6/2015 5:58 AM, Elena Gurevich wrote: > Hello dear authors, > > I am new in iWARP and need your help to clear my misunderstanding of RFCs. > > RFC5041 Section 5.4 stands that: > > “At the Data Sink, DDP MUST Deliver a DDP Message if and only if all of > the following are true: > > * the last DDP Segment of the DDP Message had its Last flag set, > > * all of the DDP Segments of the DDP Message have been Placed, > > * all preceding DDP Messages have been Placed, and > > * each preceding DDP Message has been Delivered to the ULP.” > > Let’s assume that data sink receives sequence of “rdma_write” and send” > messages. > > According to RFC even if miss ordering happens “send” can be delivered > to RDMAP only after last segment of rdma_write arrives and is placed. > > So RDMAP layer completes “send” only after full rdma_write assembly. These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend. > > If so why we need to generate new “Immediate request type” and cannot > reuse 8 bytes length “send” ? I don't understand the question. An Immediate is another behavior, and which is used by upper layers rather differently. Also, what "8 bytes" are you referring to? Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it. Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates. From nobody Mon Sep 7 02:49:02 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EC8001B3E3B for ; Mon, 7 Sep 2015 02:49:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.798 X-Spam-Level: X-Spam-Status: No, score=0.798 tagged_above=-999 required=5 tests=[BAYES_50=0.8, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X44eoOUZiUKv for ; Mon, 7 Sep 2015 02:48:58 -0700 (PDT) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0660.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::660]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 66CD61ACE9C for ; Mon, 7 Sep 2015 02:48:58 -0700 (PDT) Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) with Microsoft SMTP Server (TLS) id 15.1.262.15; Mon, 7 Sep 2015 09:48:39 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Mon, 7 Sep 2015 09:48:39 +0000 From: Elena Gurevich To: Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] Send vs. Immediate Thread-Index: AdDoie0+xkvsjYh7T92FPZdqNGMGkAAHrK+AACkxPhA= Date: Mon, 7 Sep 2015 09:48:38 +0000 Message-ID: References: <55EC40AD.4080005@talpey.com> In-Reply-To: <55EC40AD.4080005@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0652; 5:s0Pi/mVvNKRaOwB4cM4Iy/AA39FWoaXOPPcaS7u7yY7WinIucDMiGzG3UtMDyXJTPZlq4ItPtz0MCxgZp0EGc+W8jF4aOuJAxXP9llyjD5YHF7wI4FVwgEL8uFOJFWZbQ28IpubIUjiaCnNapNHN3w==; 24:PMfMX+LXH4kdChzWTxepuJQLwXC3EqGkC3xFBI/dfJH4wt56ouSWyTMnz+WA/iAU4eYy7YZXDWZCU6aAhBy4aD2ycAmWW0M8R1SMUmaUA8k=; 20:D9ryYZpduKHeItbehawm+ukkQwaz00JU8hxzlepILPYsPaX46//Euv8W9A70pCC7CAcB9mPcU8iQFGRhRJBKqA== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0652; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(8121501046)(5005006)(3002001); SRVR:HE1PR02MB0652; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0652; x-forefront-prvs: 069255B8B8 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(199003)(189002)(13464003)(34854003)(479174004)(42154003)(377454003)(24454002)(5004730100002)(81156007)(106356001)(19580405001)(86362001)(2900100001)(5001860100001)(5007970100001)(50986999)(40100003)(102836002)(15975445007)(87936001)(5001830100001)(4001540100001)(105586002)(62966003)(2501003)(122556002)(5003600100002)(2950100001)(97736004)(33656002)(19580395003)(77096005)(76176999)(77156002)(66066001)(5002640100001)(64706001)(74316001)(5001770100001)(101416001)(10400500002)(5890100001)(107886002)(189998001)(68736005)(76576001)(92566002)(54356999)(46102003)(5001960100002)(14773001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0652; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: toganetworks.com X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Sep 2015 09:48:38.8687 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0652 Archived-At: Subject: Re: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 09:49:01 -0000 Hello Tom, thanks for a prompt response, my question was little bit differe= nt. I failed to found differences in behavior between RDMA Write followed by Im= mediate Data and RDMA Write followed by Send. Both consume untagged buffer of queue #0 - so at data sink this buffer shou= ld be preposted in both cases. According to RFCs except DDP message format both operation has to be proces= sed similarly as at data Source as at data Sink. According to ordering and completion rules, specified in RFC 5040 and 7306= : First |Second | Placement | Placement | Orderi= ng Operation| Operation | Guarantee at | Guarantee at | Guarantee at | | Remote Peer | Local Peer |= Remote Peer -------------+---------------+----------------------+--------------------+-= --------------- RDMA | Send | No placement | Not applicable | Not applicabl= e Write | | guarantee. If | = | | | guarantee is | = | | | necessary, see | = | | | footnote 1. | = | -------------+---------------+----------------------+--------------------+-= ----------------- RDMA | Immediate| No Placement | Not | Immediate Da= ta Write | Data | Guarantee | Applicable | is Com= pleted | | | = | after RDMA | | | = | Write is Placed | | | = | and Delivered But even if the cell "Ordering Guarantee at Remote Peer" is different for = RDMA Write followed by Immediate Data and RDMA Write followed by Send. At the end behavior has to be the same, because of specification of RFC5041= Section 5.4. ( see details in my original e-mail) So why we need Immediate Data operation and cannot just reuse Send operatio= n ? Best regards, Lena -----Original Message----- From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey Sent: Sunday, September 06, 2015 4:34 PM To: storm@ietf.org Subject: Re: [storm] Send vs. Immediate On 9/6/2015 5:58 AM, Elena Gurevich wrote: > Hello dear authors, > > I am new in iWARP and need your help to clear my misunderstanding of RFC= s. > > RFC5041 Section 5.4 stands that: > > "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all > of the following are true: > > * the last DDP Segment of the DDP Message had its Last flag set, > > * all of the DDP Segments of the DDP Message have been Placed, > > * all preceding DDP Messages have been Placed, and > > * each preceding DDP Message has been Delivered to the ULP." > > Let's assume that data sink receives sequence of "rdma_write" and send" > messages. > > According to RFC even if miss ordering happens "send" can be delivered > to RDMAP only after last segment of rdma_write arrives and is placed. > > So RDMAP layer completes "send" only after full rdma_write assembly. These statements are correct, the Send is delivered only after Placing all = the previous RDMA Write segments. This is a key behavior upon which most up= per layers depend. > > If so why we need to generate new "Immediate request type" and cannot > reuse 8 bytes length "send" ? I don't understand the question. An Immediate is another behavior, and whic= h is used by upper layers rather differently. Also, what "8 bytes" are you referring to? Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immedi= ate? Some upper layers do, I believe Lustre does, for example, but the orig= inal iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC= 7306 extension, published last year, adds it. Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a la= rger, separate message to complete an operation, and so use a full Send. Th= ese protocols use iWARP, without Immediates. _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- From nobody Mon Sep 7 08:25:36 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8BD3A1B524F for ; Mon, 7 Sep 2015 08:25:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jTRLm3EoiC1h for ; Mon, 7 Sep 2015 08:25:34 -0700 (PDT) Received: from p3plsmtpa11-10.prod.phx3.secureserver.net (p3plsmtpa11-10.prod.phx3.secureserver.net [68.178.252.111]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 924B61B54EC for ; Mon, 7 Sep 2015 08:25:33 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa11-10.prod.phx3.secureserver.net with id EFRX1r00h1n35Pc01FRY64; Mon, 07 Sep 2015 08:25:33 -0700 To: Elena Gurevich , "storm@ietf.org" References: <55EC40AD.4080005@talpey.com> From: Tom Talpey Message-ID: <55EDAC65.2090000@talpey.com> Date: Mon, 7 Sep 2015 11:25:25 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 15:25:35 -0000 > So why we need Immediate Data operation and cannot just reuse Send operation ? Using either one is a choice by the upper layer (RDMA consumer), and it's absolutely possible to just reuse a Send. As mentioned, many upper layers simply use Send. Others, for various reasons, use Immediate data. DDP, and the extended RDMAP (RFC7306) support either one. On 9/7/2015 5:48 AM, Elena Gurevich wrote: > Hello Tom, thanks for a prompt response, my question was little bit different. > > I failed to found differences in behavior between RDMA Write followed by Immediate Data and RDMA Write followed by Send. > Both consume untagged buffer of queue #0 - so at data sink this buffer should be preposted in both cases. > > According to RFCs except DDP message format both operation has to be processed similarly as at data Source as at data Sink. > > According to ordering and completion rules, specified in RFC 5040 and 7306: > > First |Second | Placement | Placement | Ordering > Operation| Operation | Guarantee at | Guarantee at | Guarantee at > | | Remote Peer | Local Peer | Remote Peer > -------------+---------------+----------------------+--------------------+---------------- > RDMA | Send | No placement | Not applicable | Not applicable > Write | | guarantee. If | | > | | guarantee is | | > | | necessary, see | | > | | footnote 1. | | > -------------+---------------+----------------------+--------------------+------------------ > RDMA | Immediate| No Placement | Not | Immediate Data > Write | Data | Guarantee | Applicable | is Completed > | | | | after RDMA > | | | | Write is Placed > | | | | and Delivered > > But even if the cell "Ordering Guarantee at Remote Peer" is different for RDMA Write followed by Immediate Data and RDMA Write followed by Send. > At the end behavior has to be the same, because of specification of RFC5041 Section 5.4. ( see details in my original e-mail) > > So why we need Immediate Data operation and cannot just reuse Send operation ? > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey > Sent: Sunday, September 06, 2015 4:34 PM > To: storm@ietf.org > Subject: Re: [storm] Send vs. Immediate > > On 9/6/2015 5:58 AM, Elena Gurevich wrote: >> Hello dear authors, >> >> I am new in iWARP and need your help to clear my misunderstanding of RFCs. >> >> RFC5041 Section 5.4 stands that: >> >> "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all >> of the following are true: >> >> * the last DDP Segment of the DDP Message had its Last flag set, >> >> * all of the DDP Segments of the DDP Message have been Placed, >> >> * all preceding DDP Messages have been Placed, and >> >> * each preceding DDP Message has been Delivered to the ULP." >> >> Let's assume that data sink receives sequence of "rdma_write" and send" >> messages. >> >> According to RFC even if miss ordering happens "send" can be delivered >> to RDMAP only after last segment of rdma_write arrives and is placed. >> >> So RDMAP layer completes "send" only after full rdma_write assembly. > > These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend. > >> >> If so why we need to generate new "Immediate request type" and cannot >> reuse 8 bytes length "send" ? > > I don't understand the question. An Immediate is another behavior, and which is used by upper layers rather differently. Also, what > "8 bytes" are you referring to? > > Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it. > > Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates. > From nobody Mon Sep 7 22:56:18 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6A9BC1B2B11 for ; Mon, 7 Sep 2015 22:56:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oeJJI-xztv2m for ; Mon, 7 Sep 2015 22:56:15 -0700 (PDT) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0631.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::631]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7A13D1B2A71 for ; Mon, 7 Sep 2015 22:56:14 -0700 (PDT) Received: from HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) by HE1PR02MB1020.eurprd02.prod.outlook.com (10.163.172.150) with Microsoft SMTP Server (TLS) id 15.1.262.15; Tue, 8 Sep 2015 05:55:57 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) with Microsoft SMTP Server (TLS) id 15.1.262.15; Tue, 8 Sep 2015 05:55:55 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Tue, 8 Sep 2015 05:55:55 +0000 From: Elena Gurevich To: Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] Send vs. Immediate Thread-Index: AdDoie0+xkvsjYh7T92FPZdqNGMGkAAHrK+AACkxPhAADQGHgAAeMEsQ Date: Tue, 8 Sep 2015 05:55:54 +0000 Message-ID: References: <55EC40AD.4080005@talpey.com> <55EDAC65.2090000@talpey.com> In-Reply-To: <55EDAC65.2090000@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0650; 5:RUnxrIXMOrEzqC0b1wC1HZn57AwHApeUU3SLb4A5wz6SlQu2H09h8khrmD1M5hmMsBPPt2AecH4AXDADxjSvOjvRgHYZSO691kCe10Zf4o/YR3U8snfOBwKa84Qpk6RmffFU0rPw6x+RgcvswgHuwQ==; 24:6otOJZbzOwzrwrHinMPyM239nQDyua8/Z//Mwlp+OgIwC6JvmUYpvsvHuCSayEKk9TWWNIxxzZsxMlOiBMa7PhjxUnyXzQBM1+58okSZXB8=; 20:qxMBVLsj1WO8oasMmujtloQcxeJ18PMa7szfMM5t3TIWHTGxwK6SnuxPJGgEWS/y8E7I2IpgQPJWk8JiAh5bSw== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0650; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(8121501046)(3002001); SRVR:HE1PR02MB0650; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0650; x-forefront-prvs: 069373DFB6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(479174004)(189002)(199003)(377454003)(13464003)(24454002)(42154003)(34854003)(93886004)(10400500002)(19580395003)(5007970100001)(92566002)(81156007)(5002640100001)(5004730100002)(102836002)(46102003)(62966003)(106356001)(5001770100001)(2950100001)(5001830100001)(5001860100001)(40100003)(107886002)(5001960100002)(2900100001)(5890100001)(189998001)(2501003)(105586002)(76576001)(86362001)(19580405001)(54356999)(97736004)(50986999)(74316001)(66066001)(77096005)(87936001)(68736005)(76176999)(4001540100001)(5003600100002)(101416001)(122556002)(77156002)(64706001)(33656002)(14773001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0650; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Sep 2015 05:55:54.5807 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0650 X-Microsoft-Exchange-Diagnostics: 1; HE1PR02MB1020; 2:QZUKX0hve207YexiOXtRHcJTC823+jyVPGBt2Ic/ybGgNN/ftGSGMjy/D/K8co4M798ivsw0oUil6AUswrZedkuowj2ABN5Rik+0Kq6IjbbbsLQWKyQov7AhSKI3lwsDLhuVKXWUD8WjAcSec1pyw+Fh3vUquij4yexuU3hTrec=; 23:5wlqPTExkS+6le8f9xijeP2B/JXB2koY7Q8mX1ZvEsU3XrLD4FRPYHgitzhpyegvEboOaeKDYNZTEtNjclzmWtPkUvRHF4w+MZx72CzrD9mYF+bwDjSXrX9Gsd5/0YA3gw8Mw6GH5A8m2WfULL2lzdOWpjuNWVsTwGAlIeaGLZtpiLwSg+0oUMoABCMYz/Uf X-OriginatorOrg: toganetworks.com Archived-At: Subject: Re: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 05:56:17 -0000 Thanks you again, If so, the statement "Not applicable" in cell "Ordering at Remote Peer" for= RDMA Write followed by Send operation in Appendix B of RFC5040 is inaccura= te And should be changed as "Send is completed after RDMA Write placed and del= ivered". Best regards, Elena -----Original Message----- From: Tom Talpey [mailto:tom@talpey.com] Sent: Monday, September 07, 2015 6:25 PM To: Elena Gurevich; storm@ietf.org Subject: Re: [storm] Send vs. Immediate > So why we need Immediate Data operation and cannot just reuse Send opera= tion ? Using either one is a choice by the upper layer (RDMA consumer), and it's a= bsolutely possible to just reuse a Send. As mentioned, many upper layers si= mply use Send. Others, for various reasons, use Immediate data. DDP, and th= e extended RDMAP (RFC7306) support either one. On 9/7/2015 5:48 AM, Elena Gurevich wrote: > Hello Tom, thanks for a prompt response, my question was little bit diffe= rent. > > I failed to found differences in behavior between RDMA Write followed by = Immediate Data and RDMA Write followed by Send. > Both consume untagged buffer of queue #0 - so at data sink this buffer sh= ould be preposted in both cases. > > According to RFCs except DDP message format both operation has to be proc= essed similarly as at data Source as at data Sink. > > According to ordering and completion rules, specified in RFC 5040 and 73= 06: > > First |Second | Placement | Placement | Orde= ring > Operation| Operation | Guarantee at | Guarantee at | Guarantee at > | | Remote Peer | Local Peer = | Remote Peer > -------------+---------------+----------------------+--------------------= +---------------- > RDMA | Send | No placement | Not applicable | Not applica= ble > Write | | guarantee. If | = | > | | guarantee is | = | > | | necessary, see | = | > | | footnote 1. | = | > -------------+---------------+----------------------+--------------------= +------------------ > RDMA | Immediate| No Placement | Not | Immediate = Data > Write | Data | Guarantee | Applicable | is C= ompleted > | | | = | after RDMA > | | | = | Write is Placed > | | | = | and Delivered > > But even if the cell "Ordering Guarantee at Remote Peer" is different fo= r RDMA Write followed by Immediate Data and RDMA Write followed by Send. > At the end behavior has to be the same, because of specification of > RFC5041 Section 5.4. ( see details in my original e-mail) > > So why we need Immediate Data operation and cannot just reuse Send operat= ion ? > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey > Sent: Sunday, September 06, 2015 4:34 PM > To: storm@ietf.org > Subject: Re: [storm] Send vs. Immediate > > On 9/6/2015 5:58 AM, Elena Gurevich wrote: >> Hello dear authors, >> >> I am new in iWARP and need your help to clear my misunderstanding of RF= Cs. >> >> RFC5041 Section 5.4 stands that: >> >> "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all >> of the following are true: >> >> * the last DDP Segment of the DDP Message had its Last flag set, >> >> * all of the DDP Segments of the DDP Message have been Placed, >> >> * all preceding DDP Messages have been Placed, and >> >> * each preceding DDP Message has been Delivered to the ULP." >> >> Let's assume that data sink receives sequence of "rdma_write" and send" >> messages. >> >> According to RFC even if miss ordering happens "send" can be >> delivered to RDMAP only after last segment of rdma_write arrives and is = placed. >> >> So RDMAP layer completes "send" only after full rdma_write assembly. > > These statements are correct, the Send is delivered only after Placing al= l the previous RDMA Write segments. This is a key behavior upon which most = upper layers depend. > >> >> If so why we need to generate new "Immediate request type" and cannot >> reuse 8 bytes length "send" ? > > I don't understand the question. An Immediate is another behavior, and > which is used by upper layers rather differently. Also, what > "8 bytes" are you referring to? > > Perhaps you mean, why doesn't the upper layer use an RDMA Write with Imme= diate? Some upper layers do, I believe Lustre does, for example, but the or= iginal iWARP RDMAP (RFC5040) protocol did not support that operation. The R= FC7306 extension, published last year, adds it. > > Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a = larger, separate message to complete an operation, and so use a full Send. = These protocols use iWARP, without Immediates. > ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- From nobody Tue Sep 8 03:02:58 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E88301B301D for ; Tue, 8 Sep 2015 03:02:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yqNU9FfjoFLf for ; Tue, 8 Sep 2015 03:02:54 -0700 (PDT) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0667.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::667]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C1F641A904D for ; Tue, 8 Sep 2015 03:02:53 -0700 (PDT) Received: from HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) by HE1PR02MB0746.eurprd02.prod.outlook.com (10.161.114.23) with Microsoft SMTP Server (TLS) id 15.1.262.15; Tue, 8 Sep 2015 10:02:33 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) with Microsoft SMTP Server (TLS) id 15.1.262.15; Tue, 8 Sep 2015 10:02:28 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Tue, 8 Sep 2015 10:02:28 +0000 From: Elena Gurevich To: "storm@ietf.org" Thread-Topic: DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNg== Date: Tue, 8 Sep 2015 10:02:28 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0650; 5:r29RKRXfD6/IkTCwmiG3GqBfu9q9RLtvIuRA9KIBYtAGM0dIl1OeR9QXiZfOTzg9ccRQ04NWede5+/kIn5lsxFDGNBP4EMxRR0GUmm561Ecguh71UTlcbMddUlFxbHQP4HNsn68KEM3J7iL2zrtU0g==; 24:+bYmVTisMzTXwPTBlqw5uOUlZoXGmnEzHSKOrwU/n3E5rSwDej6fHolNhcZnUb3tCd4iZYcfQO7WtpEEVorSmXStxbmhOD87ECBWO+7k9zs=; 20:9LKFL8v7dXBpkFyQ5brlcWoisrDudJdmMc+cD2jU3eRiJcBkWw/Q1yOdIp2s5aKcDIYiMzfvBerUd37QcfyYMg== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0650; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(108003899814671); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(8121501046)(3002001); SRVR:HE1PR02MB0650; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0650; x-forefront-prvs: 069373DFB6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(189002)(199003)(252514010)(42154003)(34854003)(10400500002)(450100001)(5007970100001)(10710500003)(81156007)(19580395003)(92566002)(229853001)(102836002)(5004730100002)(2351001)(5002640100001)(46102003)(62966003)(106356001)(110136002)(2420400006)(5001830100001)(107886002)(5001860100001)(5001960100002)(40100003)(15975445007)(5890100001)(2900100001)(105586002)(189998001)(2501003)(76576001)(86362001)(50986999)(19580405001)(7110500001)(54356999)(97736004)(77096005)(74316001)(66066001)(19300405004)(16236675004)(87936001)(68736005)(4001540100001)(5003600100002)(122556002)(101416001)(64706001)(77156002)(33656002)(19625215002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0650; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:3; A:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_HE1PR02MB0652BA180545E2D5E21AB8FEF9530HE1PR02MB0652eurp_" MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Sep 2015 10:02:28.1775 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0650 X-Microsoft-Exchange-Diagnostics: 1; HE1PR02MB0746; 2:CImnOyPXUTlE4cLGjBsFL0EG7LZZaIbRlZ/+0h5QOjSpw/GQ3rJJhJFDbQBK3im4W/n0SEyZiuVDtSJQnUJaj6jbkTpwJa7ZXXYynF1QJ/ShgtzirNqAINvaZgfxBi0UFmjbPtmB9hUDVH+FFXEeC8q+jr5IcvZhUHio90QvhW4=; 23:g9PB7gaMZXpxmax3+uc+872pY/tl/hScMMHReKtg0Ca/jpN7RRReGegfUeznK+xGadhVPkhN/OET0XOFkK83eEWq1E2CEt5s3dzY/OdhmlzwkxpvCOGhsIxI+20ox2GzUwQoyUXaBWdJ3fcb4DpR1EWLCk6EOGe7BaW8l8tvl/FaSyh2ZAFd6uQ9nI7da9PR X-OriginatorOrg: toganetworks.com Archived-At: Subject: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 10:02:57 -0000 --_000_HE1PR02MB0652BA180545E2D5E21AB8FEF9530HE1PR02MB0652eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, RFC 5041 section 5.3 stands that ---------------------------------- 5.3 Ordering Among DDP Messages Messages passed through the DDP MUST conform to the ordering rules defined in this section. At the Data Source, DDP: * MUST transmit DDP Messages in the order they were submitted to the DDP layer, * SHOULD transmit DDP Segments within a DDP Message in increasing MO order for Untagged DDP Messages, and in increasing TO order for Tagged DDP Messages. ---------------------------------- Does this mean that transmitter MUST not interleave DDM segments related to= consequent DDP messages ? Best regards, Elena Gurevich --------------------------------------------------------- Software Architector at Toga Networks Ltd. Email: Elena.Gurevich@toganetworks.com --------------------------------------------------------- ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- --_000_HE1PR02MB0652BA180545E2D5E21AB8FEF9530HE1PR02MB0652eurp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

RFC 5041 section 5.3 s= tands that

 

----------------------------------

5.3 Ordering Among DDP Messages=

 

Messages passed through the DDP MUST confo= rm to the ordering rules

defined in this section.=

At the Data Source, DDP:=

* MUST transmit DDP Messages in the order = they were submitted to

the DDP layer,

* SHOULD transmit DDP Segments within a DD= P Message in increasing

MO order for Untagged DDP Messages, and in= increasing TO order

for Tagged DDP Messages.

----------------------------------

 

 

Does this mean that transmitter MUST not interleave = DDM segments related to consequent DDP messages ?

 

Best regards,

Elena Gurevich

----------------------------------------------------= -----

Software Architector at Toga Networks Ltd.

 

Email: Elena.Gurevich@toganetworks.com

----------------------------------------------------= -----

 

---------------------------------------------= ---------------------------------------------------------------------------= -------------------------

This email and any files transmitted and/or attachment= s with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed.
If you have received this email in error please notify the system manager. = This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not
the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on
the contents of this information is strictly prohibited.

---------------------------------------------= ---------------------------------------------------------------------------= ------------------------


--_000_HE1PR02MB0652BA180545E2D5E21AB8FEF9530HE1PR02MB0652eurp_-- From nobody Tue Sep 8 07:08:27 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A5CE61B3CE5 for ; Tue, 8 Sep 2015 07:08:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cg3Ns8h7MyPj for ; Tue, 8 Sep 2015 07:08:20 -0700 (PDT) Received: from p3plsmtpa09-03.prod.phx3.secureserver.net (p3plsmtpa09-03.prod.phx3.secureserver.net [173.201.193.232]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2057A1B4993 for ; Tue, 8 Sep 2015 07:08:12 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa09-03.prod.phx3.secureserver.net with id Ee8B1r00A1n35Pc01e8Byf; Tue, 08 Sep 2015 07:08:12 -0700 To: Elena Gurevich , "storm@ietf.org" References: <55EC40AD.4080005@talpey.com> <55EDAC65.2090000@talpey.com> From: Tom Talpey Message-ID: <55EEEBC5.4050906@talpey.com> Date: Tue, 8 Sep 2015 10:08:05 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [storm] Send vs. Immediate X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 14:08:23 -0000 On 9/8/2015 1:55 AM, Elena Gurevich wrote: > Thanks you again, > > If so, the statement "Not applicable" in cell "Ordering at Remote Peer" for RDMA Write followed by Send operation in Appendix B of RFC5040 is inaccurate > And should be changed as "Send is completed after RDMA Write placed and delivered". I disagree, it is accurate because the table is defining the behavior as visible from the *local* peer, i.e. the initiator. You will note that the only messages with anything but "not applicable" are the RDMA Read responses. Note the last words of the first sentence: Appendix B. Ordering and Completion Table The following table summarizes the ordering relationships that are defined in Section 5.5, "Ordering and Completions", from the standpoint of the local peer issuing the two Operations. The state of the *remote* peer depends upon it taking completions, which is not something the local peer can determine without an upper layer exchange. Tom. > > Best regards, > Elena > > -----Original Message----- > From: Tom Talpey [mailto:tom@talpey.com] > Sent: Monday, September 07, 2015 6:25 PM > To: Elena Gurevich; storm@ietf.org > Subject: Re: [storm] Send vs. Immediate > > > So why we need Immediate Data operation and cannot just reuse Send operation ? > > Using either one is a choice by the upper layer (RDMA consumer), and it's absolutely possible to just reuse a Send. As mentioned, many upper layers simply use Send. Others, for various reasons, use Immediate data. DDP, and the extended RDMAP (RFC7306) support either one. > > > On 9/7/2015 5:48 AM, Elena Gurevich wrote: >> Hello Tom, thanks for a prompt response, my question was little bit different. >> >> I failed to found differences in behavior between RDMA Write followed by Immediate Data and RDMA Write followed by Send. >> Both consume untagged buffer of queue #0 - so at data sink this buffer should be preposted in both cases. >> >> According to RFCs except DDP message format both operation has to be processed similarly as at data Source as at data Sink. >> >> According to ordering and completion rules, specified in RFC 5040 and 7306: >> >> First |Second | Placement | Placement | Ordering >> Operation| Operation | Guarantee at | Guarantee at | Guarantee at >> | | Remote Peer | Local Peer | Remote Peer >> -------------+---------------+----------------------+--------------------+---------------- >> RDMA | Send | No placement | Not applicable | Not applicable >> Write | | guarantee. If | | >> | | guarantee is | | >> | | necessary, see | | >> | | footnote 1. | | >> -------------+---------------+----------------------+--------------------+------------------ >> RDMA | Immediate| No Placement | Not | Immediate Data >> Write | Data | Guarantee | Applicable | is Completed >> | | | | after RDMA >> | | | | Write is Placed >> | | | | and Delivered >> >> But even if the cell "Ordering Guarantee at Remote Peer" is different for RDMA Write followed by Immediate Data and RDMA Write followed by Send. >> At the end behavior has to be the same, because of specification of >> RFC5041 Section 5.4. ( see details in my original e-mail) >> >> So why we need Immediate Data operation and cannot just reuse Send operation ? >> >> Best regards, >> Lena >> >> -----Original Message----- >> From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey >> Sent: Sunday, September 06, 2015 4:34 PM >> To: storm@ietf.org >> Subject: Re: [storm] Send vs. Immediate >> >> On 9/6/2015 5:58 AM, Elena Gurevich wrote: >>> Hello dear authors, >>> >>> I am new in iWARP and need your help to clear my misunderstanding of RFCs. >>> >>> RFC5041 Section 5.4 stands that: >>> >>> "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all >>> of the following are true: >>> >>> * the last DDP Segment of the DDP Message had its Last flag set, >>> >>> * all of the DDP Segments of the DDP Message have been Placed, >>> >>> * all preceding DDP Messages have been Placed, and >>> >>> * each preceding DDP Message has been Delivered to the ULP." >>> >>> Let's assume that data sink receives sequence of "rdma_write" and send" >>> messages. >>> >>> According to RFC even if miss ordering happens "send" can be >>> delivered to RDMAP only after last segment of rdma_write arrives and is placed. >>> >>> So RDMAP layer completes "send" only after full rdma_write assembly. >> >> These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend. >> >>> >>> If so why we need to generate new "Immediate request type" and cannot >>> reuse 8 bytes length "send" ? >> >> I don't understand the question. An Immediate is another behavior, and >> which is used by upper layers rather differently. Also, what >> "8 bytes" are you referring to? >> >> Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it. >> >> Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates. >> > ------------------------------------------------------------------------------------------------------------------------------------------------- > This email and any files transmitted and/or attachments with it are confidential and proprietary information of > Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. This message contains confidential > information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named > addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately > by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not > the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on > the contents of this information is strictly prohibited. > ------------------------------------------------------------------------------------------------------------------------------------------------ > > > From nobody Tue Sep 8 07:15:43 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 16A5B1B4B22 for ; Tue, 8 Sep 2015 07:15:42 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 00D6Ot_pKuSZ for ; Tue, 8 Sep 2015 07:15:36 -0700 (PDT) Received: from p3plsmtpa09-04.prod.phx3.secureserver.net (p3plsmtpa09-04.prod.phx3.secureserver.net [173.201.193.233]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 569D31B4B1F for ; Tue, 8 Sep 2015 07:15:36 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa09-04.prod.phx3.secureserver.net with id EeFb1r0031n35Pc01eFbVg; Tue, 08 Sep 2015 07:15:36 -0700 To: storm@ietf.org References: From: Tom Talpey Message-ID: <55EEED81.2060508@talpey.com> Date: Tue, 8 Sep 2015 10:15:29 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 14:15:42 -0000 On 9/8/2015 6:02 AM, Elena Gurevich wrote: > Hello, > > RFC 5041 section 5.3 stands that > > ---------------------------------- > > 5.3 Ordering Among DDP Messages > > Messages passed through the DDP MUST conform to the ordering rules > > defined in this section. > > At the Data Source, DDP: > > * MUST transmit DDP Messages in the order they were submitted to > > the DDP layer, > > * SHOULD transmit DDP Segments within a DDP Message in increasing > > MO order for Untagged DDP Messages, and in increasing TO order > > for Tagged DDP Messages. > > ---------------------------------- > > Does this mean that transmitter MUST not interleave DDM segments related > to consequent DDP messages ? It depends on what you mean by "the transmitter". It's certainly possible that network delivery and TCP retransmission can reorder segments. But the first rule requires that DDP not interleave two separate operations when passing them to the TCP transport. Tom. From nobody Tue Sep 8 07:40:14 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D77961B44C2 for ; Tue, 8 Sep 2015 07:40:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -7.011 X-Spam-Level: X-Spam-Status: No, score=-7.011 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YGcu989oliMV for ; Tue, 8 Sep 2015 07:40:07 -0700 (PDT) Received: from ausxipps301.us.dell.com (ausxipps301.us.dell.com [143.166.148.223]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BE1271B4475 for ; Tue, 8 Sep 2015 07:40:07 -0700 (PDT) DomainKey-Signature: s=smtpout; d=dell.com; c=nofws; q=dns; h=X-LoopCount0:X-IronPort-AV:From:To:CC:Subject: Thread-Topic:Thread-Index:Date:Message-ID:References: In-Reply-To:Accept-Language:Content-Language: X-MS-Has-Attach:X-MS-TNEF-Correlator:x-originating-ip: Content-Type:Content-ID:Content-Transfer-Encoding: MIME-Version:Return-Path; b=AL1b6edVjTwhJ93qosoe0ZMkGOt14BaxcmQ7v4nArrZmP9T617ox7o+g Fi4+mQ9ijXFQqMskLR/umI/+/PO8Qd99sG6gxCmZP0KteBg1rnVcMwOWS XhXVqmC/vphiPlS1uNzY2nbGmcFiiaGJUeDWCbb5QB87kXq4qVWoXMd7q M=; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dell.com; i=@dell.com; q=dns/txt; s=smtpout; t=1441723208; x=1473259208; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=8+LgcMhm0mjg3d47q7meSE/c/dfv9rgWfHjo7a2bsKs=; b=dnw+fQ5G2WglX0CIRnEcqgh2lHS3/jMBg0WUFTDSQ6uzz5wK2PN9y/u3 4zCDU4R+MKfLo1dthhbl98pJIALX2YhJ4uchpnp+lzqkQ6WJ/thP7T1Xv bk+CcuO6KjbjSIgg+HN8mFv9wBF+LGUS5YW01D2b3OhCjPD2g6lM3LgbR 4=; X-LoopCount0: from 10.175.216.251 X-IronPort-AV: E=Sophos;i="5.17,490,1437454800"; d="scan'208";a="701669100" From: To: Thread-Topic: [storm] DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNgATXO+AAADbpYA= Date: Tue, 8 Sep 2015 14:40:04 +0000 Message-ID: References: <55EEED81.2060508@talpey.com> In-Reply-To: <55EEED81.2060508@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.177.90.68] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Archived-At: Cc: storm@ietf.org Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 14:40:14 -0000 > On Sep 8, 2015, at 10:15 AM, Tom Talpey wrote: >=20 > On 9/8/2015 6:02 AM, Elena Gurevich wrote: >> ... >> Does this mean that transmitter MUST not interleave DDM segments related >> to consequent DDP messages ? >=20 > It depends on what you mean by "the transmitter". It's certainly > possible that network delivery and TCP retransmission can reorder > segments. No, it isn't. TCP provides two services: (1) ordered delivery, and (2) gua= ranteed delivery or report of failure to do so. Network delivery and TCP retransmission will NOT reorder the data delivered= by TCP to the layer above. paul From nobody Wed Sep 9 06:07:41 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 09C861B2A2A for ; Wed, 9 Sep 2015 06:07:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6X_P4uQQA54G for ; Wed, 9 Sep 2015 06:07:39 -0700 (PDT) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0604.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe04::604]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8DABB1A888F for ; Wed, 9 Sep 2015 06:07:38 -0700 (PDT) Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0651.eurprd02.prod.outlook.com (10.161.118.11) with Microsoft SMTP Server (TLS) id 15.1.262.15; Wed, 9 Sep 2015 13:07:15 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Wed, 9 Sep 2015 13:07:15 +0000 From: Elena Gurevich To: Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNgAI4rqAAC+dhtA= Date: Wed, 9 Sep 2015 13:07:15 +0000 Message-ID: References: <55EEED81.2060508@talpey.com> In-Reply-To: <55EEED81.2060508@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0651; 5:OeUZsXJo6WdAurvzSufbJ6lOdDg4haQgjykCrhsF6PJhWvehaQ5f8/j3ofI+zAqTGkTxzvWj3k3Tf49QsZwLI2WUYess91GNR5oGTu2dQVr2QckPyo1iNV9SY9ySlWu1BKI2CWN2r2dWw1So5SQNxA==; 24:guihUvnZs9/HIJ8XIUb/PEC16uqmioRiny237RwpFs6lEnrcI72Absh4WAN8hViXBHvBWCYVVKe4iX0MGsZ8l1uQJIppdTa7QxwVUVAQD/A=; 20:BYJrkr+eAw5+r9wsAyhnxTHqVHDm4tdBrpdeNCP4cDg/Wn9mK5arx1lfWkcxUdqcgZuW2BpqruhZ4uQrebRxow== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0651; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(8121501046)(5005006)(3002001); SRVR:HE1PR02MB0651; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0651; x-forefront-prvs: 0694C54398 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377454003)(189002)(34854003)(42154003)(24454002)(199003)(13464003)(479174004)(2900100001)(62966003)(68736005)(86362001)(4001540100001)(5002640100001)(107886002)(10400500002)(54356999)(101416001)(81156007)(5001960100002)(122556002)(5001830100001)(11100500001)(66066001)(5001860100001)(97736004)(189998001)(77096005)(15975445007)(64706001)(5001770100001)(46102003)(33656002)(102836002)(2950100001)(5890100001)(87936001)(5007970100001)(74316001)(5004730100002)(50986999)(76176999)(40100003)(19580405001)(106356001)(76576001)(19580395003)(92566002)(2501003)(77156002)(105586002)(5003600100002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0651; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: toganetworks.com X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Sep 2015 13:07:15.1547 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0651 Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 13:07:41 -0000 Hello, During my testing of some iWARP adaptor I discovered that RNIC interleaves = DDP segments of Send and Read Response messages. According to your previous response this behavior is forbidden, is it ? Best regards, Lena -----Original Message----- From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey Sent: Tuesday, September 08, 2015 5:15 PM To: storm@ietf.org Subject: Re: [storm] DDP messages ordering On 9/8/2015 6:02 AM, Elena Gurevich wrote: > Hello, > > RFC 5041 section 5.3 stands that > > ---------------------------------- > > 5.3 Ordering Among DDP Messages > > Messages passed through the DDP MUST conform to the ordering rules > > defined in this section. > > At the Data Source, DDP: > > * MUST transmit DDP Messages in the order they were submitted to > > the DDP layer, > > * SHOULD transmit DDP Segments within a DDP Message in increasing > > MO order for Untagged DDP Messages, and in increasing TO order > > for Tagged DDP Messages. > > ---------------------------------- > > Does this mean that transmitter MUST not interleave DDM segments > related to consequent DDP messages ? It depends on what you mean by "the transmitter". It's certainly possible t= hat network delivery and TCP retransmission can reorder segments. But the f= irst rule requires that DDP not interleave two separate operations when pas= sing them to the TCP transport. Tom. _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- From nobody Wed Sep 9 08:53:49 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3120D1B2C2C for ; Wed, 9 Sep 2015 08:53:47 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2UOmKFRbFgK7 for ; Wed, 9 Sep 2015 08:53:45 -0700 (PDT) Received: from p3plsmtpa09-03.prod.phx3.secureserver.net (p3plsmtpa09-03.prod.phx3.secureserver.net [173.201.193.232]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A10BF1A891C for ; Wed, 9 Sep 2015 08:53:45 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa09-03.prod.phx3.secureserver.net with id F3tj1r00B1n35Pc013tkp9; Wed, 09 Sep 2015 08:53:45 -0700 To: Elena Gurevich , "storm@ietf.org" References: <55EEED81.2060508@talpey.com> From: Tom Talpey Message-ID: <55F055FF.2050506@talpey.com> Date: Wed, 9 Sep 2015 11:53:35 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 15:53:47 -0000 On 9/9/2015 9:07 AM, Elena Gurevich wrote: > Hello, > > During my testing of some iWARP adaptor I discovered that RNIC interleaves DDP segments of Send and Read Response messages. > According to your previous response this behavior is forbidden, is it ? Not necessarily, because an RDMA Read response is special. It's injected into the DDP stream within the RNIC by RDMAP, and therefore it follows the RDMAP rules. See section 5.5 of RFC5040. Tom. > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey > Sent: Tuesday, September 08, 2015 5:15 PM > To: storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > On 9/8/2015 6:02 AM, Elena Gurevich wrote: >> Hello, >> >> RFC 5041 section 5.3 stands that >> >> ---------------------------------- >> >> 5.3 Ordering Among DDP Messages >> >> Messages passed through the DDP MUST conform to the ordering rules >> >> defined in this section. >> >> At the Data Source, DDP: >> >> * MUST transmit DDP Messages in the order they were submitted to >> >> the DDP layer, >> >> * SHOULD transmit DDP Segments within a DDP Message in increasing >> >> MO order for Untagged DDP Messages, and in increasing TO order >> >> for Tagged DDP Messages. >> >> ---------------------------------- >> >> Does this mean that transmitter MUST not interleave DDM segments >> related to consequent DDP messages ? > > It depends on what you mean by "the transmitter". It's certainly possible that network delivery and TCP retransmission can reorder segments. But the first rule requires that DDP not interleave two separate operations when passing them to the TCP transport. > > Tom. > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > ------------------------------------------------------------------------------------------------------------------------------------------------- > This email and any files transmitted and/or attachments with it are confidential and proprietary information of > Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. This message contains confidential > information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named > addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately > by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not > the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on > the contents of this information is strictly prohibited. > ------------------------------------------------------------------------------------------------------------------------------------------------ > > > From nobody Thu Sep 10 05:03:00 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C553D1A883F for ; Thu, 10 Sep 2015 05:02:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xh2Y0QMpeb79 for ; Thu, 10 Sep 2015 05:02:56 -0700 (PDT) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0667.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe04::667]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C205E1B55BB for ; Thu, 10 Sep 2015 05:02:55 -0700 (PDT) Received: from HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) by HE1PR02MB0954.eurprd02.prod.outlook.com (10.163.172.24) with Microsoft SMTP Server (TLS) id 15.1.262.15; Thu, 10 Sep 2015 12:02:35 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0650.eurprd02.prod.outlook.com (10.161.117.28) with Microsoft SMTP Server (TLS) id 15.1.262.15; Thu, 10 Sep 2015 12:02:33 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0262.011; Thu, 10 Sep 2015 12:02:33 +0000 From: Elena Gurevich To: Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNgAI4rqAAC+dhtAABhopgAApL59w Date: Thu, 10 Sep 2015 12:02:32 +0000 Message-ID: References: <55EEED81.2060508@talpey.com> <55F055FF.2050506@talpey.com> In-Reply-To: <55F055FF.2050506@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0650; 5:CTa4Ygk1guYxnHSMOEoGQZqkTQD+RTyj4rrDL0+qEAIjvFsK7eBfzPLtR4tSgCUXlEQaLdHy85QX14xhS+u1PxTOHCHmibahv2v3EAtAgTvK8FQQRr/J9X5wJXCWTjOGd2uR85aFFwFS5Qoy5T9NyA==; 24:igXTlyfXjP/FLhwOFu7v1h90p6+biRGkH9lZsaYvfhYulxeE6KnIrBeBmkoDQOwIt2P45Y0tUAXj1iJ3Q29yNvnFx76DLtZml7K6NIY+IzY=; 20:GAj/5B6PiWwDd5sO5e4iCWenPWgpWh85yHu8Mp0ADkapQrG6gyBxvkoJ2hYghpI3t3FmvE09K2AXyHLxxdFEKA== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0650; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(8121501046)(5005006)(3002001); SRVR:HE1PR02MB0650; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0650; x-forefront-prvs: 06952FC175 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(479174004)(377454003)(199003)(189002)(13464003)(24454002)(34854003)(42154003)(15975445007)(102836002)(40100003)(10400500002)(106356001)(62966003)(2950100001)(46102003)(66066001)(93886004)(5002640100001)(81156007)(5007970100001)(5001860100001)(2900100001)(2501003)(5001770100001)(5001960100002)(5004730100002)(5890100001)(76576001)(107886002)(189998001)(105586002)(92566002)(86362001)(5001830100001)(5001920100001)(74316001)(54356999)(77096005)(87936001)(50986999)(76176999)(19580395003)(19580405001)(68736005)(77156002)(122556002)(101416001)(5003600100002)(4001540100001)(33656002)(64706001)(11100500001)(97736004); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0650; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Sep 2015 12:02:32.7488 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0650 X-Microsoft-Exchange-Diagnostics: 1; HE1PR02MB0954; 2:7SwO1uPCvqkdxBzARnoH9q+rHVoiyeTAePJZZqvtZ/2/NXM3VasBS168/kHUXdp0w9zBO1O0DBcV31cg9qiucBcBfuxlWgYT7f7QpHGhS+InfGlQS1LOGcHziTszrbYZTSe+YHUwCQ9408DtS1uFlgpehsChQBdAbSFljbU43/c=; 23:/E3Gtp+KTxBJPDrQPoVbECjuriYN8bwh8n98a7rlqG8YaQjpPG7LtOQG3xXgZz9t5t3s783qfYSOoOyeaCT4g7khWVTTlAmZaPfNjrJJ6MSEQchxZwvNfntJJFIwK/6CewBdU5Phttvc8RhWEhzYDJ55G3FeBxCbIVxG4+5qr0/SvcRIB6f0oXSKVLnIZF+K X-OriginatorOrg: toganetworks.com Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2015 12:02:59 -0000 Hello Tom, Section 5.5 just specifies ordering rules of RDMAP requester and responder. According to such rules RDMAP generates RDMA messages and passes them to DD= P layer one by one for future processing. RDMAP's granularity and visibility is a particular RDMA message. DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP m= essages performs segmentation and passed them to transport layer. So DDP segments interleaving/rules cannot be part of RDMAP , it is clearly = DDP layer behavior. And according to RFC 5041 Section 5.3 and already mentioned by you "first r= ule" cannot interleave DDP segments of different operations. Best regards, Lena -----Original Message----- From: Tom Talpey [mailto:tom@talpey.com] Sent: Wednesday, September 09, 2015 6:54 PM To: Elena Gurevich; storm@ietf.org Subject: Re: [storm] DDP messages ordering On 9/9/2015 9:07 AM, Elena Gurevich wrote: > Hello, > > During my testing of some iWARP adaptor I discovered that RNIC interleave= s DDP segments of Send and Read Response messages. > According to your previous response this behavior is forbidden, is it ? Not necessarily, because an RDMA Read response is special. It's injected in= to the DDP stream within the RNIC by RDMAP, and therefore it follows the RD= MAP rules. See section 5.5 of RFC5040. Tom. > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey > Sent: Tuesday, September 08, 2015 5:15 PM > To: storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > On 9/8/2015 6:02 AM, Elena Gurevich wrote: >> Hello, >> >> RFC 5041 section 5.3 stands that >> >> ---------------------------------- >> >> 5.3 Ordering Among DDP Messages >> >> Messages passed through the DDP MUST conform to the ordering rules >> >> defined in this section. >> >> At the Data Source, DDP: >> >> * MUST transmit DDP Messages in the order they were submitted to >> >> the DDP layer, >> >> * SHOULD transmit DDP Segments within a DDP Message in increasing >> >> MO order for Untagged DDP Messages, and in increasing TO order >> >> for Tagged DDP Messages. >> >> ---------------------------------- >> >> Does this mean that transmitter MUST not interleave DDM segments >> related to consequent DDP messages ? > > It depends on what you mean by "the transmitter". It's certainly possible= that network delivery and TCP retransmission can reorder segments. But the= first rule requires that DDP not interleave two separate operations when p= assing them to the TCP transport. > > Tom. > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > ---------------------------------------------------------------------- > ---------------------------------------------------------------------- > ----- This email and any files transmitted and/or attachments with it > are confidential and proprietary information of Toga Networks Ltd., > and intended solely for the use of the individual or entity to whom they = are addressed. > If you have received this email in error please notify the system > manager. This message contains confidential information of Toga > Networks Ltd., and is intended only for the individual named. If you > are not the named addressee you should not disseminate, distribute or > copy this e-mail. Please notify the sender immediately by e-mail if > you have received this e-mail by mistake and delete this e-mail from your= system. If you are not the intended recipient you are notified that disclo= sing, copying, distributing or taking any action in reliance on the content= s of this information is strictly prohibited. > ---------------------------------------------------------------------- > ---------------------------------------------------------------------- > ---- > > > ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- From nobody Thu Sep 17 04:53:27 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8726C1B2D6D for ; Thu, 17 Sep 2015 04:53:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pNNi61-Ondsl for ; Thu, 17 Sep 2015 04:53:23 -0700 (PDT) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0645.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::645]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 32D691A6F2D for ; Thu, 17 Sep 2015 04:53:21 -0700 (PDT) Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) with Microsoft SMTP Server (TLS) id 15.1.268.17; Thu, 17 Sep 2015 11:53:01 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0268.017; Thu, 17 Sep 2015 11:53:01 +0000 From: Elena Gurevich To: Elena Gurevich , Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNgAI4rqAAC+dhtAABhopgAApL59wAWC74YA= Date: Thu, 17 Sep 2015 11:53:01 +0000 Message-ID: References: <55EEED81.2060508@talpey.com> <55F055FF.2050506@talpey.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [84.94.204.35] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0652; 5:sWwSG1Lcf+2T41yDPd0XkTQi2fmfNdCeZPmgPSjk8/ixR07+DJd8Kb6PCPs+fLfh0FTDWNntok0e16o3zD3D1bYW7rHUBQ6jwsh13LQauQc95jn2NpDGpT7Qp5Fc2RLEqp+kFlfCVrlR2NLfLtIxug==; 24:/2rumqbybPOd86BHCfe9kxK+sPZhTo2yPYkTP43k/sf8mGqWZqyvGtLzyO2NTbuOyo82ZQvHsJ8hd+XxlxyBg8JcfkrZD7yKH3OLZQS9JWU=; 20:jlaJTlYjynrQgpcnz6SkY9fP/slY6oyb53I/IladLNdoK/6OvZJUKBpMaqwZvKXpiqqSN6t1zdCs/6bP4bp8hw== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0652; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(8121501046)(520058)(520078)(5005006)(520075)(3002001); SRVR:HE1PR02MB0652; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0652; x-forefront-prvs: 07025866F6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(6009001)(13464003)(34854003)(42154003)(24454002)(377454003)(479174004)(199003)(189002)(15975445007)(102836002)(74316001)(189998001)(107886002)(50986999)(101416001)(76176999)(5890100001)(40100003)(106356001)(87936001)(86362001)(105586002)(10400500002)(2501003)(54356999)(77156002)(11100500001)(62966003)(93886004)(19580405001)(5007970100001)(66066001)(2950100001)(97736004)(68736005)(19580395003)(2900100001)(81156007)(5001860100001)(5001960100002)(122556002)(4001540100001)(5003600100002)(76576001)(5004730100002)(77096005)(46102003)(5001830100001)(5002640100001)(33656002)(5001920100001)(64706001)(5001770100001)(92566002)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0652; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: toganetworks.com X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Sep 2015 11:53:01.4530 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0652 Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Sep 2015 11:53:25 -0000 Hello Tom, Please reference exact place that explicitly or implicitly allows/forbids t= he behavior in question. Best regards, Lena -----Original Message----- From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Elena Gurevich Sent: Thursday, September 10, 2015 3:03 PM To: Tom Talpey; storm@ietf.org Subject: Re: [storm] DDP messages ordering Hello Tom, Section 5.5 just specifies ordering rules of RDMAP requester and responder. According to such rules RDMAP generates RDMA messages and passes them to DD= P layer one by one for future processing. RDMAP's granularity and visibility is a particular RDMA message. DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP m= essages performs segmentation and passed them to transport layer. So DDP segments interleaving/rules cannot be part of RDMAP , it is clearly = DDP layer behavior. And according to RFC 5041 Section 5.3 and already mentioned by you "first r= ule" cannot interleave DDP segments of different operations. Best regards, Lena -----Original Message----- From: Tom Talpey [mailto:tom@talpey.com] Sent: Wednesday, September 09, 2015 6:54 PM To: Elena Gurevich; storm@ietf.org Subject: Re: [storm] DDP messages ordering On 9/9/2015 9:07 AM, Elena Gurevich wrote: > Hello, > > During my testing of some iWARP adaptor I discovered that RNIC interleave= s DDP segments of Send and Read Response messages. > According to your previous response this behavior is forbidden, is it ? Not necessarily, because an RDMA Read response is special. It's injected in= to the DDP stream within the RNIC by RDMAP, and therefore it follows the RD= MAP rules. See section 5.5 of RFC5040. Tom. > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey > Sent: Tuesday, September 08, 2015 5:15 PM > To: storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > On 9/8/2015 6:02 AM, Elena Gurevich wrote: >> Hello, >> >> RFC 5041 section 5.3 stands that >> >> ---------------------------------- >> >> 5.3 Ordering Among DDP Messages >> >> Messages passed through the DDP MUST conform to the ordering rules >> >> defined in this section. >> >> At the Data Source, DDP: >> >> * MUST transmit DDP Messages in the order they were submitted to >> >> the DDP layer, >> >> * SHOULD transmit DDP Segments within a DDP Message in increasing >> >> MO order for Untagged DDP Messages, and in increasing TO order >> >> for Tagged DDP Messages. >> >> ---------------------------------- >> >> Does this mean that transmitter MUST not interleave DDM segments >> related to consequent DDP messages ? > > It depends on what you mean by "the transmitter". It's certainly possible= that network delivery and TCP retransmission can reorder segments. But the= first rule requires that DDP not interleave two separate operations when p= assing them to the TCP transport. > > Tom. > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > ---------------------------------------------------------------------- > ---------------------------------------------------------------------- > ----- This email and any files transmitted and/or attachments with it > are confidential and proprietary information of Toga Networks Ltd., > and intended solely for the use of the individual or entity to whom they = are addressed. > If you have received this email in error please notify the system > manager. This message contains confidential information of Toga > Networks Ltd., and is intended only for the individual named. If you > are not the named addressee you should not disseminate, distribute or > copy this e-mail. Please notify the sender immediately by e-mail if > you have received this e-mail by mistake and delete this e-mail from your= system. If you are not the intended recipient you are notified that disclo= sing, copying, distributing or taking any action in reliance on the content= s of this information is strictly prohibited. > ---------------------------------------------------------------------- > ---------------------------------------------------------------------- > ---- > > > ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solel= y for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and i= s intended only for the individual named. If you are not the named addresse= e you should not disseminate, distribute or copy this e-mail. Please notify= the sender immediately by e-mail if you have received this e-mail by mista= ke and delete this e-mail from your system. If you are not the intended rec= ipient you are notified that disclosing, copying, distributing or taking an= y action in reliance on the contents of this information is strictly prohib= ited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solel= y for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and i= s intended only for the individual named. If you are not the named addresse= e you should not disseminate, distribute or copy this e-mail. Please notify= the sender immediately by e-mail if you have received this e-mail by mista= ke and delete this e-mail from your system. If you are not the intended rec= ipient you are notified that disclosing, copying, distributing or taking an= y action in reliance on the contents of this information is strictly prohib= ited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= --------------------------------------------------------------------- From nobody Mon Sep 28 09:16:37 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 720541ACE34 for ; Mon, 28 Sep 2015 09:16:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.8 X-Spam-Level: X-Spam-Status: No, score=0.8 tagged_above=-999 required=5 tests=[BAYES_50=0.8, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xZ6j2krNqejn for ; Mon, 28 Sep 2015 09:16:35 -0700 (PDT) Received: from p3plsmtpa12-05.prod.phx3.secureserver.net (p3plsmtpa12-05.prod.phx3.secureserver.net [68.178.252.234]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0974B1ACE32 for ; Mon, 28 Sep 2015 09:16:35 -0700 (PDT) Received: from [192.168.0.59] ([24.218.177.82]) by p3plsmtpa12-05.prod.phx3.secureserver.net with id NgGZ1r00d1n35Pc01gGZb2; Mon, 28 Sep 2015 09:16:34 -0700 To: Elena Gurevich , "storm@ietf.org" References: <55EEED81.2060508@talpey.com> <55F055FF.2050506@talpey.com> From: Tom Talpey Message-ID: <560967E0.4030706@talpey.com> Date: Mon, 28 Sep 2015 12:16:32 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2015 16:16:36 -0000 I was traveling last week so this is a delayed response. I want to say that I am not answering in the role of STORM WG co-chair. This is simply my interpretation of the specifications. The behavior you observed is allowed by the RDMA Read processing rules. In RFC5040 section 4.5, the RDMA Read Response is defined to be simply a DDP message. The key is that the sender (i.e. the RDMA Read responder) can and does respond to a single RDMA Read with multiple Responses at the RDMAP layer. The "Last" flag, defined in RFC5041 section 4.1, is used to indicate the final Response in such a sequence. The RDMA Read Requestor simply processes the responses until the Last flag is received, at which point it completes the RDMA Read operation back to the application. Because each RDMA Read Response is a separate DDP message, it is perfectly legal for the Responder to interleave other DDP messages while processing the RDMA Read. There is no requirement that DDP pause the stream until the Last flag is sent. I'd actually expect most RNICs to do this for large RDMA Reads, for example if they are larger than a single Ethernet frame, or any other size if desired. Because the responding RNIC needs to perform multiple PCI Reads, and because RNICs don't usually buffer tagged data, it will be very convenient for them to issue interim RDMA Read Responses. Does this help? Tom. On 9/17/2015 7:53 AM, Elena Gurevich wrote: > Hello Tom, > > Please reference exact place that explicitly or implicitly allows/forbids the behavior in question. > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Elena Gurevich > Sent: Thursday, September 10, 2015 3:03 PM > To: Tom Talpey; storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > Hello Tom, > > Section 5.5 just specifies ordering rules of RDMAP requester and responder. > According to such rules RDMAP generates RDMA messages and passes them to DDP layer one by one for future processing. > RDMAP's granularity and visibility is a particular RDMA message. > DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP messages performs segmentation and passed them to transport layer. > So DDP segments interleaving/rules cannot be part of RDMAP , it is clearly DDP layer behavior. > And according to RFC 5041 Section 5.3 and already mentioned by you "first rule" cannot interleave DDP segments of different operations. > > Best regards, > Lena > > -----Original Message----- > From: Tom Talpey [mailto:tom@talpey.com] > Sent: Wednesday, September 09, 2015 6:54 PM > To: Elena Gurevich; storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > On 9/9/2015 9:07 AM, Elena Gurevich wrote: >> Hello, >> >> During my testing of some iWARP adaptor I discovered that RNIC interleaves DDP segments of Send and Read Response messages. >> According to your previous response this behavior is forbidden, is it ? > > Not necessarily, because an RDMA Read response is special. It's injected into the DDP stream within the RNIC by RDMAP, and therefore it follows the RDMAP rules. See section 5.5 of RFC5040. > > Tom. > >> >> Best regards, >> Lena >> >> -----Original Message----- >> From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey >> Sent: Tuesday, September 08, 2015 5:15 PM >> To: storm@ietf.org >> Subject: Re: [storm] DDP messages ordering >> >> On 9/8/2015 6:02 AM, Elena Gurevich wrote: >>> Hello, >>> >>> RFC 5041 section 5.3 stands that >>> >>> ---------------------------------- >>> >>> 5.3 Ordering Among DDP Messages >>> >>> Messages passed through the DDP MUST conform to the ordering rules >>> >>> defined in this section. >>> >>> At the Data Source, DDP: >>> >>> * MUST transmit DDP Messages in the order they were submitted to >>> >>> the DDP layer, >>> >>> * SHOULD transmit DDP Segments within a DDP Message in increasing >>> >>> MO order for Untagged DDP Messages, and in increasing TO order >>> >>> for Tagged DDP Messages. >>> >>> ---------------------------------- >>> >>> Does this mean that transmitter MUST not interleave DDM segments >>> related to consequent DDP messages ? >> >> It depends on what you mean by "the transmitter". It's certainly possible that network delivery and TCP retransmission can reorder segments. But the first rule requires that DDP not interleave two separate operations when passing them to the TCP transport. >> >> Tom. From nobody Mon Sep 28 23:18:38 2015 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 99C2D1A1AC6 for ; Mon, 28 Sep 2015 23:18:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gwlfg3y4SThk for ; Mon, 28 Sep 2015 23:18:35 -0700 (PDT) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0617.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe04::617]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B84461A1AC1 for ; Mon, 28 Sep 2015 23:18:34 -0700 (PDT) Received: from HE1PR02MB0651.eurprd02.prod.outlook.com (10.161.118.11) by HE1PR02MB0874.eurprd02.prod.outlook.com (10.161.118.24) with Microsoft SMTP Server (TLS) id 15.1.280.20; Tue, 29 Sep 2015 06:18:16 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com (10.161.118.12) by HE1PR02MB0651.eurprd02.prod.outlook.com (10.161.118.11) with Microsoft SMTP Server (TLS) id 15.1.280.20; Tue, 29 Sep 2015 06:18:12 +0000 Received: from HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) by HE1PR02MB0652.eurprd02.prod.outlook.com ([10.161.118.12]) with mapi id 15.01.0280.017; Tue, 29 Sep 2015 06:18:12 +0000 From: Elena Gurevich To: Tom Talpey , "storm@ietf.org" Thread-Topic: [storm] DDP messages ordering Thread-Index: AdDqHUWNk/8QkGT5SXeSvBvi6thoNgAI4rqAAC+dhtAABhopgAApL59wAWC74YACMmsnAAAchDmQ Date: Tue, 29 Sep 2015 06:18:11 +0000 Message-ID: References: <55EEED81.2060508@talpey.com> <55F055FF.2050506@talpey.com> <560967E0.4030706@talpey.com> In-Reply-To: <560967E0.4030706@talpey.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=elena.gurevich@toganetworks.com; x-originating-ip: [79.180.193.233] x-microsoft-exchange-diagnostics: 1; HE1PR02MB0651; 5:/HBP/RBNTcSsmsZX4K6ZVv18dN9zSfMcnLZMCx4P3NxSQ30RF0aNKjKvoIIvtWZFpyjsFQ75zXkAnb7wF7VouJ2HcA2rVA7gqG6tiHf7pMh7ia4F/i86RPOK79V2/MluSurw+FYgCs8vn5ffqAuQiQ==; 24:DjzbON8s/oenzQrrAf7XzeLBCGCW4EzdI8jP/kp86SPnj3S+6i0FGHF6CaWV5Z15MU6TV8Bwu+TJaF7xbBjQLqwgRzeMyVuOBar/YQoEGfk=; 20:TNM/NstKYPr33qzFFJQLO6uYz+sptzgwUbSyBQmzh8+m95EZv5titW76EwbzdDMkRTCmSMeqhuRs2+F9PIMiYQ== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0651; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(520078)(5005006)(3002001); SRVR:HE1PR02MB0651; BCL:0; PCL:0; RULEID:; SRVR:HE1PR02MB0651; x-forefront-prvs: 0714841678 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377454003)(34854003)(51444003)(13464003)(24454002)(42154003)(199003)(479174004)(189002)(92566002)(4001540100001)(33656002)(93886004)(86362001)(19580405001)(81156007)(97736004)(2900100001)(19580395003)(5001830100001)(40100003)(5007970100001)(5004730100002)(66066001)(122556002)(11100500001)(2950100001)(64706001)(5003600100002)(2501003)(10400500002)(5890100001)(189998001)(101416001)(54356999)(5001860100001)(76176999)(74316001)(105586002)(5001770100001)(50986999)(87936001)(76576001)(68736005)(102836002)(77096005)(77156002)(107886002)(62966003)(46102003)(106356001)(5001960100002)(5002640100001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR02MB0651; H:HE1PR02MB0652.eurprd02.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: toganetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Sep 2015 06:18:11.9297 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 73f7e7df-ca98-4f08-bf85-f137b447da96 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0651 X-Microsoft-Exchange-Diagnostics: 1; HE1PR02MB0874; 2:stGPrK8KNMNiyhcBJpPCxstrMOk58tNSAkr2vr15tVYUNsLvQBrpDr2D6i96r8UkKdKsc3UNRrONeNUnV3nhgXqs1Ra6GqV2WUuTVzM08G/ai5A9OXTJC8or39T1NUHed96sXgj9gmJkPatstP7MTUnh3Vb3FLV2s20Tx+s6RvI=; 23:k7rnJYNu/448+mfqWidGjUYzH7tkwHWMhj0lwvRSHEGmtKH0ZwtbkhwLcMfGNVduhuLIC9FtFXUPU1lUCL3dxXTr6I2Az+MHTtJGSLSSX2CBByfXJig/Y6GXu7YLM62OArLO5PpCrgcfhF8T4W6Viow4Pj7GLlM7Qs1wvvjqzkfbb+A2n6L3Dk9kEMGDxX7A X-OriginatorOrg: toganetworks.com Archived-At: Subject: Re: [storm] DDP messages ordering X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 06:18:37 -0000 Hi Tom, that you for your responses you are the only person that reacts to= my question :(. If we back to my question: according to RFC 5040 ------------------------------------------------------------- 5.2 RDMA Read Operation The RDMA Read operation MUST consist of a single RDMA Read Request Message = and a single RDMA Read Response Message. ------------------------------------------------------------- And "the last flag" you mentioned relates to DDP layer to sign last segment= of the same DDP message ( RDMA message is mapped one-by-one to DDP message= ). Personally I think that for transmitter it will be more simple to allow int= erleaving of DDP segments of different RDMA messages related to different t= ransmit queues: SQ and IRRQ, but for receiver this behavior complicates the= processing. So if we allow this behavior RFC 5041 must more accurate define this rule, = or I still miss something. Best regards, Elena -----Original Message----- From: Tom Talpey [mailto:tom@talpey.com] Sent: Monday, September 28, 2015 7:17 PM To: Elena Gurevich; storm@ietf.org Subject: Re: [storm] DDP messages ordering I was traveling last week so this is a delayed response. I want to say that= I am not answering in the role of STORM WG co-chair. This is simply my interpretation of the specifications. The behavior you observed is allowed by the RDMA Read processing rules. In = RFC5040 section 4.5, the RDMA Read Response is defined to be simply a DDP m= essage. The key is that the sender (i.e. the RDMA Read responder) can and d= oes respond to a single RDMA Read with multiple Responses at the RDMAP laye= r. The "Last" flag, defined in RFC5041 section 4.1, is used to indicate the final Response in such a seque= nce. The RDMA Read Requestor simply processes the responses until the Last = flag is received, at which point it completes the RDMA Read operation back = to the application. Because each RDMA Read Response is a separate DDP message, it is perfectly = legal for the Responder to interleave other DDP messages while processing t= he RDMA Read. There is no requirement that DDP pause the stream until the L= ast flag is sent. I'd actually expect most RNICs to do this for large RDMA Reads, for example= if they are larger than a single Ethernet frame, or any other size if desi= red. Because the responding RNIC needs to perform multiple PCI Reads, and b= ecause RNICs don't usually buffer tagged data, it will be very convenient f= or them to issue interim RDMA Read Responses. Does this help? Tom. On 9/17/2015 7:53 AM, Elena Gurevich wrote: > Hello Tom, > > Please reference exact place that explicitly or implicitly allows/forbids= the behavior in question. > > Best regards, > Lena > > -----Original Message----- > From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Elena > Gurevich > Sent: Thursday, September 10, 2015 3:03 PM > To: Tom Talpey; storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > Hello Tom, > > Section 5.5 just specifies ordering rules of RDMAP requester and responde= r. > According to such rules RDMAP generates RDMA messages and passes them to = DDP layer one by one for future processing. > RDMAP's granularity and visibility is a particular RDMA message. > DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP= messages performs segmentation and passed them to transport layer. > So DDP segments interleaving/rules cannot be part of RDMAP , it is clearl= y DDP layer behavior. > And according to RFC 5041 Section 5.3 and already mentioned by you "first= rule" cannot interleave DDP segments of different operations. > > Best regards, > Lena > > -----Original Message----- > From: Tom Talpey [mailto:tom@talpey.com] > Sent: Wednesday, September 09, 2015 6:54 PM > To: Elena Gurevich; storm@ietf.org > Subject: Re: [storm] DDP messages ordering > > On 9/9/2015 9:07 AM, Elena Gurevich wrote: >> Hello, >> >> During my testing of some iWARP adaptor I discovered that RNIC interleav= es DDP segments of Send and Read Response messages. >> According to your previous response this behavior is forbidden, is it ? > > Not necessarily, because an RDMA Read response is special. It's injected = into the DDP stream within the RNIC by RDMAP, and therefore it follows the = RDMAP rules. See section 5.5 of RFC5040. > > Tom. > >> >> Best regards, >> Lena >> >> -----Original Message----- >> From: storm [mailto:storm-bounces@ietf.org] On Behalf Of Tom Talpey >> Sent: Tuesday, September 08, 2015 5:15 PM >> To: storm@ietf.org >> Subject: Re: [storm] DDP messages ordering >> >> On 9/8/2015 6:02 AM, Elena Gurevich wrote: >>> Hello, >>> >>> RFC 5041 section 5.3 stands that >>> >>> ---------------------------------- >>> >>> 5.3 Ordering Among DDP Messages >>> >>> Messages passed through the DDP MUST conform to the ordering rules >>> >>> defined in this section. >>> >>> At the Data Source, DDP: >>> >>> * MUST transmit DDP Messages in the order they were submitted to >>> >>> the DDP layer, >>> >>> * SHOULD transmit DDP Segments within a DDP Message in increasing >>> >>> MO order for Untagged DDP Messages, and in increasing TO order >>> >>> for Tagged DDP Messages. >>> >>> ---------------------------------- >>> >>> Does this mean that transmitter MUST not interleave DDM segments >>> related to consequent DDP messages ? >> >> It depends on what you mean by "the transmitter". It's certainly possibl= e that network delivery and TCP retransmission can reorder segments. But th= e first rule requires that DDP not interleave two separate operations when = passing them to the TCP transport. >> >> Tom. ---------------------------------------------------------------------------= ---------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confide= ntial and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or en= tity to whom they are addressed. If you have received this email in error please notify the system manager. = This message contains confidential information of Toga Networks Ltd., and is intended only for the individual = named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Pleas= e notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mai= l from your system. If you are not the intended recipient you are notified that disclosing, copying, distribut= ing or taking any action in reliance on the contents of this information is strictly prohibited. ---------------------------------------------------------------------------= ---------------------------------------------------------------------