Re: [nfsv4] layout stateid handling/processing updates

["Mike Eisler" <mre-ietf at eisler.com>]

>> Let's pursue this notion a bit more though. As Jason points out,
>> technically it is the summation of i=1..N of  ca_maxrequests(i) *
>> (ca_maxoperations(i) - 1) where N is the number of forechannels
>> (the - 1 is because SEQUENCE uses a operation in a COMPOUND).
>> That however might be too large a number. Or it might not. It
>> depends on
>> the server (keeping in mind that this is a global limit per client
>> ID, not per client ID/fh pair). So the range of parallelism is
>> thus between summation of i=1..N ca_maxoperations(i) - 1,
>> and summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
>>
>> It is fair game for the client to attempt to play in this range, with
>> the understanding that the server MUST support summation of i=1..N
>> ca_maxoperations(i) - 1, and MAY support up to summation of i=1..N
>> (ca_maxoperations(i) - 1) * ca_requests(i).
>>
>> Of course,
>>  ca_maxrequestsize;
>>  ca_maxresponsesize;
>>  ca_maxresponsesize_cached;
>>
>> might limit the MAY further, because layout operations are not
>> exactly compact.
>>
>> But regardless server can limit the range by simply
>> returning NFS4ERR_DELAY on a layout operation that
>> returns a layout stateid.

[...]
>> I am having trouble imagining servers that support 10 million
>> outstanding, parallel layout operations. But I think we are
>> talking about four separate concepts:
>>
>> 1. the minimum number of parallel layout operations a client
>>    MUST be allowed
>>
>> 2. the maximum number a parallel layout operations a server MAY
>>    allow
>>
>> 3. what values of seqid get _BAD_STATEID returned?
>>
>> 4. what values of seqid get _OLD_STATEID returned?
>>
>>
>> I described 1 and 2.
>
> Note that parallel in the above definition is the current over-the-wire
> parallelism.  In the case of LAYOUT processing, the specification speaks
> of fully processed responses and requests.  Allowing for flexibility
> in the client implementation, the client may have reused a slot
> for the next request without fully processing the response to
> LAYOUTGET/LAYOUTRETURN.  Technically, the range of parallelism is
> expanded beyond that of the calculations based on the session's
> parameters.

OK, so the range of parallelism is not
between summation of i=1..N ca_maxoperations(i) - 1,
and summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i). Instead
it is between

MIN( (summation of i=1..N ca_maxoperations(i) - 1), 2 ^ 32 - 1 )

   and

2 ^ 32 - 1

E.g., let's say client issues a LAYOUTGET after an OPEN, and
gets a layout stateid with seqid set to 1. The client then
issues 2^32 parallel LAYOUTGETs, each with the layout stateid with seqid
set to 1.

The server has to return a unique seqid for all the LAYOUTGETs.
After sending a reply with with the seqid set to 2^32 - 1, the next
reply has to have seqid set to 1. The server cannot do that because
the client has not fully processed the reply that has seqid set to 2,
so seqid = 1 is not available. So
the server has to return NFS4ERR_DELAY. The client has just learned the
server's current value of PAST_SEQ.

So nothing fundamental is altered in my proposal.

>
>> I think 3 and 4 determined by the number of seqids in the
>> past. This number, call it, PAST_SEQ, is determined by the maximum
>> value of
>> summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
>> that has ever existed on the client ID. This allows for additional
>> fore channels to be added to the client ID and then deleted without
>> introducing problems, as well as the highest slot number on a fore
>> channel
>> to be increased and reduced on a server.
>>
>> Of course, a server implementer might decide: "to heck with that,
>> I'm just going to set PAST_SEQ to 10 million." Fine, as long as it
>> is never
>> less than maximum value of
>>   summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
>>
>> I willing to believe that the "to heck with that" server dude
>> can make that guarantee by doing stuff like limiting the maximum
>> number
>> of fore channels to 100, and the maximum number of slots per
>> fore channel to 100. 10K is much less than 1M, and 10K slots is
>> probably enough for most MDS clients. Though I think THWT ought to
>> be more elegant with his #define for PAST_SEQ.
>>
>>> The major point is that the server should be be very broad in
>>> its interpretation of what is in the past to allow for the
>>> parallelism.
>>
>> But the point that trumps it is that one server's definition of
>> broad will inevitably be at odds with that of one client's. Since
>> we don't want to codify 1M in the spec, and instead
>> can deterministically define PAST_SEQ, and without .x changes,
>> we should do it.
>
> Even with my statement above, I agree that defining PAST_SEQ
> is useful.  That value can then be used in describing the
> processing of layout stateid.seqid values and defining
> the client's recovery from receipt of OLD_STATEID (client
> has exceeded degree of allowable parallelism).

OK. New text coming soon?

_______________________________________________
nfsv4 mailing list
nfsv4 at ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4