idnits 2.17.1 draft-ietf-nfsv4-migration-issues-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 21, 2013) is 4026 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) == Outdated reference: A later version (-35) exists of draft-ietf-nfsv4-rfc3530bis-25 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 D. Noveck, Ed. 3 Internet-Draft EMC 4 Intended status: Informational P. Shivam 5 Expires: September 22, 2013 C. Lever 6 B. Baker 7 ORACLE 8 March 21, 2013 10 NFSv4 migration: Implementation experience and spec issues to resolve 11 draft-ietf-nfsv4-migration-issues-03 13 Abstract 15 The migration feature of NFSv4 provides for moving responsibility for 16 a single filesystem from one server to another, without disruption to 17 clients. Recent implementation experience has shown problems in the 18 existing specification for this feature. This document discusses the 19 issues which have arisen, explores the options available for curing 20 the issues, and explains the choices made in updating the NFSv4.0 and 21 NFSv4.1 specifications, to address migration. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on September 22, 2013. 40 Copyright Notice 42 Copyright (c) 2013 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. NFSv4.0 Implementation Experience . . . . . . . . . . . . . . 4 60 3.1. Implementation issues . . . . . . . . . . . . . . . . . . 4 61 3.1.1. Failure to free migrated state on client reboot . . . 4 62 3.1.2. Server reboots resulting in a confused lease 63 situation . . . . . . . . . . . . . . . . . . . . . . 5 64 3.1.3. Client complexity issues . . . . . . . . . . . . . . 6 65 3.2. Sources of Protocol difficulties . . . . . . . . . . . . 8 66 3.2.1. Issues with nfs_client_id4 generation and use . . . . 8 67 3.2.2. Issues with lease proliferation . . . . . . . . . . . 9 68 4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . 10 69 4.1. Possible changes to nfs_client_id4 client-string . . . . 10 70 4.2. Possible changes to handle differing nfs_client_id4 71 string values . . . . . . . . . . . . . . . . . . . . . . 11 72 4.3. Other issues within migration-state sections . . . . . . 12 73 4.4. Issues within other sections . . . . . . . . . . . . . . 12 74 5. Proposed resolution of NFSv4.0 protocol difficulties . . . . 13 75 5.1. Proposed changes: nfs_client_id4 client-string . . . . . 13 76 5.2. Proposed changes: merged (vs. synchronized) leases . . . 13 77 5.3. Other proposed changes to migration-state sections . . . 15 78 5.3.1. Proposed changes: Client ID migration . . . . . . . . 15 79 5.3.2. Proposed changes: Callback re-establishment . . . . . 16 80 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . 16 81 5.4. Proposed changes to other sections . . . . . . . . . . . 17 82 5.4.1. Proposed changes: callback update . . . . . . . . . . 17 83 5.4.2. Proposed changes: clientid4 handling . . . . . . . . 17 84 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE . . . . . . . . 19 85 6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 19 86 6.1. Results: Failure to free migrated state on client reboot 20 87 6.2. Results: Server reboots resulting in confused lease 88 situation . . . . . . . . . . . . . . . . . . . . . . . . 20 89 6.3. Results: Client complexity issues . . . . . . . . . . . . 22 90 6.4. Result summary . . . . . . . . . . . . . . . . . . . . . 23 91 7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 23 92 7.1. Addressing state merger in NFSv4.1 . . . . . . . . . . . 23 93 7.2. Addressing pNFS relationship with migration . . . . . . . 24 94 7.3. Addressing server owner changes in NFSv4.1 . . . . . . . 24 95 8. Security Considerations . . . . . . . . . . . . . . . . . . . 26 96 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 97 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 98 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 99 11.1. Normative References . . . . . . . . . . . . . . . . . . 26 100 11.2. Informative References . . . . . . . . . . . . . . . . . 27 101 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 103 1. Introduction 105 This document is in the informational category, and while the facts 106 it reports may have normative implications, any such normative 107 significance reflects the readers' preferences. For example, we may 108 report that the reboot of a client with migrated state results in 109 state not being promptly cleared and that this will prevent granting 110 of conflicting lock requests at least for the lease time, which is a 111 fact. While it is to be expected that client and server implementers 112 will judge this to be a situation that is best avoided, the judgment 113 as to how pressing this issue should be considered is a judgment for 114 the reader, and eventually the nfsv4 working group to make. 116 We do explore possible ways in which such issues can be avoided, with 117 minimal negative effects, given that the working group has decided to 118 address these issues, but the choice of exactly how to address these 119 is best given effect in one or more standards-track documents and/or 120 errata. 122 This document focuses on NFSv4.0, since that is where the majority of 123 implementation experience has been. Nevertheless, there is 124 discussion of the implications of the NFSv4.0 experience for 125 migration in NFSv4.1, as well as discussion of other issues with 126 regard to the treatment of migration in NFSv4.1. 128 2. Conventions 130 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 131 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 132 document are to be interpreted as described in [RFC2119]. 134 In the context of this informational document, these normative 135 keywords will always occur in the context of a quotation, most often 136 direct but sometimes indirect. The context will make it clear 137 whether the quotation is from: 139 o The current definitive definition of the NFSv4.0 protocol, whether 140 that is the original NFSv4.0 specification [RFC3530], or its 141 expected successor [RFC3530bis]. 143 As the identity of that document may change during the lifetime of 144 this document, we will often refer to the current or pending 145 definition of NFSv4.0 and quote from portions of the documents 146 that are identical among all existing drafts. Given that RFC3530 147 and all RFC3530bis drafts agree as to the issues under discussion, 148 this should not cause undue difficulty. Note that to simplify 149 document maintenance, section names rather than section numbers 150 are used when referring to sections in existing documents so that 151 only minimal changes will be necessary as the identity of the 152 document defining NFSv4.0 changes. 154 o The current definitive definition of the NFSv4.1 protocol 155 [RFC5661]. 157 o A proposed or possible text to serve as a replacement for the 158 current definitive document text. Sometimes, a number of possible 159 alternative texts may be listed and benefits and detriments of 160 each examined in turn. 162 3. NFSv4.0 Implementation Experience 164 3.1. Implementation issues 166 Note that the examples below reflect current experience which arises 167 from clients implementing the recommendation to use different 168 nfs_client_id4 id strings for different server addresses, i.e. using 169 what is later referred to herein as the "non-uniform client-string 170 approach." 172 This is simply because that is the experience implementers have had. 173 The reader should not assume that in all cases, this practice is the 174 source of the difficulty. It may be so in some cases but clearly it 175 is not in all cases. 177 3.1.1. Failure to free migrated state on client reboot 179 The following sort of situation has proved troublesome: 181 o A client C establishes a clientid4 C1 with server ABC specifying 182 an nfs_client_id4 with id string value "C-ABC" and boot verifier 183 0x111. 185 o The client begins to access files in filesystem F on server ABC, 186 resulting in generating stateids S1, S2, etc. under the lease for 187 clientid C1. It may also access files on other filesystems on the 188 same server. 190 o The filesystem is migrated from server ABC to server XYZ. When 191 transparent state migration is in effect, stateids S1 and S2 and 192 clientid4 C1 are now available for use by client C at server XYZ. 194 o Client C reboots and attempts to access data on server XYZ, 195 whether in filesystem F or another. It does a SETCLIENTID with an 196 nfs_client_id4 with id string value "C-XYZ" and boot verifier 197 0x112. There is thus no occasion to free stateids S1 and S2 since 198 they are associated with a different client name and so lease 199 expiration is the only way that they can be gotten rid of. 201 Note here that while it seems clear to us in this example that C-XYZ 202 and C-ABC are from the same client, the server has no way to 203 determine the structure of the "opaque" id string. In the protocol, 204 it really is treated as opaque. Only the client knows which 205 nfs_client_id4 values designate the same client on a different 206 server. 208 3.1.2. Server reboots resulting in a confused lease situation 210 Further problems arise from scenarios like the following. 212 o Client C talks to server ABC using an nfs_client_id4 id string 213 such as "C-ABC" and a boot verifier v1. As a result, a lease with 214 clientid4 c.i is established: {v1, "C-ABC", c.i}. 216 o fs_a1 migrates from server ABC to server XYZ along with its state. 217 Now server XYZ also has a lease: {v1, "C-ABC", c.i}. 219 o Server ABC reboots. 221 o Client C talks to server ABC using an nfs_client_id4 id string 222 such as "C-ABC" and a boot verifier v1. As a result, a lease with 223 clientid4 c.j is established: {v1, "C-ABC", c.j}. 225 o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also 226 has a lease: {v1, "C-ABC", c.j}. 228 o Now server XYZ has two leases that match {v1, "C-ABC", *}, when 229 the protocol clearly assumes there can be only one. 231 Note that if the client used "C" (rather than "C-ABC") as the 232 nfs_client_id4 id string, the exact same situation would arise. 234 One of the first cases in which this sort of situation has resulted 235 in difficulties is in connection with doing a SETCLIENTID for 236 callback update. 238 The SETCLIENTID for callback update only includes the nfs_client_id4, 239 assuming there can only be one such with a given nfs_client_id4 240 value. If there were multiple, confirmed client records with 241 identical nfs_client_id4 id string values, there would be no way to 242 map the callback update request to the correct client record. Apart 243 from the migration handling specified in [RFC3530] and [RFC3530bis], 244 such a situation cannot arise. 246 One possible accommodation for this particular issue that has been 247 used is to add a RENEW operation along with SETCLIENTID (on a 248 callback update) to disambiguate the client. 250 When the client updates the callback info to the destination, the 251 client would, by convention, send a compound like this: 253 { RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb } 255 The presence of the clientid4 in the compound would allow the server 256 to differentiate among the various leases that it knows of, all with 257 the same nfs_client_id4 value. 259 While this would be a reasonable patch for an isolated protocol 260 weakness, interoperable clients and servers would require that the 261 protocol truly be updated to allow such a situation, specifically 262 that of multiple clientid4's with the same nfs_client_id4 value. The 263 protocol is currently designed and implemented assuming this cannot 264 happen. We need to either prevent the situation from happening, or 265 fully adapt to the possibilities which can arise. See Section 4 for 266 a discussion of such issues. 268 3.1.3. Client complexity issues 270 Consider the following situation: 272 o There are a set of clients C1 through Cn accessing servers S1 273 through Sm. Each server manages some significant number of 274 filesystems with the filesystem count L being significantly 275 greater than m. 277 o Each client Cx will access a subset of the servers and so will 278 have up to m clientids, which we will call Cxy for server Sy. 280 o Now assume that for load-balancing or other operational reasons, 281 numbers of filesystems are migrated among the servers. As a 282 result, each client-server pair will have up to m clientids and 283 each client will have up to m**2 clientids. If we add the 284 possibility of server reboot, the only bound on a client's 285 clientid count is L. 287 Now, instead of a clientid4 identifying a client-server pair, we have 288 many more entities for the client to deal with. In addition, it 289 isn't clear how new state is to be incorporated in this structure. 291 The limitations of the migrated state (inability to be freed on 292 reboot) would argue against adding more such state but trying to 293 avoid that would run into its own difficulties. For example, a 294 single lockowner string presented under two different clientids would 295 appear as two different entities. 297 Thus we have to choose between: 299 o indefinite prolongation of foreign clientids even after all 300 transferred state is gone. 302 o having multiple requests for the same lockowner-string-named 303 entity carried on in parallel by separate identically named 304 lockowners under different clientid4's 306 o Adding serialization at the lock-owner string level, in addition 307 to that at the lockowner level. 309 In any case, we have gone (in adding migration as it was described) 310 from a situation in which 312 o Each client has a single clientid4/lease for each server it talks 313 to. 315 o Each client has a single nfs_client_id4 for each server it talks 316 to. 318 o Every state id can be mapped to an associated lease based on the 319 server it was obtained from. 321 To one in which 323 o Each client may have multiple clientid4's for a single server. 325 o For each stateid, the client must separately record the clientid4 326 that it is assigned to, or it must manage separate "state blobs" 327 for each fsid and map those to clientid4's. 329 o Before doing an operation that can result in a stateid, the client 330 must either find a "state blob" based on fsid or create a new one, 331 possibly with a new clientid4. 333 o There may be multiple clientid4's all connected to the same server 334 and using the same nfs_clientid4. 336 This sort of additional client complexity is troublesome and needs to 337 be eliminated. 339 3.2. Sources of Protocol difficulties 341 3.2.1. Issues with nfs_client_id4 generation and use 343 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 344 and [RFC3530bis] both agree. The section entitled "Client ID" says: 346 The second field, id is a variable length string that uniquely 347 defines the client. 349 There are two possible interpretations of the phrase "uniquely 350 defines" in the above: 352 o The relation between strings and clients is a function from such 353 strings to clients so that each string designates a single client. 355 o The relation between strings and clients is a bijection between 356 such strings and clients so that each string designates a single 357 client and each client is named by a single string. 359 The first interpretation would make these client-strings like phone 360 numbers (a single person can have several) while the second would 361 make them like social security numbers. 363 Endless debate about the true meaning of "uniquely defines" in this 364 context is quite possible but not very helpful. The following points 365 should be noted though: 367 o The second interpretation is more consistent with the way 368 "uniquely defines" is used elsewhere in the spec. 370 o The spec as now written intends the first interpretation (or is 371 internally inconsistent). In fact, it recommends, although it 372 doesn't "RECOMMEND" that a single client have at least as many 373 client-strings as server addresses that it interacts with. It 374 says, in the third bullet point regarding construction of the 375 string (which we shall henceforth refer to as client-string-BP3): 377 The string should be different for each server network address 378 that the client accesses, rather than common to all server 379 network addresses. 381 o If internode interactions are limited to those between a client 382 and its servers, there is no occasion for servers to be concerned 383 with the question of whether two client-strings designate the same 384 client, so that there is no occasion for the difference in 385 interpretation to matter. 387 o When transparent migration of client state occurs between two 388 servers, it becomes important to determine when state on two 389 different servers is for the same client or not, and this 390 distinction becomes very important. 392 Given the need for the server to be aware of client identity with 393 regard to migrated state, either client-string construction rules 394 will have to change or there will be a need to get around current 395 issues, or perhaps a combination of these two will be required. 396 Later sections will examine the options and propose a solution. 398 One consideration that may indicate that this cannot remain exactly 399 as it is today has to do with the fact that the current explanation 400 for this behavior is not correct. The current definitive definitions 401 of the NFSv4.0 protocol, [RFC3530] and [RFC3530bis] both agree. The 402 section entitled "Client ID" says: 404 The reason is that it may not be possible for the client to tell 405 if the same server is listening on multiple network addresses. If 406 the client issues SETCLIENTID with the same id string to each 407 network address of such a server, the server will think it is the 408 same client, and each successive SETCLIENTID will cause the server 409 to begin the process of removing the client's previous leased 410 state. 412 In point of fact, a "SETCLIENTID with the same id string" sent to 413 multiple network addresses will be treated as all from the same 414 client but will not "cause the server to begin the process of 415 removing the client's previous leased state" unless the server 416 believes it is a different instance of the same client, i.e. if the 417 id string is the same and there is a different boot verifier. If the 418 client does not reboot, the verifier should not change. If it does 419 reboot, the verifier will change, and the server should "begin the 420 process of removing the client's previous leased state. 422 The situation of multiple SETCLIENTID requests received by a server 423 on multiple network addresses is exactly the same, from the protocol 424 design point of view, as when multiple (i.e. duplicate) SETCLIENTID 425 requests are received by the server on a single network address. The 426 same protocol mechanisms that prevent erroneous state deletion in the 427 latter case prevent it in the former case. There is no reason for 428 special handling of the multiple-network-appearance case, in this 429 regard. 431 3.2.2. Issues with lease proliferation 433 It is often felt that this is a consequence of the client-string 434 construction issues, and it is certainly the case that the two are 435 closely connected in that non-uniform client-strings make it 436 impossible for the server to appropriately combine leases from the 437 same client. 439 However, even where the server could combine leases from the same 440 client, it needs to be clear how and when it will do so, so that the 441 client will be prepared. These issues will have to be addressed at 442 various places in the spec. 444 This could be enough only if we are prepared to do away with the 445 "should" recommending non-uniform client-strings and replace it with 446 a "should not" or even a "SHOULD NOT". Current client implementation 447 patterns make this an unpalatable choice for use as a general 448 solution, but it is reasonable to "RECOMMEND" this choice for a well- 449 defined subset of clients. One alternative would be to create a way 450 for the server to infer from client behavior which leases are held by 451 the same client and use this information to do appropriate lease 452 mergers. Prototyping and detailed specification work has shown that 453 this could be done but the resulting complexity is such that a better 454 choice is to "RECOMMEND" use of the uniform client-string approach 455 for clients supporting the migration feature. 457 Because of the discussion of client-string construction in [RFC3530] 458 and [RFC3530bis], most existing clients implement the non-uniform 459 client-string approach. As a result, existing servers may not have 460 been tested with clients implementing uniform client-strings. As a 461 consequence, care must be taken to preserve interoperability between 462 UCS-capable clients and servers that don't tolerate uniform client 463 strings for one reason or another. 465 4. Issues to be resolved in NFSv4.0 467 4.1. Possible changes to nfs_client_id4 client-string 469 The fact that the reason given in client-string-BP3 is not valid 470 makes the existing "should" insupportable. We can't either 472 o Keep a reason we know is invalid. 474 o Keep saying "should" without giving a reason. 476 What are often presented as reasons that motivate use of the non- 477 uniform approach always turn out to be cases in which, if the uniform 478 approach were used, the server will treat a client which accesses 479 that server via two different IP addresses as part of a single 480 client, as it in fact is. This may be disconcerting to a client 481 unaware that the two IP addresses connect to the same server. This 482 is not a reason to use the non-uniform approach but is better thought 483 of as an illustration of the fact that those using the uniform 484 approach need to be aware of the possibility of server trunking and 485 its effect on server behavior. 487 If it is possible to reliably infer the existence of trunking of 488 server IP addresses from observed server behavior, use of the uniform 489 approach would be more desirable, although compatibility issues would 490 have to be dealt with. 492 It is always possible that a valid new reason will be found, but so 493 far none has been proposed. Given the history, the burden of proof 494 should be on those asserting the validity of a proposed new reason. 496 So we will assume for now that the "should" will have to go. The 497 question is what to replace it with. 499 o We can't say "MUST NOT", despite the problems this raises for 500 migration since this is pretty late in the day for such a change. 501 Many currently operating clients obey the existing "should". 502 Similar considerations would apply for "SHOULD NOT" or "should 503 not". 505 o Dropping client-string-BP3 entirely is a possibility but, given 506 the context and history, it would just be a confusing version of 507 "SHOULD NOT". 509 o Using "MAY" would clearly specify that both ways of doing this are 510 valid choices for clients and that servers will have to deal with 511 clients that make either choice. 513 o This might be modified by a "SHOULD" (or even a "MUST") for 514 particular groups of clients. 516 o There will have to be some text explaining why a client might make 517 either choice but, except for the particular cases referred to 518 above, we will have to make sure that it is truly descriptive, and 519 not slanted in either direction. 521 4.2. Possible changes to handle differing nfs_client_id4 string values 523 Given the difficulties caused by having different nfs_client_id4 524 client-string values for the same client, we have two choices: 526 o Deprecate the existing treatment and basically say the client is 527 on its own doing migration, if it follows it. 529 o Introduce a way of having the client provide client identity 530 information to the server, if it can be done compatibly while 531 staying within the bounds of v4.0. 533 4.3. Other issues within migration-state sections 535 There are a number of issues where the existing text is unclear and/ 536 or wrong and needs to be fixed in some way. 538 o Lack of clarity in the discussion of moving clientids (as well as 539 stateids) as part of moving state for migration. 541 o The discussion of synchronized leases is wrong in that there is no 542 way to determine (in the current spec) when leases are for the 543 same client and also wrong in suggesting a benefit from leases 544 synchronized at the point of transfer. What is needed is merger 545 of leases, which is necessary to keep client complexity 546 requirements from getting out of hand. 548 o Lack of clarity in the discussion of LEASE_MOVED handling, 549 including failure to fully address situations in which transparent 550 state migration did not occur. 552 4.4. Issues within other sections 554 There are a number of cases in which certain sections, not 555 specifically related to migration, require additional clarification. 556 This is generally because text that is clear in a context in which 557 leases and clientids are created in one place and live there forever 558 may need further refinement in the more dynamic environment that 559 arises as part of migration. 561 Some examples: 563 o Some people are under the impression that updating callback 564 endpoint information for an existing client, as used during 565 migration, may cause the destination server to free existing 566 state. There need to be additions to clarify the situation. 568 o The handling of the sets of clientid4's maintained by each server 569 needs to be clarified. In particular, the issue of how the client 570 adapts to the presumably independent and uncoordinated clientid4 571 sets needs to be clearly addressed 573 o Statements regarding handling of invalid clientid4's need to be 574 clarified and/or refined in light of the possibilities that arise 575 due to lease motion and merger. 577 o Confusion and lack of clarity about NFS4ERR_CLID_INUSE. 579 5. Proposed resolution of NFSv4.0 protocol difficulties 581 This section lists the changes which we believe are necessary to 582 resolve the difficulties mentioned above. Such change, along with 583 other clarifications found to be desirable during drafting and review 584 are contained in [migr-v4.0-update]. 586 5.1. Proposed changes: nfs_client_id4 client-string 588 We propose replacing client-string-BP3 with the following text and 589 adding the following proposed to provide implementation guidance. 591 The string MAY be different for each server network address that 592 the client accesses, rather than common to all server network 593 addresses. 595 In addition, given the importance of the issue of client identity and 596 the fact that both client string-approaches are to be considered 597 valid, a greatly expanded treatment of client identity desirable. It 598 should have the following major elements. 600 o It should fully describe the consequences of making the string 601 different for each network address (the non-uniform client-string 602 approach) and of making it the same for all network addresses (the 603 uniform client string approach). 605 o It should give helpful guidance about the factors that might 606 affect client implementation choice between these approaches. 608 o It should describe the compatibility issues that might cause 609 servers to be incompatible with the uniform approach and give 610 guidance about dealing with these. 612 o It should describe how a client using the uniform approach might 613 use server behavior to determine server address trunking patterns. 615 o It should present a clearer and more complete set of 616 recommendations to guide client string construction. 618 5.2. Proposed changes: merged (vs. synchronized) leases 620 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 621 and [RFC3530bis] both agree. The section entitled "Migration and 622 State" says: 624 As part of the transfer of information between servers, leases 625 would be transferred as well. The leases being transferred to the 626 new server will typically have a different expiration time from 627 those for the same client, previously on the old server. To 628 maintain the property that all leases on a given server for a 629 given client expire at the same time, the server should advance 630 the expiration time to the later of the leases being transferred 631 or the leases already present. This allows the client to maintain 632 lease renewal of both classes without special effort: 634 There are a number of problems with this and any resolution of our 635 difficulties must address them somehow. 637 o The current v4.0 spec recommends that the client make it 638 essentially impossible to determine when two leases are from "the 639 same client". 641 o It is not appropriate to speak of "maintain[ing] the property that 642 all leases on a given server for a given client expire at the same 643 time", since this is not a property that holds even in the absence 644 of migration. A server listening on multiple network addresses 645 may have the same client appear as multiple clients with no way to 646 recognize the client as the same. 648 o Even if the client identity issue could be resolved, advancing the 649 lease time at the point of migration would not maintain the 650 desired synchronization property. The leases would be 651 synchronized until one of them was renewed, after which they would 652 be unsynchronized again. 654 To avoid client complexity, we need to have no more than one lease 655 between a single client and a single server. This requires merger of 656 leases since there is no real help from synchronizing them at a 657 single instant. 659 For the uniform approach, the destination server would simply merge 660 leases as part of state transfer, since two leases with the same 661 nfs_client_id4 values must be for the same client. 663 We have made the following decisions as far as proposed normative 664 statements regarding for state merger. They reflect the facts that 665 we want to support fully migration support in the simplest way 666 possible and that we can't say MUST since we have older clients and 667 servers to deal with. 669 o Clients SHOULD use the uniform client-string approach in order to 670 get good migration support. 672 o Servers SHOULD provide automatic lease merger during state 673 migration so that clients using the uniform id approach get the 674 support automatically. 676 If the clients and the servers obey the SHOULD's, having more than a 677 single lease for a given client-server pair will be a transient 678 situation, cleaned up as part of adapting to use of migrated state. 680 Since clients and servers will be a mixture of old and new and 681 because nothing is a MUST we have to ensure that no combination will 682 show worse behavior than is exhibited by current (i.e. old) clients 683 and servers. 685 5.3. Other proposed changes to migration-state sections 687 5.3.1. Proposed changes: Client ID migration 689 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 690 and [RFC3530bis] both agree. The section entitled "Migration and 691 State" says: 693 In the case of migration, the servers involved in the migration of 694 a filesystem SHOULD transfer all server state from the original to 695 the new server. This must be done in a way that is transparent to 696 the client. This state transfer will ease the client's transition 697 when a filesystem migration occurs. If the servers are successful 698 in transferring all state, the client will continue to use 699 stateids assigned by the original server. Therefore the new 700 server must recognize these stateids as valid. This holds true 701 for the client ID as well. Since responsibility for an entire 702 filesystem is transferred with a migration event, there is no 703 possibility that conflicts will arise on the new server as a 704 result of the transfer of locks. 706 This poses some difficulties, mostly because the part about "client 707 ID" is not clear: 709 o It isn't clear what part of the paragraph the "this" in the 710 statement "this holds true ..." is meant to signify. 712 o The phrase "the client ID" is ambiguous, possibly indicating the 713 clientid4 and possibly indicating the nfs_client_id4. 715 o If the text means to suggest that the same clientid4 must be used, 716 the logic is not clear since the issue is not the same as for 717 stateids of which there might be many. Adapting to the change of 718 a single clientid, as might happen as a part of lease migration, 719 is relatively easy for the client. 721 We have decided that it is best to address this issue as follows: 723 o Make it clear that both clientid4 and nfs_client_id4 (including 724 both id string and boot verifier) are to be transferred. 726 o Indicate that the initial transfer will result in the same 727 clientid4 after transfer but this is not guaranteed since there 728 may conflict with an existing clientid4 on the destination server 729 and because lease merger can result in a change of the clientid4. 731 5.3.2. Proposed changes: Callback re-establishment 733 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 734 and [RFC3530bis] both agree. The section entitled "Migration and 735 State" says: 737 A client SHOULD re-establish new callback information with the new 738 server as soon as possible, according to sequences described in 739 sections "Operation 35: SETCLIENTID - Negotiate Client ID" and 740 "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID". This 741 ensures that server operations are not blocked by the inability to 742 recall delegations. 744 The above will need to be fixed to reflect the possibility of merging 745 of leases, 747 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework 749 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 750 and [RFC3530bis] both agree. The section entitled "Notification of 751 Migrated Lease" says: 753 Upon receiving the NFS4ERR_LEASE_MOVED error, a client that 754 supports filesystem migration MUST probe all filesystems from that 755 server on which it holds open state. Once the client has 756 successfully probed all those filesystems which are migrated, the 757 server MUST resume normal handling of stateful requests from that 758 client. 760 There is a lack of clarity that is prompted by ambiguity about what 761 exactly probing is and what the interlock between client and server 762 must be. This has led to some worry about the scalability of the 763 probing process, and although the time required does scale linearly 764 with the number of filesystems that the client may have state for 765 with respect to a given server, the actual process can be done 766 efficiently. 768 To address these issues we propose rewriting the above to be more 769 clear and to give suggestions about how to do the required scanning 770 efficiently. 772 5.4. Proposed changes to other sections 774 5.4.1. Proposed changes: callback update 776 Some changes are necessary to reduce confusion about the process of 777 callback information update and in particular to make it clear that 778 no state is freed as a result: 780 o Make it clear that after migration there are confirmed entries for 781 transferred clientid4/nfs_client_id4 pairs. 783 o Be explicit in the sections headed "otherwise," in the 784 descriptions of SETCLIENTID and SETCLIENTID_CONFIRM, that these 785 don't apply in the cases we are concerned about. 787 5.4.2. Proposed changes: clientid4 handling 789 To address both of the clientid4-related issues mentioned in 790 Section 4.4, we propose replacing the last three paragraphs of the 791 section entitled "Client ID" with the following: 793 Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has 794 successfully completed, the client uses the shorthand client 795 identifier, of type clientid4, instead of the longer and less 796 compact nfs_client_id4 structure. This shorthand client 797 identifier (a client ID) is assigned by the server and should be 798 chosen so that it will not conflict with a client ID previously 799 assigned by same server. This applies across server restarts or 800 reboots. 802 Distinct servers MAY assign clientid4's independently, and will 803 generally do so. Therefore, a client has to be prepared to deal 804 with multiple instances of the same clientid4 value received on 805 distinct IP addresses, denoting separate entities. When trunking 806 of server IP addresses is not a consideration, a client should 807 keep track of (IP-address, clientid4) pairs, so that each pair is 808 distinct. In the face of possible trunking of server IP 809 addresses, the client will use the receipt of the same clientid4 810 from multiple IP-addresses, as an indication that the two IP- 811 addresses may be trunked and proceed to determine, from the 812 observed server behavior whether the two addresses are in fact 813 trunked. 815 When a clientid4 is presented to a server and that clientid4 is 816 not recognized, the server will reject the request with the error 817 NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons: 819 * A server reboot causing loss of the server's knowledge of the 820 client 822 * Client error sending an incorrect clientid4 or a valid 823 clientid4 to the wrong server. 825 * Loss of lease state due to lease expiration. 827 * Client or server error causing the server to believe that the 828 client has rebooted (i.e. receiving a SETCLIENTID with an 829 nfs_client_id4 which has a matching id string and a non- 830 matching boot verifier). 832 * Migration of all state under the associated lease causes its 833 non-existence to be recognized on the source server. 835 * Merger of state under the associated lease with another lease 836 under a different clientid causes the clientid4 serving as the 837 source of the merge to cease being recognized on its server. 839 In the event of a server reboot, or loss of lease state due to 840 lease expiration, the client must obtain a new clientid4 by use of 841 the SETCLIENTID operation and then proceed to any other necessary 842 recovery for the server reboot case (See the section entitled 843 "Server Failure and Recovery"). In cases of server or client 844 error resulting in this error, use of SETCLIENTID to establish a 845 new lease is desirable as well. 847 In the last two cases, different recovery procedures are required. 848 Note that in cases in which there is any uncertainty about which 849 sort of handling is applicable, the distinguishing characteristic 850 is that in reboot-like cases, the clientid4 and all associated 851 stateids cease to exist while in migration-related cases, the 852 clientid4 ceases to exist while the stateids are still valid. 854 The client must also employ the SETCLIENTID operation when it 855 receives a NFS4ERR_STALE_STATEID error using a stateid derived 856 from its current clientid4, since this indicates a situation, such 857 as server reboot which has invalidated the existing clientid4 and 858 associated stateids (see the section entitled "lock-owner" for 859 details). 861 See the detailed descriptions of SETCLIENTID and 862 SETCLIENTID_CONFIRM for a complete specification of the 863 operations. 865 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE 867 It appears to be the intention that only a single principal be used 868 for client establishment between any client-server pair. However: 870 o There is no explicit statement to this effect. 872 o The error that indicates a principal conflict has a name which 873 does not clarify this issue: NFS4ERR_CLID_INUSE. 875 o The definition of the error is also not very helpful: "The 876 SETCLIENTID operation has found that a client id is already in use 877 by another client". 879 As a result, servers exist which reject a SETCLIENTID simply because 880 there already exists a clientid for the same client, established 881 using a different IP address. Although this is generally understood 882 to be erroneous, such servers still exist and the spec should make 883 the correct behavior clear. 885 Although the error name cannot be changed, the following changes 886 should be made to avoid confusion: 888 o The definition of the error should be changed to read as follows: 890 The SETCLIENTID operation has found that the specified 891 nfs_client_id4 was previously presented with a different 892 principal and that client instance currently holds an active 893 lease. A server MAY return this error if the same principal is 894 used but a change in authentication flavor gives good reason to 895 reject the new SETCLIENTID operation as not bona fide. 897 o In the description of SETCLIENTID, the phrase "then the server 898 returns a NFS4ERR_CLID_INUSE error" should be expanded to read 899 "then the server returns a NFS4ERR_CLID_INUSE error, since use of 900 a single client with multiple principals is not allowed." 902 6. Results of proposed changes for NFSv4.0 904 The purpose of this section is to examine the troubling results 905 reported in Section 3.1. We will look at the scenarios as they would 906 be handled within the proposal. 908 Because the choice of uniform vs. non-uniform nfs_client_id4 id 909 strings is a "SHOULD" in these cases, we will designate clients that 910 follow this recommendation by SHOULD-UF-CID. 912 We will also have to take account of any merger-related "SHOULD" 913 clauses to better understand how they have addressed the issues seen. 914 We abbreviate as follows: 916 o SHOULD-SVR-AM refers to the server obeying the SHOULD which 917 RECOMMENDS that they merge leases with identical nfs_client_id4 id 918 strings and boot verifiers. 920 6.1. Results: Failure to free migrated state on client reboot 922 Let's look at the troublesome situation cited in Section 3.1.1. We 923 have already seen what happens when SHOULD-UF-CID does not hold. Now 924 let's look at the situation in which SHOULD-UF-CID holds, whether 925 SHOULD-SVR-AM is in effect or not. 927 o A client C establishes a clientid4 C1 with server ABC specifying 928 an nfs_client_id4 with id string value "C" and boot verifier 929 0x111. 931 o The client begins to access files in filesystem F on server ABC, 932 resulting in generating stateids S1, S2, etc. under the lease for 933 clientid C1. It may also access files on other filesystems on the 934 same server. 936 o The filesystem is migrated from ABC to server XYZ. When 937 transparent state migration is in effect, stateids S1 and S2 and 938 lease {0x111, "C", C1} are now available for use by client C at 939 server XYZ. 941 o Client C reboots and attempts to access data on server XYZ, 942 whether in filesystem F or another. It does a SETCLIENTID with an 943 nfs_client_id4 with id string value "C" and boot verifier 0x112. 944 The state associated with lease {0x111, "C", C1} is deleted as 945 part of creating {0x112, "C", C2}. No problem. 947 The correctness signature for this issue is 949 SHOULD-UF-CID 951 so if you have clients and servers that obey the SHOULD clauses, the 952 problem is gone regardless of the choice on the MAY. 954 6.2. Results: Server reboots resulting in confused lease situation 955 Now let's consider the scenario given in Section 3.1.2. We have 956 already seen what happens when SHOULD-UF-CID does not hold . Now 957 let's look at the situation in which SHOULD-UF-CID holds and SHOULD- 958 SVR-AM holds as well. 960 o Client C talks to server ABC using an nfs_client_id4 id string 961 such as "C-ABC" and boot verifier v1. As a result a lease with 962 clientid4 c.i established: {v1, "C-ABC", c.i}. 964 o Filesystem fs_a1 migrates from server ABC to server XYZ along with 965 its state. Now server XYZ also has a lease: {v1, "C-ABC", c.i} 967 o Server ABC reboots. 969 o Client C talks to server ABC using an nfs_client_id4 id string 970 such as "C-ABC" and boot verifier v1. As a result a lease with 971 clientid4 c.j established: {v1, "C-ABC", c.j}. 973 o fs_a2 migrates from server ABC to server XYZ. As part of 974 migration the incoming lease is seen to denote same nfs_client_id4 975 and so is merged with {v1, "C-ABC, c.i}. 977 o Now server XYZ has only one lease that matches {v1, "C_ABC", *}, 978 so the problem is solved 980 Now let's consider the same scenario in the situation in which 981 SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well. 983 o Client C talks to server ABC using an nfs_client_id4 id string "C" 984 and boot verifier v1. As a result a lease with clientid4 c.i is 985 established: {v1, "C", c.i}. 987 o fs_a1 migrates from server ABC to server XYZ along with its state. 988 Now XYZ also has a lease: {v1, "C", c.i} 990 o Server ABC reboots. 992 o Client C talks to server ABC using an nfs_client_id4 id string "C" 993 and boot verifier v1. As a result a lease with clientid4 c.j is 994 established: {v1, "C", c.j}. 996 o fs_a2 migrates from server ABC to server XYZ. As part of 997 migration the incoming lease is seen to denote the same 998 nfs_client_id4 and so is merged with {v1, "C", c.i}. 1000 o Now server XYZ has only one lease that matches {v1, "C", *}, so 1001 the problem is solved 1003 The correctness signature for this issue is 1005 SHOULD-SVR-AM 1007 so if you have clients and servers that obey the SHOULD clauses, the 1008 problem is gone regardless of the choice on the MAY. 1010 6.3. Results: Client complexity issues 1012 Consider the following situation: 1014 o There are a set of clients C1 through Cn accessing servers S1 1015 through Sm. Each server manages some significant number of 1016 filesystems with the filesystem count L being significantly 1017 greater than m. 1019 o Each client Cx will access a subset of the servers and so will 1020 have up to m clientids, which we will call Cxy for server Sy. 1022 o Now assume that for load-balancing or other operational reasons, 1023 numbers of filesystems are migrated among the servers. As a 1024 result, depending on how this handled, the number of clientids may 1025 explode. See below. 1027 Now look what will happen under various scenarios: 1029 o We have previously (in Section 3.1.3) looked at this in case of 1030 client following the non-uniform client-string approach. In that 1031 case, each client-server pair could have up to m clientids and 1032 each client will have up to m**2 clientids. If we add the 1033 possibility of server reboot, the only bound on a client's 1034 clientid count is L. 1036 o If we look at this in the SHOULD-UF-CID case in which the SHOULD- 1037 SVR_AM condition holds, the situation is no different. Although 1038 the server has the client identity information that could enable 1039 same-client-same-server leases to be combined, it does not do so. 1040 We still have up to L clientids per client. 1042 o On the other hand, if we look at the SHOULD-UF-CID case in which 1043 SHOULD-SVR-AM holds, the problem is gone. There can be no more 1044 than m clientids per client, and n clientids per server. 1046 The correctness signature for this issue is 1048 (SHOULD-UF-CID & SHOULD-SVR-AM) 1050 so if you have clients and servers that obey the SHOULD clauses, the 1051 problem is gone regardless of the choice on the MAY. 1053 6.4. Result summary 1055 We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to 1056 solve the problems people have experienced. 1058 7. Issues for NFSv4.1 1060 Because NFSv4.1 embraces the uniform client-string approach, 1061 addressing migration issues is simpler. In the terms of Section 6, 1062 we already have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4 1063 of [RFC5661], simplifying the work to be done. 1065 Nevertheless, there are some issues that will have to be addressed. 1066 Some examples: 1068 o The other necessary part of addressing migration issues, which we 1069 call above SHOULD-SVR-AM, is not currently addressed by NFSv4.1 1070 and changes need to be made to make it clear that state needs to 1071 be appropriately merged as part of migration, to avoid multiple 1072 clientids between a client-server pair. 1074 o There needs to be some clarification of how migration, and 1075 particularly transparent state migration, should interact with 1076 pNFS layouts. 1078 o The current discussion (in [RFC5661]), of the possibility of 1079 server_owner changes is incomplete and confusing. 1081 Discussion of how to resolve these issues will appear in the sections 1082 below. 1084 7.1. Addressing state merger in NFSv4.1 1086 The existing treatment of state transfer in [RFC5661], has similar 1087 problems to that in [RFC3530] and [RFC3530bis] in that it assumes 1088 that the state for multiple filesystems on different servers will not 1089 be merged to so that it appears under a single common clientid. 1090 We've already seen the reasons that this is a problem, with regard to 1091 NFSv4.0. 1093 Although we don't have the problems stemming from the non-uniform 1094 client-string approach, there are a number of complexities in the 1095 existing treatment of state management in the section entitled "Lock 1096 State and File System Transitions" in [RFC5661] that make this non- 1097 trivial to address: 1099 o Migration is currently treated together with other sorts of 1100 filesystem transitions including transitioning between replicas 1101 without any NFS4ERR_MOVED errors. 1103 o There is separate handling and discussion of the cases of matching 1104 and non-matching server scopes. 1106 o In the case of matching server scopes, the text calls for an 1107 impossible degree of transparency. 1109 o In the case of non-matching server scopes, the text does not 1110 mention transparent state migration at all, resulting in a 1111 functional regression from NFSV4.0 1113 7.2. Addressing pNFS relationship with migration 1115 This is made difficult because, within the PNFS framework, migration 1116 might mean any of several things: 1118 o Transfer of the MDS, leaving DS's alone. 1120 This would be minimally disruptive to those using layouts but 1121 would require the pNFS control protocol to support the DS being 1122 directed to a new MDS. 1124 o Transfer of a DS, leaving everything else in place. 1126 Such a transfer can be handled without using migration at all. 1127 The server can recall/revoke layouts, as appropriate. 1129 o Transfer of the filesystem to a new filesystem with both MDS and 1130 DS's moving. 1132 In such a transfer, an entirely different set of DS's will be at 1133 the target location. There may even be no pNFS support on the 1134 destination filesystem at all. 1136 Migration needs to support both the first and last of these models. 1138 7.3. Addressing server owner changes in NFSv4.1 1140 Section 2.10.5 of [RFC5661] states the following. 1142 The client should be prepared for the possibility that 1143 eir_server_owner values may be different on subsequent EXCHANGE_ID 1144 requests made to the same network address, as a result of various 1145 sorts of reconfiguration events. When this happens and the 1146 changes result in the invalidation of previously valid forms of 1147 trunking, the client should cease to use those forms, either by 1148 dropping connections or by adding sessions. For a discussion of 1149 lock reclaim as it relates to such reconfiguration events, see 1150 Section 8.4.2.1. 1152 While this paragraph is literally true in that such reconfiguration 1153 events can happen and clients have to deal with them, it is confusing 1154 in that it can be read as suggesting that clients have to deal with 1155 them without disruption, which in general is impossible. 1157 A clearer alternative would be: 1159 It is always possible that, as a result of various sorts of 1160 reconfiguration events, eir_server_scope and eir_server_owner 1161 values may be different on subsequent EXCHANGE_ID requests made to 1162 the same network address. 1164 In most cases such reconfiguration events will be disruptive and 1165 indicate that an IP address formerly connected to one server is 1166 now connected to an entirely different one. 1168 Some guidelines on client handling of such situations follow: 1170 * When eir_server_scope changes, the client has no assurance that 1171 any id's it obtained previously (e.g. file handles) can be 1172 validly used on the new server, and, even if the new server 1173 accepts them, there is no assurance that this is not due to 1174 accident. Thus it is best to treat all such state as lost/ 1175 stale although a client may assume that the probability of 1176 inadvertent acceptance is low and treat this situation as 1177 within the next case. 1179 * When eir_server_scope remains the same and 1180 eir_server_owner.so_major_id changes, the client can use 1181 filehandles it has and attempt reclaims. It may find that 1182 these are now stale but if NFS4ERR_STALE is not received, he 1183 can proceed to reclaim his opens. 1185 * When eir_server_scope and eir_server_owner.so_major_id remain 1186 the same, the client has to use the now-current values of 1187 eir_server-owner.so_minor_id in deciding on appropriate forms 1188 of trunking. 1190 8. Security Considerations 1192 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 1193 and [RFC3530bis] both agree. The section entitled "Security 1194 Considerations" encourages that clients protect the integrity of the 1195 SECINFO operation, any GETATTR operation for the fs_locations 1196 attribute, and the operations SETCLIENTID/SETCLIENTID_CONFIRM. A 1197 migration recovery event can use any or all of these operations. We 1198 do not recommend any change here. 1200 9. IANA Considerations 1202 This document does not require actions by IANA. 1204 10. Acknowledgements 1206 The editor and authors of this document gratefully acknowledge the 1207 contributions of Trond Myklebust of NetApp and Robert Thurlow of 1208 Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of 1209 Microsoft for their guidance and suggestions. 1211 Special thanks go to members of the Oracle Solaris NFS team, 1212 especially Rick Mesta and James Wahlig, for their work implementing 1213 an NFSv4.0 migration prototype and identifying many of the issues 1214 documented here. 1216 11. References 1218 11.1. Normative References 1220 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1221 Requirement Levels", BCP 14, RFC 2119, March 1997. 1223 [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., 1224 Beame, C., Eisler, M., and D. Noveck, "Network File System 1225 (NFS) version 4 Protocol", RFC 3530, April 2003. 1227 [RFC3530bis] 1228 Haynes, T., Ed. and D. Noveck, Ed., "Network File System 1229 (NFS) Version 4 Protocol", 2011, . 1232 Work in progress. 1234 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 1235 System (NFS) Version 4 Minor Version 1 Protocol", RFC 1236 5661, January 2010. 1238 11.2. Informative References 1240 [migr-v4.0-update] 1241 Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, 1242 "NFSv4.0 migration: Specification Update", 2013, . 1246 Work in progress. 1248 Authors' Addresses 1250 David Noveck (editor) 1251 EMC Corporation 1252 228 South Street 1253 Hopkinton, MA 01748 1254 US 1256 Phone: +1 508 249 5748 1257 Email: david.noveck@emc.com 1259 Piyush Shivam 1260 Oracle Corporation 1261 5300 Riata Park Ct. 1262 Austin, TX 78727 1263 US 1265 Phone: +1 512 401 1019 1266 Email: piyush.shivam@oracle.com 1268 Charles Lever 1269 Oracle Corporation 1270 1015 Granger Avenue 1271 Ann Arbor, MI 48104 1272 US 1274 Phone: +1 248 614 5091 1275 Email: chuck.lever@oracle.com 1276 Bill Baker 1277 Oracle Corporation 1278 5300 Riata Park Ct. 1279 Austin, TX 78727 1280 US 1282 Phone: +1 512 401 1081 1283 Email: bill.baker@oracle.com