idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-bidirection-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 20, 2017) is 2653 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-09 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Standards Track January 20, 2017 5 Expires: July 24, 2017 7 Bi-directional Remote Procedure Call On RPC-over-RDMA Transports 8 draft-ietf-nfsv4-rpcrdma-bidirection-06 10 Abstract 12 Minor versions of NFSv4 newer than NFSv4.0 work best when ONC RPC 13 transports can send Remote Procedure Call transactions in both 14 directions on the same connection. This document describes how RPC- 15 over-RDMA transport endpoints convey RPCs in both directions on a 16 single connection. 18 Requirements Language 20 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 21 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 22 document are to be interpreted as described in [RFC2119]. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on July 24, 2017. 41 Copyright Notice 43 Copyright (c) 2017 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 2. Understanding RPC Direction . . . . . . . . . . . . . . . . . 2 60 3. Immediate Uses Of Bi-Directional RPC-over-RDMA . . . . . . . 4 61 4. Flow Control . . . . . . . . . . . . . . . . . . . . . . . . 6 62 5. Sending And Receiving Backward Operations . . . . . . . . . . 8 63 6. In the Absence of Backward Direction Support . . . . . . . . 10 64 7. Considerations For Upper Layer Bindings . . . . . . . . . . . 11 65 8. Security Considerations . . . . . . . . . . . . . . . . . . . 11 66 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 67 10. Normative References . . . . . . . . . . . . . . . . . . . . 12 68 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 12 69 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 71 1. Introduction 73 The purpose of this document is to enable concurrent operation in 74 both directions on a single transport connection using RPC-over-RDMA 75 protocol versions that do not have specific facilities for backward 76 direction operation. 78 Backward direction RPC transactions are necessary for the operation 79 of NFSv4.1, and in particular, of Parallel NFS (pNFS) [RFC5661], 80 though any Upper Layer Protocol implementation may make use of them. 81 An Upper Layer Binding for NFSv4.x callback operation is additionally 82 required (see Section 7), but is not provided in this document. 84 For example, using the approach described herein, RPC transactions 85 can be conveyed in both directions on the same RPC-over-RDMA Version 86 One connection without changes to the XDR description of RPC-over- 87 RDMA Version One. This document does not modify the XDR or protocol 88 described in [I-D.ietf-nfsv4-rfc5666bis]. Future versions of RPC- 89 over-RDMA may adopt the approach described herein, or may replace it 90 with a different approach. 92 2. Understanding RPC Direction 94 The ONC RPC protocol as described in [RFC5531] is architected as a 95 message-passing protocol between one server and one or more clients. 96 ONC RPC transactions are made up of two types of messages. 98 A CALL message, or "Call", requests work. A Call is designated by 99 the value CALL in the message's msg_type field. An arbitrary unique 100 value is placed in the message's xid field. A host that originates a 101 Call is referred to in this document as a "Requester." 103 A REPLY message, or "Reply", reports the results of work requested by 104 a Call. A Reply is designated by the value REPLY in the message's 105 msg_type field. The value contained in the message's xid field is 106 copied from the Call whose results are being returned. A host that 107 emits a Reply is referred to as a "Responder." 109 Typically, a Call results in a corresponding Reply. A Reply is never 110 sent without a corresponding Call. 112 RPC-over-RDMA is a connection-oriented RPC transport. In all cases, 113 when a connection-oriented transport is used, ONC RPC client 114 endpoints are responsible for initiating transport connections, while 115 ONC RPC service endpoints passively await incoming connection 116 requests. 118 RPC direction on connectionless RPC transports is not addressed in 119 this document. 121 2.1. Forward Direction 123 Traditionally, an ONC RPC client acts as a Requester, while an ONC 124 RPC service acts as a Responder. This form of message passing is 125 referred to as "forward direction" operation. 127 2.2. Backward Direction 129 The ONC RPC specification [RFC5531] does not forbid passing messages 130 in the other direction. An ONC RPC service endpoint can act as a 131 Requester, in which case an ONC RPC client endpoint acts as a 132 Responder. This form of message passing is referred to as "backward 133 direction" operation. 135 During backward direction operation, the ONC RPC client is 136 responsible for establishing transport connections, even though ONC 137 RPC Calls come from the ONC RPC server. 139 ONC RPC clients and services are optimized to perform and scale well 140 while handling traffic in the forward direction, and might not be 141 prepared to handle operation in the backward direction. Not until 142 NFSv4.1 [RFC5661] has there been a strong need to handle backward 143 direction operation. 145 2.3. Bi-directional Operation 147 A pair of connected RPC endpoints may choose to use only forward or 148 only backward direction operations on a particular transport. Or, 149 these endpoints may send Calls in both directions concurrently on the 150 same transport. 152 "Bi-directional operation" occurs when both transport endpoints act 153 as a Requester and a Responder at the same time. 155 Bi-directionality is an extension of RPC transport connection 156 sharing. Two RPC endpoints wish to exchange independent RPC messages 157 over a shared connection, but in opposite directions. These messages 158 may or may not be related to the same workloads or RPC Programs. 160 2.4. XID Values 162 Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, 163 or "xid" for short. The value of an xid is interpreted in the 164 context of the message's msg_type field. 166 o The xid of a Call is arbitrary but is unique among outstanding 167 Calls from that Requester. 169 o The xid of a Reply always matches that of the initiating Call. 171 When receiving a Reply, a Requester matches the xid value in the 172 Reply with a Call it previously sent. 174 2.4.1. XID Generation 176 During bi-directional operation, forward and backward direction XIDs 177 are typically generated on distinct hosts by possibly different 178 algorithms. There is no co-ordination between forward and backward 179 direction XID generation. 181 Therefore, a forward direction Requester MAY use the same xid value 182 at the same time as a backward direction Requester on the same 183 transport connection. Though such concurrent requests use the same 184 xid value, they represent distinct ONC RPC transactions. 186 3. Immediate Uses Of Bi-Directional RPC-over-RDMA 188 3.1. NFSv4.0 Callback Operation 190 An NFSv4.0 client employs a traditional ONC RPC client to send NFS 191 requests to an NFSv4.0 server's traditional ONC RPC service 192 [RFC7530]. NFSv4.0 requests flow in the forward direction on a 193 connection established by the client. This connection is referred to 194 as a "forechannel" connection. 196 An NFSv4 "delegation" is simply a promise made by a server that it 197 will notify a client before another client or program running on the 198 server is allowed access to a file. With this guarantee, that client 199 can operate as sole accessor of the file. In particular, it can 200 manage the file's data and metadata caches aggressively. 202 To administer file delegations, NFSv4.0 introduces the use of 203 callback operations, or "callbacks", in Section 10.2 of [RFC7530]. 204 An NFSv4.0 server sets up a traditional ONC RPC client, and an 205 NFSv4.0 client sets up a traditional ONC RPC service. Callbacks flow 206 in the forward direction on a connection established between the 207 server's callback client, and the client's callback server. This 208 connection is distinct from connections being used as forechannels, 209 and is referred to as a "backchannel connection." 211 When an RDMA transport is used as a forechannel, an NFSv4.0 client 212 typically provides a TCP callback service. The client's SETCLIENTID 213 operation advertises the callback service endpoint with a "tcp" or 214 "tcp6" netid. The server then connects to this service using a TCP 215 socket. 217 NFSv4.0 implementations can function without a backchannel in place. 218 In this case, the server does not grant file delegations. This might 219 result in a negative performance effect, but correctness is not 220 affected. 222 3.2. NFSv4.1 Callback Operation 224 NFSv4.1 supports file delegation in a similar fashion to NFSv4.0, and 225 extends the callback mechanism to manage pNFS layouts, as discussed 226 in Section 12 of [RFC5661]. 228 NFSv4.1 transport connections are initiated by NFSv4.1 clients. 229 Therefore NFSv4.1 servers send callbacks to clients in the backward 230 direction on connections established by NFSv4.1 clients. 232 NFSv4.1 clients and servers indicate to their peers that a 233 backchannel capability is available on a given transport in the 234 arguments and results of NFS CREATE_SESSION or BIND_CONN_TO_SESSION 235 operations. 237 NFSv4.1 clients may establish distinct transport connections for 238 forechannel and backchannel operation, or they may combine 239 forechannel and backchannel operation on one transport connection 240 using bi-directional operation. 242 Without a backward direction RPC-over-RDMA capability, an NFSv4.1 243 client must additionally connect using a transport with backward 244 direction capability to use as a backchannel. TCP is the only choice 245 for an NFSv4.1 backchannel connection in this case. 247 Implementations often find it more convenient to use a single 248 combined transport (ie. a transport that is capable of bi-directional 249 operation). This simplifies connection establishment and recovery 250 during network partitions or when one endpoint restarts. This can 251 also enable better scaling by using fewer transport connections to 252 perform the same work. 254 As with NFSv4.0, if a backchannel is not in use, an NFSv4.1 server 255 does not grant delegations. Because NFSv4.1 relies on callbacks to 256 manage pNFS layout state, pNFS operation is not possible without a 257 backchannel. 259 4. Flow Control 261 For an RDMA Send operation to work properly, the receiving peer must 262 have posted a receive buffer in which to accept the incoming message. 263 If a receiver hasn't posted enough buffers to accommodate each 264 incoming Send operation, the receiving RDMA provider is allowed to 265 terminate the RDMA connection. 267 RPC-over-RDMA transport protocols provide built-in send flow control 268 to prevent overrunning the number of pre-posted receive buffers on a 269 connection's receive endpoint. For RPC-over-RDMA Version One, this 270 is discussed in Section 4.3 of [I-D.ietf-nfsv4-rfc5666bis]. 272 4.1. Backward Credits 274 RPC-over-RDMA credits work the same way in the backward direction as 275 they do in the forward direction. However, forward direction credits 276 and backward direction credits on the same connection are accounted 277 separately. 279 The forward direction credit value retains the same meaning whether 280 or not there are backward direction resources associated with an RPC- 281 over-RDMA transport connection. This is the number of RPC requests 282 the forward direction responder (the RPC server) is prepared to 283 receive concurrently. 285 The backward direction credit value is the number of RPC requests the 286 backward direction responder (the RPC client) is prepared to receive 287 concurrently. The backward direction credit value MAY be different 288 than the forward direction credit value. 290 During bi-directional operation, each receiver has to decide whether 291 an incoming message contains a credit request (the receiver is acting 292 as a responder) or a credit grant (the receiver is acting as a 293 requester) and apply the credit value accordingly. 295 When message direction is not fully determined by context (e.g., 296 suggested by the definition of the RPC-over-RDMA version that is in 297 use) or by an accompanying RPC message payload with a call direction 298 field, it is not possible for the receiver to tell with certainty 299 whether the header credit value is a request or grant. In such 300 cases, the receiver MUST ignore the header's credit value. 302 4.2. Inline Thresholds 304 Forward and backward operation on the same connection share the same 305 receive buffers. Therefore the inline threshold values for the 306 forward direction and the backward direction are the same. The call 307 inline threshold for the backward direction is the same as the reply 308 inline threshold for the forward direction, and vice versa. For more 309 information, see Section 4.3.2 of [I-D.ietf-nfsv4-rfc5666bis]. 311 4.3. Managing Receive Buffers 313 An RPC-over-RDMA transport endpoint must pre-post receive buffers 314 before it can receive and process incoming RPC-over-RDMA messages. 315 If a sender transmits a message for a receiver which has no posted 316 receive buffer, the RDMA provider is allowed to drop the RDMA 317 connection. 319 4.3.1. Client Receive Buffers 321 Typically an RPC-over-RDMA Requester posts only as many receive 322 buffers as there are outstanding RPC Calls. A client endpoint 323 without backward direction support might therefore at times have no 324 pre-posted receive buffers. 326 To receive incoming backward direction Calls, an RPC-over-RDMA client 327 endpoint must pre-post enough additional receive buffers to match its 328 advertised backward direction credit value. Each outstanding forward 329 direction RPC requires an additional receive buffer above this 330 minimum. 332 When an RDMA transport connection is lost, all active receive buffers 333 are flushed and are no longer available to receive incoming messages. 334 When a fresh transport connection is established, a client endpoint 335 must re-post a receive buffer to handle the Reply for each 336 retransmitted forward direction Call, and a full set of receive 337 buffers to handle backward direction Calls. 339 4.3.2. Server Receive Buffers 341 A forward direction RPC-over-RDMA service endpoint posts as many 342 receive buffers as it expects incoming forward direction Calls. That 343 is, it posts no fewer buffers than the number of credits granted in 344 the rdma_credit field of forward direction RPC replies. 346 To receive incoming backward direction replies, an RPC-over-RDMA 347 server endpoint must pre-post enough additional receive buffers to 348 handle replies for each backward direction Call it sends. 350 When the existing transport connection is lost, all active receive 351 buffers are flushed and are no longer available to receive incoming 352 messages. When a fresh transport connection is established, a server 353 endpoint must re-post a receive buffer to handle the Reply for each 354 retransmitted backward direction Call, and a full set of receive 355 buffers for receiving forward direction Calls. 357 5. Sending And Receiving Backward Operations 359 The operation of RPC-over-RDMA transports in the forward direction is 360 defined in [RFC5531] and [I-D.ietf-nfsv4-rfc5666bis]. In this 361 section, a mechanism for backward direction operation on RPC-over- 362 RDMA is defined. Backward operation used in combination with forward 363 operation enables bi-directional communication on a common RPC 364 transport connection. 366 Certain fields in the RPC-over-RDMA header are fixed for all versions 367 of RPC-over-RDMA. The XDR description of these fields is contained 368 in Section 5.1 of [I-D.ietf-nfsv4-rfc5666bis]. 370 5.1. Sending A Backward Direction Call 372 To form a backward direction RPC-over-RDMA Call message, an ONC RPC 373 service endpoint constructs an RPC-over-RDMA header containing a 374 fresh RPC XID in the rdma_xid field (see Section 2.4 for full 375 requirements). 377 The rdma_vers field MUST contain the same value in backward and 378 forward direction Call messages on the same connection. 380 The number of requested backward direction credits is placed in the 381 rdma_credit field (see Section 4). 383 Whether presented inline or as a separate chunk, the ONC RPC Call 384 header MUST start with the same XID value that is present in the RPC- 385 over-RDMA header, and the RPC header's msg_type field MUST contain 386 the value CALL. 388 5.2. Sending A Backward Direction Reply 390 To form a backward direction RPC-over-RDMA Reply message, an ONC RPC 391 client endpoint constructs an RPC-over-RDMA header containing a copy 392 of the matching ONC RPC Call's RPC XID in the rdma_xid field (see 393 Section 2.4 for full requirements). 395 The rdma_vers field MUST contain the same value in a backward 396 direction Reply message as in the matching Call message. 398 The number of granted backward direction credits is placed in the 399 rdma_credit field (see Section 4). 401 Whether presented inline or as a separate chunk, the ONC RPC Reply 402 header MUST start with the same XID value that is present in the RPC- 403 over-RDMA header, and the RPC header's msg_type field MUST contain 404 the value REPLY. 406 5.3. Backward Direction Chunks 408 Chunks MAY be used in the backward direction. They operate the same 409 way as in the forward direction (see [I-D.ietf-nfsv4-rfc5666bis] for 410 details). 412 An implementation might not support any Upper Layer Protocol that has 413 DDP-eligible data items. The Upper Layer Protocol may also use only 414 small messages, or it may have a native mechanism for restricting the 415 size of backward direction RPC messages, obviating the need to handle 416 Long Messages in the backward direction. 418 When there is no Upper Layer Protocol requirement for chunks, 419 implementers can choose not to provide support for chunks in the 420 backward direction. This avoids the complexity of adding support for 421 performing RDMA Reads and Writes in the backward direction. 423 When chunks are not implemented, RPC messages in the backward 424 direction are always sent using RDMA_MSG, and therefore can be no 425 larger than what can be sent inline (that is, without chunks). 426 Sending an inline message larger than the inline threshold can result 427 in loss of connection. 429 If a backward direction requester provides a non-empty chunk list to 430 a responder that does not support chunks, the responder MUST reply 431 with an RDMA_ERROR message with rdma_err field set to ERR_CHUNK. 433 5.4. Backward Direction Retransmission 435 In rare cases, an ONC RPC transaction cannot be completed within a 436 certain time. This can be because the transport connection was lost, 437 the Call or Reply message was dropped, or because the Upper Layer 438 consumer delayed or dropped the ONC RPC request. Typically, the 439 Requester sends the transaction again, reusing the same RPC XID. 440 This is known as an "RPC retransmission". 442 In the forward direction, the Requester is the ONC RPC client. The 443 client is always responsible for establishing a transport connection 444 before sending again. 446 In the backward direction, the Requester is the ONC RPC server. 447 Because an ONC RPC server does not establish transport connections 448 with clients, it cannot send a retransmission if there is no 449 transport connection. It must wait for the ONC RPC client to re- 450 establish the transport connection before it can retransmit ONC RPC 451 transactions in the backward direction. 453 If an ONC RPC client has no work to do, it may be some time before it 454 re-establishes a transport connection. Backward direction Requesters 455 must be prepared to wait indefinitely for a connection to be 456 established before a pending backward direction ONC RPC Call can be 457 retransmitted. 459 Forward direction Requesters are responsible for maintaining a 460 transport connection as long as there is the possibility of backward 461 direction requests. For example, an NFSv4.1 client with open 462 delegated files or active pNFS layouts should maintain a transport 463 connection so the server can send callback operations. 465 6. In the Absence of Backward Direction Support 467 An RPC-over-RDMA transport endpoint might not support backward 468 direction operation (and thus it does not support bi-directional 469 operation). There might be no mechanism in the transport 470 implementation to do so. Or in an implementation that can support 471 operation in the backward direction, the Upper Layer Protocol 472 consumer might not yet have configured or enabled the transport to 473 handle backward direction traffic. 475 If an endpoint is not prepared to receive an incoming backward 476 direction message, loss of the RDMA connection might result. Thus 477 denial of service could result if a sender continues to send backward 478 direction messages after every transport reconnect to an endpoint 479 that is not prepared to receive them. 481 When dealing with the possibility that the remote peer has no 482 transport level support for backward direction operation, the Upper 483 Layer Protocol becomes responsible for informing peers when backward 484 direction operation is supported. Otherwise even a simple backward 485 direction NULL probe from a peer could result in a lost connection. 487 Therefore, an Upper Layer Protocol consumer MUST NOT perform backward 488 direction ONC RPC operations unless the peer consumer has indicated 489 it is prepared to handle them. A description of Upper Layer Protocol 490 mechanisms used for this indication is outside the scope of this 491 document. 493 For example, an NFSv4.1 server does not send backchannel messages to 494 an NFSv4.1 client before the NFSv4.1 client has sent a CREATE_SESSION 495 or a BIND_CONN_TO_SESSION operation. As long as an NFSv4.1 client 496 has prepared appropriate backchannel resources before sending one of 497 these operations announcing support for backchannel operation, denial 498 of service is avoided. 500 7. Considerations For Upper Layer Bindings 502 An Upper Layer Protocol that operates on RPC-over-RDMA transports may 503 have procedures that include DDP-eligible data items. DDP- 504 eligibility is specified in an Upper Layer Binding. Direction of 505 operation does not obviate the need for DDP-eligibility statements. 507 Backward-only operation requires the client endpoint to establish a 508 fresh connection. The Upper Layer Binding can specify appropriate 509 RPC binding parameters for such connections. 511 Bi-directional operation occurs on an already-established connection. 512 Specification of RPC binding parameters is usually not necessary in 513 this case. 515 For bi-directional operation, other considerations about sharing an 516 RPC-over-RDMA transport with another ULP may apply. Consult 517 Section 6 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else 518 may be contained in an Upper Layer Binding. 520 8. Security Considerations 522 Security considerations for operation on RPC-over-RDMA transports are 523 outlined in Section 9 of [I-D.ietf-nfsv4-rfc5666bis]. 525 9. IANA Considerations 527 This document does not require actions by IANA. 529 10. Normative References 531 [I-D.ietf-nfsv4-rfc5666bis] 532 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 533 Memory Access Transport for Remote Procedure Call, Version 534 One", draft-ietf-nfsv4-rfc5666bis-09 (work in progress), 535 January 2017. 537 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 538 Requirement Levels", BCP 14, RFC 2119, March 1997. 540 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 541 Specification Version 2", RFC 5531, May 2009. 543 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 544 System (NFS) Version 4 Minor Version 1 Protocol", 545 RFC 5661, January 2010. 547 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 548 Version 4 Protocol", RFC 7530, March 2015. 550 Appendix A. Acknowledgements 552 Tom Talpey was an indispensable resource, in addition to creating the 553 foundation upon which this work is based. Our warmest regards go to 554 him for his help and support. 556 Dave Noveck provided excellent review, constructive suggestions, and 557 navigational guidance throughout the process of drafting this 558 document. 560 Dai Ngo was a solid partner and collaborator. Together we 561 constructed and tested independent prototypes of the changes 562 described in this document. 564 The author wishes to thank Bill Baker for his unwavering support of 565 this work. In addition, the author gratefully acknowledges the 566 expert contributions of Karen Deitke, Chunli Zhang, Mahesh 567 Siddheshwar, Steve Wise, and Tom Tucker. 569 Special thanks go to Transport Area Director Spencer Dawkins, nfsv4 570 Working Group and document shepherd Chair Spencer Shepler, and nfsv4 571 Working Group Secretary Tom Haynes for their support. 573 Author's Address 575 Charles Lever 576 Oracle Corporation 577 1015 Granger Avenue 578 Ann Arbor, MI 48104 579 USA 581 Phone: +1 248 816 6463 582 Email: chuck.lever@oracle.com