Network Woking Group Danny Cohen Myricom Internet Draft Craig Lund Mercury Computers Expire in six months Tony Skjellum, Thom McMahon, Robert George Mississippi State University May 1998 Enumerations for the PacketWay Protocol for High-Performance Interconnection of Computer Clusters Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). Table of Contents: This page....................................1 PacketWay at a Glance ("cheat-sheet")........2 A note about the PacketWay documents.........3 Notations....................................4 (1) PacketWay Packet Types...................5 (2) RRP messages.............................6 (3) RRP records..............................6 (4) Error messages...........................7 (5) Node capabilities........................7 (6) Optional header fields types (OH)........8 (7) Byte order (endianness)..................8 (8) Symbol types (ST)........................9 Appendix-A: Glossary........................10 Appendix-B: Acronyms and Abbreviations......12 Cohen et al. Experimental [Page 1] Internet-Draft PacketWay Enumerations May 1998 PacketWay at a Glance --------------------- 2 6 type 24 16 16 PW-Hdr+-+------+-------+--------+---------+--------+--------+--------+--------+ PH1|V| P | Destination-Type | Type-Extension | Packet-Type | +-+-+---++--------------------------+-+------+--------+-----------------+ PH2| E | PL| Data-Length (8B-words) |h| RZ |0 Source-Address | +---+---+--------+--------+---------+-+------+--------+--------+--------+ 4 3 25 1 7 1 23 type = 0xxx Physical Address 10xx L2RH 110x Reserved 1110 Logical Address 1111 Symbols 2 6 2 6 8 8 8 8 8 8 +--------+--------+--------+--------+--------+--------+--------+--------+ L2RH |V| P |10LLLLLL| SR01 | SR02 |........|........|........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ Length 2 6 4 4 8 8 8 8 8 8 +--------+--------+--------+--------+--------+--------+--------+--------+ Symbol|V| P |1111ssss|ssssssss|ssssssss| Length | data |........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ <---- Symbol Type ---> 2 6 8 8 8 8 8 8 8 Opt'l +--------+--------+--------+--------+--------+--------+--------+--------+ hdr |TCtttttt|LLLLLLLL| data |........|........|........|........|........| fields+--------+--------+--------+--------+--------+--------+--------+--------+ T: 0=optional, 1=mandatory; C: 0=more OH-fields follow, 1=last OH-field 8 8 8 8 8 8 8 8 RRP +--------+--------+--------+--------+--------+--------+--------+--------+ Record| RTyp | PL | RL |........|........|........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ RRP-messages: GVL2, L2SR, RDRC, TELL, INFO, HRTO, WRU, GVRT, RTBL; RTyp: ADDR, NAME, CAPA, LADR, SRQR, MTUR, RCVF, RTHD; Cohen et al. Experimental [Page 2] Internet-Draft PacketWay RRP May 1998 A note about the PacketWay Documents ------------------------------------ The PacketWay protocol is defined by a series of documents: * EEP (End-to-End Protocol) * RRP-1 (basic Router-to-Router Protocol) * RRP-2 (dynamic inter-SAN routing) * PacketWay enumerations Each of these documents should include the same "PacketWay at a Glance (Cheat=Sheet)", this note, and the Notations page. They should include also (as appendices) a copy of the PacketWay glossary of terms and its acronyms and abbreviations list. The EEP and the RRP documents will be published first as Internet-Drafts and later as Proposed-Standards, Draft-Standards, and Standards. The Enumeration Document will be first published as an "Informational-RFC" and later will be maintained by IANA. The enumeration document may be attached to the EEP/RRP documents, as a matter of convenience. The enumeration is NOT a part of the PacketWay standard, just as RFC0739 (the original "Assigned Numbers" RFC) is not a part of RFC0791, that defines IP. Similarly, the EEP-documenmt has "Appendix-A: A Recommendation for PacketWay Address Assignment" which is a recommendation only and NOT a part of the PacketWay standard, just as IP-address-assignment is not a part of RFC0791, that defines IP. Cohen et al. Experimental [Page 3] Internet-Draft PacketWay RRP May 1998 Notations --------- 8B means "8-byte" (64 bits). 0x indicates hexadecimal values, e.g., 0x0100 is 2^8=256(decimal). 0b indicates binary values, e.g., 0b0100 is 4(decimal). xxx indicate a field that is discarded without any checking (e.g., padding). [fff] indicates that fff is an optional field, when appropriate. [exp], in equations, is the integral part of `exp`, e.g., [17/8]=2. All length fields do not include themselves, and therefore may be zero. Fields lengths are specified either (a) by byte count, with following padding bytes to fill 8B-words, or (b) by 8B-word count, and PL, the number of trailing padding bytes (with 0<=PL<8). Cohen et al. Experimental [Page 4] Internet-Draft PacketWay RRP May 1998 PacketWay Enumerations ---------------------- (1) PacketWay Packet Types +------------------------- The EEP header reserves 4 bytes for signaling from the source node directly to the destination node. They are the PACKET TYPE (PT), and the TYPE EXTENSION (TE), 2 bytes each. This list defines values for the PACKET-TYPE (PT) 2B-field. Each packet-type has its own interpretation of the TE and the h-fields. 2B-Code Packet Type +---------- ---------------------- 0 Illegal 1 RRP 2 Embedded PacketWay Packet 3 MEM-READ 4 MEM-WRITE Higher level protocols: 21 IP 22 SNMP 23 ATM Link layer Protocols 50 Ethernet (E10) 51 Ethernet (E100) 52 Ethernet (E1000) 53 Myrinet 54 Fibre Channel 55 RACEway 56 SCI 57 VME Application level protocols: 81 MPI 82 PVM Secure Protocols 121 Secure (1) 122 Secure (2) 123 Secure (3) 1,024-2,047 User defined 65,535 ERR (for Error) More values will be assigned. "Ether-types" should be added with a pointer to those used by the Internet. Cohen et al. Experimental [Page 5] Internet-Draft PacketWay RRP May 1998 Explanation of the MEM-WRITE and MEM-READ (that a memory board may need): PT Meaning --------- -------------------------------------------------------- MEM-WRITE -- Treat the first 8 bytes of the Data Block as a local memory-address and write the remaining data into memory. MEM-READ -- Treat the data block as 2 8B-memory-addresses and an 8B-byte count. Generate a return WRITE packet containing the first address, followed by the appropriate data, that was read from the second address. (2) RRP Messages (Type Extensions of PT="RRP) +--------------------------------------------- RRP- Type Code Description +------ ---- ---------------------------------------------------- 0 Illegal GVL2 21 Please give me L2-routes from you to node (address) L2SR 22 Here are L2-routes to node (address) RDRC 23 Re-direct to node (address) via a neighbor HR (address) TELL 24 Please tell about node (address | name | capabilities) INFO 25 Info about node (address, name, capabilities) HRTO 26 Which HR should I use for node (address) WRU? 27 Who/what-Are-You? GVRT 28 Please give me your RTs RTBL 29 Here is an RT Throughout this document the RRP messages are indicated by their type (e.g., RDRC for re-direct). In actual messages the code is used (e.g., 2 for RDRC). (3) RRP records +-------------- RTyp Code Description +------ ---- ---------------------------------------------------- 0 Illegal ADDR 41 Address record for one or many nodes NAME 42 Node Name record CAPA 43 Node Capability record LADR 44 Node Logical Addresses record SRQR 45 Source Route record and its Quality (SR, Q) MTUR 46 MTU record (for the previous SRQR) Throughout this document the RRP records are indicated by their RTyp (e.g., ADDR for address). In actual messages the code is used (e.g.,41 for ADDR). Cohen et al. Experimental [Page 6] Internet-Draft PacketWay RRP May 1998 (4) Error Messages +----------------- Subtype Code Description --------- ---- ---------------------------------------------- 0 Illegal UNK 71 Unknown (address) HRDOWN 72 HR-Down (and the links associated with it) LINKDOWN 73 Link-Down (between two HRs) GENERAL 74 General error message Throughout this document the error messages are indicated by their subtype (e.g., LINKDOWN for Link-Down). In actual messages the code is used (e.g., 3 for LINKDOW). (5) PacketWay Node Capabilities +------------------------------ Code Capability Parameters +--- ------------------------ -------------------------------------- 0 Illegal 1 GP Computing Node 2 Router SAN-IDs, 1+3 Bytes each 3 PacketWay Server 4 Network Multicast Server 5 NFS 6 NPS (Paging Server) 7 Floating-point DSP IEEE word-sizes (in bytes), 1B per size 8 Fixed-point DSP word-sizes (in bytes), 1B per size 9 Printer 250 SRVLOC server 253 Secure PacketWay HR 254 Multicast agent for its SAN 255 SAN Cohen et al. Experimental [Page 7] Internet-Draft PacketWay RRP May 1998 (6) Optional Header Fields Types (OH) +------------------------------------ The MSbit of the type field (T) is the type-of-type. Its assignment is: 0b0: Optional (may drop this OH if its type, tttttt, is unknown) 0b1: Mandatory (should not process this OH if its type is unknown) The next bit is the "Completion bit" (C). Its assignment is: 0b0: More options follow 0b1: This is the last option field The 6 LSbits, tttttt, are the type field. Their assignment is: 0x00: Illegal 0x01: TBD 0x02: CRC32 here 0x03: CRC32 following in the OT (after the DB) 0x04: CRC64 here 0x05: CRC64 following in the OT (after the DB) 0x06: There is an OT (Optional Trailer) 0x07-0x3D: TBD 0x3E: Cryptographic data (7) Byte Order (Endianness) +-------------------------- A 4 bit field (E) is used to indicate Endianness, with e being its first bit (MSbit). e=0: Big-Endian order e=1: Little-Endian order The 3 LSbits indicate the size of data chucks (must be the same for the entire data block) to allow hardware swapping e000: don't swap, it's 8-bit data e001: swap as if all the data is 16-bit words e010: swap as if all the data is 32-bit words e011: swap as if all the data is 64-bit words e100: swap as if all the data is 128-bit words e101: illegal and reserved for future use e110: illegal and reserved for future use e111: illegal and reserved for future use Cohen et al. Experimental [Page 8] Internet-Draft PacketWay RRP May 1998 (8) Symbol Types (ST) +-------------------- The format of Symbols is: +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|1111ssss|ssssssss|ssssssss| Length | data |........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ <---- Symbol Type ---> Code Symbol Type -------- ----------------- 0x00000: Reserved 0x00001: Multicast 0x00002: Secure PacketWay Cohen et al. Experimental [Page 9] Internet-Draft PacketWay RRP May 1998 Appendix-A: Glossary --------------------------- (Last update: Aug-24, 1997) Address: A unique designation of a node (actually an interface to that node) or a SAN. Buddy-HR: HRs are "buddies" if they are on the same SAN. Cut-Thru: See wormhole. Destination: The node to which a packet is intended Dynamic-Routing: Routing according to dynamic information (i.e., acquired at run time, rather than pre-set). Endianness: The property of being Big-Endian or Little-Endian (transmission order, etc.) Ethertype: A 16-bit value designating the type of Level-3 packets carried by a Level-2 communication system. HR: Half-Router, the part of a router that handles one network only. L2-Forwarding: Forwarding based on Level-2 (i.e., data-link layer of the ISORM) information, e.g., the native technique of each SAN or LAN. Also called "source routing." L3-Forwarding: Forwarding based on end-to-end Level-3 (i.e., network layer of the ISORM) addresses. Also called "destination routing." Map: The topology of a network. Mapper: A node on a SAN/LAN that has the map and an RT for that network. It is expected that the mapper dynamically updates the map and the RT. Multi-homed Node: A node with more than one network interface, where each interface has another address. Node: Whatever can send and receive packets (e.g., a computer, an MPP, a software process, etc.) Node structure: A C-struct (or equivalent) containing values for some attributes of a node. Planned Transfer: Transfer of information, occurs after an initial phase in which the sender decides which Level-2 route to use for that transfer. Cohen et al. Experimental [Page 10] Internet-Draft PacketWay RRP May 1998 RCVF: The "Received From" set includes all the physical addresses through which an RT was disseminated, starting with that of the mapper that created that RT. Re-direct-message: A message that tells nodes which HR should be used in order to get to a certain remote address (or range of). Router: The inter-SAN communication device Security Context: A relationship between 2 (or more) nodes that defines how the nodes utilize security services to communicate securely. Source: The node that created a packet. Source-Route: A Level-2 route that is chosen for a packet by its source. Symbol: Data preceeding the EEP header of a PacketWay message, interleaving with the L2RHs. Twin-HR: Two HRs are twins if they both are parts of the same inter-SAN router. Wormhole-routing: (aka cut-thru routing) forwarding packets out of switches as soon as possible, without storing that entire packet in the switch (unlike Stop-and-forward). Zero-copy TCP: A TCP system that copies data directly between the user area and the network device, bypassing OS copies. Cohen et al. Experimental [Page 11] Internet-Draft PacketWay RRP May 1998 Appendix-B: Acronyms and Abbreviations -------------------------------------- (Last update: August 24, 1997) 0bNNNN The binary number NNNN (e.g., 0b0100 is 4-decimal) 0xNNNN The hexadecimal number NNNN (e.g., 0x0100 is 256-decimal) 8B 8 byte (64 bits) entity ADDR The Address-record of RRP API Application/Program Interface AT Address Type ATM Asynchronous Transmission Mode B Byte (e.g., 4B) b bit (e.g., 32b) BC Byte Count (of parameters) BER Bit Error Rate CAPA The CAPAbility-record of RRP CC Capability Code CSR Common Source-Route DA Destination Address DB Data Block DL Data Length (in 8B words) DSP Digital Signal Processore DT Destination-Type e The MSbit of E E The Endianness field (in the EEP header) EEP End/End Protocol EI Error Indication GP General Purpose GVL2 An RRP message, requesting L2 route to a given destination GVRT An RRP message asking an HR to give its routing tables h Optional header fields flag HR Half Router HRTO An RRP message asking which HR to use for a given destination ID Identification IGMP Internet Group Management Protocol INFO An RRP message providing information about nodes IP The Internet protocol ISORM The ISO Reference Model L Length field (exclusive of itself) L2 Level-2 of the ISORM (Link) L2RH Level-2 Routing Header L2SR Source Route L3 Level-3 of the ISORM (Network) LA Logical Address LADR The Logical-addresses-record of RRP LAN Local Area Network LRT Local Routing Table LSbit Least Significant bit LSbyte Least Significant byte MAC Message Authentication Code / Media Access Control MPI Message Passing Interface MPP Massively Parallel Processing system Cohen et al. Experimental [Page 12] Internet-Draft PacketWay RRP May 1998 MSbit Most Significant bit MSbyte Most Significant byte MSU Mississippi State University MTU Maximum Transmission Unit MTUR The MTU-record of RRP M/C Multicast NAME The name-record of RRP NFS Network File Server OH Optional Header field OH-TYPE The Type of an Optional Header field OT Optional Trailer field P The Priority field PAD Padding After Data PBD Padding Before Data PCI The Peripheral Component Interconnect "standard" PH PacketWay Header PL Padding Length (always in bytes) PPP The Point-to-Point Protocol PROM Programmable ROM (Read-Only-Memory) PT Packet Type (2B) PVM Parallel Virtual Machine PW The Myrinet Packet Type assigned to PacketWay (PW=0x0300) Q Quality (of a path) RCVF Received-From list, or the Received-From record of RRP RDRC A re-direct message of RRP RH Routing Header RID Record ID RL Record Length (in 8B-words) RRP Router/Router Protocol RT-hd RT (Routing Table) header RT Routing Table RTBL An RRP message proving a Routing Table RTHD The Routing-Table-Header record of RRP RTyp RRP's Record Type RZ The Reserved field (in the EEP header) SA Source Address SAN System Area Network SAN-ID The 24-bit PacketWay-address of a SAN SAR Segmentation and Reassembly SN Serial Number SNID SAN-ID SNMP Simple Network Management Protocol SR Source Route (always at Level-2) SRQR The Source-Route-and-Q-record of RRP ST Symbol Type TAIL PacketWay EEP Trailer TE Type Extension (2B) TELL An RRP message requesting information about nodes partially specified UNK Unknown V Version WRU? An RRP message asking its recipient to identify itself XRT External Routing Table xxx A padding byte Cohen et al. Experimental [Page 13] Internet-Draft PacketWay RRP May 1998 Editor's Address Anthony Skjellum Computer Science Department Box 9637 Mississippi State University Mississippi State, MS 39762-9637 Phone: 601-325-8435 Fax: 601-325-8997 Email: tony@cs.msstate.edu