idnits 2.17.1 draft-ietf-ipoib-ip-over-infiniband-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 12 longer pages, the longest (page 2) being 60 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 13 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 6 instances of too long lines in the document, the longest one being 7 characters in excess of 72. == There are 1 instance of lines with non-RFC2606-compliant FQDNs in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (February 6, 2002) is 8113 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: '23-0' is mentioned on line 365, but not defined == Missing Reference: '127-96' is mentioned on line 367, but not defined == Missing Reference: '95-64' is mentioned on line 369, but not defined == Missing Reference: '63-32' is mentioned on line 371, but not defined == Missing Reference: '31-0' is mentioned on line 373, but not defined == Unused Reference: 'RFC2375' is defined on line 539, but no explicit reference was found in the text == Unused Reference: 'RFC1700' is defined on line 543, but no explicit reference was found in the text == Unused Reference: 'RFC2434' is defined on line 545, but no explicit reference was found in the text == Unused Reference: 'IANA' is defined on line 551, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2373 (Obsoleted by RFC 3513) ** Downref: Normative reference to an Informational RFC: RFC 2375 ** Obsolete normative reference: RFC 1700 (Obsoleted by RFC 3232) ** Obsolete normative reference: RFC 2434 (Obsoleted by RFC 5226) ** Obsolete normative reference: RFC 2461 (Obsoleted by RFC 4861) ** Obsolete normative reference: RFC 3041 (Obsoleted by RFC 4941) -- Possible downref: Non-RFC (?) normative reference: ref. 'IANA' Summary: 11 errors (**), 0 flaws (~~), 14 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET DRAFT Vivek Kashyap 3 IBM 4 Expiration Date: August 6, 2002 H.K. Jerry Chu 5 Sun Microsystems 7 February 6, 2002 9 IP encapsulation and address resolution over InfiniBand networks 11 Status of this memo 13 This document is an Internet-Draft and is in full conformance 14 with all provisions of Section 10 of RFC 2026. 16 Internet-Drafts are working documents of the Internet 17 Engineering Task Force (IETF), its areas, and its working 18 groups. Note that other groups may also distribute working 19 documents as Internet- Drafts. 21 Internet-Drafts are draft documents valid for a maximum of six 22 months and may be updated, replaced, or obsoleted by other 23 documents at any time. It is inappropriate to use 24 Internet-Drafts as Reference material or to cite them other 25 than as ``work in progress''. 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/ietf/1id-abstracts.txt 30 The list of Internet-Draft Shadow Directories can be accessed 31 at http://www.ietf.org/shadow.html 33 This memo provides information for the Internet community. 34 This memo does not specify an Internet standard of any kind. 35 Distribution of this memo is unlimited. 37 Copyright Notice 39 Copyright (C) The Internet Society (2001). All Rights Reserved. 41 Abstract 43 This document specifies the frame format for transmission of 44 IP and ARP packets over InfiniBand networks. Unless explicitly 45 specified, the term 'IP' refers to both IPv4 and IPv6. The 46 term 'ARP' refers to all the ARP protocols/op-codes such as 47 ARP/RARP. This document also describes the method of forming 48 IPv6 link-local addresses, and the content of the 49 source/target link layer address option used in Neighbour 50 solicitation and advertisement, router advertisement, router 51 redirect and router solicitation on IPv6 over InfiniBand. 53 Table of Contents 55 1.0 Introduction 56 2.0 InfiniBand Datalink 57 2.1 IP Support on IPoIB Link 58 3.0 Maximum Transmission Unit 59 4.0 Frame Format 60 5.0 IPv6 Stateless Autoconfiguration 61 5.1 IPv6 Link Local Address 62 6.0 Address Mapping - Unicast 63 6.1 Link-Information 64 6.1.1 Link Layer Address/Hardware Address 65 6.1.2 Auxiliary Link Information 66 6.2 Address Resolution in IPv4 Subnets 67 6.3 Link-Layer Address in IPv6 68 7.0 IANA Considerations 69 8.0 Security Considerations 70 9.0 Acknowledgements 71 10.0 References 72 11.0 Authors' Addresses 74 1.0 Introduction 76 The InfiniBand specification[IB_ARCH] can be found at 77 www.infinibandta.org. The document [IPoIB_ARCH] provides a 78 short overview of InfiniBand architecture along with 79 considerations for specifying IP over InfiniBand networks. The 80 document [IPoIB_MCAST] defines the configuration of IPoIB 81 links and the support of IP multicast over InfiniBand 82 networks. 84 The InfiniBand architecture(IBA) defines multiple modes of 85 transport over which IP may be implemented. The unreliable 86 datagram(UD) transport method best matches the needs of IP and 87 the need for universality in general as described 88 in[IPoIB_ARCH]. 90 This document specifies IPoIB over IB's unreliable 91 datagram(UD) mode. A separate document will describe the 92 implementation of IP subnets over IB's other transport 93 mechanisms. 95 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 96 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 97 "OPTIONAL" in this document are to be interpreted as described 98 in RFC 2119. 100 2.0 InfiniBand Datalink 102 The document [IPoIB_MCAST] defines the IPoIB link, its setup, 103 and IP multicast over InfiniBand in detail. The following 104 discussion gives a short overview. 106 An IB subnet is formed by a network of IB nodes interconnected 107 either directly or via IB switches. IB subnets may be 108 connected using IB routers to form a fabric made of multiple 109 IB subnets. Multiple IP subnets may be overlaid over this IB 110 cloud. The boundary of this IP subnet is arbitrary and not 111 associated with a physical demarcation. The IPoIB nodes that 112 are members of this subnet are interconnected by an abstract 113 'link'. The link is defined by its members and common 114 characteristics such as the P_Key, link MTU and Q_Key that are 115 defined per 'link'. 117 IPv4 defines a limited-broadcast address over the link. All 118 IPv4 hosts that are members of the IPv4 subnet are members of 119 this address. IPv6 defines a multicast address referred to as 120 the all-IP hosts address. IPoIB associates a multicast GID 121 with these addresses[IPoIB_MCAST]. This multicast GID will 122 henceforth be referred to as the broadcast-GID. The 123 broadcast-GID is required to be setup for an IPoIB subnet to 124 be formed. 126 Every IPoIB interface MUST join the broadcast-GID. This 127 operation returns the MTU and the Q_Key associated with the 128 IPoIB link. Thus the IPoIB subnet(and the link) is formed by 129 the IPoIB nodes joining the broadcast GID. 131 The P_Key is a configuration parameter that must be known 132 before the broadcast-GID can be formed[IPoIB_MCAST]. 134 2.1 IP Support on IPoIB Link 136 The unreliable datagram(UD) mode of communication is supported 137 by all IB elements be they IB routers, HCAs or TCAs. In 138 addition to being the only universal transmission method it 139 supports multicasting, partitioning and a 32-bit CRC[IB_ARCH]. 140 Though multicasting support is optional in IB fabrics, IPoIB 141 architecture requires the participating components to support 142 it [IPoIB_MCAST]. 144 All IPoIB implementations MUST support IP over the unreliable 145 datagram(UD) transport mode of IBA. 147 [ 148 Note to WG: There is an ongoing discussion in the WG with respect to 149 packet encapsulation. A consensus call by the chair on the 150 'ethertype' discussion is awaited. The final draft of this 151 document will reflect the consensus. 153 The decision on 'ethertype' may effect the following two 154 sections: 156 3.0 Maximum Transmission Unit 157 4.0 Frame Format 158 ] 160 3.0 Maximum Transmission Unit 162 The IB architecture supports multiple MTU values: 256, 512, 163 1024, 2048, 4096 bytes. An implementation determines the IPoIB 164 link MTU from the MTU listed in the MCGroupRecord of the 165 broadcast GID[IPoIB_MCAST]. The IPoIB link does not have a 166 default MTU. 168 It is RECOMMENDED that the IP MTU be set equal to that of the 169 IPoIB link MTU. 171 In IPv6 subnets the IP MTU derived from the IPoIB link MTU may 172 be reduced by a Router Advertisement[RFC2461] containing an 173 MTU option which specifies a smaller MTU, or by manual 174 configuration of each node. If a Router Advertisement received 175 on an IPoIB interface has an MTU option specifying an MTU 176 larger than the link MTU or larger than a manually configured 177 value, that MTU option may be logged to system management but 178 must be otherwise ignored. 180 Similarly, the IPv4 MTU may also be reduced from the link MTU 181 value by manual configuration of each node. 183 For purposes of this document, information received from DHCP 184 is considered "manually configured". 186 Ethernet LANs, which are very common, support an MTU of 1500 187 bytes. The IPv6 specification further requires a minimum MTU 188 of 1280 bytes. Therefore it is very appropriate to set the IP 189 MTU to these values depending on the networking needs. It must 190 however be ensured that the IPoIB link MTU is at least 2048 191 bytes. IBA MTUs of smaller values are not optimal for 192 internetworking to other IP subnets. 194 4.0 Frame Format 196 The IP and ARP datagrams are directly encapsulated in IB's 197 Unreliable Datagrams payload. 199 |<------ IB Frame headers -------->|Payload|<-IB trailers -->| 200 +-------+------+---------+---------+-------+---------+-------+ 201 |Local | |Base |Datagram | |Invariant|Variant| 202 |Routing| GRH* |Transport|Extended |IPv4/v6| CRC | CRC | 203 |Header |Header|Header |Transport| /ARP | | | 204 | | | |Header | | | | 205 +-------+------+---------+---------+-------+---------+-------+ 206 Figure 1 208 The InfiniBand specification requires the use of Global 209 Routing Header(GRH)[IPoIB_ARCH] when multicasting or when an 210 InfiniBand packet traverses from one IB subnet to another 211 through an IB router. Its use is optional when used for 212 unicast transmission between nodes within an IB subnet. The 213 IPoIB implementation MUST be able to handle packets received 214 with or without the use of GRH. 216 The IP/ARP datagrams SHALL be encapsulated in IB unreliable 217 datagrams in the payload. The QPs advertised for IP 218 communication MUST NOT be used for other protocols. 220 5.0 IPv6 Stateless Autoconfiguration 222 IB architecture associates an EUI-64 identifier termed the 223 GUID (Globally Unique Identifier) [IPoIB_ARCH, IB_ARCH] with 224 each port. The LID (16 bits) is unique within an IB subnet 225 only. 227 The interface identifier may be chosen from: 229 1) The EUI-64 compliant Globally unique 230 identifier(GUID) assigned by the manufacturer. 232 2) If the IPoIB subnet is fully contained within an IB 233 subnet any of the unique 16-bit LIDs of the port 234 associated with the IPoIB interface. 236 The LID values of a port may change after a 237 reboot/power-cycle of the IB node. Therefore, if a 238 persistent value is desired, it would be prudent to 239 not use the LID to form the interface identifier. 241 On the other hand, the LID provides an identifier 242 that can be used to create a more anonymous IPv6 243 address since the LID is not globally unique and is 244 subject to change over time. 246 It is RECOMMENDED that the link-local address be constructed 247 from the port's EUI-64 identifier as per the rules specified 248 in [RFC2373]. 250 The interface identifier may also be chosen as per the 251 guidelines specified in [RFC3041]. 253 5.1 IPv6 Link Local Address 255 The IPv6 link local address for an IPoIB interface is formed 256 in accordance with the guidelines in [RFC2373]. The link local 257 address is of the format: 259 10 bits 54 bits 64 bits 260 +----------+-----------------------+----------------------------+ 261 |1111111010| (zeros) | Interface Identifier | 262 +----------+-----------------------+----------------------------+ 264 Figure 2 266 6.0 Address Mapping - Unicast 268 Address resolution in IPv4 subnets is accomplished through 269 Address Resolution protocol (ARP)[RFC826]. It is accomplished 270 in IPv6 subnets using the Neighbor discovery 271 protocol[RFC2461]. 273 6.1 Link Information 275 An InfiniBand packet over the UD mode includes multiple 276 headers such as the LRH(local route header), GRH(global route 277 header), BTH(base transport header), DETH(datagram extended 278 header) as depicted in Figure 1 and specified in the 279 InfiniBand architecture[IB_ARCH]. All these headers comprise 280 the link-layer in an IPoIB link. 282 The parameters needed in these IBA headers constitute the 283 link-layer information that needs to be determined before an 284 IP packet may be transmitted across the IPoIB link. 286 The parameters that need to be determined are: 288 a) LID (local identifier) 290 The LID is always needed. A packet always includes the 291 LRH that is targeted at the remote node's LID, or an 292 IB router's LID to get to the remote node in another 293 IB subnet. 295 b) GID (global identifier) 297 The GID is not needed when exchanging information 298 within an IB subnet though it may be included in any 299 packet. It is an absolute necessity when transmitting 300 across the IB subnet since the IB routers use the GID 301 to correctly forward the packets. The source and 302 destination GIDs are fields included in the GRH. 304 The GID, if formed using the GUID, can be used to 305 unambiguously identify an endpoint. 307 c) QPN (queue pair number) 309 Every unicast UD communication is always directed to a 310 particular queue pair(QP) at the peer. 312 d) Q_Key 314 A Q_Key is associated with each unreliable datagram 315 QPN. The received packets must contain a Q_Key that 316 matches the QP's Q_Key to be accepted. 318 e) P_Key 320 A successful communication between two IB nodes using 321 UD mode can occur only if the two nodes have 322 compatible P_Keys. This is referred to as being in the 323 same partition[IB_ARCH]. P_Keys are checked at the 324 receiving channel adapter and may be optionally 325 checked at intermediate switches/IB routers. If the 326 P_Key in the packet does not match the expected P_Key 327 the packet is dropped. 329 f) SL (service level) 331 Every IBA packet contains an SL value. A path in IBA 332 is defined by the three-tuple (source LID, destination 333 LID, SL). The SL in turns is mapped to a virtual 334 lane(VL) at every xCA, switch that sends/forwards the 335 packet [IPoIB_ARCH]. Multiple SLs may be used between 336 two endpoints to provide for load-balancing, SLs may 337 be used for providing a QoS infrastructure, or may be 338 used to avoid deadlocks in the IBA fabric. 340 Another auxiliary piece of information, not included in the 341 IBA headers, is : 343 g) Path rate 345 The InfiniBand architecture defines multiple link 346 speeds. A higher speed transmitter can swamp 347 switches/xCAs. To avoid such congestion every source 348 transmitting at greater than 1x speeds is required to 349 determine the 'path rate' before the data may be 350 transmitted [IB_ARCH]. 352 6.1.1 Link Layer Address/Hardware Address 354 Though the list of information required for a successful 355 transmittal of an IPoIB packet is large not all the 356 information need be determined during the IP address 357 resolution process. 359 The IPoIB link-layer address used in the source/target 360 link-layer address option in IPv6 and the 'hardware address' 361 in IPv4/ARP has the same format. The format is as described 362 below: 364 +--------+--------+--------+--------+ 365 |Reserved| QPN[23-0] | 366 +--------+--------+--------+--------+ 367 | GID[127-96] | 368 + + 369 | GID[95-64] | 370 + + 371 | GID[63-32] | 372 + + 373 | GID[31-0] | 374 +--------+--------+--------+--------+ 376 Figure 3 378 a) Reserved Flags 380 These 8 bits are reserved for future use. These bits 381 MUST be set to zero on send and ignored on receive 382 unless specified differently in a future document. 384 b) Queue Pair Number (QPN) 386 Every unicast communication in IB architecture is 387 directed to a specific queue pair(QP)[IB_ARCH]. This 388 QP number is included in the link description. All IP 389 communication to the relevant IPoIB interface MUST be 390 directed to this QPN. In the case of IPv4 subnets the 391 address resolution protocol(ARP) reply packets are 392 also directed to the same QPN. 394 The choice of the QPN for IP/ARP communication is up 395 to the implementation. 397 c) Global Identifier (GID) 399 This is one of the Global Identifiers(GIDs)[IB_ARCH] 400 of the port associated with the IPoIB interface. IB 401 associates multiple GIDs with a port. It is 402 RECOMMENDED that the GID formed by the combination of 403 the IB subnet prefix and the port's GUID be included 404 in the link-layer/hardware address. 406 6.1.2 Auxiliary Link Information 408 The rest of the parameters are determined as follows: 410 a) Local Identifier(LID) 412 The method of determining the peer's LID is not 413 defined in this document. It is up to the 414 implementation to use any of the IBA approved methods 415 to determine the destination LID. One such method is 416 to use the GID determined during the address 417 resolution, to retrieve the associated LID from the IB 418 routing infrastructure or the SA. 420 It is the responsibility of the administrator to 421 ensure that the IB subnet(s) have unicast connectivity 422 between the IPoIB nodes. The GID exchanged between two 423 endpoints in a multicast message(ARP/ND) does not 424 guarantee the existence of a unicast path between the 425 two. This has to be ensured by the fabric 426 administrator. 428 There may be multiple LIDs, and hence paths, between 429 the endpoints. The criteria for selection of the LIDs 430 are beyond the scope of this document. 432 b) Q_Key 434 The Q_Key received on joining the broadcast-GID MUST 435 be used for all IPoIB communication over the 436 particular IPoIB link. 438 c) P_Key 440 The network administrator is required to setup an 441 IPoIB link by setting up an IB partition and assigning 442 it a unique P_Key[IPoIB_MCAST]. 444 Thus the P_Key to be used in the IP subnet is not 445 discovered but is a configuration parameter. 447 d) Service Level(SL) 449 The method of determining the SL is not defined in 450 this document. The SL is determined by any of the IBA 451 approved methods. 453 e) Path rate 455 The implementation must leverage IB methods to 456 determine the path rate as required. 458 6.2 Address Resolution in IPv4 Subnets 460 The ARP packet header is as defined in [RFC826]. The hardware 461 type is set to 32(decimal) as specified by Internet Assigned 462 Numbers Authority(IANA). The rest of the fields are used as 463 per RFC826. 465 16 bits: hardware type 466 16 bits: protocol 467 8 bits: length of hardware address 468 8 bits: length of protocol address 469 16 bits: ARP operation 471 The remaining fields in the packet hold the sender/target 472 hardware and protocol addresses. 474 [ sender hardware address ] 475 [ sender protocol address ] 476 [ target hardware address ] 477 [ target protocol address ] 479 The hardware address included in the ARP packet will be as 480 specified in section 6.1.1 and depicted in Figure 3. 482 The length of the hardware address used in ARP packet header 483 therefore is 20. 485 6.3 Link-Layer Address in IPv6 487 The Source/Target Link-layer address option is used in Router 488 Solicit, Router advertisements, Redirect, Neighbour 489 Solicitation and Neighbour Advertisement messages when such 490 messages are transmitted on InfiniBand networks. 492 The source/target address option is specified as follows: 494 Type: 495 Source Link-layer address 1 496 Target Link-layer address 2 498 Length: 3 500 Link-layer address: 502 The link-layer address is as specified in section 503 6.1.1 and depicted in Figure 3. 505 7.0 IANA Considerations 507 To support ARP over InfiniBand a value for the Address 508 Resolution Parameter 'Number Hardware Type (hrd)' is required. 509 IANA has assigned the number '32' to indicate 510 InfiniBand[IANA_ARP]. 512 8.0 Security Considerations 514 This document specifies IP transmission over a multicast 515 network. Any network of this kind is vulnerable to a sender 516 claiming another's identity and forge traffic or eavesdrop. It 517 is the responsibility of the higher layers or applications to 518 implement suitable counter-measures if this is a problem. 520 9.0 Acknowledgements 522 The authors would like to thank Bruce Beukema, David Brean, 523 Dan Cassiday, Yaron Haviv, Thomas Narten, Erik Nordmark, Greg 524 Pfister, Jim Pinkerton, Renato Recio, Kevin Reilly, Madhu 525 Talluri and Satya Sharma for their suggestions and many 526 clarifications on the IBA specification. 528 10.0 References 530 [IB_ARCH] InfiniBand Architecture Specification, Volume 1.0a 531 www.infinibandta.org 533 [IPoIB_ARCH] draft-ietf-ipoib-architecture-01.txt 535 [IPoIB_MCAST] draft-ietf-ipoib-link-multicast-00.txt 537 [RFC2373] IP Version 6 Addressing Architecture 539 [RFC2375] IPv6 Multicast Address Assignments 541 [RFC826] An Ethernet Address Resolution Protocol 543 [RFC1700] Assigned Numbers. 545 [RFC2434] Guidelines for Writing an IANA Considerations Section in RFCs 547 [RFC2461] Neighbor Discovery for IP version 6 (IPv6) 549 [RFC3041] Extensions to IPv6 Address Autoconfiguration 551 [IANA] Internet assigned numbers authority, www.iana.org 553 [IANA_ARP] www.iana.org/assignments/arp-parameters 555 11.0 Authors' Address 557 Vivek Kashyap 559 15450, SW Koll Parkway 560 Beaverton, OR 97006 561 USA 563 Phone: +1 503 578 3422 564 Email: vivk@us.ibm.com 565 H.K. Jerry Chu 567 901 San Antonio Road, UMPK17-201 568 Palo Alto, CA 94303-4900 569 USA 571 Phone: +1 650 786-5146 572 Email: jerry.chu@sun.com 574 Full Copyright Statement 576 Copyright (C) The Internet Society (2001). All Rights Reserved. 578 This document and translations of it may be copied and 579 furnished to others, and derivative works that comment on or 580 otherwise explain it or assist in its implementation may be 581 prepared, copied, published and distributed, in whole or in 582 part, without restriction of any kind, provided that the above 583 copyright notice and this paragraph are included on all such 584 copies and derivative works. However, this document itself may 585 not be modified in any way, such as by removing the copyright 586 notice or references to the Internet Society or other Internet 587 organizations, except as needed for the purpose of developing 588 Internet standards in which case the procedures for copyrights 589 defined in the Internet Standards process must be followed, or 590 as required to translate it into languages other than 591 English. 593 The limited permissions granted above are perpetual and will 594 not be revoked by the Internet Society or its successors or 595 assigns. 597 This document and the information contained herein is provided 598 on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET 599 ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR 600 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE 601 USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR 602 ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A 603 PARTICULAR PURPOSE. 605 -- 606 Vivek Kashyap 607 Linux Technology Center, IBM 608 kashyapv@us.ibm.com 609 vivk@us.ibm.com 610 503 578 3422 (o)