| < draft-boutros-l2vpn-evpn-vpws-01.txt | draft-boutros-l2vpn-evpn-vpws-02.txt > | |||
|---|---|---|---|---|
| INTERNET-DRAFT Sami Boutros | INTERNET-DRAFT Sami Boutros | |||
| Intended Status: Standard Track Ali Sajassi | Intended Status: Standard Track Ali Sajassi | |||
| Samer Salam | Samer Salam | |||
| Cisco Systems | Cisco Systems | |||
| John Drake | John Drake | |||
| Juniper Networks | Juniper Networks | |||
| Jeff Tantsura | Jeff Tantsura | |||
| Ericsson | Ericsson | |||
| Expires: August 28, 2013 February 24, 2013 | Expires: April 24, 2014 October 21, 2013 | |||
| VPWS support in E-VPN | VPWS support in E-VPN | |||
| draft-boutros-l2vpn-evpn-vpws-01.txt | draft-boutros-l2vpn-evpn-vpws-02.txt | |||
| Abstract | Abstract | |||
| This document describes how E-VPN can be used to support virtual | This document describes how E-VPN can be used to support virtual | |||
| private wire service (VPWS) in MPLS/IP networks. E-VPN enables the | private wire service (VPWS) in MPLS/IP networks. E-VPN enables the | |||
| following characteristics for VPWS: active/standby as well as | following characteristics for VPWS: single-active as well as all- | |||
| active/active multi-homing with flow-based load-balancing, eliminates | active multi-homing with flow-based load-balancing, eliminates the | |||
| the need for single-segment and multi-segment PW signaling, and | need for single-segment and multi-segment PW signaling, and provides | |||
| provides fast protection using data-plane prefix independent | fast protection using data-plane prefix independent convergence upon | |||
| convergence upon node or link failure. | node or link failure. | |||
| Status of this Memo | Status of this Memo | |||
| This Internet-Draft is submitted to IETF in full conformance with the | This Internet-Draft is submitted to IETF in full conformance with the | |||
| provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
| Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
| Task Force (IETF), its areas, and its working groups. Note that | Task Force (IETF), its areas, and its working groups. Note that | |||
| other groups may also distribute working documents as | other groups may also distribute working documents as | |||
| Internet-Drafts. | Internet-Drafts. | |||
| skipping to change at page 2, line 26 ¶ | skipping to change at page 2, line 26 ¶ | |||
| include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
| the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
| described in the Simplified BSD License. | described in the Simplified BSD License. | |||
| Table of Contents | Table of Contents | |||
| 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
| 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
| 2. BGP Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4 | 2. BGP Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4 | |||
| 3 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 | 3 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 | |||
| 4 E-VPN Comparison to PW Signaling . . . . . . . . . . . . . . . . 5 | 4 EVPN Comparison to PW Signaling . . . . . . . . . . . . . . . . 5 | |||
| 5 ESI Bandwidth Attribute . . . . . . . . . . . . . . . . . . . . 5 | 5 ESI Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . 6 | |||
| 6 VPWS with multiple sites . . . . . . . . . . . . . . . . . . . . 6 | 6 ESI value derivation . . . . . . . . . . . . . . . . . . . . . . 6 | |||
| 7 Security Considerations . . . . . . . . . . . . . . . . . . . . 6 | 7 VPWS with multiple sites . . . . . . . . . . . . . . . . . . . . 6 | |||
| 8 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 6 | 8 Security Considerations . . . . . . . . . . . . . . . . . . . . 6 | |||
| 9 References . . . . . . . . . . . . . . . . . . . . . . . . . . 6 | 9 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 | |||
| 9.1 Normative References . . . . . . . . . . . . . . . . . . . 6 | 10 References . . . . . . . . . . . . . . . . . . . . . . . . . . 7 | |||
| 9.2 Informative References . . . . . . . . . . . . . . . . . . 7 | 10.1 Normative References . . . . . . . . . . . . . . . . . . . 7 | |||
| 10.2 Informative References . . . . . . . . . . . . . . . . . . 7 | ||||
| Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 7 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 7 | |||
| 1 Introduction | 1 Introduction | |||
| This document describes how E-VPN can be used to support virtual | This document describes how EVPN can be used to support virtual | |||
| private wire service (VPWS) in MPLS/IP networks. The use of E-VPN | private wire service (VPWS) in MPLS/IP networks. The use of EVPN | |||
| mechanisms for VPWS applies the benefits of E-VPN to p2p services. | mechanisms for VPWS brings the benefits of EVPN to p2p services. | |||
| These benefits include active/standby AC redundancy as well as | These benefits include single-active redundancy as well as all-active | |||
| active/active multi-homing with flow-based load-balancing. | redundancy with flow-based load-balancing. Furthermore, the use of | |||
| Furthermore, the use of E-VPN for VPWS eliminates the need for | EVPN for VPWS eliminates the need for signaling single-segment and | |||
| signaling single-segment and multi-segment PWs for p2p Ethernet | multi-segment PWs for p2p Ethernet services. | |||
| services. | ||||
| [E-VPN] has the ability to forward customer traffic to/from a given | [EVPN] has the ability to forward customer traffic to/from a given | |||
| customer Attachment Circuit (AC), aka Ethernet Segment in E-VPN | customer Attachment Circuit (AC), aka Ethernet Segment in EVPN | |||
| terminology, without any MAC lookup. This capability is ideal in | terminology, without any MAC lookup. This capability is ideal in | |||
| providing p2p services (aka VPWS services). [MEF] defines EVPL | providing p2p services (aka VPWS services). [MEF] defines Ethernet | |||
| service as p2p service between a pair of ACs (designated by VLANs). | Virtual Private Line (EVPL) service as p2p service between a pair of | |||
| EVPL can be considered as a VPWS with only two ACs. In delivering an | ACs (designated by VLANs). EVPL can be considered as a VPWS with only | |||
| EVPL service, the traffic forwarding capability of E-VPN using only a | two ACs. In delivering an EVPL service, the traffic forwarding | |||
| pair of Ethernet AD routes is used; whereas, for more general VPWS, | capability of EVPN based on the exchange of a pair of Ethernet AD | |||
| traffic forwarding capability of E-VPN using a group of Ethernet AD | routes is used; whereas, for more general VPWS, traffic forwarding | |||
| routes (one Ethernet AD route per AC/segment) is used. Since in VPWS | capability of EVPN based on the exchange of a group of Ethernet AD | |||
| services, the traffic from an originating Ethernet Segment can go | routes (one Ethernet AD route per AC/segment) is used. In a VPWS | |||
| only to a single destination Ethernet Segment, no MAC lookup is | service, the traffic from an originating Ethernet Segment can be | |||
| needed and the MPLS label associated with the per-EVI Ethernet AD | forwarded only to a single destination Ethernet Segment; hence, no | |||
| route can be used in forwarding user traffic to the destination AC. | MAC lookup is needed and the MPLS label associated with the per-EVI | |||
| Ethernet AD route can be used in forwarding user traffic to the | ||||
| destination AC. | ||||
| In current PW redundancy mechanisms, convergence time is a function | In current PW redundancy mechanisms, convergence time is a function | |||
| of control plane convergence characteristics. However, with E-VPN it | of control plane convergence characteristics. However, with EVPN it | |||
| is possible to attain faster convergence through the use of data- | is possible to attain faster convergence through the use of data- | |||
| plane prefix independent convergence upon node or link failure. | plane prefix independent convergence, upon node or link failure. | |||
| This document proposes the use of the Ethernet AD route to signal | This document proposes the use of the Ethernet AD route to signal | |||
| labels for P2P Ethernet services. As with E-VPN, the Ethernet Segment | labels for P2P Ethernet services. As with EVPN, the Ethernet Segment | |||
| route can be used to synchronize state between the PEs attached to | route can be used to synchronize state between the PEs attached to | |||
| the same multi-homed Segment. | the same multi-homed Ethernet Segment. | |||
| 1.1 Terminology | 1.1 Terminology | |||
| The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
| "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | |||
| document are to be interpreted as described in RFC 2119 [RFC2119]. | document are to be interpreted as described in RFC 2119 [RFC2119]. | |||
| MAC: Media Access Control | MAC: Media Access Control | |||
| MPLS: Multi Protocol Label Switching. | MPLS: Multi Protocol Label Switching. | |||
| OAM: Operations, Administration and Maintenance. | OAM: Operations, Administration and Maintenance. | |||
| PE: Provide Edge Node. | PE: Provide Edge Node. | |||
| CE: Customer Edge device e.g., host or router or switch. | CE: Customer Edge device e.g., host or router or switch. | |||
| EVI: E-VPN Instance. | EVI: EVPN Instance. | |||
| Single-Active Mode: When a device or a network is multi-homed to two | ||||
| or more PEs and when only a single PE in such redundancy group can | ||||
| forward traffic to/from the multi-homed device or network for a given | ||||
| VLAN, then such multi-homing or redundancy is referred to as "Single- | ||||
| Active". | ||||
| All-Active: When a device is multi-homed to two or more PEs and when | ||||
| all PEs in such redundancy group can forward traffic to/from the | ||||
| multi-homed device for a given VLAN, then such multi-homing or | ||||
| redundancy is referred to as "All-Active". | ||||
| 2. BGP Extensions | 2. BGP Extensions | |||
| [E-VPN] defines a new BGP NLRI for advertising different route types | [EVPN] defines a new BGP NLRI for advertising different route types | |||
| for E-VPN operation. This document does not define any new BGP | for EVPN operation. This document does not define any new BGP | |||
| messages, but rather re-purposes one of the routes as described next. | messages, but rather re-purposes one of the routes as described next. | |||
| This document proposes the use of the per EVI Ethernet AD route to | This document proposes the use of the per EVI Ethernet AD route to | |||
| signal P2P services. The Ethernet Segment Identifier field is set to | signal P2P services. The Ethernet Segment Identifier field is set to | |||
| the ESI of the attachment circuit of the VPWS service instance. The | the ESI of the attachment circuit of the VPWS service instance. The | |||
| Ethernet Tag field is set to 0 in the case of an Ethernet Private | Ethernet Tag field is set to 0 in the case of an Ethernet Private | |||
| Wire service, and to the VLAN identifier associated with the service | Wire service, and to the VLAN identifier associated with the service | |||
| for Ethernet Virtual Private Wire service. The route is associated | for Ethernet Virtual Private Wire service. The route is associated | |||
| with a Route-Target (RT) extended community attribute that identifies | with a Route-Target (RT) extended community attribute that identifies | |||
| the service instance (together with the Ethernet Tag field when non- | the service instance (together with the Ethernet Tag field when non- | |||
| zero). | zero). | |||
| 3 Operation | 3 Operation | |||
| The following figure shows an example of a P2P service deployed with | The following figure shows an example of a P2P service deployed with | |||
| E-VPN. | EVPN. | |||
| Ethernet Ethernet | Ethernet Ethernet | |||
| Native |<---------E-VPN Instance------------>| Native | Native |<--------- EVPN Instance ----------->| Native | |||
| Service | | Service | Service | | Service | |||
| (AC) | |<-PSN1->| |<-PSN2->| | (AC) | (AC) | |<-PSN1->| |<-PSN2->| | (AC) | |||
| | V V V V V V | | | V V V V V V | | |||
| | +-----+ +-----+ +-----+ +-----+ | | | +-----+ +-----+ +-----+ +-----+ | | |||
| +----+ | | PE1 |======|ASBR1|==|ASBR2|===| PE3 | | +----+ | +----+ | | PE1 |======|ASBR1|==|ASBR2|===| PE3 | | +----+ | |||
| | |-------+-----+ +-----+ +-----+ +-----+-------| | | | |-------+-----+ +-----+ +-----+ +-----+-------| | | |||
| | CE1| | | |CE2 | | | CE1| | | |CE2 | | |||
| | |-------+-----+ +-----+ +-----+ +-----+-------| | | | |-------+-----+ +-----+ +-----+ +-----+-------| | | |||
| +----+ | | PE2 |======|ASBR3|==|ASBR4|===| PE4 | | +----+ | +----+ | | PE2 |======|ASBR3|==|ASBR4|===| PE4 | | +----+ | |||
| ^ +-----+ +-----+ +-----+ +-----+ ^ | ^ +-----+ +-----+ +-----+ +-----+ ^ | |||
| | Provider Edge 1 ^ Provider Edge 2 | | | Provider Edge 1 ^ Provider Edge 2 | | |||
| | | | | | | | | |||
| | | | | | | | | |||
| | E-VPN Inter-provider point | | | EVPN Inter-provider point | | |||
| | | | | | | |||
| |<---------------- Emulated Service -------------------->| | |<---------------- Emulated Service -------------------->| | |||
| iBGP sessions will be established between PE1, PE2, ASBR1 and ASBR3, | iBGP sessions are established between PE1, PE2, ASBR1 and ASBR3, | |||
| possibly via a BGP route-reflector. Similarly, iBGP sessions will be | possibly via a BGP route-reflector. Similarly, iBGP sessions are | |||
| established between PE3, PE4, ASBR2 and ASBR4. eBGP sessions will be | established between PE3, PE4, ASBR2 and ASBR4. eBGP sessions are | |||
| established among ASBR1, ASBR2, ASBR3, and ASBR4. | established among ASBR1, ASBR2, ASBR3, and ASBR4. | |||
| All PEs and ASBRs are enabled for the E-VPN SAFI, and exchange E-VPN | All PEs and ASBRs are enabled for the EVPN SAFI, and exchange EVPN | |||
| Ethernet A-D routes - one route per AC. The ASBRs re-advertise the | Ethernet A-D routes - one route per AC. The ASBRs re-advertise the | |||
| Ethernet A-D routes with Next Hop attribute set to their IP | Ethernet A-D routes with Next Hop attribute set to their IP | |||
| addresses. The link between the CE and the PE is either a C-tagged or | addresses. The link between the CE and the PE is either a C-tagged or | |||
| S-tagged interface, as described in [802.1Q], that can carry a single | S-tagged interface, as described in [802.1Q], that can carry a single | |||
| VLAN tag or two nested VLAN tags. This interface is set up as a trunk | VLAN tag or two nested VLAN tags. This interface is set up as a trunk | |||
| with multiple VLANs. | with multiple VLANs. | |||
| A VPWS with multiple sites or multiple EVPL services on the same CE | A VPWS with multiple sites or multiple EVPL services on the same CE | |||
| port can be included in one EVI between 2 or more PEs. An Ethernet | port can be included in one EVI between 2 or more PEs. An Ethernet | |||
| Tag corresponding to each P2P connection and known to both PEs is | Tag corresponding to each P2P connection and known to both PEs is | |||
| used to identify the services multiplexed in the same EVI. | used to identify the services multiplexed in the same EVI. | |||
| For CE multi-homing, the Ethernet AD Route encodes the ESI associated | For CE multi-homing, the Ethernet AD Route encodes the ESI associated | |||
| with the CE. This allows flow-based load-balancing of traffic between | with the CE. This allows flow-based load-balancing of traffic between | |||
| PEs connected to the same multi-homed CE. The VPN ID MUST be the same | PEs connected to the same multi-homed CE. The VPN ID MUST be the same | |||
| on both PEs attached to the site. The Ethernet Segment route may be | on both PEs attached to the site. The Ethernet Segment route may be | |||
| used too, for discovery of multi-homed CEs. In all cases traffic | used too, for discovery of multi-homed CEs. In all cases traffic | |||
| follows the transport paths, which may be asymmetric. | follows the transport paths, which may be asymmetric. | |||
| 4 E-VPN Comparison to PW Signaling | 4 EVPN Comparison to PW Signaling | |||
| In E-VPN, service endpoint discovery and label signaling are done | In EVPN, service endpoint discovery and label signaling are done | |||
| concurrently using BGP. Whereas, with VPWS based on [RFC4448], label | concurrently using BGP. Whereas, with VPWS based on [RFC4448], label | |||
| signaling is done via LDP and service endpoint discovery is either | signaling is done via LDP and service endpoint discovery is either | |||
| through manual provisioning or through BGP. In VPWS, redundancy is | through manual provisioning or through BGP. In existing | |||
| limited to Active/Standby mode, while with E-VPN both Active/Active | implementation of VPWS using pseudowires(PWs), redundancy is limited | |||
| and Active/Standby redundancy modes can be supported. In VPWS, backup | to single-active mode, while with EVPN implementation of VPWS both | |||
| PWs are not used to carry traffic, while E-VPN traffic can be load- | single-active and all-active redundancy modes can be supported. In | |||
| balanced among primary and secondary PEs. On link or node failure, E- | existing implementation with PWs, backup PWs are not used to carry | |||
| VPN can trigger failover with the withdrawal of a single BGP route | traffic, while with EVPN, traffic can be load-balanced among primary | |||
| per service, whereas with VPWS PW redundancy, the failover sequence | and secondary PEs. Upon link or node failure, EVPN can trigger | |||
| requires exchange of two control plane messages: one message to | failover with the withdrawal of a single BGP route per service, | |||
| deactivate the group of primary PWs and a second message to activate | whereas with VPWS PW redundancy, the failover sequence requires | |||
| the group of backup PWs associated with the access link. Finally, E- | exchange of two control plane messages: one message to deactivate the | |||
| VPN may employ data plane local repair mechanisms not available in | group of primary PWs and a second message to activate the group of | |||
| VPWS. | backup PWs associated with the access link. Finally, EVPN may employ | |||
| data plane local repair mechanisms not available in VPWS. | ||||
| 5 ESI Bandwidth Attribute | ||||
| The ESI Bandwidth Attribute is a new optional BGP attribute that will | 5 ESI Bandwidth | |||
| be associated with the Ethernet AD route used to realize the EVPL | ||||
| services. | ||||
| +---------------------------------------+ | The ESI Bandwidth will be encoded using the Link Bandwidth Extended | |||
| | Type (2 octets) | | community defined in [draft-ietf-idr-link-bandwidth] and associated | |||
| +---------------------------------------+ | with the Ethernet AD route used to realize the EVPL services. | |||
| | Length (2 octets) | | ||||
| +---------------------------------------+ | ||||
| | Flags (1 Octet) | | ||||
| +---------------------------------------+ | ||||
| | Reserved=0(1 Octet) | | ||||
| +---------------------------------------+ | ||||
| | Reverse SENDER_TSPEC | | ||||
| +---------------------------------------+ | ||||
| The content of the SENDER_TSPEC are as defined in [RFC 2210] section | ||||
| 3.1. | ||||
| When a PE receives this attribute for a given EVPL it MUST request | When a PE receives this attribute for a given EVPL it MUST request | |||
| the appropriate resources described in the SENDER_TSPEC from the PSN | the required bandwidth from the PSN towards the other EVPL service | |||
| towards the other EVPL service destination PE originating the | destination PE originating the message. When resources are allocated | |||
| message. When resources are allocated from the PSN for a given EVPL | from the PSN for a given EVPL service, then the PSN SHOULD account | |||
| service, then the PSN SHOULD account for the Bandwidth requested by | for the Bandwidth requested by this EVPL service. | |||
| this EVPL service. | ||||
| In the case where PSN resources are not available, the PE receiving | In the case where PSN resources are not available, the PE receiving | |||
| this attribute MUST re-send its local Ethernet AD routes for this | this attribute MUST re-send its local Ethernet AD routes for this | |||
| EVPL service with the ESI Bandwidth attribute and with the Flags set | EVPL service with the ESI Bandwidth = All FFs to declare that the | |||
| to 1 "PSN Resources Unavailable". | "PSN Resources Unavailable". | |||
| 6 VPWS with multiple sites | 6 ESI value derivation | |||
| The 10 bytes ESI value will contain:- | ||||
| 1) 6-byte System-ID that is globally unique. These 6 bytes can be | ||||
| auto derived using a mechanism similar to the one used for | ||||
| automating B-MAC Address Assignment in [PBB-EVPN]. | ||||
| 2) 4-byte Local-AC-ID that is unique within each PE. | ||||
| The combination of System-ID and Local-AC-ID makes the associated AC- | ||||
| ID globally unique. A pair of such globally unique AC-ID identifies a | ||||
| point-to-point service (EVPL or EPL) uniquely in the provider | ||||
| network. | ||||
| 7 VPWS with multiple sites | ||||
| The future revision of this draft will describe how a VPWS among | The future revision of this draft will describe how a VPWS among | |||
| multiple sites (full mesh of P2P connections - one per pair of sites) | multiple sites (full mesh of P2P connections - one per pair of sites) | |||
| can be setup automatically without any explicit provisioning of P2P | can be setup automatically without any explicit provisioning of P2P | |||
| connections among the sites. | connections among the sites. | |||
| 7 Security Considerations | 8 Security Considerations | |||
| This document does not introduce any additional security constraints. | This document does not introduce any additional security constraints. | |||
| 8 IANA Considerations | 9 IANA Considerations | |||
| TBD | TBD | |||
| 9 References | 10 References | |||
| 9.1 Normative References | 10.1 Normative References | |||
| [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate | [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate | |||
| Requirement Levels", BCP 14, RFC 2119, March 1997. | Requirement Levels", BCP 14, RFC 2119, March 1997. | |||
| [RFC 2210] Wroclawski, J. "The Use of RSVP with IETF Integrated | 10.2 Informative References | |||
| Services", RFC 2210, September 1997 | ||||
| 9.2 Informative References | ||||
| [EVPN-REQ] A. Sajassi, R. Aggarwal et. al., "Requirements for | [EVPN-REQ] A. Sajassi, R. Aggarwal et. al., "Requirements for | |||
| Ethernet VPN", draft-ietf-l2vpn-evpn-req-00.txt. | Ethernet VPN", draft-ietf-l2vpn-evpn-req-00.txt. | |||
| [EVPN] A. Sajassi, R. Aggarwal et. al., "BGP MPLS Based Ethernet | [EVPN] A. Sajassi, R. Aggarwal et. al., "BGP MPLS Based Ethernet | |||
| VPN", draft-ietf-l2vpn-evpn-00.txt. | VPN", draft-ietf-l2vpn-evpn-04.txt. | |||
| [PBB-EVPN] A. Sajassi et. al., "PBB-EVPN", draft-ietf-l2vpn-pbb-evpn- | ||||
| 05.txt. | ||||
| [draft-ietf-idr-link-bandwidth] P. Mohapatra, R. Fernando, "BGP Link | ||||
| Bandwidth Extended Community", draft-ietf-idr-link-bandwidth-06.txt | ||||
| Authors' Addresses | Authors' Addresses | |||
| Sami Boutros | Sami Boutros | |||
| Cisco | Cisco | |||
| 170 West Tasman Drive | 170 West Tasman Drive | |||
| San Jose, CA 95134, US | San Jose, CA 95134, US | |||
| Email: sboutros@cisco.com | Email: sboutros@cisco.com | |||
| Ali Sajassi | Ali Sajassi | |||
| End of changes. 34 change blocks. | ||||
| 100 lines changed or deleted | 118 lines changed or added | |||
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ | ||||