idnits 2.17.1 draft-iab-bgparch-01.txt: ** The Abstract section seems to be numbered Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document is more than 15 pages and seems to lack a Table of Contents. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 136 instances of too long lines in the document, the longest one being 1 character in excess of 72. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 987: '... protocol design SHOULD therefore cons...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (May 2001) is 8382 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '11' is mentioned on line 1033, but not defined ** Obsolete normative reference: RFC 2460 (ref. '3') (Obsoleted by RFC 8200) ** Obsolete normative reference: RFC 1519 (ref. '5') (Obsoleted by RFC 4632) ** Obsolete normative reference: RFC 1771 (ref. '7') (Obsoleted by RFC 4271) == Outdated reference: A later version (-13) exists of draft-ietf-idr-as4bytes-02 Summary: 13 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Architecture Board G. Huston 2 Internet Draft Telstra 3 Document: draft-iab-bgparch-01.txt May 2001 4 Category: Informational 6 Architectural Requirements for 7 Inter-Domain Routing in the Internet 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance with 12 all provisions of Section 10 of RFC 2026 [1]. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. Internet-Drafts are draft documents valid for a maximum of 18 six months and may be updated, replaced, or obsoleted by other 19 documents at any time. It is inappropriate to use Internet- Drafts 20 as reference material or to cite them other than as "work in 21 progress." 23 The list of current Internet-Drafts can be accessed at 24 http://www.ietf.org/ietf/1id-abstracts.txt 26 The list of Internet-Draft Shadow Directories can be accessed at 27 http://www.ietf.org/shadow.html. 29 1. Abstract 31 This draft examines the various longer term trends visible within 32 the characteristics of the Internet's BGP table and identifies a 33 number of operational practices and protocol factors that contribute 34 to these trends. The potential impacts of these practices and 35 protocol properties on the scaling properties of the inter-domain 36 routing space are examined. 38 These impacts include the potential for exhaustion of the existing 39 Autonomous System number space, increasing convergence times for 40 selection of stable alternate paths following withdrawal of route 41 announcements, the stability of table entries, and the average 42 prefix length of entries in the BGP table. The larger long term 43 issue is that of an increasingly denser inter-connectivity mesh 44 between AS's, causing a finer degree of granularity of inter-domain 45 policy and finer levels of control to undertake inter-domain traffic 46 engineering. 48 Various approaches to a refinement of the inter-domain routing 49 protocol and associated operating practices that may provide 50 superior scaling properties are identified as an area for further 51 investigation. 53 2. Network Scale and Inter-Domain Routing 55 Are there inherent scaling limitations in the technology of the 56 Internet, or its architecture of deployment, that may impact on the 57 ability of the Internet to meet escalating levels of demand? There 58 are a number of potential areas to search for such limitations. 59 These include the capacity of transmission systems, packet switching 60 capacity, the continued availability of protocol addresses, and the 61 capability of the routing system to produce a stable view of the 62 overall topology of the network. In this study we will look at this 63 latter capability with a view to identifying some aspects of the 64 scaling properties of the Internet's routing system. 66 The basic structure of the Internet is a collection of networks, or 67 Autonomous Systems (AS's) that are interconnected to form a 68 connected domain. Each AS uses an interior routing system to 69 maintain a coherent view of the topology within the AS, and uses an 70 exterior routing system to maintain adjacency information with 71 neighboring AS's and thereby create a view of the connectivity of 72 the entire system. This network-wide connectivity is described in 73 the routing table used by the BGP4 protocol (referred to as the 74 Routing Information Base, or RIB). Each entry in the table refers to 75 a distinct route. The attributes of the route are used to determine 76 the best path from the local AS to the AS that is originating the 77 route. Determining the 'best path' in this case is determining which 78 routing advertisement and associated next hop address is the most 79 preferred by the local AS. Within each local BGP-speaking router 80 this preferred route is then loaded into the Forwarding Information 81 Base (or FIB), for use by the local router's forwarding engine. The 82 BGP routing system is not aware of finer level of topology within 83 the local AS or within any remote AS. From this perspective BGP can 84 be seen as an inter-AS connectivity maintenance protocol, as 85 distinct from a link level topology management protocol, and the BGP 86 routing table can be viewed as a description of the current 87 connectivity of the Internet using an AS as the basic element of 88 connectivity computation. 90 There is an associated dimension of policy determination within the 91 routing table. If an AS advertises a route to a neighboring AS, the 92 local AS is offering to accept traffic from the neighboring AS which 93 is ultimately destined to addresses described by the advertised 94 routing entry. If the local AS does not originate the route, then 95 the inference is that the local AS is willing to undertake the role 96 of transit provider for this traffic on behalf of some third party. 97 Similarly, an AS may or may not choose to accept a route from a 98 neighbor. Accepting a route implies that under some circumstances, 99 as determined by the local route selection parameters, the local AS 100 will use the neighboring AS to reach addresses spanned by the route. 101 The BGP routing domain is intended to maintain a coherent view of 102 the connectivity of the inter-AS domain, where connectivity is 103 expressed as a preference for 'shortest paths' to reach any 104 destination address as modulated by the connectivity policies 105 expressed by each AS, and coherence is expressed as a global 106 constraint that none of the paths contains loops or dead ends. The 107 elements of the BGP routing domain are routing entries, expressed as 108 a span of addresses. All addresses advertised within each routing 109 entry share a common origin AS and a common connectivity policy. The 110 total size of the BGP table is therefore a metric of the number of 111 distinct routes within the Internet, where each route describes a 112 contiguous set of addresses that share a common origin AS and a 113 common reachability policy. 115 When the scaling properties of the Internet were studied in the 116 early 1990s two critical factors identified in the study were, not 117 surprisingly, routing and addressing [2]. As more devices connect to 118 the Internet they consume addresses, and the associated function of 119 maintaining reachability information for these addresses, with an 120 assumption of an associated growth in the number of distinct 121 provider networks and the number of distinct connectivity policies, 122 implies ever larger routing tables. The work in studying the 123 limitations of the 32 bit IPv4 address space produced a number of 124 outcomes, including the specification of IPv6 [3], as well as the 125 refinement of techniques of network address translation [4] intended 126 to allow some degree of transparent interaction between two networks 127 using different address realms. Growth in the routing system is not 128 directly addressed by these approaches, as the routing space is the 129 cross product of the complexity of the inter-AS topology of the 130 network, multiplied by the number of distinct connectivity policies 131 multiplied by the degree of fragmentation of the address space. For 132 example, use of NAT may reduce the pressure on the number of public 133 addresses required by a single connected network, but it does not 134 necessarily imply that the network's connectivity policies can be 135 subsumed within the aggregated policy of a single upstream provider. 137 When an AS advertises a block of addresses into the exterior routing 138 space this entry is generally carried across the entire exterior 139 routing domain of the Internet. To measure the common 140 characteristics of the global routing table, it is necessary to 141 establish a point in the default-free part of the exterior routing 142 domain and examine the BGP routing table that is visible at that 143 point. 145 3. Measurements of the total size of the BGP Table 147 Measurements of the size of the routing table were somewhat sporadic 148 to start, and a number of measurements were taken at approximate 149 monthly intervals from 1988 until 1992 by Merit [5]. This effort was 150 resumed in 1994 by Erik-Jan Bos at Surfnet in the Netherlands, who 151 commenced measuring the size of the BGP table at hourly intervals in 152 1994. This measurement technique was adopted by the author in 1997, 153 using a measurement point located at the edge of AS 1221 at Telstra 154 in Australia, again using an hourly interval for the measurement. 155 The initial measurements were of the number of routing entries 156 contained within the set of selected best paths. These measurements 157 were expanded to include the number of AS numbers, number of AS 158 paths, and a set of measurements relating to the prefix size of 159 routing table entries. 161 We now have a view of the dynamics of the Internet's routing table 162 growth, that spans some 13 years, and a very detailed view spanning 163 the most recent seven years [6]. Looking at just the total size of 164 the BGP routing table over this period, it is possible to identify 165 four distinct phases of inter-AS routing practice in the Internet. 167 3.1 Pre-CIDR Growth 169 The initial characteristics of the routing table size from 1988 170 until April 1994 show definite characteristics of exponential 171 growth. If continued unchecked, this growth would have lead to 172 saturation of the available BGP routing table space in the non- 173 default routers of the time within a small number of years. 175 Estimates of the time at which this would've happened varied 176 somewhat from study to study, but the overall general theme of these 177 observations was that the growth rates of the BGP routing table were 178 exceeding the growth in hardware and software capability of the 179 deployed network, and that at some point in the mid-1990's, the BGP 180 table size would have grown to the point where it was larger than 181 the capabilities of available equipment to support. 183 3.2 CIDR Deployment 185 The response from the engineering community was the introduction of 186 a hierarchy into the inter-domain routing system. The intent of the 187 hierarchical routing structure was to allow a provider to merge the 188 routing entries for its customers into a single routing entry that 189 spanned its entire customer base. The practical aspects of this 190 change was the introduction of routing protocols that dispensed with 191 the requirement for the Class A, B and C address delineation, 192 replacing this scheme with a routing system that carried an address 193 prefix and an associated prefix length. This approached was termed 194 Classless Inter-Domain Routing (CIDR) [5]. 196 A concerted effort was undertaken in 1994 and 1995 to deploy CIDR 197 routing in the Internet, based on encouraging deployment of the 198 CIDR-capable version of the BGP protocol, BGP4 [7]. The effects of 199 this effort are visible in the history of the routing table, where 200 the routing table remained constant for some 14 months at 20,000 201 entries in 1994 and 1995. 203 The intention of CIDR was one of hierarchical provider address 204 aggregation, where a network provider was allocated an address block 205 from an address registry, and the provider announced this entire 206 block into the exterior routing domain as a single entry with a 207 single routing policy. Customers of the provider were encouraged to 208 use a sub-allocation from the provider's address block, and these 209 smaller routing elements were aggregated by the provider and not 210 directly passed into the exterior routing domain. During 1994 the 211 size of the routing table remained relatively constant at some 212 20,000 entries as the growth in the number of providers announcing 213 address blocks was matched by a corresponding reduction in the 214 number of address announcements as a result of CIDR aggregation. 216 3.3 CIDR Growth 218 For the next four years until the start of 1998, CIDR proved 219 effective in damping unconstrained growth in the BGP routing table. 220 During this period, the BGP table grew at an approximate linear 221 rate, adding some 10,000 entries per year. 223 A close examination of the table reveals a greater level of 224 stability in the routing system at this time. The short term 225 (hourly) variation in the number of announced routes reduced, both 226 as a percentage of the number of announced routes, and also in 227 absolute terms. One of the other benefits of using large aggregate 228 address blocks is that instability at the edge of the network is not 229 immediately propagated into the routing core. The instability at the 230 last hop is absorbed at the point where an aggregate route is used 231 in place of a collection of more specific routes. This, coupled with 232 widespread adoption of BGP route flap damping, was been every 233 effective in reducing the short term instability in the routing 234 space during this period. 236 3.4 Current Growth 237 In late 1998 the trend of growth in the BGP table size changed 238 radically, and the growth for the past two years is again showing 239 all the signs of a re-establishment of a growth trend with strong 240 correlation to an exponential growth model. This change in the 241 growth trend appears to indicate that pressure to use hierarchical 242 address allocations and CIDR has been unable to keep pace with the 243 levels of growth of the Internet, and some additional factors are 244 became apparent in the Internet. This has lead to a growth pattern 245 in the total size of the BGP table that has more in common with a 246 compound growth model than a linear model. A good fit of the data 247 for the period from January 1999 until December 2000 is a compound 248 growth model of 42% growth per year. 250 An initial observation is that this growth pattern points to some 251 weakening of the hierarchical model of connectivity and routing 252 within the Internet. To identify the characteristics of this recent 253 trend it is necessary to look at a number of related characteristics 254 of the routing table. 256 BGP table size data for the first quarter of 2001 shows different 257 trends at various measurement points in the Internet. Some 258 measurement points where the local AS has a relative larger number 259 of more specific routes show a steady state for the first quarter of 260 2001 with no appreciable growth, while other measurement points 261 where the local AS has a lower number of more specific routes 262 initially show a continued growth at the same trend rate for the 263 first quarter of 2001. Data for April 2001 has resumed the compound 264 growth trend, indicating that the data for the first three months of 265 2001 is most likely a short term hiatus of the underlying growth 266 factors. 268 4. Related Measurements derived from BGP Table 270 The level of analysis of the BGP routing table has been extended in 271 an effort to identify the factors contributing to this growth, and 272 to determine whether this leads to some limiting factors in the 273 potential size of the routing space. Analysis includes measuring the 274 number of AS's in the routing system, and the number of distinct AS 275 paths, the range of addresses spanned by the table and average span 276 of each routing entry. 278 4.1 AS Number Consumption 280 Each network that is multi-homed within the topology of the Internet 281 and wishes to express a distinct external routing policy must use a 282 unique AS number to associate its advertised addresses with such a 283 policy. In general, each network is associated with a single AS, and 284 the number of AS's in the default-free routing table tracks the 285 number of entities that have unique routing policies. There are some 286 exceptions to this, including large global transit providers with 287 varying regional policies, where multiple AS's are associated with a 288 single network, but such exceptions are relatively uncommon. 290 The number of unique AS's present in the BGP table has been tracked 291 since late 1996, and the trend of AS number deployment over the past 292 four years is also one that matches a compound growth model with a 293 growth rate of 51% per year. As of the start of May 2001 there were 294 some 10,700 AS's visible in the BGP table. At a continued rate of 295 growth of 51% p.a., the 16 bit AS number space will be fully 296 deployed by August 2005. Work is underway within the IETF to modify 297 the BGP protocol to carry AS numbers in a 32-bit field. [8] While 298 the protocol modifications are relatively straightforward, the major 299 responsibility rests with the operations community to devise a 300 transition plan that will allow gradual transition into this larger 301 AS number space. 303 4.2 Address Consumption 305 It is also possible to track the total amount of address space 306 advertised within the BGP routing table. At the start of 2001 the 307 routing table encompassed 1,081,131,733 addresses, or some 25.17% of 308 the total IPv4 address space. This has grown from 1,019,484,655 309 addresses in November 1999. However, there are a number of /8 310 prefixes that are periodically announced and withdrawn from the BGP 311 table, and if the effects of these prefixes is removed, a compound 312 growth model against the previous 12 months of data of this metric 313 yields a best fit model of growth of 7% per year in the total number 314 of addresses spanned by the routing table. 316 Compared to the 42% growth in the number of routing advertisements, 317 it would appear that much of the growth of the Internet in terms of 318 growth in the number of connected devices is occurring behind 319 various forms of NAT gateways. In terms of solving the perceived 320 finite nature of the address space identified just under a decade 321 ago, the Internet appears so far to have embraced the approach of 322 using NATs, irrespective of their various perceived functional 323 shortcomings. [9] This also supports the observation of smaller 324 address fragments supporting distinct policies in the BGP table, as 325 such small address blocks may encompass arbitrarily large networks 326 located behind one or more NAT gateways. 328 4.3 Granularity of Table Entries 330 The intent of CIDR aggregation was to support the use of large 331 aggregate address announcements in the BGP routing table. To check 332 whether this is still the case the average span of each BGP 333 announcement has been tracked for the past 12 months. The data 334 indicates a decline in the average span of a BGP advertisement from 335 16,000 individual addresses in November 1999 to 12,100 in December 336 2000. This corresponds to an increase in the average prefix length 337 from /18.03 to /18.44. Separate observations of the average prefix 338 length used to route traffic in operation networks in late 2000 339 indicate an average length of 18.1 [11]. This trend is potentially 340 cause for concern as it implies the increasing spread of traffic 341 over greater numbers of increasingly finer forwarding table entries. 342 This, in turn, has implications for the design of high speed core 343 routers, particularly when extensive use is made of a small number 344 of very high speed cached forwarding entries within the switching 345 subsystem of a router's design. 347 A similar observation can be made regarding the number of addresses 348 advertised per AS. In December 1999 each AS advertised an average of 349 161,900 addresses (equivalent to a prefix length of /14.69, and in 350 January 2001 this average has fallen to 115,800 addresses, an 351 equivalent prefix length of /15.18. 353 This points to increasingly finer levels of routing detail being 354 announced into the global routing domain. This, in turn, supports 355 the observation that the efficiencies of hierarchical routing 356 structures are no longer being realized within the deployed 357 Internet. Instead, increasingly finer levels of routing detail are 358 being announced globally in the BGP tables. The most likely cause of 359 this trend of finer levels of routing granularity is an increasingly 360 dense interconnection mesh, where more networks are moving from a 361 single-homed connection with hierarchical addressing and routing 362 into multi-homed connections without any hierarchical structure. The 363 spur for this increasingly dense connectivity mesh in the Internet 364 may well be the declining unit costs of communications bearer 365 services coupled with a common perception that richer sets of 366 adjacencies yields greater levels of service resilience. 368 4.4 Prefix Length Distribution 370 In addition to looking at the average prefix length, the analysis of 371 the BGP table also includes an examination of the number of 372 advertisements of each prefix length. 374 An extensive program commenced in the mid-nineties to move away from 375 intense use of the Class C space and to encourage providers to 376 advertise larger address blocks, as part of the CIDR effort. This 377 has been reinforced by the address registries who have used provider 378 allocation blocks that correspond to a prefix length of /19 and, 379 more recently, /20. 381 These measures were introduced in the mid-90's when there were some 382 20,000 - 30,000 entries in the BGP table. Some six years later in 383 April 2001 it is interesting to note that of the 108,000 entries in 384 the routing table, some 59,000 entries have a /24 prefix. In 385 absolute terms the /24 prefix set is the fastest growing set in the 386 BGP routing table. The routing entries of these smaller address 387 blocks also show a much higher level of change on an hourly basis. 388 While a large number of BGP routing points perform route flap 389 damping, nevertheless there is still a very high level of 390 announcements and withdrawals of these entries in this particular 391 area of the routing table when viewed using a perspective of route 392 updates per prefix length. Given that the numbers of these small 393 prefixes are growing rapidly, there is cause for some concern that 394 the total level of BGP flux, in terms of the number of announcements 395 and withdrawals per second may be increasing, despite the pressures 396 from flap damping. This concern is coupled with the observation 397 that, in terms of BGP stability under scaling pressure, it is not 398 the absolute size of the BGP table that is of prime importance, but 399 the rate of dynamic path recomputations that occur in the wake of 400 announcements and withdrawals. Withdrawals are of particular concern 401 due to the number of transient intermediate states that the BGP 402 distance vector algorithm explores in processing a withdrawal. 403 Current experimental observations indicate a typical convergence 404 time of some 2 minutes to propagate a route withdrawal across the 405 BGP domain. [10] 407 An increase in the density of the BGP mesh, coupled with an increase 408 in the rate of such dynamic changes, does have serious implications 409 in maintaining the overall stability of the BGP system as it 410 continues to grow. The registry allocation policies also have had 411 some impact on the routing table prefix distribution. The original 412 registry practice was to use a minimum allocation unit of a /19, and 413 the 10,000 prefix entries in the /17 to /19 range are a consequence 414 of this policy decision. More recently, the allocation policy now 415 allows for a minimum allocation unit of a /20 prefix, and the /20 416 prefix is used by some 4,300 entries as of January 2001, and in 417 relative terms is one of the fastest growing prefix sets. The number 418 of entries corresponding to very small address blocks (smaller than 419 a /24), while small in number as a proportion of the total BGP 420 routing table, is the fastest growing in relative terms. The number 421 of /25 through /32 prefixes in the routing table is growing faster, 422 in terms of percentage change, than any other area of the routing 423 table. If prefix length filtering were in widespread use, the 424 practice of announcing a very small address block with a distinct 425 routing policy would have no particular beneficial outcome, as the 426 address block would not be passed throughout the global BGP routing 427 domain and the propagation of the associated policy would be limited 428 in scope. The growth of the number of these small address blocks, 429 and the diversity of AS paths associated with these routing entries, 430 points to a relatively limited use of prefix length filtering in 431 today's Internet. In the absence of any corrective pressure in the 432 form of widespread adoption of prefix length filtering, the very 433 rapid growth of global announcements of very small address blocks is 434 likely to continue. In percentage terms, the set of prefixes 435 spanning /25 to /32 show the largest growth rates. 437 4.5 Aggregation and Holes 439 With the CIDR routing structure it is possible to advertise a more 440 specific prefix of an existing aggregate. The purpose of this more 441 specific announcement is to punch a 'hole' in the policy of the 442 larger aggregate announcement, creating a different policy for the 443 specifically referenced address prefix. 445 Another use of this mechanism is to perform a rudimentary form of 446 load balancing and mutual backup for multi-homed networks. In this 447 model a network may advertise the same aggregate advertisement along 448 each connection, but then advertise a set of specific advertisements 449 for each connection, altering the specific advertisements such that 450 the load on each connection is approximately balanced. The two forms 451 of holes can be readily discerned in the routing table - while the 452 approach of policy differentiation uses an AS path that is different 453 from the aggregate advertisement, the load balancing and mutual 454 backup configuration uses the same As path for both the aggregate 455 and the specific advertisements. While it is difficult to understand 456 whether the use of such more specific advertisements was intended to 457 be an exception to a more general rule or not within the original 458 intent of CIDR deployment, there appears to be very widespread use 459 of this mechanism within the routing table. Some 59,000 460 advertisements, or 55% of the total number of routing table entries, 461 are being used to punch policy holes in existing aggregate 462 announcements. Of these the overall majority of some 42,000 routes 463 use distinct AS paths, so that it does appear that this is evidence 464 of finer levels of granularity of connection policy in a densely 465 interconnected space. While long term data is not available for the 466 relative level of such advertisements as a proportion of the full 467 routing table, the growth level does strongly indicate that policy 468 differentiation at a fine level within existing provider aggregates 469 is a significant driver of overall table growth. 471 5. Current State of inter-AS routing in the Internet 473 The resumption of compound growth trends within the BGP table, and 474 the associated aspects of finer granularity of routing entries 475 within the table form adequate grounds for consideration of 476 potential refinements to the Internet's exterior routing protocols 477 and potential refinements to current operating practices of inter-AS 478 connectivity. With the exception of the 16 bit AS number space, 479 there is no particular finite limit to any aspect of the BGP table. 480 The motivation for such activity is that a long term pattern of 481 continued growth at current rates may once again pose a potential 482 condition where the capacity of the available processors may be 483 exceeded by some aspect of the Internet routing table. 485 5.1 A denser interconnectivity mesh 487 The decreasing unit cost of communications bearers in many part of 488 the Internet is creating a rapidly expanding market in exchange 489 points and other forms of inter-provider peering. A model of 490 extensive interconnection at the edges of the Internet is rapidly 491 supplanting the deployment model of a single-homed network with a 492 single upstream provider. The underlying deployment model of CIDR 493 was that of a single-homed network, allowing for a strict hierarchy 494 of supply providers. The business imperatives driving this denser 495 mesh of interconnection in the Internet are substantial, and the 496 casualty in this case is the CIDR-induced dampened growth of the BGP 497 routing table. 499 5.2 Multi-Homed small networks and service resiliency 501 It would appear that one of the major drivers of the recent growth 502 of the BGP table is that of small networks advertised as a /24 503 prefix entry in the routing table are multi-homing with a number of 504 peers and a number of upstream providers. In the appropriate 505 environment where there are a number of networks in relatively close 506 proximity, using peer relationships can reduce total connectivity 507 costs, as compared to using a single upstream service provider. 508 Equally significantly, multi-homing with a number of upstream 509 providers is seen as a means of improving the overall availability 510 of the service. In essence, multi-homing is seen as an acceptable 511 substitute for upstream service resiliency. This has a potential 512 side effect that when multi-homing is seen as a preferable 513 substitute for upstream provider resiliency, the upstream provider 514 cannot command a price premium for proving resiliency as an 515 attribute of the provided service, and therefore has little economic 516 incentive to spend the additional money required to engineer 517 resiliency into the network. The actions of the network's multi- 518 homed clients then become self-fulfilling. One way to characterize 519 this behavior is that service resiliency in the Internet is becoming 520 the responsibility of the customer, not the service provider. 522 In such an environment resiliency still exists, but rather than 523 being a function of the bearer or switching subsystem, resiliency is 524 provided through the function of the BGP routing system. The 525 question is not whether this is feasible or desirable in the 526 individual case, but whether the BGP routing system can scale 527 adequately to continue to undertake this role. 529 5.3 Traffic Engineering via Routing 531 Further driving this growth in the routing table is the use of 532 selective advertisement of smaller prefixes along different paths in 533 an effort to undertake traffic engineering within a multi-homed 534 environment. While there is considerable effort being undertaken to 535 develop traffic engineering tools within a single network using MPLS 536 as the base flow management tool, inter-provider tools to achieve 537 similar outcomes are considerably more complex when using such 538 switching techniques. 540 At this stage the only tool being used for inter-provider traffic 541 engineering is that of the BGP routing table, further exacerbating 542 the growth and stability pressures being placed on the BGP routing 543 domain. 545 5.4 Lack of Common Operational Practices 547 There is considerable evidence of a lack of uniformity of 548 operational practices within the inter-domain routing space. This 549 includes the use and setting of prefix filters, the use and setting 550 of route damping parameters and level of verification undertaken on 551 BGP advertisements by both the advertiser and the recipient. There 552 is some extent of 'noise' in the routing table where advertisements 553 appear to be propagated well beyond their intended domain of 554 applicability, and also where withdrawals and advertisements are not 555 being adequately damped close to the origin of the route flap. This 556 diversity of operating practices also extends to policies of 557 accepting advertisements that are more specific advertisements of 558 existing provider blocks. 560 5.5 CIDR and Hierarchical Routing 562 The current growth factors at play in the BGP table are not easily 563 susceptible to another round of CIDR deployment pressure within the 564 operator community. The denser interconnectivity mesh, the 565 increasing use of multi-homing with smaller address prefixes, the 566 extension of the use of BGP to perform roles related to inter-domain 567 traffic engineering and the lack of common operating practices all 568 point to a continuation of the trend of growth in the total size of 569 the BGP routing table, with this growth most apparent with 570 advertisements of smaller address blocks, and an increasing trend 571 for these small advertisements to be punching a connectivity policy 572 'hole' in an existing provider aggregate advertisement. 574 It may be appropriate to consider how to operate an Internet with a 575 BGP routing table that has millions of small entries, rather than 576 the expectation of a hierarchical routing space with at most tens of 577 thousands of larger entries in the global routing table. 579 6. Future Requirements for the Exterior Routing System 581 It is beyond the scope of this document to define a scaleable inter- 582 domain routing environment and associated routing protocols and 583 operating practices. A more modest goal is to look at the attributes 584 of routing systems as understood and identify those aspects of such 585 systems that may be applicable to the inter-domain environment as a 586 potential set of requirements for inter-domain routing tools. 588 6.1 Scalability 590 The overall intent is scalability of the routing environment. 591 Scalability can be expressed in many dimensions, including number of 592 discrete network layer reachability entries, number of discrete 593 route policy entries, level of dynamic change over a unit of time of 594 these entries, time to converge to a coherent view of the 595 connectivity of the network following changes, and so on. 597 The basic objective behind this expressed requirement for 598 scalability is that the most likely near to medium trend in the 599 structure of the Internet is a continuation in the pattern of dense 600 interconnectivity between a large number of discrete network 601 entities, and little impetus behind hierarchical aggregating 602 structures. It is not an objective to place any particular metrics 603 on scalability within this examination of requirements, aside from 604 indicating that a prudent view would encompass a scale of 605 connectivity in the inter-domain space that is at least two orders 606 of magnitude larger than comparable metrics of the current 607 environment. 609 6.2 Stability and Predictability 611 Any routing system should behave in a stable and predictable 612 fashion. What is inferred from the predictability requirement is the 613 behavior that under identical environmental conditions the routing 614 system should converge to the same state. Stability implies that the 615 routing state should be maintained for as long as the environmental 616 conditions remain constant. Stability also implies a qualitative 617 property that minor variations in the network's state should not 618 cause large scale instability across the entire network while a new 619 stable routing state is reached. Instead, routing changes should be 620 propagated only as far as necessary to reach a new stable state, so 621 that the global requirement for stability implies some degree of 622 locality in the behavior of the system. 624 6.3 Convergence 626 Any routing system should have adequate convergence properties. By 627 adequate it is implied that within a finite time following a change 628 in the external environment, the routing system will have reached a 629 shared common description of the network's topology that accurately 630 describes the current state of the network and is stable. In this 631 case finite time implies a time limit that is bounded by some upper 632 limit, and this upper limit reflects the requirements of the routing 633 system. In the case of the Internet this convergence time is 634 currently of the order of hundreds of seconds as an upper bound on 635 convergence. This long convergence time is perceived as having a 636 negative impact on various applications, particularly those that are 637 time critical. A more useful upper bound for convergence is of the 638 order of seconds or lower if it is desired to support a broad range 639 of application classes. 641 It is not a requirement to be able to undertake full convergence of 642 the inter-domain routing system in the sub-second timescale. 644 6.4 Routing Overhead 646 The greater the amount of information passed within the routing 647 system, and the greater the frequency of such information exchanges, 648 the greater the level of expectation that the routing system can 649 maintain an accurate view of the connectivity of the network. 650 Equally, the greater the amount of information passed within the 651 routing system, and the higher the frequency of information 652 exchange, the higher the level of overhead consumed by operation of 653 the routing system. There is an element of design compromise in a 654 routing system to pass enough information across the system to allow 655 each routing element to have adequate local information to reach a 656 coherent local view of the network, yet ensure that the total 657 routing overhead is low. 659 7. Architectural approaches to a scaleable Exterior Routing Protocol 661 This document does not attempt to define an inter-domain routing 662 protocol that possess all the attributes as listed above, but a 663 number of architectural considerations can be identified that would 664 form an integral part of the protocol design process. 666 7.1 Policy opaqueness vs. policy transparency 668 The two major approaches to routing protocols are distance vector 669 and link state. 671 In the distance vector protocol a routing node gathers information 672 from its neighbors, applies local policy to this information and 673 then distributes this updated information to its neighbors. In this 674 model the nature of the local policy applied to the routing 675 information is not necessarily visible to the node's neighbors, and 676 the process of converting received route advertisements into 677 advertised route advertisements uses a local policy process whose 678 policy rules are not visible externally. This scenario can be 679 described as 'policy opaque'. The side effect of such an environment 680 is that a third party cannot remotely compute which routes a network 681 may accept and which may be re-advertised to each neighbor. 683 In link state protocols a routing node effectively broadcasts its 684 local adjancies, and the policies it has with respect to these 685 adjancies, to all nodes within the link state domain. Every node can 686 perform an identical computation upon this set of adjancies and 687 associated policies in order to compute the local forwarding table. 688 The essential attribute of this environment is that the routing node 689 has to announce its routing policies, in order to allow a remote 690 node to compute which routes will be accepted from which neighbor, 691 and which routes will be advertised to each neighbor and what, if 692 any, attributes are placed on the advertisement. Within an interior 693 routing domain the local policies are in effect metrics of each link 694 and these polices can be announced within the routing domain without 695 any consequent impact. 697 In the exterior routing domain it is not the case that 698 interconnection policies between networks are always fully 699 transparent. Various permutations of supplier / customer 700 relationships and peering relationships have associated policy 701 qualifications that are not publicly announced for business 702 competitive reasons. The current diversity of interconnection 703 arrangements appears to be predicated on policy opaqueness, and to 704 mandate a change to a model of open interconnection policies may be 705 contrary to operational business imperatives. 707 An inter-domain routing tool should be able to support models of 708 interconnection where the policy associated with the interconnection 709 is not visible to any third party. This consideration would appear 710 to favor the continued use of a distance vector approach to inter- 711 domain routing. This, in turn, has implications on the convergence 712 properties and stability of the inter-domain routing environment. 714 7.2 The number of routing objects 716 The current issues with the trend behaviors of the BGP space can be 717 coarsely summarized as the growth in the number of distinct routing 718 objects, the increased level of dynamic behaviors of these objects 719 (in the form of announcements and withdrawals). 721 This entails evaluating possible measures that can address the 722 growth rate in the number of objects in the inter-domain routing 723 table, and separately examining measures that can reduce the level 724 of dynamic change in the routing table. The current routing 725 architecture defines a basic unit of a route object as an 726 originating AS number and an address prefix. 728 In looking at the growth rate in the number of route objects, the 729 salient observation is that the number of route objects is the 730 byproduct of the density of the interconnection mesh and the number 731 of discrete points where policy is imposed of route objects. One 732 approach to reduce the growth in the number of objects is to allow 733 each object to describe larger segments of infrastructure. Such an 734 approach could use a single route object to describe a set of 735 address prefixes, or a collection of ASs, or a combination of the 736 two. The most direct form of extension would be to preserve the 737 assumption that each routing object represents an indivisible policy 738 entity. However, given that one of the drivers of the increasing 739 number of route objects is a proliferation of discrete route 740 objects, it is not immediately apparent that this form of 741 aggregation will prove capable in addressing the growth in the 742 number of route objects. 744 If single route objects are to be used that encompass a set of 745 address prefixes and a collection of ASs, then it appears necessary 746 to define additional attributes within the route object to further 747 qualify the policies associated with the object in terms of specific 748 prefixes, specific ASs and specific policy semantics that may be 749 considered as policy exceptions to the overall aggregate 751 Another approach to reduce the number of route objects is to reduce 752 the scope of advertisement of each routing object, allowing the 753 object to be removed and proxy aggregated into some larger object 754 once the logical scope of the object has been reached. This approach 755 would entail the addition of route attributes that could be used to 756 define the circumstances where a specific route object would be 757 subsumed by an aggregate route object without impacting the policy 758 objectives associated with the original set of advertisements. 760 7.3 Inter-domain Traffic Engineering 761 Attempting to place greater levels of detail into route objects is 762 intended to address the dual role of the current BGP system as both 763 an inter-domain connectivity maintenance protocol and as an implicit 764 traffic engineering tool. 766 In the current environment, advertisement of more specific prefixes 767 with unique policy but with the same origin AS is often intended to 768 create a traffic engineering response, where incoming traffic to an 769 AS may be balanced across multiple paths. The outcome is that the 770 control of the relative profile of load is placed with the 771 originating AS. The way this is achieved is by using limited 772 knowledge of the remote AS's route selection policy to explicitly 773 limit the number of egress choices available to a remote AS. The 774 most common route selection policy is the preference for more 775 specific prefixes over larger address blocks. By advertising 776 specific prefixes along specific neighbor AS connections with 777 specific route attributes, traffic destined to these addresses is 778 passed through the selected transit paths. This limitation of choice 779 allows the originating AS to override the potential policy choices 780 of all other ASs, imposing its traffic import policies at a higher 781 level than the remote AS's egress policies. 783 An alternative approach is the use of a class of traffic engineering 784 attributes that are attached to an aggregate route object. The 785 intent of such attributes is to direct each remote AS to respond to 786 the route object in a manner that equates to the current response to 787 more specific advertisements, but without the need to advertise 788 specific prefix route objects. However, even this approach uses 789 route objects to communicate traffic engineering policy, and the 790 same risk remains that the route table is used to carry fine- 791 detailed traffic path policies. 793 An alternative direction is to separate the functions of 794 connectivity maintenance and traffic engineering, using the routing 795 protocol to identify a number of viable paths from a source AS to a 796 destination AS, and use a distinct collection of traffic engineering 797 tools to allow a traffic source AS to make egress path selections 798 that match the desired traffic service profile for the traffic. 800 There is one critical difference between traffic engineering 801 approaches as used in intra-domain environments and the current 802 inter-domain operating practices. Whereas the intra-domain 803 environment uses the ingress network element to make the appropriate 804 path choice to the egress point, the inter domain traffic 805 engineering has the opposite intent, where a downstream AS (or 806 egress point) is attempting to influence the path choice of an 807 upstream AS (or ingress point). If explicit traffic engineering were 808 undertaken within the inter-domain space, it is highly likely that 809 the current structure would be altered. Instead of the downstream 810 element attempting to constrain the path choices of an upstream 811 element, a probable approach is the downstream element placing a 812 number of advisory constraints on the upstream elements, and the 813 upstream elements using a combination of these advisory constraints, 814 dynamic information relating to path service characteristics and 815 local policies to make an egress choice. 817 From the perspective of the inter-domain routing environment, such 818 measures offer the potential to remove the advertisement of specific 819 routes for traffic engineering purposes. However, there is a need to 820 adding traffic engineering information into advertised route blocks, 821 requiring the definition of the syntax and semantics of traffic 822 engineering attributes that can be attached to route objects. 824 7.4 Hierarchical Routing Models 826 The CIDR routing model assumed a hierarchy of providers, where at 827 each level in the hierarchy the routing policies and address space 828 of networks at the lower level of hierarchy were subsumed by the 829 next level up (or `upstream') provider. The connectivity policy 830 assumed by this model is also a hierarchical model, where horizontal 831 connections within a single level of the hierarchy are not visible 832 beyond the networks of the two parties. 834 A number of external factors are increasing the density of 835 interconnection including decreasing unit costs of communications 836 services and the increasing use of exchange points to augment point- 837 to-point connectivity models with point-to-multipoint facilities. 839 The outcome of these external factors is a significant reduction in 840 the hierarchical nature of the inter-domain space. The outcomes of 841 this characteristic of the Internet in terms of the routing space is 842 the increasing number of distinct route policies that are associated 843 with each multi-homed network within the Internet. 845 One way to limit the proliferation of such policies across the 846 entire inter-domain space is to associate attributes to such 847 advertisements that specify the conditions whereby a remote transit 848 AS may proxy-aggregate this route object with other route objects. 850 7.5 Extend or Replace BGP 852 A final consideration is to consider whether these requirements can 853 best be met by an approach of a set of upward-compatible extensions 854 to BGP, or by a replacement to BGP. No recommendation is made here, 855 and this is a topic requiring further investigation. 857 The general approach in extending BGP appears to lie in increasing 858 the number of supported transitive route attributes, allowing the 859 route originator greater control in specifying the level of 860 propagation of the route and the intended outcome in terms of policy 861 and traffic engineering. It is also be necessary to allow BGP 862 sessions to negotiate an enhanced capability to improve the 863 convergence behavior of the protocol. Whether such changes can 864 produce a scaleable and useful outcome in terms of inter-domain 865 routing remains, at this stage, an open question. 867 An alternative approach is that of a replacement protocol, and such 868 an approach may well be based on the adoption of a link-state 869 behavior. The issues of policy opaqueness and link-state protocols 870 have been described above. The other major issue with such an 871 approach is the need to limit the extent of link state flooding, 872 where the inter-domain space would need some further levels of 873 imposed structure similar to intra-domain areas. Such structure may 874 well imply the need for an additional set of operator inter- 875 relationships such as mutual transit, and this may prove challenging 876 to adapt to existing practices. 878 The potential sets of actions include more than extend or replace 879 the BGP protocol. A third approach is to continue to use BGP as the 880 basic means of propagating route objects and their associated AS 881 paths and other attributes, and use one or more overlay protocols to 882 support inter-domain traffic engineering and other forms of inter- 883 domain policy negotiation. This approach would appear to offer a 884 means of transition for the large installed base currently using 885 BGP4 as their inter-domain routing protocol, placing additional 886 functionality in the overlay protocols while leaving the basic 887 functionality of BGP4 intact. The resultant inter-dependencies 888 between BGP and the overlay protocols would require very careful 889 attention, as this would be the most critical aspect of such an 890 approach. 892 8. Directions for Further Activity 894 While there may exist short term actions based on providing various 895 incentives for network operators to remove redundant or 896 inefficiently grouped entries from the BGP routing table, such 897 actions are short term palliative measures, and will not provide 898 long term answers to the need to a scaleable inter-domain routing 899 protocol. 901 One potential short term protocol refinement is to allow a set of 902 grouped advertisements to be aggregated into a single route 903 advertisement. This form of proxy aggregation would take a set of 904 bit-wise aligned routing entries with matching route attributes, and 905 under certain well identified circumstances, aggregated these 906 routing entries into a single re-advertised aggregate routing entry. 907 This technique removes information from the routing system, and some 908 care must be taken to define a set of proxy aggregation conditions 909 that do not materially alter the flow of traffic, or the ability of 910 originating AS's to announce routing policy. 912 A further refinement to this approach is to consider the definition 913 of the syntax and semantics of a number of additional route 914 attributes. Such attributes could define the extent to which 915 specific route advertisements should be propagated in the inter- 916 domain space, allowing the advertisement to be subsumed by a larger 917 aggregate advertisement at the boundary of this domain. This could 918 be used to form part of the preconditions of automated proxy 919 aggregation of specific routes, and also limit the extent to which 920 announcement and withdrawals are propagated across the routing 921 domain. 923 It is unclear that such measures would result in substantial longer 924 term changes to the scaling and convergence properties of BGP4. 925 Taking the requirement set enumerated in section 6 of this document, 926 one approach to the longer term requirements may be to preserve a 927 number of attributes of the current BGP protocol, while refine other 928 aspects of the protocol to improve its scaling and convergence 929 properties. A minimal set of alterations could retain the Autonomous 930 System concept to allow for boundaries of information summarization, 931 as well as retaining the approach of associating each prefix 932 advertisement with an originating AS. The concept of policy 933 opaqueness would also be retained in such an approach, implying that 934 each AS accepts a set of route advertisements, applies local policy 935 constraints, and re-advertises those advertisements permitted by the 936 local policy constraints. It could be feasible to consider 937 alterations to the distance vector path selection algorithm, 938 particular as it relates to intermediate states during processing of 939 a route withdrawal. It is also feasible to consider the use of 940 compound route attributes, allowing a route object to include an 941 aggregate route, and a number of specifics of the aggregate route, 942 and attach attributes that may apply to the aggregate or a specific 943 address prefix. Such route attributes could be used to support 944 multi-homing and inter-domain traffic engineering mechanisms. The 945 overall intent of this approach is to address the major requirements 946 in the inter-domain routing space without using an increasing set of 947 globally propagated specific route objects. 949 A potential applied research topic is to consider the feasibility of 950 decoupling the requirements of inter-domain connectivity management 951 with the applications of policy constraints and the issues of 952 sender- and/or receiver-managed traffic engineering requirements. 953 Such an approach may use a link state protocol as a means of 954 maintaining a consistent view of the topology of inter-domain 955 network, and then use some form of overlay protocol to negotiate 956 policy requirements of each AS, and use a further overlay to support 957 inter-domain traffic engineering requirements. The underlying 958 assumption of such an approach is that by dividing up the functional 959 role of inter-domain routing into distinct components each component 960 will have superior scaling and convergence properties which in turn 961 to result in superior properties for the entire routing system. 962 Obviously, this assumption requires some testing. 964 Research topics with potential longer term application include the 965 approach of drawing a distinction between a network's identity, a 966 network's location relative to other networks, and a feasible path 967 between a source and destination network that satisfies various 968 policy and traffic engineering constraints. Again the intent of such 969 an approach would be to divide the current routing function into a 970 number of distinct scaleable components. 972 9. Security Considerations 974 Any adopted inter-domain routing protocol needs to be secure against 975 disruption. Disruption comes from two primary sources: 976 - Accidental misconfiguration 977 - Malicious attacks 979 Given past experience with routing protocols, both can be 980 significant sources of harm. 982 Given that it is not reasonable to guarantee the security of all the 983 routers involved in the global Internet inter-domain routing system, 984 there is also every reason to believe that malicious attacks may 985 come from peer routers, in addition to coming from external sources. 987 A protocol design SHOULD therefore consider how to minimize the 988 damage to the overall routing computation that can be caused by a 989 single or small set of misbehaving routers. 991 The routing system itself needs to be resilient against accidental 992 or malicious advertisements of a route object by a route server not 993 entitled to generate such an advertisement. This implies several 994 things, including the need for cryptographic validation of 995 announcements, cryptographic protection of various critical routing 996 messages and an accurate and trusted database of routing assignments 997 via which authorization can be checked. 999 References 1001 [1] Bradner, S., "The Internet Standards Process -- Revision 3", 1002 BCP 9, RFC 2026, October 1996. 1004 [2] Clark, D., Chapin, L., Cerf, V., Braden, R., Hobby, R., "Towards 1005 the Future Internet Architecture", RFC 1287, December 1991. 1007 [3] Deering, S., Hinden, R., "Internet Protocol, Version 6 (IPv6) 1008 Specification, RFC 2460, December 1998. 1010 [4] Srisuresh, P., Egevang, K., "Traditional IP Network Address 1011 Translator (Traditional NAT)", RFC 3022, January 2001. 1013 [5] Fuller, V., Li, T., Yu, J., Varadhan, K., "Classless Inter- 1014 Domain Routing (CIDR): an Address Assignment and Aggregation 1015 Strategy", RFC 1519, September 1993. 1017 [6] Huston, G., "The BGP Routing Table", The Internet Protocol 1018 Journal, vol. 4, No. 1, March 2001. 1020 [7] Rekhter, Y., Li, T., "A Border Gateway Protocol 4 (BGP-4)", RFC 1021 1771, March 1995. 1023 [8] Vohara, Q., Chen, E., "BGP support for four-octet AS number 1024 space", work in progress, draft-ietf-idr-as4bytes-02.txt, April 1025 2001. 1027 [9] Hain, T., "Architectural Implications of NAT", RFC 2993, 1028 November 2000. 1030 [10] Labovitz, C., "Delayed Internet Routing Convergence", 1031 Proceedings ACM SIGCOMM 2000, August 2000. 1033 .ti 4 [11] Lothberg, P., personal communication, December 2000. 1035 Acknowledgements 1037 This document is the outcome of a collaborative effort of the IAB, 1038 and the author acknowledges the contributions of the members of the 1039 IAB in the preparation of the document. The contribution of John 1040 Leslie to this document is also acknowledged. 1042 Author 1044 Geoff Huston 1045 Telstra 1046 EMail: gih@telstra.net 1048 Full Copyright Statement 1050 Copyright (C) The Internet Society (2000). All Rights Reserved. 1052 This document and translations of it may be copied and furnished to 1053 others, and derivative works that comment on or otherwise explain it 1054 or assist in its implementation may be prepared, copied, published 1055 and distributed, in whole or in part, without restriction of any 1056 kind, provided that the above copyright notice and this paragraph 1057 are included on all such copies and derivative works. However, this 1058 document itself may not be modified in any way, such as by removing 1059 the copyright notice or references to the Internet Society or other 1060 Internet organizations, except as needed for the purpose of 1061 developing Internet standards in which case the procedures for 1062 copyrights defined in the Internet Standards process must be 1063 followed, or as required to translate it into languages other than 1064 English. 1066 The limited permissions granted above are perpetual and will not be 1067 revoked by the Internet Society or its successors or assigns. 1069 This document and the information contained herein is provided on an 1070 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 1071 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 1072 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 1073 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 1074 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1076 Acknowledgement 1078 Funding for the RFC Editor function is currently provided by the 1079 Internet Society.