idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-15.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 30, 2020) is 1338 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 589 -- Looks like a reference, but probably isn't: '1' on line 590 -- Looks like a reference, but probably isn't: '3' on line 590 -- Looks like a reference, but probably isn't: '6' on line 559 == Unused Reference: 'RFC8605' is defined on line 902, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 4 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: January 31, 2021 S. Hollenbeck 6 Verisign Labs 7 July 30, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-15 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on January 31, 2021. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.1.1. RDAP Conformance . . . . . . . . . . . . . . . . . . 6 64 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 65 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 66 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 8 67 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 14 68 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 16 69 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 16 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 17 71 4. Implementation Considerations . . . . . . . . . . . . . . . . 18 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 73 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 18 74 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 19 75 6.2. APNIC . . . . . . . . . . . . . . . . . . . . . . . . . . 19 76 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 77 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 78 8.1. Normative References . . . . . . . . . . . . . . . . . . 20 79 8.2. Informative References . . . . . . . . . . . . . . . . . 21 80 Appendix A. JSONPath operators . . . . . . . . . . . . . . . . . 22 81 Appendix B. Approaches to Result Pagination . . . . . . . . . . 24 82 B.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 25 83 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 26 84 Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 87 1. Introduction 89 The availability of functionality for result sorting and paging 90 provides benefits to both clients and servers in the implementation 91 of RESTful services [REST]. These benefits include: 93 o reducing the server response bandwidth requirements; 94 o improving server response time; 95 o improving query precision and, consequently, obtaining more 96 reliable results; 98 o decreasing server query processing load; 99 o reducing client response processing time. 101 Approaches to implementing features for result sorting and paging can 102 be grouped into two main categories: 104 1. sorting and paging are implemented through the introduction of 105 additional parameters in the query string (e.g. ODATA protocol 106 [OData-Part1]); 108 2. information related to the number of results and the specific 109 portion of the result set to be returned, in addition to a set of 110 ready-made links for the result set scrolling, are inserted in 111 the HTTP header of the request/response. 113 However, there are some drawbacks associated with the use of the HTTP 114 header. First, the header properties cannot be set directly from a 115 web browser. Moreover, in an HTTP session, the information on the 116 status (i.e. the session identifier) is usually inserted in the 117 header or a cookie, while the information on the resource 118 identification or the search type is included in the query string. 119 The second approach is therefore not compliant with the HTTP standard 120 [RFC7230]. As a result, this document describes a specification 121 based on the use of query parameters. 123 Currently, the RDAP protocol [RFC7482] defines two query types: 125 o lookup: the server returns only one object; 126 o search: the server returns a collection of objects. 128 While the lookup query does not raise issues regarding response size 129 management, the search query can potentially generate a large result 130 set that could be truncated according to server limits. Besides, it 131 is not possible to obtain the total number of objects found that 132 might be returned in a search query response [RFC7483]. Lastly, 133 there is no way to specify sort criteria to return the most relevant 134 objects at the beginning of the result set. Therefore, the client 135 might traverse the whole result set to find the relevant objects or, 136 due to truncation, might not find them at all. 138 The specification described in this document extends RDAP query 139 capabilities to enable result sorting and paging, by adding new query 140 parameters that can be applied to RDAP search path segments. The 141 service is implemented using the Hypertext Transfer Protocol (HTTP) 142 [RFC7230] and the conventions described in [RFC7480]. 144 The implementation of the new parameters is technically feasible, as 145 operators for counting, sorting and paging rows are currently 146 supported by the major relational database management systems. 148 1.1. Conventions Used in This Document 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 152 "OPTIONAL" in this document are to be interpreted as described in BCP 153 14 [RFC2119] [RFC8174] when, and only when, they appear in all 154 capitals, as shown here. 156 2. RDAP Query Parameter Specification 158 The new query parameters are OPTIONAL extensions of path segments 159 defined in [RFC7482]. They are as follows: 161 o "count": a boolean value that allows a client to request the 162 return of the total number of objects found; 164 o "sort": a string value that allows a client to request a specific 165 sort order for the result set; 167 o "cursor": a string value representing a pointer to a specific 168 fixed size portion of the result set. 170 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 171 sections to describe the formal syntax of these new parameters. 173 2.1. Sorting and Paging Metadata 175 According to most advanced principles in REST design, collectively 176 known as HATEOAS (Hypermedia as the Engine of Application State) 177 [HATEOAS], a client entering a REST application through an initial 178 URI should use server-provided links to dynamically discover 179 available actions and access the resources it needs. In this way, 180 the client is not required to have prior knowledge of the service 181 and, consequently, to hard code the URIs of different resources. 182 This allows the server to make URI changes as the API evolves without 183 breaking clients. Definitively, a REST service should be as self- 184 descriptive as possible. 186 Therefore, servers implementing the query parameters described in 187 this specification SHOULD provide additional information in their 188 responses about both the available sorting criteria and possible 189 pagination. Such information is collected in two OPTIONAL response 190 elements named "sorting_metadata" and "paging_metadata". 192 The "sorting_metadata" element contains the following properties: 194 o "currentSort": "String" (OPTIONAL) either the value of sort 195 "parameter" as specified in the query string or the sort applied 196 by default, if any; 198 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of 199 objects, with each element describing an available sort criterion. 200 Members are: 202 * "property": "String" (REQUIRED) the name that can be used by 203 the client to request the sort criterion; 204 * "default": "Boolean" (REQUIRED) whether the sort criterion is 205 applied by default; 206 * "jsonPath": "String" (OPTIONAL) the JSONPath of the RDAP field 207 corresponding to the property; 208 * "links": "Link[]" (OPTIONAL) an array of links as described in 209 [RFC8288] containing the query string that applies the sort 210 criterion. 212 At least one of the "currentSort" and "availableSorts" properties 213 MUST be present. 215 The "paging_metadata" element contains the following fields: 217 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 218 the total number of objects found. It MUST be provided if and 219 only if the query string contains the "count" parameter; 221 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 222 number of objects returned in the current page. It MUST be 223 provided if and only if the total number of objects exceeds the 224 page size. This property is redundant for RDAP clients because 225 the page size can be derived from the length of the search results 226 array but, it can be helpful if the end user interacts with the 227 server through a web browser; 229 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 230 the number of the current page in the result set. It MUST be 231 provided if and only if the total number of objects found exceeds 232 the page size; 234 o "links": "Link[]" (OPTIONAL) an array of links as described in 235 [RFC8288] containing the reference to the next page. In this 236 specification, only forward pagination is described because it is 237 all that is necessary to traverse the result set. 239 2.1.1. RDAP Conformance 241 Servers returning the "paging_metadata" element in their response 242 MUST include the string literal "paging" in the rdapConformance 243 array. Servers returning the "sorting_metadata" element MUST include 244 the string literal "sorting". 246 2.2. "count" Parameter 248 Currently, the RDAP protocol does not allow a client to determine the 249 total number of the results in a query response when the result set 250 is truncated. This is inefficient because the user cannot determine 251 if the result set is complete. 253 The "count" parameter provides additional functionality (Figure 1) 254 that allows a client to request information from the server that 255 specifies the total number of objects matching the search pattern. 257 https://example.com/rdap/domains?name=*nr.com&count=true 259 Figure 1: Example of RDAP query reporting the "count" parameter 261 The ABNF syntax is the following: 263 count = "count=" ( trueValue / falseValue ) 264 trueValue = ("true" / "yes" / "1") 265 falseValue = ("false" / "no" / "0") 267 A trueValue means that the server MUST provide the total number of 268 the objects in the "totalCount" field of the "paging_metadata" 269 element (Figure 2). A falseValue means that the server MUST NOT 270 provide this number. 272 { 273 "rdapConformance": [ 274 "rdap_level_0", 275 "paging" 276 ], 277 ... 278 "paging_metadata": { 279 "totalCount": 43 280 }, 281 "domainSearchResults": [ 282 ... 283 ] 284 } 286 Figure 2: Example of RDAP response with "paging_metadata" element 287 containing the "totalCount" field 289 2.3. "sort" Parameter 291 The RDAP protocol does not provide any capability to specify the 292 result set sort criteria. A server could implement a default sorting 293 scheme according to the object class, but this feature is not 294 mandatory and might not meet user requirements. Sorting can be 295 addressed by the client, but this solution is rather inefficient. 296 Sorting features provided by the RDAP server could help avoid 297 truncation of relevant results. 299 The "sort" parameter allows the client to ask the server to sort the 300 results according to the values of one or more properties and 301 according to the sort direction of each property. The ABNF syntax is 302 the following: 304 sort = "sort=" sortItem *( "," sortItem ) 305 sortItem = property-ref [":" ( "a" / "d" ) ] 306 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 308 "a" means that an ascending sort MUST be applied, "d" means that a 309 descending sort MUST be applied. If the sort direction is absent, an 310 ascending sort MUST be applied (Figure 3). 312 https://example.com/rdap/domains?name=*nr.com&sort=name 314 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 316 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 318 Figure 3: Examples of RDAP query reporting the "sort" parameter 320 Except for sorting IP addresses, servers MUST implement sorting 321 according to the JSON value type of the RDAP field the sorting 322 property refers to. That is, JSON strings MUST be sorted 323 lexicographically and JSON numbers MUST be sorted numerically. If IP 324 addresses are represented as JSON strings, they MUST be sorted based 325 on their numeric conversion. 327 The conversion of an IPv4 address to a number is possible since each 328 dotted format IPv4 address is a representation of a number written in 329 a 256-based manner: 192.168.0.1 means 1*256^0 + 0*256^1 + 168*256^2 + 330 192*256^3 = 3232235521. Similarly, an IPv6 address can be converted 331 into a number by applying the base 65536. Therefore, the numerical 332 representation of the IPv6 address 333 2001:0db8:85a3:0000:0000:8a2e:0370:7334 is 334 42540766452641154071740215577757643572. Builtin functions and 335 libraries for converting IP addresses into numbers are available in 336 most known programming languages and relational database management 337 systems. 339 If the "sort" parameter reports an allowed sorting property, it MUST 340 be provided in the "currentSort" field of the "sorting_metadata" 341 element. 343 2.3.1. Sorting Properties Declaration 345 In the "sort" parameter ABNF syntax, property-ref represents a 346 reference to a property of an RDAP object. Such a reference could be 347 expressed by using a JSONPath. The JSONPath in a JSON document 348 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 349 XML document. For example, the JSONPath to select the value of the 350 ASCII name inside an RDAP domain object is "$.ldhName", where $ 351 identifies the root of the document object model (DOM). Another way 352 to select a value inside a JSON document is the JSON Pointer 353 [RFC6901]. While JSONPath or JSON Pointer are both standard ways to 354 select any value inside JSON data, neither is particularly easy to 355 use (e.g. "$.events[?(@.eventAction='registration')].eventDate" is 356 the JSONPath expression of the registration date in an RDAP domain 357 object). 359 Therefore, this specification defines property-ref in terms of RDAP 360 properties. However, not all the RDAP properties are suitable to be 361 used in sort criteria, such as: 363 o properties providing service information (e.g. links, notices, 364 remarks); 366 o multivalued properties (e.g. status, roles, variants); 367 o properties representing relationships to other objects (e.g. 368 entities). 370 On the contrary, properties expressed as values of other properties 371 (e.g. registration date) could be used in such a context. The list 372 of properties an RDAP server MAY implement are divided into two 373 groups: object common properties and object specific properties. 375 o Object common properties. Object common properties are derived 376 from merging the "eventAction" and the "eventDate" properties. 377 The following values of the "sort" parameter are defined: 379 * registrationDate 380 * reregistrationDate 381 * lastChangedDate 382 * expirationDate 383 * deletionDate 384 * reinstantiationDate 385 * transferDate 386 * lockedDate 387 * unlockedDate 389 o Note that some of the object specific properties are also defined 390 as query paths. The object specific properties include: 392 * Domain: name 393 * Nameserver: name, ipV4, ipV6. 394 * Entity: fn, handle, org, email, voice, country, cc, city. 396 The correspondence between these sorting properties and the RDAP 397 object classes is shown in Table 1: 399 +------------+------------+-----------------+-------+-------+-------+ 400 | Object | Sorting | RDAP property | RFC | RFC | RFC | 401 | class | property | | 7483 | 6350 | 8605 | 402 +------------+------------+-----------------+-------+-------+-------+ 403 | Searchable | Common | eventAction | 4.5 | | | 404 | objects | properties | values suffixed | | | | 405 | | | by "Date" | | | | 406 | | | | | | | 407 | Domain | name | unicodeName/ | 5.3 | | | 408 | | | ldhName | | | | 409 | | | | | | | 410 | Nameserver | name | unicodeName/ | 5.2 | | | 411 | | | ldhName | | | | 412 | | ipV4 | v4 ipAddress | 5.2 | | | 413 | | ipV6 | v6 ipAddress | 5.2 | | | 414 | | | | | | | 415 | Entity | handle | handle | 5.1 | | | 416 | | fn | vcard fn | 5.1 | 6.2.1 | | 417 | | org | vcard org | 5.1 | 6.6.4 | | 418 | | voice | vcard tel with | 5.1 | 6.4.1 | | 419 | | | type="voice" | | | | 420 | | email | vcard email | 5.1 | 6.4.2 | | 421 | | country | country name in | 5.1 | 6.3.1 | | 422 | | | vcard adr | | | | 423 | | cc | country code in | 5.1 | | 3.1 | 424 | | | vcard adr | | | | 425 | | city | locality in | 5.1 | 6.3.1 | | 426 | | | vcard adr | | | | 427 +------------+------------+-----------------+-------+-------+-------+ 429 Table 1: Sorting properties definition 431 Regarding the definitions in Table 1, some further considerations are 432 needed to disambiguate some cases: 434 o Since the response to a search on either domains or nameservers 435 might include both A-labels and U-labels [RFC5890] in general, a 436 consistent sorting policy MUST treat the unicodeName and ldhName 437 as two representations of the same value. By default, the 438 unicodeName value MUST be used while sorting. When the 439 unicodeName is unavailable, the value of the ldhName MUST be used 440 instead; 442 o The jCard "sort-as" parameter MUST be ignored for the sorting 443 capability described in this document; 445 o Even if a nameserver can have multiple IPv4 and IPv6 addresses, 446 the most common configuration includes one address for each IP 447 version. Therefore, the assumption of having a single IPv4 and/or 448 IPv6 value for a nameserver cannot be considered too stringent. 449 When more than one address per IP version is reported, sorting 450 MUST be applied to the first value; 452 o Multiple events with a given action on an object might be 453 returned. If this occurs, sorting MUST be applied to the most 454 recent event; 456 o Except for handle values, all the sorting properties defined for 457 entity objects can be multivalued according to the definition of 458 vCard as given in [RFC6350]. When more than one value is 459 reported, sorting MUST be applied to the preferred value 460 identified by the parameter pref="1". If the pref parameter is 461 missing, sorting MUST be applied to the first value. 463 The "jsonPath" field in the "sorting_metadata" element is used to 464 clarify the RDAP field the sorting property refers to. The mapping 465 between the sorting properties and the JSONPaths of the RDAP fields 466 is shown below: 468 o Searchable objects 470 registrationDate 472 $.domainSearchResults[*].events[?(@.eventAction=="registration" 473 )].eventDate 475 reregistrationDate 477 $.domainSearchResults[*].events[?(@.eventAction=="reregistratio 478 n")].eventDate 480 lastChangedDate 482 $.domainSearchResults[*].events[?(@.eventAction=="last 483 changed")].eventDate 485 expirationDate 487 $.domainSearchResults[*].events[?(@.eventAction=="expiration")] 488 .eventDate 490 deletionDate 492 $.domainSearchResults[*].events[?(@.eventAction=="deletion")].e 493 ventDate 495 reinstantiationDate 497 $.domainSearchResults[*].events[?(@.eventAction=="reinstantiati 498 on")].eventDate 500 transferDate 502 $.domainSearchResults[*].events[?(@.eventAction=="transfer")].e 503 ventDate 505 lockedDate 507 $.domainSearchResults[*].events[?(@.eventAction=="locked")].eve 508 ntDate 510 unlockedDate 512 $.domainSearchResults[*].events[?(@.eventAction=="unlocked")].e 513 ventDate 515 o Domain 517 name 519 $.domainSearchResults[*].unicodeName 521 o Nameserver 523 name 525 $.domainSearchResults[*].unicodeName 527 ipV4 529 $.nameserverSearchResults[*].ipAddresses.v4[0] 531 ipV6 533 $.nameserverSearchResults[*].ipAddresses.v6[0] 535 o Entity 537 handle 539 $.entitySearchResults[*].handle 541 fn 542 $.entitySearchResults[*].vcardArray[1][?(@[0]=="fn")][3] 544 org 546 $.entitySearchResults[*].vcardArray[1][?(@[0]=="org")][3] 548 voice 550 $.entitySearchResults[*].vcardArray[1][?(@[0]=="tel" && 551 @[1].type=="voice")][3] 553 email 555 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email")][3] 557 country 559 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][6] 561 cc 563 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][1].cc 565 city 567 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][3] 569 The JSONPaths are provided according to the Goessner v.0.8.0 570 specification [GOESSNER-JSON-PATH]. Further documentation about 571 JSONPath operators used in this specification is included in 572 Appendix A. 574 Additional notes on the reported JSONPaths: 576 o those related to the event dates are defined only for the "domain" 577 object. To obtain the equivalent JSONPaths for "entity" and 578 "nameserver", the path segment "domainSearchResults" must be 579 replaced with "entitySearchResults" and "nameserverSearchResults" 580 respectively; 582 o those related to vCard elements are specified without taking into 583 account the "pref" parameter. Servers that sort those values 584 identified by the pref parameter SHOULD update a JSONPath by 585 adding an appropriate filter. For example, if the email values 586 identified by pref="1" are considered for sorting, the JSONPath of 587 the "email" sorting property should be: 589 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" && 590 @[1].pref=="1")][3] 592 2.3.2. Representing Sorting Links 594 An RDAP server MAY use the "links" array of the "sorting_metadata" 595 element to provide ready-made references [RFC8288] to the available 596 sort criteria (Figure 4). Each link represents a reference to an 597 alternate view of the results. 599 { 600 "rdapConformance": [ 601 "rdap_level_0", 602 "sorting" 603 ], 604 ... 605 "sorting_metadata": { 606 "currentSort": "name", 607 "availableSorts": [ 608 { 609 "property": "registrationDate", 610 "jsonPath": "$.domainSearchResults[*] 611 .events[?(@.eventAction==\"registration\")].eventDate", 612 "default": false, 613 "links": [ 614 { 615 "value": "https://example.com/rdap/domains?name=*nr.com 616 &sort=name", 617 "rel": "alternate", 618 "href": "https://example.com/rdap/domains?name=*nr.com 619 &sort=registrationDate", 620 "title": "Result Ascending Sort Link", 621 "type": "application/rdap+json" 622 }, 623 { 624 "value": "https://example.com/rdap/domains?name=*nr.com 625 &sort=name", 626 "rel": "alternate", 627 "href": "https://example.com/rdap/domains?name=*nr.com 628 &sort=registrationDate:d", 629 "title": "Result Descending Sort Link", 630 "type": "application/rdap+json" 631 } 632 ] 633 }, 634 ... 635 ] 636 }, 637 "domainSearchResults": [ 638 ... 639 ] 640 } 642 Figure 4: Example of a "sorting_metadata" instance to implement 643 result sorting 645 2.4. "cursor" Parameter 647 The cursor parameter defined in this specification can be used to 648 encode information about any pagination method. For example, in the 649 case of a simple implementation of the cursor parameter to represent 650 offset pagination information, the cursor value 651 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the Base64 encoding of 652 "offset=100,limit=50". Likewise, in a simple implementation to 653 represent keyset pagination information, the cursor value 654 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the Base64 655 encoding of "key=thelastdomainofthepage.com" whereby the key value 656 identifies the last row of the current page. 658 This solution lets RDAP providers implement a pagination method 659 according to their needs, a user's access level, and the submitted 660 query. Besides, servers can change the method over time without 661 announcing anything to clients. The considerations that have led to 662 this solution are reported in more detail in Appendix B. 664 The ABNF syntax of the cursor parameter is the following: 666 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 668 https://example.com/rdap/domains?name=*nr.com 669 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 671 Figure 5: An example of an RDAP query reporting the "cursor" 672 parameter 674 2.4.1. Representing Paging Links 676 An RDAP server SHOULD use the "links" array of the "paging_metadata" 677 element to provide a ready-made reference [RFC8288] to the next page 678 of the result set (Figure 6). Examples of additional "rel" values a 679 server MAY implement are "first", "last", and "prev". 681 { 682 "rdapConformance": [ 683 "rdap_level_0", 684 "paging" 685 ], 686 ... 687 "notices": [ 688 { 689 "title": "Search query limits", 690 "type": "result set truncated due to excessive load", 691 "description": [ 692 "search results for domains are limited to 50" 693 ] 694 } 695 ], 696 "paging_metadata": { 697 "totalCount": 73, 698 "pageSize": 50, 699 "pageNumber": 1, 700 "links": [ 701 { 702 "value": "https://example.com/rdap/domains?name=*nr.com", 703 "rel": "next", 704 "href": "https://example.com/rdap/domains?name=*nr.com 705 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 706 "title": "Result Pagination Link", 707 "type": "application/rdap+json" 708 } 709 ] 710 }, 711 "domainSearchResults": [ 712 ... 713 ] 714 } 716 Figure 6: Example of a "paging_metadata" instance to implement cursor 717 pagination 719 3. Negative Answers 721 The value constraints for the parameters are defined by their ABNF 722 syntax. Therefore, each request that includes an invalid value for a 723 parameter SHOULD produce an HTTP 400 (Bad Request) response code. 724 The same response SHOULD be returned in the following cases: 726 o If in both single and multi sort the client provides an 727 unsupported value for the "sort" parameter, as well as a value 728 related to an object property not included in the response; 730 o If the client submits an invalid value for the "cursor" parameter. 732 Optionally, the response MAY include additional information regarding 733 the negative answer in the HTTP entity body. 735 4. Implementation Considerations 737 Implementation of the new parameters is technically feasible, as 738 operators for counting, sorting and paging are currently supported by 739 the major relational database management systems. Similar operators 740 are completely or partially supported by the most well-known NoSQL 741 databases (e.g. MongoDB, CouchDB, HBase, Cassandra, Hadoop). 743 5. IANA Considerations 745 IANA is requested to register the following values in the RDAP 746 Extensions Registry: 748 Extension identifier: paging 749 Registry operator: Any 750 Published specification: This document. 751 Contact: IETF 752 Intended usage: This extension describes best practice for result 753 set paging. 755 Extension identifier: sorting 756 Registry operator: Any 757 Published specification: This document. 758 Contact: IETF 759 Intended usage: This extension describes best practice for result 760 set sorting. 762 6. Implementation Status 764 NOTE: Please remove this section and the reference to RFC 7942 prior 765 to publication as an RFC. 767 This section records the status of known implementations of the 768 protocol defined by this specification at the time of posting of this 769 Internet-Draft, and is based on a proposal described in [RFC7942]. 770 The description of implementations in this section is intended to 771 assist the IETF in its decision processes in progressing drafts to 772 RFCs. Please note that the listing of any individual implementation 773 here does not imply endorsement by the IETF. Furthermore, no effort 774 has been spent to verify the information presented here that was 775 supplied by IETF contributors. This is not intended as, and must not 776 be construed to be, a catalog of available implementations or their 777 features. Readers are advised to note that other implementations may 778 exist. 780 According to RFC 7942, "this will allow reviewers and working groups 781 to assign due consideration to documents that have the benefit of 782 running code, which may serve as evidence of valuable experimentation 783 and feedback that have made the implemented protocols more mature. 784 It is up to the individual working groups to use this information as 785 they see fit". 787 6.1. IIT-CNR/Registro.it 789 Responsible Organization: Institute of Informatics and Telematics 790 of the National Research Council (IIT-CNR)/Registro.it 791 Location: https://rdap.pubtest.nic.it/ 792 Description: This implementation includes support for RDAP queries 793 using data from .it public test environment. 794 Level of Maturity: This is an "alpha" test implementation. 795 Coverage: This implementation includes all of the features 796 described in this specification. 797 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 799 6.2. APNIC 801 Responsible Organization: Asia-Pacific Network Information Centre 802 Location: https://github.com/APNIC-net/rdap-rmp-demo/tree/sorting- 803 and-paging 804 Description: A proof-of-concept for RDAP mirroring. 805 Level of Maturity: This is a proof-of-concept implementation. 806 Coverage: This implementation includes all of the features 807 described in the specification except for nameserver sorting and 808 unicodeName sorting. 809 Contact Information: Tom Harrison, tomh@apnic.net 811 7. Security Considerations 813 Security services for the operations specified in this document are 814 described in [RFC7481]. 816 A search query typically requires more server resources (such as 817 memory, CPU cycles, and network bandwidth) when compared to a lookup 818 query. This increases the risk of server resource exhaustion and 819 subsequent denial of service due to abuse. This risk can be 820 mitigated by either restricting search functionality or limiting the 821 rate of search requests. Servers can also reduce their load by 822 truncating the results in a response. However, this last security 823 policy can result in a higher inefficiency if the RDAP server does 824 not provide any functionality to return the truncated results. 826 The new parameters presented in this document provide RDAP operators 827 with a way to implement a server that reduces inefficiency risks. 828 The "count" parameter gives the client the ability to evaluate the 829 completeness of a response. The "sort" parameter allows the client 830 to obtain the most relevant information at the beginning of the 831 result set. This can reduce the number of unnecessary search 832 requests. Finally, the "cursor" parameter enables the user to scroll 833 the result set by submitting a sequence of sustainable queries within 834 server-acceptable limits. 836 8. References 838 8.1. Normative References 840 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 841 Requirement Levels", BCP 14, RFC 2119, 842 DOI 10.17487/RFC2119, March 1997, 843 . 845 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 846 Specifications: ABNF", STD 68, RFC 5234, 847 DOI 10.17487/RFC5234, January 2008, 848 . 850 [RFC5890] Klensin, J., "Internationalized Domain Names for 851 Applications (IDNA): Definitions and Document Framework", 852 RFC 5890, DOI 10.17487/RFC5890, August 2010, 853 . 855 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 856 DOI 10.17487/RFC6350, August 2011, 857 . 859 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 860 Protocol (HTTP/1.1): Message Syntax and Routing", 861 RFC 7230, DOI 10.17487/RFC7230, June 2014, 862 . 864 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 865 Registration Data Access Protocol (RDAP)", RFC 7480, 866 DOI 10.17487/RFC7480, March 2015, 867 . 869 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 870 Registration Data Access Protocol (RDAP)", RFC 7481, 871 DOI 10.17487/RFC7481, March 2015, 872 . 874 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 875 Protocol (RDAP) Query Format", RFC 7482, 876 DOI 10.17487/RFC7482, March 2015, 877 . 879 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 880 Registration Data Access Protocol (RDAP)", RFC 7483, 881 DOI 10.17487/RFC7483, March 2015, 882 . 884 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 885 Code: The Implementation Status Section", BCP 205, 886 RFC 7942, DOI 10.17487/RFC7942, July 2016, 887 . 889 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 890 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 891 May 2017, . 893 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 894 Interchange Format", STD 90, RFC 8259, 895 DOI 10.17487/RFC8259, December 2017, 896 . 898 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 899 DOI 10.17487/RFC8288, October 2017, 900 . 902 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 903 ICANN Extensions for the Registration Data Access Protocol 904 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 905 . 907 [W3C.CR-xpath-31-20161213] 908 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 909 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 910 31-20161213, December 2016, 911 . 913 8.2. Informative References 915 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 916 Pagination", July 2014, . 919 [CURSOR-API1] 920 facebook.com, "facebook for developers - Using the Graph 921 API", July 2017, . 924 [CURSOR-API2] 925 twitter.com, "Pagination", 2017, 926 . 929 [GOESSNER-JSON-PATH] 930 Goessner, S., "JSONPath - XPath for JSON", 2007, 931 . 933 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 934 . 937 [OData-Part1] 938 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 939 Part 1: Protocol Plus Errata 03", June 2016, 940 . 945 [REST] Fredrich, T., "RESTful Service Best Practices, 946 Recommendations for Creating Web Services", April 2012, 947 . 950 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 951 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 952 DOI 10.17487/RFC6901, April 2013, 953 . 955 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 956 With Limit and Offset is Slow?", July 2017, 957 . 960 Appendix A. JSONPath operators 962 A JSONPath expression represents a path to find an element (or a set 963 of elements) in a JSON content. 965 The base JSONPath specification requires that implementations support 966 a set of "basic operators". These operators are used to access the 967 elements of a JSON structure like objects and arrays, and their 968 subelements, respectively, object members and array items. No 969 operations are defined for retrieving parent or sibling elements of a 970 given element. The root element is always referred to as $ 971 regardless of it being an object or array. 973 Additionally, the specification permits implementations to support 974 arbitrary script expressions. These can be used to index into an 975 object or array, or to filter elements from an array. While script 976 expression behavior is implementation-defined, most implementations 977 support the basic relational and logical operators, as well as both 978 object member and array item access, sufficiently similar for the 979 purpose of this document. Commonly-supported operators/functions 980 divided into "top-level operators" and "filter operators" are 981 documented in Table 2 and Table 3 respectively. 983 +-------------------+-----------------------------------------+ 984 | Operator | Descritpion | 985 +-------------------+-----------------------------------------+ 986 | $ | Root element | 987 | . | Object member access (dot-notation) | 988 | [''] | Object member access (bracket-notation) | 989 | [] | Array item access | 990 | * | All elements within the specified scope | 991 | [?()] | Filter expression | 992 +-------------------+-----------------------------------------+ 994 Table 2: JSONPath Top-Level Operators 996 +------------+----------------------------------------+ 997 | Operator | Descritpion | 998 +------------+----------------------------------------+ 999 | @ | Current element being processed | 1000 | . | Object member access | 1001 | [] | Array item access | 1002 | == | Left is equal to right | 1003 | != | Left is not equal to right | 1004 | < | Left is less than right | 1005 | <= | Left is less than or equal to right | 1006 | > | Left is greater than right | 1007 | >= | Left is greater than or equal to right | 1008 | && | Logical conjunction | 1009 | || | Logical disjunction | 1010 +------------+----------------------------------------+ 1012 Table 3: JSONPath Filter Operators 1014 Appendix B. Approaches to Result Pagination 1016 An RDAP query could return a response with hundreds, even thousands, 1017 of objects, especially when partial matching is used. For this 1018 reason, the cursor parameter addressing result pagination is defined 1019 to make responses easier to handle. 1021 Presently, the most popular methods to implement pagination in a REST 1022 API include offset pagination and keyset pagination. Neither 1023 pagination method requires the server to handle the result set in a 1024 storage area across multiple requests since a new result set is 1025 generated each time a request is submitted. Therefore, they are 1026 preferred to any other method requiring the management of a REST 1027 session. 1029 Using limit and offset operators represents the traditionally used 1030 method to implement result pagination. Both of them can be used 1031 individually: 1033 o "limit": means that the server MUST return the first N objects of 1034 the result set; 1036 o "offset": means that the server MUST skip the first N objects and 1037 MUST return objects starting from position N+1. 1039 When limit and offset are used together, they provide the ability to 1040 identify a specific portion of the result set. For example, the pair 1041 "offset=100,limit=50" returns the first 50 objects starting from 1042 position 101 of the result set. 1044 Though easy to implement, offset pagination also includes drawbacks: 1046 o When offset has a very high value, scrolling the result set could 1047 take some time; 1049 o It always requires fetching all rows before dropping as many rows 1050 as specified by offset; 1052 o It may return inconsistent pages when data are frequently updated 1053 (i.e. real-time data). 1055 Keyset pagination [SEEK] adds a query condition that enables the 1056 selection of the only data not yet returned. This method has been 1057 taken as the basis for the implementation of a "cursor" parameter 1058 [CURSOR] by some REST API providers [CURSOR-API1] [CURSOR-API2]. The 1059 cursor is an opaque URL-safe string representing a logical pointer to 1060 the first result of the next page (Figure 5). 1062 Nevertheless, even keyset pagination can be troublesome: 1064 o It needs at least one key field; 1066 o It does not allow sorting simply by any field because the sorting 1067 criterion must contain a key; 1069 o It works best with full composite values support by DBMS (i.e. 1070 [x,y]>[a,b]), emulation is possible but inelegant and less 1071 efficient; 1073 o It does not allow direct navigation to arbitrary pages because the 1074 result set must be scrolled in sequential order starting from the 1075 initial page; 1077 o Implementing bi-directional navigation is tedious because all 1078 comparison and sort operations have to be reversed. 1080 B.1. Specific Issues Raised by RDAP 1082 Furthermore, in the RDAP context, some additional considerations can 1083 be made: 1085 o An RDAP object is a conceptual aggregation of information 1086 generally collected from more than one data structure (e.g. table) 1087 and this makes it even harder to implement keyset pagination, a 1088 task that is already quite difficult. For example, the entity 1089 object can include information from different data structures 1090 (registrars, registrants, contacts, resellers), each one with its 1091 key field mapping the RDAP entity handle; 1093 o Depending on the number of page results as well as the number and 1094 the complexity of the properties of each RDAP object in the 1095 response, the time required by offset pagination to skip the 1096 previous pages could be much faster than the processing time 1097 needed to build the current page. In fact, RDAP objects are 1098 usually formed by information belonging to multiple data 1099 structures and containing multivalued properties (i.e. arrays) 1100 and, therefore, data selection might therefore be a time consuming 1101 process. This situation occurs even though the selection is 1102 supported by indexes; 1104 o Depending on the access levels defined by each RDAP operator, the 1105 increase in complexity and the decrease in flexibility of keyset 1106 pagination in comparison to offset pagination could be considered 1107 impractical. 1109 Ultimately, both pagination methods have benefits and drawbacks. 1111 Acknowledgements 1113 The authors would like to acknowledge Brian Mountford, Tom Harrison, 1114 Karl Heinz Wolf and Jasdip Singh for their contribution to the 1115 development of this document. 1117 Change Log 1119 00: Initial working group version ported from draft-loffredo-regext- 1120 rdap-sorting-and-paging-05 1121 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1122 consistent between the pagination methods. Renamed 1123 "Considerations about Paging Implementation" section in ""cursor" 1124 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1125 detailed description of both "sorting_metadata" and 1126 "paging_metadata" objects. 1127 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1128 of the cursor parameter. Rearranged the layout of some sections. 1129 Removed some items from "Informative References" section. Changed 1130 "IANA Considerations" section. 1131 03: Added "cc" to the list of sorting properties in "Sorting 1132 Properties Declaration" section. Added RFC8605 to the list of 1133 "Informative References". 1134 04: Replaced "ldhName" with "name" in the "Sorting Properties 1135 Declaration" section. Clarified the sorting logic for the JSON 1136 value types and the sorting policy for multivalued fields. 1137 05: Clarified the logic of sorting on IP addresses. Clarified the 1138 mapping between the sorting properties and the RDAP fields. 1139 Updated "Acknowledgements" section. 1140 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1141 "paging_metadata" object. 1142 07: Added "Paging Responses to POST Requests" section. 1143 08: Added "Approaches to Result Pagination" section to appendix. 1144 Added the case of requesting a sort on a property not included in 1145 the response to the errors listed in the "Negative Answers" 1146 section. 1147 09: Updated the "Implementation Status" section to include APNIC 1148 implementation. Moved the "RDAP Conformance" section up in the 1149 document. Removed the "Paging Responses to POST Requests" 1150 section. Updated the "Acknowledgements" section. Removed unused 1151 references. In the "Sorting Properties Declaration" section: 1153 * clarified the logic of sorting on events; 1154 * corrected the JSONPath of the "lastChanged" sorting property; 1155 * provided a JSONPath example taking into account the vCard 1156 "pref" parameter. 1157 10: Corrected the JSONPaths of both "fn" and "org" sorting 1158 properties in Table 2. Corrected JSON content in Figure 4. Moved 1160 [W3C.CR-xpath-31-20161213] and [RFC7942] to the "Normative 1161 References". Changed the rdapConformance tags "sorting_level_0" 1162 and "paging_level_0" to "sorting" and "paging" respectively. 1163 11: Added the "JSONPath operators" section to appendix. 1164 12: Changed the content of "JSONPath operators" section. 1165 13: Minor pre-AD review edits. 1166 14: Additionl minor pre-AD review edits. 1167 15: In section ""sort" Parameter" added a paragraph providing 1168 conversions of IP addresses into their numerical representations. 1169 In section "Sorting Properties Declaration" rearranged Table 2 in 1170 a list to make the content more readable. Other minor edits due 1171 to AD review. 1173 Authors' Addresses 1175 Mario Loffredo 1176 IIT-CNR/Registro.it 1177 Via Moruzzi,1 1178 Pisa 56124 1179 IT 1181 Email: mario.loffredo@iit.cnr.it 1182 URI: http://www.iit.cnr.it 1184 Maurizio Martinelli 1185 IIT-CNR/Registro.it 1186 Via Moruzzi,1 1187 Pisa 56124 1188 IT 1190 Email: maurizio.martinelli@iit.cnr.it 1191 URI: http://www.iit.cnr.it 1193 Scott Hollenbeck 1194 Verisign Labs 1195 12061 Bluemont Way 1196 Reston, VA 20190 1197 USA 1199 Email: shollenbeck@verisign.com 1200 URI: https://www.verisignlabs.com/