idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 20, 2020) is 1338 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 592 -- Looks like a reference, but probably isn't: '1' on line 593 -- Looks like a reference, but probably isn't: '3' on line 593 -- Looks like a reference, but probably isn't: '6' on line 562 == Unused Reference: 'RFC8605' is defined on line 906, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 4 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: February 21, 2021 S. Hollenbeck 6 Verisign Labs 7 August 20, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-16 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on February 21, 2021. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.1.1. RDAP Conformance . . . . . . . . . . . . . . . . . . 6 64 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 65 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 66 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 8 67 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 14 68 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 16 69 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 16 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 17 71 4. Implementation Considerations . . . . . . . . . . . . . . . . 18 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 73 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 18 74 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 19 75 6.2. APNIC . . . . . . . . . . . . . . . . . . . . . . . . . . 19 76 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 77 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 78 8.1. Normative References . . . . . . . . . . . . . . . . . . 20 79 8.2. Informative References . . . . . . . . . . . . . . . . . 21 80 Appendix A. JSONPath operators . . . . . . . . . . . . . . . . . 22 81 Appendix B. Approaches to Result Pagination . . . . . . . . . . 24 82 B.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 25 83 Appendix C. Additional Implementation Notes . . . . . . . . . . 26 84 C.1. Sorting . . . . . . . . . . . . . . . . . . . . . . . . . 26 85 C.2. Counting . . . . . . . . . . . . . . . . . . . . . . . . 26 86 C.3. Paging . . . . . . . . . . . . . . . . . . . . . . . . . 26 87 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 27 88 Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 91 1. Introduction 93 The availability of functionality for result sorting and paging 94 provides benefits to both clients and servers in the implementation 95 of RESTful services [REST]. These benefits include: 97 o reducing the server response bandwidth requirements; 98 o improving server response time; 99 o improving query precision and, consequently, obtaining more 100 reliable results; 101 o decreasing server query processing load; 102 o reducing client response processing time. 104 Approaches to implementing features for result sorting and paging can 105 be grouped into two main categories: 107 1. sorting and paging are implemented through the introduction of 108 additional parameters in the query string (e.g. ODATA protocol 109 [OData-Part1]); 111 2. information related to the number of results and the specific 112 portion of the result set to be returned, in addition to a set of 113 ready-made links for the result set scrolling, are inserted in 114 the HTTP header of the request/response. 116 However, there are some drawbacks associated with the use of the HTTP 117 header. First, the header properties cannot be set directly from a 118 web browser. Moreover, in an HTTP session, the information on the 119 status (i.e. the session identifier) is usually inserted in the 120 header or a cookie, while the information on the resource 121 identification or the search type is included in the query string. 122 The second approach is therefore not compliant with the HTTP standard 123 [RFC7230]. As a result, this document describes a specification 124 based on the use of query parameters. 126 Currently, the RDAP protocol [RFC7482] defines two query types: 128 o lookup: the server returns only one object; 129 o search: the server returns a collection of objects. 131 While the lookup query does not raise issues regarding response size 132 management, the search query can potentially generate a large result 133 set that is often truncated according to server limits. Besides, it 134 is not possible to obtain the total number of objects found that 135 might be returned in a search query response [RFC7483]. Lastly, 136 there is no way to specify sort criteria to return the most relevant 137 objects at the beginning of the result set. Therefore, the client 138 might traverse the whole result set to find the relevant objects or, 139 due to truncation, might not find them at all. 141 The specification described in this document extends RDAP query 142 capabilities to enable result sorting and paging, by adding new query 143 parameters that can be applied to RDAP search path segments. The 144 service is implemented using the Hypertext Transfer Protocol (HTTP) 145 [RFC7230] and the conventions described in [RFC7480]. 147 The implementation of the new parameters is technically feasible, as 148 operators for counting, sorting and paging rows are currently 149 supported by the major relational database management systems. 151 1.1. Conventions Used in This Document 153 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 154 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 155 "OPTIONAL" in this document are to be interpreted as described in BCP 156 14 [RFC2119] [RFC8174] when, and only when, they appear in all 157 capitals, as shown here. 159 2. RDAP Query Parameter Specification 161 The new query parameters are OPTIONAL extensions of path segments 162 defined in [RFC7482]. They are as follows: 164 o "count": a boolean value that allows a client to request the 165 return of the total number of objects found; 167 o "sort": a string value that allows a client to request a specific 168 sort order for the result set; 170 o "cursor": a string value representing a pointer to a specific 171 fixed size portion of the result set. 173 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 174 sections to describe the formal syntax of these new parameters. 176 2.1. Sorting and Paging Metadata 178 According to most advanced principles in REST design, collectively 179 known as HATEOAS (Hypermedia as the Engine of Application State) 180 [HATEOAS], a client entering a REST application through an initial 181 URI should use server-provided links to dynamically discover 182 available actions and access the resources it needs. In this way, 183 the client is not required to have prior knowledge of the service 184 and, consequently, to hard code the URIs of different resources. 185 This allows the server to make URI changes as the API evolves without 186 breaking clients. Definitively, a REST service should be as self- 187 descriptive as possible. 189 Therefore, servers implementing the query parameters described in 190 this specification SHOULD provide additional information in their 191 responses about both the available sorting criteria and possible 192 pagination. Such information is collected in two OPTIONAL response 193 elements named "sorting_metadata" and "paging_metadata". 195 The "sorting_metadata" element contains the following properties: 197 o "currentSort": "String" (OPTIONAL) either the value of sort 198 "parameter" as specified in the query string or the sort applied 199 by default, if any; 201 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of 202 objects, with each element describing an available sort criterion. 203 Members are: 205 * "property": "String" (REQUIRED) the name that can be used by 206 the client to request the sort criterion; 207 * "default": "Boolean" (REQUIRED) whether the sort criterion is 208 applied by default; 209 * "jsonPath": "String" (OPTIONAL) the JSONPath of the RDAP field 210 corresponding to the property; 211 * "links": "Link[]" (OPTIONAL) an array of links as described in 212 [RFC8288] containing the query string that applies the sort 213 criterion. 215 At least one of the "currentSort" and "availableSorts" properties 216 MUST be present. 218 The "paging_metadata" element contains the following fields: 220 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 221 the total number of objects found. It MUST be provided if and 222 only if the query string contains the "count" parameter; 224 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 225 number of objects returned in the current page. It MUST be 226 provided if and only if the total number of objects exceeds the 227 page size. This property is redundant for RDAP clients because 228 the page size can be derived from the length of the search results 229 array but, it can be helpful if the end user interacts with the 230 server through a web browser; 232 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 233 the number of the current page in the result set. It MUST be 234 provided if and only if the total number of objects found exceeds 235 the page size; 237 o "links": "Link[]" (OPTIONAL) an array of links as described in 238 [RFC8288] containing the reference to the next page. In this 239 specification, only forward pagination is described because it is 240 all that is necessary to traverse the result set. 242 2.1.1. RDAP Conformance 244 Servers returning the "paging_metadata" element in their response 245 MUST include the string literal "paging" in the rdapConformance 246 array. Servers returning the "sorting_metadata" element MUST include 247 the string literal "sorting". 249 2.2. "count" Parameter 251 Currently, the RDAP protocol does not allow a client to determine the 252 total number of the results in a query response when the result set 253 is truncated. This is inefficient because the user cannot determine 254 if the result set is complete. 256 The "count" parameter provides additional functionality (Figure 1) 257 that allows a client to request information from the server that 258 specifies the total number of objects matching the search pattern. 260 https://example.com/rdap/domains?name=*nr.com&count=true 262 Figure 1: Example of RDAP query reporting the "count" parameter 264 The ABNF syntax is the following: 266 count = "count=" ( trueValue / falseValue ) 267 trueValue = ("true" / "yes" / "1") 268 falseValue = ("false" / "no" / "0") 270 A trueValue means that the server MUST provide the total number of 271 the objects in the "totalCount" field of the "paging_metadata" 272 element (Figure 2). A falseValue means that the server MUST NOT 273 provide this number. 275 { 276 "rdapConformance": [ 277 "rdap_level_0", 278 "paging" 279 ], 280 ... 281 "paging_metadata": { 282 "totalCount": 43 283 }, 284 "domainSearchResults": [ 285 ... 286 ] 287 } 289 Figure 2: Example of RDAP response with "paging_metadata" element 290 containing the "totalCount" field 292 2.3. "sort" Parameter 294 The RDAP protocol does not provide any capability to specify the 295 result set sort criteria. A server could implement a default sorting 296 scheme according to the object class, but this feature is not 297 mandatory and might not meet user requirements. Sorting can be 298 addressed by the client, but this solution is rather inefficient. 299 Sorting features provided by the RDAP server could help avoid 300 truncation of relevant results. 302 The "sort" parameter allows the client to ask the server to sort the 303 results according to the values of one or more properties and 304 according to the sort direction of each property. The ABNF syntax is 305 the following: 307 sort = "sort=" sortItem *( "," sortItem ) 308 sortItem = property-ref [":" ( "a" / "d" ) ] 309 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 311 "a" means that an ascending sort MUST be applied, "d" means that a 312 descending sort MUST be applied. If the sort direction is absent, an 313 ascending sort MUST be applied (Figure 3). 315 https://example.com/rdap/domains?name=*nr.com&sort=name 317 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 319 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 321 Figure 3: Examples of RDAP query reporting the "sort" parameter 323 Except for sorting IP addresses, servers MUST implement sorting 324 according to the JSON value type of the RDAP field the sorting 325 property refers to. That is, JSON strings MUST be sorted 326 lexicographically and JSON numbers MUST be sorted numerically. If IP 327 addresses are represented as JSON strings, they MUST be sorted based 328 on their numeric conversion. 330 The conversion of an IPv4 address to a number is possible since each 331 dotted format IPv4 address is a representation of a number written in 332 a 256-based manner: 192.168.0.1 means 1*256^0 + 0*256^1 + 168*256^2 + 333 192*256^3 = 3232235521. Similarly, an IPv6 address can be converted 334 into a number by applying the base 65536. Therefore, the numerical 335 representation of the IPv6 address 336 2001:0db8:85a3:0000:0000:8a2e:0370:7334 is 337 42540766452641154071740215577757643572. Builtin functions and 338 libraries for converting IP addresses into numbers are available in 339 most known programming languages and relational database management 340 systems. 342 If the "sort" parameter reports an allowed sorting property, it MUST 343 be provided in the "currentSort" field of the "sorting_metadata" 344 element. 346 2.3.1. Sorting Properties Declaration 348 In the "sort" parameter ABNF syntax, property-ref represents a 349 reference to a property of an RDAP object. Such a reference could be 350 expressed by using a JSONPath. The JSONPath in a JSON document 351 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 352 XML document. For example, the JSONPath to select the value of the 353 ASCII name inside an RDAP domain object is "$.ldhName", where $ 354 identifies the root of the document object model (DOM). Another way 355 to select a value inside a JSON document is the JSON Pointer 356 [RFC6901]. While JSONPath or JSON Pointer are both standard ways to 357 select any value inside JSON data, neither is particularly easy to 358 use (e.g. "$.events[?(@.eventAction='registration')].eventDate" is 359 the JSONPath expression of the registration date in an RDAP domain 360 object). 362 Therefore, this specification defines property-ref in terms of RDAP 363 properties. However, not all the RDAP properties are suitable to be 364 used in sort criteria, such as: 366 o properties providing service information (e.g. links, notices, 367 remarks); 369 o multivalued properties (e.g. status, roles, variants); 370 o properties representing relationships to other objects (e.g. 371 entities). 373 On the contrary, properties expressed as values of other properties 374 (e.g. registration date) could be used in such a context. The list 375 of properties an RDAP server MAY implement are divided into two 376 groups: object common properties and object specific properties. 378 o Object common properties. Object common properties are derived 379 from merging the "eventAction" and the "eventDate" properties. 380 The following values of the "sort" parameter are defined: 382 * registrationDate 383 * reregistrationDate 384 * lastChangedDate 385 * expirationDate 386 * deletionDate 387 * reinstantiationDate 388 * transferDate 389 * lockedDate 390 * unlockedDate 392 o Note that some of the object specific properties are also defined 393 as query paths. The object specific properties include: 395 * Domain: name 396 * Nameserver: name, ipV4, ipV6. 397 * Entity: fn, handle, org, email, voice, country, cc, city. 399 The correspondence between these sorting properties and the RDAP 400 object classes is shown in Table 1: 402 +------------+------------+-----------------+-------+-------+-------+ 403 | Object | Sorting | RDAP property | RFC | RFC | RFC | 404 | class | property | | 7483 | 6350 | 8605 | 405 +------------+------------+-----------------+-------+-------+-------+ 406 | Searchable | Common | eventAction | 4.5 | | | 407 | objects | properties | values suffixed | | | | 408 | | | by "Date" | | | | 409 | | | | | | | 410 | Domain | name | unicodeName/ | 5.3 | | | 411 | | | ldhName | | | | 412 | | | | | | | 413 | Nameserver | name | unicodeName/ | 5.2 | | | 414 | | | ldhName | | | | 415 | | ipV4 | v4 ipAddress | 5.2 | | | 416 | | ipV6 | v6 ipAddress | 5.2 | | | 417 | | | | | | | 418 | Entity | handle | handle | 5.1 | | | 419 | | fn | vcard fn | 5.1 | 6.2.1 | | 420 | | org | vcard org | 5.1 | 6.6.4 | | 421 | | voice | vcard tel with | 5.1 | 6.4.1 | | 422 | | | type="voice" | | | | 423 | | email | vcard email | 5.1 | 6.4.2 | | 424 | | country | country name in | 5.1 | 6.3.1 | | 425 | | | vcard adr | | | | 426 | | cc | country code in | 5.1 | | 3.1 | 427 | | | vcard adr | | | | 428 | | city | locality in | 5.1 | 6.3.1 | | 429 | | | vcard adr | | | | 430 +------------+------------+-----------------+-------+-------+-------+ 432 Table 1: Sorting properties definition 434 Regarding the definitions in Table 1, some further considerations are 435 needed to disambiguate some cases: 437 o Since the response to a search on either domains or nameservers 438 might include both A-labels and U-labels [RFC5890] in general, a 439 consistent sorting policy MUST treat the unicodeName and ldhName 440 as two representations of the same value. By default, the 441 unicodeName value MUST be used while sorting. When the 442 unicodeName is unavailable, the value of the ldhName MUST be used 443 instead; 445 o The jCard "sort-as" parameter MUST be ignored for the sorting 446 capability described in this document; 448 o Even if a nameserver can have multiple IPv4 and IPv6 addresses, 449 the most common configuration includes one address for each IP 450 version. Therefore, the assumption of having a single IPv4 and/or 451 IPv6 value for a nameserver cannot be considered too stringent. 452 When more than one address per IP version is reported, sorting 453 MUST be applied to the first value; 455 o Multiple events with a given action on an object might be 456 returned. If this occurs, sorting MUST be applied to the most 457 recent event; 459 o Except for handle values, all the sorting properties defined for 460 entity objects can be multivalued according to the definition of 461 vCard as given in [RFC6350]. When more than one value is 462 reported, sorting MUST be applied to the preferred value 463 identified by the parameter pref="1". If the pref parameter is 464 missing, sorting MUST be applied to the first value. 466 The "jsonPath" field in the "sorting_metadata" element is used to 467 clarify the RDAP field the sorting property refers to. The mapping 468 between the sorting properties and the JSONPaths of the RDAP fields 469 is shown below: 471 o Searchable objects 473 registrationDate 475 $.domainSearchResults[*].events[?(@.eventAction=="registration" 476 )].eventDate 478 reregistrationDate 480 $.domainSearchResults[*].events[?(@.eventAction=="reregistratio 481 n")].eventDate 483 lastChangedDate 485 $.domainSearchResults[*].events[?(@.eventAction=="last 486 changed")].eventDate 488 expirationDate 490 $.domainSearchResults[*].events[?(@.eventAction=="expiration")] 491 .eventDate 493 deletionDate 495 $.domainSearchResults[*].events[?(@.eventAction=="deletion")].e 496 ventDate 498 reinstantiationDate 500 $.domainSearchResults[*].events[?(@.eventAction=="reinstantiati 501 on")].eventDate 503 transferDate 505 $.domainSearchResults[*].events[?(@.eventAction=="transfer")].e 506 ventDate 508 lockedDate 510 $.domainSearchResults[*].events[?(@.eventAction=="locked")].eve 511 ntDate 513 unlockedDate 515 $.domainSearchResults[*].events[?(@.eventAction=="unlocked")].e 516 ventDate 518 o Domain 520 name 522 $.domainSearchResults[*].unicodeName 524 o Nameserver 526 name 528 $.domainSearchResults[*].unicodeName 530 ipV4 532 $.nameserverSearchResults[*].ipAddresses.v4[0] 534 ipV6 536 $.nameserverSearchResults[*].ipAddresses.v6[0] 538 o Entity 540 handle 542 $.entitySearchResults[*].handle 544 fn 545 $.entitySearchResults[*].vcardArray[1][?(@[0]=="fn")][3] 547 org 549 $.entitySearchResults[*].vcardArray[1][?(@[0]=="org")][3] 551 voice 553 $.entitySearchResults[*].vcardArray[1][?(@[0]=="tel" && 554 @[1].type=="voice")][3] 556 email 558 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email")][3] 560 country 562 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][6] 564 cc 566 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][1].cc 568 city 570 $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][3] 572 The JSONPaths are provided according to the Goessner v.0.8.0 573 specification [GOESSNER-JSON-PATH]. Further documentation about 574 JSONPath operators used in this specification is included in 575 Appendix A. 577 Additional notes on the reported JSONPaths: 579 o those related to the event dates are defined only for the "domain" 580 object. To obtain the equivalent JSONPaths for "entity" and 581 "nameserver", the path segment "domainSearchResults" must be 582 replaced with "entitySearchResults" and "nameserverSearchResults" 583 respectively; 585 o those related to vCard elements are specified without taking into 586 account the "pref" parameter. Servers that sort those values 587 identified by the pref parameter SHOULD update a JSONPath by 588 adding an appropriate filter. For example, if the email values 589 identified by pref="1" are considered for sorting, the JSONPath of 590 the "email" sorting property should be: 592 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" && 593 @[1].pref=="1")][3] 595 2.3.2. Representing Sorting Links 597 An RDAP server MAY use the "links" array of the "sorting_metadata" 598 element to provide ready-made references [RFC8288] to the available 599 sort criteria (Figure 4). Each link represents a reference to an 600 alternate view of the results. 602 { 603 "rdapConformance": [ 604 "rdap_level_0", 605 "sorting" 606 ], 607 ... 608 "sorting_metadata": { 609 "currentSort": "name", 610 "availableSorts": [ 611 { 612 "property": "registrationDate", 613 "jsonPath": "$.domainSearchResults[*] 614 .events[?(@.eventAction==\"registration\")].eventDate", 615 "default": false, 616 "links": [ 617 { 618 "value": "https://example.com/rdap/domains?name=*nr.com 619 &sort=name", 620 "rel": "alternate", 621 "href": "https://example.com/rdap/domains?name=*nr.com 622 &sort=registrationDate", 623 "title": "Result Ascending Sort Link", 624 "type": "application/rdap+json" 625 }, 626 { 627 "value": "https://example.com/rdap/domains?name=*nr.com 628 &sort=name", 629 "rel": "alternate", 630 "href": "https://example.com/rdap/domains?name=*nr.com 631 &sort=registrationDate:d", 632 "title": "Result Descending Sort Link", 633 "type": "application/rdap+json" 634 } 635 ] 636 }, 637 ... 638 ] 639 }, 640 "domainSearchResults": [ 641 ... 642 ] 643 } 645 Figure 4: Example of a "sorting_metadata" instance to implement 646 result sorting 648 2.4. "cursor" Parameter 650 The cursor parameter defined in this specification can be used to 651 encode information about any pagination method. For example, in the 652 case of a simple implementation of the cursor parameter to represent 653 offset pagination information, the cursor value 654 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the Base64 encoding of 655 "offset=100,limit=50". Likewise, in a simple implementation to 656 represent keyset pagination information, the cursor value 657 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the Base64 658 encoding of "key=thelastdomainofthepage.com" whereby the key value 659 identifies the last row of the current page. 661 This solution lets RDAP providers implement a pagination method 662 according to their needs, a user's access level, and the submitted 663 query. Besides, servers can change the method over time without 664 announcing anything to clients. The considerations that have led to 665 this solution are reported in more detail in Appendix B. 667 The ABNF syntax of the cursor parameter is the following: 669 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 671 https://example.com/rdap/domains?name=*nr.com 672 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 674 Figure 5: An example of an RDAP query reporting the "cursor" 675 parameter 677 2.4.1. Representing Paging Links 679 An RDAP server SHOULD use the "links" array of the "paging_metadata" 680 element to provide a ready-made reference [RFC8288] to the next page 681 of the result set (Figure 6). Examples of additional "rel" values a 682 server MAY implement are "first", "last", and "prev". 684 { 685 "rdapConformance": [ 686 "rdap_level_0", 687 "paging" 688 ], 689 ... 690 "notices": [ 691 { 692 "title": "Search query limits", 693 "type": "result set truncated due to excessive load", 694 "description": [ 695 "search results for domains are limited to 50" 696 ] 697 } 698 ], 699 "paging_metadata": { 700 "totalCount": 73, 701 "pageSize": 50, 702 "pageNumber": 1, 703 "links": [ 704 { 705 "value": "https://example.com/rdap/domains?name=*nr.com", 706 "rel": "next", 707 "href": "https://example.com/rdap/domains?name=*nr.com 708 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 709 "title": "Result Pagination Link", 710 "type": "application/rdap+json" 711 } 712 ] 713 }, 714 "domainSearchResults": [ 715 ... 716 ] 717 } 719 Figure 6: Example of a "paging_metadata" instance to implement cursor 720 pagination 722 3. Negative Answers 724 The value constraints for the parameters are defined by their ABNF 725 syntax. Therefore, each request that includes an invalid value for a 726 parameter SHOULD produce an HTTP 400 (Bad Request) response code. 727 The same response SHOULD be returned in the following cases: 729 o If in both single and multi sort the client provides an 730 unsupported value for the "sort" parameter, as well as a value 731 related to an object property not included in the response; 733 o If the client submits an invalid value for the "cursor" parameter. 735 Optionally, the response MAY include additional information regarding 736 the negative answer in the HTTP entity body. 738 4. Implementation Considerations 740 Implementation of the new parameters is technically feasible, as 741 operators for counting, sorting and paging are currently supported by 742 the major relational database management systems. Similar operators 743 are completely or partially supported by the most well-known NoSQL 744 databases (e.g. MongoDB, CouchDB, HBase, Cassandra, Hadoop). 745 Additional implementation notes are included in Appendix C. 747 5. IANA Considerations 749 IANA is requested to register the following values in the RDAP 750 Extensions Registry: 752 Extension identifier: paging 753 Registry operator: Any 754 Published specification: This document. 755 Contact: IETF 756 Intended usage: This extension describes best practice for result 757 set paging. 759 Extension identifier: sorting 760 Registry operator: Any 761 Published specification: This document. 762 Contact: IETF 763 Intended usage: This extension describes best practice for result 764 set sorting. 766 6. Implementation Status 768 NOTE: Please remove this section and the reference to RFC 7942 prior 769 to publication as an RFC. 771 This section records the status of known implementations of the 772 protocol defined by this specification at the time of posting of this 773 Internet-Draft, and is based on a proposal described in [RFC7942]. 774 The description of implementations in this section is intended to 775 assist the IETF in its decision processes in progressing drafts to 776 RFCs. Please note that the listing of any individual implementation 777 here does not imply endorsement by the IETF. Furthermore, no effort 778 has been spent to verify the information presented here that was 779 supplied by IETF contributors. This is not intended as, and must not 780 be construed to be, a catalog of available implementations or their 781 features. Readers are advised to note that other implementations may 782 exist. 784 According to RFC 7942, "this will allow reviewers and working groups 785 to assign due consideration to documents that have the benefit of 786 running code, which may serve as evidence of valuable experimentation 787 and feedback that have made the implemented protocols more mature. 788 It is up to the individual working groups to use this information as 789 they see fit". 791 6.1. IIT-CNR/Registro.it 793 Responsible Organization: Institute of Informatics and Telematics 794 of the National Research Council (IIT-CNR)/Registro.it 795 Location: https://rdap.pubtest.nic.it/ 796 Description: This implementation includes support for RDAP queries 797 using data from .it public test environment. 798 Level of Maturity: This is an "alpha" test implementation. 799 Coverage: This implementation includes all of the features 800 described in this specification. 801 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 803 6.2. APNIC 805 Responsible Organization: Asia-Pacific Network Information Centre 806 Location: https://github.com/APNIC-net/rdap-rmp-demo/tree/sorting- 807 and-paging 808 Description: A proof-of-concept for RDAP mirroring. 809 Level of Maturity: This is a proof-of-concept implementation. 810 Coverage: This implementation includes all of the features 811 described in the specification except for nameserver sorting and 812 unicodeName sorting. 813 Contact Information: Tom Harrison, tomh@apnic.net 815 7. Security Considerations 817 Security services for the operations specified in this document are 818 described in [RFC7481]. 820 A search query typically requires more server resources (such as 821 memory, CPU cycles, and network bandwidth) when compared to a lookup 822 query. This increases the risk of server resource exhaustion and 823 subsequent denial of service due to abuse. This risk can be 824 mitigated by either restricting search functionality or limiting the 825 rate of search requests. Servers can also reduce their load by 826 truncating the results in a response. However, this last security 827 policy can result in a higher inefficiency if the RDAP server does 828 not provide any functionality to return the truncated results. 830 The new parameters presented in this document provide RDAP operators 831 with a way to implement a server that reduces inefficiency risks. 832 The "count" parameter gives the client the ability to evaluate the 833 completeness of a response. The "sort" parameter allows the client 834 to obtain the most relevant information at the beginning of the 835 result set. This can reduce the number of unnecessary search 836 requests. Finally, the "cursor" parameter enables the user to scroll 837 the result set by submitting a sequence of sustainable queries within 838 server-acceptable limits. 840 8. References 842 8.1. Normative References 844 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 845 Requirement Levels", BCP 14, RFC 2119, 846 DOI 10.17487/RFC2119, March 1997, 847 . 849 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 850 Specifications: ABNF", STD 68, RFC 5234, 851 DOI 10.17487/RFC5234, January 2008, 852 . 854 [RFC5890] Klensin, J., "Internationalized Domain Names for 855 Applications (IDNA): Definitions and Document Framework", 856 RFC 5890, DOI 10.17487/RFC5890, August 2010, 857 . 859 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 860 DOI 10.17487/RFC6350, August 2011, 861 . 863 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 864 Protocol (HTTP/1.1): Message Syntax and Routing", 865 RFC 7230, DOI 10.17487/RFC7230, June 2014, 866 . 868 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 869 Registration Data Access Protocol (RDAP)", RFC 7480, 870 DOI 10.17487/RFC7480, March 2015, 871 . 873 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 874 Registration Data Access Protocol (RDAP)", RFC 7481, 875 DOI 10.17487/RFC7481, March 2015, 876 . 878 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 879 Protocol (RDAP) Query Format", RFC 7482, 880 DOI 10.17487/RFC7482, March 2015, 881 . 883 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 884 Registration Data Access Protocol (RDAP)", RFC 7483, 885 DOI 10.17487/RFC7483, March 2015, 886 . 888 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 889 Code: The Implementation Status Section", BCP 205, 890 RFC 7942, DOI 10.17487/RFC7942, July 2016, 891 . 893 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 894 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 895 May 2017, . 897 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 898 Interchange Format", STD 90, RFC 8259, 899 DOI 10.17487/RFC8259, December 2017, 900 . 902 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 903 DOI 10.17487/RFC8288, October 2017, 904 . 906 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 907 ICANN Extensions for the Registration Data Access Protocol 908 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 909 . 911 [W3C.CR-xpath-31-20161213] 912 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 913 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 914 31-20161213, December 2016, 915 . 917 8.2. Informative References 919 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 920 Pagination", July 2014, . 923 [CURSOR-API1] 924 facebook.com, "facebook for developers - Using the Graph 925 API", July 2017, . 928 [CURSOR-API2] 929 twitter.com, "Pagination", 2017, 930 . 933 [GOESSNER-JSON-PATH] 934 Goessner, S., "JSONPath - XPath for JSON", 2007, 935 . 937 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 938 . 941 [OData-Part1] 942 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 943 Part 1: Protocol Plus Errata 03", June 2016, 944 . 949 [REST] Fredrich, T., "RESTful Service Best Practices, 950 Recommendations for Creating Web Services", April 2012, 951 . 954 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 955 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 956 DOI 10.17487/RFC6901, April 2013, 957 . 959 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 960 With Limit and Offset is Slow?", July 2017, 961 . 964 Appendix A. JSONPath operators 966 A JSONPath expression represents a path to find an element (or a set 967 of elements) in a JSON content. 969 The base JSONPath specification requires that implementations support 970 a set of "basic operators". These operators are used to access the 971 elements of a JSON structure like objects and arrays, and their 972 subelements, respectively, object members and array items. No 973 operations are defined for retrieving parent or sibling elements of a 974 given element. The root element is always referred to as $ 975 regardless of it being an object or array. 977 Additionally, the specification permits implementations to support 978 arbitrary script expressions. These can be used to index into an 979 object or array, or to filter elements from an array. While script 980 expression behavior is implementation-defined, most implementations 981 support the basic relational and logical operators, as well as both 982 object member and array item access, sufficiently similar for the 983 purpose of this document. Commonly-supported operators/functions 984 divided into "top-level operators" and "filter operators" are 985 documented in Table 2 and Table 3 respectively. 987 +-------------------+-----------------------------------------+ 988 | Operator | Descritpion | 989 +-------------------+-----------------------------------------+ 990 | $ | Root element | 991 | . | Object member access (dot-notation) | 992 | [''] | Object member access (bracket-notation) | 993 | [] | Array item access | 994 | * | All elements within the specified scope | 995 | [?()] | Filter expression | 996 +-------------------+-----------------------------------------+ 998 Table 2: JSONPath Top-Level Operators 1000 +------------+----------------------------------------+ 1001 | Operator | Descritpion | 1002 +------------+----------------------------------------+ 1003 | @ | Current element being processed | 1004 | . | Object member access | 1005 | [] | Array item access | 1006 | == | Left is equal to right | 1007 | != | Left is not equal to right | 1008 | < | Left is less than right | 1009 | <= | Left is less than or equal to right | 1010 | > | Left is greater than right | 1011 | >= | Left is greater than or equal to right | 1012 | && | Logical conjunction | 1013 | || | Logical disjunction | 1014 +------------+----------------------------------------+ 1016 Table 3: JSONPath Filter Operators 1018 Appendix B. Approaches to Result Pagination 1020 An RDAP query could return a response with hundreds, even thousands, 1021 of objects, especially when partial matching is used. For this 1022 reason, the cursor parameter addressing result pagination is defined 1023 to make responses easier to handle. 1025 Presently, the most popular methods to implement pagination in a REST 1026 API include offset pagination and keyset pagination. Neither 1027 pagination method requires the server to handle the result set in a 1028 storage area across multiple requests since a new result set is 1029 generated each time a request is submitted. Therefore, they are 1030 preferred to any other method requiring the management of a REST 1031 session. 1033 Using limit and offset operators represents the traditionally used 1034 method to implement result pagination. Both of them can be used 1035 individually: 1037 o "limit": means that the server MUST return the first N objects of 1038 the result set; 1040 o "offset": means that the server MUST skip the first N objects and 1041 MUST return objects starting from position N+1. 1043 When limit and offset are used together, they provide the ability to 1044 identify a specific portion of the result set. For example, the pair 1045 "offset=100,limit=50" returns the first 50 objects starting from 1046 position 101 of the result set. 1048 Though easy to implement, offset pagination also includes drawbacks: 1050 o When offset has a very high value, scrolling the result set could 1051 take some time; 1053 o It always requires fetching all rows before dropping as many rows 1054 as specified by offset; 1056 o It may return inconsistent pages when data are frequently updated 1057 (i.e. real-time data). 1059 Keyset pagination [SEEK] adds a query condition that enables the 1060 selection of the only data not yet returned. This method has been 1061 taken as the basis for the implementation of a "cursor" parameter 1062 [CURSOR] by some REST API providers [CURSOR-API1] [CURSOR-API2]. The 1063 cursor is an opaque URL-safe string representing a logical pointer to 1064 the first result of the next page (Figure 5). 1066 Nevertheless, even keyset pagination can be troublesome: 1068 o It needs at least one key field; 1070 o It does not allow sorting simply by any field because the sorting 1071 criterion must contain a key; 1073 o It works best with full composite values support by data base 1074 management systems (i.e. [x,y]>[a,b]), emulation is possible but 1075 inelegant and less efficient; 1077 o It does not allow direct navigation to arbitrary pages because the 1078 result set must be scrolled in sequential order starting from the 1079 initial page; 1081 o Implementing bi-directional navigation is tedious because all 1082 comparison and sort operations have to be reversed. 1084 B.1. Specific Issues Raised by RDAP 1086 Furthermore, in the RDAP context, some additional considerations can 1087 be made: 1089 o An RDAP object is a conceptual aggregation of information 1090 generally collected from more than one data structure (e.g. table) 1091 and this makes it even harder to implement keyset pagination, a 1092 task that is already quite difficult. For example, the entity 1093 object can include information from different data structures 1094 (registrars, registrants, contacts, resellers), each one with its 1095 key field mapping the RDAP entity handle; 1097 o Depending on the number of page results as well as the number and 1098 the complexity of the properties of each RDAP object in the 1099 response, the time required by offset pagination to skip the 1100 previous pages could be much faster than the processing time 1101 needed to build the current page. In fact, RDAP objects are 1102 usually formed by information belonging to multiple data 1103 structures and containing multivalued properties (i.e. arrays) 1104 and, therefore, data selection might therefore be a time consuming 1105 process. This situation occurs even though the selection is 1106 supported by indexes; 1108 o Depending on the access levels defined by each RDAP operator, the 1109 increase in complexity and the decrease in flexibility of keyset 1110 pagination in comparison to offset pagination could be considered 1111 impractical. 1113 Ultimately, both pagination methods have benefits and drawbacks. 1115 Appendix C. Additional Implementation Notes 1117 This section contains an overview of the main choices made during the 1118 implementation of the capabilities defined above in the RDAP public 1119 test server of Registro.it at the Institute of Informatics and 1120 Telematics of the National Research Counci (IIT-CNR). The content of 1121 this section can represent a guidance for those implementers who plan 1122 to provide RDAP users with those capabilities. The RDAP public test 1123 server can be accessed at https://rdap.pubtest.nic.it/. Further 1124 documentation about the server features is available at 1125 https://rdap.pubtest.nic.it/doc/README.html. 1127 C.1. Sorting 1129 If no sort criterion is specified in the query string, the results 1130 are sorted by a default property: "name" for domains and nameservers, 1131 "handle" for entities. The server supports multiple property sorting 1132 but the "sorting_metadata" object includes only the links to 1133 alternative result set views sorted by a single property just to show 1134 the list of sorting properties allowed for each searchable object. 1135 The server supports all the object specific sorting properties 1136 described in the specification except for nameserver sorting based on 1137 unicodeName, that is, the "name" sorting property is mapped onto the 1138 "ldhName" response field. Regarding the object common properties, 1139 the sorting by registrationDate, expirationDate, lastChangedDate and 1140 transferDate is supported. 1142 C.2. Counting 1144 The counting operation is implemented through a separate query. Some 1145 relational database management systems support custom operators to 1146 get the total count together with the rows, but the resulting query 1147 can be considerably more expensive than that performed without the 1148 total count. Therefore, as "totalCount" is an optional response 1149 information, fetching always the total number of rows has been 1150 considered an inefficient solution. Furthermore, to avoid the 1151 processing of unnecessary queries, when the "count" parameter is 1152 included in the submitted query, it is not also repeated in the query 1153 strings of the "links" array provided in both "paging_metadata" and 1154 "sorting_metadata" objects. 1156 C.3. Paging 1158 The server implements the cursor pagination through the keyset 1159 pagination when sorting by a unique property is requested or the 1160 default sort is applied, through offset pagination otherwise. As 1161 most of the relational database management systems don't support the 1162 comparison of full composite values natively, the implementation of 1163 full keyset pagination seem to be troublesome so, at least initially, 1164 a selective applicability of keyset pagination is advisable. 1165 Moreover, the "cursor" value encodes not only information about 1166 pagination but also about the search pattern and the other query 1167 parameters in order to check the consistency of the entire query 1168 string. If the "cursor" value is inconsistent with the rest of the 1169 query string, the server returns an error response. 1171 Acknowledgements 1173 The authors would like to acknowledge Brian Mountford, Tom Harrison, 1174 Karl Heinz Wolf and Jasdip Singh for their contribution to the 1175 development of this document. 1177 Change Log 1179 00: Initial working group version ported from draft-loffredo-regext- 1180 rdap-sorting-and-paging-05 1181 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1182 consistent between the pagination methods. Renamed 1183 "Considerations about Paging Implementation" section in ""cursor" 1184 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1185 detailed description of both "sorting_metadata" and 1186 "paging_metadata" objects. 1187 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1188 of the cursor parameter. Rearranged the layout of some sections. 1189 Removed some items from "Informative References" section. Changed 1190 "IANA Considerations" section. 1191 03: Added "cc" to the list of sorting properties in "Sorting 1192 Properties Declaration" section. Added RFC8605 to the list of 1193 "Informative References". 1194 04: Replaced "ldhName" with "name" in the "Sorting Properties 1195 Declaration" section. Clarified the sorting logic for the JSON 1196 value types and the sorting policy for multivalued fields. 1197 05: Clarified the logic of sorting on IP addresses. Clarified the 1198 mapping between the sorting properties and the RDAP fields. 1199 Updated "Acknowledgements" section. 1200 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1201 "paging_metadata" object. 1202 07: Added "Paging Responses to POST Requests" section. 1203 08: Added "Approaches to Result Pagination" section to appendix. 1204 Added the case of requesting a sort on a property not included in 1205 the response to the errors listed in the "Negative Answers" 1206 section. 1207 09: Updated the "Implementation Status" section to include APNIC 1208 implementation. Moved the "RDAP Conformance" section up in the 1209 document. Removed the "Paging Responses to POST Requests" 1210 section. Updated the "Acknowledgements" section. Removed unused 1211 references. In the "Sorting Properties Declaration" section: 1213 * clarified the logic of sorting on events; 1214 * corrected the JSONPath of the "lastChanged" sorting property; 1215 * provided a JSONPath example taking into account the vCard 1216 "pref" parameter. 1217 10: Corrected the JSONPaths of both "fn" and "org" sorting 1218 properties in Table 2. Corrected JSON content in Figure 4. Moved 1219 [W3C.CR-xpath-31-20161213] and [RFC7942] to the "Normative 1220 References". Changed the rdapConformance tags "sorting_level_0" 1221 and "paging_level_0" to "sorting" and "paging" respectively. 1222 11: Added the "JSONPath operators" section to appendix. 1223 12: Changed the content of "JSONPath operators" section. 1224 13: Minor pre-AD review edits. 1225 14: Additionl minor pre-AD review edits. 1226 15: In section ""sort" Parameter" added a paragraph providing 1227 conversions of IP addresses into their numerical representations. 1228 In section "Sorting Properties Declaration" rearranged Table 2 in 1229 a list to make the content more readable. Other minor edits due 1230 to AD review. 1231 16: In section "Introduction" replaced "... large result set that 1232 could be truncated ..." with "... large result set that is often 1233 truncated ..." as suggested by Gen-ART reviewer. Added 1234 Appendix C. 1236 Authors' Addresses 1238 Mario Loffredo 1239 IIT-CNR/Registro.it 1240 Via Moruzzi,1 1241 Pisa 56124 1242 IT 1244 Email: mario.loffredo@iit.cnr.it 1245 URI: http://www.iit.cnr.it 1247 Maurizio Martinelli 1248 IIT-CNR/Registro.it 1249 Via Moruzzi,1 1250 Pisa 56124 1251 IT 1253 Email: maurizio.martinelli@iit.cnr.it 1254 URI: http://www.iit.cnr.it 1255 Scott Hollenbeck 1256 Verisign Labs 1257 12061 Bluemont Way 1258 Reston, VA 20190 1259 USA 1261 Email: shollenbeck@verisign.com 1262 URI: https://www.verisignlabs.com/