idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-13.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (May 29, 2020) is 1425 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 529 -- Looks like a reference, but probably isn't: '1' on line 530 -- Looks like a reference, but probably isn't: '3' on line 530 -- Looks like a reference, but probably isn't: '6' on line 506 == Unused Reference: 'RFC8605' is defined on line 848, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 4 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: November 30, 2020 S. Hollenbeck 6 Verisign Labs 7 May 29, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-13 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on November 30, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.1.1. RDAP Conformance . . . . . . . . . . . . . . . . . . 6 64 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 65 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 66 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 8 67 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 13 68 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 15 69 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 15 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 16 71 4. Implementation Considerations . . . . . . . . . . . . . . . . 17 72 5. Implementation Status . . . . . . . . . . . . . . . . . . . . 17 73 5.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 17 74 5.2. APNIC . . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 76 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 77 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 78 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 79 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 80 9.2. Informative References . . . . . . . . . . . . . . . . . 21 81 Appendix A. JSONPath operators . . . . . . . . . . . . . . . . . 22 82 Appendix B. Approaches to Result Pagination . . . . . . . . . . 23 83 B.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 24 84 Appendix C. Change Log . . . . . . . . . . . . . . . . . . . . . 25 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 87 1. Introduction 89 The availability of functionality for result sorting and paging 90 provides benefits to both clients and servers in the implementation 91 of RESTful services [REST]. These benefits include: 93 o reducing the server response bandwidth requirements; 94 o improving server response time; 95 o improving query precision and, consequently, obtaining more 96 reliable results; 98 o decreasing server query processing load; 99 o reducing client response processing time. 101 Approaches to implementing features for result sorting and paging can 102 be grouped into two main categories: 104 1. Sorting and paging are implemented through the introduction of 105 additional parameters in the query string (i.e. ODATA protocol 106 [OData-Part1]); 108 2. Information related to the number of results and the specific 109 portion of the result set to be returned, in addition to a set of 110 ready-made links for the result set scrolling, are inserted in 111 the HTTP header of the request/response. 113 However, there are some drawbacks associated with the use of the HTTP 114 header. First, the header properties cannot be set directly from a 115 web browser. Moreover, in an HTTP session, the information on the 116 status (i.e. the session identifier) is usually inserted in the 117 header or in a cookie, while the information on the resource 118 identification or the search type is included in the query string. 119 The second approach is therefore not compliant with the HTTP standard 120 [RFC7230]. As a result, this document describes a specification 121 based on the use of query parameters. 123 Currently, the RDAP protocol [RFC7482] defines two query types: 125 o lookup: the server returns only one object; 126 o search: the server returns a collection of objects. 128 While the lookup query does not raise issues in response size 129 management, the search query can potentially generate a large result 130 set that could be truncated according to server limits. In addition, 131 it is not possible to obtain the total number of objects found that 132 might be returned in a search query response [RFC7483]. Lastly, 133 there is no way to specify sort criteria to return the most relevant 134 objects at the beginning of the result set. Therefore, the client 135 might traverse the whole result set to find the relevant objects or, 136 due to truncation, might not find them at all. 138 The specification described in this document extends RDAP query 139 capabilities to enable result sorting and paging, by adding new query 140 parameters that can be applied to RDAP search path segments. The 141 service is implemented using the Hypertext Transfer Protocol (HTTP) 142 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 144 The implementation of the new parameters is technically feasible, as 145 operators for counting, sorting and paging rows are currently 146 supported by the major RDBMSs. 148 1.1. Conventions Used in This Document 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in BCP 14 [RFC2119] 153 [RFC8174] when, and only when, they appear in all capitals, as shown 154 here. 156 2. RDAP Query Parameter Specification 158 The new query parameters are OPTIONAL extensions of path segments 159 defined in RFC 7482 [RFC7482]. They are as follows: 161 o "count": a boolean value that allows a client to request return of 162 the total number of objects found; 164 o "sort": a string value that allows a client to request a specific 165 sort order for the result set; 167 o "cursor": a string value representing a pointer to a specific 168 fixed size portion of the result set. 170 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 171 sections to describe the formal syntax of these new parameters. 173 2.1. Sorting and Paging Metadata 175 According to most advanced principles in REST design, collectively 176 known as HATEOAS (Hypermedia as the Engine of Application State) 177 ([HATEOAS]), a client entering a REST application through an initial 178 URI should use server-provided links to dynamically discover 179 available actions and access the resources it needs. In this way, 180 the client is not requested to have prior knowledge of the service 181 and, consequently, to hard code the URIs of different resources. 182 This allows the server to make URI changes as the API evolves without 183 breaking clients. Definitively, a REST service should be as self- 184 descriptive as possible. 186 Therefore, servers implementing the query parameters described in 187 this specification SHOULD provide additional information in their 188 responses about both the available sorting criteria and possible 189 pagination. Such information is collected in two OPTIONAL response 190 elements named, respectively, "sorting_metadata" and 191 "paging_metadata". 193 The "sorting_metadata" element contains the following properties: 195 o "currentSort": "String" (OPTIONAL) either the value of sort 196 "parameter" as specified in the query string or the sort applied 197 by default, if any; 199 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of 200 objects, with each element describing an available sort criterion. 201 Members are: 203 * "property": "String" (REQUIRED) the name that can be used by 204 the client to request the sort criterion; 205 * "default": "Boolean" (REQUIRED) whether the sort criterion is 206 applied by default; 207 * "jsonPath": "String" (OPTIONAL) the JSONPath of the RDAP field 208 corresponding to the property; 209 * "links": "Link[]" (OPTIONAL) an array of links as described in 210 RFC 8288 [RFC8288] containing the query string that applies the 211 sort criterion. 213 At least one of the "currentSort" and "availableSorts" properties 214 MUST be present. 216 The "paging_metadata" element contains the following fields: 218 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 219 the total number of objects found. It MUST be provided if the 220 query string contains the "count" parameter; 222 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 223 number of objects returned in the current page. It MUST be 224 provided when the total number of objects exceeds the page size. 225 This property is redundant for clients because the page size can 226 be derived from the length of the search results array but it can 227 be helpful if the end user interacts with the server through a web 228 browser; 230 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 231 the number of the current page in the result set. It MUST be 232 provided when the total number of objects found exceeds the page 233 size; 235 o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 236 8288 [RFC8288] containing the reference to the next page. In this 237 specification, only forward pagination is described because it is 238 all that is necessary to traverse the result set. 240 2.1.1. RDAP Conformance 242 Servers returning the "paging_metadata" element in their response 243 MUST include the string literal "paging" in the rdapConformance 244 array. Servers returning the "sorting_metadata" element MUST include 245 the string literal "sorting". 247 2.2. "count" Parameter 249 Currently, the RDAP protocol does not allow a client to determine the 250 total number of the results in a query response when the result set 251 is truncated. This is inefficient because the user cannot determine 252 if the result set is complete. 254 The "count" parameter provides additional functionality (Figure 1) 255 that allows a client to request information from the server that 256 specifies the total number of objects matching the search pattern. 258 https://example.com/rdap/domains?name=*nr.com&count=true 260 Figure 1: Example of RDAP query reporting the "count" parameter 262 The ABNF syntax is the following: 264 count = "count=" ( trueValue / falseValue ) 265 trueValue = ("true" / "yes" / "1") 266 falseValue = ("false" / "no" / "0") 268 A trueValue means that the server MUST provide the total number of 269 the objects in the "totalCount" field of the "paging_metadata" 270 element (Figure 2). A falseValue means that the server MUST NOT 271 provide this number. 273 { 274 "rdapConformance": [ 275 "rdap_level_0", 276 "paging" 277 ], 278 ... 279 "paging_metadata": { 280 "totalCount": 43 281 }, 282 "domainSearchResults": [ 283 ... 284 ] 285 } 287 Figure 2: Example of RDAP response with "paging_metadata" element 288 containing the "totalCount" field 290 2.3. "sort" Parameter 292 The RDAP protocol does not provide any capability to specify result 293 set sort criteria. A server could implement a default sorting scheme 294 according to the object class, but this feature is not mandatory and 295 might not meet user requirements. Sorting can be addressed by the 296 client, but this solution is rather inefficient. Sorting features 297 provided by the RDAP server could help avoid truncation of relevant 298 results. 300 The "sort" parameter allows the client to ask the server to sort the 301 results according to the values of one or more properties and 302 according to the sort direction of each property. The ABNF syntax is 303 the following: 305 sort = "sort=" sortItem *( "," sortItem ) 306 sortItem = property-ref [":" ( "a" / "d" ) ] 307 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 309 "a" means that an ascending sort MUST be applied, "d" means that a 310 descending sort MUST be applied. If the sort direction is absent, an 311 ascending sort MUST be applied (Figure 3). 313 https://example.com/rdap/domains?name=*nr.com&sort=name 315 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 317 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 319 Figure 3: Examples of RDAP query reporting the "sort" parameter 321 With the exception of sorting IP addresses, servers MUST implement 322 sorting according to the JSON value type of the RDAP field the 323 sorting property refers to. That is, JSON strings MUST be sorted 324 lexicographically and JSON numbers MUST be sorted numerically. If IP 325 addresses are represented as JSON strings, they MUST be sorted based 326 on their numeric conversion. 328 If the "sort" parameter reports an allowed sorting property, it MUST 329 be provided in the "currentSort" field of the "sorting_metadata" 330 element. 332 2.3.1. Sorting Properties Declaration 334 In the "sort" parameter ABNF syntax, property-ref represents a 335 reference to a property of an RDAP object. Such a reference could be 336 expressed by using a JSONPath. The JSONPath in a JSON document 337 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 338 XML document. For example, the JSONPath to select the value of the 339 ASCII name inside an RDAP domain object is "$.ldhName", where $ 340 identifies the root of the document (DOM). Another way to select a 341 value inside a JSON document is the JSON Pointer [RFC6901]. While 342 JSONPath or JSON Pointer are both standard ways to select any value 343 inside JSON data, neither is particularly easy to use (e.g. 344 "$.events[?(@.eventAction='registration')].eventDate" is the JSONPath 345 expression of the registration date in an RDAP domain object). 347 Therefore, this specification provides a definition of property-ref 348 in terms of RDAP properties. However, not all the RDAP properties 349 are suitable to be used in sort criteria, such as: 351 o properties providing service information (e.g. links, notices, 352 remarks, etc.); 354 o multivalued properties (e.g. status, roles, variants, etc.); 356 o properties modeling relationships to other objects (e.g. 357 entities). 359 On the contrary, properties expressed as values of other properties 360 (e.g. registration date) could be used in such a context. The list 361 of properties an RDAP server MAY implement are divided into two 362 groups: object common properties and object specific properties. 364 o Object common properties. Object common properties are derived 365 from the merge of the "eventAction" and the "eventDate" 366 properties. The following values of the "sort" parameter are 367 defined: 369 * registrationDate 370 * reregistrationDate 371 * lastChangedDate 372 * expirationDate 373 * deletionDate 374 * reinstantiationDate 375 * transferDate 376 * lockedDate 377 * unlockedDate 379 o Note that some of the object specific properties are also defined 380 as query paths. The object specific properties include: 382 * Domain: name 383 * Nameserver: name, ipV4, ipV6. 384 * Entity: fn, handle, org, email, voice, country, cc, city. 386 The correspondence between these sorting properties and the RDAP 387 object classes is shown in Table 1: 389 +-----------+-----------+---------------------+------+-------+------+ 390 | Object | Sorting | RDAP property | RFC | RFC | RFC | 391 | class | property | | 7483 | 6350 | 8605 | 392 +-----------+-----------+---------------------+------+-------+------+ 393 | Searchabl | Common pr | eventAction values | 4.5. | | | 394 | e objects | operties | suffixed by "Date" | | | | 395 | | | | | | | 396 | Domain | name | unicodeName/ldhName | 5.3. | | | 397 | | | | | | | 398 | Nameserve | name | unicodeName/ldhName | 5.2. | | | 399 | r | | | | | | 400 | | ipV4 | v4 ipAddress | 5.2. | | | 401 | | ipV6 | v6 ipAddress | 5.2. | | | 402 | | | | | | | 403 | Entity | handle | handle | 5.1. | | | 404 | | fn | vcard fn | 5.1. | 6.2.1 | | 405 | | org | vcard org | 5.1. | 6.6.4 | | 406 | | voice | vcard tel with | 5.1. | 6.4.1 | | 407 | | | type="voice" | | | | 408 | | email | vcard email | 5.1. | 6.4.2 | | 409 | | country | country name in | 5.1. | 6.3.1 | | 410 | | | vcard adr | | | | 411 | | cc | country code in | 5.1. | | 3.1 | 412 | | | vcard adr | | | | 413 | | city | locality in vcard | 5.1. | 6.3.1 | | 414 | | | adr | | | | 415 +-----------+-----------+---------------------+------+-------+------+ 417 Table 1: Sorting properties definition 419 With regard to the definitions in Table 1, some further 420 considerations are needed to disambiguate some cases: 422 o Since the response to a search on either domains or nameservers 423 might include both A-labels and U-labels ([RFC5890]) in general, a 424 consistent sorting policy MUST treat the unicodeName and ldhName 425 as two representations of the same value. By default, the 426 unicodeName value MUST be used while sorting. When unicodeName is 427 unavailable, the value of ldhName MUST be used instead; 429 o The jCard "sort-as" parameter MUST be ignored for the purpose of 430 the sorting capability described in this document; 432 o Even if a nameserver can have multiple IPv4 and IPv6 addresses, 433 the most common configuration includes one address for each IP 434 version. Therefore, the assumption of having a single IPv4 and/or 435 IPv6 value for a nameserver cannot be considered too stringent. 437 When more than one address per IP version is reported, sorting 438 MUST be applied to the first value; 440 o Multiple events with a given action on an object might be 441 returned. If this occurs, sorting MUST be applied to the most 442 recent event; 444 o With the exception of handle values, all the sorting properties 445 defined for entity objects can be multivalued according to the 446 definition of vCard as given in RFC 6350 [RFC6350]. When more 447 than one value is reported, sorting MUST be applied to the 448 preferred value identified by the parameter pref="1". If the pref 449 parameter is missing, sorting MUST be applied to the first value. 451 The "jsonPath" field in the "sorting_metadata" element is used to 452 clarify the RDAP field the sorting property refers to. The mapping 453 between the sorting properties and the JSONPaths of the RDAP fields 454 is shown in Table 2. The JSONPaths are provided according to the 455 Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]). Further 456 documentation about JSONPath operators used in Table 2 is included in 457 Appendix A. 459 +-------+-------------+---------------------------------------------+ 460 | Objec | Sorting | JSONPath | 461 | t | property | | 462 | class | | | 463 +-------+-------------+---------------------------------------------+ 464 | Searc | registratio | $.domainSearchResults[*].events[?(@.eventAc | 465 | hable | nDate | tion=="registration")].eventDate | 466 | objec | | | 467 | ts | | | 468 | | reregistrat | $.domainSearchResults[*].events[?(@.eventAc | 469 | | ionDate | tion=="reregistration")].eventDate | 470 | | lastChanged | $.domainSearchResults[*].events[?(@.eventAc | 471 | | Date | tion=="last changed")].eventDate | 472 | | expirationD | $.domainSearchResults[*].events[?(@.eventAc | 473 | | ate | tion=="expiration")].eventDate | 474 | | deletionDat | $.domainSearchResults[*].events[?(@.eventAc | 475 | | e | tion=="deletion")].eventDate | 476 | | reinstantia | $.domainSearchResults[*].events[?(@.eventAc | 477 | | tionDate | tion=="reinstantiation")].eventDate | 478 | | transferDat | $.domainSearchResults[*].events[?(@.eventAc | 479 | | e | tion=="transfer")].eventDate | 480 | | lockedDate | $.domainSearchResults[*].events[?(@.eventAc | 481 | | | tion=="locked")].eventDate | 482 | | unlockedDat | $.domainSearchResults[*].events[?(@.eventAc | 483 | | e | tion=="unlocked")].eventDate | 484 | | | | 485 | Domai | name | $.domainSearchResults[*].unicodeName | 486 | n | | | 487 | | | | 488 | Names | name | $.nameserverSearchResults[*].unicodeName | 489 | erver | | | 490 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 491 | | | [0] | 492 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 493 | | | [0] | 494 | | | | 495 | Entit | handle | $.entitySearchResults[*].handle | 496 | y | | | 497 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 498 | | | 0]=="fn")][3] | 499 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 500 | | | 0]=="org")][3] | 501 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 502 | | | 0]=="tel" && @[1].type=="voice")][3] | 503 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 504 | | | 0]=="email")][3] | 505 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 506 | | | 0]=="adr")][3][6] | 507 | | cc | $.entitySearchResults[*].vcardArray[1][?(@[ | 508 | | | 0]=="adr")][1].cc | 509 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 510 | | | 0]=="adr")][3][3] | 511 +-------+-------------+---------------------------------------------+ 513 Table 2: Sorting properties - JSONPath Mapping 515 Table 2 JSONPath notes: 517 o Those related to the event dates are defined only for the "domain" 518 object. To obtain the equivalent JSONPaths for "entity" and 519 "nameserver", the path segment "domainSearchResults" must be 520 replaced with "entitySearchResults" and "nameserverSearchResults" 521 respectively; 523 o Those related to vCard elements are specified without taking into 524 account the "pref" parameter. Servers that sort those values 525 identified by the pref parameter SHOULD update a JSONPath by 526 adding an appropriate filter. For example, if the email values 527 identified by pref="1" are considered for sorting, the JSONPath of 528 the "email" sorting property should be: 529 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" && 530 @[1].pref=="1")][3] 532 2.3.2. Representing Sorting Links 534 An RDAP server MAY use the "links" array of the "sorting_metadata" 535 element to provide ready-made references [RFC8288] to the available 536 sort criteria (Figure 4). Each link represents a reference to an 537 alternate view of the results. 539 { 540 "rdapConformance": [ 541 "rdap_level_0", 542 "sorting" 543 ], 544 ... 545 "sorting_metadata": { 546 "currentSort": "name", 547 "availableSorts": [ 548 { 549 "property": "registrationDate", 550 "jsonPath": "$.domainSearchResults[*] 551 .events[?(@.eventAction==\"registration\")].eventDate", 552 "default": false, 553 "links": [ 554 { 555 "value": "https://example.com/rdap/domains?name=*nr.com 556 &sort=name", 557 "rel": "alternate", 558 "href": "https://example.com/rdap/domains?name=*nr.com 559 &sort=registrationDate", 560 "title": "Result Ascending Sort Link", 561 "type": "application/rdap+json" 562 }, 563 { 564 "value": "https://example.com/rdap/domains?name=*nr.com 565 &sort=name", 566 "rel": "alternate", 567 "href": "https://example.com/rdap/domains?name=*nr.com 568 &sort=registrationDate:d", 569 "title": "Result Descending Sort Link", 570 "type": "application/rdap+json" 571 } 572 ] 573 }, 574 ... 575 ] 576 }, 577 "domainSearchResults": [ 578 ... 579 ] 580 } 582 Figure 4: Example of a "sorting_metadata" instance to implement 583 result sorting 585 2.4. "cursor" Parameter 587 The cursor parameter defined in this specification can be used to 588 encode information about any pagination method. For example, in the 589 case of a simple implementation of the cursor parameter to represent 590 offset pagination information, the cursor value 591 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the Base64 encoding of 592 "offset=100,limit=50". Likewise, in a simple implementation to 593 represent keyset pagination information, the cursor value 594 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the Base64 595 encoding of "key=thelastdomainofthepage.com" whereby the key value 596 identifies the last row of the current page. 598 This solution lets RDAP providers implement a pagination method 599 according to their needs, a user's access level, and the submitted 600 query. In addition, servers can change the method over time without 601 announcing anything to clients. The considerations that has led to 602 this solution are reported in more detail in Appendix B. 604 The ABNF syntax of the cursor paramter is the following: 606 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 608 https://example.com/rdap/domains?name=*nr.com 609 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 611 Figure 5: An example of RDAP query reporting the "cursor" parameter 613 2.4.1. Representing Paging Links 615 An RDAP server SHOULD use the "links" array of the "paging_metadata" 616 element to provide a ready-made reference [RFC8288] to the next page 617 of the result set (Figure 6). Examples of additional "rel" values a 618 server MAY implement are "first", "last", and "prev". 620 { 621 "rdapConformance": [ 622 "rdap_level_0", 623 "paging" 624 ], 625 ... 626 "notices": [ 627 { 628 "title": "Search query limits", 629 "type": "result set truncated due to excessive load", 630 "description": [ 631 "search results for domains are limited to 50" 632 ] 633 } 634 ], 635 "paging_metadata": { 636 "totalCount": 73, 637 "pageSize": 50, 638 "pageNumber": 1, 639 "links": [ 640 { 641 "value": "https://example.com/rdap/domains?name=*nr.com", 642 "rel": "next", 643 "href": "https://example.com/rdap/domains?name=*nr.com 644 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 645 "title": "Result Pagination Link", 646 "type": "application/rdap+json" 647 } 648 ] 649 }, 650 "domainSearchResults": [ 651 ... 652 ] 653 } 655 Figure 6: Example of a "paging_metadata" instance to implement cursor 656 pagination 658 3. Negative Answers 660 The value constraints for the parameters are defined by their ABNF 661 syntax. Therefore, each request that includes an invalid value for a 662 parameter SHOULD produce an HTTP 400 (Bad Request) response code. 663 The same response SHOULD be returned in the following cases: 665 o If in both single and multi sort the client provides an 666 unsupported value for the "sort" parameter, as well as a value 667 related to an object property not included in the response; 669 o If the client submits an invalid value for the "cursor" parameter. 671 Optionally, the response MAY include additional information regarding 672 the negative answer in the HTTP entity body. 674 4. Implementation Considerations 676 Implementation of the new parameters is technically feasible, as 677 operators for counting, sorting and paging are currently supported by 678 the major RDBMSs. Similar operators are completely or partially 679 supported by the most known NoSQL databases (MongoDB, CouchDB, HBase, 680 Cassandra, Hadoop). 682 5. Implementation Status 684 NOTE: Please remove this section and the reference to RFC 7942 prior 685 to publication as an RFC. 687 This section records the status of known implementations of the 688 protocol defined by this specification at the time of posting of this 689 Internet-Draft, and is based on a proposal described in RFC 7942 690 [RFC7942]. The description of implementations in this section is 691 intended to assist the IETF in its decision processes in progressing 692 drafts to RFCs. Please note that the listing of any individual 693 implementation here does not imply endorsement by the IETF. 694 Furthermore, no effort has been spent to verify the information 695 presented here that was supplied by IETF contributors. This is not 696 intended as, and must not be construed to be, a catalog of available 697 implementations or their features. Readers are advised to note that 698 other implementations may exist. 700 According to RFC 7942, "this will allow reviewers and working groups 701 to assign due consideration to documents that have the benefit of 702 running code, which may serve as evidence of valuable experimentation 703 and feedback that have made the implemented protocols more mature. 704 It is up to the individual working groups to use this information as 705 they see fit". 707 5.1. IIT-CNR/Registro.it 709 Responsible Organization: Institute of Informatics and Telematics 710 of National Research Council (IIT-CNR)/Registro.it 711 Location: https://rdap.pubtest.nic.it/ 712 Description: This implementation includes support for RDAP queries 713 using data from .it public test environment. 714 Level of Maturity: This is an "alpha" test implementation. 715 Coverage: This implementation includes all of the features 716 described in this specification. 718 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 720 5.2. APNIC 722 Responsible Organization: Asia-Pacific Network Information Centre 723 Location: https://github.com/APNIC-net/rdap-rmp-demo/tree/sorting- 724 and-paging 725 Description: A proof-of-concept for RDAP mirroring. 726 Level of Maturity: This is a proof-of-concept implementation. 727 Coverage: This implementation includes all of the features 728 described in the specification except for nameserver sorting and 729 unicodeName sorting. 730 Contact Information: Tom Harrison, tomh@apnic.net 732 6. IANA Considerations 734 IANA is requested to register the following values in the RDAP 735 Extensions Registry: 737 Extension identifier: paging 738 Registry operator: Any 739 Published specification: This document. 740 Contact: IESG 741 Intended usage: This extension describes a best practice for 742 result set paging. 744 Extension identifier: sorting 745 Registry operator: Any 746 Published specification: This document. 747 Contact: IESG 748 Intended usage: This extension describes a best practice for 749 result set sorting. 751 7. Security Considerations 753 Security services for the operations specified in this document are 754 described in RFC 7481 [RFC7481]. 756 A search query typically requires more server resources (such as 757 memory, CPU cycles, and network bandwidth) when compared to a lookup 758 query. This increases the risk of server resource exhaustion and 759 subsequent denial of service due to abuse. This risk can be 760 mitigated by either restricting search functionality or limiting the 761 rate of search requests. Servers can also reduce their load by 762 truncating the results in a response. However, this last security 763 policy can result in a higher inefficiency if the RDAP server does 764 not provide any functionality to return the truncated results. 766 The new parameters presented in this document provide RDAP operators 767 with a way to implement a server that reduces inefficiency risks. 768 The "count" parameter gives the client te ability to evaluate the 769 completeness of a response. The "sort" parameter allows the client 770 to obtain the most relevant information at the beginning of the 771 result set. This can reduce the amount of unnecessary search 772 requests. Finally, the "cursor" parameter enables the user to scroll 773 the result set by submitting a sequence of sustainable queries within 774 server-acceptable limits. 776 8. Acknowledgements 778 The authors would like to acknowledge Brian Mountford, Tom Harrison, 779 Karl Heinz Wolf and Jasdip Singh for their contribution to the 780 development of this document. 782 9. References 784 9.1. Normative References 786 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 787 Requirement Levels", BCP 14, RFC 2119, 788 DOI 10.17487/RFC2119, March 1997, 789 . 791 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 792 Specifications: ABNF", STD 68, RFC 5234, 793 DOI 10.17487/RFC5234, January 2008, 794 . 796 [RFC5890] Klensin, J., "Internationalized Domain Names for 797 Applications (IDNA): Definitions and Document Framework", 798 RFC 5890, DOI 10.17487/RFC5890, August 2010, 799 . 801 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 802 DOI 10.17487/RFC6350, August 2011, 803 . 805 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 806 Protocol (HTTP/1.1): Message Syntax and Routing", 807 RFC 7230, DOI 10.17487/RFC7230, June 2014, 808 . 810 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 811 Registration Data Access Protocol (RDAP)", RFC 7480, 812 DOI 10.17487/RFC7480, March 2015, 813 . 815 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 816 Registration Data Access Protocol (RDAP)", RFC 7481, 817 DOI 10.17487/RFC7481, March 2015, 818 . 820 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 821 Protocol (RDAP) Query Format", RFC 7482, 822 DOI 10.17487/RFC7482, March 2015, 823 . 825 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 826 Registration Data Access Protocol (RDAP)", RFC 7483, 827 DOI 10.17487/RFC7483, March 2015, 828 . 830 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 831 Code: The Implementation Status Section", BCP 205, 832 RFC 7942, DOI 10.17487/RFC7942, July 2016, 833 . 835 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 836 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 837 May 2017, . 839 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 840 Interchange Format", STD 90, RFC 8259, 841 DOI 10.17487/RFC8259, December 2017, 842 . 844 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 845 DOI 10.17487/RFC8288, October 2017, 846 . 848 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 849 ICANN Extensions for the Registration Data Access Protocol 850 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 851 . 853 [W3C.CR-xpath-31-20161213] 854 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 855 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 856 31-20161213, December 2016, 857 . 859 9.2. Informative References 861 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 862 Pagination", July 2014, . 865 [CURSOR-API1] 866 facebook.com, "facebook for developers - Using the Graph 867 API", July 2017, . 870 [CURSOR-API2] 871 twitter.com, "Pagination", 2017, 872 . 875 [GOESSNER-JSON-PATH] 876 Goessner, S., "JSONPath - XPath for JSON", 2007, 877 . 879 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 880 . 883 [OData-Part1] 884 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 885 Part 1: Protocol Plus Errata 03", June 2016, 886 . 891 [REST] Fredrich, T., "RESTful Service Best Practices, 892 Recommendations for Creating Web Services", April 2012, 893 . 896 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 897 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 898 DOI 10.17487/RFC6901, April 2013, 899 . 901 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 902 With Limit and Offset is Slow?", July 2017, 903 . 906 Appendix A. JSONPath operators 908 A JSONPath expression represents a path to find an element (or a set 909 of elements) in a JSON content. 911 The base JSONPath specification requires that implementations support 912 a set of "basic operators". These operators are used to access the 913 elements of a JSON structure like objects and arrays, and their 914 subelements, respectively, object members and array items. No 915 operations are defined for retrieving parent or sibling elements of a 916 given element. The root element is always referred to as $ 917 regardless of it being an object or array. 919 Additionally, the specification permits implementations to support 920 arbitrary script expressions. These can be used to index into an 921 object or an array, or to filter elements from an array. While 922 script expression behaviour is implementation-defined, most 923 implementations support the basic relational and logical operators, 924 as well as both object member and array item access, sufficiently 925 similarly for the purposes of this document. Commonly-supported 926 operators/functions divided into "top-level operators" and "filter 927 operators" are documented in Table 3 and Table 4 respectively. 929 +-------------------+-----------------------------------------+ 930 | Operator | Descritpion | 931 +-------------------+-----------------------------------------+ 932 | $ | Root element | 933 | . | Object member access (dot-notation) | 934 | [''] | Object member access (bracket-notation) | 935 | [] | Array item access | 936 | * | All elements within the specified scope | 937 | [?()] | Filter expression | 938 +-------------------+-----------------------------------------+ 940 Table 3: JSONPath Top-Level Operators 942 +------------+----------------------------------------+ 943 | Operator | Descritpion | 944 +------------+----------------------------------------+ 945 | @ | Current element being processed | 946 | . | Object member access | 947 | [] | Array item access | 948 | == | Left is equal to right | 949 | != | Left is not equal to right | 950 | < | Left is less than right | 951 | <= | Left is less than or equal to right | 952 | > | Left is greater than right | 953 | >= | Left is greater than or equal to right | 954 | && | Logical conjunction | 955 | || | Logical disjunction | 956 +------------+----------------------------------------+ 958 Table 4: JSONPath Filter Operators 960 Appendix B. Approaches to Result Pagination 962 An RDAP query could return a response with hundreds, even thousands, 963 of objects, especially when partial matching is used. For that 964 reason, the cursor parameter addressing result pagination is defined 965 to make responses easier to handle. 967 Presently, the most popular methods to implement pagination in a REST 968 API include offset pagination and keyset pagination. Neither 969 pagination method requires the server to handle the result set in a 970 storage area across multiple requests since a new result set is 971 generated each time a request is submitted. Therefore, they are 972 preferred in comparison to any other method requiring the management 973 of a REST session. 975 Using limit and offset operators represents the traditionally used 976 method to implement results pagination. Both of them can be used 977 individually: 979 o "limit": means that the server MUST return the first N objects of 980 the result set; 982 o "offset": means that the server MUST skip the first N objects and 983 MUST return objects starting from position N+1. 985 When limit and offset are used together, they provide the ability to 986 identify a specific portion of the result set. For example, the pair 987 "offset=100,limit=50" returns the first 50 objects starting from 988 position 101 of the result set. 990 Though easy to implement, offset pagination also includes drawbacks: 992 o When offset has a very high value, scrolling the result set could 993 take some time; 995 o It always requires fetching all rows before dropping as many rows 996 as specified by offset; 998 o It may return inconsistent pages when data are frequently updated 999 (i.e. real-time data). 1001 Keyset pagination [SEEK] adds a query condition that enables the 1002 selection of the only data not yet returned. This method has been 1003 taken as the basis for the implementation of a "cursor" parameter 1004 [CURSOR] by some REST API providers (e.g. 1005 [CURSOR-API1],[CURSOR-API2]). The cursor is an opaque URL-safe 1006 string representing a logical pointer to the first result of the next 1007 page (Figure 5). 1009 Nevertheless, even keyset pagination can be troublesome: 1011 o It needs at least one key field; 1013 o It does not allow to sort just by any field because the sorting 1014 criterion must contain a key; 1016 o It works best with full composite values support by DBMS (i.e. 1017 [x,y]>[a,b]), emulation is possible but ugly and less performant; 1019 o It does not allow direct navigation to arbitrary pages because the 1020 result set must be scrolled in sequential order starting from the 1021 initial page; 1023 o Implementing bi-directional navigation is tedious because all 1024 comparison and sort operations have to be reversed. 1026 B.1. Specific Issues Raised by RDAP 1028 Furthermore, in the RDAP context, some additional considerations can 1029 be made: 1031 o An RDAP object is a conceptual aggregation of information 1032 generally collected from more than one data structure (e.g. table) 1033 and this makes it even harder to implement keyset pagination, a 1034 task that is already quite difficult. For example, the entity 1035 object can include information from different data structures 1036 (registrars, registrants, contacts, resellers, and so on), each 1037 one with its own key field mapping the RDAP entity handle; 1039 o Depending on the number of page results as well as the number and 1040 the complexity of the properties of each RDAP object in the 1041 response, the time required by offset pagination to skip the 1042 previous pages could be much faster than the processing time 1043 needed to build the current page. In fact, RDAP objects are 1044 usually formed by information belonging to multiple data 1045 structures and containing multivalued properties (i.e. arrays) 1046 and, therefore, data selection might be a time consuming process. 1047 This situation occurs even though the selection is supported by 1048 indexes; 1050 o Depending on the access levels defined by each RDAP operator, the 1051 increase of complexity and the decrease of flexibility of keyset 1052 pagination with respect to offset pagination could be considered 1053 impractical. 1055 Ultimately, both pagination methods have benefits and drawbacks. 1057 Appendix C. Change Log 1059 00: Initial working group version ported from draft-loffredo-regext- 1060 rdap-sorting-and-paging-05 1061 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1062 consistent between the pagination methods. Renamed 1063 "Considerations about Paging Implementation" section in ""cursor" 1064 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1065 detailed description of both "sorting_metadata" and 1066 "paging_metadata" objects. 1067 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1068 of cursor parameter. Rearranged the layout of some sections. 1069 Removed some items from "Informative References" section. Changed 1070 "IANA Considerations" section. 1071 03: Added "cc" to the list of sorting properties in "Sorting 1072 Properties Declaration" section. Added RFC8605 to the list of 1073 "Informative References". 1074 04: Replaced "ldhName" with "name" in the "Sorting Properties 1075 Declaration" section. Clarified the sorting logic with respect to 1076 the JSON value types and the sorting policy for multivalued 1077 fields. 1078 05: Clarified the logic of sorting on IP addresses. Clarified the 1079 mapping between the sorting properties and the RDAP fields. 1080 Updated "Acknowledgements" section. 1081 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1082 "paging_metadata" object. 1083 07: Added "Paging Responses to POST Requests" section. 1084 08: Added "Approaches to Result Pagination" section to appendix. 1085 Added the case of requesting a sort on a property not included in 1086 the response to the errors listed in the "Negative Answers" 1087 section. 1088 09: Updated the "Implementation Status" section to include APNIC 1089 implementation. Moved the "RDAP Conformance" section up in the 1090 document. Removed the "Paging Responses to POST Requests" 1091 section. Updated the "Acknowledgements" section. Removed unused 1092 references. In the "Sorting Properties Declaration" section: 1094 * clarified the logic of sorting on events; 1095 * corrected the JSONPath of the "lastChanged" sorting property; 1096 * provided a JSONPath example taking into account the vCard 1097 "pref" parameter. 1098 10: Corrected the JSONPaths of both "fn" and "org" sorting 1099 properties in Table 2. Corrected JSON content in Figure 4. Moved 1100 [W3C.CR-xpath-31-20161213] and [RFC7942] to the "Normative 1101 References". Changed the rdapConformance tags "sorting_level_0" 1102 and "paging_level_0" to "sorting" and "paging" respectively. 1103 11: Added the "JSONPath operators" section to appendix. 1104 12: Changed the content of "JSONPath operators" section. 1105 13: Minor pre-AD review edits. 1107 Authors' Addresses 1109 Mario Loffredo 1110 IIT-CNR/Registro.it 1111 Via Moruzzi,1 1112 Pisa 56124 1113 IT 1115 Email: mario.loffredo@iit.cnr.it 1116 URI: http://www.iit.cnr.it 1118 Maurizio Martinelli 1119 IIT-CNR/Registro.it 1120 Via Moruzzi,1 1121 Pisa 56124 1122 IT 1124 Email: maurizio.martinelli@iit.cnr.it 1125 URI: http://www.iit.cnr.it 1126 Scott Hollenbeck 1127 Verisign Labs 1128 12061 Bluemont Way 1129 Reston, VA 20190 1130 USA 1132 Email: shollenbeck@verisign.com 1133 URI: https://www.verisignlabs.com/