idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 14, 2020) is 1444 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 535 -- Looks like a reference, but probably isn't: '1' on line 536 -- Looks like a reference, but probably isn't: '3' on line 536 -- Looks like a reference, but probably isn't: '6' on line 512 == Unused Reference: 'RFC8605' is defined on line 854, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 4 errors (**), 0 flaws (~~), 2 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: October 16, 2020 S. Hollenbeck 6 Verisign Labs 7 April 14, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-10 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on October 16, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.1.1. RDAP Conformance . . . . . . . . . . . . . . . . . . 6 64 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 65 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 66 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 8 67 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 13 68 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 15 69 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 15 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 16 71 4. Implementation Considerations . . . . . . . . . . . . . . . . 17 72 5. Implementation Status . . . . . . . . . . . . . . . . . . . . 17 73 5.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 17 74 5.2. APNIC . . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 76 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 77 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 78 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 79 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 80 9.2. Informative References . . . . . . . . . . . . . . . . . 21 81 Appendix A. Approaches to Result Pagination . . . . . . . . . . 22 82 A.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 23 83 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 24 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 86 1. Introduction 88 The availability of functionality for result sorting and paging 89 provides benefits to both clients and servers in the implementation 90 of RESTful services [REST]. These benefits include: 92 o reducing the server response bandwidth requirements; 93 o improving server response time; 94 o improving query precision and, consequently, obtaining more 95 reliable results; 96 o decreasing server query processing load; 97 o reducing client response processing time. 99 Approaches to implementing features for result sorting and paging can 100 be grouped into two main categories: 102 1. Sorting and paging are implemented through the introduction of 103 additional parameters in the query string (i.e. ODATA protocol 104 [OData-Part1]); 106 2. Information related to the number of results and the specific 107 portion of the result set to be returned, in addition to a set of 108 ready-made links for the result set scrolling, are inserted in 109 the HTTP header of the request/response. 111 However, there are some drawbacks associated with the use of the HTTP 112 header. First, the header properties cannot be set directly from a 113 web browser. Moreover, in an HTTP session, the information on the 114 status (i.e. the session identifier) is usually inserted in the 115 header or in the cookies, while the information on the resource 116 identification or the search type is included in the query string. 117 The second approach is therefore not compliant with the HTTP standard 118 [RFC7230]. As a result, this document describes a specification 119 based on the use of query parameters. 121 Currently, the RDAP protocol [RFC7482] defines two query types: 123 o lookup: the server returns only one object; 124 o search: the server returns a collection of objects. 126 While the lookup query does not raise issues in the response 127 management, the search query can potentially generate a large result 128 set that could be truncated according to the server limits. In 129 addition, it is not possible to obtain the total number of the 130 objects found that might be returned in a search query response 131 [RFC7483]. Lastly, there is no way to specify sort criteria to 132 return the most relevant objects at the beginning of the result set. 133 Therefore, the client might traverse the whole result set to find the 134 relevant objects or, due to truncation, could not find them at all. 136 The specification described in this document extends RDAP query 137 capabilities to enable result sorting and paging, by adding new query 138 parameters that can be applied to RDAP search path segments. The 139 service is implemented using the Hypertext Transfer Protocol (HTTP) 140 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 142 The implementation of the new parameters is technically feasible, as 143 operators for counting, sorting and paging rows are currently 144 supported by the major RDBMSs. 146 1.1. Conventions Used in This Document 148 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 149 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 150 document are to be interpreted as described in [RFC2119]. 152 2. RDAP Query Parameter Specification 154 The new query parameters are OPTIONAL extensions of path segments 155 defined in RFC 7482 [RFC7482]. They are as follows: 157 o "count": a boolean value that allows a client to request the total 158 number of objects found (that due to truncation can be different 159 from the number of returned objects); 161 o "sort": a string value that allows a client to request a specific 162 sort order for the result set; 164 o "cursor": a string value representing a pointer to a specific 165 fixed size portion of the result set. 167 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 168 sections to describe the formal syntax of these new parameters. 170 2.1. Sorting and Paging Metadata 172 According to most advanced principles in REST design, collectively 173 known as HATEOAS (Hypermedia as the Engine of Application State) 174 ([HATEOAS]), a client entering a REST application through an initial 175 URI should use the server-provided links to dynamically discover 176 available actions and access the resources it needs. In this way, 177 the client is not requested to have prior knowledge of the service 178 and, consequently, to hard code the URIs of different resources. 179 This would allow the server to make URI changes as the API evolves 180 without breaking the clients. Definitively, a REST service should be 181 as self-descriptive as possible. 183 Therefore, servers implementing the query parameters described in 184 this specification SHOULD provide additional information in their 185 responses about both the available sorting criteria and the possible 186 pagination. Such information is collected in two OPTIONAL response 187 elements named, respectively, "sorting_metadata" and 188 "paging_metadata". 190 The "sorting_metadata" element contains the following properties: 192 o "currentSort": "String" (OPTIONAL) either the value of sort 193 "parameter" as specified in the query string or the sort applied 194 by default, if any; 196 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of objects 197 each one describing an alternate available sorting criterion. 198 Members are: 200 * "property": "String" (REQUIRED) the name that can be used by 201 the client to request the sorting criterion; 202 * "default": "Boolean" (REQUIRED) whether the sorting criterion 203 is applied by default; 204 * "jsonPath": "String" (OPTIONAL) the JSON Path of the RDAP field 205 corresponding to the property; 206 * "links": "Link[]" (OPTIONAL) an array of links as described in 207 RFC 8288 [RFC8288] containing the query string that applies the 208 sorting criterion. 210 At least one between "currentSort" and "availableSorts" MUST be 211 present. 213 The "paging_metadata" element contains the following fields: 215 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 216 the total number of objects found. It MUST be provided if the 217 query string contains the "count" parameter; 219 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 220 number of objects returned in the current page. It MUST be 221 provided when the total number of objects exceeds the page size. 222 This property is redundant for clients because the page size can 223 be derived from the length of the search results array but it can 224 be helpful if the end user interacts with the server through a web 225 browser; 227 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 228 the number of the current page in the result set. It MUST be 229 provided when the total number of objects found exceeds the page 230 size; 232 o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 233 8288 [RFC8288] containing the reference to the next page. In this 234 specification, only the forward pagination is dealt because it is 235 considered satisfactory in order to traverse the result set. 236 Examples of additional references are to: the previous page, the 237 first page, the last page. 239 2.1.1. RDAP Conformance 241 Servers returning the "paging_metadata" element in their response 242 MUST include "paging" in the rdapConformance array as well as servers 243 returning the "sorting_metadata" element MUST include "sorting". 245 2.2. "count" Parameter 247 Currently, the RDAP protocol does not allow a client to determine the 248 total number of the results in a query response when the result set 249 is truncated. This is rather inefficient because the user cannot 250 evaluate the query precision and, at the same time, cannot receive 251 information that could be relevant. 253 The "count" parameter provides additional functionality (Figure 1) 254 that allows a client to request information from the server that 255 specifies the total number of objects matching the search pattern. 257 https://example.com/rdap/domains?name=*nr.com&count=true 259 Figure 1: Example of RDAP query reporting the "count" parameter 261 The ABNF syntax is the following: 263 count = "count=" ( trueValue / falseValue ) 264 trueValue = ("true" / "yes" / "1") 265 falseValue = ("false" / "no" / "0") 267 A trueValue means that the server MUST provide the total number of 268 the objects in the "totalCount" field of the "paging_metadata" 269 element (Figure 2). A falseValue means that the server MUST NOT 270 provide this number. 272 { 273 "rdapConformance": [ 274 "rdap_level_0", 275 "paging" 276 ], 277 ... 278 "paging_metadata": { 279 "totalCount": 43 280 }, 281 "domainSearchResults": [ 282 ... 283 ] 284 } 286 Figure 2: Example of RDAP response with "paging_metadata" element 287 containing the "totalCount" field 289 2.3. "sort" Parameter 291 The RDAP protocol does not provide any capability to specify results 292 sort criteria. A server could implement a default sorting scheme 293 according to the object class, but this feature is not mandatory and 294 might not meet user requirements. Sorting can be addressed by the 295 client, but this solution is rather inefficient. Sorting features 296 provided by the RDAP server could help avoid truncation of relevant 297 results. 299 The "sort" parameter allows the client to ask the server to sort the 300 results according to the values of one or more properties and 301 according to the sort direction of each property. The ABNF syntax is 302 the following: 304 sort = "sort=" sortItem *( "," sortItem ) 305 sortItem = property-ref [":" ( "a" / "d" ) ] 306 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 308 "a" means that the ascending sort MUST be applied, "d" means that the 309 descending sort MUST be applied. If the sort direction is absent, an 310 ascending sort MUST be applied (Figure 3). 312 https://example.com/rdap/domains?name=*nr.com&sort=name 314 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 316 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 318 Figure 3: Examples of RDAP query reporting the "sort" parameter 320 With the only exception of the sort on IP addresses, servers MUST 321 implement sorting according to the JSON value type of the RDAP field 322 the sorting property refers to: JSON strings MUST be sorted 323 lexicographically and JSON numbers MUST be sorted numerically. Even 324 if IP addresses are represented as JSON strings, they MUST be sorted 325 based on their numeric conversion. 327 If the "sort" parameter reports an allowed sorting property, it MUST 328 be provided in the "currentSort" field of the "sorting_metadata" 329 element. 331 2.3.1. Sorting Properties Declaration 333 In the "sort" parameter ABNF syntax, property-ref represents a 334 reference to a property of an RDAP object. Such a reference could be 335 expressed by using a JSON Path. The JSON Path in a JSON document 336 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 337 XML document. For example, the JSON Path to select the value of the 338 ASCII name inside an RDAP domain object is "$.ldhName", whereby $ 339 identifies the root of the document (DOM). Another way to select a 340 value inside a JSON document is the JSON Pointer [RFC6901]. While 341 JSON Path or JSON Pointer are both standard ways to select any value 342 inside JSON data, neither is particularly easy to use (e.g. 343 "$.events[?(@.eventAction='registration')].eventDate" is the JSON 344 Path expression of the registration date in an RDAP domain object). 346 Therefore, this specification provides a definition of property-ref 347 in terms of RDAP properties. However, not all the RDAP properties 348 are suitable to be used in sort criteria, such as: 350 o properties providing service information (e.g. links, notices, 351 remarks, etc.); 353 o multivalued properties (e.g. status, roles, variants, etc.); 355 o properties modeling relationships to other objects (e.g. 356 entities). 358 On the contrary, some properties expressed as values of other 359 properties (e.g. registration date) could be used in such a context. 361 In the following, a list of properties an RDAP server MAY implement 362 is presented. The properties are divided into two groups: object 363 common properties and object specific properties. 365 o Object common properties. Object common properties are derived 366 from the merge of the "eventAction" and the "eventDate" 367 properties. The following values of the "sort" parameter are 368 defined: 370 * registrationDate 371 * reregistrationDate 372 * lastChangedDate 373 * expirationDate 374 * deletionDate 375 * reinstantiationDate 376 * transferDate 377 * lockedDate 378 * unlockedDate 380 o Object specific properties. With regard to the specific 381 properties, some of them are already defined among the query 382 paths. In the following a list of possible sorting properties, 383 grouped by objects, is shown: 385 * Domain: name 386 * Nameserver: name, ipV4, ipV6. 387 * Entity: fn, handle, org, email, voice, country, cc, city. 389 The correspondence between the sorting properties and the RDAP fields 390 is shown in Table 1: 392 +-----------+-----------+---------------------+------+-------+------+ 393 | Object | Sorting | RDAP property | RFC | RFC | RFC | 394 | class | property | | 7483 | 6350 | 8605 | 395 +-----------+-----------+---------------------+------+-------+------+ 396 | Searchabl | Common pr | eventAction values | 4.5. | | | 397 | e objects | operties | suffixed by "Date" | | | | 398 | | | | | | | 399 | Domain | name | unicodeName/ldhName | 5.3. | | | 400 | | | | | | | 401 | Nameserve | name | unicodeName/ldhName | 5.2. | | | 402 | r | | | | | | 403 | | ipV4 | v4 ipAddress | 5.2. | | | 404 | | ipV6 | v6 ipAddress | 5.2. | | | 405 | | | | | | | 406 | Entity | handle | handle | 5.1. | | | 407 | | fn | vcard fn | 5.1. | 6.2.1 | | 408 | | org | vcard org | 5.1. | 6.6.4 | | 409 | | voice | vcard tel with | 5.1. | 6.4.1 | | 410 | | | type="voice" | | | | 411 | | email | vcard email | 5.1. | 6.4.2 | | 412 | | country | country name in | 5.1. | 6.3.1 | | 413 | | | vcard adr | | | | 414 | | cc | country code in | 5.1. | | 3.1 | 415 | | | vcard adr | | | | 416 | | city | locality in vcard | 5.1. | 6.3.1 | | 417 | | | adr | | | | 418 +-----------+-----------+---------------------+------+-------+------+ 420 Table 1: Sorting properties definition 422 With regard to the definitions in Table 1, some further 423 considerations must be made to disambiguate some cases: 425 o since the response to a search on either domains or nameservers 426 might include both A-labels and U-labels ([RFC5890]) in general, a 427 consistent sorting policy shall take unicodeName and ldhName as 428 two formats of the same value rather than separately. Therefore, 429 the unicodeName value MUST be taken while sorting, when 430 unicodeName is missing, the value of ldhName MUST be considered 431 instead; 433 o the jCard "sort-as" parameter MUST be ignored for the purpose of 434 the sorting capability as described in this document; 436 o even if a nameserver can have multiple IPv4 and IPv6 addresses, 437 the most common configuration includes one address for each IP 438 version. Therefore, the assumption of having a single IPv4 and/or 439 IPv6 value for a nameserver cannot be considered too stringent. 441 When more than one address per IP version is reported, sorting 442 MUST be applied to the first value; 444 o multiple events with a given action on an object might be 445 returned. When it occurs, sorting MUST be applied to the most 446 recent event; 448 o with the exception of handle values, all the sorting properties 449 defined for entity objects can be multivalued according to the 450 definition of vCard as given in RFC 6350 [RFC6350]. When more 451 than one value is reported, sorting MUST be applied to the 452 preferred value identified by the parameter pref="1". If the pref 453 parameter is missing, sorting MUST be applied to the first value. 455 Each RDAP provider MAY define other sorting properties than those 456 shown in this document as well as it MAY map those sorting properties 457 onto different locations. 459 The "jsonPath" field in the "sorting_metadata" element is used to 460 clarify the RDAP field the sorting property refers to. The mapping 461 between the sorting properties and the JSON Paths of the RDAP fields 462 is shown in Table 2. The JSON Paths are provided according to the 463 Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]). 465 +-------+-------------+---------------------------------------------+ 466 | Objec | Sorting | JSON Path | 467 | t | property | | 468 | class | | | 469 +-------+-------------+---------------------------------------------+ 470 | Searc | registratio | $.domainSearchResults[*].events[?(@.eventAc | 471 | hable | nDate | tion=="registration")].eventDate | 472 | objec | | | 473 | ts | | | 474 | | reregistrat | $.domainSearchResults[*].events[?(@.eventAc | 475 | | ionDate | tion=="reregistration")].eventDate | 476 | | lastChanged | $.domainSearchResults[*].events[?(@.eventAc | 477 | | Date | tion=="last changed")].eventDate | 478 | | expirationD | $.domainSearchResults[*].events[?(@.eventAc | 479 | | ate | tion=="expiration")].eventDate | 480 | | deletionDat | $.domainSearchResults[*].events[?(@.eventAc | 481 | | e | tion=="deletion")].eventDate | 482 | | reinstantia | $.domainSearchResults[*].events[?(@.eventAc | 483 | | tionDate | tion=="reinstantiation")].eventDate | 484 | | transferDat | $.domainSearchResults[*].events[?(@.eventAc | 485 | | e | tion=="transfer")].eventDate | 486 | | lockedDate | $.domainSearchResults[*].events[?(@.eventAc | 487 | | | tion=="locked")].eventDate | 488 | | unlockedDat | $.domainSearchResults[*].events[?(@.eventAc | 489 | | e | tion=="unlocked")].eventDate | 490 | | | | 491 | Domai | name | $.domainSearchResults[*].unicodeName | 492 | n | | | 493 | | | | 494 | Names | name | $.nameserverSearchResults[*].unicodeName | 495 | erver | | | 496 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 497 | | | [0] | 498 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 499 | | | [0] | 500 | | | | 501 | Entit | handle | $.entitySearchResults[*].handle | 502 | y | | | 503 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 504 | | | 0]=="fn")][3] | 505 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 506 | | | 0]=="org")][3] | 507 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 508 | | | 0]=="tel" && @[1].type=="voice")][3] | 509 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 510 | | | 0]=="email")][3] | 511 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 512 | | | 0]=="adr")][3][6] | 513 | | cc | $.entitySearchResults[*].vcardArray[1][?(@[ | 514 | | | 0]=="adr")][1].cc | 515 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 516 | | | 0]=="adr")][3][3] | 517 +-------+-------------+---------------------------------------------+ 519 Table 2: Sorting properties - JSON Path Mapping 521 Note about the JSON Paths of Table 2 that: 523 o those related to the event dates are defined only for the "domain" 524 object. To obtain the equivalent JSON Paths for "entity" and 525 "nameserver", the path segment "domainSearchResults" must be 526 replaced with "entitySearchResults" and "nameserverSearchResults" 527 respectively; 529 o those related to vCard elements are specified without taking into 530 account the "pref" parameter. Servers always applying sorting to 531 those values identified by the pref parameter SHOULD update a JSON 532 Path by adding an appropriate filter. For example, if the email 533 values identified by pref="1" are considered for sorting, the JSON 534 Path of the "email" sorting property should be: 535 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" && 536 @[1].pref=="1")][3] 538 2.3.2. Representing Sorting Links 540 An RDAP server MAY use the "links" array of the "sorting_metadata" 541 element to provide ready-made references [RFC8288] to the available 542 sort criteria (Figure 4). Each link represents a reference to an 543 alternate view of the results. 545 { 546 "rdapConformance": [ 547 "rdap_level_0", 548 "sorting" 549 ], 550 ... 551 "sorting_metadata": { 552 "currentSort": "name", 553 "availableSorts": [ 554 { 555 "property": "registrationDate", 556 "jsonPath": "$.domainSearchResults[*] 557 .events[?(@.eventAction==\"registration\")].eventDate", 558 "default": false, 559 "links": [ 560 { 561 "value": "https://example.com/rdap/domains?name=*nr.com 562 &sort=name", 563 "rel": "alternate", 564 "href": "https://example.com/rdap/domains?name=*nr.com 565 &sort=registrationDate", 566 "title": "Result Ascending Sort Link", 567 "type": "application/rdap+json" 568 }, 569 { 570 "value": "https://example.com/rdap/domains?name=*nr.com 571 &sort=name", 572 "rel": "alternate", 573 "href": "https://example.com/rdap/domains?name=*nr.com 574 &sort=registrationDate:d", 575 "title": "Result Descending Sort Link", 576 "type": "application/rdap+json" 577 } 578 ] 579 }, 580 ... 581 ] 582 }, 583 "domainSearchResults": [ 584 ... 585 ] 586 } 588 Figure 4: Example of a "sorting_metadata" instance to implement 589 result sorting 591 2.4. "cursor" Parameter 593 The cursor parameter defined in this specification can be used to 594 encode information about any pagination method. For example, in the 595 case of a simple implementation of the cursor parameter to represent 596 offset pagination information, the cursor value 597 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the mere Base64 encoding of 598 "offset=100,limit=50". Likewise, in a simple implementation to 599 represent keyset pagination information, the cursor value 600 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the mere 601 Base64 encoding of "key=thelastdomainofthepage.com" whereby the key 602 value identifies the last row of the current page. 604 This solution lets RDAP providers to implement a pagination method 605 according to their needs, the user access levels, the submitted 606 queries. In addition, servers can change the method over time 607 without announcing anything to the clients. The considerations that 608 has led to this solution are reported in more detail in Appendix A. 610 The ABNF syntax of the cursor paramter is the following: 612 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 614 https://example.com/rdap/domains?name=*nr.com 615 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 617 Figure 5: An example of RDAP query reporting the "cursor" parameter 619 2.4.1. Representing Paging Links 621 An RDAP server SHOULD use the "links" array of the "paging_metadata" 622 element to provide a ready-made reference [RFC8288] to the next page 623 of the result set (Figure 6). Examples of additional "rel" values a 624 server MAY implements are "first", "last", "prev". 626 { 627 "rdapConformance": [ 628 "rdap_level_0", 629 "paging" 630 ], 631 ... 632 "notices": [ 633 { 634 "title": "Search query limits", 635 "type": "result set truncated due to excessive load", 636 "description": [ 637 "search results for domains are limited to 50" 638 ] 639 } 640 ], 641 "paging_metadata": { 642 "totalCount": 73, 643 "pageSize": 50, 644 "pageNumber": 1, 645 "links": [ 646 { 647 "value": "https://example.com/rdap/domains?name=*nr.com", 648 "rel": "next", 649 "href": "https://example.com/rdap/domains?name=*nr.com 650 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 651 "title": "Result Pagination Link", 652 "type": "application/rdap+json" 653 } 654 ] 655 }, 656 "domainSearchResults": [ 657 ... 658 ] 659 } 661 Figure 6: Example of a "paging_metadata" instance to implement cursor 662 pagination 664 3. Negative Answers 666 The value constraints for the parameters are defined by their ABNF 667 syntax. Therefore, each request including an invalid value for a 668 parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The 669 same response SHOULD be returned in the following cases: 671 o if in both single and multi sort the client provides an 672 unsupported value for the "sort" parameter as well as a value 673 related to an object property not included in the response; 675 o if the client submits an invalid value for the "cursor" parameter. 677 Optionally, the response MAY include additional information regarding 678 the negative answer in the HTTP entity body. 680 4. Implementation Considerations 682 The implementation of the new parameters is technically feasible, as 683 operators for counting, sorting and paging are currently supported by 684 the major RDBMSs. 686 Similar operators are completely or partially supported by the most 687 known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so 688 the implementation of the new parameters seems to be practicable by 689 servers working without the use of an RDBMS. 691 5. Implementation Status 693 NOTE: Please remove this section and the reference to RFC 7942 prior 694 to publication as an RFC. 696 This section records the status of known implementations of the 697 protocol defined by this specification at the time of posting of this 698 Internet-Draft, and is based on a proposal described in RFC 7942 699 [RFC7942]. The description of implementations in this section is 700 intended to assist the IETF in its decision processes in progressing 701 drafts to RFCs. Please note that the listing of any individual 702 implementation here does not imply endorsement by the IETF. 703 Furthermore, no effort has been spent to verify the information 704 presented here that was supplied by IETF contributors. This is not 705 intended as, and must not be construed to be, a catalog of available 706 implementations or their features. Readers are advised to note that 707 other implementations may exist. 709 According to RFC 7942, "this will allow reviewers and working groups 710 to assign due consideration to documents that have the benefit of 711 running code, which may serve as evidence of valuable experimentation 712 and feedback that have made the implemented protocols more mature. 713 It is up to the individual working groups to use this information as 714 they see fit". 716 5.1. IIT-CNR/Registro.it 718 Responsible Organization: Institute of Informatics and Telematics 719 of National Research Council (IIT-CNR)/Registro.it 720 Location: https://rdap.pubtest.nic.it/ 721 Description: This implementation includes support for RDAP queries 722 using data from .it public test environment. 724 Level of Maturity: This is an "alpha" test implementation. 725 Coverage: This implementation includes all of the features 726 described in this specification. 727 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 729 5.2. APNIC 731 Responsible Organization: Asia-Pacific Network Information Centre 732 Location: https://github.com/APNIC-net/rdap-rmp-demo/tree/sorting- 733 and-paging 734 Description: A proof-of-concept for RDAP mirroring. 735 Level of Maturity: This is a proof-of-concept implementation. 736 Coverage: This implementation includes all of the features 737 described in the specification except for nameserver sorting and 738 unicodeName sorting. 739 Contact Information: Tom Harrison, tomh@apnic.net 741 6. IANA Considerations 743 IANA is requested to register the following values in the RDAP 744 Extensions Registry: 746 Extension identifier: paging 747 Registry operator: Any 748 Published specification: This document. 749 Contact: IESG 750 Intended usage: This extension describes a best practice for 751 result set paging. 753 Extension identifier: sorting 754 Registry operator: Any 755 Published specification: This document. 756 Contact: IESG 757 Intended usage: This extension describes a best practice for 758 result set sorting. 760 7. Security Considerations 762 Security services for the operations specified in this document are 763 described in RFC 7481 [RFC7481]. 765 The search query typically requires more server resources (such as 766 memory, CPU cycles, and network bandwidth) when compared to the 767 lookup query. This increases the risk of server resource exhaustion 768 and subsequent denial of service due to abuse. This risk can be 769 mitigated by either restricting search functionality or limiting the 770 rate of search requests. Servers can also reduce their load by 771 truncating the results in the response. However, this last security 772 policy can result in a higher inefficiency if the RDAP server does 773 not provide any functionality to return the truncated results. 775 The new parameters presented in this document provide the RDAP 776 operators with a way to implement a secure server without penalizing 777 its efficiency. The "count" parameter gives the user a measure to 778 evaluate the query precision and, at the same time, returns a 779 significant information. The "sort" parameter allows the user to 780 obtain the most relevant information at the beginning of the result 781 set. In both cases, the user doesn't need to submit further 782 unnecessary search requests. Finally, the "cursor" parameter enables 783 the user to scroll the result set by submitting a sequence of 784 sustainable queries according to the server limits. 786 8. Acknowledgements 788 The authors would like to acknowledge Brian Mountford, Tom Harrison, 789 Karl Heinz Wolf and Jasdip Singh for their contribution to the 790 development of this document. 792 9. References 794 9.1. Normative References 796 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 797 Requirement Levels", BCP 14, RFC 2119, 798 DOI 10.17487/RFC2119, March 1997, 799 . 801 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 802 Specifications: ABNF", STD 68, RFC 5234, 803 DOI 10.17487/RFC5234, January 2008, 804 . 806 [RFC5890] Klensin, J., "Internationalized Domain Names for 807 Applications (IDNA): Definitions and Document Framework", 808 RFC 5890, DOI 10.17487/RFC5890, August 2010, 809 . 811 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 812 DOI 10.17487/RFC6350, August 2011, 813 . 815 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 816 Protocol (HTTP/1.1): Message Syntax and Routing", 817 RFC 7230, DOI 10.17487/RFC7230, June 2014, 818 . 820 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 821 Registration Data Access Protocol (RDAP)", RFC 7480, 822 DOI 10.17487/RFC7480, March 2015, 823 . 825 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 826 Registration Data Access Protocol (RDAP)", RFC 7481, 827 DOI 10.17487/RFC7481, March 2015, 828 . 830 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 831 Protocol (RDAP) Query Format", RFC 7482, 832 DOI 10.17487/RFC7482, March 2015, 833 . 835 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 836 Registration Data Access Protocol (RDAP)", RFC 7483, 837 DOI 10.17487/RFC7483, March 2015, 838 . 840 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 841 Code: The Implementation Status Section", BCP 205, 842 RFC 7942, DOI 10.17487/RFC7942, July 2016, 843 . 845 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 846 Interchange Format", STD 90, RFC 8259, 847 DOI 10.17487/RFC8259, December 2017, 848 . 850 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 851 DOI 10.17487/RFC8288, October 2017, 852 . 854 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 855 ICANN Extensions for the Registration Data Access Protocol 856 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 857 . 859 [W3C.CR-xpath-31-20161213] 860 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 861 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 862 31-20161213, December 2016, 863 . 865 9.2. Informative References 867 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 868 Pagination", July 2014, . 871 [CURSOR-API1] 872 facebook.com, "facebook for developers - Using the Graph 873 API", July 2017, . 876 [CURSOR-API2] 877 twitter.com, "Pagination", 2017, 878 . 881 [GOESSNER-JSON-PATH] 882 Goessner, S., "JSONPath - XPath for JSON", 2007, 883 . 885 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 886 . 889 [OData-Part1] 890 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 891 Part 1: Protocol Plus Errata 03", June 2016, 892 . 897 [REST] Fredrich, T., "RESTful Service Best Practices, 898 Recommendations for Creating Web Services", April 2012, 899 . 902 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 903 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 904 DOI 10.17487/RFC6901, April 2013, 905 . 907 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 908 With Limit and Offset is Slow?", July 2017, 909 . 912 Appendix A. Approaches to Result Pagination 914 An RDAP query could return a response with hundreds, even thousands, 915 of objects, especially when partial matching is used. For that 916 reason, the cursor parameter addressing result pagination is defined 917 to make responses easier to handle. 919 Presently, the most popular methods to implement pagination in REST 920 API are: offset pagination and keyset pagination. Both two 921 pagination methods don't require the server to handle the result set 922 in a storage area across the requests since a new result set is 923 generated each time a request is submitted. Therefore, they are 924 preferred in comparison to any other method requiring the management 925 of a REST session. 927 Using limit and offset operators represents the traditionally used 928 method to implement results pagination. Both of them can be used 929 individually: 931 o "limit": means that the server must return the first N objects of 932 the result set; 934 o "offset": means that the server must skip the first N objects and 935 must return objects starting from position N+1. 937 When limit and offset are used together, they allow to identify a 938 specific portion of the result set. For example, the pair 939 "offset=100,limit=50" returns first 50 objects starting from position 940 101 of the result set. 942 Despite its easiness of implementation, offset pagination raises some 943 well known drawbacks: 945 o when offset has a very high value, scrolling the result set could 946 take some time; 948 o it always requires to fetch all the rows before dropping as many 949 rows as specified by offset; 951 o it may return inconsistent pages when data are frequently updated 952 (i.e. real-time data) but this doesn't seem the case of 953 registration data. 955 The keyset pagination [SEEK] consists in adding a query condition 956 that enables the selection of the only data not yet returned. This 957 method has been taken as the basis for the implementation of a 958 "cursor" parameter [CURSOR] by some REST API providers (e.g. 959 [CURSOR-API1],[CURSOR-API2]). The cursor is an opaque URL-safe 960 string representing a logical pointer to the first result of the next 961 page (Figure 5). 963 Nevertheless, even keyset pagination can be troublesome: 965 o it needs at least one key field; 967 o it does not allow to sort just by any field because the sorting 968 criterion must contain a key; 970 o it works best with full composite values support by DBMS (i.e. 971 [x,y]>[a,b]), emulation is possible but ugly and less performant; 973 o it does not allow to directly navigate to arbitrary pages because 974 the result set must be scrolled in sequential order starting from 975 the initial page; 977 o implementing the bi-directional navigation is tedious because all 978 comparison and sort operations have to be reversed. 980 A.1. Specific Issues Raised by RDAP 982 Furthermore, in the RDAP context, some additional considerations can 983 be made: 985 o an RDAP object is a conceptual aggregation of information 986 generally collected from more than one data structure (e.g. table) 987 and this makes even harder for the developers the implementation 988 of the keyset pagination that is already quite difficult. For 989 example, the entity object can gather information from different 990 data structures (registrars, registrants, contacts, resellers, and 991 so on), each one with its own key field mapping the RDAP entity 992 handle; 994 o depending on the number of the page results as well as the number 995 and the complexity of the properties of each RDAP object in the 996 response, the time required by offset pagination to skip the 997 previous pages could be much faster than the processing time 998 needed to build the current page. In fact, RDAP objects are 999 usually formed by information belonging to multiple data 1000 structures and containing multivalued properties (i.e. arrays) 1001 and, therefore, data selection might be a time consuming process. 1002 This situation occurs even though the selection is supported by 1003 indexes; 1005 o depending on the access levels defined by each RDAP operator, the 1006 increase of complexity and the decrease of flexibility of keyset 1007 pagination with respect to the offset pagination could be 1008 considered impractical. 1010 Ultimately, both pagination methods have benefits and drawbacks. 1012 Appendix B. Change Log 1014 00: Initial working group version ported from draft-loffredo-regext- 1015 rdap-sorting-and-paging-05 1016 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1017 consistent between the pagination methods. Renamed 1018 "Considerations about Paging Implementation" section in ""cursor" 1019 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1020 detailed description of both "sorting_metadata" and 1021 "paging_metadata" objects. 1022 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1023 of cursor parameter. Rearranged the layout of some sections. 1024 Removed some items from "Informative References" section. Changed 1025 "IANA Considerations" section. 1026 03: Added "cc" to the list of sorting properties in "Sorting 1027 Properties Declaration" section. Added RFC8605 to the list of 1028 "Informative References". 1029 04: Replaced "ldhName" with "name" in the "Sorting Properties 1030 Declaration" section. Clarified the sorting logic with respect to 1031 the JSON value types and the sorting policy for multivalued 1032 fields. 1033 05: Clarified the logic of sorting on IP addresses. Clarified the 1034 mapping between the sorting properties and the RDAP fields. 1035 Updated "Acknowledgements" section. 1036 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1037 "paging_metadata" object. 1038 07: Added "Paging Responses to POST Requests" section. 1039 08: Added "Approaches to Result Pagination" section to the appendix. 1040 Added the case of requesting a sort on a property not included in 1041 the response to the errors listed in the "Negative Answers" 1042 section. 1043 09: Updated the "Implementation Status" section to include APNIC 1044 implementation. Moved the "RDAP Conformance" section up in the 1045 document. Removed the "Paging Responses to POST Requests" 1046 section. Updated the "Acknowledgements" section. Removed unused 1047 references. In the "Sorting Properties Declaration" section: 1049 * clarified the logic of sorting on events; 1050 * corrected the JSON Path of the "lastChanged" sorting property; 1051 * provided a JSON Path example taking into account the vCard 1052 "pref" parameter. 1053 10: Corrected the JSON Paths of both "fn" and "org" sorting 1054 properties in Table 2. Corrected JSON content in Figure 4. Moved 1056 [W3C.CR-xpath-31-20161213] and [RFC7942] to the "Normative 1057 References". Changed the rdapConformance tags "sorting_level_0" 1058 and "paging_level_0" to "sorting" and "paging" respectively. 1060 Authors' Addresses 1062 Mario Loffredo 1063 IIT-CNR/Registro.it 1064 Via Moruzzi,1 1065 Pisa 56124 1066 IT 1068 Email: mario.loffredo@iit.cnr.it 1069 URI: http://www.iit.cnr.it 1071 Maurizio Martinelli 1072 IIT-CNR/Registro.it 1073 Via Moruzzi,1 1074 Pisa 56124 1075 IT 1077 Email: maurizio.martinelli@iit.cnr.it 1078 URI: http://www.iit.cnr.it 1080 Scott Hollenbeck 1081 Verisign Labs 1082 12061 Bluemont Way 1083 Reston, VA 20190 1084 USA 1086 Email: shollenbeck@verisign.com 1087 URI: https://www.verisignlabs.com/