idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 17, 2020) is 1468 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 540 -- Looks like a reference, but probably isn't: '1' on line 541 -- Looks like a reference, but probably isn't: '3' on line 541 -- Looks like a reference, but probably isn't: '6' on line 516 == Unused Reference: 'RFC8605' is defined on line 859, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 4 errors (**), 0 flaws (~~), 2 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: October 19, 2020 S. Hollenbeck 6 Verisign Labs 7 April 17, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-12 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on October 19, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.1.1. RDAP Conformance . . . . . . . . . . . . . . . . . . 6 64 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 65 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 66 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 8 67 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 13 68 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 15 69 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 15 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 16 71 4. Implementation Considerations . . . . . . . . . . . . . . . . 17 72 5. Implementation Status . . . . . . . . . . . . . . . . . . . . 17 73 5.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 17 74 5.2. APNIC . . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 76 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 77 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 78 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 79 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 80 9.2. Informative References . . . . . . . . . . . . . . . . . 21 81 Appendix A. JSONPath operators . . . . . . . . . . . . . . . . . 22 82 Appendix B. Approaches to Result Pagination . . . . . . . . . . 23 83 B.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 24 84 Appendix C. Change Log . . . . . . . . . . . . . . . . . . . . . 25 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 87 1. Introduction 89 The availability of functionality for result sorting and paging 90 provides benefits to both clients and servers in the implementation 91 of RESTful services [REST]. These benefits include: 93 o reducing the server response bandwidth requirements; 94 o improving server response time; 95 o improving query precision and, consequently, obtaining more 96 reliable results; 98 o decreasing server query processing load; 99 o reducing client response processing time. 101 Approaches to implementing features for result sorting and paging can 102 be grouped into two main categories: 104 1. Sorting and paging are implemented through the introduction of 105 additional parameters in the query string (i.e. ODATA protocol 106 [OData-Part1]); 108 2. Information related to the number of results and the specific 109 portion of the result set to be returned, in addition to a set of 110 ready-made links for the result set scrolling, are inserted in 111 the HTTP header of the request/response. 113 However, there are some drawbacks associated with the use of the HTTP 114 header. First, the header properties cannot be set directly from a 115 web browser. Moreover, in an HTTP session, the information on the 116 status (i.e. the session identifier) is usually inserted in the 117 header or in the cookies, while the information on the resource 118 identification or the search type is included in the query string. 119 The second approach is therefore not compliant with the HTTP standard 120 [RFC7230]. As a result, this document describes a specification 121 based on the use of query parameters. 123 Currently, the RDAP protocol [RFC7482] defines two query types: 125 o lookup: the server returns only one object; 126 o search: the server returns a collection of objects. 128 While the lookup query does not raise issues in the response 129 management, the search query can potentially generate a large result 130 set that could be truncated according to the server limits. In 131 addition, it is not possible to obtain the total number of the 132 objects found that might be returned in a search query response 133 [RFC7483]. Lastly, there is no way to specify sort criteria to 134 return the most relevant objects at the beginning of the result set. 135 Therefore, the client might traverse the whole result set to find the 136 relevant objects or, due to truncation, could not find them at all. 138 The specification described in this document extends RDAP query 139 capabilities to enable result sorting and paging, by adding new query 140 parameters that can be applied to RDAP search path segments. The 141 service is implemented using the Hypertext Transfer Protocol (HTTP) 142 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 144 The implementation of the new parameters is technically feasible, as 145 operators for counting, sorting and paging rows are currently 146 supported by the major RDBMSs. 148 1.1. Conventions Used in This Document 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in [RFC2119]. 154 2. RDAP Query Parameter Specification 156 The new query parameters are OPTIONAL extensions of path segments 157 defined in RFC 7482 [RFC7482]. They are as follows: 159 o "count": a boolean value that allows a client to request the total 160 number of objects found (that due to truncation can be different 161 from the number of returned objects); 163 o "sort": a string value that allows a client to request a specific 164 sort order for the result set; 166 o "cursor": a string value representing a pointer to a specific 167 fixed size portion of the result set. 169 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 170 sections to describe the formal syntax of these new parameters. 172 2.1. Sorting and Paging Metadata 174 According to most advanced principles in REST design, collectively 175 known as HATEOAS (Hypermedia as the Engine of Application State) 176 ([HATEOAS]), a client entering a REST application through an initial 177 URI should use the server-provided links to dynamically discover 178 available actions and access the resources it needs. In this way, 179 the client is not requested to have prior knowledge of the service 180 and, consequently, to hard code the URIs of different resources. 181 This would allow the server to make URI changes as the API evolves 182 without breaking the clients. Definitively, a REST service should be 183 as self-descriptive as possible. 185 Therefore, servers implementing the query parameters described in 186 this specification SHOULD provide additional information in their 187 responses about both the available sorting criteria and the possible 188 pagination. Such information is collected in two OPTIONAL response 189 elements named, respectively, "sorting_metadata" and 190 "paging_metadata". 192 The "sorting_metadata" element contains the following properties: 194 o "currentSort": "String" (OPTIONAL) either the value of sort 195 "parameter" as specified in the query string or the sort applied 196 by default, if any; 198 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of objects 199 each one describing an alternate available sorting criterion. 200 Members are: 202 * "property": "String" (REQUIRED) the name that can be used by 203 the client to request the sorting criterion; 204 * "default": "Boolean" (REQUIRED) whether the sorting criterion 205 is applied by default; 206 * "jsonPath": "String" (OPTIONAL) the JSONPath of the RDAP field 207 corresponding to the property; 208 * "links": "Link[]" (OPTIONAL) an array of links as described in 209 RFC 8288 [RFC8288] containing the query string that applies the 210 sorting criterion. 212 At least one between "currentSort" and "availableSorts" MUST be 213 present. 215 The "paging_metadata" element contains the following fields: 217 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 218 the total number of objects found. It MUST be provided if the 219 query string contains the "count" parameter; 221 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 222 number of objects returned in the current page. It MUST be 223 provided when the total number of objects exceeds the page size. 224 This property is redundant for clients because the page size can 225 be derived from the length of the search results array but it can 226 be helpful if the end user interacts with the server through a web 227 browser; 229 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 230 the number of the current page in the result set. It MUST be 231 provided when the total number of objects found exceeds the page 232 size; 234 o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 235 8288 [RFC8288] containing the reference to the next page. In this 236 specification, only the forward pagination is dealt because it is 237 considered satisfactory in order to traverse the result set. 238 Examples of additional references are to: the previous page, the 239 first page, the last page. 241 2.1.1. RDAP Conformance 243 Servers returning the "paging_metadata" element in their response 244 MUST include "paging" in the rdapConformance array as well as servers 245 returning the "sorting_metadata" element MUST include "sorting". 247 2.2. "count" Parameter 249 Currently, the RDAP protocol does not allow a client to determine the 250 total number of the results in a query response when the result set 251 is truncated. This is rather inefficient because the user cannot 252 evaluate the query precision and, at the same time, cannot receive 253 information that could be relevant. 255 The "count" parameter provides additional functionality (Figure 1) 256 that allows a client to request information from the server that 257 specifies the total number of objects matching the search pattern. 259 https://example.com/rdap/domains?name=*nr.com&count=true 261 Figure 1: Example of RDAP query reporting the "count" parameter 263 The ABNF syntax is the following: 265 count = "count=" ( trueValue / falseValue ) 266 trueValue = ("true" / "yes" / "1") 267 falseValue = ("false" / "no" / "0") 269 A trueValue means that the server MUST provide the total number of 270 the objects in the "totalCount" field of the "paging_metadata" 271 element (Figure 2). A falseValue means that the server MUST NOT 272 provide this number. 274 { 275 "rdapConformance": [ 276 "rdap_level_0", 277 "paging" 278 ], 279 ... 280 "paging_metadata": { 281 "totalCount": 43 282 }, 283 "domainSearchResults": [ 284 ... 285 ] 286 } 288 Figure 2: Example of RDAP response with "paging_metadata" element 289 containing the "totalCount" field 291 2.3. "sort" Parameter 293 The RDAP protocol does not provide any capability to specify results 294 sort criteria. A server could implement a default sorting scheme 295 according to the object class, but this feature is not mandatory and 296 might not meet user requirements. Sorting can be addressed by the 297 client, but this solution is rather inefficient. Sorting features 298 provided by the RDAP server could help avoid truncation of relevant 299 results. 301 The "sort" parameter allows the client to ask the server to sort the 302 results according to the values of one or more properties and 303 according to the sort direction of each property. The ABNF syntax is 304 the following: 306 sort = "sort=" sortItem *( "," sortItem ) 307 sortItem = property-ref [":" ( "a" / "d" ) ] 308 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 310 "a" means that the ascending sort MUST be applied, "d" means that the 311 descending sort MUST be applied. If the sort direction is absent, an 312 ascending sort MUST be applied (Figure 3). 314 https://example.com/rdap/domains?name=*nr.com&sort=name 316 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 318 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 320 Figure 3: Examples of RDAP query reporting the "sort" parameter 322 With the only exception of the sort on IP addresses, servers MUST 323 implement sorting according to the JSON value type of the RDAP field 324 the sorting property refers to: JSON strings MUST be sorted 325 lexicographically and JSON numbers MUST be sorted numerically. Even 326 if IP addresses are represented as JSON strings, they MUST be sorted 327 based on their numeric conversion. 329 If the "sort" parameter reports an allowed sorting property, it MUST 330 be provided in the "currentSort" field of the "sorting_metadata" 331 element. 333 2.3.1. Sorting Properties Declaration 335 In the "sort" parameter ABNF syntax, property-ref represents a 336 reference to a property of an RDAP object. Such a reference could be 337 expressed by using a JSONPath. The JSONPath in a JSON document 338 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 339 XML document. For example, the JSONPath to select the value of the 340 ASCII name inside an RDAP domain object is "$.ldhName", whereby $ 341 identifies the root of the document (DOM). Another way to select a 342 value inside a JSON document is the JSON Pointer [RFC6901]. While 343 JSONPath or JSON Pointer are both standard ways to select any value 344 inside JSON data, neither is particularly easy to use (e.g. 345 "$.events[?(@.eventAction='registration')].eventDate" is the JSONPath 346 expression of the registration date in an RDAP domain object). 348 Therefore, this specification provides a definition of property-ref 349 in terms of RDAP properties. However, not all the RDAP properties 350 are suitable to be used in sort criteria, such as: 352 o properties providing service information (e.g. links, notices, 353 remarks, etc.); 355 o multivalued properties (e.g. status, roles, variants, etc.); 357 o properties modeling relationships to other objects (e.g. 358 entities). 360 On the contrary, some properties expressed as values of other 361 properties (e.g. registration date) could be used in such a context. 363 In the following, a list of properties an RDAP server MAY implement 364 is presented. The properties are divided into two groups: object 365 common properties and object specific properties. 367 o Object common properties. Object common properties are derived 368 from the merge of the "eventAction" and the "eventDate" 369 properties. The following values of the "sort" parameter are 370 defined: 372 * registrationDate 373 * reregistrationDate 374 * lastChangedDate 375 * expirationDate 376 * deletionDate 377 * reinstantiationDate 378 * transferDate 379 * lockedDate 380 * unlockedDate 382 o Object specific properties. With regard to the specific 383 properties, some of them are already defined among the query 384 paths. In the following a list of possible sorting properties, 385 grouped by objects, is shown: 387 * Domain: name 388 * Nameserver: name, ipV4, ipV6. 389 * Entity: fn, handle, org, email, voice, country, cc, city. 391 The correspondence between the sorting properties and the RDAP fields 392 is shown in Table 1: 394 +-----------+-----------+---------------------+------+-------+------+ 395 | Object | Sorting | RDAP property | RFC | RFC | RFC | 396 | class | property | | 7483 | 6350 | 8605 | 397 +-----------+-----------+---------------------+------+-------+------+ 398 | Searchabl | Common pr | eventAction values | 4.5. | | | 399 | e objects | operties | suffixed by "Date" | | | | 400 | | | | | | | 401 | Domain | name | unicodeName/ldhName | 5.3. | | | 402 | | | | | | | 403 | Nameserve | name | unicodeName/ldhName | 5.2. | | | 404 | r | | | | | | 405 | | ipV4 | v4 ipAddress | 5.2. | | | 406 | | ipV6 | v6 ipAddress | 5.2. | | | 407 | | | | | | | 408 | Entity | handle | handle | 5.1. | | | 409 | | fn | vcard fn | 5.1. | 6.2.1 | | 410 | | org | vcard org | 5.1. | 6.6.4 | | 411 | | voice | vcard tel with | 5.1. | 6.4.1 | | 412 | | | type="voice" | | | | 413 | | email | vcard email | 5.1. | 6.4.2 | | 414 | | country | country name in | 5.1. | 6.3.1 | | 415 | | | vcard adr | | | | 416 | | cc | country code in | 5.1. | | 3.1 | 417 | | | vcard adr | | | | 418 | | city | locality in vcard | 5.1. | 6.3.1 | | 419 | | | adr | | | | 420 +-----------+-----------+---------------------+------+-------+------+ 422 Table 1: Sorting properties definition 424 With regard to the definitions in Table 1, some further 425 considerations must be made to disambiguate some cases: 427 o since the response to a search on either domains or nameservers 428 might include both A-labels and U-labels ([RFC5890]) in general, a 429 consistent sorting policy shall take unicodeName and ldhName as 430 two formats of the same value rather than separately. Therefore, 431 the unicodeName value MUST be taken while sorting, when 432 unicodeName is missing, the value of ldhName MUST be considered 433 instead; 435 o the jCard "sort-as" parameter MUST be ignored for the purpose of 436 the sorting capability as described in this document; 438 o even if a nameserver can have multiple IPv4 and IPv6 addresses, 439 the most common configuration includes one address for each IP 440 version. Therefore, the assumption of having a single IPv4 and/or 441 IPv6 value for a nameserver cannot be considered too stringent. 443 When more than one address per IP version is reported, sorting 444 MUST be applied to the first value; 446 o multiple events with a given action on an object might be 447 returned. When it occurs, sorting MUST be applied to the most 448 recent event; 450 o with the exception of handle values, all the sorting properties 451 defined for entity objects can be multivalued according to the 452 definition of vCard as given in RFC 6350 [RFC6350]. When more 453 than one value is reported, sorting MUST be applied to the 454 preferred value identified by the parameter pref="1". If the pref 455 parameter is missing, sorting MUST be applied to the first value. 457 Each RDAP provider MAY define other sorting properties than those 458 shown in this document as well as it MAY map those sorting properties 459 onto different locations. 461 The "jsonPath" field in the "sorting_metadata" element is used to 462 clarify the RDAP field the sorting property refers to. The mapping 463 between the sorting properties and the JSONPaths of the RDAP fields 464 is shown in Table 2. The JSONPaths are provided according to the 465 Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]). Further 466 documentation about JSONPath operators used in Table 2 is included in 467 Appendix A. 469 +-------+-------------+---------------------------------------------+ 470 | Objec | Sorting | JSONPath | 471 | t | property | | 472 | class | | | 473 +-------+-------------+---------------------------------------------+ 474 | Searc | registratio | $.domainSearchResults[*].events[?(@.eventAc | 475 | hable | nDate | tion=="registration")].eventDate | 476 | objec | | | 477 | ts | | | 478 | | reregistrat | $.domainSearchResults[*].events[?(@.eventAc | 479 | | ionDate | tion=="reregistration")].eventDate | 480 | | lastChanged | $.domainSearchResults[*].events[?(@.eventAc | 481 | | Date | tion=="last changed")].eventDate | 482 | | expirationD | $.domainSearchResults[*].events[?(@.eventAc | 483 | | ate | tion=="expiration")].eventDate | 484 | | deletionDat | $.domainSearchResults[*].events[?(@.eventAc | 485 | | e | tion=="deletion")].eventDate | 486 | | reinstantia | $.domainSearchResults[*].events[?(@.eventAc | 487 | | tionDate | tion=="reinstantiation")].eventDate | 488 | | transferDat | $.domainSearchResults[*].events[?(@.eventAc | 489 | | e | tion=="transfer")].eventDate | 490 | | lockedDate | $.domainSearchResults[*].events[?(@.eventAc | 491 | | | tion=="locked")].eventDate | 492 | | unlockedDat | $.domainSearchResults[*].events[?(@.eventAc | 493 | | e | tion=="unlocked")].eventDate | 494 | | | | 495 | Domai | name | $.domainSearchResults[*].unicodeName | 496 | n | | | 497 | | | | 498 | Names | name | $.nameserverSearchResults[*].unicodeName | 499 | erver | | | 500 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 501 | | | [0] | 502 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 503 | | | [0] | 504 | | | | 505 | Entit | handle | $.entitySearchResults[*].handle | 506 | y | | | 507 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 508 | | | 0]=="fn")][3] | 509 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 510 | | | 0]=="org")][3] | 511 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 512 | | | 0]=="tel" && @[1].type=="voice")][3] | 513 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 514 | | | 0]=="email")][3] | 515 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 516 | | | 0]=="adr")][3][6] | 517 | | cc | $.entitySearchResults[*].vcardArray[1][?(@[ | 518 | | | 0]=="adr")][1].cc | 519 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 520 | | | 0]=="adr")][3][3] | 521 +-------+-------------+---------------------------------------------+ 523 Table 2: Sorting properties - JSONPath Mapping 525 Note about the JSONPaths of Table 2 that: 527 o those related to the event dates are defined only for the "domain" 528 object. To obtain the equivalent JSONPaths for "entity" and 529 "nameserver", the path segment "domainSearchResults" must be 530 replaced with "entitySearchResults" and "nameserverSearchResults" 531 respectively; 533 o those related to vCard elements are specified without taking into 534 account the "pref" parameter. Servers always applying sorting to 535 those values identified by the pref parameter SHOULD update a 536 JSONPath by adding an appropriate filter. For example, if the 537 email values identified by pref="1" are considered for sorting, 538 the JSONPath of the "email" sorting property should be: 540 $.entitySearchResults[*].vcardArray[1][?(@[0]=="email" && 541 @[1].pref=="1")][3] 543 2.3.2. Representing Sorting Links 545 An RDAP server MAY use the "links" array of the "sorting_metadata" 546 element to provide ready-made references [RFC8288] to the available 547 sort criteria (Figure 4). Each link represents a reference to an 548 alternate view of the results. 550 { 551 "rdapConformance": [ 552 "rdap_level_0", 553 "sorting" 554 ], 555 ... 556 "sorting_metadata": { 557 "currentSort": "name", 558 "availableSorts": [ 559 { 560 "property": "registrationDate", 561 "jsonPath": "$.domainSearchResults[*] 562 .events[?(@.eventAction==\"registration\")].eventDate", 563 "default": false, 564 "links": [ 565 { 566 "value": "https://example.com/rdap/domains?name=*nr.com 567 &sort=name", 568 "rel": "alternate", 569 "href": "https://example.com/rdap/domains?name=*nr.com 570 &sort=registrationDate", 571 "title": "Result Ascending Sort Link", 572 "type": "application/rdap+json" 573 }, 574 { 575 "value": "https://example.com/rdap/domains?name=*nr.com 576 &sort=name", 577 "rel": "alternate", 578 "href": "https://example.com/rdap/domains?name=*nr.com 579 &sort=registrationDate:d", 580 "title": "Result Descending Sort Link", 581 "type": "application/rdap+json" 582 } 583 ] 584 }, 585 ... 586 ] 587 }, 588 "domainSearchResults": [ 589 ... 590 ] 591 } 593 Figure 4: Example of a "sorting_metadata" instance to implement 594 result sorting 596 2.4. "cursor" Parameter 598 The cursor parameter defined in this specification can be used to 599 encode information about any pagination method. For example, in the 600 case of a simple implementation of the cursor parameter to represent 601 offset pagination information, the cursor value 602 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the mere Base64 encoding of 603 "offset=100,limit=50". Likewise, in a simple implementation to 604 represent keyset pagination information, the cursor value 605 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the mere 606 Base64 encoding of "key=thelastdomainofthepage.com" whereby the key 607 value identifies the last row of the current page. 609 This solution lets RDAP providers to implement a pagination method 610 according to their needs, the user access levels, the submitted 611 queries. In addition, servers can change the method over time 612 without announcing anything to the clients. The considerations that 613 has led to this solution are reported in more detail in Appendix B. 615 The ABNF syntax of the cursor paramter is the following: 617 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 619 https://example.com/rdap/domains?name=*nr.com 620 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 622 Figure 5: An example of RDAP query reporting the "cursor" parameter 624 2.4.1. Representing Paging Links 626 An RDAP server SHOULD use the "links" array of the "paging_metadata" 627 element to provide a ready-made reference [RFC8288] to the next page 628 of the result set (Figure 6). Examples of additional "rel" values a 629 server MAY implements are "first", "last", "prev". 631 { 632 "rdapConformance": [ 633 "rdap_level_0", 634 "paging" 635 ], 636 ... 637 "notices": [ 638 { 639 "title": "Search query limits", 640 "type": "result set truncated due to excessive load", 641 "description": [ 642 "search results for domains are limited to 50" 643 ] 644 } 645 ], 646 "paging_metadata": { 647 "totalCount": 73, 648 "pageSize": 50, 649 "pageNumber": 1, 650 "links": [ 651 { 652 "value": "https://example.com/rdap/domains?name=*nr.com", 653 "rel": "next", 654 "href": "https://example.com/rdap/domains?name=*nr.com 655 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 656 "title": "Result Pagination Link", 657 "type": "application/rdap+json" 658 } 659 ] 660 }, 661 "domainSearchResults": [ 662 ... 663 ] 664 } 666 Figure 6: Example of a "paging_metadata" instance to implement cursor 667 pagination 669 3. Negative Answers 671 The value constraints for the parameters are defined by their ABNF 672 syntax. Therefore, each request including an invalid value for a 673 parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The 674 same response SHOULD be returned in the following cases: 676 o if in both single and multi sort the client provides an 677 unsupported value for the "sort" parameter as well as a value 678 related to an object property not included in the response; 680 o if the client submits an invalid value for the "cursor" parameter. 682 Optionally, the response MAY include additional information regarding 683 the negative answer in the HTTP entity body. 685 4. Implementation Considerations 687 The implementation of the new parameters is technically feasible, as 688 operators for counting, sorting and paging are currently supported by 689 the major RDBMSs. 691 Similar operators are completely or partially supported by the most 692 known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so 693 the implementation of the new parameters seems to be practicable by 694 servers working without the use of an RDBMS. 696 5. Implementation Status 698 NOTE: Please remove this section and the reference to RFC 7942 prior 699 to publication as an RFC. 701 This section records the status of known implementations of the 702 protocol defined by this specification at the time of posting of this 703 Internet-Draft, and is based on a proposal described in RFC 7942 704 [RFC7942]. The description of implementations in this section is 705 intended to assist the IETF in its decision processes in progressing 706 drafts to RFCs. Please note that the listing of any individual 707 implementation here does not imply endorsement by the IETF. 708 Furthermore, no effort has been spent to verify the information 709 presented here that was supplied by IETF contributors. This is not 710 intended as, and must not be construed to be, a catalog of available 711 implementations or their features. Readers are advised to note that 712 other implementations may exist. 714 According to RFC 7942, "this will allow reviewers and working groups 715 to assign due consideration to documents that have the benefit of 716 running code, which may serve as evidence of valuable experimentation 717 and feedback that have made the implemented protocols more mature. 718 It is up to the individual working groups to use this information as 719 they see fit". 721 5.1. IIT-CNR/Registro.it 723 Responsible Organization: Institute of Informatics and Telematics 724 of National Research Council (IIT-CNR)/Registro.it 725 Location: https://rdap.pubtest.nic.it/ 726 Description: This implementation includes support for RDAP queries 727 using data from .it public test environment. 729 Level of Maturity: This is an "alpha" test implementation. 730 Coverage: This implementation includes all of the features 731 described in this specification. 732 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 734 5.2. APNIC 736 Responsible Organization: Asia-Pacific Network Information Centre 737 Location: https://github.com/APNIC-net/rdap-rmp-demo/tree/sorting- 738 and-paging 739 Description: A proof-of-concept for RDAP mirroring. 740 Level of Maturity: This is a proof-of-concept implementation. 741 Coverage: This implementation includes all of the features 742 described in the specification except for nameserver sorting and 743 unicodeName sorting. 744 Contact Information: Tom Harrison, tomh@apnic.net 746 6. IANA Considerations 748 IANA is requested to register the following values in the RDAP 749 Extensions Registry: 751 Extension identifier: paging 752 Registry operator: Any 753 Published specification: This document. 754 Contact: IESG 755 Intended usage: This extension describes a best practice for 756 result set paging. 758 Extension identifier: sorting 759 Registry operator: Any 760 Published specification: This document. 761 Contact: IESG 762 Intended usage: This extension describes a best practice for 763 result set sorting. 765 7. Security Considerations 767 Security services for the operations specified in this document are 768 described in RFC 7481 [RFC7481]. 770 The search query typically requires more server resources (such as 771 memory, CPU cycles, and network bandwidth) when compared to the 772 lookup query. This increases the risk of server resource exhaustion 773 and subsequent denial of service due to abuse. This risk can be 774 mitigated by either restricting search functionality or limiting the 775 rate of search requests. Servers can also reduce their load by 776 truncating the results in the response. However, this last security 777 policy can result in a higher inefficiency if the RDAP server does 778 not provide any functionality to return the truncated results. 780 The new parameters presented in this document provide the RDAP 781 operators with a way to implement a secure server without penalizing 782 its efficiency. The "count" parameter gives the user a measure to 783 evaluate the query precision and, at the same time, returns a 784 significant information. The "sort" parameter allows the user to 785 obtain the most relevant information at the beginning of the result 786 set. In both cases, the user doesn't need to submit further 787 unnecessary search requests. Finally, the "cursor" parameter enables 788 the user to scroll the result set by submitting a sequence of 789 sustainable queries according to the server limits. 791 8. Acknowledgements 793 The authors would like to acknowledge Brian Mountford, Tom Harrison, 794 Karl Heinz Wolf and Jasdip Singh for their contribution to the 795 development of this document. 797 9. References 799 9.1. Normative References 801 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 802 Requirement Levels", BCP 14, RFC 2119, 803 DOI 10.17487/RFC2119, March 1997, 804 . 806 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 807 Specifications: ABNF", STD 68, RFC 5234, 808 DOI 10.17487/RFC5234, January 2008, 809 . 811 [RFC5890] Klensin, J., "Internationalized Domain Names for 812 Applications (IDNA): Definitions and Document Framework", 813 RFC 5890, DOI 10.17487/RFC5890, August 2010, 814 . 816 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 817 DOI 10.17487/RFC6350, August 2011, 818 . 820 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 821 Protocol (HTTP/1.1): Message Syntax and Routing", 822 RFC 7230, DOI 10.17487/RFC7230, June 2014, 823 . 825 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 826 Registration Data Access Protocol (RDAP)", RFC 7480, 827 DOI 10.17487/RFC7480, March 2015, 828 . 830 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 831 Registration Data Access Protocol (RDAP)", RFC 7481, 832 DOI 10.17487/RFC7481, March 2015, 833 . 835 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 836 Protocol (RDAP) Query Format", RFC 7482, 837 DOI 10.17487/RFC7482, March 2015, 838 . 840 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 841 Registration Data Access Protocol (RDAP)", RFC 7483, 842 DOI 10.17487/RFC7483, March 2015, 843 . 845 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 846 Code: The Implementation Status Section", BCP 205, 847 RFC 7942, DOI 10.17487/RFC7942, July 2016, 848 . 850 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 851 Interchange Format", STD 90, RFC 8259, 852 DOI 10.17487/RFC8259, December 2017, 853 . 855 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 856 DOI 10.17487/RFC8288, October 2017, 857 . 859 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 860 ICANN Extensions for the Registration Data Access Protocol 861 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 862 . 864 [W3C.CR-xpath-31-20161213] 865 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 866 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 867 31-20161213, December 2016, 868 . 870 9.2. Informative References 872 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 873 Pagination", July 2014, . 876 [CURSOR-API1] 877 facebook.com, "facebook for developers - Using the Graph 878 API", July 2017, . 881 [CURSOR-API2] 882 twitter.com, "Pagination", 2017, 883 . 886 [GOESSNER-JSON-PATH] 887 Goessner, S., "JSONPath - XPath for JSON", 2007, 888 . 890 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 891 . 894 [OData-Part1] 895 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 896 Part 1: Protocol Plus Errata 03", June 2016, 897 . 902 [REST] Fredrich, T., "RESTful Service Best Practices, 903 Recommendations for Creating Web Services", April 2012, 904 . 907 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 908 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 909 DOI 10.17487/RFC6901, April 2013, 910 . 912 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 913 With Limit and Offset is Slow?", July 2017, 914 . 917 Appendix A. JSONPath operators 919 A JSONPath expression represents a path to find an element (or a set 920 of elements) in a JSON content. 922 The base JSONPath specification requires that implementations support 923 a set of "basic operators". These operators are used to access the 924 elements of a JSON structure like objects and arrays, and their 925 subelements, respectively, object members and array items. No 926 operations are defined for retrieving parent or sibling elements of a 927 given element. The root element is always referred to as $ 928 regardless if it is an object or array. 930 Additionally, the specification permits implementations to support 931 arbitrary script expressions: these can be used to index into an 932 object or an array, or to filter elements from an array. While 933 script expression behaviour is implementation-defined, most 934 implementations support the basic relational and logical operators, 935 as well as both object member and array item access, sufficiently 936 similarly for the purposes of this document. Commonly-supported 937 operators/functions divided into "top-level operators" and "filter 938 operators" are documented in Table 3 and Table 4 respectively. 940 +-------------------+-----------------------------------------+ 941 | Operator | Descritpion | 942 +-------------------+-----------------------------------------+ 943 | $ | Root element | 944 | . | Object member access (dot-notation) | 945 | [''] | Object member access (bracket-notation) | 946 | [] | Array item access | 947 | * | All elements within the specified scope | 948 | [?()] | Filter expression | 949 +-------------------+-----------------------------------------+ 951 Table 3: JSONPath Top-Level Operators 953 +------------+----------------------------------------+ 954 | Operator | Descritpion | 955 +------------+----------------------------------------+ 956 | @ | Current element being processed | 957 | . | Object member access | 958 | [] | Array item access | 959 | == | Left is equal to right | 960 | != | Left is not equal to right | 961 | < | Left is less than right | 962 | <= | Left is less than or equal to right | 963 | > | Left is greater than right | 964 | >= | Left is greater than or equal to right | 965 | && | Logical conjunction | 966 | || | Logical disjunction | 967 +------------+----------------------------------------+ 969 Table 4: JSONPath Filter Operators 971 Appendix B. Approaches to Result Pagination 973 An RDAP query could return a response with hundreds, even thousands, 974 of objects, especially when partial matching is used. For that 975 reason, the cursor parameter addressing result pagination is defined 976 to make responses easier to handle. 978 Presently, the most popular methods to implement pagination in REST 979 API are: offset pagination and keyset pagination. Both two 980 pagination methods don't require the server to handle the result set 981 in a storage area across the requests since a new result set is 982 generated each time a request is submitted. Therefore, they are 983 preferred in comparison to any other method requiring the management 984 of a REST session. 986 Using limit and offset operators represents the traditionally used 987 method to implement results pagination. Both of them can be used 988 individually: 990 o "limit": means that the server must return the first N objects of 991 the result set; 993 o "offset": means that the server must skip the first N objects and 994 must return objects starting from position N+1. 996 When limit and offset are used together, they allow to identify a 997 specific portion of the result set. For example, the pair 998 "offset=100,limit=50" returns first 50 objects starting from position 999 101 of the result set. 1001 Despite its easiness of implementation, offset pagination raises some 1002 well known drawbacks: 1004 o when offset has a very high value, scrolling the result set could 1005 take some time; 1007 o it always requires to fetch all the rows before dropping as many 1008 rows as specified by offset; 1010 o it may return inconsistent pages when data are frequently updated 1011 (i.e. real-time data) but this doesn't seem the case of 1012 registration data. 1014 The keyset pagination [SEEK] consists in adding a query condition 1015 that enables the selection of the only data not yet returned. This 1016 method has been taken as the basis for the implementation of a 1017 "cursor" parameter [CURSOR] by some REST API providers (e.g. 1018 [CURSOR-API1],[CURSOR-API2]). The cursor is an opaque URL-safe 1019 string representing a logical pointer to the first result of the next 1020 page (Figure 5). 1022 Nevertheless, even keyset pagination can be troublesome: 1024 o it needs at least one key field; 1026 o it does not allow to sort just by any field because the sorting 1027 criterion must contain a key; 1029 o it works best with full composite values support by DBMS (i.e. 1030 [x,y]>[a,b]), emulation is possible but ugly and less performant; 1032 o it does not allow to directly navigate to arbitrary pages because 1033 the result set must be scrolled in sequential order starting from 1034 the initial page; 1036 o implementing the bi-directional navigation is tedious because all 1037 comparison and sort operations have to be reversed. 1039 B.1. Specific Issues Raised by RDAP 1041 Furthermore, in the RDAP context, some additional considerations can 1042 be made: 1044 o an RDAP object is a conceptual aggregation of information 1045 generally collected from more than one data structure (e.g. table) 1046 and this makes even harder for the developers the implementation 1047 of the keyset pagination that is already quite difficult. For 1048 example, the entity object can gather information from different 1049 data structures (registrars, registrants, contacts, resellers, and 1050 so on), each one with its own key field mapping the RDAP entity 1051 handle; 1053 o depending on the number of the page results as well as the number 1054 and the complexity of the properties of each RDAP object in the 1055 response, the time required by offset pagination to skip the 1056 previous pages could be much faster than the processing time 1057 needed to build the current page. In fact, RDAP objects are 1058 usually formed by information belonging to multiple data 1059 structures and containing multivalued properties (i.e. arrays) 1060 and, therefore, data selection might be a time consuming process. 1061 This situation occurs even though the selection is supported by 1062 indexes; 1064 o depending on the access levels defined by each RDAP operator, the 1065 increase of complexity and the decrease of flexibility of keyset 1066 pagination with respect to the offset pagination could be 1067 considered impractical. 1069 Ultimately, both pagination methods have benefits and drawbacks. 1071 Appendix C. Change Log 1073 00: Initial working group version ported from draft-loffredo-regext- 1074 rdap-sorting-and-paging-05 1075 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1076 consistent between the pagination methods. Renamed 1077 "Considerations about Paging Implementation" section in ""cursor" 1078 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1079 detailed description of both "sorting_metadata" and 1080 "paging_metadata" objects. 1081 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1082 of cursor parameter. Rearranged the layout of some sections. 1083 Removed some items from "Informative References" section. Changed 1084 "IANA Considerations" section. 1085 03: Added "cc" to the list of sorting properties in "Sorting 1086 Properties Declaration" section. Added RFC8605 to the list of 1087 "Informative References". 1088 04: Replaced "ldhName" with "name" in the "Sorting Properties 1089 Declaration" section. Clarified the sorting logic with respect to 1090 the JSON value types and the sorting policy for multivalued 1091 fields. 1092 05: Clarified the logic of sorting on IP addresses. Clarified the 1093 mapping between the sorting properties and the RDAP fields. 1094 Updated "Acknowledgements" section. 1095 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1096 "paging_metadata" object. 1098 07: Added "Paging Responses to POST Requests" section. 1099 08: Added "Approaches to Result Pagination" section to appendix. 1100 Added the case of requesting a sort on a property not included in 1101 the response to the errors listed in the "Negative Answers" 1102 section. 1103 09: Updated the "Implementation Status" section to include APNIC 1104 implementation. Moved the "RDAP Conformance" section up in the 1105 document. Removed the "Paging Responses to POST Requests" 1106 section. Updated the "Acknowledgements" section. Removed unused 1107 references. In the "Sorting Properties Declaration" section: 1109 * clarified the logic of sorting on events; 1110 * corrected the JSONPath of the "lastChanged" sorting property; 1111 * provided a JSONPath example taking into account the vCard 1112 "pref" parameter. 1113 10: Corrected the JSONPaths of both "fn" and "org" sorting 1114 properties in Table 2. Corrected JSON content in Figure 4. Moved 1115 [W3C.CR-xpath-31-20161213] and [RFC7942] to the "Normative 1116 References". Changed the rdapConformance tags "sorting_level_0" 1117 and "paging_level_0" to "sorting" and "paging" respectively. 1118 11: Added the "JSONPath operators" section to appendix. 1119 12: Changed the content of "JSONPath operators" section. 1121 Authors' Addresses 1123 Mario Loffredo 1124 IIT-CNR/Registro.it 1125 Via Moruzzi,1 1126 Pisa 56124 1127 IT 1129 Email: mario.loffredo@iit.cnr.it 1130 URI: http://www.iit.cnr.it 1132 Maurizio Martinelli 1133 IIT-CNR/Registro.it 1134 Via Moruzzi,1 1135 Pisa 56124 1136 IT 1138 Email: maurizio.martinelli@iit.cnr.it 1139 URI: http://www.iit.cnr.it 1140 Scott Hollenbeck 1141 Verisign Labs 1142 12061 Bluemont Way 1143 Reston, VA 20190 1144 USA 1146 Email: shollenbeck@verisign.com 1147 URI: https://www.verisignlabs.com/