idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 11, 2020) is 1529 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 491 -- Looks like a reference, but probably isn't: '1' on line 507 -- Looks like a reference, but probably isn't: '3' on line 508 -- Looks like a reference, but probably isn't: '6' on line 504 == Unused Reference: 'RFC5226' is defined on line 847, but no explicit reference was found in the text == Unused Reference: 'RFC8605' is defined on line 900, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 5 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: August 14, 2020 S. Hollenbeck 6 Verisign Labs 7 February 11, 2020 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-08 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on August 14, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 64 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 65 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 7 66 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 11 67 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 13 68 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 13 69 2.4.2. Paging Responses to POST Requests . . . . . . . . . . 14 70 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 16 71 4. RDAP Conformance . . . . . . . . . . . . . . . . . . . . . . 17 72 5. Implementation Considerations . . . . . . . . . . . . . . . . 17 73 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 17 74 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 17 75 6.2. Google Registry . . . . . . . . . . . . . . . . . . . . . 18 76 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 77 8. Security Considerations . . . . . . . . . . . . . . . . . . . 18 78 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 79 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 80 10.1. Normative References . . . . . . . . . . . . . . . . . . 19 81 10.2. Informative References . . . . . . . . . . . . . . . . . 21 82 Appendix A. Approaches to Result Pagination . . . . . . . . . . 22 83 A.1. Specific Issues Raised by RDAP . . . . . . . . . . . . . 23 84 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 24 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 87 1. Introduction 89 The availability of functionality for result sorting and paging 90 provides benefits to both clients and servers in the implementation 91 of RESTful services [REST]. These benefits include: 93 o reducing the server response bandwidth requirements; 94 o improving server response time; 95 o improving query precision and, consequently, obtaining more 96 reliable results; 98 o decreasing server query processing load; 99 o reducing client response processing time. 101 Approaches to implementing features for result sorting and paging can 102 be grouped into two main categories: 104 1. Sorting and paging are implemented through the introduction of 105 additional parameters in the query string (i.e. ODATA protocol 106 [OData-Part1]); 108 2. Information related to the number of results and the specific 109 portion of the result set to be returned, in addition to a set of 110 ready-made links for the result set scrolling, are inserted in 111 the HTTP header of the request/response. 113 However, there are some drawbacks associated with the use of the HTTP 114 header. First, the header properties cannot be set directly from a 115 web browser. Moreover, in an HTTP session, the information on the 116 status (i.e. the session identifier) is usually inserted in the 117 header or in the cookies, while the information on the resource 118 identification or the search type is included in the query string. 119 The second approach is therefore not compliant with the HTTP standard 120 [RFC7230]. As a result, this document describes a specification 121 based on the use of query parameters. 123 Currently, the RDAP protocol [RFC7482] defines two query types: 125 o lookup: the server returns only one object; 126 o search: the server returns a collection of objects. 128 While the lookup query does not raise issues in the response 129 management, the search query can potentially generate a large result 130 set that could be truncated according to the server limits. In 131 addition, it is not possible to obtain the total number of the 132 objects found that might be returned in a search query response 133 [RFC7483]. Lastly, there is no way to specify sort criteria to 134 return the most relevant objects at the beginning of the result set. 135 Therefore, the client might traverse the whole result set to find the 136 relevant objects or, due to truncation, could not find them at all. 138 The specification described in this document extends RDAP query 139 capabilities to enable result sorting and paging, by adding new query 140 parameters that can be applied to RDAP search path segments. The 141 service is implemented using the Hypertext Transfer Protocol (HTTP) 142 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 144 The implementation of the new parameters is technically feasible, as 145 operators for counting, sorting and paging rows are currently 146 supported by the major RDBMSs. 148 1.1. Conventions Used in This Document 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in [RFC2119]. 154 2. RDAP Query Parameter Specification 156 The new query parameters are OPTIONAL extensions of path segments 157 defined in RFC 7482 [RFC7482]. They are as follows: 159 o "count": a boolean value that allows a client to request the total 160 number of objects found (that due to truncation can be different 161 from the number of returned objects); 163 o "sort": a string value that allows a client to request a specific 164 sort order for the result set; 166 o "cursor": a string value representing a pointer to a specific 167 fixed size portion of the result set. 169 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 170 sections to describe the formal syntax of these new parameters. 172 2.1. Sorting and Paging Metadata 174 According to most advanced principles in REST design, collectively 175 known as HATEOAS (Hypermedia as the Engine of Application State) 176 ([HATEOAS]), a client entering a REST application through an initial 177 URI should use the server-provided links to dynamically discover 178 available actions and access the resources it needs. In this way, 179 the client is not requested to have prior knowledge of the service 180 and, consequently, to hard code the URIs of different resources. 181 This would allow the server to make URI changes as the API evolves 182 without breaking the clients. Definitively, a REST service should be 183 as self-descriptive as possible. 185 Therefore, servers implementing the query parameters described in 186 this specification SHOULD provide additional information in their 187 responses about both the available sorting criteria and the possible 188 pagination. Such information is collected in two OPTIONAL response 189 elements named, respectively, "sorting_metadata" and 190 "paging_metadata". 192 The "sorting_metadata" element contains the following properties: 194 o "currentSort": "String" (OPTIONAL) either the value of sort 195 "parameter" as specified in the query string or the sort applied 196 by default, if any; 198 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of objects 199 each one describing an alternate available sorting criterion. 200 Members are: 202 * "property": "String" (REQUIRED) the name that can be used by 203 the client to request the sorting criterion; 204 * "default": "Boolean" (REQUIRED) whether the sorting criterion 205 is applied by default; 206 * "jsonPath": "String" (OPTIONAL) the JSON Path of the RDAP field 207 corresponding to the property; 208 * "links": "Link[]" (OPTIONAL) an array of links as described in 209 RFC 8288 [RFC8288] containing the query string that applies the 210 sorting criterion. 212 At least one between "currentSort" and "availableSorts" MUST be 213 present. 215 The "paging_metadata" element contains the following fields: 217 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 218 the total number of objects found. It MUST be provided if the 219 query string contains the "count" parameter; 221 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 222 number of objects returned in the current page. It MUST be 223 provided when the total number of objects exceeds the page size. 224 This property is redundant for clients because the page size can 225 be derived from the length of the search results array but it can 226 be helpful if the end user interacts with the server through a web 227 browser; 229 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 230 the number of the current page in the result set.It MUST be 231 provided when the total number of objects found exceeds the page 232 size; 234 o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 235 8288 [RFC8288] containing the reference to the next page. In this 236 specification, only the forward pagination is dealt because it is 237 considered satisfactory in order to traverse the result set. 238 Examples of additional references are to: the previous page, the 239 first page, the last page. 241 2.2. "count" Parameter 243 Currently, the RDAP protocol does not allow a client to determine the 244 total number of the results in a query response when the result set 245 is truncated. This is rather inefficient because the user cannot 246 evaluate the query precision and, at the same time, cannot receive 247 information that could be relevant. 249 The "count" parameter provides additional functionality (Figure 1) 250 that allows a client to request information from the server that 251 specifies the total number of elements matching the search pattern. 253 https://example.com/rdap/domains?name=*nr.com&count=true 255 Figure 1: Example of RDAP query reporting the "count" parameter 257 The ABNF syntax is the following: 259 count = "count=" ( trueValue / falseValue ) 260 trueValue = ("true" / "yes" / "1") 261 falseValue = ("false" / "no" / "0") 263 A trueValue means that the server MUST provide the total number of 264 the objects in the "totalCount" field of the "paging_metadata" 265 element (Figure 2). A falseValue means that the server MUST NOT 266 provide this number. 268 { 269 "rdapConformance": [ 270 "rdap_level_0", 271 "paging_level_0" 272 ], 273 ... 274 "paging_metadata": { 275 "totalCount": 43 276 }, 277 "domainSearchResults": [ 278 ... 279 ] 280 } 282 Figure 2: Example of RDAP response with "paging_metadata" element 283 containing the "totalCount" field 285 2.3. "sort" Parameter 287 The RDAP protocol does not provide any capability to specify results 288 sort criteria. A server could implement a default sorting scheme 289 according to the object class, but this feature is not mandatory and 290 might not meet user requirements. Sorting can be addressed by the 291 client, but this solution is rather inefficient. Sorting features 292 provided by the RDAP server could help avoid truncation of relevant 293 results. 295 The "sort" parameter allows the client to ask the server to sort the 296 results according to the values of one or more properties and 297 according to the sort direction of each property. The ABNF syntax is 298 the following: 300 sort = "sort=" sortItem *( "," sortItem ) 301 sortItem = property-ref [":" ( "a" / "d" ) ] 302 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 304 "a" means that the ascending sort MUST be applied, "d" means that the 305 descending sort MUST be applied. If the sort direction is absent, an 306 ascending sort MUST be applied (Figure 3). 308 https://example.com/rdap/domains?name=*nr.com&sort=name 310 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 312 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 314 Figure 3: Examples of RDAP query reporting the "sort" parameter 316 With the only exception of the sort on IP addresses, servers MUST 317 implement sorting according to the JSON value type of the RDAP field 318 the sorting property refers to: JSON strings MUST be sorted 319 lexicographically and JSON numbers MUST be sorted numerically. Even 320 if IP addresses are represented as JSON strings, they MUST be sorted 321 based on their numeric conversion. 323 If the "sort" parameter reports an allowed sorting property, it MUST 324 be provided in the "currentSort" field of the "sorting_metadata" 325 element. 327 2.3.1. Sorting Properties Declaration 329 In the "sort" parameter ABNF syntax, property-ref represents a 330 reference to a property of an RDAP object. Such a reference could be 331 expressed by using a JSON Path. The JSON Path in a JSON document 333 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 334 XML document. For example, the JSON Path to select the value of the 335 ASCII name inside an RDAP domain object is "$.ldhName", whereby $ 336 identifies the root of the document (DOM). Another way to select a 337 value inside a JSON document is the JSON Pointer [RFC6901]. While 338 JSON Path or JSON Pointer are both standard ways to select any value 339 inside JSON data, neither is particularly easy to use (e.g. 340 "$.events[?(@.eventAction='registration')].eventDate" is the JSON 341 Path expression of the registration date in an RDAP domain object). 343 Therefore, this specification provides a definition of property-ref 344 in terms of RDAP properties. However, not all the RDAP properties 345 are suitable to be used in sort criteria, such as: 347 o properties providing service information (e.g. links, notices, 348 remarks, etc.); 350 o multivalued properties (e.g. status, roles, variants, etc.); 352 o properties modeling relationships to other objects (e.g. 353 entities). 355 On the contrary, some properties expressed as values of other 356 properties (e.g. registration date) could be used in such a context. 358 In the following, a list of properties an RDAP server MAY implement 359 is presented. The properties are divided into two groups: object 360 common properties and object specific properties. 362 o Object common properties. Object common properties are derived 363 from the merge of the "eventAction" and the "eventDate" 364 properties. The following values of the "sort" parameter are 365 defined: 367 * registrationDate 368 * reregistrationDate 369 * lastChangedDate 370 * expirationDate 371 * deletionDate 372 * reinstantiationDate 373 * transferDate 374 * lockedDate 375 * unlockedDate 377 o Object specific properties. With regard to the specific 378 properties, some of them are already defined among the query 379 paths. In the following a list of possible sorting properties, 380 grouped by objects, is shown: 382 * Domain: name 383 * Nameserver: name, ipV4, ipV6. 384 * Entity: fn, handle, org, email, voice, country, cc, city. 386 The correspondence between the sorting properties and the RDAP fields 387 is shown in Table 1: 389 +-----------+-----------+---------------------+------+-------+------+ 390 | Object | Sorting | RDAP property | RFC | RFC | RFC | 391 | class | property | | 7483 | 6350 | 8605 | 392 +-----------+-----------+---------------------+------+-------+------+ 393 | Searchabl | Common pr | eventAction values | 4.5. | | | 394 | e objects | operties | suffixed by "Date" | | | | 395 | | | | | | | 396 | Domain | name | unicodeName/ldhName | 5.3. | | | 397 | | | | | | | 398 | Nameserve | name | unicodeName/ldhName | 5.2. | | | 399 | r | | | | | | 400 | | ipV4 | v4 ipAddress | 5.2. | | | 401 | | ipV6 | v6 ipAddress | 5.2. | | | 402 | | | | | | | 403 | Entity | handle | handle | 5.1. | | | 404 | | fn | vcard fn | 5.1. | 6.2.1 | | 405 | | org | vcard org | 5.1. | 6.6.4 | | 406 | | voice | vcard tel with | 5.1. | 6.4.1 | | 407 | | | type="voice" | | | | 408 | | email | vcard email | 5.1. | 6.4.2 | | 409 | | country | country name in | 5.1. | 6.3.1 | | 410 | | | vcard adr | | | | 411 | | cc | country code in | 5.1. | | 3.1 | 412 | | | vcard adr | | | | 413 | | city | locality in vcard | 5.1. | 6.3.1 | | 414 | | | adr | | | | 415 +-----------+-----------+---------------------+------+-------+------+ 417 Table 1: Sorting properties definition 419 With regard to the definitions in Table 1, some further 420 considerations must be made to disambiguate some cases: 422 o since the response to a search on either domains or nameservers 423 might include both A-labels and U-labels ([RFC5890]) in general, a 424 consistent sorting policy shall take unicodeName and ldhName as 425 two formats of the same value rather than separately. Therefore, 426 the unicodeName value MUST be taken while sorting, when 427 unicodeName is missing, the value of ldhName MUST be considered 428 instead; 430 o the jCard "sort-as" parameter MUST be ignored for the purpose of 431 the sorting capability as described in this document; 433 o even if a nameserver can have multiple IPv4 and IPv6 addresses, 434 the most common configuration includes one address for each IP 435 version. Therefore, the assumption of having a single IPv4 and/or 436 IPv6 value for a nameserver cannot be considered too stringent. 437 When more than one address per IP version is reported, sorting 438 MUST be applied to the first value; 440 o with the exception of handle values, all the sorting properties 441 defined for entity objects can be multivalued according to the 442 definition of vCard as given in RFC6350 [RFC6350]. When more than 443 one value is reported, sorting MUST be applied to the preferred 444 value identified by the parameter pref="1". If the pref parameter 445 is missing, sorting MUST be applied to the first value. 447 Each RDAP provider MAY define other sorting properties than those 448 shown in this document as well as it MAY map those sorting properties 449 onto different locations. 451 The "jsonPath" field in the "sorting_metadata" element is used to 452 clarify the RDAP field the sorting property refers to. The mapping 453 between the sorting properties and the JSON Paths of the RDAP fields 454 is shown in Table 2. The JSON Paths are provided according to the 455 Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]): 457 +-------+-------------+---------------------------------------------+ 458 | Objec | Sorting | JSON Path | 459 | t | property | | 460 | class | | | 461 +-------+-------------+---------------------------------------------+ 462 | Searc | registratio | "$.domainSearchResults[*].events[?(@.eventA | 463 | hable | nDate | ction=="registration")].eventDate | 464 | objec | | | 465 | ts | | | 466 | | reregistrat | "$.domainSearchResults[*].events[?(@.eventA | 467 | | ionDate | ction=="reregistration")].eventDate | 468 | | lastChanged | "$.domainSearchResults[*].events[?(@.eventA | 469 | | Date | ction=="lastChanged")].eventDate | 470 | | expirationD | "$.domainSearchResults[*].events[?(@.eventA | 471 | | ate | ction=="expiration")].eventDate | 472 | | deletionDat | "$.domainSearchResults[*].events[?(@.eventA | 473 | | e | ction=="deletion")].eventDate | 474 | | reinstantia | "$.domainSearchResults[*].events[?(@.eventA | 475 | | tionDate | ction=="reinstantiation")].eventDate | 476 | | transferDat | "$.domainSearchResults[*].events[?(@.eventA | 477 | | e | ction=="transfer")].eventDate | 478 | | lockedDate | "$.domainSearchResults[*].events[?(@.eventA | 479 | | | ction=="locked")].eventDate | 480 | | unlockedDat | "$.domainSearchResults[*].events[?(@.eventA | 481 | | e | ction=="unlocked")].eventDate | 482 | | | | 483 | Domai | name | $.domainSearchResults[*].unicodeName | 484 | n | | | 485 | | | | 486 | Names | name | $.nameserverSearchResults[*].unicodeName | 487 | erver | | | 488 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 489 | | | [0] | 490 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 491 | | | [0] | 492 | | | | 493 | Entit | handle | $.entitySearchResults[*].handle | 494 | y | | | 495 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 496 | | | 0]="fn")][3] | 497 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 498 | | | 0]="org")][3] | 499 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 500 | | | 0]=="tel" && @[1].type=="voice")][3] | 501 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 502 | | | 0]=="email")][3] | 503 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 504 | | | 0]=="adr")][3][6] | 505 | | cc | $.entitySearchResults[*].vcardArray[1][?(@[ | 506 | | | 0]=="adr")][1].cc | 507 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 508 | | | 0]=="adr")][3][3] | 509 +-------+-------------+---------------------------------------------+ 511 Table 2: Sorting properties - JSON Path Mapping 513 2.3.2. Representing Sorting Links 515 An RDAP server MAY use the "links" array of the "sorting_metadata" 516 element to provide ready-made references [RFC8288] to the available 517 sort criteria (Figure 4). Each link represents a reference to an 518 alternate view of the results. 520 { 521 "rdapConformance": [ 522 "rdap_level_0", 523 "sorting_level_0" 524 ], 525 ... 526 "sorting_metadata": { 527 "currentSort": "name", 528 "availableSorts": [ 529 { 530 "property": "registrationDate", 531 "jsonPath": "$.domainSearchResults[*] 532 .events[?(@.eventAction==\"registration\")].eventDate", 533 "default": false, 534 "links": [ 535 { 536 "value": "https://example.com/rdap/domains?name=*nr.com 537 &sort=name", 538 "rel": "alternate", 539 "href": "https://example.com/rdap/domains?name=*nr.com 540 &sort=registrationDate", 541 "title": "Result Ascending Sort Link", 542 "type": "application/rdap+json" 543 }, 544 { 545 "value": "https://example.com/rdap/domains?name=*nr.com 546 &sort=name", 547 "rel": "alternate", 548 "href": "https://example.com/rdap/domains?name=*nr.com 549 &sort=registrationDate:d", 550 "title": "Result Descending Sort Link", 551 "type": "application/rdap+json" 552 } 553 ], 554 ... 555 }, 556 "domainSearchResults": [ 557 ... 558 ] 559 } 561 Figure 4: Example of a "sorting_metadata" instance to implement 562 result sorting 564 2.4. "cursor" Parameter 566 The cursor parameter defined in this specification can be used to 567 encode information about any pagination method. For example, in the 568 case of a simple implementation of the cursor parameter to represent 569 offset pagination information, the cursor value 570 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the mere Base64 encoding of 571 "offset=100,limit=50". Likewise, in a simple implementation to 572 represent keyset pagination information, the cursor value 573 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the mere 574 Base64 encoding of "key=thelastdomainofthepage.com" whereby the key 575 value identifies the last row of the current page. 577 This solution lets RDAP providers to implement a pagination method 578 according to their needs, the user access levels, the submitted 579 queries. In addition, servers can change the method over time 580 without announcing anything to the clients. The considerations that 581 has led to this solution are reported in more detail in Appendix A. 583 The ABNF syntax of the cursor paramter is the following: 585 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 587 https://example.com/rdap/domains?name=*nr.com 588 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 590 Figure 5: An example of RDAP query reporting the "cursor" parameter 592 2.4.1. Representing Paging Links 594 An RDAP server SHOULD use the "links" array of the "paging_metadata" 595 element to provide a ready-made reference [RFC8288] to the next page 596 of the result set (Figure 6). Examples of additional "rel" values a 597 server MAY implements are "first", "last", "prev". 599 { 600 "rdapConformance": [ 601 "rdap_level_0", 602 "paging_level_0" 603 ], 604 ... 605 "notices": [ 606 { 607 "title": "Search query limits", 608 "type": "result set truncated due to excessive load", 609 "description": [ 610 "search results for domains are limited to 50" 611 ] 612 } 613 ], 614 "paging_metadata": { 615 "totalCount": 73, 616 "pageSize": 50, 617 "pageNumber": 1, 618 "links": [ 619 { 620 "value": "https://example.com/rdap/domains?name=*nr.com", 621 "rel": "next", 622 "href": "https://example.com/rdap/domains?name=*nr.com 623 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 624 "title": "Result Pagination Link", 625 "type": "application/rdap+json" 626 } 627 ] 628 }, 629 "domainSearchResults": [ 630 ... 631 ] 632 } 634 Figure 6: Example of a "paging_metadata" instance to implement cursor 635 pagination 637 2.4.2. Paging Responses to POST Requests 639 In this specification, pagination is implemented by providing the 640 user with a web link through a GET request [RFC7482]. However, GET 641 could not be the only request method supported by an RDAP server in 642 the future. 644 A possible use case requiring POST might be the submission of a 645 complex search condition including predicates joined by boolean 646 operators (i.e. OR, AND, NOT). According to the search format and 647 complexity, the solution of providing a link by GET seems to be 648 pretty inefficient. In fact, GET isn't suitable for supporting 649 either very long or URL-unsafe query strings. It would be much more 650 appropriate to send the search pattern and the optional query 651 parameters by POST. Therefore, an RDAP response element which is 652 meant to represent the pagination information should also consider 653 the POST method. 655 As a consequence, the "paging_metadata" element MUST include an 656 additional property, alternate to "links", that contains the cursor 657 values used for pagination. Such property is defined as in the 658 following: 660 "cursors": "String[String]" (OPTIONAL) a map of cursor values 661 pointing to specific fixed size portions of the result set. Ths 662 property MUST be used instead of "links" when the request is 663 submitted via POST. The map keys MUST contain the "rel" values 664 used for generating the paging links (Figure 7). Examples are: 665 "next", "prev", "last". The link to the first page is 666 unnecessary. 668 { 669 "rdapConformance": [ 670 "rdap_level_0", 671 "paging_level_0" 672 ], 673 ... 674 "notices": [ 675 { 676 "title": "Search query limits", 677 "type": "result set truncated due to excessive load", 678 "description": [ 679 "search results for domains are limited to 50" 680 ] 681 } 682 ], 683 "paging_metadata": { 684 "totalCount": 73, 685 "pageSize": 50, 686 "pageNumber": 1, 687 "cursors": { 688 "next":"wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=" 689 } 690 }, 691 "domainSearchResults": [ 692 ... 693 ] 694 } 696 Figure 7: Example of a "paging_metadata" instance when the request is 697 submitted via POST 699 3. Negative Answers 701 The value constraints for the parameters are defined by their ABNF 702 syntax. Therefore, each request including an invalid value for a 703 parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The 704 same response SHOULD be returned in the following cases: 706 o if in both single and multi sort the client provides an 707 unsupported value for the "sort" parameter as well as a value 708 related to an object property not included in the response; 710 o if the client submits an invalid value for the "cursor" parameter. 712 Optionally, the response MAY include additional information regarding 713 the negative answer in the HTTP entity body. 715 4. RDAP Conformance 717 Servers returning the "paging_metadata" element in their response 718 MUST include "paging_level_0" in the rdapConformance array as well as 719 servers returning the "sorting_metadata" element MUST include 720 "sorting_level_0". 722 5. Implementation Considerations 724 The implementation of the new parameters is technically feasible, as 725 operators for counting, sorting and paging are currently supported by 726 the major RDBMSs. 728 Similar operators are completely or partially supported by the most 729 known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so 730 the implementation of the new parameters seems to be practicable by 731 servers working without the use of an RDBMS. 733 6. Implementation Status 735 NOTE: Please remove this section and the reference to RFC 7942 prior 736 to publication as an RFC. 738 This section records the status of known implementations of the 739 protocol defined by this specification at the time of posting of this 740 Internet-Draft, and is based on a proposal described in RFC 7942 741 [RFC7942]. The description of implementations in this section is 742 intended to assist the IETF in its decision processes in progressing 743 drafts to RFCs. Please note that the listing of any individual 744 implementation here does not imply endorsement by the IETF. 745 Furthermore, no effort has been spent to verify the information 746 presented here that was supplied by IETF contributors. This is not 747 intended as, and must not be construed to be, a catalog of available 748 implementations or their features. Readers are advised to note that 749 other implementations may exist. 751 According to RFC 7942, "this will allow reviewers and working groups 752 to assign due consideration to documents that have the benefit of 753 running code, which may serve as evidence of valuable experimentation 754 and feedback that have made the implemented protocols more mature. 755 It is up to the individual working groups to use this information as 756 they see fit". 758 6.1. IIT-CNR/Registro.it 760 Responsible Organization: Institute of Informatics and Telematics 761 of National Research Council (IIT-CNR)/Registro.it 762 Location: https://rdap.pubtest.nic.it/ 763 Description: This implementation includes support for RDAP queries 764 using data from .it public test environment. 765 Level of Maturity: This is an "alpha" test implementation. 766 Coverage: This implementation includes all of the features 767 described in this specification. 768 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 770 6.2. Google Registry 772 Responsible Organization: Google Registry 773 Location: https://www.registry.google/rdap/ 774 Description: This implementation includes support for RDAP queries 775 for TLDs such as .google, .how, .soy, and others. The RDAP server 776 implements cursor pagination. The link used to request the next 777 page is included in the notice section of the response. 778 Level of Maturity: Production. 779 Coverage: This implementation includes the "cursor" parameter 780 described in this specification. 781 Contact Information: Brian Mountford, mountford@google.com 783 7. IANA Considerations 785 IANA is requested to register the following values in the RDAP 786 Extensions Registry: 788 Extension identifier: paging 789 Registry operator: Any 790 Published specification: This document. 791 Contact: IESG 792 Intended usage: This extension describes a best practice for 793 result set paging. 795 Extension identifier: sorting 796 Registry operator: Any 797 Published specification: This document. 798 Contact: IESG 799 Intended usage: This extension describes a best practice for 800 result set sorting. 802 8. Security Considerations 804 Security services for the operations specified in this document are 805 described in RFC 7481 [RFC7481]. 807 The search query typically requires more server resources (such as 808 memory, CPU cycles, and network bandwidth) when compared to the 809 lookup query. This increases the risk of server resource exhaustion 810 and subsequent denial of service due to abuse. This risk can be 811 mitigated by either restricting search functionality and limiting the 812 rate of search requests. Servers can also reduce their load by 813 truncating the results in the response. However, this last security 814 policy can result in a higher inefficiency if the RDAP server does 815 not provide any functionality to return the truncated results. 817 The new parameters presented in this document provide the RDAP 818 operators with a way to implement a secure server without penalizing 819 its efficiency. The "count" parameter gives the user a measure to 820 evaluate the query precision and, at the same time, returns a 821 significant information. The "sort" parameter allows the user to 822 obtain the most relevant information at the beginning of the result 823 set. In both cases, the user doesn't need to submit further 824 unnecessary search requests. Finally, the "cursor" parameter enables 825 the user to scroll the result set by submitting a sequence of 826 sustainable queries according to the server limits. 828 9. Acknowledgements 830 The authors would like to acknowledge Brian Mountford and Tom 831 Harrison for their contribution to the development of this document. 833 10. References 835 10.1. Normative References 837 [ISO.3166.1988] 838 International Organization for Standardization, "Codes for 839 the representation of names of countries, 3rd edition", 840 ISO Standard 3166, August 1988. 842 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 843 Requirement Levels", BCP 14, RFC 2119, 844 DOI 10.17487/RFC2119, March 1997, 845 . 847 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 848 IANA Considerations Section in RFCs", RFC 5226, 849 DOI 10.17487/RFC5226, May 2008, 850 . 852 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 853 Specifications: ABNF", STD 68, RFC 5234, 854 DOI 10.17487/RFC5234, January 2008, 855 . 857 [RFC5890] Klensin, J., "Internationalized Domain Names for 858 Applications (IDNA): Definitions and Document Framework", 859 RFC 5890, DOI 10.17487/RFC5890, August 2010, 860 . 862 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 863 DOI 10.17487/RFC6350, August 2011, 864 . 866 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 867 Protocol (HTTP/1.1): Message Syntax and Routing", 868 RFC 7230, DOI 10.17487/RFC7230, June 2014, 869 . 871 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 872 Registration Data Access Protocol (RDAP)", RFC 7480, 873 DOI 10.17487/RFC7480, March 2015, 874 . 876 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 877 Registration Data Access Protocol (RDAP)", RFC 7481, 878 DOI 10.17487/RFC7481, March 2015, 879 . 881 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 882 Protocol (RDAP) Query Format", RFC 7482, 883 DOI 10.17487/RFC7482, March 2015, 884 . 886 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 887 Registration Data Access Protocol (RDAP)", RFC 7483, 888 DOI 10.17487/RFC7483, March 2015, 889 . 891 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 892 Interchange Format", STD 90, RFC 8259, 893 DOI 10.17487/RFC8259, December 2017, 894 . 896 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 897 DOI 10.17487/RFC8288, October 2017, 898 . 900 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 901 ICANN Extensions for the Registration Data Access Protocol 902 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 903 . 905 10.2. Informative References 907 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 908 Pagination", July 2014, . 911 [CURSOR-API1] 912 facebook.com, "facebook for developers - Using the Graph 913 API", July 2017, . 916 [CURSOR-API2] 917 twitter.com, "Pagination", 2017, 918 . 921 [GOESSNER-JSON-PATH] 922 Goessner, S., "JSONPath - XPath for JSON", 2007, 923 . 925 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 926 . 929 [OData-Part1] 930 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 931 Part 1: Protocol Plus Errata 03", June 2016, 932 . 937 [REST] Fredrich, T., "RESTful Service Best Practices, 938 Recommendations for Creating Web Services", April 2012, 939 . 942 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 943 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 944 DOI 10.17487/RFC6901, April 2013, 945 . 947 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 948 Code: The Implementation Status Section", BCP 205, 949 RFC 7942, DOI 10.17487/RFC7942, July 2016, 950 . 952 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 953 With Limit and Offset is Slow?", July 2017, 954 . 957 [W3C.CR-xpath-31-20161213] 958 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 959 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 960 31-20161213, December 2016, 961 . 963 Appendix A. Approaches to Result Pagination 965 An RDAP query could return a response with hundreds, even thousands, 966 of objects, especially when partial matching is used. For that 967 reason, the cursor parameter addressing result pagination is defined 968 to make responses easier to handle. 970 Presently, the most popular methods to implement pagination in REST 971 API are: offset pagination and keyset pagination. Both two 972 pagination methods don't require the server to handle the result set 973 in a storage area across the requests since a new result set is 974 generated each time a request is submitted. Therefore, they are 975 preferred in comparison to any other method requiring the management 976 of a REST session. 978 Using limit and offset operators represents the traditionally used 979 method to implement results pagination. Both of them can be used 980 individually: 982 o "limit": means that the server must return the first N objects of 983 the result set; 985 o "offset": means that the server must skip the first N objects and 986 must return objects starting from position N+1. 988 When limit and offset are used together, they allow to identify a 989 specific portion of the result set. For example, the pair 990 "offset=100,limit=50" returns first 50 objects starting from position 991 101 of the result set. 993 Despite its easiness of implementation, offset pagination raises some 994 well known drawbacks: 996 o when offset has a very high value, scrolling the result set could 997 take some time; 999 o it always requires to fetch all the rows before dropping as many 1000 rows as specified by offset; 1002 o it may return inconsistent pages when data are frequently updated 1003 (i.e. real-time data) but this doesn't seem the case of 1004 registration data. 1006 The keyset pagination [SEEK] consists in adding a query condition 1007 that enables the selection of the only data not yet returned. This 1008 method has been taken as the basis for the implementation of a 1009 "cursor" parameter [CURSOR] by some REST API providers (e.g. 1010 [CURSOR-API1],[CURSOR-API2]). The cursor is an opaque URL-safe 1011 string representing a logical pointer to the first result of the next 1012 page (Figure 5). 1014 Nevertheless, even keyset pagination can be troublesome: 1016 o it needs at least one key field; 1018 o it does not allow to sort just by any field because the sorting 1019 criterion must contain a key; 1021 o it works best with full composite values support by DBMS (i.e. 1022 [x,y]>[a,b]), emulation is possible but ugly and less performant; 1024 o it does not allow to directly navigate to arbitrary pages because 1025 the result set must be scrolled in sequential order starting from 1026 the initial page; 1028 o implementing the bi-directional navigation is tedious because all 1029 comparison and sort operations have to be reversed. 1031 A.1. Specific Issues Raised by RDAP 1033 Furthermore, in the RDAP context, some additional considerations can 1034 be made: 1036 o an RDAP object is a conceptual aggregation of information 1037 generally collected from more than one data structure (e.g. table) 1038 and this makes even harder for the developers the implementation 1039 of the keyset pagination that is already quite difficult. For 1040 example, the entity object can gather information from different 1041 data structures (registrars, registrants, contacts, resellers, and 1042 so on), each one with its own key field mapping the RDAP entity 1043 handle; 1045 o depending on the number of the page results as well as the number 1046 and the complexity of the properties of each RDAP object in the 1047 response, the time required by offset pagination to skip the 1048 previous pages could be much faster than the processing time 1049 needed to build the current page. In fact, RDAP objects are 1050 usually formed by information belonging to multiple data 1051 structures and containing multivalued properties (i.e. arrays) 1052 and, therefore, data selection might be a time consuming process. 1053 This situation occurs even though the selection is supported by 1054 indexes; 1056 o depending on the access levels defined by each RDAP operator, the 1057 increase of complexity and the decrease of flexibility of keyset 1058 pagination with respect to the offset pagination could be 1059 considered impractical. 1061 Ultimately, both pagination methods have benefits and drawbacks. 1063 Appendix B. Change Log 1065 00: Initial working group version ported from draft-loffredo-regext- 1066 rdap-sorting-and-paging-05 1067 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 1068 consistent between the pagination methods. Renamed 1069 "Considerations about Paging Implementation" section in ""cursor" 1070 Parameter". Removed "FOR DISCUSSION" items. Provided a more 1071 detailed description of both "sorting_metadata" and 1072 "paging_metadata" elements. 1073 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1074 of cursor parameter. Rearranged the layout of some sections. 1075 Removed some items from "Informative References" section. Changed 1076 "IANA Considerations" section. 1077 03: Added "cc" to the list of sorting properties in "Sorting 1078 Properties Declaration" section. Added RFC8605 to the list of 1079 "Informative References". 1080 04: Replaced "ldhName" with "name" in the "Sorting Properties 1081 Declaration" section. Clarified the sorting logic with respect to 1082 the JSON value types and the sorting policy for multivalued 1083 fields. 1084 05: Clarified the logic of sorting on IP addresses. Clarified the 1085 mapping between the sorting properties and the RDAP fields. 1086 Updated "Acknowledgements" section. 1087 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1088 "paging_metadata" element. 1089 07: Added "Paging Responses to POST Requests" section. 1090 08: Added "Approaches to Result Pagination" section in the appendix. 1091 Added the case of requesting a sort on a property not included in 1092 the response to the errors listed in the "Negative Answers" 1093 section . 1095 Authors' Addresses 1097 Mario Loffredo 1098 IIT-CNR/Registro.it 1099 Via Moruzzi,1 1100 Pisa 56124 1101 IT 1103 Email: mario.loffredo@iit.cnr.it 1104 URI: http://www.iit.cnr.it 1106 Maurizio Martinelli 1107 IIT-CNR/Registro.it 1108 Via Moruzzi,1 1109 Pisa 56124 1110 IT 1112 Email: maurizio.martinelli@iit.cnr.it 1113 URI: http://www.iit.cnr.it 1115 Scott Hollenbeck 1116 Verisign Labs 1117 12061 Bluemont Way 1118 Reston, VA 20190 1119 USA 1121 Email: shollenbeck@verisign.com 1122 URI: https://www.verisignlabs.com/