idnits 2.17.1 draft-ietf-regext-rdap-sorting-and-paging-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 7, 2019) is 1662 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 486 -- Looks like a reference, but probably isn't: '1' on line 502 -- Looks like a reference, but probably isn't: '3' on line 503 -- Looks like a reference, but probably isn't: '6' on line 499 == Unused Reference: 'RFC5226' is defined on line 875, but no explicit reference was found in the text == Unused Reference: 'RFC8605' is defined on line 928, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) ** Downref: Normative reference to an Informational RFC: RFC 8605 Summary: 5 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: April 9, 2020 S. Hollenbeck 6 Verisign Labs 7 October 7, 2019 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-ietf-regext-rdap-sorting-and-paging-06 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing large 21 responses. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on April 9, 2020. 42 Copyright Notice 44 Copyright (c) 2019 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 64 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 65 2.3.1. Sorting Properties Declaration . . . . . . . . . . . 7 66 2.3.2. Representing Sorting Links . . . . . . . . . . . . . 11 67 2.4. "cursor" Parameter . . . . . . . . . . . . . . . . . . . 13 68 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 15 69 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 16 70 4. RDAP Conformance . . . . . . . . . . . . . . . . . . . . . . 17 71 5. Implementation Considerations . . . . . . . . . . . . . . . . 17 72 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 17 73 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 18 74 6.2. Google Registry . . . . . . . . . . . . . . . . . . . . . 18 75 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 76 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 77 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 19 80 10.2. Informative References . . . . . . . . . . . . . . . . . 21 81 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 22 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 84 1. Introduction 86 The availability of functionality for result sorting and paging 87 provides benefits to both clients and servers in the implementation 88 of RESTful services [REST]. These benefits include: 90 o reducing the server response bandwidth requirements; 91 o improving server response time; 92 o improving query precision and, consequently, obtaining more 93 reliable results; 94 o decreasing server query processing load; 95 o reducing client response processing time. 97 Approaches to implementing features for result sorting and paging can 98 be grouped into two main categories: 100 1. Sorting and paging are implemented through the introduction of 101 additional parameters in the query string (i.e. ODATA protocol 102 [OData-Part1]); 104 2. Information related to the number of results and the specific 105 portion of the result set to be returned, in addition to a set of 106 ready-made links for the result set scrolling, are inserted in 107 the HTTP header of the request/response. 109 However, there are some drawbacks associated with the use of the HTTP 110 header. First, the header properties cannot be set directly from a 111 web browser. Moreover, in an HTTP session, the information on the 112 status (i.e. the session identifier) is usually inserted in the 113 header or in the cookies, while the information on the resource 114 identification or the search type is included in the query string. 115 The second approach is therefore not compliant with the HTTP standard 116 [RFC7230]. As a result, this document describes a specification 117 based on the use of query parameters. 119 Currently, the RDAP protocol [RFC7482] defines two query types: 121 o lookup: the server returns only one object; 122 o search: the server returns a collection of objects. 124 While the lookup query does not raise issues in the response 125 management, the search query can potentially generate a large result 126 set that could be truncated according to the server limits. In 127 addition, it is not possible to obtain the total number of the 128 objects found that might be returned in a search query response 129 [RFC7483]. Lastly, there is no way to specify sort criteria to 130 return the most relevant objects at the beginning of the result set. 131 Therefore, the client might traverse the whole result set to find the 132 relevant objects or, due to truncation, could not find them at all. 134 The specification described in this document extends RDAP query 135 capabilities to enable result sorting and paging, by adding new query 136 parameters that can be applied to RDAP search path segments. The 137 service is implemented using the Hypertext Transfer Protocol (HTTP) 138 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 140 The implementation of the new parameters is technically feasible, as 141 operators for counting, sorting and paging rows are currently 142 supported by the major RDBMSs. 144 1.1. Conventions Used in This Document 146 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 147 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 148 document are to be interpreted as described in [RFC2119]. 150 2. RDAP Query Parameter Specification 152 The new query parameters are OPTIONAL extensions of path segments 153 defined in RFC 7482 [RFC7482]. They are as follows: 155 o "count": a boolean value that allows a client to request the total 156 number of objects found (that due to truncation can be different 157 from the number of returned objects); 159 o "sort": a string value that allows a client to request a specific 160 sort order for the result set; 162 o "cursor": a string value representing a pointer to a specific 163 fixed size portion of the result set. 165 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 166 sections to describe the formal syntax of these new parameters. 168 2.1. Sorting and Paging Metadata 170 According to most advanced principles in REST design, collectively 171 known as HATEOAS (Hypermedia as the Engine of Application State) 172 ([HATEOAS]), a client entering a REST application through an initial 173 URI should use the server-provided links to dynamically discover 174 available actions and access the resources it needs. In this way, 175 the client is not requested to have prior knowledge of the service 176 and, consequently, to hard code the URIs of different resources. 177 This would allow the server to make URI changes as the API evolves 178 without breaking the clients. Definitively, a REST service should be 179 as self-descriptive as possible. 181 Therefore, servers implementing the query parameters described in 182 this specification SHOULD provide additional information in their 183 responses about both the available sorting criteria and the possible 184 pagination. Such information is collected in two OPTIONAL response 185 elements named, respectively, "sorting_metadata" and 186 "paging_metadata". 188 The "sorting_metadata" element contains the following properties: 190 o "currentSort": "String" (OPTIONAL) either the value of sort 191 "parameter" as specified in the query string or the sort applied 192 by default, if any; 194 o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of objects 195 each one describing an alternate available sorting criterion. 196 Members are: 198 * "property": "String" (REQUIRED) the name that can be used by 199 the client to request the sorting criterion; 200 * "default": "Boolean" (REQUIRED) whether the sorting criterion 201 is applied by default; 202 * "jsonPath": "String" (OPTIONAL) the JSON Path of the RDAP field 203 corresponding to the property; 204 * "links": "Link[]" (OPTIONAL) an array of links as described in 205 RFC 8288 [RFC8288] containing the query string that applies the 206 sorting criterion. 208 At least one between "currentSort" and "availableSorts" MUST be 209 present. 211 The "paging_metadata" element contains the following fields: 213 o "totalCount": "Numeric" (OPTIONAL) a numeric value representing 214 the total number of objects found. It is provided if the query 215 string contains the "count" parameter; 217 o "pageSize": "Numeric" (OPTIONAL) a numeric value representing the 218 number of objects returned in the current page. It is provided 219 when the total number of objects exceeds the page size. This 220 property is redundant for clients because the page size can be 221 derived from the length of the search results array but it can be 222 helpful if the end user interacts with the server through a web 223 browser; 225 o "pageNumber": "Numeric" (OPTIONAL) a numeric value representing 226 the number of the current page in the result set.It is provided 227 when the total number of objects found exceeds the page size; 229 o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 230 8288 [RFC8288] containing the reference to the next page. In this 231 specification, only the forward pagination is dealt because it is 232 considered satisfactory in order to traverse the result set. 233 Examples of additional references are to: the previous page, the 234 first page, the last page. 236 At least one between "totalCount" and "links" MUST be present. 238 2.2. "count" Parameter 240 Currently, the RDAP protocol does not allow a client to determine the 241 total number of the results in a query response when the result set 242 is truncated. This is rather inefficient because the user cannot 243 evaluate the query precision and, at the same time, cannot receive 244 information that could be relevant. 246 The "count" parameter provides additional functionality (Figure 1) 247 that allows a client to request information from the server that 248 specifies the total number of elements matching the search pattern. 250 https://example.com/rdap/domains?name=*nr.com&count=true 252 Figure 1: Example of RDAP query reporting the "count" parameter 254 The ABNF syntax is the following: 256 count = "count=" ( trueValue / falseValue ) 257 trueValue = ("true" / "yes" / "1") 258 falseValue = ("false" / "no" / "0") 260 A trueValue means that the server MUST provide the total number of 261 the objects in the "totalCount" field of the "paging_metadata" 262 element (Figure 2). A falseValue means that the server MUST NOT 263 provide this number. 265 { 266 "rdapConformance": [ 267 "rdap_level_0", 268 "paging_level_0" 269 ], 270 ... 271 "paging_metadata": { 272 "totalCount": 43 273 }, 274 "domainSearchResults": [ 275 ... 276 ] 277 } 279 Figure 2: Example of RDAP response with "paging_metadata" element 280 containing the "totalCount" field 282 2.3. "sort" Parameter 284 The RDAP protocol does not provide any capability to specify results 285 sort criteria. A server could implement a default sorting scheme 286 according to the object class, but this feature is not mandatory and 287 might not meet user requirements. Sorting can be addressed by the 288 client, but this solution is rather inefficient. Sorting features 289 provided by the RDAP server could help avoid truncation of relevant 290 results. 292 The "sort" parameter allows the client to ask the server to sort the 293 results according to the values of one or more properties and 294 according to the sort direction of each property. The ABNF syntax is 295 the following: 297 sort = "sort=" sortItem *( "," sortItem ) 298 sortItem = property-ref [":" ( "a" / "d" ) ] 299 property-ref = ALPHA *( ALPHA / DIGIT / "_" ) 301 "a" means that the ascending sort MUST be applied, "d" means that the 302 descending sort MUST be applied. If the sort direction is absent, an 303 ascending sort MUST be applied (Figure 3). 305 https://example.com/rdap/domains?name=*nr.com&sort=name 307 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 309 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,name 311 Figure 3: Examples of RDAP query reporting the "sort" parameter 313 With the only exception of the sort on IP addresses, servers MUST 314 implement sorting according to the JSON value type of the RDAP field 315 the sorting property refers to: JSON strings MUST be sorted 316 lexicographically and JSON numbers MUST be sorted numerically. Even 317 if IP addresses are represented as JSON strings, they MUST be sorted 318 based on their numeric conversion. 320 If the "sort" parameter reports an allowed sorting property, it MUST 321 be provided in the "currentSort" field of the "sorting_metadata" 322 element. 324 2.3.1. Sorting Properties Declaration 326 In the "sort" parameter ABNF syntax, property-ref represents a 327 reference to a property of an RDAP object. Such a reference could be 328 expressed by using a JSON Path. The JSON Path in a JSON document 330 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 331 XML document. For example, the JSON Path to select the value of the 332 ASCII name inside an RDAP domain object is "$.ldhName", whereby $ 333 identifies the root of the document (DOM). Another way to select a 334 value inside a JSON document is the JSON Pointer [RFC6901]. While 335 JSON Path or JSON Pointer are both standard ways to select any value 336 inside JSON data, neither is particularly easy to use (e.g. 337 "$.events[?(@.eventAction='registration')].eventDate" is the JSON 338 Path expression of the registration date in an RDAP domain object). 340 Therefore, this specification provides a definition of property-ref 341 in terms of RDAP properties. However, not all the RDAP properties 342 are suitable to be used in sort criteria, such as: 344 o properties providing service information (e.g. links, notices, 345 remarks, etc.); 347 o multivalued properties (e.g. status, roles, variants, etc.); 349 o properties modeling relationships to other objects (e.g. 350 entities). 352 On the contrary, some properties expressed as values of other 353 properties (e.g. registration date) could be used in such a context. 355 In the following, a list of properties an RDAP server MAY implement 356 is presented. The properties are divided into two groups: object 357 common properties and object specific properties. 359 o Object common properties. Object common properties are derived 360 from the merge of the "eventAction" and the "eventDate" 361 properties. The following values of the "sort" parameter are 362 defined: 364 * registrationDate 365 * reregistrationDate 366 * lastChangedDate 367 * expirationDate 368 * deletionDate 369 * reinstantiationDate 370 * transferDate 371 * lockedDate 372 * unlockedDate 374 o Object specific properties. With regard to the specific 375 properties, some of them are already defined among the query 376 paths. In the following a list of possible sorting properties, 377 grouped by objects, is shown: 379 * Domain: name 380 * Nameserver: name, ipV4, ipV6. 381 * Entity: fn, handle, org, email, voice, country, cc, city. 383 The correspondence between the sorting properties and the RDAP fields 384 is shown in Table 1: 386 +-----------+-----------+---------------------+------+-------+------+ 387 | Object | Sorting | RDAP property | RFC | RFC | RFC | 388 | class | property | | 7483 | 6350 | 8605 | 389 +-----------+-----------+---------------------+------+-------+------+ 390 | Searchabl | Common pr | eventAction values | 4.5. | | | 391 | e objects | operties | suffixed by "Date" | | | | 392 | | | | | | | 393 | Domain | name | unicodeName/ldhName | 5.3. | | | 394 | | | | | | | 395 | Nameserve | name | unicodeName/ldhName | 5.2. | | | 396 | r | | | | | | 397 | | ipV4 | v4 ipAddress | 5.2. | | | 398 | | ipV6 | v6 ipAddress | 5.2. | | | 399 | | | | | | | 400 | Entity | handle | handle | 5.1. | | | 401 | | fn | vcard fn | 5.1. | 6.2.1 | | 402 | | org | vcard org | 5.1. | 6.6.4 | | 403 | | voice | vcard tel with | 5.1. | 6.4.1 | | 404 | | | type="voice" | | | | 405 | | email | vcard email | 5.1. | 6.4.2 | | 406 | | country | country name in | 5.1. | 6.3.1 | | 407 | | | vcard adr | | | | 408 | | cc | country code in | 5.1. | | 3.1 | 409 | | | vcard adr | | | | 410 | | city | locality in vcard | 5.1. | 6.3.1 | | 411 | | | adr | | | | 412 +-----------+-----------+---------------------+------+-------+------+ 414 Table 1: Sorting properties definition 416 With regard to the definitions in Table 1, some further 417 considerations must be made to disambiguate some cases: 419 o since the response to a search on either domains or nameservers 420 might include both A-labels and U-labels ([RFC5890]) in general, a 421 consistent sorting policy shall take unicodeName and ldhName as 422 two formats of the same value rather than separately. Therefore, 423 the unicodeName value MUST be taken while sorting, when 424 unicodeName is missing, the value of ldhName MUST be considered 425 instead; 427 o the jCard "sort-as" parameter MUST be ignored for the purpose of 428 the sorting capability as described in this document; 430 o even if a nameserver can have multiple IPv4 and IPv6 addresses, 431 the most common configuration includes one address for each IP 432 version. Therefore, the assumption of having a single IPv4 and/or 433 IPv6 value for a nameserver cannot be considered too stringent; 435 o with the exception of handle values, all the sorting properties 436 defined for entity objects can be multivalued according to the 437 definition of vCard as given in RFC6350 [RFC6350]. When more than 438 one value is reported, sorting will be applied to the preferred 439 value identified by the parameter pref="1". If the pref parameter 440 is missing, sorting will be applied to the first value. 442 Each RDAP provider MAY define other sorting properties than those 443 shown in this document as well as it MAY map those sorting properties 444 onto different locations. 446 The "jsonPath" field in the "sorting_metadata" element is used to 447 clarify the RDAP field the sorting property refers to. The mapping 448 between the sorting properties and the JSON Paths of the RDAP fields 449 is shown in Table 2. The JSON Paths are provided according to the 450 Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]): 452 +-------+-------------+---------------------------------------------+ 453 | Objec | Sorting | JSON Path | 454 | t | property | | 455 | class | | | 456 +-------+-------------+---------------------------------------------+ 457 | Searc | registratio | "$.domainSearchResults[*].events[?(@.eventA | 458 | hable | nDate | ction=="registration")].eventDate | 459 | objec | | | 460 | ts | | | 461 | | reregistrat | "$.domainSearchResults[*].events[?(@.eventA | 462 | | ionDate | ction=="reregistration")].eventDate | 463 | | lastChanged | "$.domainSearchResults[*].events[?(@.eventA | 464 | | Date | ction=="lastChanged")].eventDate | 465 | | expirationD | "$.domainSearchResults[*].events[?(@.eventA | 466 | | ate | ction=="expiration")].eventDate | 467 | | deletionDat | "$.domainSearchResults[*].events[?(@.eventA | 468 | | e | ction=="deletion")].eventDate | 469 | | reinstantia | "$.domainSearchResults[*].events[?(@.eventA | 470 | | tionDate | ction=="reinstantiation")].eventDate | 471 | | transferDat | "$.domainSearchResults[*].events[?(@.eventA | 472 | | e | ction=="transfer")].eventDate | 473 | | lockedDate | "$.domainSearchResults[*].events[?(@.eventA | 474 | | | ction=="locked")].eventDate | 475 | | unlockedDat | "$.domainSearchResults[*].events[?(@.eventA | 476 | | e | ction=="unlocked")].eventDate | 477 | | | | 478 | Domai | name | $.domainSearchResults[*].unicodeName | 479 | n | | | 480 | | | | 481 | Names | name | $.nameserverSearchResults[*].unicodeName | 482 | erver | | | 483 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 484 | | | [0] | 485 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 486 | | | [0] | 487 | | | | 488 | Entit | handle | $.entitySearchResults[*].handle | 489 | y | | | 490 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 491 | | | 0]="fn")][3] | 492 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 493 | | | 0]="org")][3] | 494 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 495 | | | 0]=="tel" && @[1].type=="voice")][3] | 496 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 497 | | | 0]=="email")][3] | 498 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 499 | | | 0]=="adr")][3][6] | 500 | | cc | $.entitySearchResults[*].vcardArray[1][?(@[ | 501 | | | 0]=="adr")][1].cc | 502 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 503 | | | 0]=="adr")][3][3] | 504 +-------+-------------+---------------------------------------------+ 506 Table 2: Sorting properties - JSON Path Mapping 508 2.3.2. Representing Sorting Links 510 An RDAP server MAY use the "links" array of the "sorting_metadata" 511 element to provide ready-made references [RFC8288] to the available 512 sort criteria (Figure 4). Each link represents a reference to an 513 alternate view of the results. 515 { 516 "rdapConformance": [ 517 "rdap_level_0", 518 "sorting_level_0" 519 ], 520 ... 521 "sorting_metadata": { 522 "currentSort": "name", 523 "availableSorts": [ 524 { 525 "property": "registrationDate", 526 "jsonPath": "$.domainSearchResults[*] 527 .events[?(@.eventAction==\"registration\")].eventDate", 528 "default": false, 529 "links": [ 530 { 531 "value": "https://example.com/rdap/domains?name=*nr.com 532 &sort=name", 533 "rel": "alternate", 534 "href": "https://example.com/rdap/domains?name=*nr.com 535 &sort=registrationDate", 536 "title": "Result Ascending Sort Link", 537 "type": "application/rdap+json" 538 }, 539 { 540 "value": "https://example.com/rdap/domains?name=*nr.com 541 &sort=name", 542 "rel": "alternate", 543 "href": "https://example.com/rdap/domains?name=*nr.com 544 &sort=registrationDate:d", 545 "title": "Result Descending Sort Link", 546 "type": "application/rdap+json" 547 } 548 ], 549 ... 550 }, 551 "domainSearchResults": [ 552 ... 553 ] 554 } 556 Figure 4: Example of a "sorting_metadata" instance to implement 557 result sorting 559 2.4. "cursor" Parameter 561 An RDAP query could return a response with hundreds, even thousands, 562 of objects, especially when partial matching is used. For that 563 reason, the cursor parameter addressing result pagination is defined 564 to make responses easier to handle. 566 Presently, the most popular methods to implement pagination in REST 567 API are: offset pagination and keyset pagination. Both two 568 pagination methods don't require the server to handle the result set 569 in a storage area across the requests since a new result set is 570 generated each time a request is submitted. Therefore, they are 571 preferred in comparison to any other method requiring the management 572 of a REST session. 574 Using limit and offset operators represents the traditionally used 575 method to implement results pagination. Both of them can be used 576 individually: 578 o "limit": means that the server must return the first N objects of 579 the result set; 581 o "offset": means that the server must skip the first N objects and 582 must return objects starting from position N+1. 584 When limit and offset are used together, they allow to identify a 585 specific portion of the result set. For example, the pair 586 "offset=100,limit=50" returns first 50 objects starting from position 587 101 of the result set. 589 Despite its easiness of implementation, offset pagination raises some 590 well known drawbacks: 592 o when offset has a very high value, scrolling the result set could 593 take some time; 595 o it always requires to fetch all the rows before dropping as many 596 rows as specified by offset; 598 o it may return inconsistent pages when data are frequently updated 599 (i.e. real-time data) but this doesn't seem the case of 600 registration data. 602 The keyset pagination [SEEK] consists in adding a query condition 603 that enables the selection of the only data not yet returned. This 604 method has been taken as the basis for the implementation of a 605 "cursor" parameter [CURSOR] by some REST API providers (e.g. 606 [CURSOR-API1],[CURSOR-API2]). The cursor is an opaque URL-safe 607 string representing a logical pointer to the first result of the next 608 page (Figure 5). 610 Nevertheless, even keyset pagination can be troublesome: 612 o it needs at least one key field; 614 o it does not allow to sort just by any field because the sorting 615 criterion must contain a key; 617 o it works best with full composite values support by DBMS (i.e. 618 [x,y]>[a,b]), emulation is possible but ugly and less performant; 620 o it does not allow to directly navigate to arbitrary pages because 621 the result set must be scrolled in sequential order starting from 622 the initial page; 624 o implementing the bi-directional navigation is tedious because all 625 comparison and sort operations have to be reversed. 627 Furthermore, in the RDAP context, some additional considerations can 628 be made: 630 o an RDAP object is a conceptual aggregation of information 631 generally collected from more than one data structure (e.g. table) 632 and this makes even harder for the developers the implementation 633 of the keyset pagination that is already quite difficult. For 634 example, the entity object can gather information from different 635 data structures (registrars, registrants, contacts, resellers, and 636 so on), each one with its own key field mapping the RDAP entity 637 handle; 639 o depending on the number of the page results as well as the number 640 and the complexity of the properties of each RDAP object in the 641 response, the time required by offset pagination to skip the 642 previous pages could be much faster than the processing time 643 needed to build the current page. In fact, RDAP objects are 644 usually formed by information belonging to multiple data 645 structures and containing multivalued properties (i.e. arrays) 646 and, therefore, data selection might be a time consuming process. 647 This situation occurs even though the selection is supported by 648 indexes; 650 o depending on the access levels defined by each RDAP operator, the 651 increase of complexity and the decrease of flexibility of keyset 652 pagination with respect to the offset pagination could be 653 considered impractical. 655 Ultimately, both pagination methods have benefits and drawbacks. 657 That said, the cursor parameter defined in this specification can be 658 used to encode information about any pagination method. For example, 659 in the case of a simple implementation of the cursor parameter to 660 represent offset pagination information, the cursor value 661 "b2Zmc2V0PTEwMCxsaW1pdD01MAo=" is the mere Base64 encoding of 662 "offset=100,limit=50". Likewise, in a simple implementation to 663 represent keyset pagination information, the cursor value 664 "a2V5PXRoZWxhc3Rkb21haW5vZnRoZXBhZ2UuY29t=" represents the mere 665 Base64 encoding of "key=thelastdomainofthepage.com" whereby the key 666 value identifies the last row of the current page. 668 This solution lets RDAP providers to implement a pagination method 669 according to their needs, the user access levels, the submitted 670 queries. In addition, servers can change the method over time 671 without announcing anything to the clients. 673 The ABNF syntax of the cursor paramter is the following: 675 cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" ) 677 https://example.com/rdap/domains?name=*nr.com 678 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= 680 Figure 5: An example of RDAP query reporting the "cursor" parameter 682 2.4.1. Representing Paging Links 684 An RDAP server SHOULD use the "links" array of the "paging_metadata" 685 element to provide a ready-made reference [RFC8288] to the next page 686 of the result set (Figure 6). Examples of additional "rel" values a 687 server MAY implements are "first", "last", "prev". 689 { 690 "rdapConformance": [ 691 "rdap_level_0", 692 "paging_level_0" 693 ], 694 ... 695 "notices": [ 696 { 697 "title": "Search query limits", 698 "type": "result set truncated due to excessive load", 699 "description": [ 700 "search results for domains are limited to 10" 701 ] 702 } 703 ], 704 "paging_metadata": { 705 "totalCount": 73, 706 "pageSize": 50, 707 "pageNumber": 1, 708 "links": [ 709 { 710 "value": "https://example.com/rdap/domains?name=*nr.com", 711 "rel": "next", 712 "href": "https://example.com/rdap/domains?name=*nr.com 713 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 714 "title": "Result Pagination Link", 715 "type": "application/rdap+json" 716 } 717 ] 718 }, 719 "domainSearchResults": [ 720 ... 721 ] 722 } 724 Figure 6: Example of a "paging_metadata" instance to implement cursor 725 pagination 727 3. Negative Answers 729 The value constraints for the parameters are defined by their ABNF 730 syntax. Therefore, each request including an invalid value for a 731 parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The 732 same response SHOULD be returned in the following cases: 734 o if the client provides an unsupported value for the "sort" 735 parameter in both single and multi sort; 737 o if the client submits an invalid value for the "cursor" parameter. 739 Optionally, the response MAY include additional information regarding 740 the negative answer in the HTTP entity body. 742 4. RDAP Conformance 744 Servers returning the "paging_metadata" element in their response 745 MUST include "paging_level_0" in the rdapConformance array as well as 746 servers returning the "sorting_metadata" element MUST include 747 "sorting_level_0". 749 5. Implementation Considerations 751 The implementation of the new parameters is technically feasible, as 752 operators for counting, sorting and paging are currently supported by 753 the major RDBMSs. 755 Similar operators are completely or partially supported by the most 756 known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so 757 the implementation of the new parameters seems to be practicable by 758 servers working without the use of an RDBMS. 760 6. Implementation Status 762 NOTE: Please remove this section and the reference to RFC 7942 prior 763 to publication as an RFC. 765 This section records the status of known implementations of the 766 protocol defined by this specification at the time of posting of this 767 Internet-Draft, and is based on a proposal described in RFC 7942 768 [RFC7942]. The description of implementations in this section is 769 intended to assist the IETF in its decision processes in progressing 770 drafts to RFCs. Please note that the listing of any individual 771 implementation here does not imply endorsement by the IETF. 772 Furthermore, no effort has been spent to verify the information 773 presented here that was supplied by IETF contributors. This is not 774 intended as, and must not be construed to be, a catalog of available 775 implementations or their features. Readers are advised to note that 776 other implementations may exist. 778 According to RFC 7942, "this will allow reviewers and working groups 779 to assign due consideration to documents that have the benefit of 780 running code, which may serve as evidence of valuable experimentation 781 and feedback that have made the implemented protocols more mature. 782 It is up to the individual working groups to use this information as 783 they see fit". 785 6.1. IIT-CNR/Registro.it 787 Responsible Organization: Institute of Informatics and Telematics 788 of National Research Council (IIT-CNR)/Registro.it 789 Location: https://rdap.pubtest.nic.it/ 790 Description: This implementation includes support for RDAP queries 791 using data from .it public test environment. 792 Level of Maturity: This is a "proof of concept" research 793 implementation. 794 Coverage: This implementation includes all of the features 795 described in this specification. 796 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 798 6.2. Google Registry 800 Responsible Organization: Google Registry 801 Location: https://www.registry.google/rdap/ 802 Description: This implementation includes support for RDAP queries 803 for TLDs such as .google, .how, .soy, and .xn--q9jyb4c . The RDAP 804 server implements cursor pagination. The link used to request the 805 next page is included in the notice section of the response. 806 Level of Maturity: Production. 807 Coverage: This implementation includes the "cursor" parameter 808 described in this specification. 809 Contact Information: Brian Mountford, mountford@google.com 811 7. IANA Considerations 813 IANA is requested to register the following values in the RDAP 814 Extensions Registry: 816 Extension identifier: paging 817 Registry operator: Any 818 Published specification: This document. 819 Contact: IESG 820 Intended usage: This extension describes a best practice for 821 result set paging. 823 Extension identifier: sorting 824 Registry operator: Any 825 Published specification: This document. 826 Contact: IESG 827 Intended usage: This extension describes a best practice for 828 result set sorting. 830 8. Security Considerations 832 Security services for the operations specified in this document are 833 described in RFC 7481 [RFC7481]. 835 The search query typically requires more server resources (such as 836 memory, CPU cycles, and network bandwidth) when compared to the 837 lookup query. This increases the risk of server resource exhaustion 838 and subsequent denial of service due to abuse. This risk can be 839 mitigated by either restricting search functionality and limiting the 840 rate of search requests. Servers can also reduce their load by 841 truncating the results in the response. However, this last security 842 policy can result in a higher inefficiency if the RDAP server does 843 not provide any functionality to return the truncated results. 845 The new parameters presented in this document provide the RDAP 846 operators with a way to implement a secure server without penalizing 847 its efficiency. The "count" parameter gives the user a measure to 848 evaluate the query precision and, at the same time, returns a 849 significant information. The "sort" parameter allows the user to 850 obtain the most relevant information at the beginning of the result 851 set. In both cases, the user doesn't need to submit further 852 unnecessary search requests. Finally, the "cursor" parameter enables 853 the user to scroll the result set by submitting a sequence of 854 sustainable queries according to the server limits. 856 9. Acknowledgements 858 The authors would like to acknowledge Brian Mountford and Tom 859 Harrison for their contribution to the development of this document. 861 10. References 863 10.1. Normative References 865 [ISO.3166.1988] 866 International Organization for Standardization, "Codes for 867 the representation of names of countries, 3rd edition", 868 ISO Standard 3166, August 1988. 870 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 871 Requirement Levels", BCP 14, RFC 2119, 872 DOI 10.17487/RFC2119, March 1997, 873 . 875 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 876 IANA Considerations Section in RFCs", RFC 5226, 877 DOI 10.17487/RFC5226, May 2008, 878 . 880 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 881 Specifications: ABNF", STD 68, RFC 5234, 882 DOI 10.17487/RFC5234, January 2008, 883 . 885 [RFC5890] Klensin, J., "Internationalized Domain Names for 886 Applications (IDNA): Definitions and Document Framework", 887 RFC 5890, DOI 10.17487/RFC5890, August 2010, 888 . 890 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 891 DOI 10.17487/RFC6350, August 2011, 892 . 894 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 895 Protocol (HTTP/1.1): Message Syntax and Routing", 896 RFC 7230, DOI 10.17487/RFC7230, June 2014, 897 . 899 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 900 Registration Data Access Protocol (RDAP)", RFC 7480, 901 DOI 10.17487/RFC7480, March 2015, 902 . 904 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 905 Registration Data Access Protocol (RDAP)", RFC 7481, 906 DOI 10.17487/RFC7481, March 2015, 907 . 909 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 910 Protocol (RDAP) Query Format", RFC 7482, 911 DOI 10.17487/RFC7482, March 2015, 912 . 914 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 915 Registration Data Access Protocol (RDAP)", RFC 7483, 916 DOI 10.17487/RFC7483, March 2015, 917 . 919 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 920 Interchange Format", STD 90, RFC 8259, 921 DOI 10.17487/RFC8259, December 2017, 922 . 924 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 925 DOI 10.17487/RFC8288, October 2017, 926 . 928 [RFC8605] Hollenbeck, S. and R. Carney, "vCard Format Extensions: 929 ICANN Extensions for the Registration Data Access Protocol 930 (RDAP)", RFC 8605, DOI 10.17487/RFC8605, May 2019, 931 . 933 10.2. Informative References 935 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 936 Pagination", July 2014, . 939 [CURSOR-API1] 940 facebook.com, "facebook for developers - Using the Graph 941 API", July 2017, . 944 [CURSOR-API2] 945 twitter.com, "Pagination", 2017, 946 . 949 [GOESSNER-JSON-PATH] 950 Goessner, S., "JSONPath - XPath for JSON", 2007, 951 . 953 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 954 . 957 [OData-Part1] 958 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 959 Part 1: Protocol Plus Errata 03", June 2016, 960 . 965 [REST] Fredrich, T., "RESTful Service Best Practices, 966 Recommendations for Creating Web Services", April 2012, 967 . 970 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 971 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 972 DOI 10.17487/RFC6901, April 2013, 973 . 975 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 976 Code: The Implementation Status Section", BCP 205, 977 RFC 7942, DOI 10.17487/RFC7942, July 2016, 978 . 980 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 981 With Limit and Offset is Slow?", July 2017, 982 . 985 [W3C.CR-xpath-31-20161213] 986 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 987 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 988 31-20161213, December 2016, 989 . 991 Appendix A. Change Log 993 00: Initial working group version ported from draft-loffredo-regext- 994 rdap-sorting-and-paging-05 995 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" 996 consistent between the pagination methods. Renamed 997 "Considerations about Paging Implementation" section in ""cursor" 998 Parameter". Removed "FOR DISCUSSION" items. Provided a more 999 detailed description of both "sorting_metadata" and 1000 "paging_metadata" elements. 1001 02: Removed both "offset" and "limit" parameters. Added ABNF syntax 1002 of cursor parameter. Rearranged the layout of some sections. 1003 Removed some items from "Informative References" section. Changed 1004 "IANA Considerations" section. 1005 03: Added "cc" to the list of sorting properties in "Sorting 1006 Properties Declaration" section. Added RFC8605 to the list of 1007 "Informative References". 1008 04: Replaced "ldhName" with "name" in the "Sorting Properties 1009 Declaration" section. Clarified the sorting logic with respect to 1010 the JSON value types and the sorting policy for multivalued 1011 fields. 1012 05: Clarified the logic of sorting on IP addresses. Clarified the 1013 mapping between the sorting properties and the RDAP fields. 1014 Updated "Acknowledgements" section. 1015 06: Renamed "pageCount" to "pageSize" and added "pageNumber" in the 1016 "paging_metadata" element. 1018 Authors' Addresses 1020 Mario Loffredo 1021 IIT-CNR/Registro.it 1022 Via Moruzzi,1 1023 Pisa 56124 1024 IT 1026 Email: mario.loffredo@iit.cnr.it 1027 URI: http://www.iit.cnr.it 1029 Maurizio Martinelli 1030 IIT-CNR/Registro.it 1031 Via Moruzzi,1 1032 Pisa 56124 1033 IT 1035 Email: maurizio.martinelli@iit.cnr.it 1036 URI: http://www.iit.cnr.it 1038 Scott Hollenbeck 1039 Verisign Labs 1040 12061 Bluemont Way 1041 Reston, VA 20190 1042 USA 1044 Email: shollenbeck@verisign.com 1045 URI: https://www.verisignlabs.com/