idnits 2.17.1 draft-loffredo-regext-rdap-sorting-and-paging-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 25 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: If pagination is implemented by using a cursor, both "offset" and "nextOffset" fields MUST not be included in the "paging_metadata" section. -- The document date (September 21, 2018) is 2036 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 469 -- Looks like a reference, but probably isn't: '1' on line 483 -- Looks like a reference, but probably isn't: '3' on line 484 -- Looks like a reference, but probably isn't: '6' on line 482 == Unused Reference: 'RFC5226' is defined on line 925, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7482 (Obsoleted by RFC 9082) ** Obsolete normative reference: RFC 7483 (Obsoleted by RFC 9083) Summary: 5 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Registration Protocols Extensions M. Loffredo 3 Internet-Draft M. Martinelli 4 Intended status: Standards Track IIT-CNR/Registro.it 5 Expires: March 25, 2019 S. Hollenbeck 6 Verisign Labs 7 September 21, 2018 9 Registration Data Access Protocol (RDAP) Query Parameters for Result 10 Sorting and Paging 11 draft-loffredo-regext-rdap-sorting-and-paging-05 13 Abstract 15 The Registration Data Access Protocol (RDAP) does not include core 16 functionality for clients to provide sorting and paging parameters 17 for control of large result sets. This omission can lead to 18 unpredictable server processing of queries and client processing of 19 responses. This unpredictability can be greatly reduced if clients 20 can provide servers with their preferences for managing response 21 values. This document describes RDAP query extensions that allow 22 clients to specify their preferences for sorting and paging result 23 sets. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on March 25, 2019. 42 Copyright Notice 44 Copyright (c) 2018 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 61 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 62 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 63 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 64 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 65 2.3.1. Representing Sorting Links . . . . . . . . . . . . . 11 66 2.4. "limit" and "offset" Parameters . . . . . . . . . . . . . 12 67 2.4.1. Representing Paging Links . . . . . . . . . . . . . . 13 68 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . . 14 69 4. RDAP Conformance . . . . . . . . . . . . . . . . . . . . . . 15 70 5. Implementation Considerations . . . . . . . . . . . . . . . . 15 71 5.1. Considerations about Paging Implementation . . . . . . . 16 72 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 19 73 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . . 19 74 6.2. Google Registry . . . . . . . . . . . . . . . . . . . . . 19 75 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 76 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 77 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 21 80 10.2. Informative References . . . . . . . . . . . . . . . . . 22 81 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 24 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 84 1. Introduction 86 The availability of functionality for result sorting and paging 87 provides benefits to both clients and servers in the implementation 88 of RESTful services [REST]. These benefits include: 90 o reducing the server response bandwidth requirements; 91 o improving server response time; 92 o improving query precision and, consequently, obtaining more 93 reliable results; 94 o decreasing server query processing load; 95 o reducing client response processing time. 97 Approaches to implementing features for result sorting and paging can 98 be grouped into two main categories: 100 1. Sorting and paging are implemented through the introduction of 101 additional parameters in the query string (i.e. ODATA protocol 102 [OData-Part1]); 104 2. Information related to the number of results and the specific 105 portion of the result set to be returned, in addition to a set of 106 ready-made links for the result set scrolling, are inserted in 107 the HTTP header of the request/response. 109 However, there are some drawbacks associated with use of the HTTP 110 header. First, the header properties cannot be set directly from a 111 web browser. Moreover, in an HTTP session, the information on the 112 status (i.e. the session identifier) is usually inserted in the 113 header or in the cookies, while the information on the resource 114 identification or the search type is included in the query string. 115 The second approach is therefore not compliant with the HTTP standard 116 [RFC7230]. As a result, this document describes a specification 117 based on use of query parameters. 119 Currently the RDAP protocol [RFC7482] defines two query types: 121 o lookup: the server returns only one object; 122 o search: the server returns a collection of objects. 124 While the lookup query does not raise issues in the management of 125 large result sets, the search query can potentially generate a large 126 result set that could be truncated according to the limits of the 127 server. In addition, it is not possible to obtain the total number 128 of the objects found that might be returned in a search query 129 response [RFC7483]. Lastly, there is no way to specify sort criteria 130 to return the most relevant objects at the beginning of the result 131 set. Therefore, the client could traverse the whole result set to 132 find the relevant objects or, due to truncation, could not find them 133 at all. 135 The protocol described in this specification extends RDAP query 136 capabilities to enable result sorting and paging, by adding new query 137 parameters that can be applied to RDAP search path segments. The 138 service is implemented using the Hypertext Transfer Protocol (HTTP) 139 [RFC7230] and the conventions described in RFC 7480 [RFC7480]. 141 The implementation of these parameters is technically feasible, as 142 operators for counting, sorting and paging rows are currently 143 supported by the major RDBMSs. 145 1.1. Conventions Used in This Document 147 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 148 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 149 document are to be interpreted as described in [RFC2119]. 151 2. RDAP Query Parameter Specification 153 The new query parameters are OPTIONAL extensions of path segments 154 defined in RFC 7482 [RFC7482]. They are as follows: 156 o "count": a boolean value that allows a client to request the total 157 number of objects found (that due to truncation can be different 158 from the number of returned objects); 160 o "sort": a string value that allows a client to request a specific 161 sort order for the result set; 163 o "limit" and "offset": numeric values that allow a client to 164 request a specific portion of the entire result set. 166 Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following 167 sections to describe the formal syntax of these new parameters. 169 2.1. Sorting and Paging Metadata 171 According to most advanced principles in REST design, collectively 172 known as HATEOAS (Hypermedia as the Engine of Application State) 173 ([HATEOAS]), a client entering a REST application through an initial 174 URI should use the server-provided links to dynamically discover 175 available actions and access the resources it needs. In this way, 176 the client is not requested to have prior knowledge of the service 177 and, consequently, to hard code the URIs of different resources. 178 This would allow the server to make URI changes as the API evolves 179 without breaking the clients. Definitively, a REST service should be 180 self-descriptive as much as possible. 182 Therefore, the implementation of the query parameters described in 183 this specification recommends servers to provide additional 184 information in their responses about both the available sorting 185 criteria and the possible pagination. Such information is collected 186 in two new data structures named, respectively, "sorting_metadata" 187 and "paging_metadata". 189 Obviously, both the new data structures are OPTIONAL because their 190 presence in the response not only depends on the implementation of 191 sorting and paging query capabilities but also on some situations 192 related to the results. For example, it is quite natural to expect 193 that the "paging_metadata" section will not be present at the last 194 result page when the server implements only the forward pagination. 196 The "sorting_metadata" structure contains the following fields: 198 o "currentSort": the value of sort parameter as specified in the 199 query string; 201 o "availableSorts": an array of objects each one describing an 202 available sorting criterion: 204 * "property": the name that can be used by the client to request 205 the sorting criterion; 206 * "jsonPath": the JSON Path of the RDAP field corresponding to 207 the property; 208 * "default": whether the sorting criterion is applied by default; 209 * "links": an array of links as described in RFC 8288 [RFC8288] 210 containing the query string that applies the sorting criterion. 212 Both "currentSort" and "availableSorts" are OPTIONAL fields of the 213 "sorting_metadata" structure. In particular, the "currentSort" field 214 is provided when the query string contains a valid value for sort 215 parameter, while the "availableSorts" field SHOULD be provided when 216 the sort parameter is missing in the query string or when it is 217 present and the server implements more than a sorting criterion for 218 the RDAP object. At least the "property" field is REQUIRED in each 219 item of the "availableSorts" array while the other fields are 220 RECOMMENDED. 222 The "paging_metadata" structure contains the following fields: 224 o "totalCount": a numeric value representing the total number of 225 objects found; 227 o "pageCount": a numeric value representing the number of objects 228 returned in the current page; 230 o "offset": a numeric value identifying the start of current page in 231 the result set; 233 o "nextOffset": a numeric value identifying the start of the next 234 page in the result set or null if the result set has been 235 completely scrolled; 237 o "links": an array of links as described in RFC 8288 [RFC8288] 238 containing the reference to next page. 240 Only the "pageCount" field is REQUIRED in the "paging_metadata" 241 structure. The other fields appear when pagination occurs. In this 242 specification, only the forward pagination is dealt because it is 243 considered satisfactory in order to traverse the result set. If a 244 server should also implement backward pagination, an appropriate 245 field (e.g. "prevOffset") identifying the start of the previous page 246 is RECOMMENDED. Finally, the "totalCount" field is provided if the 247 query string contains the count parameter. 249 FOR DISCUSSION: Should the metadata described in this specification 250 be part of a more general "metadata" section including other contents 251 (e.g rate limits, information about the server, information about the 252 response, metadata information related to other parameters)? 254 2.2. "count" Parameter 256 Currently the RDAP protocol does not allow a client to determine the 257 total number of the results in a query response when the result set 258 is truncated. This is rather inefficient because the user cannot 259 evaluate the query precision and, at the same time, cannot receive 260 information that could be relevant. 262 The count parameter provides additional functionality (Figure 1) that 263 allows a client to request information from the server that specifies 264 the total number of elements matching a particular search pattern. 266 https://example.com/rdap/domains?name=*nr.com&count=true 268 Figure 1: Example of RDAP query reporting the count parameter 270 The ABNF syntax is the following: 272 count = "count" EQ ( trueValue / falseValue ) 273 trueValue = ("true" / "yes" / "1") 274 falseValue = ("false" / "no" / "0") 275 EQ = "=" 277 A trueValue means that the server MUST provide the total number of 278 the objects in the "totalCount" field of the "paging_metadata" 279 section (Figure 2). A falseValue means that the server MUST NOT 280 provide this number. 282 { 283 "rdapConformance": [ 284 "rdap_level_0", 285 "paging_level_0" 286 ], 287 ... 288 "paging_metadata": { 289 "totalCount": 73 290 }, 291 "domainSearchResults": [ 292 ... 293 ] 294 } 296 Figure 2: Example of RDAP response with "paging_metadata" section 297 containing the "totalCount" field 299 2.3. "sort" Parameter 301 The RDAP protocol does not provide any capability to specify results 302 sort criteria. A server could implement a default sorting scheme 303 according to the object class, but this feature is not mandatory and 304 might not meet user requirements. Sorting can be addressed by the 305 client, but this solution is rather inefficient. Sorting and paging 306 features provided by the RDAP server could help avoid truncation of 307 relevant results and allow for scrolling the result set using 308 subsequent queries. 310 The sort parameter allows the client to ask the server to sort the 311 results according to the values of one or more properties and 312 according to the sort direction of each property. The ABNF syntax is 313 the following: 315 sort = "sort" EQ sortItem *( "," sortItem ) 316 sortItem = property-ref [":" ( "a" / "d" ) ] 318 "a" means that the ascending sort MUST be applied, "d" means that the 319 descending sort MUST be applied. If the sort direction is absent, an 320 ascending sort MUST be applied (Figure 3). 322 In the sort parameter ABNF syntax, property-ref represents a 323 reference to a property of an RDAP object. Such a reference could be 324 expressed by using a JSON Path. The JSON Path in a JSON document 325 [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a 326 XML document. For example, the JSON Path to select the value of the 327 ASCII name inside an RDAP domain object is "$.ldhName", where $ 328 identifies the root of the document (DOM). Another way to select a 329 value inside a JSON document is the JSON Pointer [RFC6901]. While 330 JSON Path or JSON Pointer are both standard ways to select any value 331 inside JSON data, neither is particularly easy to use (e.g. 332 "$.events[?(@.eventAction='registration')].eventDate" is the JSON 333 Path expression of the registration date in a RDAP domain object). 335 Therefore, this specification provides a definition of property-ref 336 in terms of RDAP properties. However, not all the RDAP properties 337 are suitable to be used in sort criteria, such as: 339 o properties providing service information (e.g. links, notices, 340 remarks, etc.); 341 o multivalued properties (e.g. status, roles, variants, etc.); 342 o properties modeling relationships to other objects (e.g. 343 entities). 345 On the contrary, some properties expressed as values of other 346 properties (e.g. registration date) could be used in such a context. 348 In the following, a list of the proposed properties for sort criteria 349 is presented. The properties are divided in two groups: object 350 common properties and object specific properties. 352 o Object common properties. Object common properties are derived 353 from the merge of the "eventAction" and the "eventDate" 354 properties. The following values of the sort parameter are 355 defined: 357 * registrationDate 358 * reregistrationDate 359 * lastChangedDate 360 * expirationDate 361 * deletionDate 362 * reinstantiationDate 363 * transferDate 364 * lockedDate 365 * unlockedDate 367 o Object specific properties. With regard to the specific 368 properties, some of them are already defined among the query 369 paths. In the following the list of the proposed sorting 370 properties, grouped by objects, is shown: 372 * Domain: ldhName 373 * Nameserver: ldhName, ipV4, ipV6. 374 * Entity: fn, handle, org, email, voice, country, city. 376 In the following, the correspondence between the sorting properties 377 and the RDAP fields is shown (Table 1): 379 +------------+------------+---------------+------------+------------+ 380 | Object | Sorting | RDAP property | Reference | Reference | 381 | class | property | | in RFC | in RFC | 382 | | | | 7483 | 6350 | 383 +------------+------------+---------------+------------+------------+ 384 | Searchable | Common | eventAction | 4.5. | | 385 | objects | properties | values | | | 386 | | | suffixed by | | | 387 | | | "Date" | | | 388 | | | | | | 389 | Domain | ldhName | ldhName | 5.3. | | 390 | | | | | | 391 | Nameserver | ldhName | ldhName | 5.2. | | 392 | | ipV4 | v4 ipAddress | 5.2. | | 393 | | ipV6 | v6 ipAddress | 5.2. | | 394 | | | | | | 395 | Entity | handle | handle | 5.1. | | 396 | | fn | vcard fn | 5.1. | 6.2.1 | 397 | | org | vcard org | 5.1. | 6.6.4 | 398 | | voice | vcard tel | 5.1. | 6.4.1 | 399 | | | with | | | 400 | | | type="voice" | | | 401 | | email | vcard email | 5.1. | 6.4.2 | 402 | | country | country name | 5.1. | 6.3.1 | 403 | | | in vcard adr | | | 404 | | city | locality in | 5.1. | 6.3.1 | 405 | | | vcard adr | | | 406 +------------+------------+---------------+------------+------------+ 408 Table 1: Sorting properties definition 410 With regard to the definitions in Table 1, some further 411 considerations must be made to disambiguate cases where the RDAP 412 property is multivalued: 414 o Even if a nameserver can have multiple IPv4 and IPv6 addresses, 415 the most common configuration includes one address for each IP 416 version. Therefore, the assumption of having a single IPv4 and/or 417 IPv6 value for a nameserver cannot be considered too stringent. 419 o With the exception of handle values, all the sorting properties 420 defined for entity objects can be multivalued according to the 421 definition of vCard as given in RFC6350 [RFC6350]. When more than 422 a value is reported, sorting can be applied to the preferred value 423 identified by the parameter pref="1". 425 Each RDAP provider MAY define other sorting properties than those 426 shown in this document. 428 The "jsonPath" field in the "sorting_metadata" section is used to 429 clarify the RDAP field the sorting property refers to. In the 430 following, the mapping between the sorting properties and the JSON 431 Paths of the RDAP fields is shown (Table 2). The JSON Paths are 432 provided according to the Goessner v.0.8.0 specification 433 ([GOESSNER-JSON-PATH]): 435 +-------+-------------+---------------------------------------------+ 436 | Objec | Sorting | JSON Path | 437 | t | property | | 438 | class | | | 439 +-------+-------------+---------------------------------------------+ 440 | Searc | registratio | "$.domainSearchResults[*].events[?(@.eventA | 441 | hable | nDate | ction=="registration")].eventDate | 442 | objec | | | 443 | ts | | | 444 | | reregistrat | "$.domainSearchResults[*].events[?(@.eventA | 445 | | ionDate | ction=="reregistration")].eventDate | 446 | | lastChanged | "$.domainSearchResults[*].events[?(@.eventA | 447 | | Date | ction=="lastChanged")].eventDate | 448 | | expirationD | "$.domainSearchResults[*].events[?(@.eventA | 449 | | ate | ction=="expiration")].eventDate | 450 | | deletionDat | "$.domainSearchResults[*].events[?(@.eventA | 451 | | e | ction=="deletion")].eventDate | 452 | | reinstantia | "$.domainSearchResults[*].events[?(@.eventA | 453 | | tionDate | ction=="reinstantiation")].eventDate | 454 | | transferDat | "$.domainSearchResults[*].events[?(@.eventA | 455 | | e | ction=="transfer")].eventDate | 456 | | lockedDate | "$.domainSearchResults[*].events[?(@.eventA | 457 | | | ction=="locked")].eventDate | 458 | | unlockedDat | "$.domainSearchResults[*].events[?(@.eventA | 459 | | e | ction=="unlocked")].eventDate | 460 | | | | 461 | Domai | ldhName | $.domainSearchResults[*].ldhName | 462 | n | | | 463 | | | | 464 | Names | ldhName | $.nameserverSearchResults[*].ldhName | 465 | erver | | | 466 | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | 467 | | | [0] | 468 | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | 469 | | | [0] | 470 | | | | 471 | Entit | handle | $.entitySearchResults[*].handle | 472 | y | | | 473 | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | 474 | | | 0]="fn")][3] | 475 | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | 476 | | | 0]="org")][3] | 477 | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | 478 | | | 0]=="tel" && @[1].type=="voice")][3] | 479 | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | 480 | | | 0]=="email")][3] | 481 | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | 482 | | | 0]=="adr")][3][6] | 483 | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | 484 | | | 0]=="adr")][3][3] | 485 +-------+-------------+---------------------------------------------+ 487 Table 2: Sorting properties - JSON Path Mapping 489 If the sort parameter reports an allowed sorting property, it MUST be 490 provided in the "currentSort" field of the "sorting_metadata" 491 structure. 493 https://example.com/rdap/domains?name=*nr.com&sort=ldhName 495 https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d 497 https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,ldhName 499 Figure 3: Examples of RDAP query reporting the sort parameter 501 2.3.1. Representing Sorting Links 503 An RDAP server MAY use the "links" array of the "sorting_metadata" 504 section to provide ready-made references [RFC8288] to the available 505 sort criteria (Figure 4). Each link represents a reference to an 506 alternate view of the results. 508 { 509 "rdapConformance": [ 510 "rdap_level_0", 511 "sorting_level_0" 512 ], 513 ... 514 "sorting_metadata": { 515 "currentSort": "ldhName", 516 "availableSorts": [ 517 { 518 "property": "registrationDate", 519 "jsonPath": "$.domainSearchResults[*].events[?(@.eventAction==\"registration\")].eventDate", 520 "default": false, 521 "links": [ 522 { 523 "value": "https://example.com/rdap/domains?name=*nr.com 524 &sort=ldhName", 525 "rel": "alternate", 526 "href": "https://example.com/rdap/domains?name=*nr.com 527 &sort=registrationDate", 528 "title": "Result Ascending Sort Link", 529 "type": "application/rdap+json" 530 }, 531 { 532 "value": "https://example.com/rdap/domains?name=*nr.com 533 &sort=ldhName", 534 "rel": "alternate", 535 "href": "https://example.com/rdap/domains?name=*nr.com 536 &sort=registrationDate:d", 537 "title": "Result Descending Sort Link", 538 "type": "application/rdap+json" 539 } 540 ] 541 }, 542 "domainSearchResults": [ 543 ... 544 ] 545 } 547 Figure 4: Example of a "sorting_metadata" instance to implement 548 result sorting 550 2.4. "limit" and "offset" Parameters 552 An RDAP query could return a response with hundreds of objects, 553 especially when partial matching is used. For that reason, two 554 parameters addressing result pagination are defined to make responses 555 easier to handle: 557 o "limit": means that the server MUST return the first N objects of 558 the result set in the response; 559 o "offset": means that the server MUST skip the first N objects and 560 MUST return objects starting from position N+1. 562 The ABNF syntax is the following: 564 EQ = "=" 565 limit = "limit" EQ positive-number 566 offset = "offset" EQ positive-number 567 positive-number = non-zero-digit *digit 568 non-zero-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / 569 "9" 570 digit = "0" / non-zero-digit 572 When limit and offset are used together, they allow implementation of 573 result pagination. The following examples illustrate requests to 574 return, respectively, the first 5 objects, the set of objects 575 starting from position 6, and first 5 objects starting from position 576 11 of the result set (Figure 5). 578 https://example.com/rdap/domains?name=*nr.com&limit=5 580 https://example.com/rdap/domains?name=*nr.com&offset=5 582 https://example.com/rdap/domains?name=*nr.com&limit=5&offset=10 584 Figure 5: Examples of RDAP query reporting the limit and offset 585 parameters 587 2.4.1. Representing Paging Links 589 An RDAP server MAY use the "links" array of the "paging_metadata" 590 section to provide a ready-made reference [RFC8288] to the next page 591 of the result set (Figure 6). Examples of additional "rel" values 592 are "first", "last", "prev". 594 { 595 "rdapConformance": [ 596 "rdap_level_0", 597 "paging_level_0" 598 ], 599 ... 600 "notices": [ 601 { 602 "title": "Search query limits", 603 "type": "result set truncated due to excessive load", 604 "description": [ 605 "search results for domains are limited to 10" 606 ] 607 } 608 ], 609 "paging_metadata": { 610 "totalCount": 73, 611 "pageCount": 10, 612 "offset": 10, 613 "nextOffset": 20, 614 "links": [ 615 { 616 "value": "https://example.com/rdap/domains?name=*nr.com", 617 "rel": "next", 618 "href": "https://example.com/rdap/domains?name=*nr.com&limit=10 619 &offset=10", 620 "title": "Result Pagination Link", 621 "type": "application/rdap+json" 622 } 623 ] 624 }, 625 "domainSearchResults": [ 626 ... 627 ] 628 } 630 Figure 6: Example of a "paging_metadata" instance to implement result 631 pagination based on offset and limit 633 3. Negative Answers 635 The value constraints for the parameters are defined by their ABNF 636 syntax. Therefore, each request providing an invalid value for a 637 parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The 638 same response SHOULD be returned if the client provides an 639 unsupported value for the sort parameter in both single and multi 640 sort. 642 The server can provide a different response when it supports the 643 limit and/or offset parameters and the client submits values that are 644 out of the valid ranges. The possible cases are: 646 o If the client submits a value for the limit parameter that is 647 greater than the number of objects to be processed, it is 648 RECOMMENDED that server returns a response including only the 649 processed objects. 651 o If the client submits a value for the offset parameter that is 652 greater than the number of objects to be processed, it is 653 RECOMMENDED that server returns an HTTP 404 (Not Found) response 654 code. 656 Optionally, the response MAY include additional information regarding 657 the negative answer in the HTTP entity body. 659 4. RDAP Conformance 661 Servers returning the "paging_metadata" section in their responses 662 MUST include "paging_level_0" in the rdapConformance array as well as 663 servers returning the "sorting_metadata" section MUST include 664 "sorting_level_0". 666 5. Implementation Considerations 668 The implementation of the new parameters is technically feasible, as 669 operators for counting, sorting and paging are currently supported by 670 the major RDBMSs. 672 In the following, the match between the new defined parameters and 673 the SQL operators is shown (Table 3): 675 +----------------+--------------------------------------------------+ 676 | New query | SQL operator | 677 | parameter | | 678 +----------------+--------------------------------------------------+ 679 | count | count(*) query without offset, limit and order | 680 | | by | 681 | | [MYSQL-COUNT],[POSTGRES-COUNT],[ORACLE-COUNT] | 682 | | | 683 | sort | order by | 684 | | [MYSQL-SORT],[POSTGRES-SORT],[ORACLE-SORT] | 685 | | | 686 | limit | limit n (in MySql [MYSQL-LIMIT] and Postgres | 687 | | [POSTGRES-LIMIT]) | 688 | | FETCH FIRST n ROWS ONLY (in Oracle | 689 | | [ORACLE-LIMIT]) | 690 | | | 691 | offset | offset m (in Postgres) | 692 | | OFFSET m ROWS (in Oracle) | 693 | | | 694 | limit + offset | limit n offset m (in MySql and Postgres) | 695 | | OFFSET m ROWS FETCH NEXT n ROWS ONLY (in Oracle) | 696 +----------------+--------------------------------------------------+ 698 Table 3: New query parameters vs. SQL operators 700 With regard to Oracle, Table 3 reports only one of the three methods 701 that can be used to implement limit and offset parameters. The 702 others are described in [ORACLE-ROWNUM] and [ORACLE-ROW-NUMBER]. 704 In addition, similar operators are completely or partially supported 705 by the most known NoSQL databases (MongoDB, CouchDB, HBase, 706 Cassandra, Hadoop) so the implementation of the new parameters seems 707 to be practicable by servers working without the use of an RDBMS. 709 5.1. Considerations about Paging Implementation 711 The use of limit and offset operators represents the most common way 712 to implement results pagination. However, when offset has a high 713 value, scrolling the result set could take some time. In addition, 714 offset pagination may return inconsistent pages when data are 715 frequently updated (i.e. real-time data) but this is not the case of 716 registration data. An alternative approach to offset pagination is 717 the keyset pagination, a.k.a. seek-method [SEEK] or cursor based 718 pagination. This method has been taken as the basis for the 719 implementation of a cursor parameter [CURSOR] by some REST API 720 providers (e.g. [CURSOR-API1],[CURSOR-API2]). The cursor parameter 721 is an opaque URL-safe string representing a logical pointer to the 722 first result of the next page (Figure 7). 724 { 725 "rdapConformance": [ 726 "rdap_level_0", 727 "paging_level_0" 728 ], 729 ... 730 "notices": [ 731 { 732 "title": "Search query limits", 733 "type": "result set truncated due to excessive load", 734 "description": [ 735 "search results for domains are limited to 10" 736 ] 737 } 738 ], 739 "paging_metadata": { 740 "totalCount": 73, 741 "pageCount": 10, 742 "links": [ 743 { 744 "value": "https://example.com/rdap/domains?name=*nr.com", 745 "rel": "next", 746 "href": "https://example.com/rdap/domains?name=*nr.com&limit=10 747 &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", 748 "title": "Result Pagination Link", 749 "type": "application/rdap+json" 750 } 751 ] 752 }, 753 "domainSearchResults": [ 754 ... 755 ] 756 } 758 Figure 7: Example of a "paging_metadata" instance to implement keyset 759 pagination 761 But keyset pagination raises some drawbacks with respect to offset 762 pagination: 764 o it needs at least one key field; 765 o it does not allow to sort by any field and paginate the results 766 because sorting has to be made on the key field; 767 o it does not allow to skip pages because they have to be scrolled 768 in sequential order starting from the initial page; 769 o it makes very hard the navigation of the result set in both 770 directions because all comparison and sort operations have to be 771 reversed. 773 Furthermore, in the RDAP context, some additional considerations can 774 be made: 776 o an RDAP object is a conceptual aggregation of information 777 collected from more than one data structure (e.g. table) and this 778 makes even harder for the developers the implementation of the 779 seek-method that is already quite difficult. In fact, for 780 example, the entity object can gather information from different 781 data structures (registrars, registrants, contacts, resellers, and 782 so on), each one with its own key field mapping the RDAP entity 783 handle; 785 o depending on the number of the page results as well as the number 786 and the complexity of the properties of each RDAP object in the 787 response, the time required by offset pagination to skip the 788 previous pages could be much faster than the processing time 789 needed to build the current page. In fact, RDAP objects are 790 usually formed by information belonging to multiple data 791 structures and containing multivalued properties (e.g. arrays) 792 and, therefore, data selection is a time consuming process. This 793 situation occurs even though the data selection process makes use 794 of indexes; 796 o depending on the access levels defined by each RDAP operator, the 797 increase of complexity and the decrease of flexibility of keyset 798 pagination with respect to the offset pagination could be 799 considered impractical. 801 Finally, the keyset pagination is not fully compliant with the 802 additional RDAP capabilities proposed by this document. In fact, the 803 presence of a possible cursor parameter does not seem to be 804 consistent with both the sorting capability and the possibility to 805 implement additional ready-made links besides the classic "next page" 806 link. But, while the provisioning of more paging links can be 807 superfluous, dropping the sorting capability seems quite 808 unreasonable. 810 If pagination is implemented by using a cursor, both "offset" and 811 "nextOffset" fields MUST not be included in the "paging_metadata" 812 section. 814 FOR DISCUSSION: Should RDAP specification reports both offset and 815 cursor parameters and let operators to implement pagination according 816 to their needs, the user access levels, the submitted queries? 818 6. Implementation Status 820 NOTE: Please remove this section and the reference to RFC 7942 prior 821 to publication as an RFC. 823 This section records the status of known implementations of the 824 protocol defined by this specification at the time of posting of this 825 Internet-Draft, and is based on a proposal described in RFC 7942 826 [RFC7942]. The description of implementations in this section is 827 intended to assist the IETF in its decision processes in progressing 828 drafts to RFCs. Please note that the listing of any individual 829 implementation here does not imply endorsement by the IETF. 830 Furthermore, no effort has been spent to verify the information 831 presented here that was supplied by IETF contributors. This is not 832 intended as, and must not be construed to be, a catalog of available 833 implementations or their features. Readers are advised to note that 834 other implementations may exist. 836 According to RFC 7942, "this will allow reviewers and working groups 837 to assign due consideration to documents that have the benefit of 838 running code, which may serve as evidence of valuable experimentation 839 and feedback that have made the implemented protocols more mature. 840 It is up to the individual working groups to use this information as 841 they see fit". 843 6.1. IIT-CNR/Registro.it 845 Responsible Organization: Institute of Informatics and Telematics 846 of National Research Council (IIT-CNR)/Registro.it 847 Location: https://rdap.pubtest.nic.it/ 848 Description: This implementation includes support for RDAP queries 849 using data from the public test environment of .it ccTLD. The 850 RDAP server does not implement any security policy because data 851 returned by this server are only for experimental testing 852 purposes. The RDAP server implements both offset and cursor based 853 pagination (the latter only when sort and offset parameters are 854 not present in the query string). 855 Level of Maturity: This is a "proof of concept" research 856 implementation. 857 Coverage: This implementation includes all of the features 858 described in this specification. 859 Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 861 6.2. Google Registry 863 Responsible Organization: Google Registry 864 Location: https://www.registry.google/rdap/ 865 Description: This implementation includes support for RDAP queries 866 for TLDs such as .GOOGLE, .HOW, .SOY, and .xn--q9jyb4c . The RDAP 867 server implements cursor based pagination (the number of objects 868 per page is fixed so the limit parameter is not available). The 869 link used to request the next page is included in the notice 870 section of the response. 871 Level of Maturity: Production. 872 Coverage: This implementation includes the cursor parameter 873 described in this specification. 874 Contact Information: Brian Mountford, mountford@google.com 876 7. Security Considerations 878 Security services for the operations specified in this document are 879 described in RFC 7481 [RFC7481]. 881 Search query typically requires more server resources (such as 882 memory, CPU cycles, and network bandwidth) when compared to lookup 883 query. This increases the risk of server resource exhaustion and 884 subsequent denial of service due to abuse. This risk can be 885 mitigated by either restricting search functionality and limiting the 886 rate of search requests. Servers can also reduce their load by 887 truncating the results in the response. However, this last security 888 policy can result in a higher inefficiency if the RDAP server does 889 not provide any functionality to return the truncated results. 891 The new parameters presented in this document provide the RDAP 892 operators with a way to implement a secure server without penalizing 893 its efficiency. The "count" parameter gives the user a measure to 894 evaluate the query precision and, at the same time, return a 895 significant information. The sort parameter allows the user to 896 obtain the most relevant information at the beginning of the result 897 set. In both cases, the user doesn't need to submit further 898 unnecessary search requests. Finally, the limit and offset 899 parameters enable the user to scroll the result set by submitting a 900 sequence of sustainable queries according to the server limits. 902 8. IANA Considerations 904 This document has no actions for IANA. 906 9. Acknowledgements 908 The authors would like to acknowledge Brian Mountford for his 909 contribution to the development of this document. 911 10. References 913 10.1. Normative References 915 [ISO.3166.1988] 916 International Organization for Standardization, "Codes for 917 the representation of names of countries, 3rd edition", 918 ISO Standard 3166, August 1988. 920 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 921 Requirement Levels", BCP 14, RFC 2119, 922 DOI 10.17487/RFC2119, March 1997, 923 . 925 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 926 IANA Considerations Section in RFCs", RFC 5226, 927 DOI 10.17487/RFC5226, May 2008, 928 . 930 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 931 Specifications: ABNF", STD 68, RFC 5234, 932 DOI 10.17487/RFC5234, January 2008, 933 . 935 [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, 936 DOI 10.17487/RFC6350, August 2011, 937 . 939 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 940 Protocol (HTTP/1.1): Message Syntax and Routing", 941 RFC 7230, DOI 10.17487/RFC7230, June 2014, 942 . 944 [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the 945 Registration Data Access Protocol (RDAP)", RFC 7480, 946 DOI 10.17487/RFC7480, March 2015, 947 . 949 [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the 950 Registration Data Access Protocol (RDAP)", RFC 7481, 951 DOI 10.17487/RFC7481, March 2015, 952 . 954 [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access 955 Protocol (RDAP) Query Format", RFC 7482, 956 DOI 10.17487/RFC7482, March 2015, 957 . 959 [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the 960 Registration Data Access Protocol (RDAP)", RFC 7483, 961 DOI 10.17487/RFC7483, March 2015, 962 . 964 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 965 Interchange Format", STD 90, RFC 8259, 966 DOI 10.17487/RFC8259, December 2017, 967 . 969 [RFC8288] Nottingham, M., "Web Linking", RFC 8288, 970 DOI 10.17487/RFC8288, October 2017, 971 . 973 10.2. Informative References 975 [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset 976 Pagination", July 2014, . 979 [CURSOR-API1] 980 facebook.com, "facebook for developers - Using the Graph 981 API", July 2017, . 984 [CURSOR-API2] 985 twitter.com, "Pagination", 2017, 986 . 989 [GOESSNER-JSON-PATH] 990 Goessner, S., "JSONPath - XPath for JSON", 2007, 991 . 993 [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, 994 . 997 [MYSQL-COUNT] 998 mysql.com, "MySQL 5.7 Reference Manual, Counting Rows", 999 October 2015, . 1002 [MYSQL-LIMIT] 1003 mysql.com, "MySQL 5.7 Reference Manual, SELECT Syntax", 1004 October 2015, 1005 . 1007 [MYSQL-SORT] 1008 mysql.com, "MySQL 5.7 Reference Manual, Sorting Rows", 1009 October 2015, . 1012 [OData-Part1] 1013 Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. 1014 Part 1: Protocol Plus Errata 03", June 2016, 1015 . 1020 [ORACLE-COUNT] 1021 Oracle Corporation, "Database SQL Language Reference, 1022 COUNT", March 2016, 1023 . 1025 [ORACLE-LIMIT] 1026 Oracle Corporation, "Database SQL Language Reference, 1027 SELECT, Row limiting clause", March 2016, 1028 . 1030 [ORACLE-ROW-NUMBER] 1031 Oracle Corporation, "Database SQL Language Reference, 1032 SELECT, ROW_NUMBER", March 2016, 1033 . 1036 [ORACLE-ROWNUM] 1037 Oracle Corporation, "Database SQL Language Reference, 1038 SELECT, ROWNUM Pseudocolumn", March 2016, 1039 . 1042 [ORACLE-SORT] 1043 Oracle Corporation, "Database SQL Language Reference, 1044 SELECT, Order by clause", March 2016, 1045 . 1047 [POSTGRES-COUNT] 1048 postgresql.org, "PostgresSQL, Aggregate Functions", 1049 September 2016, 1050 . 1053 [POSTGRES-LIMIT] 1054 postgresql.org, "PostgresSQL, LIMIT and OFFSET", September 1055 2016, . 1058 [POSTGRES-SORT] 1059 postgresql.org, "PostgresSQL, Sorting Rows", September 1060 2016, . 1063 [REST] Fredrich, T., "RESTful Service Best Practices, 1064 Recommendations for Creating Web Services", April 2012, 1065 . 1068 [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., 1069 "JavaScript Object Notation (JSON) Pointer", RFC 6901, 1070 DOI 10.17487/RFC6901, April 2013, 1071 . 1073 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 1074 Code: The Implementation Status Section", BCP 205, 1075 RFC 7942, DOI 10.17487/RFC7942, July 2016, 1076 . 1078 [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By 1079 With Limit and Offset is Slow?", July 2017, 1080 . 1083 [W3C.CR-xpath-31-20161213] 1084 Robie, J., Dyck, M., and J. Spiegel, "XML Path Language 1085 (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 1086 31-20161213, December 2016, 1087 . 1089 Appendix A. Change Log 1091 00: Initial version. 1092 01: Added the paragraph "Considerations about Paging Implementation" 1093 to "Implementation Considerations" section. Added "Implementation 1094 Status" section. Added acknowledgements. Renamed the property 1095 reporting the paging links. 1096 02: Corrected the value of "title" field in "paging_links" property. 1097 Updated references to RFC5988 (obsoleted by RFC 8288) and RFC7159 1098 (obsoleted by RFC 8259). Revised some sentences. 1099 03: Added the paragraph "Google Registry" to "Implementation Status" 1100 section. 1102 04: Rearranged the information about pagination included in RDAP 1103 responses. Added the section "Paging Metadata". Replaced the 1104 wrong reference to RFC 5266 with the correct reference to RFC 1105 5226. 1106 05: Renamed "sortby" parameter in "sort". Removed "country" from 1107 the list of sorting properties. Added "sorting_level_0" into the 1108 "rdapConformance" array. Changed the title of section "Paging 1109 Metadata" in "Sorting and Paging Metadata". Changed the "IANA 1110 Considerations" section. Added "Representing Sorting Links" 1111 section. Changed the name of some sorting properties to be 1112 compliant with EPP. 1114 Authors' Addresses 1116 Mario Loffredo 1117 IIT-CNR/Registro.it 1118 Via Moruzzi,1 1119 Pisa 56124 1120 IT 1122 Email: mario.loffredo@iit.cnr.it 1123 URI: http://www.iit.cnr.it 1125 Maurizio Martinelli 1126 IIT-CNR/Registro.it 1127 Via Moruzzi,1 1128 Pisa 56124 1129 IT 1131 Email: maurizio.martinelli@iit.cnr.it 1132 URI: http://www.iit.cnr.it 1134 Scott Hollenbeck 1135 Verisign Labs 1136 12061 Bluemont Way 1137 Reston, VA 20190 1138 USA 1140 Email: shollenbeck@verisign.com 1141 URI: https://www.verisignlabs.com/