idnits 2.17.1 draft-ietf-payload-vp9-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 19, 2015) is 3112 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational draft: draft-grange-vp9-bitstream (ref. 'I-D.grange-vp9-bitstream') ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: April 21, 2016 Google 6 J. Lennox 7 D. Hong 8 Vidyo 9 October 19, 2015 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-01 14 Abstract 16 This memo describes an RTP payload format for the VP9 video codec. 17 The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on April 21, 2016. 39 Copyright Notice 41 Copyright (c) 2015 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 2 58 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 59 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 60 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 61 4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 6 62 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 10 63 4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 11 64 4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12 65 4.5. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 12 66 5. Using VP9 with RPSI and SLI Feedback . . . . . . . . . . . . 12 67 5.1. RPSI . . . . . . . . . . . . . . . . . . . . . . . . . . 12 68 5.2. SLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 69 5.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 13 70 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 15 71 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 15 72 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 17 73 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 17 74 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 17 75 7. Security Considerations . . . . . . . . . . . . . . . . . . . 17 76 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 18 77 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 18 80 10.2. Informative References . . . . . . . . . . . . . . . . . 19 81 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 83 1. Introduction 85 This memo describes an RTP payload specification applicable to the 86 transmission of video streams encoded using the VP9 video codec 87 [I-D.grange-vp9-bitstream]. The format described in this document 88 can be used both in peer-to-peer and video conferencing applications. 90 TODO: VP9 description. Please see [I-D.grange-vp9-bitstream]. 92 2. Conventions, Definitions and Acronyms 94 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 95 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 96 document are to be interpreted as described in [RFC2119]. 98 TODO: Cite terminology from [I-D.grange-vp9-bitstream]. 100 3. Media Format Description 102 The VP9 codec can maintain up to eight reference frames, of which up 103 to three can be referenced or updated by any new frame. 105 VP9 also allows a reference frame to be resampled and used as a 106 reference for another frame of a different resolution. This allows 107 internal resolution changes without requiring the use of key frames. 109 These features together enable an encoder to implement various forms 110 of coarse-grained scalability, including temporal, spatial and 111 quality scalability modes, as well as combinations of these, without 112 the need for explicit scalable coding tools. 114 Temporal layers define different frame rates of video; spatial and 115 quality layers define different and possibly dependent 116 representations of a single input frame. Spatial layers allow a 117 frame to be encoded at different resolutions, whereas quality layers 118 allow a frame to be encoded at the same resolution but at different 119 qualities (and thus with different amounts of coding error). VP9 120 supports quality layers as spatial layers without any resolution 121 changes; hereinafter, the term "spatial layer" is used to represent 122 both spatial and quality layers. 124 This payload format specification defines how such temporal and 125 spatial scalability layers can be described and communicated. 127 Temporal and spatial scalability layers are associated with non- 128 negative integer IDs. The lowest layer of either type has an ID of 129 0. 131 Layers are designed (and MUST be encoded) such that if any layer, and 132 all higher layers, are removed from the bitstream along any of the 133 two dimensions, the remaining bitstream is still correctly decodable. 135 For terminology, this document uses the term "layer frame" to refer 136 to a single encoded VP9 frame for a particular resolution/quality, 137 and "super frame" to refer to all the representations (layer frames) 138 at a single instant in time. A super frame thus consists of one or 139 more layer frames, encoding different spatial layers. 141 Within a super frame, a layer frame with spatial layer ID equal to S, 142 where S > 0, can depend on a layer frame of the same super frame with 143 a lower spatial layer ID. This "inter-layer" dependency can result 144 in additional coding gain compared to the case where only traditional 145 "inter-picture" dependency is used, where a frame depends on 146 previously coded frame in time. For simplicity, this payload format 147 assumes that, within a super frame and if inter-layer dependency is 148 used, a spatial layer S frame can only depend on spatial layer S-1 149 frame when S > 0. Additionally, if inter-picture dependency is used, 150 spatial layer S frame is assumed to only depend on previously coded 151 spatial layer S frame. 153 TODO: Describe how simulcast can be supported? 155 Given above simplifications for inter-layer and inter-picture 156 dependencies, a flag (the D bit described below) is used to indicate 157 whether a spatial layer S frame depends on spatial layer S-1 frame. 158 Given the D bit, a receiver only needs to additionally know the 159 inter-picture dependency structure for a given spatial layer frame in 160 order to determine its decodability. Two modes of describing the 161 inter-picture dependency structure are possible: "flexible mode" and 162 "non-flexible mode". An encoder can only switch between the two on 163 the very first packet of a key frame with temporal layer ID equal to 164 0. 166 In flexible mode, each packet can contain up to 3 reference indices, 167 which identify all frames referenced by the frame transmitted in the 168 current packet for inter-picture prediction. This (along with the D 169 bit) enables a receiver to identify if a frame is decodable or not 170 and helps it understand the temporal layer structure. Since this is 171 signaled in each packet it makes it possible to have very flexible 172 temporal layer hierarchies and patterns which are changing 173 dynamically. 175 In non-flexible mode, the inter-picture dependency (the reference 176 indices) of a group of frames (GOF) MUST be pre-specified as part of 177 the scalability structure (SS) data. In this mode, each packet MUST 178 have an index to refer to one of the described frames in the GOF, 179 from which the frames referenced by the frame transmitted in the 180 current packet for inter-picture prediction can be identified. 182 The SS data can also be used to specify the resolution of each 183 spatial layer present in the VP9 stream for both flexible and non- 184 flexible modes. 186 4. Payload Format 188 This section describes how the encoded VP9 bitstream is encapsulated 189 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 190 RECOMMENDED. All integer fields in the specifications are encoded as 191 unsigned integers in network octet order. 193 4.1. RTP Header Usage 195 The general RTP payload format for VP9 is depicted below. 197 0 1 2 3 198 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 199 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 200 |V=2|P|X| CC |M| PT | sequence number | 201 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 202 | timestamp | 203 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 204 | synchronization source (SSRC) identifier | 205 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 206 | contributing source (CSRC) identifiers | 207 | .... | 208 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 209 | VP9 payload descriptor (integer #octets) | 210 : : 211 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 212 | : VP9 pyld hdr | | 213 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 214 | | 215 + | 216 : Bytes 2..N of VP9 payload : 217 | | 218 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 219 | : OPTIONAL RTP padding | 220 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 222 The VP9 payload descriptor and VP9 payload header will be described 223 in Section 4.2 and Section 4.3. OPTIONAL RTP padding MUST NOT be 224 included unless the P bit is set. The figure specifically shows the 225 format for the first packet in a frame. Subsequent packets will not 226 contain the VP9 payload header, and will have later octets in the 227 frame payload. 229 Figure 1 231 Marker bit (M): MUST be set to 1 for the final packet of the highest 232 spatial layer frame (the final packet of the super frame), and 0 233 otherwise. Unless spatial scalability is in use for this super 234 frame, this will have the same value as the E bit described below. 235 Note this bit MUST be set to 1 for the target spatial layer frame 236 if a stream is being rewritten to remove higher spatial layers. 238 Payload Type (TP): In line with the policy in Section 3 of 239 [RFC3551], applications using the VP9 RTP payload profile MUST 240 assign a dynamic payload type number to be used in each RTP 241 session and provide a mechanism to indicate the mapping. See 242 Section 6.2 for the mechanism to be used with the Session 243 Description Protocol (SDP) [RFC4566]. 245 Timestamp: The RTP timestamp indicates the time when the input frame 246 was sampled, at a clock rate of 90 kHz. If the input frame is 247 encoded with multiple layer frames, all of the layer frames of the 248 super frame MUST have the same timestamp. 250 The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, 251 SSRC and CSRC identifiers) are used as specified in Section 5.1 of 252 [RFC3550]. 254 4.2. VP9 Payload Description 256 In flexible mode (with the F bit below set to 1), The first octets 257 after the RTP header are the VP9 payload descriptor, with the 258 following structure. 260 0 1 2 3 4 5 6 7 261 +-+-+-+-+-+-+-+-+ 262 |I|P|L|F|B|E|V|-| (REQUIRED) 263 +-+-+-+-+-+-+-+-+ 264 I: |M| PICTURE ID | (REQUIRED) 265 +-+-+-+-+-+-+-+-+ 266 M: | EXTENDED PID | (RECOMMENDED) 267 +-+-+-+-+-+-+-+-+ 268 L: | T |U| S |D| (CONDITIONALLY RECOMMENDED) 269 +-+-+-+-+-+-+-+-+ -\ 270 P,F: | P_DIFF |N| (CONDITIONALLY REQUIRED) - up to 3 times 271 +-+-+-+-+-+-+-+-+ -/ 272 V: | SS | 273 | .. | 274 +-+-+-+-+-+-+-+-+ 276 Figure 2 278 In non-flexible mode (with the F bit below set to 0), The first 279 octets after the RTP header are the VP9 payload descriptor, with the 280 following structure. 282 0 1 2 3 4 5 6 7 283 +-+-+-+-+-+-+-+-+ 284 |I|P|L|F|B|E|V|-| (REQUIRED) 285 +-+-+-+-+-+-+-+-+ 286 I: |M| PICTURE ID | (RECOMMENDED) 287 +-+-+-+-+-+-+-+-+ 288 M: | EXTENDED PID | (RECOMMENDED) 289 +-+-+-+-+-+-+-+-+ 290 L: | T |U| S |D| (CONDITIONALLY RECOMMENDED) 291 +-+-+-+-+-+-+-+-+ 292 | TL0PICIDX | (CONDITIONALLY REQUIRED) 293 +-+-+-+-+-+-+-+-+ 294 V: | SS | 295 | .. | 296 +-+-+-+-+-+-+-+-+ 298 Figure 3 300 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 301 be present after the mandatory first octet and specified as below. 302 Otherwise, PID MUST NOT be present. 304 P: Inter-picture predicted layer frame. When set to zero, the layer 305 frame does not utilize inter-picture prediction. In this case, 306 up-switching to current spatial layer's frame is possible from 307 directly lower spatial layer frame. P SHOULD also be set to zero 308 when encoding a layer synchronization frame in response to an LRR 309 [I-D.lennox-avtext-lrr]. When P is set to zero, the T bit 310 (described below) MUST also be set to 0 (if present). 312 L: Layer indices present. When set to one, the one or two octets 313 following the mandatory first octet and the PID (if present) is as 314 described by "Layer indices" below. If the F bit (described 315 below) is set to 1 (indicating flexible mode), then only one octet 316 is present for the layer indices. Otherwise if the F bit is set 317 to 0 (indicating non-flexible mode), then two octets are present 318 for the layer indices. 320 F: Flexible mode. F set to one indicates flexible mode and if the P 321 bit is also set to one, then the octets following the mandatory 322 first octet, the PID, and layer indices (if present) are as 323 described by "Reference indices" below. This MUST only be set to 324 1 if the I bit is also set to one; if the I bit is set to zero, 325 then this MUST also be set to zero and ignored by receivers. The 326 value of this F bit CAN ONLY CHANGE on the very first packet of a 327 key picture. This is a packet with the P bit equal to zero, S or 328 D bit (described below) equal to zero, and B bit (described below) 329 equal to 1. 331 B: Start of a layer frame. MUST be set to 1 if the first payload 332 octet of the RTP packet is the beginning of a new VP9 layer frame, 333 and MUST NOT be 1 otherwise. Note that this layer frame might not 334 be the very first layer frame of a super frame. 336 E: End of a layer frame. MUST be set to 1 for the final RTP packet 337 of a VP9 layer frame, and 0 otherwise. This enables a decoder to 338 finish decoding the layer frame, where it otherwise may need to 339 wait for the next packet to explicitly know that the layer frame 340 is complete. Note that, if spatial scalability is in use, more 341 layer frames from the same super frame may follow; see the 342 description of the M bit above. 344 V: Scalability structure (SS) data present. When set to one, the 345 OPTIONAL SS data MUST be present in the payload descriptor. 346 Otherwise, the SS data MUST NOT be present. 348 -: Bit reserved for future use. MUST be set to zero and MUST be 349 ignored by the receiver. 351 The mandatory first octet is followed by the extension data fields 352 that are enabled: 354 M: The most significant bit of the first octet is an extension flag. 355 The field MUST be present if the I bit is equal to one. If set, 356 the PID field MUST contain 15 bits; otherwise, it MUST contain 7 357 bits. See PID below. 359 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 360 on the M bit. This is a running index of the pictures. The field 361 MUST be present if the I bit is equal to one. If M is set to 362 zero, 7 bits carry the PID; else if M is set to one, 15 bits carry 363 the PID in network byte order. The sender may choose between a 7- 364 or 15-bit index. The PID SHOULD start on a random number, and 365 MUST wrap after reaching the maximum ID. The receiver MUST NOT 366 assume that the number of bits in PID stay the same through the 367 session. 369 In the non-flexible mode (when the F bit is set to 0), this PID is 370 used as an index to the GOF specified in the SS data bleow. In 371 this mode, the PID of the key frame corresponds to the very first 372 specified frame in the GOF. Then subsequent PIDs are mapped to 373 subsequently specified frames in the GOF (modulo N_G, specified in 374 the SS data below), respectively. 376 Layer indices: This information is optional but recommended whenever 377 encoding with layers. For both flexible and non-flexible modes, 378 one octet is used to specify a layer frame's temporal layer ID (T) 379 and spatial layer ID (S) as shown both in Figure 2 and Figure 3. 380 Additionally, a bit (U) is used to indicate that the current frame 381 is a "switching up point" frame. Another bit (D) is used to 382 indicate whether inter-layer prediction is used for the current 383 layer frame. 385 In the non-flexible mode (when the F bit is set to 0), another 386 octet is used to represent temporal layer 0 index (TL0PICIDX), as 387 depicted in Figure 3. The TL0PICIDX is present so that all 388 minimally required frames - the base temporal layer frames - can 389 be tracked. 391 The T and S fields indicate the temporal and spatial layers and 392 can help middleboxes and and endpoints quickly identify which 393 layer a packet belongs to. 395 T: The temporal layer ID of current frame. In the case of non- 396 flexible mode, if PID is mapped to a frame in a specified GOF, 397 then the value of T MUST match the corresponding T value of the 398 mapped frame in the GOF. 400 U: Switching up point. If this bit is set to 1 for the current 401 frame with temporal layer ID equal to T, then "switch up" to a 402 higher frame rate is possible as subsequent higher temporal 403 layer frames will not depend on any frame before the current 404 frame (in coding time) with temporal layer ID greater than T. 406 S: The spatial layer ID of current frame. Note that frames with 407 spatial layer S > 0 may be dependent on decoded spatial layer 408 S-1 frame within the same super frame. 410 D: Inter-layer dependency used. MUST be set to one if current 411 spatial layer S frame depends on spatial layer S-1 frame of the 412 same super frame. MUST only be set to zero if current spatial 413 layer S frame does not depend on spatial layer S-1 frame of the 414 same super frame. For the base layer frame with S equal to 0, 415 this D bit MUST be set to zero. 417 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 418 present in the non-flexible mode (F = 0). This is a running 419 index for the temporal base layer frames, i.e., the frames with 420 T set to 0. If T is larger than 0, TL0PICIDX indicates which 421 temporal base layer frame the current frame depends on. 422 TL0PICIDX MUST be incremented when T is equal to 0. The index 423 SHOULD start on a random number, and MUST restart at 0 after 424 reaching the maximum number 255. 426 Reference indices: When P and F are both set to one, indicating a 427 non-key frame in flexible mode, then at least one reference index 428 has to be specified as below. Additional reference indices (total 429 of up to 3 reference indices are allowed) may be specified using 430 the N bit below. When either P or F is set to zero, then no 431 reference index is specified. 433 P_DIFF: The reference index (in 7 bits) specified as the relative 434 PID from the current frame. For example, when P_DIFF=3 on a 435 packet containing the frame with PID 112 means that the frame 436 refers back to the frame with PID 109. This calculation is 437 done modulo the size of the PID field, i.e., either 7 or 15 438 bits. 440 N: 1 if there is additional P_DIFF following the current P_DIFF. 442 4.2.1. Scalability Structure (SS): 444 The scalability structure (SS) data describes the resolution of each 445 layer frame within a super frame as well as the inter-picture 446 dependencies for a group of frames (GOF). If the VP9 payload 447 descriptor's "V" bit is set, the SS data is present in the position 448 indicated in Figure 2 and Figure 3. 450 +-+-+-+-+-+-+-+-+ 451 V: | N_S |Y|G|-|-|-| 452 +-+-+-+-+-+-+-+-+ -\ 453 Y: | WIDTH | (OPTIONAL) . 454 + + . 455 | | (OPTIONAL) . 456 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 457 | HEIGHT | (OPTIONAL) . 458 + + . 459 | | (OPTIONAL) . 460 +-+-+-+-+-+-+-+-+ -/ -\ 461 G: | N_G | (OPTIONAL) 462 +-+-+-+-+-+-+-+-+ -\ 463 N_G: | T |U| R |-|-| (OPTIONAL) . 464 +-+-+-+-+-+-+-+-+ -\ . - N_G times 465 | P_DIFF | (OPTIONAL) . - R times . 466 +-+-+-+-+-+-+-+-+ -/ -/ 468 Figure 4 470 N_S: N_S + 1 indicates the number of spatial layers present in the 471 VP9 stream. 473 Y: Each spatial layer's frame resolution present. When set to one, 474 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 475 present for each layer frame. Otherwise, the resolution MUST NOT 476 be present. 478 G: GOF description present flag. 480 -: Bit reserved for future use. MUST be set to zero and MUST be 481 ignored by the receiver. 483 N_G: N_G indicates the number of frames in a GOF. If N_G is greater 484 than 0, then the SS data allows the inter-picture dependency 485 structure of the VP9 stream to be pre-declared, rather than 486 indicating it on the fly with every packet. If N_G is greater 487 than 0, then for N_G pictures in the GOF, each frame's temporal 488 layer ID (T), switch up point (U), and the R reference indices 489 (P_DIFFs) are specified. 491 The very first frame specified in the GOF MUST have T set to 0. 493 G set to 0 or N_G set to 0 indicates that either there is only one 494 temporal layer or no fixed inter-picture dependency information is 495 present going forward in the bitstream. 497 Note that for a given super frame, all layer frames follow the 498 same inter-picture dependency structure. However, the frame rate 499 of each spatial layer can be different from each other and this 500 can be controlled with the use of the D bit described above. The 501 specified dependency structure in the SS data MUST be for the 502 highest frame rate layer. 504 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 505 included in the first packet of every key frame. This is a packet 506 with P bit equal to zero, S or D bit equal to zero, and B bit equal 507 to 1. The SS data MUST only be changed on the frame that corresponds 508 to the very first frame specified in the previous SS data's GOF (if 509 the previous SS data's N_G was greater than 0). 511 4.3. VP9 Payload Header 513 TODO: need to describe VP9 payload header. 515 4.4. Frame Fragmentation 517 VP9 frames are fragmented into packets, in RTP sequence number order, 518 beginning with a packet with the B bit set, and ending with a packet 519 with the RTP marker bit M set. There is no mechanism for finer- 520 grained access to parts of a VP9 frame. 522 4.5. Examples of VP9 RTP Stream 524 TODO 526 5. Using VP9 with RPSI and SLI Feedback 528 The VP9 payload descriptor defined in Section 4.2 above contains an 529 optional PictureID parameter. One use of this parameter is to enable 530 use of the reference picture selection index (RPSI) and slice loss 531 indication (SLI) RTCP feedback messages, both defined in [RFC4585]. 533 5.1. RPSI 535 TODO: Update to indicate which frame within the picture. 537 The reference picture selection index is a payload-specific feedback 538 message defined within the RTCP-based feedback format. The RPSI 539 message is generated by a receiver and can be used in two ways. 540 Either it can signal a preferred reference picture when a loss has 541 been detected by the decoder -- preferably then a reference that the 542 decoder knows is perfect -- or, it can be used as positive feedback 543 information to acknowledge correct decoding of certain reference 544 pictures. The positive feedback method is useful for VP9 used for 545 point to point (unicast) communication. The use of RPSI for VP9 is 546 preferably combined with a special update pattern of the codec's two 547 special reference frames -- the golden frame and the altref frame -- 548 in which they are updated in an alternating leapfrog fashion. When a 549 receiver has received and correctly decoded a golden or altref frame, 550 and that frame had a PictureID in the payload descriptor, the 551 receiver can acknowledge this simply by sending an RPSI message back 552 to the sender. The message body (i.e., the "native RPSI bit string" 553 in [RFC4585]) is simply the PictureID of the received frame. 555 5.2. SLI 557 TODO: Update to indicate which frame within the picture. 559 The slice loss indication is another payload-specific feedback 560 message defined within the RTCP-based feedback format. The SLI 561 message is generated by the receiver when a loss or corruption is 562 detected in a frame. The format of the SLI message is as follows 563 [RFC4585]: 565 0 1 2 3 566 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 567 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 568 | First | Number | PictureID | 569 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 571 Figure 5 573 Here, First is the macroblock address (in scan order) of the first 574 lost block and Number is the number of lost blocks, as defined in 575 [RFC4585]. PictureID is the six least significant bits of the codec- 576 specific picture identifier in which the loss or corruption has 577 occurred. For VP9, this codec-specific identifier is naturally the 578 PictureID of the current frame, as read from the payload descriptor. 579 If the payload descriptor of the current frame does not have a 580 PictureID, the receiver MAY send the last received PictureID+1 in the 581 SLI message. The receiver MAY set the First parameter to 0, and the 582 Number parameter to the total number of macroblocks per frame, even 583 though only part of the frame is corrupted. When the sender receives 584 an SLI message, it can make use of the knowledge from the latest 585 received RPSI message. Knowing that the last golden or altref frame 586 was successfully received, it can encode the next frame with 587 reference to that established reference. 589 5.3. Example 591 TODO: this example is copied from the VP8 payload format 592 specification, and has not been updated for VP9. It may be 593 incorrect. 595 The use of RPSI and SLI is best illustrated in an example. In this 596 example, the encoder may not update the altref frame until the last 597 sent golden frame has been acknowledged with an RPSI message. If an 598 update is not received within some time, a new golden frame update is 599 sent instead. Once the new golden frame is established and 600 acknowledged, the same rule applies when updating the altref frame. 602 +-------+-------------------+-------------------------+-------------+ 603 | Event | Sender | Receiver | Established | 604 | | | | reference | 605 +-------+-------------------+-------------------------+-------------+ 606 | 1000 | Send golden frame | | | 607 | | PictureID = 0 | | | 608 | | | | | 609 | | | Receive and decode | | 610 | | | golden frame | | 611 | | | | | 612 | 1001 | | Send RPSI(0) | | 613 | | | | | 614 | 1002 | Receive RPSI(0) | | golden | 615 | | | | | 616 | ... | (sending regular | | | 617 | | frames) | | | 618 | | | | | 619 | 1100 | Send altref frame | | | 620 | | PictureID = 100 | | | 621 | | | | | 622 | | | Altref corrupted or | golden | 623 | | | lost | | 624 | | | | | 625 | 1101 | | Send SLI(100) | golden | 626 | | | | | 627 | 1102 | Receive SLI(100) | | | 628 | | | | | 629 | 1103 | Send frame with | | | 630 | | reference to | | | 631 | | golden | | | 632 | | | | | 633 | | | Receive and decode | golden | 634 | | | frame (decoder state | | 635 | | | restored) | | 636 | | | | | 637 | ... | (sending regular | | | 638 | | frames) | | | 639 | | | | | 640 | 1200 | Send altref frame | | | 641 | | PictureID = 200 | | | 642 | | | | | 643 | | | Receive and decode | golden | 644 | | | altref frame | | 645 | | | | | 646 | 1201 | | Send RPSI(200) | | 647 | | | | | 648 | 1202 | Receive RPSI(200) | | altref | 649 | | | | | 650 | ... | (sending regular | | | 651 | | frames) | | | 652 | | | | | 653 | 1300 | Send golden frame | | | 654 | | PictureID = 300 | | | 655 | | | | | 656 | | | Receive and decode | altref | 657 | | | golden frame | | 658 | | | | | 659 | 1301 | | Send RPSI(300) | altref | 660 | | | | | 661 | 1302 | RPSI lost | | | 662 | | | | | 663 | 1400 | Send golden frame | | | 664 | | PictureID = 400 | | | 665 | | | | | 666 | | | Receive and decode | altref | 667 | | | golden frame | | 668 | | | | | 669 | 1401 | | Send RPSI(400) | | 670 | | | | | 671 | 1402 | Receive RPSI(400) | | golden | 672 +-------+-------------------+-------------------------+-------------+ 674 Table 1: Example signaling between sender and receiver 676 Note that the scheme is robust to loss of the feedback messages. If 677 the RPSI is lost, the sender will try to update the golden (or 678 altref) again after a while, without releasing the established 679 reference. Also, if an SLI is lost, the receiver can keep sending 680 SLI messages at any interval allowed by the RTCP sending timing 681 restrictions as specified in [RFC4585], as long as the picture is 682 corrupted. 684 6. Payload Format Parameters 686 This payload format has two optional parameters. 688 6.1. Media Type Definition 690 This registration is done using the template defined in [RFC6838] and 691 following [RFC4855]. 693 Type name: video 695 Subtype name: VP9 697 Required parameters: None. 699 Optional parameters: 700 These parameters are used to signal the capabilities of a receiver 701 implementation. If the implementation is willing to receive 702 media, both parameters MUST be provided. These parameters MUST 703 NOT be used for any other purpose. 705 max-fr: The value of max-fr is an integer indicating the maximum 706 frame rate in units of frames per second that the decoder is 707 capable of decoding. 709 max-fs: The value of max-fs is an integer indicating the maximum 710 frame size in units of macroblocks that the decoder is capable 711 of decoding. 713 The decoder is capable of decoding this frame size as long as 714 the width and height of the frame in macroblocks are less than 715 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 716 of supporting 640x480 resolution) will support widths and 717 heights up to 1552 pixels (97 macroblocks). 719 Encoding considerations: 720 This media type is framed in RTP and contains binary data; see 721 Section 4.8 of [RFC6838]. 723 Security considerations: See Section 7 of RFC xxxx. 724 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 725 with the number assigned to this document and remove this note.] 727 Interoperability considerations: None. 729 Published specification: VP9 bitstream format 730 [I-D.grange-vp9-bitstream] and RFC XXXX. 731 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 732 with the number assigned to this document and remove this note.] 734 Applications which use this media type: 735 For example: Video over IP, video conferencing. 737 Fragment identifier considerations: N/A. 739 Additional information: None. 741 Person & email address to contact for further information: 742 TODO [Pick a contact] 744 Intended usage: COMMON 746 Restrictions on usage: 747 This media type depends on RTP framing, and hence is only defined 748 for transfer via RTP [RFC3550]. 750 Author: TODO [Pick a contact] 752 Change controller: 754 IETF Payload Working Group delegated from the IESG. 756 6.2. SDP Parameters 758 The receiver MUST ignore any fmtp parameter unspecified in this memo. 760 6.2.1. Mapping of Media Subtype Parameters to SDP 762 The media type video/VP9 string is mapped to fields in the Session 763 Description Protocol (SDP) [RFC4566] as follows: 765 o The media name in the "m=" line of SDP MUST be video. 767 o The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 768 media subtype). 770 o The clock rate in the "a=rtpmap" line MUST be 90000. 772 o The parameters "max-fs", and "max-fr", MUST be included in the 773 "a=fmtp" line of SDP if SDP is used to declare receiver 774 capabilities. These parameters are expressed as a media subtype 775 string, in the form of a semicolon separated list of 776 parameter=value pairs. 778 6.2.1.1. Example 780 An example of media representation in SDP is as follows: 782 m=video 49170 RTP/AVPF 98 783 a=rtpmap:98 VP9/90000 784 a=fmtp:98 max-fr=30; max-fs=3600; 786 6.2.2. Offer/Answer Considerations 788 TODO: Update this for VP9 790 7. Security Considerations 792 RTP packets using the payload format defined in this specification 793 are subject to the security considerations discussed in the RTP 794 specification [RFC3550], and in any applicable RTP profile such as 795 RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/ 796 SAVPF [RFC5124]. SAVPF [RFC5124]. However, as "Securing the RTP 797 Protocol Framework: Why RTP Does Not Mandate a Single Media Security 798 Solution" [RFC7202] discusses, it is not an RTP payload format's 799 responsibility to discuss or mandate what solutions are used to meet 800 the basic security goals like confidentiality, integrity and source 801 authenticity for RTP in general. This responsibility lays on anyone 802 using RTP in an application. They can find guidance on available 803 security mechanisms in Options for Securing RTP Sessions [RFC7201]. 804 Applications SHOULD use one or more appropriate strong security 805 mechanisms. The rest of this security consideration section 806 discusses the security impacting properties of the payload format 807 itself. 809 This RTP payload format and its media decoder do not exhibit any 810 significant non-uniformity in the receiver-side computational 811 complexity for packet processing, and thus are unlikely to pose a 812 denial-of-service threat due to the receipt of pathological data. 813 Nor does the RTP payload format contain any active content. 815 8. Congestion Control 817 Congestion control for RTP SHALL be used in accordance with RFC 3550 818 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 819 [RFC3551]. The congestion control mechanism can, in a real-time 820 encoding scenario, adapt the transmission rate by instructing the 821 encoder to encode at a certain target rate. Media aware network 822 elements MAY use the information in the VP9 payload descriptor in 823 Section 4.2 to identify non-reference frames and discard them in 824 order to reduce network congestion. Note that discarding of non- 825 reference frames cannot be done if the stream is encrypted (because 826 the non-reference marker is encrypted). 828 9. IANA Considerations 830 The IANA is requested to register the following values: 831 - Media type registration as described in Section 6.1. 833 10. References 835 10.1. Normative References 837 [I-D.grange-vp9-bitstream] 838 Grange, A. and H. Alvestrand, "A VP9 Bitstream Overview", 839 draft-grange-vp9-bitstream-00 (work in progress), February 840 2013. 842 [I-D.lennox-avtext-lrr] 843 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 844 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 845 Message", draft-lennox-avtext-lrr-00 (work in progress), 846 March 2015. 848 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 849 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 850 RFC2119, March 1997, 851 . 853 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 854 Jacobson, "RTP: A Transport Protocol for Real-Time 855 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 856 July 2003, . 858 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 859 Description Protocol", RFC 4566, DOI 10.17487/RFC4566, 860 July 2006, . 862 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 863 "Extended RTP Profile for Real-time Transport Control 864 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 865 10.17487/RFC4585, July 2006, 866 . 868 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 869 Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, 870 . 872 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 873 Specifications and Registration Procedures", BCP 13, RFC 874 6838, DOI 10.17487/RFC6838, January 2013, 875 . 877 10.2. Informative References 879 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 880 Video Conferences with Minimal Control", STD 65, RFC 3551, 881 DOI 10.17487/RFC3551, July 2003, 882 . 884 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 885 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 886 RFC 3711, DOI 10.17487/RFC3711, March 2004, 887 . 889 [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for 890 Real-time Transport Control Protocol (RTCP)-Based Feedback 891 (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 892 2008, . 894 [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP 895 Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, 896 . 898 [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP 899 Framework: Why RTP Does Not Mandate a Single Media 900 Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 901 2014, . 903 Authors' Addresses 905 Justin Uberti 906 Google, Inc. 907 747 6th Street South 908 Kirkland, WA 98033 909 USA 911 Email: justin@uberti.name 913 Stefan Holmer 914 Google, Inc. 915 Kungsbron 2 916 Stockholm 111 22 917 Sweden 919 Email: holmer@google.com 921 Magnus Flodman 922 Google, Inc. 923 Kungsbron 2 924 Stockholm 111 22 925 Sweden 927 Email: mflodman@google.com 929 Jonathan Lennox 930 Vidyo, Inc. 931 433 Hackensack Avenue 932 Seventh Floor 933 Hackensack, NJ 07601 934 US 936 Email: jonathan@vidyo.com 937 Danny Hong 938 Vidyo, Inc. 939 433 Hackensack Avenue 940 Seventh Floor 941 Hackensack, NJ 07601 942 US 944 Email: danny@vidyo.com