idnits 2.17.1 draft-ietf-payload-vp9-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (2 February 2021) is 1172 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'VP9-BITSTREAM' Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 AVTCore Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: 6 August 2021 D. Hong 6 Google 7 J. Lennox 8 8x8 / Jitsi 9 2 February 2021 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-11 14 Abstract 16 This memo describes an RTP payload format for the VP9 video codec. 17 The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on 6 August 2021. 39 Copyright Notice 41 Copyright (c) 2021 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 46 license-info) in effect on the date of publication of this document. 47 Please review these documents carefully, as they describe your rights 48 and restrictions with respect to this document. Code Components 49 extracted from this document must include Simplified BSD License text 50 as described in Section 4.e of the Trust Legal Provisions and are 51 provided without warranty as described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 57 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 58 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 5 59 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 60 4.2. VP9 Payload Descriptor . . . . . . . . . . . . . . . . . 6 61 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 11 62 4.3. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12 63 4.4. Scalable encoding considerations . . . . . . . . . . . . 13 64 4.5. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 13 65 4.5.1. Reference picture use for scalable structure . . . . 13 66 5. Feedback Messages and Header Extensions . . . . . . . . . . . 14 67 5.1. Reference Picture Selection Indication (RPSI) . . . . . . 14 68 5.2. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 15 69 5.3. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 15 70 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 16 71 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 16 72 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 18 73 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 19 74 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 19 75 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 76 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 20 77 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 78 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21 79 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 80 11.1. Normative References . . . . . . . . . . . . . . . . . . 21 81 11.2. Informative References . . . . . . . . . . . . . . . . . 22 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 84 1. Introduction 86 This memo describes an RTP payload specification applicable to the 87 transmission of video streams encoded using the VP9 video codec 88 [VP9-BITSTREAM]. The format described in this document can be used 89 both in peer-to-peer and video conferencing applications. 91 The VP9 video codec was developed by Google, and is the successor to 92 its earlier VP8 [RFC6386] codec. Above the compression improvements 93 and other general enhancements above VP8, VP9 is also designed in a 94 way that allows spatially-scalable video encoding. 96 2. Conventions, Definitions and Acronyms 98 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 99 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 100 document are to be interpreted as described in [RFC2119]. 102 3. Media Format Description 104 The VP9 codec can maintain up to eight reference frames, of which up 105 to three can be referenced by any new frame. 107 VP9 also allows a frame to use another frame of a different 108 resolution as a reference frame. (Specifically, a frame may use any 109 references whose width and height are between 1/16th that of the 110 current frame and twice that of the current frame, inclusive.) This 111 allows internal resolution changes without requiring the use of key 112 frames. 114 These features together enable an encoder to implement various forms 115 of coarse-grained scalability, including temporal, spatial and 116 quality scalability modes, as well as combinations of these, without 117 the need for explicit scalable coding tools. 119 Temporal layers define different frame rates of video; spatial and 120 quality layers define different and possibly dependent 121 representations of a single input frame. Spatial layers allow a 122 frame to be encoded at different resolutions, whereas quality layers 123 allow a frame to be encoded at the same resolution but at different 124 qualities (and thus with different amounts of coding error). VP9 125 supports quality layers as spatial layers without any resolution 126 changes; hereinafter, the term "spatial layer" is used to represent 127 both spatial and quality layers. 129 This payload format specification defines how such temporal and 130 spatial scalability layers can be described and communicated. 132 Temporal and spatial scalability layers are associated with non- 133 negative integer IDs. The lowest layer of either type has an ID of 134 0, and is sometimes referred to as the "base" temporal or spatial 135 layer. 137 Layers are designed (and MUST be encoded) such that if any layer, and 138 all higher layers, are removed from the bitstream along either of the 139 two dimensions, the remaining bitstream is still correctly decodable. 141 For terminology, this document uses the term "frame" to refer to a 142 single encoded VP9 frame for a particular resolution/quality, and 143 "picture" to refer to all the representations (frames) at a single 144 instant in time. A picture thus consists of one or more frames, 145 encoding different spatial layers. 147 Within a picture, a frame with spatial layer ID equal to SID, where 148 SID > 0, can depend on a frame of the same picture with a lower 149 spatial layer ID. This "inter-layer" dependency can result in 150 additional coding gain compared to the case where only traditional 151 "inter-picture" dependency is used, where a frame depends on 152 previously coded frame in time. For simplicity, this payload format 153 assumes that, within a picture and if inter-layer dependency is used, 154 a spatial layer SID frame can depend only on the immediately previous 155 spatial layer SID-1 frame, when S > 0. Additionally, if inter- 156 picture dependency is used, a spatial layer SID frame is assumed to 157 only depend on a previously coded spatial layer SID frame. 159 Given above simplifications for inter-layer and inter-picture 160 dependencies, a flag (the D bit described below) is used to indicate 161 whether a spatial layer SID frame depends on the spatial layer SID-1 162 frame. Given the D bit, a receiver only needs to additionally know 163 the inter-picture dependency structure for a given spatial layer 164 frame in order to determine its decodability. Two modes of 165 describing the inter-picture dependency structure are possible: 166 "flexible mode" and "non-flexible mode". An encoder can only switch 167 between the two on the first packet of a key frame with temporal 168 layer ID equal to 0. 170 In flexible mode, each packet can contain up to 3 reference indices, 171 which identify all frames referenced by the frame transmitted in the 172 current packet for inter-picture prediction. This (along with the D 173 bit) enables a receiver to identify if a frame is decodable or not 174 and helps it understand the temporal layer structure. Since this is 175 signaled in each packet it makes it possible to have very flexible 176 temporal layer hierarchies and patterns which are changing 177 dynamically. 179 In non-flexible mode, the inter-picture dependency (the reference 180 indices) of a Picture Group (PG) MUST be pre-specified as part of the 181 scalability structure (SS) data. In this mode, each packet has an 182 index to refer to one of the described pictures in the PG, from which 183 the pictures referenced by the picture transmitted in the current 184 packet for inter-picture prediction can be identified. 186 (Editor's Note: A "Picture Group", as used in this document, is not 187 the same thing as a the term "Group of Pictures" as it is 188 traditionally used in video coding, i.e. to mean an independently- 189 decoadable run of pictures beginning with a keyframe. Suggestions 190 for better terminology are welcome.) 192 The SS data can also be used to specify the resolution of each 193 spatial layer present in the VP9 stream for both flexible and non- 194 flexible modes. 196 4. Payload Format 198 This section describes how the encoded VP9 bitstream is encapsulated 199 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 200 RECOMMENDED. All integer fields in the specifications are encoded as 201 unsigned integers in network octet order. 203 4.1. RTP Header Usage 205 The general RTP payload format for VP9 is depicted below. 207 0 1 2 3 208 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 209 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 210 |V=2|P|X| CC |M| PT | sequence number | 211 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 212 | timestamp | 213 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 214 | synchronization source (SSRC) identifier | 215 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 216 | contributing source (CSRC) identifiers | 217 | .... | 218 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 219 | VP9 payload descriptor (integer #octets) | 220 : : 221 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 222 | : VP9 pyld hdr | | 223 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 224 | | 225 + | 226 : Bytes 2..N of VP9 payload : 227 | | 228 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 229 | : OPTIONAL RTP padding | 230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 231 Figure 1 233 The VP9 payload descriptor will be described in Section 4.2; the VP9 234 payload header is described in [VP9-BITSTREAM]. OPTIONAL RTP padding 235 MUST NOT be included unless the P bit is set. The figure 236 specifically shows the format for the first packet in a frame. 237 Subsequent packets will not contain the VP9 payload header, and will 238 have later octets in the frame payload. 240 Marker bit (M): MUST be set to 1 for the final packet of the highest 241 spatial layer frame (the final packet of the picture), and 0 242 otherwise. Unless spatial scalability is in use for this picture, 243 this will have the same value as the E bit described below. Note 244 this bit MUST be set to 1 for the target spatial layer frame if a 245 stream is being rewritten to remove higher spatial layers. 247 Payload Type (PT): In line with the policy in Section 3 of 248 [RFC3551], applications using the VP9 RTP payload profile MUST 249 assign a dynamic payload type number to be used in each RTP 250 session and provide a mechanism to indicate the mapping. See 251 Section 6.2 for the mechanism to be used with the Session 252 Description Protocol (SDP) [RFC8866]. 254 Timestamp: The RTP timestamp indicates the time when the input frame 255 was sampled, at a clock rate of 90 kHz. If the input picture is 256 encoded with multiple layer frames, all of the frames of the 257 picture MUST have the same timestamp. 259 If a frame has the VP9 show_frame field set to 0 (i.e., it is 260 meant only to populate a reference buffer, without being output) 261 its timestamp MAY alternately be set to be the same as the 262 subsequent frame with show_frame equal to 1. (This will be 263 convenient for playing out pre-encoded content packaged with VP9 264 "superframes", which typically bundle show_frame==0 frames with a 265 subsequent show_frame==1 frame.) Every frame with show_frame==1, 266 however, MUST have a unique timestamp modulo the 2^32 wrap of the 267 field. 269 The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, 270 SSRC and CSRC identifiers) are used as specified in Section 5.1 of 271 [RFC3550]. 273 4.2. VP9 Payload Descriptor 275 In flexible mode (with the F bit below set to 1), The first octets 276 after the RTP header are the VP9 payload descriptor, with the 277 following structure. 279 0 1 2 3 4 5 6 7 280 +-+-+-+-+-+-+-+-+ 281 |I|P|L|F|B|E|V|Z| (REQUIRED) 282 +-+-+-+-+-+-+-+-+ 283 I: |M| PICTURE ID | (REQUIRED) 284 +-+-+-+-+-+-+-+-+ 285 M: | EXTENDED PID | (RECOMMENDED) 286 +-+-+-+-+-+-+-+-+ 287 L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED) 288 +-+-+-+-+-+-+-+-+ -\ 289 P,F: | P_DIFF |N| (CONDITIONALLY REQUIRED) - up to 3 times 290 +-+-+-+-+-+-+-+-+ -/ 291 V: | SS | 292 | .. | 293 +-+-+-+-+-+-+-+-+ 295 Figure 2 297 In non-flexible mode (with the F bit below set to 0), The first 298 octets after the RTP header are the VP9 payload descriptor, with the 299 following structure. 301 0 1 2 3 4 5 6 7 302 +-+-+-+-+-+-+-+-+ 303 |I|P|L|F|B|E|V|Z| (REQUIRED) 304 +-+-+-+-+-+-+-+-+ 305 I: |M| PICTURE ID | (RECOMMENDED) 306 +-+-+-+-+-+-+-+-+ 307 M: | EXTENDED PID | (RECOMMENDED) 308 +-+-+-+-+-+-+-+-+ 309 L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED) 310 +-+-+-+-+-+-+-+-+ 311 | TL0PICIDX | (CONDITIONALLY REQUIRED) 312 +-+-+-+-+-+-+-+-+ 313 V: | SS | 314 | .. | 315 +-+-+-+-+-+-+-+-+ 317 Figure 3 319 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 320 be present after the mandatory first octet and specified as below. 321 Otherwise, PID MUST NOT be present. If the SS field was present 322 in the stream's most recent start of a keyframe (i.e., non- 323 flexible scalability mode is in use), then the PID MUST also be 324 present in every packet. 326 P: Inter-picture predicted frame. When set to zero, the frame does 327 not utilize inter-picture prediction. In this case, up-switching 328 to a current spatial layer's frame is possible from directly lower 329 spatial layer frame. P SHOULD also be set to zero when encoding a 330 layer synchronization frame in response to an LRR 331 [I-D.ietf-avtext-lrr] message (see Section 5.3). When P is set to 332 zero, the TID field (described below) MUST also be set to 0 (if 333 present). Note that the P bit does not forbid intra-picture, 334 inter-layer prediction from earlier frames of the same picture, if 335 any. 337 L: Layer indices present. When set to one, the one or two octets 338 following the mandatory first octet and the PID (if present) is as 339 described by "Layer indices" below. If the F bit (described 340 below) is set to 1 (indicating flexible mode), then only one octet 341 is present for the layer indices. Otherwise if the F bit is set 342 to 0 (indicating non-flexible mode), then two octets are present 343 for the layer indices. 345 F: Flexible mode. F set to one indicates flexible mode and if the P 346 bit is also set to one, then the octets following the mandatory 347 first octet, the PID, and layer indices (if present) are as 348 described by "Reference indices" below. This MUST only be set to 349 1 if the I bit is also set to one; if the I bit is set to zero, 350 then this MUST also be set to zero and ignored by receivers. The 351 value of this F bit MUST only change on the first packet of a key 352 picture. A key picture is a picture whose base spatial layer 353 frame is a key frame, and which thus completely resets the encoder 354 state. This packet will have its P bit equal to zero, SID or D 355 bit (described below) equal to zero, and B bit (described below) 356 equal to 1. 358 B: Start of a frame. MUST be set to 1 if the first payload octet of 359 the RTP packet is the beginning of a new VP9 frame, and MUST NOT 360 be 1 otherwise. Note that this frame might not be the first frame 361 of a picture. 363 E: End of a frame. MUST be set to 1 for the final RTP packet of a 364 VP9 frame, and 0 otherwise. This enables a decoder to finish 365 decoding the frame, where it otherwise may need to wait for the 366 next packet to explicitly know that the frame is complete. Note 367 that, if spatial scalability is in use, more frames from the same 368 picture may follow; see the description of the M bit above. 370 V: Scalability structure (SS) data present. When set to one, the 371 OPTIONAL SS data MUST be present in the payload descriptor. 372 Otherwise, the SS data MUST NOT be present. 374 Z: Not a reference frame for upper spatial layers. If set to 1, 375 indicates that frames with higher spatial layers SID+1 of the 376 current and following pictures do not depend on the current 377 spatial layer SID frame. This enables a decoder which is 378 targeting a higher spatial layer to know that it can safely 379 discard this packet's frame without processing it, without having 380 to wait for the "D" bit in the higher-layer frame (see below). 382 The mandatory first octet is followed by the extension data fields 383 that are enabled: 385 M: The most significant bit of the first octet is an extension flag. 386 The field MUST be present if the I bit is equal to one. If set, 387 the PID field MUST contain 15 bits; otherwise, it MUST contain 7 388 bits. See PID below. 390 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 391 on the M bit. This is a running index of the pictures. The field 392 MUST be present if the I bit is equal to one. If M is set to 393 zero, 7 bits carry the PID; else if M is set to one, 15 bits carry 394 the PID in network byte order. The sender may choose between a 7- 395 or 15-bit index. The PID SHOULD start on a random number, and 396 MUST wrap after reaching the maximum ID. The receiver MUST NOT 397 assume that the number of bits in PID stay the same through the 398 session. 400 In the non-flexible mode (when the F bit is set to 0), this PID is 401 used as an index to the picture group (PG) specified in the SS 402 data below. In this mode, the PID of the key frame corresponds to 403 the first specified frame in the PG. Then subsequent PIDs are 404 mapped to subsequently specified frames in the PG (modulo N_G, 405 specified in the SS data below), respectively. 407 All frames of the same picture MUST have the same PID value. 409 Frames (and their corresponding pictures) with the VP9 show_frame 410 field equal to 0 MUST have distinct PID values from subsequent 411 pictures with show_frame equal to 1. Thus, a Picture as defined 412 in this specification is different than a VP9 Superframe. 414 All frames of the same picture MUST have the same value for 415 show_frame. 417 Layer indices: This information is optional but recommended whenever 418 encoding with layers. For both flexible and non-flexible modes, 419 one octet is used to specify a layer frame's temporal layer ID 420 (TID) and spatial layer ID (SID) as shown both in Figure 2 and 421 Figure 3. Additionally, a bit (U) is used to indicate that the 422 current frame is a "switching up point" frame. Another bit (D) is 423 used to indicate whether inter-layer prediction is used for the 424 current frame. 426 In the non-flexible mode (when the F bit is set to 0), another 427 octet is used to represent temporal layer 0 index (TL0PICIDX), as 428 depicted in Figure 3. The TL0PICIDX is present so that all 429 minimally required frames - the base temporal layer frames - can 430 be tracked. 432 The TID and SID fields indicate the temporal and spatial layers 433 and can help middleboxes and and endpoints quickly identify which 434 layer a packet belongs to. 436 TID: The temporal layer ID of current frame. In the case of non- 437 flexible mode, if PID is mapped to a picture in a specified PG, 438 then the value of TID MUST match the corresponding TID value of 439 the mapped picture in the PG. 441 U: Switching up point. If this bit is set to 1 for the current 442 picture with temporal layer ID equal to TID, then "switch up" 443 to a higher frame rate is possible as subsequent higher 444 temporal layer pictures will not depend on any picture before 445 the current picture (in coding order) with temporal layer ID 446 greater than TID. 448 SID: The spatial layer ID of current frame. Note that frames 449 with spatial layer SDI > 0 may be dependent on decoded spatial 450 layer SID-1 frame within the same picture. Different frames of 451 the same picture MUST have distinct spatial layer IDs, and 452 frames' spatial layers MUST appear in increasing order within 453 the frame. 455 D: Inter-layer dependency used. MUST be set to one if current 456 spatial layer SID frame depends on spatial layer SID-1 frame of 457 the same picture. MUST only be set to zero if current spatial 458 layer SID frame does not depend on spatial layer SID-1 frame of 459 the same picture. For the base layer frame (with SID equal to 460 0), this D bit MUST be set to zero. 462 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 463 present in the non-flexible mode (F = 0). This is a running 464 index for the temporal base layer pictures, i.e., the pictures 465 with TID set to 0. If TID is larger than 0, TL0PICIDX 466 indicates which temporal base layer picture the current picture 467 depends on. TL0PICIDX MUST be incremented when TID is equal to 468 0. The index SHOULD start on a random number, and MUST restart 469 at 0 after reaching the maximum number 255. 471 Reference indices: When P and F are both set to one, indicating a 472 non-key frame in flexible mode, then at least one reference index 473 has to be specified as below. Additional reference indices (total 474 of up to 3 reference indices are allowed) may be specified using 475 the N bit below. When either P or F is set to zero, then no 476 reference index is specified. 478 P_DIFF: The reference index (in 7 bits) specified as the relative 479 PID from the current picture. For example, when P_DIFF=3 on a 480 packet containing the picture with PID 112 means that the 481 picture refers back to the picture with PID 109. This 482 calculation is done modulo the size of the PID field, i.e., 483 either 7 or 15 bits. 485 N: 1 if there is additional P_DIFF following the current P_DIFF. 487 4.2.1. Scalability Structure (SS): 489 The scalability structure (SS) data describes the resolution of each 490 frame within a picture as well as the inter-picture dependencies for 491 a picture group (PG). If the VP9 payload descriptor's "V" bit is 492 set, the SS data is present in the position indicated in Figure 2 and 493 Figure 3. 495 +-+-+-+-+-+-+-+-+ 496 V: | N_S |Y|G|-|-|-| 497 +-+-+-+-+-+-+-+-+ -\ 498 Y: | WIDTH | (OPTIONAL) . 499 + + . 500 | | (OPTIONAL) . 501 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 502 | HEIGHT | (OPTIONAL) . 503 + + . 504 | | (OPTIONAL) . 505 +-+-+-+-+-+-+-+-+ -/ 506 G: | N_G | (OPTIONAL) 507 +-+-+-+-+-+-+-+-+ -\ 508 N_G: | TID |U| R |-|-| (OPTIONAL) . 509 +-+-+-+-+-+-+-+-+ -\ . - N_G times 510 | P_DIFF | (OPTIONAL) . - R times . 511 +-+-+-+-+-+-+-+-+ -/ -/ 513 Figure 4 515 N_S: N_S + 1 indicates the number of spatial layers present in the 516 VP9 stream. 518 Y: Each spatial layer's frame resolution present. When set to one, 519 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 520 present for each layer frame. Otherwise, the resolution MUST NOT 521 be present. 523 G: PG description present flag. 525 -: Bit reserved for future use. MUST be set to zero and MUST be 526 ignored by the receiver. 528 N_G: N_G indicates the number of pictures in a Picture Group (PG). 529 If N_G is greater than 0, then the SS data allows the inter- 530 picture dependency structure of the VP9 stream to be pre-declared, 531 rather than indicating it on the fly with every packet. If N_G is 532 greater than 0, then for N_G pictures in the PG, each picture's 533 temporal layer ID (TID), switch up point (U), and the R reference 534 indices (P_DIFFs) are specified. 536 The first picture specified in the PG MUST have TID set to 0. 538 G set to 0 or N_G set to 0 indicates that either there is only one 539 temporal layer or no fixed inter-picture dependency information is 540 present going forward in the bitstream. 542 Note that for a given picture, all frames follow the same inter- 543 picture dependency structure. However, the frame rate of each 544 spatial layer can be different from each other and this can be 545 controlled with the use of the D bit described above. The 546 specified dependency structure in the SS data MUST be for the 547 highest frame rate layer. 549 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 550 included in the first packet of every key frame. This is a packet 551 with P bit equal to zero, SID or D bit equal to zero, and B bit equal 552 to 1. The SS data MUST only be changed on the picture that 553 corresponds to the first picture specified in the previous SS data's 554 PG (if the previous SS data's N_G was greater than 0). 556 4.3. Frame Fragmentation 558 VP9 frames are fragmented into packets, in RTP sequence number order, 559 beginning with a packet with the B bit set, and ending with a packet 560 with the E bit set. There is no mechanism for finer-grained access 561 to parts of a VP9 frame. 563 4.4. Scalable encoding considerations 565 In addition to the use of reference frames, VP9 has several 566 additional forms of inter-frame dependencies, largely involving 567 probability tables for the entropy and tree encoders. In VP9 syntax, 568 the syntax element "error_resilient_mode" resets this additional 569 inter-frame data, allowing a frame's syntax to be decoded 570 independently. 572 Due to the requirements of scalable streams, a VP9 encoder producing 573 a scalable stream needs to ensure that a frame does not depend on a 574 previous frame (of the same or a previous picture) that can 575 legitimately be removed from the stream. Thus, a frame that follows 576 a removable frame (in full decode order) MUST be encoded with 577 "error_resilient_mode" set to true. 579 For spatially-scalable streams, this means that 580 "error_resilient_mode" needs to be turned on for the base spatial 581 layer; it can however be turned off for higher spatial layers, 582 assuming they are sent with inter-layer dependency (i.e. with the "D" 583 bit set). For streams that are only temporally-scalable without 584 spatial scalability, "error_resilient_mode" can additionally be 585 turned off for any picture that immediately follows a temporal layer 586 0 frame. 588 4.5. Examples of VP9 RTP Stream 590 4.5.1. Reference picture use for scalable structure 592 As discussed in Section 3, the VP9 codec can maintain up to eight 593 reference frames, of which up to three can be referenced or updated 594 by any new frame. This section illustrates one way that a scalable 595 structure (with three spatial layers and three temporal layers) can 596 be constructed using these reference frames. 598 +==========+=========+============+=========+ 599 | Temporal | Spatial | References | Updates | 600 +==========+=========+============+=========+ 601 | 0 | 0 | 0 | 0 | 602 +----------+---------+------------+---------+ 603 | 0 | 1 | 0,1 | 1 | 604 +----------+---------+------------+---------+ 605 | 0 | 2 | 1,2 | 2 | 606 +----------+---------+------------+---------+ 607 | 2 | 0 | 0 | 6 | 608 +----------+---------+------------+---------+ 609 | 2 | 1 | 1,6 | 7 | 610 +----------+---------+------------+---------+ 611 | 2 | 2 | 2,7 | - | 612 +----------+---------+------------+---------+ 613 | 1 | 0 | 0 | 3 | 614 +----------+---------+------------+---------+ 615 | 1 | 1 | 1,3 | 4 | 616 +----------+---------+------------+---------+ 617 | 1 | 2 | 2,4 | 5 | 618 +----------+---------+------------+---------+ 619 | 2 | 0 | 3 | 6 | 620 +----------+---------+------------+---------+ 621 | 2 | 1 | 4,6 | 7 | 622 +----------+---------+------------+---------+ 623 | 2 | 2 | 5,7 | - | 624 +----------+---------+------------+---------+ 626 Table 1: Example scalability structure 628 This structure is constructed such that the "U" bit can always be 629 set. 631 5. Feedback Messages and Header Extensions 633 5.1. Reference Picture Selection Indication (RPSI) 635 The reference picture selection index is a payload-specific feedback 636 message defined within the RTCP-based feedback format. The RPSI 637 message is generated by a receiver and can be used in two ways. 638 Either it can signal a preferred reference picture when a loss has 639 been detected by the decoder -- preferably then a reference that the 640 decoder knows is perfect -- or, it can be used as positive feedback 641 information to acknowledge correct decoding of certain reference 642 pictures. The positive feedback method is useful for VP9 used for 643 point to point (unicast) communication. The use of RPSI for VP9 is 644 preferably combined with a special update pattern of the codec's two 645 special reference frames -- the golden frame and the altref frame -- 646 in which they are updated in an alternating leapfrog fashion. When a 647 receiver has received and correctly decoded a golden or altref frame, 648 and that frame had a PictureID in the payload descriptor, the 649 receiver can acknowledge this simply by sending an RPSI message back 650 to the sender. The message body (i.e., the "native RPSI bit string" 651 in [RFC4585]) is simply the PictureID of the received frame. 653 Note: because all frames of the same picture must have the same 654 inter-picture reference structure, there is no need for a message to 655 specify which frame is being selected. 657 5.2. Full Intra Request (FIR) 659 The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a 660 receiver to request a full state refresh of an encoded stream. 662 Upon receipt of an FIR request, a VP9 sender MUST send a picture with 663 a keyframe for its spatial layer 0 layer frame, and then send frames 664 without inter-picture prediction (P=0) for any higher layer frames. 666 5.3. Layer Refresh Request (LRR) 668 The Layer Refresh Request [I-D.ietf-avtext-lrr] allows a receiver to 669 request a single layer of a spatially or temporally encoded stream to 670 be refreshed, without necessarily affecting the stream's other 671 layers. 673 +---------------+---------------+ 674 |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7| 675 +---------------+---------+-----+ 676 | RES | TID | RES | SID | 677 +---------------+---------+-----+ 679 Figure 5 681 Figure 5 shows the format of LRR's layer index fields for VP9 682 streams. The two "RES" fields MUST be set to 0 on transmission and 683 ingnored on reception. See Section 4.2 for details on the TID and 684 SID fields. 686 Identification of a layer refresh frame can be derived from the 687 reference IDs of each frame by backtracking the dependency chain 688 until reaching a point where only decodable frames are being 689 referenced. Therefore it's recommended for both the flexible and the 690 non-flexible mode that, when upgrade frames are being encoded in 691 response to a LRR, those packets should contain layer indices and the 692 reference fields so that the decoder or an MCU can make this 693 derivation. 695 Example: 697 LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying 698 {1,0} to a receiver and which wants to upgrade to {2,1}. In response 699 the encoder should encode the next frames in layers {1,1} and {2,1} 700 by only referring to frames in {1,0}, or {0,0}. 702 In the non-flexible mode, periodic upgrade frames can be defined by 703 the layer structure of the SS, thus periodic upgrade frames can be 704 automatically identified by the picture ID. 706 6. Payload Format Parameters 708 This payload format has two optional parameters. 710 6.1. Media Type Definition 712 This registration is done using the template defined in [RFC6838] and 713 following [RFC4855]. 715 Type name: 716 video 718 Subtype name: 719 VP9 721 Required parameters: 722 None. 724 Optional parameters: 725 These parameters are used to signal the capabilities of a receiver 726 implementation. If the implementation is willing to receive 727 media, both parameters MUST be provided. These parameters MUST 728 NOT be used for any other purpose. 730 max-fr: The value of max-fr is an integer indicating the maximum 731 frame rate in units of frames per second that the decoder is 732 capable of decoding. 734 max-fs: The value of max-fs is an integer indicating the maximum 735 frame size in units of macroblocks that the decoder is capable 736 of decoding. 738 The decoder is capable of decoding this frame size as long as 739 the width and height of the frame in macroblocks are less than 740 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 741 of supporting 640x480 resolution) will support widths and 742 heights up to 1552 pixels (97 macroblocks). 744 profile-id: The value of profile-id is an integer indicating the 745 default coding profile, the subset of coding tools that may 746 have been used to generate the stream or that the receiver 747 supports). Table 2 lists all of the profiles defined in 748 section 7.2 of [VP9-BITSTREAM] and the corresponding integer 749 values to be used. 751 If no profile-id is present, Profile 0 MUST be inferred. 753 Informative note: See Table 3 for capabilities of coding 754 profiles defined in section 7.2 of [VP9-BITSTREAM]. 756 Encoding considerations: 757 This media type is framed in RTP and contains binary data; see 758 Section 4.8 of [RFC6838]. 760 Security considerations: 761 See Section 7 of RFC xxxx. 763 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 764 with the number assigned to this document and remove this note.] 766 Interoperability considerations: 767 None. 769 Published specification: 770 VP9 bitstream format [VP9-BITSTREAM] and RFC XXXX. 772 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 773 with the number assigned to this document and remove this note.] 775 Applications which use this media type: 776 For example: Video over IP, video conferencing. 778 Fragment identifier considerations: 779 N/A. 781 Additional information: 782 None. 784 Person & email address to contact for further information: 785 Jonathan Lennox 787 Intended usage: 788 COMMON 790 Restrictions on usage: 791 This media type depends on RTP framing, and hence is only defined 792 for transfer via RTP [RFC3550]. 794 Author: 795 Jonathan Lennox 797 Change controller: 798 IETF AVTCore Working Group delegated from the IESG. 800 +=========+============+ 801 | Profile | profile-id | 802 +=========+============+ 803 | 0 | 0 | 804 +---------+------------+ 805 | 1 | 1 | 806 +---------+------------+ 807 | 2 | 2 | 808 +---------+------------+ 809 | 3 | 3 | 810 +---------+------------+ 812 Table 2: Table 1. 813 Table of profile-id 814 integer values 815 representing the VP9 816 profile corresponding 817 to the set of coding 818 tools supported. 820 +=========+===========+=================+==========================+ 821 | Profile | Bit Depth | SRGB Colorspace | Chroma Subsampling | 822 +=========+===========+=================+==========================+ 823 | 0 | 8 | No | YUV 4:2:0 | 824 +---------+-----------+-----------------+--------------------------+ 825 | 1 | 8 | Yes | YUV 4:2:0,4:4:0 or 4:4:4 | 826 +---------+-----------+-----------------+--------------------------+ 827 | 2 | 10 or 12 | No | YUV 4:2:0 | 828 +---------+-----------+-----------------+--------------------------+ 829 | 3 | 10 or 12 | Yes | YUV 4:2:0,4:4:0 or 4:4:4 | 830 +---------+-----------+-----------------+--------------------------+ 832 Table 3: Table 2. Table of profile capabilities. 834 6.2. SDP Parameters 836 The receiver MUST ignore any fmtp parameter unspecified in this memo. 838 6.2.1. Mapping of Media Subtype Parameters to SDP 840 The media type video/VP9 string is mapped to fields in the Session 841 Description Protocol (SDP) [RFC8866] as follows: 843 * The media name in the "m=" line of SDP MUST be video. 845 * The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 846 media subtype). 848 * The clock rate in the "a=rtpmap" line MUST be 90000. 850 * The parameters "max-fs", and "max-fr", MUST be included in the 851 "a=fmtp" line of SDP if SDP is used to declare receiver 852 capabilities. These parameters are expressed as a media subtype 853 string, in the form of a semicolon separated list of 854 parameter=value pairs. 856 * The OPTIONAL parameter profile-id, when present, SHOULD be 857 included in the "a=fmtp" line of SDP. This parameter is expressed 858 as a media subtype string, in the form of a parameter=value pair. 859 When the parameter is not present, a value of 0 MUST be used for 860 profile-id. 862 6.2.1.1. Example 864 An example of media representation in SDP is as follows: 866 m=video 49170 RTP/AVPF 98 a=rtpmap:98 VP9/90000 a=fmtp:98 max-fr=30; 867 max-fs=3600; profile-id=0; 869 6.2.2. Offer/Answer Considerations 871 When VP9 is offered over RTP using SDP in an Offer/Answer model 872 [RFC3264] for negotiation for unicast usage, the following 873 limitations and rules apply: 875 * The parameter identifying a media format configuration for VP9 is 876 profile-id. This media format configuration parameter MUST be 877 used symmetrically; that is, the answerer MUST either maintain all 878 configuration parameters or remove the media format (payload type) 879 completely if one or more of the parameter values are not 880 supported. 882 * To simplify the handling and matching of these configurations, the 883 same RTP payload type number used in the offer SHOULD also be used 884 in the answer, as specified in [RFC3264]. An answer MUST NOT 885 contain the payload type number used in the offer unless the 886 configuration is exactly the same as in the offer. 888 7. Security Considerations 890 RTP packets using the payload format defined in this specification 891 are subject to the security considerations discussed in the RTP 892 specification [RFC3550], and in any applicable RTP profile such as 893 RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/ 894 SAVPF [RFC5124]. SAVPF [RFC5124]. However, as "Securing the RTP 895 Protocol Framework: Why RTP Does Not Mandate a Single Media Security 896 Solution" [RFC7202] discusses, it is not an RTP payload format's 897 responsibility to discuss or mandate what solutions are used to meet 898 the basic security goals like confidentiality, integrity and source 899 authenticity for RTP in general. This responsibility lays on anyone 900 using RTP in an application. They can find guidance on available 901 security mechanisms in Options for Securing RTP Sessions [RFC7201]. 902 Applications SHOULD use one or more appropriate strong security 903 mechanisms. The rest of this security consideration section 904 discusses the security impacting properties of the payload format 905 itself. 907 This RTP payload format and its media decoder do not exhibit any 908 significant non-uniformity in the receiver-side computational 909 complexity for packet processing, and thus are unlikely to pose a 910 denial-of-service threat due to the receipt of pathological data. 911 Nor does the RTP payload format contain any active content. 913 8. Congestion Control 915 Congestion control for RTP SHALL be used in accordance with RFC 3550 916 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 917 [RFC3551]. The congestion control mechanism can, in a real-time 918 encoding scenario, adapt the transmission rate by instructing the 919 encoder to encode at a certain target rate. Media aware network 920 elements MAY use the information in the VP9 payload descriptor in 921 Section 4.2 to identify non-reference frames and discard them in 922 order to reduce network congestion. Note that discarding of non- 923 reference frames cannot be done if the stream is encrypted (because 924 the non-reference marker is encrypted). 926 9. IANA Considerations 928 The IANA is requested to register the media type registration "video/ 929 vp9" as specified in Section 6.1. The media type is also requested 930 to be added to the IANA registry for "RTP Payload Format MIME types" 931 . 933 10. Acknowledgments 935 Alex Eleftheriadis, Yuki Ito, Won Kap Jang, Sergio Garcia Murillo, 936 Roi Sasson, Timothy Terriberry, Emircan Uysaler, and Thomas Volkert 937 commented on the development of this document and provided helpful 938 comments and feedback. 940 11. References 942 11.1. Normative References 944 [I-D.ietf-avtext-lrr] 945 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 946 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 947 Message", Work in Progress, Internet-Draft, draft-ietf- 948 avtext-lrr-07, 2 July 2017, . 951 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 952 Requirement Levels", BCP 14, RFC 2119, 953 DOI 10.17487/RFC2119, March 1997, 954 . 956 [RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model 957 with Session Description Protocol (SDP)", RFC 3264, 958 DOI 10.17487/RFC3264, June 2002, 959 . 961 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 962 Jacobson, "RTP: A Transport Protocol for Real-Time 963 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 964 July 2003, . 966 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 967 "Extended RTP Profile for Real-time Transport Control 968 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 969 DOI 10.17487/RFC4585, July 2006, 970 . 972 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 973 Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, 974 . 976 [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, 977 "Codec Control Messages in the RTP Audio-Visual Profile 978 with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, 979 February 2008, . 981 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 982 Specifications and Registration Procedures", BCP 13, 983 RFC 6838, DOI 10.17487/RFC6838, January 2013, 984 . 986 [RFC8866] Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP: 987 Session Description Protocol", RFC 8866, 988 DOI 10.17487/RFC8866, January 2021, 989 . 991 [VP9-BITSTREAM] 992 Grange, A., de Rivaz, P., and J. Hunt, "VP9 Bitstream & 993 Decoding Process Specification", Version 0.6, 31 March 994 2016, 995 . 999 11.2. Informative References 1001 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1002 Video Conferences with Minimal Control", STD 65, RFC 3551, 1003 DOI 10.17487/RFC3551, July 2003, 1004 . 1006 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1007 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1008 RFC 3711, DOI 10.17487/RFC3711, March 2004, 1009 . 1011 [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for 1012 Real-time Transport Control Protocol (RTCP)-Based Feedback 1013 (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 1014 2008, . 1016 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 1017 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 1018 Guide", RFC 6386, DOI 10.17487/RFC6386, November 2011, 1019 . 1021 [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP 1022 Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, 1023 . 1025 [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP 1026 Framework: Why RTP Does Not Mandate a Single Media 1027 Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 1028 2014, . 1030 Authors' Addresses 1032 Justin Uberti 1033 Google, Inc. 1034 747 6th Street South 1035 Kirkland, WA 98033 1036 United States of America 1038 Email: justin@uberti.name 1040 Stefan Holmer 1041 Google, Inc. 1042 Kungsbron 2 1043 SE-111 22 Stockholm 1044 Sweden 1046 Email: holmer@google.com 1048 Magnus Flodman 1049 Google, Inc. 1050 Kungsbron 2 1051 SE-111 22 Stockholm 1052 Sweden 1054 Email: mflodman@google.com 1056 Danny Hong 1057 Google, Inc. 1058 1585 Charleston Road 1059 Mountain View, CA 94043 1060 United States of America 1062 Email: dannyhong@google.com 1063 Jonathan Lennox 1064 8x8, Inc. / Jitsi 1065 1350 Broadway 1066 New York, NY 10018 1067 United States of America 1069 Email: jonathan.lennox@8x8.com