idnits 2.17.1 draft-ietf-payload-vp9-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (3 June 2021) is 1058 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'VP9-BITSTREAM' Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 AVTCore Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: 5 December 2021 D. Hong 6 Google 7 J. Lennox 8 8x8 / Jitsi 9 3 June 2021 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-14 14 Abstract 16 This specification describes an RTP payload format for the VP9 video 17 codec. The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on 5 December 2021. 39 Copyright Notice 41 Copyright (c) 2021 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 46 license-info) in effect on the date of publication of this document. 47 Please review these documents carefully, as they describe your rights 48 and restrictions with respect to this document. Code Components 49 extracted from this document must include Simplified BSD License text 50 as described in Section 4.e of the Trust Legal Provisions and are 51 provided without warranty as described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 57 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 58 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 5 59 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 60 4.2. VP9 Payload Descriptor . . . . . . . . . . . . . . . . . 6 61 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 11 62 4.3. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 13 63 4.4. Scalable encoding considerations . . . . . . . . . . . . 13 64 4.5. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 13 65 4.5.1. Reference picture use for scalable structure . . . . 14 66 5. Feedback Messages and Header Extensions . . . . . . . . . . . 14 67 5.1. Reference Picture Selection Indication (RPSI) . . . . . . 15 68 5.2. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 15 69 5.3. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 15 70 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 16 71 6.1. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 18 72 6.1.1. Mapping of Media Subtype Parameters to SDP . . . . . 18 73 6.1.2. Offer/Answer Considerations . . . . . . . . . . . . . 19 74 7. Media Type Definition . . . . . . . . . . . . . . . . . . . . 19 75 8. Security Considerations . . . . . . . . . . . . . . . . . . . 21 76 9. Congestion Control . . . . . . . . . . . . . . . . . . . . . 21 77 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 78 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 79 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 80 12.1. Normative References . . . . . . . . . . . . . . . . . . 22 81 12.2. Informative References . . . . . . . . . . . . . . . . . 23 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 84 1. Introduction 86 This specification describes an RTP [RFC3550] payload specification 87 applicable to the transmission of video streams encoded using the VP9 88 video codec [VP9-BITSTREAM]. The format described in this document 89 can be used both in peer-to-peer and video conferencing applications. 91 The VP9 video codec was developed by Google, and is the successor to 92 its earlier VP8 [RFC6386] codec. Above the compression improvements 93 and other general enhancements above VP8, VP9 is also designed in a 94 way that allows spatially-scalable video encoding. 96 2. Conventions, Definitions and Acronyms 98 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 99 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 100 "OPTIONAL" in this document are to be interpreted as described in BCP 101 14 [RFC2119] [RFC8174] when, and only when, they appear in all 102 capitals, as shown here. 104 3. Media Format Description 106 The VP9 codec can maintain up to eight reference frames, of which up 107 to three can be referenced by any new frame. 109 VP9 also allows a frame to use another frame of a different 110 resolution as a reference frame. (Specifically, a frame may use any 111 references whose width and height are between 1/16th that of the 112 current frame and twice that of the current frame, inclusive.) This 113 allows internal resolution changes without requiring the use of key 114 frames. 116 These features together enable an encoder to implement various forms 117 of coarse-grained scalability, including temporal, spatial and 118 quality scalability modes, as well as combinations of these, without 119 the need for explicit scalable coding tools. 121 Temporal layers define different frame rates of video; spatial and 122 quality layers define different and possibly dependent 123 representations of a single input frame. Spatial layers allow a 124 frame to be encoded at different resolutions, whereas quality layers 125 allow a frame to be encoded at the same resolution but at different 126 qualities (and thus with different amounts of coding error). VP9 127 supports quality layers as spatial layers without any resolution 128 changes; hereinafter, the term "spatial layer" is used to represent 129 both spatial and quality layers. 131 This payload format specification defines how such temporal and 132 spatial scalability layers can be described and communicated. 134 Temporal and spatial scalability layers are associated with non- 135 negative integer IDs. The lowest layer of either type has an ID of 136 0, and is sometimes referred to as the "base" temporal or spatial 137 layer. 139 Layers are designed, and MUST be encoded, such that if any layer, and 140 all higher layers, are removed from the bitstream along either the 141 spatial or temporal dimension, the remaining bitstream is still 142 correctly decodable. 144 For terminology, this document uses the term "frame" to refer to a 145 single encoded VP9 frame for a particular resolution/quality, and 146 "picture" to refer to all the representations (frames) at a single 147 instant in time. A picture thus consists of one or more frames, 148 encoding different spatial layers. 150 Within a picture, a frame with spatial layer ID equal to SID, where 151 SID > 0, can depend on a frame of the same picture with a lower 152 spatial layer ID. This "inter-layer" dependency can result in 153 additional coding gain compared to the case where only traditional 154 "inter-picture" dependency is used, where a frame depends on 155 previously coded frame in time. For simplicity, this payload format 156 assumes that, within a picture and if inter-layer dependency is used, 157 a spatial layer SID frame can depend only on the immediately previous 158 spatial layer SID-1 frame, when S > 0. Additionally, if inter- 159 picture dependency is used, a spatial layer SID frame is assumed to 160 only depend on a previously coded spatial layer SID frame. 162 Given above simplifications for inter-layer and inter-picture 163 dependencies, a flag (the D bit described below) is used to indicate 164 whether a spatial layer SID frame depends on the spatial layer SID-1 165 frame. Given the D bit, a receiver only needs to additionally know 166 the inter-picture dependency structure for a given spatial layer 167 frame in order to determine its decodability. Two modes of 168 describing the inter-picture dependency structure are possible: 169 "flexible mode" and "non-flexible mode". An encoder can only switch 170 between the two on the first packet of a key frame with temporal 171 layer ID equal to 0. 173 In flexible mode, each packet can contain up to 3 reference indices, 174 which identify all frames referenced by the frame transmitted in the 175 current packet for inter-picture prediction. This (along with the D 176 bit) enables a receiver to identify if a frame is decodable or not 177 and helps it understand the temporal layer structure. Since this is 178 signaled in each packet it makes it possible to have very flexible 179 temporal layer hierarchies, and scalability structures which are 180 changing dynamically. 182 In non-flexible mode, frames are encoded using a fixed, recurring 183 pattern of dependencies; the set of pictures that recur in this 184 pattern is known as a Picture Group (PG). In this mode, the inter- 185 picture dependencies (the reference indices) of the Picture Group 186 MUST be pre-specified as part of the scalability structure (SS) data. 187 A Picture Group is a recurring pattern of spatial and temporal 188 dependencies which In this mode, each packet has an index to refer to 189 one of the described pictures in the PG, from which the pictures 190 referenced by the picture transmitted in the current packet for 191 inter-picture prediction can be identified. 193 (Note: A "Picture Group", as used in this document, is not the same 194 thing as the term "Group of Pictures" as it is traditionally used in 195 video coding, i.e. to mean an independently-decoadable run of 196 pictures beginning with a keyframe.) 198 The SS data can also be used to specify the resolution of each 199 spatial layer present in the VP9 stream for both flexible and non- 200 flexible modes. 202 4. Payload Format 204 This section describes how the encoded VP9 bitstream is encapsulated 205 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 206 RECOMMENDED. All integer fields in the specifications are encoded as 207 unsigned integers in network octet order. 209 4.1. RTP Header Usage 211 The general RTP payload format for VP9 is depicted below. 213 0 1 2 3 214 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 215 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 216 |V=2|P|X| CC |M| PT | sequence number | 217 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 218 | timestamp | 219 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 220 | synchronization source (SSRC) identifier | 221 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 222 | contributing source (CSRC) identifiers | 223 | .... | 224 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 225 | VP9 payload descriptor (integer #octets) | 226 : : 227 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 228 | : | 229 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 230 | | 231 + | 232 : VP9 payload : 233 | | 234 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 235 | : OPTIONAL RTP padding | 236 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 237 Figure 1 239 The VP9 payload descriptor will be described in Section 4.2; the VP9 240 payload is described in [VP9-BITSTREAM]. OPTIONAL RTP padding MUST 241 NOT be included unless the P bit is set. 243 Marker bit (M): MUST be set to 1 for the final packet of the highest 244 spatial layer frame (the final packet of the picture), and 0 245 otherwise. Unless spatial scalability is in use for this picture, 246 this will have the same value as the E bit described below. Note 247 this bit MUST be set to 1 for the target spatial layer frame if a 248 stream is being rewritten to remove higher spatial layers. 250 Payload Type (PT): In line with the policy in Section 3 of 251 [RFC3551], applications using the VP9 RTP payload profile MUST 252 assign a dynamic payload type number to be used in each RTP 253 session and provide a mechanism to indicate the mapping. See 254 Section 6.1 for the mechanism to be used with the Session 255 Description Protocol (SDP) [RFC8866]. 257 Timestamp: The RTP timestamp [RFC3550] indicates the time when the 258 input frame was sampled, at a clock rate of 90 kHz. If the input 259 picture is encoded with multiple layer frames, all of the frames 260 of the picture MUST have the same timestamp. 262 If a frame has the VP9 show_frame field set to 0 (i.e., it is 263 meant only to populate a reference buffer, without being output) 264 its timestamp MAY alternatively be set to be the same as the 265 subsequent frame with show_frame equal to 1. (This will be 266 convenient for playing out pre-encoded content packaged with VP9 267 "superframes", which typically bundle show_frame==0 frames with a 268 subsequent show_frame==1 frame.) Every frame with show_frame==1, 269 however, MUST have a unique timestamp modulo the 2^32 wrap of the 270 field. 272 The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, 273 SSRC and CSRC identifiers) are used as specified in Section 5.1 of 274 [RFC3550]. 276 4.2. VP9 Payload Descriptor 278 In flexible mode (with the F bit below set to 1), the first octets 279 after the RTP header are the VP9 payload descriptor, with the 280 following structure. 282 0 1 2 3 4 5 6 7 283 +-+-+-+-+-+-+-+-+ 284 |I|P|L|F|B|E|V|Z| (REQUIRED) 285 +-+-+-+-+-+-+-+-+ 286 I: |M| PICTURE ID | (REQUIRED) 287 +-+-+-+-+-+-+-+-+ 288 M: | EXTENDED PID | (RECOMMENDED) 289 +-+-+-+-+-+-+-+-+ 290 L: | TID |U| SID |D| (Conditionally RECOMMENDED) 291 +-+-+-+-+-+-+-+-+ -\ 292 P,F: | P_DIFF |N| (Conditionally REQUIRED) - up to 3 times 293 +-+-+-+-+-+-+-+-+ -/ 294 V: | SS | 295 | .. | 296 +-+-+-+-+-+-+-+-+ 298 Figure 2 300 In non-flexible mode (with the F bit below set to 0), The first 301 octets after the RTP header are the VP9 payload descriptor, with the 302 following structure. 304 0 1 2 3 4 5 6 7 305 +-+-+-+-+-+-+-+-+ 306 |I|P|L|F|B|E|V|Z| (REQUIRED) 307 +-+-+-+-+-+-+-+-+ 308 I: |M| PICTURE ID | (RECOMMENDED) 309 +-+-+-+-+-+-+-+-+ 310 M: | EXTENDED PID | (RECOMMENDED) 311 +-+-+-+-+-+-+-+-+ 312 L: | TID |U| SID |D| (Conditionally RECOMMENDED) 313 +-+-+-+-+-+-+-+-+ 314 | TL0PICIDX | (Conditionally REQUIRED) 315 +-+-+-+-+-+-+-+-+ 316 V: | SS | 317 | .. | 318 +-+-+-+-+-+-+-+-+ 320 Figure 3 322 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 323 be present after the mandatory first octet and specified as below. 324 Otherwise, PID MUST NOT be present. If the V bit was set in the 325 stream's most recent start of a keyframe (i.e. the SS field was 326 present, and non-flexible scalability mode is in use), then this 327 bit MUST be set on every packet. 329 P: Inter-picture predicted frame. When set to zero, the frame does 330 not utilize inter-picture prediction. In this case, up-switching 331 to a current spatial layer's frame is possible from directly lower 332 spatial layer frame. P SHOULD also be set to zero when encoding a 333 layer synchronization frame in response to an LRR 334 [I-D.ietf-avtext-lrr] message (see Section 5.3). When P is set to 335 zero, the TID field (described below) MUST also be set to 0 (if 336 present). Note that the P bit does not forbid intra-picture, 337 inter-layer prediction from earlier frames of the same picture, if 338 any. 340 L: Layer indices present. When set to one, the one or two octets 341 following the mandatory first octet and the PID (if present) is as 342 described by "Layer indices" below. If the F bit (described 343 below) is set to 1 (indicating flexible mode), then only one octet 344 is present for the layer indices. Otherwise if the F bit is set 345 to 0 (indicating non-flexible mode), then two octets are present 346 for the layer indices. 348 F: Flexible mode. F set to one indicates flexible mode and if the P 349 bit is also set to one, then the octets following the mandatory 350 first octet, the PID, and layer indices (if present) are as 351 described by "Reference indices" below. This MUST only be set to 352 1 if the I bit is also set to one; if the I bit is set to zero, 353 then this MUST also be set to zero and ignored by receivers. 354 (Flexible mode's Reference indices are defined as offsets from the 355 Picture ID field, so they would have no meaning if I were not 356 set.) The value of this F bit MUST only change on the first 357 packet of a key picture. A key picture is a picture whose base 358 spatial layer frame is a key frame, and which thus completely 359 resets the encoder state. This packet will have its P bit equal 360 to zero, SID or L bit (described below) equal to zero, and B bit 361 (described below) equal to 1. 363 B: Start of a frame. MUST be set to 1 if the first payload octet of 364 the RTP packet is the beginning of a new VP9 frame, and MUST NOT 365 be 1 otherwise. Note that this frame might not be the first frame 366 of a picture. 368 E: End of a frame. MUST be set to 1 for the final RTP packet of a 369 VP9 frame, and 0 otherwise. This enables a decoder to finish 370 decoding the frame, where it otherwise may need to wait for the 371 next packet to explicitly know that the frame is complete. Note 372 that, if spatial scalability is in use, more frames from the same 373 picture may follow; see the description of the M bit above. 375 V: Scalability structure (SS) data present. When set to one, the 376 OPTIONAL SS data MUST be present in the payload descriptor. 377 Otherwise, the SS data MUST NOT be present. 379 Z: Not a reference frame for upper spatial layers. If set to 1, 380 indicates that frames with higher spatial layers SID+1 and greater 381 of the current and following pictures do not depend on the current 382 spatial layer SID frame. This enables a decoder which is 383 targeting a higher spatial layer to know that it can safely 384 discard this packet's frame without processing it, without having 385 to wait for the "D" bit in the higher-layer frame (see below). 387 The mandatory first octet is followed by the extension data fields 388 that are enabled: 390 M: The most significant bit of the first octet is an extension flag. 391 The field MUST be present if the I bit is equal to one. If M is 392 set, the PID field MUST contain 15 bits; otherwise, it MUST 393 contain 7 bits. See PID below. 395 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 396 on the M bit. This is a running index of the pictures, where the 397 sender increments the value by 1 for each picture it sends. (Note 398 however that because a middlebox can discard pictures where 399 permitted by the scalability structure, Picture IDs as received by 400 a receiver might not be contiguous.) This field MUST be present 401 if the I bit is equal to one. If M is set to zero, 7 bits carry 402 the PID; else if M is set to one, 15 bits carry the PID in network 403 byte order. The sender may choose between a 7- or 15-bit index. 404 The PID SHOULD start on a random number, and MUST wrap after 405 reaching the maximum ID (0x7f or 0x7fff depending on the index 406 size chosen). The receiver MUST NOT assume that the number of 407 bits in PID stay the same through the session. If this field 408 transitions from 7-bits to 15-bits, the value is zero-extended 409 (i.e. the value after 0x6e is 0x006f); if the field transitions 410 from 15 bits to 7 bits, it is truncated (i.e. the value after 411 0x1bbe is 0xbf). 413 In the non-flexible mode (when the F bit is set to 0), this PID is 414 used as an index to the picture group (PG) specified in the SS 415 data below. In this mode, the PID of the key frame corresponds to 416 the first specified frame in the PG. Then subsequent PIDs are 417 mapped to subsequently specified frames in the PG (modulo N_G, 418 specified in the SS data below), respectively. 420 All frames of the same picture MUST have the same PID value. 422 Frames (and their corresponding pictures) with the VP9 show_frame 423 field equal to 0 MUST have distinct PID values from subsequent 424 pictures with show_frame equal to 1. Thus, a Picture as defined 425 in this specification is different than a VP9 Superframe. 427 All frames of the same picture MUST have the same value for 428 show_frame. 430 Layer indices: This information is optional but RECOMMENDED whenever 431 encoding with layers. For both flexible and non-flexible modes, 432 one octet is used to specify a layer frame's temporal layer ID 433 (TID) and spatial layer ID (SID) as shown both in Figure 2 and 434 Figure 3. Additionally, a bit (U) is used to indicate that the 435 current frame is a "switching up point" frame. Another bit (D) is 436 used to indicate whether inter-layer prediction is used for the 437 current frame. 439 In the non-flexible mode (when the F bit is set to 0), another 440 octet is used to represent temporal layer 0 index (TL0PICIDX), as 441 depicted in Figure 3. The TL0PICIDX is present so that all 442 minimally required frames - the base temporal layer frames - can 443 be tracked. 445 The TID and SID fields indicate the temporal and spatial layers 446 and can help middleboxes and endpoints quickly identify which 447 layer a packet belongs to. 449 TID: The temporal layer ID of current frame. In the case of non- 450 flexible mode, if PID is mapped to a picture in a specified PG, 451 then the value of TID MUST match the corresponding TID value of 452 the mapped picture in the PG. 454 U: Switching up point. If this bit is set to 1 for the current 455 picture with temporal layer ID equal to TID, then "switch up" 456 to a higher frame rate is possible as subsequent higher 457 temporal layer pictures will not depend on any picture before 458 the current picture (in coding order) with temporal layer ID 459 greater than TID. 461 SID: The spatial layer ID of current frame. Note that frames 462 with spatial layer SID > 0 may be dependent on decoded spatial 463 layer SID-1 frame within the same picture. Different frames of 464 the same picture MUST have distinct spatial layer IDs, and 465 frames' spatial layers MUST appear in increasing order within 466 the frame. 468 D: Inter-layer dependency used. MUST be set to one if and only 469 if the current spatial layer SID frame depends on spatial layer 470 SID-1 frame of the same picture, otherwise MUST be set to zero. 471 For the base layer frame (with SID equal to 0), this D bit MUST 472 be set to zero. 474 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 475 present in the non-flexible mode (F = 0). This is a running 476 index for the temporal base layer pictures, i.e., the pictures 477 with TID set to 0. If TID is larger than 0, TL0PICIDX 478 indicates which temporal base layer picture the current picture 479 depends on. TL0PICIDX MUST be incremented by 1 when TID is 480 equal to 0. The index SHOULD start on a random number, and 481 MUST restart at 0 after reaching the maximum number 255. 483 Reference indices: When P and F are both set to one, indicating a 484 non-key frame in flexible mode, then at least one reference index 485 MUST be specified as below. Additional reference indices (total 486 of up to 3 reference indices are allowed) may be specified using 487 the N bit below. When either P or F is set to zero, then no 488 reference index is specified. 490 P_DIFF: The reference index (in 7 bits) specified as the relative 491 PID from the current picture. For example, when P_DIFF=3 on a 492 packet containing the picture with PID 112 means that the 493 picture refers back to the picture with PID 109. This 494 calculation is done modulo the size of the PID field, i.e., 495 either 7 or 15 bits. A P_DIFF value of 0 is invalid. 497 N: 1 if there is additional P_DIFF following the current P_DIFF. 499 4.2.1. Scalability Structure (SS): 501 The scalability structure (SS) data describes the resolution of each 502 frame within a picture as well as the inter-picture dependencies for 503 a picture group (PG). If the VP9 payload descriptor's "V" bit is 504 set, the SS data is present in the position indicated in Figure 2 and 505 Figure 3. 507 +-+-+-+-+-+-+-+-+ 508 V: | N_S |Y|G|-|-|-| 509 +-+-+-+-+-+-+-+-+ -\ 510 Y: | WIDTH | (OPTIONAL) . 511 + + . 512 | | (OPTIONAL) . 513 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 514 | HEIGHT | (OPTIONAL) . 515 + + . 516 | | (OPTIONAL) . 517 +-+-+-+-+-+-+-+-+ -/ 518 G: | N_G | (OPTIONAL) 519 +-+-+-+-+-+-+-+-+ -\ 520 N_G: | TID |U| R |-|-| (OPTIONAL) . 521 +-+-+-+-+-+-+-+-+ -\ . - N_G times 522 | P_DIFF | (OPTIONAL) . - R times . 523 +-+-+-+-+-+-+-+-+ -/ -/ 525 Figure 4 527 N_S: N_S + 1 indicates the number of spatial layers present in the 528 VP9 stream. 530 Y: Each spatial layer's frame resolution present. When set to one, 531 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 532 present for each layer frame. Otherwise, the resolution MUST NOT 533 be present. 535 G: PG description present flag. 537 -: Bit reserved for future use. MUST be set to zero and MUST be 538 ignored by the receiver. 540 N_G: N_G indicates the number of pictures in a Picture Group (PG). 541 If N_G is greater than 0, then the SS data allows the inter- 542 picture dependency structure of the VP9 stream to be pre-declared, 543 rather than indicating it on the fly with every packet. If N_G is 544 greater than 0, then for N_G pictures in the PG, each picture's 545 temporal layer ID (TID), switch up point (U), and the Reference 546 indices (P_DIFFs) are specified. 548 The first picture specified in the PG MUST have TID set to 0. 550 G set to 0 or N_G set to 0 indicates that either there is only one 551 temporal layer (for non-flexible mode) or no fixed inter-picture 552 dependency information is present (for flexible mode) going 553 forward in the bitstream. 555 Note that for a given picture, all frames follow the same inter- 556 picture dependency structure. However, the frame rate of each 557 spatial layer can be different from each other and this can be 558 described with the use of the D bit described above. The 559 specified dependency structure in the SS data MUST be for the 560 highest frame rate layer. 562 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 563 included in the first packet of every key frame. This is a packet 564 with P bit equal to zero, SID or Lis not the bit equal to zero, and B 565 bit equal to 1. The SS data MUST only be changed on the picture that 566 corresponds to the first picture specified in the previous SS data's 567 PG (if the previous SS data's N_G was greater than 0). 569 4.3. Frame Fragmentation 571 VP9 frames are fragmented into packets, in RTP sequence number order, 572 beginning with a packet with the B bit set, and ending with a packet 573 with the E bit set. There is no mechanism for finer-grained access 574 to parts of a VP9 frame. 576 4.4. Scalable encoding considerations 578 In addition to the use of reference frames, VP9 has several 579 additional forms of inter-frame dependencies, largely involving 580 probability tables for the entropy and tree encoders. In VP9 syntax, 581 the syntax element "error_resilient_mode" resets this additional 582 inter-frame data, allowing a frame's syntax to be decoded 583 independently. 585 Due to the requirements of scalable streams, a VP9 encoder producing 586 a scalable stream needs to ensure that a frame does not depend on a 587 previous frame (of the same or a previous picture) that can 588 legitimately be removed from the stream. Thus, a frame that follows 589 a frame that might be removed (in full decode order) MUST be encoded 590 with "error_resilient_mode" set to true. 592 For spatially-scalable streams, this means that 593 "error_resilient_mode" needs to be turned on for the base spatial 594 layer; it can however be turned off for higher spatial layers, 595 assuming they are sent with inter-layer dependency (i.e. with the "D" 596 bit set). For streams that are only temporally-scalable without 597 spatial scalability, "error_resilient_mode" can additionally be 598 turned off for any picture that immediately follows a temporal layer 599 0 frame. 601 4.5. Examples of VP9 RTP Stream 602 4.5.1. Reference picture use for scalable structure 604 As discussed in Section 3, the VP9 codec can maintain up to eight 605 reference frames, of which up to three can be referenced or updated 606 by any new frame. This section illustrates one way that a scalable 607 structure (with three spatial layers and three temporal layers) can 608 be constructed using these reference frames. 610 +==========+=========+============+=========+ 611 | Temporal | Spatial | References | Updates | 612 +==========+=========+============+=========+ 613 | 0 | 0 | 0 | 0 | 614 +----------+---------+------------+---------+ 615 | 0 | 1 | 0,1 | 1 | 616 +----------+---------+------------+---------+ 617 | 0 | 2 | 1,2 | 2 | 618 +----------+---------+------------+---------+ 619 | 2 | 0 | 0 | 6 | 620 +----------+---------+------------+---------+ 621 | 2 | 1 | 1,6 | 7 | 622 +----------+---------+------------+---------+ 623 | 2 | 2 | 2,7 | - | 624 +----------+---------+------------+---------+ 625 | 1 | 0 | 0 | 3 | 626 +----------+---------+------------+---------+ 627 | 1 | 1 | 1,3 | 4 | 628 +----------+---------+------------+---------+ 629 | 1 | 2 | 2,4 | 5 | 630 +----------+---------+------------+---------+ 631 | 2 | 0 | 3 | 6 | 632 +----------+---------+------------+---------+ 633 | 2 | 1 | 4,6 | 7 | 634 +----------+---------+------------+---------+ 635 | 2 | 2 | 5,7 | - | 636 +----------+---------+------------+---------+ 638 Table 1: Example scalability structure 640 This structure is constructed such that the "U" bit can always be 641 set. 643 5. Feedback Messages and Header Extensions 644 5.1. Reference Picture Selection Indication (RPSI) 646 The reference picture selection index is a payload-specific feedback 647 message defined within the RTCP-based feedback format. The RPSI 648 message is generated by a receiver and can be used in two ways. 649 Either it can signal a preferred reference picture when a loss has 650 been detected by the decoder -- preferably then a reference that the 651 decoder knows is perfect -- or, it can be used as positive feedback 652 information to acknowledge correct decoding of certain reference 653 pictures. The positive feedback method is useful for VP9 used for 654 point to point (unicast) communication. The use of RPSI for VP9 is 655 preferably combined with a special update pattern of the codec's two 656 special reference frames -- the golden frame and the altref frame -- 657 in which they are updated in an alternating leapfrog fashion. When a 658 receiver has received and correctly decoded a golden or altref frame, 659 and that frame had a Picture ID in the payload descriptor, the 660 receiver can acknowledge this simply by sending an RPSI message back 661 to the sender. The message body (i.e., the "native RPSI bit string" 662 in [RFC4585]) is simply the (7 or 15 bit) Picture ID of the received 663 frame. 665 Note: because all frames of the same picture must have the same 666 inter-picture reference structure, there is no need for a message to 667 specify which frame is being selected. 669 5.2. Full Intra Request (FIR) 671 The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a 672 receiver to request a full state refresh of an encoded stream. 674 Upon receipt of an FIR request, a VP9 sender MUST send a picture with 675 a keyframe for its spatial layer 0 layer frame, and then send frames 676 without inter-picture prediction (P=0) for any higher layer frames. 678 5.3. Layer Refresh Request (LRR) 680 The Layer Refresh Request (LRR) [I-D.ietf-avtext-lrr] allows a 681 receiver to request a single layer of a spatially or temporally 682 encoded stream to be refreshed, without necessarily affecting the 683 stream's other layers. 685 +---------------+---------------+ 686 |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7| 687 +---------------+---------+-----+ 688 | RES | TID | RES | SID | 689 +---------------+---------+-----+ 691 Figure 5 693 Figure 5 shows the format of LRR's layer index fields for VP9 694 streams. The two "RES" fields MUST be set to 0 on transmission and 695 ingnored on reception. See Section 4.2 for details on the TID and 696 SID fields. 698 Identification of a layer refresh frame can be derived from the 699 reference IDs of each frame by backtracking the dependency chain 700 until reaching a point where only decodable frames are being 701 referenced. Therefore it's recommended for both the flexible and the 702 non-flexible mode that, when switching up points are being encoded in 703 response to a LRR, those packets should contain layer indices and the 704 reference field(s) so that the decoder or a selective forwarding 705 middleboxes [RFC7667] can make this derivation. 707 Example: 709 LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying 710 {1,0} to a receiver and which wants to upgrade to {2,1}. In response 711 the encoder should encode the next frames in layers {1,1} and {2,1} 712 by only referring to frames in {1,0}, or {0,0}. 714 In the non-flexible mode, periodic upgrade frames can be defined by 715 the layer structure of the SS, thus periodic upgrade frames can be 716 automatically identified by the picture ID. 718 6. Payload Format Parameters 720 This payload format has three optional parameters, "max-fr", "max- 721 fs", and "profile-id". 723 The max-fr and max-fs parameters are used to signal the capabilities 724 of a receiver implementation. If the implementation is willing to 725 receive media, both parameters MUST be provided. These parameters 726 MUST NOT be used for any other purpose. A media sender SHOULD NOT 727 send media with a frame rate or frame size exceeding the max-fr and 728 max-fs values signaled. (There may be scenarios, such as pre-encoded 729 media or selective forwarding middleboxes [RFC7667], where a media 730 sender does not have media available that fits within a receivers 731 max-fs and max-fr value; in such scenarios, a sender MAY exceed the 732 signaled values.) 734 max-fr: The value of max-fr is an integer indicating the maximum 735 frame rate in units of frames per second that the decoder is 736 capable of decoding. 738 max-fs: The value of max-fs is an integer indicating the maximum 739 frame size in units of macroblocks that the decoder is capable of 740 decoding. 742 The decoder is capable of decoding this frame size as long as the 743 width and height of the frame in macroblocks are less than 744 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable of 745 supporting 640x480 resolution) will support widths and heights up 746 to 1552 pixels (97 macroblocks). 748 profile-id: The value of profile-id is an integer indicating the 749 default coding profile, the subset of coding tools that may have 750 been used to generate the stream or that the receiver supports). 751 Table 2 lists all of the profiles defined in section 7.2 of 752 [VP9-BITSTREAM] and the corresponding integer values to be used. 754 If no profile-id is present, Profile 0 MUST be inferred. (The 755 profile-id parameter was added relatively late in the development 756 of this specification, so some existing implementations may not 757 send it.) 759 Informative note: See Table 3 for capabilities of coding profiles 760 defined in section 7.2 of [VP9-BITSTREAM]. 762 A receiver MUST ignore any parameter unspecified in this 763 specification. 765 +=========+============+ 766 | Profile | profile-id | 767 +=========+============+ 768 | 0 | 0 | 769 +---------+------------+ 770 | 1 | 1 | 771 +---------+------------+ 772 | 2 | 2 | 773 +---------+------------+ 774 | 3 | 3 | 775 +---------+------------+ 777 Table 2: Table of 778 profile-id integer 779 values representing 780 the VP9 profile 781 corresponding to the 782 set of coding tools 783 supported. 785 +=========+===========+=================+==========================+ 786 | Profile | Bit Depth | SRGB Colorspace | Chroma Subsampling | 787 +=========+===========+=================+==========================+ 788 | 0 | 8 | No | YUV 4:2:0 | 789 +---------+-----------+-----------------+--------------------------+ 790 | 1 | 8 | Yes | YUV 4:2:2,4:4:0 or 4:4:4 | 791 +---------+-----------+-----------------+--------------------------+ 792 | 2 | 10 or 12 | No | YUV 4:2:0 | 793 +---------+-----------+-----------------+--------------------------+ 794 | 3 | 10 or 12 | Yes | YUV 4:2:2,4:4:0 or 4:4:4 | 795 +---------+-----------+-----------------+--------------------------+ 797 Table 3: Table of profile capabilities. 799 6.1. SDP Parameters 801 6.1.1. Mapping of Media Subtype Parameters to SDP 803 The media type video/VP9 string is mapped to fields in the Session 804 Description Protocol (SDP) [RFC8866] as follows: 806 * The media name in the "m=" line of SDP MUST be video. 808 * The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 809 media subtype). 811 * The clock rate in the "a=rtpmap" line MUST be 90000. 813 * The parameters "max-fr" and "max-fs" MUST be included in the 814 "a=fmtp" line of SDP if the receiver wishes to declare its 815 receiver capabilities. These parameters are expressed as a media 816 subtype string, in the form of a semicolon separated list of 817 parameter=value pairs. 819 * The OPTIONAL parameter profile-id, when present, SHOULD be 820 included in the "a=fmtp" line of SDP. This parameter is expressed 821 as a media subtype string, in the form of a parameter=value pair. 822 When the parameter is not present, a value of 0 MUST be inferred 823 for profile-id. 825 6.1.1.1. Example 827 An example of media representation in SDP is as follows: 829 m=video 49170 RTP/AVPF 98 830 a=rtpmap:98 VP9/90000 831 a=fmtp:98 max-fr=30;max-fs=3600;profile-id=0 833 6.1.2. Offer/Answer Considerations 835 When VP9 is offered over RTP using SDP in an Offer/Answer model 836 [RFC3264] for negotiation for unicast usage, the following 837 limitations and rules apply: 839 * The parameter identifying a media format configuration for VP9 is 840 profile-id. This media format configuration parameter MUST be 841 used symmetrically; that is, the answerer MUST either maintain 842 this configuration parameter or remove the media format (payload 843 type) completely if it is not supported. 845 * The max-fr and max-fs parameters are used declaratively to 846 describe receiver capabilities, even in the Offer/Answer model. 847 The values in an answer are used to describe the answerer's 848 capabilities, and thus their values are set independently of the 849 values in the offer. 851 * To simplify the handling and matching of these configurations, the 852 same RTP payload type number used in the offer SHOULD also be used 853 in the answer and in a subsequent offer, as specified in 854 [RFC3264]. An answer or subsequent offer MUST NOT contain the 855 payload type number used in the offer unless the profile-id value 856 is exactly the same as in the original offer. However, max-fr and 857 max-fs parameters MAY be changed in subsequent offers and answers, 858 with the same payload type number, if an endpoint wishes to change 859 its declared receiver capabilities. 861 7. Media Type Definition 863 This registration is done using the template defined in [RFC6838] and 864 following [RFC4855]. 866 Type name: 867 video 869 Subtype name: 870 VP9 872 Required parameters: 873 N/A. 875 Optional parameters: 876 There are three optional parameters, "max-fr", "max-fs", and 877 "profile-id". See Section 6 for their definition. 879 Encoding considerations: 880 This media type is framed in RTP and contains binary data; see 881 Section 4.8 of [RFC6838]. 883 Security considerations: 884 See Section 8 of RFC xxxx. 886 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 887 with the number assigned to this document and remove this note.] 889 Interoperability considerations: 890 None. 892 Published specification: 893 VP9 bitstream format [VP9-BITSTREAM] and RFC XXXX. 895 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 896 with the number assigned to this document and remove this note.] 898 Applications which use this media type: 899 For example: Video over IP, video conferencing. 901 Fragment identifier considerations: 902 N/A. 904 Additional information: 905 None. 907 Person & email address to contact for further information: 908 Jonathan Lennox 910 Intended usage: 911 COMMON 913 Restrictions on usage: 914 This media type depends on RTP framing, and hence is only defined 915 for transfer via RTP [RFC3550]. 917 Author: 918 Jonathan Lennox 920 Change controller: 921 IETF AVTCore Working Group delegated from the IESG. 923 8. Security Considerations 925 RTP packets using the payload format defined in this specification 926 are subject to the security considerations discussed in the RTP 927 specification [RFC3550], and in any applicable RTP profile such as 928 RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/ 929 SAVPF [RFC5124]. However, as "Securing the RTP Protocol Framework: 930 Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] 931 discusses, it is not an RTP payload format's responsibility to 932 discuss or mandate what solutions are used to meet the basic security 933 goals like confidentiality, integrity and source authenticity for RTP 934 in general. This responsibility lays on anyone using RTP in an 935 application. They can find guidance on available security mechanisms 936 in Options for Securing RTP Sessions [RFC7201]. Applications SHOULD 937 use one or more appropriate strong security mechanisms. The rest of 938 this security consideration section discusses the security impacting 939 properties of the payload format itself. 941 Implementations of this RTP payload format need to take appropriate 942 security considerations into account. It is extremely important for 943 the decoder to be robust against malicious or malformed payloads and 944 ensure that they do not cause the decoder to overrun its allocated 945 memory or otherwise mis-behave. An overrun in allocated memory could 946 lead to arbitrary code execution by an attacker. The same applies to 947 the encoder, even though problems in encoders are typically rarer. 949 This RTP payload format and its media decoder do not exhibit any 950 significant non-uniformity in the receiver-side computational 951 complexity for packet processing, and thus are unlikely to pose a 952 denial-of-service threat due to the receipt of pathological data. 953 Nor does the RTP payload format contain any active content. 955 9. Congestion Control 957 Congestion control for RTP SHALL be used in accordance with RFC 3550 958 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 959 [RFC3551]. The congestion control mechanism can, in a real-time 960 encoding scenario, adapt the transmission rate by instructing the 961 encoder to encode at a certain target rate. Media aware network 962 elements MAY use the information in the VP9 payload descriptor in 963 Section 4.2 to identify non-reference frames and discard them in 964 order to reduce network congestion. Note that discarding of non- 965 reference frames cannot be done if the stream is encrypted (because 966 the non-reference marker is encrypted). 968 10. IANA Considerations 970 The IANA is requested to register the media type registration "video/ 971 vp9" as specified in Section 7. The media type is also requested to 972 be added to the IANA registry for "RTP Payload Format MIME types" 973 . 975 11. Acknowledgments 977 Alex Eleftheriadis, Yuki Ito, Won Kap Jang, Sergio Garcia Murillo, 978 Roi Sasson, Timothy Terriberry, Emircan Uysaler, and Thomas Volkert 979 commented on the development of this document and provided helpful 980 comments and feedback. 982 12. References 984 12.1. Normative References 986 [I-D.ietf-avtext-lrr] 987 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 988 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 989 Message", Work in Progress, Internet-Draft, draft-ietf- 990 avtext-lrr-07, 2 July 2017, 991 . 994 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 995 Requirement Levels", BCP 14, RFC 2119, 996 DOI 10.17487/RFC2119, March 1997, 997 . 999 [RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model 1000 with Session Description Protocol (SDP)", RFC 3264, 1001 DOI 10.17487/RFC3264, June 2002, 1002 . 1004 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1005 Jacobson, "RTP: A Transport Protocol for Real-Time 1006 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1007 July 2003, . 1009 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1010 "Extended RTP Profile for Real-time Transport Control 1011 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 1012 DOI 10.17487/RFC4585, July 2006, 1013 . 1015 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 1016 Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, 1017 . 1019 [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, 1020 "Codec Control Messages in the RTP Audio-Visual Profile 1021 with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, 1022 February 2008, . 1024 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 1025 Specifications and Registration Procedures", BCP 13, 1026 RFC 6838, DOI 10.17487/RFC6838, January 2013, 1027 . 1029 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1030 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1031 May 2017, . 1033 [RFC8866] Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP: 1034 Session Description Protocol", RFC 8866, 1035 DOI 10.17487/RFC8866, January 2021, 1036 . 1038 [VP9-BITSTREAM] 1039 Grange, A., de Rivaz, P., and J. Hunt, "VP9 Bitstream & 1040 Decoding Process Specification", Version 0.6, 31 March 1041 2016, 1042 . 1046 12.2. Informative References 1048 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1049 Video Conferences with Minimal Control", STD 65, RFC 3551, 1050 DOI 10.17487/RFC3551, July 2003, 1051 . 1053 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1054 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1055 RFC 3711, DOI 10.17487/RFC3711, March 2004, 1056 . 1058 [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for 1059 Real-time Transport Control Protocol (RTCP)-Based Feedback 1060 (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 1061 2008, . 1063 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 1064 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 1065 Guide", RFC 6386, DOI 10.17487/RFC6386, November 2011, 1066 . 1068 [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP 1069 Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, 1070 . 1072 [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP 1073 Framework: Why RTP Does Not Mandate a Single Media 1074 Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 1075 2014, . 1077 [RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667, 1078 DOI 10.17487/RFC7667, November 2015, 1079 . 1081 Authors' Addresses 1083 Justin Uberti 1084 Google, Inc. 1085 747 6th Street South 1086 Kirkland, WA 98033 1087 United States of America 1089 Email: justin@uberti.name 1091 Stefan Holmer 1092 Google, Inc. 1093 Kungsbron 2 1094 SE-111 22 Stockholm 1095 Sweden 1097 Email: holmer@google.com 1099 Magnus Flodman 1100 Google, Inc. 1101 Kungsbron 2 1102 SE-111 22 Stockholm 1103 Sweden 1105 Email: mflodman@google.com 1106 Danny Hong 1107 Google, Inc. 1108 1585 Charleston Road 1109 Mountain View, CA 94043 1110 United States of America 1112 Email: dannyhong@google.com 1114 Jonathan Lennox 1115 8x8, Inc. / Jitsi 1116 1350 Broadway 1117 New York, NY 10018 1118 United States of America 1120 Email: jonathan.lennox@8x8.com