idnits 2.17.1 draft-ietf-payload-vp9-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 2, 2018) is 2122 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-16) exists of draft-ietf-avtext-framemarking-07 ** Downref: Normative reference to an Experimental draft: draft-ietf-avtext-framemarking (ref. 'I-D.ietf-avtext-framemarking') ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) -- Possible downref: Non-RFC (?) normative reference: ref. 'VP9-BITSTREAM' Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: January 3, 2019 Google 6 J. Lennox 7 D. Hong 8 Vidyo 9 July 2, 2018 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-06 14 Abstract 16 This memo describes an RTP payload format for the VP9 video codec. 17 The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on January 3, 2019. 39 Copyright Notice 41 Copyright (c) 2018 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 58 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 59 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 5 60 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 61 4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 7 62 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 11 63 4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 13 64 4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 13 65 4.5. Scalable encoding considerations . . . . . . . . . . . . 13 66 4.6. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 14 67 4.6.1. Reference picture use for scalable structure . . . . 14 68 5. Feedback Messages and Header Extensions . . . . . . . . . . . 15 69 5.1. Reference Picture Selection Indication (RPSI) . . . . . . 15 70 5.2. Slice Loss Indication (SLI) . . . . . . . . . . . . . . . 15 71 5.3. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 16 72 5.4. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 16 73 5.5. Frame Marking . . . . . . . . . . . . . . . . . . . . . . 17 74 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 17 75 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 18 76 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 19 77 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 19 78 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 20 79 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 80 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 20 81 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 82 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 83 10.1. Normative References . . . . . . . . . . . . . . . . . . 21 84 10.2. Informative References . . . . . . . . . . . . . . . . . 22 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 87 1. Introduction 89 This memo describes an RTP payload specification applicable to the 90 transmission of video streams encoded using the VP9 video codec 91 [VP9-BITSTREAM]. The format described in this document can be used 92 both in peer-to-peer and video conferencing applications. 94 TODO: VP9 description. Please see [VP9-BITSTREAM]. 96 2. Conventions, Definitions and Acronyms 98 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 99 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 100 document are to be interpreted as described in [RFC2119]. 102 TODO: Cite terminology from [VP9-BITSTREAM]. 104 3. Media Format Description 106 The VP9 codec can maintain up to eight reference frames, of which up 107 to three can be referenced by any new frame. 109 VP9 also allows a frame to use another frame of a different 110 resolution as a reference frame. (Specifically, a frame may use any 111 references whose width and height are between 1/16th that of the 112 current frame and twice that of the current frame, inclusive.) This 113 allows internal resolution changes without requiring the use of key 114 frames. 116 These features together enable an encoder to implement various forms 117 of coarse-grained scalability, including temporal, spatial and 118 quality scalability modes, as well as combinations of these, without 119 the need for explicit scalable coding tools. 121 Temporal layers define different frame rates of video; spatial and 122 quality layers define different and possibly dependent 123 representations of a single input frame. Spatial layers allow a 124 frame to be encoded at different resolutions, whereas quality layers 125 allow a frame to be encoded at the same resolution but at different 126 qualities (and thus with different amounts of coding error). VP9 127 supports quality layers as spatial layers without any resolution 128 changes; hereinafter, the term "spatial layer" is used to represent 129 both spatial and quality layers. 131 This payload format specification defines how such temporal and 132 spatial scalability layers can be described and communicated. 134 Temporal and spatial scalability layers are associated with non- 135 negative integer IDs. The lowest layer of either type has an ID of 136 0, and is sometimes referred to as the "base" temporal or spatial 137 layer. 139 Layers are designed (and MUST be encoded) such that if any layer, and 140 all higher layers, are removed from the bitstream along either of the 141 two dimensions, the remaining bitstream is still correctly decodable. 143 For terminology, this document uses the term "frame" to refer to a 144 single encoded VP9 frame for a particular resolution/quality, and 145 "picture" to refer to all the representations (frames) at a single 146 instant in time. A picture thus consists of one or more frames, 147 encoding different spatial layers. 149 Within a picture, a frame with spatial layer ID equal to SID, where 150 SID > 0, can depend on a frame of the same picture with a lower 151 spatial layer ID. This "inter-layer" dependency can result in 152 additional coding gain compared to the case where only traditional 153 "inter-picture" dependency is used, where a frame depends on 154 previously coded frame in time. For simplicity, this payload format 155 assumes that, within a picture and if inter-layer dependency is used, 156 a spatial layer SID frame can depend only on the immediately previous 157 spatial layer SID-1 frame, when S > 0. Additionally, if inter- 158 picture dependency is used, a spatial layer SID frame is assumed to 159 only depend on a previously coded spatial layer SID frame. 161 Given above simplifications for inter-layer and inter-picture 162 dependencies, a flag (the D bit described below) is used to indicate 163 whether a spatial layer SID frame depends on the spatial layer SID-1 164 frame. Given the D bit, a receiver only needs to additionally know 165 the inter-picture dependency structure for a given spatial layer 166 frame in order to determine its decodability. Two modes of 167 describing the inter-picture dependency structure are possible: 168 "flexible mode" and "non-flexible mode". An encoder can only switch 169 between the two on the first packet of a key frame with temporal 170 layer ID equal to 0. 172 In flexible mode, each packet can contain up to 3 reference indices, 173 which identify all frames referenced by the frame transmitted in the 174 current packet for inter-picture prediction. This (along with the D 175 bit) enables a receiver to identify if a frame is decodable or not 176 and helps it understand the temporal layer structure. Since this is 177 signaled in each packet it makes it possible to have very flexible 178 temporal layer hierarchies and patterns which are changing 179 dynamically. 181 In non-flexible mode, the inter-picture dependency (the reference 182 indices) of a Picture Group (PG) MUST be pre-specified as part of the 183 scalability structure (SS) data. In this mode, each packet has an 184 index to refer to one of the described pictures in the PG, from which 185 the pictures referenced by the picture transmitted in the current 186 packet for inter-picture prediction can be identified. 188 (Editor's Note: A "Picture Group", as used in this document, is not 189 the same thing as a the term "Group of Pictures" as it is 190 traditionally used in video coding, i.e. to mean an independently- 191 decoadable run of pictures beginning with a keyframe. Suggestions 192 for better terminology are welcome.) 194 The SS data can also be used to specify the resolution of each 195 spatial layer present in the VP9 stream for both flexible and non- 196 flexible modes. 198 4. Payload Format 200 This section describes how the encoded VP9 bitstream is encapsulated 201 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 202 RECOMMENDED. All integer fields in the specifications are encoded as 203 unsigned integers in network octet order. 205 4.1. RTP Header Usage 206 The general RTP payload format for VP9 is depicted below. 208 0 1 2 3 209 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 210 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 211 |V=2|P|X| CC |M| PT | sequence number | 212 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 213 | timestamp | 214 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 215 | synchronization source (SSRC) identifier | 216 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 217 | contributing source (CSRC) identifiers | 218 | .... | 219 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 220 | VP9 payload descriptor (integer #octets) | 221 : : 222 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 223 | : VP9 pyld hdr | | 224 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 225 | | 226 + | 227 : Bytes 2..N of VP9 payload : 228 | | 229 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 230 | : OPTIONAL RTP padding | 231 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 233 The VP9 payload descriptor and VP9 payload header will be described 234 in Section 4.2 and Section 4.3. OPTIONAL RTP padding MUST NOT be 235 included unless the P bit is set. The figure specifically shows the 236 format for the first packet in a frame. Subsequent packets will not 237 contain the VP9 payload header, and will have later octets in the 238 frame payload. 240 Figure 1 242 Marker bit (M): MUST be set to 1 for the final packet of the highest 243 spatial layer frame (the final packet of the picture), and 0 244 otherwise. Unless spatial scalability is in use for this picture, 245 this will have the same value as the E bit described below. Note 246 this bit MUST be set to 1 for the target spatial layer frame if a 247 stream is being rewritten to remove higher spatial layers. 249 Payload Type (PT): In line with the policy in Section 3 of 250 [RFC3551], applications using the VP9 RTP payload profile MUST 251 assign a dynamic payload type number to be used in each RTP 252 session and provide a mechanism to indicate the mapping. See 253 Section 6.2 for the mechanism to be used with the Session 254 Description Protocol (SDP) [RFC4566]. 256 Timestamp: The RTP timestamp indicates the time when the input frame 257 was sampled, at a clock rate of 90 kHz. If the input picture is 258 encoded with multiple layer frames, all of the frames of the 259 picture MUST have the same timestamp. 261 If a frame has the VP9 show_frame field set to 0 (i.e., it is 262 meant only to populate a reference buffer, without being output) 263 its timestamp MAY alternately be set to be the same as the 264 subsequent frame with show_frame equal to 1. (This will be 265 convenient for playing out pre-encoded content packaged with VP9 266 "superframes", which typically bundle show_frame==0 frames with a 267 subsequent show_frame==1 frame.) Every frame with show_frame==1, 268 however, MUST have a unique timestamp modulo the 2^32 wrap of the 269 field. 271 The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, 272 SSRC and CSRC identifiers) are used as specified in Section 5.1 of 273 [RFC3550]. 275 4.2. VP9 Payload Description 277 In flexible mode (with the F bit below set to 1), The first octets 278 after the RTP header are the VP9 payload descriptor, with the 279 following structure. 281 0 1 2 3 4 5 6 7 282 +-+-+-+-+-+-+-+-+ 283 |I|P|L|F|B|E|V|-| (REQUIRED) 284 +-+-+-+-+-+-+-+-+ 285 I: |M| PICTURE ID | (REQUIRED) 286 +-+-+-+-+-+-+-+-+ 287 M: | EXTENDED PID | (RECOMMENDED) 288 +-+-+-+-+-+-+-+-+ 289 L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED) 290 +-+-+-+-+-+-+-+-+ -\ 291 P,F: | P_DIFF |N| (CONDITIONALLY REQUIRED) - up to 3 times 292 +-+-+-+-+-+-+-+-+ -/ 293 V: | SS | 294 | .. | 295 +-+-+-+-+-+-+-+-+ 297 Figure 2 299 In non-flexible mode (with the F bit below set to 0), The first 300 octets after the RTP header are the VP9 payload descriptor, with the 301 following structure. 303 0 1 2 3 4 5 6 7 304 +-+-+-+-+-+-+-+-+ 305 |I|P|L|F|B|E|V|-| (REQUIRED) 306 +-+-+-+-+-+-+-+-+ 307 I: |M| PICTURE ID | (RECOMMENDED) 308 +-+-+-+-+-+-+-+-+ 309 M: | EXTENDED PID | (RECOMMENDED) 310 +-+-+-+-+-+-+-+-+ 311 L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED) 312 +-+-+-+-+-+-+-+-+ 313 | TL0PICIDX | (CONDITIONALLY REQUIRED) 314 +-+-+-+-+-+-+-+-+ 315 V: | SS | 316 | .. | 317 +-+-+-+-+-+-+-+-+ 319 Figure 3 321 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 322 be present after the mandatory first octet and specified as below. 323 Otherwise, PID MUST NOT be present. If the SS field was present 324 in the stream's most recent start of a keyframe (i.e., non- 325 flexible scalability mode is in use), then the PID MUST also be 326 present in every packet. 328 P: Inter-picture predicted frame. When set to zero, the frame does 329 not utilize inter-picture prediction. In this case, up-switching 330 to a current spatial layer's frame is possible from directly lower 331 spatial layer frame. P SHOULD also be set to zero when encoding a 332 layer synchronization frame in response to an LRR 333 [I-D.ietf-avtext-lrr] message (see Section 5.4). When P is set to 334 zero, the TID field (described below) MUST also be set to 0 (if 335 present). Note that the P bit does not forbid intra-picture, 336 inter-layer prediction from earlier frames of the same picture, if 337 any. 339 L: Layer indices present. When set to one, the one or two octets 340 following the mandatory first octet and the PID (if present) is as 341 described by "Layer indices" below. If the F bit (described 342 below) is set to 1 (indicating flexible mode), then only one octet 343 is present for the layer indices. Otherwise if the F bit is set 344 to 0 (indicating non-flexible mode), then two octets are present 345 for the layer indices. 347 F: Flexible mode. F set to one indicates flexible mode and if the P 348 bit is also set to one, then the octets following the mandatory 349 first octet, the PID, and layer indices (if present) are as 350 described by "Reference indices" below. This MUST only be set to 351 1 if the I bit is also set to one; if the I bit is set to zero, 352 then this MUST also be set to zero and ignored by receivers. The 353 value of this F bit MUST only change on the first packet of a key 354 picture. A key picture is a picture whose base spatial layer 355 frame is a key frame, and which thus completely resets the encoder 356 state. This packet will have its P bit equal to zero, SID or D 357 bit (described below) equal to zero, and B bit (described below) 358 equal to 1. 360 B: Start of a frame. MUST be set to 1 if the first payload octet of 361 the RTP packet is the beginning of a new VP9 frame, and MUST NOT 362 be 1 otherwise. Note that this frame might not be the first frame 363 of a picture. 365 E: End of a frame. MUST be set to 1 for the final RTP packet of a 366 VP9 frame, and 0 otherwise. This enables a decoder to finish 367 decoding the frame, where it otherwise may need to wait for the 368 next packet to explicitly know that the frame is complete. Note 369 that, if spatial scalability is in use, more frames from the same 370 picture may follow; see the description of the M bit above. 372 V: Scalability structure (SS) data present. When set to one, the 373 OPTIONAL SS data MUST be present in the payload descriptor. 374 Otherwise, the SS data MUST NOT be present. 376 -: Bit reserved for future use. MUST be set to zero and MUST be 377 ignored by the receiver. 379 The mandatory first octet is followed by the extension data fields 380 that are enabled: 382 M: The most significant bit of the first octet is an extension flag. 383 The field MUST be present if the I bit is equal to one. If set, 384 the PID field MUST contain 15 bits; otherwise, it MUST contain 7 385 bits. See PID below. 387 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 388 on the M bit. This is a running index of the pictures. The field 389 MUST be present if the I bit is equal to one. If M is set to 390 zero, 7 bits carry the PID; else if M is set to one, 15 bits carry 391 the PID in network byte order. The sender may choose between a 7- 392 or 15-bit index. The PID SHOULD start on a random number, and 393 MUST wrap after reaching the maximum ID. The receiver MUST NOT 394 assume that the number of bits in PID stay the same through the 395 session. 397 In the non-flexible mode (when the F bit is set to 0), this PID is 398 used as an index to the picture group (PG) specified in the SS 399 data below. In this mode, the PID of the key frame corresponds to 400 the first specified frame in the PG. Then subsequent PIDs are 401 mapped to subsequently specified frames in the PG (modulo N_G, 402 specified in the SS data below), respectively. 404 All frames of the same picture MUST have the same PID value. 406 Frames (and their corresponding pictures) with the VP9 show_frame 407 field equal to 0 MUST have distinct PID values from subsequent 408 pictures with show_frame equal to 1. Thus, a Picture as defined 409 in this specification is different than a VP9 Superframe. 411 All frames of the same picture MUST have the same value for 412 show_frame. 414 Layer indices: This information is optional but recommended whenever 415 encoding with layers. For both flexible and non-flexible modes, 416 one octet is used to specify a layer frame's temporal layer ID 417 (TID) and spatial layer ID (SID) as shown both in Figure 2 and 418 Figure 3. Additionally, a bit (U) is used to indicate that the 419 current frame is a "switching up point" frame. Another bit (D) is 420 used to indicate whether inter-layer prediction is used for the 421 current frame. 423 In the non-flexible mode (when the F bit is set to 0), another 424 octet is used to represent temporal layer 0 index (TL0PICIDX), as 425 depicted in Figure 3. The TL0PICIDX is present so that all 426 minimally required frames - the base temporal layer frames - can 427 be tracked. 429 The TID and SID fields indicate the temporal and spatial layers 430 and can help middleboxes and and endpoints quickly identify which 431 layer a packet belongs to. 433 TID: The temporal layer ID of current frame. In the case of non- 434 flexible mode, if PID is mapped to a picture in a specified PG, 435 then the value of TID MUST match the corresponding TID value of 436 the mapped picture in the PG. 438 U: Switching up point. If this bit is set to 1 for the current 439 picture with temporal layer ID equal to TID, then "switch up" 440 to a higher frame rate is possible as subsequent higher 441 temporal layer pictures will not depend on any picture before 442 the current picture (in coding order) with temporal layer ID 443 greater than TID. 445 SID: The spatial layer ID of current frame. Note that frames 446 with spatial layer SDI > 0 may be dependent on decoded spatial 447 layer SID-1 frame within the same picture. Different frames of 448 the same picture MUST have distinct spatial layer IDs, and 449 frames' spatial layers MUST appear in increasing order within 450 the frame. 452 D: Inter-layer dependency used. MUST be set to one if current 453 spatial layer SID frame depends on spatial layer SID-1 frame of 454 the same picture. MUST only be set to zero if current spatial 455 layer SID frame does not depend on spatial layer SID-1 frame of 456 the same picture. For the base layer frame (with SID equal to 457 0), this D bit MUST be set to zero. 459 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 460 present in the non-flexible mode (F = 0). This is a running 461 index for the temporal base layer pictures, i.e., the pictures 462 with TID set to 0. If TID is larger than 0, TL0PICIDX 463 indicates which temporal base layer picture the current picture 464 depends on. TL0PICIDX MUST be incremented when TID is equal to 465 0. The index SHOULD start on a random number, and MUST restart 466 at 0 after reaching the maximum number 255. 468 Reference indices: When P and F are both set to one, indicating a 469 non-key frame in flexible mode, then at least one reference index 470 has to be specified as below. Additional reference indices (total 471 of up to 3 reference indices are allowed) may be specified using 472 the N bit below. When either P or F is set to zero, then no 473 reference index is specified. 475 P_DIFF: The reference index (in 7 bits) specified as the relative 476 PID from the current picture. For example, when P_DIFF=3 on a 477 packet containing the picture with PID 112 means that the 478 picture refers back to the picture with PID 109. This 479 calculation is done modulo the size of the PID field, i.e., 480 either 7 or 15 bits. 482 N: 1 if there is additional P_DIFF following the current P_DIFF. 484 4.2.1. Scalability Structure (SS): 486 The scalability structure (SS) data describes the resolution of each 487 frame within a picture as well as the inter-picture dependencies for 488 a picture group (PG). If the VP9 payload descriptor's "V" bit is 489 set, the SS data is present in the position indicated in Figure 2 and 490 Figure 3. 492 +-+-+-+-+-+-+-+-+ 493 V: | N_S |Y|G|-|-|-| 494 +-+-+-+-+-+-+-+-+ -\ 495 Y: | WIDTH | (OPTIONAL) . 496 + + . 497 | | (OPTIONAL) . 498 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 499 | HEIGHT | (OPTIONAL) . 500 + + . 501 | | (OPTIONAL) . 502 +-+-+-+-+-+-+-+-+ -/ -\ 503 G: | N_G | (OPTIONAL) 504 +-+-+-+-+-+-+-+-+ -\ 505 N_G: | TID |U| R |-|-| (OPTIONAL) . 506 +-+-+-+-+-+-+-+-+ -\ . - N_G times 507 | P_DIFF | (OPTIONAL) . - R times . 508 +-+-+-+-+-+-+-+-+ -/ -/ 510 Figure 4 512 N_S: N_S + 1 indicates the number of spatial layers present in the 513 VP9 stream. 515 Y: Each spatial layer's frame resolution present. When set to one, 516 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 517 present for each layer frame. Otherwise, the resolution MUST NOT 518 be present. 520 G: PG description present flag. 522 -: Bit reserved for future use. MUST be set to zero and MUST be 523 ignored by the receiver. 525 N_G: N_G indicates the number of pictures in a Picture Group (PG). 526 If N_G is greater than 0, then the SS data allows the inter- 527 picture dependency structure of the VP9 stream to be pre-declared, 528 rather than indicating it on the fly with every packet. If N_G is 529 greater than 0, then for N_G pictures in the PG, each picture's 530 temporal layer ID (TID), switch up point (U), and the R reference 531 indices (P_DIFFs) are specified. 533 The first picture specified in the PG MUST have TID set to 0. 535 G set to 0 or N_G set to 0 indicates that either there is only one 536 temporal layer or no fixed inter-picture dependency information is 537 present going forward in the bitstream. 539 Note that for a given picture, all frames follow the same inter- 540 picture dependency structure. However, the frame rate of each 541 spatial layer can be different from each other and this can be 542 controlled with the use of the D bit described above. The 543 specified dependency structure in the SS data MUST be for the 544 highest frame rate layer. 546 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 547 included in the first packet of every key frame. This is a packet 548 with P bit equal to zero, SID or D bit equal to zero, and B bit equal 549 to 1. The SS data MUST only be changed on the picture that 550 corresponds to the first picture specified in the previous SS data's 551 PG (if the previous SS data's N_G was greater than 0). 553 4.3. VP9 Payload Header 555 TODO: need to describe VP9 payload header. 557 4.4. Frame Fragmentation 559 VP9 frames are fragmented into packets, in RTP sequence number order, 560 beginning with a packet with the B bit set, and ending with a packet 561 with the E bit set. There is no mechanism for finer-grained access 562 to parts of a VP9 frame. 564 4.5. Scalable encoding considerations 566 In addition to the use of reference frames, VP9 has several 567 additional forms of inter-frame dependencies, largely involving 568 probability tables for the entropy and tree encoders. In VP9 syntax, 569 the syntax element "error_resilient_mode" resets this additional 570 inter-frame data, allowing a frame's syntax to be decoded 571 independently. 573 Due to the requirements of scalable streams, a VP9 encoder producing 574 a scalable stream needs to ensure that a frame does not depend on a 575 previous frame (of the same or a previous picture) that can 576 legitimately be removed from the stream. Thus, a frame that follows 577 a removable frame (in full decode order) MUST be encoded with 578 "error_resilient_mode" to true. 580 For spatially-scalable streams, this means that 581 "error_resilient_mode" needs to be turned on for the base spatial 582 layer; it can however be turned off for higher spatial layers, 583 assuming they are sent with inter-layer dependency (i.e. with the "D" 584 bit set). For streams that are only temporally-scalable without 585 spatial scalability, "error_resilient_mode" can additionally be 586 turned off for any picture that immediately follows a temporal layer 587 0 frame. 589 4.6. Examples of VP9 RTP Stream 591 TODO: Examples of packet layouts 593 4.6.1. Reference picture use for scalable structure 595 As discussed in Section 3, the VP9 codec can maintain up to eight 596 reference frames, of which up to three can be referenced or updated 597 by any new frame. This section illustrates one way that a scalable 598 structure (with three spatial layers and three temporal layers) can 599 be constructed using these reference frames. 601 +----------+---------+------------+---------+ 602 | Temporal | Spatial | References | Updates | 603 +----------+---------+------------+---------+ 604 | 0 | 0 | 0 | 0 | 605 | | | | | 606 | 0 | 1 | 0,1 | 1 | 607 | | | | | 608 | 0 | 2 | 1,2 | 2 | 609 | | | | | 610 | 2 | 0 | 0 | 6 | 611 | | | | | 612 | 2 | 1 | 1,6 | 7 | 613 | | | | | 614 | 2 | 2 | 2,7 | - | 615 | | | | | 616 | 1 | 0 | 0 | 3 | 617 | | | | | 618 | 1 | 1 | 1,3 | 4 | 619 | | | | | 620 | 1 | 2 | 2,4 | 5 | 621 | | | | | 622 | 2 | 0 | 3 | 6 | 623 | | | | | 624 | 2 | 1 | 4,6 | 7 | 625 | | | | | 626 | 2 | 2 | 5,7 | - | 627 +----------+---------+------------+---------+ 629 Example scalability structure 631 This structure is constructed such that the "U" bit can always be 632 set. 634 5. Feedback Messages and Header Extensions 636 5.1. Reference Picture Selection Indication (RPSI) 638 The reference picture selection index is a payload-specific feedback 639 message defined within the RTCP-based feedback format. The RPSI 640 message is generated by a receiver and can be used in two ways. 641 Either it can signal a preferred reference picture when a loss has 642 been detected by the decoder -- preferably then a reference that the 643 decoder knows is perfect -- or, it can be used as positive feedback 644 information to acknowledge correct decoding of certain reference 645 pictures. The positive feedback method is useful for VP9 used for 646 point to point (unicast) communication. The use of RPSI for VP9 is 647 preferably combined with a special update pattern of the codec's two 648 special reference frames -- the golden frame and the altref frame -- 649 in which they are updated in an alternating leapfrog fashion. When a 650 receiver has received and correctly decoded a golden or altref frame, 651 and that frame had a PictureID in the payload descriptor, the 652 receiver can acknowledge this simply by sending an RPSI message back 653 to the sender. The message body (i.e., the "native RPSI bit string" 654 in [RFC4585]) is simply the PictureID of the received frame. 656 Note: because all frames of the same picture must have the same 657 inter-picture reference structure, there is no need for a message to 658 specify which frame is being selected. 660 5.2. Slice Loss Indication (SLI) 662 TODO: Update to indicate which frame within the picture. 664 The slice loss indication is another payload-specific feedback 665 message defined within the RTCP-based feedback format. The SLI 666 message is generated by the receiver when a loss or corruption is 667 detected in a frame. The format of the SLI message is as follows 668 [RFC4585]: 670 0 1 2 3 671 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 672 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 673 | First | Number | PictureID | 674 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 676 Figure 5 678 Here, First is the macroblock address (in scan order) of the first 679 lost block and Number is the number of lost blocks, as defined in 680 [RFC4585]. PictureID is the six least significant bits of the codec- 681 specific picture identifier in which the loss or corruption has 682 occurred. For VP9, this codec-specific identifier is naturally the 683 PictureID of the current frame, as read from the payload descriptor. 684 If the payload descriptor of the current frame does not have a 685 PictureID, the receiver MAY send the last received PictureID+1 in the 686 SLI message. The receiver MAY set the First parameter to 0, and the 687 Number parameter to the total number of macroblocks per frame, even 688 though only part of the frame is corrupted. When the sender receives 689 an SLI message, it can make use of the knowledge from the latest 690 received RPSI message. Knowing that the last golden or altref frame 691 was successfully received, it can encode the next frame with 692 reference to that established reference. 694 5.3. Full Intra Request (FIR) 696 The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a 697 receiver to request a full state refresh of an encoded stream. 699 Upon receipt of an FIR request, a VP9 sender MUST send a picture with 700 a keyframe for its spatial layer 0 layer frame, and then send frames 701 without inter-picture prediction (P=0) for any higher layer frames. 703 5.4. Layer Refresh Request (LRR) 705 The Layer Refresh Request [I-D.ietf-avtext-lrr] allows a receiver to 706 request a single layer of a spatially or temporally encoded stream to 707 be refreshed, without necessarily affecting the stream's other 708 layers. 710 +---------------+---------------+ 711 |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7| 712 +---------------+---------+-----+ 713 | RES | TID | RES | SID | 714 +---------------+---------+-----+ 716 Figure 6 718 Figure 6 shows the format of LRR's layer index fields for VP9 719 streams. The two "RES" fields MUST be set to 0 on transmission and 720 ingnored on reception. See Section 4.2 for details on the TID and 721 SID fields. 723 Identification of a layer refresh frame can be derived from the 724 reference IDs of each frame by backtracking the dependency chain 725 until reaching a point where only decodable frames are being 726 referenced. Therefore it's recommended for both the flexible and the 727 non-flexible mode that, when upgrade frames are being encoded in 728 response to a LRR, those packets should contain layer indices and the 729 reference fields so that the decoder or an MCU can make this 730 derivation. 732 Example: 734 LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying 735 {1,0} to a receiver and which wants to upgrade to {2,1}. In response 736 the encoder should encode the next frames in layers {1,1} and {2,1} 737 by only referring to frames in {1,0}, or {0,0}. 739 In the non-flexible mode, periodic upgrade frames can be defined by 740 the layer structure of the SS, thus periodic upgrade frames can be 741 automatically identified by the picture ID. 743 5.5. Frame Marking 745 The Frame Marking RTP header extension [I-D.ietf-avtext-framemarking] 746 is a mechanism to provide information about frames of video streams 747 in a largely codec-independent manner. However, for its extension 748 for scalable codecs, the specific manner in which codec layers are 749 identified needs to be specified specifically for each codec. This 750 section defines how frame marking is used with VP9. 752 0 1 2 3 753 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 754 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 755 | ID=2 | L=2 |S|E|I|D|B| TID |0|0|0|0|0| SID | TL0PICIDX | 756 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 758 Figure 7 760 When this header extension is used with VP9, the TID and SID fields 761 MUST match the values in the packet which the header extension is 762 attached to; see Section 4.2 for details on these fields. 764 See [I-D.ietf-avtext-framemarking] for explanations of the other 765 fields, which are generic. 767 6. Payload Format Parameters 769 This payload format has two optional parameters. 771 6.1. Media Type Definition 773 This registration is done using the template defined in [RFC6838] and 774 following [RFC4855]. 776 Type name: video 778 Subtype name: VP9 780 Required parameters: None. 782 Optional parameters: 783 These parameters are used to signal the capabilities of a receiver 784 implementation. If the implementation is willing to receive 785 media, both parameters MUST be provided. These parameters MUST 786 NOT be used for any other purpose. 788 max-fr: The value of max-fr is an integer indicating the maximum 789 frame rate in units of frames per second that the decoder is 790 capable of decoding. 792 max-fs: The value of max-fs is an integer indicating the maximum 793 frame size in units of macroblocks that the decoder is capable 794 of decoding. 796 The decoder is capable of decoding this frame size as long as 797 the width and height of the frame in macroblocks are less than 798 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 799 of supporting 640x480 resolution) will support widths and 800 heights up to 1552 pixels (97 macroblocks). 802 Encoding considerations: 803 This media type is framed in RTP and contains binary data; see 804 Section 4.8 of [RFC6838]. 806 Security considerations: See Section 7 of RFC xxxx. 807 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 808 with the number assigned to this document and remove this note.] 810 Interoperability considerations: None. 812 Published specification: VP9 bitstream format [VP9-BITSTREAM] and 813 RFC XXXX. 814 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 815 with the number assigned to this document and remove this note.] 817 Applications which use this media type: 818 For example: Video over IP, video conferencing. 820 Fragment identifier considerations: N/A. 822 Additional information: None. 824 Person & email address to contact for further information: 825 TODO [Pick a contact] 827 Intended usage: COMMON 829 Restrictions on usage: 830 This media type depends on RTP framing, and hence is only defined 831 for transfer via RTP [RFC3550]. 833 Author: TODO [Pick a contact] 835 Change controller: 836 IETF Payload Working Group delegated from the IESG. 838 6.2. SDP Parameters 840 The receiver MUST ignore any fmtp parameter unspecified in this memo. 842 6.2.1. Mapping of Media Subtype Parameters to SDP 844 The media type video/VP9 string is mapped to fields in the Session 845 Description Protocol (SDP) [RFC4566] as follows: 847 o The media name in the "m=" line of SDP MUST be video. 849 o The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 850 media subtype). 852 o The clock rate in the "a=rtpmap" line MUST be 90000. 854 o The parameters "max-fs", and "max-fr", MUST be included in the 855 "a=fmtp" line of SDP if SDP is used to declare receiver 856 capabilities. These parameters are expressed as a media subtype 857 string, in the form of a semicolon separated list of 858 parameter=value pairs. 860 6.2.1.1. Example 862 An example of media representation in SDP is as follows: 864 m=video 49170 RTP/AVPF 98 865 a=rtpmap:98 VP9/90000 866 a=fmtp:98 max-fr=30; max-fs=3600; 868 6.2.2. Offer/Answer Considerations 870 TODO: Update this for VP9 872 7. Security Considerations 874 RTP packets using the payload format defined in this specification 875 are subject to the security considerations discussed in the RTP 876 specification [RFC3550], and in any applicable RTP profile such as 877 RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/ 878 SAVPF [RFC5124]. SAVPF [RFC5124]. However, as "Securing the RTP 879 Protocol Framework: Why RTP Does Not Mandate a Single Media Security 880 Solution" [RFC7202] discusses, it is not an RTP payload format's 881 responsibility to discuss or mandate what solutions are used to meet 882 the basic security goals like confidentiality, integrity and source 883 authenticity for RTP in general. This responsibility lays on anyone 884 using RTP in an application. They can find guidance on available 885 security mechanisms in Options for Securing RTP Sessions [RFC7201]. 886 Applications SHOULD use one or more appropriate strong security 887 mechanisms. The rest of this security consideration section 888 discusses the security impacting properties of the payload format 889 itself. 891 This RTP payload format and its media decoder do not exhibit any 892 significant non-uniformity in the receiver-side computational 893 complexity for packet processing, and thus are unlikely to pose a 894 denial-of-service threat due to the receipt of pathological data. 895 Nor does the RTP payload format contain any active content. 897 8. Congestion Control 899 Congestion control for RTP SHALL be used in accordance with RFC 3550 900 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 901 [RFC3551]. The congestion control mechanism can, in a real-time 902 encoding scenario, adapt the transmission rate by instructing the 903 encoder to encode at a certain target rate. Media aware network 904 elements MAY use the information in the VP9 payload descriptor in 905 Section 4.2 to identify non-reference frames and discard them in 906 order to reduce network congestion. Note that discarding of non- 907 reference frames cannot be done if the stream is encrypted (because 908 the non-reference marker is encrypted). 910 9. IANA Considerations 912 The IANA is requested to register the following values: 913 - Media type registration as described in Section 6.1. 915 10. References 917 10.1. Normative References 919 [I-D.ietf-avtext-framemarking] 920 Zanaty, M., Berger, E., and S. Nandakumar, "Frame Marking 921 RTP Header Extension", draft-ietf-avtext-framemarking-07 922 (work in progress), April 2018. 924 [I-D.ietf-avtext-lrr] 925 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 926 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 927 Message", draft-ietf-avtext-lrr-07 (work in progress), 928 July 2017. 930 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 931 Requirement Levels", BCP 14, RFC 2119, 932 DOI 10.17487/RFC2119, March 1997, 933 . 935 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 936 Jacobson, "RTP: A Transport Protocol for Real-Time 937 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 938 July 2003, . 940 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 941 Description Protocol", RFC 4566, DOI 10.17487/RFC4566, 942 July 2006, . 944 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 945 "Extended RTP Profile for Real-time Transport Control 946 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 947 DOI 10.17487/RFC4585, July 2006, 948 . 950 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 951 Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, 952 . 954 [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, 955 "Codec Control Messages in the RTP Audio-Visual Profile 956 with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, 957 February 2008, . 959 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 960 Specifications and Registration Procedures", BCP 13, 961 RFC 6838, DOI 10.17487/RFC6838, January 2013, 962 . 964 [VP9-BITSTREAM] 965 Grange, A., de Rivaz, P., and J. Hunt, "VP9 Bitstream & 966 Decoding Process Specification", Version 0.6, March 2016, 967 . 971 10.2. Informative References 973 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 974 Video Conferences with Minimal Control", STD 65, RFC 3551, 975 DOI 10.17487/RFC3551, July 2003, 976 . 978 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 979 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 980 RFC 3711, DOI 10.17487/RFC3711, March 2004, 981 . 983 [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for 984 Real-time Transport Control Protocol (RTCP)-Based Feedback 985 (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 986 2008, . 988 [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP 989 Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, 990 . 992 [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP 993 Framework: Why RTP Does Not Mandate a Single Media 994 Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 995 2014, . 997 Authors' Addresses 999 Justin Uberti 1000 Google, Inc. 1001 747 6th Street South 1002 Kirkland, WA 98033 1003 USA 1005 Email: justin@uberti.name 1006 Stefan Holmer 1007 Google, Inc. 1008 Kungsbron 2 1009 Stockholm 111 22 1010 Sweden 1012 Email: holmer@google.com 1014 Magnus Flodman 1015 Google, Inc. 1016 Kungsbron 2 1017 Stockholm 111 22 1018 Sweden 1020 Email: mflodman@google.com 1022 Jonathan Lennox 1023 Vidyo, Inc. 1024 433 Hackensack Avenue 1025 Seventh Floor 1026 Hackensack, NJ 07601 1027 US 1029 Email: jonathan@vidyo.com 1031 Danny Hong 1032 Vidyo, Inc. 1033 433 Hackensack Avenue 1034 Seventh Floor 1035 Hackensack, NJ 07601 1036 US 1038 Email: danny@vidyo.com