idnits 2.17.1 draft-ietf-payload-vp9-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 3, 2017) is 2490 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-16) exists of draft-ietf-avtext-framemarking-04 ** Downref: Normative reference to an Experimental draft: draft-ietf-avtext-framemarking (ref. 'I-D.ietf-avtext-framemarking') == Outdated reference: A later version (-07) exists of draft-ietf-avtext-lrr-06 ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) -- Possible downref: Non-RFC (?) normative reference: ref. 'VP9-BITSTREAM' Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: January 4, 2018 Google 6 J. Lennox 7 D. Hong 8 Vidyo 9 July 3, 2017 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-04 14 Abstract 16 This memo describes an RTP payload format for the VP9 video codec. 17 The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on January 4, 2018. 39 Copyright Notice 41 Copyright (c) 2017 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 58 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 59 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 5 60 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 61 4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 6 62 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 10 63 4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 12 64 4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12 65 4.5. Scalable encoding considerations . . . . . . . . . . . . 12 66 4.6. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 13 67 4.6.1. Reference picture use for scalable structure . . . . 13 68 5. Feedback Messages and Header Extensions . . . . . . . . . . . 14 69 5.1. Reference Picture Selection Indication (RPSI) . . . . . . 14 70 5.2. Slice Loss Indication (SLI) . . . . . . . . . . . . . . . 14 71 5.3. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 15 72 5.4. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 15 73 5.5. Frame Marking . . . . . . . . . . . . . . . . . . . . . . 16 74 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 16 75 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 16 76 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 18 77 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 18 78 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 18 79 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 80 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 19 81 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 82 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 83 10.1. Normative References . . . . . . . . . . . . . . . . . . 20 84 10.2. Informative References . . . . . . . . . . . . . . . . . 21 85 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 87 1. Introduction 89 This memo describes an RTP payload specification applicable to the 90 transmission of video streams encoded using the VP9 video codec 91 [VP9-BITSTREAM]. The format described in this document can be used 92 both in peer-to-peer and video conferencing applications. 94 TODO: VP9 description. Please see [VP9-BITSTREAM]. 96 2. Conventions, Definitions and Acronyms 98 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 99 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 100 document are to be interpreted as described in [RFC2119]. 102 TODO: Cite terminology from [VP9-BITSTREAM]. 104 3. Media Format Description 106 The VP9 codec can maintain up to eight reference frames, of which up 107 to three can be referenced or updated by any new frame. 109 VP9 also allows a reference frame to be resampled and used as a 110 reference for another frame of a different resolution. This allows 111 internal resolution changes without requiring the use of key frames. 113 These features together enable an encoder to implement various forms 114 of coarse-grained scalability, including temporal, spatial and 115 quality scalability modes, as well as combinations of these, without 116 the need for explicit scalable coding tools. 118 Temporal layers define different frame rates of video; spatial and 119 quality layers define different and possibly dependent 120 representations of a single input frame. Spatial layers allow a 121 frame to be encoded at different resolutions, whereas quality layers 122 allow a frame to be encoded at the same resolution but at different 123 qualities (and thus with different amounts of coding error). VP9 124 supports quality layers as spatial layers without any resolution 125 changes; hereinafter, the term "spatial layer" is used to represent 126 both spatial and quality layers. 128 This payload format specification defines how such temporal and 129 spatial scalability layers can be described and communicated. 131 Temporal and spatial scalability layers are associated with non- 132 negative integer IDs. The lowest layer of either type has an ID of 133 0, and is sometimes referred to as the "base" temporal or spatial 134 layer. 136 Layers are designed (and MUST be encoded) such that if any layer, and 137 all higher layers, are removed from the bitstream along either of the 138 two dimensions, the remaining bitstream is still correctly decodable. 140 For terminology, this document uses the term "frame" to refer to a 141 single encoded VP9 frame for a particular resolution/quality, and 142 "picture" to refer to all the representations (frames) at a single 143 instant in time. A picture thus consists of one or more frames, 144 encoding different spatial layers. 146 Within a picture, a frame with spatial layer ID equal to S, where S > 147 0, can depend on a frame of the same picture with a lower spatial 148 layer ID. This "inter-layer" dependency can result in additional 149 coding gain compared to the case where only traditional "inter- 150 picture" dependency is used, where a frame depends on previously 151 coded frame in time. For simplicity, this payload format assumes 152 that, within a picture and if inter-layer dependency is used, a 153 spatial layer S frame can only depend on spatial layer S-1 frame when 154 S > 0. Additionally, if inter-picture dependency is used, spatial 155 layer S frame is assumed to only depend on a previously coded spatial 156 layer S frame. 158 Given above simplifications for inter-layer and inter-picture 159 dependencies, a flag (the D bit described below) is used to indicate 160 whether a spatial layer S frame depends on spatial layer S-1 frame. 161 Given the D bit, a receiver only needs to additionally know the 162 inter-picture dependency structure for a given spatial layer frame in 163 order to determine its decodability. Two modes of describing the 164 inter-picture dependency structure are possible: "flexible mode" and 165 "non-flexible mode". An encoder can only switch between the two on 166 the first packet of a key frame with temporal layer ID equal to 0. 168 In flexible mode, each packet can contain up to 3 reference indices, 169 which identify all frames referenced by the frame transmitted in the 170 current packet for inter-picture prediction. This (along with the D 171 bit) enables a receiver to identify if a frame is decodable or not 172 and helps it understand the temporal layer structure. Since this is 173 signaled in each packet it makes it possible to have very flexible 174 temporal layer hierarchies and patterns which are changing 175 dynamically. 177 In non-flexible mode, the inter-picture dependency (the reference 178 indices) of a Picture Group (PG) MUST be pre-specified as part of the 179 scalability structure (SS) data. In this mode, each packet MUST have 180 an index to refer to one of the described pictures in the PG, from 181 which the pictures referenced by the picture transmitted in the 182 current packet for inter-picture prediction can be identified. 184 (Editor's Note: A "Picture Group", as used in this document, is not 185 the same thing as a "Group of Pictures" as traditionally used in 186 video coding. Suggestions for better terminology are welcome.) 188 The SS data can also be used to specify the resolution of each 189 spatial layer present in the VP9 stream for both flexible and non- 190 flexible modes. 192 4. Payload Format 194 This section describes how the encoded VP9 bitstream is encapsulated 195 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 196 RECOMMENDED. All integer fields in the specifications are encoded as 197 unsigned integers in network octet order. 199 4.1. RTP Header Usage 201 The general RTP payload format for VP9 is depicted below. 203 0 1 2 3 204 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 205 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 206 |V=2|P|X| CC |M| PT | sequence number | 207 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 208 | timestamp | 209 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 210 | synchronization source (SSRC) identifier | 211 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 212 | contributing source (CSRC) identifiers | 213 | .... | 214 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 215 | VP9 payload descriptor (integer #octets) | 216 : : 217 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 218 | : VP9 pyld hdr | | 219 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 220 | | 221 + | 222 : Bytes 2..N of VP9 payload : 223 | | 224 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 225 | : OPTIONAL RTP padding | 226 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 228 The VP9 payload descriptor and VP9 payload header will be described 229 in Section 4.2 and Section 4.3. OPTIONAL RTP padding MUST NOT be 230 included unless the P bit is set. The figure specifically shows the 231 format for the first packet in a frame. Subsequent packets will not 232 contain the VP9 payload header, and will have later octets in the 233 frame payload. 235 Figure 1 237 Marker bit (M): MUST be set to 1 for the final packet of the highest 238 spatial layer frame (the final packet of the picture), and 0 239 otherwise. Unless spatial scalability is in use for this picture, 240 this will have the same value as the E bit described below. Note 241 this bit MUST be set to 1 for the target spatial layer frame if a 242 stream is being rewritten to remove higher spatial layers. 244 Payload Type (PT): In line with the policy in Section 3 of 245 [RFC3551], applications using the VP9 RTP payload profile MUST 246 assign a dynamic payload type number to be used in each RTP 247 session and provide a mechanism to indicate the mapping. See 248 Section 6.2 for the mechanism to be used with the Session 249 Description Protocol (SDP) [RFC4566]. 251 Timestamp: The RTP timestamp indicates the time when the input frame 252 was sampled, at a clock rate of 90 kHz. If the input picture is 253 encoded with multiple layer frames, all of the frames of the 254 picture MUST have the same timestamp. 256 The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, 257 SSRC and CSRC identifiers) are used as specified in Section 5.1 of 258 [RFC3550]. 260 4.2. VP9 Payload Description 262 In flexible mode (with the F bit below set to 1), The first octets 263 after the RTP header are the VP9 payload descriptor, with the 264 following structure. 266 0 1 2 3 4 5 6 7 267 +-+-+-+-+-+-+-+-+ 268 |I|P|L|F|B|E|V|-| (REQUIRED) 269 +-+-+-+-+-+-+-+-+ 270 I: |M| PICTURE ID | (REQUIRED) 271 +-+-+-+-+-+-+-+-+ 272 M: | EXTENDED PID | (RECOMMENDED) 273 +-+-+-+-+-+-+-+-+ 274 L: | T |U| S |D| (CONDITIONALLY RECOMMENDED) 275 +-+-+-+-+-+-+-+-+ -\ 276 P,F: | P_DIFF |N| (CONDITIONALLY REQUIRED) - up to 3 times 277 +-+-+-+-+-+-+-+-+ -/ 278 V: | SS | 279 | .. | 280 +-+-+-+-+-+-+-+-+ 282 Figure 2 284 In non-flexible mode (with the F bit below set to 0), The first 285 octets after the RTP header are the VP9 payload descriptor, with the 286 following structure. 288 0 1 2 3 4 5 6 7 289 +-+-+-+-+-+-+-+-+ 290 |I|P|L|F|B|E|V|-| (REQUIRED) 291 +-+-+-+-+-+-+-+-+ 292 I: |M| PICTURE ID | (RECOMMENDED) 293 +-+-+-+-+-+-+-+-+ 294 M: | EXTENDED PID | (RECOMMENDED) 295 +-+-+-+-+-+-+-+-+ 296 L: | T |U| S |D| (CONDITIONALLY RECOMMENDED) 297 +-+-+-+-+-+-+-+-+ 298 | TL0PICIDX | (CONDITIONALLY REQUIRED) 299 +-+-+-+-+-+-+-+-+ 300 V: | SS | 301 | .. | 302 +-+-+-+-+-+-+-+-+ 304 Figure 3 306 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 307 be present after the mandatory first octet and specified as below. 308 Otherwise, PID MUST NOT be present. 310 P: Inter-picture predicted picture. When set to zero, the picture 311 does not utilize inter-picture prediction. In this case, up- 312 switching to a current spatial layer's frame is possible from 313 directly lower spatial layer frame. P SHOULD also be set to zero 314 when encoding a layer synchronization frame in response to an LRR 315 [I-D.ietf-avtext-lrr] message (see Section 5.4). When P is set to 316 zero, the T field (described below) MUST also be set to 0 (if 317 present). Note that the P bit does not forbid intra-picture, 318 inter-layer prediction from earlier frames of the same picture, if 319 any. 321 L: Layer indices present. When set to one, the one or two octets 322 following the mandatory first octet and the PID (if present) is as 323 described by "Layer indices" below. If the F bit (described 324 below) is set to 1 (indicating flexible mode), then only one octet 325 is present for the layer indices. Otherwise if the F bit is set 326 to 0 (indicating non-flexible mode), then two octets are present 327 for the layer indices. 329 F: Flexible mode. F set to one indicates flexible mode and if the P 330 bit is also set to one, then the octets following the mandatory 331 first octet, the PID, and layer indices (if present) are as 332 described by "Reference indices" below. This MUST only be set to 333 1 if the I bit is also set to one; if the I bit is set to zero, 334 then this MUST also be set to zero and ignored by receivers. The 335 value of this F bit MUST only change on the first packet of a key 336 picture. A key picture is a picture whose base spatial layer 337 frame is a key frame, and which thus completely resets the encoder 338 state. This packet will have its P bit equal to zero, S or D bit 339 (described below) equal to zero, and B bit (described below) equal 340 to 1. 342 B: Start of a frame. MUST be set to 1 if the first payload octet of 343 the RTP packet is the beginning of a new VP9 frame, and MUST NOT 344 be 1 otherwise. Note that this frame might not be the first frame 345 of a picture. 347 E: End of a frame. MUST be set to 1 for the final RTP packet of a 348 VP9 frame, and 0 otherwise. This enables a decoder to finish 349 decoding the frame, where it otherwise may need to wait for the 350 next packet to explicitly know that the frame is complete. Note 351 that, if spatial scalability is in use, more frames from the same 352 picture may follow; see the description of the M bit above. 354 V: Scalability structure (SS) data present. When set to one, the 355 OPTIONAL SS data MUST be present in the payload descriptor. 356 Otherwise, the SS data MUST NOT be present. 358 -: Bit reserved for future use. MUST be set to zero and MUST be 359 ignored by the receiver. 361 The mandatory first octet is followed by the extension data fields 362 that are enabled: 364 M: The most significant bit of the first octet is an extension flag. 365 The field MUST be present if the I bit is equal to one. If set, 366 the PID field MUST contain 15 bits; otherwise, it MUST contain 7 367 bits. See PID below. 369 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 370 on the M bit. This is a running index of the pictures. The field 371 MUST be present if the I bit is equal to one. If M is set to 372 zero, 7 bits carry the PID; else if M is set to one, 15 bits carry 373 the PID in network byte order. The sender may choose between a 7- 374 or 15-bit index. The PID SHOULD start on a random number, and 375 MUST wrap after reaching the maximum ID. The receiver MUST NOT 376 assume that the number of bits in PID stay the same through the 377 session. 379 In the non-flexible mode (when the F bit is set to 0), this PID is 380 used as an index to the picture group (PG) specified in the SS 381 data below. In this mode, the PID of the key frame corresponds to 382 the first specified frame in the PG. Then subsequent PIDs are 383 mapped to subsequently specified frames in the PG (modulo N_G, 384 specified in the SS data below), respectively. 386 Layer indices: This information is optional but recommended whenever 387 encoding with layers. For both flexible and non-flexible modes, 388 one octet is used to specify a layer frame's temporal layer ID (T) 389 and spatial layer ID (S) as shown both in Figure 2 and Figure 3. 390 Additionally, a bit (U) is used to indicate that the current frame 391 is a "switching up point" frame. Another bit (D) is used to 392 indicate whether inter-layer prediction is used for the current 393 frame. 395 In the non-flexible mode (when the F bit is set to 0), another 396 octet is used to represent temporal layer 0 index (TL0PICIDX), as 397 depicted in Figure 3. The TL0PICIDX is present so that all 398 minimally required frames - the base temporal layer frames - can 399 be tracked. 401 The T and S fields indicate the temporal and spatial layers and 402 can help middleboxes and and endpoints quickly identify which 403 layer a packet belongs to. 405 T: The temporal layer ID of current frame. In the case of non- 406 flexible mode, if PID is mapped to a picture in a specified PG, 407 then the value of T MUST match the corresponding T value of the 408 mapped picture in the PG. 410 U: Switching up point. If this bit is set to 1 for the current 411 picture with temporal layer ID equal to T, then "switch up" to 412 a higher frame rate is possible as subsequent higher temporal 413 layer pictures will not depend on any picture before the 414 current picture (in coding order) with temporal layer ID 415 greater than T. 417 S: The spatial layer ID of current frame. Note that frames with 418 spatial layer S > 0 may be dependent on decoded spatial layer 419 S-1 frame within the same picture. 421 D: Inter-layer dependency used. MUST be set to one if current 422 spatial layer S frame depends on spatial layer S-1 frame of the 423 same picture. MUST only be set to zero if current spatial 424 layer S frame does not depend on spatial layer S-1 frame of the 425 same picture. For the base layer frame (with S equal to 0), 426 this D bit MUST be set to zero. 428 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 429 present in the non-flexible mode (F = 0). This is a running 430 index for the temporal base layer pictures, i.e., the pictures 431 with T set to 0. If T is larger than 0, TL0PICIDX indicates 432 which temporal base layer picture the current picture depends 433 on. TL0PICIDX MUST be incremented when T is equal to 0. The 434 index SHOULD start on a random number, and MUST restart at 0 435 after reaching the maximum number 255. 437 Reference indices: When P and F are both set to one, indicating a 438 non-key frame in flexible mode, then at least one reference index 439 has to be specified as below. Additional reference indices (total 440 of up to 3 reference indices are allowed) may be specified using 441 the N bit below. When either P or F is set to zero, then no 442 reference index is specified. 444 P_DIFF: The reference index (in 7 bits) specified as the relative 445 PID from the current picture. For example, when P_DIFF=3 on a 446 packet containing the picture with PID 112 means that the 447 picture refers back to the picture with PID 109. This 448 calculation is done modulo the size of the PID field, i.e., 449 either 7 or 15 bits. 451 N: 1 if there is additional P_DIFF following the current P_DIFF. 453 4.2.1. Scalability Structure (SS): 455 The scalability structure (SS) data describes the resolution of each 456 frame within a picture as well as the inter-picture dependencies for 457 a picture group (PG). If the VP9 payload descriptor's "V" bit is 458 set, the SS data is present in the position indicated in Figure 2 and 459 Figure 3. 461 +-+-+-+-+-+-+-+-+ 462 V: | N_S |Y|G|-|-|-| 463 +-+-+-+-+-+-+-+-+ -\ 464 Y: | WIDTH | (OPTIONAL) . 465 + + . 466 | | (OPTIONAL) . 467 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 468 | HEIGHT | (OPTIONAL) . 469 + + . 470 | | (OPTIONAL) . 471 +-+-+-+-+-+-+-+-+ -/ -\ 472 G: | N_G | (OPTIONAL) 473 +-+-+-+-+-+-+-+-+ -\ 474 N_G: | T |U| R |-|-| (OPTIONAL) . 475 +-+-+-+-+-+-+-+-+ -\ . - N_G times 476 | P_DIFF | (OPTIONAL) . - R times . 477 +-+-+-+-+-+-+-+-+ -/ -/ 479 Figure 4 481 N_S: N_S + 1 indicates the number of spatial layers present in the 482 VP9 stream. 484 Y: Each spatial layer's frame resolution present. When set to one, 485 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 486 present for each layer frame. Otherwise, the resolution MUST NOT 487 be present. 489 G: PG description present flag. 491 -: Bit reserved for future use. MUST be set to zero and MUST be 492 ignored by the receiver. 494 N_G: N_G indicates the number of pictures in a Picture Group (PG). 495 If N_G is greater than 0, then the SS data allows the inter- 496 picture dependency structure of the VP9 stream to be pre-declared, 497 rather than indicating it on the fly with every packet. If N_G is 498 greater than 0, then for N_G pictures in the PG, each picture's 499 temporal layer ID (T), switch up point (U), and the R reference 500 indices (P_DIFFs) are specified. 502 The first picture specified in the PG MUST have T set to 0. 504 G set to 0 or N_G set to 0 indicates that either there is only one 505 temporal layer or no fixed inter-picture dependency information is 506 present going forward in the bitstream. 508 Note that for a given picture, all frames follow the same inter- 509 picture dependency structure. However, the frame rate of each 510 spatial layer can be different from each other and this can be 511 controlled with the use of the D bit described above. The 512 specified dependency structure in the SS data MUST be for the 513 highest frame rate layer. 515 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 516 included in the first packet of every key frame. This is a packet 517 with P bit equal to zero, S or D bit equal to zero, and B bit equal 518 to 1. The SS data MUST only be changed on the picture that 519 corresponds to the first picture specified in the previous SS data's 520 PG (if the previous SS data's N_G was greater than 0). 522 4.3. VP9 Payload Header 524 TODO: need to describe VP9 payload header. 526 4.4. Frame Fragmentation 528 VP9 frames are fragmented into packets, in RTP sequence number order, 529 beginning with a packet with the B bit set, and ending with a packet 530 with the E bit set. There is no mechanism for finer-grained access 531 to parts of a VP9 frame. 533 4.5. Scalable encoding considerations 535 In addition to the use of reference frames, VP9 has several 536 additional forms of inter-frame dependencies, largely involving 537 probability tables for the entropy and tree encoders. In VP9 syntax, 538 the syntax element "error_resilient_mode" resets this additional 539 inter-frame data, allowing a frame's syntax to be decoded 540 independently. 542 Due to the requirements of scalable streams, a VP9 encoder producing 543 a scalable stream needs to ensure that a frame does not depend on a 544 previous frame (of the same or a previous picture) that can 545 legitimately be removed from the stream. Thus, a frame that follows 546 a removable frame (in full decode order) MUST be encoded with 547 "error_resilient_mode" to true. 549 For spatially-scalable streams, this means that 550 "error_resilient_mode" needs to be turned on for the base spatial 551 layer; it can however be turned off for higher spatial layers, 552 assuming they are sent with inter-layer dependency (i.e. with the "D" 553 bit set). For streams that are only temporally-scalable without 554 spatial scalability, "error_resilient_mode" can additionally be 555 turned off for any picture that immediately follows a temporal layer 556 0 frame. 558 4.6. Examples of VP9 RTP Stream 560 TODO: Examples of packet layouts 562 4.6.1. Reference picture use for scalable structure 564 As discussed in Section 3, the VP9 codec can maintain up to eight 565 reference frames, of which up to three can be referenced or updated 566 by any new frame. This section illustrates one way that a scalable 567 structure (with three spatial layers and three temporal layers) can 568 be constructed using these reference frames. 570 +----------+---------+------------+---------+ 571 | Temporal | Spatial | References | Updates | 572 +----------+---------+------------+---------+ 573 | 0 | 0 | 0 | 0 | 574 | | | | | 575 | 0 | 1 | 0,1 | 1 | 576 | | | | | 577 | 0 | 2 | 1,2 | 2 | 578 | | | | | 579 | 2 | 0 | 0 | 6 | 580 | | | | | 581 | 2 | 1 | 1,6 | 7 | 582 | | | | | 583 | 2 | 2 | 2,7 | - | 584 | | | | | 585 | 1 | 0 | 0 | 3 | 586 | | | | | 587 | 1 | 1 | 1,3 | 4 | 588 | | | | | 589 | 1 | 2 | 2,4 | 5 | 590 | | | | | 591 | 2 | 0 | 3 | 6 | 592 | | | | | 593 | 2 | 1 | 4,6 | 7 | 594 | | | | | 595 | 2 | 2 | 5,7 | - | 596 +----------+---------+------------+---------+ 598 Example scalability structure 600 This structure is constructed such that the "U" bit can always be 601 set. 603 5. Feedback Messages and Header Extensions 605 5.1. Reference Picture Selection Indication (RPSI) 607 The reference picture selection index is a payload-specific feedback 608 message defined within the RTCP-based feedback format. The RPSI 609 message is generated by a receiver and can be used in two ways. 610 Either it can signal a preferred reference picture when a loss has 611 been detected by the decoder -- preferably then a reference that the 612 decoder knows is perfect -- or, it can be used as positive feedback 613 information to acknowledge correct decoding of certain reference 614 pictures. The positive feedback method is useful for VP9 used for 615 point to point (unicast) communication. The use of RPSI for VP9 is 616 preferably combined with a special update pattern of the codec's two 617 special reference frames -- the golden frame and the altref frame -- 618 in which they are updated in an alternating leapfrog fashion. When a 619 receiver has received and correctly decoded a golden or altref frame, 620 and that frame had a PictureID in the payload descriptor, the 621 receiver can acknowledge this simply by sending an RPSI message back 622 to the sender. The message body (i.e., the "native RPSI bit string" 623 in [RFC4585]) is simply the PictureID of the received frame. 625 Note: because all frames of the same picture must have the same 626 inter-picture reference structure, there is no need for a message to 627 specify which frame is being selected. 629 5.2. Slice Loss Indication (SLI) 631 TODO: Update to indicate which frame within the picture. 633 The slice loss indication is another payload-specific feedback 634 message defined within the RTCP-based feedback format. The SLI 635 message is generated by the receiver when a loss or corruption is 636 detected in a frame. The format of the SLI message is as follows 637 [RFC4585]: 639 0 1 2 3 640 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 641 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 642 | First | Number | PictureID | 643 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 645 Figure 5 647 Here, First is the macroblock address (in scan order) of the first 648 lost block and Number is the number of lost blocks, as defined in 649 [RFC4585]. PictureID is the six least significant bits of the codec- 650 specific picture identifier in which the loss or corruption has 651 occurred. For VP9, this codec-specific identifier is naturally the 652 PictureID of the current frame, as read from the payload descriptor. 653 If the payload descriptor of the current frame does not have a 654 PictureID, the receiver MAY send the last received PictureID+1 in the 655 SLI message. The receiver MAY set the First parameter to 0, and the 656 Number parameter to the total number of macroblocks per frame, even 657 though only part of the frame is corrupted. When the sender receives 658 an SLI message, it can make use of the knowledge from the latest 659 received RPSI message. Knowing that the last golden or altref frame 660 was successfully received, it can encode the next frame with 661 reference to that established reference. 663 5.3. Full Intra Request (FIR) 665 The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a 666 receiver to request a full state refresh of an encoded stream. 668 Upon receipt of an FIR request, a VP9 sender MUST send a picture with 669 a keyframe for its spatial layer 0 layer frame, and then send frames 670 without inter-picture prediction (P=0) for any higher layer frames. 672 5.4. Layer Refresh Request (LRR) 674 The Layer Refresh Request [I-D.ietf-avtext-lrr] allows a receiver to 675 request a single layer of a spatially or temporally encoded stream to 676 be refreshed, without necessarily affecting the stream's other 677 layers. 679 +---------------+---------------+ 680 |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7| 681 +---------------+---------+-----+ 682 | RES | T | RES | S | 683 +---------------+---------+-----+ 685 Figure 6 687 Figure 6 shows the format of LRR's layer index fields for VP9 688 streams. The two "RES" fields MUST be set to 0 on transmission and 689 ingnored on reception. See Section 4.2 for details on the T and S 690 fields. 692 Identification of a layer refresh frame can be derived from the 693 reference IDs of each frame by backtracking the dependency chain 694 until reaching a point where only decodable frames are being 695 referenced. Therefore it's recommended for both the flexible and the 696 non-flexible mode that, when upgrade frames are being encoded in 697 response to a LRR, those packets should contain layer indices and the 698 reference fields so that the decoder or an MCU can make this 699 derivation. 701 Example: 703 LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying 704 {1,0} to a receiver and which wants to upgrade to {2,1}. In response 705 the encoder should encode the next frames in layers {1,1} and {2,1} 706 by only referring to frames in {1,0}, or {0,0}. 708 In the non-flexible mode, periodic upgrade frames can be defined by 709 the layer structure of the SS, thus periodic upgrade frames can be 710 automatically identified by the picture ID. 712 5.5. Frame Marking 714 The Frame Marking RTP header extension [I-D.ietf-avtext-framemarking] 715 is a mechanism to provide information about frames of video streams 716 in a largely codec-independent manner. However, for its extension 717 for scalable codecs, the specific manner in which codec layers are 718 identified needs to be specified specifically for each codec. This 719 section defines how frame marking is used with VP9. 721 0 1 2 3 722 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 723 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 724 | ID=2 | L=2 |S|E|I|D|B| T |0|0|0|0|0| S | TL0PICIDX | 725 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 727 Figure 7 729 When this header extension is used with VP9, the T and S fields MUST 730 match the values in the packet which the header extension is attached 731 to; see Section 4.2 for details on these fields. 733 See [I-D.ietf-avtext-framemarking] for explanations of the other 734 fields, which are generic. 736 6. Payload Format Parameters 738 This payload format has two optional parameters. 740 6.1. Media Type Definition 742 This registration is done using the template defined in [RFC6838] and 743 following [RFC4855]. 745 Type name: video 746 Subtype name: VP9 748 Required parameters: None. 750 Optional parameters: 751 These parameters are used to signal the capabilities of a receiver 752 implementation. If the implementation is willing to receive 753 media, both parameters MUST be provided. These parameters MUST 754 NOT be used for any other purpose. 756 max-fr: The value of max-fr is an integer indicating the maximum 757 frame rate in units of frames per second that the decoder is 758 capable of decoding. 760 max-fs: The value of max-fs is an integer indicating the maximum 761 frame size in units of macroblocks that the decoder is capable 762 of decoding. 764 The decoder is capable of decoding this frame size as long as 765 the width and height of the frame in macroblocks are less than 766 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 767 of supporting 640x480 resolution) will support widths and 768 heights up to 1552 pixels (97 macroblocks). 770 Encoding considerations: 771 This media type is framed in RTP and contains binary data; see 772 Section 4.8 of [RFC6838]. 774 Security considerations: See Section 7 of RFC xxxx. 775 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 776 with the number assigned to this document and remove this note.] 778 Interoperability considerations: None. 780 Published specification: VP9 bitstream format [VP9-BITSTREAM] and 781 RFC XXXX. 782 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 783 with the number assigned to this document and remove this note.] 785 Applications which use this media type: 786 For example: Video over IP, video conferencing. 788 Fragment identifier considerations: N/A. 790 Additional information: None. 792 Person & email address to contact for further information: 793 TODO [Pick a contact] 795 Intended usage: COMMON 797 Restrictions on usage: 798 This media type depends on RTP framing, and hence is only defined 799 for transfer via RTP [RFC3550]. 801 Author: TODO [Pick a contact] 803 Change controller: 804 IETF Payload Working Group delegated from the IESG. 806 6.2. SDP Parameters 808 The receiver MUST ignore any fmtp parameter unspecified in this memo. 810 6.2.1. Mapping of Media Subtype Parameters to SDP 812 The media type video/VP9 string is mapped to fields in the Session 813 Description Protocol (SDP) [RFC4566] as follows: 815 o The media name in the "m=" line of SDP MUST be video. 817 o The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 818 media subtype). 820 o The clock rate in the "a=rtpmap" line MUST be 90000. 822 o The parameters "max-fs", and "max-fr", MUST be included in the 823 "a=fmtp" line of SDP if SDP is used to declare receiver 824 capabilities. These parameters are expressed as a media subtype 825 string, in the form of a semicolon separated list of 826 parameter=value pairs. 828 6.2.1.1. Example 830 An example of media representation in SDP is as follows: 832 m=video 49170 RTP/AVPF 98 833 a=rtpmap:98 VP9/90000 834 a=fmtp:98 max-fr=30; max-fs=3600; 836 6.2.2. Offer/Answer Considerations 838 TODO: Update this for VP9 840 7. Security Considerations 842 RTP packets using the payload format defined in this specification 843 are subject to the security considerations discussed in the RTP 844 specification [RFC3550], and in any applicable RTP profile such as 845 RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/ 846 SAVPF [RFC5124]. SAVPF [RFC5124]. However, as "Securing the RTP 847 Protocol Framework: Why RTP Does Not Mandate a Single Media Security 848 Solution" [RFC7202] discusses, it is not an RTP payload format's 849 responsibility to discuss or mandate what solutions are used to meet 850 the basic security goals like confidentiality, integrity and source 851 authenticity for RTP in general. This responsibility lays on anyone 852 using RTP in an application. They can find guidance on available 853 security mechanisms in Options for Securing RTP Sessions [RFC7201]. 854 Applications SHOULD use one or more appropriate strong security 855 mechanisms. The rest of this security consideration section 856 discusses the security impacting properties of the payload format 857 itself. 859 This RTP payload format and its media decoder do not exhibit any 860 significant non-uniformity in the receiver-side computational 861 complexity for packet processing, and thus are unlikely to pose a 862 denial-of-service threat due to the receipt of pathological data. 863 Nor does the RTP payload format contain any active content. 865 8. Congestion Control 867 Congestion control for RTP SHALL be used in accordance with RFC 3550 868 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 869 [RFC3551]. The congestion control mechanism can, in a real-time 870 encoding scenario, adapt the transmission rate by instructing the 871 encoder to encode at a certain target rate. Media aware network 872 elements MAY use the information in the VP9 payload descriptor in 873 Section 4.2 to identify non-reference frames and discard them in 874 order to reduce network congestion. Note that discarding of non- 875 reference frames cannot be done if the stream is encrypted (because 876 the non-reference marker is encrypted). 878 9. IANA Considerations 880 The IANA is requested to register the following values: 881 - Media type registration as described in Section 6.1. 883 10. References 884 10.1. Normative References 886 [I-D.ietf-avtext-framemarking] 887 Berger, E., Nandakumar, S., and M. Zanaty, "Frame Marking 888 RTP Header Extension", draft-ietf-avtext-framemarking-04 889 (work in progress), March 2017. 891 [I-D.ietf-avtext-lrr] 892 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 893 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 894 Message", draft-ietf-avtext-lrr-06 (work in progress), 895 June 2017. 897 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 898 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 899 RFC2119, March 1997, 900 . 902 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 903 Jacobson, "RTP: A Transport Protocol for Real-Time 904 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 905 July 2003, . 907 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 908 Description Protocol", RFC 4566, DOI 10.17487/RFC4566, 909 July 2006, . 911 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 912 "Extended RTP Profile for Real-time Transport Control 913 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 914 10.17487/RFC4585, July 2006, 915 . 917 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 918 Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, 919 . 921 [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, 922 "Codec Control Messages in the RTP Audio-Visual Profile 923 with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, 924 February 2008, . 926 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 927 Specifications and Registration Procedures", BCP 13, RFC 928 6838, DOI 10.17487/RFC6838, January 2013, 929 . 931 [VP9-BITSTREAM] 932 Grange, A., de Rivaz, P., and J. Hunt, "VP9 Bitstream & 933 Decoding Process Specification", Version 0.6, March 2016, 934 . 938 10.2. Informative References 940 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 941 Video Conferences with Minimal Control", STD 65, RFC 3551, 942 DOI 10.17487/RFC3551, July 2003, 943 . 945 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 946 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 947 RFC 3711, DOI 10.17487/RFC3711, March 2004, 948 . 950 [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for 951 Real-time Transport Control Protocol (RTCP)-Based Feedback 952 (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 953 2008, . 955 [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP 956 Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, 957 . 959 [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP 960 Framework: Why RTP Does Not Mandate a Single Media 961 Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 962 2014, . 964 Authors' Addresses 966 Justin Uberti 967 Google, Inc. 968 747 6th Street South 969 Kirkland, WA 98033 970 USA 972 Email: justin@uberti.name 973 Stefan Holmer 974 Google, Inc. 975 Kungsbron 2 976 Stockholm 111 22 977 Sweden 979 Email: holmer@google.com 981 Magnus Flodman 982 Google, Inc. 983 Kungsbron 2 984 Stockholm 111 22 985 Sweden 987 Email: mflodman@google.com 989 Jonathan Lennox 990 Vidyo, Inc. 991 433 Hackensack Avenue 992 Seventh Floor 993 Hackensack, NJ 07601 994 US 996 Email: jonathan@vidyo.com 998 Danny Hong 999 Vidyo, Inc. 1000 433 Hackensack Avenue 1001 Seventh Floor 1002 Hackensack, NJ 07601 1003 US 1005 Email: danny@vidyo.com