idnits 2.17.1 draft-ietf-payload-vp9-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 6, 2015) is 3210 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational draft: draft-grange-vp9-bitstream (ref. 'I-D.grange-vp9-bitstream') ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group J. Uberti 3 Internet-Draft S. Holmer 4 Intended status: Standards Track M. Flodman 5 Expires: January 7, 2016 Google 6 J. Lennox 7 D. Hong 8 Vidyo 9 July 6, 2015 11 RTP Payload Format for VP9 Video 12 draft-ietf-payload-vp9-00 14 Abstract 16 This memo describes an RTP payload format for the VP9 video codec. 17 The payload format has wide applicability, as it supports 18 applications from low bit-rate peer-to-peer usage, to high bit-rate 19 video conferences. It includes provisions for temporal and spatial 20 scalability. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on January 7, 2016. 39 Copyright Notice 41 Copyright (c) 2015 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 2 58 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 59 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 60 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 4 61 4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 6 62 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 10 63 4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 12 64 4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12 65 4.5. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 12 66 5. Using VP9 with RPSI and SLI Feedback . . . . . . . . . . . . 12 67 5.1. RPSI . . . . . . . . . . . . . . . . . . . . . . . . . . 12 68 5.2. SLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 69 5.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 13 70 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 15 71 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 15 72 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 17 73 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 17 74 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 17 75 7. Security Considerations . . . . . . . . . . . . . . . . . . . 17 76 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 18 77 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 18 80 10.2. Informative References . . . . . . . . . . . . . . . . . 19 81 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 83 1. Introduction 85 This memo describes an RTP payload specification applicable to the 86 transmission of video streams encoded using the VP9 video codec 87 [I-D.grange-vp9-bitstream]. The format described in this document 88 can be used both in peer-to-peer and video conferencing applications. 90 TODO: VP9 description. Please see [I-D.grange-vp9-bitstream]. 92 2. Conventions, Definitions and Acronyms 94 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 95 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 96 document are to be interpreted as described in [RFC2119]. 98 3. Media Format Description 100 The VP9 codec can maintain up to eight reference frames, of which up 101 to three can be referenced or updated by any new frame. 103 VP9 also allows a reference frame to be resampled and used as a 104 reference for another frame of a different resolution. This allows 105 internal resolution changes without requiring the use of key frames. 107 These features together enable an encoder to implement various forms 108 of coarse-grained scalability, including temporal, spatial and 109 quality scalability modes, as well as combinations of these, without 110 the need for explicit scalable coding tools. 112 Temporal layers define different frame rates of video; spatial and 113 quality layers define different and possibly dependent 114 representations of a single input frame. Spatial layers allow a 115 frame to be encoded at different resolutions, whereas quality layers 116 allow a frame to be encoded at the same resolution but at different 117 qualities (and thus with different amounts of coding error). VP9 118 supports quality layers as spatial layers without any resolution 119 changes; hereinafter, the term "spatial layer" is used to represent 120 both spatial and quality layers. 122 This payload format specification defines how such temporal and 123 spatial scalability layers can be described and communicated. 125 Layers are designed (and MUST be encoded) such that if any layer, and 126 all higher layers, are removed from the bitstream along any of the 127 two dimensions, the remaining bitstream is still correctly decodable. 129 For terminology, this document uses the term "layer frame" to refer 130 to a single encoded VP9 frame for a particular resolution/quality, 131 and "super frame" to refer to all the representations (layer frames) 132 at a single instant in time. A super frame thus consists of one or 133 more layer frames, encoding different spatial layers. 135 Within a super frame, a layer frame with spatial layer ID equal to S, 136 where S > 0, can depend on a frame with a lower spatial layer ID. 137 This "inter-layer" dependency results in additional coding gain to 138 the traditional "inter-picture" dependency, where a frame depends on 139 previously coded frame in time. For simplicity, this payload format 140 assumes that, within a super frame if inter-layer dependency is used, 141 a spatial layer S frame can only depend on spatial layer S-1 frame 142 when S > 0. Additionally, if inter-picture dependency is used, 143 spatial layer S frame is assumed to only depend on prevously coded 144 spatial layer S frame. 146 TODO: Describe how simulcast can be supported? 148 Given above simplifications for inter-layer and inter-picture 149 dependencies, a flag (the D bit described below) is used to indicate 150 whether a spatial layer S frame depends on spatial layer S-1 frame. 151 Then a receiver only needs to know the inter-picture dependency 152 structure for a given spatial layer frame in order to determine its 153 decodability. Two modes of describing the inter-picture dependency 154 structure are possible: "flexible mode" and "non-flexible mode". An 155 encoder can only switch between the two on the very first packet of a 156 key frame with temporal layer ID equal to 0. 158 In flexible mode, each packet can contain up to 3 reference indices, 159 which identifies all frames referenced by the frame transmitted in 160 the current packet for inter-picture prediction. This (along with 161 the D bit) enables a receiver to identify if a frame is decodable or 162 not and helps it understand the temporal layer structure so that it 163 can drop packets as it sees fit. Since this is signaled in each 164 packet it makes it possible to have very flexible temporal layer 165 hierarchies and patterns which are changing dynamically. 167 In non-flexible mode, the inter-picture dependency (the reference 168 indices) of a group of frames (GOF) MUST be pre-specified as part of 169 the scalability structure (SS) data. In this mode, each packet will 170 have an index to refer to one of the described frames, from which the 171 frames referenced by the frame transmitted in the current packet for 172 inter-picture prediction can be identified. 174 The SS data can also be used to specify the resolution of each 175 spatial layer present in the VP9 stream. 177 4. Payload Format 179 This section describes how the encoded VP9 bitstream is encapsulated 180 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 181 RECOMMENDED. All integer fields in the specifications are encoded as 182 unsigned integers in network octet order. 184 4.1. RTP Header Usage 185 The general RTP payload format for VP9 is depicted below. 187 0 1 2 3 188 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 190 |V=2|P|X| CC |M| PT | sequence number | 191 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 192 | timestamp | 193 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 194 | synchronization source (SSRC) identifier | 195 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 196 | contributing source (CSRC) identifiers | 197 | .... | 198 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 199 | VP9 payload descriptor (integer #bytes) | 200 : : 201 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 202 | : VP9 pyld hdr | | 203 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 204 | | 205 + | 206 : Bytes 2..N of VP9 payload : 207 | | 208 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 209 | : OPTIONAL RTP padding | 210 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 212 The VP9 payload descriptor and VP9 payload header will be described 213 in the next section. OPTIONAL RTP padding MUST NOT be included 214 unless the P bit is set. 216 Figure 1 218 Marker bit (M): MUST be set to 1 for the final packet of the highest 219 spatial layer frame (the final packet of the super frame), and 0 220 otherwise. Unless spatial scalability is in use for this super 221 frame, this will have the same value as the E bit described below. 222 Note that a MANE MUST set this value to 1 for the target spatial 223 layer frame when shaping out higher spatial layers. 225 Timestamp: The RTP timestamp indicates the time when the input frame 226 was sampled, at a clock rate of 90 kHz. If the input frame is 227 encoded with multiple layer frames, all of the layer frames of the 228 super frame MUST have the same timestamp. 230 Sequence number: The sequence numbers are monotonically increasing 231 in order of the encoded bitstream. 233 The remaining RTP header fields are used as specified in [RFC3550]. 235 4.2. VP9 Payload Description 237 In flexible mode (with the F bit below set to 1), The first octets 238 after the RTP header are the VP9 payload descriptor, with the 239 following structure. 241 0 1 2 3 4 5 6 7 242 +-+-+-+-+-+-+-+-+ 243 |I|P|L|F|B|E|V|-| (REQUIRED) 244 +-+-+-+-+-+-+-+-+ 245 I: |M| PICTURE ID | (RECOMMENDED) 246 +-+-+-+-+-+-+-+-+ 247 M: | EXTENDED PID | (RECOMMENDED) 248 +-+-+-+-+-+-+-+-+ 249 L: | T |U| S |D| (CONDITIONALLY RECOMMENDED) 250 +-+-+-+-+-+-+-+-+ -\ 251 P,F: | P_DIFF |X|N| (CONDITIONALLY RECOMMENDED) . 252 +-+-+-+-+-+-+-+-+ . - up to 3 times 253 X: |EXTENDED P_DIFF| (OPTIONAL) . 254 +-+-+-+-+-+-+-+-+ -/ 255 V: | SS | 256 | .. | 257 +-+-+-+-+-+-+-+-+ 259 Figure 2 261 In non-flexible mode (with the F bit below set to 0), The first 262 octets after the RTP header are the VP9 payload descriptor, with the 263 following structure. 265 0 1 2 3 4 5 6 7 266 +-+-+-+-+-+-+-+-+ 267 |I|P|L|F|B|E|V|-| (REQUIRED) 268 +-+-+-+-+-+-+-+-+ 269 I: |M| PICTURE ID | (RECOMMENDED) 270 +-+-+-+-+-+-+-+-+ 271 M: | EXTENDED PID | (RECOMMENDED) 272 +-+-+-+-+-+-+-+-+ 273 L: |GOF_IDX| S |D| (CONDITIONALLY RECOMMENDED) 274 +-+-+-+-+-+-+-+-+ 275 | TL0PICIDX | (CONDITIONALLY REQUIRED) 276 +-+-+-+-+-+-+-+-+ 277 V: | SS | 278 | .. | 279 +-+-+-+-+-+-+-+-+ 281 Figure 3 283 I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST 284 be present after the mandatory first octet and specified as below. 285 Otherwise, PID MUST NOT be present. 287 P: Inter-picture predicted layer frame. When set to zero, the layer 288 frame does not utilize inter-picture prediction. In this case, 289 up-switching to current spatial layer's frame is possible from 290 directly lower spatial layer frame. P SHOULD also be set to zero 291 when encoding a layer synchronization frame in response to an LRR 292 [I-D.lennox-avtext-lrr]. 294 L: Layer indices present. When set to one, the one or two octets 295 following the mandatory first octet and the PID (if present) is as 296 described by "Layer indices" below. If the F bit (described 297 below) is set to 1 (indicating flexible mode), then only one octet 298 is present for the layer indices. Otherwise if the F bit is set 299 to 0 (indicating non-flexible mode), then two octets are present 300 for the layer indices. 302 F: Flexible mode. F set to one indicates flexible mode and if the P 303 bit is also set to one, then the octets following the mandatory 304 first octet, the PID, and layer indices (if present) are as 305 described by "Reference indices" below. This MUST only be set to 306 one if the I bit is also set to one; if the I bit is set to zero, 307 then this MUST also be set to zero and ignored by receivers. The 308 value of this F bit CAN ONLY CHANGE on the very first packet of a 309 key picture. This is a packet with the P bit equal to zero, S or 310 D bit (described below) equal to zero, B bit (described below) 311 equal to 1, and temporal layer ID equal to 0. 313 B: Start of a layer frame. MUST be set to 1 if the first payload 314 octet of the RTP packet is the beginning of a new VP9 layer frame, 315 and MUST NOT be 1 otherwise. Note that this layer frame might not 316 be the very first layer frame of a super frame. 318 E: End of a layer frame. MUST be set to 1 for the final RTP packet 319 of a VP9 layer frame, and 0 otherwise. This enables a decoder to 320 finish decoding the layer frame, where it otherwise may need to 321 wait for the next packet to explicitly know that the layer frame 322 is complete. Note that, if spatial scalability is in use, more 323 layer frames from the same super frame may follow; see the 324 description of the M bit above. 326 V: Scalability structure (SS) data present. When set to one, the 327 OPTIONAL SS data MUST be present in the payload descriptor. 328 Otherwise, the SS data MUST NOT be present. 330 -: Bit reserved for future use. MUST be set to zero and MUST be 331 ignored by the receiver. 333 The mandatory first octet is followed by the extension data fields 334 that are enabled: 336 M: The most significant bit of the first octet is an extension flag. 337 The field MUST be present if the I bit is equal to one. If set, 338 the PID field MUST contain 15 bits; otherwise, it MUST contain 7 339 bits. See PID below. 341 Picture ID (PID): Picture ID represented in 7 or 15 bits, depending 342 on the M bit. This is a running index of the pictures. The field 343 MUST be present if the I bit is equal to one. If M is set to 344 zero, 7 bits carry the PID; else if M is set to one, 15 bits carry 345 the PID. The sender may choose between 7 or 15 bits index. The 346 PID SHOULD start on a random number, and MUST wrap after reaching 347 the maximum ID. The receiver MUST NOT assume that the number of 348 bits in PID stay the same through the session. 350 Layer indices: This information is optional but recommended whenever 351 encoding with layers. In the flexible mode (when the F bit is set 352 to 1), one octet is used to specify a layer frame's temporal layer 353 ID (T) and spatial layer ID (S) as shown in Figure 2. 354 Additionally, a bit (U) is used to indcate that the current frame 355 is a "switching up point" frame. Another bit (D) is used to 356 indicate whether inter-layer prediction is used for the current 357 layer frame. 359 In the non-flexible mode (when the F bit is set to 0), two octets 360 are used as depicted in Figure 3. Like the flexible mode, the 361 first byte contains the spatial layer ID and the D bit. Unlike 362 the flexible mode, instead of the T and U fields, a group of 363 frames index (GOF_IDX) is specified, which can be used to obtain 364 the values of T and U fields from the scalable structure (SS) data 365 described below. An additional octet to represent the temporal 366 layer 0 index, TL0PICIDX, is present so that all minimally 367 required frames can be tracked. 369 The T and S fields, whether obtained directly or indirectly from 370 the SS data, indicate the temporal and spatial layers and can help 371 MCUs measure bitrates per layer and can help them make a quick 372 decision on whether to relay a packet or not. They can also help 373 receivers determine what layers they are currently decoding. 375 T: The temporal layer ID of current frame. This field is only 376 present in the flexible mode (F = 1). 378 U: Switching up point. This bit is only present in the flexible 379 mode (F = 1). If this bit is set to 1 for the current frame 380 with temporal layer ID equal to T, then "switch up" to a higher 381 frame rate is possible as subsequent higher temporal layer 382 frames will not depend on any frame before the current frame 383 (in coding time) with temporal layer ID greater than T. 385 S: The spatial layer ID of current frame. Note that frames with 386 spatial layer S > 0 may be dependent on decoded spatial layer 387 S-1 frame within the same super frame. 389 D: Inter-layer dependency used. MUST be set to one if current 390 spatial layer S frame depends on spatial layer S-1 frame of the 391 same super frame. MUST only be set to zero if current spatial 392 layer S frame does not depend on spatial layer S-1 frame of the 393 same super frame. For the base layer frame with S equal to 0, 394 this D bit MUST be set to zero. 396 GOF_IDX: An index to a frame in the group of frames (GOF) 397 described by the SS data. This field is only present in the 398 non-flexible mode (F = 0). In this mode, the SS data SHOULD 399 have been received and the temporal characteristics of each 400 frame must have been speficied as group of frames in the SS 401 data (see the description of "Scalability structure" below). 402 Here, the values of the T and the U fields are derived from the 403 SS data. Additionally, the frame's inter-picture dependecy can 404 also be obtained from the SS data. In the case no SS data has 405 been received or the received SS data does not specify GOF (N_G 406 is set to 0), then GOF_IDX MUST be ignored and the stream is 407 assumed to have no temporal hierarchy with both T and U equal 408 to 0. 410 TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only 411 present in the non-flexible mode (F = 0). This is a running 412 index for the temporal base layer frames, i.e., the frames with 413 temporal layer ID (TID) set to 0. If TID is larger than 0, 414 TL0PICIDX indicates which temporal base layer frame the current 415 frame depends on. TL0PICIDX MUST be incremented when TID is 0. 416 The index SHOULD start on a random number, and MUST restart at 417 0 after reaching the maximum number 255. 419 Reference indices: These bytes are optional, but recommended when 420 encoding with temporal layers in the flexible mode. When P and F 421 are both set to one, then at least one reference index has to be 422 specified as below. Additional reference indices (total of up to 423 3 reference indices are allowed) may be specified using the N bit 424 below. When either P or F is set to zero, then no reference index 425 is specified. 427 P_DIFF: The reference index specified as the relative PID from 428 the current frame. For example, when P_DIFF=3 on a packet 429 containing the frame with PID 112 means that the frame refers 430 back to the frame with PID 109. This calculation is done 431 modulo the size of the PID field, i.e., either 7 or 15 bits. 432 For most layer structures a 6-bit relative PID will be enough; 433 however, the X bit can be used to refer to older frames. 435 X: 1 if this layer index has an extended P_DIFF. 437 N: 1 if there is additional P_DIFF following the current P_DIFF. 439 4.2.1. Scalability Structure (SS): 441 The scalability structure (SS) data describes the resolution of each 442 layer frame within a super frame as well as the inter-picture 443 dependencies for a group of frames (GOF). If the VP9 payload 444 descriptor's "V" bit is set, the SS data is present in the position 445 indicated in Figure 2 and Figure 3. 447 +-+-+-+-+-+-+-+-+ 448 V: | N_S |Y| N_G | 449 +-+-+-+-+-+-+-+-+ -\ 450 Y: | WIDTH | (OPTIONAL) . 451 + + . 452 | | (OPTIONAL) . 453 +-+-+-+-+-+-+-+-+ . - N_S + 1 times 454 | HEIGHT | (OPTIONAL) . 455 + + . 456 | | (OPTIONAL) . 457 +-+-+-+-+-+-+-+-+ -/ -\ 458 N_G: | T |U| R |-|-| (OPTIONAL) . 459 +-+-+-+-+-+-+-+-+ -\ . - N_G + 1 times 460 | P_DIFF | (OPTIONAL) . - R times . 461 +-+-+-+-+-+-+-+-+ -/ -/ 463 Figure 4 465 N_S: N_S + 1 indicates the number of spatial layers present in the 466 VP9 stream. 468 Y: Each spatial layer's frame resolution present. When set to one, 469 the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be 470 present for each layer frame. Otherwise, the resolution MUST NOT 471 be present. 473 N_G: N_G + 1 indicates the number of frames in a GOF. If N_G is 474 greater than 0, then the SS data allows the inter-picture 475 dependency structure of the VP9 stream to be pre-declared, rather 476 than indicating it on the fly with every packet. If N_G is 477 greater than 0, then for N_G + 1 pictures in the GOF, each frame's 478 temporal layer ID (T), switch up point (U), and the R reference 479 indices (P_DIFFs) are specified. 481 N_G=0 indicates that either there is only one temporal layer or no 482 fixed inter-picture dependency information is present going 483 forward in the bitstream. 485 Note that for a given super frame, all layer frames follow the 486 same inter-picture dependency structure. However, the frame rate 487 of each spatial layer can be different from each other and this 488 can be controlled with the use of the D bit described above. The 489 specified dependency structure in the SS data MUST be for the 490 highest frame rate layer. 492 In a scalable stream sent with a fixed pattern, the SS data SHOULD be 493 included in the first packet of every key frame. This is a packet 494 with P bit equal to zero, S or D bit equal to zero, B bit equal to 1, 495 and temporal layer ID (TID) equal to 0. The SS data SHOULD also be 496 included in the first packet of the first frame in which the SS 497 changes. If the SS data is included in a frame with TID not equal to 498 0, it MUST also be repeated in the first packet of the first frame 499 with a lower TID, until TID equals to 0. 501 4.3. VP9 Payload Header 503 TODO: need to describe VP9 payload header. 505 4.4. Frame Fragmentation 507 VP9 frames are fragmented into packets, in RTP sequence number order, 508 beginning with a packet with the B bit set, and ending with a packet 509 with the RTP marker bit set. There is no mechanism for finer-grained 510 access to parts of a VP9 frame. 512 4.5. Examples of VP9 RTP Stream 514 TODO 516 5. Using VP9 with RPSI and SLI Feedback 518 The VP9 payload descriptor defined in Section 4.2 above contains an 519 optional PictureID parameter. One use of this parameter is included 520 to enable use of reference picture selection index (RPSI) and slice 521 loss indication (SLI), both defined in [RFC4585]. 523 5.1. RPSI 525 TODO: Update to indicate which frame within the picture. 527 The reference picture selection index is a payload-specific feedback 528 message defined within the RTCP-based feedback format. The RPSI 529 message is generated by a receiver and can be used in two ways. 530 Either it can signal a preferred reference picture when a loss has 531 been detected by the decoder -- preferably then a reference that the 532 decoder knows is perfect -- or, it can be used as positive feedback 533 information to acknowledge correct decoding of certain reference 534 pictures. The positive feedback method is useful for VP9 used as 535 unicast. The use of RPSI for VP9 is preferably combined with a 536 special update pattern of the codec's two special reference frames -- 537 the golden frame and the altref frame -- in which they are updated in 538 an alternating leapfrog fashion. When a receiver has received and 539 correctly decoded a golden or altref frame, and that frame had a 540 PictureID in the payload descriptor, the receiver can acknowledge 541 this simply by sending an RPSI message back to the sender. The 542 message body (i.e., the "native RPSI bit string" in [RFC4585]) is 543 simply the PictureID of the received frame. 545 5.2. SLI 547 TODO: Update to indicate which frame within the picture. 549 The slice loss indication is another payload-specific feedback 550 message defined within the RTCP-based feedback format. The SLI 551 message is generated by the receiver when a loss or corruption is 552 detected in a frame. The format of the SLI message is as follows 553 [RFC4585]: 555 0 1 2 3 556 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 557 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 558 | First | Number | PictureID | 559 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 561 Figure 5 563 Here, First is the macroblock address (in scan order) of the first 564 lost block and Number is the number of lost blocks. PictureID is the 565 six least significant bits of the codec-specific picture identifier 566 in which the loss or corruption has occurred. For VP9, this codec- 567 specific identifier is naturally the PictureID of the current frame, 568 as read from the payload descriptor. If the payload descriptor of 569 the current frame does not have a PictureID, the receiver MAY send 570 the last received PictureID+1 in the SLI message. The receiver MAY 571 set the First parameter to 0, and the Number parameter to the total 572 number of macroblocks per frame, even though only parts of the frame 573 is corrupted. When the sender receives an SLI message, it can make 574 use of the knowledge from the latest received RPSI message. Knowing 575 that the last golden or altref frame was successfully received, it 576 can encode the next frame with reference to that established 577 reference. 579 5.3. Example 581 TODO: this example is copied from the VP8 payload format 582 specification, and has not been updated for VP9. It may be 583 incorrect. 585 The use of RPSI and SLI is best illustrated in an example. In this 586 example, the encoder may not update the altref frame until the last 587 sent golden frame has been acknowledged with an RPSI message. If an 588 update is not received within some time, a new golden frame update is 589 sent instead. Once the new golden frame is established and 590 acknowledged, the same rule applies when updating the altref frame. 592 +-------+-------------------+-------------------------+-------------+ 593 | Event | Sender | Receiver | Established | 594 | | | | reference | 595 +-------+-------------------+-------------------------+-------------+ 596 | 1000 | Send golden frame | | | 597 | | PictureID = 0 | | | 598 | | | | | 599 | | | Receive and decode | | 600 | | | golden frame | | 601 | | | | | 602 | 1001 | | Send RPSI(0) | | 603 | | | | | 604 | 1002 | Receive RPSI(0) | | golden | 605 | | | | | 606 | ... | (sending regular | | | 607 | | frames) | | | 608 | | | | | 609 | 1100 | Send altref frame | | | 610 | | PictureID = 100 | | | 611 | | | | | 612 | | | Altref corrupted or | golden | 613 | | | lost | | 614 | | | | | 615 | 1101 | | Send SLI(100) | golden | 616 | | | | | 617 | 1102 | Receive SLI(100) | | | 618 | | | | | 619 | 1103 | Send frame with | | | 620 | | reference to | | | 621 | | golden | | | 622 | | | | | 623 | | | Receive and decode | golden | 624 | | | frame (decoder state | | 625 | | | restored) | | 626 | | | | | 627 | ... | (sending regular | | | 628 | | frames) | | | 629 | | | | | 630 | 1200 | Send altref frame | | | 631 | | PictureID = 200 | | | 632 | | | | | 633 | | | Receive and decode | golden | 634 | | | altref frame | | 635 | | | | | 636 | 1201 | | Send RPSI(200) | | 637 | | | | | 638 | 1202 | Receive RPSI(200) | | altref | 639 | | | | | 640 | ... | (sending regular | | | 641 | | frames) | | | 642 | | | | | 643 | 1300 | Send golden frame | | | 644 | | PictureID = 300 | | | 645 | | | | | 646 | | | Receive and decode | altref | 647 | | | golden frame | | 648 | | | | | 649 | 1301 | | Send RPSI(300) | altref | 650 | | | | | 651 | 1302 | RPSI lost | | | 652 | | | | | 653 | 1400 | Send golden frame | | | 654 | | PictureID = 400 | | | 655 | | | | | 656 | | | Receive and decode | altref | 657 | | | golden frame | | 658 | | | | | 659 | 1401 | | Send RPSI(400) | | 660 | | | | | 661 | 1402 | Receive RPSI(400) | | golden | 662 +-------+-------------------+-------------------------+-------------+ 664 Table 1: Example signaling between sender and receiver 666 Note that the scheme is robust to loss of the feedback messages. If 667 the RPSI is lost, the sender will try to update the golden (or 668 altref) again after a while, without releasing the established 669 reference. Also, if an SLI is lost, the receiver can keep sending 670 SLI messages at any interval allowed by the RTCP sending timing 671 restrictions as specified in [RFC4585], as long as the picture is 672 corrupted. 674 6. Payload Format Parameters 676 This payload format has two required parameters. 678 6.1. Media Type Definition 680 This registration is done using the template defined in [RFC6838] and 681 following [RFC4855]. 683 Type name: video 684 Subtype name: VP9 686 Required parameters: 687 These parameters MUST be used to signal the capabilities of a 688 receiver implementation. These parameters MUST NOT be used for 689 any other purpose. 691 max-fr: The value of max-fr is an integer indicating the maximum 692 frame rate in units of frames per second that the decoder is 693 capable of decoding. 695 max-fs: The value of max-fs is an integer indicating the maximum 696 frame size in units of macroblocks that the decoder is capable 697 of decoding. 699 The decoder is capable of decoding this frame size as long as 700 the width and height of the frame in macroblocks are less than 701 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 702 of supporting 640x480 resolution) will support widths and 703 heights up to 1552 pixels (97 macroblocks). 705 Encoding considerations: 706 This media type is framed in RTP and contains binary data; see 707 Section 4.8 of [RFC6838]. 709 Security considerations: See Section 7 of RFC xxxx. 710 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 711 with the number assigned to this document and remove this note.] 713 Interoperability considerations: None. 715 Published specification: VP9 bitstream format 716 [I-D.grange-vp9-bitstream] and RFC XXXX. 717 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 718 with the number assigned to this document and remove this note.] 720 Applications which use this media type: 721 For example: Video over IP, video conferencing. 723 Fragment identifier considerations: N/A. 725 Additional information: None. 727 Person & email address to contact for further information: 728 TODO [Pick a contact] 730 Intended usage: COMMON 731 Restrictions on usage: 732 This media type depends on RTP framing, and hence is only defined 733 for transfer via RTP [RFC3550]. 735 Author: TODO [Pick a contact] 737 Change controller: 738 IETF Payload Working Group delegated from the IESG. 740 6.2. SDP Parameters 742 The receiver MUST ignore any fmtp parameter unspecified in this memo. 744 6.2.1. Mapping of Media Subtype Parameters to SDP 746 The media type video/VP9 string is mapped to fields in the Session 747 Description Protocol (SDP) [RFC4566] as follows: 749 o The media name in the "m=" line of SDP MUST be video. 751 o The encoding name in the "a=rtpmap" line of SDP MUST be VP9 (the 752 media subtype). 754 o The clock rate in the "a=rtpmap" line MUST be 90000. 756 o The parameters "max-fs", and "max-fr", MUST be included in the 757 "a=fmtp" line of SDP if SDP is used to declare receiver 758 capabilities. These parameters are expressed as a media subtype 759 string, in the form of a semicolon separated list of 760 parameter=value pairs. 762 6.2.1.1. Example 764 An example of media representation in SDP is as follows: 766 m=video 49170 RTP/AVPF 98 767 a=rtpmap:98 VP9/90000 768 a=fmtp:98 max-fr=30; max-fs=3600; 770 6.2.2. Offer/Answer Considerations 772 TODO: Update this for VP9 774 7. Security Considerations 776 RTP packets using the payload format defined in this specification 777 are subject to the security considerations discussed in the RTP 778 specification [RFC3550], and in any applicable RTP profile. The main 779 security considerations for the RTP packet carrying the RTP payload 780 format defined within this memo are confidentiality, integrity and 781 source authenticity. Confidentiality is achieved by encryption of 782 the RTP payload. Integrity of the RTP packets through suitable 783 cryptographic integrity protection mechanism. Cryptographic system 784 may also allow the authentication of the source of the payload. A 785 suitable security mechanism for this RTP payload format should 786 provide confidentiality, integrity protection and at least source 787 authentication capable of determining if an RTP packet is from a 788 member of the RTP session or not. Note that the appropriate 789 mechanism to provide security to RTP and payloads following this memo 790 may vary. It is dependent on the application, the transport, and the 791 signaling protocol employed. Therefore a single mechanism is not 792 sufficient, although if suitable the usage of SRTP [RFC3711] is 793 recommended. This RTP payload format and its media decoder do not 794 exhibit any significant non-uniformity in the receiver-side 795 computational complexity for packet processing, and thus are unlikely 796 to pose a denial-of-service threat due to the receipt of pathological 797 data. Nor does the RTP payload format contain any active content. 799 8. Congestion Control 801 Congestion control for RTP SHALL be used in accordance with RFC 3550 802 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 803 [RFC3551]. The congestion control mechanism can, in a real-time 804 encoding scenario, adapt the transmission rate by instructing the 805 encoder to encode at a certain target rate. Media aware network 806 elements MAY use the information in the VP9 payload descriptor in 807 Section 4.2 to identify non-reference frames and discard them in 808 order to reduce network congestion. Note that discarding of non- 809 reference frames cannot be done if the stream is encrypted (because 810 the non-reference marker is encrypted). 812 9. IANA Considerations 814 The IANA is requested to register the following values: 815 - Media type registration as described in Section 6.1. 817 10. References 819 10.1. Normative References 821 [I-D.grange-vp9-bitstream] 822 Grange, A. and H. Alvestrand, "A VP9 Bitstream Overview", 823 draft-grange-vp9-bitstream-00 (work in progress), February 824 2013. 826 [I-D.lennox-avtext-lrr] 827 Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. 828 Flodman, "The Layer Refresh Request (LRR) RTCP Feedback 829 Message", draft-lennox-avtext-lrr-00 (work in progress), 830 March 2015. 832 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 833 Requirement Levels", BCP 14, RFC 2119, March 1997. 835 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 836 Jacobson, "RTP: A Transport Protocol for Real-Time 837 Applications", STD 64, RFC 3550, July 2003. 839 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 840 Description Protocol", RFC 4566, July 2006. 842 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 843 "Extended RTP Profile for Real-time Transport Control 844 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 845 2006. 847 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 848 Formats", RFC 4855, February 2007. 850 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 851 Specifications and Registration Procedures", BCP 13, RFC 852 6838, January 2013. 854 10.2. Informative References 856 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 857 Video Conferences with Minimal Control", STD 65, RFC 3551, 858 July 2003. 860 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 861 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 862 RFC 3711, March 2004. 864 Authors' Addresses 866 Justin Uberti 867 Google, Inc. 868 747 6th Street South 869 Kirkland, WA 98033 870 USA 872 Email: justin@uberti.name 873 Stefan Holmer 874 Google, Inc. 875 Kungsbron 2 876 Stockholm 111 22 877 Sweden 879 Email: holmer@google.com 881 Magnus Flodman 882 Google, Inc. 883 Kungsbron 2 884 Stockholm 111 22 885 Sweden 887 Email: mflodman@google.com 889 Jonathan Lennox 890 Vidyo, Inc. 891 433 Hackensack Avenue 892 Seventh Floor 893 Hackensack, NJ 07601 894 US 896 Email: jonathan@vidyo.com 898 Danny Hong 899 Vidyo, Inc. 900 433 Hackensack Avenue 901 Seventh Floor 902 Hackensack, NJ 07601 903 US 905 Email: danny@vidyo.com