idnits 2.17.1 draft-ietf-payload-vp8-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 5, 2015) is 3245 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) ** Downref: Normative reference to an Informational RFC: RFC 6386 Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group P. Westin 3 Internet-Draft H. Lundin 4 Intended status: Standards Track M. Glover 5 Expires: December 7, 2015 J. Uberti 6 F. Galligan 7 Google 8 June 5, 2015 10 RTP Payload Format for VP8 Video 11 draft-ietf-payload-vp8-16 13 Abstract 15 This memo describes an RTP payload format for the VP8 video codec. 16 The payload format has wide applicability, as it supports 17 applications from low bit-rate peer-to-peer usage, to high bit-rate 18 video conferences. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on December 7, 2015. 37 Copyright Notice 39 Copyright (c) 2015 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 56 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 57 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 58 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 4 59 4.2. VP8 Payload Descriptor . . . . . . . . . . . . . . . . . 6 60 4.3. VP8 Payload Header . . . . . . . . . . . . . . . . . . . 10 61 4.4. Aggregated and Fragmented Payloads . . . . . . . . . . . 11 62 4.5. Frame reconstruction algorithm . . . . . . . . . . . . . 11 63 4.5.1. Partition reconstruction algorithm . . . . . . . . . 12 64 4.6. Examples of VP8 RTP Stream . . . . . . . . . . . . . . . 12 65 4.6.1. Key frame in a single RTP packet . . . . . . . . . . 12 66 4.6.2. Non-discardable VP8 interframe in a single RTP 67 packet; no PictureID . . . . . . . . . . . . . . . . 13 68 4.6.3. VP8 partitions in separate RTP packets . . . . . . . 14 69 4.6.4. VP8 frame fragmented across RTP packets . . . . . . . 15 70 4.6.5. VP8 frame with long PictureID . . . . . . . . . . . . 16 71 5. Using VP8 with RPSI and SLI Feedback . . . . . . . . . . . . 17 72 5.1. RPSI . . . . . . . . . . . . . . . . . . . . . . . . . . 17 73 5.2. SLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 74 5.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 20 76 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 20 77 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 22 78 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 22 79 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 22 80 7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 81 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 23 82 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 83 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 84 10.1. Normative References . . . . . . . . . . . . . . . . . . 24 85 10.2. Informative References . . . . . . . . . . . . . . . . . 24 86 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 88 1. Introduction 90 This memo describes an RTP payload specification applicable to the 91 transmission of video streams encoded using the VP8 video codec 92 [RFC6386]. The format described in this document can be used both in 93 peer-to-peer and video conferencing applications. 95 VP8 is based on decomposition of frames into square sub-blocks of 96 pixels, prediction of such sub-blocks using previously constructed 97 blocks, and adjustment of such predictions (as well as synthesis of 98 unpredicted blocks) using a discrete cosine transform (hereafter 99 abbreviated as DCT). In one special case, however, VP8 uses a 100 "Walsh-Hadamard" (hereafter abbreviated as WHT) transform instead of 101 a DCT. An encoded VP8 frame is divided into two or more partitions, 102 as described in [RFC6386]. The first partition (prediction or mode) 103 contains prediction mode parameters and motion vectors for all 104 macroblocks. The remaining partitions all contain the quantized DCT/ 105 WHT coefficients for the residuals. There can be 1, 2, 4, or 8 DCT/ 106 WHT partitions per frame, depending on encoder settings. 108 In summary, the payload format described in this document enables a 109 number of features in VP8, including: 111 o Taking partition boundaries into consideration, to improve loss 112 robustness and facilitate efficient packet loss concealment at the 113 decoder. 115 o Temporal scalability. 117 o Advanced use of reference frames to enable efficient error 118 recovery. 120 o Marking of frames that have no impact on the decoding of any other 121 frame, so that these non-reference frames can be discarded in a 122 server or media-aware network element if needed. 124 2. Conventions, Definitions and Acronyms 126 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 127 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 128 document are to be interpreted as described in [RFC2119]. 130 3. Media Format Description 132 The VP8 codec uses three different reference frames for interframe 133 prediction: the previous frame, the golden frame, and the altref 134 frame. Blocks in an interframe may be predicted using blocks in the 135 immediately previous frame as well as the most recent golden frame or 136 altref frame. Every key frame is automatically golden and altref, 137 and any interframe may optionally replace the most recent golden or 138 altref frame. Golden frames and altref frames may also be used to 139 increase the tolerance to dropped frames. The payload specification 140 in this memo has elements that enable advanced use of the reference 141 frames, e.g., for improved loss robustness. 143 One specific use case of the three reference frame types is temporal 144 scalability. By setting up the reference hierarchy in the 145 appropriate way, up to five temporal layers can be encoded. (How to 146 set up the reference hierarchy for temporal scalability is not within 147 the scope of this memo.) 149 Another property of the VP8 codec is that it applies data 150 partitioning to the encoded data. Thus, an encoded VP8 frame can be 151 divided into two or more partitions, as described in "VP8 Data Format 152 and Decoding Guide" [RFC6386]. The first partition (prediction or 153 mode) contains prediction mode parameters and motion vectors for all 154 macroblocks. The remaining partitions all contain the transform 155 coefficients for the residuals. The first partition is decodable 156 without the remaining residual partitions. The subsequent partitions 157 may be useful even if some part of the frame is lost. This memo 158 allows the partitions to be sent separately or in the same RTP 159 packet. It may be beneficial for decoder error-concealment to send 160 the partitions in different packets, even though it is not mandatory 161 according to this specification. 163 The format specification is described in Section 4. In Section 5, a 164 method to acknowledge receipt of reference frames using RTCP 165 techniques is described. 167 The payload partitioning and the acknowledging method both serve as 168 motivation for three of the fields included in the payload format: 169 the "PID", "1st partition size" and "PictureID" fields. The ability 170 to encode a temporally scalable stream motivates the "TL0PICIDX" and 171 "TID" fields. 173 4. Payload Format 175 This section describes how the encoded VP8 bitstream is encapsulated 176 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 177 RECOMMENDED. All integer fields in the specifications are encoded as 178 unsigned integers in network octet order. 180 4.1. RTP Header Usage 181 The general RTP payload format for VP8 is depicted below. 183 0 1 2 3 184 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 186 |V=2|P|X| CC |M| PT | sequence number | 187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 188 | timestamp | 189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 190 | synchronization source (SSRC) identifier | 191 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 192 | contributing source (CSRC) identifiers | 193 | .... | 194 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 195 | VP8 payload descriptor (integer #bytes) | 196 : : 197 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 198 | : VP8 payload header (3 octets) | 199 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 200 | VP8 pyld hdr : | 201 +-+-+-+-+-+-+-+-+ | 202 : Bytes 4..N of VP8 payload : 203 | | 204 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 205 | : OPTIONAL RTP padding | 206 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 208 The VP8 payload descriptor and VP8 payload header will be described 209 in the sequel. OPTIONAL RTP padding MUST NOT be included unless the 210 P bit is set. 212 Figure 1 214 Marker bit (M): MUST be set for the very last packet of each encoded 215 frame in line with the normal use of the M bit in video formats. 216 This enables a decoder to finish decoding the picture, where it 217 otherwise may need to wait for the next packet to explicitly know 218 that the frame is complete. 220 Timestamp: The RTP timestamp indicates the time when the frame was 221 sampled at a clock rate of 90 kHz. 223 Sequence number: The sequence numbers are monotonically increasing 224 and set as packets are sent. 226 The remaining RTP header fields are used as specified in 227 [RFC3550]. 229 4.2. VP8 Payload Descriptor 231 The first octets after the RTP header are the VP8 payload descriptor, 232 with the following structure. The single-octet version of the 233 PictureID is illustrated to the left (M bit set to zero), while the 234 dual-octet version (M bit set to one) is show to the right. 236 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 237 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 238 |X|R|N|S|R| PID | (REQUIRED) |X|R|N|S|R| PID | (REQUIRED) 239 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 240 X: |I|L|T|K| RSV | (OPTIONAL) X: |I|L|T|K| RSV | (OPTIONAL) 241 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 242 I: |M| PictureID | (OPTIONAL) I: |M| PictureID | (OPTIONAL) 243 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 244 L: | TL0PICIDX | (OPTIONAL) | PictureID | 245 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 246 T/K: |TID|Y| KEYIDX | (OPTIONAL) L: | TL0PICIDX | (OPTIONAL) 247 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 248 T/K: |TID|Y| KEYIDX | (OPTIONAL) 249 +-+-+-+-+-+-+-+-+ 251 Figure 2 253 X: Extended control bits present. When set to one, the extension 254 octet MUST be provided immediately after the mandatory first 255 octet. If the bit is zero, all optional fields MUST be omitted. 257 R: Bit reserved for future use. MUST be set to zero and MUST be 258 ignored by the receiver. 260 N: Non-reference frame. When set to one, the frame can be discarded 261 without affecting any other future or past frames. If the 262 reference status of the frame is unknown, this bit SHOULD be set 263 to zero to avoid discarding frames needed for reference. 265 Informative note: This document does not describe how to 266 determine if an encoded frame is non-reference. The reference 267 status of an encoded frame is preferably provided from the 268 encoder implementation. 270 S: Start of VP8 partition. SHOULD be set to 1 when the first payload 271 octet of the RTP packet is the beginning of a new VP8 partition, 272 and MUST NOT be 1 otherwise. The S bit MUST be set to 1 for the 273 first packet of each encoded frame. 275 PID: Partition index. Denotes which VP8 partition the first payload 276 octet of the packet belongs to. The first VP8 partition 277 (containing modes and motion vectors) MUST be labeled with PID = 278 0. PID SHOULD be incremented for each subsequent partition, but 279 MAY be kept at 0 for all packets. PID MUST NOT be larger than 7. 280 If more than one packet in an encoded frame contains the same PID, 281 the S bit MUST NOT be set for any other packet than the first 282 packet with that PID. 284 When the X bit is set to 1 in the first octet, the extension bit 285 field octet MUST be provided as the second octet. If the X bit is 0, 286 the extension bit field octet MUST NOT be present, and no extensions 287 (I, L, T, or K) are permitted. 289 I: PictureID present. When set to one, the PictureID MUST be present 290 after the extension bit field and specified as below. Otherwise, 291 PictureID MUST NOT be present. 293 L: TL0PICIDX present. When set to one, the TL0PICIDX MUST be present 294 and specified as below, and the T bit MUST be set to 1. 295 Otherwise, TL0PICIDX MUST NOT be present. 297 T: TID present. When set to one, the TID/Y/KEYIDX octet MUST be 298 present. The TID|Y part of the octet MUST be specified as below. 299 If K (below) is set to one but T is set to zero, the TID/Y/KEYIDX 300 octet MUST be present, but the TID field MUST be ignored. If 301 neither T nor K is set to one, the TID/Y/KEYIDX octet MUST NOT be 302 present. 304 K: KEYIDX present. When set to one, the TID/Y/KEYIDX octet MUST be 305 present. The KEYIDX part of the octet MUST be specified as below. 306 If T (above) is set to one but K is set to zero, the TID/Y/KEYIDX 307 octet MUST be present, but the KEYIDX field MUST be ignored. If 308 neither T nor K is set to one, the TID/Y/KEYIDX octet MUST NOT be 309 present. 311 RSV: Bits reserved for future use. MUST be set to zero and MUST be 312 ignored by the receiver. 314 After the extension bit field follow the extension data fields that 315 are enabled. 317 The PictureID extension: If the I bit is set to one, the PictureID 318 extension field MUST be present, and MUST NOT be present 319 otherwise. The field consists of two parts: 321 M: The most significant bit of the first octet is an extension 322 flag. If M is set, the remainder of the PictureID field MUST 323 contain 15 bits, else it MUST contain 7 bits. 325 PictureID: 7 or 15 bits (shown left and right, respectively, in 326 Figure 2) not including the M bit. This is a running index of 327 the frames, which SHOULD begin as a random number, MUST 328 increase by 1 for each subsequent frame, and MUST wrap to 0 329 after reaching the maximum ID (all bits set). The 7 or 15 bits 330 of the PictureID go from most significant to least significant, 331 beginning with the first bit after the M bit. The sender 332 chooses a 7 or 15 bit index and sets the M bit accordingly. 333 The receiver MUST NOT assume that the number of bits in 334 PictureID stay the same through the session. Having sent a 335 7-bit PictureID with all bits set to 1, the sender may either 336 wrap the PictureID to 0, or extend to 15 bits and continue 337 incrementing. 339 The TL0PICIDX extension: If the L bit is set to one, the TL0PICIDX 340 extension field MUST be present, and MUST NOT be present 341 otherwise. The field consists of one part: 343 TL0PICIDX: 8 bits temporal level zero index. TL0PICIDX is a 344 running index for the temporal base layer frames, i.e., the 345 frames with TID set to 0. If TID is larger than 0, TL0PICIDX 346 indicates which base layer frame the current image depends on. 347 TL0PICIDX MUST be incremented when TID is 0. The index SHOULD 348 start on a random number, and MUST restart at 0 after reaching 349 the maximum number 255. 351 The TID/Y/KEYIDX extension: If the any of the T or K bits are set to 352 one, the TID/Y/KEYIDX extension field MUST be present. It MUST 353 NOT be present if both T and K are zero. The field consists of 354 three parts: 356 TID: 2 bits temporal layer index. The TID field MUST be ignored 357 by the receiver when the T bit is set equal to 0. The TID 358 field indicates which temporal layer the packet represents. 359 The lowest layer, i.e., the base layer, MUST have TID set to 0. 360 Higher layers SHOULD increment the TID according to their 361 position in the layer hierarchy. 363 Y: 1 layer sync bit. The Y bit SHOULD be set to 1 if the current 364 frame depends only on the base layer (TID = 0) frame with 365 TL0PICIDX equal to that of the current frame. The Y bit MUST 366 be set to 0 if the current frame depends on any other frame 367 than the base layer (TID = 0) frame with TL0PICIDX equal to 368 that of the current frame. If the Y bit is set when the T bit 369 is equal to 0 the current frame MUST only depend on a past base 370 layer (TID=0) key frame as signaled by a change in the KEYIDX 371 field. Additionally this frame MUST NOT depend on any of the 372 three codec buffers (as defined by [RFC6386]) that have been 373 updated since the last time the KEYIDX field was changed. 375 Informative note: This document does not describe how to 376 determine the dependency status for a frame; this information 377 is preferably provided from the encoder implementation. In the 378 case of unknown status, the Y bit can safely be set to 0. 380 KEYIDX: 5 bits temporal key frame index. The KEYIDX field MUST 381 be ignored by the receiver when the K bit is set equal to 0. 382 The KEYIDX field is a running index for key frames. KEYIDX MAY 383 start on a random number, and MUST restart at 0 after reaching 384 the maximum number 31. When in use, the KEYIDX SHOULD be 385 present for both key frames and interframes. The sender MUST 386 increment KEYIDX for key frames which convey parameter updates 387 critical to the interpretation of subsequent frames, and SHOULD 388 leave the KEYIDX unchanged for key frames that do not contain 389 these critical updates. If the KEYIDX is present, a receiver 390 SHOULD NOT decode an interframe if it has not received and 391 decoded a key frame with the same KEYIDX after the last KEYIDX 392 wrap-around. 394 Informative note: This document does not describe how to 395 determine if a key frame updates critical parameters; this 396 information is preferably provided from the encoder 397 implementation. A sender that does not have this information 398 may either omit the KEYIDX field (set K equal to 0), or 399 increment the KEYIDX on every key frame. The benefit with the 400 latter is that any key frame loss will be detected by the 401 receiver, which can signal for re-transmission or request a new 402 key frame. 404 Informative note: Implementations doing splicing of VP8 streams will 405 have to make sure the rules for incrementing TL0PICIDX and KEYIDX 406 are obeyed across the splice. This will likely require rewriting 407 values of TL0PICIDX and KEYIDX after the splice. 409 4.3. VP8 Payload Header 411 The beginning of an encoded VP8 frame is referred to as an 412 "uncompressed data chunk" in [RFC6386], and also serves as a payload 413 header in this RTP format. The codec bitstream format specifies two 414 different variants of the uncompressed data chunk: a 3 octet version 415 for interframes and a 10 octet version for key frames. The first 3 416 octets are common to both variants. In the case of a key frame the 417 remaining 7 octets are considered to be part of the remaining payload 418 in this RTP format. Note that the header is present only in packets 419 which have the S bit equal to one and the PID equal to zero in the 420 payload descriptor. Subsequent packets for the same frame do not 421 carry the payload header. 423 0 1 2 3 4 5 6 7 424 +-+-+-+-+-+-+-+-+ 425 |Size0|H| VER |P| 426 +-+-+-+-+-+-+-+-+ 427 | Size1 | 428 +-+-+-+-+-+-+-+-+ 429 | Size2 | 430 +-+-+-+-+-+-+-+-+ 431 | Bytes 4..N of | 432 | VP8 payload | 433 : : 434 +-+-+-+-+-+-+-+-+ 435 | OPTIONAL RTP | 436 | padding | 437 : : 438 +-+-+-+-+-+-+-+-+ 440 Figure 3 442 H: Show frame bit as defined in [RFC6386]. 444 VER: A version number as defined in [RFC6386]. 446 P: Inverse key frame flag. When set to 0 the current frame is a key 447 frame. When set to 1 the current frame is an interframe. Defined 448 in [RFC6386] 450 SizeN: The size of the first partition in bytes is calculated from 451 the 19 bits in Size0, Size1, and Size2 as 1stPartitionSize = Size0 452 + 8 * Size1 + 2048 * Size2. [RFC6386]. 454 4.4. Aggregated and Fragmented Payloads 456 An encoded VP8 frame can be divided into two or more partitions, as 457 described in Section 1. One packet can contain a fragment of a 458 partition, a complete partition, or an aggregate of fragments and 459 partitions. In the preferred use case, the S bit and PID fields 460 described in Section 4.2 should be used to indicate what the packet 461 contains. The PID field should indicate which partition the first 462 octet of the payload belongs to, and the S bit indicates that the 463 packet starts on a new partition. Aggregation of encoded partitions 464 is done without explicit signaling. Partitions MUST be aggregated in 465 decoding order. Two fragments from different partitions MAY be 466 aggregated into the same packet. An aggregation MUST have exactly 467 one payload descriptor. Aggregated partitions MUST represent parts 468 of one and the same video frame. Consequently, an aggregated packet 469 will have one or no payload header, depending on whether the 470 aggregate contains the beginning of the first partition of a frame or 471 not, respectively. Note that the length of the first partition can 472 always be obtained from the first partition size parameter in the VP8 473 payload header. 475 The VP8 bitstream format [RFC6386] specifies that if multiple DCT/WHT 476 partitions are produced, the location of each partition start is 477 found at the end of the first (prediction/mode) partition. In this 478 RTP payload specification, the location offsets are considered to be 479 part of the first partition. 481 It is OPTIONAL for a packetizer implementing this RTP specification 482 to pay attention to the partition boundaries within an encoded frame. 483 If packetization of a frame is done without considering the partition 484 boundaries, the PID field MAY be set to zero for all packets, and the 485 S bit MUST NOT be set to one for any other packet than the first. 487 4.5. Frame reconstruction algorithm 489 Example of frame reconstruction algorithm. 491 1: Collect all packets with a given RTP timestamp. 493 2: Go through packets in order, sorted by sequence numbers, if 494 packets are missing, send NACK as defined in [RFC4585] or decode 495 with missing partitions, see Section 4.5.1 below. 497 3: A frame is complete if the frame has no missing sequence numbers, 498 the first packet in the frame contains S=1 with partId=0 and the 499 last packet in the frame has the marker bit set. 501 4.5.1. Partition reconstruction algorithm 503 Example of partition reconstruction algorithm. 505 1: Scan for the start of a new partition; S=1. 507 2: Continue scan to detect end of partition; hence a new S=1 508 (previous packet was the end of the partition) is found or the 509 marker bit is set. If a loss is detected before the end of the 510 partition, abandon all packets in this partition and continue the 511 scan repeating from step 1. 513 3: Store the packets in the complete partition, continue the scan 514 repeating from step 1 until end of frame is reached. 516 4: Send all complete partitions to the decoder. If no complete 517 partition is found discard the whole frame. 519 4.6. Examples of VP8 RTP Stream 521 A few examples of how the VP8 RTP payload can be used are included 522 below. 524 4.6.1. Key frame in a single RTP packet 526 0 1 2 3 4 5 6 7 527 +-+-+-+-+-+-+-+-+ 528 | RTP header | 529 | M = 1 | 530 +-+-+-+-+-+-+-+-+ 531 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 532 +-+-+-+-+-+-+-+-+ 533 |1|0|0|0|0 0 0 0| I = 1 534 +-+-+-+-+-+-+-+-+ 535 |0 0 0 1 0 0 0 1| PictureID = 17 536 +-+-+-+-+-+-+-+-+ 537 |Size0|1| VER |0| P = 0 538 +-+-+-+-+-+-+-+-+ 539 | Size1 | 540 +-+-+-+-+-+-+-+-+ 541 | Size2 | 542 +-+-+-+-+-+-+-+-+ 543 | VP8 payload | 544 +-+-+-+-+-+-+-+-+ 546 4.6.2. Non-discardable VP8 interframe in a single RTP packet; no 547 PictureID 549 0 1 2 3 4 5 6 7 550 +-+-+-+-+-+-+-+-+ 551 | RTP header | 552 | M = 1 | 553 +-+-+-+-+-+-+-+-+ 554 |0|0|0|1|0|0 0 0| X = 0; S = 1; PID = 0 555 +-+-+-+-+-+-+-+-+ 556 |Size0|1| VER |1| P = 1 557 +-+-+-+-+-+-+-+-+ 558 | Size1 | 559 +-+-+-+-+-+-+-+-+ 560 | Size2 | 561 +-+-+-+-+-+-+-+-+ 562 | VP8 payload | 563 +-+-+-+-+-+-+-+-+ 565 4.6.3. VP8 partitions in separate RTP packets 567 First RTP packet; complete first partition. 569 0 1 2 3 4 5 6 7 570 +-+-+-+-+-+-+-+-+ 571 | RTP header | 572 | M = 0 | 573 +-+-+-+-+-+-+-+-+ 574 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 575 +-+-+-+-+-+-+-+-+ 576 |1|0|0|0|0 0 0 0| I = 1 577 +-+-+-+-+-+-+-+-+ 578 |0 0 0 1 0 0 0 1| PictureID = 17 579 +-+-+-+-+-+-+-+-+ 580 |Size0|1| VER |1| P = 1 581 +-+-+-+-+-+-+-+-+ 582 | Size1 | 583 +-+-+-+-+-+-+-+-+ 584 | Size2 | 585 +-+-+-+-+-+-+-+-+ 586 | Bytes 4..L of | 587 | first VP8 | 588 | partition | 589 : : 590 +-+-+-+-+-+-+-+-+ 592 Second RTP packet; complete second partition. 594 0 1 2 3 4 5 6 7 595 +-+-+-+-+-+-+-+-+ 596 | RTP header | 597 | M = 1 | 598 +-+-+-+-+-+-+-+-+ 599 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 600 +-+-+-+-+-+-+-+-+ 601 |1|0|0|0|0 0 0 0| I = 1 602 +-+-+-+-+-+-+-+-+ 603 |0 0 0 1 0 0 0 1| PictureID = 17 604 +-+-+-+-+-+-+-+-+ 605 | Remaining VP8 | 606 | partitions | 607 : : 608 +-+-+-+-+-+-+-+-+ 610 4.6.4. VP8 frame fragmented across RTP packets 612 First RTP packet; complete first partition. 614 0 1 2 3 4 5 6 7 615 +-+-+-+-+-+-+-+-+ 616 | RTP header | 617 | M = 0 | 618 +-+-+-+-+-+-+-+-+ 619 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 620 +-+-+-+-+-+-+-+-+ 621 |1|0|0|0|0 0 0 0| I = 1 622 +-+-+-+-+-+-+-+-+ 623 |0 0 0 1 0 0 0 1| PictureID = 17 624 +-+-+-+-+-+-+-+-+ 625 |Size0|1| VER |1| P = 1 626 +-+-+-+-+-+-+-+-+ 627 | Size1 | 628 +-+-+-+-+-+-+-+-+ 629 | Size2 | 630 +-+-+-+-+-+-+-+-+ 631 | Complete | 632 | first | 633 | partition | 634 : : 635 +-+-+-+-+-+-+-+-+ 637 Second RTP packet; first fragment of second partition. 639 0 1 2 3 4 5 6 7 640 +-+-+-+-+-+-+-+-+ 641 | RTP header | 642 | M = 0 | 643 +-+-+-+-+-+-+-+-+ 644 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 645 +-+-+-+-+-+-+-+-+ 646 |1|0|0|0|0 0 0 0| I = 1 647 +-+-+-+-+-+-+-+-+ 648 |0 0 0 1 0 0 0 1| PictureID = 17 649 +-+-+-+-+-+-+-+-+ 650 | First fragment| 651 | of second | 652 | partition | 653 : : 654 +-+-+-+-+-+-+-+-+ 656 Third RTP packet; second fragment of second partition. 658 0 1 2 3 4 5 6 7 659 +-+-+-+-+-+-+-+-+ 660 | RTP header | 661 | M = 0 | 662 +-+-+-+-+-+-+-+-+ 663 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 664 +-+-+-+-+-+-+-+-+ 665 |1|0|0|0|0 0 0 0| I = 1 666 +-+-+-+-+-+-+-+-+ 667 |0 0 0 1 0 0 0 1| PictureID = 17 668 +-+-+-+-+-+-+-+-+ 669 | Mid fragment | 670 | of second | 671 | partition | 672 : : 673 +-+-+-+-+-+-+-+-+ 675 Fourth RTP packet; last fragment of second partition. 677 0 1 2 3 4 5 6 7 678 +-+-+-+-+-+-+-+-+ 679 | RTP header | 680 | M = 1 | 681 +-+-+-+-+-+-+-+-+ 682 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 683 +-+-+-+-+-+-+-+-+ 684 |1|0|0|0|0 0 0 0| I = 1 685 +-+-+-+-+-+-+-+-+ 686 |0 0 0 1 0 0 0 1| PictureID = 17 687 +-+-+-+-+-+-+-+-+ 688 | Last fragment | 689 | of second | 690 | partition | 691 : : 692 +-+-+-+-+-+-+-+-+ 694 4.6.5. VP8 frame with long PictureID 695 PictureID = 4711 = 001001001100111 binary (first 7 bits: 0010010, 696 last 8 bits: 01100111). 698 0 1 2 3 4 5 6 7 699 +-+-+-+-+-+-+-+-+ 700 | RTP header | 701 | M = 1 | 702 +-+-+-+-+-+-+-+-+ 703 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 704 +-+-+-+-+-+-+-+-+ 705 |1|0|0|0|0 0 0 0| I = 1; 706 +-+-+-+-+-+-+-+-+ 707 |1 0 0 1 0 0 1 0| Long PictureID flag = 1 708 |0 1 1 0 0 1 1 1| PictureID = 4711 709 +-+-+-+-+-+-+-+-+ 710 |Size0|1| VER |1| 711 +-+-+-+-+-+-+-+-+ 712 | Size1 | 713 +-+-+-+-+-+-+-+-+ 714 | Size2 | 715 +-+-+-+-+-+-+-+-+ 716 | Bytes 4..N of | 717 | VP8 payload | 718 : : 719 +-+-+-+-+-+-+-+-+ 721 5. Using VP8 with RPSI and SLI Feedback 723 The VP8 payload descriptor defined in Section 4.2 above contains an 724 optional PictureID parameter. This parameter is included mainly to 725 enable use of reference picture selection index (RPSI) and slice loss 726 indication (SLI), both defined in [RFC4585]. 728 5.1. RPSI 730 The reference picture selection index is a payload-specific feedback 731 message defined within the RTCP-based feedback format. The RPSI 732 message is generated by a receiver and can be used in two ways. 733 Either it can signal a preferred reference picture when a loss has 734 been detected by the decoder -- preferably then a reference that the 735 decoder knows is perfect -- or, it can be used as positive feedback 736 information to acknowledge correct decoding of certain reference 737 pictures. The positive feedback method is useful for VP8 used as 738 unicast. The use of RPSI for VP8 is preferably combined with a 739 special update pattern of the codec's two special reference frames -- 740 the golden frame and the altref frame -- in which they are updated in 741 an alternating leapfrog fashion. When a receiver has received and 742 correctly decoded a golden or altref frame, and that frame had a 743 PictureID in the payload descriptor, the receiver can acknowledge 744 this simply by sending an RPSI message back to the sender. The 745 message body (i.e., the "native RPSI bit string" in [RFC4585]) is 746 simply the PictureID of the received frame. 748 5.2. SLI 750 The slice loss indication is another payload-specific feedback 751 message defined within the RTCP-based feedback format. The SLI 752 message is generated by the receiver when a loss or corruption is 753 detected in a frame. The format of the SLI message is as follows 754 [RFC4585]: 756 0 1 2 3 757 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 758 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 759 | First | Number | PictureID | 760 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 762 Figure 4 764 Here, First is the macroblock address (in scan order) of the first 765 lost block and Number is the number of lost blocks. PictureID is the 766 six least significant bits of the codec-specific picture identifier 767 in which the loss or corruption has occurred. For VP8, this codec- 768 specific identifier is naturally the PictureID of the current frame, 769 as read from the payload descriptor. If the payload descriptor of 770 the current frame does not have a PictureID, the receiver MAY send 771 the last received PictureID+1 in the SLI message. The receiver MAY 772 set the First parameter to 0, and the Number parameter to the total 773 number of macroblocks per frame, even though only parts of the frame 774 is corrupted. When the sender receives an SLI message, it can make 775 use of the knowledge from the latest received RPSI message. Knowing 776 that the last golden or altref frame was successfully received, it 777 can encode the next frame with reference to that established 778 reference. 780 5.3. Example 782 The use of RPSI and SLI is best illustrated in an example. In this 783 example, the encoder may not update the altref frame until the last 784 sent golden frame has been acknowledged with an RPSI message. If an 785 update is not received within some time, a new golden frame update is 786 sent instead. Once the new golden frame is established and 787 acknowledged, the same rule applies when updating the altref frame. 789 +-------+-------------------+-------------------------+-------------+ 790 | Event | Sender | Receiver | Established | 791 | | | | reference | 792 +-------+-------------------+-------------------------+-------------+ 793 | 1000 | Send golden frame | | | 794 | | PictureID = 0 | | | 795 | | | | | 796 | | | Receive and decode | | 797 | | | golden frame | | 798 | | | | | 799 | 1001 | | Send RPSI(0) | | 800 | | | | | 801 | 1002 | Receive RPSI(0) | | golden | 802 | | | | | 803 | ... | (sending regular | | | 804 | | frames) | | | 805 | | | | | 806 | 1100 | Send altref frame | | | 807 | | PictureID = 100 | | | 808 | | | | | 809 | | | Altref corrupted or | golden | 810 | | | lost | | 811 | | | | | 812 | 1101 | | Send SLI(100) | golden | 813 | | | | | 814 | 1102 | Receive SLI(100) | | | 815 | | | | | 816 | 1103 | Send frame with | | | 817 | | reference to | | | 818 | | golden | | | 819 | | | | | 820 | | | Receive and decode | golden | 821 | | | frame (decoder state | | 822 | | | restored) | | 823 | | | | | 824 | ... | (sending regular | | | 825 | | frames) | | | 826 | | | | | 827 | 1200 | Send altref frame | | | 828 | | PictureID = 200 | | | 829 | | | | | 830 | | | Receive and decode | golden | 831 | | | altref frame | | 832 | | | | | 833 | 1201 | | Send RPSI(200) | | 834 | | | | | 835 | 1202 | Receive RPSI(200) | | altref | 836 | | | | | 837 | ... | (sending regular | | | 838 | | frames) | | | 839 | | | | | 840 | 1300 | Send golden frame | | | 841 | | PictureID = 300 | | | 842 | | | | | 843 | | | Receive and decode | altref | 844 | | | golden frame | | 845 | | | | | 846 | 1301 | | Send RPSI(300) | altref | 847 | | | | | 848 | 1302 | RPSI lost | | | 849 | | | | | 850 | 1400 | Send golden frame | | | 851 | | PictureID = 400 | | | 852 | | | | | 853 | | | Receive and decode | altref | 854 | | | golden frame | | 855 | | | | | 856 | 1401 | | Send RPSI(400) | | 857 | | | | | 858 | 1402 | Receive RPSI(400) | | golden | 859 +-------+-------------------+-------------------------+-------------+ 861 Table 1: Exemple signaling between sender and receiver 863 Note that the scheme is robust to loss of the feedback messages. If 864 the RPSI is lost, the sender will try to update the golden (or 865 altref) again after a while, without releasing the established 866 reference. Also, if an SLI is lost, the receiver can keep sending 867 SLI messages at any interval allowed by the RTCP sending timing 868 restrictions as specified in [RFC4585], as long as the picture is 869 corrupted. 871 6. Payload Format Parameters 873 This payload format has two required parameters. 875 6.1. Media Type Definition 877 This registration is done using the template defined in [RFC6838] and 878 following [RFC4855]. 880 Type name: video 882 Subtype name: VP8 884 Required parameters: None. 886 Optional parameters: 888 These parameters are used to signal the capabilities of a receiver 889 implementation. If the implementation is willing to receive 890 media, both parameters MUST be provided. These parameters MUST 891 NOT be used for any other purpose. 893 max-fr: The value of max-fr is an integer indicating the maximum 894 frame rate in units of frames per second that the decoder is 895 capable of decoding. 897 max-fs: The value of max-fs is an integer indicating the maximum 898 frame size in units of macroblocks that the decoder is capable 899 of decoding. 901 The decoder is capable of decoding this frame size as long as 902 the width and height of the frame in macroblocks are less than 903 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 904 of supporting 640x480 resolution) will support widths and 905 heights up to 1552 pixels (97 macroblocks). 907 Encoding considerations: 908 This media type is framed in RTP and contains binary data; see 909 Section 4.8 of [RFC6838]. 911 Security considerations: See Section 7 of RFC xxxx. 912 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 913 with the number assigned to this document and remove this note.] 915 Interoperability considerations: None. 917 Published specification: VP8 bitstream format [RFC6386] and RFC 918 XXXX. 919 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 920 with the number assigned to this document and remove this note.] 922 Applications which use this media type: 923 For example: Video over IP, video conferencing. 925 Fragment identifier considerations: N/A. 927 Additional information: None. 929 Person & email address to contact for further information: 930 Patrik Westin, patrik.westin@gmail.com 932 Intended usage: COMMON 934 Restrictions on usage: 936 This media type depends on RTP framing, and hence is only defined 937 for transfer via RTP [RFC3550]. 939 Author: Patrik Westin, patrik.westin@gmail.com 941 Change controller: 942 IETF Payload Working Group delegated from the IESG. 944 6.2. SDP Parameters 946 The receiver MUST ignore any fmtp parameter unspecified in this memo. 948 6.2.1. Mapping of Media Subtype Parameters to SDP 950 The media type video/VP8 string is mapped to fields in the Session 951 Description Protocol (SDP) [RFC4566] as follows: 953 o The media name in the "m=" line of SDP MUST be video. 955 o The encoding name in the "a=rtpmap" line of SDP MUST be VP8 (the 956 media subtype). 958 o The clock rate in the "a=rtpmap" line MUST be 90000. 960 o The parameters "max-fs", and "max-fr", MUST be included in the 961 "a=fmtp" line if the SDP is used to declare receiver capabilities. 962 These parameters are expressed as a media subtype string, in the 963 form of a semicolon separated list of parameter=value pairs. 965 6.2.1.1. Example 967 An example of media representation in SDP is as follows: 969 m=video 49170 RTP/AVPF 98 970 a=rtpmap:98 VP8/90000 971 a=fmtp:98 max-fr=30; max-fs=3600; 973 6.2.2. Offer/Answer Considerations 975 The VP8 codec offers a decode complexity that is roughly linear with 976 the number of pixels encoded. The parameters "max-fr" and "max-fs" 977 are defined in Section 6.1, where the macroblock size is 16x16 pixels 978 as defined in [RFC6386], the max-fs and max-fr parameters MUST be 979 used to establish these limits. 981 7. Security Considerations 983 RTP packets using the payload format defined in this specification 984 are subject to the security considerations discussed in the RTP 985 specification [RFC3550], and in any applicable RTP profile. The main 986 security considerations for the RTP packet carrying the RTP payload 987 format defined within this memo are confidentiality, integrity and 988 source authenticity. Confidentiality is achieved by encryption of 989 the RTP payload. Integrity of the RTP packets through suitable 990 cryptographic integrity protection mechanism. Cryptographic system 991 may also allow the authentication of the source of the payload. A 992 suitable security mechanism for this RTP payload format should 993 provide confidentiality, integrity protection and at least source 994 authentication capable of determining if an RTP packet is from a 995 member of the RTP session or not. Note that the appropriate 996 mechanism to provide security to RTP and payloads following this memo 997 may vary. It is dependent on the application, the transport, and the 998 signaling protocol employed. Therefore a single mechanism is not 999 sufficient, although if suitable the usage of SRTP [RFC3711] is 1000 recommended. This RTP payload format and its media decoder do not 1001 exhibit any significant non-uniformity in the receiver-side 1002 computational complexity for packet processing, and thus are unlikely 1003 to pose a denial-of-service threat due to the receipt of pathological 1004 data. Nor does the RTP payload format contain any active content. 1006 8. Congestion Control 1008 Congestion control for RTP SHALL be used in accordance with RFC 3550 1009 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 1010 [RFC3551]. The congestion control mechanism can, in a real-time 1011 encoding scenario, adapt the transmission rate by instructing the 1012 encoder to encode at a certain target rate. Media aware network 1013 elements MAY use the information in the VP8 payload descriptor in 1014 Section 4.2 to identify non-reference frames and discard them in 1015 order to reduce network congestion. Note that discarding of non- 1016 reference frames cannot be done if the stream is encrypted (because 1017 the non-reference marker is encrypted). 1019 9. IANA Considerations 1021 The IANA is requested to register the following values: 1022 - Media type registration as described in Section 6.1. 1024 10. References 1025 10.1. Normative References 1027 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1028 Requirement Levels", BCP 14, RFC 2119, March 1997. 1030 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1031 Jacobson, "RTP: A Transport Protocol for Real-Time 1032 Applications", STD 64, RFC 3550, July 2003. 1034 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 1035 Description Protocol", RFC 4566, July 2006. 1037 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1038 "Extended RTP Profile for Real-time Transport Control 1039 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 1040 2006. 1042 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 1043 Formats", RFC 4855, February 2007. 1045 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 1046 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 1047 Guide", RFC 6386, November 2011. 1049 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 1050 Specifications and Registration Procedures", BCP 13, RFC 1051 6838, January 2013. 1053 10.2. Informative References 1055 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1056 Video Conferences with Minimal Control", STD 65, RFC 3551, 1057 July 2003. 1059 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1060 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1061 RFC 3711, March 2004. 1063 Authors' Addresses 1065 Patrik Westin 1066 Google, Inc. 1067 1600 Amphitheatre Parkway 1068 Mountain View, CA 94043 1069 USA 1071 Email: patrik.westin@gmail.com 1072 Henrik F Lundin 1073 Google, Inc. 1074 Kungsbron 2 1075 Stockholm 11122 1076 Sweden 1078 Email: hlundin@google.com 1080 Michael Glover 1081 Google, Inc. 1082 5 Cambridge Center 1083 Cambridge, MA 02142 1084 USA 1086 Justin Uberti 1087 Google, Inc. 1088 747 6th Street South 1089 Kirkland, WA 98033 1090 USA 1092 Frank Galligan 1093 Google, Inc. 1094 1600 Amphitheatre Parkway 1095 Mountain View, CA 94043 1096 USA