idnits 2.17.1 draft-ietf-payload-vp8-15.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 23, 2015) is 3293 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) ** Downref: Normative reference to an Informational RFC: RFC 6386 Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group P. Westin 3 Internet-Draft H. Lundin 4 Intended status: Standards Track M. Glover 5 Expires: September 24, 2015 J. Uberti 6 F. Galligan 7 Google 8 March 23, 2015 10 RTP Payload Format for VP8 Video 11 draft-ietf-payload-vp8-15 13 Abstract 15 This memo describes an RTP payload format for the VP8 video codec. 16 The payload format has wide applicability, as it supports 17 applications from low bit-rate peer-to-peer usage, to high bit-rate 18 video conferences. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on September 24, 2015. 37 Copyright Notice 39 Copyright (c) 2015 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 56 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 57 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 58 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 4 59 4.2. VP8 Payload Descriptor . . . . . . . . . . . . . . . . . 6 60 4.3. VP8 Payload Header . . . . . . . . . . . . . . . . . . . 10 61 4.4. Aggregated and Fragmented Payloads . . . . . . . . . . . 11 62 4.5. Frame reconstruction algorithm . . . . . . . . . . . . . 11 63 4.5.1. Partition reconstruction algorithm . . . . . . . . . 12 64 4.6. Examples of VP8 RTP Stream . . . . . . . . . . . . . . . 12 65 4.6.1. Key frame in a single RTP packet . . . . . . . . . . 12 66 4.6.2. Non-discardable VP8 interframe in a single RTP 67 packet; no PictureID . . . . . . . . . . . . . . . . 13 68 4.6.3. VP8 partitions in separate RTP packets . . . . . . . 14 69 4.6.4. VP8 frame fragmented across RTP packets . . . . . . . 15 70 4.6.5. VP8 frame with long PictureID . . . . . . . . . . . . 16 71 5. Using VP8 with RPSI and SLI Feedback . . . . . . . . . . . . 17 72 5.1. RPSI . . . . . . . . . . . . . . . . . . . . . . . . . . 17 73 5.2. SLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 74 5.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 20 76 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 20 77 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 22 78 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 22 79 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 22 80 7. Security Considerations . . . . . . . . . . . . . . . . . . . 22 81 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 23 82 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 83 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 84 10.1. Normative References . . . . . . . . . . . . . . . . . . 23 85 10.2. Informative References . . . . . . . . . . . . . . . . . 24 86 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 88 1. Introduction 90 This memo describes an RTP payload specification applicable to the 91 transmission of video streams encoded using the VP8 video codec 92 [RFC6386]. The format described in this document can be used both in 93 peer-to-peer and video conferencing applications. 95 VP8 is based on decomposition of frames into square sub-blocks of 96 pixels, prediction of such sub-blocks using previously constructed 97 blocks, and adjustment of such predictions (as well as synthesis of 98 unpredicted blocks) using a discrete cosine transform (hereafter 99 abbreviated as DCT). In one special case, however, VP8 uses a 100 "Walsh-Hadamard" (hereafter abbreviated as WHT) transform instead of 101 a DCT. An encoded VP8 frame is divided into two or more partitions, 102 as described in [RFC6386]. The first partition (prediction or mode) 103 contains prediction mode parameters and motion vectors for all 104 macroblocks. The remaining partitions all contain the quantized DCT/ 105 WHT coefficients for the residuals. There can be 1, 2, 4, or 8 DCT/ 106 WHT partitions per frame, depending on encoder settings. 108 In summary, the payload format described in this document enables a 109 number of features in VP8, including: 111 o Taking partition boundaries into consideration, to improve loss 112 robustness and facilitate efficient packet loss concealment at the 113 decoder. 115 o Temporal scalability. 117 o Advanced use of reference frames to enable efficient error 118 recovery. 120 o Marking of frames that have no impact on the decoding of any other 121 frame, so that these non-reference frames can be discarded in a 122 server or media-aware network element if needed. 124 2. Conventions, Definitions and Acronyms 126 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 127 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 128 document are to be interpreted as described in [RFC2119]. 130 3. Media Format Description 132 The VP8 codec uses three different reference frames for interframe 133 prediction: the previous frame, the golden frame, and the altref 134 frame. Blocks in an interframe may be predicted using blocks in the 135 immediately previous frame as well as the most recent golden frame or 136 altref frame. Every key frame is automatically golden and altref, 137 and any interframe may optionally replace the most recent golden or 138 altref frame. Golden frames and altref frames may also be used to 139 increase the tolerance to dropped frames. The payload specification 140 in this memo has elements that enable advanced use of the reference 141 frames, e.g., for improved loss robustness. 143 One specific use case of the three reference frame types is temporal 144 scalability. By setting up the reference hierarchy in the 145 appropriate way, up to five temporal layers can be encoded. (How to 146 set up the reference hierarchy for temporal scalability is not within 147 the scope of this memo.) 149 Another property of the VP8 codec is that it applies data 150 partitioning to the encoded data. Thus, an encoded VP8 frame can be 151 divided into two or more partitions, as described in "VP8 Data Format 152 and Decoding Guide" [RFC6386]. The first partition (prediction or 153 mode) contains prediction mode parameters and motion vectors for all 154 macroblocks. The remaining partitions all contain the transform 155 coefficients for the residuals. The first partition is decodable 156 without the remaining residual partitions. The subsequent partitions 157 may be useful even if some part of the frame is lost. This memo 158 allows the partitions to be sent separately or in the same RTP 159 packet. It may be beneficial for decoder error-concealment to send 160 the partitions in different packets, even though it is not mandatory 161 according to this specification. 163 The format specification is described in Section 4. In Section 5, a 164 method to acknowledge receipt of reference frames using RTCP 165 techniques is described. 167 The payload partitioning and the acknowledging method both serve as 168 motivation for three of the fields included in the payload format: 169 the "PID", "1st partition size" and "PictureID" fields. The ability 170 to encode a temporally scalable stream motivates the "TL0PICIDX" and 171 "TID" fields. 173 4. Payload Format 175 This section describes how the encoded VP8 bitstream is encapsulated 176 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 177 RECOMMENDED. All integer fields in the specifications are encoded as 178 unsigned integers in network octet order. 180 4.1. RTP Header Usage 181 The general RTP payload format for VP8 is depicted below. 183 0 1 2 3 184 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 186 |V=2|P|X| CC |M| PT | sequence number | 187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 188 | timestamp | 189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 190 | synchronization source (SSRC) identifier | 191 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 192 | contributing source (CSRC) identifiers | 193 | .... | 194 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 195 | VP8 payload descriptor (integer #bytes) | 196 : : 197 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 198 | : VP8 payload header (3 octets) | 199 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 200 | VP8 pyld hdr : | 201 +-+-+-+-+-+-+-+-+ | 202 : Bytes 4..N of VP8 payload : 203 | | 204 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 205 | : OPTIONAL RTP padding | 206 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 208 The VP8 payload descriptor and VP8 payload header will be described 209 in the sequel. OPTIONAL RTP padding MUST NOT be included unless the 210 P bit is set. 212 Figure 1 214 Marker bit (M): MUST be set for the very last packet of each encoded 215 frame in line with the normal use of the M bit in video formats. 216 This enables a decoder to finish decoding the picture, where it 217 otherwise may need to wait for the next packet to explicitly know 218 that the frame is complete. 220 Timestamp: The RTP timestamp indicates the time when the frame was 221 sampled at a clock rate of 90 kHz. 223 Sequence number: The sequence numbers are monotonically increasing 224 and set as packets are sent. 226 The remaining RTP header fields are used as specified in 227 [RFC3550]. 229 4.2. VP8 Payload Descriptor 231 The first octets after the RTP header are the VP8 payload descriptor, 232 with the following structure. The single-octet version of the 233 PictureID is illustrated to the left (M bit set to zero), while the 234 dual-octet version (M bit set to one) is show to the right. 236 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 237 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 238 |X|R|N|S|R| PID | (REQUIRED) |X|R|N|S|R| PID | (REQUIRED) 239 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 240 X: |I|L|T|K| RSV | (OPTIONAL) X: |I|L|T|K| RSV | (OPTIONAL) 241 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 242 I: |M| PictureID | (OPTIONAL) I: |M| PictureID | (OPTIONAL) 243 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 244 L: | TL0PICIDX | (OPTIONAL) | PictureID | 245 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 246 T/K: |TID|Y| KEYIDX | (OPTIONAL) L: | TL0PICIDX | (OPTIONAL) 247 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 248 T/K: |TID|Y| KEYIDX | (OPTIONAL) 249 +-+-+-+-+-+-+-+-+ 251 Figure 2 253 X: Extended control bits present. When set to one, the extension 254 octet MUST be provided immediately after the mandatory first 255 octet. If the bit is zero, all optional fields MUST be omitted. 257 R: Bit reserved for future use. MUST be set to zero and MUST be 258 ignored by the receiver. 260 N: Non-reference frame. When set to one, the frame can be discarded 261 without affecting any other future or past frames. If the 262 reference status of the frame is unknown, this bit SHOULD be set 263 to zero to avoid discarding frames needed for reference. 265 Informative note: This document does not describe how to 266 determine if an encoded frame is non-reference. The reference 267 status of an encoded frame is preferably provided from the 268 encoder implementation. 270 S: Start of VP8 partition. SHOULD be set to 1 when the first payload 271 octet of the RTP packet is the beginning of a new VP8 partition, 272 and MUST NOT be 1 otherwise. The S bit MUST be set to 1 for the 273 first packet of each encoded frame. 275 PID: Partition index. Denotes which VP8 partition the first payload 276 octet of the packet belongs to. The first VP8 partition 277 (containing modes and motion vectors) MUST be labeled with PID = 278 0. PID SHOULD be incremented for each subsequent partition, but 279 MAY be kept at 0 for all packets. PID MUST NOT be larger than 7. 280 If more than one packet in an encoded frame contains the same PID, 281 the S bit MUST NOT be set for any other packet than the first 282 packet with that PID. 284 When the X bit is set to 1 in the first octet, the extension bit 285 field octet MUST be provided as the second octet. If the X bit is 0, 286 the extension bit field octet MUST NOT be present, and all bits below 287 MUST be implicitly interpreted as 0. 289 I: PictureID present. When set to one, the OPTIONAL PictureID MUST 290 be present after the extension bit field and specified as below. 291 Otherwise, PictureID MUST NOT be present. 293 L: TL0PICIDX present. When set to one, the OPTIONAL TL0PICIDX MUST 294 be present and specified as below, and the T bit MUST be set to 1. 295 Otherwise, TL0PICIDX MUST NOT be present. 297 T: TID present. When set to one, the OPTIONAL TID/KEYIDX octet MUST 298 be present. The TID|Y part of the octet MUST be specified as 299 below. If K (below) is set to one but T is set to zero, the TID/ 300 KEYIDX octet MUST be present, but the TID|Y field MUST be ignored. 301 If neither T nor K is set to one, the TID/KEYIDX octet MUST NOT be 302 present. 304 K: KEYIDX present. When set to one, the OPTIONAL TID/KEYIDX octet 305 MUST be present. The KEYIDX part of the octet MUST be specified 306 as below. If T (above) is set to one but K is set to zero, the 307 TID/KEYIDX octet MUST be present, but the KEYIDX field MUST be 308 ignored. If neither T nor K is set to one, the TID/KEYIDX octet 309 MUST NOT be present. 311 RSV: Bits reserved for future use. MUST be set to zero and MUST be 312 ignored by the receiver. 314 After the extension bit field follow the extension data fields that 315 are enabled. 317 M: The most significant bit of the first octet is an extension flag. 318 The field MUST be present if the I bit is equal to one. If set 319 the PictureID field MUST contain 16 bits else it MUST contain 8 320 bits including this MSB, see PictureID. 322 PictureID: 8 or 16 bits (shown left and right, respectively, in 323 Figure 2) including the M bit. This is a running index of the 324 frames. The field MUST be present if the I bit is equal to one. 326 The 7 following bits carry (parts of) the PictureID. If the 327 extension flag is one, the PictureID continues in the next octet 328 forming a 15 bit index, where the 8 bits in the second octet are 329 the least significant bits of the PictureID. If the extension 330 flag is zero, there is no extension, and the PictureID is the 7 331 remaining bits of the first (and only) octet. The sender may 332 choose 7 or 15 bits index. The PictureID SHOULD start on a random 333 number, and MUST wrap after reaching the maximum ID. The receiver 334 MUST NOT assume that the number of bits in PictureID stay the same 335 through the session. 337 TL0PICIDX: 8 bits temporal level zero index. The field MUST be 338 present if the L bit is equal to 1, and MUST NOT be present 339 otherwise. TL0PICIDX is a running index for the temporal base 340 layer frames, i.e., the frames with TID set to 0. If TID is 341 larger than 0, TL0PICIDX indicates which base layer frame the 342 current image depends on. TL0PICIDX MUST be incremented when TID 343 is 0. The index SHOULD start on a random number, and MUST restart 344 at 0 after reaching the maximum number 255. 346 TID: 2 bits temporal layer index. The TID/KEYIDX octet MUST be 347 present when either the T bit or the K bit or both are equal to 1, 348 and MUST NOT be present otherwise. The TID field MUST be ignored 349 by the receiver when the T bit is set equal to 0. The TID field 350 indicates which temporal layer the packet represents. The lowest 351 layer, i.e., the base layer, MUST have TID set to 0. Higher 352 layers SHOULD increment the TID according to their position in the 353 layer hierarchy. 355 Y: 1 layer sync bit. The TID/KEYIDX octet MUST be present when 356 either the T bit or the K bit or both are equal to 1, and MUST NOT 357 be present otherwise. The Y bit SHOULD be set to 1 if the current 358 frame depends only on the base layer (TID = 0) frame with 359 TL0PICIDX equal to that of the current frame. The Y bit MUST be 360 set to 0 if the current frame depends any other frame than the 361 base layer (TID = 0) frame with TL0PICIDX equal to that of the 362 current frame. If the Y bit is set when the T bit is equal to 0 363 the current frame MUST only depend on a past base layer (TID=0) 364 key frame as signaled by a change in the KEYIDX field. 365 Additionally this frame MUST NOT depend on any of the three codec 366 buffers (as defined by [RFC6386]) that have been updated since the 367 last time the KEYIDX field was changed. 369 Informative note: This document does not describe how to 370 determine the dependence status for a frame; this information 371 is preferably provided from the encoder implementation. In the 372 case of unknown status, the Y bit can safely be set to 0. 374 KEYIDX: 5 bits temporal key frame index. The TID/KEYIDX octet MUST 375 be present when either the T bit or the K bit or both are equal to 376 1, and MUST NOT be present otherwise. The KEYIDX field MUST be 377 ignored by the receiver when the K bit is set equal to 0. The 378 KEYIDX field is a running index for key frames. KEYIDX MAY start 379 on a random number, and MUST restart at 0 after reaching the 380 maximum number 31. When in use, the KEYIDX SHOULD be present for 381 both key frames and interframes. The sender MUST increment KEYIDX 382 for key frames which convey parameter updates critical to the 383 interpretation of subsequent frames, and SHOULD leave the KEYIDX 384 unchanged for key frames that do not contain these critical 385 updates. If the KEYIDX is present, a receiver SHOULD NOT decode 386 an interframe if it has not received and decoded a key frame with 387 the same KEYIDX after the last KEYIDX wrap-around. 389 Informative note: This document does not describe how to 390 determine if a key frame updates critical parameters; this 391 information is preferably provided from the encoder 392 implementation. A sender that does not have this information 393 may either omit the KEYIDX field (set K equal to 0), or 394 increment the KEYIDX on every key frame. The benefit with the 395 latter is that any key frame loss will be detected by the 396 receiver, which can signal for re-transmission or request a new 397 key frame. 399 Informative note: Implementations doing splicing of VP8 streams will 400 have to make sure the rules for incrementing TL0PICIDX and KEYIDX 401 are obeyed across the splice. This will likely require rewriting 402 values of TL0PICIDX and KEYIDX after the splice. 404 4.3. VP8 Payload Header 406 The beginning of an encoded VP8 frame is referred to as an 407 "uncompressed data chunk" in [RFC6386], and co-serve as payload 408 header in this RTP format. The codec bitstream format specifies two 409 different variants of the uncompressed data chunk: a 3 octet version 410 for interframes and a 10 octet version for key frames. The first 3 411 octets are common to both variants. In the case of a key frame the 412 remaining 7 octets are considered to be part of the remaining payload 413 in this RTP format. Note that the header is present only in packets 414 which have the S bit equal to one and the PID equal to zero in the 415 payload descriptor. Subsequent packets for the same frame do not 416 carry the payload header. 418 0 1 2 3 4 5 6 7 419 +-+-+-+-+-+-+-+-+ 420 |Size0|H| VER |P| 421 +-+-+-+-+-+-+-+-+ 422 | Size1 | 423 +-+-+-+-+-+-+-+-+ 424 | Size2 | 425 +-+-+-+-+-+-+-+-+ 426 | Bytes 4..N of | 427 | VP8 payload | 428 : : 429 +-+-+-+-+-+-+-+-+ 430 | OPTIONAL RTP | 431 | padding | 432 : : 433 +-+-+-+-+-+-+-+-+ 435 Figure 3 437 H: Show frame bit as defined in [RFC6386]. 439 VER: A version number as defined in [RFC6386]. 441 P: Inverse key frame flag. When set to 0 the current frame is a key 442 frame. When set to 1 the current frame is an interframe. Defined 443 in [RFC6386] 445 SizeN: The size of the first partition in bytes is calculated from 446 the 19 bits in Size0, Size1, and Size2 as 1stPartitionSize = Size0 447 + 8 * Size1 + 2048 * Size2. [RFC6386]. 449 4.4. Aggregated and Fragmented Payloads 451 An encoded VP8 frame can be divided into two or more partitions, as 452 described in Section 1. One packet can contain a fragment of a 453 partition, a complete partition, or an aggregate of fragments and 454 partitions. In the preferred use case, the S bit and PID fields 455 described in Section 4.2 should be used to indicate what the packet 456 contains. The PID field should indicate which partition the first 457 octet of the payload belongs to, and the S bit indicates that the 458 packet starts on a new partition. Aggregation of encoded partitions 459 is done without explicit signaling. Partitions MUST be aggregated in 460 decoding order. Two fragments from different partitions MAY be 461 aggregated into the same packet. An aggregation MUST have exactly 462 one payload descriptor. Aggregated partitions MUST represent parts 463 of one and the same video frame. Consequently, an aggregated packet 464 will have one or no payload header, depending on whether the 465 aggregate contains the beginning of the first partition of a frame or 466 not, respectively. Note that the length of the first partition can 467 always be obtained from the first partition size parameter in the VP8 468 payload header. 470 The VP8 bitstream format [RFC6386] specifies that if multiple DCT/WHT 471 partitions are produced, the location of each partition start is 472 found at the end of the first (prediction/mode) partition. In this 473 RTP payload specification, the location offsets are considered to be 474 part of the first partition. 476 It is OPTIONAL for a packetizer implementing this RTP specification 477 to pay attention to the partition boundaries within an encoded frame. 478 If packetization of a frame is done without considering the partition 479 boundaries, the PID field MAY be set to zero for all packets, and the 480 S bit MUST NOT be set to one for any other packet than the first. 482 4.5. Frame reconstruction algorithm 484 Example of frame reconstruction algorithm. 486 1: Collect all packets with a given RTP timestamp. 488 2: Go through packets in order, sorted by sequence numbers, if 489 packets are missing, send NACK as defined in [RFC4585] or decode 490 with missing partitions, see Section 4.5.1 below. 492 3: A frame is complete if the frame has no missing sequence numbers, 493 the first packet in the frame contains S=1 with partId=0 and the 494 last packet in the frame has the marker bit set. 496 4.5.1. Partition reconstruction algorithm 498 Example of partition reconstruction algorithm. 500 1: Scan for the start of a new partition; S=1. 502 2: Continue scan to detect end of partition; hence a new S=1 503 (previous packet was the end of the partition) is found or the 504 marker bit is set. If a loss is detected before the end of the 505 partition, abandon all packets in this partition and continue the 506 scan repeating from step 1. 508 3: Store the packets in the complete partition, continue the scan 509 repeating from step 1 until end of frame is reached. 511 4: Send all complete partitions to the decoder. If no complete 512 partition is found discard the whole frame. 514 4.6. Examples of VP8 RTP Stream 516 A few examples of how the VP8 RTP payload can be used are included 517 below. 519 4.6.1. Key frame in a single RTP packet 521 0 1 2 3 4 5 6 7 522 +-+-+-+-+-+-+-+-+ 523 | RTP header | 524 | M = 1 | 525 +-+-+-+-+-+-+-+-+ 526 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 527 +-+-+-+-+-+-+-+-+ 528 |1|0|0|0|0 0 0 0| I = 1 529 +-+-+-+-+-+-+-+-+ 530 |0 0 0 1 0 0 0 1| PictureID = 17 531 +-+-+-+-+-+-+-+-+ 532 |Size0|1| VER |0| P = 0 533 +-+-+-+-+-+-+-+-+ 534 | Size1 | 535 +-+-+-+-+-+-+-+-+ 536 | Size2 | 537 +-+-+-+-+-+-+-+-+ 538 | VP8 payload | 539 +-+-+-+-+-+-+-+-+ 541 4.6.2. Non-discardable VP8 interframe in a single RTP packet; no 542 PictureID 544 0 1 2 3 4 5 6 7 545 +-+-+-+-+-+-+-+-+ 546 | RTP header | 547 | M = 1 | 548 +-+-+-+-+-+-+-+-+ 549 |0|0|0|1|0|0 0 0| X = 0; S = 1; PID = 0 550 +-+-+-+-+-+-+-+-+ 551 |Size0|1| VER |1| P = 1 552 +-+-+-+-+-+-+-+-+ 553 | Size1 | 554 +-+-+-+-+-+-+-+-+ 555 | Size2 | 556 +-+-+-+-+-+-+-+-+ 557 | VP8 payload | 558 +-+-+-+-+-+-+-+-+ 560 4.6.3. VP8 partitions in separate RTP packets 562 First RTP packet; complete first partition. 564 0 1 2 3 4 5 6 7 565 +-+-+-+-+-+-+-+-+ 566 | RTP header | 567 | M = 0 | 568 +-+-+-+-+-+-+-+-+ 569 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 570 +-+-+-+-+-+-+-+-+ 571 |1|0|0|0|0 0 0 0| I = 1 572 +-+-+-+-+-+-+-+-+ 573 |0 0 0 1 0 0 0 1| PictureID = 17 574 +-+-+-+-+-+-+-+-+ 575 |Size0|1| VER |1| P = 1 576 +-+-+-+-+-+-+-+-+ 577 | Size1 | 578 +-+-+-+-+-+-+-+-+ 579 | Size2 | 580 +-+-+-+-+-+-+-+-+ 581 | Bytes 4..L of | 582 | first VP8 | 583 | partition | 584 : : 585 +-+-+-+-+-+-+-+-+ 587 Second RTP packet; complete second partition. 589 0 1 2 3 4 5 6 7 590 +-+-+-+-+-+-+-+-+ 591 | RTP header | 592 | M = 1 | 593 +-+-+-+-+-+-+-+-+ 594 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 595 +-+-+-+-+-+-+-+-+ 596 |1|0|0|0|0 0 0 0| I = 1 597 +-+-+-+-+-+-+-+-+ 598 |0 0 0 1 0 0 0 1| PictureID = 17 599 +-+-+-+-+-+-+-+-+ 600 | Remaining VP8 | 601 | partitions | 602 : : 603 +-+-+-+-+-+-+-+-+ 605 4.6.4. VP8 frame fragmented across RTP packets 607 First RTP packet; complete first partition. 609 0 1 2 3 4 5 6 7 610 +-+-+-+-+-+-+-+-+ 611 | RTP header | 612 | M = 0 | 613 +-+-+-+-+-+-+-+-+ 614 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 615 +-+-+-+-+-+-+-+-+ 616 |1|0|0|0|0 0 0 0| I = 1 617 +-+-+-+-+-+-+-+-+ 618 |0 0 0 1 0 0 0 1| PictureID = 17 619 +-+-+-+-+-+-+-+-+ 620 |Size0|1| VER |1| P = 1 621 +-+-+-+-+-+-+-+-+ 622 | Size1 | 623 +-+-+-+-+-+-+-+-+ 624 | Size2 | 625 +-+-+-+-+-+-+-+-+ 626 | Complete | 627 | first | 628 | partition | 629 : : 630 +-+-+-+-+-+-+-+-+ 632 Second RTP packet; first fragment of second partition. 634 0 1 2 3 4 5 6 7 635 +-+-+-+-+-+-+-+-+ 636 | RTP header | 637 | M = 0 | 638 +-+-+-+-+-+-+-+-+ 639 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 640 +-+-+-+-+-+-+-+-+ 641 |1|0|0|0|0 0 0 0| I = 1 642 +-+-+-+-+-+-+-+-+ 643 |0 0 0 1 0 0 0 1| PictureID = 17 644 +-+-+-+-+-+-+-+-+ 645 | First fragment| 646 | of second | 647 | partition | 648 : : 649 +-+-+-+-+-+-+-+-+ 651 Third RTP packet; second fragment of second partition. 653 0 1 2 3 4 5 6 7 654 +-+-+-+-+-+-+-+-+ 655 | RTP header | 656 | M = 0 | 657 +-+-+-+-+-+-+-+-+ 658 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 659 +-+-+-+-+-+-+-+-+ 660 |1|0|0|0|0 0 0 0| I = 1 661 +-+-+-+-+-+-+-+-+ 662 |0 0 0 1 0 0 0 1| PictureID = 17 663 +-+-+-+-+-+-+-+-+ 664 | Mid fragment | 665 | of second | 666 | partition | 667 : : 668 +-+-+-+-+-+-+-+-+ 670 Fourth RTP packet; last fragment of second partition. 672 0 1 2 3 4 5 6 7 673 +-+-+-+-+-+-+-+-+ 674 | RTP header | 675 | M = 1 | 676 +-+-+-+-+-+-+-+-+ 677 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 678 +-+-+-+-+-+-+-+-+ 679 |1|0|0|0|0 0 0 0| I = 1 680 +-+-+-+-+-+-+-+-+ 681 |0 0 0 1 0 0 0 1| PictureID = 17 682 +-+-+-+-+-+-+-+-+ 683 | Last fragment | 684 | of second | 685 | partition | 686 : : 687 +-+-+-+-+-+-+-+-+ 689 4.6.5. VP8 frame with long PictureID 690 PictureID = 4711 = 001001001100111 binary (first 7 bits: 0010010, 691 last 8 bits: 01100111). 693 0 1 2 3 4 5 6 7 694 +-+-+-+-+-+-+-+-+ 695 | RTP header | 696 | M = 1 | 697 +-+-+-+-+-+-+-+-+ 698 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 699 +-+-+-+-+-+-+-+-+ 700 |1|0|0|0|0 0 0 0| I = 1; 701 +-+-+-+-+-+-+-+-+ 702 |1 0 0 1 0 0 1 0| Long PictureID flag = 1 703 |0 1 1 0 0 1 1 1| PictureID = 4711 704 +-+-+-+-+-+-+-+-+ 705 |Size0|1| VER |1| 706 +-+-+-+-+-+-+-+-+ 707 | Size1 | 708 +-+-+-+-+-+-+-+-+ 709 | Size2 | 710 +-+-+-+-+-+-+-+-+ 711 | Bytes 4..N of | 712 | VP8 payload | 713 : : 714 +-+-+-+-+-+-+-+-+ 716 5. Using VP8 with RPSI and SLI Feedback 718 The VP8 payload descriptor defined in Section 4.2 above contains an 719 optional PictureID parameter. This parameter is included mainly to 720 enable use of reference picture selection index (RPSI) and slice loss 721 indication (SLI), both defined in [RFC4585]. 723 5.1. RPSI 725 The reference picture selection index is a payload-specific feedback 726 message defined within the RTCP-based feedback format. The RPSI 727 message is generated by a receiver and can be used in two ways. 728 Either it can signal a preferred reference picture when a loss has 729 been detected by the decoder -- preferably then a reference that the 730 decoder knows is perfect -- or, it can be used as positive feedback 731 information to acknowledge correct decoding of certain reference 732 pictures. The positive feedback method is useful for VP8 used as 733 unicast. The use of RPSI for VP8 is preferably combined with a 734 special update pattern of the codec's two special reference frames -- 735 the golden frame and the altref frame -- in which they are updated in 736 an alternating leapfrog fashion. When a receiver has received and 737 correctly decoded a golden or altref frame, and that frame had a 738 PictureID in the payload descriptor, the receiver can acknowledge 739 this simply by sending an RPSI message back to the sender. The 740 message body (i.e., the "native RPSI bit string" in [RFC4585]) is 741 simply the PictureID of the received frame. 743 5.2. SLI 745 The slice loss indication is another payload-specific feedback 746 message defined within the RTCP-based feedback format. The SLI 747 message is generated by the receiver when a loss or corruption is 748 detected in a frame. The format of the SLI message is as follows 749 [RFC4585]: 751 0 1 2 3 752 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 753 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 754 | First | Number | PictureID | 755 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 757 Figure 4 759 Here, First is the macroblock address (in scan order) of the first 760 lost block and Number is the number of lost blocks. PictureID is the 761 six least significant bits of the codec-specific picture identifier 762 in which the loss or corruption has occurred. For VP8, this codec- 763 specific identifier is naturally the PictureID of the current frame, 764 as read from the payload descriptor. If the payload descriptor of 765 the current frame does not have a PictureID, the receiver MAY send 766 the last received PictureID+1 in the SLI message. The receiver MAY 767 set the First parameter to 0, and the Number parameter to the total 768 number of macroblocks per frame, even though only parts of the frame 769 is corrupted. When the sender receives an SLI message, it can make 770 use of the knowledge from the latest received RPSI message. Knowing 771 that the last golden or altref frame was successfully received, it 772 can encode the next frame with reference to that established 773 reference. 775 5.3. Example 777 The use of RPSI and SLI is best illustrated in an example. In this 778 example, the encoder may not update the altref frame until the last 779 sent golden frame has been acknowledged with an RPSI message. If an 780 update is not received within some time, a new golden frame update is 781 sent instead. Once the new golden frame is established and 782 acknowledged, the same rule applies when updating the altref frame. 784 +-------+-------------------+-------------------------+-------------+ 785 | Event | Sender | Receiver | Established | 786 | | | | reference | 787 +-------+-------------------+-------------------------+-------------+ 788 | 1000 | Send golden frame | | | 789 | | PictureID = 0 | | | 790 | | | | | 791 | | | Receive and decode | | 792 | | | golden frame | | 793 | | | | | 794 | 1001 | | Send RPSI(0) | | 795 | | | | | 796 | 1002 | Receive RPSI(0) | | golden | 797 | | | | | 798 | ... | (sending regular | | | 799 | | frames) | | | 800 | | | | | 801 | 1100 | Send altref frame | | | 802 | | PictureID = 100 | | | 803 | | | | | 804 | | | Altref corrupted or | golden | 805 | | | lost | | 806 | | | | | 807 | 1101 | | Send SLI(100) | golden | 808 | | | | | 809 | 1102 | Receive SLI(100) | | | 810 | | | | | 811 | 1103 | Send frame with | | | 812 | | reference to | | | 813 | | golden | | | 814 | | | | | 815 | | | Receive and decode | golden | 816 | | | frame (decoder state | | 817 | | | restored) | | 818 | | | | | 819 | ... | (sending regular | | | 820 | | frames) | | | 821 | | | | | 822 | 1200 | Send altref frame | | | 823 | | PictureID = 200 | | | 824 | | | | | 825 | | | Receive and decode | golden | 826 | | | altref frame | | 827 | | | | | 828 | 1201 | | Send RPSI(200) | | 829 | | | | | 830 | 1202 | Receive RPSI(200) | | altref | 831 | | | | | 832 | ... | (sending regular | | | 833 | | frames) | | | 834 | | | | | 835 | 1300 | Send golden frame | | | 836 | | PictureID = 300 | | | 837 | | | | | 838 | | | Receive and decode | altref | 839 | | | golden frame | | 840 | | | | | 841 | 1301 | | Send RPSI(300) | altref | 842 | | | | | 843 | 1302 | RPSI lost | | | 844 | | | | | 845 | 1400 | Send golden frame | | | 846 | | PictureID = 400 | | | 847 | | | | | 848 | | | Receive and decode | altref | 849 | | | golden frame | | 850 | | | | | 851 | 1401 | | Send RPSI(400) | | 852 | | | | | 853 | 1402 | Receive RPSI(400) | | golden | 854 +-------+-------------------+-------------------------+-------------+ 856 Table 1: Exemple signaling between sender and receiver 858 Note that the scheme is robust to loss of the feedback messages. If 859 the RPSI is lost, the sender will try to update the golden (or 860 altref) again after a while, without releasing the established 861 reference. Also, if an SLI is lost, the receiver can keep sending 862 SLI messages at any interval allowed by the RTCP sending timing 863 restrictions as specified in [RFC4585], as long as the picture is 864 corrupted. 866 6. Payload Format Parameters 868 This payload format has two required parameters. 870 6.1. Media Type Definition 872 This registration is done using the template defined in [RFC6838] and 873 following [RFC4855]. 875 Type name: video 877 Subtype name: VP8 879 Required parameters: 881 These parameters MUST be used to signal the capabilities of a 882 receiver implementation. These parameters MUST NOT be used for 883 any other purpose. 885 max-fr: The value of max-fr is an integer indicating the maximum 886 frame rate in units of frames per second that the decoder is 887 capable of decoding. 889 max-fs: The value of max-fs is an integer indicating the maximum 890 frame size in units of macroblocks that the decoder is capable 891 of decoding. 893 The decoder is capable of decoding this frame size as long as 894 the width and height of the frame in macroblocks are less than 895 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 896 of supporting 640x480 resolution) will support widths and 897 heights up to 1552 pixels (97 macroblocks). 899 Optional parameters: none 901 Encoding considerations: 902 This media type is framed in RTP and contains binary data; see 903 Section 4.8 of [RFC6838]. 905 Security considerations: See Section 7 of RFC xxxx. 906 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 907 with the number assigned to this document and remove this note.] 909 Interoperability considerations: None. 911 Published specification: VP8 bitstream format [RFC6386] and RFC 912 XXXX. 913 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 914 with the number assigned to this document and remove this note.] 916 Applications which use this media type: 917 For example: Video over IP, video conferencing. 919 Additional information: None. 921 Person & email address to contact for further information: 922 Patrik Westin, patrik.westin@gmail.com 924 Intended usage: COMMON 926 Restrictions on usage: 927 This media type depends on RTP framing, and hence is only defined 928 for transfer via RTP [RFC3550]. 930 Author: Patrik Westin, patrik.westin@gmail.com 932 Change controller: 933 IETF Payload Working Group delegated from the IESG. 935 6.2. SDP Parameters 937 The receiver MUST ignore any fmtp parameter unspecified in this memo. 939 6.2.1. Mapping of Media Subtype Parameters to SDP 941 The media type video/VP8 string is mapped to fields in the Session 942 Description Protocol (SDP) [RFC4566] as follows: 944 o The media name in the "m=" line of SDP MUST be video. 946 o The encoding name in the "a=rtpmap" line of SDP MUST be VP8 (the 947 media subtype). 949 o The clock rate in the "a=rtpmap" line MUST be 90000. 951 o The parameters "max-fs", and "max-fr", MUST be included in the 952 "a=fmtp" line if the SDP is used to declare receiver capabilities. 953 These parameters are expressed as a media subtype string, in the 954 form of a semicolon separated list of parameter=value pairs. 956 6.2.1.1. Example 958 An example of media representation in SDP is as follows: 960 m=video 49170 RTP/AVPF 98 961 a=rtpmap:98 VP8/90000 962 a=fmtp:98 max-fr=30; max-fs=3600; 964 6.2.2. Offer/Answer Considerations 966 The VP8 codec offers a decode complexity that is roughly linear with 967 the number of pixels encoded. The parameters "max-fr" and "max-fs" 968 are defined in Section 6.1, where the macroblock size is 16x16 pixels 969 as defined in [RFC6386], the max-fs and max-fr parameters MUST be 970 used to establish these limits. 972 7. Security Considerations 974 RTP packets using the payload format defined in this specification 975 are subject to the security considerations discussed in the RTP 976 specification [RFC3550], and in any applicable RTP profile. The main 977 security considerations for the RTP packet carrying the RTP payload 978 format defined within this memo are confidentiality, integrity and 979 source authenticity. Confidentiality is achieved by encryption of 980 the RTP payload. Integrity of the RTP packets through suitable 981 cryptographic integrity protection mechanism. Cryptographic system 982 may also allow the authentication of the source of the payload. A 983 suitable security mechanism for this RTP payload format should 984 provide confidentiality, integrity protection and at least source 985 authentication capable of determining if an RTP packet is from a 986 member of the RTP session or not. Note that the appropriate 987 mechanism to provide security to RTP and payloads following this memo 988 may vary. It is dependent on the application, the transport, and the 989 signaling protocol employed. Therefore a single mechanism is not 990 sufficient, although if suitable the usage of SRTP [RFC3711] is 991 recommended. This RTP payload format and its media decoder do not 992 exhibit any significant non-uniformity in the receiver-side 993 computational complexity for packet processing, and thus are unlikely 994 to pose a denial-of-service threat due to the receipt of pathological 995 data. Nor does the RTP payload format contain any active content. 997 8. Congestion Control 999 Congestion control for RTP SHALL be used in accordance with RFC 3550 1000 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 1001 [RFC3551]. The congestion control mechanism can, in a real-time 1002 encoding scenario, adapt the transmission rate by instructing the 1003 encoder to encode at a certain target rate. Media aware network 1004 elements MAY use the information in the VP8 payload descriptor in 1005 Section 4.2 to identify non-reference frames and discard them in 1006 order to reduce network congestion. Note that discarding of non- 1007 reference frames cannot be done if the stream is encrypted (because 1008 the non-reference marker is encrypted). 1010 9. IANA Considerations 1012 The IANA is requested to register the following values: 1013 - Media type registration as described in Section 6.1. 1015 10. References 1017 10.1. Normative References 1019 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1020 Requirement Levels", BCP 14, RFC 2119, March 1997. 1022 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1023 Jacobson, "RTP: A Transport Protocol for Real-Time 1024 Applications", STD 64, RFC 3550, July 2003. 1026 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 1027 Description Protocol", RFC 4566, July 2006. 1029 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1030 "Extended RTP Profile for Real-time Transport Control 1031 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 1032 2006. 1034 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 1035 Formats", RFC 4855, February 2007. 1037 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 1038 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 1039 Guide", RFC 6386, November 2011. 1041 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 1042 Specifications and Registration Procedures", BCP 13, RFC 1043 6838, January 2013. 1045 10.2. Informative References 1047 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1048 Video Conferences with Minimal Control", STD 65, RFC 3551, 1049 July 2003. 1051 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1052 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1053 RFC 3711, March 2004. 1055 Authors' Addresses 1057 Patrik Westin 1058 Google, Inc. 1059 1600 Amphitheatre Parkway 1060 Mountain View, CA 94043 1061 USA 1063 Email: patrik.westin@gmail.com 1065 Henrik F Lundin 1066 Google, Inc. 1067 Kungsbron 2 1068 Stockholm 11122 1069 Sweden 1071 Email: hlundin@google.com 1072 Michael Glover 1073 Google, Inc. 1074 5 Cambridge Center 1075 Cambridge, MA 02142 1076 USA 1078 Justin Uberti 1079 Google, Inc. 1080 747 6th Street South 1081 Kirkland, WA 98033 1082 USA 1084 Frank Galligan 1085 Google, Inc. 1086 1600 Amphitheatre Parkway 1087 Mountain View, CA 94043 1088 USA