idnits 2.17.1 draft-ietf-payload-vp8-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 13, 2014) is 3534 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 4566 (Obsoleted by RFC 8866) ** Downref: Normative reference to an Informational RFC: RFC 6386 Summary: 3 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Payload Working Group P. Westin 3 Internet-Draft H. Lundin 4 Intended status: Standards Track M. Glover 5 Expires: February 14, 2015 J. Uberti 6 F. Galligan 7 Google 8 August 13, 2014 10 RTP Payload Format for VP8 Video 11 draft-ietf-payload-vp8-12 13 Abstract 15 This memo describes an RTP payload format for the VP8 video codec. 16 The payload format has wide applicability, as it supports 17 applications from low bit-rate peer-to-peer usage, to high bit-rate 18 video conferences. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on February 14, 2015. 37 Copyright Notice 39 Copyright (c) 2014 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 56 3. Media Format Description . . . . . . . . . . . . . . . . . . 3 57 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 58 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 4 59 4.2. VP8 Payload Descriptor . . . . . . . . . . . . . . . . . 6 60 4.3. VP8 Payload Header . . . . . . . . . . . . . . . . . . . 10 61 4.4. Aggregated and Fragmented Payloads . . . . . . . . . . . 11 62 4.5. Frame reconstruction algorithm . . . . . . . . . . . . . 11 63 4.5.1. Partition reconstruction algorithm . . . . . . . . . 12 64 4.6. Examples of VP8 RTP Stream . . . . . . . . . . . . . . . 12 65 4.6.1. Key frame in a single RTP packet . . . . . . . . . . 12 66 4.6.2. Non-discardable VP8 interframe in a single RTP 67 packet; no PictureID . . . . . . . . . . . . . . . . 13 68 4.6.3. VP8 partitions in separate RTP packets . . . . . . . 14 69 4.6.4. VP8 frame fragmented across RTP packets . . . . . . . 15 70 4.6.5. VP8 frame with long PictureID . . . . . . . . . . . . 16 71 5. Using VP8 with RPSI and SLI Feedback . . . . . . . . . . . . 17 72 5.1. RPSI . . . . . . . . . . . . . . . . . . . . . . . . . . 17 73 5.2. SLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 74 5.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 20 76 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 20 77 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 22 78 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 22 79 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 22 80 7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 81 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 23 82 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 83 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 24 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 86 1. Introduction 88 This memo describes an RTP payload specification applicable to the 89 transmission of video streams encoded using the VP8 video codec 90 [RFC6386]. The format described in this document can be used both in 91 peer-to-peer and video conferencing applications. 93 VP8 is based on decomposition of frames into square sub-blocks of 94 pixels, prediction of such sub-blocks using previously constructed 95 blocks, and adjustment of such predictions (as well as synthesis of 96 unpredicted blocks) using a discrete cosine transform (hereafter 97 abbreviated as DCT). In one special case, however, VP8 uses a 98 "Walsh-Hadamard" (hereafter abbreviated as WHT) transform instead of 99 a DCT. An encoded VP8 frame is divided into two or more partitions, 100 as described in [RFC6386]. The first partition (prediction or mode) 101 contains prediction mode parameters and motion vectors for all 102 macroblocks. The remaining partitions all contain the quantized DCT/ 103 WHT coefficients for the residuals. There can be 1, 2, 4, or 8 DCT/ 104 WHT partitions per frame, depending on encoder settings. 106 In summary, the payload format described in this document enables a 107 number of features in VP8, including: 109 o Taking partition boundaries into consideration, to improve loss 110 robustness and facilitate efficient packet loss concealment at the 111 decoder. 113 o Temporal scalability. 115 o Advanced use of reference frames to enable efficient error 116 recovery. 118 o Marking of frames that have no impact on the decoding of any other 119 frame, so that these non-reference frames can be discarded in a 120 server or media-aware network element if needed. 122 2. Conventions, Definitions and Acronyms 124 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 125 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 126 document are to be interpreted as described in [RFC2119]. 128 3. Media Format Description 130 The VP8 codec uses three different reference frames for interframe 131 prediction: the previous frame, the golden frame, and the altref 132 frame. Blocks in an interframe may be predicted using blocks in the 133 immediately previous frame as well as the most recent golden frame or 134 altref frame. Every key frame is automatically golden and altref, 135 and any interframe may optionally replace the most recent golden or 136 altref frame. Golden frames and altref frames may also be used to 137 increase the tolerance to dropped frames. The payload specification 138 in this memo has elements that enable advanced use of the reference 139 frames, e.g., for improved loss robustness. 141 One specific use case of the three reference frame types is temporal 142 scalability. By setting up the reference hierarchy in the 143 appropriate way, up to five temporal layers can be encoded. (How to 144 set up the reference hierarchy for temporal scalability is not within 145 the scope of this memo.) 147 Another property of the VP8 codec is that it applies data 148 partitioning to the encoded data. Thus, an encoded VP8 frame can be 149 divided into two or more partitions, as described in "VP8 Data Format 150 and Decoding Guide" [RFC6386]. The first partition (prediction or 151 mode) contains prediction mode parameters and motion vectors for all 152 macroblocks. The remaining partitions all contain the transform 153 coefficients for the residuals. The first partition is decodable 154 without the remaining residual partitions. The subsequent partitions 155 may be useful even if some part of the frame is lost. This memo 156 allows the partitions to be sent separately or in the same RTP 157 packet. It may be beneficial for decoder error-concealment to send 158 the partitions in different packets, even though it is not mandatory 159 according to this specification. 161 The format specification is described in Section 4. In Section 5, a 162 method to acknowledge receipt of reference frames using RTCP 163 techniques is described. 165 The payload partitioning and the acknowledging method both serve as 166 motivation for three of the fields included in the payload format: 167 the "PID", "1st partition size" and "PictureID" fields. The ability 168 to encode a temporally scalable stream motivates the "TL0PICIDX" and 169 "TID" fields. 171 4. Payload Format 173 This section describes how the encoded VP8 bitstream is encapsulated 174 in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is 175 RECOMMENDED. All integer fields in the specifications are encoded as 176 unsigned integers in network octet order. 178 4.1. RTP Header Usage 179 The general RTP payload format for VP8 is depicted below. 181 0 1 2 3 182 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 183 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 184 |V=2|P|X| CC |M| PT | sequence number | 185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 186 | timestamp | 187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 188 | synchronization source (SSRC) identifier | 189 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 190 | contributing source (CSRC) identifiers | 191 | .... | 192 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 193 | VP8 payload descriptor (integer #bytes) | 194 : : 195 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 196 | : VP8 payload header (3 octets) | 197 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 198 | VP8 pyld hdr : | 199 +-+-+-+-+-+-+-+-+ | 200 : Bytes 4..N of VP8 payload : 201 | | 202 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 203 | : OPTIONAL RTP padding | 204 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 206 The VP8 payload descriptor and VP8 payload header will be described 207 in the sequel. OPTIONAL RTP padding MUST NOT be included unless the 208 P bit is set. 210 Figure 1 212 Marker bit (M): MUST be set for the very last packet of each encoded 213 frame in line with the normal use of the M bit in video formats. 214 This enables a decoder to finish decoding the picture, where it 215 otherwise may need to wait for the next packet to explicitly know 216 that the frame is complete. 218 Timestamp: The RTP timestamp indicates the time when the frame was 219 sampled at a clock rate of 90 kHz. 221 Sequence number: The sequence numbers are monotonically increasing 222 and set as packets are sent. 224 The remaining RTP header fields are used as specified in 225 [RFC3550]. 227 4.2. VP8 Payload Descriptor 229 The first octets after the RTP header are the VP8 payload descriptor, 230 with the following structure. The single-octet version of the 231 PictureID is illustrated to the left (M bit set to zero), while the 232 dual-octet version (M bit set to one) is show to the right. 234 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 235 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 236 |X|R|N|S|R| PID | (REQUIRED) |X|R|N|S|R| PID | (REQUIRED) 237 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 238 X: |I|L|T|K| RSV | (OPTIONAL) X: |I|L|T|K| RSV | (OPTIONAL) 239 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 240 I: |M| PictureID | (OPTIONAL) I: |M| PictureID | (OPTIONAL) 241 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 242 L: | TL0PICIDX | (OPTIONAL) | PictureID | 243 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 244 T/K: |TID|Y| KEYIDX | (OPTIONAL) L: | TL0PICIDX | (OPTIONAL) 245 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 246 T/K: |TID|Y| KEYIDX | (OPTIONAL) 247 +-+-+-+-+-+-+-+-+ 249 Figure 2 251 X: Extended control bits present. When set to one, the extension 252 octet MUST be provided immediately after the mandatory first 253 octet. If the bit is zero, all optional fields MUST be omitted. 255 R: Bit reserved for future use. MUST be set to zero and MUST be 256 ignored by the receiver. 258 N: Non-reference frame. When set to one, the frame can be discarded 259 without affecting any other future or past frames. If the 260 reference status of the frame is unknown, this bit SHOULD be set 261 to zero to avoid discarding frames needed for reference. 263 Informative note: This document does not describe how to 264 determine if an encoded frame is non-reference. The reference 265 status of an encoded frame is preferably provided from the 266 encoder implementation. 268 S: Start of VP8 partition. SHOULD be set to 1 when the first payload 269 octet of the RTP packet is the beginning of a new VP8 partition, 270 and MUST NOT be 1 otherwise. The S bit MUST be set to 1 for the 271 first packet of each encoded frame. 273 PID: Partition index. Denotes which VP8 partition the first payload 274 octet of the packet belongs to. The first VP8 partition 275 (containing modes and motion vectors) MUST be labeled with PID = 276 0. PID SHOULD be incremented for each subsequent partition, but 277 MAY be kept at 0 for all packets. PID MUST NOT be larger than 7. 278 If more than one packet in an encoded frame contains the same PID, 279 the S bit MUST NOT be set for any other packet than the first 280 packet with that PID. 282 When the X bit is set to 1 in the first octet, the extension bit 283 field octet MUST be provided as the second octet. If the X bit is 0, 284 the extension bit field octet MUST NOT be present, and all bits below 285 MUST be implicitly interpreted as 0. 287 I: PictureID present. When set to one, the OPTIONAL PictureID MUST 288 be present after the extension bit field and specified as below. 289 Otherwise, PictureID MUST NOT be present. 291 L: TL0PICIDX present. When set to one, the OPTIONAL TL0PICIDX MUST 292 be present and specified as below, and the T bit MUST be set to 1. 293 Otherwise, TL0PICIDX MUST NOT be present. 295 T: TID present. When set to one, the OPTIONAL TID/KEYIDX octet MUST 296 be present. The TID|Y part of the octet MUST be specified as 297 below. If K (below) is set to one but T is set to zero, the TID/ 298 KEYIDX octet MUST be present, but the TID|Y field MUST be ignored. 299 If neither T nor K is set to one, the TID/KEYIDX octet MUST NOT be 300 present. 302 K: KEYIDX present. When set to one, the OPTIONAL TID/KEYIDX octet 303 MUST be present. The KEYIDX part of the octet MUST be specified 304 as below. If T (above) is set to one but K is set to zero, the 305 TID/KEYIDX octet MUST be present, but the KEYIDX field MUST be 306 ignored. If neither T nor K is set to one, the TID/KEYIDX octet 307 MUST NOT be present. 309 RSV: Bits reserved for future use. MUST be set to zero and MUST be 310 ignored by the receiver. 312 After the extension bit field follow the extension data fields that 313 are enabled. 315 M: The most significant bit of the first octet is an extension flag. 316 The field MUST be present if the I bit is equal to one. If set 317 the PictureID field MUST contain 16 bits else it MUST contain 8 318 bits including this MSB, see PictureID. 320 PictureID: 8 or 16 bits (shown left and right, respectively, in 321 Figure 2) including the M bit. This is a running index of the 322 frames. The field MUST be present if the I bit is equal to one. 324 The 7 following bits carry (parts of) the PictureID. If the 325 extension flag is one, the PictureID continues in the next octet 326 forming a 15 bit index, where the 8 bits in the second octet are 327 the least significant bits of the PictureID. If the extension 328 flag is zero, there is no extension, and the PictureID is the 7 329 remaining bits of the first (and only) octet. The sender may 330 choose 7 or 15 bits index. The PictureID SHOULD start on a random 331 number, and MUST wrap after reaching the maximum ID. The receiver 332 MUST NOT assume that the number of bits in PictureID stay the same 333 through the session. 335 TL0PICIDX: 8 bits temporal level zero index. The field MUST be 336 present if the L bit is equal to 1, and MUST NOT be present 337 otherwise. TL0PICIDX is a running index for the temporal base 338 layer frames, i.e., the frames with TID set to 0. If TID is 339 larger than 0, TL0PICIDX indicates which base layer frame the 340 current image depends on. TL0PICIDX MUST be incremented when TID 341 is 0. The index SHOULD start on a random number, and MUST restart 342 at 0 after reaching the maximum number 255. 344 TID: 2 bits temporal layer index. The TID/KEYIDX octet MUST be 345 present when either the T bit or the K bit or both are equal to 1, 346 and MUST NOT be present otherwise. The TID field MUST be ignored 347 by the receiver when the T bit is set equal to 0. The TID field 348 indicates which temporal layer the packet represents. The lowest 349 layer, i.e., the base layer, MUST have TID set to 0. Higher 350 layers SHOULD increment the TID according to their position in the 351 layer hierarchy. 353 Y: 1 layer sync bit. The TID/KEYIDX octet MUST be present when 354 either the T bit or the K bit or both are equal to 1, and MUST NOT 355 be present otherwise. The Y bit SHOULD be set to 1 if the current 356 frame depends only on the base layer (TID = 0) frame with 357 TL0PICIDX equal to that of the current frame. The Y bit MUST be 358 set to 0 if the current frame depends any other frame than the 359 base layer (TID = 0) frame with TL0PICIDX equal to that of the 360 current frame. If the Y bit is set when the T bit is equal to 0 361 the current frame MUST only depend on a past base layer (TID=0) 362 key frame as signaled by a change in the KEYIDX field. 363 Additionally this frame MUST NOT depend on any of the three codec 364 buffers (as defined by [RFC6386]) that have been updated since the 365 last time the KEYIDX field was changed. 367 Informative note: This document does not describe how to 368 determine the dependence status for a frame; this information 369 is preferably provided from the encoder implementation. In the 370 case of unknown status, the Y bit can safely be set to 0. 372 KEYIDX: 5 bits temporal key frame index. The TID/KEYIDX octet MUST 373 be present when either the T bit or the K bit or both are equal to 374 1, and MUST NOT be present otherwise. The KEYIDX field MUST be 375 ignored by the receiver when the K bit is set equal to 0. The 376 KEYIDX field is a running index for key frames. KEYIDX MAY start 377 on a random number, and MUST restart at 0 after reaching the 378 maximum number 31. When in use, the KEYIDX SHOULD be present for 379 both key frames and interframes. The sender MUST increment KEYIDX 380 for key frames which convey parameter updates critical to the 381 interpretation of subsequent frames, and SHOULD leave the KEYIDX 382 unchanged for key frames that do not contain these critical 383 updates. If the KEYIDX is present, a receiver SHOULD NOT decode 384 an interframe if it has not received and decoded a key frame with 385 the same KEYIDX after the last KEYIDX wrap-around. 387 Informative note: This document does not describe how to 388 determine if a key frame updates critical parameters; this 389 information is preferably provided from the encoder 390 implementation. A sender that does not have this information 391 may either omit the KEYIDX field (set K equal to 0), or 392 increment the KEYIDX on every key frame. The benefit with the 393 latter is that any key frame loss will be detected by the 394 receiver, which can signal for re-transmission or request a new 395 key frame. 397 Informative note: Implementations doing splicing of VP8 streams will 398 have to make sure the rules for incrementing TL0PICIDX and KEYIDX 399 are obeyed across the splice. This will likely require rewriting 400 values of TL0PICIDX and KEYIDX after the splice. 402 4.3. VP8 Payload Header 404 The beginning of an encoded VP8 frame is referred to as an 405 "uncompressed data chunk" in [RFC6386], and co-serve as payload 406 header in this RTP format. The codec bitstream format specifies two 407 different variants of the uncompressed data chunk: a 3 octet version 408 for interframes and a 10 octet version for key frames. The first 3 409 octets are common to both variants. In the case of a key frame the 410 remaining 7 octets are considered to be part of the remaining payload 411 in this RTP format. Note that the header is present only in packets 412 which have the S bit equal to one and the PID equal to zero in the 413 payload descriptor. Subsequent packets for the same frame do not 414 carry the payload header. 416 0 1 2 3 4 5 6 7 417 +-+-+-+-+-+-+-+-+ 418 |Size0|H| VER |P| 419 +-+-+-+-+-+-+-+-+ 420 | Size1 | 421 +-+-+-+-+-+-+-+-+ 422 | Size2 | 423 +-+-+-+-+-+-+-+-+ 424 | Bytes 4..N of | 425 | VP8 payload | 426 : : 427 +-+-+-+-+-+-+-+-+ 428 | OPTIONAL RTP | 429 | padding | 430 : : 431 +-+-+-+-+-+-+-+-+ 433 Figure 3 435 H: Show frame bit as defined in [RFC6386]. 437 VER: A version number as defined in [RFC6386]. 439 P: Inverse key frame flag. When set to 0 the current frame is a key 440 frame. When set to 1 the current frame is an interframe. Defined 441 in [RFC6386] 443 SizeN: The size of the first partition in bytes is calculated from 444 the 19 bits in Size0, Size1, and Size2 as 1stPartitionSize = Size0 445 + 8 * Size1 + 2048 * Size2. [RFC6386]. 447 4.4. Aggregated and Fragmented Payloads 449 An encoded VP8 frame can be divided into two or more partitions, as 450 described in Section 1. One packet can contain a fragment of a 451 partition, a complete partition, or an aggregate of fragments and 452 partitions. In the preferred use case, the S bit and PID fields 453 described in Section 4.2 should be used to indicate what the packet 454 contains. The PID field should indicate which partition the first 455 octet of the payload belongs to, and the S bit indicates that the 456 packet starts on a new partition. Aggregation of encoded partitions 457 is done without explicit signaling. Partitions MUST be aggregated in 458 decoding order. Two fragments from different partitions MAY be 459 aggregated into the same packet. An aggregation MUST have exactly 460 one payload descriptor. Aggregated partitions MUST represent parts 461 of one and the same video frame. Consequently, an aggregated packet 462 will have one or no payload header, depending on whether the 463 aggregate contains the beginning of the first partition of a frame or 464 not, respectively. Note that the length of the first partition can 465 always be obtained from the first partition size parameter in the VP8 466 payload header. 468 The VP8 bitstream format [RFC6386] specifies that if multiple DCT/WHT 469 partitions are produced, the location of each partition start is 470 found at the end of the first (prediction/mode) partition. In this 471 RTP payload specification, the location offsets are considered to be 472 part of the first partition. 474 It is OPTIONAL for a packetizer implementing this RTP specification 475 to pay attention to the partition boundaries within an encoded frame. 476 If packetization of a frame is done without considering the partition 477 boundaries, the PID field MAY be set to zero for all packets, and the 478 S bit MUST NOT be set to one for any other packet than the first. 480 4.5. Frame reconstruction algorithm 482 Example of frame reconstruction algorithm. 484 1: Collect all packets with a given RTP timestamp. 486 2: Go through packets in order, sorted by sequence numbers, if 487 packets are missing, send NACK as defined in [RFC4585] or decode 488 with missing partitions, see Section 4.5.1 below. 490 3: A frame is complete if the frame has no missing sequence numbers, 491 the first packet in the frame contains S=1 with partId=0 and the 492 last packet in the frame has the marker bit set. 494 4.5.1. Partition reconstruction algorithm 496 Example of partition reconstruction algorithm. 498 1: Scan for the start of a new partition; S=1. 500 2: Continue scan to detect end of partition; hence a new S=1 501 (previous packet was the end of the partition) is found or the 502 marker bit is set. If a loss is detected before the end of the 503 partition, abandon all packets in this partition and continue the 504 scan repeating from step 1. 506 3: Store the packets in the complete partition, continue the scan 507 repeating from step 1 until end of frame is reached. 509 4: Send all complete partitions to the decoder. If no complete 510 partition is found discard the whole frame. 512 4.6. Examples of VP8 RTP Stream 514 A few examples of how the VP8 RTP payload can be used are included 515 below. 517 4.6.1. Key frame in a single RTP packet 519 0 1 2 3 4 5 6 7 520 +-+-+-+-+-+-+-+-+ 521 | RTP header | 522 | M = 1 | 523 +-+-+-+-+-+-+-+-+ 524 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 525 +-+-+-+-+-+-+-+-+ 526 |1|0|0|0|0 0 0 0| I = 1 527 +-+-+-+-+-+-+-+-+ 528 |0 0 0 0 1 0 0 1| PictureID = 17 529 +-+-+-+-+-+-+-+-+ 530 |Size0|1| VER |0| P = 0 531 +-+-+-+-+-+-+-+-+ 532 | Size1 | 533 +-+-+-+-+-+-+-+-+ 534 | Size2 | 535 +-+-+-+-+-+-+-+-+ 536 | VP8 payload | 537 +-+-+-+-+-+-+-+-+ 539 4.6.2. Non-discardable VP8 interframe in a single RTP packet; no 540 PictureID 542 0 1 2 3 4 5 6 7 543 +-+-+-+-+-+-+-+-+ 544 | RTP header | 545 | M = 1 | 546 +-+-+-+-+-+-+-+-+ 547 |0|0|0|1|0|0 0 0| X = 0; S = 1; PID = 0 548 +-+-+-+-+-+-+-+-+ 549 |Size0|1| VER |1| P = 1 550 +-+-+-+-+-+-+-+-+ 551 | Size1 | 552 +-+-+-+-+-+-+-+-+ 553 | Size2 | 554 +-+-+-+-+-+-+-+-+ 555 | VP8 payload | 556 +-+-+-+-+-+-+-+-+ 558 4.6.3. VP8 partitions in separate RTP packets 560 First RTP packet; complete first partition. 562 0 1 2 3 4 5 6 7 563 +-+-+-+-+-+-+-+-+ 564 | RTP header | 565 | M = 0 | 566 +-+-+-+-+-+-+-+-+ 567 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 568 +-+-+-+-+-+-+-+-+ 569 |1|0|0|0|0 0 0 0| I = 1 570 +-+-+-+-+-+-+-+-+ 571 |0 0 0 0 1 0 0 1| PictureID = 17 572 +-+-+-+-+-+-+-+-+ 573 |Size0|1| VER |1| P = 1 574 +-+-+-+-+-+-+-+-+ 575 | Size1 | 576 +-+-+-+-+-+-+-+-+ 577 | Size2 | 578 +-+-+-+-+-+-+-+-+ 579 | Bytes 4..L of | 580 | first VP8 | 581 | partition | 582 : : 583 +-+-+-+-+-+-+-+-+ 585 Second RTP packet; complete second partition. 587 0 1 2 3 4 5 6 7 588 +-+-+-+-+-+-+-+-+ 589 | RTP header | 590 | M = 1 | 591 +-+-+-+-+-+-+-+-+ 592 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 593 +-+-+-+-+-+-+-+-+ 594 |1|0|0|0|0 0 0 0| I = 1 595 +-+-+-+-+-+-+-+-+ 596 |0 0 0 0 1 0 0 1| PictureID = 17 597 +-+-+-+-+-+-+-+-+ 598 | Remaining VP8 | 599 | partitions | 600 : : 601 +-+-+-+-+-+-+-+-+ 603 4.6.4. VP8 frame fragmented across RTP packets 605 First RTP packet; complete first partition. 607 0 1 2 3 4 5 6 7 608 +-+-+-+-+-+-+-+-+ 609 | RTP header | 610 | M = 0 | 611 +-+-+-+-+-+-+-+-+ 612 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 613 +-+-+-+-+-+-+-+-+ 614 |1|0|0|0|0 0 0 0| I = 1 615 +-+-+-+-+-+-+-+-+ 616 |0 0 0 0 1 0 0 1| PictureID = 17 617 +-+-+-+-+-+-+-+-+ 618 |Size0|1| VER |1| P = 1 619 +-+-+-+-+-+-+-+-+ 620 | Size1 | 621 +-+-+-+-+-+-+-+-+ 622 | Size2 | 623 +-+-+-+-+-+-+-+-+ 624 | Complete | 625 | first | 626 | partition | 627 : : 628 +-+-+-+-+-+-+-+-+ 630 Second RTP packet; first fragment of second partition. 632 0 1 2 3 4 5 6 7 633 +-+-+-+-+-+-+-+-+ 634 | RTP header | 635 | M = 0 | 636 +-+-+-+-+-+-+-+-+ 637 |1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1 638 +-+-+-+-+-+-+-+-+ 639 |1|0|0|0|0 0 0 0| I = 1 640 +-+-+-+-+-+-+-+-+ 641 |0 0 0 0 1 0 0 1| PictureID = 17 642 +-+-+-+-+-+-+-+-+ 643 | First fragment| 644 | of second | 645 | partition | 646 : : 647 +-+-+-+-+-+-+-+-+ 649 Third RTP packet; second fragment of second partition. 651 0 1 2 3 4 5 6 7 652 +-+-+-+-+-+-+-+-+ 653 | RTP header | 654 | M = 0 | 655 +-+-+-+-+-+-+-+-+ 656 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 657 +-+-+-+-+-+-+-+-+ 658 |1|0|0|0|0 0 0 0| I = 1 659 +-+-+-+-+-+-+-+-+ 660 |0 0 0 0 1 0 0 1| PictureID = 17 661 +-+-+-+-+-+-+-+-+ 662 | Mid fragment | 663 | of second | 664 | partition | 665 : : 666 +-+-+-+-+-+-+-+-+ 668 Fourth RTP packet; last fragment of second partition. 670 0 1 2 3 4 5 6 7 671 +-+-+-+-+-+-+-+-+ 672 | RTP header | 673 | M = 1 | 674 +-+-+-+-+-+-+-+-+ 675 |1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1 676 +-+-+-+-+-+-+-+-+ 677 |1|0|0|0|0 0 0 0| I = 1 678 +-+-+-+-+-+-+-+-+ 679 |0 0 0 0 1 0 0 1| PictureID = 17 680 +-+-+-+-+-+-+-+-+ 681 | Last fragment | 682 | of second | 683 | partition | 684 : : 685 +-+-+-+-+-+-+-+-+ 687 4.6.5. VP8 frame with long PictureID 688 PictureID = 4711 = 001001001100111 binary (first 7 bits: 0010010, 689 last 8 bits: 01100111). 691 0 1 2 3 4 5 6 7 692 +-+-+-+-+-+-+-+-+ 693 | RTP header | 694 | M = 1 | 695 +-+-+-+-+-+-+-+-+ 696 |1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0 697 +-+-+-+-+-+-+-+-+ 698 |1|0|0|0|0 0 0 0| I = 1; 699 +-+-+-+-+-+-+-+-+ 700 |1 0 0 1 0 0 1 0| Long PictureID flag = 1 701 |0 1 1 0 0 1 1 1| PictureID = 4711 702 +-+-+-+-+-+-+-+-+ 703 |Size0|1| VER |1| 704 +-+-+-+-+-+-+-+-+ 705 | Size1 | 706 +-+-+-+-+-+-+-+-+ 707 | Size2 | 708 +-+-+-+-+-+-+-+-+ 709 | Bytes 4..N of | 710 | VP8 payload | 711 : : 712 +-+-+-+-+-+-+-+-+ 714 5. Using VP8 with RPSI and SLI Feedback 716 The VP8 payload descriptor defined in Section 4.2 above contains an 717 optional PictureID parameter. This parameter is included mainly to 718 enable use of reference picture selection index (RPSI) and slice loss 719 indication (SLI), both defined in [RFC4585]. 721 5.1. RPSI 723 The reference picture selection index is a payload-specific feedback 724 message defined within the RTCP-based feedback format. The RPSI 725 message is generated by a receiver and can be used in two ways. 726 Either it can signal a preferred reference picture when a loss has 727 been detected by the decoder -- preferably then a reference that the 728 decoder knows is perfect -- or, it can be used as positive feedback 729 information to acknowledge correct decoding of certain reference 730 pictures. The positive feedback method is useful for VP8 used as 731 unicast. The use of RPSI for VP8 is preferably combined with a 732 special update pattern of the codec's two special reference frames -- 733 the golden frame and the altref frame -- in which they are updated in 734 an alternating leapfrog fashion. When a receiver has received and 735 correctly decoded a golden or altref frame, and that frame had a 736 PictureID in the payload descriptor, the receiver can acknowledge 737 this simply by sending an RPSI message back to the sender. The 738 message body (i.e., the "native RPSI bit string" in [RFC4585]) is 739 simply the PictureID of the received frame. 741 5.2. SLI 743 The slice loss indication is another payload-specific feedback 744 message defined within the RTCP-based feedback format. The SLI 745 message is generated by the receiver when a loss or corruption is 746 detected in a frame. The format of the SLI message is as follows 747 [RFC4585]: 749 0 1 2 3 750 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 751 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 752 | First | Number | PictureID | 753 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 755 Figure 4 757 Here, First is the macroblock address (in scan order) of the first 758 lost block and Number is the number of lost blocks. PictureID is the 759 six least significant bits of the codec-specific picture identifier 760 in which the loss or corruption has occurred. For VP8, this codec- 761 specific identifier is naturally the PictureID of the current frame, 762 as read from the payload descriptor. If the payload descriptor of 763 the current frame does not have a PictureID, the receiver MAY send 764 the last received PictureID+1 in the SLI message. The receiver MAY 765 set the First parameter to 0, and the Number parameter to the total 766 number of macroblocks per frame, even though only parts of the frame 767 is corrupted. When the sender receives an SLI message, it can make 768 use of the knowledge from the latest received RPSI message. Knowing 769 that the last golden or altref frame was successfully received, it 770 can encode the next frame with reference to that established 771 reference. 773 5.3. Example 775 The use of RPSI and SLI is best illustrated in an example. In this 776 example, the encoder may not update the altref frame until the last 777 sent golden frame has been acknowledged with an RPSI message. If an 778 update is not received within some time, a new golden frame update is 779 sent instead. Once the new golden frame is established and 780 acknowledged, the same rule applies when updating the altref frame. 782 +-------+-------------------+-------------------------+-------------+ 783 | Event | Sender | Receiver | Established | 784 | | | | reference | 785 +-------+-------------------+-------------------------+-------------+ 786 | 1000 | Send golden frame | | | 787 | | PictureID = 0 | | | 788 | | | | | 789 | | | Receive and decode | | 790 | | | golden frame | | 791 | | | | | 792 | 1001 | | Send RPSI(0) | | 793 | | | | | 794 | 1002 | Receive RPSI(0) | | golden | 795 | | | | | 796 | ... | (sending regular | | | 797 | | frames) | | | 798 | | | | | 799 | 1100 | Send altref frame | | | 800 | | PictureID = 100 | | | 801 | | | | | 802 | | | Altref corrupted or | golden | 803 | | | lost | | 804 | | | | | 805 | 1101 | | Send SLI(100) | golden | 806 | | | | | 807 | 1102 | Receive SLI(100) | | | 808 | | | | | 809 | 1103 | Send frame with | | | 810 | | reference to | | | 811 | | golden | | | 812 | | | | | 813 | | | Receive and decode | golden | 814 | | | frame (decoder state | | 815 | | | restored) | | 816 | | | | | 817 | ... | (sending regular | | | 818 | | frames) | | | 819 | | | | | 820 | 1200 | Send altref frame | | | 821 | | PictureID = 200 | | | 822 | | | | | 823 | | | Receive and decode | golden | 824 | | | altref frame | | 825 | | | | | 826 | 1201 | | Send RPSI(200) | | 827 | | | | | 828 | 1202 | Receive RPSI(200) | | altref | 829 | | | | | 830 | ... | (sending regular | | | 831 | | frames) | | | 832 | | | | | 833 | 1300 | Send golden frame | | | 834 | | PictureID = 300 | | | 835 | | | | | 836 | | | Receive and decode | altref | 837 | | | golden frame | | 838 | | | | | 839 | 1301 | | Send RPSI(300) | altref | 840 | | | | | 841 | 1302 | RPSI lost | | | 842 | | | | | 843 | 1400 | Send golden frame | | | 844 | | PictureID = 400 | | | 845 | | | | | 846 | | | Receive and decode | altref | 847 | | | golden frame | | 848 | | | | | 849 | 1401 | | Send RPSI(400) | | 850 | | | | | 851 | 1402 | Receive RPSI(400) | | golden | 852 +-------+-------------------+-------------------------+-------------+ 854 Table 1: Exemple signaling between sender and receiver 856 Note that the scheme is robust to loss of the feedback messages. If 857 the RPSI is lost, the sender will try to update the golden (or 858 altref) again after a while, without releasing the established 859 reference. Also, if an SLI is lost, the receiver can keep sending 860 SLI messages at any interval allowed by the RTCP sending timing 861 restrictions as specified in [RFC4585], as long as the picture is 862 corrupted. 864 6. Payload Format Parameters 866 This payload format has two required parameters. 868 6.1. Media Type Definition 870 This registration is done using the template defined in [RFC6838] and 871 following [RFC4855]. 873 Type name: video 875 Subtype name: VP8 877 Required parameters: 879 These parameters MUST be used to signal the capabilities of a 880 receiver implementation. These parameters MUST NOT be used for 881 any other purpose. 883 max-fr: The value of max-fr is an integer indicating the maximum 884 frame rate in units of frames per second that the decoder is 885 capable of decoding. 887 max-fs: The value of max-fs is an integer indicating the maximum 888 frame size in units of macroblocks that the decoder is capable 889 of decoding. 891 The decoder is capable of decoding this frame size as long as 892 the width and height of the frame in macroblocks are less than 893 int(sqrt(max-fs * 8)) - for instance, a max-fs of 1200 (capable 894 of supporting 640x480 resolution) will support widths and 895 heights up to 1552 pixels (97 macroblocks). 897 Optional parameters: none 899 Encoding considerations: 900 This media type is framed in RTP and contains binary data; see 901 Section 4.8 of [RFC6838]. 903 Security considerations: See Section 7 of RFC xxxx. 904 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 905 with the number assigned to this document and remove this note.] 907 Interoperability considerations: None. 909 Published specification: VP8 bitstream format [RFC6386] and RFC 910 XXXX. 911 [RFC Editor: Upon publication as an RFC, please replace "XXXX" 912 with the number assigned to this document and remove this note.] 914 Applications which use this media type: 915 For example: Video over IP, video conferencing. 917 Additional information: None. 919 Person & email address to contact for further information: 920 Patrik Westin, patrik.westin@gmail.com 922 Intended usage: COMMON 924 Restrictions on usage: 925 This media type depends on RTP framing, and hence is only defined 926 for transfer via RTP [RFC3550]. 928 Author: Patrik Westin, patrik.westin@gmail.com 930 Change controller: 931 IETF Payload Working Group delegated from the IESG. 933 6.2. SDP Parameters 935 The receiver MUST ignore any fmtp parameter unspecified in this memo. 937 6.2.1. Mapping of Media Subtype Parameters to SDP 939 The media type video/VP8 string is mapped to fields in the Session 940 Description Protocol (SDP) [RFC4566] as follows: 942 o The media name in the "m=" line of SDP MUST be video. 944 o The encoding name in the "a=rtpmap" line of SDP MUST be VP8 (the 945 media subtype). 947 o The clock rate in the "a=rtpmap" line MUST be 90000. 949 o The parameters "max-fs", and "max-fr", MUST be included in the 950 "a=fmtp" line of SDP. These parameters are expressed as a media 951 subtype string, in the form of a semicolon separated list of 952 parameter=value pairs. 954 6.2.1.1. Example 956 An example of media representation in SDP is as follows: 958 m=video 49170 RTP/AVPF 98 959 a=rtpmap:98 VP8/90000 960 a=fmtp:98 max-fr=30; max-fs=3600; 962 6.2.2. Offer/Answer Considerations 964 The VP8 codec offers a decode complexity that is roughly linear with 965 the number of pixels encoded. The parameters "max-fr" and "max-fs" 966 are defined in Section 6.1, where the macroblock size is 16x16 pixels 967 as defined in [RFC6386], the max-fs and max-fr parameters MUST be 968 used to establish these limits. 970 NOTE IN DRAFT: If closer control of width and height is desired, the 971 mechanism described in draft-nandakumar-payload-sdp-max-video- 972 resolution is a possible candidate for signalling, but since that 973 document appears to be far from finalization, this document does not 974 make a reference to that document. This note is only intended for 975 facilitating WG discussion, and should be deleted before publication 976 of this document as an RFC. 978 7. Security Considerations 980 RTP packets using the payload format defined in this specification 981 are subject to the security considerations discussed in the RTP 982 specification [RFC3550], and in any applicable RTP profile. The main 983 security considerations for the RTP packet carrying the RTP payload 984 format defined within this memo are confidentiality, integrity and 985 source authenticity. Confidentiality is achieved by encryption of 986 the RTP payload. Integrity of the RTP packets through suitable 987 cryptographic integrity protection mechanism. Cryptographic system 988 may also allow the authentication of the source of the payload. A 989 suitable security mechanism for this RTP payload format should 990 provide confidentiality, integrity protection and at least source 991 authentication capable of determining if an RTP packet is from a 992 member of the RTP session or not. Note that the appropriate 993 mechanism to provide security to RTP and payloads following this memo 994 may vary. It is dependent on the application, the transport, and the 995 signaling protocol employed. Therefore a single mechanism is not 996 sufficient, although if suitable the usage of SRTP [RFC3711] is 997 recommended. This RTP payload format and its media decoder do not 998 exhibit any significant non-uniformity in the receiver-side 999 computational complexity for packet processing, and thus are unlikely 1000 to pose a denial-of-service threat due to the receipt of pathological 1001 data. Nor does the RTP payload format contain any active content. 1003 8. Congestion Control 1005 Congestion control for RTP SHALL be used in accordance with RFC 3550 1006 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 1007 [RFC3551]. The congestion control mechanism can, in a real-time 1008 encoding scenario, adapt the transmission rate by instructing the 1009 encoder to encode at a certain target rate. Media aware network 1010 elements MAY use the information in the VP8 payload descriptor in 1011 Section 4.2 to identify non-reference frames and discard them in 1012 order to reduce network congestion. Note that discarding of non- 1013 reference frames cannot be done if the stream is encrypted (because 1014 the non-reference marker is encrypted). 1016 9. IANA Considerations 1018 The IANA is requested to register the following values: 1019 - Media type registration as described in Section 6.1. 1021 10. References 1023 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1024 Requirement Levels", BCP 14, RFC 2119, March 1997. 1026 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1027 Jacobson, "RTP: A Transport Protocol for Real-Time 1028 Applications", STD 64, RFC 3550, July 2003. 1030 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1031 Video Conferences with Minimal Control", STD 65, RFC 3551, 1032 July 2003. 1034 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1035 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1036 RFC 3711, March 2004. 1038 [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 1039 Description Protocol", RFC 4566, July 2006. 1041 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1042 "Extended RTP Profile for Real-time Transport Control 1043 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 1044 2006. 1046 [RFC4855] Casner, S., "Media Type Registration of RTP Payload 1047 Formats", RFC 4855, February 2007. 1049 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 1050 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 1051 Guide", RFC 6386, November 2011. 1053 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 1054 Specifications and Registration Procedures", BCP 13, RFC 1055 6838, January 2013. 1057 Authors' Addresses 1059 Patrik Westin 1060 Google, Inc. 1061 1600 Amphitheatre Parkway 1062 Mountain View, CA 94043 1063 USA 1065 Email: patrik.westin@gmail.com 1066 Henrik F Lundin 1067 Google, Inc. 1068 Kungsbron 2 1069 Stockholm 11122 1070 Sweden 1072 Email: hlundin@google.com 1074 Michael Glover 1075 Google, Inc. 1076 5 Cambridge Center 1077 Cambridge, MA 02142 1078 USA 1080 Justin Uberti 1081 Google, Inc. 1082 747 6th Street South 1083 Kirkland, WA 98033 1084 USA 1086 Frank Galligan 1087 Google, Inc. 1088 1600 Amphitheatre Parkway 1089 Mountain View, CA 94043 1090 USA