idnits 2.17.1 draft-ietf-avt-rtp-rfc3984bis-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The abstract seems to indicate that this document obsoletes RFC3984, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document seems to contain a disclaimer for pre-RFC5378 work, and may have content which was first submitted before 10 November 2008. The disclaimer is necessary when there are original authors that you have been unable to contact, or if some do not wish to grant the BCP78 rights to the IETF Trust. If you are able to get all authors (current and original) to grant those rights, you can and should remove the disclaimer; otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 21, 2009) is 5330 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' ** Obsolete normative reference: RFC 4566 (ref. '6') (Obsoleted by RFC 8866) ** Obsolete normative reference: RFC 3548 (ref. '7') (Obsoleted by RFC 4648) -- Obsolete informational reference (is this intentional?): RFC 2326 (ref. '27') (Obsoleted by RFC 7826) -- Obsolete informational reference (is this intentional?): RFC 5117 (ref. '29') (Obsoleted by RFC 7667) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Obsoletes RFC 3984 2 Audio/Video Transport WG Y.-K. Wang 3 Internet Draft Huawei Technologies 4 Intended status: Standards track R. Even 5 Expires: March 2010 Self-employed 6 T. Kristensen 7 Tandberg 8 R. Jesup 9 WorldGate Communications 10 September 21, 2009 12 RTP Payload Format for H.264 Video 13 draft-ietf-avt-rtp-rfc3984bis-08.txt 15 Status of this Memo 17 This Internet-Draft is submitted to IETF in full conformance with 18 the provisions of BCP 78 and BCP 79. This document may contain 19 material from IETF Documents or IETF Contributions published or 20 made publicly available before November 10, 2008. The person(s) 21 controlling the copyright in some of this material may not have 22 granted the IETF Trust the right to allow modifications of such 23 material outside the IETF Standards Process. Without obtaining an 24 adequate license from the person(s) controlling the copyright in 25 such materials, this document may not be modified outside the IETF 26 Standards Process, and derivative works of it may not be created 27 outside the IETF Standards Process, except to format it for 28 publication as an RFC or to translate it into languages other than 29 English. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF), its areas, and its working groups. Note that 33 other groups may also distribute working documents as Internet- 34 Drafts. 36 Internet-Drafts are draft documents valid for a maximum of six 37 months and may be updated, replaced, or obsoleted by other 38 documents at any time. It is inappropriate to use Internet-Drafts 39 as reference material or to cite them other than as "work in 40 progress". 42 The list of current Internet-Drafts can be accessed at 43 http://www.ietf.org/ietf/1id-abstracts.txt. 45 The list of Internet-Draft Shadow Directories can be accessed at 46 http://www.ietf.org/shadow.html. 48 This Internet-Draft will expire on March 21, 2009. 50 Copyright Notice 52 Copyright (c) 2009 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents in effect on the date of 57 publication of this document (http://trustee.ietf.org/license-info). 58 Please review these documents carefully, as they describe your 59 rights and restrictions with respect to this document. Code 60 Components extracted from this document must include Simplified BSD 61 License text as described in Section 4.e of the Trust Legal 62 Provisions and are provided without warranty as described in the 63 BSD License. 65 Abstract 67 This memo describes an RTP Payload format for the ITU-T 68 Recommendation H.264 video codec and the technically identical 69 ISO/IEC International Standard 14496-10 video codec, excluding the 70 Scalable Video Coding (SVC) extension and the Multivew Video Coding 71 extension, for which the RTP payload formats are defined elsewhere. 72 The RTP payload format allows for packetization of one or more 73 Network Abstraction Layer Units (NALUs), produced by an H.264 video 74 encoder, in each RTP payload. The payload format has wide 75 applicability, as it supports applications from simple low bit-rate 76 conversational usage, to Internet video streaming with interleaved 77 transmission, to high bit-rate video-on-demand. 79 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 80 in section 18. Issues on backward compatibility to RFC 3984 are 81 discussed in section 17. 83 Table of Contents 85 Abstract.........................................................2 86 1. Introduction..................................................5 87 1.1. The H.264 Codec..........................................5 88 1.2. Parameter Set Concept....................................6 89 1.3. Network Abstraction Layer Unit Types.....................7 90 2. Conventions...................................................8 91 3. Scope.........................................................8 92 4. Definitions and Abbreviations.................................8 93 4.1. Definitions..............................................8 94 4.2. Abbreviations...........................................10 95 5. RTP Payload Format...........................................11 96 5.1. RTP Header Usage........................................11 97 5.2. Payload Structures......................................13 98 5.3. NAL Unit Header Usage...................................15 99 5.4. Packetization Modes.....................................17 100 5.5. Decoding Order Number (DON).............................19 101 5.6. Single NAL Unit Packet..................................21 102 5.7. Aggregation Packets.....................................22 103 Table 4. Type field for STAPs and MTAPs........................23 104 5.7.1. Single-Time Aggregation Packet.....................24 105 5.7.2. Multi-Time Aggregation Packets (MTAPs).............26 106 5.7.3. Fragmentation Units (FUs)..........................30 107 6. Packetization Rules..........................................34 108 6.1. Common Packetization Rules..............................34 109 6.2. Single NAL Unit Mode....................................35 110 6.3. Non-Interleaved Mode....................................35 111 6.4. Interleaved Mode........................................36 112 7. De-Packetization Process.....................................36 113 7.1. Single NAL Unit and Non-Interleaved Mode................36 114 7.2. Interleaved Mode........................................37 115 7.2.1. Size of the De-interleaving Buffer.................37 116 7.2.2. De-interleaving Process............................38 117 7.3. Additional De-Packetization Guidelines..................39 118 8. Payload Format Parameters....................................40 119 8.1. Media Type Registration.................................40 120 8.2. SDP Parameters..........................................59 121 8.2.1. Mapping of Payload Type Parameters to SDP..........59 122 8.2.2. Usage with the SDP Offer/Answer Model..............60 123 8.2.3. Usage in Declarative Session Descriptions..........69 124 8.3. Examples................................................71 125 Offer SDP:......................................................76 126 Answer SDP:.....................................................77 127 8.4. Parameter Set Considerations............................78 128 8.5. Decoder Refresh Point Procedure using In-Band Transport of 129 Parameter Sets (Informative).................................81 130 8.5.1. IDR Procedure to Respond to a Request for a Decoder 131 Refresh Point.............................................81 132 8.5.2. Gradual Recovery Procedure to Respond to a Request for 133 a Decoder Refresh Point...................................82 134 9. Security Considerations......................................82 135 10. Congestion Control..........................................83 136 11. IANA Consideration..........................................84 137 12. Informative Appendix: Application Examples..................84 138 12.1. Video Telephony according to ITU-T Recommendation H.241 139 Annex A......................................................84 140 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 141 Aggregation..................................................85 142 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 143 Aggregation..................................................85 144 12.4. Video Telephony with Data Partitioning.................86 145 12.5. Video Telephony or Streaming with FUs and Forward Error 146 Correction...................................................86 147 12.6. Low Bit-Rate Streaming.................................89 148 12.7. Robust Packet Scheduling in Video Streaming............90 149 13. Informative Appendix: Rationale for Decoding Order Number...91 150 13.1. Introduction...........................................91 151 13.2. Example of Multi-Picture Slice Interleaving............91 152 13.3. Example of Robust Packet Scheduling....................92 153 13.4. Robust Transmission Scheduling of Redundant Coded Slices96 154 13.5. Remarks on Other Design Possibilities..................97 155 14. Acknowledgements............................................98 156 15. References..................................................98 157 15.1. Normative References...................................98 158 15.2. Informative References.................................99 159 16. Authors' Addresses.........................................101 160 Phone: +1-908-541-3518.........................................101 161 Phone: +972-545481099..........................................101 162 Tom Kristensen.................................................101 163 Phone: +47 67125125............................................101 164 Phone: +1-215-354-5166.........................................101 165 Email: rjesup@wgate.com........................................101 166 17. Backward Compatibility to RFC 3984.........................101 167 18. Changes from RFC 3984......................................103 169 1. Introduction 171 This memo specifies an RTP payload specification for the video 172 coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC 173 International Standard 14496 Part 10 [2] (both also known as 174 Advanced Video Coding, or AVC). In this memo the name H.264 is 175 used for the codec and the standard, but the memo is equally 176 applicable to the ISO/IEC counterpart of the coding standard. 178 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 179 in section 18. Issues on backward compatibility to RFC 3984 are 180 discussed in section 17. 182 1.1. The H.264 Codec 184 The H.264 video codec has a very broad application range that 185 covers all forms of digital compressed video, from low bit-rate 186 Internet streaming applications to HDTV broadcast and Digital 187 Cinema applications with nearly lossless coding. Compared to the 188 current state of technology, the overall performance of H.264 is 189 such that bit rate savings of 50% or more are reported. Digital 190 Satellite TV quality, for example, was reported to be achievable at 191 1.5 Mbit/s, compared to the current operation point of MPEG 2 video 192 at around 3.5 Mbit/s [10]. 194 The codec specification [1] itself distinguishes conceptually 195 between a video coding layer (VCL) and a network abstraction layer 196 (NAL). The VCL contains the signal processing functionality of the 197 codec; mechanisms such as transform, quantization, and motion 198 compensated prediction; and a loop filter. It follows the general 199 concept of most of today's video codecs, a macroblock-based coder 200 that uses inter picture prediction with motion compensation and 201 transform coding of the residual signal. The VCL encoder outputs 202 slices: a bit string that contains the macroblock data of an 203 integer number of macroblocks, and the information of the slice 204 header (containing the spatial address of the first macroblock in 205 the slice, the initial quantization parameter, and similar 206 information). Macroblocks in slices are arranged in scan order 207 unless a different macroblock allocation is specified, by using the 208 so-called Flexible Macroblock Ordering syntax. In-picture 209 prediction is used only within a slice. More information is 210 provided in [10]. 212 The Network Abstraction Layer (NAL) encoder encapsulates the slice 213 output of the VCL encoder into Network Abstraction Layer Units (NAL 214 units), which are suitable for transmission over packet networks or 215 use in packet oriented multiplex environments. Annex B of H.264 216 defines an encapsulation process to transmit such NAL units over 217 byte-stream oriented networks. In the scope of this memo, Annex B 218 is not relevant. 220 Internally, the NAL uses NAL units. A NAL unit consists of a one- 221 byte header and the payload byte string. The header indicates the 222 type of the NAL unit, the (potential) presence of bit errors or 223 syntax violations in the NAL unit payload, and information 224 regarding the relative importance of the NAL unit for the decoding 225 process. This RTP payload specification is designed to be unaware 226 of the bit string in the NAL unit payload. 228 One of the main properties of H.264 is the complete decoupling of 229 the transmission time, the decoding time, and the sampling or 230 presentation time of slices and pictures. The decoding process 231 specified in H.264 is unaware of time, and the H.264 syntax does 232 not carry information such as the number of skipped frames (as is 233 common in the form of the Temporal Reference in earlier video 234 compression standards). Also, there are NAL units that affect many 235 pictures and that are, therefore, inherently timeless. For this 236 reason, the handling of the RTP timestamp requires some special 237 considerations for NAL units for which the sampling or presentation 238 time is not defined or, at transmission time, unknown. 240 1.2. Parameter Set Concept 242 One very fundamental design concept of H.264 is to generate self- 243 contained packets, to make mechanisms such as the header 244 duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension 245 Code (HEC) [12] unnecessary. This was achieved by decoupling 246 information relevant to more than one slice from the media stream. 247 This higher layer meta information should be sent reliably, 248 asynchronously, and in advance from the RTP packet stream that 249 contains the slice packets. (Provisions for sending this 250 information in-band are also available for applications that do not 251 have an out-of-band transport channel appropriate for the purpose.) 252 The combination of the higher-level parameters is called a 253 parameter set. The H.264 specification includes two types of 254 parameter sets: sequence parameter set and picture parameter set. 255 An active sequence parameter set remains unchanged throughout a 256 coded video sequence, and an active picture parameter set remains 257 unchanged within a coded picture. The sequence and picture 258 parameter set structures contain information such as picture size, 259 optional coding modes employed, and macroblock to slice group map. 261 To be able to change picture parameters (such as the picture size) 262 without having to transmit parameter set updates synchronously to 263 the slice packet stream, the encoder and decoder can maintain a 264 list of more than one sequence and picture parameter set. Each 265 slice header contains a codeword that indicates the sequence and 266 picture parameter set to be used. 268 This mechanism allows the decoupling of the transmission of 269 parameter sets from the packet stream, and the transmission of them 270 by external means (e.g., as a side effect of the capability 271 exchange), or through a (reliable or unreliable) control protocol. 272 It may even be possible that they are never transmitted but are 273 fixed by an application design specification. 275 1.3. Network Abstraction Layer Unit Types 277 Tutorial information on the NAL design can be found in [13], [14], 278 and [15]. 280 All NAL units consist of a single NAL unit type octet, which also 281 co-serves as the payload header of this RTP payload format. The 282 payload of a NAL unit follows immediately. 284 The syntax and semantics of the NAL unit type octet are specified 285 in [1], but the essential properties of the NAL unit type octet are 286 summarized below. The NAL unit type octet has the following format: 288 +---------------+ 289 |0|1|2|3|4|5|6|7| 290 +-+-+-+-+-+-+-+-+ 291 |F|NRI| Type | 292 +---------------+ 294 The semantics of the components of the NAL unit type octet, as 295 specified in the H.264 specification, are described briefly below. 297 F: 1 bit 298 forbidden_zero_bit. The H.264 specification declares a value of 299 1 as a syntax violation. 301 NRI: 2 bits 302 nal_ref_idc. A value of 00 indicates that the content of the 303 NAL unit is not used to reconstruct reference pictures for inter 304 picture prediction. Such NAL units can be discarded without 305 risking the integrity of the reference pictures. Values greater 306 than 00 indicate that the decoding of the NAL unit is required 307 to maintain the integrity of the reference pictures. 309 Type: 5 bits 310 nal_unit_type. This component specifies the NAL unit payload 311 type as defined in Table 7-1 of [1], and later within this memo. 312 For a reference of all currently defined NAL unit types and 313 their semantics, please refer to section 7.4.1 in [1]. 315 This memo introduces new NAL unit types, which are presented in 316 section 5.2. The NAL unit types defined in this memo are marked as 317 unspecified in [1]. Moreover, this specification extends the 318 semantics of F and NRI as described in section 5.3. 320 2. Conventions 322 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 323 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 324 this document are to be interpreted as described in RFC 2119 [4]. 326 This specification uses the notion of setting and clearing a bit 327 when bit fields are handled. Setting a bit is the same as 328 assigning that bit the value of 1 (On). Clearing a bit is the same 329 as assigning that bit the value of 0 (Off). 331 3. Scope 333 This payload specification can only be used to carry the "naked" 334 H.264 NAL unit stream over RTP, and not the bitstream format 335 discussed in Annex B of H.264. Likely, the first applications of 336 this specification will be in the conversational multimedia field, 337 video telephony or video conferencing, but the payload format also 338 covers other applications, such as Internet streaming and TV over 339 IP. 341 4. Definitions and Abbreviations 343 4.1. Definitions 345 This document uses the definitions of [1]. The following terms, 346 defined in [1], are summed up for convenience: 348 access unit: A set of NAL units always containing a primary 349 coded picture. In addition to the primary coded picture, an 350 access unit may also contain one or more redundant coded 351 pictures or other NAL units not containing slices or slice data 352 partitions of a coded picture. The decoding of an access unit 353 always results in a decoded picture. 355 coded video sequence: A sequence of access units that consists, 356 in decoding order, of an instantaneous decoding refresh (IDR) 357 access unit followed by zero or more non-IDR access units 358 including all subsequent access units up to but not including 359 any subsequent IDR access unit. 361 IDR access unit: An access unit in which the primary coded 362 picture is an IDR picture. 364 IDR picture: A coded picture containing only slices with I or SI 365 slice types that causes a "reset" in the decoding process. 366 After the decoding of an IDR picture, all following coded 367 pictures in decoding order can be decoded without inter 368 prediction from any picture decoded prior to the IDR picture. 370 primary coded picture: The coded representation of a picture to 371 be used by the decoding process for a bitstream conforming to 372 H.264. The primary coded picture contains all macroblocks of 373 the picture. 375 redundant coded picture: A coded representation of a picture or 376 a part of a picture. The content of a redundant coded picture 377 shall not be used by the decoding process for a bitstream 378 conforming to H.264. The content of a redundant coded picture 379 may be used by the decoding process for a bitstream that 380 contains errors or losses. 382 VCL NAL unit: A collective term used to refer to coded slice and 383 coded data partition NAL units. 385 In addition, the following definitions apply: 387 decoding order number (DON): A field in the payload structure or 388 a derived variable indicating NAL unit decoding order. Values 389 of DON are in the range of 0 to 65535, inclusive. After 390 reaching the maximum value, the value of DON wraps around to 0. 392 NAL unit decoding order: A NAL unit order that conforms to the 393 constraints on NAL unit order given in section 7.4.1.2 in [1]. 395 NALU-time: The value that the RTP timestamp would have if the 396 NAL unit would be transported in its own RTP packet. 398 transmission order: The order of packets in ascending RTP 399 sequence number order (in modulo arithmetic). Within an 400 aggregation packet, the NAL unit transmission order is the same 401 as the order of appearance of NAL units in the packet. 403 media aware network element (MANE): A network element, such as a 404 middlebox or application layer gateway that is capable of 405 parsing certain aspects of the RTP payload headers or the RTP 406 payload and reacting to the contents. 408 Informative note: The concept of a MANE goes beyond normal 409 routers or gateways in that a MANE has to be aware of the 410 signaling (e.g., to learn about the payload type mappings of 411 the media streams), and in that it has to be trusted when 412 working with SRTP. The advantage of using MANEs is that they 413 allow packets to be dropped according to the needs of the 414 media coding. For example, if a MANE has to drop packets due 415 to congestion on a certain link, it can identify and remove 416 those packets whose elimination produces the least adverse 417 effect on the user experience. 419 static macroblock: A certain amount of macroblocks in the video 420 stream can be defined as static, as defined in section 8.3.2.8 421 in [3]. Static macroblocks free up additional processing 422 cycles for the handling of non-static macroblocks. Based on a 423 given amount of video processing resources and a given 424 resolution, a higher number of static macroblocks enables a 425 correspondingly higher frame rate. 427 default sub-profile: The subset of coding tools, which may be 428 all coding tools of one profile or the common subset of coding 429 tools of more than one profile, indicated by the profile-level- 430 id parameter. 432 default level: The level indicated by the profile-level-id 433 parameter, which consists of three octets, profile_idc, profile- 434 iop, and level_idc. The default level is indicated by level_idc 435 in most cases, and, in some cases, additionally by profile-iop. 437 4.2. Abbreviations 439 DON: Decoding Order Number 440 DONB: Decoding Order Number Base 441 DOND: Decoding Order Number Difference 442 FEC: Forward Error Correction 443 FU: Fragmentation Unit 444 IDR: Instantaneous Decoding Refresh 445 IEC: International Electrotechnical Commission 446 ISO: International Organization for Standardization 447 ITU-T: International Telecommunication Union, 448 Telecommunication Standardization Sector 449 MANE: Media Aware Network Element 450 MTAP: Multi-Time Aggregation Packet 451 MTAP16: MTAP with 16-bit timestamp offset 452 MTAP24: MTAP with 24-bit timestamp offset 453 NAL: Network Abstraction Layer 454 NALU: NAL Unit 455 SAR: Sample Aspect Ratio 456 SEI: Supplemental Enhancement Information 457 STAP: Single-Time Aggregation Packet 458 STAP-A: STAP type A 459 STAP-B: STAP type B 460 TS: Timestamp 461 VCL: Video Coding Layer 462 VUI: Video Usability Information 464 5. RTP Payload Format 466 5.1. RTP Header Usage 468 The format of the RTP header is specified in RFC 3550 [5] and 469 reprinted in Figure 1 for convenience. This payload format uses 470 the fields of the header in a manner consistent with that 471 specification. 473 When one NAL unit is encapsulated per RTP packet, the RECOMMENDED 474 RTP payload format is specified in section 5.6. The RTP payload 475 (and the settings for some RTP header bits) for aggregation packets 476 and fragmentation units are specified in sections 5.7 and 5.8, 477 respectively. 479 0 1 2 3 480 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 481 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 482 |V=2|P|X| CC |M| PT | sequence number | 483 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 484 | timestamp | 485 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 | synchronization source (SSRC) identifier | 487 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 488 | contributing source (CSRC) identifiers | 489 | .... | 490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 492 Figure 1 RTP header according to RFC 3550 494 The RTP header information to be set according to this RTP payload 495 format is set as follows: 497 Marker bit (M): 1 bit 498 Set for the very last packet of the access unit indicated by the 499 RTP timestamp, in line with the normal use of the M bit in video 500 formats, to allow an efficient playout buffer handling. For 501 aggregation packets (STAP and MTAP), the marker bit in the RTP 502 header MUST be set to the value that the marker bit of the last 503 NAL unit of the aggregation packet would have been if it were 504 transported in its own RTP packet. Decoders MAY use this bit as 505 an early indication of the last packet of an access unit, but 506 MUST NOT rely on this property. 508 Informative note: Only one M bit is associated with an 509 aggregation packet carrying multiple NAL units. Thus, if a 510 gateway has re-packetized an aggregation packet into several 511 packets, it cannot reliably set the M bit of those packets. 513 Payload type (PT): 7 bits 514 The assignment of an RTP payload type for this new packet format 515 is outside the scope of this document and will not be specified 516 here. The assignment of a payload type has to be performed 517 either through the profile used or in a dynamic way. 519 Sequence number (SN): 16 bits 520 Set and used in accordance with RFC 3550. For the single NALU 521 and non-interleaved packetization mode, the sequence number is 522 used to determine decoding order for the NALU. 524 Timestamp: 32 bits 525 The RTP timestamp is set to the sampling timestamp of the 526 content. A 90 kHz clock rate MUST be used. 528 If the NAL unit has no timing properties of its own (e.g., 529 parameter set and SEI NAL units), the RTP timestamp is set to 530 the RTP timestamp of the primary coded picture of the access 531 unit in which the NAL unit is included, according to section 532 7.4.1.2 of [1]. 534 The setting of the RTP Timestamp for MTAPs is defined in section 535 5.7.2. 537 Receivers SHOULD ignore any picture timing SEI messages included 538 in access units that have only one display timestamp. Instead, 539 receivers SHOULD use the RTP timestamp for synchronizing the 540 display process. 542 If one access unit has more than one display timestamp carried 543 in a picture timing SEI message, then the information in the SEI 544 message SHOULD be treated as relative to the RTP timestamp, with 545 the earliest event occurring at the time given by the RTP 546 timestamp, and subsequent events later, as given by the 547 difference in SEI message picture timing values. Let tSEI1, 548 tSEI2, ..., tSEIn be the display timestamps carried in the SEI 549 message of an access unit, where tSEI1 is the earliest of all 550 such timestamps. Let tmadjst() be a function that adjusts the 551 SEI messages time scale to a 90-kHz time scale. Let TS be the 552 RTP timestamp. Then, the display time for the event associated 553 with tSEI1 is TS. The display time for the event with tSEIx, 554 where x is [2..n] is TS + tmadjst (tSEIx - tSEI1). 556 Informative note: Displaying coded frames as fields is needed 557 commonly in an operation known as 3:2 pulldown, in which film 558 content that consists of coded frames is displayed on a 559 display using interlaced scanning. The picture timing SEI 560 message enables carriage of multiple timestamps for the same 561 coded picture, and therefore the 3:2 pulldown process is 562 perfectly controlled. The picture timing SEI message 563 mechanism is necessary because only one timestamp per coded 564 frame can be conveyed in the RTP timestamp. 566 5.2. Payload Structures 568 The payload format defines three different basic payload structures. 569 A receiver can identify the payload structure by the first byte of 570 the RTP packet payload, which co-serves as the RTP payload header 571 and, in some cases, as the first byte of the payload. This byte is 572 always structured as a NAL unit header. The NAL unit type field 573 indicates which structure is present. The possible structures are 574 as follows: 576 Single NAL Unit Packet: Contains only a single NAL unit in the 577 payload. The NAL header type field will be equal to the original 578 NAL unit type; i.e., in the range of 1 to 23, inclusive. Specified 579 in section 5.6. 581 Aggregation Packet: Packet type used to aggregate multiple NAL 582 units into a single RTP payload. This packet exists in four 583 versions, the Single-Time Aggregation Packet type A (STAP-A), the 584 Single-Time Aggregation Packet type B (STAP-B), Multi-Time 585 Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and Multi- 586 Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The 587 NAL unit type numbers assigned for STAP-A, STAP-B, MTAP16, and 588 MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 589 5.7. 591 Fragmentation Unit: Used to fragment a single NAL unit over 592 multiple RTP packets. Exists with two versions, FU-A and FU-B, 593 identified with the NAL unit type numbers 28 and 29, respectively. 594 Specified in section 5.8. 596 Informative note: This specification does not limit the size of 597 NAL units encapsulated in single NAL unit packets and 598 fragmentation units. The maximum size of a NAL unit 599 encapsulated in any aggregation packet is 65535 bytes. 601 Table 1 summarizes NAL unit types and the corresponding RTP packet 602 types when each of these NAL units is directly used as a packet 603 payload, and where the types are described in this memo. 605 Table 1. Summary of NAL unit types and the corresponding packet 606 types 608 NAL Unit Packet Packet Type Name Section 609 Type Type 610 --------------------------------------------------------- 611 0 reserved - 612 1-23 NAL unit Single NAL unit packet 5.6 613 24 STAP-A Single-time aggregation packet 5.7.1 614 25 STAP-B Single-time aggregation packet 5.7.1 615 26 MTAP16 Multi-time aggregation packet 5.7.2 616 27 MTAP24 Multi-time aggregation packet 5.7.2 617 28 FU-A Fragmentation unit 5.8 618 29 FU-B Fragmentation unit 5.8 619 30-31 reserved - 621 5.3. NAL Unit Header Usage 623 The structure and semantics of the NAL unit header were introduced 624 in section 1.3. For convenience, the format of the NAL unit header 625 is reprinted below: 627 +---------------+ 628 |0|1|2|3|4|5|6|7| 629 +-+-+-+-+-+-+-+-+ 630 |F|NRI| Type | 631 +---------------+ 633 This section specifies the semantics of F and NRI according to this 634 specification. 636 F: 1 bit 637 forbidden_zero_bit. A value of 0 indicates that the NAL unit 638 type octet and payload should not contain bit errors or other 639 syntax violations. A value of 1 indicates that the NAL unit 640 type octet and payload may contain bit errors or other syntax 641 violations. 643 MANEs SHOULD set the F bit to indicate detected bit errors in 644 the NAL unit. The H.264 specification requires that the F bit 645 is equal to 0. When the F bit is set, the decoder is advised 646 that bit errors or any other syntax violations may be present in 647 the payload or in the NAL unit type octet. The simplest decoder 648 reaction to a NAL unit in which the F bit is equal to 1 is to 649 discard such a NAL unit and to conceal the lost data in the 650 discarded NAL unit. 652 NRI: 2 bits 653 nal_ref_idc. The semantics of value 00 and a non-zero value 654 remain unchanged from the H.264 specification. In other words, 655 a value of 00 indicates that the content of the NAL unit is not 656 used to reconstruct reference pictures for inter picture 657 prediction. Such NAL units can be discarded without risking the 658 integrity of the reference pictures. Values greater than 00 659 indicate that the decoding of the NAL unit is required to 660 maintain the integrity of the reference pictures. 662 In addition to the specification above, according to this RTP 663 payload specification, values of NRI indicate the relative 664 transport priority, as determined by the encoder. MANEs can use 665 this information to protect more important NAL units better than 666 they do less important NAL units. The highest transport 667 priority is 11, followed by 10, and then by 01; finally, 00 is 668 the lowest. 670 Informative note: Any non-zero value of NRI is handled 671 identically in H.264 decoders. Therefore, receivers need not 672 manipulate the value of NRI when passing NAL units to the 673 decoder. 675 An H.264 encoder MUST set the value of NRI according to the 676 H.264 specification (subclause 7.4.1) when the value of 677 nal_unit_type is in the range of 1 to 12, inclusive. In 678 particular, the H.264 specification requires that the value of 679 NRI SHALL be equal to 0 for all NAL units having nal_unit_type 680 equal to 6, 9, 10, 11, or 12. 682 For NAL units having nal_unit_type equal to 7 or 8 (indicating a 683 sequence parameter set or a picture parameter set, respectively), 684 an H.264 encoder SHOULD set the value of NRI to 11 (in binary 685 format). For coded slice NAL units of a primary coded picture 686 having nal_unit_type equal to 5 (indicating a coded slice 687 belonging to an IDR picture), an H.264 encoder SHOULD set the 688 value of NRI to 11 (in binary format). 690 For a mapping of the remaining nal_unit_types to NRI values, the 691 following example MAY be used and has been shown to be efficient 692 in a certain environment [14]. Other mappings MAY also be 693 desirable, depending on the application and the H.264/AVC Annex 694 A profile in use. 696 Informative note: Data Partitioning is not available in 697 certain profiles; e.g., in the Main or Baseline profiles. 699 Consequently, the NAL unit types 2, 3, and 4 can occur only 700 if the video bitstream conforms to a profile in which data 701 partitioning is allowed and not in streams that conform to 702 the Main or Baseline profiles. 704 Table 2. Example of NRI values for coded slices and coded slice 705 data partitions of primary coded reference pictures 707 NAL Unit Type Content of NAL unit NRI (binary) 708 ---------------------------------------------------------------- 709 1 non-IDR coded slice 10 710 2 Coded slice data partition A 10 711 3 Coded slice data partition B 01 712 4 Coded slice data partition C 01 714 Informative note: As mentioned before, the NRI value of non- 715 reference pictures is 00 as mandated by H.264/AVC. 717 An H.264 encoder SHOULD set the value of NRI for coded slice and 718 coded slice data partition NAL units of redundant coded 719 reference pictures equal to 01 (in binary format). 721 Definitions of the values for NRI for NAL unit types 24 to 29, 722 inclusive, are given in sections 5.7 and 5.8 of this memo. 724 No recommendation for the value of NRI is given for NAL units 725 having nal_unit_type in the range of 13 to 23, inclusive, 726 because these values are reserved for ITU-T and ISO/IEC. No 727 recommendation for the value of NRI is given for NAL units 728 having nal_unit_type equal to 0 or in the range of 30 to 31, 729 inclusive, as the semantics of these values are not specified in 730 this memo. 732 5.4. Packetization Modes 734 This memo specifies three cases of packetization modes: 736 o Single NAL unit mode 738 o Non-interleaved mode 740 o Interleaved mode 742 The single NAL unit mode is targeted for conversational systems 743 that comply with ITU-T Recommendation H.241 [3] (see section 12.1). 744 The non-interleaved mode is targeted for conversational systems 745 that may not comply with ITU-T Recommendation H.241. In the non- 746 interleaved mode, NAL units are transmitted in NAL unit decoding 747 order. The interleaved mode is targeted for systems that do not 748 require very low end-to-end latency. The interleaved mode allows 749 transmission of NAL units out of NAL unit decoding order. 751 The packetization mode in use MAY be signaled by the value of the 752 OPTIONAL packetization-mode media type parameter. The used 753 packetization mode governs which NAL unit types are allowed in RTP 754 payloads. Table 3 summarizes the allowed packet payload types for 755 each packetization mode. Packetization modes are explained in more 756 detail in section 6. 758 Table 3. Summary of allowed NAL unit types for each packetization 759 mode (yes = allowed, no = disallowed, ig = ignore) 761 Payload Packet Single NAL Non-Interleaved Interleaved 762 Type Type Unit Mode Mode Mode 763 ------------------------------------------------------------- 764 0 reserved ig ig ig 765 1-23 NAL unit yes yes no 766 24 STAP-A no yes no 767 25 STAP-B no no yes 768 26 MTAP16 no no yes 769 27 MTAP24 no no yes 770 28 FU-A no yes yes 771 29 FU-B no no yes 772 30-31 reserved ig ig ig 774 Some NAL unit or payload type values (indicated as reserved in 775 Table 3) are reserved for future extensions. NAL units of those 776 types SHOULD NOT be sent by a sender (direct as packet payloads, or 777 as aggregation units in aggregation packets, or as fragmented units 778 in FU packets) and MUST be ignored by a receiver. For example, the 779 payload types 1-23, with the associated packet type "NAL unit", are 780 allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", 781 but disallowed in "Interleaved Mode". However, NAL units of NAL 782 unit types 1-23 can be used in "Interleaved Mode" as aggregation 783 units in STAP-B, MTAP16 and MTAP14 packets as well as fragmented 784 units in FU-A and FU-B packets. Similarly, NAL units of NAL unit 785 types 1-23 can also be used in the "Non-Interleaved Mode" as 786 aggregation units in STAP-A packets or fragmented units in FU-A 787 packets, in addition to being directly used as packet payloads. 789 5.5. Decoding Order Number (DON) 791 In the interleaved packetization mode, the transmission order of 792 NAL units is allowed to differ from the decoding order of the NAL 793 units. Decoding order number (DON) is a field in the payload 794 structure or a derived variable that indicates the NAL unit 795 decoding order. Rationale and examples of use cases for 796 transmission out of decoding order and for the use of DON are given 797 in section 13. 799 The coupling of transmission and decoding order is controlled by 800 the OPTIONAL sprop-interleaving-depth media type parameter as 801 follows. When the value of the OPTIONAL sprop-interleaving-depth 802 media type parameter is equal to 0 (explicitly or per default), the 803 transmission order of NAL units MUST conform to the NAL unit 804 decoding order. When the value of the OPTIONAL sprop-interleaving- 805 depth media type parameter is greater than 0, 807 o the order of NAL units in an MTAP16 and an MTAP24 is not 808 required to be the NAL unit decoding order, and 810 o the order of NAL units generated by de-packetizing STAP-Bs, 811 MTAPs, and FUs in two consecutive packets is not required to be 812 the NAL unit decoding order. 814 The RTP payload structures for a single NAL unit packet, an STAP-A, 815 and an FU-A do not include DON. STAP-B and FU-B structures include 816 DON, and the structure of MTAPs enables derivation of DON as 817 specified in section 5.7.2. 819 Informative note: When an FU-A occurs in interleaved mode, it 820 always follows an FU-B, which sets its DON. 822 Informative note: If a transmitter wants to encapsulate a single 823 NAL unit per packet and transmit packets out of their decoding 824 order, STAP-B packet type can be used. 826 In the single NAL unit packetization mode, the transmission order 827 of NAL units, determined by the RTP sequence number, MUST be the 828 same as their NAL unit decoding order. In the non-interleaved 829 packetization mode, the transmission order of NAL units in single 830 NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL 831 unit decoding order. The NAL units within an STAP MUST appear in 832 the NAL unit decoding order. Thus, the decoding order is first 833 provided through the implicit order within a STAP, and second 834 provided through the RTP sequence number for the order between 835 STAPs, FUs, and single NAL unit packets. 837 Signaling of the value of DON for NAL units carried in STAP-B, MTAP, 838 and a series of fragmentation units starting with an FU-B is 839 specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON 840 value of the first NAL unit in transmission order MAY be set to any 841 value. Values of DON are in the range of 0 to 65535, inclusive. 842 After reaching the maximum value, the value of DON wraps around to 843 0. 845 The decoding order of two NAL units contained in any STAP-B, MTAP, 846 or a series of fragmentation units starting with an FU-B is 847 determined as follows. Let DON(i) be the decoding order number of 848 the NAL unit having index i in the transmission order. Function 849 don_diff(m,n) is specified as follows: 851 If DON(m) == DON(n), don_diff(m,n) = 0 853 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 854 don_diff(m,n) = DON(n) - DON(m) 856 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 857 don_diff(m,n) = 65536 - DON(m) + DON(n) 859 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 860 don_diff(m,n) = - (DON(m) + 65536 - DON(n)) 862 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 863 don_diff(m,n) = - (DON(m) - DON(n)) 865 A positive value of don_diff(m,n) indicates that the NAL unit 866 having transmission order index n follows, in decoding order, the 867 NAL unit having transmission order index m. When don_diff(m,n) is 868 equal to 0, then the NAL unit decoding order of the two NAL units 869 can be in either order. A negative value of don_diff(m,n) 870 indicates that the NAL unit having transmission order index n 871 precedes, in decoding order, the NAL unit having transmission order 872 index m. 874 Values of DON related fields (DON, DONB, and DOND; see section 5.7) 875 MUST be such that the decoding order determined by the values of 876 DON, as specified above, conforms to the NAL unit decoding order. 877 If the order of two NAL units in NAL unit decoding order is 878 switched and the new order does not conform to the NAL unit 879 decoding order, the NAL units MUST NOT have the same value of DON. 881 If the order of two consecutive NAL units in the NAL unit stream is 882 switched and the new order still conforms to the NAL unit decoding 883 order, the NAL units MAY have the same value of DON. For example, 884 when arbitrary slice order is allowed by the video coding profile 885 in use, all the coded slice NAL units of a coded picture are 886 allowed to have the same value of DON. Consequently, NAL units 887 having the same value of DON can be decoded in any order, and two 888 NAL units having a different value of DON should be passed to the 889 decoder in the order specified above. When two consecutive NAL 890 units in the NAL unit decoding order have a different value of DON, 891 the value of DON for the second NAL unit in decoding order SHOULD 892 be the value of DON for the first, incremented by one. 894 An example of the de-packetization process to recover the NAL unit 895 decoding order is given in section 7. 897 Informative note: Receivers should not expect that the absolute 898 difference of values of DON for two consecutive NAL units in the 899 NAL unit decoding order will be equal to one, even in error-free 900 transmission. An increment by one is not required, as at the 901 time of associating values of DON to NAL units, it may not be 902 known whether all NAL units are delivered to the receiver. For 903 example, a gateway may not forward coded slice NAL units of non- 904 reference pictures or SEI NAL units when there is a shortage of 905 bit rate in the network to which the packets are forwarded. In 906 another example, a live broadcast is interrupted by pre-encoded 907 content, such as commercials, from time to time. The first 908 intra picture of a pre-encoded clip is transmitted in advance to 909 ensure that it is readily available in the receiver. When 910 transmitting the first intra picture, the originator does not 911 exactly know how many NAL units will be encoded before the first 912 intra picture of the pre-encoded clip follows in decoding order. 913 Thus, the values of DON for the NAL units of the first intra 914 picture of the pre-encoded clip have to be estimated when they 915 are transmitted, and gaps in values of DON may occur. 917 5.6. Single NAL Unit Packet 919 The single NAL unit packet defined here MUST contain only one NAL 920 unit, of the types defined in [1]. This means that neither an 921 aggregation packet nor a fragmentation unit can be used within a 922 single NAL unit packet. A NAL unit stream composed by de- 923 packetizing single NAL unit packets in RTP sequence number order 924 MUST conform to the NAL unit decoding order. The structure of the 925 single NAL unit packet is shown in Figure 2. 927 Informative note: The first byte of a NAL unit co-serves as the 928 RTP payload header. 930 0 1 2 3 931 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 932 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 933 |F|NRI| Type | | 934 +-+-+-+-+-+-+-+-+ | 935 | | 936 | Bytes 2..n of a Single NAL unit | 937 | | 938 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 939 | :...OPTIONAL RTP padding | 940 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 942 Figure 2 RTP payload format for single NAL unit packet 944 5.7. Aggregation Packets 946 Aggregation packets are the NAL unit aggregation scheme of this 947 payload specification. The scheme is introduced to reflect the 948 dramatically different MTU sizes of two key target networks: 949 wireline IP networks (with an MTU size that is often limited by the 950 Ethernet MTU size; roughly 1500 bytes), and IP or non-IP (e.g., 951 ITU-T H.324/M) based wireless communication systems with preferred 952 transmission unit sizes of 254 bytes or less. To prevent media 953 transcoding between the two worlds, and to avoid undesirable 954 packetization overhead, a NAL unit aggregation scheme is introduced. 956 Two types of aggregation packets are defined by this specification: 958 o Single-time aggregation packet (STAP): aggregates NAL units with 959 identical NALU-time. Two types of STAPs are defined, one 960 without DON (STAP-A) and another including DON (STAP-B). 962 o Multi-time aggregation packet (MTAP): aggregates NAL units with 963 potentially differing NALU-time. Two different MTAPs are 964 defined, differing in the length of the NAL unit timestamp 965 offset. 967 Each NAL unit to be carried in an aggregation packet is 968 encapsulated in an aggregation unit. Please see below for the four 969 different aggregation units and their characteristics. 971 The structure of the RTP payload format for aggregation packets is 972 presented in Figure 3. 974 0 1 2 3 975 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 976 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 977 |F|NRI| Type | | 978 +-+-+-+-+-+-+-+-+ | 979 | | 980 | one or more aggregation units | 981 | | 982 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 983 | :...OPTIONAL RTP padding | 984 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 986 Figure 3 RTP payload format for aggregation packets 988 MTAPs and STAPs share the following packetization rules: The RTP 989 timestamp MUST be set to the earliest of the NALU-times of all the 990 NAL units to be aggregated. The type field of the NAL unit type 991 octet MUST be set to the appropriate value, as indicated in Table 4. 992 The F bit MUST be cleared if all F bits of the aggregated NAL units 993 are zero; otherwise, it MUST be set. The value of NRI MUST be the 994 maximum of all the NAL units carried in the aggregation packet. 996 Table 4. Type field for STAPs and MTAPs 998 Type Packet Timestamp offset DON related fields 999 field length (DON, DONB, DOND) 1000 (in bits) present 1001 -------------------------------------------------------- 1002 24 STAP-A 0 no 1003 25 STAP-B 0 yes 1004 26 MTAP16 16 yes 1005 27 MTAP24 24 yes 1007 The marker bit in the RTP header is set to the value that the 1008 marker bit of the last NAL unit of the aggregated packet would have 1009 if it were transported in its own RTP packet. 1011 The payload of an aggregation packet consists of one or more 1012 aggregation units. See sections 5.7.1 and 5.7.2 for the four 1013 different types of aggregation units. An aggregation packet can 1014 carry as many aggregation units as necessary; however, the total 1015 amount of data in an aggregation packet obviously MUST fit into an 1016 IP packet, and the size SHOULD be chosen so that the resulting IP 1017 packet is smaller than the MTU size. An aggregation packet MUST 1018 NOT contain fragmentation units specified in section 5.8. 1020 Aggregation packets MUST NOT be nested; i.e., an aggregation packet 1021 MUST NOT contain another aggregation packet. 1023 5.7.1. Single-Time Aggregation Packet 1025 Single-time aggregation packet (STAP) SHOULD be used whenever NAL 1026 units are aggregated that all share the same NALU-time. The 1027 payload of an STAP-A does not include DON and consists of at least 1028 one single-time aggregation unit, as presented in Figure 4. The 1029 payload of an STAP-B consists of a 16-bit unsigned decoding order 1030 number (DON) (in network byte order) followed by at least one 1031 single-time aggregation unit, as presented in Figure 5. 1033 0 1 2 3 1034 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1035 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1036 : | 1037 +-+-+-+-+-+-+-+-+ | 1038 | | 1039 | single-time aggregation units | 1040 | | 1041 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1042 | : 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1045 Figure 4 Payload format for STAP-A 1047 0 1 2 3 1048 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1049 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1050 : decoding order number (DON) | | 1051 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1052 | | 1053 | single-time aggregation units | 1054 | | 1055 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1056 | : 1057 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 Figure 5 Payload format for STAP-B 1061 The DON field specifies the value of DON for the first NAL unit in 1062 an STAP-B in transmission order. For each successive NAL unit in 1063 appearance order in an STAP-B, the value of DON is equal to (the 1064 value of DON of the previous NAL unit in the STAP-B + 1) % 65536, 1065 in which '%' stands for the modulo operation. 1067 A single-time aggregation unit consists of 16-bit unsigned size 1068 information (in network byte order) that indicates the size of the 1069 following NAL unit in bytes (excluding these two octets, but 1070 including the NAL unit type octet of the NAL unit), followed by the 1071 NAL unit itself, including its NAL unit type byte. A single-time 1072 aggregation unit is byte aligned within the RTP payload, but it may 1073 not be aligned on a 32-bit word boundary. Figure 6 presents the 1074 structure of the single-time aggregation unit. 1076 0 1 2 3 1077 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 : NAL unit size | | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1081 | | 1082 | NAL unit | 1083 | | 1084 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1085 | : 1086 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1088 Figure 6 Structure for single-time aggregation unit 1090 Figure 7 presents an example of an RTP packet that contains an 1091 STAP-A. The STAP contains two single-time aggregation units, 1092 labeled as 1 and 2 in the figure. 1094 0 1 2 3 1095 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 | RTP Header | 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 |STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 | NALU 1 Data | 1102 : : 1103 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 | | NALU 2 Size | NALU 2 HDR | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1106 | NALU 2 Data | 1107 : : 1108 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1109 | :...OPTIONAL RTP padding | 1110 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1112 Figure 7 An example of an RTP packet including an STAP-A containing 1113 two single-time aggregation units 1115 Figure 8 presents an example of an RTP packet that contains an 1116 STAP-B. The STAP contains two single-time aggregation units, 1117 labeled as 1 and 2 in the figure. 1119 0 1 2 3 1120 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1121 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1122 | RTP Header | 1123 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1124 |STAP-B NAL HDR | DON | NALU 1 Size | 1125 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1126 | NALU 1 Size | NALU 1 HDR | NALU 1 Data | 1127 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1128 : : 1129 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1130 | | NALU 2 Size | NALU 2 HDR | 1131 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1132 | NALU 2 Data | 1133 : : 1134 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 | :...OPTIONAL RTP padding | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1138 Figure 8 An example of an RTP packet including an STAP-B containing 1139 two single-time aggregation units 1141 5.7.2. Multi-Time Aggregation Packets (MTAPs) 1143 The NAL unit payload of MTAPs consists of a 16-bit unsigned 1144 decoding order number base (DONB) (in network byte order) and one 1145 or more multi-time aggregation units, as presented in Figure 9. 1146 DONB MUST contain the value of DON for the first NAL unit in the 1147 NAL unit decoding order among the NAL units of the MTAP. 1149 Informative note: The first NAL unit in the NAL unit decoding 1150 order is not necessarily the first NAL unit in the order in 1151 which the NAL units are encapsulated in an MTAP. 1153 0 1 2 3 1154 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1155 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1156 : decoding order number base | | 1157 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1158 | | 1159 | multi-time aggregation units | 1160 | | 1161 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1162 | : 1163 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1165 Figure 9 NAL unit payload format for MTAPs 1167 Two different multi-time aggregation units are defined in this 1168 specification. Both of them consist of 16 bits unsigned size 1169 information of the following NAL unit (in network byte order), an 1170 8-bit unsigned decoding order number difference (DOND), and n bits 1171 (in network byte order) of timestamp offset (TS offset) for this 1172 NAL unit, whereby n can be 16 or 24. The choice between the 1173 different MTAP types (MTAP16 and MTAP24) is application dependent: 1174 the larger the timestamp offset is, the higher the flexibility of 1175 the MTAP, but the overhead is also higher. 1177 The structure of the multi-time aggregation units for MTAP16 and 1178 MTAP24 are presented in Figures 10 and 11, respectively. The 1179 starting or ending position of an aggregation unit within a packet 1180 is not required to be on a 32-bit word boundary. The DON of the 1181 NAL unit contained in a multi-time aggregation unit is equal to 1182 (DONB + DOND) % 65536, in which % denotes the modulo operation. 1183 This memo does not specify how the NAL units within an MTAP are 1184 ordered, but, in most cases, NAL unit decoding order SHOULD be used. 1186 The timestamp offset field MUST be set to a value equal to the 1187 value of the following formula: If the NALU-time is larger than or 1188 equal to the RTP timestamp of the packet, then the timestamp offset 1189 equals (the NALU-time of the NAL unit - the RTP timestamp of the 1190 packet). If the NALU-time is smaller than the RTP timestamp of the 1191 packet, then the timestamp offset is equal to the NALU-time + (2^32 1192 - the RTP timestamp of the packet). 1194 0 1 2 3 1195 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1196 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1197 : NAL unit size | DOND | TS offset | 1198 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1199 | TS offset | | 1200 +-+-+-+-+-+-+-+-+ NAL unit | 1201 | | 1202 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1203 | : 1204 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1206 Figure 10 Multi-time aggregation unit for MTAP16 1208 0 1 2 3 1209 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1210 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1211 : NAL unit size | DOND | TS offset | 1212 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1213 | TS offset | | 1214 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1215 | NAL unit | 1216 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1217 | : 1218 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1220 Figure 11 Multi-time aggregation unit for MTAP24 1222 For the "earliest" multi-time aggregation unit in an MTAP the 1223 timestamp offset MUST be zero. Hence, the RTP timestamp of the 1224 MTAP itself is identical to the earliest NALU-time. 1226 Informative note: The "earliest" multi-time aggregation unit is 1227 the one that would have the smallest extended RTP timestamp 1228 among all the aggregation units of an MTAP if the NAL units 1229 contained in the aggregation units were encapsulated in single 1230 NAL unit packets. An extended timestamp is a timestamp that has 1231 more than 32 bits and is capable of counting the wraparound of 1232 the timestamp field, thus enabling one to determine the smallest 1233 value if the timestamp wraps. Such an "earliest" aggregation 1234 unit may not be the first one in the order in which the 1235 aggregation units are encapsulated in an MTAP. The "earliest" 1236 NAL unit need not be the same as the first NAL unit in the NAL 1237 unit decoding order either. 1239 Figure 12 presents an example of an RTP packet that contains a 1240 multi-time aggregation packet of type MTAP16 that contains two 1241 multi-time aggregation units, labeled as 1 and 2 in the figure. 1243 0 1 2 3 1244 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1245 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1246 | RTP Header | 1247 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1248 |MTAP16 NAL HDR | decoding order number base | NALU 1 Size | 1249 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1250 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offset | 1251 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1252 | NALU 1 HDR | NALU 1 DATA | 1253 +-+-+-+-+-+-+-+-+ + 1254 : : 1255 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1256 | | NALU 2 SIZE | NALU 2 DOND | 1257 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1258 | NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA | 1259 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1260 : : 1261 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1262 | :...OPTIONAL RTP padding | 1263 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1265 Figure 12 An RTP packet including a multi-time aggregation packet 1266 of type MTAP16 containing two multi-time aggregation units 1268 Figure 13 presents an example of an RTP packet that contains a 1269 multi-time aggregation packet of type MTAP24 that contains two 1270 multi-time aggregation units, labeled as 1 and 2 in the figure. 1272 0 1 2 3 1273 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 | RTP Header | 1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1277 |MTAP24 NAL HDR | decoding order number base | NALU 1 Size | 1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1279 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offs | 1280 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1281 |NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA | 1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1283 : : 1284 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1285 | | NALU 2 SIZE | NALU 2 DOND | 1286 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1287 | NALU 2 TS offset | NALU 2 HDR | 1288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1289 | NALU 2 DATA | 1290 : : 1291 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1292 | :...OPTIONAL RTP padding | 1293 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1295 Figure 13 An RTP packet including a multi-time aggregation packet 1296 of type MTAP24 containing two multi-time aggregation units 1298 5.7.3. Fragmentation Units (FUs) 1300 This payload type allows fragmenting a NAL unit into several RTP 1301 packets. Doing so on the application layer instead of relying on 1302 lower layer fragmentation (e.g., by IP) has the following 1303 advantages: 1305 o The payload format is capable of transporting NAL units bigger 1306 than 64 kbytes over an IPv4 network that may be present in pre- 1307 recorded video, particularly in High Definition formats (there 1308 is a limit of the number of slices per picture, which results in 1309 a limit of NAL units per picture, which may result in big NAL 1310 units). 1312 o The fragmentation mechanism allows fragmenting a single NAL unit 1313 and applying generic forward error correction as described in 1314 section 12.5. 1316 Fragmentation is defined only for a single NAL unit and not for any 1317 aggregation packets. A fragment of a NAL unit consists of an 1318 integer number of consecutive octets of that NAL unit. Each octet 1319 of the NAL unit MUST be part of exactly one fragment of that NAL 1320 unit. Fragments of the same NAL unit MUST be sent in consecutive 1321 order with ascending RTP sequence numbers (with no other RTP 1322 packets within the same RTP packet stream being sent between the 1323 first and last fragment). Similarly, a NAL unit MUST be 1324 reassembled in RTP sequence number order. 1326 When a NAL unit is fragmented and conveyed within fragmentation 1327 units (FUs), it is referred to as a fragmented NAL unit. STAPs and 1328 MTAPs MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU 1329 MUST NOT contain another FU. 1331 The RTP timestamp of an RTP packet carrying an FU is set to the 1332 NALU-time of the fragmented NAL unit. 1334 Figure 14 presents the RTP payload format for FU-As. An FU-A 1335 consists of a fragmentation unit indicator of one octet, a 1336 fragmentation unit header of one octet, and a fragmentation unit 1337 payload. 1339 0 1 2 3 1340 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1341 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1342 | FU indicator | FU header | | 1343 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1344 | | 1345 | FU payload | 1346 | | 1347 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1348 | :...OPTIONAL RTP padding | 1349 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1351 Figure 14 RTP payload format for FU-A 1353 Figure 15 presents the RTP payload format for FU-Bs. An FU-B 1354 consists of a fragmentation unit indicator of one octet, a 1355 fragmentation unit header of one octet, a decoding order number 1356 (DON) (in network byte order), and a fragmentation unit payload. 1357 In other words, the structure of FU-B is the same as the structure 1358 of FU-A, except for the additional DON field. 1360 0 1 2 3 1361 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1363 | FU indicator | FU header | DON | 1364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 1365 | | 1366 | FU payload | 1367 | | 1368 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1369 | :...OPTIONAL RTP padding | 1370 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1372 Figure 15 RTP payload format for FU-B 1374 NAL unit type FU-B MUST be used in the interleaved packetization 1375 mode for the first fragmentation unit of a fragmented NAL unit. 1376 NAL unit type FU-B MUST NOT be used in any other case. In other 1377 words, in the interleaved packetization mode, each NALU that is 1378 fragmented has an FU-B as the first fragment, followed by one or 1379 more FU-A fragments. 1381 The FU indicator octet has the following format: 1383 +---------------+ 1384 |0|1|2|3|4|5|6|7| 1385 +-+-+-+-+-+-+-+-+ 1386 |F|NRI| Type | 1387 +---------------+ 1389 Values equal to 28 and 29 in the Type field of the FU indicator 1390 octet identify an FU-A and an FU-B, respectively. The use of the F 1391 bit is described in section 5.3. The value of the NRI field MUST 1392 be set according to the value of the NRI field in the fragmented 1393 NAL unit. 1395 The FU header has the following format: 1397 +---------------+ 1398 |0|1|2|3|4|5|6|7| 1399 +-+-+-+-+-+-+-+-+ 1400 |S|E|R| Type | 1401 +---------------+ 1403 S: 1 bit 1404 When set to one, the Start bit indicates the start of a 1405 fragmented NAL unit. When the following FU payload is not the 1406 start of a fragmented NAL unit payload, the Start bit is set to 1407 zero. 1409 E: 1 bit 1410 When set to one, the End bit indicates the end of a fragmented 1411 NAL unit, i.e., the last byte of the payload is also the last 1412 byte of the fragmented NAL unit. When the following FU payload 1413 is not the last fragment of a fragmented NAL unit, the End bit 1414 is set to zero. 1416 R: 1 bit 1417 The Reserved bit MUST be equal to 0 and MUST be ignored by the 1418 receiver. 1420 Type: 5 bits 1421 The NAL unit payload type as defined in Table 7-1 of [1]. 1423 The value of DON in FU-Bs is selected as described in section 5.5. 1425 Informative note: The DON field in FU-Bs allows gateways to 1426 fragment NAL units to FU-Bs without organizing the incoming NAL 1427 units to the NAL unit decoding order. 1429 A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the 1430 Start bit and End bit MUST NOT both be set to one in the same FU 1431 header. 1433 The FU payload consists of fragments of the payload of the 1434 fragmented NAL unit so that if the fragmentation unit payloads of 1435 consecutive FUs are sequentially concatenated, the payload of the 1436 fragmented NAL unit can be reconstructed. The NAL unit type octet 1437 of the fragmented NAL unit is not included as such in the 1438 fragmentation unit payload, but rather the information of the NAL 1439 unit type octet of the fragmented NAL unit is conveyed in F and NRI 1440 fields of the FU indicator octet of the fragmentation unit and in 1441 the type field of the FU header. An FU payload MAY have any number 1442 of octets and MAY be empty. 1444 Informative note: Empty FUs are allowed to reduce the latency of 1445 a certain class of senders in nearly lossless environments. 1446 These senders can be characterized in that they packetize NALU 1447 fragments before the NALU is completely generated and, hence, 1448 before the NALU size is known. If zero-length NALU fragments 1449 were not allowed, the sender would have to generate at least one 1450 bit of data of the following fragment before the current 1451 fragment could be sent. Due to the characteristics of H.264, 1452 where sometimes several macroblocks occupy zero bits, this is 1453 undesirable and can add delay. However, the (potential) use of 1454 zero-length NALU fragments should be carefully weighed against 1455 the increased risk of the loss of at least a part of the NALU 1456 because of the additional packets employed for its transmission. 1458 If a fragmentation unit is lost, the receiver SHOULD discard all 1459 following fragmentation units in transmission order corresponding 1460 to the same fragmented NAL unit. 1462 A receiver in an endpoint or in a MANE MAY aggregate the first n-1 1463 fragments of a NAL unit to an (incomplete) NAL unit, even if 1464 fragment n of that NAL unit is not received. In this case, the 1465 forbidden_zero_bit of the NAL unit MUST be set to one to indicate a 1466 syntax violation. 1468 6. Packetization Rules 1470 The packetization modes are introduced in section 5.2. The 1471 packetization rules common to more than one of the packetization 1472 modes are specified in section 6.1. The packetization rules for 1473 the single NAL unit mode, the non-interleaved mode, and the 1474 interleaved mode are specified in sections 6.2, 6.3, and 6.4, 1475 respectively. 1477 6.1. Common Packetization Rules 1479 All senders MUST enforce the following packetization rules 1480 regardless of the packetization mode in use: 1482 o Coded slice NAL units or coded slice data partition NAL units 1483 belonging to the same coded picture (and thus sharing the same 1484 RTP timestamp value) MAY be sent in any order; however, for 1485 delay-critical systems, they SHOULD be sent in their original 1486 decoding order to minimize the delay. Note that the decoding 1487 order is the order of the NAL units in the bitstream. 1489 o Parameter sets are handled in accordance with the rules and 1490 recommendations given in section 8.4. 1492 o MANEs MUST NOT duplicate any NAL unit except for sequence or 1493 picture parameter set NAL units, as neither this memo nor the 1494 H.264 specification provides means to identify duplicated NAL 1495 units. Sequence and picture parameter set NAL units MAY be 1496 duplicated to make their correct reception more probable, but 1497 any such duplication MUST NOT affect the contents of any active 1498 sequence or picture parameter set. Duplication SHOULD be 1499 performed on the application layer and not by duplicating RTP 1500 packets (with identical sequence numbers). 1502 Senders using the non-interleaved mode and the interleaved mode 1503 MUST enforce the following packetization rule: 1505 o MANEs MAY convert single NAL unit packets into one aggregation 1506 packet, convert an aggregation packet into several single NAL 1507 unit packets, or mix both concepts, in an RTP translator. The 1508 RTP translator SHOULD take into account at least the following 1509 parameters: path MTU size, unequal protection mechanisms (e.g., 1510 through packet-based FEC according to RFC 2733 [18], especially 1511 for sequence and picture parameter set NAL units and coded slice 1512 data partition A NAL units), bearable latency of the system, and 1513 buffering capabilities of the receiver. 1515 Informative note: An RTP translator is required to handle 1516 RTCP as per RFC 3550. 1518 6.2. Single NAL Unit Mode 1520 This mode is in use when the value of the OPTIONAL packetization- 1521 mode media type parameter is equal to 0 or the packetization-mode 1522 is not present. All receivers MUST support this mode. It is 1523 primarily intended for low-delay applications that are compatible 1524 with systems using ITU-T Recommendation H.241 [3] (see section 1525 12.1). Only single NAL unit packets MAY be used in this mode. 1526 STAPs, MTAPs, and FUs MUST NOT be used. The transmission order of 1527 single NAL unit packets MUST comply with the NAL unit decoding 1528 order. 1530 6.3. Non-Interleaved Mode 1532 This mode is in use when the value of the OPTIONAL packetization- 1533 mode media type parameter is equal to 1. This mode SHOULD be 1534 supported. It is primarily intended for low-delay applications. 1535 Only single NAL unit packets, STAP-As, and FU-As MAY be used in 1536 this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. The 1537 transmission order of NAL units MUST comply with the NAL unit 1538 decoding order. 1540 6.4. Interleaved Mode 1542 This mode is in use when the value of the OPTIONAL packetization- 1543 mode media type parameter is equal to 2. Some receivers MAY 1544 support this mode. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. 1545 STAP-As and single NAL unit packets MUST NOT be used. The 1546 transmission order of packets and NAL units is constrained as 1547 specified in section 5.5. 1549 7. De-Packetization Process 1551 The de-packetization process is implementation dependent. 1552 Therefore, the following description should be seen as an example 1553 of a suitable implementation. Other schemes may be used as well as 1554 long as the output for the same input is the same as the process 1555 described below. The same output means that the resulting NAL 1556 units, and their order, are identical. Optimizations relative to 1557 the described algorithms are likely possible. Section 7.1 presents 1558 the de-packetization process for the single NAL unit and non- 1559 interleaved packetization modes, whereas section 7.2 describes the 1560 process for the interleaved mode. Section 7.3 includes additional 1561 de-packetization guidelines for intelligent receivers. 1563 All normal RTP mechanisms related to buffer management apply. In 1564 particular, duplicated or outdated RTP packets (as indicated by the 1565 RTP sequence number and the RTP timestamp) are removed. To 1566 determine the exact time for decoding, factors such as a possible 1567 intentional delay to allow for proper inter-stream synchronization 1568 must be factored in. 1570 7.1. Single NAL Unit and Non-Interleaved Mode 1572 The receiver includes a receiver buffer to compensate for 1573 transmission delay jitter. The receiver stores incoming packets in 1574 reception order into the receiver buffer. Packets are de- 1575 packetized in RTP sequence number order. If a de-packetized packet 1576 is a single NAL unit packet, the NAL unit contained in the packet 1577 is passed directly to the decoder. If a de-packetized packet is an 1578 STAP-A, the NAL units contained in the packet are passed to the 1579 decoder in the order in which they are encapsulated in the packet. 1580 For all the FU-A packets containing fragments of a single NAL unit, 1581 the de-packetized fragments are concatenated in their sending order 1582 to recover the NAL unit, which is then passed to the decoder. 1584 Informative note: If the decoder supports Arbitrary Slice Order, 1585 coded slices of a picture can be passed to the decoder in any 1586 order regardless of their reception and transmission order. 1588 7.2. Interleaved Mode 1590 The general concept behind these de-packetization rules is to 1591 reorder NAL units from transmission order to the NAL unit decoding 1592 order. 1594 The receiver includes a receiver buffer, which is used to 1595 compensate for transmission delay jitter and to reorder NAL units 1596 from transmission order to the NAL unit decoding order. In this 1597 section, the receiver operation is described under the assumption 1598 that there is no transmission delay jitter. To make a difference 1599 from a practical receiver buffer that is also used for compensation 1600 of transmission delay jitter, the receiver buffer is here after 1601 called the de-interleaving buffer in this section. Receivers 1602 SHOULD also prepare for transmission delay jitter; i.e., either 1603 reserve separate buffers for transmission delay jitter buffering 1604 and de-interleaving buffering or use a receiver buffer for both 1605 transmission delay jitter and de-interleaving. Moreover, receivers 1606 SHOULD take transmission delay jitter into account in the buffering 1607 operation; e.g., by additional initial buffering before starting of 1608 decoding and playback. 1610 This section is organized as follows: subsection 7.2.1 presents how 1611 to calculate the size of the de-interleaving buffer. Subsection 1612 7.2.2 specifies the receiver process on how to organize received 1613 NAL units to the NAL unit decoding order. 1615 7.2.1. Size of the De-interleaving Buffer 1617 In either Offer/Answer or declarative SDP usage, the sprop-deint- 1618 buf-req media type parameter signals the requirement for the de- 1619 interleaving buffer size. It is therefore RECOMMENDED to set the 1620 de-interleaving buffer size, in terms of number of bytes, equal to 1621 or greater than the value of sprop-deint-buf-req media type 1622 parameter. 1624 When the SDP Offer/Answer model or any other capability exchange 1625 procedure is used in session setup, the properties of the received 1626 stream SHOULD be such that the receiver capabilities are not 1627 exceeded. In the SDP Offer/Answer model, the receiver can indicate 1628 its capabilities to allocate a de-interleaving buffer with the 1629 deint-buf-cap media type parameter. See section 8.1 for further 1630 information on deint-buf-cap and sprop-deint-buf-req media type 1631 parameters and section 8.2.2 for further information on their use 1632 in the SDP Offer/Answer model. 1634 7.2.2. De-interleaving Process 1636 There are two buffering states in the receiver: initial buffering 1637 and buffering while playing. Initial buffering occurs when the RTP 1638 session is initialized. After initial buffering, decoding and 1639 playback are started, and the buffering-while-playing mode is used. 1641 Regardless of the buffering state, the receiver stores incoming NAL 1642 units, in reception order, in the de-interleaving buffer as follows. 1643 NAL units of aggregation packets are stored in the de-interleaving 1644 buffer individually. The value of DON is calculated and stored for 1645 each NAL unit. 1647 The receiver operation is described below with the help of the 1648 following functions and constants: 1650 o Function AbsDON is specified in section 8.1. 1652 o Function don_diff is specified in section 5.5. 1654 o Constant N is the value of the OPTIONAL sprop-interleaving-depth 1655 media type parameter (see section 8.1) incremented by 1. 1657 Initial buffering lasts until one of the following conditions is 1658 fulfilled: 1660 o There are N or more VCL NAL units in the de-interleaving buffer. 1662 o If sprop-max-don-diff is present, don_diff(m,n) is greater than 1663 the value of sprop-max-don-diff, in which n corresponds to the 1664 NAL unit having the greatest value of AbsDON among the received 1665 NAL units and m corresponds to the NAL unit having the smallest 1666 value of AbsDON among the received NAL units. 1668 o Initial buffering has lasted for the duration equal to or 1669 greater than the value of the OPTIONAL sprop-init-buf-time media 1670 type parameter. 1672 The NAL units to be removed from the de-interleaving buffer are 1673 determined as follows: 1675 o If the de-interleaving buffer contains at least N VCL NAL units, 1676 NAL units are removed from the de-interleaving buffer and passed 1677 to the decoder in the order specified below until the buffer 1678 contains N-1 VCL NAL units. 1680 o If sprop-max-don-diff is present, all NAL units m for which 1681 don_diff(m,n) is greater than sprop-max-don-diff are removed 1682 from the de-interleaving buffer and passed to the decoder in the 1683 order specified below. Herein, n corresponds to the NAL unit 1684 having the greatest value of AbsDON among the NAL units in the 1685 de-interleaving buffer. 1687 The order in which NAL units are passed to the decoder is specified 1688 as follows: 1690 o Let PDON be a variable that is initialized to 0 at the beginning 1691 of the RTP session. 1693 o For each NAL unit associated with a value of DON, a DON distance 1694 is calculated as follows. If the value of DON of the NAL unit 1695 is larger than the value of PDON, the DON distance is equal to 1696 DON - PDON. Otherwise, the DON distance is equal to 65535 - 1697 PDON + DON + 1. 1699 o NAL units are delivered to the decoder in ascending order of DON 1700 distance. If several NAL units share the same value of DON 1701 distance, they can be passed to the decoder in any order. 1703 o When a desired number of NAL units have been passed to the 1704 decoder, the value of PDON is set to the value of DON for the 1705 last NAL unit passed to the decoder. 1707 7.3. Additional De-Packetization Guidelines 1709 The following additional de-packetization rules may be used to 1710 implement an operational H.264 de-packetizer: 1712 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1713 coded slice data partitions A (DPAs). If a lost DPA is detected, 1714 after taking into account possible retransmission and FEC, a 1715 gateway may decide not to send the corresponding coded slice 1716 data partitions B and C, as their information is meaningless for 1717 H.264 decoders. In this way a MANE can reduce network load by 1718 discarding useless packets without parsing a complex bitstream. 1720 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1721 FUs. If a lost FU is found, a gateway may decide not to send 1722 the following FUs of the same fragmented NAL unit, as their 1723 information is meaningless for H.264 decoders. In this way a 1724 MANE can reduce network load by discarding useless packets 1725 without parsing a complex bitstream. 1727 o Intelligent receivers having to discard packets or NALUs should 1728 first discard all packets/NALUs in which the value of the NRI 1729 field of the NAL unit type octet is equal to 0. This will 1730 minimize the impact on user experience and keep the reference 1731 pictures intact. If more packets have to be discarded, then 1732 packets with a numerically lower NRI value should be discarded 1733 before packets with a numerically higher NRI value. However, 1734 discarding any packets with an NRI bigger than 0 very likely 1735 leads to decoder drift and SHOULD be avoided. 1737 8. Payload Format Parameters 1739 This section specifies the parameters that MAY be used to select 1740 optional features of the payload format and certain features of the 1741 bitstream. The parameters are specified here as part of the media 1742 subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. 1743 A mapping of the parameters into the Session Description Protocol 1744 (SDP) [6] is also provided for applications that use SDP. 1745 Equivalent parameters could be defined elsewhere for use with 1746 control protocols that do not use SDP. 1748 Some parameters provide a receiver with the properties of the 1749 stream that will be sent. The names of all these parameters start 1750 with "sprop" for stream properties. Some of these "sprop" 1751 parameters are limited by other payload or codec configuration 1752 parameters. For example, the sprop-parameter-sets parameter is 1753 constrained by the profile-level-id parameter. The media sender 1754 selects all "sprop" parameters rather than the receiver. This 1755 uncommon characteristic of the "sprop" parameters may not be 1756 compatible with some signaling protocol concepts, in which case the 1757 use of these parameters SHOULD be avoided. 1759 8.1. Media Type Registration 1761 The media subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is 1762 allocated from the IETF tree. 1764 The receiver MUST ignore any unspecified parameter. 1766 Media Type name: video 1768 Media subtype name: H264 1770 Required parameters: none 1772 OPTIONAL parameters: 1774 profile-level-id: 1775 A base16 [7] (hexadecimal) representation of the following 1776 three bytes in the sequence parameter set NAL unit specified 1777 in [1]: 1) profile_idc, 2) a byte herein referred to as 1778 profile-iop, composed of the values of constraint_set0_flag, 1779 constraint_set1_flag,constraint_set2_flag, 1780 constraint_set3_flag, and reserved_zero_4bits in bit- 1781 significance order, starting from the most significant bit, 1782 and 3) level_idc. Note that reserved_zero_4bits is required 1783 to be equal to 0 in [1], but other values for it may be 1784 specified in the future by ITU-T or ISO/IEC. 1786 The profile-level-id parameter indicates the default sub- 1787 profile, i.e. the subset of coding tools that may have been 1788 used to generate the stream or that the receiver supports, 1789 and the default level of the stream or the receiver supports. 1791 The default sub-profile is indicated collectively by the 1792 profile_idc byte and some fields in the profile-iop byte. 1793 Depending on the values of the fields in the profile-iop byte, 1794 the default sub-profile may be the set of coding tools 1795 supported by one profile, or a common subset of coding tools 1796 of multiple profiles, as specified in subsection 7.4.2.1.1 of 1797 [1]. The default level is indicated by the level_idc byte, 1798 and, when profile_idc is equal to 66, 77 or 88 (the Baseline, 1799 Main, or Extended profile) and level_idc is equal to 11, 1800 additionally by bit 4 (constraint_set3_flag) of the profile- 1801 iop byte. When profile_idc is equal to 66, 77 or 88 (the 1802 Baseline, Main, or Extended profile) and level_idc is equal 1803 to 11, and bit 4 (constraint_set3_flag) of the profile-iop 1804 byte is equal to 1, the default level is level 1b. 1806 Table 5 lists all profiles defined in Annex A of [1] and, for 1807 each of the profiles, the possible combinations of 1808 profile_idc and profile-iop that represent the same sub- 1809 profile. 1811 Table 5. Combinations of profile_idc and profile-iop 1812 representing the same sub-profile corresponding to the 1813 full set of coding tools supported by one profile. In 1814 the following, x may be either 0 or 1, while the profile 1815 names are indicated as follows. CB: Constrained Baseline 1816 profile, B: Baseline profile, M: Main profile, E: 1817 Extended profile, H: High profile, H10: High 10 profile, 1818 H42: High 4:2:2 profile, H44: High 4:4:4 Predictive 1819 profile, H10I: High 10 Intra profile, H42I: High 4:2:2 1820 Intra profile, H44I: High 4:4:4 Intra profile, and C44I: 1821 CAVLC 4:4:4 Intra profile. 1823 Profile profile_idc profile-iop 1824 (hexadecimal) (binary) 1826 CB 42 (B) x1xx0000 1827 same as: 4D (M) 1xxx0000 1828 same as: 58 (E) 11xx0000 1829 same as: 64 (H), 6E (H10), 1xx00000 1830 7A (H42), or F4 (H44) 1831 B 42 (B) x0xx0000 1832 same as: 58 (E) 10xx0000 1833 M 4D (M) 0x0x0000 1834 same as: 64 (H), 6E (H10), 01000000 1835 7A (H42), or F4 (H44) 1836 E 58 00xx0000 1837 H 64 00000000 1838 H10 6E 00000000 1839 H42 7A 00000000 1840 H44 F4 00000000 1841 H10I 64 00010000 1842 H42I 7A 00010000 1843 H44I F4 00010000 1844 C44I 2C 00010000 1846 For example, in the table above, profile_idc equal to 58 1847 (Extended) with profile-iop equal to 11xx0000 indicates the 1848 same sub-profile corresponding to profile_idc equal to 42 1849 (Baseline) with profile-iop equal to x1xx0000. Note that 1850 other combinations of profile_idc and profile-iop (not listed 1851 in Table 5) may represent a sub-profile equivalent to the 1852 common subset of coding tools for more than one profile. 1853 Note also that a decoder conforming to a certain profile may 1854 be able to decode bitstreams conforming to other profiles. 1855 For example, a decoder conforming to the High 4:4:4 profile 1856 at certain level must be able to decode bitstreams confirming 1857 to the Constrained Baseline, Main, High, High 10 or High 1858 4:2:2 profile at the same or a lower level. 1860 If the profile-level-id parameter is used to indicate 1861 properties of a NAL unit stream, it indicates that, to decode 1862 the stream, the minimum subset of coding tools a decoder has 1863 to support is the default sub-profile, and the lowest level 1864 the decoder has to support is the default level. 1866 If the profile-level-id parameter is used for capability 1867 exchange or session setup procedure, it indicates the subset 1868 of coding tools, which is equal to the default sub-profile, 1869 that the codec supports for both receiving and sending. If 1870 max-recv-level is not present, the default level from 1871 profile-level-id indicates the highest level the codec wishes 1872 to support. If max-recv-level is present it indicates the 1873 highest level the codec supports for receiving. For either 1874 receiving or sending, all levels that are lower than the 1875 highest level supported MUST also be supported. 1877 Informative note: Capability exchange and session setup 1878 procedures should provide means to list the capabilities 1879 for each supported sub-profile separately. For example, 1880 the one-of-N codec selection procedure of the SDP 1881 Offer/Answer model can be used (section 10.2 of [8]). 1882 The one-of-N codec selection procedure may also be used 1883 to provide different combinations of profile_idc and 1884 profile-iop that represent the same sub-profile. When 1885 there are many different combinations of profile_idc and 1886 profile-iop that represent the same sub-profile, using 1887 the one-of-N codec selection procedure may result into a 1888 fairly large SDP message. Therefore, a receiver should 1889 understand the different equivalent combinations of 1890 profile_idc and profile-iop that represent the same sub- 1891 profile, and be ready to accept an offer using any of the 1892 equivalent combinations. 1894 If no profile-level-id is present, the Baseline Profile 1895 without additional constraints at Level 1 MUST be inferred. 1897 max-recv-level: 1898 This parameter MAY be used to indicate the highest level a 1899 receiver supports when the highest level is higher than the 1900 default level (the level indicated by profile-level-id). The 1901 value of max-recv-level is a base16 (hexadecimal) 1902 representation of the two bytes after the syntax element 1903 profile_idc in the sequence parameter set NAL unit specified 1904 in [1]: profile-iop (as defined above) and level_idc. If 1905 (the level_idc byte of max-recv-level is equal to 11 and bit 1906 4 of the profile-iop byte of max-recv-level is equal to 1) or 1907 (the level_idc byte of max-recv-level is equal to 9 and bit 4 1908 of the profile-iop byte of max-recv-level is equal to 0), the 1909 highest level the receiver supports is level 1b. Otherwise, 1910 the highest level the receiver supports is equal to the 1911 level_idc byte of max-recv-level divided by 10. 1913 max-recv-level MUST NOT be present if the highest level the 1914 receiver supports is not higher than the default level. 1916 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: 1917 These parameters MAY be used to signal the capabilities of a 1918 receiver implementation. These parameters MUST NOT be used 1919 for any other purpose. The highest level conveyed in the 1920 value of the profile-level-id parameter or the max-recv-level 1921 parameter MUST be such that the receiver is fully capable of 1922 supporting. max-mbps, max-smbps, max-fs, max-cpb, max-dpb, 1923 and max-br MAY be used to indicate capabilities of the 1924 receiver that extend the required capabilities of the 1925 signaled highest level, as specified below. 1927 When more than one parameter from the set (max-mbps, max- 1928 smbps , max-fs, max-cpb, max-dpb, max-br) is present, the 1929 receiver MUST support all signaled capabilities 1930 simultaneously. For example, if both max-mbps and max-br are 1931 present, the signaled highest level with the extension of 1932 both the frame rate and bit rate is supported. That is, the 1933 receiver is able to decode NAL unit streams in which the 1934 macroblock processing rate is up to max-mbps (inclusive), the 1935 bit rate is up to max-br (inclusive), the coded picture 1936 buffer size is derived as specified in the semantics of the 1937 max-br parameter below, and other properties comply with the 1938 highest level specified in the value of the profile-level-id 1939 parameter or the max-recv-level parameter. 1941 If a receiver can support all the properties of level A, the 1942 highest level specified in the value of the profile-level-id 1943 parameter or the max-recv-level parameter MUST be level A 1944 (i.e. MUST NOT be lower than level A). In other words, a 1945 receiver MUST NOT signal values of max-mbps, max-fs, max-cpb, 1946 max-dpb, and max-br that taken together meet the requirements 1947 of a higher level compared to the highest level specified in 1948 the value of the profile-level-id parameter or the max-recv- 1949 level parameter. 1951 Informative note: When the OPTIONAL media type parameters 1952 are used to signal the properties of a NAL unit stream, 1953 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br 1954 are not present, and the value of profile-level-id must 1955 always be such that the NAL unit stream complies fully 1956 with the specified profile and level. 1958 max-mbps: The value of max-mbps is an integer indicating the 1959 maximum macroblock processing rate in units of macroblocks 1960 per second. The max-mbps parameter signals that the receiver 1961 is capable of decoding video at a higher rate than is 1962 required by the signaled highest level conveyed in the value 1963 of the profile-level-id parameter or the max-recv-level 1964 parameter. When max-mbps is signaled, the receiver MUST be 1965 able to decode NAL unit streams that conform to the signaled 1966 highestlevel, with the exception that the MaxMBPS value in 1967 Table A-1 of [1] for the signaled highest level is replaced 1968 with the value of max-mbps. The value of max-mbps MUST be 1969 greater than or equal to the value of MaxMBPS given in Table 1970 A-1 of [1] for the highest level. Senders MAY use this 1971 knowledge to send pictures of a given size at a higher 1972 picture rate than is indicated in the signaled highest level. 1974 max-smbps: The value of max-smbps is an integer indicating the 1975 maximum static macroblock processing rate in units of static 1976 macroblocks per second, under the hypothetical assumption 1977 that all macroblocks are static macroblocks. When max-smbps 1978 is signalled the MaxMBPS value in Table A-1 of [1] should be 1979 replaced with the result of the following computation: 1981 o If the parameter max-mbps is signalled, set a variable 1982 MaxMacroblocksPerSecond to the value of max-mbps. 1983 Otherwise, set MaxMacroblocksPerSecond equal to the value 1984 of MaxMBPS in Table A-1 [1] for the highest level. 1986 o Set a variable P_non-static to the proportion of non- 1987 static macroblocks in picture n. 1989 o Set a variable P_static to the proportion of static 1990 macroblocks in picture n. 1992 o The value of MaxMBPS in Table A-1 of [1] should be 1993 considered by the encoder to be equal to: 1995 MaxMacroblocksPerSecond * max-smbps / (P_non-static * 1996 max-smbps + P_static * MaxMacroblocksPerSecond) 1998 The encoder should recompute this value for each picture. The 1999 value of max-smbps MUST be greater than the value of MaxMBPS 2000 given in Table A-1 of [1] for the highest level. Senders MAY 2001 use this knowledge to send pictures of a given size at a 2002 higher picture rate than is indicated in the signaled highest 2003 level. 2005 max-fs: The value of max-fs is an integer indicating the maximum 2006 frame size in units of macroblocks. The max-fs parameter 2007 signals that the receiver is capable of decoding larger 2008 picture sizes than are required by the signaled highest level 2009 conveyed in the value of the profile-level-id parameter or 2010 the max-recv-level parameter. When max-fs is signaled, the 2011 receiver MUST be able to decode NAL unit streams that conform 2012 to the signaled highest level, with the exception that the 2013 MaxFS value in Table A-1 of [1] for the signaled highest 2014 level is replaced with the value of max-fs. The value of 2015 max-fs MUST be greater than or equal to the value of MaxFS 2016 given in Table A-1 of [1] for the highest level. Senders MAY 2017 use this knowledge to send larger pictures at a 2018 proportionally lower frame rate than is indicated in the 2019 signaled highest level. 2021 max-cpb: The value of max-cpb is an integer indicating the 2022 maximum coded picture buffer size in units of 1000 bits for 2023 the VCL HRD parameters (see A.3.1 item i of [1]) and in units 2024 of 1200 bits for the NAL HRD parameters (see A.3.1 item j of 2025 [1]). The max-cpb parameter signals that the receiver has 2026 more memory than the minimum amount of coded picture buffer 2027 memory required by the signaled highest level conveyed in the 2028 value of the profile-level-id parameter or the max-recv-level 2029 parameter. When max-cpb is signaled, the receiver MUST be 2030 able to decode NAL unit streams that conform to the signaled 2031 highest level, with the exception that the MaxCPB value in 2032 Table A-1 of [1] for the signaled highest level is replaced 2033 with the value of max-cpb. The value of max-cpb MUST be 2034 greater than or equal to the value of MaxCPB given in Table 2035 A-1 of [1] for the highest level. Senders MAY use this 2036 knowledge to construct coded video streams with greater 2037 variation of bit rate than can be achieved with the MaxCPB 2038 value in Table A-1 of [1]. 2040 Informative note: The coded picture buffer is used in the 2041 hypothetical reference decoder (Annex C) of H.264. The 2042 use of the hypothetical reference decoder is recommended 2043 in H.264 encoders to verify that the produced bitstream 2044 conforms to the standard and to control the output 2045 bitrate. Thus, the coded picture buffer is conceptually 2046 independent of any other potential buffers in the 2047 receiver, including de-interleaving and de-jitter buffers. 2048 The coded picture buffer need not be implemented in 2049 decoders as specified in Annex C of H.264, but rather 2050 standard-compliant decoders can have any buffering 2051 arrangements provided that they can decode standard- 2052 compliant bitstreams. Thus, in practice, the input 2053 buffer for video decoder can be integrated with de- 2054 interleaving and de-jitter buffers of the receiver. 2056 max-dpb: The value of max-dpb is an integer indicating the 2057 maximum decoded picture buffer size in units of 1024 bytes. 2058 The max-dpb parameter signals that the receiver has more 2059 memory than the minimum amount of decoded picture buffer 2060 memory required by the signaled highest level conveyed in the 2061 value of the profile-level-id parameter or the max-recv-level 2062 parameter. When max-dpb is signaled, the receiver MUST be 2063 able to decode NAL unit streams that conform to the signaled 2064 highest level, with the exception that the MaxDPB value in 2065 Table A-1 of [1] for the signaled highest level is replaced 2066 with the value of max-dpb. Consequently, a receiver that 2067 signals max-dpb MUST be capable of storing the following 2068 number of decoded frames, complementary field pairs, and non- 2069 paired fields in its decoded picture buffer: 2071 Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs * 2072 256 * ChromaFormatFactor ), 16) 2074 PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are 2075 defined in [1]. 2077 The value of max-dpb MUST be greater than or equal to the 2078 value of MaxDPB given in Table A-1 of [1] for the highest 2079 level. Senders MAY use this knowledge to construct coded 2080 video streams with improved compression. 2082 Informative note: This parameter was added primarily to 2083 complement a similar codepoint in the ITU-T 2084 Recommendation H.245, so as to facilitate signaling 2085 gateway designs. The decoded picture buffer stores 2086 reconstructed samples. There is no relationship between 2087 the size of the decoded picture buffer and the buffers 2088 used in RTP, especially de-interleaving and de-jitter 2089 buffers. 2091 max-br: The value of max-br is an integer indicating the maximum 2092 video bit rate in units of 1000 bits per second for the VCL 2093 HRD parameters (see A.3.1 item i of [1]) and in units of 1200 2094 bits per second for the NAL HRD parameters (see A.3.1 item j 2095 of [1]). 2097 The max-br parameter signals that the video decoder of the 2098 receiver is capable of decoding video at a higher bit rate 2099 than is required by the signaled highest level conveyed in 2100 the value of the profile-level-id parameter or the max-recv- 2101 level parameter. 2103 When max-br is signaled, the video codec of the receiver MUST 2104 be able to decode NAL unit streams that conform to the 2105 signaled highest level, with the following exceptions in the 2106 limits specified by the highest level: 2108 o The value of max-br replaces the MaxBR value in Table A-1 2109 of [1] for the highest level. 2111 o When the max-cpb parameter is not present, the result of 2112 the following formula replaces the value of MaxCPB in 2113 Table A-1 of [1]: (MaxCPB of the signaled level) * max-br 2114 / (MaxBR of the signaled highest level). 2116 For example, if a receiver signals capability for Level 1.2 2117 with max-br equal to 1550, this indicates a maximum video 2118 bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum 2119 video bitrate of 1860 kbits/sec for NAL HRD parameters, and a 2120 CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000). 2122 The value of max-br MUST be greater than or equal to the 2123 value MaxBR given in Table A-1 of [1] for the signaled 2124 highest level. 2126 Senders MAY use this knowledge to send higher bitrate video 2127 as allowed in the level definition of Annex A of H.264, to 2128 achieve improved video quality. 2130 Informative note: This parameter was added primarily to 2131 complement a similar codepoint in the ITU-T 2132 Recommendation H.245, so as to facilitate signaling 2133 gateway designs. No assumption can be made from the 2134 value of this parameter that the network is capable of 2135 handling such bit rates at any given time. In particular, 2136 no conclusion can be drawn that the signaled bit rate is 2137 possible under congestion control constraints. 2139 redundant-pic-cap: 2140 This parameter signals the capabilities of a receiver 2141 implementation. When equal to 0, the parameter indicates 2142 that the receiver makes no attempt to use redundant coded 2143 pictures to correct incorrectly decoded primary coded 2144 pictures. When equal to 0, the receiver is not capable of 2145 using redundant slices; therefore, a sender SHOULD avoid 2146 sending redundant slices to save bandwidth. When equal to 1, 2147 the receiver is capable of decoding any such redundant slice 2148 that covers a corrupted area in a primary decoded picture (at 2149 least partly), and therefore a sender MAY send redundant 2150 slices. When the parameter is not present, then a value of 0 2151 MUST be used for redundant-pic-cap. When present, the value 2152 of redundant-pic-cap MUST be either 0 or 1. 2154 When the profile-level-id parameter is present in the same 2155 signaling as the redundant-pic-cap parameter, and the profile 2156 indicated in profile-level-id is such that it disallows the 2157 use of redundant coded pictures (e.g., Main Profile), the 2158 value of redundant-pic-cap MUST be equal to 0. When a 2159 receiver indicates redundant-pic-cap equal to 0, the received 2160 stream SHOULD NOT contain redundant coded pictures. 2162 Informative note: Even if redundant-pic-cap is equal to 0, 2163 the decoder is able to ignore redundant codec pictures 2164 provided that the decoder supports such a profile 2165 (Baseline, Extended) in which redundant coded pictures 2166 are allowed. 2168 Informative note: Even if redundant-pic-cap is equal to 1, 2169 the receiver may also choose other error concealment 2170 strategies to replace or complement decoding of redundant 2171 slices. 2173 sprop-parameter-sets: 2174 This parameter MAY be used to convey any sequence and picture 2175 parameter set NAL units (herein referred to as the initial 2176 parameter set NAL units) that can be placed in the NAL unit 2177 stream to precede any other NAL units in decoding order. The 2178 parameter MUST NOT be used to indicate codec capability in 2179 any capability exchange procedure. The value of the 2180 parameter is a comma (',') separated list of base64 [7] 2181 representations of parameter set NAL units as specified in 2182 sections 7.3.2.1 and 7.3.2.2 of [1]. Note that the number of 2183 bytes in a parameter set NAL unit is typically less than 10, 2184 but a picture parameter set NAL unit can contain several 2185 hundreds of bytes. 2187 Informative note: When several payload types are offered 2188 in the SDP Offer/Answer model, each with its own sprop- 2189 parameter-sets parameter, then the receiver cannot assume 2190 that those parameter sets do not use conflicting storage 2191 locations (i.e., identical values of parameter set 2192 identifiers). Therefore, a receiver should buffer all 2193 sprop-parameter-sets and make them available to the 2194 decoder instance that decodes a certain payload type. 2196 The "sprop-parameter-sets" parameter MUST only contain 2197 parameter sets that are conforming to the profile-level-id, 2198 i.e., the subset of coding tools indicated by any of the 2199 parameter sets MUST be equal to the default sub-profile, and 2200 the level indicated by any of the parameter sets MUST be 2201 equal to the default level. 2203 sprop-level-parameter-sets: 2204 This parameter MAY be used to convey any sequence and picture 2205 parameter set NAL units (herein referred to as the initial 2206 parameter set NAL units) that can be placed in the NAL unit 2207 stream to precede any other NAL units in decoding order and 2208 that are associated with one or more levels different than 2209 the default level. The parameter MUST NOT be used to 2210 indicate codec capability in any capability exchange 2211 procedure. 2213 The sprop-level-parameter-sets parameter contains parameter 2214 sets for one or more levels which are different than the 2215 default level. All parameter sets associated with one level 2216 are clustered and prefixed with a three-byte field which has 2217 the same syntax as profile-level-id. This enables the 2218 receiver to install the parameter sets for one level and 2219 discard the rest. The three-byte field is named PLId, and 2220 all parameter sets associated with one level are named PSL, 2221 which has the same syntax as sprop-parameter-sets. Parameter 2222 sets for each level are represented in the form of PLId:PSL, 2223 i.e., PLId followed by a colon (':') and the base64 [7] 2224 representation of the initial parameter set NAL units for the 2225 level. Each pair of PLId:PSL is also separated by a colon. 2226 Note that a PSL can contain multiple parameter sets for that 2227 level, separated with commas (','). 2229 The subset of coding tools indicated by each PLId field MUST 2230 be equal to the default sub-profile, and the level indicated 2231 by each PLId field MUST be different than the default level. 2232 All sequence parameter sets contained in each PSL MUST have 2233 the three bytes from profile_idc to level_idc, inclusive, 2234 equal to the preceding PLId. 2236 Informative note: This parameter allows for efficient 2237 level downgrade or upgrade in SDP Offer/Answer and out- 2238 of-band transport of parameter sets, simultaneously. 2240 use-level-src-parameter-sets: 2241 This parameter MAY be used to indicate a receiver capability. 2242 The value MAY be equal to either 0 or 1. When the parameter 2243 is not present, the value MUST be inferred to be equal to 0. 2244 The value 0 indicates that the receiver does not understand 2245 the sprop-level-parameter-sets parameter, and does not 2246 understand the "fmtp" source attribute as specified in 2247 section 6.3 of [9], and will ignore sprop-level-parameter- 2248 sets when present, and will ignore sprop-parameter-sets when 2249 conveyed using the "fmtp" source attribute. The value 1 2250 indicates that the receiver understands the sprop-level- 2251 parameter-sets parameter, and understands the "fmtp" source 2252 attribute as specified in section 6.3 of [9], and is capable 2253 of using parameter sets contained in the sprop-level- 2254 parameter-sets or contained in the sprop-parameter-sets that 2255 is conveyed using the "fmtp" source attribute. 2257 Informative note: An RFC 3984 receiver does not 2258 understand sprop-level-parameter-sets, use-level-src- 2259 parameter-sets, or the "fmtp" source attribute as 2260 specified in section 6.3 of [9]. Therefore, during SDP 2261 Offer/Answer, an RFC 3984 receiver as the answerer will 2262 simply ignore sprop-level-parameter-sets, when present in 2263 an offer, and sprop-parameter-sets conveyed using the 2264 "fmtp" source attribute as specified in section 6.3 of 2265 [9]. Assume that the offered payload type was accepted 2266 at a level lower than the default level. If the offered 2267 payload type included sprop-level-parameter-sets or 2268 included sprop-parameter-sets conveyed using the "fmtp" 2269 source attribute, and the offerer sees that the answerer 2270 has not included use-level-src-parameter-sets equal to 1 2271 in the answer, the offerer knows that in-band transport 2272 of parameter sets is needed. 2274 in-band-parameter-sets: 2275 This parameter MAY be used to indicate a receiver capability. 2276 The value MAY be equal to either 0 or 1. The value 1 2277 indicates that the receiver discards out-of-band parameter 2278 sets in sprop-parameter-sets and sprop-level-parameter-sets, 2279 therefore the sender MUST transmit all parameter sets in-band. 2280 The value 0 indicates that the receiver utilizes out-of-band 2281 parameter sets included in sprop-parameter-sets and/or sprop- 2282 level-parameter-sets. However, in this case, the sender MAY 2283 still choose to send parameter sets in-band. When in-band- 2284 parameter-sets is equal to 1, use-level-src-parameter-sets 2285 MUST NOT be present or MUST be equal to 0. When the 2286 parameter is not present, this receiver capability is not 2287 specified, and therefore the sender MAY send out-of-band 2288 parameter sets only, or it MAY send in-band-parameter-sets 2289 only, or it MAY send both. 2291 level-asymmetry-allowed: 2292 This parameter MAY be used in SDP Offer/Answer to indicate 2293 whether level asymmetry, i.e., using a different level in the 2294 offerer-to-answerer direction than the level in the answerer- 2295 to-offerer direction, is allowed. The value MAY be equal to 2296 either 0 or 1. When the parameter is not present, the value 2297 MUST be inferred to be equal to 0. The value 1 in both the 2298 offer and the answer indicates that level asymmetry is 2299 allowed. The value of 0 in either the offer or the answer 2300 indicates the level asymmetry is not allowed. 2302 If "level-asymmetry-allowed" is equal to 0 (or not present) 2303 in either the offer or the answer, level asymmetry is not 2304 allowed. In this case, the level to use in the direction 2305 from the offerer to the answerer MUST be the same as the 2306 level to use in the opposite direction. 2308 packetization-mode: 2309 This parameter signals the properties of an RTP payload type 2310 or the capabilities of a receiver implementation. Only a 2311 single configuration point can be indicated; thus, when 2312 capabilities to support more than one packetization-mode are 2313 declared, multiple configuration points (RTP payload types) 2314 must be used. 2316 When the value of packetization-mode is equal to 0 or 2317 packetization-mode is not present, the single NAL mode, as 2318 defined in section 6.2 of RFC 3984, MUST be used. This mode 2319 is in use in standards using ITU-T Recommendation H.241 [3] 2320 (see section 12.1). When the value of packetization-mode is 2321 equal to 1, the non-interleaved mode, as defined in section 2322 6.3 of RFC 3984, MUST be used. When the value of 2323 packetization-mode is equal to 2, the interleaved mode, as 2324 defined in section 6.4 of RFC 3984, MUST be used. The value 2325 of packetization-mode MUST be an integer in the range of 0 to 2326 2, inclusive. 2328 sprop-interleaving-depth: 2329 This parameter MUST NOT be present when packetization-mode is 2330 not present or the value of packetization-mode is equal to 0 2331 or 1. This parameter MUST be present when the value of 2332 packetization-mode is equal to 2. 2334 This parameter signals the properties of an RTP packet stream. 2335 It specifies the maximum number of VCL NAL units that precede 2336 any VCL NAL unit in the RTP packet stream in transmission 2337 order and follow the VCL NAL unit in decoding order. 2338 Consequently, it is guaranteed that receivers can reconstruct 2339 NAL unit decoding order when the buffer size for NAL unit 2340 decoding order recovery is at least the value of sprop- 2341 interleaving-depth + 1 in terms of VCL NAL units. 2343 The value of sprop-interleaving-depth MUST be an integer in 2344 the range of 0 to 32767, inclusive. 2346 sprop-deint-buf-req: 2347 This parameter MUST NOT be present when packetization-mode is 2348 not present or the value of packetization-mode is equal to 0 2349 or 1. It MUST be present when the value of packetization- 2350 mode is equal to 2. 2352 sprop-deint-buf-req signals the required size of the de- 2353 interleaving buffer for the RTP packet stream. The value of 2354 the parameter MUST be greater than or equal to the maximum 2355 buffer occupancy (in units of bytes) required in such a de- 2356 interleaving buffer that is specified in section 7.2 of RFC 2357 3984. It is guaranteed that receivers can perform the de- 2358 interleaving of interleaved NAL units into NAL unit decoding 2359 order, when the de-interleaving buffer size is at least the 2360 value of sprop-deint-buf-req in terms of bytes. 2362 The value of sprop-deint-buf-req MUST be an integer in the 2363 range of 0 to 4294967295, inclusive. 2365 Informative note: sprop-deint-buf-req indicates the 2366 required size of the de-interleaving buffer only. When 2367 network jitter can occur, an appropriately sized jitter 2368 buffer has to be provisioned for as well. 2370 deint-buf-cap: 2371 This parameter signals the capabilities of a receiver 2372 implementation and indicates the amount of de-interleaving 2373 buffer space in units of bytes that the receiver has 2374 available for reconstructing the NAL unit decoding order. A 2375 receiver is able to handle any stream for which the value of 2376 the sprop-deint-buf-req parameter is smaller than or equal to 2377 this parameter. 2379 If the parameter is not present, then a value of 0 MUST be 2380 used for deint-buf-cap. The value of deint-buf-cap MUST be 2381 an integer in the range of 0 to 4294967295, inclusive. 2383 Informative note: deint-buf-cap indicates the maximum 2384 possible size of the de-interleaving buffer of the 2385 receiver only. When network jitter can occur, an 2386 appropriately sized jitter buffer has to be provisioned 2387 for as well. 2389 sprop-init-buf-time: 2390 This parameter MAY be used to signal the properties of an RTP 2391 packet stream. The parameter MUST NOT be present, if the 2392 value of packetization-mode is equal to 0 or 1. 2394 The parameter signals the initial buffering time that a 2395 receiver MUST wait before starting decoding to recover the 2396 NAL unit decoding order from the transmission order. The 2397 parameter is the maximum value of (decoding time of the NAL 2398 unit - transmission time of a NAL unit), assuming reliable 2399 and instantaneous transmission, the same timeline for 2400 transmission and decoding, and that decoding starts when the 2401 first packet arrives. 2403 An example of specifying the value of sprop-init-buf-time 2404 follows. A NAL unit stream is sent in the following 2405 interleaved order, in which the value corresponds to the 2406 decoding time and the transmission order is from left to 2407 right: 2409 0 2 1 3 5 4 6 8 7 ... 2411 Assuming a steady transmission rate of NAL units, the 2412 transmission times are: 2414 0 1 2 3 4 5 6 7 8 ... 2416 Subtracting the decoding time from the transmission time 2417 column-wise results in the following series: 2419 0 -1 1 0 -1 1 0 -1 1 ... 2421 Thus, in terms of intervals of NAL unit transmission times, 2422 the value of sprop-init-buf-time in this example is 1. The 2423 parameter is coded as a non-negative base10 integer 2424 representation in clock ticks of a 90-kHz clock. If the 2425 parameter is not present, then no initial buffering time 2426 value is defined. Otherwise the value of sprop-init-buf-time 2427 MUST be an integer in the range of 0 to 4294967295, inclusive. 2429 In addition to the signaled sprop-init-buf-time, receivers 2430 SHOULD take into account the transmission delay jitter 2431 buffering, including buffering for the delay jitter caused by 2432 mixers, translators, gateways, proxies, traffic-shapers, and 2433 other network elements. 2435 sprop-max-don-diff: 2436 This parameter MAY be used to signal the properties of an RTP 2437 packet stream. It MUST NOT be used to signal transmitter or 2438 receiver or codec capabilities. The parameter MUST NOT be 2439 present if the value of packetization-mode is equal to 0 or 1. 2440 sprop-max-don-diff is an integer in the range of 0 to 32767, 2441 inclusive. If sprop-max-don-diff is not present, the value 2442 of the parameter is unspecified. sprop-max-don-diff is 2443 calculated as follows: 2445 sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)}, 2446 for any i and any j>i, 2448 where i and j indicate the index of the NAL unit in the 2449 transmission order and AbsDON denotes a decoding order number 2450 of the NAL unit that does not wrap around to 0 after 65535. 2451 In other words, AbsDON is calculated as follows: Let m and n 2452 be consecutive NAL units in transmission order. For the very 2453 first NAL unit in transmission order (whose index is 0), 2454 AbsDON(0) = DON(0). For other NAL units, AbsDON is 2455 calculated as follows: 2457 If DON(m) == DON(n), AbsDON(n) = AbsDON(m) 2459 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 2460 AbsDON(n) = AbsDON(m) + DON(n) - DON(m) 2462 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 2463 AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n) 2465 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 2466 AbsDON(n) = AbsDON(m) - (DON(m) + 65536 - DON(n)) 2468 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 2469 AbsDON(n) = AbsDON(m) - (DON(m) - DON(n)) 2471 where DON(i) is the decoding order number of the NAL unit 2472 having index i in the transmission order. The decoding order 2473 number is specified in section 5.5 of RFC 3984. 2475 Informative note: Receivers may use sprop-max-don-diff to 2476 trigger which NAL units in the receiver buffer can be 2477 passed to the decoder. 2479 max-rcmd-nalu-size: 2480 This parameter MAY be used to signal the capabilities of a 2481 receiver. The parameter MUST NOT be used for any other 2482 purposes. The value of the parameter indicates the largest 2483 NALU size in bytes that the receiver can handle efficiently. 2484 The parameter value is a recommendation, not a strict upper 2485 boundary. The sender MAY create larger NALUs but must be 2486 aware that the handling of these may come at a higher cost 2487 than NALUs conforming to the limitation. 2489 The value of max-rcmd-nalu-size MUST be an integer in the 2490 range of 0 to 4294967295, inclusive. If this parameter is 2491 not specified, no known limitation to the NALU size exists. 2492 Senders still have to consider the MTU size available between 2493 the sender and the receiver and SHOULD run MTU discovery for 2494 this purpose. 2496 This parameter is motivated by, for example, an IP to H.223 2497 video telephony gateway, where NALUs smaller than the H.223 2498 transport data unit will be more efficient. A gateway may 2499 terminate IP; thus, MTU discovery will normally not work 2500 beyond the gateway. 2502 Informative note: Setting this parameter to a lower than 2503 necessary value may have a negative impact. 2505 sar-understood: 2506 This parameter MAY be used to indicate a receiver capability 2507 and not anything else. The parameter indicates the maximum 2508 value of aspect_ratio_idc (specified in [1]) smaller than 255 2509 that the receiver understands. Table E-1 of [1] specifies 2510 aspect_ratio_idc equal to 0 as "unspecified", 1 to 16, 2511 inclusive, as specific Sample Aspect Ratios (SARs), 17 to 254, 2512 inclusive, as "reserved", and 255 as the Extended SAR, for 2513 which SAR width and SAR height are explicitly signaled. 2514 Therefore, a receiver with a decoder according to [1] 2515 understands aspect_ratio_idc in the range of 1 to 16, 2516 inclusive and aspect_ratio_idc equal to 255, in the sense 2517 that the receiver knows what exactly the SAR is. For such a 2518 receiver, the value of sar-understood is 16. If in the 2519 future Table E-1 of [1] is extended, e.g., such that the SAR 2520 for aspect_ratio_idc equal to 17 is specified, then for a 2521 receiver with a decoder that understands the extension, the 2522 value of sar-understood is 17. For a receiver with a decoder 2523 according to the 2003 version of [1], the value of sar- 2524 understood is 13, as the minimum reserved aspect_ratio_idc 2525 therein is 14. 2527 When sar-understood is not present, the value MUST be 2528 inferred to be equal to 13. 2530 sar-supported: 2531 This parameter MAY be used to indicate a receiver capability 2532 and not anything else. The value of this parameter is an 2533 integer in the range of 1 to sar-understood, inclusive, equal 2534 to 255. The value of sar-supported equal to N smaller than 2535 255 indicates that the reciever supports all the SARs 2536 corresponding to H.264 aspect_ratio_idc values (see Table E-1 2537 of [1]) in the range from 1 to N, inclusive, without 2538 geometric distortion. The value of sar-supported equal to 2539 255 indicates that the receiver supports all sample aspect 2540 ratios which are expressible using two 16-bit integer values 2541 as the numerator and denominator, i.e., those that are 2542 expressible using the H.264 aspect_ratio_idc value of 255 2543 (Extended_SAR, see Table E-1 of [1]), without geometric 2544 distortion. 2546 H.264 compliant encoders SHOULD NOT send an aspect_ratio_idc 2547 equal to 0, or an aspect_ratio_idc larger than sar-understood 2548 and smaller than 255. H.264 compliant encoders SHOULD send 2549 an aspect_ratio_idc that the receiver is able to display 2550 without geometrical distortion. However, H.264 compliant 2551 encoders MAY choose to send pictures using any SAR. 2553 Note that the actual sample aspect ratio or extended sample 2554 aspect ratio, when present, of the stream is conveyed in the 2555 Video Usability Information (VUI) part of the sequence 2556 parameter set. 2558 Encoding considerations: 2559 This type is only defined for transfer via RTP (RFC 3550). 2561 Security considerations: 2562 See section 9 of RFC xxxx. 2564 Public specification: 2565 Please refer to RFC xxxx and its section 15. 2567 Additional information: 2568 None 2570 File extensions: none 2572 Macintosh file type code: none 2574 Object identifier or OID: none 2576 Person & email address to contact for further information: 2577 Ye-Kui Wang, yekuiwang@huawei.com 2579 Intended usage: COMMON 2581 Author: 2582 Ye-Kui Wang, yekuiwang@huawei.com 2584 Change controller: 2585 IETF Audio/Video Transport working group delegated from the 2586 IESG. 2588 8.2. SDP Parameters 2590 8.2.1. Mapping of Payload Type Parameters to SDP 2592 The media type video/H264 string is mapped to fields in the Session 2593 Description Protocol (SDP) [6] as follows: 2595 o The media name in the "m=" line of SDP MUST be video. 2597 o The encoding name in the "a=rtpmap" line of SDP MUST be H264 2598 (the media subtype). 2600 o The clock rate in the "a=rtpmap" line MUST be 90000. 2602 o The OPTIONAL parameters "profile-level-id", "max-recv-level", 2603 "max-mbps", "max-smbps", "max-fs", "max-cpb", "max-dpb", "max- 2604 br", "redundant-pic-cap", "use-level-src-parameter-sets", "in- 2605 band-parameter-sets", "level-asymmetry-allowed", "packetization- 2606 mode", "sprop-interleaving-depth", "sprop-deint-buf-req", 2607 "deint-buf-cap", "sprop-init-buf-time", "sprop-max-don-diff", 2608 "max-rcmd-nalu-size", "sar-understood", and "sar-supported", 2609 when present, MUST be included in the "a=fmtp" line of SDP. 2610 These parameters are expressed as a media type string, in the 2611 form of a semicolon separated list of parameter=value pairs. 2613 o The OPTIONAL parameters "sprop-parameter-sets" and "sprop-level- 2614 parameter-sets", when present, MUST be included in the "a=fmtp" 2615 line of SDP or conveyed using the "fmtp" source attribute as 2616 specified in section 6.3 of [9]. For a particular media format 2617 (i.e., RTP payload type), a "sprop-parameter-sets" or "sprop- 2618 level-parameter-sets" MUST NOT be both included in the "a=fmtp" 2619 line of SDP and conveyed using the "fmtp" source attribute. 2620 When included in the "a=fmtp" line of SDP, these parameters are 2621 expressed as a media type string, in the form of a semicolon 2622 separated list of parameter=value pairs. When conveyed using 2623 the "fmtp" source attribute, these parameters are only 2624 associated with the given source and payload type as parts of 2625 the "fmtp" source attribute. 2627 Informative note: Conveyance of "sprop-parameter-sets" and 2628 "sprop-level-parameter-sets" using the "fmtp" source 2629 attribute allows for out-of-band transport of parameter sets 2630 in topologies like Topo-Video-switch-MCU [29]. 2632 An example of media representation in SDP is as follows (Baseline 2633 Profile, Level 3.0, some of the constraints of the Main profile may 2634 not be obeyed): 2636 m=video 49170 RTP/AVP 98 2637 a=rtpmap:98 H264/90000 2638 a=fmtp:98 profile-level-id=42A01E; 2639 packetization-mode=1; 2640 sprop-parameter-sets= 2642 8.2.2. Usage with the SDP Offer/Answer Model 2644 When H.264 is offered over RTP using SDP in an Offer/Answer model 2645 [8] for negotiation for unicast usage, the following limitations 2646 and rules apply: 2648 o The parameters identifying a media format configuration for 2649 H.264 are "profile-level-id" and "packetization-mode". These 2650 media format configuration parameters (except for the level part 2651 of "profile-level-id") MUST be used symmetrically; i.e., the 2652 answerer MUST either maintain all configuration parameters or 2653 remove the media format (payload type) completely, if one or 2654 more of the parameter values are not supported. Note that the 2655 level part of "profile-level-id" includes level_idc, and, for 2656 indication of level 1b when profile_idc is equal to 66, 77 or 88, 2657 bit 4 (constraint_set3_flag) of profile-iop. The level part of 2658 "profile-level-id" is changeable. 2660 Informative note: The requirement for symmetric use does not 2661 apply for the level part of "profile-level-id", and does not 2662 apply for the other stream properties and capability 2663 parameters. 2665 Informative note: In H.264 [1], all the levels except for 2666 level 1b are equal to the value of level_idc divided by 10. 2667 Level 1b is a level higher than level 1.0 but lower than 2668 level 1.1, and is signaled in an ad-hoc manner, due to that 2669 the level was specified after level 1.0 and level 1.1. For 2670 the Baseline, Main and Extended profiles (with profile_idc 2671 equal to 66, 77 and 88, respectively), level 1b is indicated 2672 by level_idc equal to 11 (i.e. same as level 1.1) and 2673 constraint_set3_flag equal to 1. For other profiles, level 2674 1b is indicated by level_idc equal to 9 (but note that level 2675 1b for these profiles are still higher than level 1, which 2676 has level_idc equal to 10, and lower than level 1.1). In SDP 2677 Offer/Answer, an answer to an offer may indicate a level 2678 equal to or lower than the level indicated in the offer. Due 2679 to the ad-hoc indication of level 1b, offerers and answerers 2680 must check the value of bit 4 (constraint_set3_flag) of the 2681 middle octet of the parameter "profile-level-id", when 2682 profile_idc is equal to 66, 77 or 88 and level_idc is equal 2683 to 11. 2685 To simplify handling and matching of these configurations, the 2686 same RTP payload type number used in the offer SHOULD also be 2687 used in the answer, as specified in [8]. An answer MUST NOT 2688 contain a payload type number used in the offer unless the 2689 configuration is exactly the same as in the offer. 2691 Informative note: When an offerer receives an answer, it has 2692 to compare payload types not declared in the offer based on 2693 the media type (i.e., video/H264) and the above media 2694 configuration parameters with any payload types it has 2695 already declared. This will enable it to determine whether 2696 the configuration in question is new or if it is equivalent 2697 to configuration already offered, since a different payload 2698 type number may be used in the answer. 2700 o The parameter "max-recv-level", when present, declares the 2701 highest level supported for receiving. In case "max-recv-level" 2702 is not present, the highest level supported for receiving is 2703 equal to the default level indicated by the level part of 2704 "profile-level-id". "max-recv-level", when present, MUST be 2705 higher than the default level. 2707 o The parameter "level-asymmetry-allowed" indicates whether level 2708 asymmetry is allowed. 2710 If "level-asymmetry-allowed" is equal to 0 (or not present) in 2711 either the offer or the answer, level asymmetry is not allowed. 2712 In this case, the level to use in the direction from the offerer 2713 to the answerer MUST be the same as the level to use in the 2714 opposite direction, and the common level to use is equal to the 2715 lower value of the default level in the offer and the default 2716 level in the answer. 2718 Otherwise ("level-asymmetry-allowed" equals to 1 in both the 2719 offer and the answer), level asymmetry is allowed. In this case, 2720 the level to use in the offerer-to-answerer direction MUST be 2721 equal to the highest level the answerer supports for receiving, 2722 and the level to use in the answerer-to-offerer direction MUST 2723 be equal to the highest level the offerer supports for receiving. 2725 When level asymmetry is not allowed, level upgrade is not 2726 allowed, i.e. the default level in the answer MUST be equal to 2727 or lower than the default level in the offer. 2729 o The parameters "sprop-deint-buf-req", "sprop-interleaving-depth", 2730 "sprop-max-don-diff", and "sprop-init-buf-time" describe the 2731 properties of the RTP packet stream that the offerer or answerer 2732 is sending for the media format configuration. This differs 2733 from the normal usage of the Offer/Answer parameters: normally 2734 such parameters declare the properties of the stream that the 2735 offerer or the answerer is able to receive. When dealing with 2736 H.264, the offerer assumes that the answerer will be able to 2737 receive media encoded using the configuration being offered. 2739 Informative note: The above parameters apply for any stream 2740 sent by the declaring entity with the same configuration; 2741 i.e., they are dependent on their source. Rather than being 2742 bound to the payload type, the values may have to be applied 2743 to another payload type when being sent, as they apply for 2744 the configuration. 2746 o The capability parameters "max-mbps", "max-smbps", "max-fs", 2747 "max-cpb", "max-dpb", "max-br", ,"redundant-pic-cap", "max-rcmd- 2748 nalu-size", "sar-understood", "sar-supported" MAY be used to 2749 declare further capabilities of the offerer or answerer for 2750 receiving. These parameters MUST NOT be present when the 2751 direction attribute is sendonly, and the parameters describe the 2752 limitations of what the offerer or answerer accepts for 2753 receiving streams. 2755 o An offerer has to include the size of the de-interleaving buffer, 2756 "sprop-deint-buf-req", in the offer for an interleaved H.264 2757 stream. To enable the offerer and answerer to inform each other 2758 about their capabilities for de-interleaving buffering in 2759 receiving streams, both parties are RECOMMENDED to include 2760 "deint-buf-cap". For interleaved streams, it is also 2761 RECOMMENDED to consider offering multiple payload types with 2762 different buffering requirements when the capabilities of the 2763 receiver are unknown. 2765 o The "sprop-parameter-sets" or "sprop-level-parameter-sets" 2766 parameter, when present (included in the "a=fmtp" line of SDP or 2767 conveyed using the "fmtp" source attribute as specified in 2768 section 6.3 of [9]), is used for out-of-band transport of 2769 parameter sets. However, when out-of-band transport of 2770 parameter sets is used, parameter sets MAY still be additionally 2771 transported in-band. 2773 The answerer MAY use either out-of-band or in-band transport of 2774 parameter sets for the stream it is sending, regardless of 2775 whether out-of-band parameter sets transport has been used in 2776 the offerer-to-answerer direction. Parameter sets included in 2777 an answer are independent of those parameter sets included in 2778 the offer, as they are used for decoding two different video 2779 streams, one from the answerer to the offerer, and the other in 2780 the opposite direction. 2782 The following rules apply to transport of parameter sets in the 2783 offerer-to-answerer direction. 2785 o An offer MAY include either or both of "sprop-parameter- 2786 sets" and "sprop-level-parameter-sets". If neither "sprop- 2787 parameter-sets" nor "sprop-level-parameter-sets" is present 2788 in the offer, then only in-band transport of parameter sets 2789 is used. 2791 o If the answer includes "in-band-parameter-sets" equal to 1, 2792 then the offerer MUST transmit parameter sets in-band. 2793 Otherwise, the following applies. 2795 o If the level to use in the offerer-to-answerer 2796 direction is equal to the default level in the offer, 2797 the following applies. 2799 When there is a "sprop-parameter-sets" included 2800 in the "a=fmtp" line in the offer, the answerer 2801 MUST be prepared to use the parameter sets 2802 included in the "sprop-parameter-sets" for 2803 decoding the incoming NAL unit stream. 2805 When there is a "sprop-parameter-sets" conveyed 2806 using the "fmtp" source attribute in the offer, 2807 the following applies. If the answer includes 2808 "use-level-src-parameter-sets" equal to 1 or the 2809 "fmtp" source attribute, the answerer MUST be 2810 prepared to use the parameter sets included in 2811 the "sprop-parameter-sets" for decoding the 2812 incoming NAL unit stream; Otherwise, the offerer 2813 MUST transmit parameter sets in-band. 2815 When "sprop-parameter-sets" is not present in the 2816 offer, the offerer MUST transmit parameter sets 2817 in-band. 2819 The answerer MUST ignore "sprop-level-parameter- 2820 sets", when present (either included in the 2821 "a=fmtp" line or conveyed using the "fmtp" source 2822 attribute) in the offer. 2824 o Otherwise (the level to use in the offerer-to-answerer 2825 direction is not equal to the default level in the 2826 offer), the following applies. 2828 The answerer MUST ignore "sprop-parameter-sets", 2829 when present (either included in the "a=fmtp" 2830 line or conveyed using the "fmtp" source 2831 attribute) in the offer. 2833 When neither "use-level-src-parameter-sets" equal 2834 to 1 nor the "fmtp" source attribute is present 2835 in the answer, the answerer MUST ignore "sprop- 2836 level-parameter-sets", when present in the offer, 2837 and the offerer MUST transmit parameter sets in- 2838 band. 2840 When either "use-level-src-parameter-sets" equal 2841 to 1 or the "fmtp" source attribute is present in 2842 the answer, the answerer MUST be prepared to use 2843 the parameter sets that are included in "sprop- 2844 level-parameter-sets" for the accepted level (i.e. 2845 the default level in the answer), when present in 2846 the offer, for decoding the incoming NAL unit 2847 stream, and ignore all other parameter sets 2848 included in "sprop-level-parameter-sets". 2850 When no parameter sets for the level to use in 2851 the offerer-to-answerer direction are present in 2852 "sprop-level-parameter-sets" in the offer, the 2853 offerer MUST transmit parameter sets in-band. 2855 The following rules apply to transport of parameter sets in the 2856 answerer-to-offerer direction. 2858 o An answer MAY include either "sprop-parameter-sets" or 2859 "sprop-level-parameter-sets", but MUST NOT include both of 2860 the two. If neither "sprop-parameter-sets" nor "sprop- 2861 level-parameter-sets" is present in the answer, then only 2862 in-band transport of parameter sets is used. 2864 o If the offer includes "in-band-parameter-sets" equal to 1, 2865 the answerer MUST NOT include "sprop-parameter-sets" or 2866 "sprop-level-parameter-sets" in the answer and MUST 2867 transmit parameter sets in-band. Otherwise, the following 2868 applies. 2870 o If the level to use in the answerer-to-offerer 2871 direction is equal to the default level in the answer, 2872 the following applies. 2874 When there is a "sprop-parameter-sets" included 2875 in the "a=fmtp" line in the answer, the offerer 2876 MUST be prepared to use the parameter sets 2877 included in the "sprop-parameter-sets" for 2878 decoding the incoming NAL unit stream. 2880 When there is a "sprop-parameter-sets" conveyed 2881 using the "fmtp" source attribute in the answer, 2882 the following applies. If the offer includes 2883 "use-level-src-parameter-sets" equal to 1 or the 2884 "fmtp" source attribute, the offerer MUST be 2885 prepared to use the parameter sets included in 2886 the "sprop-parameter-sets" for decoding the 2887 incoming NAL unit stream; Otherwise, the 2888 answerer MUST transmit parameter sets in-band. 2890 When "sprop-parameter-sets" is not present in the 2891 answer, the answerer MUST transmit parameter sets 2892 in-band. 2894 The offerer MUST ignore "sprop-level-parameter- 2895 sets", when present (either included in the 2896 "a=fmtp" line or conveyed using the "fmtp" source 2897 attribute) in the answer. 2899 o Otherwise (the level to use in the answerer-to-offerer 2900 direction is not equal to the default level in the 2901 answer), the following applies. 2903 The offerer MUST ignore "sprop-parameter-sets", 2904 when present (either included in the "a=fmtp" 2905 line of SDP or conveyed using the "fmtp" source 2906 attribute) in the answer. 2908 When neither "use-level-src-parameter-sets" equal 2909 to 1 nor the "fmtp" source attribute is present 2910 in the offer, the offerer MUST ignore "sprop- 2911 level-parameter-sets", when present, and the 2912 answerer MUST transmit parameter sets in-band. 2914 When either "use-level-src-parameter-sets" equal 2915 to 1 or the "fmtp" source attribute is present in 2916 the offer, the offerer MUST be prepared to use 2917 the parameter sets that are included in "sprop- 2918 level-parameter-sets" for the level to use in the 2919 answerer-to-offerer direction, when present in 2920 the answer, for decoding the incoming NAL unit 2921 stream, and ignore all other parameter sets 2922 included in "sprop-level-parameter-sets" in the 2923 answer. 2925 When no parameter sets for the level to use in 2926 the answerer-to-offerer direction are present in 2927 "sprop-level-parameter-sets" in the answer, the 2928 answerer MUST transmit parameter sets in-band. 2930 When "sprop-parameter-sets" or "sprop-level-parameter-sets" is 2931 conveyed using the "fmtp" source attribute as specified in 2932 section 6.3 of [9], the receiver of the parameters MUST store 2933 the parameter sets included in the "sprop-parameter-sets" or 2934 "sprop-level-parameter-sets" for the accepted level and 2935 associate them to the source given as a part of the "fmtp" 2936 source attribute. Parameter sets associated with one source 2937 MUST only be used to decode NAL units conveyed in RTP packets 2938 from the same source. When this mechanism is in use, SSRC 2939 collision detection and resolution MUST be performed as 2940 specified in [9]. 2942 Informative note: Conveyance of "sprop-parameter-sets" and 2943 "sprop-level-parameter-sets" using the "fmtp" source 2944 attribute may be used in topologies like Topo-Video-switch- 2945 MCU [29] to enable out-of-band transport of parameter sets. 2947 For streams being delivered over multicast, the following rules 2948 apply: 2950 o The media format configuration is identified by "profile-level- 2951 id", including the level part, and "packetization-mode". These 2952 media format configuration parameters (including the level part 2953 of "profile-level-id") MUST be used symmetrically; i.e., the 2954 answerer MUST either maintain all configuration parameters or 2955 remove the media format (payload type) completely. Note that 2956 this implies that the level part of "profile-level-id" for 2957 Offer/Answer in multicast is not changeable. 2959 To simplify handling and matching of these configurations, the 2960 same RTP payload type number used in the offer SHOULD also be 2961 used in the answer, as specified in [8]. An answer MUST NOT 2962 contain a payload type number used in the offer unless the 2963 configuration is the same as in the offer. 2965 o Parameter sets received MUST be associated with the originating 2966 source, and MUST be only used in decoding the incoming NAL unit 2967 stream from the same source. 2969 o The rules for other parameters are the same as above for unicast 2970 as long as the above rules are obeyed. 2972 Table 6 lists the interpretation of all the media type parameters 2973 that MUST be used for the different direction attributes. 2975 Table 6. Interpretation of parameters for different direction 2976 attributes. 2978 sendonly --+ 2979 recvonly --+ | 2980 sendrecv --+ | | 2981 | | | 2982 profile-level-id C C P 2983 max-recv-level R R - 2984 packetization-mode C C P 2985 sprop-deint-buf-req P - P 2986 sprop-interleaving-depth P - P 2987 sprop-max-don-diff P - P 2988 sprop-init-buf-time P - P 2989 max-mbps R R - 2990 max-smbps R R - 2991 max-fs R R - 2992 max-cpb R R - 2993 max-dpb R R - 2994 max-br R R - 2995 redundant-pic-cap R R - 2996 deint-buf-cap R R - 2997 max-rcmd-nalu-size R R - 2998 sar-understood R R - 2999 sar-supported R R - 3000 in-band-parameter-sets R R - 3001 use-level-src-parameter-sets R R - 3002 level-asymmetry-allowed O - - 3003 sprop-parameter-sets S - S 3004 sprop-level-parameter-sets S - S 3006 Legend: 3008 C: configuration for sending and receiving streams 3009 O: offer/answer mode 3010 P: properties of the stream to be sent 3011 R: receiver capabilities 3012 S: out-of-band parameter sets 3013 -: not usable, when present SHOULD be ignored 3015 Parameters used for declaring receiver capabilities are in general 3016 downgradable; i.e., they express the upper limit for a sender's 3017 possible behavior. Thus a sender MAY select to set its encoder 3018 using only lower/less or equal values of these parameters. 3020 Parameters declaring a configuration point are not changeable, with 3021 the exception of the level part of the "profile-level-id" parameter 3022 for unicast usage. This expresses values a receiver expects to be 3023 used and must be used verbatim on the sender side. 3025 When a sender's capabilities are declared, and non-downgradable 3026 parameters are used in this declaration, then these parameters 3027 express a configuration that is acceptable for the sender to 3028 receive streams. In order to achieve high interoperability levels, 3029 it is often advisable to offer multiple alternative configurations; 3030 e.g., for the packetization mode. It is impossible to offer 3031 multiple configurations in a single payload type. Thus, when 3032 multiple configuration offers are made, each offer requires its own 3033 RTP payload type associated with the offer. 3035 A receiver SHOULD understand all media type parameters, even if it 3036 only supports a subset of the payload format's functionality. This 3037 ensures that a receiver is capable of understanding when an offer 3038 to receive media can be downgraded to what is supported by the 3039 receiver of the offer. 3041 An answerer MAY extend the offer with additional media format 3042 configurations. However, to enable their usage, in most cases a 3043 second offer is required from the offerer to provide the stream 3044 property parameters that the media sender will use. This also has 3045 the effect that the offerer has to be able to receive this media 3046 format configuration, not only to send it. 3048 If an offerer wishes to have non-symmetric capabilities between 3049 sending and receiving, the offerer can allow asymmetric levels via 3050 "level-asymmetry-allowed" equal to 1. Alternatively, the offerer 3051 could offer different RTP sessions; i.e., different media lines 3052 declared as "recvonly" and "sendonly", respectively. This may have 3053 further implications on the system, and may require additional 3054 external semantics to associate the two media lines. 3056 8.2.3. Usage in Declarative Session Descriptions 3058 When H.264 over RTP is offered with SDP in a declarative style, as 3059 in RTSP [27] or SAP [28], the following considerations are 3060 necessary. 3062 o All parameters capable of indicating both stream properties and 3063 receiver capabilities are used to indicate only stream 3064 properties. For example, in this case, the parameter "profile- 3065 level-id" declares only the values used by the stream, not the 3066 capabilities for receiving streams. This results in that the 3067 following interpretation of the parameters MUST be used: 3069 Declaring actual configuration or stream properties: 3071 - profile-level-id 3072 - packetization-mode 3073 - sprop-interleaving-depth 3074 - sprop-deint-buf-req 3075 - sprop-max-don-diff 3076 - sprop-init-buf-time 3078 Out-of-band transporting of parameter sets: 3080 - sprop-parameter-sets 3081 - sprop-level-parameter-sets 3083 Not usable(when present, they SHOULD be ignored): 3085 - max-mbps 3086 - max-smbps 3087 - max-fs 3088 - max-cpb 3089 - max-dpb 3090 - max-br 3091 - max-recv-level 3092 - redundant-pic-cap 3093 - max-rcmd-nalu-size 3094 - deint-buf-cap 3095 - sar-understood 3096 - sar-supported 3097 - in-band-parameter-sets 3098 - level-asymmetry-allowed 3099 - use-level-src-parameter-sets 3101 o A receiver of the SDP is required to support all parameters and 3102 values of the parameters provided; otherwise, the receiver MUST 3103 reject (RTSP) or not participate in (SAP) the session. It falls 3104 on the creator of the session to use values that are expected to 3105 be supported by the receiving application. 3107 8.3. Examples 3109 An SDP Offer/Answer exchange wherein both parties are expected to 3110 both send and receive could look like the following. Only the 3111 media codec specific parts of the SDP are shown. Some lines are 3112 wrapped due to text constraints. 3114 Offerer -> Answerer SDP message: 3116 m=video 49170 RTP/AVP 100 99 98 3117 a=rtpmap:98 H264/90000 3118 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; 3119 sprop-parameter-sets= 3120 a=rtpmap:99 H264/90000 3121 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3122 sprop-parameter-sets= 3123 a=rtpmap:100 H264/90000 3124 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3125 sprop-parameter-sets=; 3126 sprop-interleaving-depth=45; sprop-deint-buf-req=64000; 3127 sprop-init-buf-time=102478; deint-buf-cap=128000 3129 The above offer presents the same codec configuration in three 3130 different packetization formats. PT 98 represents single NALU mode, 3131 PT 99 represents non-interleaved mode, and PT 100 indicates the 3132 interleaved mode. In the interleaved mode case, the interleaving 3133 parameters that the offerer would use if the answer indicates 3134 support for PT 100 are also included. In all three cases the 3135 parameter "sprop-parameter-sets" conveys the initial parameter sets 3136 that are required by the answerer when receiving a stream from the 3137 offerer when this configuration is accepted. Note that the value 3138 for "sprop-parameter-sets" could be different for each payload type. 3140 Answerer -> Offerer SDP message: 3142 m=video 49170 RTP/AVP 100 99 97 3143 a=rtpmap:97 H264/90000 3144 a=fmtp:97 profile-level-id=42A01E; packetization-mode=0; 3145 sprop-parameter-sets= 3146 a=rtpmap:99 H264/90000 3147 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3148 sprop-parameter-sets=; 3149 max-rcmd-nalu-size=3980 3150 a=rtpmap:100 H264/90000 3151 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3152 sprop-parameter-sets=; 3153 sprop-interleaving-depth=60; 3154 sprop-deint-buf-req=86000; sprop-init-buf-time=156320; 3155 deint-buf-cap=128000; max-rcmd-nalu-size=3980 3157 As the Offer/Answer negotiation covers both sending and receiving 3158 streams, an offer indicates the exact parameters for what the 3159 offerer is willing to receive, whereas the answer indicates the 3160 same for what the answerer accepts to receive. In this case the 3161 offerer declared that it is willing to receive payload type 98. 3162 The answerer accepts this by declaring an equivalent payload type 3163 97; i.e., it has identical values for the two parameters "profile- 3164 level-id" and "packetization-mode" (since "packetization-mode" is 3165 equal to 0, "sprop-deint-buf-req" is not present). As the offered 3166 payload type 98 is accepted, the answerer needs to store parameter 3167 sets included in sprop-parameter-sets= in 3168 case the offer finally decides to use this configuration. In the 3169 answer, the answerer includes the parameter sets in sprop- 3170 parameter-sets= that the answerer would use 3171 in the stream sent from the answerer if this configuration is 3172 finally used. 3174 The answerer also accepts the reception of the two configurations 3175 that payload types 99 and 100 represent. Again, the answerer needs 3176 to store parameter sets included in sprop-parameter-sets= and sprop-parameter-sets= in 3178 case the offer finally decides to use either of these two 3179 configurations. The answerer provides the initial parameter sets 3180 for the answerer-to-offerer direction, i.e. the parameter sets in 3181 sprop-parameter-sets= and sprop-parameter- 3182 sets=, for payload types 99 and 100, 3183 respectively, that it will use to send the payload types. The 3184 answerer also provides the offerer with its memory limit for de- 3185 interleaving operations by providing a "deint-buf-cap" parameter. 3186 This is only useful if the offerer decides on making a second offer, 3187 where it can take the new value into account. The "max-rcmd-nalu- 3188 size" indicates that the answerer can efficiently process NALUs up 3189 to the size of 3980 bytes. However, there is no guarantee that the 3190 network supports this size. 3192 In the following example, the offer is accepted without level 3193 downgrading (i.e. the default level, 3.0, is accepted), and both 3194 "sprop-parameter-sets" and "sprop-level-parameter-sets" are present 3195 in the offer. The answerer must ignore sprop-level-parameter- 3196 sets= and store parameter sets in sprop- 3197 parameter-sets= for decoding the incoming 3198 NAL unit stream. The offerer must store the parameter sets in 3199 sprop-parameter-sets= in the answer for 3200 decoding the incoming NAL unit stream. Note that in this example, 3201 parameter sets in sprop-parameter-sets= must 3202 be associated with level 3.0. 3204 Offer SDP: 3206 m=video 49170 RTP/AVP 98 3207 a=rtpmap:98 H264/90000 3208 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3209 packetization-mode=1; 3210 sprop-parameter-sets=; 3211 sprop-level-parameter-sets= 3213 Answer SDP: 3215 m=video 49170 RTP/AVP 98 3216 a=rtpmap:98 H264/90000 3217 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3218 packetization-mode=1; 3219 sprop-parameter-sets= 3221 In the following example, the offer (Baseline profile, level 1.1) 3222 is accepted with level downgrading (the accepted level is 1b), and 3223 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3224 present in the offer. The answerer must ignore sprop-parameter- 3225 sets= and all parameter sets not for the 3226 accepted level (level 1b) in sprop-level-parameter-sets=, and must store parameter sets for the accepted level 3228 (level 1b) in sprop-level-parameter-sets= 3229 for decoding the incoming NAL unit stream. The offerer must store 3230 the parameter sets in sprop-parameter-sets= 3231 in the answer for decoding the incoming NAL unit stream. Note that 3232 in this example, parameter sets in sprop-parameter-sets= must be associated with level 1b. 3235 Offer SDP: 3237 m=video 49170 RTP/AVP 98 3238 a=rtpmap:98 H264/90000 3239 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3240 packetization-mode=1; 3241 sprop-parameter-sets=; 3242 sprop-level-parameter-sets= 3244 Answer SDP: 3246 m=video 49170 RTP/AVP 98 3247 a=rtpmap:98 H264/90000 3248 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3249 packetization-mode=1; 3250 sprop-parameter-sets=; 3251 use-level-src-parameter-sets=1 3253 In the following example, the offer (Baseline profile, level 1.1) 3254 is accepted with level downgrading (the accepted level is 1b), and 3255 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3256 present in the offer. However, the answerer is a legacy RFC 3984 3257 implementation and does not understand "sprop-level-parameter-sets", 3258 hence it does not include "use-level-src-parameter-sets" (which the 3259 answerer does not understand, either) in the answer. Therefore, 3260 the answerer must ignore both sprop-parameter-sets= and sprop-level-parameter-sets=, and 3262 the offerer must transport parameter sets in-band. 3264 Offer SDP: 3266 m=video 49170 RTP/AVP 98 3267 a=rtpmap:98 H264/90000 3268 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3269 packetization-mode=1; 3270 sprop-parameter-sets=; 3271 sprop-level-parameter-sets= 3273 Answer SDP: 3275 m=video 49170 RTP/AVP 98 3276 a=rtpmap:98 H264/90000 3277 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3278 packetization-mode=1 3280 In the following example, the offer is accepted without level 3281 downgrading, and "sprop-parameter-sets" is present in the offer. 3282 Parameter sets in sprop-parameter-sets= must 3283 be stored and used used by the encoder of the offerer and the 3284 decoder of the answerer, and parameter sets in sprop-parameter- 3285 sets=must be used by the encoder of the 3286 answerer and the decoder of the offerer. Note that sprop- 3287 parameter-sets= is basically independent of 3288 sprop-parameter-sets=. 3290 Offer SDP: 3292 m=video 49170 RTP/AVP 98 3293 a=rtpmap:98 H264/90000 3294 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3295 packetization-mode=1; 3296 sprop-parameter-sets= 3298 Answer SDP: 3300 m=video 49170 RTP/AVP 98 3301 a=rtpmap:98 H264/90000 3302 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3303 packetization-mode=1; 3304 sprop-parameter-sets= 3306 In the following example, the offer is accepted without level 3307 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3308 parameter-sets" is present in the offer, meaning that there is no 3309 out-of-band transmission of parameter sets, which then have to be 3310 transported in-band. 3312 Offer SDP: 3314 m=video 49170 RTP/AVP 98 3315 a=rtpmap:98 H264/90000 3316 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3317 packetization-mode=1 3319 Answer SDP: 3321 m=video 49170 RTP/AVP 98 3322 a=rtpmap:98 H264/90000 3323 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3324 packetization-mode=1 3326 In the following example, the offer is accepted with level 3327 downgrading and "sprop-parameter-sets" is present in the offer. As 3328 sprop-parameter-sets= contains level_idc 3329 indicating Level 3.0, therefore cannot be used as the answerer 3330 wants Level 2.0 and must be ignored by the answerer, and in-band 3331 parameter sets must be used. 3333 Offer SDP: 3335 m=video 49170 RTP/AVP 98 3336 a=rtpmap:98 H264/90000 3337 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3338 packetization-mode=1; 3339 sprop-parameter-sets= 3341 Answer SDP: 3343 m=video 49170 RTP/AVP 98 3344 a=rtpmap:98 H264/90000 3345 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3346 packetization-mode=1 3348 In the following example, the offer is also accepted with level 3349 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3350 parameter-sets" is present in the offer, meaning that there is no 3351 out-of-band transmission of parameter sets, which then have to be 3352 transported in-band. 3354 Offer SDP: 3356 m=video 49170 RTP/AVP 98 3357 a=rtpmap:98 H264/90000 3358 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3359 packetization-mode=1 3361 Answer SDP: 3363 m=video 49170 RTP/AVP 98 3364 a=rtpmap:98 H264/90000 3365 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3366 packetization-mode=1 3368 In the following example, the offer is accepted with level 3369 upgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3370 parameter-sets" is present in the offer or the answer, meaning that 3371 there is no out-of-band transmission of parameter sets, which then 3372 have to be transported in-band. The level to use in the offerer- 3373 to-answerer direction is Level 3.0, and the level to use in the 3374 answerer-to-offerer direction is Level 2.0. The answerer is 3375 allowed to send at any level up to and including level 2.0, and the 3376 offerer is allowed to send at any level up to and including level 3377 3.0. 3379 Offer SDP: 3381 m=video 49170 RTP/AVP 98 3382 a=rtpmap:98 H264/90000 3383 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3384 packetization-mode=1; level-asymmetry-allowed=1 3386 Answer SDP: 3388 m=video 49170 RTP/AVP 98 3389 a=rtpmap:98 H264/90000 3390 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3391 packetization-mode=1; level-asymmetry-allowed=1 3393 In the following example, the offerer is a Multipoint Control Unit 3394 (MCU) in a Topo-Video-switch-MCU like topology [29], offering 3395 parameter sets received (using out-of-band transport) from three 3396 other participants B, C, and D, and receiving parameter sets from 3397 the participant A, which is the answerer. The participants are 3398 identified by their values of CNAME, which are mapped to different 3399 SSRC values. The same codec configuration is used by all the four 3400 participants. The participant A stores and associates the 3401 parameter sets included in , , and to participants B, C, and D, 3403 respectively, and uses for decoding NAL 3404 units carried in RTP packets originated from participant B only, 3405 uses for decoding NAL units carried in RTP 3406 packets originated from participant C only, and uses for decoding NAL units carried in RTP packets 3408 originated from participant D only. 3410 Offer SDP: 3412 m=video 49170 RTP/AVP 98 3413 a=ssrc:SSRC-B cname:CNAME-B 3414 a=ssrc:SSRC-C cname:CNAME-C 3415 a=ssrc:SSRC-D cname:CNAME-D 3416 a=ssrc:SSRC-B fmtp:98 3417 sprop-parameter-sets= 3418 a=ssrc:SSRC-C fmtp:98 3419 sprop-parameter-sets= 3420 a=ssrc:SSRC-D fmtp:98 3421 sprop-parameter-sets= 3422 a=rtpmap:98 H264/90000 3423 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3424 packetization-mode=1 3426 Answer SDP: 3428 m=video 49170 RTP/AVP 98 3429 a=ssrc:SSRC-A cname:CNAME-A 3430 a=ssrc:SSRC-A fmtp:98 3431 sprop-parameter-sets= 3432 a=rtpmap:98 H264/90000 3433 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3434 packetization-mode=1 3436 8.4. Parameter Set Considerations 3438 The H.264 parameter sets are a fundamental part of the video codec 3439 and vital to its operation; see section 1.2. Due to their 3440 characteristics and their importance for the decoding process, lost 3441 or erroneously transmitted parameter sets can hardly be concealed 3442 locally at the receiver. A reference to a corrupt parameter set 3443 has normally fatal results to the decoding process. Corruption 3444 could occur, for example, due to the erroneous transmission or loss 3445 of a parameter set NAL unit, but also due to the untimely 3446 transmission of a parameter set update. A parameter set update 3447 refers to a change of at least one parameter in a picture parameter 3448 set or sequence parameter set for which the picture parameter set 3449 or sequence parameter set identifier remains unchanged. Therefore, 3450 the following recommendations are provided as a guideline for the 3451 implementer of the RTP sender. 3453 Parameter set NALUs can be transported using three different 3454 principles: 3456 A. Using a session control protocol (out-of-band) prior to the 3457 actual RTP session. 3459 B. Using a session control protocol (out-of-band) during an ongoing 3460 RTP session. 3462 C. Within the RTP packet stream in the payload (in-band) during an 3463 ongoing RTP session. 3465 It is recommended to implement principles A and B within a session 3466 control protocol. SIP and SDP can be used as described in the SDP 3467 Offer/Answer model and in the previous sections of this memo. 3468 Section 8.2.2 includes a detailed discussion on transport of 3469 parameter sets in-band or out-of-band in SDP Offer/Answer using 3470 media type parameters "sprop-parameter-sets", "sprop-level- 3471 parameter-sets", "use-level-src-parameter-sets" and "in-band- 3472 parameter-sets". This section contains guidelines on how 3473 principles A and B should be implemented within session control 3474 protocols. It is independent of the particular protocol used. 3475 Principle C is supported by the RTP payload format defined in this 3476 specification. There are topologies like Topo-Video-switch-MCU [29] 3477 for which the use of principle C may be desirable. 3479 If in-band signaling of parameter sets is used, the picture and 3480 sequence parameter set NALUs SHOULD be transmitted in the RTP 3481 payload using a reliable method of delivering of RTP (see below), 3482 as a loss of a parameter set of either type will likely prevent 3483 decoding of a considerable portion of the corresponding RTP packet 3484 stream. 3486 If in-band signaling of parameter sets is used, the sender SHOULD 3487 take the error characteristics into account and use mechanisms to 3488 provide a high probability for delivering the parameter sets 3489 correctly. Mechanisms that increase the probability for a correct 3490 reception include packet repetition, FEC, and retransmission. The 3491 use of an unreliable, out-of-band control protocol has similar 3492 disadvantages as the in-band signaling (possible loss) and, in 3493 addition, may also lead to difficulties in the synchronization (see 3494 below). Therefore, it is NOT RECOMMENDED. 3496 Parameter sets MAY be added or updated during the lifetime of a 3497 session using principles B and C. It is required that parameter 3498 sets are present at the decoder prior to the NAL units that refer 3499 to them. Updating or adding of parameter sets can result in 3500 further problems, and therefore the following recommendations 3501 should be considered. 3503 - When parameter sets are added or updated, care SHOULD be taken 3504 to ensure that any parameter set is delivered prior to its usage. 3505 When new parameter sets are added, previously unused parameter 3506 set identifiers are used. It is common that no synchronization 3507 is present between out-of-band signaling and in-band traffic. 3508 If out-of-band signaling is used, it is RECOMMENDED that a 3509 sender does not start sending NALUs requiring the added or 3510 updated parameter sets prior to acknowledgement of delivery from 3511 the signaling protocol. 3513 - When parameter sets are updated, the following synchronization 3514 issue should be taken into account. When overwriting a 3515 parameter set at the receiver, the sender has to ensure that the 3516 parameter set in question is not needed by any NALU present in 3517 the network or receiver buffers. Otherwise, decoding with a 3518 wrong parameter set may occur. To lessen this problem, it is 3519 RECOMMENDED either to overwrite only those parameter sets that 3520 have not been used for a sufficiently long time (to ensure that 3521 all related NALUs have been consumed), or to add a new parameter 3522 set instead (which may have negative consequences for the 3523 efficiency of the video coding). 3525 Informative note: In some topologies like Topo-Video-switch- 3526 MCU [29] the origin of the whole set of parameter sets may 3527 come from multiple sources that may use non-unique parameter 3528 sets identifiers. In this case an offer may overwrite an 3529 existing parameter set if no other mechanism that enables 3530 uniqueness of the parameter sets in the out-of-band channel 3531 exists. 3533 - In a multiparty session, one participant MUST associate 3534 parameter sets coming from different sources with the source 3535 identification whenever possible, e.g. by conveying out-of-band 3536 transported parameter sets, as different sources typically use 3537 independent parameter set identifier value spaces. 3539 - Adding or modifying parameter sets by using both principles B 3540 and C in the same RTP session may lead to inconsistencies of the 3541 parameter sets because of the lack of synchronization between 3542 the control and the RTP channel. Therefore, principles B and C 3543 MUST NOT both be used in the same session unless sufficient 3544 synchronization can be provided. 3546 In some scenarios (e.g., when only the subset of this payload 3547 format specification corresponding to H.241 is used) or topologies, 3548 it is not possible to employ out-of-band parameter set transmission. 3549 In this case, parameter sets have to be transmitted in-band. Here, 3550 the synchronization with the non-parameter-set-data in the 3551 bitstream is implicit, but the possibility of a loss has to be 3552 taken into account. The loss probability should be reduced using 3553 the mechanisms discussed above. In case a loss of a parameter set 3554 is detected, recovery may be achieved by using a Decoder Refresh 3555 Point procedure, for example, using RTCP feedback Full Intra 3556 Request (FIR) [30]. Two example Decoder Refresh Point procedures 3557 are provided in the informative Section 8.5. 3559 - When parameter sets are initially provided using principle A and 3560 then later added or updated in-band (principle C), there is a 3561 risk associated with updating the parameter sets delivered out- 3562 of-band. If receivers miss some in-band updates (for example, 3563 because of a loss or a late tune-in), those receivers attempt to 3564 decode the bitstream using out-dated parameters. It is 3565 therefore RECOMMENDED that parameter set IDs be partitioned 3566 between the out-of-band and in-band parameter sets. 3568 8.5. Decoder Refresh Point Procedure using In-Band Transport of 3569 Parameter Sets (Informative) 3571 When a sender with a video encoder according to [1] receives a 3572 request for a decoder refresh point, the encoder shall enter the 3573 fast update mode by using one of the procedures specified 3574 in Section 8.5.1 or 8.5.2 below. The procedure in 8.5.1 is the 3575 preferred response in a lossless transmission environment. Both 3576 procedures satisfy the requirement to enter the fast update mode 3577 for H.264 video encoding. 3579 8.5.1. IDR Procedure to Respond to a Request for a Decoder Refresh 3580 Point 3582 This section gives one possible way to respond to a request for a 3583 decoder refresh point. 3585 The encoder shall, in the order presented here: 3587 1) Immediately prepare to send an IDR picture. 3589 2) Send a sequence parameter set to be used by the IDR picture to 3590 be sent. The encoder may optionally also send other sequence 3591 parameter sets. 3593 3) Send a picture parameter set to be used by the IDR picture to be 3594 sent. The encoder may optionally also send other picture 3595 parameter sets. 3597 4) Send the IDR picture. 3599 5) From this point forward in time, send any other sequence or 3600 picture parameter sets that have not yet been sent in this 3601 procedure, prior to their reference by any NAL unit, regardless 3602 of whether such parameter sets were previously sent prior to 3603 receiving the request for a decoder refresh point. As needed, 3604 such parameter sets may be sent in a batch, one at a time, or in 3605 any combination of these two methods. Parameter sets may be re- 3606 sent at any time for redundancy. Caution should be taken when 3607 parameter set updates are present, as described above in Section 3608 8.4. 3610 8.5.2. Gradual Recovery Procedure to Respond to a Request for a 3611 Decoder Refresh Point 3613 This section gives another possible way to respond to a request for 3614 a decoder refresh point. 3616 The encoder shall, in the order presented here: 3618 1) Send a recovery point SEI message (see Sections D.1.7 and D.2.7 3619 of [1]). 3621 2) Repeat any sequence and picture parameter sets that were sent 3622 before the recovery point SEI message, prior to their reference 3623 by a NAL unit. 3625 The encoder shall ensure that the decoder has access to all 3626 reference pictures for inter prediction of pictures at or after the 3627 recovery point, which is indicated by the recovery point SEI 3628 message, in output order, assuming that the transmission from now 3629 on is error-free. 3631 The value of the recovery_frame_cnt syntax element in the recovery 3632 point SEI message should be small enough to ensure a fast recovery. 3634 As needed, such parameter sets may be re-sent in a batch, one at a 3635 time, or in any combination of these two methods. Parameter sets 3636 may be re-sent at any time for redundancy. Caution should be taken 3637 when parameter set updates are present, as described above in 3638 Section 8.4. 3640 9. Security Considerations 3642 RTP packets using the payload format defined in this specification 3643 are subject to the security considerations discussed in the RTP 3644 specification [5], and in any appropriate RTP profile (for example, 3645 [16]). This implies that confidentiality of the media streams is 3646 achieved by encryption; for example, through the application of 3647 SRTP [26]. Because the data compression used with this payload 3648 format is applied end-to-end, any encryption needs to be performed 3649 after compression. A potential denial-of-service threat exists for 3650 data encodings using compression techniques that have non-uniform 3651 receiver-end computational load. The attacker can inject 3652 pathological datagrams into the stream that are complex to decode 3653 and that cause the receiver to be overloaded. H.264 is 3654 particularly vulnerable to such attacks, as it is extremely simple 3655 to generate datagrams containing NAL units that affect the decoding 3656 process of many future NAL units. Therefore, the usage of data 3657 origin authentication and data integrity protection of at least the 3658 RTP packet is RECOMMENDED; for example, with SRTP [26]. 3660 Note that the appropriate mechanism to ensure confidentiality and 3661 integrity of RTP packets and their payloads is very dependent on 3662 the application and on the transport and signaling protocols 3663 employed. Thus, although SRTP is given as an example above, other 3664 possible choices exist. 3666 Decoders MUST exercise caution with respect to the handling of user 3667 data SEI messages, particularly if they contain active elements, 3668 and MUST restrict their domain of applicability to the presentation 3669 containing the stream. 3671 End-to-End security with either authentication, integrity or 3672 confidentiality protection will prevent a MANE from performing 3673 media-aware operations other than discarding complete packets. And 3674 in the case of confidentiality protection it will even be prevented 3675 from performing discarding of packets in a media aware way. To 3676 allow any MANE to perform its operations, it will be required to be 3677 a trusted entity which is included in the security context 3678 establishment. 3680 10. Congestion Control 3682 Congestion control for RTP SHALL be used in accordance with RFC 3683 3550 [5], and with any applicable RTP profile; e.g., RFC 3551 [16]. 3684 An additional requirement if best-effort service is being used is: 3685 users of this payload format MUST monitor packet loss to ensure 3686 that the packet loss rate is within acceptable parameters. Packet 3687 loss is considered acceptable if a TCP flow across the same network 3688 path, and experiencing the same network conditions, would achieve 3689 an average throughput, measured on a reasonable timescale, that is 3690 not less than the RTP flow is achieving. This condition can be 3691 satisfied by implementing congestion control mechanisms to adapt 3692 the transmission rate (or the number of layers subscribed for a 3693 layered multicast session), or by arranging for a receiver to leave 3694 the session if the loss rate is unacceptably high. 3696 The bit rate adaptation necessary for obeying the congestion 3697 control principle is easily achievable when real-time encoding is 3698 used. However, when pre-encoded content is being transmitted, 3699 bandwidth adaptation requires the availability of more than one 3700 coded representation of the same content, at different bit rates, 3701 or the existence of non-reference pictures or sub-sequences [22] in 3702 the bitstream. The switching between the different representations 3703 can normally be performed in the same RTP session; e.g., by 3704 employing a concept known as SI/SP slices of the Extended Profile, 3705 or by switching streams at IDR picture boundaries. Only when non- 3706 downgradable parameters (such as the profile part of the 3707 profile/level ID) are required to be changed does it become 3708 necessary to terminate and re-start the media stream. This may be 3709 accomplished by using a different RTP payload type. 3711 MANEs MAY follow the suggestions outlined in section 7.3 and remove 3712 certain unusable packets from the packet stream when that stream 3713 was damaged due to previous packet losses. This can help reduce 3714 the network load in certain special cases. 3716 11. IANA Consideration 3718 The H264 media subtype name specified by RFC 3984 should be updated 3719 as defined in section 8.1 of this memo. 3721 12. Informative Appendix: Application Examples 3723 This payload specification is very flexible in its use, in order to 3724 cover the extremely wide application space anticipated for H.264. 3725 However, this great flexibility also makes it difficult for an 3726 implementer to decide on a reasonable packetization scheme. Some 3727 information on how to apply this specification to real-world 3728 scenarios is likely to appear in the form of academic publications 3729 and a test model software and description in the near future. 3730 However, some preliminary usage scenarios are described here as 3731 well. 3733 12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 3735 H.323-based video telephony systems that use H.264 as an optional 3736 video compression scheme are required to support H.241 Annex A [3] 3737 as a packetization scheme. The packetization mechanism defined in 3738 this Annex is technically identical with a small subset of this 3739 specification. 3741 When a system operates according to H.241 Annex A, parameter set 3742 NAL units are sent in-band. Only Single NAL unit packets are used. 3743 Many such systems are not sending IDR pictures regularly, but only 3744 when required by user interaction or by control protocol means; 3745 e.g., when switching between video channels in a Multipoint Control 3746 Unit or for error recovery requested by feedback. 3748 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 3749 Aggregation 3751 The RTP part of this scheme is implemented and tested (though not 3752 the control-protocol part; see below). 3754 In most real-world video telephony applications, picture parameters 3755 such as picture size or optional modes never change during the 3756 lifetime of a connection. Therefore, all necessary parameter sets 3757 (usually only one) are sent as a side effect of the capability 3758 exchange/announcement process, e.g., according to the SDP syntax 3759 specified in section 8.2 of this document. As all necessary 3760 parameter set information is established before the RTP session 3761 starts, there is no need for sending any parameter set NAL units. 3762 Slice data partitioning is not used, either. Thus, the RTP packet 3763 stream basically consists of NAL units that carry single coded 3764 slices. 3766 The encoder chooses the size of coded slice NAL units so that they 3767 offer the best performance. Often, this is done by adapting the 3768 coded slice size to the MTU size of the IP network. For small 3769 picture sizes, this may result in a one-picture-per-one-packet 3770 strategy. Intra refresh algorithms clean up the loss of packets 3771 and the resulting drift-related artifacts. 3773 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 3774 Aggregation 3776 This scheme allows better error concealment and is used in H.263 3777 based designs using RFC 4629 packetization [11]. It has been 3778 implemented, and good results were reported [13]. 3780 The VCL encoder codes the source picture so that all macroblocks 3781 (MBs) of one MB line are assigned to one slice. All slices with 3782 even MB row addresses are combined into one STAP, and all slices 3783 with odd MB row addresses into another. Those STAPs are 3784 transmitted as RTP packets. The establishment of the parameter 3785 sets is performed as discussed above. 3787 Note that the use of STAPs is essential here, as the high number of 3788 individual slices (18 for a CIF picture) would lead to unacceptably 3789 high IP/UDP/RTP header overhead (unless the source coding tool FMO 3790 is used, which is not assumed in this scenario). Furthermore, some 3791 wireless video transmission systems, such as H.324M and the IP- 3792 based video telephony specified in 3GPP, are likely to use 3793 relatively small transport packet size. For example, a typical MTU 3794 size of H.223 AL3 SDU is around 100 bytes [17]. Coding individual 3795 slices according to this packetization scheme provides further 3796 advantage in communication between wired and wireless networks, as 3797 individual slices are likely to be smaller than the preferred 3798 maximum packet size of wireless systems. Consequently, a gateway 3799 can convert the STAPs used in a wired network into several RTP 3800 packets with only one NAL unit, which are preferred in a wireless 3801 network, and vice versa. 3803 12.4. Video Telephony with Data Partitioning 3805 This scheme has been implemented and has been shown to offer good 3806 performance, especially at higher packet loss rates [13]. 3808 Data Partitioning is known to be useful only when some form of 3809 unequal error protection is available. Normally, in single-session 3810 RTP environments, even error characteristics are assumed; i.e., the 3811 packet loss probability of all packets of the session is the same 3812 statistically. However, there are means to reduce the packet loss 3813 probability of individual packets in an RTP session. A FEC packet 3814 according to RFC 2733 [18], for example, specifies which media 3815 packets are associated with the FEC packet. 3817 In all cases, the incurred overhead is substantial but is in the 3818 same order of magnitude as the number of bits that have otherwise 3819 been spent for intra information. However, this mechanism does not 3820 add any delay to the system. 3822 Again, the complete parameter set establishment is performed 3823 through control protocol means. 3825 12.5. Video Telephony or Streaming with FUs and Forward Error 3826 Correction 3828 This scheme has been implemented and has been shown to provide good 3829 performance, especially at higher packet loss rates [19]. 3831 The most efficient means to combat packet losses for scenarios 3832 where retransmissions are not applicable is forward error 3833 correction (FEC). Although application layer, end-to-end use of 3834 FEC is often less efficient than an FEC-based protection of 3835 individual links (especially when links of different 3836 characteristics are in the transmission path), application layer, 3837 end-to-end FEC is unavoidable in some scenarios. RFC 5109 [18] 3838 provides means to use generic, application layer, end-to-end FEC in 3839 packet-loss environments. A binary forward error correcting code 3840 is generated by applying the XOR operation to the bits at the same 3841 bit position in different packets. The binary code can be 3842 specified by the parameters (n,k) in which k is the number of 3843 information packets used in the connection and n is the total 3844 number of packets generated for k information packets; i.e., n-k 3845 parity packets are generated for k information packets. 3847 When a code is used with parameters (n,k) within the RFC 5109 3848 framework, the following properties are well known: 3850 a) If applied over one RTP packet, RFC 5109 provides only packet 3851 repetition. 3853 b) RFC 5109 is most bit rate efficient if XOR-connected packets 3854 have equal length. 3856 c) At the same packet loss probability p and for a fixed k, the 3857 greater the value of n is, the smaller the residual error 3858 probability becomes. For example, for a packet loss probability 3859 of 10%, k=1, and n=2, the residual error probability is about 1%, 3860 whereas for n=3, the residual error probability is about 0.1%. 3862 d) At the same packet loss probability p and for a fixed code rate 3863 k/n, the greater the value of n is, the smaller the residual 3864 error probability becomes. For example, at a packet loss 3865 probability of p=10%, k=1 and n=2, the residual error rate is 3866 about 1%, whereas for an extended Golay code with k=12 and n=24, 3867 the residual error rate is about 0.01%. 3869 For applying RFC 5109 in combination with H.264 baseline coded 3870 video without using FUs, several options might be considered: 3872 1) The video encoder produces NAL units for which each video frame 3873 is coded in a single slice. Applying FEC, one could use a 3874 simple code; e.g., (n=2, k=1). That is, each NAL unit would 3875 basically just be repeated. The disadvantage is obviously the 3876 bad code performance according to d), above, and the low 3877 flexibility, as only (n, k=1) codes can be used. 3879 2) The video encoder produces NAL units for which each video frame 3880 is encoded in one or more consecutive slices. Applying FEC, one 3881 could use a better code, e.g., (n=24, k=12), over a sequence of 3882 NAL units. Depending on the number of RTP packets per frame, a 3883 loss may introduce a significant delay, which is reduced when 3884 more RTP packets are used per frame. Packets of completely 3885 different length might also be connected, which decreases bit 3886 rate efficiency according to b), above. However, with some care 3887 and for slices of 1kb or larger, similar length (100-200 bytes 3888 difference) may be produced, which will not lower the bit 3889 efficiency catastrophically. 3891 3) The video encoder produces NAL units, for which a certain frame 3892 contains k slices of possibly almost equal length. Then, 3893 applying FEC, a better code, e.g., (n=24, k=12), can be used 3894 over the sequence of NAL units for each frame. The delay 3895 compared to that of 2), above, may be reduced, but several 3896 disadvantages are obvious. First, the coding efficiency of the 3897 encoded video is lowered significantly, as slice-structured 3898 coding reduces intra-frame prediction and additional slice 3899 overhead is necessary. Second, pre-encoded content or, when 3900 operating over a gateway, the video is usually not appropriately 3901 coded with k slices such that FEC can be applied. Finally, the 3902 encoding of video producing k slices of equal length is not 3903 straightforward and might require more than one encoding pass. 3905 Many of the mentioned disadvantages can be avoided by applying FUs 3906 in combination with FEC. Each NAL unit can be split into any 3907 number of FUs of basically equal length; therefore, FEC with a 3908 reasonable k and n can be applied, even if the encoder made no 3909 effort to produce slices of equal length. For example, a coded 3910 slice NAL unit containing an entire frame can be split to k FUs, 3911 and a parity check code (n=k+1, k) can be applied. However, this 3912 has the disadvantage that unless all created fragments can be 3913 recovered, the whole slice will be lost. Thus a larger section is 3914 lost than would be if the frame had been split into several slices. 3916 The presented technique makes it possible to achieve good 3917 transmission error tolerance, even if no additional source coding 3918 layer redundancy (such as periodic intra frames) is present. 3919 Consequently, the same coded video sequence can be used to achieve 3920 the maximum compression efficiency and quality over error-free 3921 transmission and for transmission over error-prone networks. 3922 Furthermore, the technique allows the application of FEC to pre- 3923 encoded sequences without adding delay. In this case, pre-encoded 3924 sequences that are not encoded for error-prone networks can still 3925 be transmitted almost reliably without adding extensive delays. In 3926 addition, FUs of equal length result in a bit rate efficient use of 3927 RFC 5109. 3929 If the error probability depends on the length of the transmitted 3930 packet (e.g., in case of mobile transmission [15]), the benefits of 3931 applying FUs with FEC are even more obvious. Basically, the 3932 flexibility of the size of FUs allows appropriate FEC to be applied 3933 for each NAL unit and unequal error protection of NAL units. 3935 When FUs and FEC are used, the incurred overhead is substantial but 3936 is in the same order of magnitude as the number of bits that have 3937 to be spent for intra-coded macroblocks if no FEC is applied. In 3938 [19], it was shown that the overall performance of the FEC-based 3939 approach enhanced quality when using the same error rate and same 3940 overall bit rate, including the overhead. 3942 12.6. Low Bit-Rate Streaming 3944 This scheme has been implemented with H.263 and non-standard RTP 3945 packetization and has given good results [20]. There is no 3946 technical reason why similarly good results could not be achievable 3947 with H.264. 3949 In today's Internet streaming, some of the offered bit rates are 3950 relatively low in order to allow terminals with dial-up modems to 3951 access the content. In wired IP networks, relatively large packets, 3952 say 500 - 1500 bytes, are preferred to smaller and more frequently 3953 occurring packets in order to reduce network congestion. Moreover, 3954 use of large packets decreases the amount of RTP/UDP/IP header 3955 overhead. For low bit-rate video, the use of large packets means 3956 that sometimes up to few pictures should be encapsulated in one 3957 packet. 3959 However, loss of a packet including many coded pictures would have 3960 drastic consequences for visual quality, as there is practically no 3961 other way to conceal a loss of an entire picture than to repeat the 3962 previous one. One way to construct relatively large packets and 3963 maintain possibilities for successful loss concealment is to 3964 construct MTAPs that contain interleaved slices from several 3965 pictures. An MTAP should not contain spatially adjacent slices 3966 from the same picture or spatially overlapping slices from any 3967 picture. If a packet is lost, it is likely that a lost slice is 3968 surrounded by spatially adjacent slices of the same picture and 3969 spatially corresponding slices of the temporally previous and 3970 succeeding pictures. Consequently, concealment of the lost slice 3971 is likely to be relatively successful. 3973 12.7. Robust Packet Scheduling in Video Streaming 3975 Robust packet scheduling has been implemented with MPEG-4 Part 2 3976 and simulated in a wireless streaming environment [21]. There is 3977 no technical reason why similar or better results could not be 3978 achievable with H.264. 3980 Streaming clients typically have a receiver buffer that is capable 3981 of storing a relatively large amount of data. Initially, when a 3982 streaming session is established, a client does not start playing 3983 the stream back immediately. Rather, it typically buffers the 3984 incoming data for a few seconds. This buffering helps maintain 3985 continuous playback, as, in case of occasional increased 3986 transmission delays or network throughput drops, the client can 3987 decode and play buffered data. Otherwise, without initial 3988 buffering, the client has to freeze the display, stop decoding, and 3989 wait for incoming data. The buffering is also necessary for either 3990 automatic or selective retransmission in any protocol level. If 3991 any part of a picture is lost, a retransmission mechanism may be 3992 used to resend the lost data. If the retransmitted data is 3993 received before its scheduled decoding or playback time, the loss 3994 is recovered perfectly. Coded pictures can be ranked according to 3995 their importance in the subjective quality of the decoded sequence. 3996 For example, non-reference pictures, such as conventional B 3997 pictures, are subjectively least important, as their absence does 3998 not affect decoding of any other pictures. In addition to non- 3999 reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard 4000 includes a temporal scalability method called sub-sequences [22]. 4001 Subjective ranking can also be made on coded slice data partition 4002 or slice group basis. Coded slices and coded slice data partitions 4003 that are subjectively the most important can be sent earlier than 4004 their decoding order indicates, whereas coded slices and coded 4005 slice data partitions that are subjectively the least important can 4006 be sent later than their natural coding order indicates. 4007 Consequently, any retransmitted parts of the most important slices 4008 and coded slice data partitions are more likely to be received 4009 before their scheduled decoding or playback time compared to the 4010 least important slices and slice data partitions. 4012 13. Informative Appendix: Rationale for Decoding Order Number 4014 13.1. Introduction 4016 The Decoding Order Number (DON) concept was introduced mainly to 4017 enable efficient multi-picture slice interleaving (see section 12.6) 4018 and robust packet scheduling (see section 12.7). In both of these 4019 applications, NAL units are transmitted out of decoding order. DON 4020 indicates the decoding order of NAL units and should be used in the 4021 receiver to recover the decoding order. Example use cases for 4022 efficient multi-picture slice interleaving and for robust packet 4023 scheduling are given in sections 13.2 and 13.3, respectively. 4024 Section 13.4 describes the benefits of the DON concept in error 4025 resiliency achieved by redundant coded pictures. Section 13.5 4026 summarizes considered alternatives to DON and justifies why DON was 4027 chosen to this RTP payload specification. 4029 13.2. Example of Multi-Picture Slice Interleaving 4031 An example of multi-picture slice interleaving follows. A subset 4032 of a coded video sequence is depicted below in output order. R 4033 denotes a reference picture, N denotes a non-reference picture, and 4034 the number indicates a relative output time. 4036 ... R1 N2 R3 N4 R5 ... 4038 The decoding order of these pictures from left to right is as 4039 follows: 4041 ... R1 R3 N2 R5 N4 ... 4043 The NAL units of pictures R1, R3, N2, R5, and N4 are marked with a 4044 DON equal to 1, 2, 3, 4, and 5, respectively. 4046 Each reference picture consists of three slice groups that are 4047 scattered as follows (a number denotes the slice group number for 4048 each macroblock in a QCIF frame): 4050 0 1 2 0 1 2 0 1 2 0 1 4051 2 0 1 2 0 1 2 0 1 2 0 4052 1 2 0 1 2 0 1 2 0 1 2 4053 0 1 2 0 1 2 0 1 2 0 1 4054 2 0 1 2 0 1 2 0 1 2 0 4055 1 2 0 1 2 0 1 2 0 1 2 4056 0 1 2 0 1 2 0 1 2 0 1 4057 2 0 1 2 0 1 2 0 1 2 0 4058 1 2 0 1 2 0 1 2 0 1 2 4060 For the sake of simplicity, we assume that all the macroblocks of a 4061 slice group are included in one slice. Three MTAPs are constructed 4062 from three consecutive reference pictures so that each MTAP 4063 contains three aggregation units, each of which contains all the 4064 macroblocks from one slice group. The first MTAP contains slice 4065 group 0 of picture R1, slice group 1 of picture R3, and slice group 4066 2 of picture R5. The second MTAP contains slice group 1 of picture 4067 R1, slice group 2 of picture R3, and slice group 0 of picture R5. 4068 The third MTAP contains slice group 2 of picture R1, slice group 0 4069 of picture R3, and slice group 1 of picture R5. Each non-reference 4070 picture is encapsulated into an STAP-B. 4072 Consequently, the transmission order of NAL units is the following: 4074 R1, slice group 0, DON 1, carried in MTAP,RTP SN: N 4075 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N 4076 R5, slice group 2, DON 4, carried in MTAP,RTP SN: N 4077 R1, slice group 1, DON 1, carried in MTAP,RTP SN: N+1 4078 R3, slice group 2, DON 2, carried in MTAP,RTP SN: N+1 4079 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+1 4080 R1, slice group 2, DON 1, carried in MTAP,RTP SN: N+2 4081 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N+2 4082 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+2 4083 N2, DON 3, carried in STAP-B, RTP SN: N+3 4084 N4, DON 5, carried in STAP-B, RTP SN: N+4 4086 The receiver is able to organize the NAL units back in decoding 4087 order based on the value of DON associated with each NAL unit. 4089 If one of the MTAPs is lost, the spatially adjacent and temporally 4090 co-located macroblocks are received and can be used to conceal the 4091 loss efficiently. If one of the STAPs is lost, the effect of the 4092 loss does not propagate temporally. 4094 13.3. Example of Robust Packet Scheduling 4096 An example of robust packet scheduling follows. The communication 4097 system used in the example consists of the following components in 4098 the order that the video is processed from source to sink: 4100 o camera and capturing 4101 o pre-encoding buffer 4102 o encoder 4103 o encoded picture buffer 4104 o transmitter 4105 o transmission channel 4106 o receiver 4107 o receiver buffer 4108 o decoder 4109 o decoded picture buffer 4110 o display 4112 The video communication system used in the example operates as 4113 follows. Note that processing of the video stream happens 4114 gradually and at the same time in all components of the system. 4115 The source video sequence is shot and captured to a pre-encoding 4116 buffer. The pre-encoding buffer can be used to order pictures from 4117 sampling order to encoding order or to analyze multiple 4118 uncompressed frames for bit rate control purposes, for example. In 4119 some cases, the pre-encoding buffer may not exist; instead, the 4120 sampled pictures are encoded right away. The encoder encodes 4121 pictures from the pre-encoding buffer and stores the output; i.e., 4122 coded pictures, to the encoded picture buffer. The transmitter 4123 encapsulates the coded pictures from the encoded picture buffer to 4124 transmission packets and sends them to a receiver through a 4125 transmission channel. The receiver stores the received packets to 4126 the receiver buffer. The receiver buffering process typically 4127 includes buffering for transmission delay jitter. The receiver 4128 buffer can also be used to recover correct decoding order of coded 4129 data. The decoder reads coded data from the receiver buffer and 4130 produces decoded pictures as output into the decoded picture buffer. 4131 The decoded picture buffer is used to recover the output (or 4132 display) order of pictures. Finally, pictures are displayed. 4134 In the following example figures, I denotes an IDR picture, R 4135 denotes a reference picture, N denotes a non-reference picture, and 4136 the number after I, R, or N indicates the sampling time relative to 4137 the previous IDR picture in decoding order. Values below the 4138 sequence of pictures indicate scaled system clock timestamps. The 4139 system clock is initialized arbitrarily in this example, and time 4140 runs from left to right. Each I, R, and N picture is mapped into 4141 the same timeline compared to the previous processing step, if any, 4142 assuming that encoding, transmission, and decoding take no time. 4143 Thus, events happening at the same time are located in the same 4144 column throughout all example figures. 4146 A subset of a sequence of coded pictures is depicted below in 4147 sampling order. 4149 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ... 4150 ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ... 4151 ... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ... 4153 Figure 16 Sequence of pictures in sampling order 4155 The sampled pictures are buffered in the pre-encoding buffer to 4156 arrange them in encoding order. In this example, we assume that 4157 the non-reference pictures are predicted from both the previous and 4158 the next reference picture in output order, except for the non- 4159 reference pictures immediately preceding an IDR picture, which are 4160 predicted only from the previous reference picture in output order. 4161 Thus, the pre-encoding buffer has to contain at least two pictures, 4162 and the buffering causes a delay of two picture intervals. The 4163 output of the pre-encoding buffering process and the encoding (and 4164 decoding) order of the pictures are as follows: 4166 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4167 ... -|---|---|---|---|---|---|---|---|- ... 4168 ... 60 61 62 63 64 65 66 67 68 ... 4170 Figure 17 Re-ordered pictures in the pre-encoding buffer 4172 The encoder or the transmitter can set the value of DON for each 4173 picture to a value of DON for the previous picture in decoding 4174 order plus one. 4176 For the sake of simplicity, let us assume that: 4178 o the frame rate of the sequence is constant, 4179 o each picture consists of only one slice, 4180 o each slice is encapsulated in a single NAL unit packet, 4181 o there is no transmission delay, and 4182 o pictures are transmitted at constant intervals (that is, 1 / 4183 (frame rate)). 4185 When pictures are transmitted in decoding order, they are received 4186 as follows: 4188 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4189 ... -|---|---|---|---|---|---|---|---|- ... 4190 ... 60 61 62 63 64 65 66 67 68 ... 4192 Figure 18 Received pictures in decoding order 4194 The OPTIONAL sprop-interleaving-depth media type parameter is set 4195 to 0, as the transmission (or reception) order is identical to the 4196 decoding order. 4198 The decoder has to buffer for one picture interval initially in its 4199 decoded picture buffer to organize pictures from decoding order to 4200 output order as depicted below: 4202 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... 4203 ... -|---|---|---|---|---|---|---|---|- ... 4204 ... 61 62 63 64 65 66 67 68 69 ... 4206 Figure 19 Output order 4208 The amount of required initial buffering in the decoded picture 4209 buffer can be signaled in the buffering period SEI message or with 4210 the num_reorder_frames syntax element of H.264 video usability 4211 information. num_reorder_frames indicates the maximum number of 4212 frames, complementary field pairs, or non-paired fields that 4213 precede any frame, complementary field pair, or non-paired field in 4214 the sequence in decoding order and that follow it in output order. 4215 For the sake of simplicity, we assume that num_reorder_frames is 4216 used to indicate the initial buffer in the decoded picture buffer. 4217 In this example, num_reorder_frames is equal to 1. 4219 It can be observed that if the IDR picture I00 is lost during 4220 transmission and a retransmission request is issued when the value 4221 of the system clock is 62, there is one picture interval of time 4222 (until the system clock reaches timestamp 63) to receive the 4223 retransmitted IDR picture I00. 4225 Let us then assume that IDR pictures are transmitted two frame 4226 intervals earlier than their decoding position; i.e., the pictures 4227 are transmitted as follows: 4229 ... I00 N58 N59 R03 N01 N02 R06 N04 N05 ... 4230 ... --|---|---|---|---|---|---|---|---|- ... 4231 ... 62 63 64 65 66 67 68 69 70 ... 4233 Figure 20 Interleaving: Early IDR pictures in sending order 4235 The OPTIONAL sprop-interleaving-depth media type parameter is set 4236 equal to 1 according to its definition. (The value of sprop- 4237 interleaving-depth in this example can be derived as follows: 4238 Picture I00 is the only picture preceding picture N58 or N59 in 4239 transmission order and following it in decoding order. Except for 4240 pictures I00, N58, and N59, the transmission order is the same as 4241 the decoding order of pictures. As a coded picture is encapsulated 4242 into exactly one NAL unit, the value of sprop-interleaving-depth is 4243 equal to the maximum number of pictures preceding any picture in 4244 transmission order and following the picture in decoding order.) 4246 The receiver buffering process contains two pictures at a time 4247 according to the value of the sprop-interleaving-depth parameter 4248 and orders pictures from the reception order to the correct 4249 decoding order based on the value of DON associated with each 4250 picture. The output of the receiver buffering process is as 4251 follows: 4253 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4254 ... -|---|---|---|---|---|---|---|---|- ... 4255 ... 63 64 65 66 67 68 69 70 71 ... 4257 Figure 21 Interleaving: Receiver buffer 4259 Again, an initial buffering delay of one picture interval is needed 4260 to organize pictures from decoding order to output order, as 4261 depicted below: 4263 ... N58 N59 I00 N01 N02 R03 N04 N05 ... 4264 ... -|---|---|---|---|---|---|---|- ... 4265 ... 64 65 66 67 68 69 70 71 ... 4267 Figure 22 Interleaving: Receiver buffer after reordering 4269 Note that the maximum delay that IDR pictures can undergo during 4270 transmission, including possible application, transport, or link 4271 layer retransmission, is equal to three picture intervals. Thus, 4272 the loss resiliency of IDR pictures is improved in systems 4273 supporting retransmission compared to the case in which pictures 4274 were transmitted in their decoding order. 4276 13.4. Robust Transmission Scheduling of Redundant Coded Slices 4278 A redundant coded picture is a coded representation of a picture or 4279 a part of a picture that is not used in the decoding process if the 4280 corresponding primary coded picture is correctly decoded. There 4281 should be no noticeable difference between any area of the decoded 4282 primary picture and a corresponding area that would result from 4283 application of the H.264 decoding process for any redundant picture 4284 in the same access unit. A redundant coded slice is a coded slice 4285 that is a part of a redundant coded picture. 4287 Redundant coded pictures can be used to provide unequal error 4288 protection in error-prone video transmission. If a primary coded 4289 representation of a picture is decoded incorrectly, a corresponding 4290 redundant coded picture can be decoded. Examples of applications 4291 and coding techniques using the redundant codec picture feature 4292 include the video redundancy coding [23] and the protection of "key 4293 pictures" in multicast streaming [24]. 4295 One property of many error-prone video communications systems is 4296 that transmission errors are often bursty. Therefore, they may 4297 affect more than one consecutive transmission packets in 4298 transmission order. In low bit-rate video communication, it is 4299 relatively common that an entire coded picture can be encapsulated 4300 into one transmission packet. Consequently, a primary coded 4301 picture and the corresponding redundant coded pictures may be 4302 transmitted in consecutive packets in transmission order. To make 4303 the transmission scheme more tolerant of bursty transmission errors, 4304 it is beneficial to transmit the primary coded picture and 4305 redundant coded picture separated by more than a single packet. 4306 The DON concept enables this. 4308 13.5. Remarks on Other Design Possibilities 4310 The slice header syntax structure of the H.264 coding standard 4311 contains the frame_num syntax element that can indicate the 4312 decoding order of coded frames. However, the usage of the 4313 frame_num syntax element is not feasible or desirable to recover 4314 the decoding order, due to the following reasons: 4316 o The receiver is required to parse at least one slice header per 4317 coded picture (before passing the coded data to the decoder). 4319 o Coded slices from multiple coded video sequences cannot be 4320 interleaved, as the frame number syntax element is reset to 0 in 4321 each IDR picture. 4323 o The coded fields of a complementary field pair share the same 4324 value of the frame_num syntax element. Thus, the decoding order 4325 of the coded fields of a complementary field pair cannot be 4326 recovered based on the frame_num syntax element or any other 4327 syntax element of the H.264 coding syntax. 4329 The RTP payload format for transport of MPEG-4 elementary streams 4330 [25] enables interleaving of access units and transmission of 4331 multiple access units in the same RTP packet. An access unit is 4332 specified in the H.264 coding standard to comprise all NAL units 4333 associated with a primary coded picture according to subclause 4334 7.4.1.2 of [1]. Consequently, slices of different pictures cannot 4335 be interleaved, and the multi-picture slice interleaving technique 4336 (see section 12.6) for improved error resilience cannot be used. 4338 14. Acknowledgements 4340 Stephan Wenger, Miska Hannuksela, Thomas Stockhammer, Magnus 4341 Westerlund, and David Singer are thanked as the authors of RFC 3984. 4342 Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, 4343 Joerg Ott, and Colin Perkins are thanked for careful review during 4344 the development of RFC 3984. Randell Jesup, Stephen Botzko, Magnus 4345 Westerlund, Alex Eleftheriadis, Thomas Schierl, Tom Taylor, Ali 4346 Begen, and Aaron Wells are thanked for their valuable comments and 4347 inputs during the development of this memo. 4349 This document was prepared using 2-Word-v2.0.template.dot. 4351 15. References 4353 15.1. Normative References 4355 [1] ITU-T Recommendation H.264, "Advanced video coding for 4356 generic audiovisual services", November 2007. 4358 [2] ISO/IEC International Standard 14496-10:2008. 4360 [3] ITU-T Recommendation H.241, "Extended video procedures and 4361 control signals for H.300 series terminals", May 2006. 4363 [4] Bradner, S., "Key words for use in RFCs to Indicate 4364 Requirement Levels", BCP 14, RFC 2119, March 1997. 4366 [5] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, 4367 "RTP: A Transport Protocol for Real-Time Applications", STD 4368 64, RFC 3550, July 2003. 4370 [6] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 4371 Description Protocol", RFC 4566, July 2006. 4373 [7] Josefsson, S., "The Base16, Base32, and Base64 Data 4374 Encodings", RFC 3548, July 2003. 4376 [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with 4377 Session Description Protocol (SDP)", RFC 3264, June 2002. 4379 [9] Lennox, J., Ott, J., and Schierl, T., "Source-Specific Media 4380 Attributes in the Session Description Protocol (SDP)", RFC 4381 5576, June 2009. 4383 15.2. Informative References 4385 [10] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special 4386 Issue on H.264/AVC. IEEE Transactions on Circuits and Systems 4387 on Video Technology, July 2003. 4389 [11] Ott, J., Bormann, C., Sullivan, G., Wenger, S., and R. Even 4390 (Ed.), "RTP Payload Format for ITU-T Rec. H.263 Video", RFC 4391 4629, January 2007. 4393 [12] ISO/IEC IS 14496-2. 4395 [13] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and 4396 Systems for Video technology, Vol. 13, No. 7, July 2003. 4398 [14] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", 4399 Proceedings Packet Video Workshop 02, April 2002. 4401 [15] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT 4402 Coding Network Abstraction Layer and IP-based Transport" in 4403 Proc. ICIP 2002, Rochester, NY, September 2002. 4405 [16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 4406 Video Conferences with Minimal Control", STD 65, RFC 3551, 4407 July 2003. 4409 [17] ITU-T Recommendation H.223, "Multiplexing protocol for low 4410 bit rate multimedia communication", July 2001. 4412 [18] Li, A., "RTP Payload Format for Generic Forward Error 4413 Correction", RFC 5109, December 2007. 4415 [19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, 4416 "Video Coding and Transport Layer Techniques for H.264/AVC- 4417 Based Transmission over Packet-Lossy Networks", IEEE 4418 International Conference on Image Processing (ICIP 2003), 4419 Barcelona, Spain, September 2003. 4421 [20] Varsa, V. and M. Karczewicz, "Slice interleaving in 4422 compressed video packetization", Packet Video Workshop 2000. 4424 [21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for 4425 wireless video streaming," International Packet Video 4426 Workshop 2002. 4428 [22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, 4429 available http://ftp3.itu.int/av-arch/video- 4430 site/0201_Gen/JVT-B042.doc, anuary 2002. 4432 [23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 4433 International Workshop on Audio-Visual Services over Packet 4434 Networks, September 1997. 4436 [24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error 4437 Resilient Video Coding Using Unequally Protected Key 4438 Pictures", in Proc. International Workshop VLBV03, September 4439 2003. 4441 [25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., 4442 and P. Gentric, "RTP Payload Format for Transport of MPEG-4 4443 Elementary Streams", RFC 3640, November 2003. 4445 [26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 4446 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 4447 RFC 3711, March 2004. 4449 [27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time 4450 Streaming Protocol (RTSP)", RFC 2326, April 1998. 4452 [28] Handley, M., Perkins, C., and E. Whelan, "Session 4453 Announcement Protocol", RFC 2974, October 2000. 4455 [29] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, 4456 January 2008. 4458 [30] Wenger, S., Chandra, U., and M. Westerlund, "Codec Control 4459 Messages in the RTP Audio-Visual Profile with Feedback 4460 (AVPF)", RFC 5104, February 2008. 4462 16. Authors' Addresses 4464 Ye-Kui Wang 4465 Huawei Technologies 4466 400 Somerset Corp Blvd, Suite 602 4467 Bridgewater, NJ 08807 4468 USA 4470 Phone: +1-908-541-3518 4471 EMail: yekuiwang@huawei.com 4473 Roni Even 4474 14 David Hamelech 4475 Tel Aviv 64953 4476 Israel 4478 Phone: +972-545481099 4479 Email: ron.even.tlv@gmail.com 4481 Tom Kristensen 4482 TANDBERG 4483 Philip Pedersens vei 22 4484 N-1366 Lysaker 4485 Norway 4487 Phone: +47 67125125 4488 Email: tom.kristensen@tandberg.com, tomkri@ifi.uio.no 4490 Randell Jesup 4491 WorldGate Communications 4492 3190 Tremont Ave 4493 Trevose, PA 19053 4494 USA 4496 Phone: +1-215-354-5166 4497 Email: rjesup@wgate.com 4499 17. Backward Compatibility to RFC 3984 4501 The current document is a revision of RFC 3984 and intends to 4502 obsolete it. This section addresses the backward compatibility 4503 issues. 4505 The technical changes are listed in section 18. 4507 Items 1), 2), 3), 7), 9), 10), 12), 13) are bug-fix type of changes, 4508 and do not incur any backward compatibility issues. 4510 Item 4), addition of six new media type parameters, does not incur 4511 any backward compatibility issues for SDP Offer/Answer based 4512 applications, as legacy RFC 3984 receivers ignore these parameters, 4513 and it is fine for legacy RFC 3984 senders not to use these 4514 parameters as they are optional. However, there is a backward 4515 compatibility issue for SDP declarative usage based applications, 4516 e.g. those using RTSP and SAP, because the SDP receiver per RFC 4517 3984 cannot accept a session for which the SDP includes an 4518 unrecognized parameter. Therefore, the RTSP or SAP server may have 4519 to prepare two sets of streams, one for legacy RFC 3984 receivers 4520 and one for receivers according to this memo. 4522 Items 5), 6) and 11) are related to out-of-band transport of 4523 parameter sets. There are following backward compatibility issues. 4525 1) When a legacy sender per RFC 3984 includes parameter sets for a 4526 level different than the default level indicated by profile- 4527 level-id to sprop-parameter-sets, the parameter value of sprop- 4528 parameter-sets is invalid to the receiver per this memo and 4529 therefore the session may be rejected. 4531 2) In SDP Offer/Answer between a legacy offerer per RFC 3984 and an 4532 answerer per this memo, when the answerer includes in the answer 4533 parameter sets that are not a superset of the parameter sets 4534 included in the offer, the parameter value of sprop-parameter- 4535 sets is invalid to offerer and the session may not be initiated 4536 properly (related to change item 11)). 4538 3) When one endpoint A per this memo includes in-band-parameter- 4539 sets equal to 1, the other side B per RFC 3984 does not 4540 understand that it must transmit parameter sets in-band and B 4541 may still exclude parameter sets in the in-band stream it is 4542 sending. Consequently endpoint A cannot decode the stream it 4543 receives. 4545 Item 7), allowance of conveying sprop-parameter-sets and sprop- 4546 level-parameter-sets using the "fmtp" source attribute as specified 4547 in section 6.3 of [9], is similar as item 4). It does not incur 4548 any backward compatibility issues for SDP Offer/Answer based 4549 applications, as legacy RFC 3984 receivers ignore the "fmtp" source 4550 attribute, and it is fine for legacy RFC 3984 senders not to use 4551 the "fmtp" source attribute as it is optional. However, there is a 4552 backward compatibility issue for SDP declarative usage based 4553 applications, e.g. those using RTSP and SAP, because the SDP 4554 receiver per RFC 3984 cannot accept a session for which the SDP 4555 includes an unrecognized parameter (i.e., the "fmtp" source 4556 attribute). Therefore, the RTSP or SAP server may have to prepare 4557 two sets of streams, one for legacy RFC 3984 receivers and one for 4558 receivers according to this memo. 4560 Item 14) removed that use of out-of-band transport of parameter 4561 sets is recommended. As out-of-band transport of parameter sets is 4562 still allowed, this change does not incur any backward 4563 compatibility issues. 4565 Item 15) does not incur any backward compatibility issues as the 4566 added subsection 8.5 is informative. 4568 Item 16) does not create any backward compatibility issues as the 4569 handling of default level is the same if either end is RFC 3984 4570 compliant, and furthermore, RFC 3984 compliant ends would simply 4571 ignore the new media type parameters, if present. 4573 18. Changes from RFC 3984 4575 Following is the list of technical changes (including bug fixes) 4576 from RFC 3984. Besides this list of technical changes, numerous 4577 editorial changes have been made, but not documented in this memo. 4579 1) In subsections 5.4, 5.5, 6.2, 6,3 and 6.4, removed that the 4580 packetization mode in use may be signaled by external means. 4582 2) In subsection 7.2.2, changed the sentence 4584 There are N VCL NAL units in the deinterleaving buffer. 4586 to 4588 There are N or more VCL NAL units in the de-interleaving buffer. 4590 3) In subsection 8.1, the semantics of sprop-init-buf-time, 4591 paragraph 2, changed the sentence 4593 The parameter is the maximum value of (transmission time of a 4594 NAL unit - decoding time of the NAL unit), assuming reliable and 4595 instantaneous transmission, the same timeline for transmission 4596 and decoding, and that decoding starts when the first packet 4597 arrives. 4599 to 4601 The parameter is the maximum value of (decoding time of the NAL 4602 unit - transmission time of a NAL unit), assuming reliable and 4603 instantaneous transmission, the same timeline for transmission 4604 and decoding, and that decoding starts when the first packet 4605 arrives. 4607 4) Added media type parameters max-smbps, sprop-level-parameter- 4608 sets, use-level-src-parameter-sets, in-band-parameter-sets, sar- 4609 understood and sar-supported. 4611 5) In subsection 8.1, removed the specification of parameter-add. 4612 Other descriptions of parameter-add (in subsections 8.2 and 8.4) 4613 are also removed. 4615 6) In subsection 8.1, added a constraint to sprop-parameter-sets 4616 such that it can only contain parameter sets for the same 4617 profile and level as indicated by profile-level-id. 4619 7) In subsection 8.2.1, added that sprop-parameter-sets and sprop- 4620 level-parameter-sets may be either included in the "a=fmtp" line 4621 of SDP or conveyed using the "fmtp" source attribute as 4622 specified in section 6.3 of [9]. 4624 8) In subsection 8.2.2, removed sprop-deint-buf-req from being part 4625 of the media format configuration in usage with the SDP 4626 Offer/Answer model. 4628 9) In subsection 8.2.2, made it clear that level is downgradable in 4629 the SDP Offer/Answer model, i.e. the use of the level part of 4630 "profile-level-id" does not need to be symmetric (the level 4631 included in the answer can be lower than or equal to the level 4632 included in the offer). 4634 10)In subsection 8.2.2, removed that the capability parameters may 4635 be used to declare encoding capabilities. 4637 11)In subsection 8.2.2, added rules on how to use sprop-parameter- 4638 sets and sprop-level-parameter-sets for out-of-band transport of 4639 parameter sets, with or without level downgrading. 4641 12)In subsection 8.2.2, clarified the rules of using the media type 4642 parameters with SDP Offer/Answer for multicast. 4644 13)In subsection 8.2.2, completed and corrected the list of how 4645 different media type parameters shall be interpreted in the 4646 different combinations of offer or answer and direction 4647 attribute. 4649 14)In subsection 8.4, changed the text such that both out-of-band 4650 and in-band transport of parameter sets are allowed and neither 4651 is recommended or required. 4653 15)Added subsection 8.5 (informative) providing example methods for 4654 decoder refresh to handle parameter set losses. 4656 16)Added media type parameters max-recv-level, and level-asymmetry- 4657 allowed, and adjusted associated text and examples for level 4658 upgrade and asymmetry.