idnits 2.17.1 draft-ietf-avt-rtp-rfc3984bis-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The abstract seems to indicate that this document obsoletes RFC3984, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document seems to contain a disclaimer for pre-RFC5378 work, and may have content which was first submitted before 10 November 2008. The disclaimer is necessary when there are original authors that you have been unable to contact, or if some do not wish to grant the BCP78 rights to the IETF Trust. If you are able to get all authors (current and original) to grant those rights, you can and should remove the disclaimer; otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 10, 2009) is 5341 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' ** Obsolete normative reference: RFC 4566 (ref. '6') (Obsoleted by RFC 8866) ** Obsolete normative reference: RFC 3548 (ref. '7') (Obsoleted by RFC 4648) -- Obsolete informational reference (is this intentional?): RFC 2326 (ref. '27') (Obsoleted by RFC 7826) -- Obsolete informational reference (is this intentional?): RFC 5117 (ref. '29') (Obsoleted by RFC 7667) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Obsoletes RFC 3984 2 Audio/Video Transport WG Y.-K. Wang 3 Internet Draft Huawei Technologies 4 Intended status: Standards track R. Even 5 Expires: March 2010 Self-employed 6 T. Kristensen 7 Tandberg 8 R. Jesup 9 WorldGate Communications 10 September 10, 2009 12 RTP Payload Format for H.264 Video 13 draft-ietf-avt-rtp-rfc3984bis-07.txt 15 Status of this Memo 17 This Internet-Draft is submitted to IETF in full conformance with 18 the provisions of BCP 78 and BCP 79. This document may contain 19 material from IETF Documents or IETF Contributions published or 20 made publicly available before November 10, 2008. The person(s) 21 controlling the copyright in some of this material may not have 22 granted the IETF Trust the right to allow modifications of such 23 material outside the IETF Standards Process. Without obtaining an 24 adequate license from the person(s) controlling the copyright in 25 such materials, this document may not be modified outside the IETF 26 Standards Process, and derivative works of it may not be created 27 outside the IETF Standards Process, except to format it for 28 publication as an RFC or to translate it into languages other than 29 English. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF), its areas, and its working groups. Note that 33 other groups may also distribute working documents as Internet- 34 Drafts. 36 Internet-Drafts are draft documents valid for a maximum of six 37 months and may be updated, replaced, or obsoleted by other 38 documents at any time. It is inappropriate to use Internet-Drafts 39 as reference material or to cite them other than as "work in 40 progress". 42 The list of current Internet-Drafts can be accessed at 43 http://www.ietf.org/ietf/1id-abstracts.txt. 45 The list of Internet-Draft Shadow Directories can be accessed at 46 http://www.ietf.org/shadow.html. 48 This Internet-Draft will expire on January 10, 2009. 50 Copyright Notice 52 Copyright (c) 2009 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents in effect on the date of 57 publication of this document (http://trustee.ietf.org/license-info). 58 Please review these documents carefully, as they describe your 59 rights and restrictions with respect to this document. 61 Abstract 63 This memo describes an RTP Payload format for the ITU-T 64 Recommendation H.264 video codec and the technically identical 65 ISO/IEC International Standard 14496-10 video codec, excluding the 66 Scalable Video Coding (SVC) extension and the Multivew Video Coding 67 extension, for which the RTP payload formats are defined elsewhere. 68 The RTP payload format allows for packetization of one or more 69 Network Abstraction Layer Units (NALUs), produced by an H.264 video 70 encoder, in each RTP payload. The payload format has wide 71 applicability, as it supports applications from simple low bit-rate 72 conversational usage, to Internet video streaming with interleaved 73 transmission, to high bit-rate video-on-demand. 75 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 76 in section 18. Issues on backward compatibility to RFC 3984 are 77 discussed in section 17. 79 Table of Contents 81 Abstract.........................................................2 82 1. Introduction..................................................4 83 1.1. The H.264 Codec..........................................5 84 1.2. Parameter Set Concept....................................6 85 1.3. Network Abstraction Layer Unit Types.....................7 87 2. Conventions...................................................8 88 3. Scope.........................................................8 89 4. Definitions and Abbreviations.................................8 90 4.1. Definitions..............................................8 91 4.2. Abbreviations...........................................10 92 5. RTP Payload Format...........................................11 93 5.1. RTP Header Usage........................................11 94 5.2. Payload Structures......................................13 95 5.3. NAL Unit Header Usage...................................14 96 5.4. Packetization Modes.....................................17 97 5.5. Decoding Order Number (DON).............................18 98 5.6. Single NAL Unit Packet..................................21 99 5.7. Aggregation Packets.....................................22 100 Table 4. Type field for STAPs and MTAPs........................23 101 5.7.1. Single-Time Aggregation Packet.....................23 102 5.7.2. Multi-Time Aggregation Packets (MTAPs).............26 103 5.7.3. Fragmentation Units (FUs)..........................30 104 6. Packetization Rules..........................................34 105 6.1. Common Packetization Rules..............................34 106 6.2. Single NAL Unit Mode....................................35 107 6.3. Non-Interleaved Mode....................................35 108 6.4. Interleaved Mode........................................36 109 7. De-Packetization Process.....................................36 110 7.1. Single NAL Unit and Non-Interleaved Mode................36 111 7.2. Interleaved Mode........................................37 112 7.2.1. Size of the De-interleaving Buffer.................37 113 7.2.2. De-interleaving Process............................38 114 7.3. Additional De-Packetization Guidelines..................39 115 8. Payload Format Parameters....................................40 116 8.1. Media Type Registration.................................40 117 8.2. SDP Parameters..........................................59 118 8.2.1. Mapping of Payload Type Parameters to SDP..........59 119 8.2.2. Usage with the SDP Offer/Answer Model..............60 120 8.2.3. Usage in Declarative Session Descriptions..........69 121 8.3. Examples................................................70 122 Offer SDP:......................................................76 123 Answer SDP:.....................................................76 124 8.4. Parameter Set Considerations............................77 125 8.5. Decoder Refresh Point Procedure using In-Band Transport of 126 Parameter Sets (Informative).................................80 127 8.5.1. IDR Procedure to Respond to a Request for a Decoder 128 Refresh Point.............................................80 129 8.5.2. Gradual Recovery Procedure to Respond to a Request for 130 a Decoder Refresh Point...................................81 131 9. Security Considerations......................................82 132 10. Congestion Control..........................................82 133 11. IANA Consideration..........................................83 134 12. Informative Appendix: Application Examples..................83 135 12.1. Video Telephony according to ITU-T Recommendation H.241 136 Annex A......................................................84 137 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 138 Aggregation..................................................84 139 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 140 Aggregation..................................................84 141 12.4. Video Telephony with Data Partitioning.................85 142 12.5. Video Telephony or Streaming with FUs and Forward Error 143 Correction...................................................86 144 12.6. Low Bit-Rate Streaming.................................88 145 12.7. Robust Packet Scheduling in Video Streaming............89 146 13. Informative Appendix: Rationale for Decoding Order Number...90 147 13.1. Introduction...........................................90 148 13.2. Example of Multi-Picture Slice Interleaving............90 149 13.3. Example of Robust Packet Scheduling....................91 150 13.4. Robust Transmission Scheduling of Redundant Coded Slices95 151 13.5. Remarks on Other Design Possibilities..................96 152 14. Acknowledgements............................................97 153 15. References..................................................97 154 15.1. Normative References...................................97 155 15.2. Informative References.................................98 156 16. Authors' Addresses.........................................100 157 Phone: +1-908-541-3518.........................................100 158 Phone: +972-545481099..........................................100 159 Tom Kristensen.................................................100 160 Phone: +47 67125125............................................100 161 Phone: +1-215-354-5166.........................................100 162 Email: rjesup@wgate.com........................................100 163 17. Backward Compatibility to RFC 3984.........................100 164 18. Changes from RFC 3984......................................102 166 1. Introduction 168 This memo specifies an RTP payload specification for the video 169 coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC 170 International Standard 14496 Part 10 [2] (both also known as 171 Advanced Video Coding, or AVC). In this memo the name H.264 is 172 used for the codec and the standard, but the memo is equally 173 applicable to the ISO/IEC counterpart of the coding standard. 175 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 176 in section 18. Issues on backward compatibility to RFC 3984 are 177 discussed in section 17. 179 1.1. The H.264 Codec 181 The H.264 video codec has a very broad application range that 182 covers all forms of digital compressed video, from low bit-rate 183 Internet streaming applications to HDTV broadcast and Digital 184 Cinema applications with nearly lossless coding. Compared to the 185 current state of technology, the overall performance of H.264 is 186 such that bit rate savings of 50% or more are reported. Digital 187 Satellite TV quality, for example, was reported to be achievable at 188 1.5 Mbit/s, compared to the current operation point of MPEG 2 video 189 at around 3.5 Mbit/s [10]. 191 The codec specification [1] itself distinguishes conceptually 192 between a video coding layer (VCL) and a network abstraction layer 193 (NAL). The VCL contains the signal processing functionality of the 194 codec; mechanisms such as transform, quantization, and motion 195 compensated prediction; and a loop filter. It follows the general 196 concept of most of today's video codecs, a macroblock-based coder 197 that uses inter picture prediction with motion compensation and 198 transform coding of the residual signal. The VCL encoder outputs 199 slices: a bit string that contains the macroblock data of an 200 integer number of macroblocks, and the information of the slice 201 header (containing the spatial address of the first macroblock in 202 the slice, the initial quantization parameter, and similar 203 information). Macroblocks in slices are arranged in scan order 204 unless a different macroblock allocation is specified, by using the 205 so-called Flexible Macroblock Ordering syntax. In-picture 206 prediction is used only within a slice. More information is 207 provided in [10]. 209 The Network Abstraction Layer (NAL) encoder encapsulates the slice 210 output of the VCL encoder into Network Abstraction Layer Units (NAL 211 units), which are suitable for transmission over packet networks or 212 use in packet oriented multiplex environments. Annex B of H.264 213 defines an encapsulation process to transmit such NAL units over 214 byte-stream oriented networks. In the scope of this memo, Annex B 215 is not relevant. 217 Internally, the NAL uses NAL units. A NAL unit consists of a one- 218 byte header and the payload byte string. The header indicates the 219 type of the NAL unit, the (potential) presence of bit errors or 220 syntax violations in the NAL unit payload, and information 221 regarding the relative importance of the NAL unit for the decoding 222 process. This RTP payload specification is designed to be unaware 223 of the bit string in the NAL unit payload. 225 One of the main properties of H.264 is the complete decoupling of 226 the transmission time, the decoding time, and the sampling or 227 presentation time of slices and pictures. The decoding process 228 specified in H.264 is unaware of time, and the H.264 syntax does 229 not carry information such as the number of skipped frames (as is 230 common in the form of the Temporal Reference in earlier video 231 compression standards). Also, there are NAL units that affect many 232 pictures and that are, therefore, inherently timeless. For this 233 reason, the handling of the RTP timestamp requires some special 234 considerations for NAL units for which the sampling or presentation 235 time is not defined or, at transmission time, unknown. 237 1.2. Parameter Set Concept 239 One very fundamental design concept of H.264 is to generate self- 240 contained packets, to make mechanisms such as the header 241 duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension 242 Code (HEC) [12] unnecessary. This was achieved by decoupling 243 information relevant to more than one slice from the media stream. 244 This higher layer meta information should be sent reliably, 245 asynchronously, and in advance from the RTP packet stream that 246 contains the slice packets. (Provisions for sending this 247 information in-band are also available for applications that do not 248 have an out-of-band transport channel appropriate for the purpose.) 249 The combination of the higher-level parameters is called a 250 parameter set. The H.264 specification includes two types of 251 parameter sets: sequence parameter set and picture parameter set. 252 An active sequence parameter set remains unchanged throughout a 253 coded video sequence, and an active picture parameter set remains 254 unchanged within a coded picture. The sequence and picture 255 parameter set structures contain information such as picture size, 256 optional coding modes employed, and macroblock to slice group map. 258 To be able to change picture parameters (such as the picture size) 259 without having to transmit parameter set updates synchronously to 260 the slice packet stream, the encoder and decoder can maintain a 261 list of more than one sequence and picture parameter set. Each 262 slice header contains a codeword that indicates the sequence and 263 picture parameter set to be used. 265 This mechanism allows the decoupling of the transmission of 266 parameter sets from the packet stream, and the transmission of them 267 by external means (e.g., as a side effect of the capability 268 exchange), or through a (reliable or unreliable) control protocol. 269 It may even be possible that they are never transmitted but are 270 fixed by an application design specification. 272 1.3. Network Abstraction Layer Unit Types 274 Tutorial information on the NAL design can be found in [13], [14], 275 and [15]. 277 All NAL units consist of a single NAL unit type octet, which also 278 co-serves as the payload header of this RTP payload format. The 279 payload of a NAL unit follows immediately. 281 The syntax and semantics of the NAL unit type octet are specified 282 in [1], but the essential properties of the NAL unit type octet are 283 summarized below. The NAL unit type octet has the following format: 285 +---------------+ 286 |0|1|2|3|4|5|6|7| 287 +-+-+-+-+-+-+-+-+ 288 |F|NRI| Type | 289 +---------------+ 291 The semantics of the components of the NAL unit type octet, as 292 specified in the H.264 specification, are described briefly below. 294 F: 1 bit 295 forbidden_zero_bit. The H.264 specification declares a value of 296 1 as a syntax violation. 298 NRI: 2 bits 299 nal_ref_idc. A value of 00 indicates that the content of the 300 NAL unit is not used to reconstruct reference pictures for inter 301 picture prediction. Such NAL units can be discarded without 302 risking the integrity of the reference pictures. Values greater 303 than 00 indicate that the decoding of the NAL unit is required 304 to maintain the integrity of the reference pictures. 306 Type: 5 bits 307 nal_unit_type. This component specifies the NAL unit payload 308 type as defined in Table 7-1 of [1], and later within this memo. 309 For a reference of all currently defined NAL unit types and 310 their semantics, please refer to section 7.4.1 in [1]. 312 This memo introduces new NAL unit types, which are presented in 313 section 5.2. The NAL unit types defined in this memo are marked as 314 unspecified in [1]. Moreover, this specification extends the 315 semantics of F and NRI as described in section 5.3. 317 2. Conventions 319 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 320 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 321 this document are to be interpreted as described in RFC 2119 [4]. 323 This specification uses the notion of setting and clearing a bit 324 when bit fields are handled. Setting a bit is the same as 325 assigning that bit the value of 1 (On). Clearing a bit is the same 326 as assigning that bit the value of 0 (Off). 328 3. Scope 330 This payload specification can only be used to carry the "naked" 331 H.264 NAL unit stream over RTP, and not the bitstream format 332 discussed in Annex B of H.264. Likely, the first applications of 333 this specification will be in the conversational multimedia field, 334 video telephony or video conferencing, but the payload format also 335 covers other applications, such as Internet streaming and TV over 336 IP. 338 4. Definitions and Abbreviations 340 4.1. Definitions 342 This document uses the definitions of [1]. The following terms, 343 defined in [1], are summed up for convenience: 345 access unit: A set of NAL units always containing a primary 346 coded picture. In addition to the primary coded picture, an 347 access unit may also contain one or more redundant coded 348 pictures or other NAL units not containing slices or slice data 349 partitions of a coded picture. The decoding of an access unit 350 always results in a decoded picture. 352 coded video sequence: A sequence of access units that consists, 353 in decoding order, of an instantaneous decoding refresh (IDR) 354 access unit followed by zero or more non-IDR access units 355 including all subsequent access units up to but not including 356 any subsequent IDR access unit. 358 IDR access unit: An access unit in which the primary coded 359 picture is an IDR picture. 361 IDR picture: A coded picture containing only slices with I or SI 362 slice types that causes a "reset" in the decoding process. 363 After the decoding of an IDR picture, all following coded 364 pictures in decoding order can be decoded without inter 365 prediction from any picture decoded prior to the IDR picture. 367 primary coded picture: The coded representation of a picture to 368 be used by the decoding process for a bitstream conforming to 369 H.264. The primary coded picture contains all macroblocks of 370 the picture. 372 redundant coded picture: A coded representation of a picture or 373 a part of a picture. The content of a redundant coded picture 374 shall not be used by the decoding process for a bitstream 375 conforming to H.264. The content of a redundant coded picture 376 may be used by the decoding process for a bitstream that 377 contains errors or losses. 379 VCL NAL unit: A collective term used to refer to coded slice and 380 coded data partition NAL units. 382 In addition, the following definitions apply: 384 decoding order number (DON): A field in the payload structure or 385 a derived variable indicating NAL unit decoding order. Values 386 of DON are in the range of 0 to 65535, inclusive. After 387 reaching the maximum value, the value of DON wraps around to 0. 389 NAL unit decoding order: A NAL unit order that conforms to the 390 constraints on NAL unit order given in section 7.4.1.2 in [1]. 392 NALU-time: The value that the RTP timestamp would have if the 393 NAL unit would be transported in its own RTP packet. 395 transmission order: The order of packets in ascending RTP 396 sequence number order (in modulo arithmetic). Within an 397 aggregation packet, the NAL unit transmission order is the same 398 as the order of appearance of NAL units in the packet. 400 media aware network element (MANE): A network element, such as a 401 middlebox or application layer gateway that is capable of 402 parsing certain aspects of the RTP payload headers or the RTP 403 payload and reacting to the contents. 405 Informative note: The concept of a MANE goes beyond normal 406 routers or gateways in that a MANE has to be aware of the 407 signaling (e.g., to learn about the payload type mappings of 408 the media streams), and in that it has to be trusted when 409 working with SRTP. The advantage of using MANEs is that they 410 allow packets to be dropped according to the needs of the 411 media coding. For example, if a MANE has to drop packets due 412 to congestion on a certain link, it can identify and remove 413 those packets whose elimination produces the least adverse 414 effect on the user experience. 416 static macroblock: A certain amount of macroblocks in the video 417 stream can be defined as static, as defined in section 8.3.2.8 418 in [3]. Static macroblocks free up additional processing 419 cycles for the handling of non-static macroblocks. Based on a 420 given amount of video processing resources and a given 421 resolution, a higher number of static macroblocks enables a 422 correspondingly higher frame rate. 424 default sub-profile: The subset of coding tools, which may be 425 all coding tools of one profile or the common subset of coding 426 tools of more than one profile, indicated by the profile-level- 427 id parameter. 429 default level: The level indicated by the profile-level-id 430 parameter, which consists of three octets, profile_idc, profile- 431 iop, and level_idc. The default level is indicated by level_idc 432 in most cases, and, in some cases, additionally by profile-iop. 434 maximum receive level: The higher of the default level (from 435 profile-level-id) and max-recv-level (if specified). 437 4.2. Abbreviations 439 DON: Decoding Order Number 440 DONB: Decoding Order Number Base 441 DOND: Decoding Order Number Difference 442 FEC: Forward Error Correction 443 FU: Fragmentation Unit 444 IDR: Instantaneous Decoding Refresh 445 IEC: International Electrotechnical Commission 446 ISO: International Organization for Standardization 447 ITU-T: International Telecommunication Union, 448 Telecommunication Standardization Sector 449 MANE: Media Aware Network Element 450 MTAP: Multi-Time Aggregation Packet 451 MTAP16: MTAP with 16-bit timestamp offset 452 MTAP24: MTAP with 24-bit timestamp offset 453 NAL: Network Abstraction Layer 454 NALU: NAL Unit 455 SAR: Sample Aspect Ratio 456 SEI: Supplemental Enhancement Information 457 STAP: Single-Time Aggregation Packet 458 STAP-A: STAP type A 459 STAP-B: STAP type B 460 TS: Timestamp 461 VCL: Video Coding Layer 462 VUI: Video Usability Information 464 5. RTP Payload Format 466 5.1. RTP Header Usage 468 The format of the RTP header is specified in RFC 3550 [5] and 469 reprinted in Figure 1 for convenience. This payload format uses 470 the fields of the header in a manner consistent with that 471 specification. 473 When one NAL unit is encapsulated per RTP packet, the RECOMMENDED 474 RTP payload format is specified in section 5.6. The RTP payload 475 (and the settings for some RTP header bits) for aggregation packets 476 and fragmentation units are specified in sections 5.7 and 5.8, 477 respectively. 479 0 1 2 3 480 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 481 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 482 |V=2|P|X| CC |M| PT | sequence number | 483 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 484 | timestamp | 485 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 | synchronization source (SSRC) identifier | 487 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 488 | contributing source (CSRC) identifiers | 489 | .... | 490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 492 Figure 1 RTP header according to RFC 3550 494 The RTP header information to be set according to this RTP payload 495 format is set as follows: 497 Marker bit (M): 1 bit 498 Set for the very last packet of the access unit indicated by the 499 RTP timestamp, in line with the normal use of the M bit in video 500 formats, to allow an efficient playout buffer handling. For 501 aggregation packets (STAP and MTAP), the marker bit in the RTP 502 header MUST be set to the value that the marker bit of the last 503 NAL unit of the aggregation packet would have been if it were 504 transported in its own RTP packet. Decoders MAY use this bit as 505 an early indication of the last packet of an access unit, but 506 MUST NOT rely on this property. 508 Informative note: Only one M bit is associated with an 509 aggregation packet carrying multiple NAL units. Thus, if a 510 gateway has re-packetized an aggregation packet into several 511 packets, it cannot reliably set the M bit of those packets. 513 Payload type (PT): 7 bits 514 The assignment of an RTP payload type for this new packet format 515 is outside the scope of this document and will not be specified 516 here. The assignment of a payload type has to be performed 517 either through the profile used or in a dynamic way. 519 Sequence number (SN): 16 bits 520 Set and used in accordance with RFC 3550. For the single NALU 521 and non-interleaved packetization mode, the sequence number is 522 used to determine decoding order for the NALU. 524 Timestamp: 32 bits 525 The RTP timestamp is set to the sampling timestamp of the 526 content. A 90 kHz clock rate MUST be used. 528 If the NAL unit has no timing properties of its own (e.g., 529 parameter set and SEI NAL units), the RTP timestamp is set to 530 the RTP timestamp of the primary coded picture of the access 531 unit in which the NAL unit is included, according to section 532 7.4.1.2 of [1]. 534 The setting of the RTP Timestamp for MTAPs is defined in section 535 5.7.2. 537 Receivers SHOULD ignore any picture timing SEI messages included 538 in access units that have only one display timestamp. Instead, 539 receivers SHOULD use the RTP timestamp for synchronizing the 540 display process. 542 If one access unit has more than one display timestamp carried 543 in a picture timing SEI message, then the information in the SEI 544 message SHOULD be treated as relative to the RTP timestamp, with 545 the earliest event occurring at the time given by the RTP 546 timestamp, and subsequent events later, as given by the 547 difference in SEI message picture timing values. Let tSEI1, 548 tSEI2, ..., tSEIn be the display timestamps carried in the SEI 549 message of an access unit, where tSEI1 is the earliest of all 550 such timestamps. Let tmadjst() be a function that adjusts the 551 SEI messages time scale to a 90-kHz time scale. Let TS be the 552 RTP timestamp. Then, the display time for the event associated 553 with tSEI1 is TS. The display time for the event with tSEIx, 554 where x is [2..n] is TS + tmadjst (tSEIx - tSEI1). 556 Informative note: Displaying coded frames as fields is needed 557 commonly in an operation known as 3:2 pulldown, in which film 558 content that consists of coded frames is displayed on a 559 display using interlaced scanning. The picture timing SEI 560 message enables carriage of multiple timestamps for the same 561 coded picture, and therefore the 3:2 pulldown process is 562 perfectly controlled. The picture timing SEI message 563 mechanism is necessary because only one timestamp per coded 564 frame can be conveyed in the RTP timestamp. 566 5.2. Payload Structures 568 The payload format defines three different basic payload structures. 569 A receiver can identify the payload structure by the first byte of 570 the RTP packet payload, which co-serves as the RTP payload header 571 and, in some cases, as the first byte of the payload. This byte is 572 always structured as a NAL unit header. The NAL unit type field 573 indicates which structure is present. The possible structures are 574 as follows: 576 Single NAL Unit Packet: Contains only a single NAL unit in the 577 payload. The NAL header type field will be equal to the original 578 NAL unit type; i.e., in the range of 1 to 23, inclusive. Specified 579 in section 5.6. 581 Aggregation Packet: Packet type used to aggregate multiple NAL 582 units into a single RTP payload. This packet exists in four 583 versions, the Single-Time Aggregation Packet type A (STAP-A), the 584 Single-Time Aggregation Packet type B (STAP-B), Multi-Time 585 Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and Multi- 586 Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The 587 NAL unit type numbers assigned for STAP-A, STAP-B, MTAP16, and 588 MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 589 5.7. 591 Fragmentation Unit: Used to fragment a single NAL unit over 592 multiple RTP packets. Exists with two versions, FU-A and FU-B, 593 identified with the NAL unit type numbers 28 and 29, respectively. 594 Specified in section 5.8. 596 Informative note: This specification does not limit the size of 597 NAL units encapsulated in single NAL unit packets and 598 fragmentation units. The maximum size of a NAL unit 599 encapsulated in any aggregation packet is 65535 bytes. 601 Table 1 summarizes NAL unit types and the corresponding RTP packet 602 types when each of these NAL units is directly used as a packet 603 payload, and where the types are described in this memo. 605 Table 1. Summary of NAL unit types and the corresponding packet 606 types 608 NAL Unit Packet Packet Type Name Section 609 Type Type 610 --------------------------------------------------------- 611 0 reserved - 612 1-23 NAL unit Single NAL unit packet 5.6 613 24 STAP-A Single-time aggregation packet 5.7.1 614 25 STAP-B Single-time aggregation packet 5.7.1 615 26 MTAP16 Multi-time aggregation packet 5.7.2 616 27 MTAP24 Multi-time aggregation packet 5.7.2 617 28 FU-A Fragmentation unit 5.8 618 29 FU-B Fragmentation unit 5.8 619 30-31 reserved - 621 5.3. NAL Unit Header Usage 623 The structure and semantics of the NAL unit header were introduced 624 in section 1.3. For convenience, the format of the NAL unit header 625 is reprinted below: 627 +---------------+ 628 |0|1|2|3|4|5|6|7| 629 +-+-+-+-+-+-+-+-+ 630 |F|NRI| Type | 631 +---------------+ 633 This section specifies the semantics of F and NRI according to this 634 specification. 636 F: 1 bit 637 forbidden_zero_bit. A value of 0 indicates that the NAL unit 638 type octet and payload should not contain bit errors or other 639 syntax violations. A value of 1 indicates that the NAL unit 640 type octet and payload may contain bit errors or other syntax 641 violations. 643 MANEs SHOULD set the F bit to indicate detected bit errors in 644 the NAL unit. The H.264 specification requires that the F bit 645 is equal to 0. When the F bit is set, the decoder is advised 646 that bit errors or any other syntax violations may be present in 647 the payload or in the NAL unit type octet. The simplest decoder 648 reaction to a NAL unit in which the F bit is equal to 1 is to 649 discard such a NAL unit and to conceal the lost data in the 650 discarded NAL unit. 652 NRI: 2 bits 653 nal_ref_idc. The semantics of value 00 and a non-zero value 654 remain unchanged from the H.264 specification. In other words, 655 a value of 00 indicates that the content of the NAL unit is not 656 used to reconstruct reference pictures for inter picture 657 prediction. Such NAL units can be discarded without risking the 658 integrity of the reference pictures. Values greater than 00 659 indicate that the decoding of the NAL unit is required to 660 maintain the integrity of the reference pictures. 662 In addition to the specification above, according to this RTP 663 payload specification, values of NRI indicate the relative 664 transport priority, as determined by the encoder. MANEs can use 665 this information to protect more important NAL units better than 666 they do less important NAL units. The highest transport 667 priority is 11, followed by 10, and then by 01; finally, 00 is 668 the lowest. 670 Informative note: Any non-zero value of NRI is handled 671 identically in H.264 decoders. Therefore, receivers need not 672 manipulate the value of NRI when passing NAL units to the 673 decoder. 675 An H.264 encoder MUST set the value of NRI according to the 676 H.264 specification (subclause 7.4.1) when the value of 677 nal_unit_type is in the range of 1 to 12, inclusive. In 678 particular, the H.264 specification requires that the value of 679 NRI SHALL be equal to 0 for all NAL units having nal_unit_type 680 equal to 6, 9, 10, 11, or 12. 682 For NAL units having nal_unit_type equal to 7 or 8 (indicating a 683 sequence parameter set or a picture parameter set, respectively), 684 an H.264 encoder SHOULD set the value of NRI to 11 (in binary 685 format). For coded slice NAL units of a primary coded picture 686 having nal_unit_type equal to 5 (indicating a coded slice 687 belonging to an IDR picture), an H.264 encoder SHOULD set the 688 value of NRI to 11 (in binary format). 690 For a mapping of the remaining nal_unit_types to NRI values, the 691 following example MAY be used and has been shown to be efficient 692 in a certain environment [14]. Other mappings MAY also be 693 desirable, depending on the application and the H.264/AVC Annex 694 A profile in use. 696 Informative note: Data Partitioning is not available in 697 certain profiles; e.g., in the Main or Baseline profiles. 698 Consequently, the NAL unit types 2, 3, and 4 can occur only 699 if the video bitstream conforms to a profile in which data 700 partitioning is allowed and not in streams that conform to 701 the Main or Baseline profiles. 703 Table 2. Example of NRI values for coded slices and coded slice 704 data partitions of primary coded reference pictures 706 NAL Unit Type Content of NAL unit NRI (binary) 707 ---------------------------------------------------------------- 708 1 non-IDR coded slice 10 709 2 Coded slice data partition A 10 710 3 Coded slice data partition B 01 711 4 Coded slice data partition C 01 713 Informative note: As mentioned before, the NRI value of non- 714 reference pictures is 00 as mandated by H.264/AVC. 716 An H.264 encoder SHOULD set the value of NRI for coded slice and 717 coded slice data partition NAL units of redundant coded 718 reference pictures equal to 01 (in binary format). 720 Definitions of the values for NRI for NAL unit types 24 to 29, 721 inclusive, are given in sections 5.7 and 5.8 of this memo. 723 No recommendation for the value of NRI is given for NAL units 724 having nal_unit_type in the range of 13 to 23, inclusive, 725 because these values are reserved for ITU-T and ISO/IEC. No 726 recommendation for the value of NRI is given for NAL units 727 having nal_unit_type equal to 0 or in the range of 30 to 31, 728 inclusive, as the semantics of these values are not specified in 729 this memo. 731 5.4. Packetization Modes 733 This memo specifies three cases of packetization modes: 735 o Single NAL unit mode 737 o Non-interleaved mode 739 o Interleaved mode 741 The single NAL unit mode is targeted for conversational systems 742 that comply with ITU-T Recommendation H.241 [3] (see section 12.1). 743 The non-interleaved mode is targeted for conversational systems 744 that may not comply with ITU-T Recommendation H.241. In the non- 745 interleaved mode, NAL units are transmitted in NAL unit decoding 746 order. The interleaved mode is targeted for systems that do not 747 require very low end-to-end latency. The interleaved mode allows 748 transmission of NAL units out of NAL unit decoding order. 750 The packetization mode in use MAY be signaled by the value of the 751 OPTIONAL packetization-mode media type parameter. The used 752 packetization mode governs which NAL unit types are allowed in RTP 753 payloads. Table 3 summarizes the allowed packet payload types for 754 each packetization mode. Packetization modes are explained in more 755 detail in section 6. 757 Table 3. Summary of allowed NAL unit types for each packetization 758 mode (yes = allowed, no = disallowed, ig = ignore) 760 Payload Packet Single NAL Non-Interleaved Interleaved 761 Type Type Unit Mode Mode Mode 762 ------------------------------------------------------------- 763 0 reserved ig ig ig 764 1-23 NAL unit yes yes no 765 24 STAP-A no yes no 766 25 STAP-B no no yes 767 26 MTAP16 no no yes 768 27 MTAP24 no no yes 769 28 FU-A no yes yes 770 29 FU-B no no yes 771 30-31 reserved ig ig ig 773 Some NAL unit or payload type values (indicated as reserved in 774 Table 3) are reserved for future extensions. NAL units of those 775 types SHOULD NOT be sent by a sender (direct as packet payloads, or 776 as aggregation units in aggregation packets, or as fragmented units 777 in FU packets) and MUST be ignored by a receiver. For example, the 778 payload types 1-23, with the associated packet type "NAL unit", are 779 allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", 780 but disallowed in "Interleaved Mode". However, NAL units of NAL 781 unit types 1-23 can be used in "Interleaved Mode" as aggregation 782 units in STAP-B, MTAP16 and MTAP14 packets as well as fragmented 783 units in FU-A and FU-B packets. Similarly, NAL units of NAL unit 784 types 1-23 can also be used in the "Non-Interleaved Mode" as 785 aggregation units in STAP-A packets or fragmented units in FU-A 786 packets, in addition to being directly used as packet payloads. 788 5.5. Decoding Order Number (DON) 790 In the interleaved packetization mode, the transmission order of 791 NAL units is allowed to differ from the decoding order of the NAL 792 units. Decoding order number (DON) is a field in the payload 793 structure or a derived variable that indicates the NAL unit 794 decoding order. Rationale and examples of use cases for 795 transmission out of decoding order and for the use of DON are given 796 in section 13. 798 The coupling of transmission and decoding order is controlled by 799 the OPTIONAL sprop-interleaving-depth media type parameter as 800 follows. When the value of the OPTIONAL sprop-interleaving-depth 801 media type parameter is equal to 0 (explicitly or per default), the 802 transmission order of NAL units MUST conform to the NAL unit 803 decoding order. When the value of the OPTIONAL sprop-interleaving- 804 depth media type parameter is greater than 0, 805 o the order of NAL units in an MTAP16 and an MTAP24 is not 806 required to be the NAL unit decoding order, and 808 o the order of NAL units generated by de-packetizing STAP-Bs, 809 MTAPs, and FUs in two consecutive packets is not required to be 810 the NAL unit decoding order. 812 The RTP payload structures for a single NAL unit packet, an STAP-A, 813 and an FU-A do not include DON. STAP-B and FU-B structures include 814 DON, and the structure of MTAPs enables derivation of DON as 815 specified in section 5.7.2. 817 Informative note: When an FU-A occurs in interleaved mode, it 818 always follows an FU-B, which sets its DON. 820 Informative note: If a transmitter wants to encapsulate a single 821 NAL unit per packet and transmit packets out of their decoding 822 order, STAP-B packet type can be used. 824 In the single NAL unit packetization mode, the transmission order 825 of NAL units, determined by the RTP sequence number, MUST be the 826 same as their NAL unit decoding order. In the non-interleaved 827 packetization mode, the transmission order of NAL units in single 828 NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL 829 unit decoding order. The NAL units within an STAP MUST appear in 830 the NAL unit decoding order. Thus, the decoding order is first 831 provided through the implicit order within a STAP, and second 832 provided through the RTP sequence number for the order between 833 STAPs, FUs, and single NAL unit packets. 835 Signaling of the value of DON for NAL units carried in STAP-B, MTAP, 836 and a series of fragmentation units starting with an FU-B is 837 specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON 838 value of the first NAL unit in transmission order MAY be set to any 839 value. Values of DON are in the range of 0 to 65535, inclusive. 840 After reaching the maximum value, the value of DON wraps around to 841 0. 843 The decoding order of two NAL units contained in any STAP-B, MTAP, 844 or a series of fragmentation units starting with an FU-B is 845 determined as follows. Let DON(i) be the decoding order number of 846 the NAL unit having index i in the transmission order. Function 847 don_diff(m,n) is specified as follows: 849 If DON(m) == DON(n), don_diff(m,n) = 0 850 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 851 don_diff(m,n) = DON(n) - DON(m) 853 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 854 don_diff(m,n) = 65536 - DON(m) + DON(n) 856 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 857 don_diff(m,n) = - (DON(m) + 65536 - DON(n)) 859 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 860 don_diff(m,n) = - (DON(m) - DON(n)) 862 A positive value of don_diff(m,n) indicates that the NAL unit 863 having transmission order index n follows, in decoding order, the 864 NAL unit having transmission order index m. When don_diff(m,n) is 865 equal to 0, then the NAL unit decoding order of the two NAL units 866 can be in either order. A negative value of don_diff(m,n) 867 indicates that the NAL unit having transmission order index n 868 precedes, in decoding order, the NAL unit having transmission order 869 index m. 871 Values of DON related fields (DON, DONB, and DOND; see section 5.7) 872 MUST be such that the decoding order determined by the values of 873 DON, as specified above, conforms to the NAL unit decoding order. 874 If the order of two NAL units in NAL unit decoding order is 875 switched and the new order does not conform to the NAL unit 876 decoding order, the NAL units MUST NOT have the same value of DON. 877 If the order of two consecutive NAL units in the NAL unit stream is 878 switched and the new order still conforms to the NAL unit decoding 879 order, the NAL units MAY have the same value of DON. For example, 880 when arbitrary slice order is allowed by the video coding profile 881 in use, all the coded slice NAL units of a coded picture are 882 allowed to have the same value of DON. Consequently, NAL units 883 having the same value of DON can be decoded in any order, and two 884 NAL units having a different value of DON should be passed to the 885 decoder in the order specified above. When two consecutive NAL 886 units in the NAL unit decoding order have a different value of DON, 887 the value of DON for the second NAL unit in decoding order SHOULD 888 be the value of DON for the first, incremented by one. 890 An example of the de-packetization process to recover the NAL unit 891 decoding order is given in section 7. 893 Informative note: Receivers should not expect that the absolute 894 difference of values of DON for two consecutive NAL units in the 895 NAL unit decoding order will be equal to one, even in error-free 896 transmission. An increment by one is not required, as at the 897 time of associating values of DON to NAL units, it may not be 898 known whether all NAL units are delivered to the receiver. For 899 example, a gateway may not forward coded slice NAL units of non- 900 reference pictures or SEI NAL units when there is a shortage of 901 bit rate in the network to which the packets are forwarded. In 902 another example, a live broadcast is interrupted by pre-encoded 903 content, such as commercials, from time to time. The first 904 intra picture of a pre-encoded clip is transmitted in advance to 905 ensure that it is readily available in the receiver. When 906 transmitting the first intra picture, the originator does not 907 exactly know how many NAL units will be encoded before the first 908 intra picture of the pre-encoded clip follows in decoding order. 909 Thus, the values of DON for the NAL units of the first intra 910 picture of the pre-encoded clip have to be estimated when they 911 are transmitted, and gaps in values of DON may occur. 913 5.6. Single NAL Unit Packet 915 The single NAL unit packet defined here MUST contain only one NAL 916 unit, of the types defined in [1]. This means that neither an 917 aggregation packet nor a fragmentation unit can be used within a 918 single NAL unit packet. A NAL unit stream composed by de- 919 packetizing single NAL unit packets in RTP sequence number order 920 MUST conform to the NAL unit decoding order. The structure of the 921 single NAL unit packet is shown in Figure 2. 923 Informative note: The first byte of a NAL unit co-serves as the 924 RTP payload header. 926 0 1 2 3 927 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 928 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 929 |F|NRI| Type | | 930 +-+-+-+-+-+-+-+-+ | 931 | | 932 | Bytes 2..n of a Single NAL unit | 933 | | 934 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 935 | :...OPTIONAL RTP padding | 936 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 938 Figure 2 RTP payload format for single NAL unit packet 940 5.7. Aggregation Packets 942 Aggregation packets are the NAL unit aggregation scheme of this 943 payload specification. The scheme is introduced to reflect the 944 dramatically different MTU sizes of two key target networks: 945 wireline IP networks (with an MTU size that is often limited by the 946 Ethernet MTU size; roughly 1500 bytes), and IP or non-IP (e.g., 947 ITU-T H.324/M) based wireless communication systems with preferred 948 transmission unit sizes of 254 bytes or less. To prevent media 949 transcoding between the two worlds, and to avoid undesirable 950 packetization overhead, a NAL unit aggregation scheme is introduced. 952 Two types of aggregation packets are defined by this specification: 954 o Single-time aggregation packet (STAP): aggregates NAL units with 955 identical NALU-time. Two types of STAPs are defined, one 956 without DON (STAP-A) and another including DON (STAP-B). 958 o Multi-time aggregation packet (MTAP): aggregates NAL units with 959 potentially differing NALU-time. Two different MTAPs are 960 defined, differing in the length of the NAL unit timestamp 961 offset. 963 Each NAL unit to be carried in an aggregation packet is 964 encapsulated in an aggregation unit. Please see below for the four 965 different aggregation units and their characteristics. 967 The structure of the RTP payload format for aggregation packets is 968 presented in Figure 3. 970 0 1 2 3 971 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 972 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 973 |F|NRI| Type | | 974 +-+-+-+-+-+-+-+-+ | 975 | | 976 | one or more aggregation units | 977 | | 978 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 979 | :...OPTIONAL RTP padding | 980 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 982 Figure 3 RTP payload format for aggregation packets 984 MTAPs and STAPs share the following packetization rules: The RTP 985 timestamp MUST be set to the earliest of the NALU-times of all the 986 NAL units to be aggregated. The type field of the NAL unit type 987 octet MUST be set to the appropriate value, as indicated in Table 4. 988 The F bit MUST be cleared if all F bits of the aggregated NAL units 989 are zero; otherwise, it MUST be set. The value of NRI MUST be the 990 maximum of all the NAL units carried in the aggregation packet. 992 Table 4. Type field for STAPs and MTAPs 994 Type Packet Timestamp offset DON related fields 995 field length (DON, DONB, DOND) 996 (in bits) present 997 -------------------------------------------------------- 998 24 STAP-A 0 no 999 25 STAP-B 0 yes 1000 26 MTAP16 16 yes 1001 27 MTAP24 24 yes 1003 The marker bit in the RTP header is set to the value that the 1004 marker bit of the last NAL unit of the aggregated packet would have 1005 if it were transported in its own RTP packet. 1007 The payload of an aggregation packet consists of one or more 1008 aggregation units. See sections 5.7.1 and 5.7.2 for the four 1009 different types of aggregation units. An aggregation packet can 1010 carry as many aggregation units as necessary; however, the total 1011 amount of data in an aggregation packet obviously MUST fit into an 1012 IP packet, and the size SHOULD be chosen so that the resulting IP 1013 packet is smaller than the MTU size. An aggregation packet MUST 1014 NOT contain fragmentation units specified in section 5.8. 1015 Aggregation packets MUST NOT be nested; i.e., an aggregation packet 1016 MUST NOT contain another aggregation packet. 1018 5.7.1. Single-Time Aggregation Packet 1020 Single-time aggregation packet (STAP) SHOULD be used whenever NAL 1021 units are aggregated that all share the same NALU-time. The 1022 payload of an STAP-A does not include DON and consists of at least 1023 one single-time aggregation unit, as presented in Figure 4. The 1024 payload of an STAP-B consists of a 16-bit unsigned decoding order 1025 number (DON) (in network byte order) followed by at least one 1026 single-time aggregation unit, as presented in Figure 5. 1028 0 1 2 3 1029 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1030 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 : | 1032 +-+-+-+-+-+-+-+-+ | 1033 | | 1034 | single-time aggregation units | 1035 | | 1036 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1037 | : 1038 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1040 Figure 4 Payload format for STAP-A 1042 0 1 2 3 1043 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1044 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1045 : decoding order number (DON) | | 1046 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1047 | | 1048 | single-time aggregation units | 1049 | | 1050 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1051 | : 1052 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1054 Figure 5 Payload format for STAP-B 1056 The DON field specifies the value of DON for the first NAL unit in 1057 an STAP-B in transmission order. For each successive NAL unit in 1058 appearance order in an STAP-B, the value of DON is equal to (the 1059 value of DON of the previous NAL unit in the STAP-B + 1) % 65536, 1060 in which '%' stands for the modulo operation. 1062 A single-time aggregation unit consists of 16-bit unsigned size 1063 information (in network byte order) that indicates the size of the 1064 following NAL unit in bytes (excluding these two octets, but 1065 including the NAL unit type octet of the NAL unit), followed by the 1066 NAL unit itself, including its NAL unit type byte. A single-time 1067 aggregation unit is byte aligned within the RTP payload, but it may 1068 not be aligned on a 32-bit word boundary. Figure 6 presents the 1069 structure of the single-time aggregation unit. 1071 0 1 2 3 1072 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1073 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 : NAL unit size | | 1075 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1076 | | 1077 | NAL unit | 1078 | | 1079 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1080 | : 1081 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1083 Figure 6 Structure for single-time aggregation unit 1085 Figure 7 presents an example of an RTP packet that contains an 1086 STAP-A. The STAP contains two single-time aggregation units, 1087 labeled as 1 and 2 in the figure. 1089 0 1 2 3 1090 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1091 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 | RTP Header | 1093 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1094 |STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR | 1095 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1096 | NALU 1 Data | 1097 : : 1098 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 | | NALU 2 Size | NALU 2 HDR | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 | NALU 2 Data | 1102 : : 1103 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 | :...OPTIONAL RTP padding | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1107 Figure 7 An example of an RTP packet including an STAP-A containing 1108 two single-time aggregation units 1110 Figure 8 presents an example of an RTP packet that contains an 1111 STAP-B. The STAP contains two single-time aggregation units, 1112 labeled as 1 and 2 in the figure. 1114 0 1 2 3 1115 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 | RTP Header | 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1119 |STAP-B NAL HDR | DON | NALU 1 Size | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1121 | NALU 1 Size | NALU 1 HDR | NALU 1 Data | 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1123 : : 1124 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1125 | | NALU 2 Size | NALU 2 HDR | 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1127 | NALU 2 Data | 1128 : : 1129 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1130 | :...OPTIONAL RTP padding | 1131 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 Figure 8 An example of an RTP packet including an STAP-B containing 1134 two single-time aggregation units 1136 5.7.2. Multi-Time Aggregation Packets (MTAPs) 1138 The NAL unit payload of MTAPs consists of a 16-bit unsigned 1139 decoding order number base (DONB) (in network byte order) and one 1140 or more multi-time aggregation units, as presented in Figure 9. 1141 DONB MUST contain the value of DON for the first NAL unit in the 1142 NAL unit decoding order among the NAL units of the MTAP. 1144 Informative note: The first NAL unit in the NAL unit decoding 1145 order is not necessarily the first NAL unit in the order in 1146 which the NAL units are encapsulated in an MTAP. 1148 0 1 2 3 1149 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1150 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1151 : decoding order number base | | 1152 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1153 | | 1154 | multi-time aggregation units | 1155 | | 1156 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1157 | : 1158 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1160 Figure 9 NAL unit payload format for MTAPs 1162 Two different multi-time aggregation units are defined in this 1163 specification. Both of them consist of 16 bits unsigned size 1164 information of the following NAL unit (in network byte order), an 1165 8-bit unsigned decoding order number difference (DOND), and n bits 1166 (in network byte order) of timestamp offset (TS offset) for this 1167 NAL unit, whereby n can be 16 or 24. The choice between the 1168 different MTAP types (MTAP16 and MTAP24) is application dependent: 1169 the larger the timestamp offset is, the higher the flexibility of 1170 the MTAP, but the overhead is also higher. 1172 The structure of the multi-time aggregation units for MTAP16 and 1173 MTAP24 are presented in Figures 10 and 11, respectively. The 1174 starting or ending position of an aggregation unit within a packet 1175 is not required to be on a 32-bit word boundary. The DON of the 1176 NAL unit contained in a multi-time aggregation unit is equal to 1177 (DONB + DOND) % 65536, in which % denotes the modulo operation. 1178 This memo does not specify how the NAL units within an MTAP are 1179 ordered, but, in most cases, NAL unit decoding order SHOULD be used. 1181 The timestamp offset field MUST be set to a value equal to the 1182 value of the following formula: If the NALU-time is larger than or 1183 equal to the RTP timestamp of the packet, then the timestamp offset 1184 equals (the NALU-time of the NAL unit - the RTP timestamp of the 1185 packet). If the NALU-time is smaller than the RTP timestamp of the 1186 packet, then the timestamp offset is equal to the NALU-time + (2^32 1187 - the RTP timestamp of the packet). 1189 0 1 2 3 1190 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1191 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1192 : NAL unit size | DOND | TS offset | 1193 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1194 | TS offset | | 1195 +-+-+-+-+-+-+-+-+ NAL unit | 1196 | | 1197 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1198 | : 1199 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1201 Figure 10 Multi-time aggregation unit for MTAP16 1203 0 1 2 3 1204 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1205 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1206 : NAL unit size | DOND | TS offset | 1207 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1208 | TS offset | | 1209 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1210 | NAL unit | 1211 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1212 | : 1213 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1215 Figure 11 Multi-time aggregation unit for MTAP24 1217 For the "earliest" multi-time aggregation unit in an MTAP the 1218 timestamp offset MUST be zero. Hence, the RTP timestamp of the 1219 MTAP itself is identical to the earliest NALU-time. 1221 Informative note: The "earliest" multi-time aggregation unit is 1222 the one that would have the smallest extended RTP timestamp 1223 among all the aggregation units of an MTAP if the NAL units 1224 contained in the aggregation units were encapsulated in single 1225 NAL unit packets. An extended timestamp is a timestamp that has 1226 more than 32 bits and is capable of counting the wraparound of 1227 the timestamp field, thus enabling one to determine the smallest 1228 value if the timestamp wraps. Such an "earliest" aggregation 1229 unit may not be the first one in the order in which the 1230 aggregation units are encapsulated in an MTAP. The "earliest" 1231 NAL unit need not be the same as the first NAL unit in the NAL 1232 unit decoding order either. 1234 Figure 12 presents an example of an RTP packet that contains a 1235 multi-time aggregation packet of type MTAP16 that contains two 1236 multi-time aggregation units, labeled as 1 and 2 in the figure. 1238 0 1 2 3 1239 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1240 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1241 | RTP Header | 1242 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1243 |MTAP16 NAL HDR | decoding order number base | NALU 1 Size | 1244 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1245 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offset | 1246 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1247 | NALU 1 HDR | NALU 1 DATA | 1248 +-+-+-+-+-+-+-+-+ + 1249 : : 1250 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1251 | | NALU 2 SIZE | NALU 2 DOND | 1252 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1253 | NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA | 1254 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1255 : : 1256 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1257 | :...OPTIONAL RTP padding | 1258 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1260 Figure 12 An RTP packet including a multi-time aggregation packet 1261 of type MTAP16 containing two multi-time aggregation units 1263 Figure 13 presents an example of an RTP packet that contains a 1264 multi-time aggregation packet of type MTAP24 that contains two 1265 multi-time aggregation units, labeled as 1 and 2 in the figure. 1267 0 1 2 3 1268 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1269 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1270 | RTP Header | 1271 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1272 |MTAP24 NAL HDR | decoding order number base | NALU 1 Size | 1273 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1274 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offs | 1275 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1276 |NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA | 1277 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1278 : : 1279 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1280 | | NALU 2 SIZE | NALU 2 DOND | 1281 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1282 | NALU 2 TS offset | NALU 2 HDR | 1283 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1284 | NALU 2 DATA | 1285 : : 1286 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1287 | :...OPTIONAL RTP padding | 1288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1290 Figure 13 An RTP packet including a multi-time aggregation packet 1291 of type MTAP24 containing two multi-time aggregation units 1293 5.7.3. Fragmentation Units (FUs) 1295 This payload type allows fragmenting a NAL unit into several RTP 1296 packets. Doing so on the application layer instead of relying on 1297 lower layer fragmentation (e.g., by IP) has the following 1298 advantages: 1300 o The payload format is capable of transporting NAL units bigger 1301 than 64 kbytes over an IPv4 network that may be present in pre- 1302 recorded video, particularly in High Definition formats (there 1303 is a limit of the number of slices per picture, which results in 1304 a limit of NAL units per picture, which may result in big NAL 1305 units). 1307 o The fragmentation mechanism allows fragmenting a single NAL unit 1308 and applying generic forward error correction as described in 1309 section 12.5. 1311 Fragmentation is defined only for a single NAL unit and not for any 1312 aggregation packets. A fragment of a NAL unit consists of an 1313 integer number of consecutive octets of that NAL unit. Each octet 1314 of the NAL unit MUST be part of exactly one fragment of that NAL 1315 unit. Fragments of the same NAL unit MUST be sent in consecutive 1316 order with ascending RTP sequence numbers (with no other RTP 1317 packets within the same RTP packet stream being sent between the 1318 first and last fragment). Similarly, a NAL unit MUST be 1319 reassembled in RTP sequence number order. 1321 When a NAL unit is fragmented and conveyed within fragmentation 1322 units (FUs), it is referred to as a fragmented NAL unit. STAPs and 1323 MTAPs MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU 1324 MUST NOT contain another FU. 1326 The RTP timestamp of an RTP packet carrying an FU is set to the 1327 NALU-time of the fragmented NAL unit. 1329 Figure 14 presents the RTP payload format for FU-As. An FU-A 1330 consists of a fragmentation unit indicator of one octet, a 1331 fragmentation unit header of one octet, and a fragmentation unit 1332 payload. 1334 0 1 2 3 1335 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1336 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1337 | FU indicator | FU header | | 1338 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1339 | | 1340 | FU payload | 1341 | | 1342 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1343 | :...OPTIONAL RTP padding | 1344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1346 Figure 14 RTP payload format for FU-A 1348 Figure 15 presents the RTP payload format for FU-Bs. An FU-B 1349 consists of a fragmentation unit indicator of one octet, a 1350 fragmentation unit header of one octet, a decoding order number 1351 (DON) (in network byte order), and a fragmentation unit payload. 1352 In other words, the structure of FU-B is the same as the structure 1353 of FU-A, except for the additional DON field. 1355 0 1 2 3 1356 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1357 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1358 | FU indicator | FU header | DON | 1359 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 1360 | | 1361 | FU payload | 1362 | | 1363 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1364 | :...OPTIONAL RTP padding | 1365 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1367 Figure 15 RTP payload format for FU-B 1369 NAL unit type FU-B MUST be used in the interleaved packetization 1370 mode for the first fragmentation unit of a fragmented NAL unit. 1371 NAL unit type FU-B MUST NOT be used in any other case. In other 1372 words, in the interleaved packetization mode, each NALU that is 1373 fragmented has an FU-B as the first fragment, followed by one or 1374 more FU-A fragments. 1376 The FU indicator octet has the following format: 1378 +---------------+ 1379 |0|1|2|3|4|5|6|7| 1380 +-+-+-+-+-+-+-+-+ 1381 |F|NRI| Type | 1382 +---------------+ 1384 Values equal to 28 and 29 in the Type field of the FU indicator 1385 octet identify an FU-A and an FU-B, respectively. The use of the F 1386 bit is described in section 5.3. The value of the NRI field MUST 1387 be set according to the value of the NRI field in the fragmented 1388 NAL unit. 1390 The FU header has the following format: 1392 +---------------+ 1393 |0|1|2|3|4|5|6|7| 1394 +-+-+-+-+-+-+-+-+ 1395 |S|E|R| Type | 1396 +---------------+ 1398 S: 1 bit 1399 When set to one, the Start bit indicates the start of a 1400 fragmented NAL unit. When the following FU payload is not the 1401 start of a fragmented NAL unit payload, the Start bit is set to 1402 zero. 1404 E: 1 bit 1405 When set to one, the End bit indicates the end of a fragmented 1406 NAL unit, i.e., the last byte of the payload is also the last 1407 byte of the fragmented NAL unit. When the following FU payload 1408 is not the last fragment of a fragmented NAL unit, the End bit 1409 is set to zero. 1411 R: 1 bit 1412 The Reserved bit MUST be equal to 0 and MUST be ignored by the 1413 receiver. 1415 Type: 5 bits 1416 The NAL unit payload type as defined in Table 7-1 of [1]. 1418 The value of DON in FU-Bs is selected as described in section 5.5. 1420 Informative note: The DON field in FU-Bs allows gateways to 1421 fragment NAL units to FU-Bs without organizing the incoming NAL 1422 units to the NAL unit decoding order. 1424 A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the 1425 Start bit and End bit MUST NOT both be set to one in the same FU 1426 header. 1428 The FU payload consists of fragments of the payload of the 1429 fragmented NAL unit so that if the fragmentation unit payloads of 1430 consecutive FUs are sequentially concatenated, the payload of the 1431 fragmented NAL unit can be reconstructed. The NAL unit type octet 1432 of the fragmented NAL unit is not included as such in the 1433 fragmentation unit payload, but rather the information of the NAL 1434 unit type octet of the fragmented NAL unit is conveyed in F and NRI 1435 fields of the FU indicator octet of the fragmentation unit and in 1436 the type field of the FU header. An FU payload MAY have any number 1437 of octets and MAY be empty. 1439 Informative note: Empty FUs are allowed to reduce the latency of 1440 a certain class of senders in nearly lossless environments. 1441 These senders can be characterized in that they packetize NALU 1442 fragments before the NALU is completely generated and, hence, 1443 before the NALU size is known. If zero-length NALU fragments 1444 were not allowed, the sender would have to generate at least one 1445 bit of data of the following fragment before the current 1446 fragment could be sent. Due to the characteristics of H.264, 1447 where sometimes several macroblocks occupy zero bits, this is 1448 undesirable and can add delay. However, the (potential) use of 1449 zero-length NALU fragments should be carefully weighed against 1450 the increased risk of the loss of at least a part of the NALU 1451 because of the additional packets employed for its transmission. 1453 If a fragmentation unit is lost, the receiver SHOULD discard all 1454 following fragmentation units in transmission order corresponding 1455 to the same fragmented NAL unit. 1457 A receiver in an endpoint or in a MANE MAY aggregate the first n-1 1458 fragments of a NAL unit to an (incomplete) NAL unit, even if 1459 fragment n of that NAL unit is not received. In this case, the 1460 forbidden_zero_bit of the NAL unit MUST be set to one to indicate a 1461 syntax violation. 1463 6. Packetization Rules 1465 The packetization modes are introduced in section 5.2. The 1466 packetization rules common to more than one of the packetization 1467 modes are specified in section 6.1. The packetization rules for 1468 the single NAL unit mode, the non-interleaved mode, and the 1469 interleaved mode are specified in sections 6.2, 6.3, and 6.4, 1470 respectively. 1472 6.1. Common Packetization Rules 1474 All senders MUST enforce the following packetization rules 1475 regardless of the packetization mode in use: 1477 o Coded slice NAL units or coded slice data partition NAL units 1478 belonging to the same coded picture (and thus sharing the same 1479 RTP timestamp value) MAY be sent in any order; however, for 1480 delay-critical systems, they SHOULD be sent in their original 1481 decoding order to minimize the delay. Note that the decoding 1482 order is the order of the NAL units in the bitstream. 1484 o Parameter sets are handled in accordance with the rules and 1485 recommendations given in section 8.4. 1487 o MANEs MUST NOT duplicate any NAL unit except for sequence or 1488 picture parameter set NAL units, as neither this memo nor the 1489 H.264 specification provides means to identify duplicated NAL 1490 units. Sequence and picture parameter set NAL units MAY be 1491 duplicated to make their correct reception more probable, but 1492 any such duplication MUST NOT affect the contents of any active 1493 sequence or picture parameter set. Duplication SHOULD be 1494 performed on the application layer and not by duplicating RTP 1495 packets (with identical sequence numbers). 1497 Senders using the non-interleaved mode and the interleaved mode 1498 MUST enforce the following packetization rule: 1500 o MANEs MAY convert single NAL unit packets into one aggregation 1501 packet, convert an aggregation packet into several single NAL 1502 unit packets, or mix both concepts, in an RTP translator. The 1503 RTP translator SHOULD take into account at least the following 1504 parameters: path MTU size, unequal protection mechanisms (e.g., 1505 through packet-based FEC according to RFC 2733 [18], especially 1506 for sequence and picture parameter set NAL units and coded slice 1507 data partition A NAL units), bearable latency of the system, and 1508 buffering capabilities of the receiver. 1510 Informative note: An RTP translator is required to handle 1511 RTCP as per RFC 3550. 1513 6.2. Single NAL Unit Mode 1515 This mode is in use when the value of the OPTIONAL packetization- 1516 mode media type parameter is equal to 0 or the packetization-mode 1517 is not present. All receivers MUST support this mode. It is 1518 primarily intended for low-delay applications that are compatible 1519 with systems using ITU-T Recommendation H.241 [3] (see section 1520 12.1). Only single NAL unit packets MAY be used in this mode. 1521 STAPs, MTAPs, and FUs MUST NOT be used. The transmission order of 1522 single NAL unit packets MUST comply with the NAL unit decoding 1523 order. 1525 6.3. Non-Interleaved Mode 1527 This mode is in use when the value of the OPTIONAL packetization- 1528 mode media type parameter is equal to 1. This mode SHOULD be 1529 supported. It is primarily intended for low-delay applications. 1530 Only single NAL unit packets, STAP-As, and FU-As MAY be used in 1531 this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. The 1532 transmission order of NAL units MUST comply with the NAL unit 1533 decoding order. 1535 6.4. Interleaved Mode 1537 This mode is in use when the value of the OPTIONAL packetization- 1538 mode media type parameter is equal to 2. Some receivers MAY 1539 support this mode. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. 1540 STAP-As and single NAL unit packets MUST NOT be used. The 1541 transmission order of packets and NAL units is constrained as 1542 specified in section 5.5. 1544 7. De-Packetization Process 1546 The de-packetization process is implementation dependent. 1547 Therefore, the following description should be seen as an example 1548 of a suitable implementation. Other schemes may be used as well as 1549 long as the output for the same input is the same as the process 1550 described below. The same output means that the resulting NAL 1551 units, and their order, are identical. Optimizations relative to 1552 the described algorithms are likely possible. Section 7.1 presents 1553 the de-packetization process for the single NAL unit and non- 1554 interleaved packetization modes, whereas section 7.2 describes the 1555 process for the interleaved mode. Section 7.3 includes additional 1556 de-packetization guidelines for intelligent receivers. 1558 All normal RTP mechanisms related to buffer management apply. In 1559 particular, duplicated or outdated RTP packets (as indicated by the 1560 RTP sequence number and the RTP timestamp) are removed. To 1561 determine the exact time for decoding, factors such as a possible 1562 intentional delay to allow for proper inter-stream synchronization 1563 must be factored in. 1565 7.1. Single NAL Unit and Non-Interleaved Mode 1567 The receiver includes a receiver buffer to compensate for 1568 transmission delay jitter. The receiver stores incoming packets in 1569 reception order into the receiver buffer. Packets are de- 1570 packetized in RTP sequence number order. If a de-packetized packet 1571 is a single NAL unit packet, the NAL unit contained in the packet 1572 is passed directly to the decoder. If a de-packetized packet is an 1573 STAP-A, the NAL units contained in the packet are passed to the 1574 decoder in the order in which they are encapsulated in the packet. 1575 For all the FU-A packets containing fragments of a single NAL unit, 1576 the de-packetized fragments are concatenated in their sending order 1577 to recover the NAL unit, which is then passed to the decoder. 1579 Informative note: If the decoder supports Arbitrary Slice Order, 1580 coded slices of a picture can be passed to the decoder in any 1581 order regardless of their reception and transmission order. 1583 7.2. Interleaved Mode 1585 The general concept behind these de-packetization rules is to 1586 reorder NAL units from transmission order to the NAL unit decoding 1587 order. 1589 The receiver includes a receiver buffer, which is used to 1590 compensate for transmission delay jitter and to reorder NAL units 1591 from transmission order to the NAL unit decoding order. In this 1592 section, the receiver operation is described under the assumption 1593 that there is no transmission delay jitter. To make a difference 1594 from a practical receiver buffer that is also used for compensation 1595 of transmission delay jitter, the receiver buffer is here after 1596 called the de-interleaving buffer in this section. Receivers 1597 SHOULD also prepare for transmission delay jitter; i.e., either 1598 reserve separate buffers for transmission delay jitter buffering 1599 and de-interleaving buffering or use a receiver buffer for both 1600 transmission delay jitter and de-interleaving. Moreover, receivers 1601 SHOULD take transmission delay jitter into account in the buffering 1602 operation; e.g., by additional initial buffering before starting of 1603 decoding and playback. 1605 This section is organized as follows: subsection 7.2.1 presents how 1606 to calculate the size of the de-interleaving buffer. Subsection 1607 7.2.2 specifies the receiver process on how to organize received 1608 NAL units to the NAL unit decoding order. 1610 7.2.1. Size of the De-interleaving Buffer 1612 In either Offer/Answer or declarative SDP usage, the sprop-deint- 1613 buf-req media type parameter signals the requirement for the de- 1614 interleaving buffer size. It is therefore RECOMMENDED to set the 1615 de-interleaving buffer size, in terms of number of bytes, equal to 1616 or greater than the value of sprop-deint-buf-req media type 1617 parameter. 1619 When the SDP Offer/Answer model or any other capability exchange 1620 procedure is used in session setup, the properties of the received 1621 stream SHOULD be such that the receiver capabilities are not 1622 exceeded. In the SDP Offer/Answer model, the receiver can indicate 1623 its capabilities to allocate a de-interleaving buffer with the 1624 deint-buf-cap media type parameter. See section 8.1 for further 1625 information on deint-buf-cap and sprop-deint-buf-req media type 1626 parameters and section 8.2.2 for further information on their use 1627 in the SDP Offer/Answer model. 1629 7.2.2. De-interleaving Process 1631 There are two buffering states in the receiver: initial buffering 1632 and buffering while playing. Initial buffering occurs when the RTP 1633 session is initialized. After initial buffering, decoding and 1634 playback are started, and the buffering-while-playing mode is used. 1636 Regardless of the buffering state, the receiver stores incoming NAL 1637 units, in reception order, in the de-interleaving buffer as follows. 1638 NAL units of aggregation packets are stored in the de-interleaving 1639 buffer individually. The value of DON is calculated and stored for 1640 each NAL unit. 1642 The receiver operation is described below with the help of the 1643 following functions and constants: 1645 o Function AbsDON is specified in section 8.1. 1647 o Function don_diff is specified in section 5.5. 1649 o Constant N is the value of the OPTIONAL sprop-interleaving-depth 1650 media type parameter (see section 8.1) incremented by 1. 1652 Initial buffering lasts until one of the following conditions is 1653 fulfilled: 1655 o There are N or more VCL NAL units in the de-interleaving buffer. 1657 o If sprop-max-don-diff is present, don_diff(m,n) is greater than 1658 the value of sprop-max-don-diff, in which n corresponds to the 1659 NAL unit having the greatest value of AbsDON among the received 1660 NAL units and m corresponds to the NAL unit having the smallest 1661 value of AbsDON among the received NAL units. 1663 o Initial buffering has lasted for the duration equal to or 1664 greater than the value of the OPTIONAL sprop-init-buf-time media 1665 type parameter. 1667 The NAL units to be removed from the de-interleaving buffer are 1668 determined as follows: 1670 o If the de-interleaving buffer contains at least N VCL NAL units, 1671 NAL units are removed from the de-interleaving buffer and passed 1672 to the decoder in the order specified below until the buffer 1673 contains N-1 VCL NAL units. 1675 o If sprop-max-don-diff is present, all NAL units m for which 1676 don_diff(m,n) is greater than sprop-max-don-diff are removed 1677 from the de-interleaving buffer and passed to the decoder in the 1678 order specified below. Herein, n corresponds to the NAL unit 1679 having the greatest value of AbsDON among the NAL units in the 1680 de-interleaving buffer. 1682 The order in which NAL units are passed to the decoder is specified 1683 as follows: 1685 o Let PDON be a variable that is initialized to 0 at the beginning 1686 of the RTP session. 1688 o For each NAL unit associated with a value of DON, a DON distance 1689 is calculated as follows. If the value of DON of the NAL unit 1690 is larger than the value of PDON, the DON distance is equal to 1691 DON - PDON. Otherwise, the DON distance is equal to 65535 - 1692 PDON + DON + 1. 1694 o NAL units are delivered to the decoder in ascending order of DON 1695 distance. If several NAL units share the same value of DON 1696 distance, they can be passed to the decoder in any order. 1698 o When a desired number of NAL units have been passed to the 1699 decoder, the value of PDON is set to the value of DON for the 1700 last NAL unit passed to the decoder. 1702 7.3. Additional De-Packetization Guidelines 1704 The following additional de-packetization rules may be used to 1705 implement an operational H.264 de-packetizer: 1707 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1708 coded slice data partitions A (DPAs). If a lost DPA is detected, 1709 after taking into account possible retransmission and FEC, a 1710 gateway may decide not to send the corresponding coded slice 1711 data partitions B and C, as their information is meaningless for 1712 H.264 decoders. In this way a MANE can reduce network load by 1713 discarding useless packets without parsing a complex bitstream. 1715 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1716 FUs. If a lost FU is found, a gateway may decide not to send 1717 the following FUs of the same fragmented NAL unit, as their 1718 information is meaningless for H.264 decoders. In this way a 1719 MANE can reduce network load by discarding useless packets 1720 without parsing a complex bitstream. 1722 o Intelligent receivers having to discard packets or NALUs should 1723 first discard all packets/NALUs in which the value of the NRI 1724 field of the NAL unit type octet is equal to 0. This will 1725 minimize the impact on user experience and keep the reference 1726 pictures intact. If more packets have to be discarded, then 1727 packets with a numerically lower NRI value should be discarded 1728 before packets with a numerically higher NRI value. However, 1729 discarding any packets with an NRI bigger than 0 very likely 1730 leads to decoder drift and SHOULD be avoided. 1732 8. Payload Format Parameters 1734 This section specifies the parameters that MAY be used to select 1735 optional features of the payload format and certain features of the 1736 bitstream. The parameters are specified here as part of the media 1737 subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. 1738 A mapping of the parameters into the Session Description Protocol 1739 (SDP) [6] is also provided for applications that use SDP. 1740 Equivalent parameters could be defined elsewhere for use with 1741 control protocols that do not use SDP. 1743 Some parameters provide a receiver with the properties of the 1744 stream that will be sent. The names of all these parameters start 1745 with "sprop" for stream properties. Some of these "sprop" 1746 parameters are limited by other payload or codec configuration 1747 parameters. For example, the sprop-parameter-sets parameter is 1748 constrained by the profile-level-id parameter. The media sender 1749 selects all "sprop" parameters rather than the receiver. This 1750 uncommon characteristic of the "sprop" parameters may not be 1751 compatible with some signaling protocol concepts, in which case the 1752 use of these parameters SHOULD be avoided. 1754 8.1. Media Type Registration 1756 The media subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is 1757 allocated from the IETF tree. 1759 The receiver MUST ignore any unspecified parameter. 1761 Media Type name: video 1763 Media subtype name: H264 1765 Required parameters: none 1767 OPTIONAL parameters: 1769 profile-level-id: 1770 A base16 [7] (hexadecimal) representation of the following 1771 three bytes in the sequence parameter set NAL unit specified 1772 in [1]: 1) profile_idc, 2) a byte herein referred to as 1773 profile-iop, composed of the values of constraint_set0_flag, 1774 constraint_set1_flag,constraint_set2_flag, 1775 constraint_set3_flag, and reserved_zero_4bits in bit- 1776 significance order, starting from the most significant bit, 1777 and 3) level_idc. Note that reserved_zero_4bits is required 1778 to be equal to 0 in [1], but other values for it may be 1779 specified in the future by ITU-T or ISO/IEC. 1781 The profile-level-id parameter indicates the default sub- 1782 profile, i.e. the subset of coding tools that may have been 1783 used to generate the stream or that the receiver supports, 1784 and the default level of the stream or the receiver supports. 1786 The default sub-profile is indicated collectively by the 1787 profile_idc byte and some fields in the profile-iop byte. 1788 Depending on the values of the fields in the profile-iop byte, 1789 the default sub-profile may be the set of coding tools 1790 supported by one profile, or a common subset of coding tools 1791 of multiple profiles, as specified in subsection 7.4.2.1.1 of 1792 [1]. The default level is indicated by the level_idc byte, 1793 and, when profile_idc is equal to 66, 77 or 88 (the Baseline, 1794 Main, or Extended profile) and level_idc is equal to 11, 1795 additionally by bit 4 (constraint_set3_flag) of the profile- 1796 iop byte. When profile_idc is equal to 66, 77 or 88 (the 1797 Baseline, Main, or Extended profile) and level_idc is equal 1798 to 11, and bit 4 (constraint_set3_flag) of the profile-iop 1799 byte is equal to 1, the default level is level 1b. 1801 Table 5 lists all profiles defined in Annex A of [1] and, for 1802 each of the profiles, the possible combinations of 1803 profile_idc and profile-iop that represent the same sub- 1804 profile. 1806 Table 5. Combinations of profile_idc and profile-iop 1807 representing the same sub-profile corresponding to the 1808 full set of coding tools supported by one profile. In 1809 the following, x may be either 0 or 1, while the profile 1810 names are indicated as follows. CB: Constrained Baseline 1811 profile, B: Baseline profile, M: Main profile, E: 1812 Extended profile, H: High profile, H10: High 10 profile, 1813 H42: High 4:2:2 profile, H44: High 4:4:4 Predictive 1814 profile, H10I: High 10 Intra profile, H42I: High 4:2:2 1815 Intra profile, H44I: High 4:4:4 Intra profile, and C44I: 1816 CAVLC 4:4:4 Intra profile. 1818 Profile profile_idc profile-iop 1819 (hexadecimal) (binary) 1821 CB 42 (B) x1xx0000 1822 same as: 4D (M) 1xxx0000 1823 same as: 58 (E) 11xx0000 1824 same as: 64 (H), 6E (H10), 1xx00000 1825 7A (H42), or F4 (H44) 1826 B 42 (B) x0xx0000 1827 same as: 58 (E) 10xx0000 1828 M 4D (M) 0x0x0000 1829 same as: 64 (H), 6E (H10), 01000000 1830 7A (H42), or F4 (H44) 1831 E 58 00xx0000 1832 H 64 00000000 1833 H10 6E 00000000 1834 H42 7A 00000000 1835 H44 F4 00000000 1836 H10I 64 00010000 1837 H42I 7A 00010000 1838 H44I F4 00010000 1839 C44I 2C 00010000 1841 For example, in the table above, profile_idc equal to 58 1842 (Extended) with profile-iop equal to 11xx0000 indicates the 1843 same sub-profile corresponding to profile_idc equal to 42 1844 (Baseline) with profile-iop equal to x1xx0000. Note that 1845 other combinations of profile_idc and profile-iop (not listed 1846 in Table 5) may represent a sub-profile equivalent to the 1847 common subset of coding tools for more than one profile. 1848 Note also that a decoder conforming to a certain profile may 1849 be able to decode bitstreams conforming to other profiles. 1850 For example, a decoder conforming to the High 4:4:4 profile 1851 at certain level must be able to decode bitstreams confirming 1852 to the Constrained Baseline, Main, High, High 10 or High 1853 4:2:2 profile at the same or a lower level. 1855 If the profile-level-id parameter is used to indicate 1856 properties of a NAL unit stream, it indicates that, to decode 1857 the stream, the minimum subset of coding tools a decoder has 1858 to support is the default sub-profile, and the lowest level 1859 the decoder has to support is the default level. 1861 If the profile-level-id parameter is used for capability 1862 exchange or session setup procedure, it indicates the subset 1863 of coding tools, which is equal to the default sub-profile, 1864 that the codec supports for both receiving and sending. If 1865 max-recv-level is not present, the default level from 1866 profile-level-id indicates the highest level the codec wishes 1867 to support. If max-recv-level is present it indicates the 1868 highest level the codec supports for receiving, for use in 1869 asymmetric exchanges. For either receiving or sending, all 1870 levels that are lower than the highest level supported MUST 1871 also be supported. 1873 Informative note: Capability exchange and session setup 1874 procedures should provide means to list the capabilities 1875 for each supported sub-profile separately. For example, 1876 the one-of-N codec selection procedure of the SDP 1877 Offer/Answer model can be used (section 10.2 of [8]). 1878 The one-of-N codec selection procedure may also be used 1879 to provide different combinations of profile_idc and 1880 profile-iop that represent the same sub-profile. When 1881 there are many different combinations of profile_idc and 1882 profile-iop that represent the same sub-profile, using 1883 the one-of-N codec selection procedure may result into a 1884 fairly large SDP message. Therefore, a receiver should 1885 understand the different equivalent combinations of 1886 profile_idc and profile-iop that represent the same sub- 1887 profile, and be ready to accept an offer using any of the 1888 equivalent combinations. 1890 If no profile-level-id is present, the Baseline Profile 1891 without additional constraints at Level 1 MUST be inferred. 1893 max-recv-level: 1894 This parameter MAY be used to indicate the highest level a 1895 receiver supports when the highest level is higher than the 1896 default level (the level indicated by profile-level-id). The 1897 value of max-recv-level is a base16 (hexadecimal) 1898 representation of the two bytes after the syntax element 1899 profile_idc in the sequence parameter set NAL unit specified 1900 in [1]: profile-iop (as defined above) and level_idc. If the 1901 level_idc byte of max-recv-level is equal to 11, and bit 4 of 1902 the profile-iop byte of max-recv-level is equal to 1, the 1903 highest level the receiver supports is level 1b. If bit 4 of 1904 the profile-iop byte of max-recv-level is equal to 0, the 1905 highest level the receiver supports is equal to the level_idc 1906 byte of max-recv-level divided by 10. 1908 max-recv-level MUST NOT be present if the highest level the 1909 receiver supports is not higher than the default level. 1911 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: 1912 These parameters MAY be used to signal the capabilities of a 1913 receiver implementation. These parameters MUST NOT be used 1914 for any other purpose. The highest level conveyed in the 1915 value of the profile-level-id parameter or the max-recv-level 1916 parameter MUST be such that the receiver is fully capable of 1917 supporting. max-mbps, max-smbps, max-fs, max-cpb, max-dpb, 1918 and max-br MAY be used to indicate capabilities of the 1919 receiver that extend the required capabilities of the 1920 signaled highest level, as specified below. 1922 When more than one parameter from the set (max-mbps, max- 1923 smbps , max-fs, max-cpb, max-dpb, max-br) is present, the 1924 receiver MUST support all signaled capabilities 1925 simultaneously. For example, if both max-mbps and max-br are 1926 present, the signaled highest level with the extension of 1927 both the frame rate and bit rate is supported. That is, the 1928 receiver is able to decode NAL unit streams in which the 1929 macroblock processing rate is up to max-mbps (inclusive), the 1930 bit rate is up to max-br (inclusive), the coded picture 1931 buffer size is derived as specified in the semantics of the 1932 max-br parameter below, and other properties comply with the 1933 highest level specified in the value of the profile-level-id 1934 parameter or the max-recv-level parameter. 1936 If a receiver can support all the properties of level A, the 1937 highest level specified in the value of the profile-level-id 1938 parameter or the max-recv-level parameter MUST be level A 1939 (i.e. MUST NOT be lower than level A). In other words, a 1940 receiver MUST NOT signal values of max-mbps, max-fs, max-cpb, 1941 max-dpb, and max-br that taken together meet the requirements 1942 of a higher level compared to the highest level specified in 1943 the value of the profile-level-id parameter or the max-recv- 1944 level parameter. 1946 Informative note: When the OPTIONAL media type parameters 1947 are used to signal the properties of a NAL unit stream, 1948 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br 1949 are not present, and the value of profile-level-id must 1950 always be such that the NAL unit stream complies fully 1951 with the specified profile and level. 1953 max-mbps: The value of max-mbps is an integer indicating the 1954 maximum macroblock processing rate in units of macroblocks 1955 per second. The max-mbps parameter signals that the receiver 1956 is capable of decoding video at a higher rate than is 1957 required by the signaled highest level conveyed in the value 1958 of the profile-level-id parameter or the max-recv-level 1959 parameter. When max-mbps is signaled, the receiver MUST be 1960 able to decode NAL unit streams that conform to the signaled 1961 highestlevel, with the exception that the MaxMBPS value in 1962 Table A-1 of [1] for the signaled highest level is replaced 1963 with the value of max-mbps. The value of max-mbps MUST be 1964 greater than or equal to the value of MaxMBPS given in Table 1965 A-1 of [1] for the highest level. Senders MAY use this 1966 knowledge to send pictures of a given size at a higher 1967 picture rate than is indicated in the signaled highest level. 1969 max-smbps: The value of max-smbps is an integer indicating the 1970 maximum static macroblock processing rate in units of static 1971 macroblocks per second, under the hypothetical assumption 1972 that all macroblocks are static macroblocks. When max-smbps 1973 is signalled the MaxMBPS value in Table A-1 of [1] should be 1974 replaced with the result of the following computation: 1976 o If the parameter max-mbps is signalled, set a variable 1977 MaxMacroblocksPerSecond to the value of max-mbps. 1978 Otherwise, set MaxMacroblocksPerSecond equal to the value 1979 of MaxMBPS in Table A-1 [1] for the highest level. 1981 o Set a variable P_non-static to the proportion of non- 1982 static macroblocks in picture n. 1984 o Set a variable P_static to the proportion of static 1985 macroblocks in picture n. 1987 o The value of MaxMBPS in Table A-1 of [1] should be 1988 considered by the encoder to be equal to: 1990 MaxMacroblocksPerSecond * max-smbps / ( P_non-static * 1991 max-smbps + P_static * MaxMacroblocksPerSecond) 1993 The encoder should recompute this value for each picture. The 1994 value of max-smbps MUST be greater than the value of MaxMBPS 1995 given in Table A-1 of [1] for the highest level. Senders MAY 1996 use this knowledge to send pictures of a given size at a 1997 higher picture rate than is indicated in the signaled highest 1998 level. 2000 max-fs: The value of max-fs is an integer indicating the maximum 2001 frame size in units of macroblocks. The max-fs parameter 2002 signals that the receiver is capable of decoding larger 2003 picture sizes than are required by the signaled highest level 2004 conveyed in the value of the profile-level-id parameter or 2005 the max-recv-level parameter. When max-fs is signaled, the 2006 receiver MUST be able to decode NAL unit streams that conform 2007 to the signaled highest level, with the exception that the 2008 MaxFS value in Table A-1 of [1] for the signaled highest 2009 level is replaced with the value of max-fs. The value of 2010 max-fs MUST be greater than or equal to the value of MaxFS 2011 given in Table A-1 of [1] for the highest level. Senders MAY 2012 use this knowledge to send larger pictures at a 2013 proportionally lower frame rate than is indicated in the 2014 signaled highest level. 2016 max-cpb: The value of max-cpb is an integer indicating the 2017 maximum coded picture buffer size in units of 1000 bits for 2018 the VCL HRD parameters (see A.3.1 item i of [1]) and in units 2019 of 1200 bits for the NAL HRD parameters (see A.3.1 item j of 2020 [1]). The max-cpb parameter signals that the receiver has 2021 more memory than the minimum amount of coded picture buffer 2022 memory required by the signaled highest level conveyed in the 2023 value of the profile-level-id parameter or the max-recv-level 2024 parameter. When max-cpb is signaled, the receiver MUST be 2025 able to decode NAL unit streams that conform to the signaled 2026 highest level, with the exception that the MaxCPB value in 2027 Table A-1 of [1] for the signaled highest level is replaced 2028 with the value of max-cpb. The value of max-cpb MUST be 2029 greater than or equal to the value of MaxCPB given in Table 2030 A-1 of [1] for the highest level. Senders MAY use this 2031 knowledge to construct coded video streams with greater 2032 variation of bit rate than can be achieved with the MaxCPB 2033 value in Table A-1 of [1]. 2035 Informative note: The coded picture buffer is used in the 2036 hypothetical reference decoder (Annex C) of H.264. The 2037 use of the hypothetical reference decoder is recommended 2038 in H.264 encoders to verify that the produced bitstream 2039 conforms to the standard and to control the output 2040 bitrate. Thus, the coded picture buffer is conceptually 2041 independent of any other potential buffers in the 2042 receiver, including de-interleaving and de-jitter buffers. 2043 The coded picture buffer need not be implemented in 2044 decoders as specified in Annex C of H.264, but rather 2045 standard-compliant decoders can have any buffering 2046 arrangements provided that they can decode standard- 2047 compliant bitstreams. Thus, in practice, the input 2048 buffer for video decoder can be integrated with de- 2049 interleaving and de-jitter buffers of the receiver. 2051 max-dpb: The value of max-dpb is an integer indicating the 2052 maximum decoded picture buffer size in units of 1024 bytes. 2053 The max-dpb parameter signals that the receiver has more 2054 memory than the minimum amount of decoded picture buffer 2055 memory required by the signaled highest level conveyed in the 2056 value of the profile-level-id parameter or the max-recv-level 2057 parameter. When max-dpb is signaled, the receiver MUST be 2058 able to decode NAL unit streams that conform to the signaled 2059 highest level, with the exception that the MaxDPB value in 2060 Table A-1 of [1] for the signaled highest level is replaced 2061 with the value of max-dpb. Consequently, a receiver that 2062 signals max-dpb MUST be capable of storing the following 2063 number of decoded frames, complementary field pairs, and non- 2064 paired fields in its decoded picture buffer: 2066 Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs * 2067 256 * ChromaFormatFactor ), 16) 2069 PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are 2070 defined in [1]. 2072 The value of max-dpb MUST be greater than or equal to the 2073 value of MaxDPB given in Table A-1 of [1] for the highest 2074 level. Senders MAY use this knowledge to construct coded 2075 video streams with improved compression. 2077 Informative note: This parameter was added primarily to 2078 complement a similar codepoint in the ITU-T 2079 Recommendation H.245, so as to facilitate signaling 2080 gateway designs. The decoded picture buffer stores 2081 reconstructed samples. There is no relationship between 2082 the size of the decoded picture buffer and the buffers 2083 used in RTP, especially de-interleaving and de-jitter 2084 buffers. 2086 max-br: The value of max-br is an integer indicating the maximum 2087 video bit rate in units of 1000 bits per second for the VCL 2088 HRD parameters (see A.3.1 item i of [1]) and in units of 1200 2089 bits per second for the NAL HRD parameters (see A.3.1 item j 2090 of [1]). 2092 The max-br parameter signals that the video decoder of the 2093 receiver is capable of decoding video at a higher bit rate 2094 than is required by the signaled highest level conveyed in 2095 the value of the profile-level-id parameter or the max-recv- 2096 level parameter. 2098 When max-br is signaled, the video codec of the receiver MUST 2099 be able to decode NAL unit streams that conform to the 2100 signaled highest level, with the following exceptions in the 2101 limits specified by the highest level: 2103 o The value of max-br replaces the MaxBR value n Table A-1 2104 of [1] for the highest level. 2106 o When the max-cpb parameter is not present, the result of 2107 the following formula replaces the value of MaxCPB in 2108 Table A-1 of [1]: (MaxCPB of the signaled level) * max-br 2109 / (MaxBR of the signaled highest level). 2111 For example, if a receiver signals capability for Level 1.2 2112 with max-br equal to 1550, this indicates a maximum video 2113 bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum 2114 video bitrate of 1860 kbits/sec for NAL HRD parameters, and a 2115 CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000). 2117 The value of max-br MUST be greater than or equal to the 2118 value MaxBR given in Table A-1 of [1] for the signaled 2119 highest level. 2121 Senders MAY use this knowledge to send higher bitrate video 2122 as allowed in the level definition of Annex A of H.264, to 2123 achieve improved video quality. 2125 Informative note: This parameter was added primarily to 2126 complement a similar codepoint in the ITU-T 2127 Recommendation H.245, so as to facilitate signaling 2128 gateway designs. No assumption can be made from the 2129 value of this parameter that the network is capable of 2130 handling such bit rates at any given time. In particular, 2131 no conclusion can be drawn that the signaled bit rate is 2132 possible under congestion control constraints. 2134 redundant-pic-cap: 2135 This parameter signals the capabilities of a receiver 2136 implementation. When equal to 0, the parameter indicates 2137 that the receiver makes no attempt to use redundant coded 2138 pictures to correct incorrectly decoded primary coded 2139 pictures. When equal to 0, the receiver is not capable of 2140 using redundant slices; therefore, a sender SHOULD avoid 2141 sending redundant slices to save bandwidth. When equal to 1, 2142 the receiver is capable of decoding any such redundant slice 2143 that covers a corrupted area in a primary decoded picture (at 2144 least partly), and therefore a sender MAY send redundant 2145 slices. When the parameter is not present, then a value of 0 2146 MUST be used for redundant-pic-cap. When present, the value 2147 of redundant-pic-cap MUST be either 0 or 1. 2149 When the profile-level-id parameter is present in the same 2150 signaling as the redundant-pic-cap parameter, and the profile 2151 indicated in profile-level-id is such that it disallows the 2152 use of redundant coded pictures (e.g., Main Profile), the 2153 value of redundant-pic-cap MUST be equal to 0. When a 2154 receiver indicates redundant-pic-cap equal to 0, the received 2155 stream SHOULD NOT contain redundant coded pictures. 2157 Informative note: Even if redundant-pic-cap is equal to 0, 2158 the decoder is able to ignore redundant codec pictures 2159 provided that the decoder supports such a profile 2160 (Baseline, Extended) in which redundant coded pictures 2161 are allowed. 2163 Informative note: Even if redundant-pic-cap is equal to 1, 2164 the receiver may also choose other error concealment 2165 strategies to replace or complement decoding of redundant 2166 slices. 2168 sprop-parameter-sets: 2169 This parameter MAY be used to convey any sequence and picture 2170 parameter set NAL units (herein referred to as the initial 2171 parameter set NAL units) that can be placed in the NAL unit 2172 stream to precede any other NAL units in decoding order. The 2173 parameter MUST NOT be used to indicate codec capability in 2174 any capability exchange procedure. The value of the 2175 parameter is a comma (',') separated list of base64 [7] 2176 representations of parameter set NAL units as specified in 2177 sections 7.3.2.1 and 7.3.2.2 of [1]. Note that the number of 2178 bytes in a parameter set NAL unit is typically less than 10, 2179 but a picture parameter set NAL unit can contain several 2180 hundreds of bytes. 2182 Informative note: When several payload types are offered 2183 in the SDP Offer/Answer model, each with its own sprop- 2184 parameter-sets parameter, then the receiver cannot assume 2185 that those parameter sets do not use conflicting storage 2186 locations (i.e., identical values of parameter set 2187 identifiers). Therefore, a receiver should buffer all 2188 sprop-parameter-sets and make them available to the 2189 decoder instance that decodes a certain payload type. 2191 The "sprop-parameter-sets" parameter MUST only contain 2192 parameter sets that are conforming to the profile-level-id, 2193 i.e., the subset of coding tools indicated by any of the 2194 parameter sets MUST be equal to the default sub-profile, and 2195 the level indicated by any of the parameter sets MUST be 2196 equal to the default level. 2198 sprop-level-parameter-sets: 2199 This parameter MAY be used to convey any sequence and picture 2200 parameter set NAL units (herein referred to as the initial 2201 parameter set NAL units) that can be placed in the NAL unit 2202 stream to precede any other NAL units in decoding order and 2203 that are associated with one or more levels different than 2204 the default level. The parameter MUST NOT be used to 2205 indicate codec capability in any capability exchange 2206 procedure. 2208 The sprop-level-parameter-sets parameter contains parameter 2209 sets for one or more levels which are different than the 2210 default level. All parameter sets associated with one level 2211 are clustered and prefixed with a three-byte field which has 2212 the same syntax as profile-level-id. This enables the 2213 receiver to install the parameter sets for one level and 2214 discard the rest. The three-byte field is named PLId, and 2215 all parameter sets associated with one level are named PSL, 2216 which has the same syntax as sprop-parameter-sets. Parameter 2217 sets for each level are represented in the form of PLId:PSL, 2218 i.e., PLId followed by a colon (':') and the base64 [7] 2219 representation of the initial parameter set NAL units for the 2220 level. Each pair of PLId:PSL is also separated by a colon. 2221 Note that a PSL can contain multiple parameter sets for that 2222 level, separated with commas (','). 2224 The subset of coding tools indicated by each PLId field MUST 2225 be equal to the default sub-profile, and the level indicated 2226 by each PLId field MUST be different than the default level. 2227 All sequence parameter sets contained in each PSL MUST have 2228 the three bytes from profile_idc to level_idc, inclusive, 2229 equal to the preceding PLId. 2231 Informative note: This parameter allows for efficient 2232 level downgrade or upgrade in SDP Offer/Answer and out- 2233 of-band transport of parameter sets, simultaneously. 2235 use-level-src-parameter-sets: 2236 This parameter MAY be used to indicate a receiver capability. 2237 The value MAY be equal to either 0 or 1. When the parameter 2238 is not present, the value MUST be inferred to be equal to 0. 2239 The value 0 indicates that the receiver does not understand 2240 the sprop-level-parameter-sets parameter, and does not 2241 understand the "fmtp" source attribute as specified in 2242 section 6.3 of [9], and will ignore sprop-level-parameter- 2243 sets when present, and will ignore sprop-parameter-sets when 2244 conveyed using the "fmtp" source attribute. The value 1 2245 indicates that the receiver understands the sprop-level- 2246 parameter-sets parameter, and understands the "fmtp" source 2247 attribute as specified in section 6.3 of [9], and is capable 2248 of using parameter sets contained in the sprop-level- 2249 parameter-sets or contained in the sprop-parameter-sets that 2250 is conveyed using the "fmtp" source attribute. 2252 Informative note: An RFC 3984 receiver does not 2253 understand sprop-level-parameter-sets, use-level-src- 2254 parameter-sets, or the "fmtp" source attribute as 2255 specified in section 6.3 of [9]. Therefore, during SDP 2256 Offer/Answer, an RFC 3984 receiver as the answerer will 2257 simply ignore sprop-level-parameter-sets, when present in 2258 an offer, and sprop-parameter-sets conveyed using the 2259 "fmtp" source attribute as specified in section 6.3 of 2260 [9]. Assume that the offered payload type was accepted 2261 at a level lower than the default level. If the offered 2262 payload type included sprop-level-parameter-sets or 2263 included sprop-parameter-sets conveyed using the "fmtp" 2264 source attribute, and the offerer sees that the answerer 2265 has not included use-level-src-parameter-sets equal to 1 2266 in the answer, the offerer knows that in-band transport 2267 of parameter sets is needed. 2269 in-band-parameter-sets: 2270 This parameter MAY be used to indicate a receiver capability. 2271 The value MAY be equal to either 0 or 1. The value 1 2272 indicates that the receiver discards out-of-band parameter 2273 sets in sprop-parameter-sets and sprop-level-parameter-sets, 2274 therefore the sender MUST transmit all parameter sets in-band. 2275 The value 0 indicates that the receiver utilizes out-of-band 2276 parameter sets included in sprop-parameter-sets and/or sprop- 2277 level-parameter-sets. However, in this case, the sender MAY 2278 still choose to send parameter sets in-band. When in-band- 2279 parameter-sets is equal to 1, use-level-src-parameter-sets 2280 MUST NOT be present or MUST be equal to 0. When the 2281 parameter is not present, this receiver capability is not 2282 specified, and therefore the sender MAY send out-of-band 2283 parameter sets only, or it MAY send in-band-parameter-sets 2284 only, or it MAY send both. 2286 level-asymmetry-allowed: 2287 This parameter MAY be used in SDP Offer/Answer to indicate 2288 whether level asymmetry, i.e., using a different level in the 2289 offerer-to-answerer direction than the level in the answerer- 2290 to-offerer direction, is allowed. The value MAY be equal to 2291 either 0 or 1. When the parameter is not present, the value 2292 MUST be inferred to be equal to 0. The value 1 in both the 2293 offer and the answer indicates that level asymmetry is 2294 allowed. The value of 0 in either the offer or the answer 2295 indicates the level asymmetry is not allowed. 2297 If "level-asymmetry-allowed" is equal to 0 (or not present) 2298 in either the offer or the answer, level asymmetry is not 2299 allowed. In this case, the level to use in the direction 2300 from the offerer to the answerer MUST be the same as the 2301 level to use in the opposite direction. 2303 packetization-mode: 2304 This parameter signals the properties of an RTP payload type 2305 or the capabilities of a receiver implementation. Only a 2306 single configuration point can be indicated; thus, when 2307 capabilities to support more than one packetization-mode are 2308 declared, multiple configuration points (RTP payload types) 2309 must be used. 2311 When the value of packetization-mode is equal to 0 or 2312 packetization-mode is not present, the single NAL mode, as 2313 defined in section 6.2 of RFC 3984, MUST be used. This mode 2314 is in use in standards using ITU-T Recommendation H.241 [3] 2315 (see section 12.1). When the value of packetization-mode is 2316 equal to 1, the non-interleaved mode, as defined in section 2317 6.3 of RFC 3984, MUST be used. When the value of 2318 packetization-mode is equal to 2, the interleaved mode, as 2319 defined in section 6.4 of RFC 3984, MUST be used. The value 2320 of packetization-mode MUST be an integer in the range of 0 to 2321 2, inclusive. 2323 sprop-interleaving-depth: 2324 This parameter MUST NOT be present when packetization-mode is 2325 not present or the value of packetization-mode is equal to 0 2326 or 1. This parameter MUST be present when the value of 2327 packetization-mode is equal to 2. 2329 This parameter signals the properties of an RTP packet stream. 2330 It specifies the maximum number of VCL NAL units that precede 2331 any VCL NAL unit in the RTP packet stream in transmission 2332 order and follow the VCL NAL unit in decoding order. 2333 Consequently, it is guaranteed that receivers can reconstruct 2334 NAL unit decoding order when the buffer size for NAL unit 2335 decoding order recovery is at least the value of sprop- 2336 interleaving-depth + 1 in terms of VCL NAL units. 2338 The value of sprop-interleaving-depth MUST be an integer in 2339 the range of 0 to 32767, inclusive. 2341 sprop-deint-buf-req: 2342 This parameter MUST NOT be present when packetization-mode is 2343 not present or the value of packetization-mode is equal to 0 2344 or 1. It MUST be present when the value of packetization- 2345 mode is equal to 2. 2347 sprop-deint-buf-req signals the required size of the de- 2348 interleaving buffer for the RTP packet stream. The value of 2349 the parameter MUST be greater than or equal to the maximum 2350 buffer occupancy (in units of bytes) required in such a de- 2351 interleaving buffer that is specified in section 7.2 of RFC 2352 3984. It is guaranteed that receivers can perform the de- 2353 interleaving of interleaved NAL units into NAL unit decoding 2354 order, when the de-interleaving buffer size is at least the 2355 value of sprop-deint-buf-req in terms of bytes. 2357 The value of sprop-deint-buf-req MUST be an integer in the 2358 range of 0 to 4294967295, inclusive. 2360 Informative note: sprop-deint-buf-req indicates the 2361 required size of the de-interleaving buffer only. When 2362 network jitter can occur, an appropriately sized jitter 2363 buffer has to be provisioned for as well. 2365 deint-buf-cap: 2366 This parameter signals the capabilities of a receiver 2367 implementation and indicates the amount of de-interleaving 2368 buffer space in units of bytes that the receiver has 2369 available for reconstructing the NAL unit decoding order. A 2370 receiver is able to handle any stream for which the value of 2371 the sprop-deint-buf-req parameter is smaller than or equal to 2372 this parameter. 2374 If the parameter is not present, then a value of 0 MUST be 2375 used for deint-buf-cap. The value of deint-buf-cap MUST be 2376 an integer in the range of 0 to 4294967295, inclusive. 2378 Informative note: deint-buf-cap indicates the maximum 2379 possible size of the de-interleaving buffer of the 2380 receiver only. When network jitter can occur, an 2381 appropriately sized jitter buffer has to be provisioned 2382 for as well. 2384 sprop-init-buf-time: 2385 This parameter MAY be used to signal the properties of an RTP 2386 packet stream. The parameter MUST NOT be present, if the 2387 value of packetization-mode is equal to 0 or 1. 2389 The parameter signals the initial buffering time that a 2390 receiver MUST wait before starting decoding to recover the 2391 NAL unit decoding order from the transmission order. The 2392 parameter is the maximum value of (decoding time of the NAL 2393 unit - transmission time of a NAL unit), assuming reliable 2394 and instantaneous transmission, the same timeline for 2395 transmission and decoding, and that decoding starts when the 2396 first packet arrives. 2398 An example of specifying the value of sprop-init-buf-time 2399 follows. A NAL unit stream is sent in the following 2400 interleaved order, in which the value corresponds to the 2401 decoding time and the transmission order is from left to 2402 right: 2404 0 2 1 3 5 4 6 8 7 ... 2406 Assuming a steady transmission rate of NAL units, the 2407 transmission times are: 2409 0 1 2 3 4 5 6 7 8 ... 2411 Subtracting the decoding time from the transmission time 2412 column-wise results in the following series: 2414 0 -1 1 0 -1 1 0 -1 1 ... 2416 Thus, in terms of intervals of NAL unit transmission times, 2417 the value of sprop-init-buf-time in this example is 1. The 2418 parameter is coded as a non-negative base10 integer 2419 representation in clock ticks of a 90-kHz clock. If the 2420 parameter is not present, then no initial buffering time 2421 value is defined. Otherwise the value of sprop-init-buf-time 2422 MUST be an integer in the range of 0 to 4294967295, inclusive. 2424 In addition to the signaled sprop-init-buf-time, receivers 2425 SHOULD take into account the transmission delay jitter 2426 buffering, including buffering for the delay jitter caused by 2427 mixers, translators, gateways, proxies, traffic-shapers, and 2428 other network elements. 2430 sprop-max-don-diff: 2431 This parameter MAY be used to signal the properties of an RTP 2432 packet stream. It MUST NOT be used to signal transmitter or 2433 receiver or codec capabilities. The parameter MUST NOT be 2434 present if the value of packetization-mode is equal to 0 or 1. 2435 sprop-max-don-diff is an integer in the range of 0 to 32767, 2436 inclusive. If sprop-max-don-diff is not present, the value 2437 of the parameter is unspecified. sprop-max-don-diff is 2438 calculated as follows: 2440 sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)}, 2441 for any i and any j>i, 2443 where i and j indicate the index of the NAL unit in the 2444 transmission order and AbsDON denotes a decoding order number 2445 of the NAL unit that does not wrap around to 0 after 65535. 2446 In other words, AbsDON is calculated as follows: Let m and n 2447 be consecutive NAL units in transmission order. For the very 2448 first NAL unit in transmission order (whose index is 0), 2449 AbsDON(0) = DON(0). For other NAL units, AbsDON is 2450 calculated as follows: 2452 If DON(m) == DON(n), AbsDON(n) = AbsDON(m) 2454 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 2455 AbsDON(n) = AbsDON(m) + DON(n) - DON(m) 2457 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 2458 AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n) 2460 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 2461 AbsDON(n) = AbsDON(m) - (DON(m) + 65536 - DON(n)) 2463 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 2464 AbsDON(n) = AbsDON(m) - (DON(m) - DON(n)) 2466 where DON(i) is the decoding order number of the NAL unit 2467 having index i in the transmission order. The decoding order 2468 number is specified in section 5.5 of RFC 3984. 2470 Informative note: Receivers may use sprop-max-don-diff to 2471 trigger which NAL units in the receiver buffer can be 2472 passed to the decoder. 2474 max-rcmd-nalu-size: 2475 This parameter MAY be used to signal the capabilities of a 2476 receiver. The parameter MUST NOT be used for any other 2477 purposes. The value of the parameter indicates the largest 2478 NALU size in bytes that the receiver can handle efficiently. 2479 The parameter value is a recommendation, not a strict upper 2480 boundary. The sender MAY create larger NALUs but must be 2481 aware that the handling of these may come at a higher cost 2482 than NALUs conforming to the limitation. 2484 The value of max-rcmd-nalu-size MUST be an integer in the 2485 range of 0 to 4294967295, inclusive. If this parameter is 2486 not specified, no known limitation to the NALU size exists. 2487 Senders still have to consider the MTU size available between 2488 the sender and the receiver and SHOULD run MTU discovery for 2489 this purpose. 2491 This parameter is motivated by, for example, an IP to H.223 2492 video telephony gateway, where NALUs smaller than the H.223 2493 transport data unit will be more efficient. A gateway may 2494 terminate IP; thus, MTU discovery will normally not work 2495 beyond the gateway. 2497 Informative note: Setting this parameter to a lower than 2498 necessary value may have a negative impact. 2500 sar-understood: 2501 This parameter MAY be used to indicate a receiver capability 2502 and not anything else. The parameter indicates the maximum 2503 value of aspect_ratio_idc (specified in [1]) smaller than 255 2504 that the receiver understands. Table E-1 of [1] specifies 2505 aspect_ratio_idc equal to 0 as "unspecified", 1 to 16, 2506 inclusive, as specific Sample Aspect Ratios (SARs), 17 to 254, 2507 inclusive, as "reserved", and 255 as the Extended SAR, for 2508 which SAR width and SAR height are explicitly signaled. 2509 Therefore, a receiver with a decoder according to [1] 2510 understands aspect_ratio_idc in the range of 1 to 16, 2511 inclusive and aspect_ratio_idc equal to 255, in the sense 2512 that the receiver knows what exactly the SAR is. For such a 2513 receiver, the value of sar-understood is 16. If in the 2514 future Table E-1 of [1] is extended, e.g., such that the SAR 2515 for aspect_ratio_idc equal to 17 is specified, then for a 2516 receiver with a decoder that understands the extension, the 2517 value of sar-understood is 17. For a receiver with a decoder 2518 according to the 2003 version of [1], the value of sar- 2519 understood is 13, as the minimum reserved aspect_ratio_idc 2520 therein is 14. 2522 When sar-understood is not present, the value MUST be 2523 inferred to be equal to 13. 2525 sar-supported: 2526 This parameter MAY be used to indicate a receiver capability 2527 and not anything else. The value of this parameter is an 2528 integer in the range of 1 to sar-understood, inclusive, equal 2529 to 255. The value of sar-supported equal to N smaller than 2530 255 indicates that the reciever supports all the SARs 2531 corresponding to H.264 aspect_ratio_idc values (see Table E-1 2532 of [1]) in the range from 1 to N, inclusive, without 2533 geometric distortion. The value of sar-supported equal to 2534 255 indicates that the receiver supports all sample aspect 2535 ratios which are expressible using two 16-bit integer values 2536 as the numerator and denominator, i.e., those that are 2537 expressible using the H.264 aspect_ratio_idc value of 255 2538 (Extended_SAR, see Table E-1 of [1]), without geometric 2539 distortion. 2541 H.264 compliant encoders SHOULD NOT send an aspect_ratio_idc 2542 equal to 0, or an aspect_ratio_idc larger than sar-understood 2543 and smaller than 255. H.264 compliant encoders SHOULD send 2544 an aspect_ratio_idc that the receiver is able to display 2545 without geometrical distortion. However, H.264 compliant 2546 encoders MAY choose to send pictures using any SAR. 2548 Note that the actual sample aspect ratio or extended sample 2549 aspect ratio, when present, of the stream is conveyed in the 2550 Video Usability Information (VUI) part of the sequence 2551 parameter set. 2553 Encoding considerations: 2554 This type is only defined for transfer via RTP (RFC 3550). 2556 Security considerations: 2557 See section 9 of RFC xxxx. 2559 Public specification: 2560 Please refer to RFC xxxx and its section 15. 2562 Additional information: 2563 None 2565 File extensions: none 2567 Macintosh file type code: none 2569 Object identifier or OID: none 2571 Person & email address to contact for further information: 2572 Ye-Kui Wang, yekuiwang@huawei.com 2574 Intended usage: COMMON 2576 Author: 2577 Ye-Kui Wang, yekuiwang@huawei.com 2579 Change controller: 2580 IETF Audio/Video Transport working group delegated from the 2581 IESG. 2583 8.2. SDP Parameters 2585 8.2.1. Mapping of Payload Type Parameters to SDP 2587 The media type video/H264 string is mapped to fields in the Session 2588 Description Protocol (SDP) [6] as follows: 2590 o The media name in the "m=" line of SDP MUST be video. 2592 o The encoding name in the "a=rtpmap" line of SDP MUST be H264 2593 (the media subtype). 2595 o The clock rate in the "a=rtpmap" line MUST be 90000. 2597 o The OPTIONAL parameters "profile-level-id", "max-recv-level", 2598 "max-mbps", "max-smbps", "max-fs", "max-cpb", "max-dpb", "max- 2599 br", "redundant-pic-cap", "use-level-src-parameter-sets", "in- 2600 band-parameter-sets", "level-asymmetry-allowed", "packetization- 2601 mode", "sprop-interleaving-depth", "sprop-deint-buf-req", 2602 "deint-buf-cap", "sprop-init-buf-time", "sprop-max-don-diff", 2603 "max-rcmd-nalu-size", "sar-understood", and "sar-supported", 2604 when present, MUST be included in the "a=fmtp" line of SDP. 2605 These parameters are expressed as a media type string, in the 2606 form of a semicolon separated list of parameter=value pairs. 2608 o The OPTIONAL parameters "sprop-parameter-sets" and "sprop-level- 2609 parameter-sets", when present, MUST be included in the "a=fmtp" 2610 line of SDP or conveyed using the "fmtp" source attribute as 2611 specified in section 6.3 of [9]. For a particular media format 2612 (i.e., RTP payload type), a "sprop-parameter-sets" or "sprop- 2613 level-parameter-sets" MUST NOT be both included in the "a=fmtp" 2614 line of SDP and conveyed using the "fmtp" source attribute. 2615 When included in the "a=fmtp" line of SDP, these parameters are 2616 expressed as a media type string, in the form of a semicolon 2617 separated list of parameter=value pairs. When conveyed using 2618 the "fmtp" source attribute, these parameters are only 2619 associated with the given source and payload type as parts of 2620 the "fmtp" source attribute. 2622 Informative note: Conveyance of "sprop-parameter-sets" and 2623 "sprop-level-parameter-sets" using the "fmtp" source 2624 attribute allows for out-of-band transport of parameter sets 2625 in topologies like Topo-Video-switch-MCU [29]. 2627 An example of media representation in SDP is as follows (Baseline 2628 Profile, Level 3.0, some of the constraints of the Main profile may 2629 not be obeyed): 2631 m=video 49170 RTP/AVP 98 2632 a=rtpmap:98 H264/90000 2633 a=fmtp:98 profile-level-id=42A01E; 2634 packetization-mode=1; 2635 sprop-parameter-sets= 2637 8.2.2. Usage with the SDP Offer/Answer Model 2639 When H.264 is offered over RTP using SDP in an Offer/Answer model 2640 [8] for negotiation for unicast usage, the following limitations 2641 and rules apply: 2643 o The parameters identifying a media format configuration for 2644 H.264 are "profile-level-id" and "packetization-mode". These 2645 media format configuration parameters (except for the level part 2646 of "profile-level-id") MUST be used symmetrically; i.e., the 2647 answerer MUST either maintain all configuration parameters or 2648 remove the media format (payload type) completely, if one or 2649 more of the parameter values are not supported. Note that the 2650 level part of "profile-level-id" includes level_idc, and, for 2651 indication of level 1b when profile_idc is equal to 66, 77 or 88, 2652 bit 4 (constraint_set3_flag) of profile-iop. The level part of 2653 "profile-level-id" is changeable. 2655 Informative note: The requirement for symmetric use does not 2656 apply for the level part of "profile-level-id", and does not 2657 apply for the other stream properties and capability 2658 parameters. 2660 Informative note: In H.264 [1], all the levels except for 2661 level 1b are equal to the value of level_idc divided by 10. 2662 Level 1b is a level higher than level 1.0 but lower than 2663 level 1.1, and is signaled in an ad-hoc manner, due to that 2664 the level was specified after level 1.0 and level 1.1. For 2665 the Baseline, Main and Extended profiles (with profile_idc 2666 equal to 66, 77 and 88, respectively), level 1b is indicated 2667 by level_idc equal to 11 (i.e. same as level 1.1) and 2668 constraint_set3_flag equal to 1. For other profiles, level 2669 1b is indicated by level_idc equal to 9 (but note that level 2670 1b for these profiles are still higher than level 1, which 2671 has level_idc equal to 10, and lower than level 1.1). In SDP 2672 Offer/Answer, an answer to an offer may indicate a level 2673 equal to or lower than the level indicated in the offer. Due 2674 to the ad-hoc indication of level 1b, offerers and answerers 2675 must check the value of bit 4 (constraint_set3_flag) of the 2676 middle octet of the parameter "profile-level-id", when 2677 profile_idc is equal to 66, 77 or 88 and level_idc is equal 2678 to 11. 2680 To simplify handling and matching of these configurations, the 2681 same RTP payload type number used in the offer SHOULD also be 2682 used in the answer, as specified in [8]. An answer MUST NOT 2683 contain a payload type number used in the offer unless the 2684 configuration is exactly the same as in the offer. 2686 Informative note: When an offerer receives an answer, it has 2687 to compare payload types not declared in the offer based on 2688 the media type (i.e., video/H264) and the above media 2689 configuration parameters with any payload types it has 2690 already declared. This will enable it to determine whether 2691 the configuration in question is new or if it is equivalent 2692 to configuration already offered, since a different payload 2693 type number may be used in the answer. 2695 o The parameter "max-recv-level", when present, declares the 2696 highest level supported for receiving. In case "max-recv-level" 2697 is not present, the highest level supported for receiving is 2698 equal to the default level indicated by the level part of 2699 "profile-level-id". "max-recv-level" MUST be higher than the 2700 default level. 2702 o The parameter "level-asymmetry-allowed" indicates whether level 2703 asymmetry is allowed. 2705 If "level-asymmetry-allowed" is equal to 0 (or not present) in 2706 either the offer or the answer, level asymmetry is not allowed. 2707 In this case, the level to use in the direction from the offerer 2708 to the answerer MUST be the same as the level to use in the 2709 opposite direction, and the common level to use is equal to the 2710 lower value of the default level in the offer and the default 2711 level in the answer. 2713 Otherwise ("level-asymmetry-allowed" equals to 1 in both the 2714 offer and the answer), level asymmetry is allowed. In this case, 2715 the level to use in the offerer-to-answerer direction MUST be 2716 equal to the highest level the answerer supports for receiving, 2717 and the level to use in the answerer-to-offerer direction MUST 2718 be equal to the highest level the offerer supports for receiving. 2720 When level asymmetry is not allowed, level upgrade is not 2721 allowed, i.e. the default level in the answer MUST be equal to 2722 or lower than the default level in the offer. 2724 o The parameters "sprop-deint-buf-req", "sprop-interleaving-depth", 2725 "sprop-max-don-diff", and "sprop-init-buf-time" describe the 2726 properties of the RTP packet stream that the offerer or answerer 2727 is sending for the media format configuration. This differs 2728 from the normal usage of the Offer/Answer parameters: normally 2729 such parameters declare the properties of the stream that the 2730 offerer or the answerer is able to receive. When dealing with 2731 H.264, the offerer assumes that the answerer will be able to 2732 receive media encoded using the configuration being offered. 2734 Informative note: The above parameters apply for any stream 2735 sent by the declaring entity with the same configuration; 2736 i.e., they are dependent on their source. Rather than being 2737 bound to the payload type, the values may have to be applied 2738 to another payload type when being sent, as they apply for 2739 the configuration. 2741 o The capability parameters "max-mbps", "max-smbps", "max-fs", 2742 "max-cpb", "max-dpb", "max-br", ,"redundant-pic-cap", "max-rcmd- 2743 nalu-size", "sar-understood", "sar-supported" MAY be used to 2744 declare further capabilities of the offerer or answerer for 2745 receiving. These parameters MUST NOT be present when the 2746 direction attribute is sendonly, and the parameters describe the 2747 limitations of what the offerer or answerer accepts for 2748 receiving streams. 2750 o An offerer has to include the size of the de-interleaving buffer, 2751 "sprop-deint-buf-req", in the offer for an interleaved H.264 2752 stream. To enable the offerer and answerer to inform each other 2753 about their capabilities for de-interleaving buffering in 2754 receiving streams, both parties are RECOMMENDED to include 2755 "deint-buf-cap". For interleaved streams, it is also 2756 RECOMMENDED to consider offering multiple payload types with 2757 different buffering requirements when the capabilities of the 2758 receiver are unknown. 2760 o The "sprop-parameter-sets" or "sprop-level-parameter-sets" 2761 parameter, when present (included in the "a=fmtp" line of SDP or 2762 conveyed using the "fmtp" source attribute as specified in 2763 section 6.3 of [9]), is used for out-of-band transport of 2764 parameter sets. However, when out-of-band transport of 2765 parameter sets is used, parameter sets MAY still be additionally 2766 transported in-band. 2768 The answerer MAY use either out-of-band or in-band transport of 2769 parameter sets for the stream it is sending, regardless of 2770 whether out-of-band parameter sets transport has been used in 2771 the offerer-to-answerer direction. Parameter sets included in 2772 an answer are independent of those parameter sets included in 2773 the offer, as they are used for decoding two different video 2774 streams, one from the answerer to the offerer, and the other in 2775 the opposite direction. 2777 The following rules apply to transport of parameter sets in the 2778 offerer-to-answerer direction. 2780 o An offer MAY include either or both of "sprop-parameter- 2781 sets" and "sprop-level-parameter-sets". If neither "sprop- 2782 parameter-sets" nor "sprop-level-parameter-sets" is present 2783 in the offer, then only in-band transport of parameter sets 2784 is used. 2786 o If the answer includes "in-band-parameter-sets" equal to 1, 2787 then the offerer MUST transmit parameter sets in-band. 2788 Otherwise, the following applies. 2790 o If the level to use in the offerer-to-answerer 2791 direction is equal to the default level in the offer, 2792 the following applies. 2794 When there is a "sprop-parameter-sets" included 2795 in the "a=fmtp" line in the offer, the answerer 2796 MUST be prepared to use the parameter sets 2797 included in the "sprop-parameter-sets" for 2798 decoding the incoming NAL unit stream. 2800 When there is a "sprop-parameter-sets" conveyed 2801 using the "fmtp" source attribute in the offer, 2802 the following applies. If the answer includes 2803 "use-level-src-parameter-sets" equal to 1 or the 2804 "fmtp" source attribute, the answerer MUST be 2805 prepared to use the parameter sets included in 2806 the "sprop-parameter-sets" for decoding the 2807 incoming NAL unit stream; Otherwise, the offerer 2808 MUST transmit parameter sets in-band. 2810 When "sprop-parameter-sets" is not present in the 2811 offer, the offerer MUST transmit parameter sets 2812 in-band. 2814 The answerer MUST ignore "sprop-level-parameter- 2815 sets", when present (either included in the 2816 "a=fmtp" line or conveyed using the "fmtp" source 2817 attribute) in the offer. 2819 o Otherwise (the level to use in the offerer-to-answerer 2820 direction is not equal to the default level in the 2821 offer, the following applies. 2823 The answerer MUST ignore "sprop-parameter-sets", 2824 when present (either included in the "a=fmtp" 2825 line or conveyed using the "fmtp" source 2826 attribute) in the offer. 2828 When neither "use-level-src-parameter-sets" equal 2829 to 1 nor the "fmtp" source attribute is present 2830 in the answer, the answerer MUST ignore "sprop- 2831 level-parameter-sets", when present in the offer, 2832 and the offerer MUST transmit parameter sets in- 2833 band. 2835 When either "use-level-src-parameter-sets" equal 2836 to 1 or the "fmtp" source attribute is present in 2837 the answer, the answerer MUST be prepared to use 2838 the parameter sets that are included in "sprop- 2839 level-parameter-sets" for the accepted level (i.e. 2840 the default level in the answer), when present in 2841 the offer, for decoding the incoming NAL unit 2842 stream, and ignore all other parameter sets 2843 included in "sprop-level-parameter-sets". 2845 When no parameter sets for the level to use in 2846 the offerer-to-answerer direction are present in 2847 "sprop-level-parameter-sets" in the offer, the 2848 offerer MUST transmit parameter sets in-band. 2850 The following rules apply to transport of parameter sets in the 2851 answerer-to-offerer direction. 2853 o An answer MAY include either "sprop-parameter-sets" or 2854 "sprop-level-parameter-sets", but MUST NOT include both of 2855 the two. If neither "sprop-parameter-sets" nor "sprop- 2856 level-parameter-sets" is present in the answer, then only 2857 in-band transport of parameter sets is used. 2859 o If the offer includes "in-band-parameter-sets" equal to 1, 2860 the answerer MUST NOT include "sprop-parameter-sets" or 2861 "sprop-level-parameter-sets" in the answer and MUST 2862 transmit parameter sets in-band. Otherwise, the following 2863 applies. 2865 o If the level to use in the answerer-to-offerer 2866 direction is equal to the default level in the answer, 2867 the following applies. 2869 When there is a "sprop-parameter-sets" included 2870 in the "a=fmtp" line in the answer, the offerer 2871 MUST be prepared to use the parameter sets 2872 included in the "sprop-parameter-sets" for 2873 decoding the incoming NAL unit stream. 2875 When there is a "sprop-parameter-sets" conveyed 2876 using the "fmtp" source attribute in the answer, 2877 the following applies. If the offer includes 2878 "use-level-src-parameter-sets" equal to 1 or the 2879 "fmtp" source attribute, the offerer MUST be 2880 prepared to use the parameter sets included in 2881 the "sprop-parameter-sets" for decoding the 2882 incoming NAL unit stream; Otherwise, the 2883 answerer MUST transmit parameter sets in-band. 2885 When "sprop-parameter-sets" is not present in the 2886 answer, the answerer MUST transmit parameter sets 2887 in-band. 2889 The offerer MUST ignore "sprop-level-parameter- 2890 sets", when present (either included in the 2891 "a=fmtp" line or conveyed using the "fmtp" source 2892 attribute) in the answer. 2894 o Otherwise (the level to use in the answerer-to-offerer 2895 direction is not equal to the default level in the 2896 answer, the following applies. 2898 The offerer MUST ignore "sprop-parameter-sets", 2899 when present (either included in the "a=fmtp" 2900 line of SDP or conveyed using the "fmtp" source 2901 attribute) in the answer. 2903 When neither "use-level-src-parameter-sets" equal 2904 to 1 nor the "fmtp" source attribute is present 2905 in the offer, the offerer MUST ignore "sprop- 2906 level-parameter-sets", when present, and the 2907 answerer MUST transmit parameter sets in-band. 2909 When either "use-level-src-parameter-sets" equal 2910 to 1 or the "fmtp" source attribute is present in 2911 the offer, the offerer MUST be prepared to use 2912 the parameter sets that are included in "sprop- 2913 level-parameter-sets" for the level to use in the 2914 answerer-to-offerer direction, when present in 2915 the answer, for decoding the incoming NAL unit 2916 stream, and ignore all other parameter sets 2917 included in "sprop-level-parameter-sets" in the 2918 answer. 2920 When no parameter sets for the level to use in 2921 the answerer-to-offerer direction are present in 2922 "sprop-level-parameter-sets" in the answer, the 2923 answerer MUST transmit parameter sets in-band. 2925 When "sprop-parameter-sets" or "sprop-level-parameter-sets" is 2926 conveyed using the "fmtp" source attribute in as specified in 2927 section 6.3 of [9], the receiver of the parameters MUST store 2928 the parameter sets included in the "sprop-parameter-sets" or 2929 "sprop-level-parameter-sets" for the accepted level and 2930 associate them to the source given as a part of the "fmtp" 2931 source attribute. Parameter sets associated with one source 2932 MUST only be used to decode NAL units conveyed in RTP packets 2933 from the same source. When this mechanism is in use, SSRC 2934 collision detection and resolution MUST be performed as 2935 specified in [9]. 2937 Informative note: Conveyance of "sprop-parameter-sets" and 2938 "sprop-level-parameter-sets" using the "fmtp" source 2939 attribute may be used in topologies like Topo-Video-switch- 2940 MCU [29] to enable out-of-band transport of parameter sets. 2942 For streams being delivered over multicast, the following rules 2943 apply: 2945 o The media format configuration is identified by "profile-level- 2946 id", including the level part, and "packetization-mode". These 2947 media format configuration parameters (including the level part 2948 of "profile-level-id") MUST be used symmetrically; i.e., the 2949 answerer MUST either maintain all configuration parameters or 2950 remove the media format (payload type) completely. Note that 2951 this implies that the level part of "profile-level-id" for 2952 Offer/Answer in multicast is not changeable. 2954 To simplify handling and matching of these configurations, the 2955 same RTP payload type number used in the offer SHOULD also be 2956 used in the answer, as specified in [8]. An answer MUST NOT 2957 contain a payload type number used in the offer unless the 2958 configuration is the same as in the offer. 2960 o Parameter sets received MUST be associated with the originating 2961 source, and MUST be only used in decoding the incoming NAL unit 2962 stream from the same source. 2964 o The rules for other parameters are the same as above for unicast 2965 as long as the above rules are obeyed. 2967 Table 6 lists the interpretation of all the media type parameters 2968 that MUST be used for the different direction attributes. 2970 Table 6. Interpretation of parameters for different direction 2971 attributes. 2973 sendonly --+ 2974 recvonly --+ | 2975 sendrecv --+ | | 2976 | | | 2977 profile-level-id C C P 2978 max-recv-level R R - 2979 packetization-mode C C P 2980 sprop-deint-buf-req P - P 2981 sprop-interleaving-depth P - P 2982 sprop-max-don-diff P - P 2983 sprop-init-buf-time P - P 2984 max-mbps R R - 2985 max-smbps R R - 2986 max-fs R R - 2987 max-cpb R R - 2988 max-dpb R R - 2989 max-br R R - 2990 redundant-pic-cap R R - 2991 deint-buf-cap R R - 2992 max-rcmd-nalu-size R R - 2993 sar-understood R R - 2994 sar-supported R R - 2995 in-band-parameter-sets R R - 2996 use-level-src-parameter-sets R R - 2997 level-asymmetry-allowed O - - 2998 sprop-parameter-sets S - S 2999 sprop-level-parameter-sets S - S 3001 Legend: 3003 C: configuration for sending and receiving streams 3004 O: offer/answer mode 3005 P: properties of the stream to be sent 3006 R: receiver capabilities 3007 S: out-of-band parameter sets 3008 -: not usable, when present SHOULD be ignored 3010 Parameters used for declaring receiver capabilities are in general 3011 downgradable; i.e., they express the upper limit for a sender's 3012 possible behavior. Thus a sender MAY select to set its encoder 3013 using only lower/less or equal values of these parameters. 3015 Parameters declaring a configuration point are not changeable, with 3016 the exception of the level part of the "profile-level-id" parameter 3017 for unicast usage. This expresses values a receiver expects to be 3018 used and must be used verbatim on the sender side. 3020 When a sender's capabilities are declared, and non-downgradable 3021 parameters are used in this declaration, then these parameters 3022 express a configuration that is acceptable for the sender to 3023 receive streams. In order to achieve high interoperability levels, 3024 it is often advisable to offer multiple alternative configurations; 3025 e.g., for the packetization mode. It is impossible to offer 3026 multiple configurations in a single payload type. Thus, when 3027 multiple configuration offers are made, each offer requires its own 3028 RTP payload type associated with the offer. 3030 A receiver SHOULD understand all media type parameters, even if it 3031 only supports a subset of the payload format's functionality. This 3032 ensures that a receiver is capable of understanding when an offer 3033 to receive media can be downgraded to what is supported by the 3034 receiver of the offer. 3036 An answerer MAY extend the offer with additional media format 3037 configurations. However, to enable their usage, in most cases a 3038 second offer is required from the offerer to provide the stream 3039 property parameters that the media sender will use. This also has 3040 the effect that the offerer has to be able to receive this media 3041 format configuration, not only to send it. 3043 If an offerer wishes to have non-symmetric capabilities between 3044 sending and receiving, the offerer can allow asymmetric levels via 3045 "level-asymmetry-allowed" equal to 1. Alternatively, the offerer 3046 could offer different RTP sessions; i.e., different media lines 3047 declared as "recvonly" and "sendonly", respectively. This may have 3048 further implications on the system, and may require additional 3049 external semantics to associate the two media lines. 3051 8.2.3. Usage in Declarative Session Descriptions 3053 When H.264 over RTP is offered with SDP in a declarative style, as 3054 in RTSP [27] or SAP [28], the following considerations are 3055 necessary. 3057 o All parameters capable of indicating both stream properties and 3058 receiver capabilities are used to indicate only stream 3059 properties. For example, in this case, the parameter "profile- 3060 level-id" declares only the values used by the stream, not the 3061 capabilities for receiving streams. This results in that the 3062 following interpretation of the parameters MUST be used: 3064 Declaring actual configuration or stream properties: 3066 - profile-level-id 3067 - packetization-mode 3068 - sprop-interleaving-depth 3069 - sprop-deint-buf-req 3070 - sprop-max-don-diff 3071 - sprop-init-buf-time 3073 Out-of-band transporting of parameter sets: 3075 - sprop-parameter-sets 3076 - sprop-level-parameter-sets 3078 Not usable(when present, they SHOULD be ignored): 3080 - max-mbps 3081 - max-smbps 3082 - max-fs 3083 - max-cpb 3084 - max-dpb 3085 - max-br 3086 - max-recv-level 3087 - redundant-pic-cap 3088 - max-rcmd-nalu-size 3089 - deint-buf-cap 3090 - sar-understood 3091 - sar-supported 3092 - in-band-parameter-sets 3093 - level-asymmetry-allowed 3094 - use-level-src-parameter-sets 3096 o A receiver of the SDP is required to support all parameters and 3097 values of the parameters provided; otherwise, the receiver MUST 3098 reject (RTSP) or not participate in (SAP) the session. It falls 3099 on the creator of the session to use values that are expected to 3100 be supported by the receiving application. 3102 8.3. Examples 3104 An SDP Offer/Answer exchange wherein both parties are expected to 3105 both send and receive could look like the following. Only the 3106 media codec specific parts of the SDP are shown. Some lines are 3107 wrapped due to text constraints. 3109 Offerer -> Answerer SDP message: 3111 m=video 49170 RTP/AVP 100 99 98 3112 a=rtpmap:98 H264/90000 3113 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; 3114 sprop-parameter-sets= 3115 a=rtpmap:99 H264/90000 3116 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3117 sprop-parameter-sets= 3118 a=rtpmap:100 H264/90000 3119 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3120 sprop-parameter-sets=; 3121 sprop-interleaving-depth=45; sprop-deint-buf-req=64000; 3122 sprop-init-buf-time=102478; deint-buf-cap=128000 3124 The above offer presents the same codec configuration in three 3125 different packetization formats. PT 98 represents single NALU mode, 3126 PT 99 represents non-interleaved mode, and PT 100 indicates the 3127 interleaved mode. In the interleaved mode case, the interleaving 3128 parameters that the offerer would use if the answer indicates 3129 support for PT 100 are also included. In all three cases the 3130 parameter "sprop-parameter-sets" conveys the initial parameter sets 3131 that are required by the answerer when receiving a stream from the 3132 offerer when this configuration is accepted. Note that the value 3133 for "sprop-parameter-sets" could be different for each payload type. 3135 Answerer -> Offerer SDP message: 3137 m=video 49170 RTP/AVP 100 99 97 3138 a=rtpmap:97 H264/90000 3139 a=fmtp:97 profile-level-id=42A01E; packetization-mode=0; 3140 sprop-parameter-sets= 3141 a=rtpmap:99 H264/90000 3142 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3143 sprop-parameter-sets=; 3144 max-rcmd-nalu-size=3980 3145 a=rtpmap:100 H264/90000 3146 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3147 sprop-parameter-sets=; 3148 sprop-interleaving-depth=60; 3149 sprop-deint-buf-req=86000; sprop-init-buf-time=156320; 3150 deint-buf-cap=128000; max-rcmd-nalu-size=3980 3152 As the Offer/Answer negotiation covers both sending and receiving 3153 streams, an offer indicates the exact parameters for what the 3154 offerer is willing to receive, whereas the answer indicates the 3155 same for what the answerer accepts to receive. In this case the 3156 offerer declared that it is willing to receive payload type 98. 3157 The answerer accepts this by declaring an equivalent payload type 3158 97; i.e., it has identical values for the two parameters "profile- 3159 level-id" and "packetization-mode" (since "packetization-mode" is 3160 equal to 0, "sprop-deint-buf-req" is not present). As the offered 3161 payload type 98 is accepted, the answerer needs to store parameter 3162 sets included in sprop-parameter-sets= in 3163 case the offer finally decides to use this configuration. In the 3164 answer, the answerer includes the parameter sets in sprop- 3165 parameter-sets= that the answerer would use 3166 in the stream sent from the answerer if this configuration is 3167 finally used. 3169 The answerer also accepts the reception of the two configurations 3170 that payload types 99 and 100 represent. Again, the answerer needs 3171 to store parameter sets included in sprop-parameter-sets= and sprop-parameter-sets= in 3173 case the offer finally decides to use either of these two 3174 configurations. The answerer provides the initial parameter sets 3175 for the answerer-to-offerer direction, i.e. the parameter sets in 3176 sprop-parameter-sets= and sprop-parameter- 3177 sets=, for payload types 99 and 100, 3178 respectively, that it will use to send the payload types. The 3179 answerer also provides the offerer with its memory limit for de- 3180 interleaving operations by providing a "deint-buf-cap" parameter. 3181 This is only useful if the offerer decides on making a second offer, 3182 where it can take the new value into account. The "max-rcmd-nalu- 3183 size" indicates that the answerer can efficiently process NALUs up 3184 to the size of 3980 bytes. However, there is no guarantee that the 3185 network supports this size. 3187 In the following example, the offer is accepted without level 3188 downgrading (i.e. the default level, 3.0, is accepted), and both 3189 "sprop-parameter-sets" and "sprop-level-parameter-sets" are present 3190 in the offer. The answerer must ignore sprop-level-parameter- 3191 sets= and store parameter sets in sprop- 3192 parameter-sets= for decoding the incoming 3193 NAL unit stream. The offerer must store the parameter sets in 3194 sprop-parameter-sets= in the answer for 3195 decoding the incoming NAL unit stream. Note that in this example, 3196 parameter sets in sprop-parameter-sets= must 3197 be associated with level 3.0. 3199 Offer SDP: 3201 m=video 49170 RTP/AVP 98 3202 a=rtpmap:98 H264/90000 3203 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3204 packetization-mode=1; 3205 sprop-parameter-sets=; 3206 sprop-level-parameter-sets= 3208 Answer SDP: 3210 m=video 49170 RTP/AVP 98 3211 a=rtpmap:98 H264/90000 3212 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3213 packetization-mode=1; 3214 sprop-parameter-sets= 3216 In the following example, the offer (Baseline profile, level 1.1) 3217 is accepted with level downgrading (the accepted level is 1b), and 3218 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3219 present in the offer. The answerer must ignore sprop-parameter- 3220 sets= and all parameter sets not for the 3221 accepted level (level 1b) in sprop-level-parameter-sets=, and must store parameter sets for the accepted level 3223 (level 1b) in sprop-level-parameter-sets= 3224 for decoding the incoming NAL unit stream. The offerer must store 3225 the parameter sets in sprop-parameter-sets= 3226 in the answer for decoding the incoming NAL unit stream. Note that 3227 in this example, parameter sets in sprop-parameter-sets= must be associated with level 1b. 3230 Offer SDP: 3232 m=video 49170 RTP/AVP 98 3233 a=rtpmap:98 H264/90000 3234 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3235 packetization-mode=1; 3236 sprop-parameter-sets=; 3237 sprop-level-parameter-sets= 3239 Answer SDP: 3241 m=video 49170 RTP/AVP 98 3242 a=rtpmap:98 H264/90000 3243 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3244 packetization-mode=1; 3245 sprop-parameter-sets=; 3246 use-level-src-parameter-sets=1 3248 In the following example, the offer (Baseline profile, level 1.1) 3249 is accepted with level downgrading (the accepted level is 1b), and 3250 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3251 present in the offer. However, the answerer is a legacy RFC 3984 3252 implementation and does not understand "sprop-level-parameter-sets", 3253 hence it does not include "use-level-src-parameter-sets" (which the 3254 answerer does not understand, either) in the answer. Therefore, 3255 the answerer must ignore both sprop-parameter-sets= and sprop-level-parameter-sets=, and 3257 the offerer must transport parameter sets in-band. 3259 Offer SDP: 3261 m=video 49170 RTP/AVP 98 3262 a=rtpmap:98 H264/90000 3263 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3264 packetization-mode=1; 3265 sprop-parameter-sets=; 3266 sprop-level-parameter-sets= 3268 Answer SDP: 3270 m=video 49170 RTP/AVP 98 3271 a=rtpmap:98 H264/90000 3272 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3273 packetization-mode=1 3275 In the following example, the offer is accepted without level 3276 downgrading, and "sprop-parameter-sets" is present in the offer. 3277 Parameter sets in sprop-parameter-sets= must 3278 be stored and used used by the encoder of the offerer and the 3279 decoder of the answerer, and parameter sets in sprop-parameter- 3280 sets=must be used by the encoder of the 3281 answerer and the decoder of the offerer. Note that sprop- 3282 parameter-sets= is basically independent of 3283 sprop-parameter-sets=. 3285 Offer SDP: 3287 m=video 49170 RTP/AVP 98 3288 a=rtpmap:98 H264/90000 3289 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3290 packetization-mode=1; 3291 sprop-parameter-sets= 3293 Answer SDP: 3295 m=video 49170 RTP/AVP 98 3296 a=rtpmap:98 H264/90000 3297 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3298 packetization-mode=1; 3299 sprop-parameter-sets= 3301 In the following example, the offer is accepted without level 3302 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3303 parameter-sets" is present in the offer, meaning that there is no 3304 out-of-band transmission of parameter sets, which then have to be 3305 transported in-band. 3307 Offer SDP: 3309 m=video 49170 RTP/AVP 98 3310 a=rtpmap:98 H264/90000 3311 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3312 packetization-mode=1 3314 Answer SDP: 3316 m=video 49170 RTP/AVP 98 3317 a=rtpmap:98 H264/90000 3318 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3319 packetization-mode=1 3321 In the following example, the offer is accepted with level 3322 downgrading and "sprop-parameter-sets" is present in the offer. As 3323 sprop-parameter-sets= contains level_idc 3324 indicating Level 3.0, therefore cannot be used as the answerer 3325 wants Level 2.0 and must be ignored by the answerer, and in-band 3326 parameter sets must be used. 3328 Offer SDP: 3330 m=video 49170 RTP/AVP 98 3331 a=rtpmap:98 H264/90000 3332 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3333 packetization-mode=1; 3334 sprop-parameter-sets= 3336 Answer SDP: 3338 m=video 49170 RTP/AVP 98 3339 a=rtpmap:98 H264/90000 3340 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3341 packetization-mode=1 3343 In the following example, the offer is also accepted with level 3344 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3345 parameter-sets" is present in the offer, meaning that there is no 3346 out-of-band transmission of parameter sets, which then have to be 3347 transported in-band. 3349 Offer SDP: 3351 m=video 49170 RTP/AVP 98 3352 a=rtpmap:98 H264/90000 3353 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3354 packetization-mode=1 3356 Answer SDP: 3358 m=video 49170 RTP/AVP 98 3359 a=rtpmap:98 H264/90000 3360 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3361 packetization-mode=1 3363 In the following example, the offer is accepted with level 3364 upgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3365 parameter-sets" is present in the offer or the answer, meaning that 3366 there is no out-of-band transmission of parameter sets, which then 3367 have to be transported in-band. The level to use in the offerer- 3368 to-answerer direction is Level 3.0, and the level to use in the 3369 answerer-to-offerer direction is Level 2.0. The answerer is 3370 allowed to send at any level up to and including level 2.0, and the 3371 offerer is allowed to send at any level up to and including level 3372 3.0. 3374 Offer SDP: 3376 m=video 49170 RTP/AVP 98 3377 a=rtpmap:98 H264/90000 3378 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3379 packetization-mode=1; level-asymmetry-allowed=1 3381 Answer SDP: 3383 m=video 49170 RTP/AVP 98 3384 a=rtpmap:98 H264/90000 3385 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3386 packetization-mode=1; level-asymmetry-allowed=1 3388 In the following example, the offerer is a Multipoint Control Unit 3389 (MCU) in a Topo-Video-switch-MCU like topology [29], offering 3390 parameter sets received (using out-of-band transport) from three 3391 other participants B, C, and D, and receiving parameter sets from 3392 the participant A, which is the answerer. The participants are 3393 identified by their values of CNAME, which are mapped to different 3394 SSRC values. The same codec configuration is used by all the four 3395 participants. The participant A stores and associates the 3396 parameter sets included in , , and to participants B, C, and D, 3398 respectively, and uses for decoding NAL 3399 units carried in RTP packets originated from participant B only, 3400 uses for decoding NAL units carried in RTP 3401 packets originated from participant C only, and uses for decoding NAL units carried in RTP packets 3403 originated from participant D only. 3405 Offer SDP: 3407 m=video 49170 RTP/AVP 98 3408 a=ssrc:SSRC-B cname:CNAME-B 3409 a=ssrc:SSRC-C cname:CNAME-C 3410 a=ssrc:SSRC-D cname:CNAME-D 3411 a=ssrc:SSRC-B fmtp:98 3412 sprop-parameter-sets= 3413 a=ssrc:SSRC-C fmtp:98 3414 sprop-parameter-sets= 3415 a=ssrc:SSRC-D fmtp:98 3416 sprop-parameter-sets= 3417 a=rtpmap:98 H264/90000 3418 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3419 packetization-mode=1 3421 Answer SDP: 3423 m=video 49170 RTP/AVP 98 3424 a=ssrc:SSRC-A cname:CNAME-A 3425 a=ssrc:SSRC-A fmtp:98 3426 sprop-parameter-sets= 3427 a=rtpmap:98 H264/90000 3428 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3429 packetization-mode=1 3431 8.4. Parameter Set Considerations 3433 The H.264 parameter sets are a fundamental part of the video codec 3434 and vital to its operation; see section 1.2. Due to their 3435 characteristics and their importance for the decoding process, lost 3436 or erroneously transmitted parameter sets can hardly be concealed 3437 locally at the receiver. A reference to a corrupt parameter set 3438 has normally fatal results to the decoding process. Corruption 3439 could occur, for example, due to the erroneous transmission or loss 3440 of a parameter set NAL unit, but also due to the untimely 3441 transmission of a parameter set update. A parameter set update 3442 refers to a change of at least one parameter in a picture parameter 3443 set or sequence parameter set for which the picture parameter set 3444 or sequence parameter set identifier remains unchanged. Therefore, 3445 the following recommendations are provided as a guideline for the 3446 implementer of the RTP sender. 3448 Parameter set NALUs can be transported using three different 3449 principles: 3451 A. Using a session control protocol (out-of-band) prior to the 3452 actual RTP session. 3454 B. Using a session control protocol (out-of-band) during an ongoing 3455 RTP session. 3457 C. Within the RTP packet stream in the payload (in-band) during an 3458 ongoing RTP session. 3460 It is recommended to implement principles A and B within a session 3461 control protocol. SIP and SDP can be used as described in the SDP 3462 Offer/Answer model and in the previous sections of this memo. 3463 Section 8.2.2 includes a detailed discussion on transport of 3464 parameter sets in-band or out-of-band in SDP Offer/Answer using 3465 media type parameters "sprop-parameter-sets", "sprop-level- 3466 parameter-sets", "use-level-src-parameter-sets" and "in-band- 3467 parameter-sets". This section contains guidelines on how 3468 principles A and B should be implemented within session control 3469 protocols. It is independent of the particular protocol used. 3470 Principle C is supported by the RTP payload format defined in this 3471 specification. There are topologies like Topo-Video-switch-MCU [29] 3472 for which the use of principle C may be desirable. 3474 If in-band signaling of parameter sets is used, the picture and 3475 sequence parameter set NALUs SHOULD be transmitted in the RTP 3476 payload using a reliable method of delivering of RTP (see below), 3477 as a loss of a parameter set of either type will likely prevent 3478 decoding of a considerable portion of the corresponding RTP packet 3479 stream. 3481 If in-band signaling of parameter sets is used, the sender SHOULD 3482 take the error characteristics into account and use mechanisms to 3483 provide a high probability for delivering the parameter sets 3484 correctly. Mechanisms that increase the probability for a correct 3485 reception include packet repetition, FEC, and retransmission. The 3486 use of an unreliable, out-of-band control protocol has similar 3487 disadvantages as the in-band signaling (possible loss) and, in 3488 addition, may also lead to difficulties in the synchronization (see 3489 below). Therefore, it is NOT RECOMMENDED. 3491 Parameter sets MAY be added or updated during the lifetime of a 3492 session using principles B and C. It is required that parameter 3493 sets are present at the decoder prior to the NAL units that refer 3494 to them. Updating or adding of parameter sets can result in 3495 further problems, and therefore the following recommendations 3496 should be considered. 3498 - When parameter sets are added or updated, care SHOULD be taken 3499 to ensure that any parameter set is delivered prior to its usage. 3500 When new parameter sets are added, previously unused parameter 3501 set identifiers are used. It is common that no synchronization 3502 is present between out-of-band signaling and in-band traffic. 3503 If out-of-band signaling is used, it is RECOMMENDED that a 3504 sender does not start sending NALUs requiring the added or 3505 updated parameter sets prior to acknowledgement of delivery from 3506 the signaling protocol. 3508 - When parameter sets are updated, the following synchronization 3509 issue should be taken into account. When overwriting a 3510 parameter set at the receiver, the sender has to ensure that the 3511 parameter set in question is not needed by any NALU present in 3512 the network or receiver buffers. Otherwise, decoding with a 3513 wrong parameter set may occur. To lessen this problem, it is 3514 RECOMMENDED either to overwrite only those parameter sets that 3515 have not been used for a sufficiently long time (to ensure that 3516 all related NALUs have been consumed), or to add a new parameter 3517 set instead (which may have negative consequences for the 3518 efficiency of the video coding). 3520 Informative note: In some topologies like Topo-Video-switch- 3521 MCU [29] the origin of the whole set of parameter sets may 3522 come from multiple sources that may use non-unique parameter 3523 sets identifiers. In this case an offer may overwrite an 3524 existing parameter set if no other mechanism that enables 3525 uniqueness of the parameter sets in the out-of-band channel 3526 exists. 3528 - In a multiparty session, one participant MUST associate 3529 parameter sets coming from different sources with the source 3530 identification whenever possible, e.g. by conveying out-of-band 3531 transported parameter sets, as different sources typically use 3532 independent parameter set identifier value spaces. 3534 - Adding or modifying parameter sets by using both principles B 3535 and C in the same RTP session may lead to inconsistencies of the 3536 parameter sets because of the lack of synchronization between 3537 the control and the RTP channel. Therefore, principles B and C 3538 MUST NOT both be used in the same session unless sufficient 3539 synchronization can be provided. 3541 In some scenarios (e.g., when only the subset of this payload 3542 format specification corresponding to H.241 is used) or topologies, 3543 it is not possible to employ out-of-band parameter set transmission. 3544 In this case, parameter sets have to be transmitted in-band. Here, 3545 the synchronization with the non-parameter-set-data in the 3546 bitstream is implicit, but the possibility of a loss has to be 3547 taken into account. The loss probability should be reduced using 3548 the mechanisms discussed above. In case a loss of a parameter set 3549 is detected, recovery may be achieved by using a Decoder Refresh 3550 Point procedure, for example, using RTCP feedback Full Intra 3551 Request (FIR) [30]. Two example Decoder Refresh Point procedures 3552 are provided in the informative Section 8.5. 3554 - When parameter sets are initially provided using principle A and 3555 then later added or updated in-band (principle C), there is a 3556 risk associated with updating the parameter sets delivered out- 3557 of-band. If receivers miss some in-band updates (for example, 3558 because of a loss or a late tune-in), those receivers attempt to 3559 decode the bitstream using out-dated parameters. It is 3560 therefore RECOMMENDED that parameter set IDs be partitioned 3561 between the out-of-band and in-band parameter sets. 3563 8.5. Decoder Refresh Point Procedure using In-Band Transport of 3564 Parameter Sets (Informative) 3566 When a sender with a video encoder according to [1] receives a 3567 request for a decoder refresh point, the encoder shall enter the 3568 fast update mode by using one of the procedures specified 3569 in Section 8.5.1 or 8.5.2 below. The procedure in 8.5.1 is the 3570 preferred response in a lossless transmission environment. Both 3571 procedures satisfy the requirement to enter the fast update mode 3572 for H.264 video encoding. 3574 8.5.1. IDR Procedure to Respond to a Request for a Decoder Refresh 3575 Point 3577 This section gives one possible way to respond to a request for a 3578 decoder refresh point. 3580 The encoder shall, in the order presented here: 3582 1) Immediately prepare to send an IDR picture. 3584 2) Send a sequence parameter set to be used by the IDR picture to 3585 be sent. The encoder may optionally also send other sequence 3586 parameter sets. 3588 3) Send a picture parameter set to be used by the IDR picture to be 3589 sent. The encoder may optionally also send other picture 3590 parameter sets. 3592 4) Send the IDR picture. 3594 5) From this point forward in time, send any other sequence or 3595 picture parameter sets that have not yet been sent in this 3596 procedure, prior to their reference by any NAL unit, regardless 3597 of whether such parameter sets were previously sent prior to 3598 receiving the request for a decoder refresh point. As needed, 3599 such parameter sets may be sent in a batch, one at a time, or in 3600 any combination of these two methods. Parameter sets may be re- 3601 sent at any time for redundancy. Caution should be taken when 3602 parameter set updates are present, as described above in Section 3603 8.4. 3605 8.5.2. Gradual Recovery Procedure to Respond to a Request for a 3606 Decoder Refresh Point 3608 This section gives another possible way to respond to a request for 3609 a decoder refresh point. 3611 The encoder shall, in the order presented here: 3613 1) Send a recovery point SEI message (see Sections D.1.7 and D.2.7 3614 of [1]). 3616 2) Repeat any sequence and picture parameter sets that were sent 3617 before the recovery point SEI message, prior to their reference 3618 by a NAL unit. 3620 The encoder shall ensure that the decoder has access to all 3621 reference pictures for inter prediction of pictures at or after the 3622 recovery point, which is indicated by the recovery point SEI 3623 message, in output order, assuming that the transmission from now 3624 on is error-free. 3626 The value of the recovery_frame_cnt syntax element in the recovery 3627 point SEI message should be small enough to ensure a fast recovery. 3629 As needed, such parameter sets may be re-sent in a batch, one at a 3630 time, or in any combination of these two methods. Parameter sets 3631 may be re-sent at any time for redundancy. Caution should be taken 3632 when parameter set updates are present, as described above in 3633 Section 8.4. 3635 9. Security Considerations 3637 RTP packets using the payload format defined in this specification 3638 are subject to the security considerations discussed in the RTP 3639 specification [5], and in any appropriate RTP profile (for example, 3640 [16]). This implies that confidentiality of the media streams is 3641 achieved by encryption; for example, through the application of 3642 SRTP [26]. Because the data compression used with this payload 3643 format is applied end-to-end, any encryption needs to be performed 3644 after compression. A potential denial-of-service threat exists for 3645 data encodings using compression techniques that have non-uniform 3646 receiver-end computational load. The attacker can inject 3647 pathological datagrams into the stream that are complex to decode 3648 and that cause the receiver to be overloaded. H.264 is 3649 particularly vulnerable to such attacks, as it is extremely simple 3650 to generate datagrams containing NAL units that affect the decoding 3651 process of many future NAL units. Therefore, the usage of data 3652 origin authentication and data integrity protection of at least the 3653 RTP packet is RECOMMENDED; for example, with SRTP [26]. 3655 Note that the appropriate mechanism to ensure confidentiality and 3656 integrity of RTP packets and their payloads is very dependent on 3657 the application and on the transport and signaling protocols 3658 employed. Thus, although SRTP is given as an example above, other 3659 possible choices exist. 3661 Decoders MUST exercise caution with respect to the handling of user 3662 data SEI messages, particularly if they contain active elements, 3663 and MUST restrict their domain of applicability to the presentation 3664 containing the stream. 3666 End-to-End security with either authentication, integrity or 3667 confidentiality protection will prevent a MANE from performing 3668 media-aware operations other than discarding complete packets. And 3669 in the case of confidentiality protection it will even be prevented 3670 from performing discarding of packets in a media aware way. To 3671 allow any MANE to perform its operations, it will be required to be 3672 a trusted entity which is included in the security context 3673 establishment. 3675 10. Congestion Control 3677 Congestion control for RTP SHALL be used in accordance with RFC 3678 3550 [5], and with any applicable RTP profile; e.g., RFC 3551 [16]. 3679 An additional requirement if best-effort service is being used is: 3680 users of this payload format MUST monitor packet loss to ensure 3681 that the packet loss rate is within acceptable parameters. Packet 3682 loss is considered acceptable if a TCP flow across the same network 3683 path, and experiencing the same network conditions, would achieve 3684 an average throughput, measured on a reasonable timescale, that is 3685 not less than the RTP flow is achieving. This condition can be 3686 satisfied by implementing congestion control mechanisms to adapt 3687 the transmission rate (or the number of layers subscribed for a 3688 layered multicast session), or by arranging for a receiver to leave 3689 the session if the loss rate is unacceptably high. 3691 The bit rate adaptation necessary for obeying the congestion 3692 control principle is easily achievable when real-time encoding is 3693 used. However, when pre-encoded content is being transmitted, 3694 bandwidth adaptation requires the availability of more than one 3695 coded representation of the same content, at different bit rates, 3696 or the existence of non-reference pictures or sub-sequences [22] in 3697 the bitstream. The switching between the different representations 3698 can normally be performed in the same RTP session; e.g., by 3699 employing a concept known as SI/SP slices of the Extended Profile, 3700 or by switching streams at IDR picture boundaries. Only when non- 3701 downgradable parameters (such as the profile part of the 3702 profile/level ID) are required to be changed does it become 3703 necessary to terminate and re-start the media stream. This may be 3704 accomplished by using a different RTP payload type. 3706 MANEs MAY follow the suggestions outlined in section 7.3 and remove 3707 certain unusable packets from the packet stream when that stream 3708 was damaged due to previous packet losses. This can help reduce 3709 the network load in certain special cases. 3711 11. IANA Consideration 3713 The H264 media subtype name specified by RFC 3984 should be updated 3714 as defined in section 8.1 of this memo. 3716 12. Informative Appendix: Application Examples 3718 This payload specification is very flexible in its use, in order to 3719 cover the extremely wide application space anticipated for H.264. 3720 However, this great flexibility also makes it difficult for an 3721 implementer to decide on a reasonable packetization scheme. Some 3722 information on how to apply this specification to real-world 3723 scenarios is likely to appear in the form of academic publications 3724 and a test model software and description in the near future. 3725 However, some preliminary usage scenarios are described here as 3726 well. 3728 12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 3730 H.323-based video telephony systems that use H.264 as an optional 3731 video compression scheme are required to support H.241 Annex A [3] 3732 as a packetization scheme. The packetization mechanism defined in 3733 this Annex is technically identical with a small subset of this 3734 specification. 3736 When a system operates according to H.241 Annex A, parameter set 3737 NAL units are sent in-band. Only Single NAL unit packets are used. 3738 Many such systems are not sending IDR pictures regularly, but only 3739 when required by user interaction or by control protocol means; 3740 e.g., when switching between video channels in a Multipoint Control 3741 Unit or for error recovery requested by feedback. 3743 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 3744 Aggregation 3746 The RTP part of this scheme is implemented and tested (though not 3747 the control-protocol part; see below). 3749 In most real-world video telephony applications, picture parameters 3750 such as picture size or optional modes never change during the 3751 lifetime of a connection. Therefore, all necessary parameter sets 3752 (usually only one) are sent as a side effect of the capability 3753 exchange/announcement process, e.g., according to the SDP syntax 3754 specified in section 8.2 of this document. As all necessary 3755 parameter set information is established before the RTP session 3756 starts, there is no need for sending any parameter set NAL units. 3757 Slice data partitioning is not used, either. Thus, the RTP packet 3758 stream basically consists of NAL units that carry single coded 3759 slices. 3761 The encoder chooses the size of coded slice NAL units so that they 3762 offer the best performance. Often, this is done by adapting the 3763 coded slice size to the MTU size of the IP network. For small 3764 picture sizes, this may result in a one-picture-per-one-packet 3765 strategy. Intra refresh algorithms clean up the loss of packets 3766 and the resulting drift-related artifacts. 3768 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 3769 Aggregation 3771 This scheme allows better error concealment and is used in H.263 3772 based designs using RFC 4629 packetization [11]. It has been 3773 implemented, and good results were reported [13]. 3775 The VCL encoder codes the source picture so that all macroblocks 3776 (MBs) of one MB line are assigned to one slice. All slices with 3777 even MB row addresses are combined into one STAP, and all slices 3778 with odd MB row addresses into another. Those STAPs are 3779 transmitted as RTP packets. The establishment of the parameter 3780 sets is performed as discussed above. 3782 Note that the use of STAPs is essential here, as the high number of 3783 individual slices (18 for a CIF picture) would lead to unacceptably 3784 high IP/UDP/RTP header overhead (unless the source coding tool FMO 3785 is used, which is not assumed in this scenario). Furthermore, some 3786 wireless video transmission systems, such as H.324M and the IP- 3787 based video telephony specified in 3GPP, are likely to use 3788 relatively small transport packet size. For example, a typical MTU 3789 size of H.223 AL3 SDU is around 100 bytes [17]. Coding individual 3790 slices according to this packetization scheme provides further 3791 advantage in communication between wired and wireless networks, as 3792 individual slices are likely to be smaller than the preferred 3793 maximum packet size of wireless systems. Consequently, a gateway 3794 can convert the STAPs used in a wired network into several RTP 3795 packets with only one NAL unit, which are preferred in a wireless 3796 network, and vice versa. 3798 12.4. Video Telephony with Data Partitioning 3800 This scheme has been implemented and has been shown to offer good 3801 performance, especially at higher packet loss rates [13]. 3803 Data Partitioning is known to be useful only when some form of 3804 unequal error protection is available. Normally, in single-session 3805 RTP environments, even error characteristics are assumed; i.e., the 3806 packet loss probability of all packets of the session is the same 3807 statistically. However, there are means to reduce the packet loss 3808 probability of individual packets in an RTP session. A FEC packet 3809 according to RFC 2733 [18], for example, specifies which media 3810 packets are associated with the FEC packet. 3812 In all cases, the incurred overhead is substantial but is in the 3813 same order of magnitude as the number of bits that have otherwise 3814 been spent for intra information. However, this mechanism does not 3815 add any delay to the system. 3817 Again, the complete parameter set establishment is performed 3818 through control protocol means. 3820 12.5. Video Telephony or Streaming with FUs and Forward Error 3821 Correction 3823 This scheme has been implemented and has been shown to provide good 3824 performance, especially at higher packet loss rates [19]. 3826 The most efficient means to combat packet losses for scenarios 3827 where retransmissions are not applicable is forward error 3828 correction (FEC). Although application layer, end-to-end use of 3829 FEC is often less efficient than an FEC-based protection of 3830 individual links (especially when links of different 3831 characteristics are in the transmission path), application layer, 3832 end-to-end FEC is unavoidable in some scenarios. RFC 5109 [18] 3833 provides means to use generic, application layer, end-to-end FEC in 3834 packet-loss environments. A binary forward error correcting code 3835 is generated by applying the XOR operation to the bits at the same 3836 bit position in different packets. The binary code can be 3837 specified by the parameters (n,k) in which k is the number of 3838 information packets used in the connection and n is the total 3839 number of packets generated for k information packets; i.e., n-k 3840 parity packets are generated for k information packets. 3842 When a code is used with parameters (n,k) within the RFC 5109 3843 framework, the following properties are well known: 3845 a) If applied over one RTP packet, RFC 5109 provides only packet 3846 repetition. 3848 b) RFC 5109 is most bit rate efficient if XOR-connected packets 3849 have equal length. 3851 c) At the same packet loss probability p and for a fixed k, the 3852 greater the value of n is, the smaller the residual error 3853 probability becomes. For example, for a packet loss probability 3854 of 10%, k=1, and n=2, the residual error probability is about 1%, 3855 whereas for n=3, the residual error probability is about 0.1%. 3857 d) At the same packet loss probability p and for a fixed code rate 3858 k/n, the greater the value of n is, the smaller the residual 3859 error probability becomes. For example, at a packet loss 3860 probability of p=10%, k=1 and n=2, the residual error rate is 3861 about 1%, whereas for an extended Golay code with k=12 and n=24, 3862 the residual error rate is about 0.01%. 3864 For applying RFC 5109 in combination with H.264 baseline coded 3865 video without using FUs, several options might be considered: 3867 1) The video encoder produces NAL units for which each video frame 3868 is coded in a single slice. Applying FEC, one could use a 3869 simple code; e.g., (n=2, k=1). That is, each NAL unit would 3870 basically just be repeated. The disadvantage is obviously the 3871 bad code performance according to d), above, and the low 3872 flexibility, as only (n, k=1) codes can be used. 3874 2) The video encoder produces NAL units for which each video frame 3875 is encoded in one or more consecutive slices. Applying FEC, one 3876 could use a better code, e.g., (n=24, k=12), over a sequence of 3877 NAL units. Depending on the number of RTP packets per frame, a 3878 loss may introduce a significant delay, which is reduced when 3879 more RTP packets are used per frame. Packets of completely 3880 different length might also be connected, which decreases bit 3881 rate efficiency according to b), above. However, with some care 3882 and for slices of 1kb or larger, similar length (100-200 bytes 3883 difference) may be produced, which will not lower the bit 3884 efficiency catastrophically. 3886 3) The video encoder produces NAL units, for which a certain frame 3887 contains k slices of possibly almost equal length. Then, 3888 applying FEC, a better code, e.g., (n=24, k=12), can be used 3889 over the sequence of NAL units for each frame. The delay 3890 compared to that of 2), above, may be reduced, but several 3891 disadvantages are obvious. First, the coding efficiency of the 3892 encoded video is lowered significantly, as slice-structured 3893 coding reduces intra-frame prediction and additional slice 3894 overhead is necessary. Second, pre-encoded content or, when 3895 operating over a gateway, the video is usually not appropriately 3896 coded with k slices such that FEC can be applied. Finally, the 3897 encoding of video producing k slices of equal length is not 3898 straightforward and might require more than one encoding pass. 3900 Many of the mentioned disadvantages can be avoided by applying FUs 3901 in combination with FEC. Each NAL unit can be split into any 3902 number of FUs of basically equal length; therefore, FEC with a 3903 reasonable k and n can be applied, even if the encoder made no 3904 effort to produce slices of equal length. For example, a coded 3905 slice NAL unit containing an entire frame can be split to k FUs, 3906 and a parity check code (n=k+1, k) can be applied. However, this 3907 has the disadvantage that unless all created fragments can be 3908 recovered, the whole slice will be lost. Thus a larger section is 3909 lost than would be if the frame had been split into several slices. 3911 The presented technique makes it possible to achieve good 3912 transmission error tolerance, even if no additional source coding 3913 layer redundancy (such as periodic intra frames) is present. 3914 Consequently, the same coded video sequence can be used to achieve 3915 the maximum compression efficiency and quality over error-free 3916 transmission and for transmission over error-prone networks. 3917 Furthermore, the technique allows the application of FEC to pre- 3918 encoded sequences without adding delay. In this case, pre-encoded 3919 sequences that are not encoded for error-prone networks can still 3920 be transmitted almost reliably without adding extensive delays. In 3921 addition, FUs of equal length result in a bit rate efficient use of 3922 RFC 5109. 3924 If the error probability depends on the length of the transmitted 3925 packet (e.g., in case of mobile transmission [15]), the benefits of 3926 applying FUs with FEC are even more obvious. Basically, the 3927 flexibility of the size of FUs allows appropriate FEC to be applied 3928 for each NAL unit and unequal error protection of NAL units. 3930 When FUs and FEC are used, the incurred overhead is substantial but 3931 is in the same order of magnitude as the number of bits that have 3932 to be spent for intra-coded macroblocks if no FEC is applied. In 3933 [19], it was shown that the overall performance of the FEC-based 3934 approach enhanced quality when using the same error rate and same 3935 overall bit rate, including the overhead. 3937 12.6. Low Bit-Rate Streaming 3939 This scheme has been implemented with H.263 and non-standard RTP 3940 packetization and has given good results [20]. There is no 3941 technical reason why similarly good results could not be achievable 3942 with H.264. 3944 In today's Internet streaming, some of the offered bit rates are 3945 relatively low in order to allow terminals with dial-up modems to 3946 access the content. In wired IP networks, relatively large packets, 3947 say 500 - 1500 bytes, are preferred to smaller and more frequently 3948 occurring packets in order to reduce network congestion. Moreover, 3949 use of large packets decreases the amount of RTP/UDP/IP header 3950 overhead. For low bit-rate video, the use of large packets means 3951 that sometimes up to few pictures should be encapsulated in one 3952 packet. 3954 However, loss of a packet including many coded pictures would have 3955 drastic consequences for visual quality, as there is practically no 3956 other way to conceal a loss of an entire picture than to repeat the 3957 previous one. One way to construct relatively large packets and 3958 maintain possibilities for successful loss concealment is to 3959 construct MTAPs that contain interleaved slices from several 3960 pictures. An MTAP should not contain spatially adjacent slices 3961 from the same picture or spatially overlapping slices from any 3962 picture. If a packet is lost, it is likely that a lost slice is 3963 surrounded by spatially adjacent slices of the same picture and 3964 spatially corresponding slices of the temporally previous and 3965 succeeding pictures. Consequently, concealment of the lost slice 3966 is likely to be relatively successful. 3968 12.7. Robust Packet Scheduling in Video Streaming 3970 Robust packet scheduling has been implemented with MPEG-4 Part 2 3971 and simulated in a wireless streaming environment [21]. There is 3972 no technical reason why similar or better results could not be 3973 achievable with H.264. 3975 Streaming clients typically have a receiver buffer that is capable 3976 of storing a relatively large amount of data. Initially, when a 3977 streaming session is established, a client does not start playing 3978 the stream back immediately. Rather, it typically buffers the 3979 incoming data for a few seconds. This buffering helps maintain 3980 continuous playback, as, in case of occasional increased 3981 transmission delays or network throughput drops, the client can 3982 decode and play buffered data. Otherwise, without initial 3983 buffering, the client has to freeze the display, stop decoding, and 3984 wait for incoming data. The buffering is also necessary for either 3985 automatic or selective retransmission in any protocol level. If 3986 any part of a picture is lost, a retransmission mechanism may be 3987 used to resend the lost data. If the retransmitted data is 3988 received before its scheduled decoding or playback time, the loss 3989 is recovered perfectly. Coded pictures can be ranked according to 3990 their importance in the subjective quality of the decoded sequence. 3991 For example, non-reference pictures, such as conventional B 3992 pictures, are subjectively least important, as their absence does 3993 not affect decoding of any other pictures. In addition to non- 3994 reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard 3995 includes a temporal scalability method called sub-sequences [22]. 3996 Subjective ranking can also be made on coded slice data partition 3997 or slice group basis. Coded slices and coded slice data partitions 3998 that are subjectively the most important can be sent earlier than 3999 their decoding order indicates, whereas coded slices and coded 4000 slice data partitions that are subjectively the least important can 4001 be sent later than their natural coding order indicates. 4002 Consequently, any retransmitted parts of the most important slices 4003 and coded slice data partitions are more likely to be received 4004 before their scheduled decoding or playback time compared to the 4005 least important slices and slice data partitions. 4007 13. Informative Appendix: Rationale for Decoding Order Number 4009 13.1. Introduction 4011 The Decoding Order Number (DON) concept was introduced mainly to 4012 enable efficient multi-picture slice interleaving (see section 12.6) 4013 and robust packet scheduling (see section 12.7). In both of these 4014 applications, NAL units are transmitted out of decoding order. DON 4015 indicates the decoding order of NAL units and should be used in the 4016 receiver to recover the decoding order. Example use cases for 4017 efficient multi-picture slice interleaving and for robust packet 4018 scheduling are given in sections 13.2 and 13.3, respectively. 4019 Section 13.4 describes the benefits of the DON concept in error 4020 resiliency achieved by redundant coded pictures. Section 13.5 4021 summarizes considered alternatives to DON and justifies why DON was 4022 chosen to this RTP payload specification. 4024 13.2. Example of Multi-Picture Slice Interleaving 4026 An example of multi-picture slice interleaving follows. A subset 4027 of a coded video sequence is depicted below in output order. R 4028 denotes a reference picture, N denotes a non-reference picture, and 4029 the number indicates a relative output time. 4031 ... R1 N2 R3 N4 R5 ... 4033 The decoding order of these pictures from left to right is as 4034 follows: 4036 ... R1 R3 N2 R5 N4 ... 4038 The NAL units of pictures R1, R3, N2, R5, and N4 are marked with a 4039 DON equal to 1, 2, 3, 4, and 5, respectively. 4041 Each reference picture consists of three slice groups that are 4042 scattered as follows (a number denotes the slice group number for 4043 each macroblock in a QCIF frame): 4045 0 1 2 0 1 2 0 1 2 0 1 4046 2 0 1 2 0 1 2 0 1 2 0 4047 1 2 0 1 2 0 1 2 0 1 2 4048 0 1 2 0 1 2 0 1 2 0 1 4049 2 0 1 2 0 1 2 0 1 2 0 4050 1 2 0 1 2 0 1 2 0 1 2 4051 0 1 2 0 1 2 0 1 2 0 1 4052 2 0 1 2 0 1 2 0 1 2 0 4053 1 2 0 1 2 0 1 2 0 1 2 4055 For the sake of simplicity, we assume that all the macroblocks of a 4056 slice group are included in one slice. Three MTAPs are constructed 4057 from three consecutive reference pictures so that each MTAP 4058 contains three aggregation units, each of which contains all the 4059 macroblocks from one slice group. The first MTAP contains slice 4060 group 0 of picture R1, slice group 1 of picture R3, and slice group 4061 2 of picture R5. The second MTAP contains slice group 1 of picture 4062 R1, slice group 2 of picture R3, and slice group 0 of picture R5. 4063 The third MTAP contains slice group 2 of picture R1, slice group 0 4064 of picture R3, and slice group 1 of picture R5. Each non-reference 4065 picture is encapsulated into an STAP-B. 4067 Consequently, the transmission order of NAL units is the following: 4069 R1, slice group 0, DON 1, carried in MTAP,RTP SN: N 4070 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N 4071 R5, slice group 2, DON 4, carried in MTAP,RTP SN: N 4072 R1, slice group 1, DON 1, carried in MTAP,RTP SN: N+1 4073 R3, slice group 2, DON 2, carried in MTAP,RTP SN: N+1 4074 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+1 4075 R1, slice group 2, DON 1, carried in MTAP,RTP SN: N+2 4076 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N+2 4077 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+2 4078 N2, DON 3, carried in STAP-B, RTP SN: N+3 4079 N4, DON 5, carried in STAP-B, RTP SN: N+4 4081 The receiver is able to organize the NAL units back in decoding 4082 order based on the value of DON associated with each NAL unit. 4084 If one of the MTAPs is lost, the spatially adjacent and temporally 4085 co-located macroblocks are received and can be used to conceal the 4086 loss efficiently. If one of the STAPs is lost, the effect of the 4087 loss does not propagate temporally. 4089 13.3. Example of Robust Packet Scheduling 4091 An example of robust packet scheduling follows. The communication 4092 system used in the example consists of the following components in 4093 the order that the video is processed from source to sink: 4095 o camera and capturing 4096 o pre-encoding buffer 4097 o encoder 4098 o encoded picture buffer 4099 o transmitter 4100 o transmission channel 4101 o receiver 4102 o receiver buffer 4103 o decoder 4104 o decoded picture buffer 4105 o display 4107 The video communication system used in the example operates as 4108 follows. Note that processing of the video stream happens 4109 gradually and at the same time in all components of the system. 4110 The source video sequence is shot and captured to a pre-encoding 4111 buffer. The pre-encoding buffer can be used to order pictures from 4112 sampling order to encoding order or to analyze multiple 4113 uncompressed frames for bit rate control purposes, for example. In 4114 some cases, the pre-encoding buffer may not exist; instead, the 4115 sampled pictures are encoded right away. The encoder encodes 4116 pictures from the pre-encoding buffer and stores the output; i.e., 4117 coded pictures, to the encoded picture buffer. The transmitter 4118 encapsulates the coded pictures from the encoded picture buffer to 4119 transmission packets and sends them to a receiver through a 4120 transmission channel. The receiver stores the received packets to 4121 the receiver buffer. The receiver buffering process typically 4122 includes buffering for transmission delay jitter. The receiver 4123 buffer can also be used to recover correct decoding order of coded 4124 data. The decoder reads coded data from the receiver buffer and 4125 produces decoded pictures as output into the decoded picture buffer. 4126 The decoded picture buffer is used to recover the output (or 4127 display) order of pictures. Finally, pictures are displayed. 4129 In the following example figures, I denotes an IDR picture, R 4130 denotes a reference picture, N denotes a non-reference picture, and 4131 the number after I, R, or N indicates the sampling time relative to 4132 the previous IDR picture in decoding order. Values below the 4133 sequence of pictures indicate scaled system clock timestamps. The 4134 system clock is initialized arbitrarily in this example, and time 4135 runs from left to right. Each I, R, and N picture is mapped into 4136 the same timeline compared to the previous processing step, if any, 4137 assuming that encoding, transmission, and decoding take no time. 4138 Thus, events happening at the same time are located in the same 4139 column throughout all example figures. 4141 A subset of a sequence of coded pictures is depicted below in 4142 sampling order. 4144 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ... 4145 ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ... 4146 ... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ... 4148 Figure 16 Sequence of pictures in sampling order 4150 The sampled pictures are buffered in the pre-encoding buffer to 4151 arrange them in encoding order. In this example, we assume that 4152 the non-reference pictures are predicted from both the previous and 4153 the next reference picture in output order, except for the non- 4154 reference pictures immediately preceding an IDR picture, which are 4155 predicted only from the previous reference picture in output order. 4156 Thus, the pre-encoding buffer has to contain at least two pictures, 4157 and the buffering causes a delay of two picture intervals. The 4158 output of the pre-encoding buffering process and the encoding (and 4159 decoding) order of the pictures are as follows: 4161 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4162 ... -|---|---|---|---|---|---|---|---|- ... 4163 ... 60 61 62 63 64 65 66 67 68 ... 4165 Figure 17 Re-ordered pictures in the pre-encoding buffer 4167 The encoder or the transmitter can set the value of DON for each 4168 picture to a value of DON for the previous picture in decoding 4169 order plus one. 4171 For the sake of simplicity, let us assume that: 4173 o the frame rate of the sequence is constant, 4174 o each picture consists of only one slice, 4175 o each slice is encapsulated in a single NAL unit packet, 4176 o there is no transmission delay, and 4177 o pictures are transmitted at constant intervals (that is, 1 / 4178 (frame rate)). 4180 When pictures are transmitted in decoding order, they are received 4181 as follows: 4183 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4184 ... -|---|---|---|---|---|---|---|---|- ... 4185 ... 60 61 62 63 64 65 66 67 68 ... 4187 Figure 18 Received pictures in decoding order 4189 The OPTIONAL sprop-interleaving-depth media type parameter is set 4190 to 0, as the transmission (or reception) order is identical to the 4191 decoding order. 4193 The decoder has to buffer for one picture interval initially in its 4194 decoded picture buffer to organize pictures from decoding order to 4195 output order as depicted below: 4197 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... 4198 ... -|---|---|---|---|---|---|---|---|- ... 4199 ... 61 62 63 64 65 66 67 68 69 ... 4201 Figure 19 Output order 4203 The amount of required initial buffering in the decoded picture 4204 buffer can be signaled in the buffering period SEI message or with 4205 the num_reorder_frames syntax element of H.264 video usability 4206 information. num_reorder_frames indicates the maximum number of 4207 frames, complementary field pairs, or non-paired fields that 4208 precede any frame, complementary field pair, or non-paired field in 4209 the sequence in decoding order and that follow it in output order. 4210 For the sake of simplicity, we assume that num_reorder_frames is 4211 used to indicate the initial buffer in the decoded picture buffer. 4212 In this example, num_reorder_frames is equal to 1. 4214 It can be observed that if the IDR picture I00 is lost during 4215 transmission and a retransmission request is issued when the value 4216 of the system clock is 62, there is one picture interval of time 4217 (until the system clock reaches timestamp 63) to receive the 4218 retransmitted IDR picture I00. 4220 Let us then assume that IDR pictures are transmitted two frame 4221 intervals earlier than their decoding position; i.e., the pictures 4222 are transmitted as follows: 4224 ... I00 N58 N59 R03 N01 N02 R06 N04 N05 ... 4225 ... --|---|---|---|---|---|---|---|---|- ... 4226 ... 62 63 64 65 66 67 68 69 70 ... 4228 Figure 20 Interleaving: Early IDR pictures in sending order 4230 The OPTIONAL sprop-interleaving-depth media type parameter is set 4231 equal to 1 according to its definition. (The value of sprop- 4232 interleaving-depth in this example can be derived as follows: 4234 Picture I00 is the only picture preceding picture N58 or N59 in 4235 transmission order and following it in decoding order. Except for 4236 pictures I00, N58, and N59, the transmission order is the same as 4237 the decoding order of pictures. As a coded picture is encapsulated 4238 into exactly one NAL unit, the value of sprop-interleaving-depth is 4239 equal to the maximum number of pictures preceding any picture in 4240 transmission order and following the picture in decoding order.) 4242 The receiver buffering process contains two pictures at a time 4243 according to the value of the sprop-interleaving-depth parameter 4244 and orders pictures from the reception order to the correct 4245 decoding order based on the value of DON associated with each 4246 picture. The output of the receiver buffering process is as 4247 follows: 4249 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4250 ... -|---|---|---|---|---|---|---|---|- ... 4251 ... 63 64 65 66 67 68 69 70 71 ... 4253 Figure 21 Interleaving: Receiver buffer 4255 Again, an initial buffering delay of one picture interval is needed 4256 to organize pictures from decoding order to output order, as 4257 depicted below: 4259 ... N58 N59 I00 N01 N02 R03 N04 N05 ... 4260 ... -|---|---|---|---|---|---|---|- ... 4261 ... 64 65 66 67 68 69 70 71 ... 4263 Figure 22 Interleaving: Receiver buffer after reordering 4265 Note that the maximum delay that IDR pictures can undergo during 4266 transmission, including possible application, transport, or link 4267 layer retransmission, is equal to three picture intervals. Thus, 4268 the loss resiliency of IDR pictures is improved in systems 4269 supporting retransmission compared to the case in which pictures 4270 were transmitted in their decoding order. 4272 13.4. Robust Transmission Scheduling of Redundant Coded Slices 4274 A redundant coded picture is a coded representation of a picture or 4275 a part of a picture that is not used in the decoding process if the 4276 corresponding primary coded picture is correctly decoded. There 4277 should be no noticeable difference between any area of the decoded 4278 primary picture and a corresponding area that would result from 4279 application of the H.264 decoding process for any redundant picture 4280 in the same access unit. A redundant coded slice is a coded slice 4281 that is a part of a redundant coded picture. 4283 Redundant coded pictures can be used to provide unequal error 4284 protection in error-prone video transmission. If a primary coded 4285 representation of a picture is decoded incorrectly, a corresponding 4286 redundant coded picture can be decoded. Examples of applications 4287 and coding techniques using the redundant codec picture feature 4288 include the video redundancy coding [23] and the protection of "key 4289 pictures" in multicast streaming [24]. 4291 One property of many error-prone video communications systems is 4292 that transmission errors are often bursty. Therefore, they may 4293 affect more than one consecutive transmission packets in 4294 transmission order. In low bit-rate video communication, it is 4295 relatively common that an entire coded picture can be encapsulated 4296 into one transmission packet. Consequently, a primary coded 4297 picture and the corresponding redundant coded pictures may be 4298 transmitted in consecutive packets in transmission order. To make 4299 the transmission scheme more tolerant of bursty transmission errors, 4300 it is beneficial to transmit the primary coded picture and 4301 redundant coded picture separated by more than a single packet. 4302 The DON concept enables this. 4304 13.5. Remarks on Other Design Possibilities 4306 The slice header syntax structure of the H.264 coding standard 4307 contains the frame_num syntax element that can indicate the 4308 decoding order of coded frames. However, the usage of the 4309 frame_num syntax element is not feasible or desirable to recover 4310 the decoding order, due to the following reasons: 4312 o The receiver is required to parse at least one slice header per 4313 coded picture (before passing the coded data to the decoder). 4315 o Coded slices from multiple coded video sequences cannot be 4316 interleaved, as the frame number syntax element is reset to 0 in 4317 each IDR picture. 4319 o The coded fields of a complementary field pair share the same 4320 value of the frame_num syntax element. Thus, the decoding order 4321 of the coded fields of a complementary field pair cannot be 4322 recovered based on the frame_num syntax element or any other 4323 syntax element of the H.264 coding syntax. 4325 The RTP payload format for transport of MPEG-4 elementary streams 4326 [25] enables interleaving of access units and transmission of 4327 multiple access units in the same RTP packet. An access unit is 4328 specified in the H.264 coding standard to comprise all NAL units 4329 associated with a primary coded picture according to subclause 4330 7.4.1.2 of [1]. Consequently, slices of different pictures cannot 4331 be interleaved, and the multi-picture slice interleaving technique 4332 (see section 12.6) for improved error resilience cannot be used. 4334 14. Acknowledgements 4336 Stephan Wenger, Miska Hannuksela, Thomas Stockhammer, Magnus 4337 Westerlund, and David Singer are thanked as the authors of RFC 3984. 4338 Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, 4339 Joerg Ott, and Colin Perkins are thanked for careful review during 4340 the development of RFC 3984. Randell Jesup, Stephen Botzko, Magnus 4341 Westerlund, Alex Eleftheriadis, Thomas Schierl, Tom Taylor, Ali 4342 Begen, and Aaron Wells are thanked for their valuable comments and 4343 inputs during the development of this memo. 4345 This document was prepared using 2-Word-v2.0.template.dot. 4347 15. References 4349 15.1. Normative References 4351 [1] ITU-T Recommendation H.264, "Advanced video coding for 4352 generic audiovisual services", November 2007. 4354 [2] ISO/IEC International Standard 14496-10:2008. 4356 [3] ITU-T Recommendation H.241, "Extended video procedures and 4357 control signals for H.300 series terminals", May 2006. 4359 [4] Bradner, S., "Key words for use in RFCs to Indicate 4360 Requirement Levels", BCP 14, RFC 2119, March 1997. 4362 [5] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, 4363 "RTP: A Transport Protocol for Real-Time Applications", STD 4364 64, RFC 3550, July 2003. 4366 [6] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 4367 Description Protocol", RFC 4566, July 2006. 4369 [7] Josefsson, S., "The Base16, Base32, and Base64 Data 4370 Encodings", RFC 3548, July 2003. 4372 [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with 4373 Session Description Protocol (SDP)", RFC 3264, June 2002. 4375 [9] Lennox, J., Ott, J., and Schierl, T., "Source-Specific Media 4376 Attributes in the Session Description Protocol", draft-ietf- 4377 mmusic-sdp-source-attributes-02 (work in progress), October 4378 2008. 4380 15.2. Informative References 4382 [10] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special 4383 Issue on H.264/AVC. IEEE Transactions on Circuits and Systems 4384 on Video Technology, July 2003. 4386 [11] Ott, J., Bormann, C., Sullivan, G., Wenger, S., and R. Even 4387 (Ed.), "RTP Payload Format for ITU-T Rec. H.263 Video", RFC 4388 4629, January 2007. 4390 [12] ISO/IEC IS 14496-2. 4392 [13] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and 4393 Systems for Video technology, Vol. 13, No. 7, July 2003. 4395 [14] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", 4396 Proceedings Packet Video Workshop 02, April 2002. 4398 [15] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT 4399 Coding Network Abstraction Layer and IP-based Transport" in 4400 Proc. ICIP 2002, Rochester, NY, September 2002. 4402 [16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 4403 Video Conferences with Minimal Control", STD 65, RFC 3551, 4404 July 2003. 4406 [17] ITU-T Recommendation H.223, "Multiplexing protocol for low 4407 bit rate multimedia communication", July 2001. 4409 [18] Li, A., "RTP Payload Format for Generic Forward Error 4410 Correction", RFC 5109, December 2007. 4412 [19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, 4413 "Video Coding and Transport Layer Techniques for H.264/AVC- 4414 Based Transmission over Packet-Lossy Networks", IEEE 4415 International Conference on Image Processing (ICIP 2003), 4416 Barcelona, Spain, September 2003. 4418 [20] Varsa, V. and M. Karczewicz, "Slice interleaving in 4419 compressed video packetization", Packet Video Workshop 2000. 4421 [21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for 4422 wireless video streaming," International Packet Video 4423 Workshop 2002. 4425 [22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, 4426 available http://ftp3.itu.int/av-arch/video- 4427 site/0201_Gen/JVT-B042.doc, anuary 2002. 4429 [23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 4430 International Workshop on Audio-Visual Services over Packet 4431 Networks, September 1997. 4433 [24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error 4434 Resilient Video Coding Using Unequally Protected Key 4435 Pictures", in Proc. International Workshop VLBV03, September 4436 2003. 4438 [25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., 4439 and P. Gentric, "RTP Payload Format for Transport of MPEG-4 4440 Elementary Streams", RFC 3640, November 2003. 4442 [26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 4443 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 4444 RFC 3711, March 2004. 4446 [27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time 4447 Streaming Protocol (RTSP)", RFC 2326, April 1998. 4449 [28] Handley, M., Perkins, C., and E. Whelan, "Session 4450 Announcement Protocol", RFC 2974, October 2000. 4452 [29] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, 4453 January 2008. 4455 [30] Wenger, S., Chandra, U., and M. Westerlund, "Codec Control 4456 Messages in the RTP Audio-Visual Profile with Feedback 4457 (AVPF)", RFC 5104, February 2008. 4459 16. Authors' Addresses 4461 Ye-Kui Wang 4462 Huawei Technologies 4463 400 Somerset Corp Blvd, Suite 602 4464 Bridgewater, NJ 08807 4465 USA 4467 Phone: +1-908-541-3518 4468 EMail: yekuiwang@huawei.com 4470 Roni Even 4471 14 David Hamelech 4472 Tel Aviv 64953 4473 Israel 4475 Phone: +972-545481099 4476 Email: ron.even.tlv@gmail.com 4478 Tom Kristensen 4479 TANDBERG 4480 Philip Pedersens vei 22 4481 N-1366 Lysaker 4482 Norway 4484 Phone: +47 67125125 4485 Email: tom.kristensen@tandberg.com, tomkri@ifi.uio.no 4487 Randell Jesup 4488 WorldGate Communications 4489 3190 Tremont Ave 4490 Trevose, PA 19053 4491 USA 4493 Phone: +1-215-354-5166 4494 Email: rjesup@wgate.com 4496 17. Backward Compatibility to RFC 3984 4498 The current document is a revision of RFC 3984 and intends to 4499 obsolete it. This section addresses the backward compatibility 4500 issues. 4502 The technical changes are listed in section 18. 4504 Items 1), 2), 3), 7), 9), 10), 12), 13) are bug-fix type of changes, 4505 and do not incur any backward compatibility issues. 4507 Item 4), addition of six new media type parameters, does not incur 4508 any backward compatibility issues for SDP Offer/Answer based 4509 applications, as legacy RFC 3984 receivers ignore these parameters, 4510 and it is fine for legacy RFC 3984 senders not to use these 4511 parameters as they are optional. However, there is a backward 4512 compatibility issue for SDP declarative usage based applications, 4513 e.g. those using RTSP and SAP, because the SDP receiver per RFC 4514 3984 cannot accept a session for which the SDP includes an 4515 unrecognized parameter. Therefore, the RTSP or SAP server may have 4516 to prepare two sets of streams, one for legacy RFC 3984 receivers 4517 and one for receivers according to this memo. 4519 Items 5), 6) and 11) are related to out-of-band transport of 4520 parameter sets. There are following backward compatibility issues. 4522 1) When a legacy sender per RFC 3984 includes parameter sets for a 4523 level different than the default level indicated by profile- 4524 level-id to sprop-parameter-sets, the parameter value of sprop- 4525 parameter-sets is invalid to the receiver per this memo and 4526 therefore the session may be rejected. 4528 2) In SDP Offer/Answer between a legacy offerer per RFC 3984 and an 4529 answerer per this memo, when the answerer includes in the answer 4530 parameter sets that are not a superset of the parameter sets 4531 included in the offer, the parameter value of sprop-parameter- 4532 sets is invalid to offerer and the session may not be initiated 4533 properly (related to change item 11)). 4535 3) When one endpoint A per this memo includes in-band-parameter- 4536 sets equal to 1, the other side B per RFC 3984 does not 4537 understand that it must transmit parameter sets in-band and B 4538 may still exclude parameter sets in the in-band stream it is 4539 sending. Consequently endpoint A cannot decode the stream it 4540 receives. 4542 Item 7), allowance of conveying sprop-parameter-sets and sprop- 4543 level-parameter-sets using the "fmtp" source attribute as specified 4544 in section 6.3 of [9], is similar as item 4). It does not incur 4545 any backward compatibility issues for SDP Offer/Answer based 4546 applications, as legacy RFC 3984 receivers ignore the "fmtp" source 4547 attribute, and it is fine for legacy RFC 3984 senders not to use 4548 the "fmtp" source attribute as it is optional. However, there is a 4549 backward compatibility issue for SDP declarative usage based 4550 applications, e.g. those using RTSP and SAP, because the SDP 4551 receiver per RFC 3984 cannot accept a session for which the SDP 4552 includes an unrecognized parameter (i.e., the "fmtp" source 4553 attribute). Therefore, the RTSP or SAP server may have to prepare 4554 two sets of streams, one for legacy RFC 3984 receivers and one for 4555 receivers according to this memo. 4557 Item 14) removed that use of out-of-band transport of parameter 4558 sets is recommended. As out-of-band transport of parameter sets is 4559 still allowed, this change does not incur any backward 4560 compatibility issues. 4562 Item 15) does not incur any backward compatibility issues as the 4563 added subsection 8.5 is informative. 4565 Item 16) does not create any backward compatibility issues as the 4566 handling of default level is the same if either end is RFC 3984 4567 compliant, and furthermore, RFC 3984 compliant ends would simply 4568 ignore the new media type parameters, if present. 4570 18. Changes from RFC 3984 4572 Following is the list of technical changes (including bug fixes) 4573 from RFC 3984. Besides this list of technical changes, numerous 4574 editorial changes have been made, but not documented in this memo. 4576 1) In subsections 5.4, 5.5, 6.2, 6,3 and 6.4, removed that the 4577 packetization mode in use may be signaled by external means. 4579 2) In subsection 7.2.2, changed the sentence 4581 There are N VCL NAL units in the deinterleaving buffer. 4583 to 4585 There are N or more VCL NAL units in the de-interleaving buffer. 4587 3) In subsection 8.1, the semantics of sprop-init-buf-time, 4588 paragraph 2, changed the sentence 4590 The parameter is the maximum value of (transmission time of a 4591 NAL unit - decoding time of the NAL unit), assuming reliable and 4592 instantaneous transmission, the same timeline for transmission 4593 and decoding, and that decoding starts when the first packet 4594 arrives. 4596 to 4598 The parameter is the maximum value of (decoding time of the NAL 4599 unit - transmission time of a NAL unit), assuming reliable and 4600 instantaneous transmission, the same timeline for transmission 4601 and decoding, and that decoding starts when the first packet 4602 arrives. 4604 4) Added media type parameters max-smbps, sprop-level-parameter- 4605 sets, use-level-src-parameter-sets, in-band-parameter-sets, sar- 4606 understood and sar-supported. 4608 5) In subsection 8.1, removed the specification of parameter-add. 4609 Other descriptions of parameter-add (in subsections 8.2 and 8.4) 4610 are also removed. 4612 6) In subsection 8.1, added a constraint to sprop-parameter-sets 4613 such that it can only contain parameter sets for the same 4614 profile and level as indicated by profile-level-id. 4616 7) In subsection 8.2.1, added that sprop-parameter-sets and sprop- 4617 level-parameter-sets may be either included in the "a=fmtp" line 4618 of SDP or conveyed using the "fmtp" source attribute as 4619 specified in section 6.3 of [9]. 4621 8) In subsection 8.2.2, removed sprop-deint-buf-req from being part 4622 of the media format configuration in usage with the SDP 4623 Offer/Answer model. 4625 9) In subsection 8.2.2, made it clear that level is downgradable in 4626 the SDP Offer/Answer model, i.e. the use of the level part of 4627 "profile-level-id" does not need to be symmetric (the level 4628 included in the answer can be lower than or equal to the level 4629 included in the offer). 4631 10)In subsection 8.2.2, removed that the capability parameters may 4632 be used to declare encoding capabilities. 4634 11)In subsection 8.2.2, added rules on how to use sprop-parameter- 4635 sets and sprop-level-parameter-sets for out-of-band transport of 4636 parameter sets, with or without level downgrading. 4638 12)In subsection 8.2.2, clarified the rules of using the media type 4639 parameters with SDP Offer/Answer for multicast. 4641 13)In subsection 8.2.2, completed and corrected the list of how 4642 different media type parameters shall be interpreted in the 4643 different combinations of offer or answer and direction 4644 attribute. 4646 14)In subsection 8.4, changed the text such that both out-of-band 4647 and in-band transport of parameter sets are allowed and neither 4648 is recommended or required. 4650 15)Added subsection 8.5 (informative) providing example methods for 4651 decoder refresh to handle parameter set losses. 4653 16)Added media type parameters max-recv-level, and level-asymmetry- 4654 allowed, and adjusted associated text and examples for level 4655 upgrade and asymmetry.