idnits 2.17.1 draft-ietf-avt-rtp-rfc3984bis-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The abstract seems to indicate that this document obsoletes RFC3984, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1975 has weird spacing: '...licitly as th...' == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 9, 2010) is 4938 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' ** Obsolete normative reference: RFC 4566 (ref. '6') (Obsoleted by RFC 8866) -- Obsolete informational reference (is this intentional?): RFC 2326 (ref. '27') (Obsoleted by RFC 7826) -- Obsolete informational reference (is this intentional?): RFC 5117 (ref. '29') (Obsoleted by RFC 7667) Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Obsoletes RFC 3984 2 Audio/Video Transport WG Y.-K. Wang 3 Internet Draft Huawei Technologies 4 Intended status: Standards track R. Even 5 Expires: April 2011 Self-employed 6 T. Kristensen 7 Tandberg 8 R. Jesup 9 WorldGate Communications 10 October 9, 2010 12 RTP Payload Format for H.264 Video 13 draft-ietf-avt-rtp-rfc3984bis-12.txt 15 Status of this Memo 17 This Internet-Draft is submitted to IETF in full conformance with 18 the provisions of BCP 78 and BCP 79. 20 Internet-Drafts are working documents of the Internet Engineering 21 Task Force (IETF), its areas, and its working groups. Note that 22 other groups may also distribute working documents as Internet- 23 Drafts. 25 Internet-Drafts are draft documents valid for a maximum of six 26 months and may be updated, replaced, or obsoleted by other 27 documents at any time. It is inappropriate to use Internet-Drafts 28 as reference material or to cite them other than as "work in 29 progress." 31 The list of current Internet-Drafts can be accessed at 32 http://www.ietf.org/ietf/1id-abstracts.txt. 34 The list of Internet-Draft Shadow Directories can be accessed at 35 http://www.ietf.org/shadow.html. 37 This Internet-Draft will expire on April 9, 2009. 39 Copyright and License Notice 41 Copyright (c) 2010 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with 49 respect to this document. Code Components extracted from this 50 document must include Simplified BSD License text as described in 51 Section 4.e of the Trust Legal Provisions and are provided without 52 warranty as described in the BSD License. 54 Abstract 56 This memo describes an RTP Payload format for the ITU-T 57 Recommendation H.264 video codec and the technically identical 58 ISO/IEC International Standard 14496-10 video codec, excluding the 59 Scalable Video Coding (SVC) extension and the Multivew Video Coding 60 extension, for which the RTP payload formats are defined elsewhere. 61 The RTP payload format allows for packetization of one or more 62 Network Abstraction Layer Units (NALUs), produced by an H.264 video 63 encoder, in each RTP payload. The payload format has wide 64 applicability, as it supports applications from simple low bit-rate 65 conversational usage, to Internet video streaming with interleaved 66 transmission, to high bit-rate video-on-demand. 68 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 69 in section 15. Issues on backward compatibility to RFC 3984 are 70 discussed in section 14. 72 Table of Contents 74 Table of Contents................................................2 75 1. Introduction..................................................4 76 1.1. The H.264 Codec..........................................4 77 1.2. Parameter Set Concept....................................6 78 1.3. Network Abstraction Layer Unit Types.....................6 79 2. Conventions...................................................7 80 3. Scope.........................................................8 81 4. Definitions and Abbreviations.................................8 82 4.1. Definitions..............................................8 83 4.2. Abbreviations...........................................10 84 5. RTP Payload Format...........................................11 85 5.1. RTP Header Usage........................................11 86 5.2. Payload Structures......................................13 87 5.3. NAL Unit Header Usage...................................14 88 5.4. Packetization Modes.....................................16 89 5.5. Decoding Order Number (DON).............................18 90 5.6. Single NAL Unit Packet..................................20 91 5.7. Aggregation Packets.....................................21 92 Table 4. Type field for STAPs and MTAPs........................22 93 5.7.1. Single-Time Aggregation Packet.....................23 94 5.7.2. Multi-Time Aggregation Packets (MTAPs).............26 95 5.7.3. Fragmentation Units (FUs)..........................29 96 6. Packetization Rules..........................................33 97 6.1. Common Packetization Rules..............................33 98 6.2. Single NAL Unit Mode....................................34 99 6.3. Non-Interleaved Mode....................................34 100 6.4. Interleaved Mode........................................35 101 7. De-Packetization Process.....................................35 102 7.1. Single NAL Unit and Non-Interleaved Mode................35 103 7.2. Interleaved Mode........................................36 104 7.2.1. Size of the De-interleaving Buffer.................36 105 7.2.2. De-interleaving Process............................37 106 7.3. Additional De-Packetization Guidelines..................38 107 8. Payload Format Parameters....................................39 108 8.1. Media Type Registration.................................39 109 8.2. SDP Parameters..........................................58 110 8.2.1. Mapping of Payload Type Parameters to SDP..........58 111 8.2.2. Usage with the SDP Offer/Answer Model..............59 112 8.2.3. Usage in Declarative Session Descriptions..........69 113 8.3. Examples................................................70 114 Offer SDP:......................................................76 115 Answer SDP:.....................................................76 116 8.4. Parameter Set Considerations............................77 117 8.5. Decoder Refresh Point Procedure using In-Band Transport of 118 Parameter Sets (Informative).................................80 119 8.5.1. IDR Procedure to Respond to a Request for a Decoder 120 Refresh Point.............................................80 121 8.5.2. Gradual Recovery Procedure to Respond to a Request for 122 a Decoder Refresh Point...................................81 123 9. Security Considerations......................................82 124 10. Congestion Control..........................................82 125 11. IANA Consideration..........................................83 126 12. Informative Appendix: Application Examples..................83 127 12.1. Video Telephony according to ITU-T Recommendation H.241 128 Annex A......................................................84 129 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 130 Aggregation..................................................84 131 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 132 Aggregation..................................................84 133 12.4. Video Telephony with Data Partitioning.................85 134 12.5. Video Telephony or Streaming with FUs and Forward Error 135 Correction...................................................86 136 12.6. Low Bit-Rate Streaming.................................88 137 12.7. Robust Packet Scheduling in Video Streaming............89 138 13. Informative Appendix: Rationale for Decoding Order Number...90 139 13.1. Introduction...........................................90 140 13.2. Example of Multi-Picture Slice Interleaving............90 141 13.3. Example of Robust Packet Scheduling....................92 142 13.4. Robust Transmission Scheduling of Redundant Coded Slices96 143 13.5. Remarks on Other Design Possibilities..................96 144 14. Backward Compatibility to RFC 3984..........................97 145 15. Changes from RFC 3984.......................................99 146 16. Acknowledgements...........................................101 147 17. References.................................................101 148 17.1. Normative References..................................101 149 17.2. Informative References................................102 150 18. Authors' Addresses.........................................104 152 1. Introduction 154 This memo specifies an RTP payload specification for the video 155 coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC 156 International Standard 14496 Part 10 [2] (both also known as 157 Advanced Video Coding, or AVC). In this memo the name H.264 is 158 used for the codec and the standard, but the memo is equally 159 applicable to the ISO/IEC counterpart of the coding standard. 161 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 162 in section 15. Issues on backward compatibility to RFC 3984 are 163 discussed in section 14. 165 1.1. The H.264 Codec 167 The H.264 video codec has a very broad application range that 168 covers all forms of digital compressed video, from low bit-rate 169 Internet streaming applications to HDTV broadcast and Digital 170 Cinema applications with nearly lossless coding. Compared to the 171 current state of technology, the overall performance of H.264 is 172 such that bit rate savings of 50% or more are reported. Digital 173 Satellite TV quality, for example, was reported to be achievable at 174 1.5 Mbit/s, compared to the current operation point of MPEG 2 video 175 at around 3.5 Mbit/s [10]. 177 The codec specification [1] itself distinguishes conceptually 178 between a video coding layer (VCL) and a network abstraction layer 179 (NAL). The VCL contains the signal processing functionality of the 180 codec; mechanisms such as transform, quantization, and motion 181 compensated prediction; and a loop filter. It follows the general 182 concept of most of today's video codecs, a macroblock-based coder 183 that uses inter picture prediction with motion compensation and 184 transform coding of the residual signal. The VCL encoder outputs 185 slices: a bit string that contains the macroblock data of an 186 integer number of macroblocks, and the information of the slice 187 header (containing the spatial address of the first macroblock in 188 the slice, the initial quantization parameter, and similar 189 information). Macroblocks in slices are arranged in scan order 190 unless a different macroblock allocation is specified, by using the 191 so-called Flexible Macroblock Ordering syntax. In-picture 192 prediction is used only within a slice. More information is 193 provided in [10]. 195 The Network Abstraction Layer (NAL) encoder encapsulates the slice 196 output of the VCL encoder into Network Abstraction Layer Units (NAL 197 units), which are suitable for transmission over packet networks or 198 use in packet oriented multiplex environments. Annex B of H.264 199 defines an encapsulation process to transmit such NAL units over 200 byte-stream oriented networks. In the scope of this memo, Annex B 201 is not relevant. 203 Internally, the NAL uses NAL units. A NAL unit consists of a one- 204 byte header and the payload byte string. The header indicates the 205 type of the NAL unit, the (potential) presence of bit errors or 206 syntax violations in the NAL unit payload, and information 207 regarding the relative importance of the NAL unit for the decoding 208 process. This RTP payload specification is designed to be unaware 209 of the bit string in the NAL unit payload. 211 One of the main properties of H.264 is the complete decoupling of 212 the transmission time, the decoding time, and the sampling or 213 presentation time of slices and pictures. The decoding process 214 specified in H.264 is unaware of time, and the H.264 syntax does 215 not carry information such as the number of skipped frames (as is 216 common in the form of the Temporal Reference in earlier video 217 compression standards). Also, there are NAL units that affect many 218 pictures and that are, therefore, inherently timeless. For this 219 reason, the handling of the RTP timestamp requires some special 220 considerations for NAL units for which the sampling or presentation 221 time is not defined or, at transmission time, unknown. 223 1.2. Parameter Set Concept 225 One very fundamental design concept of H.264 is to generate self- 226 contained packets, to make mechanisms such as the header 227 duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension 228 Code (HEC) [12] unnecessary. This was achieved by decoupling 229 information relevant to more than one slice from the media stream. 230 This higher layer meta information should be sent reliably, 231 asynchronously, and in advance from the RTP packet stream that 232 contains the slice packets. (Provisions for sending this 233 information in-band are also available for applications that do not 234 have an out-of-band transport channel appropriate for the purpose.) 235 The combination of the higher-level parameters is called a 236 parameter set. The H.264 specification includes two types of 237 parameter sets: sequence parameter set and picture parameter set. 238 An active sequence parameter set remains unchanged throughout a 239 coded video sequence, and an active picture parameter set remains 240 unchanged within a coded picture. The sequence and picture 241 parameter set structures contain information such as picture size, 242 optional coding modes employed, and macroblock to slice group map. 244 To be able to change picture parameters (such as the picture size) 245 without having to transmit parameter set updates synchronously to 246 the slice packet stream, the encoder and decoder can maintain a 247 list of more than one sequence and picture parameter set. Each 248 slice header contains a codeword that indicates the sequence and 249 picture parameter set to be used. 251 This mechanism allows the decoupling of the transmission of 252 parameter sets from the packet stream, and the transmission of them 253 by external means (e.g., as a side effect of the capability 254 exchange), or through a (reliable or unreliable) control protocol. 255 It may even be possible that they are never transmitted but are 256 fixed by an application design specification. 258 1.3. Network Abstraction Layer Unit Types 260 Tutorial information on the NAL design can be found in [13], [14], 261 and [15]. 263 All NAL units consist of a single NAL unit type octet, which also 264 co-serves as the payload header of this RTP payload format. The 265 payload of a NAL unit follows immediately. 267 The syntax and semantics of the NAL unit type octet are specified 268 in [1], but the essential properties of the NAL unit type octet are 269 summarized below. The NAL unit type octet has the following format: 271 +---------------+ 272 |0|1|2|3|4|5|6|7| 273 +-+-+-+-+-+-+-+-+ 274 |F|NRI| Type | 275 +---------------+ 277 The semantics of the components of the NAL unit type octet, as 278 specified in the H.264 specification, are described briefly below. 280 F: 1 bit 281 forbidden_zero_bit. The H.264 specification declares a value of 282 1 as a syntax violation. 284 NRI: 2 bits 285 nal_ref_idc. A value of 00 indicates that the content of the 286 NAL unit is not used to reconstruct reference pictures for inter 287 picture prediction. Such NAL units can be discarded without 288 risking the integrity of the reference pictures. Values greater 289 than 00 indicate that the decoding of the NAL unit is required 290 to maintain the integrity of the reference pictures. 292 Type: 5 bits 293 nal_unit_type. This component specifies the NAL unit payload 294 type as defined in Table 7-1 of [1], and later within this memo. 295 For a reference of all currently defined NAL unit types and 296 their semantics, please refer to section 7.4.1 in [1]. 298 This memo introduces new NAL unit types, which are presented in 299 section 5.2. The NAL unit types defined in this memo are marked as 300 unspecified in [1]. Moreover, this specification extends the 301 semantics of F and NRI as described in section 5.3. 303 2. Conventions 305 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 306 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 307 this document are to be interpreted as described in RFC 2119 [4]. 309 This specification uses the notion of setting and clearing a bit 310 when bit fields are handled. Setting a bit is the same as 311 assigning that bit the value of 1 (On). Clearing a bit is the same 312 as assigning that bit the value of 0 (Off). 314 3. Scope 316 This payload specification can only be used to carry the "naked" 317 H.264 NAL unit stream over RTP, and not the bitstream format 318 discussed in Annex B of H.264. Likely, the first applications of 319 this specification will be in the conversational multimedia field, 320 video telephony or video conferencing, but the payload format also 321 covers other applications, such as Internet streaming and TV over 322 IP. 324 4. Definitions and Abbreviations 326 4.1. Definitions 328 This document uses the definitions of [1]. The following terms, 329 defined in [1], are summed up for convenience: 331 access unit: A set of NAL units always containing a primary 332 coded picture. In addition to the primary coded picture, an 333 access unit may also contain one or more redundant coded 334 pictures or other NAL units not containing slices or slice data 335 partitions of a coded picture. The decoding of an access unit 336 always results in a decoded picture. 338 coded video sequence: A sequence of access units that consists, 339 in decoding order, of an instantaneous decoding refresh (IDR) 340 access unit followed by zero or more non-IDR access units 341 including all subsequent access units up to but not including 342 any subsequent IDR access unit. 344 IDR access unit: An access unit in which the primary coded 345 picture is an IDR picture. 347 IDR picture: A coded picture containing only slices with I or SI 348 slice types that causes a "reset" in the decoding process. 349 After the decoding of an IDR picture, all following coded 350 pictures in decoding order can be decoded without inter 351 prediction from any picture decoded prior to the IDR picture. 353 primary coded picture: The coded representation of a picture to 354 be used by the decoding process for a bitstream conforming to 355 H.264. The primary coded picture contains all macroblocks of 356 the picture. 358 redundant coded picture: A coded representation of a picture or 359 a part of a picture. The content of a redundant coded picture 360 shall not be used by the decoding process for a bitstream 361 conforming to H.264. The content of a redundant coded picture 362 may be used by the decoding process for a bitstream that 363 contains errors or losses. 365 VCL NAL unit: A collective term used to refer to coded slice and 366 coded data partition NAL units. 368 In addition, the following definitions apply: 370 decoding order number (DON): A field in the payload structure or 371 a derived variable indicating NAL unit decoding order. Values 372 of DON are in the range of 0 to 65535, inclusive. After 373 reaching the maximum value, the value of DON wraps around to 0. 375 NAL unit decoding order: A NAL unit order that conforms to the 376 constraints on NAL unit order given in section 7.4.1.2 in [1]. 378 NALU-time: The value that the RTP timestamp would have if the 379 NAL unit would be transported in its own RTP packet. 381 transmission order: The order of packets in ascending RTP 382 sequence number order (in modulo arithmetic). Within an 383 aggregation packet, the NAL unit transmission order is the same 384 as the order of appearance of NAL units in the packet. 386 media aware network element (MANE): A network element, such as a 387 middlebox or application layer gateway that is capable of 388 parsing certain aspects of the RTP payload headers or the RTP 389 payload and reacting to the contents. 391 Informative note: The concept of a MANE goes beyond normal 392 routers or gateways in that a MANE has to be aware of the 393 signaling (e.g., to learn about the payload type mappings of 394 the media streams), and in that it has to be trusted when 395 working with SRTP. The advantage of using MANEs is that they 396 allow packets to be dropped according to the needs of the 397 media coding. For example, if a MANE has to drop packets due 398 to congestion on a certain link, it can identify and remove 399 those packets whose elimination produces the least adverse 400 effect on the user experience. 402 static macroblock: A certain amount of macroblocks in the video 403 stream can be defined as static, as defined in section 8.3.2.8 404 in [3]. Static macroblocks free up additional processing 405 cycles for the handling of non-static macroblocks. Based on a 406 given amount of video processing resources and a given 407 resolution, a higher number of static macroblocks enables a 408 correspondingly higher frame rate. 410 default sub-profile: The subset of coding tools, which may be 411 all coding tools of one profile or the common subset of coding 412 tools of more than one profile, indicated by the profile-level- 413 id parameter. 415 default level: The level indicated by the profile-level-id 416 parameter, which consists of three octets, profile_idc, profile- 417 iop, and level_idc. The default level is indicated by level_idc 418 in most cases, and, in some cases, additionally by profile-iop. 420 4.2. Abbreviations 422 DON: Decoding Order Number 423 DONB: Decoding Order Number Base 424 DOND: Decoding Order Number Difference 425 FEC: Forward Error Correction 426 FU: Fragmentation Unit 427 IDR: Instantaneous Decoding Refresh 428 IEC: International Electrotechnical Commission 429 ISO: International Organization for Standardization 430 ITU-T: International Telecommunication Union, 431 Telecommunication Standardization Sector 432 MANE: Media Aware Network Element 433 MTAP: Multi-Time Aggregation Packet 434 MTAP16: MTAP with 16-bit timestamp offset 435 MTAP24: MTAP with 24-bit timestamp offset 436 NAL: Network Abstraction Layer 437 NALU: NAL Unit 438 SAR: Sample Aspect Ratio 439 SEI: Supplemental Enhancement Information 440 STAP: Single-Time Aggregation Packet 441 STAP-A: STAP type A 442 STAP-B: STAP type B 443 TS: Timestamp 444 VCL: Video Coding Layer 445 VUI: Video Usability Information 447 5. RTP Payload Format 449 5.1. RTP Header Usage 451 The format of the RTP header is specified in RFC 3550 [5] and 452 reprinted in Figure 1 for convenience. This payload format uses 453 the fields of the header in a manner consistent with that 454 specification. 456 When one NAL unit is encapsulated per RTP packet, the RECOMMENDED 457 RTP payload format is specified in section 5.6. The RTP payload 458 (and the settings for some RTP header bits) for aggregation packets 459 and fragmentation units are specified in sections 5.7 and 5.8, 460 respectively. 462 0 1 2 3 463 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 465 |V=2|P|X| CC |M| PT | sequence number | 466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 467 | timestamp | 468 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 469 | synchronization source (SSRC) identifier | 470 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 471 | contributing source (CSRC) identifiers | 472 | .... | 473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 475 Figure 1 RTP header according to RFC 3550 477 The RTP header information to be set according to this RTP payload 478 format is set as follows: 480 Marker bit (M): 1 bit 481 Set for the very last packet of the access unit indicated by the 482 RTP timestamp, in line with the normal use of the M bit in video 483 formats, to allow an efficient playout buffer handling. For 484 aggregation packets (STAP and MTAP), the marker bit in the RTP 485 header MUST be set to the value that the marker bit of the last 486 NAL unit of the aggregation packet would have been if it were 487 transported in its own RTP packet. Decoders MAY use this bit as 488 an early indication of the last packet of an access unit, but 489 MUST NOT rely on this property. 491 Informative note: Only one M bit is associated with an 492 aggregation packet carrying multiple NAL units. Thus, if a 493 gateway has re-packetized an aggregation packet into several 494 packets, it cannot reliably set the M bit of those packets. 496 Payload type (PT): 7 bits 497 The assignment of an RTP payload type for this new packet format 498 is outside the scope of this document and will not be specified 499 here. The assignment of a payload type has to be performed 500 either through the profile used or in a dynamic way. 502 Sequence number (SN): 16 bits 503 Set and used in accordance with RFC 3550. For the single NALU 504 and non-interleaved packetization mode, the sequence number is 505 used to determine decoding order for the NALU. 507 Timestamp: 32 bits 508 The RTP timestamp is set to the sampling timestamp of the 509 content. A 90 kHz clock rate MUST be used. 511 If the NAL unit has no timing properties of its own (e.g., 512 parameter set and SEI NAL units), the RTP timestamp is set to 513 the RTP timestamp of the primary coded picture of the access 514 unit in which the NAL unit is included, according to section 515 7.4.1.2 of [1]. 517 The setting of the RTP Timestamp for MTAPs is defined in section 518 5.7.2. 520 Receivers SHOULD ignore any picture timing SEI messages included 521 in access units that have only one display timestamp. Instead, 522 receivers SHOULD use the RTP timestamp for synchronizing the 523 display process. 525 If one access unit has more than one display timestamp carried 526 in a picture timing SEI message, then the information in the SEI 527 message SHOULD be treated as relative to the RTP timestamp, with 528 the earliest event occurring at the time given by the RTP 529 timestamp, and subsequent events later, as given by the 530 difference in SEI message picture timing values. Let tSEI1, 531 tSEI2, ..., tSEIn be the display timestamps carried in the SEI 532 message of an access unit, where tSEI1 is the earliest of all 533 such timestamps. Let tmadjst() be a function that adjusts the 534 SEI messages time scale to a 90-kHz time scale. Let TS be the 535 RTP timestamp. Then, the display time for the event associated 536 with tSEI1 is TS. The display time for the event with tSEIx, 537 where x is [2..n] is TS + tmadjst (tSEIx - tSEI1). 539 Informative note: Displaying coded frames as fields is needed 540 commonly in an operation known as 3:2 pulldown, in which film 541 content that consists of coded frames is displayed on a 542 display using interlaced scanning. The picture timing SEI 543 message enables carriage of multiple timestamps for the same 544 coded picture, and therefore the 3:2 pulldown process is 545 perfectly controlled. The picture timing SEI message 546 mechanism is necessary because only one timestamp per coded 547 frame can be conveyed in the RTP timestamp. 549 5.2. Payload Structures 551 The payload format defines three different basic payload structures. 552 A receiver can identify the payload structure by the first byte of 553 the RTP packet payload, which co-serves as the RTP payload header 554 and, in some cases, as the first byte of the payload. This byte is 555 always structured as a NAL unit header. The NAL unit type field 556 indicates which structure is present. The possible structures are 557 as follows: 559 Single NAL Unit Packet: Contains only a single NAL unit in the 560 payload. The NAL header type field will be equal to the original 561 NAL unit type; i.e., in the range of 1 to 23, inclusive. Specified 562 in section 5.6. 564 Aggregation Packet: Packet type used to aggregate multiple NAL 565 units into a single RTP payload. This packet exists in four 566 versions, the Single-Time Aggregation Packet type A (STAP-A), the 567 Single-Time Aggregation Packet type B (STAP-B), Multi-Time 568 Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and Multi- 569 Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The 570 NAL unit type numbers assigned for STAP-A, STAP-B, MTAP16, and 571 MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 572 5.7. 574 Fragmentation Unit: Used to fragment a single NAL unit over 575 multiple RTP packets. Exists with two versions, FU-A and FU-B, 576 identified with the NAL unit type numbers 28 and 29, respectively. 577 Specified in section 5.8. 579 Informative note: This specification does not limit the size of 580 NAL units encapsulated in single NAL unit packets and 581 fragmentation units. The maximum size of a NAL unit 582 encapsulated in any aggregation packet is 65535 bytes. 584 Table 1 summarizes NAL unit types and the corresponding RTP packet 585 types when each of these NAL units is directly used as a packet 586 payload, and where the types are described in this memo. 588 Table 1. Summary of NAL unit types and the corresponding packet 589 types 591 NAL Unit Packet Packet Type Name Section 592 Type Type 593 --------------------------------------------------------- 594 0 reserved - 595 1-23 NAL unit Single NAL unit packet 5.6 596 24 STAP-A Single-time aggregation packet 5.7.1 597 25 STAP-B Single-time aggregation packet 5.7.1 598 26 MTAP16 Multi-time aggregation packet 5.7.2 599 27 MTAP24 Multi-time aggregation packet 5.7.2 600 28 FU-A Fragmentation unit 5.8 601 29 FU-B Fragmentation unit 5.8 602 30-31 reserved - 604 5.3. NAL Unit Header Usage 606 The structure and semantics of the NAL unit header were introduced 607 in section 1.3. For convenience, the format of the NAL unit header 608 is reprinted below: 610 +---------------+ 611 |0|1|2|3|4|5|6|7| 612 +-+-+-+-+-+-+-+-+ 613 |F|NRI| Type | 614 +---------------+ 616 This section specifies the semantics of F and NRI according to this 617 specification. 619 F: 1 bit 620 forbidden_zero_bit. A value of 0 indicates that the NAL unit 621 type octet and payload should not contain bit errors or other 622 syntax violations. A value of 1 indicates that the NAL unit 623 type octet and payload may contain bit errors or other syntax 624 violations. 626 MANEs SHOULD set the F bit to indicate detected bit errors in 627 the NAL unit. The H.264 specification requires that the F bit 628 is equal to 0. When the F bit is set, the decoder is advised 629 that bit errors or any other syntax violations may be present in 630 the payload or in the NAL unit type octet. The simplest decoder 631 reaction to a NAL unit in which the F bit is equal to 1 is to 632 discard such a NAL unit and to conceal the lost data in the 633 discarded NAL unit. 635 NRI: 2 bits 636 nal_ref_idc. The semantics of value 00 and a non-zero value 637 remain unchanged from the H.264 specification. In other words, 638 a value of 00 indicates that the content of the NAL unit is not 639 used to reconstruct reference pictures for inter picture 640 prediction. Such NAL units can be discarded without risking the 641 integrity of the reference pictures. Values greater than 00 642 indicate that the decoding of the NAL unit is required to 643 maintain the integrity of the reference pictures. 645 In addition to the specification above, according to this RTP 646 payload specification, values of NRI indicate the relative 647 transport priority, as determined by the encoder. MANEs can use 648 this information to protect more important NAL units better than 649 they do less important NAL units. The highest transport 650 priority is 11, followed by 10, and then by 01; finally, 00 is 651 the lowest. 653 Informative note: Any non-zero value of NRI is handled 654 identically in H.264 decoders. Therefore, receivers need not 655 manipulate the value of NRI when passing NAL units to the 656 decoder. 658 An H.264 encoder MUST set the value of NRI according to the 659 H.264 specification (subclause 7.4.1) when the value of 660 nal_unit_type is in the range of 1 to 12, inclusive. In 661 particular, the H.264 specification requires that the value of 662 NRI SHALL be equal to 0 for all NAL units having nal_unit_type 663 equal to 6, 9, 10, 11, or 12. 665 For NAL units having nal_unit_type equal to 7 or 8 (indicating a 666 sequence parameter set or a picture parameter set, respectively), 667 an H.264 encoder SHOULD set the value of NRI to 11 (in binary 668 format). For coded slice NAL units of a primary coded picture 669 having nal_unit_type equal to 5 (indicating a coded slice 670 belonging to an IDR picture), an H.264 encoder SHOULD set the 671 value of NRI to 11 (in binary format). 673 For a mapping of the remaining nal_unit_types to NRI values, the 674 following example MAY be used and has been shown to be efficient 675 in a certain environment [14]. Other mappings MAY also be 676 desirable, depending on the application and the H.264/AVC Annex 677 A profile in use. 679 Informative note: Data Partitioning is not available in 680 certain profiles; e.g., in the Main or Baseline profiles. 681 Consequently, the NAL unit types 2, 3, and 4 can occur only 682 if the video bitstream conforms to a profile in which data 683 partitioning is allowed and not in streams that conform to 684 the Main or Baseline profiles. 686 Table 2. Example of NRI values for coded slices and coded slice 687 data partitions of primary coded reference pictures 689 NAL Unit Type Content of NAL unit NRI (binary) 690 ---------------------------------------------------------------- 691 1 non-IDR coded slice 10 692 2 Coded slice data partition A 10 693 3 Coded slice data partition B 01 694 4 Coded slice data partition C 01 696 Informative note: As mentioned before, the NRI value of non- 697 reference pictures is 00 as mandated by H.264/AVC. 699 An H.264 encoder SHOULD set the value of NRI for coded slice and 700 coded slice data partition NAL units of redundant coded 701 reference pictures equal to 01 (in binary format). 703 Definitions of the values for NRI for NAL unit types 24 to 29, 704 inclusive, are given in sections 5.7 and 5.8 of this memo. 706 No recommendation for the value of NRI is given for NAL units 707 having nal_unit_type in the range of 13 to 23, inclusive, 708 because these values are reserved for ITU-T and ISO/IEC. No 709 recommendation for the value of NRI is given for NAL units 710 having nal_unit_type equal to 0 or in the range of 30 to 31, 711 inclusive, as the semantics of these values are not specified in 712 this memo. 714 5.4. Packetization Modes 716 This memo specifies three cases of packetization modes: 718 o Single NAL unit mode 720 o Non-interleaved mode 721 o Interleaved mode 723 The single NAL unit mode is targeted for conversational systems 724 that comply with ITU-T Recommendation H.241 [3] (see section 12.1). 725 The non-interleaved mode is targeted for conversational systems 726 that may not comply with ITU-T Recommendation H.241. In the non- 727 interleaved mode, NAL units are transmitted in NAL unit decoding 728 order. The interleaved mode is targeted for systems that do not 729 require very low end-to-end latency. The interleaved mode allows 730 transmission of NAL units out of NAL unit decoding order. 732 The packetization mode in use MAY be signaled by the value of the 733 OPTIONAL packetization-mode media type parameter. The used 734 packetization mode governs which NAL unit types are allowed in RTP 735 payloads. Table 3 summarizes the allowed packet payload types for 736 each packetization mode. Packetization modes are explained in more 737 detail in section 6. 739 Table 3. Summary of allowed NAL unit types for each packetization 740 mode (yes = allowed, no = disallowed, ig = ignore) 742 Payload Packet Single NAL Non-Interleaved Interleaved 743 Type Type Unit Mode Mode Mode 744 ------------------------------------------------------------- 745 0 reserved ig ig ig 746 1-23 NAL unit yes yes no 747 24 STAP-A no yes no 748 25 STAP-B no no yes 749 26 MTAP16 no no yes 750 27 MTAP24 no no yes 751 28 FU-A no yes yes 752 29 FU-B no no yes 753 30-31 reserved ig ig ig 755 Some NAL unit or payload type values (indicated as reserved in 756 Table 3) are reserved for future extensions. NAL units of those 757 types SHOULD NOT be sent by a sender (direct as packet payloads, or 758 as aggregation units in aggregation packets, or as fragmented units 759 in FU packets) and MUST be ignored by a receiver. For example, the 760 payload types 1-23, with the associated packet type "NAL unit", are 761 allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", 762 but disallowed in "Interleaved Mode". However, NAL units of NAL 763 unit types 1-23 can be used in "Interleaved Mode" as aggregation 764 units in STAP-B, MTAP16 and MTAP24 packets as well as fragmented 765 units in FU-A and FU-B packets. Similarly, NAL units of NAL unit 766 types 1-23 can also be used in the "Non-Interleaved Mode" as 767 aggregation units in STAP-A packets or fragmented units in FU-A 768 packets, in addition to being directly used as packet payloads. 770 5.5. Decoding Order Number (DON) 772 In the interleaved packetization mode, the transmission order of 773 NAL units is allowed to differ from the decoding order of the NAL 774 units. Decoding order number (DON) is a field in the payload 775 structure or a derived variable that indicates the NAL unit 776 decoding order. Rationale and examples of use cases for 777 transmission out of decoding order and for the use of DON are given 778 in section 13. 780 The coupling of transmission and decoding order is controlled by 781 the OPTIONAL sprop-interleaving-depth media type parameter as 782 follows. When the value of the OPTIONAL sprop-interleaving-depth 783 media type parameter is equal to 0 (explicitly or per default), the 784 transmission order of NAL units MUST conform to the NAL unit 785 decoding order. When the value of the OPTIONAL sprop-interleaving- 786 depth media type parameter is greater than 0, 788 o the order of NAL units in an MTAP16 and an MTAP24 is not 789 required to be the NAL unit decoding order, and 791 o the order of NAL units generated by de-packetizing STAP-Bs, 792 MTAPs, and FUs in two consecutive packets is not required to be 793 the NAL unit decoding order. 795 The RTP payload structures for a single NAL unit packet, an STAP-A, 796 and an FU-A do not include DON. STAP-B and FU-B structures include 797 DON, and the structure of MTAPs enables derivation of DON as 798 specified in section 5.7.2. 800 Informative note: When an FU-A occurs in interleaved mode, it 801 always follows an FU-B, which sets its DON. 803 Informative note: If a transmitter wants to encapsulate a single 804 NAL unit per packet and transmit packets out of their decoding 805 order, STAP-B packet type can be used. 807 In the single NAL unit packetization mode, the transmission order 808 of NAL units, determined by the RTP sequence number, MUST be the 809 same as their NAL unit decoding order. In the non-interleaved 810 packetization mode, the transmission order of NAL units in single 811 NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL 812 unit decoding order. The NAL units within an STAP MUST appear in 813 the NAL unit decoding order. Thus, the decoding order is first 814 provided through the implicit order within a STAP, and second 815 provided through the RTP sequence number for the order between 816 STAPs, FUs, and single NAL unit packets. 818 Signaling of the value of DON for NAL units carried in STAP-B, MTAP, 819 and a series of fragmentation units starting with an FU-B is 820 specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON 821 value of the first NAL unit in transmission order MAY be set to any 822 value. Values of DON are in the range of 0 to 65535, inclusive. 823 After reaching the maximum value, the value of DON wraps around to 824 0. 826 The decoding order of two NAL units contained in any STAP-B, MTAP, 827 or a series of fragmentation units starting with an FU-B is 828 determined as follows. Let DON(i) be the decoding order number of 829 the NAL unit having index i in the transmission order. Function 830 don_diff(m,n) is specified as follows: 832 If DON(m) == DON(n), don_diff(m,n) = 0 834 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 835 don_diff(m,n) = DON(n) - DON(m) 837 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 838 don_diff(m,n) = 65536 - DON(m) + DON(n) 840 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 841 don_diff(m,n) = - (DON(m) + 65536 - DON(n)) 843 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 844 don_diff(m,n) = - (DON(m) - DON(n)) 846 A positive value of don_diff(m,n) indicates that the NAL unit 847 having transmission order index n follows, in decoding order, the 848 NAL unit having transmission order index m. When don_diff(m,n) is 849 equal to 0, then the NAL unit decoding order of the two NAL units 850 can be in either order. A negative value of don_diff(m,n) 851 indicates that the NAL unit having transmission order index n 852 precedes, in decoding order, the NAL unit having transmission order 853 index m. 855 Values of DON related fields (DON, DONB, and DOND; see section 5.7) 856 MUST be such that the decoding order determined by the values of 857 DON, as specified above, conforms to the NAL unit decoding order. 859 If the order of two NAL units in NAL unit decoding order is 860 switched and the new order does not conform to the NAL unit 861 decoding order, the NAL units MUST NOT have the same value of DON. 862 If the order of two consecutive NAL units in the NAL unit stream is 863 switched and the new order still conforms to the NAL unit decoding 864 order, the NAL units MAY have the same value of DON. For example, 865 when arbitrary slice order is allowed by the video coding profile 866 in use, all the coded slice NAL units of a coded picture are 867 allowed to have the same value of DON. Consequently, NAL units 868 having the same value of DON can be decoded in any order, and two 869 NAL units having a different value of DON should be passed to the 870 decoder in the order specified above. When two consecutive NAL 871 units in the NAL unit decoding order have a different value of DON, 872 the value of DON for the second NAL unit in decoding order SHOULD 873 be the value of DON for the first, incremented by one. 875 An example of the de-packetization process to recover the NAL unit 876 decoding order is given in section 7. 878 Informative note: Receivers should not expect that the absolute 879 difference of values of DON for two consecutive NAL units in the 880 NAL unit decoding order will be equal to one, even in error-free 881 transmission. An increment by one is not required, as at the 882 time of associating values of DON to NAL units, it may not be 883 known whether all NAL units are delivered to the receiver. For 884 example, a gateway may not forward coded slice NAL units of non- 885 reference pictures or SEI NAL units when there is a shortage of 886 bit rate in the network to which the packets are forwarded. In 887 another example, a live broadcast is interrupted by pre-encoded 888 content, such as commercials, from time to time. The first 889 intra picture of a pre-encoded clip is transmitted in advance to 890 ensure that it is readily available in the receiver. When 891 transmitting the first intra picture, the originator does not 892 exactly know how many NAL units will be encoded before the first 893 intra picture of the pre-encoded clip follows in decoding order. 894 Thus, the values of DON for the NAL units of the first intra 895 picture of the pre-encoded clip have to be estimated when they 896 are transmitted, and gaps in values of DON may occur. 898 5.6. Single NAL Unit Packet 900 The single NAL unit packet defined here MUST contain only one NAL 901 unit, of the types defined in [1]. This means that neither an 902 aggregation packet nor a fragmentation unit can be used within a 903 single NAL unit packet. A NAL unit stream composed by de- 904 packetizing single NAL unit packets in RTP sequence number order 905 MUST conform to the NAL unit decoding order. The structure of the 906 single NAL unit packet is shown in Figure 2. 908 Informative note: The first byte of a NAL unit co-serves as the 909 RTP payload header. 911 0 1 2 3 912 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 913 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 914 |F|NRI| Type | | 915 +-+-+-+-+-+-+-+-+ | 916 | | 917 | Bytes 2..n of a Single NAL unit | 918 | | 919 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 920 | :...OPTIONAL RTP padding | 921 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 923 Figure 2 RTP payload format for single NAL unit packet 925 5.7. Aggregation Packets 927 Aggregation packets are the NAL unit aggregation scheme of this 928 payload specification. The scheme is introduced to reflect the 929 dramatically different MTU sizes of two key target networks: 930 wireline IP networks (with an MTU size that is often limited by the 931 Ethernet MTU size; roughly 1500 bytes), and IP or non-IP (e.g., 932 ITU-T H.324/M) based wireless communication systems with preferred 933 transmission unit sizes of 254 bytes or less. To prevent media 934 transcoding between the two worlds, and to avoid undesirable 935 packetization overhead, a NAL unit aggregation scheme is introduced. 937 Two types of aggregation packets are defined by this specification: 939 o Single-time aggregation packet (STAP): aggregates NAL units with 940 identical NALU-time. Two types of STAPs are defined, one 941 without DON (STAP-A) and another including DON (STAP-B). 943 o Multi-time aggregation packet (MTAP): aggregates NAL units with 944 potentially differing NALU-time. Two different MTAPs are 945 defined, differing in the length of the NAL unit timestamp 946 offset. 948 Each NAL unit to be carried in an aggregation packet is 949 encapsulated in an aggregation unit. Please see below for the four 950 different aggregation units and their characteristics. 952 The structure of the RTP payload format for aggregation packets is 953 presented in Figure 3. 955 0 1 2 3 956 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 957 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 958 |F|NRI| Type | | 959 +-+-+-+-+-+-+-+-+ | 960 | | 961 | one or more aggregation units | 962 | | 963 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 964 | :...OPTIONAL RTP padding | 965 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 967 Figure 3 RTP payload format for aggregation packets 969 MTAPs and STAPs share the following packetization rules: The RTP 970 timestamp MUST be set to the earliest of the NALU-times of all the 971 NAL units to be aggregated. The type field of the NAL unit type 972 octet MUST be set to the appropriate value, as indicated in Table 4. 973 The F bit MUST be cleared if all F bits of the aggregated NAL units 974 are zero; otherwise, it MUST be set. The value of NRI MUST be the 975 maximum of all the NAL units carried in the aggregation packet. 977 Table 4. Type field for STAPs and MTAPs 979 Type Packet Timestamp offset DON related fields 980 field length (DON, DONB, DOND) 981 (in bits) present 982 -------------------------------------------------------- 983 24 STAP-A 0 no 984 25 STAP-B 0 yes 985 26 MTAP16 16 yes 986 27 MTAP24 24 yes 988 The marker bit in the RTP header is set to the value that the 989 marker bit of the last NAL unit of the aggregated packet would have 990 if it were transported in its own RTP packet. 992 The payload of an aggregation packet consists of one or more 993 aggregation units. See sections 5.7.1 and 5.7.2 for the four 994 different types of aggregation units. An aggregation packet can 995 carry as many aggregation units as necessary; however, the total 996 amount of data in an aggregation packet obviously MUST fit into an 997 IP packet, and the size SHOULD be chosen so that the resulting IP 998 packet is smaller than the MTU size. An aggregation packet MUST 999 NOT contain fragmentation units specified in section 5.8. 1000 Aggregation packets MUST NOT be nested; i.e., an aggregation packet 1001 MUST NOT contain another aggregation packet. 1003 5.7.1. Single-Time Aggregation Packet 1005 Single-time aggregation packet (STAP) SHOULD be used whenever NAL 1006 units are aggregated that all share the same NALU-time. The 1007 payload of an STAP-A does not include DON and consists of at least 1008 one single-time aggregation unit, as presented in Figure 4. The 1009 payload of an STAP-B consists of a 16-bit unsigned decoding order 1010 number (DON) (in network byte order) followed by at least one 1011 single-time aggregation unit, as presented in Figure 5. 1013 0 1 2 3 1014 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1015 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1016 : | 1017 +-+-+-+-+-+-+-+-+ | 1018 | | 1019 | single-time aggregation units | 1020 | | 1021 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1022 | : 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1025 Figure 4 Payload format for STAP-A 1027 0 1 2 3 1028 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1029 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1030 : decoding order number (DON) | | 1031 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1032 | | 1033 | single-time aggregation units | 1034 | | 1035 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1036 | : 1037 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1039 Figure 5 Payload format for STAP-B 1041 The DON field specifies the value of DON for the first NAL unit in 1042 an STAP-B in transmission order. For each successive NAL unit in 1043 appearance order in an STAP-B, the value of DON is equal to (the 1044 value of DON of the previous NAL unit in the STAP-B + 1) % 65536, 1045 in which '%' stands for the modulo operation. 1047 A single-time aggregation unit consists of 16-bit unsigned size 1048 information (in network byte order) that indicates the size of the 1049 following NAL unit in bytes (excluding these two octets, but 1050 including the NAL unit type octet of the NAL unit), followed by the 1051 NAL unit itself, including its NAL unit type byte. A single-time 1052 aggregation unit is byte aligned within the RTP payload, but it may 1053 not be aligned on a 32-bit word boundary. Figure 6 presents the 1054 structure of the single-time aggregation unit. 1056 0 1 2 3 1057 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1058 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 : NAL unit size | | 1060 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1061 | | 1062 | NAL unit | 1063 | | 1064 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1065 | : 1066 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1068 Figure 6 Structure for single-time aggregation unit 1070 Figure 7 presents an example of an RTP packet that contains an 1071 STAP-A. The STAP contains two single-time aggregation units, 1072 labeled as 1 and 2 in the figure. 1074 0 1 2 3 1075 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 | RTP Header | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 |STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | NALU 1 Data | 1082 : : 1083 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 | | NALU 2 Size | NALU 2 HDR | 1085 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1086 | NALU 2 Data | 1087 : : 1088 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1089 | :...OPTIONAL RTP padding | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 Figure 7 An example of an RTP packet including an STAP-A containing 1093 two single-time aggregation units 1095 Figure 8 presents an example of an RTP packet that contains an 1096 STAP-B. The STAP contains two single-time aggregation units, 1097 labeled as 1 and 2 in the figure. 1099 0 1 2 3 1100 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1101 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 | RTP Header | 1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 |STAP-B NAL HDR | DON | NALU 1 Size | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1106 | NALU 1 Size | NALU 1 HDR | NALU 1 Data | 1107 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1108 : : 1109 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1110 | | NALU 2 Size | NALU 2 HDR | 1111 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1112 | NALU 2 Data | 1113 : : 1114 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1115 | :...OPTIONAL RTP padding | 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1118 Figure 8 An example of an RTP packet including an STAP-B containing 1119 two single-time aggregation units 1121 5.7.2. Multi-Time Aggregation Packets (MTAPs) 1123 The NAL unit payload of MTAPs consists of a 16-bit unsigned 1124 decoding order number base (DONB) (in network byte order) and one 1125 or more multi-time aggregation units, as presented in Figure 9. 1126 DONB MUST contain the value of DON for the first NAL unit in the 1127 NAL unit decoding order among the NAL units of the MTAP. 1129 Informative note: The first NAL unit in the NAL unit decoding 1130 order is not necessarily the first NAL unit in the order in 1131 which the NAL units are encapsulated in an MTAP. 1133 0 1 2 3 1134 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1135 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1136 : decoding order number base | | 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1138 | | 1139 | multi-time aggregation units | 1140 | | 1141 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1142 | : 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1145 Figure 9 NAL unit payload format for MTAPs 1147 Two different multi-time aggregation units are defined in this 1148 specification. Both of them consist of 16 bits unsigned size 1149 information of the following NAL unit (in network byte order), an 1150 8-bit unsigned decoding order number difference (DOND), and n bits 1151 (in network byte order) of timestamp offset (TS offset) for this 1152 NAL unit, whereby n can be 16 or 24. The choice between the 1153 different MTAP types (MTAP16 and MTAP24) is application dependent: 1154 the larger the timestamp offset is, the higher the flexibility of 1155 the MTAP, but the overhead is also higher. 1157 The structure of the multi-time aggregation units for MTAP16 and 1158 MTAP24 are presented in Figures 10 and 11, respectively. The 1159 starting or ending position of an aggregation unit within a packet 1160 is not required to be on a 32-bit word boundary. The DON of the 1161 NAL unit contained in a multi-time aggregation unit is equal to 1162 (DONB + DOND) % 65536, in which % denotes the modulo operation. 1163 This memo does not specify how the NAL units within an MTAP are 1164 ordered, but, in most cases, NAL unit decoding order SHOULD be used. 1166 The timestamp offset field MUST be set to a value equal to the 1167 value of the following formula: If the NALU-time is larger than or 1168 equal to the RTP timestamp of the packet, then the timestamp offset 1169 equals (the NALU-time of the NAL unit - the RTP timestamp of the 1170 packet). If the NALU-time is smaller than the RTP timestamp of the 1171 packet, then the timestamp offset is equal to the NALU-time + (2^32 1172 - the RTP timestamp of the packet). 1174 0 1 2 3 1175 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1176 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 : NAL unit size | DOND | TS offset | 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1179 | TS offset | | 1180 +-+-+-+-+-+-+-+-+ NAL unit | 1181 | | 1182 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1183 | : 1184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1186 Figure 10 Multi-time aggregation unit for MTAP16 1188 0 1 2 3 1189 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1190 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1191 : NAL unit size | DOND | TS offset | 1192 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1193 | TS offset | | 1194 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1195 | NAL unit | 1196 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1197 | : 1198 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1200 Figure 11 Multi-time aggregation unit for MTAP24 1202 For the "earliest" multi-time aggregation unit in an MTAP the 1203 timestamp offset MUST be zero. Hence, the RTP timestamp of the 1204 MTAP itself is identical to the earliest NALU-time. 1206 Informative note: The "earliest" multi-time aggregation unit is 1207 the one that would have the smallest extended RTP timestamp 1208 among all the aggregation units of an MTAP if the NAL units 1209 contained in the aggregation units were encapsulated in single 1210 NAL unit packets. An extended timestamp is a timestamp that has 1211 more than 32 bits and is capable of counting the wraparound of 1212 the timestamp field, thus enabling one to determine the smallest 1213 value if the timestamp wraps. Such an "earliest" aggregation 1214 unit may not be the first one in the order in which the 1215 aggregation units are encapsulated in an MTAP. The "earliest" 1216 NAL unit need not be the same as the first NAL unit in the NAL 1217 unit decoding order either. 1219 Figure 12 presents an example of an RTP packet that contains a 1220 multi-time aggregation packet of type MTAP16 that contains two 1221 multi-time aggregation units, labeled as 1 and 2 in the figure. 1223 0 1 2 3 1224 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1225 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1226 | RTP Header | 1227 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1228 |MTAP16 NAL HDR | decoding order number base | NALU 1 Size | 1229 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1230 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offset | 1231 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1232 | NALU 1 HDR | NALU 1 DATA | 1233 +-+-+-+-+-+-+-+-+ + 1234 : : 1235 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1236 | | NALU 2 SIZE | NALU 2 DOND | 1237 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1238 | NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA | 1239 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1240 : : 1241 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1242 | :...OPTIONAL RTP padding | 1243 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1245 Figure 12 An RTP packet including a multi-time aggregation packet 1246 of type MTAP16 containing two multi-time aggregation units 1248 Figure 13 presents an example of an RTP packet that contains a 1249 multi-time aggregation packet of type MTAP24 that contains two 1250 multi-time aggregation units, labeled as 1 and 2 in the figure. 1252 0 1 2 3 1253 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1254 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1255 | RTP Header | 1256 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1257 |MTAP24 NAL HDR | decoding order number base | NALU 1 Size | 1258 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1259 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offs | 1260 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1261 |NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA | 1262 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1263 : : 1264 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1265 | | NALU 2 SIZE | NALU 2 DOND | 1266 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1267 | NALU 2 TS offset | NALU 2 HDR | 1268 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1269 | NALU 2 DATA | 1270 : : 1271 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1272 | :...OPTIONAL RTP padding | 1273 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 Figure 13 An RTP packet including a multi-time aggregation packet 1276 of type MTAP24 containing two multi-time aggregation units 1278 5.7.3. Fragmentation Units (FUs) 1280 This payload type allows fragmenting a NAL unit into several RTP 1281 packets. Doing so on the application layer instead of relying on 1282 lower layer fragmentation (e.g., by IP) has the following 1283 advantages: 1285 o The payload format is capable of transporting NAL units bigger 1286 than 64 kbytes over an IPv4 network that may be present in pre- 1287 recorded video, particularly in High Definition formats (there 1288 is a limit of the number of slices per picture, which results in 1289 a limit of NAL units per picture, which may result in big NAL 1290 units). 1292 o The fragmentation mechanism allows fragmenting a single NAL unit 1293 and applying generic forward error correction as described in 1294 section 12.5. 1296 Fragmentation is defined only for a single NAL unit and not for any 1297 aggregation packets. A fragment of a NAL unit consists of an 1298 integer number of consecutive octets of that NAL unit. Each octet 1299 of the NAL unit MUST be part of exactly one fragment of that NAL 1300 unit. Fragments of the same NAL unit MUST be sent in consecutive 1301 order with ascending RTP sequence numbers (with no other RTP 1302 packets within the same RTP packet stream being sent between the 1303 first and last fragment). Similarly, a NAL unit MUST be 1304 reassembled in RTP sequence number order. 1306 When a NAL unit is fragmented and conveyed within fragmentation 1307 units (FUs), it is referred to as a fragmented NAL unit. STAPs and 1308 MTAPs MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU 1309 MUST NOT contain another FU. 1311 The RTP timestamp of an RTP packet carrying an FU is set to the 1312 NALU-time of the fragmented NAL unit. 1314 Figure 14 presents the RTP payload format for FU-As. An FU-A 1315 consists of a fragmentation unit indicator of one octet, a 1316 fragmentation unit header of one octet, and a fragmentation unit 1317 payload. 1319 0 1 2 3 1320 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1322 | FU indicator | FU header | | 1323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1324 | | 1325 | FU payload | 1326 | | 1327 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1328 | :...OPTIONAL RTP padding | 1329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1331 Figure 14 RTP payload format for FU-A 1333 Figure 15 presents the RTP payload format for FU-Bs. An FU-B 1334 consists of a fragmentation unit indicator of one octet, a 1335 fragmentation unit header of one octet, a decoding order number 1336 (DON) (in network byte order), and a fragmentation unit payload. 1337 In other words, the structure of FU-B is the same as the structure 1338 of FU-A, except for the additional DON field. 1340 0 1 2 3 1341 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1342 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1343 | FU indicator | FU header | DON | 1344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 1345 | | 1346 | FU payload | 1347 | | 1348 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1349 | :...OPTIONAL RTP padding | 1350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1352 Figure 15 RTP payload format for FU-B 1354 NAL unit type FU-B MUST be used in the interleaved packetization 1355 mode for the first fragmentation unit of a fragmented NAL unit. 1356 NAL unit type FU-B MUST NOT be used in any other case. In other 1357 words, in the interleaved packetization mode, each NALU that is 1358 fragmented has an FU-B as the first fragment, followed by one or 1359 more FU-A fragments. 1361 The FU indicator octet has the following format: 1363 +---------------+ 1364 |0|1|2|3|4|5|6|7| 1365 +-+-+-+-+-+-+-+-+ 1366 |F|NRI| Type | 1367 +---------------+ 1369 Values equal to 28 and 29 in the Type field of the FU indicator 1370 octet identify an FU-A and an FU-B, respectively. The use of the F 1371 bit is described in section 5.3. The value of the NRI field MUST 1372 be set according to the value of the NRI field in the fragmented 1373 NAL unit. 1375 The FU header has the following format: 1377 +---------------+ 1378 |0|1|2|3|4|5|6|7| 1379 +-+-+-+-+-+-+-+-+ 1380 |S|E|R| Type | 1381 +---------------+ 1383 S: 1 bit 1384 When set to one, the Start bit indicates the start of a 1385 fragmented NAL unit. When the following FU payload is not the 1386 start of a fragmented NAL unit payload, the Start bit is set to 1387 zero. 1389 E: 1 bit 1390 When set to one, the End bit indicates the end of a fragmented 1391 NAL unit, i.e., the last byte of the payload is also the last 1392 byte of the fragmented NAL unit. When the following FU payload 1393 is not the last fragment of a fragmented NAL unit, the End bit 1394 is set to zero. 1396 R: 1 bit 1397 The Reserved bit MUST be equal to 0 and MUST be ignored by the 1398 receiver. 1400 Type: 5 bits 1401 The NAL unit payload type as defined in Table 7-1 of [1]. 1403 The value of DON in FU-Bs is selected as described in section 5.5. 1405 Informative note: The DON field in FU-Bs allows gateways to 1406 fragment NAL units to FU-Bs without organizing the incoming NAL 1407 units to the NAL unit decoding order. 1409 A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the 1410 Start bit and End bit MUST NOT both be set to one in the same FU 1411 header. 1413 The FU payload consists of fragments of the payload of the 1414 fragmented NAL unit so that if the fragmentation unit payloads of 1415 consecutive FUs are sequentially concatenated, the payload of the 1416 fragmented NAL unit can be reconstructed. The NAL unit type octet 1417 of the fragmented NAL unit is not included as such in the 1418 fragmentation unit payload, but rather the information of the NAL 1419 unit type octet of the fragmented NAL unit is conveyed in F and NRI 1420 fields of the FU indicator octet of the fragmentation unit and in 1421 the type field of the FU header. An FU payload MAY have any number 1422 of octets and MAY be empty. 1424 Informative note: Empty FUs are allowed to reduce the latency of 1425 a certain class of senders in nearly lossless environments. 1426 These senders can be characterized in that they packetize NALU 1427 fragments before the NALU is completely generated and, hence, 1428 before the NALU size is known. If zero-length NALU fragments 1429 were not allowed, the sender would have to generate at least one 1430 bit of data of the following fragment before the current 1431 fragment could be sent. Due to the characteristics of H.264, 1432 where sometimes several macroblocks occupy zero bits, this is 1433 undesirable and can add delay. However, the (potential) use of 1434 zero-length NALU fragments should be carefully weighed against 1435 the increased risk of the loss of at least a part of the NALU 1436 because of the additional packets employed for its transmission. 1438 If a fragmentation unit is lost, the receiver SHOULD discard all 1439 following fragmentation units in transmission order corresponding 1440 to the same fragmented NAL unit. 1442 A receiver in an endpoint or in a MANE MAY aggregate the first n-1 1443 fragments of a NAL unit to an (incomplete) NAL unit, even if 1444 fragment n of that NAL unit is not received. In this case, the 1445 forbidden_zero_bit of the NAL unit MUST be set to one to indicate a 1446 syntax violation. 1448 6. Packetization Rules 1450 The packetization modes are introduced in section 5.2. The 1451 packetization rules common to more than one of the packetization 1452 modes are specified in section 6.1. The packetization rules for 1453 the single NAL unit mode, the non-interleaved mode, and the 1454 interleaved mode are specified in sections 6.2, 6.3, and 6.4, 1455 respectively. 1457 6.1. Common Packetization Rules 1459 All senders MUST enforce the following packetization rules 1460 regardless of the packetization mode in use: 1462 o Coded slice NAL units or coded slice data partition NAL units 1463 belonging to the same coded picture (and thus sharing the same 1464 RTP timestamp value) MAY be sent in any order; however, for 1465 delay-critical systems, they SHOULD be sent in their original 1466 decoding order to minimize the delay. Note that the decoding 1467 order is the order of the NAL units in the bitstream. 1469 o Parameter sets are handled in accordance with the rules and 1470 recommendations given in section 8.4. 1472 o MANEs MUST NOT duplicate any NAL unit except for sequence or 1473 picture parameter set NAL units, as neither this memo nor the 1474 H.264 specification provides means to identify duplicated NAL 1475 units. Sequence and picture parameter set NAL units MAY be 1476 duplicated to make their correct reception more probable, but 1477 any such duplication MUST NOT affect the contents of any active 1478 sequence or picture parameter set. Duplication SHOULD be 1479 performed on the application layer and not by duplicating RTP 1480 packets (with identical sequence numbers). 1482 Senders using the non-interleaved mode and the interleaved mode 1483 MUST enforce the following packetization rule: 1485 o MANEs MAY convert single NAL unit packets into one aggregation 1486 packet, convert an aggregation packet into several single NAL 1487 unit packets, or mix both concepts, in an RTP translator. The 1488 RTP translator SHOULD take into account at least the following 1489 parameters: path MTU size, unequal protection mechanisms (e.g., 1490 through packet-based FEC according to RFC 2733 [18], especially 1491 for sequence and picture parameter set NAL units and coded slice 1492 data partition A NAL units), bearable latency of the system, and 1493 buffering capabilities of the receiver. 1495 Informative note: An RTP translator is required to handle 1496 RTCP as per RFC 3550. 1498 6.2. Single NAL Unit Mode 1500 This mode is in use when the value of the OPTIONAL packetization- 1501 mode media type parameter is equal to 0 or the packetization-mode 1502 is not present. All receivers MUST support this mode. It is 1503 primarily intended for low-delay applications that are compatible 1504 with systems using ITU-T Recommendation H.241 [3] (see section 1505 12.1). Only single NAL unit packets MAY be used in this mode. 1506 STAPs, MTAPs, and FUs MUST NOT be used. The transmission order of 1507 single NAL unit packets MUST comply with the NAL unit decoding 1508 order. 1510 6.3. Non-Interleaved Mode 1512 This mode is in use when the value of the OPTIONAL packetization- 1513 mode media type parameter is equal to 1. This mode SHOULD be 1514 supported. It is primarily intended for low-delay applications. 1515 Only single NAL unit packets, STAP-As, and FU-As MAY be used in 1516 this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. The 1517 transmission order of NAL units MUST comply with the NAL unit 1518 decoding order. 1520 6.4. Interleaved Mode 1522 This mode is in use when the value of the OPTIONAL packetization- 1523 mode media type parameter is equal to 2. Some receivers MAY 1524 support this mode. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. 1525 STAP-As and single NAL unit packets MUST NOT be used. The 1526 transmission order of packets and NAL units is constrained as 1527 specified in section 5.5. 1529 7. De-Packetization Process 1531 The de-packetization process is implementation dependent. 1532 Therefore, the following description should be seen as an example 1533 of a suitable implementation. Other schemes may be used as well as 1534 long as the output for the same input is the same as the process 1535 described below. The same output means that the resulting NAL 1536 units, and their order, are identical. Optimizations relative to 1537 the described algorithms are likely possible. Section 7.1 presents 1538 the de-packetization process for the single NAL unit and non- 1539 interleaved packetization modes, whereas section 7.2 describes the 1540 process for the interleaved mode. Section 7.3 includes additional 1541 de-packetization guidelines for intelligent receivers. 1543 All normal RTP mechanisms related to buffer management apply. In 1544 particular, duplicated or outdated RTP packets (as indicated by the 1545 RTP sequence number and the RTP timestamp) are removed. To 1546 determine the exact time for decoding, factors such as a possible 1547 intentional delay to allow for proper inter-stream synchronization 1548 must be factored in. 1550 7.1. Single NAL Unit and Non-Interleaved Mode 1552 The receiver includes a receiver buffer to compensate for 1553 transmission delay jitter. The receiver stores incoming packets in 1554 reception order into the receiver buffer. Packets are de- 1555 packetized in RTP sequence number order. If a de-packetized packet 1556 is a single NAL unit packet, the NAL unit contained in the packet 1557 is passed directly to the decoder. If a de-packetized packet is an 1558 STAP-A, the NAL units contained in the packet are passed to the 1559 decoder in the order in which they are encapsulated in the packet. 1560 For all the FU-A packets containing fragments of a single NAL unit, 1561 the de-packetized fragments are concatenated in their sending order 1562 to recover the NAL unit, which is then passed to the decoder. 1564 Informative note: If the decoder supports Arbitrary Slice Order, 1565 coded slices of a picture can be passed to the decoder in any 1566 order regardless of their reception and transmission order. 1568 7.2. Interleaved Mode 1570 The general concept behind these de-packetization rules is to 1571 reorder NAL units from transmission order to the NAL unit decoding 1572 order. 1574 The receiver includes a receiver buffer, which is used to 1575 compensate for transmission delay jitter and to reorder NAL units 1576 from transmission order to the NAL unit decoding order. In this 1577 section, the receiver operation is described under the assumption 1578 that there is no transmission delay jitter. To make a difference 1579 from a practical receiver buffer that is also used for compensation 1580 of transmission delay jitter, the receiver buffer is here after 1581 called the de-interleaving buffer in this section. Receivers 1582 SHOULD also prepare for transmission delay jitter; i.e., either 1583 reserve separate buffers for transmission delay jitter buffering 1584 and de-interleaving buffering or use a receiver buffer for both 1585 transmission delay jitter and de-interleaving. Moreover, receivers 1586 SHOULD take transmission delay jitter into account in the buffering 1587 operation; e.g., by additional initial buffering before starting of 1588 decoding and playback. 1590 This section is organized as follows: subsection 7.2.1 presents how 1591 to calculate the size of the de-interleaving buffer. Subsection 1592 7.2.2 specifies the receiver process on how to organize received 1593 NAL units to the NAL unit decoding order. 1595 7.2.1. Size of the De-interleaving Buffer 1597 In either Offer/Answer or declarative SDP usage, the sprop-deint- 1598 buf-req media type parameter signals the requirement for the de- 1599 interleaving buffer size. It is therefore RECOMMENDED to set the 1600 de-interleaving buffer size, in terms of number of bytes, equal to 1601 or greater than the value of sprop-deint-buf-req media type 1602 parameter. 1604 When the SDP Offer/Answer model or any other capability exchange 1605 procedure is used in session setup, the properties of the received 1606 stream SHOULD be such that the receiver capabilities are not 1607 exceeded. In the SDP Offer/Answer model, the receiver can indicate 1608 its capabilities to allocate a de-interleaving buffer with the 1609 deint-buf-cap media type parameter. See section 8.1 for further 1610 information on deint-buf-cap and sprop-deint-buf-req media type 1611 parameters and section 8.2.2 for further information on their use 1612 in the SDP Offer/Answer model. 1614 7.2.2. De-interleaving Process 1616 There are two buffering states in the receiver: initial buffering 1617 and buffering while playing. Initial buffering occurs when the RTP 1618 session is initialized. After initial buffering, decoding and 1619 playback are started, and the buffering-while-playing mode is used. 1621 Regardless of the buffering state, the receiver stores incoming NAL 1622 units, in reception order, in the de-interleaving buffer as follows. 1623 NAL units of aggregation packets are stored in the de-interleaving 1624 buffer individually. The value of DON is calculated and stored for 1625 each NAL unit. 1627 The receiver operation is described below with the help of the 1628 following functions and constants: 1630 o Function AbsDON is specified in section 8.1. 1632 o Function don_diff is specified in section 5.5. 1634 o Constant N is the value of the OPTIONAL sprop-interleaving-depth 1635 media type parameter (see section 8.1) incremented by 1. 1637 Initial buffering lasts until one of the following conditions is 1638 fulfilled: 1640 o There are N or more VCL NAL units in the de-interleaving buffer. 1642 o If sprop-max-don-diff is present, don_diff(m,n) is greater than 1643 the value of sprop-max-don-diff, in which n corresponds to the 1644 NAL unit having the greatest value of AbsDON among the received 1645 NAL units and m corresponds to the NAL unit having the smallest 1646 value of AbsDON among the received NAL units. 1648 o Initial buffering has lasted for the duration equal to or 1649 greater than the value of the OPTIONAL sprop-init-buf-time media 1650 type parameter. 1652 The NAL units to be removed from the de-interleaving buffer are 1653 determined as follows: 1655 o If the de-interleaving buffer contains at least N VCL NAL units, 1656 NAL units are removed from the de-interleaving buffer and passed 1657 to the decoder in the order specified below until the buffer 1658 contains N-1 VCL NAL units. 1660 o If sprop-max-don-diff is present, all NAL units m for which 1661 don_diff(m,n) is greater than sprop-max-don-diff are removed 1662 from the de-interleaving buffer and passed to the decoder in the 1663 order specified below. Herein, n corresponds to the NAL unit 1664 having the greatest value of AbsDON among the NAL units in the 1665 de-interleaving buffer. 1667 The order in which NAL units are passed to the decoder is specified 1668 as follows: 1670 o Let PDON be a variable that is initialized to 0 at the beginning 1671 of the RTP session. 1673 o For each NAL unit associated with a value of DON, a DON distance 1674 is calculated as follows. If the value of DON of the NAL unit 1675 is larger than the value of PDON, the DON distance is equal to 1676 DON - PDON. Otherwise, the DON distance is equal to 65535 - 1677 PDON + DON + 1. 1679 o NAL units are delivered to the decoder in ascending order of DON 1680 distance. If several NAL units share the same value of DON 1681 distance, they can be passed to the decoder in any order. 1683 o When a desired number of NAL units have been passed to the 1684 decoder, the value of PDON is set to the value of DON for the 1685 last NAL unit passed to the decoder. 1687 7.3. Additional De-Packetization Guidelines 1689 The following additional de-packetization rules may be used to 1690 implement an operational H.264 de-packetizer: 1692 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1693 coded slice data partitions A (DPAs). If a lost DPA is detected, 1694 after taking into account possible retransmission and FEC, a 1695 gateway may decide not to send the corresponding coded slice 1696 data partitions B and C, as their information is meaningless for 1697 H.264 decoders. In this way a MANE can reduce network load by 1698 discarding useless packets without parsing a complex bitstream. 1700 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1701 FUs. If a lost FU is found, a gateway may decide not to send 1702 the following FUs of the same fragmented NAL unit, as their 1703 information is meaningless for H.264 decoders. In this way a 1704 MANE can reduce network load by discarding useless packets 1705 without parsing a complex bitstream. 1707 o Intelligent receivers having to discard packets or NALUs should 1708 first discard all packets/NALUs in which the value of the NRI 1709 field of the NAL unit type octet is equal to 0. This will 1710 minimize the impact on user experience and keep the reference 1711 pictures intact. If more packets have to be discarded, then 1712 packets with a numerically lower NRI value should be discarded 1713 before packets with a numerically higher NRI value. However, 1714 discarding any packets with an NRI bigger than 0 very likely 1715 leads to decoder drift and SHOULD be avoided. 1717 8. Payload Format Parameters 1719 This section specifies the parameters that MAY be used to select 1720 optional features of the payload format and certain features of the 1721 bitstream. The parameters are specified here as part of the media 1722 subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. 1723 A mapping of the parameters into the Session Description Protocol 1724 (SDP) [6] is also provided for applications that use SDP. 1725 Equivalent parameters could be defined elsewhere for use with 1726 control protocols that do not use SDP. 1728 Some parameters provide a receiver with the properties of the 1729 stream that will be sent. The names of all these parameters start 1730 with "sprop" for stream properties. Some of these "sprop" 1731 parameters are limited by other payload or codec configuration 1732 parameters. For example, the sprop-parameter-sets parameter is 1733 constrained by the profile-level-id parameter. 1735 8.1. Media Type Registration 1737 The media subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is 1738 allocated from the IETF tree. 1740 Media Type name: video 1742 Media subtype name: H264 1744 Required parameters: none 1745 OPTIONAL parameters: 1747 profile-level-id: 1748 A base16 [7] (hexadecimal) representation of the following 1749 three bytes in the sequence parameter set NAL unit specified 1750 in [1]: 1) profile_idc, 2) a byte herein referred to as 1751 profile-iop, composed of the values of constraint_set0_flag, 1752 constraint_set1_flag,constraint_set2_flag, 1753 constraint_set3_flag, and reserved_zero_4bits in bit- 1754 significance order, starting from the most significant bit, 1755 and 3) level_idc. Note that reserved_zero_4bits is required 1756 to be equal to 0 in [1], but other values for it may be 1757 specified in the future by ITU-T or ISO/IEC. 1759 The profile-level-id parameter indicates the default sub- 1760 profile, i.e. the subset of coding tools that may have been 1761 used to generate the stream or that the receiver supports, 1762 and the default level of the stream or the receiver supports. 1764 The default sub-profile is indicated collectively by the 1765 profile_idc byte and some fields in the profile-iop byte. 1766 Depending on the values of the fields in the profile-iop byte, 1767 the default sub-profile may be the set of coding tools 1768 supported by one profile, or a common subset of coding tools 1769 of multiple profiles, as specified in subsection 7.4.2.1.1 of 1770 [1]. The default level is indicated by the level_idc byte, 1771 and, when profile_idc is equal to 66, 77 or 88 (the Baseline, 1772 Main, or Extended profile) and level_idc is equal to 11, 1773 additionally by bit 4 (constraint_set3_flag) of the profile- 1774 iop byte. When profile_idc is equal to 66, 77 or 88 (the 1775 Baseline, Main, or Extended profile) and level_idc is equal 1776 to 11, and bit 4 (constraint_set3_flag) of the profile-iop 1777 byte is equal to 1, the default level is level 1b. 1779 Table 5 lists all profiles defined in Annex A of [1] and, for 1780 each of the profiles, the possible combinations of 1781 profile_idc and profile-iop that represent the same sub- 1782 profile. 1784 Table 5. Combinations of profile_idc and profile-iop 1785 representing the same sub-profile corresponding to the 1786 full set of coding tools supported by one profile. In 1787 the following, x may be either 0 or 1, while the profile 1788 names are indicated as follows. CB: Constrained Baseline 1789 profile, B: Baseline profile, M: Main profile, E: 1790 Extended profile, H: High profile, H10: High 10 profile, 1791 H42: High 4:2:2 profile, H44: High 4:4:4 Predictive 1792 profile, H10I: High 10 Intra profile, H42I: High 4:2:2 1793 Intra profile, H44I: High 4:4:4 Intra profile, and C44I: 1794 CAVLC 4:4:4 Intra profile. 1796 Profile profile_idc profile-iop 1797 (hexadecimal) (binary) 1799 CB 42 (B) x1xx0000 1800 same as: 4D (M) 1xxx0000 1801 same as: 58 (E) 11xx0000 1802 same as: 64 (H), 6E (H10), 1xx00000 1803 7A (H42), or F4 (H44) 1804 B 42 (B) x0xx0000 1805 same as: 58 (E) 10xx0000 1806 M 4D (M) 0x0x0000 1807 same as: 64 (H), 6E (H10), 01000000 1808 7A (H42), or F4 (H44) 1809 E 58 00xx0000 1810 H 64 00000000 1811 H10 6E 00000000 1812 H42 7A 00000000 1813 H44 F4 00000000 1814 H10I 64 00010000 1815 H42I 7A 00010000 1816 H44I F4 00010000 1817 C44I 2C 00010000 1819 For example, in the table above, profile_idc equal to 58 1820 (Extended) with profile-iop equal to 11xx0000 indicates the 1821 same sub-profile corresponding to profile_idc equal to 42 1822 (Baseline) with profile-iop equal to x1xx0000. Note that 1823 other combinations of profile_idc and profile-iop (not listed 1824 in Table 5) may represent a sub-profile equivalent to the 1825 common subset of coding tools for more than one profile. 1826 Note also that a decoder conforming to a certain profile may 1827 be able to decode bitstreams conforming to other profiles. 1828 For example, a decoder conforming to the High 4:4:4 profile 1829 at certain level must be able to decode bitstreams conforming 1830 to the Constrained Baseline, Main, High, High 10 or High 1831 4:2:2 profile at the same or a lower level. 1833 If the profile-level-id parameter is used to indicate 1834 properties of a NAL unit stream, it indicates that, to decode 1835 the stream, the minimum subset of coding tools a decoder has 1836 to support is the default sub-profile, and the lowest level 1837 the decoder has to support is the default level. 1839 If the profile-level-id parameter is used for capability 1840 exchange or session setup procedure, it indicates the subset 1841 of coding tools, which is equal to the default sub-profile, 1842 that the codec supports for both receiving and sending. If 1843 max-recv-level is not present, the default level from 1844 profile-level-id indicates the highest level the codec wishes 1845 to support. If max-recv-level is present it indicates the 1846 highest level the codec supports for receiving. For either 1847 receiving or sending, all levels that are lower than the 1848 highest level supported MUST also be supported. 1850 Informative note: Capability exchange and session setup 1851 procedures should provide means to list the capabilities 1852 for each supported sub-profile separately. For example, 1853 the one-of-N codec selection procedure of the SDP 1854 Offer/Answer model can be used (section 10.2 of [8]). 1855 The one-of-N codec selection procedure may also be used 1856 to provide different combinations of profile_idc and 1857 profile-iop that represent the same sub-profile. When 1858 there are many different combinations of profile_idc and 1859 profile-iop that represent the same sub-profile, using 1860 the one-of-N codec selection procedure may result into a 1861 fairly large SDP message. Therefore, a receiver should 1862 understand the different equivalent combinations of 1863 profile_idc and profile-iop that represent the same sub- 1864 profile, and be ready to accept an offer using any of the 1865 equivalent combinations. 1867 If no profile-level-id is present, the Baseline Profile 1868 without additional constraints at Level 1 MUST be inferred. 1870 max-recv-level: 1871 This parameter MAY be used to indicate the highest level a 1872 receiver supports when the highest level is higher than the 1873 default level (the level indicated by profile-level-id). The 1874 value of max-recv-level is a base16 (hexadecimal) 1875 representation of the two bytes after the syntax element 1876 profile_idc in the sequence parameter set NAL unit specified 1877 in [1]: profile-iop (as defined above) and level_idc. If 1878 (the level_idc byte of max-recv-level is equal to 11 and bit 1879 4 of the profile-iop byte of max-recv-level is equal to 1) or 1880 (the level_idc byte of max-recv-level is equal to 9 and bit 4 1881 of the profile-iop byte of max-recv-level is equal to 0), the 1882 highest level the receiver supports is level 1b. Otherwise, 1883 the highest level the receiver supports is equal to the 1884 level_idc byte of max-recv-level divided by 10. 1886 max-recv-level MUST NOT be present if the highest level the 1887 receiver supports is not higher than the default level. 1889 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: 1890 These parameters MAY be used to signal the capabilities of a 1891 receiver implementation. These parameters MUST NOT be used 1892 for any other purpose. The highest level conveyed in the 1893 value of the profile-level-id parameter or the max-recv-level 1894 parameter MUST be such that the receiver is fully capable of 1895 supporting. max-mbps, max-smbps, max-fs, max-cpb, max-dpb, 1896 and max-br MAY be used to indicate capabilities of the 1897 receiver that extend the required capabilities of the 1898 signaled highest level, as specified below. 1900 When more than one parameter from the set (max-mbps, max- 1901 smbps , max-fs, max-cpb, max-dpb, max-br) is present, the 1902 receiver MUST support all signaled capabilities 1903 simultaneously. For example, if both max-mbps and max-br are 1904 present, the signaled highest level with the extension of 1905 both the frame rate and bit rate is supported. That is, the 1906 receiver is able to decode NAL unit streams in which the 1907 macroblock processing rate is up to max-mbps (inclusive), the 1908 bit rate is up to max-br (inclusive), the coded picture 1909 buffer size is derived as specified in the semantics of the 1910 max-br parameter below, and other properties comply with the 1911 highest level specified in the value of the profile-level-id 1912 parameter or the max-recv-level parameter. 1914 If a receiver can support all the properties of level A, the 1915 highest level specified in the value of the profile-level-id 1916 parameter or the max-recv-level parameter MUST be level A 1917 (i.e. MUST NOT be lower than level A). In other words, a 1918 receiver MUST NOT signal values of max-mbps, max-fs, max-cpb, 1919 max-dpb, and max-br that taken together meet the requirements 1920 of a higher level compared to the highest level specified in 1921 the value of the profile-level-id parameter or the max-recv- 1922 level parameter. 1924 Informative note: When the OPTIONAL media type parameters 1925 are used to signal the properties of a NAL unit stream, 1926 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br 1927 are not present, and the value of profile-level-id must 1928 always be such that the NAL unit stream complies fully 1929 with the specified profile and level. 1931 max-mbps: The value of max-mbps is an integer indicating the 1932 maximum macroblock processing rate in units of macroblocks 1933 per second. The max-mbps parameter signals that the receiver 1934 is capable of decoding video at a higher rate than is 1935 required by the signaled highest level conveyed in the value 1936 of the profile-level-id parameter or the max-recv-level 1937 parameter. When max-mbps is signaled, the receiver MUST be 1938 able to decode NAL unit streams that conform to the signaled 1939 highest level, with the exception that the MaxMBPS value in 1940 Table A-1 of [1] for the signaled highest level is replaced 1941 with the value of max-mbps. The value of max-mbps MUST be 1942 greater than or equal to the value of MaxMBPS given in Table 1943 A-1 of [1] for the highest level. Senders MAY use this 1944 knowledge to send pictures of a given size at a higher 1945 picture rate than is indicated in the signaled highest level. 1947 max-smbps: The value of max-smbps is an integer indicating the 1948 maximum static macroblock processing rate in units of static 1949 macroblocks per second, under the hypothetical assumption 1950 that all macroblocks are static macroblocks. When max-smbps 1951 is signaled the MaxMBPS value in Table A-1 of [1] should be 1952 replaced with the result of the following computation: 1954 o If the parameter max-mbps is signaled, set a variable 1955 MaxMacroblocksPerSecond to the value of max-mbps. 1956 Otherwise, set MaxMacroblocksPerSecond equal to the value 1957 of MaxMBPS in Table A-1 [1] for the signaled highest level 1958 conveyed in the value of the profile-level-id parameter or 1959 the max-recv-level parameter. 1961 o Set a variable P_non-static to the proportion of non- 1962 static macroblocks in picture n. 1964 o Set a variable P_static to the proportion of static 1965 macroblocks in picture n. 1967 o The value of MaxMBPS in Table A-1 of [1] should be 1968 considered by the encoder to be equal to: 1970 MaxMacroblocksPerSecond * max-smbps / (P_non-static * 1971 max-smbps + P_static * MaxMacroblocksPerSecond) 1973 The encoder should recompute this value for each picture. The 1974 value of max-smbps MUST be greater than or equal to the value 1975 of MaxMBPS given explicitly as the value of the max-mbps 1976 parameter or implicitly in Table A-1 of [1] for the signaled 1977 highest level. Senders MAY use this knowledge to send 1978 pictures of a given size at a higher picture rate than is 1979 indicated in the signaled highest level. 1981 max-fs: The value of max-fs is an integer indicating the maximum 1982 frame size in units of macroblocks. The max-fs parameter 1983 signals that the receiver is capable of decoding larger 1984 picture sizes than are required by the signaled highest level 1985 conveyed in the value of the profile-level-id parameter or 1986 the max-recv-level parameter. When max-fs is signaled, the 1987 receiver MUST be able to decode NAL unit streams that conform 1988 to the signaled highest level, with the exception that the 1989 MaxFS value in Table A-1 of [1] for the signaled highest 1990 level is replaced with the value of max-fs. The value of 1991 max-fs MUST be greater than or equal to the value of MaxFS 1992 given in Table A-1 of [1] for the highest level. Senders MAY 1993 use this knowledge to send larger pictures at a 1994 proportionally lower frame rate than is indicated in the 1995 signaled highest level. 1997 max-cpb: The value of max-cpb is an integer indicating the 1998 maximum coded picture buffer size in units of 1000 bits for 1999 the VCL HRD parameters (see A.3.1 item i of [1]) and in units 2000 of 1200 bits for the NAL HRD parameters (see A.3.1 item j of 2001 [1]). The max-cpb parameter signals that the receiver has 2002 more memory than the minimum amount of coded picture buffer 2003 memory required by the signaled highest level conveyed in the 2004 value of the profile-level-id parameter or the max-recv-level 2005 parameter. When max-cpb is signaled, the receiver MUST be 2006 able to decode NAL unit streams that conform to the signaled 2007 highest level, with the exception that the MaxCPB value in 2008 Table A-1 of [1] for the signaled highest level is replaced 2009 with the value of max-cpb. The value of max-cpb MUST be 2010 greater than or equal to the value of MaxCPB given in Table 2011 A-1 of [1] for the highest level. Senders MAY use this 2012 knowledge to construct coded video streams with greater 2013 variation of bit rate than can be achieved with the MaxCPB 2014 value in Table A-1 of [1]. 2016 Informative note: The coded picture buffer is used in the 2017 hypothetical reference decoder (Annex C) of H.264. The 2018 use of the hypothetical reference decoder is recommended 2019 in H.264 encoders to verify that the produced bitstream 2020 conforms to the standard and to control the output 2021 bitrate. Thus, the coded picture buffer is conceptually 2022 independent of any other potential buffers in the 2023 receiver, including de-interleaving and de-jitter buffers. 2024 The coded picture buffer need not be implemented in 2025 decoders as specified in Annex C of H.264, but rather 2026 standard-compliant decoders can have any buffering 2027 arrangements provided that they can decode standard- 2028 compliant bitstreams. Thus, in practice, the input 2029 buffer for video decoder can be integrated with de- 2030 interleaving and de-jitter buffers of the receiver. 2032 max-dpb: The value of max-dpb is an integer indicating the 2033 maximum decoded picture buffer size in units of 1024 bytes. 2034 The max-dpb parameter signals that the receiver has more 2035 memory than the minimum amount of decoded picture buffer 2036 memory required by the signaled highest level conveyed in the 2037 value of the profile-level-id parameter or the max-recv-level 2038 parameter. When max-dpb is signaled, the receiver MUST be 2039 able to decode NAL unit streams that conform to the signaled 2040 highest level, with the exception that the MaxDPB value in 2041 Table A-1 of [1] for the signaled highest level is replaced 2042 with the value of max-dpb. Consequently, a receiver that 2043 signals max-dpb MUST be capable of storing the following 2044 number of decoded frames, complementary field pairs, and non- 2045 paired fields in its decoded picture buffer: 2047 Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs * 2048 256 * ChromaFormatFactor ), 16) 2050 PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are 2051 defined in [1]. 2053 The value of max-dpb MUST be greater than or equal to the 2054 value of MaxDPB given in Table A-1 of [1] for the highest 2055 level. Senders MAY use this knowledge to construct coded 2056 video streams with improved compression. 2058 Informative note: This parameter was added primarily to 2059 complement a similar codepoint in the ITU-T 2060 Recommendation H.245, so as to facilitate signaling 2061 gateway designs. The decoded picture buffer stores 2062 reconstructed samples. There is no relationship between 2063 the size of the decoded picture buffer and the buffers 2064 used in RTP, especially de-interleaving and de-jitter 2065 buffers. 2067 max-br: The value of max-br is an integer indicating the maximum 2068 video bit rate in units of 1000 bits per second for the VCL 2069 HRD parameters (see A.3.1 item i of [1]) and in units of 1200 2070 bits per second for the NAL HRD parameters (see A.3.1 item j 2071 of [1]). 2073 The max-br parameter signals that the video decoder of the 2074 receiver is capable of decoding video at a higher bit rate 2075 than is required by the signaled highest level conveyed in 2076 the value of the profile-level-id parameter or the max-recv- 2077 level parameter. 2079 When max-br is signaled, the video codec of the receiver MUST 2080 be able to decode NAL unit streams that conform to the 2081 signaled highest level, with the following exceptions in the 2082 limits specified by the highest level: 2084 o The value of max-br replaces the MaxBR value in Table A-1 2085 of [1] for the highest level. 2087 o When the max-cpb parameter is not present, the result of 2088 the following formula replaces the value of MaxCPB in 2089 Table A-1 of [1]: (MaxCPB of the signaled level) * max-br 2090 / (MaxBR of the signaled highest level). 2092 For example, if a receiver signals capability for Level 1.2 2093 with max-br equal to 1550, this indicates a maximum video 2094 bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum 2095 video bitrate of 1860 kbits/sec for NAL HRD parameters, and a 2096 CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000). 2098 The value of max-br MUST be greater than or equal to the 2099 value MaxBR given in Table A-1 of [1] for the signaled 2100 highest level. 2102 Senders MAY use this knowledge to send higher bitrate video 2103 as allowed in the level definition of Annex A of H.264, to 2104 achieve improved video quality. 2106 Informative note: This parameter was added primarily to 2107 complement a similar codepoint in the ITU-T 2108 Recommendation H.245, so as to facilitate signaling 2109 gateway designs. No assumption can be made from the 2110 value of this parameter that the network is capable of 2111 handling such bit rates at any given time. In particular, 2112 no conclusion can be drawn that the signaled bit rate is 2113 possible under congestion control constraints. 2115 redundant-pic-cap: 2116 This parameter signals the capabilities of a receiver 2117 implementation. When equal to 0, the parameter indicates 2118 that the receiver makes no attempt to use redundant coded 2119 pictures to correct incorrectly decoded primary coded 2120 pictures. When equal to 0, the receiver is not capable of 2121 using redundant slices; therefore, a sender SHOULD avoid 2122 sending redundant slices to save bandwidth. When equal to 1, 2123 the receiver is capable of decoding any such redundant slice 2124 that covers a corrupted area in a primary decoded picture (at 2125 least partly), and therefore a sender MAY send redundant 2126 slices. When the parameter is not present, then a value of 0 2127 MUST be used for redundant-pic-cap. When present, the value 2128 of redundant-pic-cap MUST be either 0 or 1. 2130 When the profile-level-id parameter is present in the same 2131 signaling as the redundant-pic-cap parameter, and the profile 2132 indicated in profile-level-id is such that it disallows the 2133 use of redundant coded pictures (e.g., Main Profile), the 2134 value of redundant-pic-cap MUST be equal to 0. When a 2135 receiver indicates redundant-pic-cap equal to 0, the received 2136 stream SHOULD NOT contain redundant coded pictures. 2138 Informative note: Even if redundant-pic-cap is equal to 0, 2139 the decoder is able to ignore redundant codec pictures 2140 provided that the decoder supports such a profile 2141 (Baseline, Extended) in which redundant coded pictures 2142 are allowed. 2144 Informative note: Even if redundant-pic-cap is equal to 1, 2145 the receiver may also choose other error concealment 2146 strategies to replace or complement decoding of redundant 2147 slices. 2149 sprop-parameter-sets: 2150 This parameter MAY be used to convey any sequence and picture 2151 parameter set NAL units (herein referred to as the initial 2152 parameter set NAL units) that can be placed in the NAL unit 2153 stream to precede any other NAL units in decoding order. The 2154 parameter MUST NOT be used to indicate codec capability in 2155 any capability exchange procedure. The value of the 2156 parameter is a comma (',') separated list of base64 [7] 2157 representations of parameter set NAL units as specified in 2158 sections 7.3.2.1 and 7.3.2.2 of [1]. Note that the number of 2159 bytes in a parameter set NAL unit is typically less than 10, 2160 but a picture parameter set NAL unit can contain several 2161 hundreds of bytes. 2163 Informative note: When several payload types are offered 2164 in the SDP Offer/Answer model, each with its own sprop- 2165 parameter-sets parameter, then the receiver cannot assume 2166 that those parameter sets do not use conflicting storage 2167 locations (i.e., identical values of parameter set 2168 identifiers). Therefore, a receiver should buffer all 2169 sprop-parameter-sets and make them available to the 2170 decoder instance that decodes a certain payload type. 2172 The "sprop-parameter-sets" parameter MUST only contain 2173 parameter sets that are conforming to the profile-level-id, 2174 i.e., the subset of coding tools indicated by any of the 2175 parameter sets MUST be equal to the default sub-profile, and 2176 the level indicated by any of the parameter sets MUST be 2177 equal to the default level. 2179 sprop-level-parameter-sets: 2180 This parameter MAY be used to convey any sequence and picture 2181 parameter set NAL units (herein referred to as the initial 2182 parameter set NAL units) that can be placed in the NAL unit 2183 stream to precede any other NAL units in decoding order and 2184 that are associated with one or more levels different than 2185 the default level. The parameter MUST NOT be used to 2186 indicate codec capability in any capability exchange 2187 procedure. 2189 The sprop-level-parameter-sets parameter contains parameter 2190 sets for one or more levels which are different than the 2191 default level. All parameter sets associated with one level 2192 are clustered and prefixed with a three-byte field which has 2193 the same syntax as profile-level-id. This enables the 2194 receiver to install the parameter sets for one level and 2195 discard the rest. The three-byte field is named PLId, and 2196 all parameter sets associated with one level are named PSL, 2197 which has the same syntax as sprop-parameter-sets. Parameter 2198 sets for each level are represented in the form of PLId:PSL, 2199 i.e., PLId followed by a colon (':') and the base64 [7] 2200 representation of the initial parameter set NAL units for the 2201 level. Each pair of PLId:PSL is also separated by a colon. 2202 Note that a PSL can contain multiple parameter sets for that 2203 level, separated with commas (','). 2205 The subset of coding tools indicated by each PLId field MUST 2206 be equal to the default sub-profile, and the level indicated 2207 by each PLId field MUST be different than the default level. 2208 All sequence parameter sets contained in each PSL MUST have 2209 the three bytes from profile_idc to level_idc, inclusive, 2210 equal to the preceding PLId. 2212 Informative note: This parameter allows for efficient 2213 level downgrade or upgrade in SDP Offer/Answer and out- 2214 of-band transport of parameter sets, simultaneously. 2216 use-level-src-parameter-sets: 2217 This parameter MAY be used to indicate a receiver capability. 2218 The value MAY be equal to either 0 or 1. When the parameter 2219 is not present, the value MUST be inferred to be equal to 0. 2220 The value 0 indicates that the receiver does not understand 2221 the sprop-level-parameter-sets parameter, and does not 2222 understand the "fmtp" source attribute as specified in 2223 section 6.3 of [9], and will ignore sprop-level-parameter- 2224 sets when present, and will ignore sprop-parameter-sets when 2225 conveyed using the "fmtp" source attribute. The value 1 2226 indicates that the receiver understands the sprop-level- 2227 parameter-sets parameter, and understands the "fmtp" source 2228 attribute as specified in section 6.3 of [9], and is capable 2229 of using parameter sets contained in the sprop-level- 2230 parameter-sets or contained in the sprop-parameter-sets that 2231 is conveyed using the "fmtp" source attribute. 2233 Informative note: An RFC 3984 receiver does not 2234 understand sprop-level-parameter-sets, use-level-src- 2235 parameter-sets, or the "fmtp" source attribute as 2236 specified in section 6.3 of [9]. Therefore, during SDP 2237 Offer/Answer, an RFC 3984 receiver as the answerer will 2238 simply ignore sprop-level-parameter-sets, when present in 2239 an offer, and sprop-parameter-sets conveyed using the 2240 "fmtp" source attribute as specified in section 6.3 of 2242 [9]. Assume that the offered payload type was accepted 2243 at a level lower than the default level. If the offered 2244 payload type included sprop-level-parameter-sets or 2245 included sprop-parameter-sets conveyed using the "fmtp" 2246 source attribute, and the offerer sees that the answerer 2247 has not included use-level-src-parameter-sets equal to 1 2248 in the answer, the offerer knows that in-band transport 2249 of parameter sets is needed. 2251 in-band-parameter-sets: 2252 This parameter MAY be used to indicate a receiver capability. 2253 The value MAY be equal to either 0 or 1. The value 1 2254 indicates that the receiver discards out-of-band parameter 2255 sets in sprop-parameter-sets and sprop-level-parameter-sets, 2256 therefore the sender MUST transmit all parameter sets in-band. 2257 The value 0 indicates that the receiver utilizes out-of-band 2258 parameter sets included in sprop-parameter-sets and/or sprop- 2259 level-parameter-sets. However, in this case, the sender MAY 2260 still choose to send parameter sets in-band. When in-band- 2261 parameter-sets is equal to 1, use-level-src-parameter-sets 2262 MUST NOT be present or MUST be equal to 0. When the 2263 parameter is not present, this receiver capability is not 2264 specified, and therefore the sender MAY send out-of-band 2265 parameter sets only, or it MAY send in-band-parameter-sets 2266 only, or it MAY send both. 2268 level-asymmetry-allowed: 2269 This parameter MAY be used in SDP Offer/Answer to indicate 2270 whether level asymmetry, i.e., sending media encoded at a 2271 different level in the offerer-to-answerer direction than the 2272 level in the answerer-to-offerer direction, is allowed. The 2273 value MAY be equal to either 0 or 1. When the parameter is 2274 not present, the value MUST be inferred to be equal to 0. 2275 The value 1 in both the offer and the answer indicates that 2276 level asymmetry is allowed. The value of 0 in either the 2277 offer or the answer indicates the level asymmetry is not 2278 allowed. 2280 If "level-asymmetry-allowed" is equal to 0 (or not present) 2281 in either the offer or the answer, level asymmetry is not 2282 allowed. In this case, the level to use in the direction 2283 from the offerer to the answerer MUST be the same as the 2284 level to use in the opposite direction. 2286 packetization-mode: 2287 This parameter signals the properties of an RTP payload type 2288 or the capabilities of a receiver implementation. Only a 2289 single configuration point can be indicated; thus, when 2290 capabilities to support more than one packetization-mode are 2291 declared, multiple configuration points (RTP payload types) 2292 must be used. 2294 When the value of packetization-mode is equal to 0 or 2295 packetization-mode is not present, the single NAL mode MUST 2296 be used. This mode is in use in standards using ITU-T 2297 Recommendation H.241 [3] (see section 12.1). When the value 2298 of packetization-mode is equal to 1, the non-interleaved mode 2299 MUST be used. When the value of packetization-mode is equal 2300 to 2, the interleaved mode MUST be used. The value of 2301 packetization-mode MUST be an integer in the range of 0 to 2, 2302 inclusive. 2304 sprop-interleaving-depth: 2305 This parameter MUST NOT be present when packetization-mode is 2306 not present or the value of packetization-mode is equal to 0 2307 or 1. This parameter MUST be present when the value of 2308 packetization-mode is equal to 2. 2310 This parameter signals the properties of an RTP packet stream. 2311 It specifies the maximum number of VCL NAL units that precede 2312 any VCL NAL unit in the RTP packet stream in transmission 2313 order and follow the VCL NAL unit in decoding order. 2314 Consequently, it is guaranteed that receivers can reconstruct 2315 NAL unit decoding order when the buffer size for NAL unit 2316 decoding order recovery is at least the value of sprop- 2317 interleaving-depth + 1 in terms of VCL NAL units. 2319 The value of sprop-interleaving-depth MUST be an integer in 2320 the range of 0 to 32767, inclusive. 2322 sprop-deint-buf-req: 2323 This parameter MUST NOT be present when packetization-mode is 2324 not present or the value of packetization-mode is equal to 0 2325 or 1. It MUST be present when the value of packetization- 2326 mode is equal to 2. 2328 sprop-deint-buf-req signals the required size of the de- 2329 interleaving buffer for the RTP packet stream. The value of 2330 the parameter MUST be greater than or equal to the maximum 2331 buffer occupancy (in units of bytes) required in such a de- 2332 interleaving buffer that is specified in section 7.2. It is 2333 guaranteed that receivers can perform the de-interleaving of 2334 interleaved NAL units into NAL unit decoding order, when the 2335 de-interleaving buffer size is at least the value of sprop- 2336 deint-buf-req in terms of bytes. 2338 The value of sprop-deint-buf-req MUST be an integer in the 2339 range of 0 to 4294967295, inclusive. 2341 Informative note: sprop-deint-buf-req indicates the 2342 required size of the de-interleaving buffer only. When 2343 network jitter can occur, an appropriately sized jitter 2344 buffer has to be provisioned for as well. 2346 deint-buf-cap: 2347 This parameter signals the capabilities of a receiver 2348 implementation and indicates the amount of de-interleaving 2349 buffer space in units of bytes that the receiver has 2350 available for reconstructing the NAL unit decoding order. A 2351 receiver is able to handle any stream for which the value of 2352 the sprop-deint-buf-req parameter is smaller than or equal to 2353 this parameter. 2355 If the parameter is not present, then a value of 0 MUST be 2356 used for deint-buf-cap. The value of deint-buf-cap MUST be 2357 an integer in the range of 0 to 4294967295, inclusive. 2359 Informative note: deint-buf-cap indicates the maximum 2360 possible size of the de-interleaving buffer of the 2361 receiver only. When network jitter can occur, an 2362 appropriately sized jitter buffer has to be provisioned 2363 for as well. 2365 sprop-init-buf-time: 2366 This parameter MAY be used to signal the properties of an RTP 2367 packet stream. The parameter MUST NOT be present, if the 2368 value of packetization-mode is equal to 0 or 1. 2370 The parameter signals the initial buffering time that a 2371 receiver MUST wait before starting decoding to recover the 2372 NAL unit decoding order from the transmission order. The 2373 parameter is the maximum value of (decoding time of the NAL 2374 unit - transmission time of a NAL unit), assuming reliable 2375 and instantaneous transmission, the same timeline for 2376 transmission and decoding, and that decoding starts when the 2377 first packet arrives. 2379 An example of specifying the value of sprop-init-buf-time 2380 follows. A NAL unit stream is sent in the following 2381 interleaved order, in which the value corresponds to the 2382 decoding time and the transmission order is from left to 2383 right: 2385 0 2 1 3 5 4 6 8 7 ... 2387 Assuming a steady transmission rate of NAL units, the 2388 transmission times are: 2390 0 1 2 3 4 5 6 7 8 ... 2392 Subtracting the decoding time from the transmission time 2393 column-wise results in the following series: 2395 0 -1 1 0 -1 1 0 -1 1 ... 2397 Thus, in terms of intervals of NAL unit transmission times, 2398 the value of sprop-init-buf-time in this example is 1. The 2399 parameter is coded as a non-negative base10 integer 2400 representation in clock ticks of a 90-kHz clock. If the 2401 parameter is not present, then no initial buffering time 2402 value is defined. Otherwise the value of sprop-init-buf-time 2403 MUST be an integer in the range of 0 to 4294967295, inclusive. 2405 In addition to the signaled sprop-init-buf-time, receivers 2406 SHOULD take into account the transmission delay jitter 2407 buffering, including buffering for the delay jitter caused by 2408 mixers, translators, gateways, proxies, traffic-shapers, and 2409 other network elements. 2411 sprop-max-don-diff: 2412 This parameter MAY be used to signal the properties of an RTP 2413 packet stream. It MUST NOT be used to signal transmitter or 2414 receiver or codec capabilities. The parameter MUST NOT be 2415 present if the value of packetization-mode is equal to 0 or 1. 2416 sprop-max-don-diff is an integer in the range of 0 to 32767, 2417 inclusive. If sprop-max-don-diff is not present, the value 2418 of the parameter is unspecified. sprop-max-don-diff is 2419 calculated as follows: 2421 sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)}, 2422 for any i and any j>i, 2424 where i and j indicate the index of the NAL unit in the 2425 transmission order and AbsDON denotes a decoding order number 2426 of the NAL unit that does not wrap around to 0 after 65535. 2427 In other words, AbsDON is calculated as follows: Let m and n 2428 be consecutive NAL units in transmission order. For the very 2429 first NAL unit in transmission order (whose index is 0), 2430 AbsDON(0) = DON(0). For other NAL units, AbsDON is 2431 calculated as follows: 2433 If DON(m) == DON(n), AbsDON(n) = AbsDON(m) 2435 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 2436 AbsDON(n) = AbsDON(m) + DON(n) - DON(m) 2438 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 2439 AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n) 2441 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 2442 AbsDON(n) = AbsDON(m) - (DON(m) + 65536 - DON(n)) 2444 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 2445 AbsDON(n) = AbsDON(m) - (DON(m) - DON(n)) 2447 where DON(i) is the decoding order number of the NAL unit 2448 having index i in the transmission order. The decoding order 2449 number is specified in section 5.5. 2451 Informative note: Receivers may use sprop-max-don-diff to 2452 trigger which NAL units in the receiver buffer can be 2453 passed to the decoder. 2455 max-rcmd-nalu-size: 2456 This parameter MAY be used to signal the capabilities of a 2457 receiver. The parameter MUST NOT be used for any other 2458 purposes. The value of the parameter indicates the largest 2459 NALU size in bytes that the receiver can handle efficiently. 2460 The parameter value is a recommendation, not a strict upper 2461 boundary. The sender MAY create larger NALUs but must be 2462 aware that the handling of these may come at a higher cost 2463 than NALUs conforming to the limitation. 2465 The value of max-rcmd-nalu-size MUST be an integer in the 2466 range of 0 to 4294967295, inclusive. If this parameter is 2467 not specified, no known limitation to the NALU size exists. 2468 Senders still have to consider the MTU size available between 2469 the sender and the receiver and SHOULD run MTU discovery for 2470 this purpose. 2472 This parameter is motivated by, for example, an IP to H.223 2473 video telephony gateway, where NALUs smaller than the H.223 2474 transport data unit will be more efficient. A gateway may 2475 terminate IP; thus, MTU discovery will normally not work 2476 beyond the gateway. 2478 Informative note: Setting this parameter to a lower than 2479 necessary value may have a negative impact. 2481 sar-understood: 2482 This parameter MAY be used to indicate a receiver capability 2483 and not anything else. The parameter indicates the maximum 2484 value of aspect_ratio_idc (specified in [1]) smaller than 255 2485 that the receiver understands. Table E-1 of [1] specifies 2486 aspect_ratio_idc equal to 0 as "unspecified", 1 to 16, 2487 inclusive, as specific Sample Aspect Ratios (SARs), 17 to 254, 2488 inclusive, as "reserved", and 255 as the Extended SAR, for 2489 which SAR width and SAR height are explicitly signaled. 2490 Therefore, a receiver with a decoder according to [1] 2491 understands aspect_ratio_idc in the range of 1 to 16, 2492 inclusive and aspect_ratio_idc equal to 255, in the sense 2493 that the receiver knows what exactly the SAR is. For such a 2494 receiver, the value of sar-understood is 16. If in the 2495 future Table E-1 of [1] is extended, e.g., such that the SAR 2496 for aspect_ratio_idc equal to 17 is specified, then for a 2497 receiver with a decoder that understands the extension, the 2498 value of sar-understood is 17. For a receiver with a decoder 2499 according to the 2003 version of [1], the value of sar- 2500 understood is 13, as the minimum reserved aspect_ratio_idc 2501 therein is 14. 2503 When sar-understood is not present, the value MUST be 2504 inferred to be equal to 13. 2506 sar-supported: 2507 This parameter MAY be used to indicate a receiver capability 2508 and not anything else. The value of this parameter is an 2509 integer in the range of 1 to sar-understood, inclusive, equal 2510 to 255. The value of sar-supported equal to N smaller than 2511 255 indicates that the receiver supports all the SARs 2512 corresponding to H.264 aspect_ratio_idc values (see Table E-1 2513 of [1]) in the range from 1 to N, inclusive, without 2514 geometric distortion. The value of sar-supported equal to 2515 255 indicates that the receiver supports all sample aspect 2516 ratios which are expressible using two 16-bit integer values 2517 as the numerator and denominator, i.e., those that are 2518 expressible using the H.264 aspect_ratio_idc value of 255 2519 (Extended_SAR, see Table E-1 of [1]), without geometric 2520 distortion. 2522 H.264 compliant encoders SHOULD NOT send an aspect_ratio_idc 2523 equal to 0, or an aspect_ratio_idc larger than sar-understood 2524 and smaller than 255. H.264 compliant encoders SHOULD send 2525 an aspect_ratio_idc that the receiver is able to display 2526 without geometrical distortion. However, H.264 compliant 2527 encoders MAY choose to send pictures using any SAR. 2529 Note that the actual sample aspect ratio or extended sample 2530 aspect ratio, when present, of the stream is conveyed in the 2531 Video Usability Information (VUI) part of the sequence 2532 parameter set. 2534 Encoding considerations: 2535 This type is only defined for transfer via RTP (RFC 3550). 2537 Security considerations: 2538 See section 9 of RFC xxxx. 2540 Public specification: 2541 Please refer to RFC xxxx and its section 15. 2543 Additional information: 2544 None 2546 File extensions: none 2548 Macintosh file type code: none 2550 Object identifier or OID: none 2552 Person & email address to contact for further information: 2553 Ye-Kui Wang, yekuiwang@huawei.com 2555 Intended usage: COMMON 2557 Author: 2558 Ye-Kui Wang, yekuiwang@huawei.com 2560 Change controller: 2561 IETF Audio/Video Transport working group delegated from the 2562 IESG. 2564 8.2. SDP Parameters 2566 The receiver MUST ignore any parameter unspecified in this memo. 2568 8.2.1. Mapping of Payload Type Parameters to SDP 2570 The media type video/H264 string is mapped to fields in the Session 2571 Description Protocol (SDP) [6] as follows: 2573 o The media name in the "m=" line of SDP MUST be video. 2575 o The encoding name in the "a=rtpmap" line of SDP MUST be H264 2576 (the media subtype). 2578 o The clock rate in the "a=rtpmap" line MUST be 90000. 2580 o The OPTIONAL parameters "profile-level-id", "max-recv-level", 2581 "max-mbps", "max-smbps", "max-fs", "max-cpb", "max-dpb", "max- 2582 br", "redundant-pic-cap", "use-level-src-parameter-sets", "in- 2583 band-parameter-sets", "level-asymmetry-allowed", "packetization- 2584 mode", "sprop-interleaving-depth", "sprop-deint-buf-req", 2585 "deint-buf-cap", "sprop-init-buf-time", "sprop-max-don-diff", 2586 "max-rcmd-nalu-size", "sar-understood", and "sar-supported", 2587 when present, MUST be included in the "a=fmtp" line of SDP. 2588 These parameters are expressed as a media type string, in the 2589 form of a semicolon separated list of parameter=value pairs. 2591 o The OPTIONAL parameters "sprop-parameter-sets" and "sprop-level- 2592 parameter-sets", when present, MUST be included in the "a=fmtp" 2593 line of SDP or conveyed using the "fmtp" source attribute as 2594 specified in section 6.3 of [9]. For a particular media format 2595 (i.e., RTP payload type), a "sprop-parameter-sets" or "sprop- 2596 level-parameter-sets" MUST NOT be both included in the "a=fmtp" 2597 line of SDP and conveyed using the "fmtp" source attribute. 2598 When included in the "a=fmtp" line of SDP, these parameters are 2599 expressed as a media type string, in the form of a semicolon 2600 separated list of parameter=value pairs. When conveyed using 2601 the "fmtp" source attribute, these parameters are only 2602 associated with the given source and payload type as parts of 2603 the "fmtp" source attribute. 2605 Informative note: Conveyance of "sprop-parameter-sets" and 2606 "sprop-level-parameter-sets" using the "fmtp" source 2607 attribute allows for out-of-band transport of parameter sets 2608 in topologies like Topo-Video-switch-MCU [29]. 2610 An example of media representation in SDP is as follows (Baseline 2611 Profile, Level 3.0, some of the constraints of the Main profile may 2612 not be obeyed): 2614 m=video 49170 RTP/AVP 98 2615 a=rtpmap:98 H264/90000 2616 a=fmtp:98 profile-level-id=42A01E; 2617 packetization-mode=1; 2618 sprop-parameter-sets= 2620 8.2.2. Usage with the SDP Offer/Answer Model 2622 When H.264 is offered over RTP using SDP in an Offer/Answer model 2623 [8] for negotiation for unicast usage, the following limitations 2624 and rules apply: 2626 o The parameters identifying a media format configuration for 2627 H.264 are "profile-level-id" and "packetization-mode". These 2628 media format configuration parameters (except for the level part 2629 of "profile-level-id") MUST be used symmetrically; i.e., the 2630 answerer MUST either maintain all configuration parameters or 2631 remove the media format (payload type) completely, if one or 2632 more of the parameter values are not supported. Note that the 2633 level part of "profile-level-id" includes level_idc, and, for 2634 indication of level 1b when profile_idc is equal to 66, 77 or 88, 2635 bit 4 (constraint_set3_flag) of profile-iop. The level part of 2636 "profile-level-id" is changeable. 2638 Informative note: The requirement for symmetric use does not 2639 apply for the level part of "profile-level-id", and does not 2640 apply for the other stream properties and capability 2641 parameters. 2643 Informative note: In H.264 [1], all the levels except for 2644 level 1b are equal to the value of level_idc divided by 10. 2645 Level 1b is a level higher than level 1.0 but lower than 2646 level 1.1, and is signaled in an ad-hoc manner, due to that 2647 the level was specified after level 1.0 and level 1.1. For 2648 the Baseline, Main and Extended profiles (with profile_idc 2649 equal to 66, 77 and 88, respectively), level 1b is indicated 2650 by level_idc equal to 11 (i.e. same as level 1.1) and 2651 constraint_set3_flag equal to 1. For other profiles, level 2652 1b is indicated by level_idc equal to 9 (but note that level 2653 1b for these profiles are still higher than level 1, which 2654 has level_idc equal to 10, and lower than level 1.1). In SDP 2655 Offer/Answer, an answer to an offer may indicate a level 2656 equal to or lower than the level indicated in the offer. Due 2657 to the ad-hoc indication of level 1b, offerers and answerers 2658 must check the value of bit 4 (constraint_set3_flag) of the 2659 middle octet of the parameter "profile-level-id", when 2660 profile_idc is equal to 66, 77 or 88 and level_idc is equal 2661 to 11. 2663 To simplify handling and matching of these configurations, the 2664 same RTP payload type number used in the offer SHOULD also be 2665 used in the answer, as specified in [8]. An answer MUST NOT 2666 contain a payload type number used in the offer unless the 2667 configuration is exactly the same as in the offer. 2669 Informative note: When an offerer receives an answer, it has 2670 to compare payload types not declared in the offer based on 2671 the media type (i.e., video/H264) and the above media 2672 configuration parameters with any payload types it has 2673 already declared. This will enable it to determine whether 2674 the configuration in question is new or if it is equivalent 2675 to configuration already offered, since a different payload 2676 type number may be used in the answer. 2678 o The parameter "max-recv-level", when present, declares the 2679 highest level supported for receiving. In case "max-recv-level" 2680 is not present, the highest level supported for receiving is 2681 equal to the default level indicated by the level part of 2682 "profile-level-id". "max-recv-level", when present, MUST be 2683 higher than the default level. 2685 o The parameter "level-asymmetry-allowed" indicates whether level 2686 asymmetry is allowed. 2688 If "level-asymmetry-allowed" is equal to 0 (or not present) in 2689 either the offer or the answer, level asymmetry is not allowed. 2690 In this case, the level to use in the direction from the offerer 2691 to the answerer MUST be the same as the level to use in the 2692 opposite direction, and the common level to use is equal to the 2693 lower value of the default level in the offer and the default 2694 level in the answer. 2696 Otherwise ("level-asymmetry-allowed" equals to 1 in both the 2697 offer and the answer), level asymmetry is allowed. In this case, 2698 the level to use in the offerer-to-answerer direction MUST be 2699 equal to the highest level the answerer supports for receiving, 2700 and the level to use in the answerer-to-offerer direction MUST 2701 be equal to the highest level the offerer supports for receiving. 2703 When level asymmetry is not allowed, level upgrade is not 2704 allowed, i.e. the default level in the answer MUST be equal to 2705 or lower than the default level in the offer. 2707 o The parameters "sprop-deint-buf-req", "sprop-interleaving-depth", 2708 "sprop-max-don-diff", and "sprop-init-buf-time" describe the 2709 properties of the RTP packet stream that the offerer or answerer 2710 is sending for the media format configuration. This differs 2711 from the normal usage of the Offer/Answer parameters: normally 2712 such parameters declare the properties of the stream that the 2713 offerer or the answerer is able to receive. When dealing with 2714 H.264, the offerer assumes that the answerer will be able to 2715 receive media encoded using the configuration being offered. 2717 Informative note: The above parameters apply for any stream 2718 sent by the declaring entity with the same configuration; 2719 i.e., they are dependent on their source. Rather than being 2720 bound to the payload type, the values may have to be applied 2721 to another payload type when being sent, as they apply for 2722 the configuration. 2724 o The capability parameters "max-mbps", "max-smbps", "max-fs", 2725 "max-cpb", "max-dpb", "max-br", ,"redundant-pic-cap", "max-rcmd- 2726 nalu-size", "sar-understood", "sar-supported" MAY be used to 2727 declare further capabilities of the offerer or answerer for 2728 receiving. These parameters MUST NOT be present when the 2729 direction attribute is sendonly, and the parameters describe the 2730 limitations of what the offerer or answerer accepts for 2731 receiving streams. 2733 o An offerer has to include the size of the de-interleaving buffer, 2734 "sprop-deint-buf-req", in the offer for an interleaved H.264 2735 stream. To enable the offerer and answerer to inform each other 2736 about their capabilities for de-interleaving buffering in 2737 receiving streams, both parties are RECOMMENDED to include 2738 "deint-buf-cap". For interleaved streams, it is also 2739 RECOMMENDED to consider offering multiple payload types with 2740 different buffering requirements when the capabilities of the 2741 receiver are unknown. 2743 o The "sprop-parameter-sets" or "sprop-level-parameter-sets" 2744 parameter, when present (included in the "a=fmtp" line of SDP or 2745 conveyed using the "fmtp" source attribute as specified in 2746 section 6.3 of [9]), is used for out-of-band transport of 2747 parameter sets. However, when out-of-band transport of 2748 parameter sets is used, parameter sets MAY still be additionally 2749 transported in-band. 2751 The answerer MAY use either out-of-band or in-band transport of 2752 parameter sets for the stream it is sending, regardless of 2753 whether out-of-band parameter sets transport has been used in 2754 the offerer-to-answerer direction. Parameter sets included in 2755 an answer are independent of those parameter sets included in 2756 the offer, as they are used for decoding two different video 2757 streams, one from the answerer to the offerer, and the other in 2758 the opposite direction. 2760 The following rules apply to transport of parameter sets in the 2761 offerer-to-answerer direction. 2763 o An offer MAY include either or both of "sprop-parameter- 2764 sets" and "sprop-level-parameter-sets". If neither "sprop- 2765 parameter-sets" nor "sprop-level-parameter-sets" is present 2766 in the offer, then only in-band transport of parameter sets 2767 is used. 2769 o If the answer includes "in-band-parameter-sets" equal to 1, 2770 then the offerer MUST transmit parameter sets in-band. 2771 Otherwise, the following applies. 2773 o If the level to use in the offerer-to-answerer 2774 direction is equal to the default level in the offer, 2775 the following applies. 2777 When there is a "sprop-parameter-sets" included 2778 in the "a=fmtp" line in the offer, the answerer 2779 MUST be prepared to use the parameter sets 2780 included in the "sprop-parameter-sets" for 2781 decoding the incoming NAL unit stream. 2783 When there is a "sprop-parameter-sets" conveyed 2784 using the "fmtp" source attribute in the offer, 2785 the following applies. If the answer includes 2786 "use-level-src-parameter-sets" equal to 1 or the 2787 "fmtp" source attribute, the answerer MUST be 2788 prepared to use the parameter sets included in 2789 the "sprop-parameter-sets" for decoding the 2790 incoming NAL unit stream; Otherwise, the offerer 2791 MUST transmit parameter sets in-band. 2793 When "sprop-parameter-sets" is not present in the 2794 offer, the offerer MUST transmit parameter sets 2795 in-band. 2797 The answerer MUST ignore "sprop-level-parameter- 2798 sets", when present (either included in the 2799 "a=fmtp" line or conveyed using the "fmtp" source 2800 attribute) in the offer. 2802 o Otherwise (the level to use in the offerer-to-answerer 2803 direction is not equal to the default level in the 2804 offer), the following applies. 2806 The answerer MUST ignore "sprop-parameter-sets", 2807 when present (either included in the "a=fmtp" 2808 line or conveyed using the "fmtp" source 2809 attribute) in the offer. 2811 When neither "use-level-src-parameter-sets" equal 2812 to 1 nor the "fmtp" source attribute is present 2813 in the answer, the answerer MUST ignore "sprop- 2814 level-parameter-sets", when present in the offer, 2815 and the offerer MUST transmit parameter sets in- 2816 band. 2818 When either "use-level-src-parameter-sets" equal 2819 to 1 or the "fmtp" source attribute is present in 2820 the answer, the answerer MUST be prepared to use 2821 the parameter sets that are included in "sprop- 2822 level-parameter-sets" for the accepted level (i.e. 2823 the default level in the answer), when present in 2824 the offer, for decoding the incoming NAL unit 2825 stream, and ignore all other parameter sets 2826 included in "sprop-level-parameter-sets". 2828 When no parameter sets for the level to use in 2829 the offerer-to-answerer direction are present in 2830 "sprop-level-parameter-sets" in the offer, the 2831 offerer MUST transmit parameter sets in-band. 2833 The following rules apply to transport of parameter sets in the 2834 answerer-to-offerer direction. 2836 o An answer MAY include either "sprop-parameter-sets" or 2837 "sprop-level-parameter-sets", but MUST NOT include both of 2838 the two. If neither "sprop-parameter-sets" nor "sprop- 2839 level-parameter-sets" is present in the answer, then only 2840 in-band transport of parameter sets is used. 2842 o If the offer includes "in-band-parameter-sets" equal to 1, 2843 the answerer MUST NOT include "sprop-parameter-sets" or 2844 "sprop-level-parameter-sets" in the answer and MUST 2845 transmit parameter sets in-band. Otherwise, the following 2846 applies. 2848 o If the level to use in the answerer-to-offerer 2849 direction is equal to the default level in the answer, 2850 the following applies. 2852 When there is a "sprop-parameter-sets" included 2853 in the "a=fmtp" line in the answer, the offerer 2854 MUST be prepared to use the parameter sets 2855 included in the "sprop-parameter-sets" for 2856 decoding the incoming NAL unit stream. 2858 When there is a "sprop-parameter-sets" conveyed 2859 using the "fmtp" source attribute in the answer, 2860 the following applies. If the offer includes 2861 "use-level-src-parameter-sets" equal to 1 or the 2862 "fmtp" source attribute, the offerer MUST be 2863 prepared to use the parameter sets included in 2864 the "sprop-parameter-sets" for decoding the 2865 incoming NAL unit stream; Otherwise, the 2866 answerer MUST transmit parameter sets in-band. 2868 When "sprop-parameter-sets" is not present in the 2869 answer, the answerer MUST transmit parameter sets 2870 in-band. 2872 The offerer MUST ignore "sprop-level-parameter- 2873 sets", when present (either included in the 2874 "a=fmtp" line or conveyed using the "fmtp" source 2875 attribute) in the answer. 2877 o Otherwise (the level to use in the answerer-to-offerer 2878 direction is not equal to the default level in the 2879 answer), the following applies. 2881 The offerer MUST ignore "sprop-parameter-sets", 2882 when present (either included in the "a=fmtp" 2883 line of SDP or conveyed using the "fmtp" source 2884 attribute) in the answer. 2886 When neither "use-level-src-parameter-sets" equal 2887 to 1 nor the "fmtp" source attribute is present 2888 in the offer, the offerer MUST ignore "sprop- 2889 level-parameter-sets", when present, and the 2890 answerer MUST transmit parameter sets in-band. 2892 When either "use-level-src-parameter-sets" equal 2893 to 1 or the "fmtp" source attribute is present in 2894 the offer, the offerer MUST be prepared to use 2895 the parameter sets that are included in "sprop- 2896 level-parameter-sets" for the level to use in the 2897 answerer-to-offerer direction, when present in 2898 the answer, for decoding the incoming NAL unit 2899 stream, and ignore all other parameter sets 2900 included in "sprop-level-parameter-sets" in the 2901 answer. 2903 When no parameter sets for the level to use in 2904 the answerer-to-offerer direction are present in 2905 "sprop-level-parameter-sets" in the answer, the 2906 answerer MUST transmit parameter sets in-band. 2908 When "sprop-parameter-sets" or "sprop-level-parameter-sets" is 2909 conveyed using the "fmtp" source attribute as specified in 2910 section 6.3 of [9], the receiver of the parameters MUST store 2911 the parameter sets included in the "sprop-parameter-sets" or 2912 "sprop-level-parameter-sets" for the accepted level and 2913 associate them to the source given as a part of the "fmtp" 2914 source attribute. Parameter sets associated with one source 2915 MUST only be used to decode NAL units conveyed in RTP packets 2916 from the same source. When this mechanism is in use, SSRC 2917 collision detection and resolution MUST be performed as 2918 specified in [9]. 2920 Informative note: Conveyance of "sprop-parameter-sets" and 2921 "sprop-level-parameter-sets" using the "fmtp" source 2922 attribute may be used in topologies like Topo-Video-switch- 2923 MCU [29] to enable out-of-band transport of parameter sets. 2925 For streams being delivered over multicast, the following rules 2926 apply: 2928 o The media format configuration is identified by "profile-level- 2929 id", including the level part, and "packetization-mode". These 2930 media format configuration parameters (including the level part 2931 of "profile-level-id") MUST be used symmetrically; i.e., the 2932 answerer MUST either maintain all configuration parameters or 2933 remove the media format (payload type) completely. Note that 2934 this implies that the level part of "profile-level-id" for 2935 Offer/Answer in multicast is not changeable. 2937 To simplify handling and matching of these configurations, the 2938 same RTP payload type number used in the offer SHOULD also be 2939 used in the answer, as specified in [8]. An answer MUST NOT 2940 contain a payload type number used in the offer unless the 2941 configuration is the same as in the offer. 2943 o Parameter sets received MUST be associated with the originating 2944 source, and MUST be only used in decoding the incoming NAL unit 2945 stream from the same source. 2947 o The rules for other parameters are the same as above for unicast 2948 as long as the above rules are obeyed. 2950 Table 6 lists the interpretation of all the media type parameters 2951 that MUST be used for the different direction attributes. 2953 Table 6. Interpretation of parameters for different direction 2954 attributes. 2956 sendonly --+ 2957 recvonly --+ | 2958 sendrecv --+ | | 2959 | | | 2960 profile-level-id C C P 2961 max-recv-level R R - 2962 packetization-mode C C P 2963 sprop-deint-buf-req P - P 2964 sprop-interleaving-depth P - P 2965 sprop-max-don-diff P - P 2966 sprop-init-buf-time P - P 2967 max-mbps R R - 2968 max-smbps R R - 2969 max-fs R R - 2970 max-cpb R R - 2971 max-dpb R R - 2972 max-br R R - 2973 redundant-pic-cap R R - 2974 deint-buf-cap R R - 2975 max-rcmd-nalu-size R R - 2976 sar-understood R R - 2977 sar-supported R R - 2978 in-band-parameter-sets R R - 2979 use-level-src-parameter-sets R R - 2980 level-asymmetry-allowed O - - 2981 sprop-parameter-sets S - S 2982 sprop-level-parameter-sets S - S 2984 Legend: 2986 C: configuration for sending and receiving streams 2987 O: offer/answer mode 2988 P: properties of the stream to be sent 2989 R: receiver capabilities 2990 S: out-of-band parameter sets 2991 -: not usable, when present SHOULD be ignored 2993 Parameters used for declaring receiver capabilities are in general 2994 downgradable; i.e., they express the upper limit for a sender's 2995 possible behavior. Thus a sender MAY select to set its encoder 2996 using only lower/less or equal values of these parameters. 2998 Parameters declaring a configuration point are not changeable, with 2999 the exception of the level part of the "profile-level-id" parameter 3000 for unicast usage. These express values a receiver expects to be 3001 used and must be used verbatim on the sender side. 3003 When a sender's capabilities are declared, and non-downgradable 3004 parameters are used in this declaration, then these parameters 3005 express a configuration that is acceptable for the sender to 3006 receive streams. In order to achieve high interoperability levels, 3007 it is often advisable to offer multiple alternative configurations; 3008 e.g., for the packetization mode. It is impossible to offer 3009 multiple configurations in a single payload type. Thus, when 3010 multiple configuration offers are made, each offer requires its own 3011 RTP payload type associated with the offer. 3013 A receiver SHOULD understand all media type parameters, even if it 3014 only supports a subset of the payload format's functionality. This 3015 ensures that a receiver is capable of understanding when an offer 3016 to receive media can be downgraded to what is supported by the 3017 receiver of the offer. 3019 An answerer MAY extend the offer with additional media format 3020 configurations. However, to enable their usage, in most cases a 3021 second offer is required from the offerer to provide the stream 3022 property parameters that the media sender will use. This also has 3023 the effect that the offerer has to be able to receive this media 3024 format configuration, not only to send it. 3026 If an offerer wishes to have non-symmetric capabilities between 3027 sending and receiving, the offerer can allow asymmetric levels via 3028 "level-asymmetry-allowed" equal to 1. Alternatively, the offerer 3029 could offer different RTP sessions; i.e., different media lines 3030 declared as "recvonly" and "sendonly", respectively. This may have 3031 further implications on the system, and may require additional 3032 external semantics to associate the two media lines. 3034 8.2.3. Usage in Declarative Session Descriptions 3036 When H.264 over RTP is offered with SDP in a declarative style, as 3037 in RTSP [27] or SAP [28], the following considerations are 3038 necessary. 3040 o All parameters capable of indicating both stream properties and 3041 receiver capabilities are used to indicate only stream 3042 properties. For example, in this case, the parameter "profile- 3043 level-id" declares only the values used by the stream, not the 3044 capabilities for receiving streams. This results in that the 3045 following interpretation of the parameters MUST be used: 3047 Declaring actual configuration or stream properties: 3049 - profile-level-id 3050 - packetization-mode 3051 - sprop-interleaving-depth 3052 - sprop-deint-buf-req 3053 - sprop-max-don-diff 3054 - sprop-init-buf-time 3056 Out-of-band transporting of parameter sets: 3058 - sprop-parameter-sets 3059 - sprop-level-parameter-sets 3061 Not usable(when present, they SHOULD be ignored): 3063 - max-mbps 3064 - max-smbps 3065 - max-fs 3066 - max-cpb 3067 - max-dpb 3068 - max-br 3069 - max-recv-level 3070 - redundant-pic-cap 3071 - max-rcmd-nalu-size 3072 - deint-buf-cap 3073 - sar-understood 3074 - sar-supported 3075 - in-band-parameter-sets 3076 - level-asymmetry-allowed 3077 - use-level-src-parameter-sets 3079 o A receiver of the SDP is required to support all parameters and 3080 values of the parameters provided; otherwise, the receiver MUST 3081 reject (RTSP) or not participate in (SAP) the session. It falls 3082 on the creator of the session to use values that are expected to 3083 be supported by the receiving application. 3085 8.3. Examples 3087 An SDP Offer/Answer exchange wherein both parties are expected to 3088 both send and receive could look like the following. Only the 3089 media codec specific parts of the SDP are shown. Some lines are 3090 wrapped due to text constraints. 3092 Offerer -> Answerer SDP message: 3094 m=video 49170 RTP/AVP 100 99 98 3095 a=rtpmap:98 H264/90000 3096 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; 3097 sprop-parameter-sets= 3098 a=rtpmap:99 H264/90000 3099 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3100 sprop-parameter-sets= 3101 a=rtpmap:100 H264/90000 3102 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3103 sprop-parameter-sets=; 3104 sprop-interleaving-depth=45; sprop-deint-buf-req=64000; 3105 sprop-init-buf-time=102478; deint-buf-cap=128000 3107 The above offer presents the same codec configuration in three 3108 different packetization formats. PT 98 represents single NALU mode, 3109 PT 99 represents non-interleaved mode, and PT 100 indicates the 3110 interleaved mode. In the interleaved mode case, the interleaving 3111 parameters that the offerer would use if the answer indicates 3112 support for PT 100 are also included. In all three cases the 3113 parameter "sprop-parameter-sets" conveys the initial parameter sets 3114 that are required by the answerer when receiving a stream from the 3115 offerer when this configuration is accepted. Note that the value 3116 for "sprop-parameter-sets" could be different for each payload type. 3118 Answerer -> Offerer SDP message: 3120 m=video 49170 RTP/AVP 100 99 97 3121 a=rtpmap:97 H264/90000 3122 a=fmtp:97 profile-level-id=42A01E; packetization-mode=0; 3123 sprop-parameter-sets= 3124 a=rtpmap:99 H264/90000 3125 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3126 sprop-parameter-sets=; 3127 max-rcmd-nalu-size=3980 3128 a=rtpmap:100 H264/90000 3129 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3130 sprop-parameter-sets=; 3131 sprop-interleaving-depth=60; 3132 sprop-deint-buf-req=86000; sprop-init-buf-time=156320; 3133 deint-buf-cap=128000; max-rcmd-nalu-size=3980 3135 As the Offer/Answer negotiation covers both sending and receiving 3136 streams, an offer indicates the exact parameters for what the 3137 offerer is willing to receive, whereas the answer indicates the 3138 same for what the answerer accepts to receive. In this case the 3139 offerer declared that it is willing to receive payload type 98. 3140 The answerer accepts this by declaring an equivalent payload type 3141 97; i.e., it has identical values for the two parameters "profile- 3142 level-id" and "packetization-mode" (since "packetization-mode" is 3143 equal to 0, "sprop-deint-buf-req" is not present). As the offered 3144 payload type 98 is accepted, the answerer needs to store parameter 3145 sets included in sprop-parameter-sets= in 3146 case the offer finally decides to use this configuration. In the 3147 answer, the answerer includes the parameter sets in sprop- 3148 parameter-sets= that the answerer would use 3149 in the stream sent from the answerer if this configuration is 3150 finally used. 3152 The answerer also accepts the reception of the two configurations 3153 that payload types 99 and 100 represent. Again, the answerer needs 3154 to store parameter sets included in sprop-parameter-sets= and sprop-parameter-sets= in 3156 case the offer finally decides to use either of these two 3157 configurations. The answerer provides the initial parameter sets 3158 for the answerer-to-offerer direction, i.e. the parameter sets in 3159 sprop-parameter-sets= and sprop-parameter- 3160 sets=, for payload types 99 and 100, 3161 respectively, that it will use to send the payload types. The 3162 answerer also provides the offerer with its memory limit for de- 3163 interleaving operations by providing a "deint-buf-cap" parameter. 3164 This is only useful if the offerer decides on making a second offer, 3165 where it can take the new value into account. The "max-rcmd-nalu- 3166 size" indicates that the answerer can efficiently process NALUs up 3167 to the size of 3980 bytes. However, there is no guarantee that the 3168 network supports this size. 3170 In the following example, the offer is accepted without level 3171 downgrading (i.e. the default level, 3.0, is accepted), and both 3172 "sprop-parameter-sets" and "sprop-level-parameter-sets" are present 3173 in the offer. The answerer must ignore sprop-level-parameter- 3174 sets= and store parameter sets in sprop- 3175 parameter-sets= for decoding the incoming 3176 NAL unit stream. The offerer must store the parameter sets in 3177 sprop-parameter-sets= in the answer for 3178 decoding the incoming NAL unit stream. Note that in this example, 3179 parameter sets in sprop-parameter-sets= must 3180 be associated with level 3.0. 3182 Offer SDP: 3184 m=video 49170 RTP/AVP 98 3185 a=rtpmap:98 H264/90000 3186 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3187 packetization-mode=1; 3188 sprop-parameter-sets=; 3189 sprop-level-parameter-sets= 3191 Answer SDP: 3193 m=video 49170 RTP/AVP 98 3194 a=rtpmap:98 H264/90000 3195 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3196 packetization-mode=1; 3197 sprop-parameter-sets= 3199 In the following example, the offer (Baseline profile, level 1.1) 3200 is accepted with level downgrading (the accepted level is 1b), and 3201 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3202 present in the offer. The answerer must ignore sprop-parameter- 3203 sets= and all parameter sets not for the 3204 accepted level (level 1b) in sprop-level-parameter-sets=, and must store parameter sets for the accepted level 3206 (level 1b) in sprop-level-parameter-sets= 3207 for decoding the incoming NAL unit stream. The offerer must store 3208 the parameter sets in sprop-parameter-sets= 3209 in the answer for decoding the incoming NAL unit stream. Note that 3210 in this example, parameter sets in sprop-parameter-sets= must be associated with level 1b. 3213 Offer SDP: 3215 m=video 49170 RTP/AVP 98 3216 a=rtpmap:98 H264/90000 3217 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3218 packetization-mode=1; 3219 sprop-parameter-sets=; 3220 sprop-level-parameter-sets= 3222 Answer SDP: 3224 m=video 49170 RTP/AVP 98 3225 a=rtpmap:98 H264/90000 3226 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3227 packetization-mode=1; 3228 sprop-parameter-sets=; 3229 use-level-src-parameter-sets=1 3231 In the following example, the offer (Baseline profile, level 1.1) 3232 is accepted with level downgrading (the accepted level is 1b), and 3233 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3234 present in the offer. However, the answerer is a legacy RFC 3984 3235 implementation and does not understand "sprop-level-parameter-sets", 3236 hence it does not include "use-level-src-parameter-sets" (which the 3237 answerer does not understand, either) in the answer. Therefore, 3238 the answerer must ignore both sprop-parameter-sets= and sprop-level-parameter-sets=, and 3240 the offerer must transport parameter sets in-band. 3242 Offer SDP: 3244 m=video 49170 RTP/AVP 98 3245 a=rtpmap:98 H264/90000 3246 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3247 packetization-mode=1; 3248 sprop-parameter-sets=; 3249 sprop-level-parameter-sets= 3251 Answer SDP: 3253 m=video 49170 RTP/AVP 98 3254 a=rtpmap:98 H264/90000 3255 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3256 packetization-mode=1 3258 In the following example, the offer is accepted without level 3259 downgrading, and "sprop-parameter-sets" is present in the offer. 3261 Parameter sets in sprop-parameter-sets= must 3262 be stored and used used by the encoder of the offerer and the 3263 decoder of the answerer, and parameter sets in sprop-parameter- 3264 sets=must be used by the encoder of the 3265 answerer and the decoder of the offerer. Note that sprop- 3266 parameter-sets= is basically independent of 3267 sprop-parameter-sets=. 3269 Offer SDP: 3271 m=video 49170 RTP/AVP 98 3272 a=rtpmap:98 H264/90000 3273 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3274 packetization-mode=1; 3275 sprop-parameter-sets= 3277 Answer SDP: 3279 m=video 49170 RTP/AVP 98 3280 a=rtpmap:98 H264/90000 3281 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3282 packetization-mode=1; 3283 sprop-parameter-sets= 3285 In the following example, the offer is accepted without level 3286 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3287 parameter-sets" is present in the offer, meaning that there is no 3288 out-of-band transmission of parameter sets, which then have to be 3289 transported in-band. 3291 Offer SDP: 3293 m=video 49170 RTP/AVP 98 3294 a=rtpmap:98 H264/90000 3295 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3296 packetization-mode=1 3298 Answer SDP: 3300 m=video 49170 RTP/AVP 98 3301 a=rtpmap:98 H264/90000 3302 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3303 packetization-mode=1 3305 In the following example, the offer is accepted with level 3306 downgrading and "sprop-parameter-sets" is present in the offer. As 3307 sprop-parameter-sets= contains level_idc 3308 indicating Level 3.0, therefore cannot be used as the answerer 3309 wants Level 2.0 and must be ignored by the answerer, and in-band 3310 parameter sets must be used. 3312 Offer SDP: 3314 m=video 49170 RTP/AVP 98 3315 a=rtpmap:98 H264/90000 3316 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3317 packetization-mode=1; 3318 sprop-parameter-sets= 3320 Answer SDP: 3322 m=video 49170 RTP/AVP 98 3323 a=rtpmap:98 H264/90000 3324 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3325 packetization-mode=1 3327 In the following example, the offer is also accepted with level 3328 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3329 parameter-sets" is present in the offer, meaning that there is no 3330 out-of-band transmission of parameter sets, which then have to be 3331 transported in-band. 3333 Offer SDP: 3335 m=video 49170 RTP/AVP 98 3336 a=rtpmap:98 H264/90000 3337 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3338 packetization-mode=1 3340 Answer SDP: 3342 m=video 49170 RTP/AVP 98 3343 a=rtpmap:98 H264/90000 3344 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3345 packetization-mode=1 3347 In the following example, the offer is accepted with level 3348 upgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3349 parameter-sets" is present in the offer or the answer, meaning that 3350 there is no out-of-band transmission of parameter sets, which then 3351 have to be transported in-band. The level to use in the offerer- 3352 to-answerer direction is Level 3.0, and the level to use in the 3353 answerer-to-offerer direction is Level 2.0. The answerer is 3354 allowed to send at any level up to and including level 2.0, and the 3355 offerer is allowed to send at any level up to and including level 3356 3.0. 3358 Offer SDP: 3360 m=video 49170 RTP/AVP 98 3361 a=rtpmap:98 H264/90000 3362 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3363 packetization-mode=1; level-asymmetry-allowed=1 3365 Answer SDP: 3367 m=video 49170 RTP/AVP 98 3368 a=rtpmap:98 H264/90000 3369 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3370 packetization-mode=1; level-asymmetry-allowed=1 3372 In the following example, the offerer is a Multipoint Control Unit 3373 (MCU) in a Topo-Video-switch-MCU like topology [29], offering 3374 parameter sets received (using out-of-band transport) from three 3375 other participants B, C, and D, and receiving parameter sets from 3376 the participant A, which is the answerer. The participants are 3377 identified by their values of CNAME, which are mapped to different 3378 SSRC values. The same codec configuration is used by all the four 3379 participants. The participant A stores and associates the 3380 parameter sets included in , , and to participants B, C, and D, 3382 respectively, and uses for decoding NAL 3383 units carried in RTP packets originated from participant B only, 3384 uses for decoding NAL units carried in RTP 3385 packets originated from participant C only, and uses for decoding NAL units carried in RTP packets 3387 originated from participant D only. 3389 Offer SDP: 3391 m=video 49170 RTP/AVP 98 3392 a=ssrc:SSRC-B cname:CNAME-B 3393 a=ssrc:SSRC-C cname:CNAME-C 3394 a=ssrc:SSRC-D cname:CNAME-D 3395 a=ssrc:SSRC-B fmtp:98 3396 sprop-parameter-sets= 3397 a=ssrc:SSRC-C fmtp:98 3398 sprop-parameter-sets= 3399 a=ssrc:SSRC-D fmtp:98 3400 sprop-parameter-sets= 3401 a=rtpmap:98 H264/90000 3402 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3403 packetization-mode=1 3405 Answer SDP: 3407 m=video 49170 RTP/AVP 98 3408 a=ssrc:SSRC-A cname:CNAME-A 3409 a=ssrc:SSRC-A fmtp:98 3410 sprop-parameter-sets= 3411 a=rtpmap:98 H264/90000 3412 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3413 packetization-mode=1 3415 8.4. Parameter Set Considerations 3417 The H.264 parameter sets are a fundamental part of the video codec 3418 and vital to its operation; see section 1.2. Due to their 3419 characteristics and their importance for the decoding process, lost 3420 or erroneously transmitted parameter sets can hardly be concealed 3421 locally at the receiver. A reference to a corrupt parameter set 3422 has normally fatal results to the decoding process. Corruption 3423 could occur, for example, due to the erroneous transmission or loss 3424 of a parameter set NAL unit, but also due to the untimely 3425 transmission of a parameter set update. A parameter set update 3426 refers to a change of at least one parameter in a picture parameter 3427 set or sequence parameter set for which the picture parameter set 3428 or sequence parameter set identifier remains unchanged. Therefore, 3429 the following recommendations are provided as a guideline for the 3430 implementer of the RTP sender. 3432 Parameter set NALUs can be transported using three different 3433 principles: 3435 A. Using a session control protocol (out-of-band) prior to the 3436 actual RTP session. 3438 B. Using a session control protocol (out-of-band) during an ongoing 3439 RTP session. 3441 C. Within the RTP packet stream in the payload (in-band) during an 3442 ongoing RTP session. 3444 It is recommended to implement principles A and B within a session 3445 control protocol. SIP and SDP can be used as described in the SDP 3446 Offer/Answer model and in the previous sections of this memo. 3447 Section 8.2.2 includes a detailed discussion on transport of 3448 parameter sets in-band or out-of-band in SDP Offer/Answer using 3449 media type parameters "sprop-parameter-sets", "sprop-level- 3450 parameter-sets", "use-level-src-parameter-sets" and "in-band- 3451 parameter-sets". This section contains guidelines on how 3452 principles A and B should be implemented within session control 3453 protocols. It is independent of the particular protocol used. 3454 Principle C is supported by the RTP payload format defined in this 3455 specification. There are topologies like Topo-Video-switch-MCU [29] 3456 for which the use of principle C may be desirable. 3458 If in-band signaling of parameter sets is used, the picture and 3459 sequence parameter set NALUs SHOULD be transmitted in the RTP 3460 payload using a reliable method of delivering of RTP (see below), 3461 as a loss of a parameter set of either type will likely prevent 3462 decoding of a considerable portion of the corresponding RTP packet 3463 stream. 3465 If in-band signaling of parameter sets is used, the sender SHOULD 3466 take the error characteristics into account and use mechanisms to 3467 provide a high probability for delivering the parameter sets 3468 correctly. Mechanisms that increase the probability for a correct 3469 reception include packet repetition, FEC, and retransmission. The 3470 use of an unreliable, out-of-band control protocol has similar 3471 disadvantages as the in-band signaling (possible loss) and, in 3472 addition, may also lead to difficulties in the synchronization (see 3473 below). Therefore, it is NOT RECOMMENDED. 3475 Parameter sets MAY be added or updated during the lifetime of a 3476 session using principles B and C. It is required that parameter 3477 sets are present at the decoder prior to the NAL units that refer 3478 to them. Updating or adding of parameter sets can result in 3479 further problems, and therefore the following recommendations 3480 should be considered. 3482 - When parameter sets are added or updated, care SHOULD be taken 3483 to ensure that any parameter set is delivered prior to its usage. 3484 When new parameter sets are added, previously unused parameter 3485 set identifiers are used. It is common that no synchronization 3486 is present between out-of-band signaling and in-band traffic. 3487 If out-of-band signaling is used, it is RECOMMENDED that a 3488 sender does not start sending NALUs requiring the added or 3489 updated parameter sets prior to acknowledgement of delivery from 3490 the signaling protocol. 3492 - When parameter sets are updated, the following synchronization 3493 issue should be taken into account. When overwriting a 3494 parameter set at the receiver, the sender has to ensure that the 3495 parameter set in question is not needed by any NALU present in 3496 the network or receiver buffers. Otherwise, decoding with a 3497 wrong parameter set may occur. To lessen this problem, it is 3498 RECOMMENDED either to overwrite only those parameter sets that 3499 have not been used for a sufficiently long time (to ensure that 3500 all related NALUs have been consumed), or to add a new parameter 3501 set instead (which may have negative consequences for the 3502 efficiency of the video coding). 3504 Informative note: In some topologies like Topo-Video-switch- 3505 MCU [29] the origin of the whole set of parameter sets may 3506 come from multiple sources that may use non-unique parameter 3507 sets identifiers. In this case an offer may overwrite an 3508 existing parameter set if no other mechanism that enables 3509 uniqueness of the parameter sets in the out-of-band channel 3510 exists. 3512 - In a multiparty session, one participant MUST associate 3513 parameter sets coming from different sources with the source 3514 identification whenever possible, e.g. by conveying out-of-band 3515 transported parameter sets, as different sources typically use 3516 independent parameter set identifier value spaces. 3518 - Adding or modifying parameter sets by using both principles B 3519 and C in the same RTP session may lead to inconsistencies of the 3520 parameter sets because of the lack of synchronization between 3521 the control and the RTP channel. Therefore, principles B and C 3522 MUST NOT both be used in the same session unless sufficient 3523 synchronization can be provided. 3525 In some scenarios (e.g., when only the subset of this payload 3526 format specification corresponding to H.241 is used) or topologies, 3527 it is not possible to employ out-of-band parameter set transmission. 3529 In this case, parameter sets have to be transmitted in-band. Here, 3530 the synchronization with the non-parameter-set-data in the 3531 bitstream is implicit, but the possibility of a loss has to be 3532 taken into account. The loss probability should be reduced using 3533 the mechanisms discussed above. In case a loss of a parameter set 3534 is detected, recovery may be achieved by using a Decoder Refresh 3535 Point procedure, for example, using RTCP feedback Full Intra 3536 Request (FIR) [30]. Two example Decoder Refresh Point procedures 3537 are provided in the informative Section 8.5. 3539 - When parameter sets are initially provided using principle A and 3540 then later added or updated in-band (principle C), there is a 3541 risk associated with updating the parameter sets delivered out- 3542 of-band. If receivers miss some in-band updates (for example, 3543 because of a loss or a late tune-in), those receivers attempt to 3544 decode the bitstream using out-dated parameters. It is 3545 therefore RECOMMENDED that parameter set IDs be partitioned 3546 between the out-of-band and in-band parameter sets. 3548 8.5. Decoder Refresh Point Procedure using In-Band Transport of 3549 Parameter Sets (Informative) 3551 When a sender with a video encoder according to [1] receives a 3552 request for a decoder refresh point, the encoder shall enter the 3553 fast update mode by using one of the procedures specified 3554 in Section 8.5.1 or 8.5.2 below. The procedure in 8.5.1 is the 3555 preferred response in a lossless transmission environment. Both 3556 procedures satisfy the requirement to enter the fast update mode 3557 for H.264 video encoding. 3559 8.5.1. IDR Procedure to Respond to a Request for a Decoder Refresh 3560 Point 3562 This section gives one possible way to respond to a request for a 3563 decoder refresh point. 3565 The encoder shall, in the order presented here: 3567 1) Immediately prepare to send an IDR picture. 3569 2) Send a sequence parameter set to be used by the IDR picture to 3570 be sent. The encoder may optionally also send other sequence 3571 parameter sets. 3573 3) Send a picture parameter set to be used by the IDR picture to be 3574 sent. The encoder may optionally also send other picture 3575 parameter sets. 3577 4) Send the IDR picture. 3579 5) From this point forward in time, send any other sequence or 3580 picture parameter sets that have not yet been sent in this 3581 procedure, prior to their reference by any NAL unit, regardless 3582 of whether such parameter sets were previously sent prior to 3583 receiving the request for a decoder refresh point. As needed, 3584 such parameter sets may be sent in a batch, one at a time, or in 3585 any combination of these two methods. Parameter sets may be re- 3586 sent at any time for redundancy. Caution should be taken when 3587 parameter set updates are present, as described above in Section 3588 8.4. 3590 8.5.2. Gradual Recovery Procedure to Respond to a Request for a 3591 Decoder Refresh Point 3593 This section gives another possible way to respond to a request for 3594 a decoder refresh point. 3596 The encoder shall, in the order presented here: 3598 1) Send a recovery point SEI message (see Sections D.1.7 and D.2.7 3599 of [1]). 3601 2) Repeat any sequence and picture parameter sets that were sent 3602 before the recovery point SEI message, prior to their reference 3603 by a NAL unit. 3605 The encoder shall ensure that the decoder has access to all 3606 reference pictures for inter prediction of pictures at or after the 3607 recovery point, which is indicated by the recovery point SEI 3608 message, in output order, assuming that the transmission from now 3609 on is error-free. 3611 The value of the recovery_frame_cnt syntax element in the recovery 3612 point SEI message should be small enough to ensure a fast recovery. 3614 As needed, such parameter sets may be re-sent in a batch, one at a 3615 time, or in any combination of these two methods. Parameter sets 3616 may be re-sent at any time for redundancy. Caution should be taken 3617 when parameter set updates are present, as described above in 3618 Section 8.4. 3620 9. Security Considerations 3622 RTP packets using the payload format defined in this specification 3623 are subject to the security considerations discussed in the RTP 3624 specification [5], and in any appropriate RTP profile (for example, 3625 [16]). This implies that confidentiality of the media streams is 3626 achieved by encryption; for example, through the application of 3627 SRTP [26]. Because the data compression used with this payload 3628 format is applied end-to-end, any encryption needs to be performed 3629 after compression. A potential denial-of-service threat exists for 3630 data encodings using compression techniques that have non-uniform 3631 receiver-end computational load. The attacker can inject 3632 pathological datagrams into the stream that are complex to decode 3633 and that cause the receiver to be overloaded. H.264 is 3634 particularly vulnerable to such attacks, as it is extremely simple 3635 to generate datagrams containing NAL units that affect the decoding 3636 process of many future NAL units. Therefore, the usage of data 3637 origin authentication and data integrity protection of at least the 3638 RTP packet is RECOMMENDED; for example, with SRTP [26]. 3640 Note that the appropriate mechanism to ensure confidentiality and 3641 integrity of RTP packets and their payloads is very dependent on 3642 the application and on the transport and signaling protocols 3643 employed. Thus, although SRTP is given as an example above, other 3644 possible choices exist. 3646 Decoders MUST exercise caution with respect to the handling of user 3647 data SEI messages, particularly if they contain active elements, 3648 and MUST restrict their domain of applicability to the presentation 3649 containing the stream. 3651 End-to-End security with either authentication, integrity or 3652 confidentiality protection will prevent a MANE from performing 3653 media-aware operations other than discarding complete packets. And 3654 in the case of confidentiality protection it will even be prevented 3655 from performing discarding of packets in a media aware way. To 3656 allow any MANE to perform its operations, it will be required to be 3657 a trusted entity which is included in the security context 3658 establishment. 3660 10. Congestion Control 3662 Congestion control for RTP SHALL be used in accordance with RFC 3663 3550 [5], and with any applicable RTP profile; e.g., RFC 3551 [16]. 3664 An additional requirement if best-effort service is being used is: 3665 users of this payload format MUST monitor packet loss to ensure 3666 that the packet loss rate is within acceptable parameters. Packet 3667 loss is considered acceptable if a TCP flow across the same network 3668 path, and experiencing the same network conditions, would achieve 3669 an average throughput, measured on a reasonable timescale, that is 3670 not less than the RTP flow is achieving. This condition can be 3671 satisfied by implementing congestion control mechanisms to adapt 3672 the transmission rate (or the number of layers subscribed for a 3673 layered multicast session), or by arranging for a receiver to leave 3674 the session if the loss rate is unacceptably high. 3676 The bit rate adaptation necessary for obeying the congestion 3677 control principle is easily achievable when real-time encoding is 3678 used. However, when pre-encoded content is being transmitted, 3679 bandwidth adaptation requires the availability of more than one 3680 coded representation of the same content, at different bit rates, 3681 or the existence of non-reference pictures or sub-sequences [22] in 3682 the bitstream. The switching between the different representations 3683 can normally be performed in the same RTP session; e.g., by 3684 employing a concept known as SI/SP slices of the Extended Profile, 3685 or by switching streams at IDR picture boundaries. Only when non- 3686 downgradable parameters (such as the profile part of the 3687 profile/level ID) are required to be changed does it become 3688 necessary to terminate and re-start the media stream. This may be 3689 accomplished by using a different RTP payload type. 3691 MANEs MAY follow the suggestions outlined in section 7.3 and remove 3692 certain unusable packets from the packet stream when that stream 3693 was damaged due to previous packet losses. This can help reduce 3694 the network load in certain special cases. 3696 11. IANA Consideration 3698 The H264 media subtype name specified by RFC 3984 should be updated 3699 as defined in section 8.1 of this memo. 3701 12. Informative Appendix: Application Examples 3703 This payload specification is very flexible in its use, in order to 3704 cover the extremely wide application space anticipated for H.264. 3705 However, this great flexibility also makes it difficult for an 3706 implementer to decide on a reasonable packetization scheme. Some 3707 information on how to apply this specification to real-world 3708 scenarios is likely to appear in the form of academic publications 3709 and a test model software and description in the near future. 3710 However, some preliminary usage scenarios are described here as 3711 well. 3713 12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 3715 H.323-based video telephony systems that use H.264 as an optional 3716 video compression scheme are required to support H.241 Annex A [3] 3717 as a packetization scheme. The packetization mechanism defined in 3718 this Annex is technically identical with a small subset of this 3719 specification. 3721 When a system operates according to H.241 Annex A, parameter set 3722 NAL units are sent in-band. Only Single NAL unit packets are used. 3723 Many such systems are not sending IDR pictures regularly, but only 3724 when required by user interaction or by control protocol means; 3725 e.g., when switching between video channels in a Multipoint Control 3726 Unit or for error recovery requested by feedback. 3728 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 3729 Aggregation 3731 The RTP part of this scheme is implemented and tested (though not 3732 the control-protocol part; see below). 3734 In most real-world video telephony applications, picture parameters 3735 such as picture size or optional modes never change during the 3736 lifetime of a connection. Therefore, all necessary parameter sets 3737 (usually only one) are sent as a side effect of the capability 3738 exchange/announcement process, e.g., according to the SDP syntax 3739 specified in section 8.2 of this document. As all necessary 3740 parameter set information is established before the RTP session 3741 starts, there is no need for sending any parameter set NAL units. 3742 Slice data partitioning is not used, either. Thus, the RTP packet 3743 stream basically consists of NAL units that carry single coded 3744 slices. 3746 The encoder chooses the size of coded slice NAL units so that they 3747 offer the best performance. Often, this is done by adapting the 3748 coded slice size to the MTU size of the IP network. For small 3749 picture sizes, this may result in a one-picture-per-one-packet 3750 strategy. Intra refresh algorithms clean up the loss of packets 3751 and the resulting drift-related artifacts. 3753 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 3754 Aggregation 3756 This scheme allows better error concealment and is used in H.263 3757 based designs using RFC 4629 packetization [11]. It has been 3758 implemented, and good results were reported [13]. 3760 The VCL encoder codes the source picture so that all macroblocks 3761 (MBs) of one MB line are assigned to one slice. All slices with 3762 even MB row addresses are combined into one STAP, and all slices 3763 with odd MB row addresses into another. Those STAPs are 3764 transmitted as RTP packets. The establishment of the parameter 3765 sets is performed as discussed above. 3767 Note that the use of STAPs is essential here, as the high number of 3768 individual slices (18 for a CIF picture) would lead to unacceptably 3769 high IP/UDP/RTP header overhead (unless the source coding tool FMO 3770 is used, which is not assumed in this scenario). Furthermore, some 3771 wireless video transmission systems, such as H.324M and the IP- 3772 based video telephony specified in 3GPP, are likely to use 3773 relatively small transport packet size. For example, a typical MTU 3774 size of H.223 AL3 SDU is around 100 bytes [17]. Coding individual 3775 slices according to this packetization scheme provides further 3776 advantage in communication between wired and wireless networks, as 3777 individual slices are likely to be smaller than the preferred 3778 maximum packet size of wireless systems. Consequently, a gateway 3779 can convert the STAPs used in a wired network into several RTP 3780 packets with only one NAL unit, which are preferred in a wireless 3781 network, and vice versa. 3783 12.4. Video Telephony with Data Partitioning 3785 This scheme has been implemented and has been shown to offer good 3786 performance, especially at higher packet loss rates [13]. 3788 Data Partitioning is known to be useful only when some form of 3789 unequal error protection is available. Normally, in single-session 3790 RTP environments, even error characteristics are assumed; i.e., the 3791 packet loss probability of all packets of the session is the same 3792 statistically. However, there are means to reduce the packet loss 3793 probability of individual packets in an RTP session. A FEC packet 3794 according to RFC 2733 [18], for example, specifies which media 3795 packets are associated with the FEC packet. 3797 In all cases, the incurred overhead is substantial but is in the 3798 same order of magnitude as the number of bits that have otherwise 3799 been spent for intra information. However, this mechanism does not 3800 add any delay to the system. 3802 Again, the complete parameter set establishment is performed 3803 through control protocol means. 3805 12.5. Video Telephony or Streaming with FUs and Forward Error 3806 Correction 3808 This scheme has been implemented and has been shown to provide good 3809 performance, especially at higher packet loss rates [19]. 3811 The most efficient means to combat packet losses for scenarios 3812 where retransmissions are not applicable is forward error 3813 correction (FEC). Although application layer, end-to-end use of 3814 FEC is often less efficient than an FEC-based protection of 3815 individual links (especially when links of different 3816 characteristics are in the transmission path), application layer, 3817 end-to-end FEC is unavoidable in some scenarios. RFC 5109 [18] 3818 provides means to use generic, application layer, end-to-end FEC in 3819 packet-loss environments. A binary forward error correcting code 3820 is generated by applying the XOR operation to the bits at the same 3821 bit position in different packets. The binary code can be 3822 specified by the parameters (n,k) in which k is the number of 3823 information packets used in the connection and n is the total 3824 number of packets generated for k information packets; i.e., n-k 3825 parity packets are generated for k information packets. 3827 When a code is used with parameters (n,k) within the RFC 5109 3828 framework, the following properties are well known: 3830 a) If applied over one RTP packet, RFC 5109 provides only packet 3831 repetition. 3833 b) RFC 5109 is most bit rate efficient if XOR-connected packets 3834 have equal length. 3836 c) At the same packet loss probability p and for a fixed k, the 3837 greater the value of n is, the smaller the residual error 3838 probability becomes. For example, for a packet loss probability 3839 of 10%, k=1, and n=2, the residual error probability is about 1%, 3840 whereas for n=3, the residual error probability is about 0.1%. 3842 d) At the same packet loss probability p and for a fixed code rate 3843 k/n, the greater the value of n is, the smaller the residual 3844 error probability becomes. For example, at a packet loss 3845 probability of p=10%, k=1 and n=2, the residual error rate is 3846 about 1%, whereas for an extended Golay code with k=12 and n=24, 3847 the residual error rate is about 0.01%. 3849 For applying RFC 5109 in combination with H.264 baseline coded 3850 video without using FUs, several options might be considered: 3852 1) The video encoder produces NAL units for which each video frame 3853 is coded in a single slice. Applying FEC, one could use a 3854 simple code; e.g., (n=2, k=1). That is, each NAL unit would 3855 basically just be repeated. The disadvantage is obviously the 3856 bad code performance according to d), above, and the low 3857 flexibility, as only (n, k=1) codes can be used. 3859 2) The video encoder produces NAL units for which each video frame 3860 is encoded in one or more consecutive slices. Applying FEC, one 3861 could use a better code, e.g., (n=24, k=12), over a sequence of 3862 NAL units. Depending on the number of RTP packets per frame, a 3863 loss may introduce a significant delay, which is reduced when 3864 more RTP packets are used per frame. Packets of completely 3865 different length might also be connected, which decreases bit 3866 rate efficiency according to b), above. However, with some care 3867 and for slices of 1kb or larger, similar length (100-200 bytes 3868 difference) may be produced, which will not lower the bit 3869 efficiency catastrophically. 3871 3) The video encoder produces NAL units, for which a certain frame 3872 contains k slices of possibly almost equal length. Then, 3873 applying FEC, a better code, e.g., (n=24, k=12), can be used 3874 over the sequence of NAL units for each frame. The delay 3875 compared to that of 2), above, may be reduced, but several 3876 disadvantages are obvious. First, the coding efficiency of the 3877 encoded video is lowered significantly, as slice-structured 3878 coding reduces intra-frame prediction and additional slice 3879 overhead is necessary. Second, pre-encoded content or, when 3880 operating over a gateway, the video is usually not appropriately 3881 coded with k slices such that FEC can be applied. Finally, the 3882 encoding of video producing k slices of equal length is not 3883 straightforward and might require more than one encoding pass. 3885 Many of the mentioned disadvantages can be avoided by applying FUs 3886 in combination with FEC. Each NAL unit can be split into any 3887 number of FUs of basically equal length; therefore, FEC with a 3888 reasonable k and n can be applied, even if the encoder made no 3889 effort to produce slices of equal length. For example, a coded 3890 slice NAL unit containing an entire frame can be split to k FUs, 3891 and a parity check code (n=k+1, k) can be applied. However, this 3892 has the disadvantage that unless all created fragments can be 3893 recovered, the whole slice will be lost. Thus a larger section is 3894 lost than would be if the frame had been split into several slices. 3896 The presented technique makes it possible to achieve good 3897 transmission error tolerance, even if no additional source coding 3898 layer redundancy (such as periodic intra frames) is present. 3899 Consequently, the same coded video sequence can be used to achieve 3900 the maximum compression efficiency and quality over error-free 3901 transmission and for transmission over error-prone networks. 3902 Furthermore, the technique allows the application of FEC to pre- 3903 encoded sequences without adding delay. In this case, pre-encoded 3904 sequences that are not encoded for error-prone networks can still 3905 be transmitted almost reliably without adding extensive delays. In 3906 addition, FUs of equal length result in a bit rate efficient use of 3907 RFC 5109. 3909 If the error probability depends on the length of the transmitted 3910 packet (e.g., in case of mobile transmission [15]), the benefits of 3911 applying FUs with FEC are even more obvious. Basically, the 3912 flexibility of the size of FUs allows appropriate FEC to be applied 3913 for each NAL unit and unequal error protection of NAL units. 3915 When FUs and FEC are used, the incurred overhead is substantial but 3916 is in the same order of magnitude as the number of bits that have 3917 to be spent for intra-coded macroblocks if no FEC is applied. In 3918 [19], it was shown that the overall performance of the FEC-based 3919 approach enhanced quality when using the same error rate and same 3920 overall bit rate, including the overhead. 3922 12.6. Low Bit-Rate Streaming 3924 This scheme has been implemented with H.263 and non-standard RTP 3925 packetization and has given good results [20]. There is no 3926 technical reason why similarly good results could not be achievable 3927 with H.264. 3929 In today's Internet streaming, some of the offered bit rates are 3930 relatively low in order to allow terminals with dial-up modems to 3931 access the content. In wired IP networks, relatively large packets, 3932 say 500 - 1500 bytes, are preferred to smaller and more frequently 3933 occurring packets in order to reduce network congestion. Moreover, 3934 use of large packets decreases the amount of RTP/UDP/IP header 3935 overhead. For low bit-rate video, the use of large packets means 3936 that sometimes up to few pictures should be encapsulated in one 3937 packet. 3939 However, loss of a packet including many coded pictures would have 3940 drastic consequences for visual quality, as there is practically no 3941 other way to conceal a loss of an entire picture than to repeat the 3942 previous one. One way to construct relatively large packets and 3943 maintain possibilities for successful loss concealment is to 3944 construct MTAPs that contain interleaved slices from several 3945 pictures. An MTAP should not contain spatially adjacent slices 3946 from the same picture or spatially overlapping slices from any 3947 picture. If a packet is lost, it is likely that a lost slice is 3948 surrounded by spatially adjacent slices of the same picture and 3949 spatially corresponding slices of the temporally previous and 3950 succeeding pictures. Consequently, concealment of the lost slice 3951 is likely to be relatively successful. 3953 12.7. Robust Packet Scheduling in Video Streaming 3955 Robust packet scheduling has been implemented with MPEG-4 Part 2 3956 and simulated in a wireless streaming environment [21]. There is 3957 no technical reason why similar or better results could not be 3958 achievable with H.264. 3960 Streaming clients typically have a receiver buffer that is capable 3961 of storing a relatively large amount of data. Initially, when a 3962 streaming session is established, a client does not start playing 3963 the stream back immediately. Rather, it typically buffers the 3964 incoming data for a few seconds. This buffering helps maintain 3965 continuous playback, as, in case of occasional increased 3966 transmission delays or network throughput drops, the client can 3967 decode and play buffered data. Otherwise, without initial 3968 buffering, the client has to freeze the display, stop decoding, and 3969 wait for incoming data. The buffering is also necessary for either 3970 automatic or selective retransmission in any protocol level. If 3971 any part of a picture is lost, a retransmission mechanism may be 3972 used to resend the lost data. If the retransmitted data is 3973 received before its scheduled decoding or playback time, the loss 3974 is recovered perfectly. Coded pictures can be ranked according to 3975 their importance in the subjective quality of the decoded sequence. 3976 For example, non-reference pictures, such as conventional B 3977 pictures, are subjectively least important, as their absence does 3978 not affect decoding of any other pictures. In addition to non- 3979 reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard 3980 includes a temporal scalability method called sub-sequences [22]. 3981 Subjective ranking can also be made on coded slice data partition 3982 or slice group basis. Coded slices and coded slice data partitions 3983 that are subjectively the most important can be sent earlier than 3984 their decoding order indicates, whereas coded slices and coded 3985 slice data partitions that are subjectively the least important can 3986 be sent later than their natural coding order indicates. 3987 Consequently, any retransmitted parts of the most important slices 3988 and coded slice data partitions are more likely to be received 3989 before their scheduled decoding or playback time compared to the 3990 least important slices and slice data partitions. 3992 13. Informative Appendix: Rationale for Decoding Order Number 3994 13.1. Introduction 3996 The Decoding Order Number (DON) concept was introduced mainly to 3997 enable efficient multi-picture slice interleaving (see section 12.6) 3998 and robust packet scheduling (see section 12.7). In both of these 3999 applications, NAL units are transmitted out of decoding order. DON 4000 indicates the decoding order of NAL units and should be used in the 4001 receiver to recover the decoding order. Example use cases for 4002 efficient multi-picture slice interleaving and for robust packet 4003 scheduling are given in sections 13.2 and 13.3, respectively. 4004 Section 13.4 describes the benefits of the DON concept in error 4005 resiliency achieved by redundant coded pictures. Section 13.5 4006 summarizes considered alternatives to DON and justifies why DON was 4007 chosen to this RTP payload specification. 4009 13.2. Example of Multi-Picture Slice Interleaving 4011 An example of multi-picture slice interleaving follows. A subset 4012 of a coded video sequence is depicted below in output order. R 4013 denotes a reference picture, N denotes a non-reference picture, and 4014 the number indicates a relative output time. 4016 ... R1 N2 R3 N4 R5 ... 4018 The decoding order of these pictures from left to right is as 4019 follows: 4021 ... R1 R3 N2 R5 N4 ... 4023 The NAL units of pictures R1, R3, N2, R5, and N4 are marked with a 4024 DON equal to 1, 2, 3, 4, and 5, respectively. 4026 Each reference picture consists of three slice groups that are 4027 scattered as follows (a number denotes the slice group number for 4028 each macroblock in a QCIF frame): 4030 0 1 2 0 1 2 0 1 2 0 1 4031 2 0 1 2 0 1 2 0 1 2 0 4032 1 2 0 1 2 0 1 2 0 1 2 4033 0 1 2 0 1 2 0 1 2 0 1 4034 2 0 1 2 0 1 2 0 1 2 0 4035 1 2 0 1 2 0 1 2 0 1 2 4036 0 1 2 0 1 2 0 1 2 0 1 4037 2 0 1 2 0 1 2 0 1 2 0 4038 1 2 0 1 2 0 1 2 0 1 2 4040 For the sake of simplicity, we assume that all the macroblocks of a 4041 slice group are included in one slice. Three MTAPs are constructed 4042 from three consecutive reference pictures so that each MTAP 4043 contains three aggregation units, each of which contains all the 4044 macroblocks from one slice group. The first MTAP contains slice 4045 group 0 of picture R1, slice group 1 of picture R3, and slice group 4046 2 of picture R5. The second MTAP contains slice group 1 of picture 4047 R1, slice group 2 of picture R3, and slice group 0 of picture R5. 4048 The third MTAP contains slice group 2 of picture R1, slice group 0 4049 of picture R3, and slice group 1 of picture R5. Each non-reference 4050 picture is encapsulated into an STAP-B. 4052 Consequently, the transmission order of NAL units is the following: 4054 R1, slice group 0, DON 1, carried in MTAP,RTP SN: N 4055 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N 4056 R5, slice group 2, DON 4, carried in MTAP,RTP SN: N 4057 R1, slice group 1, DON 1, carried in MTAP,RTP SN: N+1 4058 R3, slice group 2, DON 2, carried in MTAP,RTP SN: N+1 4059 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+1 4060 R1, slice group 2, DON 1, carried in MTAP,RTP SN: N+2 4061 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N+2 4062 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+2 4063 N2, DON 3, carried in STAP-B, RTP SN: N+3 4064 N4, DON 5, carried in STAP-B, RTP SN: N+4 4066 The receiver is able to organize the NAL units back in decoding 4067 order based on the value of DON associated with each NAL unit. 4069 If one of the MTAPs is lost, the spatially adjacent and temporally 4070 co-located macroblocks are received and can be used to conceal the 4071 loss efficiently. If one of the STAPs is lost, the effect of the 4072 loss does not propagate temporally. 4074 13.3. Example of Robust Packet Scheduling 4076 An example of robust packet scheduling follows. The communication 4077 system used in the example consists of the following components in 4078 the order that the video is processed from source to sink: 4080 o camera and capturing 4081 o pre-encoding buffer 4082 o encoder 4083 o encoded picture buffer 4084 o transmitter 4085 o transmission channel 4086 o receiver 4087 o receiver buffer 4088 o decoder 4089 o decoded picture buffer 4090 o display 4092 The video communication system used in the example operates as 4093 follows. Note that processing of the video stream happens 4094 gradually and at the same time in all components of the system. 4095 The source video sequence is shot and captured to a pre-encoding 4096 buffer. The pre-encoding buffer can be used to order pictures from 4097 sampling order to encoding order or to analyze multiple 4098 uncompressed frames for bit rate control purposes, for example. In 4099 some cases, the pre-encoding buffer may not exist; instead, the 4100 sampled pictures are encoded right away. The encoder encodes 4101 pictures from the pre-encoding buffer and stores the output; i.e., 4102 coded pictures, to the encoded picture buffer. The transmitter 4103 encapsulates the coded pictures from the encoded picture buffer to 4104 transmission packets and sends them to a receiver through a 4105 transmission channel. The receiver stores the received packets to 4106 the receiver buffer. The receiver buffering process typically 4107 includes buffering for transmission delay jitter. The receiver 4108 buffer can also be used to recover correct decoding order of coded 4109 data. The decoder reads coded data from the receiver buffer and 4110 produces decoded pictures as output into the decoded picture buffer. 4111 The decoded picture buffer is used to recover the output (or 4112 display) order of pictures. Finally, pictures are displayed. 4114 In the following example figures, I denotes an IDR picture, R 4115 denotes a reference picture, N denotes a non-reference picture, and 4116 the number after I, R, or N indicates the sampling time relative to 4117 the previous IDR picture in decoding order. Values below the 4118 sequence of pictures indicate scaled system clock timestamps. The 4119 system clock is initialized arbitrarily in this example, and time 4120 runs from left to right. Each I, R, and N picture is mapped into 4121 the same timeline compared to the previous processing step, if any, 4122 assuming that encoding, transmission, and decoding take no time. 4123 Thus, events happening at the same time are located in the same 4124 column throughout all example figures. 4126 A subset of a sequence of coded pictures is depicted below in 4127 sampling order. 4129 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ... 4130 ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ... 4131 ... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ... 4133 Figure 16 Sequence of pictures in sampling order 4135 The sampled pictures are buffered in the pre-encoding buffer to 4136 arrange them in encoding order. In this example, we assume that 4137 the non-reference pictures are predicted from both the previous and 4138 the next reference picture in output order, except for the non- 4139 reference pictures immediately preceding an IDR picture, which are 4140 predicted only from the previous reference picture in output order. 4141 Thus, the pre-encoding buffer has to contain at least two pictures, 4142 and the buffering causes a delay of two picture intervals. The 4143 output of the pre-encoding buffering process and the encoding (and 4144 decoding) order of the pictures are as follows: 4146 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4147 ... -|---|---|---|---|---|---|---|---|- ... 4148 ... 60 61 62 63 64 65 66 67 68 ... 4150 Figure 17 Re-ordered pictures in the pre-encoding buffer 4152 The encoder or the transmitter can set the value of DON for each 4153 picture to a value of DON for the previous picture in decoding 4154 order plus one. 4156 For the sake of simplicity, let us assume that: 4158 o the frame rate of the sequence is constant, 4159 o each picture consists of only one slice, 4160 o each slice is encapsulated in a single NAL unit packet, 4161 o there is no transmission delay, and 4162 o pictures are transmitted at constant intervals (that is, 1 / 4163 (frame rate)). 4165 When pictures are transmitted in decoding order, they are received 4166 as follows: 4168 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4169 ... -|---|---|---|---|---|---|---|---|- ... 4170 ... 60 61 62 63 64 65 66 67 68 ... 4172 Figure 18 Received pictures in decoding order 4174 The OPTIONAL sprop-interleaving-depth media type parameter is set 4175 to 0, as the transmission (or reception) order is identical to the 4176 decoding order. 4178 The decoder has to buffer for one picture interval initially in its 4179 decoded picture buffer to organize pictures from decoding order to 4180 output order as depicted below: 4182 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... 4183 ... -|---|---|---|---|---|---|---|---|- ... 4184 ... 61 62 63 64 65 66 67 68 69 ... 4186 Figure 19 Output order 4188 The amount of required initial buffering in the decoded picture 4189 buffer can be signaled in the buffering period SEI message or with 4190 the num_reorder_frames syntax element of H.264 video usability 4191 information. num_reorder_frames indicates the maximum number of 4192 frames, complementary field pairs, or non-paired fields that 4193 precede any frame, complementary field pair, or non-paired field in 4194 the sequence in decoding order and that follow it in output order. 4195 For the sake of simplicity, we assume that num_reorder_frames is 4196 used to indicate the initial buffer in the decoded picture buffer. 4197 In this example, num_reorder_frames is equal to 1. 4199 It can be observed that if the IDR picture I00 is lost during 4200 transmission and a retransmission request is issued when the value 4201 of the system clock is 62, there is one picture interval of time 4202 (until the system clock reaches timestamp 63) to receive the 4203 retransmitted IDR picture I00. 4205 Let us then assume that IDR pictures are transmitted two frame 4206 intervals earlier than their decoding position; i.e., the pictures 4207 are transmitted as follows: 4209 ... I00 N58 N59 R03 N01 N02 R06 N04 N05 ... 4210 ... --|---|---|---|---|---|---|---|---|- ... 4211 ... 62 63 64 65 66 67 68 69 70 ... 4213 Figure 20 Interleaving: Early IDR pictures in sending order 4215 The OPTIONAL sprop-interleaving-depth media type parameter is set 4216 equal to 1 according to its definition. (The value of sprop- 4217 interleaving-depth in this example can be derived as follows: 4218 Picture I00 is the only picture preceding picture N58 or N59 in 4219 transmission order and following it in decoding order. Except for 4220 pictures I00, N58, and N59, the transmission order is the same as 4221 the decoding order of pictures. As a coded picture is encapsulated 4222 into exactly one NAL unit, the value of sprop-interleaving-depth is 4223 equal to the maximum number of pictures preceding any picture in 4224 transmission order and following the picture in decoding order.) 4226 The receiver buffering process contains two pictures at a time 4227 according to the value of the sprop-interleaving-depth parameter 4228 and orders pictures from the reception order to the correct 4229 decoding order based on the value of DON associated with each 4230 picture. The output of the receiver buffering process is as 4231 follows: 4233 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4234 ... -|---|---|---|---|---|---|---|---|- ... 4235 ... 63 64 65 66 67 68 69 70 71 ... 4237 Figure 21 Interleaving: Receiver buffer 4239 Again, an initial buffering delay of one picture interval is needed 4240 to organize pictures from decoding order to output order, as 4241 depicted below: 4243 ... N58 N59 I00 N01 N02 R03 N04 N05 ... 4244 ... -|---|---|---|---|---|---|---|- ... 4245 ... 64 65 66 67 68 69 70 71 ... 4247 Figure 22 Interleaving: Receiver buffer after reordering 4249 Note that the maximum delay that IDR pictures can undergo during 4250 transmission, including possible application, transport, or link 4251 layer retransmission, is equal to three picture intervals. Thus, 4252 the loss resiliency of IDR pictures is improved in systems 4253 supporting retransmission compared to the case in which pictures 4254 were transmitted in their decoding order. 4256 13.4. Robust Transmission Scheduling of Redundant Coded Slices 4258 A redundant coded picture is a coded representation of a picture or 4259 a part of a picture that is not used in the decoding process if the 4260 corresponding primary coded picture is correctly decoded. There 4261 should be no noticeable difference between any area of the decoded 4262 primary picture and a corresponding area that would result from 4263 application of the H.264 decoding process for any redundant picture 4264 in the same access unit. A redundant coded slice is a coded slice 4265 that is a part of a redundant coded picture. 4267 Redundant coded pictures can be used to provide unequal error 4268 protection in error-prone video transmission. If a primary coded 4269 representation of a picture is decoded incorrectly, a corresponding 4270 redundant coded picture can be decoded. Examples of applications 4271 and coding techniques using the redundant codec picture feature 4272 include the video redundancy coding [23] and the protection of "key 4273 pictures" in multicast streaming [24]. 4275 One property of many error-prone video communications systems is 4276 that transmission errors are often bursty. Therefore, they may 4277 affect more than one consecutive transmission packets in 4278 transmission order. In low bit-rate video communication, it is 4279 relatively common that an entire coded picture can be encapsulated 4280 into one transmission packet. Consequently, a primary coded 4281 picture and the corresponding redundant coded pictures may be 4282 transmitted in consecutive packets in transmission order. To make 4283 the transmission scheme more tolerant of bursty transmission errors, 4284 it is beneficial to transmit the primary coded picture and 4285 redundant coded picture separated by more than a single packet. 4286 The DON concept enables this. 4288 13.5. Remarks on Other Design Possibilities 4290 The slice header syntax structure of the H.264 coding standard 4291 contains the frame_num syntax element that can indicate the 4292 decoding order of coded frames. However, the usage of the 4293 frame_num syntax element is not feasible or desirable to recover 4294 the decoding order, due to the following reasons: 4296 o The receiver is required to parse at least one slice header per 4297 coded picture (before passing the coded data to the decoder). 4299 o Coded slices from multiple coded video sequences cannot be 4300 interleaved, as the frame number syntax element is reset to 0 in 4301 each IDR picture. 4303 o The coded fields of a complementary field pair share the same 4304 value of the frame_num syntax element. Thus, the decoding order 4305 of the coded fields of a complementary field pair cannot be 4306 recovered based on the frame_num syntax element or any other 4307 syntax element of the H.264 coding syntax. 4309 The RTP payload format for transport of MPEG-4 elementary streams 4310 [25] enables interleaving of access units and transmission of 4311 multiple access units in the same RTP packet. An access unit is 4312 specified in the H.264 coding standard to comprise all NAL units 4313 associated with a primary coded picture according to subclause 4314 7.4.1.2 of [1]. Consequently, slices of different pictures cannot 4315 be interleaved, and the multi-picture slice interleaving technique 4316 (see section 12.6) for improved error resilience cannot be used. 4318 14. Backward Compatibility to RFC 3984 4320 The current document is a revision of RFC 3984 and obsoletes it. 4321 The technical changes relative to RFC 3984 are listed in section 15. 4322 This section addresses the backward compatibility issues. 4324 It should be noted that for the majority of cases, there will be no 4325 compatibility issues for legacy implementations per RFC 3984 and 4326 new implementations per this document to interwork. Compatibility 4327 issues may only occur when both of the following conditions are 4328 true: 1) legacy implementations and new implementations are 4329 interworking; and 2) parameter sets are transported out of band. 4330 Even when such compatibility issues occur, it is easy to debug and 4331 find out the reason according to the following analyses. 4333 Items 1), 2), 3), 7), 9), 10), 12) and 13) are bug-fix type of 4334 changes, and do not incur any backward compatibility issues. 4336 Item 4), addition of six new media type parameters, does not incur 4337 any backward compatibility issues for SDP Offer/Answer based 4338 applications, as legacy RFC 3984 receivers ignore these parameters, 4339 and it is fine for legacy RFC 3984 senders not to use these 4340 parameters as they are optional. However, there is a backward 4341 compatibility issue for SDP declarative usage based applications 4342 (only for the parameter sprop-level-parameter-sets as the other 4343 five parameters are not usable in declarative usage), e.g. those 4344 using RTSP and SAP, because the SDP receiver per RFC 3984 cannot 4345 accept a session for which the SDP includes an unrecognized 4346 parameter. Therefore, the RTSP or SAP server may have to prepare 4347 two sets of streams, one for legacy RFC 3984 receivers and one for 4348 receivers according to this memo. 4350 Items 5), 6), and 11) are related to out-of-band transport of 4351 parameter sets. There are following backward compatibility issues. 4353 1) When a legacy sender per RFC 3984 includes parameter sets for a 4354 level different than the default level indicated by profile- 4355 level-id to sprop-parameter-sets, the parameter value of sprop- 4356 parameter-sets is invalid to the receiver per this memo and 4357 therefore the session may be rejected. 4359 2) In SDP Offer/Answer between a legacy offerer per RFC 3984 and an 4360 answerer per this memo, when the answerer includes in the answer 4361 parameter sets that are not a superset of the parameter sets 4362 included in the offer, the parameter value of sprop-parameter- 4363 sets is invalid to the offerer and the session may not be 4364 initiated properly (related to change item 11). 4366 3) When one endpoint A per this memo includes in-band-parameter- 4367 sets equal to 1, the other side B per RFC 3984 does not 4368 understand that it must transmit parameter sets in-band and B 4369 may still exclude parameter sets in the in-band stream it is 4370 sending. Consequently endpoint A cannot decode the stream it 4371 receives. 4373 Item7), allowance of conveying sprop-parameter-sets and sprop- 4374 level-parameter-sets using the "fmtp" source attribute as specified 4375 in section 6.3 of [9], is similar as item 4). It does not incur 4376 any backward compatibility issues for SDP Offer/Answer based 4377 applications, as legacy RFC 3984 receivers ignore the "fmtp" source 4378 attribute, and it is fine for legacy RFC 3984 senders not to use 4379 the "fmtp" source attribute as it is optional. However, there is a 4380 backward compatibility issue for SDP declarative usage based 4381 applications, e.g. those using RTSP and SAP, because the SDP 4382 receiver per RFC 3984 cannot accept a session for which the SDP 4383 includes an unrecognized parameter (i.e., the "fmtp" source 4384 attribute). Therefore, the RTSP or SAP server may have to prepare 4385 two sets of streams, one for legacy RFC 3984 receivers and one for 4386 receivers according to this memo. 4388 Item 14) removed that use of out-of-band transport of parameter 4389 sets is recommended. As out-of-band transport of parameter sets is 4390 still allowed, this change does not incur any backward 4391 compatibility issues. 4393 Item 15) does not incur any backward compatibility issues as the 4394 added subsection 8.5 is informative. 4396 Item 16) does not create any backward compatibility issues as the 4397 handling of default level is the same if either end is RFC 3984 4398 compliant, and furthermore, RFC 3984 compliant ends would simply 4399 ignore the new media type parameters, if present. 4401 15. Changes from RFC 3984 4403 Following is the list of technical changes (including bug fixes) 4404 from RFC 3984. Besides this list of technical changes, numerous 4405 editorial changes have been made, but not documented in this 4406 section. Note that section 8.2.2 is where much of the important 4407 changes in this memo occurs and deserves particular attention. 4409 1) In subsections 5.4, 5.5, 6.2, 6,3 and 6.4, removed that the 4410 packetization mode in use may be signaled by external means. 4412 2) In subsection 7.2.2, changed the sentence 4414 There are N VCL NAL units in the deinterleaving buffer. 4416 to 4418 There are N or more VCL NAL units in the de-interleaving buffer. 4420 3) In subsection 8.1, the semantics of sprop-init-buf-time, 4421 paragraph 2, changed the sentence 4423 The parameter is the maximum value of (transmission time of a 4424 NAL unit - decoding time of the NAL unit), assuming reliable and 4425 instantaneous transmission, the same timeline for transmission 4426 and decoding, and that decoding starts when the first packet 4427 arrives. 4429 to 4431 The parameter is the maximum value of (decoding time of the NAL 4432 unit - transmission time of a NAL unit), assuming reliable and 4433 instantaneous transmission, the same timeline for transmission 4434 and decoding, and that decoding starts when the first packet 4435 arrives. 4437 4) Added media type parameters max-smbps, sprop-level-parameter- 4438 sets, use-level-src-parameter-sets, in-band-parameter-sets, sar- 4439 understood and sar-supported. 4441 5) In subsection 8.1, removed the specification of parameter-add. 4442 Other descriptions of parameter-add (in subsections 8.2 and 8.4) 4443 are also removed. 4445 6) In subsection 8.1, added a constraint to sprop-parameter-sets 4446 such that it can only contain parameter sets for the same 4447 profile and level as indicated by profile-level-id. 4449 7) In subsection 8.2.1, added that sprop-parameter-sets and sprop- 4450 level-parameter-sets may be either included in the "a=fmtp" line 4451 of SDP or conveyed using the "fmtp" source attribute as 4452 specified in section 6.3 of [9]. 4454 8) In subsection 8.2.2, removed sprop-deint-buf-req from being part 4455 of the media format configuration in usage with the SDP 4456 Offer/Answer model. 4458 9) In subsection 8.2.2, made it clear that level is downgradable in 4459 the SDP Offer/Answer model, i.e. the use of the level part of 4460 "profile-level-id" does not need to be symmetric (the level 4461 included in the answer can be lower than or equal to the level 4462 included in the offer). 4464 10)In subsection 8.2.2, removed that the capability parameters may 4465 be used to declare encoding capabilities. 4467 11)In subsection 8.2.2, added rules on how to use sprop-parameter- 4468 sets and sprop-level-parameter-sets for out-of-band transport of 4469 parameter sets, with or without level downgrading. 4471 12)In subsection 8.2.2, clarified the rules of using the media type 4472 parameters with SDP Offer/Answer for multicast. 4474 13)In subsection 8.2.2, completed and corrected the list of how 4475 different media type parameters shall be interpreted in the 4476 different combinations of offer or answer and direction 4477 attribute. 4479 14)In subsection 8.4, changed the text such that both out-of-band 4480 and in-band transport of parameter sets are allowed and neither 4481 is recommended or required. 4483 15)Added subsection 8.5 (informative) providing example methods for 4484 decoder refresh to handle parameter set losses. 4486 16)Added media type parameters max-recv-level, and level-asymmetry- 4487 allowed, and adjusted associated text and examples for level 4488 upgrade and asymmetry. 4490 16. Acknowledgements 4492 Stephan Wenger, Miska Hannuksela, Thomas Stockhammer, Magnus 4493 Westerlund, and David Singer are thanked as the authors of RFC 3984. 4494 Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, 4495 Joerg Ott, and Colin Perkins are thanked for careful review during 4496 the development of RFC 3984. Stephen Botzko, Magnus Westerlund, 4497 Alex Eleftheriadis, Thomas Schierl, Tom Taylor, Ali Begen, Aaron 4498 Wells, Stuart Taylor, Robert Sparks, Dan Romascanu, and Niclas 4499 Comstedt are thanked for their valuable comments and inputs during 4500 the development of this memo. 4502 This document was prepared using 2-Word-v2.0.template.dot. 4504 17. References 4506 17.1. Normative References 4508 [1] ITU-T Recommendation H.264, "Advanced video coding for 4509 generic audiovisual services", November 2007. 4511 [2] ISO/IEC International Standard 14496-10:2008. 4513 [3] ITU-T Recommendation H.241, "Extended video procedures and 4514 control signals for H.300 series terminals", May 2006. 4516 [4] Bradner, S., "Key words for use in RFCs to Indicate 4517 Requirement Levels", BCP 14, RFC 2119, March 1997. 4519 [5] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, 4520 "RTP: A Transport Protocol for Real-Time Applications", STD 4521 64, RFC 3550, July 2003. 4523 [6] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 4524 Description Protocol", RFC 4566, July 2006. 4526 [7] Josefsson, S., "The Base16, Base32, and Base64 Data 4527 Encodings", RFC 4648, October 2006. 4529 [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with 4530 Session Description Protocol (SDP)", RFC 3264, June 2002. 4532 [9] Lennox, J., Ott, J., and Schierl, T., "Source-Specific Media 4533 Attributes in the Session Description Protocol (SDP)", RFC 4534 5576, June 2009. 4536 17.2. Informative References 4538 [10] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special 4539 Issue on H.264/AVC. IEEE Transactions on Circuits and Systems 4540 on Video Technology, July 2003. 4542 [11] Ott, J., Bormann, C., Sullivan, G., Wenger, S., and R. Even 4543 (Ed.), "RTP Payload Format for ITU-T Rec. H.263 Video", RFC 4544 4629, January 2007. 4546 [12] ISO/IEC IS 14496-2. 4548 [13] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and 4549 Systems for Video technology, Vol. 13, No. 7, July 2003. 4551 [14] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", 4552 Proceedings Packet Video Workshop 02, April 2002. 4554 [15] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT 4555 Coding Network Abstraction Layer and IP-based Transport" in 4556 Proc. ICIP 2002, Rochester, NY, September 2002. 4558 [16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 4559 Video Conferences with Minimal Control", STD 65, RFC 3551, 4560 July 2003. 4562 [17] ITU-T Recommendation H.223, "Multiplexing protocol for low 4563 bit rate multimedia communication", July 2001. 4565 [18] Li, A., "RTP Payload Format for Generic Forward Error 4566 Correction", RFC 5109, December 2007. 4568 [19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, 4569 "Video Coding and Transport Layer Techniques for H.264/AVC- 4570 Based Transmission over Packet-Lossy Networks", IEEE 4571 International Conference on Image Processing (ICIP 2003), 4572 Barcelona, Spain, September 2003. 4574 [20] Varsa, V. and M. Karczewicz, "Slice interleaving in 4575 compressed video packetization", Packet Video Workshop 2000. 4577 [21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for 4578 wireless video streaming," International Packet Video 4579 Workshop 2002. 4581 [22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, 4582 available http://ftp3.itu.int/av-arch/video- 4583 site/0201_Gen/JVT-B042.doc, anuary 2002. 4585 [23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 4586 International Workshop on Audio-Visual Services over Packet 4587 Networks, September 1997. 4589 [24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error 4590 Resilient Video Coding Using Unequally Protected Key 4591 Pictures", in Proc. International Workshop VLBV03, September 4592 2003. 4594 [25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., 4595 and P. Gentric, "RTP Payload Format for Transport of MPEG-4 4596 Elementary Streams", RFC 3640, November 2003. 4598 [26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 4599 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 4600 RFC 3711, March 2004. 4602 [27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time 4603 Streaming Protocol (RTSP)", RFC 2326, April 1998. 4605 [28] Handley, M., Perkins, C., and E. Whelan, "Session 4606 Announcement Protocol", RFC 2974, October 2000. 4608 [29] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, 4609 January 2008. 4611 [30] Wenger, S., Chandra, U., and M. Westerlund, "Codec Control 4612 Messages in the RTP Audio-Visual Profile with Feedback 4613 (AVPF)", RFC 5104, February 2008. 4615 18. Authors' Addresses 4617 Ye-Kui Wang 4618 Huawei Technologies 4619 400 Somerset Corp Blvd, Suite 602 4620 Bridgewater, NJ 08807 4621 USA 4623 Phone: +1-908-541-3518 4624 EMail: yekuiwang@huawei.com 4626 Roni Even 4627 14 David Hamelech 4628 Tel Aviv 64953 4629 Israel 4631 Phone: +972-545481099 4632 Email: ron.even.tlv@gmail.com 4634 Tom Kristensen 4635 TANDBERG 4636 Philip Pedersens vei 22 4637 N-1366 Lysaker 4638 Norway 4640 Phone: +47 67125125 4641 Email: tom.kristensen@tandberg.com, tomkri@ifi.uio.no 4643 Randell Jesup 4644 WorldGate Communications 4645 3190 Tremont Ave 4646 Trevose, PA 19053 4647 USA 4649 Phone: +1-215-354-5166 4650 Email: rjesup@wgate.com