idnits 2.17.1 draft-ietf-avt-rtp-rfc3984bis-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The abstract seems to indicate that this document obsoletes RFC3984, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 4, 2010) is 5135 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' ** Obsolete normative reference: RFC 4566 (ref. '6') (Obsoleted by RFC 8866) ** Obsolete normative reference: RFC 3548 (ref. '7') (Obsoleted by RFC 4648) -- Obsolete informational reference (is this intentional?): RFC 2326 (ref. '27') (Obsoleted by RFC 7826) -- Obsolete informational reference (is this intentional?): RFC 5117 (ref. '29') (Obsoleted by RFC 7667) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Obsoletes RFC 3984 2 Audio/Video Transport WG Y.-K. Wang 3 Internet Draft Huawei Technologies 4 Intended status: Standards track R. Even 5 Expires: October 2010 Self-employed 6 T. Kristensen 7 Tandberg 8 R. Jesup 9 WorldGate Communications 10 April 4, 2010 12 RTP Payload Format for H.264 Video 13 draft-ietf-avt-rtp-rfc3984bis-10.txt 15 Status of this Memo 17 This Internet-Draft is submitted to IETF in full conformance with 18 the provisions of BCP 78 and BCP 79. 20 Internet-Drafts are working documents of the Internet Engineering 21 Task Force (IETF), its areas, and its working groups. Note that 22 other groups may also distribute working documents as Internet- 23 Drafts. 25 Internet-Drafts are draft documents valid for a maximum of six 26 months and may be updated, replaced, or obsoleted by other 27 documents at any time. It is inappropriate to use Internet-Drafts 28 as reference material or to cite them other than as "work in 29 progress." 31 The list of current Internet-Drafts can be accessed at 32 http://www.ietf.org/ietf/1id-abstracts.txt. 34 The list of Internet-Draft Shadow Directories can be accessed at 35 http://www.ietf.org/shadow.html. 37 This Internet-Draft will expire on October 4, 2010. 39 Copyright Notice 41 Copyright (c) 2010 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with 49 respect to this document. Code Components extracted from this 50 document must include Simplified BSD License text as described in 51 Section 4.e of the Trust Legal Provisions and are provided without 52 warranty as described in the BSD License. 54 Abstract 56 This memo describes an RTP Payload format for the ITU-T 57 Recommendation H.264 video codec and the technically identical 58 ISO/IEC International Standard 14496-10 video codec, excluding the 59 Scalable Video Coding (SVC) extension and the Multivew Video Coding 60 extension, for which the RTP payload formats are defined elsewhere. 61 The RTP payload format allows for packetization of one or more 62 Network Abstraction Layer Units (NALUs), produced by an H.264 video 63 encoder, in each RTP payload. The payload format has wide 64 applicability, as it supports applications from simple low bit-rate 65 conversational usage, to Internet video streaming with interleaved 66 transmission, to high bit-rate video-on-demand. 68 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 69 in section 15. Issues on backward compatibility to RFC 3984 are 70 discussed in section 14. 72 Table of Contents 74 Table of Contents................................................2 75 1. Introduction..................................................4 76 1.1. The H.264 Codec..........................................4 77 1.2. Parameter Set Concept....................................6 78 1.3. Network Abstraction Layer Unit Types.....................6 79 2. Conventions...................................................7 80 3. Scope.........................................................8 81 4. Definitions and Abbreviations.................................8 82 4.1. Definitions..............................................8 83 4.2. Abbreviations...........................................10 84 5. RTP Payload Format...........................................11 85 5.1. RTP Header Usage........................................11 86 5.2. Payload Structures......................................13 87 5.3. NAL Unit Header Usage...................................14 88 5.4. Packetization Modes.....................................16 89 5.5. Decoding Order Number (DON).............................18 90 5.6. Single NAL Unit Packet..................................20 91 5.7. Aggregation Packets.....................................21 92 Table 4. Type field for STAPs and MTAPs........................22 93 5.7.1. Single-Time Aggregation Packet.....................23 94 5.7.2. Multi-Time Aggregation Packets (MTAPs).............26 95 5.7.3. Fragmentation Units (FUs)..........................29 96 6. Packetization Rules..........................................33 97 6.1. Common Packetization Rules..............................33 98 6.2. Single NAL Unit Mode....................................34 99 6.3. Non-Interleaved Mode....................................34 100 6.4. Interleaved Mode........................................35 101 7. De-Packetization Process.....................................35 102 7.1. Single NAL Unit and Non-Interleaved Mode................35 103 7.2. Interleaved Mode........................................36 104 7.2.1. Size of the De-interleaving Buffer.................36 105 7.2.2. De-interleaving Process............................37 106 7.3. Additional De-Packetization Guidelines..................38 107 8. Payload Format Parameters....................................39 108 8.1. Media Type Registration.................................39 109 8.2. SDP Parameters..........................................57 110 8.2.1. Mapping of Payload Type Parameters to SDP..........57 111 8.2.2. Usage with the SDP Offer/Answer Model..............59 112 8.2.3. Usage in Declarative Session Descriptions..........68 113 8.3. Examples................................................70 114 Offer SDP:......................................................75 115 Answer SDP:.....................................................76 116 8.4. Parameter Set Considerations............................77 117 8.5. Decoder Refresh Point Procedure using In-Band Transport of 118 Parameter Sets (Informative).................................80 119 8.5.1. IDR Procedure to Respond to a Request for a Decoder 120 Refresh Point.............................................80 121 8.5.2. Gradual Recovery Procedure to Respond to a Request for 122 a Decoder Refresh Point...................................81 123 9. Security Considerations......................................81 124 10. Congestion Control..........................................82 125 11. IANA Consideration..........................................83 126 12. Informative Appendix: Application Examples..................83 127 12.1. Video Telephony according to ITU-T Recommendation H.241 128 Annex A......................................................83 129 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 130 Aggregation..................................................84 131 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 132 Aggregation..................................................84 133 12.4. Video Telephony with Data Partitioning.................85 134 12.5. Video Telephony or Streaming with FUs and Forward Error 135 Correction...................................................85 136 12.6. Low Bit-Rate Streaming.................................88 137 12.7. Robust Packet Scheduling in Video Streaming............89 138 13. Informative Appendix: Rationale for Decoding Order Number...90 139 13.1. Introduction...........................................90 140 13.2. Example of Multi-Picture Slice Interleaving............90 141 13.3. Example of Robust Packet Scheduling....................92 142 13.4. Robust Transmission Scheduling of Redundant Coded Slices96 143 13.5. Remarks on Other Design Possibilities..................96 144 14. Backward Compatibility to RFC 3984..........................97 145 15. Changes from RFC 3984.......................................98 146 16. Acknowledgements...........................................100 147 17. References.................................................101 148 17.1. Normative References..................................101 149 17.2. Informative References................................101 150 18. Authors' Addresses.........................................103 152 1. Introduction 154 This memo specifies an RTP payload specification for the video 155 coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC 156 International Standard 14496 Part 10 [2] (both also known as 157 Advanced Video Coding, or AVC). In this memo the name H.264 is 158 used for the codec and the standard, but the memo is equally 159 applicable to the ISO/IEC counterpart of the coding standard. 161 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 162 in section 15. Issues on backward compatibility to RFC 3984 are 163 discussed in section 14. 165 1.1. The H.264 Codec 167 The H.264 video codec has a very broad application range that 168 covers all forms of digital compressed video, from low bit-rate 169 Internet streaming applications to HDTV broadcast and Digital 170 Cinema applications with nearly lossless coding. Compared to the 171 current state of technology, the overall performance of H.264 is 172 such that bit rate savings of 50% or more are reported. Digital 173 Satellite TV quality, for example, was reported to be achievable at 174 1.5 Mbit/s, compared to the current operation point of MPEG 2 video 175 at around 3.5 Mbit/s [10]. 177 The codec specification [1] itself distinguishes conceptually 178 between a video coding layer (VCL) and a network abstraction layer 179 (NAL). The VCL contains the signal processing functionality of the 180 codec; mechanisms such as transform, quantization, and motion 181 compensated prediction; and a loop filter. It follows the general 182 concept of most of today's video codecs, a macroblock-based coder 183 that uses inter picture prediction with motion compensation and 184 transform coding of the residual signal. The VCL encoder outputs 185 slices: a bit string that contains the macroblock data of an 186 integer number of macroblocks, and the information of the slice 187 header (containing the spatial address of the first macroblock in 188 the slice, the initial quantization parameter, and similar 189 information). Macroblocks in slices are arranged in scan order 190 unless a different macroblock allocation is specified, by using the 191 so-called Flexible Macroblock Ordering syntax. In-picture 192 prediction is used only within a slice. More information is 193 provided in [10]. 195 The Network Abstraction Layer (NAL) encoder encapsulates the slice 196 output of the VCL encoder into Network Abstraction Layer Units (NAL 197 units), which are suitable for transmission over packet networks or 198 use in packet oriented multiplex environments. Annex B of H.264 199 defines an encapsulation process to transmit such NAL units over 200 byte-stream oriented networks. In the scope of this memo, Annex B 201 is not relevant. 203 Internally, the NAL uses NAL units. A NAL unit consists of a one- 204 byte header and the payload byte string. The header indicates the 205 type of the NAL unit, the (potential) presence of bit errors or 206 syntax violations in the NAL unit payload, and information 207 regarding the relative importance of the NAL unit for the decoding 208 process. This RTP payload specification is designed to be unaware 209 of the bit string in the NAL unit payload. 211 One of the main properties of H.264 is the complete decoupling of 212 the transmission time, the decoding time, and the sampling or 213 presentation time of slices and pictures. The decoding process 214 specified in H.264 is unaware of time, and the H.264 syntax does 215 not carry information such as the number of skipped frames (as is 216 common in the form of the Temporal Reference in earlier video 217 compression standards). Also, there are NAL units that affect many 218 pictures and that are, therefore, inherently timeless. For this 219 reason, the handling of the RTP timestamp requires some special 220 considerations for NAL units for which the sampling or presentation 221 time is not defined or, at transmission time, unknown. 223 1.2. Parameter Set Concept 225 One very fundamental design concept of H.264 is to generate self- 226 contained packets, to make mechanisms such as the header 227 duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension 228 Code (HEC) [12] unnecessary. This was achieved by decoupling 229 information relevant to more than one slice from the media stream. 230 This higher layer meta information should be sent reliably, 231 asynchronously, and in advance from the RTP packet stream that 232 contains the slice packets. (Provisions for sending this 233 information in-band are also available for applications that do not 234 have an out-of-band transport channel appropriate for the purpose.) 235 The combination of the higher-level parameters is called a 236 parameter set. The H.264 specification includes two types of 237 parameter sets: sequence parameter set and picture parameter set. 238 An active sequence parameter set remains unchanged throughout a 239 coded video sequence, and an active picture parameter set remains 240 unchanged within a coded picture. The sequence and picture 241 parameter set structures contain information such as picture size, 242 optional coding modes employed, and macroblock to slice group map. 244 To be able to change picture parameters (such as the picture size) 245 without having to transmit parameter set updates synchronously to 246 the slice packet stream, the encoder and decoder can maintain a 247 list of more than one sequence and picture parameter set. Each 248 slice header contains a codeword that indicates the sequence and 249 picture parameter set to be used. 251 This mechanism allows the decoupling of the transmission of 252 parameter sets from the packet stream, and the transmission of them 253 by external means (e.g., as a side effect of the capability 254 exchange), or through a (reliable or unreliable) control protocol. 255 It may even be possible that they are never transmitted but are 256 fixed by an application design specification. 258 1.3. Network Abstraction Layer Unit Types 260 Tutorial information on the NAL design can be found in [13], [14], 261 and [15]. 263 All NAL units consist of a single NAL unit type octet, which also 264 co-serves as the payload header of this RTP payload format. The 265 payload of a NAL unit follows immediately. 267 The syntax and semantics of the NAL unit type octet are specified 268 in [1], but the essential properties of the NAL unit type octet are 269 summarized below. The NAL unit type octet has the following format: 271 +---------------+ 272 |0|1|2|3|4|5|6|7| 273 +-+-+-+-+-+-+-+-+ 274 |F|NRI| Type | 275 +---------------+ 277 The semantics of the components of the NAL unit type octet, as 278 specified in the H.264 specification, are described briefly below. 280 F: 1 bit 281 forbidden_zero_bit. The H.264 specification declares a value of 282 1 as a syntax violation. 284 NRI: 2 bits 285 nal_ref_idc. A value of 00 indicates that the content of the 286 NAL unit is not used to reconstruct reference pictures for inter 287 picture prediction. Such NAL units can be discarded without 288 risking the integrity of the reference pictures. Values greater 289 than 00 indicate that the decoding of the NAL unit is required 290 to maintain the integrity of the reference pictures. 292 Type: 5 bits 293 nal_unit_type. This component specifies the NAL unit payload 294 type as defined in Table 7-1 of [1], and later within this memo. 295 For a reference of all currently defined NAL unit types and 296 their semantics, please refer to section 7.4.1 in [1]. 298 This memo introduces new NAL unit types, which are presented in 299 section 5.2. The NAL unit types defined in this memo are marked as 300 unspecified in [1]. Moreover, this specification extends the 301 semantics of F and NRI as described in section 5.3. 303 2. Conventions 305 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 306 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 307 this document are to be interpreted as described in RFC 2119 [4]. 309 This specification uses the notion of setting and clearing a bit 310 when bit fields are handled. Setting a bit is the same as 311 assigning that bit the value of 1 (On). Clearing a bit is the same 312 as assigning that bit the value of 0 (Off). 314 3. Scope 316 This payload specification can only be used to carry the "naked" 317 H.264 NAL unit stream over RTP, and not the bitstream format 318 discussed in Annex B of H.264. Likely, the first applications of 319 this specification will be in the conversational multimedia field, 320 video telephony or video conferencing, but the payload format also 321 covers other applications, such as Internet streaming and TV over 322 IP. 324 4. Definitions and Abbreviations 326 4.1. Definitions 328 This document uses the definitions of [1]. The following terms, 329 defined in [1], are summed up for convenience: 331 access unit: A set of NAL units always containing a primary 332 coded picture. In addition to the primary coded picture, an 333 access unit may also contain one or more redundant coded 334 pictures or other NAL units not containing slices or slice data 335 partitions of a coded picture. The decoding of an access unit 336 always results in a decoded picture. 338 coded video sequence: A sequence of access units that consists, 339 in decoding order, of an instantaneous decoding refresh (IDR) 340 access unit followed by zero or more non-IDR access units 341 including all subsequent access units up to but not including 342 any subsequent IDR access unit. 344 IDR access unit: An access unit in which the primary coded 345 picture is an IDR picture. 347 IDR picture: A coded picture containing only slices with I or SI 348 slice types that causes a "reset" in the decoding process. 349 After the decoding of an IDR picture, all following coded 350 pictures in decoding order can be decoded without inter 351 prediction from any picture decoded prior to the IDR picture. 353 primary coded picture: The coded representation of a picture to 354 be used by the decoding process for a bitstream conforming to 355 H.264. The primary coded picture contains all macroblocks of 356 the picture. 358 redundant coded picture: A coded representation of a picture or 359 a part of a picture. The content of a redundant coded picture 360 shall not be used by the decoding process for a bitstream 361 conforming to H.264. The content of a redundant coded picture 362 may be used by the decoding process for a bitstream that 363 contains errors or losses. 365 VCL NAL unit: A collective term used to refer to coded slice and 366 coded data partition NAL units. 368 In addition, the following definitions apply: 370 decoding order number (DON): A field in the payload structure or 371 a derived variable indicating NAL unit decoding order. Values 372 of DON are in the range of 0 to 65535, inclusive. After 373 reaching the maximum value, the value of DON wraps around to 0. 375 NAL unit decoding order: A NAL unit order that conforms to the 376 constraints on NAL unit order given in section 7.4.1.2 in [1]. 378 NALU-time: The value that the RTP timestamp would have if the 379 NAL unit would be transported in its own RTP packet. 381 transmission order: The order of packets in ascending RTP 382 sequence number order (in modulo arithmetic). Within an 383 aggregation packet, the NAL unit transmission order is the same 384 as the order of appearance of NAL units in the packet. 386 media aware network element (MANE): A network element, such as a 387 middlebox or application layer gateway that is capable of 388 parsing certain aspects of the RTP payload headers or the RTP 389 payload and reacting to the contents. 391 Informative note: The concept of a MANE goes beyond normal 392 routers or gateways in that a MANE has to be aware of the 393 signaling (e.g., to learn about the payload type mappings of 394 the media streams), and in that it has to be trusted when 395 working with SRTP. The advantage of using MANEs is that they 396 allow packets to be dropped according to the needs of the 397 media coding. For example, if a MANE has to drop packets due 398 to congestion on a certain link, it can identify and remove 399 those packets whose elimination produces the least adverse 400 effect on the user experience. 402 static macroblock: A certain amount of macroblocks in the video 403 stream can be defined as static, as defined in section 8.3.2.8 404 in [3]. Static macroblocks free up additional processing 405 cycles for the handling of non-static macroblocks. Based on a 406 given amount of video processing resources and a given 407 resolution, a higher number of static macroblocks enables a 408 correspondingly higher frame rate. 410 default sub-profile: The subset of coding tools, which may be 411 all coding tools of one profile or the common subset of coding 412 tools of more than one profile, indicated by the profile-level- 413 id parameter. 415 default level: The level indicated by the profile-level-id 416 parameter, which consists of three octets, profile_idc, profile- 417 iop, and level_idc. The default level is indicated by level_idc 418 in most cases, and, in some cases, additionally by profile-iop. 420 4.2. Abbreviations 422 DON: Decoding Order Number 423 DONB: Decoding Order Number Base 424 DOND: Decoding Order Number Difference 425 FEC: Forward Error Correction 426 FU: Fragmentation Unit 427 IDR: Instantaneous Decoding Refresh 428 IEC: International Electrotechnical Commission 429 ISO: International Organization for Standardization 430 ITU-T: International Telecommunication Union, 431 Telecommunication Standardization Sector 432 MANE: Media Aware Network Element 433 MTAP: Multi-Time Aggregation Packet 434 MTAP16: MTAP with 16-bit timestamp offset 435 MTAP24: MTAP with 24-bit timestamp offset 436 NAL: Network Abstraction Layer 437 NALU: NAL Unit 438 SAR: Sample Aspect Ratio 439 SEI: Supplemental Enhancement Information 440 STAP: Single-Time Aggregation Packet 441 STAP-A: STAP type A 442 STAP-B: STAP type B 443 TS: Timestamp 444 VCL: Video Coding Layer 445 VUI: Video Usability Information 447 5. RTP Payload Format 449 5.1. RTP Header Usage 451 The format of the RTP header is specified in RFC 3550 [5] and 452 reprinted in Figure 1 for convenience. This payload format uses 453 the fields of the header in a manner consistent with that 454 specification. 456 When one NAL unit is encapsulated per RTP packet, the RECOMMENDED 457 RTP payload format is specified in section 5.6. The RTP payload 458 (and the settings for some RTP header bits) for aggregation packets 459 and fragmentation units are specified in sections 5.7 and 5.8, 460 respectively. 462 0 1 2 3 463 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 465 |V=2|P|X| CC |M| PT | sequence number | 466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 467 | timestamp | 468 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 469 | synchronization source (SSRC) identifier | 470 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 471 | contributing source (CSRC) identifiers | 472 | .... | 473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 475 Figure 1 RTP header according to RFC 3550 477 The RTP header information to be set according to this RTP payload 478 format is set as follows: 480 Marker bit (M): 1 bit 481 Set for the very last packet of the access unit indicated by the 482 RTP timestamp, in line with the normal use of the M bit in video 483 formats, to allow an efficient playout buffer handling. For 484 aggregation packets (STAP and MTAP), the marker bit in the RTP 485 header MUST be set to the value that the marker bit of the last 486 NAL unit of the aggregation packet would have been if it were 487 transported in its own RTP packet. Decoders MAY use this bit as 488 an early indication of the last packet of an access unit, but 489 MUST NOT rely on this property. 491 Informative note: Only one M bit is associated with an 492 aggregation packet carrying multiple NAL units. Thus, if a 493 gateway has re-packetized an aggregation packet into several 494 packets, it cannot reliably set the M bit of those packets. 496 Payload type (PT): 7 bits 497 The assignment of an RTP payload type for this new packet format 498 is outside the scope of this document and will not be specified 499 here. The assignment of a payload type has to be performed 500 either through the profile used or in a dynamic way. 502 Sequence number (SN): 16 bits 503 Set and used in accordance with RFC 3550. For the single NALU 504 and non-interleaved packetization mode, the sequence number is 505 used to determine decoding order for the NALU. 507 Timestamp: 32 bits 508 The RTP timestamp is set to the sampling timestamp of the 509 content. A 90 kHz clock rate MUST be used. 511 If the NAL unit has no timing properties of its own (e.g., 512 parameter set and SEI NAL units), the RTP timestamp is set to 513 the RTP timestamp of the primary coded picture of the access 514 unit in which the NAL unit is included, according to section 515 7.4.1.2 of [1]. 517 The setting of the RTP Timestamp for MTAPs is defined in section 518 5.7.2. 520 Receivers SHOULD ignore any picture timing SEI messages included 521 in access units that have only one display timestamp. Instead, 522 receivers SHOULD use the RTP timestamp for synchronizing the 523 display process. 525 If one access unit has more than one display timestamp carried 526 in a picture timing SEI message, then the information in the SEI 527 message SHOULD be treated as relative to the RTP timestamp, with 528 the earliest event occurring at the time given by the RTP 529 timestamp, and subsequent events later, as given by the 530 difference in SEI message picture timing values. Let tSEI1, 531 tSEI2, ..., tSEIn be the display timestamps carried in the SEI 532 message of an access unit, where tSEI1 is the earliest of all 533 such timestamps. Let tmadjst() be a function that adjusts the 534 SEI messages time scale to a 90-kHz time scale. Let TS be the 535 RTP timestamp. Then, the display time for the event associated 536 with tSEI1 is TS. The display time for the event with tSEIx, 537 where x is [2..n] is TS + tmadjst (tSEIx - tSEI1). 539 Informative note: Displaying coded frames as fields is needed 540 commonly in an operation known as 3:2 pulldown, in which film 541 content that consists of coded frames is displayed on a 542 display using interlaced scanning. The picture timing SEI 543 message enables carriage of multiple timestamps for the same 544 coded picture, and therefore the 3:2 pulldown process is 545 perfectly controlled. The picture timing SEI message 546 mechanism is necessary because only one timestamp per coded 547 frame can be conveyed in the RTP timestamp. 549 5.2. Payload Structures 551 The payload format defines three different basic payload structures. 552 A receiver can identify the payload structure by the first byte of 553 the RTP packet payload, which co-serves as the RTP payload header 554 and, in some cases, as the first byte of the payload. This byte is 555 always structured as a NAL unit header. The NAL unit type field 556 indicates which structure is present. The possible structures are 557 as follows: 559 Single NAL Unit Packet: Contains only a single NAL unit in the 560 payload. The NAL header type field will be equal to the original 561 NAL unit type; i.e., in the range of 1 to 23, inclusive. Specified 562 in section 5.6. 564 Aggregation Packet: Packet type used to aggregate multiple NAL 565 units into a single RTP payload. This packet exists in four 566 versions, the Single-Time Aggregation Packet type A (STAP-A), the 567 Single-Time Aggregation Packet type B (STAP-B), Multi-Time 568 Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and Multi- 569 Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The 570 NAL unit type numbers assigned for STAP-A, STAP-B, MTAP16, and 571 MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 572 5.7. 574 Fragmentation Unit: Used to fragment a single NAL unit over 575 multiple RTP packets. Exists with two versions, FU-A and FU-B, 576 identified with the NAL unit type numbers 28 and 29, respectively. 577 Specified in section 5.8. 579 Informative note: This specification does not limit the size of 580 NAL units encapsulated in single NAL unit packets and 581 fragmentation units. The maximum size of a NAL unit 582 encapsulated in any aggregation packet is 65535 bytes. 584 Table 1 summarizes NAL unit types and the corresponding RTP packet 585 types when each of these NAL units is directly used as a packet 586 payload, and where the types are described in this memo. 588 Table 1. Summary of NAL unit types and the corresponding packet 589 types 591 NAL Unit Packet Packet Type Name Section 592 Type Type 593 --------------------------------------------------------- 594 0 reserved - 595 1-23 NAL unit Single NAL unit packet 5.6 596 24 STAP-A Single-time aggregation packet 5.7.1 597 25 STAP-B Single-time aggregation packet 5.7.1 598 26 MTAP16 Multi-time aggregation packet 5.7.2 599 27 MTAP24 Multi-time aggregation packet 5.7.2 600 28 FU-A Fragmentation unit 5.8 601 29 FU-B Fragmentation unit 5.8 602 30-31 reserved - 604 5.3. NAL Unit Header Usage 606 The structure and semantics of the NAL unit header were introduced 607 in section 1.3. For convenience, the format of the NAL unit header 608 is reprinted below: 610 +---------------+ 611 |0|1|2|3|4|5|6|7| 612 +-+-+-+-+-+-+-+-+ 613 |F|NRI| Type | 614 +---------------+ 616 This section specifies the semantics of F and NRI according to this 617 specification. 619 F: 1 bit 620 forbidden_zero_bit. A value of 0 indicates that the NAL unit 621 type octet and payload should not contain bit errors or other 622 syntax violations. A value of 1 indicates that the NAL unit 623 type octet and payload may contain bit errors or other syntax 624 violations. 626 MANEs SHOULD set the F bit to indicate detected bit errors in 627 the NAL unit. The H.264 specification requires that the F bit 628 is equal to 0. When the F bit is set, the decoder is advised 629 that bit errors or any other syntax violations may be present in 630 the payload or in the NAL unit type octet. The simplest decoder 631 reaction to a NAL unit in which the F bit is equal to 1 is to 632 discard such a NAL unit and to conceal the lost data in the 633 discarded NAL unit. 635 NRI: 2 bits 636 nal_ref_idc. The semantics of value 00 and a non-zero value 637 remain unchanged from the H.264 specification. In other words, 638 a value of 00 indicates that the content of the NAL unit is not 639 used to reconstruct reference pictures for inter picture 640 prediction. Such NAL units can be discarded without risking the 641 integrity of the reference pictures. Values greater than 00 642 indicate that the decoding of the NAL unit is required to 643 maintain the integrity of the reference pictures. 645 In addition to the specification above, according to this RTP 646 payload specification, values of NRI indicate the relative 647 transport priority, as determined by the encoder. MANEs can use 648 this information to protect more important NAL units better than 649 they do less important NAL units. The highest transport 650 priority is 11, followed by 10, and then by 01; finally, 00 is 651 the lowest. 653 Informative note: Any non-zero value of NRI is handled 654 identically in H.264 decoders. Therefore, receivers need not 655 manipulate the value of NRI when passing NAL units to the 656 decoder. 658 An H.264 encoder MUST set the value of NRI according to the 659 H.264 specification (subclause 7.4.1) when the value of 660 nal_unit_type is in the range of 1 to 12, inclusive. In 661 particular, the H.264 specification requires that the value of 662 NRI SHALL be equal to 0 for all NAL units having nal_unit_type 663 equal to 6, 9, 10, 11, or 12. 665 For NAL units having nal_unit_type equal to 7 or 8 (indicating a 666 sequence parameter set or a picture parameter set, respectively), 667 an H.264 encoder SHOULD set the value of NRI to 11 (in binary 668 format). For coded slice NAL units of a primary coded picture 669 having nal_unit_type equal to 5 (indicating a coded slice 670 belonging to an IDR picture), an H.264 encoder SHOULD set the 671 value of NRI to 11 (in binary format). 673 For a mapping of the remaining nal_unit_types to NRI values, the 674 following example MAY be used and has been shown to be efficient 675 in a certain environment [14]. Other mappings MAY also be 676 desirable, depending on the application and the H.264/AVC Annex 677 A profile in use. 679 Informative note: Data Partitioning is not available in 680 certain profiles; e.g., in the Main or Baseline profiles. 681 Consequently, the NAL unit types 2, 3, and 4 can occur only 682 if the video bitstream conforms to a profile in which data 683 partitioning is allowed and not in streams that conform to 684 the Main or Baseline profiles. 686 Table 2. Example of NRI values for coded slices and coded slice 687 data partitions of primary coded reference pictures 689 NAL Unit Type Content of NAL unit NRI (binary) 690 ---------------------------------------------------------------- 691 1 non-IDR coded slice 10 692 2 Coded slice data partition A 10 693 3 Coded slice data partition B 01 694 4 Coded slice data partition C 01 696 Informative note: As mentioned before, the NRI value of non- 697 reference pictures is 00 as mandated by H.264/AVC. 699 An H.264 encoder SHOULD set the value of NRI for coded slice and 700 coded slice data partition NAL units of redundant coded 701 reference pictures equal to 01 (in binary format). 703 Definitions of the values for NRI for NAL unit types 24 to 29, 704 inclusive, are given in sections 5.7 and 5.8 of this memo. 706 No recommendation for the value of NRI is given for NAL units 707 having nal_unit_type in the range of 13 to 23, inclusive, 708 because these values are reserved for ITU-T and ISO/IEC. No 709 recommendation for the value of NRI is given for NAL units 710 having nal_unit_type equal to 0 or in the range of 30 to 31, 711 inclusive, as the semantics of these values are not specified in 712 this memo. 714 5.4. Packetization Modes 716 This memo specifies three cases of packetization modes: 718 o Single NAL unit mode 720 o Non-interleaved mode 721 o Interleaved mode 723 The single NAL unit mode is targeted for conversational systems 724 that comply with ITU-T Recommendation H.241 [3] (see section 12.1). 725 The non-interleaved mode is targeted for conversational systems 726 that may not comply with ITU-T Recommendation H.241. In the non- 727 interleaved mode, NAL units are transmitted in NAL unit decoding 728 order. The interleaved mode is targeted for systems that do not 729 require very low end-to-end latency. The interleaved mode allows 730 transmission of NAL units out of NAL unit decoding order. 732 The packetization mode in use MAY be signaled by the value of the 733 OPTIONAL packetization-mode media type parameter. The used 734 packetization mode governs which NAL unit types are allowed in RTP 735 payloads. Table 3 summarizes the allowed packet payload types for 736 each packetization mode. Packetization modes are explained in more 737 detail in section 6. 739 Table 3. Summary of allowed NAL unit types for each packetization 740 mode (yes = allowed, no = disallowed, ig = ignore) 742 Payload Packet Single NAL Non-Interleaved Interleaved 743 Type Type Unit Mode Mode Mode 744 ------------------------------------------------------------- 745 0 reserved ig ig ig 746 1-23 NAL unit yes yes no 747 24 STAP-A no yes no 748 25 STAP-B no no yes 749 26 MTAP16 no no yes 750 27 MTAP24 no no yes 751 28 FU-A no yes yes 752 29 FU-B no no yes 753 30-31 reserved ig ig ig 755 Some NAL unit or payload type values (indicated as reserved in 756 Table 3) are reserved for future extensions. NAL units of those 757 types SHOULD NOT be sent by a sender (direct as packet payloads, or 758 as aggregation units in aggregation packets, or as fragmented units 759 in FU packets) and MUST be ignored by a receiver. For example, the 760 payload types 1-23, with the associated packet type "NAL unit", are 761 allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", 762 but disallowed in "Interleaved Mode". However, NAL units of NAL 763 unit types 1-23 can be used in "Interleaved Mode" as aggregation 764 units in STAP-B, MTAP16 and MTAP24 packets as well as fragmented 765 units in FU-A and FU-B packets. Similarly, NAL units of NAL unit 766 types 1-23 can also be used in the "Non-Interleaved Mode" as 767 aggregation units in STAP-A packets or fragmented units in FU-A 768 packets, in addition to being directly used as packet payloads. 770 5.5. Decoding Order Number (DON) 772 In the interleaved packetization mode, the transmission order of 773 NAL units is allowed to differ from the decoding order of the NAL 774 units. Decoding order number (DON) is a field in the payload 775 structure or a derived variable that indicates the NAL unit 776 decoding order. Rationale and examples of use cases for 777 transmission out of decoding order and for the use of DON are given 778 in section 13. 780 The coupling of transmission and decoding order is controlled by 781 the OPTIONAL sprop-interleaving-depth media type parameter as 782 follows. When the value of the OPTIONAL sprop-interleaving-depth 783 media type parameter is equal to 0 (explicitly or per default), the 784 transmission order of NAL units MUST conform to the NAL unit 785 decoding order. When the value of the OPTIONAL sprop-interleaving- 786 depth media type parameter is greater than 0, 788 o the order of NAL units in an MTAP16 and an MTAP24 is not 789 required to be the NAL unit decoding order, and 791 o the order of NAL units generated by de-packetizing STAP-Bs, 792 MTAPs, and FUs in two consecutive packets is not required to be 793 the NAL unit decoding order. 795 The RTP payload structures for a single NAL unit packet, an STAP-A, 796 and an FU-A do not include DON. STAP-B and FU-B structures include 797 DON, and the structure of MTAPs enables derivation of DON as 798 specified in section 5.7.2. 800 Informative note: When an FU-A occurs in interleaved mode, it 801 always follows an FU-B, which sets its DON. 803 Informative note: If a transmitter wants to encapsulate a single 804 NAL unit per packet and transmit packets out of their decoding 805 order, STAP-B packet type can be used. 807 In the single NAL unit packetization mode, the transmission order 808 of NAL units, determined by the RTP sequence number, MUST be the 809 same as their NAL unit decoding order. In the non-interleaved 810 packetization mode, the transmission order of NAL units in single 811 NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL 812 unit decoding order. The NAL units within an STAP MUST appear in 813 the NAL unit decoding order. Thus, the decoding order is first 814 provided through the implicit order within a STAP, and second 815 provided through the RTP sequence number for the order between 816 STAPs, FUs, and single NAL unit packets. 818 Signaling of the value of DON for NAL units carried in STAP-B, MTAP, 819 and a series of fragmentation units starting with an FU-B is 820 specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON 821 value of the first NAL unit in transmission order MAY be set to any 822 value. Values of DON are in the range of 0 to 65535, inclusive. 823 After reaching the maximum value, the value of DON wraps around to 824 0. 826 The decoding order of two NAL units contained in any STAP-B, MTAP, 827 or a series of fragmentation units starting with an FU-B is 828 determined as follows. Let DON(i) be the decoding order number of 829 the NAL unit having index i in the transmission order. Function 830 don_diff(m,n) is specified as follows: 832 If DON(m) == DON(n), don_diff(m,n) = 0 834 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 835 don_diff(m,n) = DON(n) - DON(m) 837 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 838 don_diff(m,n) = 65536 - DON(m) + DON(n) 840 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 841 don_diff(m,n) = - (DON(m) + 65536 - DON(n)) 843 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 844 don_diff(m,n) = - (DON(m) - DON(n)) 846 A positive value of don_diff(m,n) indicates that the NAL unit 847 having transmission order index n follows, in decoding order, the 848 NAL unit having transmission order index m. When don_diff(m,n) is 849 equal to 0, then the NAL unit decoding order of the two NAL units 850 can be in either order. A negative value of don_diff(m,n) 851 indicates that the NAL unit having transmission order index n 852 precedes, in decoding order, the NAL unit having transmission order 853 index m. 855 Values of DON related fields (DON, DONB, and DOND; see section 5.7) 856 MUST be such that the decoding order determined by the values of 857 DON, as specified above, conforms to the NAL unit decoding order. 859 If the order of two NAL units in NAL unit decoding order is 860 switched and the new order does not conform to the NAL unit 861 decoding order, the NAL units MUST NOT have the same value of DON. 862 If the order of two consecutive NAL units in the NAL unit stream is 863 switched and the new order still conforms to the NAL unit decoding 864 order, the NAL units MAY have the same value of DON. For example, 865 when arbitrary slice order is allowed by the video coding profile 866 in use, all the coded slice NAL units of a coded picture are 867 allowed to have the same value of DON. Consequently, NAL units 868 having the same value of DON can be decoded in any order, and two 869 NAL units having a different value of DON should be passed to the 870 decoder in the order specified above. When two consecutive NAL 871 units in the NAL unit decoding order have a different value of DON, 872 the value of DON for the second NAL unit in decoding order SHOULD 873 be the value of DON for the first, incremented by one. 875 An example of the de-packetization process to recover the NAL unit 876 decoding order is given in section 7. 878 Informative note: Receivers should not expect that the absolute 879 difference of values of DON for two consecutive NAL units in the 880 NAL unit decoding order will be equal to one, even in error-free 881 transmission. An increment by one is not required, as at the 882 time of associating values of DON to NAL units, it may not be 883 known whether all NAL units are delivered to the receiver. For 884 example, a gateway may not forward coded slice NAL units of non- 885 reference pictures or SEI NAL units when there is a shortage of 886 bit rate in the network to which the packets are forwarded. In 887 another example, a live broadcast is interrupted by pre-encoded 888 content, such as commercials, from time to time. The first 889 intra picture of a pre-encoded clip is transmitted in advance to 890 ensure that it is readily available in the receiver. When 891 transmitting the first intra picture, the originator does not 892 exactly know how many NAL units will be encoded before the first 893 intra picture of the pre-encoded clip follows in decoding order. 894 Thus, the values of DON for the NAL units of the first intra 895 picture of the pre-encoded clip have to be estimated when they 896 are transmitted, and gaps in values of DON may occur. 898 5.6. Single NAL Unit Packet 900 The single NAL unit packet defined here MUST contain only one NAL 901 unit, of the types defined in [1]. This means that neither an 902 aggregation packet nor a fragmentation unit can be used within a 903 single NAL unit packet. A NAL unit stream composed by de- 904 packetizing single NAL unit packets in RTP sequence number order 905 MUST conform to the NAL unit decoding order. The structure of the 906 single NAL unit packet is shown in Figure 2. 908 Informative note: The first byte of a NAL unit co-serves as the 909 RTP payload header. 911 0 1 2 3 912 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 913 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 914 |F|NRI| Type | | 915 +-+-+-+-+-+-+-+-+ | 916 | | 917 | Bytes 2..n of a Single NAL unit | 918 | | 919 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 920 | :...OPTIONAL RTP padding | 921 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 923 Figure 2 RTP payload format for single NAL unit packet 925 5.7. Aggregation Packets 927 Aggregation packets are the NAL unit aggregation scheme of this 928 payload specification. The scheme is introduced to reflect the 929 dramatically different MTU sizes of two key target networks: 930 wireline IP networks (with an MTU size that is often limited by the 931 Ethernet MTU size; roughly 1500 bytes), and IP or non-IP (e.g., 932 ITU-T H.324/M) based wireless communication systems with preferred 933 transmission unit sizes of 254 bytes or less. To prevent media 934 transcoding between the two worlds, and to avoid undesirable 935 packetization overhead, a NAL unit aggregation scheme is introduced. 937 Two types of aggregation packets are defined by this specification: 939 o Single-time aggregation packet (STAP): aggregates NAL units with 940 identical NALU-time. Two types of STAPs are defined, one 941 without DON (STAP-A) and another including DON (STAP-B). 943 o Multi-time aggregation packet (MTAP): aggregates NAL units with 944 potentially differing NALU-time. Two different MTAPs are 945 defined, differing in the length of the NAL unit timestamp 946 offset. 948 Each NAL unit to be carried in an aggregation packet is 949 encapsulated in an aggregation unit. Please see below for the four 950 different aggregation units and their characteristics. 952 The structure of the RTP payload format for aggregation packets is 953 presented in Figure 3. 955 0 1 2 3 956 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 957 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 958 |F|NRI| Type | | 959 +-+-+-+-+-+-+-+-+ | 960 | | 961 | one or more aggregation units | 962 | | 963 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 964 | :...OPTIONAL RTP padding | 965 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 967 Figure 3 RTP payload format for aggregation packets 969 MTAPs and STAPs share the following packetization rules: The RTP 970 timestamp MUST be set to the earliest of the NALU-times of all the 971 NAL units to be aggregated. The type field of the NAL unit type 972 octet MUST be set to the appropriate value, as indicated in Table 4. 973 The F bit MUST be cleared if all F bits of the aggregated NAL units 974 are zero; otherwise, it MUST be set. The value of NRI MUST be the 975 maximum of all the NAL units carried in the aggregation packet. 977 Table 4. Type field for STAPs and MTAPs 979 Type Packet Timestamp offset DON related fields 980 field length (DON, DONB, DOND) 981 (in bits) present 982 -------------------------------------------------------- 983 24 STAP-A 0 no 984 25 STAP-B 0 yes 985 26 MTAP16 16 yes 986 27 MTAP24 24 yes 988 The marker bit in the RTP header is set to the value that the 989 marker bit of the last NAL unit of the aggregated packet would have 990 if it were transported in its own RTP packet. 992 The payload of an aggregation packet consists of one or more 993 aggregation units. See sections 5.7.1 and 5.7.2 for the four 994 different types of aggregation units. An aggregation packet can 995 carry as many aggregation units as necessary; however, the total 996 amount of data in an aggregation packet obviously MUST fit into an 997 IP packet, and the size SHOULD be chosen so that the resulting IP 998 packet is smaller than the MTU size. An aggregation packet MUST 999 NOT contain fragmentation units specified in section 5.8. 1000 Aggregation packets MUST NOT be nested; i.e., an aggregation packet 1001 MUST NOT contain another aggregation packet. 1003 5.7.1. Single-Time Aggregation Packet 1005 Single-time aggregation packet (STAP) SHOULD be used whenever NAL 1006 units are aggregated that all share the same NALU-time. The 1007 payload of an STAP-A does not include DON and consists of at least 1008 one single-time aggregation unit, as presented in Figure 4. The 1009 payload of an STAP-B consists of a 16-bit unsigned decoding order 1010 number (DON) (in network byte order) followed by at least one 1011 single-time aggregation unit, as presented in Figure 5. 1013 0 1 2 3 1014 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1015 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1016 : | 1017 +-+-+-+-+-+-+-+-+ | 1018 | | 1019 | single-time aggregation units | 1020 | | 1021 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1022 | : 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1025 Figure 4 Payload format for STAP-A 1027 0 1 2 3 1028 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1029 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1030 : decoding order number (DON) | | 1031 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1032 | | 1033 | single-time aggregation units | 1034 | | 1035 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1036 | : 1037 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1039 Figure 5 Payload format for STAP-B 1041 The DON field specifies the value of DON for the first NAL unit in 1042 an STAP-B in transmission order. For each successive NAL unit in 1043 appearance order in an STAP-B, the value of DON is equal to (the 1044 value of DON of the previous NAL unit in the STAP-B + 1) % 65536, 1045 in which '%' stands for the modulo operation. 1047 A single-time aggregation unit consists of 16-bit unsigned size 1048 information (in network byte order) that indicates the size of the 1049 following NAL unit in bytes (excluding these two octets, but 1050 including the NAL unit type octet of the NAL unit), followed by the 1051 NAL unit itself, including its NAL unit type byte. A single-time 1052 aggregation unit is byte aligned within the RTP payload, but it may 1053 not be aligned on a 32-bit word boundary. Figure 6 presents the 1054 structure of the single-time aggregation unit. 1056 0 1 2 3 1057 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1058 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 : NAL unit size | | 1060 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1061 | | 1062 | NAL unit | 1063 | | 1064 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1065 | : 1066 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1068 Figure 6 Structure for single-time aggregation unit 1070 Figure 7 presents an example of an RTP packet that contains an 1071 STAP-A. The STAP contains two single-time aggregation units, 1072 labeled as 1 and 2 in the figure. 1074 0 1 2 3 1075 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 | RTP Header | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 |STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | NALU 1 Data | 1082 : : 1083 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 | | NALU 2 Size | NALU 2 HDR | 1085 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1086 | NALU 2 Data | 1087 : : 1088 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1089 | :...OPTIONAL RTP padding | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 Figure 7 An example of an RTP packet including an STAP-A containing 1093 two single-time aggregation units 1095 Figure 8 presents an example of an RTP packet that contains an 1096 STAP-B. The STAP contains two single-time aggregation units, 1097 labeled as 1 and 2 in the figure. 1099 0 1 2 3 1100 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1101 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 | RTP Header | 1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 |STAP-B NAL HDR | DON | NALU 1 Size | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1106 | NALU 1 Size | NALU 1 HDR | NALU 1 Data | 1107 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1108 : : 1109 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1110 | | NALU 2 Size | NALU 2 HDR | 1111 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1112 | NALU 2 Data | 1113 : : 1114 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1115 | :...OPTIONAL RTP padding | 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1118 Figure 8 An example of an RTP packet including an STAP-B containing 1119 two single-time aggregation units 1121 5.7.2. Multi-Time Aggregation Packets (MTAPs) 1123 The NAL unit payload of MTAPs consists of a 16-bit unsigned 1124 decoding order number base (DONB) (in network byte order) and one 1125 or more multi-time aggregation units, as presented in Figure 9. 1126 DONB MUST contain the value of DON for the first NAL unit in the 1127 NAL unit decoding order among the NAL units of the MTAP. 1129 Informative note: The first NAL unit in the NAL unit decoding 1130 order is not necessarily the first NAL unit in the order in 1131 which the NAL units are encapsulated in an MTAP. 1133 0 1 2 3 1134 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1135 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1136 : decoding order number base | | 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1138 | | 1139 | multi-time aggregation units | 1140 | | 1141 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1142 | : 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1145 Figure 9 NAL unit payload format for MTAPs 1147 Two different multi-time aggregation units are defined in this 1148 specification. Both of them consist of 16 bits unsigned size 1149 information of the following NAL unit (in network byte order), an 1150 8-bit unsigned decoding order number difference (DOND), and n bits 1151 (in network byte order) of timestamp offset (TS offset) for this 1152 NAL unit, whereby n can be 16 or 24. The choice between the 1153 different MTAP types (MTAP16 and MTAP24) is application dependent: 1154 the larger the timestamp offset is, the higher the flexibility of 1155 the MTAP, but the overhead is also higher. 1157 The structure of the multi-time aggregation units for MTAP16 and 1158 MTAP24 are presented in Figures 10 and 11, respectively. The 1159 starting or ending position of an aggregation unit within a packet 1160 is not required to be on a 32-bit word boundary. The DON of the 1161 NAL unit contained in a multi-time aggregation unit is equal to 1162 (DONB + DOND) % 65536, in which % denotes the modulo operation. 1163 This memo does not specify how the NAL units within an MTAP are 1164 ordered, but, in most cases, NAL unit decoding order SHOULD be used. 1166 The timestamp offset field MUST be set to a value equal to the 1167 value of the following formula: If the NALU-time is larger than or 1168 equal to the RTP timestamp of the packet, then the timestamp offset 1169 equals (the NALU-time of the NAL unit - the RTP timestamp of the 1170 packet). If the NALU-time is smaller than the RTP timestamp of the 1171 packet, then the timestamp offset is equal to the NALU-time + (2^32 1172 - the RTP timestamp of the packet). 1174 0 1 2 3 1175 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1176 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 : NAL unit size | DOND | TS offset | 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1179 | TS offset | | 1180 +-+-+-+-+-+-+-+-+ NAL unit | 1181 | | 1182 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1183 | : 1184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1186 Figure 10 Multi-time aggregation unit for MTAP16 1188 0 1 2 3 1189 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1190 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1191 : NAL unit size | DOND | TS offset | 1192 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1193 | TS offset | | 1194 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1195 | NAL unit | 1196 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1197 | : 1198 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1200 Figure 11 Multi-time aggregation unit for MTAP24 1202 For the "earliest" multi-time aggregation unit in an MTAP the 1203 timestamp offset MUST be zero. Hence, the RTP timestamp of the 1204 MTAP itself is identical to the earliest NALU-time. 1206 Informative note: The "earliest" multi-time aggregation unit is 1207 the one that would have the smallest extended RTP timestamp 1208 among all the aggregation units of an MTAP if the NAL units 1209 contained in the aggregation units were encapsulated in single 1210 NAL unit packets. An extended timestamp is a timestamp that has 1211 more than 32 bits and is capable of counting the wraparound of 1212 the timestamp field, thus enabling one to determine the smallest 1213 value if the timestamp wraps. Such an "earliest" aggregation 1214 unit may not be the first one in the order in which the 1215 aggregation units are encapsulated in an MTAP. The "earliest" 1216 NAL unit need not be the same as the first NAL unit in the NAL 1217 unit decoding order either. 1219 Figure 12 presents an example of an RTP packet that contains a 1220 multi-time aggregation packet of type MTAP16 that contains two 1221 multi-time aggregation units, labeled as 1 and 2 in the figure. 1223 0 1 2 3 1224 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1225 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1226 | RTP Header | 1227 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1228 |MTAP16 NAL HDR | decoding order number base | NALU 1 Size | 1229 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1230 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offset | 1231 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1232 | NALU 1 HDR | NALU 1 DATA | 1233 +-+-+-+-+-+-+-+-+ + 1234 : : 1235 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1236 | | NALU 2 SIZE | NALU 2 DOND | 1237 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1238 | NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA | 1239 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1240 : : 1241 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1242 | :...OPTIONAL RTP padding | 1243 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1245 Figure 12 An RTP packet including a multi-time aggregation packet 1246 of type MTAP16 containing two multi-time aggregation units 1248 Figure 13 presents an example of an RTP packet that contains a 1249 multi-time aggregation packet of type MTAP24 that contains two 1250 multi-time aggregation units, labeled as 1 and 2 in the figure. 1252 0 1 2 3 1253 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1254 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1255 | RTP Header | 1256 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1257 |MTAP24 NAL HDR | decoding order number base | NALU 1 Size | 1258 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1259 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offs | 1260 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1261 |NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA | 1262 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1263 : : 1264 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1265 | | NALU 2 SIZE | NALU 2 DOND | 1266 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1267 | NALU 2 TS offset | NALU 2 HDR | 1268 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1269 | NALU 2 DATA | 1270 : : 1271 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1272 | :...OPTIONAL RTP padding | 1273 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 Figure 13 An RTP packet including a multi-time aggregation packet 1276 of type MTAP24 containing two multi-time aggregation units 1278 5.7.3. Fragmentation Units (FUs) 1280 This payload type allows fragmenting a NAL unit into several RTP 1281 packets. Doing so on the application layer instead of relying on 1282 lower layer fragmentation (e.g., by IP) has the following 1283 advantages: 1285 o The payload format is capable of transporting NAL units bigger 1286 than 64 kbytes over an IPv4 network that may be present in pre- 1287 recorded video, particularly in High Definition formats (there 1288 is a limit of the number of slices per picture, which results in 1289 a limit of NAL units per picture, which may result in big NAL 1290 units). 1292 o The fragmentation mechanism allows fragmenting a single NAL unit 1293 and applying generic forward error correction as described in 1294 section 12.5. 1296 Fragmentation is defined only for a single NAL unit and not for any 1297 aggregation packets. A fragment of a NAL unit consists of an 1298 integer number of consecutive octets of that NAL unit. Each octet 1299 of the NAL unit MUST be part of exactly one fragment of that NAL 1300 unit. Fragments of the same NAL unit MUST be sent in consecutive 1301 order with ascending RTP sequence numbers (with no other RTP 1302 packets within the same RTP packet stream being sent between the 1303 first and last fragment). Similarly, a NAL unit MUST be 1304 reassembled in RTP sequence number order. 1306 When a NAL unit is fragmented and conveyed within fragmentation 1307 units (FUs), it is referred to as a fragmented NAL unit. STAPs and 1308 MTAPs MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU 1309 MUST NOT contain another FU. 1311 The RTP timestamp of an RTP packet carrying an FU is set to the 1312 NALU-time of the fragmented NAL unit. 1314 Figure 14 presents the RTP payload format for FU-As. An FU-A 1315 consists of a fragmentation unit indicator of one octet, a 1316 fragmentation unit header of one octet, and a fragmentation unit 1317 payload. 1319 0 1 2 3 1320 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1322 | FU indicator | FU header | | 1323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1324 | | 1325 | FU payload | 1326 | | 1327 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1328 | :...OPTIONAL RTP padding | 1329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1331 Figure 14 RTP payload format for FU-A 1333 Figure 15 presents the RTP payload format for FU-Bs. An FU-B 1334 consists of a fragmentation unit indicator of one octet, a 1335 fragmentation unit header of one octet, a decoding order number 1336 (DON) (in network byte order), and a fragmentation unit payload. 1337 In other words, the structure of FU-B is the same as the structure 1338 of FU-A, except for the additional DON field. 1340 0 1 2 3 1341 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1342 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1343 | FU indicator | FU header | DON | 1344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 1345 | | 1346 | FU payload | 1347 | | 1348 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1349 | :...OPTIONAL RTP padding | 1350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1352 Figure 15 RTP payload format for FU-B 1354 NAL unit type FU-B MUST be used in the interleaved packetization 1355 mode for the first fragmentation unit of a fragmented NAL unit. 1356 NAL unit type FU-B MUST NOT be used in any other case. In other 1357 words, in the interleaved packetization mode, each NALU that is 1358 fragmented has an FU-B as the first fragment, followed by one or 1359 more FU-A fragments. 1361 The FU indicator octet has the following format: 1363 +---------------+ 1364 |0|1|2|3|4|5|6|7| 1365 +-+-+-+-+-+-+-+-+ 1366 |F|NRI| Type | 1367 +---------------+ 1369 Values equal to 28 and 29 in the Type field of the FU indicator 1370 octet identify an FU-A and an FU-B, respectively. The use of the F 1371 bit is described in section 5.3. The value of the NRI field MUST 1372 be set according to the value of the NRI field in the fragmented 1373 NAL unit. 1375 The FU header has the following format: 1377 +---------------+ 1378 |0|1|2|3|4|5|6|7| 1379 +-+-+-+-+-+-+-+-+ 1380 |S|E|R| Type | 1381 +---------------+ 1383 S: 1 bit 1384 When set to one, the Start bit indicates the start of a 1385 fragmented NAL unit. When the following FU payload is not the 1386 start of a fragmented NAL unit payload, the Start bit is set to 1387 zero. 1389 E: 1 bit 1390 When set to one, the End bit indicates the end of a fragmented 1391 NAL unit, i.e., the last byte of the payload is also the last 1392 byte of the fragmented NAL unit. When the following FU payload 1393 is not the last fragment of a fragmented NAL unit, the End bit 1394 is set to zero. 1396 R: 1 bit 1397 The Reserved bit MUST be equal to 0 and MUST be ignored by the 1398 receiver. 1400 Type: 5 bits 1401 The NAL unit payload type as defined in Table 7-1 of [1]. 1403 The value of DON in FU-Bs is selected as described in section 5.5. 1405 Informative note: The DON field in FU-Bs allows gateways to 1406 fragment NAL units to FU-Bs without organizing the incoming NAL 1407 units to the NAL unit decoding order. 1409 A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the 1410 Start bit and End bit MUST NOT both be set to one in the same FU 1411 header. 1413 The FU payload consists of fragments of the payload of the 1414 fragmented NAL unit so that if the fragmentation unit payloads of 1415 consecutive FUs are sequentially concatenated, the payload of the 1416 fragmented NAL unit can be reconstructed. The NAL unit type octet 1417 of the fragmented NAL unit is not included as such in the 1418 fragmentation unit payload, but rather the information of the NAL 1419 unit type octet of the fragmented NAL unit is conveyed in F and NRI 1420 fields of the FU indicator octet of the fragmentation unit and in 1421 the type field of the FU header. An FU payload MAY have any number 1422 of octets and MAY be empty. 1424 Informative note: Empty FUs are allowed to reduce the latency of 1425 a certain class of senders in nearly lossless environments. 1426 These senders can be characterized in that they packetize NALU 1427 fragments before the NALU is completely generated and, hence, 1428 before the NALU size is known. If zero-length NALU fragments 1429 were not allowed, the sender would have to generate at least one 1430 bit of data of the following fragment before the current 1431 fragment could be sent. Due to the characteristics of H.264, 1432 where sometimes several macroblocks occupy zero bits, this is 1433 undesirable and can add delay. However, the (potential) use of 1434 zero-length NALU fragments should be carefully weighed against 1435 the increased risk of the loss of at least a part of the NALU 1436 because of the additional packets employed for its transmission. 1438 If a fragmentation unit is lost, the receiver SHOULD discard all 1439 following fragmentation units in transmission order corresponding 1440 to the same fragmented NAL unit. 1442 A receiver in an endpoint or in a MANE MAY aggregate the first n-1 1443 fragments of a NAL unit to an (incomplete) NAL unit, even if 1444 fragment n of that NAL unit is not received. In this case, the 1445 forbidden_zero_bit of the NAL unit MUST be set to one to indicate a 1446 syntax violation. 1448 6. Packetization Rules 1450 The packetization modes are introduced in section 5.2. The 1451 packetization rules common to more than one of the packetization 1452 modes are specified in section 6.1. The packetization rules for 1453 the single NAL unit mode, the non-interleaved mode, and the 1454 interleaved mode are specified in sections 6.2, 6.3, and 6.4, 1455 respectively. 1457 6.1. Common Packetization Rules 1459 All senders MUST enforce the following packetization rules 1460 regardless of the packetization mode in use: 1462 o Coded slice NAL units or coded slice data partition NAL units 1463 belonging to the same coded picture (and thus sharing the same 1464 RTP timestamp value) MAY be sent in any order; however, for 1465 delay-critical systems, they SHOULD be sent in their original 1466 decoding order to minimize the delay. Note that the decoding 1467 order is the order of the NAL units in the bitstream. 1469 o Parameter sets are handled in accordance with the rules and 1470 recommendations given in section 8.4. 1472 o MANEs MUST NOT duplicate any NAL unit except for sequence or 1473 picture parameter set NAL units, as neither this memo nor the 1474 H.264 specification provides means to identify duplicated NAL 1475 units. Sequence and picture parameter set NAL units MAY be 1476 duplicated to make their correct reception more probable, but 1477 any such duplication MUST NOT affect the contents of any active 1478 sequence or picture parameter set. Duplication SHOULD be 1479 performed on the application layer and not by duplicating RTP 1480 packets (with identical sequence numbers). 1482 Senders using the non-interleaved mode and the interleaved mode 1483 MUST enforce the following packetization rule: 1485 o MANEs MAY convert single NAL unit packets into one aggregation 1486 packet, convert an aggregation packet into several single NAL 1487 unit packets, or mix both concepts, in an RTP translator. The 1488 RTP translator SHOULD take into account at least the following 1489 parameters: path MTU size, unequal protection mechanisms (e.g., 1490 through packet-based FEC according to RFC 2733 [18], especially 1491 for sequence and picture parameter set NAL units and coded slice 1492 data partition A NAL units), bearable latency of the system, and 1493 buffering capabilities of the receiver. 1495 Informative note: An RTP translator is required to handle 1496 RTCP as per RFC 3550. 1498 6.2. Single NAL Unit Mode 1500 This mode is in use when the value of the OPTIONAL packetization- 1501 mode media type parameter is equal to 0 or the packetization-mode 1502 is not present. All receivers MUST support this mode. It is 1503 primarily intended for low-delay applications that are compatible 1504 with systems using ITU-T Recommendation H.241 [3] (see section 1505 12.1). Only single NAL unit packets MAY be used in this mode. 1506 STAPs, MTAPs, and FUs MUST NOT be used. The transmission order of 1507 single NAL unit packets MUST comply with the NAL unit decoding 1508 order. 1510 6.3. Non-Interleaved Mode 1512 This mode is in use when the value of the OPTIONAL packetization- 1513 mode media type parameter is equal to 1. This mode SHOULD be 1514 supported. It is primarily intended for low-delay applications. 1515 Only single NAL unit packets, STAP-As, and FU-As MAY be used in 1516 this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. The 1517 transmission order of NAL units MUST comply with the NAL unit 1518 decoding order. 1520 6.4. Interleaved Mode 1522 This mode is in use when the value of the OPTIONAL packetization- 1523 mode media type parameter is equal to 2. Some receivers MAY 1524 support this mode. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. 1525 STAP-As and single NAL unit packets MUST NOT be used. The 1526 transmission order of packets and NAL units is constrained as 1527 specified in section 5.5. 1529 7. De-Packetization Process 1531 The de-packetization process is implementation dependent. 1532 Therefore, the following description should be seen as an example 1533 of a suitable implementation. Other schemes may be used as well as 1534 long as the output for the same input is the same as the process 1535 described below. The same output means that the resulting NAL 1536 units, and their order, are identical. Optimizations relative to 1537 the described algorithms are likely possible. Section 7.1 presents 1538 the de-packetization process for the single NAL unit and non- 1539 interleaved packetization modes, whereas section 7.2 describes the 1540 process for the interleaved mode. Section 7.3 includes additional 1541 de-packetization guidelines for intelligent receivers. 1543 All normal RTP mechanisms related to buffer management apply. In 1544 particular, duplicated or outdated RTP packets (as indicated by the 1545 RTP sequence number and the RTP timestamp) are removed. To 1546 determine the exact time for decoding, factors such as a possible 1547 intentional delay to allow for proper inter-stream synchronization 1548 must be factored in. 1550 7.1. Single NAL Unit and Non-Interleaved Mode 1552 The receiver includes a receiver buffer to compensate for 1553 transmission delay jitter. The receiver stores incoming packets in 1554 reception order into the receiver buffer. Packets are de- 1555 packetized in RTP sequence number order. If a de-packetized packet 1556 is a single NAL unit packet, the NAL unit contained in the packet 1557 is passed directly to the decoder. If a de-packetized packet is an 1558 STAP-A, the NAL units contained in the packet are passed to the 1559 decoder in the order in which they are encapsulated in the packet. 1560 For all the FU-A packets containing fragments of a single NAL unit, 1561 the de-packetized fragments are concatenated in their sending order 1562 to recover the NAL unit, which is then passed to the decoder. 1564 Informative note: If the decoder supports Arbitrary Slice Order, 1565 coded slices of a picture can be passed to the decoder in any 1566 order regardless of their reception and transmission order. 1568 7.2. Interleaved Mode 1570 The general concept behind these de-packetization rules is to 1571 reorder NAL units from transmission order to the NAL unit decoding 1572 order. 1574 The receiver includes a receiver buffer, which is used to 1575 compensate for transmission delay jitter and to reorder NAL units 1576 from transmission order to the NAL unit decoding order. In this 1577 section, the receiver operation is described under the assumption 1578 that there is no transmission delay jitter. To make a difference 1579 from a practical receiver buffer that is also used for compensation 1580 of transmission delay jitter, the receiver buffer is here after 1581 called the de-interleaving buffer in this section. Receivers 1582 SHOULD also prepare for transmission delay jitter; i.e., either 1583 reserve separate buffers for transmission delay jitter buffering 1584 and de-interleaving buffering or use a receiver buffer for both 1585 transmission delay jitter and de-interleaving. Moreover, receivers 1586 SHOULD take transmission delay jitter into account in the buffering 1587 operation; e.g., by additional initial buffering before starting of 1588 decoding and playback. 1590 This section is organized as follows: subsection 7.2.1 presents how 1591 to calculate the size of the de-interleaving buffer. Subsection 1592 7.2.2 specifies the receiver process on how to organize received 1593 NAL units to the NAL unit decoding order. 1595 7.2.1. Size of the De-interleaving Buffer 1597 In either Offer/Answer or declarative SDP usage, the sprop-deint- 1598 buf-req media type parameter signals the requirement for the de- 1599 interleaving buffer size. It is therefore RECOMMENDED to set the 1600 de-interleaving buffer size, in terms of number of bytes, equal to 1601 or greater than the value of sprop-deint-buf-req media type 1602 parameter. 1604 When the SDP Offer/Answer model or any other capability exchange 1605 procedure is used in session setup, the properties of the received 1606 stream SHOULD be such that the receiver capabilities are not 1607 exceeded. In the SDP Offer/Answer model, the receiver can indicate 1608 its capabilities to allocate a de-interleaving buffer with the 1609 deint-buf-cap media type parameter. See section 8.1 for further 1610 information on deint-buf-cap and sprop-deint-buf-req media type 1611 parameters and section 8.2.2 for further information on their use 1612 in the SDP Offer/Answer model. 1614 7.2.2. De-interleaving Process 1616 There are two buffering states in the receiver: initial buffering 1617 and buffering while playing. Initial buffering occurs when the RTP 1618 session is initialized. After initial buffering, decoding and 1619 playback are started, and the buffering-while-playing mode is used. 1621 Regardless of the buffering state, the receiver stores incoming NAL 1622 units, in reception order, in the de-interleaving buffer as follows. 1623 NAL units of aggregation packets are stored in the de-interleaving 1624 buffer individually. The value of DON is calculated and stored for 1625 each NAL unit. 1627 The receiver operation is described below with the help of the 1628 following functions and constants: 1630 o Function AbsDON is specified in section 8.1. 1632 o Function don_diff is specified in section 5.5. 1634 o Constant N is the value of the OPTIONAL sprop-interleaving-depth 1635 media type parameter (see section 8.1) incremented by 1. 1637 Initial buffering lasts until one of the following conditions is 1638 fulfilled: 1640 o There are N or more VCL NAL units in the de-interleaving buffer. 1642 o If sprop-max-don-diff is present, don_diff(m,n) is greater than 1643 the value of sprop-max-don-diff, in which n corresponds to the 1644 NAL unit having the greatest value of AbsDON among the received 1645 NAL units and m corresponds to the NAL unit having the smallest 1646 value of AbsDON among the received NAL units. 1648 o Initial buffering has lasted for the duration equal to or 1649 greater than the value of the OPTIONAL sprop-init-buf-time media 1650 type parameter. 1652 The NAL units to be removed from the de-interleaving buffer are 1653 determined as follows: 1655 o If the de-interleaving buffer contains at least N VCL NAL units, 1656 NAL units are removed from the de-interleaving buffer and passed 1657 to the decoder in the order specified below until the buffer 1658 contains N-1 VCL NAL units. 1660 o If sprop-max-don-diff is present, all NAL units m for which 1661 don_diff(m,n) is greater than sprop-max-don-diff are removed 1662 from the de-interleaving buffer and passed to the decoder in the 1663 order specified below. Herein, n corresponds to the NAL unit 1664 having the greatest value of AbsDON among the NAL units in the 1665 de-interleaving buffer. 1667 The order in which NAL units are passed to the decoder is specified 1668 as follows: 1670 o Let PDON be a variable that is initialized to 0 at the beginning 1671 of the RTP session. 1673 o For each NAL unit associated with a value of DON, a DON distance 1674 is calculated as follows. If the value of DON of the NAL unit 1675 is larger than the value of PDON, the DON distance is equal to 1676 DON - PDON. Otherwise, the DON distance is equal to 65535 - 1677 PDON + DON + 1. 1679 o NAL units are delivered to the decoder in ascending order of DON 1680 distance. If several NAL units share the same value of DON 1681 distance, they can be passed to the decoder in any order. 1683 o When a desired number of NAL units have been passed to the 1684 decoder, the value of PDON is set to the value of DON for the 1685 last NAL unit passed to the decoder. 1687 7.3. Additional De-Packetization Guidelines 1689 The following additional de-packetization rules may be used to 1690 implement an operational H.264 de-packetizer: 1692 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1693 coded slice data partitions A (DPAs). If a lost DPA is detected, 1694 after taking into account possible retransmission and FEC, a 1695 gateway may decide not to send the corresponding coded slice 1696 data partitions B and C, as their information is meaningless for 1697 H.264 decoders. In this way a MANE can reduce network load by 1698 discarding useless packets without parsing a complex bitstream. 1700 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1701 FUs. If a lost FU is found, a gateway may decide not to send 1702 the following FUs of the same fragmented NAL unit, as their 1703 information is meaningless for H.264 decoders. In this way a 1704 MANE can reduce network load by discarding useless packets 1705 without parsing a complex bitstream. 1707 o Intelligent receivers having to discard packets or NALUs should 1708 first discard all packets/NALUs in which the value of the NRI 1709 field of the NAL unit type octet is equal to 0. This will 1710 minimize the impact on user experience and keep the reference 1711 pictures intact. If more packets have to be discarded, then 1712 packets with a numerically lower NRI value should be discarded 1713 before packets with a numerically higher NRI value. However, 1714 discarding any packets with an NRI bigger than 0 very likely 1715 leads to decoder drift and SHOULD be avoided. 1717 8. Payload Format Parameters 1719 This section specifies the parameters that MAY be used to select 1720 optional features of the payload format and certain features of the 1721 bitstream. The parameters are specified here as part of the media 1722 subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. 1723 A mapping of the parameters into the Session Description Protocol 1724 (SDP) [6] is also provided for applications that use SDP. 1725 Equivalent parameters could be defined elsewhere for use with 1726 control protocols that do not use SDP. 1728 Some parameters provide a receiver with the properties of the 1729 stream that will be sent. The names of all these parameters start 1730 with "sprop" for stream properties. Some of these "sprop" 1731 parameters are limited by other payload or codec configuration 1732 parameters. For example, the sprop-parameter-sets parameter is 1733 constrained by the profile-level-id parameter. 1735 8.1. Media Type Registration 1737 The media subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is 1738 allocated from the IETF tree. 1740 Media Type name: video 1742 Media subtype name: H264 1744 Required parameters: none 1745 OPTIONAL parameters: 1747 profile-level-id: 1748 A base16 [7] (hexadecimal) representation of the following 1749 three bytes in the sequence parameter set NAL unit specified 1750 in [1]: 1) profile_idc, 2) a byte herein referred to as 1751 profile-iop, composed of the values of constraint_set0_flag, 1752 constraint_set1_flag,constraint_set2_flag, 1753 constraint_set3_flag, and reserved_zero_4bits in bit- 1754 significance order, starting from the most significant bit, 1755 and 3) level_idc. Note that reserved_zero_4bits is required 1756 to be equal to 0 in [1], but other values for it may be 1757 specified in the future by ITU-T or ISO/IEC. 1759 The profile-level-id parameter indicates the default sub- 1760 profile, i.e. the subset of coding tools that may have been 1761 used to generate the stream or that the receiver supports, 1762 and the default level of the stream or the receiver supports. 1764 The default sub-profile is indicated collectively by the 1765 profile_idc byte and some fields in the profile-iop byte. 1766 Depending on the values of the fields in the profile-iop byte, 1767 the default sub-profile may be the set of coding tools 1768 supported by one profile, or a common subset of coding tools 1769 of multiple profiles, as specified in subsection 7.4.2.1.1 of 1770 [1]. The default level is indicated by the level_idc byte, 1771 and, when profile_idc is equal to 66, 77 or 88 (the Baseline, 1772 Main, or Extended profile) and level_idc is equal to 11, 1773 additionally by bit 4 (constraint_set3_flag) of the profile- 1774 iop byte. When profile_idc is equal to 66, 77 or 88 (the 1775 Baseline, Main, or Extended profile) and level_idc is equal 1776 to 11, and bit 4 (constraint_set3_flag) of the profile-iop 1777 byte is equal to 1, the default level is level 1b. 1779 Table 5 lists all profiles defined in Annex A of [1] and, for 1780 each of the profiles, the possible combinations of 1781 profile_idc and profile-iop that represent the same sub- 1782 profile. 1784 Table 5. Combinations of profile_idc and profile-iop 1785 representing the same sub-profile corresponding to the 1786 full set of coding tools supported by one profile. In 1787 the following, x may be either 0 or 1, while the profile 1788 names are indicated as follows. CB: Constrained Baseline 1789 profile, B: Baseline profile, M: Main profile, E: 1790 Extended profile, H: High profile, H10: High 10 profile, 1791 H42: High 4:2:2 profile, H44: High 4:4:4 Predictive 1792 profile, H10I: High 10 Intra profile, H42I: High 4:2:2 1793 Intra profile, H44I: High 4:4:4 Intra profile, and C44I: 1794 CAVLC 4:4:4 Intra profile. 1796 Profile profile_idc profile-iop 1797 (hexadecimal) (binary) 1799 CB 42 (B) x1xx0000 1800 same as: 4D (M) 1xxx0000 1801 same as: 58 (E) 11xx0000 1802 same as: 64 (H), 6E (H10), 1xx00000 1803 7A (H42), or F4 (H44) 1804 B 42 (B) x0xx0000 1805 same as: 58 (E) 10xx0000 1806 M 4D (M) 0x0x0000 1807 same as: 64 (H), 6E (H10), 01000000 1808 7A (H42), or F4 (H44) 1809 E 58 00xx0000 1810 H 64 00000000 1811 H10 6E 00000000 1812 H42 7A 00000000 1813 H44 F4 00000000 1814 H10I 64 00010000 1815 H42I 7A 00010000 1816 H44I F4 00010000 1817 C44I 2C 00010000 1819 For example, in the table above, profile_idc equal to 58 1820 (Extended) with profile-iop equal to 11xx0000 indicates the 1821 same sub-profile corresponding to profile_idc equal to 42 1822 (Baseline) with profile-iop equal to x1xx0000. Note that 1823 other combinations of profile_idc and profile-iop (not listed 1824 in Table 5) may represent a sub-profile equivalent to the 1825 common subset of coding tools for more than one profile. 1826 Note also that a decoder conforming to a certain profile may 1827 be able to decode bitstreams conforming to other profiles. 1828 For example, a decoder conforming to the High 4:4:4 profile 1829 at certain level must be able to decode bitstreams conforming 1830 to the Constrained Baseline, Main, High, High 10 or High 1831 4:2:2 profile at the same or a lower level. 1833 If the profile-level-id parameter is used to indicate 1834 properties of a NAL unit stream, it indicates that, to decode 1835 the stream, the minimum subset of coding tools a decoder has 1836 to support is the default sub-profile, and the lowest level 1837 the decoder has to support is the default level. 1839 If the profile-level-id parameter is used for capability 1840 exchange or session setup procedure, it indicates the subset 1841 of coding tools, which is equal to the default sub-profile, 1842 that the codec supports for both receiving and sending. If 1843 max-recv-level is not present, the default level from 1844 profile-level-id indicates the highest level the codec wishes 1845 to support. If max-recv-level is present it indicates the 1846 highest level the codec supports for receiving. For either 1847 receiving or sending, all levels that are lower than the 1848 highest level supported MUST also be supported. 1850 Informative note: Capability exchange and session setup 1851 procedures should provide means to list the capabilities 1852 for each supported sub-profile separately. For example, 1853 the one-of-N codec selection procedure of the SDP 1854 Offer/Answer model can be used (section 10.2 of [8]). 1855 The one-of-N codec selection procedure may also be used 1856 to provide different combinations of profile_idc and 1857 profile-iop that represent the same sub-profile. When 1858 there are many different combinations of profile_idc and 1859 profile-iop that represent the same sub-profile, using 1860 the one-of-N codec selection procedure may result into a 1861 fairly large SDP message. Therefore, a receiver should 1862 understand the different equivalent combinations of 1863 profile_idc and profile-iop that represent the same sub- 1864 profile, and be ready to accept an offer using any of the 1865 equivalent combinations. 1867 If no profile-level-id is present, the Baseline Profile 1868 without additional constraints at Level 1 MUST be inferred. 1870 max-recv-level: 1871 This parameter MAY be used to indicate the highest level a 1872 receiver supports when the highest level is higher than the 1873 default level (the level indicated by profile-level-id). The 1874 value of max-recv-level is a base16 (hexadecimal) 1875 representation of the two bytes after the syntax element 1876 profile_idc in the sequence parameter set NAL unit specified 1877 in [1]: profile-iop (as defined above) and level_idc. If 1878 (the level_idc byte of max-recv-level is equal to 11 and bit 1879 4 of the profile-iop byte of max-recv-level is equal to 1) or 1880 (the level_idc byte of max-recv-level is equal to 9 and bit 4 1881 of the profile-iop byte of max-recv-level is equal to 0), the 1882 highest level the receiver supports is level 1b. Otherwise, 1883 the highest level the receiver supports is equal to the 1884 level_idc byte of max-recv-level divided by 10. 1886 max-recv-level MUST NOT be present if the highest level the 1887 receiver supports is not higher than the default level. 1889 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: 1890 These parameters MAY be used to signal the capabilities of a 1891 receiver implementation. These parameters MUST NOT be used 1892 for any other purpose. The highest level conveyed in the 1893 value of the profile-level-id parameter or the max-recv-level 1894 parameter MUST be such that the receiver is fully capable of 1895 supporting. max-mbps, max-smbps, max-fs, max-cpb, max-dpb, 1896 and max-br MAY be used to indicate capabilities of the 1897 receiver that extend the required capabilities of the 1898 signaled highest level, as specified below. 1900 When more than one parameter from the set (max-mbps, max- 1901 smbps , max-fs, max-cpb, max-dpb, max-br) is present, the 1902 receiver MUST support all signaled capabilities 1903 simultaneously. For example, if both max-mbps and max-br are 1904 present, the signaled highest level with the extension of 1905 both the frame rate and bit rate is supported. That is, the 1906 receiver is able to decode NAL unit streams in which the 1907 macroblock processing rate is up to max-mbps (inclusive), the 1908 bit rate is up to max-br (inclusive), the coded picture 1909 buffer size is derived as specified in the semantics of the 1910 max-br parameter below, and other properties comply with the 1911 highest level specified in the value of the profile-level-id 1912 parameter or the max-recv-level parameter. 1914 If a receiver can support all the properties of level A, the 1915 highest level specified in the value of the profile-level-id 1916 parameter or the max-recv-level parameter MUST be level A 1917 (i.e. MUST NOT be lower than level A). In other words, a 1918 receiver MUST NOT signal values of max-mbps, max-fs, max-cpb, 1919 max-dpb, and max-br that taken together meet the requirements 1920 of a higher level compared to the highest level specified in 1921 the value of the profile-level-id parameter or the max-recv- 1922 level parameter. 1924 Informative note: When the OPTIONAL media type parameters 1925 are used to signal the properties of a NAL unit stream, 1926 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br 1927 are not present, and the value of profile-level-id must 1928 always be such that the NAL unit stream complies fully 1929 with the specified profile and level. 1931 max-mbps: The value of max-mbps is an integer indicating the 1932 maximum macroblock processing rate in units of macroblocks 1933 per second. The max-mbps parameter signals that the receiver 1934 is capable of decoding video at a higher rate than is 1935 required by the signaled highest level conveyed in the value 1936 of the profile-level-id parameter or the max-recv-level 1937 parameter. When max-mbps is signaled, the receiver MUST be 1938 able to decode NAL unit streams that conform to the signaled 1939 highestlevel, with the exception that the MaxMBPS value in 1940 Table A-1 of [1] for the signaled highest level is replaced 1941 with the value of max-mbps. The value of max-mbps MUST be 1942 greater than or equal to the value of MaxMBPS given in Table 1943 A-1 of [1] for the highest level. Senders MAY use this 1944 knowledge to send pictures of a given size at a higher 1945 picture rate than is indicated in the signaled highest level. 1947 max-smbps: The value of max-smbps is an integer indicating the 1948 maximum static macroblock processing rate in units of static 1949 macroblocks per second, under the hypothetical assumption 1950 that all macroblocks are static macroblocks. When max-smbps 1951 is signalled the MaxMBPS value in Table A-1 of [1] should be 1952 replaced with the result of the following computation: 1954 o If the parameter max-mbps is signalled, set a variable 1955 MaxMacroblocksPerSecond to the value of max-mbps. 1956 Otherwise, set MaxMacroblocksPerSecond equal to the value 1957 of MaxMBPS in Table A-1 [1] for the highest level. 1959 o Set a variable P_non-static to the proportion of non- 1960 static macroblocks in picture n. 1962 o Set a variable P_static to the proportion of static 1963 macroblocks in picture n. 1965 o The value of MaxMBPS in Table A-1 of [1] should be 1966 considered by the encoder to be equal to: 1968 MaxMacroblocksPerSecond * max-smbps / (P_non-static * 1969 max-smbps + P_static * MaxMacroblocksPerSecond) 1971 The encoder should recompute this value for each picture. The 1972 value of max-smbps MUST be greater than the value of MaxMBPS 1973 given in Table A-1 of [1] for the highest level. Senders MAY 1974 use this knowledge to send pictures of a given size at a 1975 higher picture rate than is indicated in the signaled highest 1976 level. 1978 max-fs: The value of max-fs is an integer indicating the maximum 1979 frame size in units of macroblocks. The max-fs parameter 1980 signals that the receiver is capable of decoding larger 1981 picture sizes than are required by the signaled highest level 1982 conveyed in the value of the profile-level-id parameter or 1983 the max-recv-level parameter. When max-fs is signaled, the 1984 receiver MUST be able to decode NAL unit streams that conform 1985 to the signaled highest level, with the exception that the 1986 MaxFS value in Table A-1 of [1] for the signaled highest 1987 level is replaced with the value of max-fs. The value of 1988 max-fs MUST be greater than or equal to the value of MaxFS 1989 given in Table A-1 of [1] for the highest level. Senders MAY 1990 use this knowledge to send larger pictures at a 1991 proportionally lower frame rate than is indicated in the 1992 signaled highest level. 1994 max-cpb: The value of max-cpb is an integer indicating the 1995 maximum coded picture buffer size in units of 1000 bits for 1996 the VCL HRD parameters (see A.3.1 item i of [1]) and in units 1997 of 1200 bits for the NAL HRD parameters (see A.3.1 item j of 1998 [1]). The max-cpb parameter signals that the receiver has 1999 more memory than the minimum amount of coded picture buffer 2000 memory required by the signaled highest level conveyed in the 2001 value of the profile-level-id parameter or the max-recv-level 2002 parameter. When max-cpb is signaled, the receiver MUST be 2003 able to decode NAL unit streams that conform to the signaled 2004 highest level, with the exception that the MaxCPB value in 2005 Table A-1 of [1] for the signaled highest level is replaced 2006 with the value of max-cpb. The value of max-cpb MUST be 2007 greater than or equal to the value of MaxCPB given in Table 2008 A-1 of [1] for the highest level. Senders MAY use this 2009 knowledge to construct coded video streams with greater 2010 variation of bit rate than can be achieved with the MaxCPB 2011 value in Table A-1 of [1]. 2013 Informative note: The coded picture buffer is used in the 2014 hypothetical reference decoder (Annex C) of H.264. The 2015 use of the hypothetical reference decoder is recommended 2016 in H.264 encoders to verify that the produced bitstream 2017 conforms to the standard and to control the output 2018 bitrate. Thus, the coded picture buffer is conceptually 2019 independent of any other potential buffers in the 2020 receiver, including de-interleaving and de-jitter buffers. 2021 The coded picture buffer need not be implemented in 2022 decoders as specified in Annex C of H.264, but rather 2023 standard-compliant decoders can have any buffering 2024 arrangements provided that they can decode standard- 2025 compliant bitstreams. Thus, in practice, the input 2026 buffer for video decoder can be integrated with de- 2027 interleaving and de-jitter buffers of the receiver. 2029 max-dpb: The value of max-dpb is an integer indicating the 2030 maximum decoded picture buffer size in units of 1024 bytes. 2031 The max-dpb parameter signals that the receiver has more 2032 memory than the minimum amount of decoded picture buffer 2033 memory required by the signaled highest level conveyed in the 2034 value of the profile-level-id parameter or the max-recv-level 2035 parameter. When max-dpb is signaled, the receiver MUST be 2036 able to decode NAL unit streams that conform to the signaled 2037 highest level, with the exception that the MaxDPB value in 2038 Table A-1 of [1] for the signaled highest level is replaced 2039 with the value of max-dpb. Consequently, a receiver that 2040 signals max-dpb MUST be capable of storing the following 2041 number of decoded frames, complementary field pairs, and non- 2042 paired fields in its decoded picture buffer: 2044 Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs * 2045 256 * ChromaFormatFactor ), 16) 2047 PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are 2048 defined in [1]. 2050 The value of max-dpb MUST be greater than or equal to the 2051 value of MaxDPB given in Table A-1 of [1] for the highest 2052 level. Senders MAY use this knowledge to construct coded 2053 video streams with improved compression. 2055 Informative note: This parameter was added primarily to 2056 complement a similar codepoint in the ITU-T 2057 Recommendation H.245, so as to facilitate signaling 2058 gateway designs. The decoded picture buffer stores 2059 reconstructed samples. There is no relationship between 2060 the size of the decoded picture buffer and the buffers 2061 used in RTP, especially de-interleaving and de-jitter 2062 buffers. 2064 max-br: The value of max-br is an integer indicating the maximum 2065 video bit rate in units of 1000 bits per second for the VCL 2066 HRD parameters (see A.3.1 item i of [1]) and in units of 1200 2067 bits per second for the NAL HRD parameters (see A.3.1 item j 2068 of [1]). 2070 The max-br parameter signals that the video decoder of the 2071 receiver is capable of decoding video at a higher bit rate 2072 than is required by the signaled highest level conveyed in 2073 the value of the profile-level-id parameter or the max-recv- 2074 level parameter. 2076 When max-br is signaled, the video codec of the receiver MUST 2077 be able to decode NAL unit streams that conform to the 2078 signaled highest level, with the following exceptions in the 2079 limits specified by the highest level: 2081 o The value of max-br replaces the MaxBR value in Table A-1 2082 of [1] for the highest level. 2084 o When the max-cpb parameter is not present, the result of 2085 the following formula replaces the value of MaxCPB in 2086 Table A-1 of [1]: (MaxCPB of the signaled level) * max-br 2087 / (MaxBR of the signaled highest level). 2089 For example, if a receiver signals capability for Level 1.2 2090 with max-br equal to 1550, this indicates a maximum video 2091 bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum 2092 video bitrate of 1860 kbits/sec for NAL HRD parameters, and a 2093 CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000). 2095 The value of max-br MUST be greater than or equal to the 2096 value MaxBR given in Table A-1 of [1] for the signaled 2097 highest level. 2099 Senders MAY use this knowledge to send higher bitrate video 2100 as allowed in the level definition of Annex A of H.264, to 2101 achieve improved video quality. 2103 Informative note: This parameter was added primarily to 2104 complement a similar codepoint in the ITU-T 2105 Recommendation H.245, so as to facilitate signaling 2106 gateway designs. No assumption can be made from the 2107 value of this parameter that the network is capable of 2108 handling such bit rates at any given time. In particular, 2109 no conclusion can be drawn that the signaled bit rate is 2110 possible under congestion control constraints. 2112 redundant-pic-cap: 2113 This parameter signals the capabilities of a receiver 2114 implementation. When equal to 0, the parameter indicates 2115 that the receiver makes no attempt to use redundant coded 2116 pictures to correct incorrectly decoded primary coded 2117 pictures. When equal to 0, the receiver is not capable of 2118 using redundant slices; therefore, a sender SHOULD avoid 2119 sending redundant slices to save bandwidth. When equal to 1, 2120 the receiver is capable of decoding any such redundant slice 2121 that covers a corrupted area in a primary decoded picture (at 2122 least partly), and therefore a sender MAY send redundant 2123 slices. When the parameter is not present, then a value of 0 2124 MUST be used for redundant-pic-cap. When present, the value 2125 of redundant-pic-cap MUST be either 0 or 1. 2127 When the profile-level-id parameter is present in the same 2128 signaling as the redundant-pic-cap parameter, and the profile 2129 indicated in profile-level-id is such that it disallows the 2130 use of redundant coded pictures (e.g., Main Profile), the 2131 value of redundant-pic-cap MUST be equal to 0. When a 2132 receiver indicates redundant-pic-cap equal to 0, the received 2133 stream SHOULD NOT contain redundant coded pictures. 2135 Informative note: Even if redundant-pic-cap is equal to 0, 2136 the decoder is able to ignore redundant codec pictures 2137 provided that the decoder supports such a profile 2138 (Baseline, Extended) in which redundant coded pictures 2139 are allowed. 2141 Informative note: Even if redundant-pic-cap is equal to 1, 2142 the receiver may also choose other error concealment 2143 strategies to replace or complement decoding of redundant 2144 slices. 2146 sprop-parameter-sets: 2147 This parameter MAY be used to convey any sequence and picture 2148 parameter set NAL units (herein referred to as the initial 2149 parameter set NAL units) that can be placed in the NAL unit 2150 stream to precede any other NAL units in decoding order. The 2151 parameter MUST NOT be used to indicate codec capability in 2152 any capability exchange procedure. The value of the 2153 parameter is a comma (',') separated list of base64 [7] 2154 representations of parameter set NAL units as specified in 2155 sections 7.3.2.1 and 7.3.2.2 of [1]. Note that the number of 2156 bytes in a parameter set NAL unit is typically less than 10, 2157 but a picture parameter set NAL unit can contain several 2158 hundreds of bytes. 2160 Informative note: When several payload types are offered 2161 in the SDP Offer/Answer model, each with its own sprop- 2162 parameter-sets parameter, then the receiver cannot assume 2163 that those parameter sets do not use conflicting storage 2164 locations (i.e., identical values of parameter set 2165 identifiers). Therefore, a receiver should buffer all 2166 sprop-parameter-sets and make them available to the 2167 decoder instance that decodes a certain payload type. 2169 The "sprop-parameter-sets" parameter MUST only contain 2170 parameter sets that are conforming to the profile-level-id, 2171 i.e., the subset of coding tools indicated by any of the 2172 parameter sets MUST be equal to the default sub-profile, and 2173 the level indicated by any of the parameter sets MUST be 2174 equal to the default level. 2176 sprop-level-parameter-sets: 2177 This parameter MAY be used to convey any sequence and picture 2178 parameter set NAL units (herein referred to as the initial 2179 parameter set NAL units) that can be placed in the NAL unit 2180 stream to precede any other NAL units in decoding order and 2181 that are associated with one or more levels different than 2182 the default level. The parameter MUST NOT be used to 2183 indicate codec capability in any capability exchange 2184 procedure. 2186 The sprop-level-parameter-sets parameter contains parameter 2187 sets for one or more levels which are different than the 2188 default level. All parameter sets associated with one level 2189 are clustered and prefixed with a three-byte field which has 2190 the same syntax as profile-level-id. This enables the 2191 receiver to install the parameter sets for one level and 2192 discard the rest. The three-byte field is named PLId, and 2193 all parameter sets associated with one level are named PSL, 2194 which has the same syntax as sprop-parameter-sets. Parameter 2195 sets for each level are represented in the form of PLId:PSL, 2196 i.e., PLId followed by a colon (':') and the base64 [7] 2197 representation of the initial parameter set NAL units for the 2198 level. Each pair of PLId:PSL is also separated by a colon. 2199 Note that a PSL can contain multiple parameter sets for that 2200 level, separated with commas (','). 2202 The subset of coding tools indicated by each PLId field MUST 2203 be equal to the default sub-profile, and the level indicated 2204 by each PLId field MUST be different than the default level. 2205 All sequence parameter sets contained in each PSL MUST have 2206 the three bytes from profile_idc to level_idc, inclusive, 2207 equal to the preceding PLId. 2209 Informative note: This parameter allows for efficient 2210 level downgrade or upgrade in SDP Offer/Answer and out- 2211 of-band transport of parameter sets, simultaneously. 2213 use-level-src-parameter-sets: 2214 This parameter MAY be used to indicate a receiver capability. 2215 The value MAY be equal to either 0 or 1. When the parameter 2216 is not present, the value MUST be inferred to be equal to 0. 2217 The value 0 indicates that the receiver does not understand 2218 the sprop-level-parameter-sets parameter, and does not 2219 understand the "fmtp" source attribute as specified in 2220 section 6.3 of [9], and will ignore sprop-level-parameter- 2221 sets when present, and will ignore sprop-parameter-sets when 2222 conveyed using the "fmtp" source attribute. The value 1 2223 indicates that the receiver understands the sprop-level- 2224 parameter-sets parameter, and understands the "fmtp" source 2225 attribute as specified in section 6.3 of [9], and is capable 2226 of using parameter sets contained in the sprop-level- 2227 parameter-sets or contained in the sprop-parameter-sets that 2228 is conveyed using the "fmtp" source attribute. 2230 Informative note: An RFC 3984 receiver does not 2231 understand sprop-level-parameter-sets, use-level-src- 2232 parameter-sets, or the "fmtp" source attribute as 2233 specified in section 6.3 of [9]. Therefore, during SDP 2234 Offer/Answer, an RFC 3984 receiver as the answerer will 2235 simply ignore sprop-level-parameter-sets, when present in 2236 an offer, and sprop-parameter-sets conveyed using the 2237 "fmtp" source attribute as specified in section 6.3 of 2238 [9]. Assume that the offered payload type was accepted 2239 at a level lower than the default level. If the offered 2240 payload type included sprop-level-parameter-sets or 2241 included sprop-parameter-sets conveyed using the "fmtp" 2242 source attribute, and the offerer sees that the answerer 2243 has not included use-level-src-parameter-sets equal to 1 2244 in the answer, the offerer knows that in-band transport 2245 of parameter sets is needed. 2247 in-band-parameter-sets: 2248 This parameter MAY be used to indicate a receiver capability. 2249 The value MAY be equal to either 0 or 1. The value 1 2250 indicates that the receiver discards out-of-band parameter 2251 sets in sprop-parameter-sets and sprop-level-parameter-sets, 2252 therefore the sender MUST transmit all parameter sets in-band. 2253 The value 0 indicates that the receiver utilizes out-of-band 2254 parameter sets included in sprop-parameter-sets and/or sprop- 2255 level-parameter-sets. However, in this case, the sender MAY 2256 still choose to send parameter sets in-band. When in-band- 2257 parameter-sets is equal to 1, use-level-src-parameter-sets 2258 MUST NOT be present or MUST be equal to 0. When the 2259 parameter is not present, this receiver capability is not 2260 specified, and therefore the sender MAY send out-of-band 2261 parameter sets only, or it MAY send in-band-parameter-sets 2262 only, or it MAY send both. 2264 level-asymmetry-allowed: 2265 This parameter MAY be used in SDP Offer/Answer to indicate 2266 whether level asymmetry, i.e., sending media encoded at a 2267 different level in the offerer-to-answerer direction than the 2268 level in the answerer-to-offerer direction, is allowed. The 2269 value MAY be equal to either 0 or 1. When the parameter is 2270 not present, the value MUST be inferred to be equal to 0. 2271 The value 1 in both the offer and the answer indicates that 2272 level asymmetry is allowed. The value of 0 in either the 2273 offer or the answer indicates the level asymmetry is not 2274 allowed. 2276 If "level-asymmetry-allowed" is equal to 0 (or not present) 2277 in either the offer or the answer, level asymmetry is not 2278 allowed. In this case, the level to use in the direction 2279 from the offerer to the answerer MUST be the same as the 2280 level to use in the opposite direction. 2282 packetization-mode: 2283 This parameter signals the properties of an RTP payload type 2284 or the capabilities of a receiver implementation. Only a 2285 single configuration point can be indicated; thus, when 2286 capabilities to support more than one packetization-mode are 2287 declared, multiple configuration points (RTP payload types) 2288 must be used. 2290 When the value of packetization-mode is equal to 0 or 2291 packetization-mode is not present, the single NAL mode MUST 2292 be used. This mode is in use in standards using ITU-T 2293 Recommendation H.241 [3] (see section 12.1). When the value 2294 of packetization-mode is equal to 1, the non-interleaved mode 2295 MUST be used. When the value of packetization-mode is equal 2296 to 2, the interleaved mode MUST be used. The value of 2297 packetization-mode MUST be an integer in the range of 0 to 2, 2298 inclusive. 2300 sprop-interleaving-depth: 2301 This parameter MUST NOT be present when packetization-mode is 2302 not present or the value of packetization-mode is equal to 0 2303 or 1. This parameter MUST be present when the value of 2304 packetization-mode is equal to 2. 2306 This parameter signals the properties of an RTP packet stream. 2307 It specifies the maximum number of VCL NAL units that precede 2308 any VCL NAL unit in the RTP packet stream in transmission 2309 order and follow the VCL NAL unit in decoding order. 2310 Consequently, it is guaranteed that receivers can reconstruct 2311 NAL unit decoding order when the buffer size for NAL unit 2312 decoding order recovery is at least the value of sprop- 2313 interleaving-depth + 1 in terms of VCL NAL units. 2315 The value of sprop-interleaving-depth MUST be an integer in 2316 the range of 0 to 32767, inclusive. 2318 sprop-deint-buf-req: 2319 This parameter MUST NOT be present when packetization-mode is 2320 not present or the value of packetization-mode is equal to 0 2321 or 1. It MUST be present when the value of packetization- 2322 mode is equal to 2. 2324 sprop-deint-buf-req signals the required size of the de- 2325 interleaving buffer for the RTP packet stream. The value of 2326 the parameter MUST be greater than or equal to the maximum 2327 buffer occupancy (in units of bytes) required in such a de- 2328 interleaving buffer that is specified in section 7.2. It is 2329 guaranteed that receivers can perform the de-interleaving of 2330 interleaved NAL units into NAL unit decoding order, when the 2331 de-interleaving buffer size is at least the value of sprop- 2332 deint-buf-req in terms of bytes. 2334 The value of sprop-deint-buf-req MUST be an integer in the 2335 range of 0 to 4294967295, inclusive. 2337 Informative note: sprop-deint-buf-req indicates the 2338 required size of the de-interleaving buffer only. When 2339 network jitter can occur, an appropriately sized jitter 2340 buffer has to be provisioned for as well. 2342 deint-buf-cap: 2343 This parameter signals the capabilities of a receiver 2344 implementation and indicates the amount of de-interleaving 2345 buffer space in units of bytes that the receiver has 2346 available for reconstructing the NAL unit decoding order. A 2347 receiver is able to handle any stream for which the value of 2348 the sprop-deint-buf-req parameter is smaller than or equal to 2349 this parameter. 2351 If the parameter is not present, then a value of 0 MUST be 2352 used for deint-buf-cap. The value of deint-buf-cap MUST be 2353 an integer in the range of 0 to 4294967295, inclusive. 2355 Informative note: deint-buf-cap indicates the maximum 2356 possible size of the de-interleaving buffer of the 2357 receiver only. When network jitter can occur, an 2358 appropriately sized jitter buffer has to be provisioned 2359 for as well. 2361 sprop-init-buf-time: 2362 This parameter MAY be used to signal the properties of an RTP 2363 packet stream. The parameter MUST NOT be present, if the 2364 value of packetization-mode is equal to 0 or 1. 2366 The parameter signals the initial buffering time that a 2367 receiver MUST wait before starting decoding to recover the 2368 NAL unit decoding order from the transmission order. The 2369 parameter is the maximum value of (decoding time of the NAL 2370 unit - transmission time of a NAL unit), assuming reliable 2371 and instantaneous transmission, the same timeline for 2372 transmission and decoding, and that decoding starts when the 2373 first packet arrives. 2375 An example of specifying the value of sprop-init-buf-time 2376 follows. A NAL unit stream is sent in the following 2377 interleaved order, in which the value corresponds to the 2378 decoding time and the transmission order is from left to 2379 right: 2381 0 2 1 3 5 4 6 8 7 ... 2383 Assuming a steady transmission rate of NAL units, the 2384 transmission times are: 2386 0 1 2 3 4 5 6 7 8 ... 2388 Subtracting the decoding time from the transmission time 2389 column-wise results in the following series: 2391 0 -1 1 0 -1 1 0 -1 1 ... 2393 Thus, in terms of intervals of NAL unit transmission times, 2394 the value of sprop-init-buf-time in this example is 1. The 2395 parameter is coded as a non-negative base10 integer 2396 representation in clock ticks of a 90-kHz clock. If the 2397 parameter is not present, then no initial buffering time 2398 value is defined. Otherwise the value of sprop-init-buf-time 2399 MUST be an integer in the range of 0 to 4294967295, inclusive. 2401 In addition to the signaled sprop-init-buf-time, receivers 2402 SHOULD take into account the transmission delay jitter 2403 buffering, including buffering for the delay jitter caused by 2404 mixers, translators, gateways, proxies, traffic-shapers, and 2405 other network elements. 2407 sprop-max-don-diff: 2408 This parameter MAY be used to signal the properties of an RTP 2409 packet stream. It MUST NOT be used to signal transmitter or 2410 receiver or codec capabilities. The parameter MUST NOT be 2411 present if the value of packetization-mode is equal to 0 or 1. 2412 sprop-max-don-diff is an integer in the range of 0 to 32767, 2413 inclusive. If sprop-max-don-diff is not present, the value 2414 of the parameter is unspecified. sprop-max-don-diff is 2415 calculated as follows: 2417 sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)}, 2418 for any i and any j>i, 2420 where i and j indicate the index of the NAL unit in the 2421 transmission order and AbsDON denotes a decoding order number 2422 of the NAL unit that does not wrap around to 0 after 65535. 2423 In other words, AbsDON is calculated as follows: Let m and n 2424 be consecutive NAL units in transmission order. For the very 2425 first NAL unit in transmission order (whose index is 0), 2426 AbsDON(0) = DON(0). For other NAL units, AbsDON is 2427 calculated as follows: 2429 If DON(m) == DON(n), AbsDON(n) = AbsDON(m) 2431 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 2432 AbsDON(n) = AbsDON(m) + DON(n) - DON(m) 2434 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 2435 AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n) 2437 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 2438 AbsDON(n) = AbsDON(m) - (DON(m) + 65536 - DON(n)) 2440 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 2441 AbsDON(n) = AbsDON(m) - (DON(m) - DON(n)) 2443 where DON(i) is the decoding order number of the NAL unit 2444 having index i in the transmission order. The decoding order 2445 number is specified in section 5.5. 2447 Informative note: Receivers may use sprop-max-don-diff to 2448 trigger which NAL units in the receiver buffer can be 2449 passed to the decoder. 2451 max-rcmd-nalu-size: 2452 This parameter MAY be used to signal the capabilities of a 2453 receiver. The parameter MUST NOT be used for any other 2454 purposes. The value of the parameter indicates the largest 2455 NALU size in bytes that the receiver can handle efficiently. 2456 The parameter value is a recommendation, not a strict upper 2457 boundary. The sender MAY create larger NALUs but must be 2458 aware that the handling of these may come at a higher cost 2459 than NALUs conforming to the limitation. 2461 The value of max-rcmd-nalu-size MUST be an integer in the 2462 range of 0 to 4294967295, inclusive. If this parameter is 2463 not specified, no known limitation to the NALU size exists. 2464 Senders still have to consider the MTU size available between 2465 the sender and the receiver and SHOULD run MTU discovery for 2466 this purpose. 2468 This parameter is motivated by, for example, an IP to H.223 2469 video telephony gateway, where NALUs smaller than the H.223 2470 transport data unit will be more efficient. A gateway may 2471 terminate IP; thus, MTU discovery will normally not work 2472 beyond the gateway. 2474 Informative note: Setting this parameter to a lower than 2475 necessary value may have a negative impact. 2477 sar-understood: 2478 This parameter MAY be used to indicate a receiver capability 2479 and not anything else. The parameter indicates the maximum 2480 value of aspect_ratio_idc (specified in [1]) smaller than 255 2481 that the receiver understands. Table E-1 of [1] specifies 2482 aspect_ratio_idc equal to 0 as "unspecified", 1 to 16, 2483 inclusive, as specific Sample Aspect Ratios (SARs), 17 to 254, 2484 inclusive, as "reserved", and 255 as the Extended SAR, for 2485 which SAR width and SAR height are explicitly signaled. 2486 Therefore, a receiver with a decoder according to [1] 2487 understands aspect_ratio_idc in the range of 1 to 16, 2488 inclusive and aspect_ratio_idc equal to 255, in the sense 2489 that the receiver knows what exactly the SAR is. For such a 2490 receiver, the value of sar-understood is 16. If in the 2491 future Table E-1 of [1] is extended, e.g., such that the SAR 2492 for aspect_ratio_idc equal to 17 is specified, then for a 2493 receiver with a decoder that understands the extension, the 2494 value of sar-understood is 17. For a receiver with a decoder 2495 according to the 2003 version of [1], the value of sar- 2496 understood is 13, as the minimum reserved aspect_ratio_idc 2497 therein is 14. 2499 When sar-understood is not present, the value MUST be 2500 inferred to be equal to 13. 2502 sar-supported: 2503 This parameter MAY be used to indicate a receiver capability 2504 and not anything else. The value of this parameter is an 2505 integer in the range of 1 to sar-understood, inclusive, equal 2506 to 255. The value of sar-supported equal to N smaller than 2507 255 indicates that the reciever supports all the SARs 2508 corresponding to H.264 aspect_ratio_idc values (see Table E-1 2509 of [1]) in the range from 1 to N, inclusive, without 2510 geometric distortion. The value of sar-supported equal to 2511 255 indicates that the receiver supports all sample aspect 2512 ratios which are expressible using two 16-bit integer values 2513 as the numerator and denominator, i.e., those that are 2514 expressible using the H.264 aspect_ratio_idc value of 255 2515 (Extended_SAR, see Table E-1 of [1]), without geometric 2516 distortion. 2518 H.264 compliant encoders SHOULD NOT send an aspect_ratio_idc 2519 equal to 0, or an aspect_ratio_idc larger than sar-understood 2520 and smaller than 255. H.264 compliant encoders SHOULD send 2521 an aspect_ratio_idc that the receiver is able to display 2522 without geometrical distortion. However, H.264 compliant 2523 encoders MAY choose to send pictures using any SAR. 2525 Note that the actual sample aspect ratio or extended sample 2526 aspect ratio, when present, of the stream is conveyed in the 2527 Video Usability Information (VUI) part of the sequence 2528 parameter set. 2530 Encoding considerations: 2531 This type is only defined for transfer via RTP (RFC 3550). 2533 Security considerations: 2534 See section 9 of RFC xxxx. 2536 Public specification: 2537 Please refer to RFC xxxx and its section 15. 2539 Additional information: 2540 None 2542 File extensions: none 2544 Macintosh file type code: none 2546 Object identifier or OID: none 2548 Person & email address to contact for further information: 2549 Ye-Kui Wang, yekuiwang@huawei.com 2551 Intended usage: COMMON 2553 Author: 2554 Ye-Kui Wang, yekuiwang@huawei.com 2556 Change controller: 2557 IETF Audio/Video Transport working group delegated from the 2558 IESG. 2560 8.2. SDP Parameters 2562 The receiver MUST ignore any parameter unspecified in this memo. 2564 8.2.1. Mapping of Payload Type Parameters to SDP 2566 The media type video/H264 string is mapped to fields in the Session 2567 Description Protocol (SDP) [6] as follows: 2569 o The media name in the "m=" line of SDP MUST be video. 2571 o The encoding name in the "a=rtpmap" line of SDP MUST be H264 2572 (the media subtype). 2574 o The clock rate in the "a=rtpmap" line MUST be 90000. 2576 o The OPTIONAL parameters "profile-level-id", "max-recv-level", 2577 "max-mbps", "max-smbps", "max-fs", "max-cpb", "max-dpb", "max- 2578 br", "redundant-pic-cap", "use-level-src-parameter-sets", "in- 2579 band-parameter-sets", "level-asymmetry-allowed", "packetization- 2580 mode", "sprop-interleaving-depth", "sprop-deint-buf-req", 2581 "deint-buf-cap", "sprop-init-buf-time", "sprop-max-don-diff", 2582 "max-rcmd-nalu-size", "sar-understood", and "sar-supported", 2583 when present, MUST be included in the "a=fmtp" line of SDP. 2584 These parameters are expressed as a media type string, in the 2585 form of a semicolon separated list of parameter=value pairs. 2587 o The OPTIONAL parameters "sprop-parameter-sets" and "sprop-level- 2588 parameter-sets", when present, MUST be included in the "a=fmtp" 2589 line of SDP or conveyed using the "fmtp" source attribute as 2590 specified in section 6.3 of [9]. For a particular media format 2591 (i.e., RTP payload type), a "sprop-parameter-sets" or "sprop- 2592 level-parameter-sets" MUST NOT be both included in the "a=fmtp" 2593 line of SDP and conveyed using the "fmtp" source attribute. 2594 When included in the "a=fmtp" line of SDP, these parameters are 2595 expressed as a media type string, in the form of a semicolon 2596 separated list of parameter=value pairs. When conveyed using 2597 the "fmtp" source attribute, these parameters are only 2598 associated with the given source and payload type as parts of 2599 the "fmtp" source attribute. 2601 Informative note: Conveyance of "sprop-parameter-sets" and 2602 "sprop-level-parameter-sets" using the "fmtp" source 2603 attribute allows for out-of-band transport of parameter sets 2604 in topologies like Topo-Video-switch-MCU [29]. 2606 An example of media representation in SDP is as follows (Baseline 2607 Profile, Level 3.0, some of the constraints of the Main profile may 2608 not be obeyed): 2610 m=video 49170 RTP/AVP 98 2611 a=rtpmap:98 H264/90000 2612 a=fmtp:98 profile-level-id=42A01E; 2613 packetization-mode=1; 2614 sprop-parameter-sets= 2616 8.2.2. Usage with the SDP Offer/Answer Model 2618 When H.264 is offered over RTP using SDP in an Offer/Answer model 2619 [8] for negotiation for unicast usage, the following limitations 2620 and rules apply: 2622 o The parameters identifying a media format configuration for 2623 H.264 are "profile-level-id" and "packetization-mode". These 2624 media format configuration parameters (except for the level part 2625 of "profile-level-id") MUST be used symmetrically; i.e., the 2626 answerer MUST either maintain all configuration parameters or 2627 remove the media format (payload type) completely, if one or 2628 more of the parameter values are not supported. Note that the 2629 level part of "profile-level-id" includes level_idc, and, for 2630 indication of level 1b when profile_idc is equal to 66, 77 or 88, 2631 bit 4 (constraint_set3_flag) of profile-iop. The level part of 2632 "profile-level-id" is changeable. 2634 Informative note: The requirement for symmetric use does not 2635 apply for the level part of "profile-level-id", and does not 2636 apply for the other stream properties and capability 2637 parameters. 2639 Informative note: In H.264 [1], all the levels except for 2640 level 1b are equal to the value of level_idc divided by 10. 2641 Level 1b is a level higher than level 1.0 but lower than 2642 level 1.1, and is signaled in an ad-hoc manner, due to that 2643 the level was specified after level 1.0 and level 1.1. For 2644 the Baseline, Main and Extended profiles (with profile_idc 2645 equal to 66, 77 and 88, respectively), level 1b is indicated 2646 by level_idc equal to 11 (i.e. same as level 1.1) and 2647 constraint_set3_flag equal to 1. For other profiles, level 2648 1b is indicated by level_idc equal to 9 (but note that level 2649 1b for these profiles are still higher than level 1, which 2650 has level_idc equal to 10, and lower than level 1.1). In SDP 2651 Offer/Answer, an answer to an offer may indicate a level 2652 equal to or lower than the level indicated in the offer. Due 2653 to the ad-hoc indication of level 1b, offerers and answerers 2654 must check the value of bit 4 (constraint_set3_flag) of the 2655 middle octet of the parameter "profile-level-id", when 2656 profile_idc is equal to 66, 77 or 88 and level_idc is equal 2657 to 11. 2659 To simplify handling and matching of these configurations, the 2660 same RTP payload type number used in the offer SHOULD also be 2661 used in the answer, as specified in [8]. An answer MUST NOT 2662 contain a payload type number used in the offer unless the 2663 configuration is exactly the same as in the offer. 2665 Informative note: When an offerer receives an answer, it has 2666 to compare payload types not declared in the offer based on 2667 the media type (i.e., video/H264) and the above media 2668 configuration parameters with any payload types it has 2669 already declared. This will enable it to determine whether 2670 the configuration in question is new or if it is equivalent 2671 to configuration already offered, since a different payload 2672 type number may be used in the answer. 2674 o The parameter "max-recv-level", when present, declares the 2675 highest level supported for receiving. In case "max-recv-level" 2676 is not present, the highest level supported for receiving is 2677 equal to the default level indicated by the level part of 2678 "profile-level-id". "max-recv-level", when present, MUST be 2679 higher than the default level. 2681 o The parameter "level-asymmetry-allowed" indicates whether level 2682 asymmetry is allowed. 2684 If "level-asymmetry-allowed" is equal to 0 (or not present) in 2685 either the offer or the answer, level asymmetry is not allowed. 2686 In this case, the level to use in the direction from the offerer 2687 to the answerer MUST be the same as the level to use in the 2688 opposite direction, and the common level to use is equal to the 2689 lower value of the default level in the offer and the default 2690 level in the answer. 2692 Otherwise ("level-asymmetry-allowed" equals to 1 in both the 2693 offer and the answer), level asymmetry is allowed. In this case, 2694 the level to use in the offerer-to-answerer direction MUST be 2695 equal to the highest level the answerer supports for receiving, 2696 and the level to use in the answerer-to-offerer direction MUST 2697 be equal to the highest level the offerer supports for receiving. 2699 When level asymmetry is not allowed, level upgrade is not 2700 allowed, i.e. the default level in the answer MUST be equal to 2701 or lower than the default level in the offer. 2703 o The parameters "sprop-deint-buf-req", "sprop-interleaving-depth", 2704 "sprop-max-don-diff", and "sprop-init-buf-time" describe the 2705 properties of the RTP packet stream that the offerer or answerer 2706 is sending for the media format configuration. This differs 2707 from the normal usage of the Offer/Answer parameters: normally 2708 such parameters declare the properties of the stream that the 2709 offerer or the answerer is able to receive. When dealing with 2710 H.264, the offerer assumes that the answerer will be able to 2711 receive media encoded using the configuration being offered. 2713 Informative note: The above parameters apply for any stream 2714 sent by the declaring entity with the same configuration; 2715 i.e., they are dependent on their source. Rather than being 2716 bound to the payload type, the values may have to be applied 2717 to another payload type when being sent, as they apply for 2718 the configuration. 2720 o The capability parameters "max-mbps", "max-smbps", "max-fs", 2721 "max-cpb", "max-dpb", "max-br", ,"redundant-pic-cap", "max-rcmd- 2722 nalu-size", "sar-understood", "sar-supported" MAY be used to 2723 declare further capabilities of the offerer or answerer for 2724 receiving. These parameters MUST NOT be present when the 2725 direction attribute is sendonly, and the parameters describe the 2726 limitations of what the offerer or answerer accepts for 2727 receiving streams. 2729 o An offerer has to include the size of the de-interleaving buffer, 2730 "sprop-deint-buf-req", in the offer for an interleaved H.264 2731 stream. To enable the offerer and answerer to inform each other 2732 about their capabilities for de-interleaving buffering in 2733 receiving streams, both parties are RECOMMENDED to include 2734 "deint-buf-cap". For interleaved streams, it is also 2735 RECOMMENDED to consider offering multiple payload types with 2736 different buffering requirements when the capabilities of the 2737 receiver are unknown. 2739 o The "sprop-parameter-sets" or "sprop-level-parameter-sets" 2740 parameter, when present (included in the "a=fmtp" line of SDP or 2741 conveyed using the "fmtp" source attribute as specified in 2742 section 6.3 of [9]), is used for out-of-band transport of 2743 parameter sets. However, when out-of-band transport of 2744 parameter sets is used, parameter sets MAY still be additionally 2745 transported in-band. 2747 The answerer MAY use either out-of-band or in-band transport of 2748 parameter sets for the stream it is sending, regardless of 2749 whether out-of-band parameter sets transport has been used in 2750 the offerer-to-answerer direction. Parameter sets included in 2751 an answer are independent of those parameter sets included in 2752 the offer, as they are used for decoding two different video 2753 streams, one from the answerer to the offerer, and the other in 2754 the opposite direction. 2756 The following rules apply to transport of parameter sets in the 2757 offerer-to-answerer direction. 2759 o An offer MAY include either or both of "sprop-parameter- 2760 sets" and "sprop-level-parameter-sets". If neither "sprop- 2761 parameter-sets" nor "sprop-level-parameter-sets" is present 2762 in the offer, then only in-band transport of parameter sets 2763 is used. 2765 o If the answer includes "in-band-parameter-sets" equal to 1, 2766 then the offerer MUST transmit parameter sets in-band. 2767 Otherwise, the following applies. 2769 o If the level to use in the offerer-to-answerer 2770 direction is equal to the default level in the offer, 2771 the following applies. 2773 When there is a "sprop-parameter-sets" included 2774 in the "a=fmtp" line in the offer, the answerer 2775 MUST be prepared to use the parameter sets 2776 included in the "sprop-parameter-sets" for 2777 decoding the incoming NAL unit stream. 2779 When there is a "sprop-parameter-sets" conveyed 2780 using the "fmtp" source attribute in the offer, 2781 the following applies. If the answer includes 2782 "use-level-src-parameter-sets" equal to 1 or the 2783 "fmtp" source attribute, the answerer MUST be 2784 prepared to use the parameter sets included in 2785 the "sprop-parameter-sets" for decoding the 2786 incoming NAL unit stream; Otherwise, the offerer 2787 MUST transmit parameter sets in-band. 2789 When "sprop-parameter-sets" is not present in the 2790 offer, the offerer MUST transmit parameter sets 2791 in-band. 2793 The answerer MUST ignore "sprop-level-parameter- 2794 sets", when present (either included in the 2795 "a=fmtp" line or conveyed using the "fmtp" source 2796 attribute) in the offer. 2798 o Otherwise (the level to use in the offerer-to-answerer 2799 direction is not equal to the default level in the 2800 offer), the following applies. 2802 The answerer MUST ignore "sprop-parameter-sets", 2803 when present (either included in the "a=fmtp" 2804 line or conveyed using the "fmtp" source 2805 attribute) in the offer. 2807 When neither "use-level-src-parameter-sets" equal 2808 to 1 nor the "fmtp" source attribute is present 2809 in the answer, the answerer MUST ignore "sprop- 2810 level-parameter-sets", when present in the offer, 2811 and the offerer MUST transmit parameter sets in- 2812 band. 2814 When either "use-level-src-parameter-sets" equal 2815 to 1 or the "fmtp" source attribute is present in 2816 the answer, the answerer MUST be prepared to use 2817 the parameter sets that are included in "sprop- 2818 level-parameter-sets" for the accepted level (i.e. 2819 the default level in the answer), when present in 2820 the offer, for decoding the incoming NAL unit 2821 stream, and ignore all other parameter sets 2822 included in "sprop-level-parameter-sets". 2824 When no parameter sets for the level to use in 2825 the offerer-to-answerer direction are present in 2826 "sprop-level-parameter-sets" in the offer, the 2827 offerer MUST transmit parameter sets in-band. 2829 The following rules apply to transport of parameter sets in the 2830 answerer-to-offerer direction. 2832 o An answer MAY include either "sprop-parameter-sets" or 2833 "sprop-level-parameter-sets", but MUST NOT include both of 2834 the two. If neither "sprop-parameter-sets" nor "sprop- 2835 level-parameter-sets" is present in the answer, then only 2836 in-band transport of parameter sets is used. 2838 o If the offer includes "in-band-parameter-sets" equal to 1, 2839 the answerer MUST NOT include "sprop-parameter-sets" or 2840 "sprop-level-parameter-sets" in the answer and MUST 2841 transmit parameter sets in-band. Otherwise, the following 2842 applies. 2844 o If the level to use in the answerer-to-offerer 2845 direction is equal to the default level in the answer, 2846 the following applies. 2848 When there is a "sprop-parameter-sets" included 2849 in the "a=fmtp" line in the answer, the offerer 2850 MUST be prepared to use the parameter sets 2851 included in the "sprop-parameter-sets" for 2852 decoding the incoming NAL unit stream. 2854 When there is a "sprop-parameter-sets" conveyed 2855 using the "fmtp" source attribute in the answer, 2856 the following applies. If the offer includes 2857 "use-level-src-parameter-sets" equal to 1 or the 2858 "fmtp" source attribute, the offerer MUST be 2859 prepared to use the parameter sets included in 2860 the "sprop-parameter-sets" for decoding the 2861 incoming NAL unit stream; Otherwise, the 2862 answerer MUST transmit parameter sets in-band. 2864 When "sprop-parameter-sets" is not present in the 2865 answer, the answerer MUST transmit parameter sets 2866 in-band. 2868 The offerer MUST ignore "sprop-level-parameter- 2869 sets", when present (either included in the 2870 "a=fmtp" line or conveyed using the "fmtp" source 2871 attribute) in the answer. 2873 o Otherwise (the level to use in the answerer-to-offerer 2874 direction is not equal to the default level in the 2875 answer), the following applies. 2877 The offerer MUST ignore "sprop-parameter-sets", 2878 when present (either included in the "a=fmtp" 2879 line of SDP or conveyed using the "fmtp" source 2880 attribute) in the answer. 2882 When neither "use-level-src-parameter-sets" equal 2883 to 1 nor the "fmtp" source attribute is present 2884 in the offer, the offerer MUST ignore "sprop- 2885 level-parameter-sets", when present, and the 2886 answerer MUST transmit parameter sets in-band. 2888 When either "use-level-src-parameter-sets" equal 2889 to 1 or the "fmtp" source attribute is present in 2890 the offer, the offerer MUST be prepared to use 2891 the parameter sets that are included in "sprop- 2892 level-parameter-sets" for the level to use in the 2893 answerer-to-offerer direction, when present in 2894 the answer, for decoding the incoming NAL unit 2895 stream, and ignore all other parameter sets 2896 included in "sprop-level-parameter-sets" in the 2897 answer. 2899 When no parameter sets for the level to use in 2900 the answerer-to-offerer direction are present in 2901 "sprop-level-parameter-sets" in the answer, the 2902 answerer MUST transmit parameter sets in-band. 2904 When "sprop-parameter-sets" or "sprop-level-parameter-sets" is 2905 conveyed using the "fmtp" source attribute as specified in 2906 section 6.3 of [9], the receiver of the parameters MUST store 2907 the parameter sets included in the "sprop-parameter-sets" or 2908 "sprop-level-parameter-sets" for the accepted level and 2909 associate them to the source given as a part of the "fmtp" 2910 source attribute. Parameter sets associated with one source 2911 MUST only be used to decode NAL units conveyed in RTP packets 2912 from the same source. When this mechanism is in use, SSRC 2913 collision detection and resolution MUST be performed as 2914 specified in [9]. 2916 Informative note: Conveyance of "sprop-parameter-sets" and 2917 "sprop-level-parameter-sets" using the "fmtp" source 2918 attribute may be used in topologies like Topo-Video-switch- 2919 MCU [29] to enable out-of-band transport of parameter sets. 2921 For streams being delivered over multicast, the following rules 2922 apply: 2924 o The media format configuration is identified by "profile-level- 2925 id", including the level part, and "packetization-mode". These 2926 media format configuration parameters (including the level part 2927 of "profile-level-id") MUST be used symmetrically; i.e., the 2928 answerer MUST either maintain all configuration parameters or 2929 remove the media format (payload type) completely. Note that 2930 this implies that the level part of "profile-level-id" for 2931 Offer/Answer in multicast is not changeable. 2933 To simplify handling and matching of these configurations, the 2934 same RTP payload type number used in the offer SHOULD also be 2935 used in the answer, as specified in [8]. An answer MUST NOT 2936 contain a payload type number used in the offer unless the 2937 configuration is the same as in the offer. 2939 o Parameter sets received MUST be associated with the originating 2940 source, and MUST be only used in decoding the incoming NAL unit 2941 stream from the same source. 2943 o The rules for other parameters are the same as above for unicast 2944 as long as the above rules are obeyed. 2946 Table 6 lists the interpretation of all the media type parameters 2947 that MUST be used for the different direction attributes. 2949 Table 6. Interpretation of parameters for different direction 2950 attributes. 2952 sendonly --+ 2953 recvonly --+ | 2954 sendrecv --+ | | 2955 | | | 2956 profile-level-id C C P 2957 max-recv-level R R - 2958 packetization-mode C C P 2959 sprop-deint-buf-req P - P 2960 sprop-interleaving-depth P - P 2961 sprop-max-don-diff P - P 2962 sprop-init-buf-time P - P 2963 max-mbps R R - 2964 max-smbps R R - 2965 max-fs R R - 2966 max-cpb R R - 2967 max-dpb R R - 2968 max-br R R - 2969 redundant-pic-cap R R - 2970 deint-buf-cap R R - 2971 max-rcmd-nalu-size R R - 2972 sar-understood R R - 2973 sar-supported R R - 2974 in-band-parameter-sets R R - 2975 use-level-src-parameter-sets R R - 2976 level-asymmetry-allowed O - - 2977 sprop-parameter-sets S - S 2978 sprop-level-parameter-sets S - S 2980 Legend: 2982 C: configuration for sending and receiving streams 2983 O: offer/answer mode 2984 P: properties of the stream to be sent 2985 R: receiver capabilities 2986 S: out-of-band parameter sets 2987 -: not usable, when present SHOULD be ignored 2989 Parameters used for declaring receiver capabilities are in general 2990 downgradable; i.e., they express the upper limit for a sender's 2991 possible behavior. Thus a sender MAY select to set its encoder 2992 using only lower/less or equal values of these parameters. 2994 Parameters declaring a configuration point are not changeable, with 2995 the exception of the level part of the "profile-level-id" parameter 2996 for unicast usage. These express values a receiver expects to be 2997 used and must be used verbatim on the sender side. 2999 When a sender's capabilities are declared, and non-downgradable 3000 parameters are used in this declaration, then these parameters 3001 express a configuration that is acceptable for the sender to 3002 receive streams. In order to achieve high interoperability levels, 3003 it is often advisable to offer multiple alternative configurations; 3004 e.g., for the packetization mode. It is impossible to offer 3005 multiple configurations in a single payload type. Thus, when 3006 multiple configuration offers are made, each offer requires its own 3007 RTP payload type associated with the offer. 3009 A receiver SHOULD understand all media type parameters, even if it 3010 only supports a subset of the payload format's functionality. This 3011 ensures that a receiver is capable of understanding when an offer 3012 to receive media can be downgraded to what is supported by the 3013 receiver of the offer. 3015 An answerer MAY extend the offer with additional media format 3016 configurations. However, to enable their usage, in most cases a 3017 second offer is required from the offerer to provide the stream 3018 property parameters that the media sender will use. This also has 3019 the effect that the offerer has to be able to receive this media 3020 format configuration, not only to send it. 3022 If an offerer wishes to have non-symmetric capabilities between 3023 sending and receiving, the offerer can allow asymmetric levels via 3024 "level-asymmetry-allowed" equal to 1. Alternatively, the offerer 3025 could offer different RTP sessions; i.e., different media lines 3026 declared as "recvonly" and "sendonly", respectively. This may have 3027 further implications on the system, and may require additional 3028 external semantics to associate the two media lines. 3030 8.2.3. Usage in Declarative Session Descriptions 3032 When H.264 over RTP is offered with SDP in a declarative style, as 3033 in RTSP [27] or SAP [28], the following considerations are 3034 necessary. 3036 o All parameters capable of indicating both stream properties and 3037 receiver capabilities are used to indicate only stream 3038 properties. For example, in this case, the parameter "profile- 3039 level-id" declares only the values used by the stream, not the 3040 capabilities for receiving streams. This results in that the 3041 following interpretation of the parameters MUST be used: 3043 Declaring actual configuration or stream properties: 3045 - profile-level-id 3046 - packetization-mode 3047 - sprop-interleaving-depth 3048 - sprop-deint-buf-req 3049 - sprop-max-don-diff 3050 - sprop-init-buf-time 3052 Out-of-band transporting of parameter sets: 3054 - sprop-parameter-sets 3055 - sprop-level-parameter-sets 3057 Not usable(when present, they SHOULD be ignored): 3059 - max-mbps 3060 - max-smbps 3061 - max-fs 3062 - max-cpb 3063 - max-dpb 3064 - max-br 3065 - max-recv-level 3066 - redundant-pic-cap 3067 - max-rcmd-nalu-size 3068 - deint-buf-cap 3069 - sar-understood 3070 - sar-supported 3071 - in-band-parameter-sets 3072 - level-asymmetry-allowed 3073 - use-level-src-parameter-sets 3075 o A receiver of the SDP is required to support all parameters and 3076 values of the parameters provided; otherwise, the receiver MUST 3077 reject (RTSP) or not participate in (SAP) the session. It falls 3078 on the creator of the session to use values that are expected to 3079 be supported by the receiving application. 3081 8.3. Examples 3083 An SDP Offer/Answer exchange wherein both parties are expected to 3084 both send and receive could look like the following. Only the 3085 media codec specific parts of the SDP are shown. Some lines are 3086 wrapped due to text constraints. 3088 Offerer -> Answerer SDP message: 3090 m=video 49170 RTP/AVP 100 99 98 3091 a=rtpmap:98 H264/90000 3092 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; 3093 sprop-parameter-sets= 3094 a=rtpmap:99 H264/90000 3095 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3096 sprop-parameter-sets= 3097 a=rtpmap:100 H264/90000 3098 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3099 sprop-parameter-sets=; 3100 sprop-interleaving-depth=45; sprop-deint-buf-req=64000; 3101 sprop-init-buf-time=102478; deint-buf-cap=128000 3103 The above offer presents the same codec configuration in three 3104 different packetization formats. PT 98 represents single NALU mode, 3105 PT 99 represents non-interleaved mode, and PT 100 indicates the 3106 interleaved mode. In the interleaved mode case, the interleaving 3107 parameters that the offerer would use if the answer indicates 3108 support for PT 100 are also included. In all three cases the 3109 parameter "sprop-parameter-sets" conveys the initial parameter sets 3110 that are required by the answerer when receiving a stream from the 3111 offerer when this configuration is accepted. Note that the value 3112 for "sprop-parameter-sets" could be different for each payload type. 3114 Answerer -> Offerer SDP message: 3116 m=video 49170 RTP/AVP 100 99 97 3117 a=rtpmap:97 H264/90000 3118 a=fmtp:97 profile-level-id=42A01E; packetization-mode=0; 3119 sprop-parameter-sets= 3120 a=rtpmap:99 H264/90000 3121 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3122 sprop-parameter-sets=; 3123 max-rcmd-nalu-size=3980 3124 a=rtpmap:100 H264/90000 3125 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3126 sprop-parameter-sets=; 3127 sprop-interleaving-depth=60; 3128 sprop-deint-buf-req=86000; sprop-init-buf-time=156320; 3129 deint-buf-cap=128000; max-rcmd-nalu-size=3980 3131 As the Offer/Answer negotiation covers both sending and receiving 3132 streams, an offer indicates the exact parameters for what the 3133 offerer is willing to receive, whereas the answer indicates the 3134 same for what the answerer accepts to receive. In this case the 3135 offerer declared that it is willing to receive payload type 98. 3136 The answerer accepts this by declaring an equivalent payload type 3137 97; i.e., it has identical values for the two parameters "profile- 3138 level-id" and "packetization-mode" (since "packetization-mode" is 3139 equal to 0, "sprop-deint-buf-req" is not present). As the offered 3140 payload type 98 is accepted, the answerer needs to store parameter 3141 sets included in sprop-parameter-sets= in 3142 case the offer finally decides to use this configuration. In the 3143 answer, the answerer includes the parameter sets in sprop- 3144 parameter-sets= that the answerer would use 3145 in the stream sent from the answerer if this configuration is 3146 finally used. 3148 The answerer also accepts the reception of the two configurations 3149 that payload types 99 and 100 represent. Again, the answerer needs 3150 to store parameter sets included in sprop-parameter-sets= and sprop-parameter-sets= in 3152 case the offer finally decides to use either of these two 3153 configurations. The answerer provides the initial parameter sets 3154 for the answerer-to-offerer direction, i.e. the parameter sets in 3155 sprop-parameter-sets= and sprop-parameter- 3156 sets=, for payload types 99 and 100, 3157 respectively, that it will use to send the payload types. The 3158 answerer also provides the offerer with its memory limit for de- 3159 interleaving operations by providing a "deint-buf-cap" parameter. 3160 This is only useful if the offerer decides on making a second offer, 3161 where it can take the new value into account. The "max-rcmd-nalu- 3162 size" indicates that the answerer can efficiently process NALUs up 3163 to the size of 3980 bytes. However, there is no guarantee that the 3164 network supports this size. 3166 In the following example, the offer is accepted without level 3167 downgrading (i.e. the default level, 3.0, is accepted), and both 3168 "sprop-parameter-sets" and "sprop-level-parameter-sets" are present 3169 in the offer. The answerer must ignore sprop-level-parameter- 3170 sets= and store parameter sets in sprop- 3171 parameter-sets= for decoding the incoming 3172 NAL unit stream. The offerer must store the parameter sets in 3173 sprop-parameter-sets= in the answer for 3174 decoding the incoming NAL unit stream. Note that in this example, 3175 parameter sets in sprop-parameter-sets= must 3176 be associated with level 3.0. 3178 Offer SDP: 3180 m=video 49170 RTP/AVP 98 3181 a=rtpmap:98 H264/90000 3182 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3183 packetization-mode=1; 3184 sprop-parameter-sets=; 3185 sprop-level-parameter-sets= 3187 Answer SDP: 3189 m=video 49170 RTP/AVP 98 3190 a=rtpmap:98 H264/90000 3191 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3192 packetization-mode=1; 3193 sprop-parameter-sets= 3195 In the following example, the offer (Baseline profile, level 1.1) 3196 is accepted with level downgrading (the accepted level is 1b), and 3197 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3198 present in the offer. The answerer must ignore sprop-parameter- 3199 sets= and all parameter sets not for the 3200 accepted level (level 1b) in sprop-level-parameter-sets=, and must store parameter sets for the accepted level 3202 (level 1b) in sprop-level-parameter-sets= 3203 for decoding the incoming NAL unit stream. The offerer must store 3204 the parameter sets in sprop-parameter-sets= 3205 in the answer for decoding the incoming NAL unit stream. Note that 3206 in this example, parameter sets in sprop-parameter-sets= must be associated with level 1b. 3209 Offer SDP: 3211 m=video 49170 RTP/AVP 98 3212 a=rtpmap:98 H264/90000 3213 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3214 packetization-mode=1; 3215 sprop-parameter-sets=; 3216 sprop-level-parameter-sets= 3218 Answer SDP: 3220 m=video 49170 RTP/AVP 98 3221 a=rtpmap:98 H264/90000 3222 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3223 packetization-mode=1; 3224 sprop-parameter-sets=; 3225 use-level-src-parameter-sets=1 3227 In the following example, the offer (Baseline profile, level 1.1) 3228 is accepted with level downgrading (the accepted level is 1b), and 3229 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3230 present in the offer. However, the answerer is a legacy RFC 3984 3231 implementation and does not understand "sprop-level-parameter-sets", 3232 hence it does not include "use-level-src-parameter-sets" (which the 3233 answerer does not understand, either) in the answer. Therefore, 3234 the answerer must ignore both sprop-parameter-sets= and sprop-level-parameter-sets=, and 3236 the offerer must transport parameter sets in-band. 3238 Offer SDP: 3240 m=video 49170 RTP/AVP 98 3241 a=rtpmap:98 H264/90000 3242 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3243 packetization-mode=1; 3244 sprop-parameter-sets=; 3245 sprop-level-parameter-sets= 3247 Answer SDP: 3249 m=video 49170 RTP/AVP 98 3250 a=rtpmap:98 H264/90000 3251 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3252 packetization-mode=1 3254 In the following example, the offer is accepted without level 3255 downgrading, and "sprop-parameter-sets" is present in the offer. 3256 Parameter sets in sprop-parameter-sets= must 3257 be stored and used used by the encoder of the offerer and the 3258 decoder of the answerer, and parameter sets in sprop-parameter- 3259 sets=must be used by the encoder of the 3260 answerer and the decoder of the offerer. Note that sprop- 3261 parameter-sets= is basically independent of 3262 sprop-parameter-sets=. 3264 Offer SDP: 3266 m=video 49170 RTP/AVP 98 3267 a=rtpmap:98 H264/90000 3268 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3269 packetization-mode=1; 3270 sprop-parameter-sets= 3272 Answer SDP: 3274 m=video 49170 RTP/AVP 98 3275 a=rtpmap:98 H264/90000 3276 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3277 packetization-mode=1; 3278 sprop-parameter-sets= 3280 In the following example, the offer is accepted without level 3281 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3282 parameter-sets" is present in the offer, meaning that there is no 3283 out-of-band transmission of parameter sets, which then have to be 3284 transported in-band. 3286 Offer SDP: 3288 m=video 49170 RTP/AVP 98 3289 a=rtpmap:98 H264/90000 3290 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3291 packetization-mode=1 3293 Answer SDP: 3295 m=video 49170 RTP/AVP 98 3296 a=rtpmap:98 H264/90000 3297 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3298 packetization-mode=1 3300 In the following example, the offer is accepted with level 3301 downgrading and "sprop-parameter-sets" is present in the offer. As 3302 sprop-parameter-sets= contains level_idc 3303 indicating Level 3.0, therefore cannot be used as the answerer 3304 wants Level 2.0 and must be ignored by the answerer, and in-band 3305 parameter sets must be used. 3307 Offer SDP: 3309 m=video 49170 RTP/AVP 98 3310 a=rtpmap:98 H264/90000 3311 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3312 packetization-mode=1; 3313 sprop-parameter-sets= 3315 Answer SDP: 3317 m=video 49170 RTP/AVP 98 3318 a=rtpmap:98 H264/90000 3319 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3320 packetization-mode=1 3322 In the following example, the offer is also accepted with level 3323 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3324 parameter-sets" is present in the offer, meaning that there is no 3325 out-of-band transmission of parameter sets, which then have to be 3326 transported in-band. 3328 Offer SDP: 3330 m=video 49170 RTP/AVP 98 3331 a=rtpmap:98 H264/90000 3332 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3333 packetization-mode=1 3335 Answer SDP: 3337 m=video 49170 RTP/AVP 98 3338 a=rtpmap:98 H264/90000 3339 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3340 packetization-mode=1 3342 In the following example, the offer is accepted with level 3343 upgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3344 parameter-sets" is present in the offer or the answer, meaning that 3345 there is no out-of-band transmission of parameter sets, which then 3346 have to be transported in-band. The level to use in the offerer- 3347 to-answerer direction is Level 3.0, and the level to use in the 3348 answerer-to-offerer direction is Level 2.0. The answerer is 3349 allowed to send at any level up to and including level 2.0, and the 3350 offerer is allowed to send at any level up to and including level 3351 3.0. 3353 Offer SDP: 3355 m=video 49170 RTP/AVP 98 3356 a=rtpmap:98 H264/90000 3357 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3358 packetization-mode=1; level-asymmetry-allowed=1 3360 Answer SDP: 3362 m=video 49170 RTP/AVP 98 3363 a=rtpmap:98 H264/90000 3364 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3365 packetization-mode=1; level-asymmetry-allowed=1 3367 In the following example, the offerer is a Multipoint Control Unit 3368 (MCU) in a Topo-Video-switch-MCU like topology [29], offering 3369 parameter sets received (using out-of-band transport) from three 3370 other participants B, C, and D, and receiving parameter sets from 3371 the participant A, which is the answerer. The participants are 3372 identified by their values of CNAME, which are mapped to different 3373 SSRC values. The same codec configuration is used by all the four 3374 participants. The participant A stores and associates the 3375 parameter sets included in , , and to participants B, C, and D, 3377 respectively, and uses for decoding NAL 3378 units carried in RTP packets originated from participant B only, 3379 uses for decoding NAL units carried in RTP 3380 packets originated from participant C only, and uses for decoding NAL units carried in RTP packets 3382 originated from participant D only. 3384 Offer SDP: 3386 m=video 49170 RTP/AVP 98 3387 a=ssrc:SSRC-B cname:CNAME-B 3388 a=ssrc:SSRC-C cname:CNAME-C 3389 a=ssrc:SSRC-D cname:CNAME-D 3390 a=ssrc:SSRC-B fmtp:98 3391 sprop-parameter-sets= 3392 a=ssrc:SSRC-C fmtp:98 3393 sprop-parameter-sets= 3394 a=ssrc:SSRC-D fmtp:98 3395 sprop-parameter-sets= 3396 a=rtpmap:98 H264/90000 3397 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3398 packetization-mode=1 3400 Answer SDP: 3402 m=video 49170 RTP/AVP 98 3403 a=ssrc:SSRC-A cname:CNAME-A 3404 a=ssrc:SSRC-A fmtp:98 3405 sprop-parameter-sets= 3406 a=rtpmap:98 H264/90000 3407 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3408 packetization-mode=1 3410 8.4. Parameter Set Considerations 3412 The H.264 parameter sets are a fundamental part of the video codec 3413 and vital to its operation; see section 1.2. Due to their 3414 characteristics and their importance for the decoding process, lost 3415 or erroneously transmitted parameter sets can hardly be concealed 3416 locally at the receiver. A reference to a corrupt parameter set 3417 has normally fatal results to the decoding process. Corruption 3418 could occur, for example, due to the erroneous transmission or loss 3419 of a parameter set NAL unit, but also due to the untimely 3420 transmission of a parameter set update. A parameter set update 3421 refers to a change of at least one parameter in a picture parameter 3422 set or sequence parameter set for which the picture parameter set 3423 or sequence parameter set identifier remains unchanged. Therefore, 3424 the following recommendations are provided as a guideline for the 3425 implementer of the RTP sender. 3427 Parameter set NALUs can be transported using three different 3428 principles: 3430 A. Using a session control protocol (out-of-band) prior to the 3431 actual RTP session. 3433 B. Using a session control protocol (out-of-band) during an ongoing 3434 RTP session. 3436 C. Within the RTP packet stream in the payload (in-band) during an 3437 ongoing RTP session. 3439 It is recommended to implement principles A and B within a session 3440 control protocol. SIP and SDP can be used as described in the SDP 3441 Offer/Answer model and in the previous sections of this memo. 3442 Section 8.2.2 includes a detailed discussion on transport of 3443 parameter sets in-band or out-of-band in SDP Offer/Answer using 3444 media type parameters "sprop-parameter-sets", "sprop-level- 3445 parameter-sets", "use-level-src-parameter-sets" and "in-band- 3446 parameter-sets". This section contains guidelines on how 3447 principles A and B should be implemented within session control 3448 protocols. It is independent of the particular protocol used. 3449 Principle C is supported by the RTP payload format defined in this 3450 specification. There are topologies like Topo-Video-switch-MCU [29] 3451 for which the use of principle C may be desirable. 3453 If in-band signaling of parameter sets is used, the picture and 3454 sequence parameter set NALUs SHOULD be transmitted in the RTP 3455 payload using a reliable method of delivering of RTP (see below), 3456 as a loss of a parameter set of either type will likely prevent 3457 decoding of a considerable portion of the corresponding RTP packet 3458 stream. 3460 If in-band signaling of parameter sets is used, the sender SHOULD 3461 take the error characteristics into account and use mechanisms to 3462 provide a high probability for delivering the parameter sets 3463 correctly. Mechanisms that increase the probability for a correct 3464 reception include packet repetition, FEC, and retransmission. The 3465 use of an unreliable, out-of-band control protocol has similar 3466 disadvantages as the in-band signaling (possible loss) and, in 3467 addition, may also lead to difficulties in the synchronization (see 3468 below). Therefore, it is NOT RECOMMENDED. 3470 Parameter sets MAY be added or updated during the lifetime of a 3471 session using principles B and C. It is required that parameter 3472 sets are present at the decoder prior to the NAL units that refer 3473 to them. Updating or adding of parameter sets can result in 3474 further problems, and therefore the following recommendations 3475 should be considered. 3477 - When parameter sets are added or updated, care SHOULD be taken 3478 to ensure that any parameter set is delivered prior to its usage. 3479 When new parameter sets are added, previously unused parameter 3480 set identifiers are used. It is common that no synchronization 3481 is present between out-of-band signaling and in-band traffic. 3482 If out-of-band signaling is used, it is RECOMMENDED that a 3483 sender does not start sending NALUs requiring the added or 3484 updated parameter sets prior to acknowledgement of delivery from 3485 the signaling protocol. 3487 - When parameter sets are updated, the following synchronization 3488 issue should be taken into account. When overwriting a 3489 parameter set at the receiver, the sender has to ensure that the 3490 parameter set in question is not needed by any NALU present in 3491 the network or receiver buffers. Otherwise, decoding with a 3492 wrong parameter set may occur. To lessen this problem, it is 3493 RECOMMENDED either to overwrite only those parameter sets that 3494 have not been used for a sufficiently long time (to ensure that 3495 all related NALUs have been consumed), or to add a new parameter 3496 set instead (which may have negative consequences for the 3497 efficiency of the video coding). 3499 Informative note: In some topologies like Topo-Video-switch- 3500 MCU [29] the origin of the whole set of parameter sets may 3501 come from multiple sources that may use non-unique parameter 3502 sets identifiers. In this case an offer may overwrite an 3503 existing parameter set if no other mechanism that enables 3504 uniqueness of the parameter sets in the out-of-band channel 3505 exists. 3507 - In a multiparty session, one participant MUST associate 3508 parameter sets coming from different sources with the source 3509 identification whenever possible, e.g. by conveying out-of-band 3510 transported parameter sets, as different sources typically use 3511 independent parameter set identifier value spaces. 3513 - Adding or modifying parameter sets by using both principles B 3514 and C in the same RTP session may lead to inconsistencies of the 3515 parameter sets because of the lack of synchronization between 3516 the control and the RTP channel. Therefore, principles B and C 3517 MUST NOT both be used in the same session unless sufficient 3518 synchronization can be provided. 3520 In some scenarios (e.g., when only the subset of this payload 3521 format specification corresponding to H.241 is used) or topologies, 3522 it is not possible to employ out-of-band parameter set transmission. 3523 In this case, parameter sets have to be transmitted in-band. Here, 3524 the synchronization with the non-parameter-set-data in the 3525 bitstream is implicit, but the possibility of a loss has to be 3526 taken into account. The loss probability should be reduced using 3527 the mechanisms discussed above. In case a loss of a parameter set 3528 is detected, recovery may be achieved by using a Decoder Refresh 3529 Point procedure, for example, using RTCP feedback Full Intra 3530 Request (FIR) [30]. Two example Decoder Refresh Point procedures 3531 are provided in the informative Section 8.5. 3533 - When parameter sets are initially provided using principle A and 3534 then later added or updated in-band (principle C), there is a 3535 risk associated with updating the parameter sets delivered out- 3536 of-band. If receivers miss some in-band updates (for example, 3537 because of a loss or a late tune-in), those receivers attempt to 3538 decode the bitstream using out-dated parameters. It is 3539 therefore RECOMMENDED that parameter set IDs be partitioned 3540 between the out-of-band and in-band parameter sets. 3542 8.5. Decoder Refresh Point Procedure using In-Band Transport of 3543 Parameter Sets (Informative) 3545 When a sender with a video encoder according to [1] receives a 3546 request for a decoder refresh point, the encoder shall enter the 3547 fast update mode by using one of the procedures specified 3548 in Section 8.5.1 or 8.5.2 below. The procedure in 8.5.1 is the 3549 preferred response in a lossless transmission environment. Both 3550 procedures satisfy the requirement to enter the fast update mode 3551 for H.264 video encoding. 3553 8.5.1. IDR Procedure to Respond to a Request for a Decoder Refresh 3554 Point 3556 This section gives one possible way to respond to a request for a 3557 decoder refresh point. 3559 The encoder shall, in the order presented here: 3561 1) Immediately prepare to send an IDR picture. 3563 2) Send a sequence parameter set to be used by the IDR picture to 3564 be sent. The encoder may optionally also send other sequence 3565 parameter sets. 3567 3) Send a picture parameter set to be used by the IDR picture to be 3568 sent. The encoder may optionally also send other picture 3569 parameter sets. 3571 4) Send the IDR picture. 3573 5) From this point forward in time, send any other sequence or 3574 picture parameter sets that have not yet been sent in this 3575 procedure, prior to their reference by any NAL unit, regardless 3576 of whether such parameter sets were previously sent prior to 3577 receiving the request for a decoder refresh point. As needed, 3578 such parameter sets may be sent in a batch, one at a time, or in 3579 any combination of these two methods. Parameter sets may be re- 3580 sent at any time for redundancy. Caution should be taken when 3581 parameter set updates are present, as described above in Section 3582 8.4. 3584 8.5.2. Gradual Recovery Procedure to Respond to a Request for a 3585 Decoder Refresh Point 3587 This section gives another possible way to respond to a request for 3588 a decoder refresh point. 3590 The encoder shall, in the order presented here: 3592 1) Send a recovery point SEI message (see Sections D.1.7 and D.2.7 3593 of [1]). 3595 2) Repeat any sequence and picture parameter sets that were sent 3596 before the recovery point SEI message, prior to their reference 3597 by a NAL unit. 3599 The encoder shall ensure that the decoder has access to all 3600 reference pictures for inter prediction of pictures at or after the 3601 recovery point, which is indicated by the recovery point SEI 3602 message, in output order, assuming that the transmission from now 3603 on is error-free. 3605 The value of the recovery_frame_cnt syntax element in the recovery 3606 point SEI message should be small enough to ensure a fast recovery. 3608 As needed, such parameter sets may be re-sent in a batch, one at a 3609 time, or in any combination of these two methods. Parameter sets 3610 may be re-sent at any time for redundancy. Caution should be taken 3611 when parameter set updates are present, as described above in 3612 Section 8.4. 3614 9. Security Considerations 3616 RTP packets using the payload format defined in this specification 3617 are subject to the security considerations discussed in the RTP 3618 specification [5], and in any appropriate RTP profile (for example, 3619 [16]). This implies that confidentiality of the media streams is 3620 achieved by encryption; for example, through the application of 3621 SRTP [26]. Because the data compression used with this payload 3622 format is applied end-to-end, any encryption needs to be performed 3623 after compression. A potential denial-of-service threat exists for 3624 data encodings using compression techniques that have non-uniform 3625 receiver-end computational load. The attacker can inject 3626 pathological datagrams into the stream that are complex to decode 3627 and that cause the receiver to be overloaded. H.264 is 3628 particularly vulnerable to such attacks, as it is extremely simple 3629 to generate datagrams containing NAL units that affect the decoding 3630 process of many future NAL units. Therefore, the usage of data 3631 origin authentication and data integrity protection of at least the 3632 RTP packet is RECOMMENDED; for example, with SRTP [26]. 3634 Note that the appropriate mechanism to ensure confidentiality and 3635 integrity of RTP packets and their payloads is very dependent on 3636 the application and on the transport and signaling protocols 3637 employed. Thus, although SRTP is given as an example above, other 3638 possible choices exist. 3640 Decoders MUST exercise caution with respect to the handling of user 3641 data SEI messages, particularly if they contain active elements, 3642 and MUST restrict their domain of applicability to the presentation 3643 containing the stream. 3645 End-to-End security with either authentication, integrity or 3646 confidentiality protection will prevent a MANE from performing 3647 media-aware operations other than discarding complete packets. And 3648 in the case of confidentiality protection it will even be prevented 3649 from performing discarding of packets in a media aware way. To 3650 allow any MANE to perform its operations, it will be required to be 3651 a trusted entity which is included in the security context 3652 establishment. 3654 10. Congestion Control 3656 Congestion control for RTP SHALL be used in accordance with RFC 3657 3550 [5], and with any applicable RTP profile; e.g., RFC 3551 [16]. 3658 An additional requirement if best-effort service is being used is: 3659 users of this payload format MUST monitor packet loss to ensure 3660 that the packet loss rate is within acceptable parameters. Packet 3661 loss is considered acceptable if a TCP flow across the same network 3662 path, and experiencing the same network conditions, would achieve 3663 an average throughput, measured on a reasonable timescale, that is 3664 not less than the RTP flow is achieving. This condition can be 3665 satisfied by implementing congestion control mechanisms to adapt 3666 the transmission rate (or the number of layers subscribed for a 3667 layered multicast session), or by arranging for a receiver to leave 3668 the session if the loss rate is unacceptably high. 3670 The bit rate adaptation necessary for obeying the congestion 3671 control principle is easily achievable when real-time encoding is 3672 used. However, when pre-encoded content is being transmitted, 3673 bandwidth adaptation requires the availability of more than one 3674 coded representation of the same content, at different bit rates, 3675 or the existence of non-reference pictures or sub-sequences [22] in 3676 the bitstream. The switching between the different representations 3677 can normally be performed in the same RTP session; e.g., by 3678 employing a concept known as SI/SP slices of the Extended Profile, 3679 or by switching streams at IDR picture boundaries. Only when non- 3680 downgradable parameters (such as the profile part of the 3681 profile/level ID) are required to be changed does it become 3682 necessary to terminate and re-start the media stream. This may be 3683 accomplished by using a different RTP payload type. 3685 MANEs MAY follow the suggestions outlined in section 7.3 and remove 3686 certain unusable packets from the packet stream when that stream 3687 was damaged due to previous packet losses. This can help reduce 3688 the network load in certain special cases. 3690 11. IANA Consideration 3692 The H264 media subtype name specified by RFC 3984 should be updated 3693 as defined in section 8.1 of this memo. 3695 12. Informative Appendix: Application Examples 3697 This payload specification is very flexible in its use, in order to 3698 cover the extremely wide application space anticipated for H.264. 3699 However, this great flexibility also makes it difficult for an 3700 implementer to decide on a reasonable packetization scheme. Some 3701 information on how to apply this specification to real-world 3702 scenarios is likely to appear in the form of academic publications 3703 and a test model software and description in the near future. 3704 However, some preliminary usage scenarios are described here as 3705 well. 3707 12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 3709 H.323-based video telephony systems that use H.264 as an optional 3710 video compression scheme are required to support H.241 Annex A [3] 3711 as a packetization scheme. The packetization mechanism defined in 3712 this Annex is technically identical with a small subset of this 3713 specification. 3715 When a system operates according to H.241 Annex A, parameter set 3716 NAL units are sent in-band. Only Single NAL unit packets are used. 3717 Many such systems are not sending IDR pictures regularly, but only 3718 when required by user interaction or by control protocol means; 3719 e.g., when switching between video channels in a Multipoint Control 3720 Unit or for error recovery requested by feedback. 3722 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 3723 Aggregation 3725 The RTP part of this scheme is implemented and tested (though not 3726 the control-protocol part; see below). 3728 In most real-world video telephony applications, picture parameters 3729 such as picture size or optional modes never change during the 3730 lifetime of a connection. Therefore, all necessary parameter sets 3731 (usually only one) are sent as a side effect of the capability 3732 exchange/announcement process, e.g., according to the SDP syntax 3733 specified in section 8.2 of this document. As all necessary 3734 parameter set information is established before the RTP session 3735 starts, there is no need for sending any parameter set NAL units. 3736 Slice data partitioning is not used, either. Thus, the RTP packet 3737 stream basically consists of NAL units that carry single coded 3738 slices. 3740 The encoder chooses the size of coded slice NAL units so that they 3741 offer the best performance. Often, this is done by adapting the 3742 coded slice size to the MTU size of the IP network. For small 3743 picture sizes, this may result in a one-picture-per-one-packet 3744 strategy. Intra refresh algorithms clean up the loss of packets 3745 and the resulting drift-related artifacts. 3747 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 3748 Aggregation 3750 This scheme allows better error concealment and is used in H.263 3751 based designs using RFC 4629 packetization [11]. It has been 3752 implemented, and good results were reported [13]. 3754 The VCL encoder codes the source picture so that all macroblocks 3755 (MBs) of one MB line are assigned to one slice. All slices with 3756 even MB row addresses are combined into one STAP, and all slices 3757 with odd MB row addresses into another. Those STAPs are 3758 transmitted as RTP packets. The establishment of the parameter 3759 sets is performed as discussed above. 3761 Note that the use of STAPs is essential here, as the high number of 3762 individual slices (18 for a CIF picture) would lead to unacceptably 3763 high IP/UDP/RTP header overhead (unless the source coding tool FMO 3764 is used, which is not assumed in this scenario). Furthermore, some 3765 wireless video transmission systems, such as H.324M and the IP- 3766 based video telephony specified in 3GPP, are likely to use 3767 relatively small transport packet size. For example, a typical MTU 3768 size of H.223 AL3 SDU is around 100 bytes [17]. Coding individual 3769 slices according to this packetization scheme provides further 3770 advantage in communication between wired and wireless networks, as 3771 individual slices are likely to be smaller than the preferred 3772 maximum packet size of wireless systems. Consequently, a gateway 3773 can convert the STAPs used in a wired network into several RTP 3774 packets with only one NAL unit, which are preferred in a wireless 3775 network, and vice versa. 3777 12.4. Video Telephony with Data Partitioning 3779 This scheme has been implemented and has been shown to offer good 3780 performance, especially at higher packet loss rates [13]. 3782 Data Partitioning is known to be useful only when some form of 3783 unequal error protection is available. Normally, in single-session 3784 RTP environments, even error characteristics are assumed; i.e., the 3785 packet loss probability of all packets of the session is the same 3786 statistically. However, there are means to reduce the packet loss 3787 probability of individual packets in an RTP session. A FEC packet 3788 according to RFC 2733 [18], for example, specifies which media 3789 packets are associated with the FEC packet. 3791 In all cases, the incurred overhead is substantial but is in the 3792 same order of magnitude as the number of bits that have otherwise 3793 been spent for intra information. However, this mechanism does not 3794 add any delay to the system. 3796 Again, the complete parameter set establishment is performed 3797 through control protocol means. 3799 12.5. Video Telephony or Streaming with FUs and Forward Error 3800 Correction 3802 This scheme has been implemented and has been shown to provide good 3803 performance, especially at higher packet loss rates [19]. 3805 The most efficient means to combat packet losses for scenarios 3806 where retransmissions are not applicable is forward error 3807 correction (FEC). Although application layer, end-to-end use of 3808 FEC is often less efficient than an FEC-based protection of 3809 individual links (especially when links of different 3810 characteristics are in the transmission path), application layer, 3811 end-to-end FEC is unavoidable in some scenarios. RFC 5109 [18] 3812 provides means to use generic, application layer, end-to-end FEC in 3813 packet-loss environments. A binary forward error correcting code 3814 is generated by applying the XOR operation to the bits at the same 3815 bit position in different packets. The binary code can be 3816 specified by the parameters (n,k) in which k is the number of 3817 information packets used in the connection and n is the total 3818 number of packets generated for k information packets; i.e., n-k 3819 parity packets are generated for k information packets. 3821 When a code is used with parameters (n,k) within the RFC 5109 3822 framework, the following properties are well known: 3824 a) If applied over one RTP packet, RFC 5109 provides only packet 3825 repetition. 3827 b) RFC 5109 is most bit rate efficient if XOR-connected packets 3828 have equal length. 3830 c) At the same packet loss probability p and for a fixed k, the 3831 greater the value of n is, the smaller the residual error 3832 probability becomes. For example, for a packet loss probability 3833 of 10%, k=1, and n=2, the residual error probability is about 1%, 3834 whereas for n=3, the residual error probability is about 0.1%. 3836 d) At the same packet loss probability p and for a fixed code rate 3837 k/n, the greater the value of n is, the smaller the residual 3838 error probability becomes. For example, at a packet loss 3839 probability of p=10%, k=1 and n=2, the residual error rate is 3840 about 1%, whereas for an extended Golay code with k=12 and n=24, 3841 the residual error rate is about 0.01%. 3843 For applying RFC 5109 in combination with H.264 baseline coded 3844 video without using FUs, several options might be considered: 3846 1) The video encoder produces NAL units for which each video frame 3847 is coded in a single slice. Applying FEC, one could use a 3848 simple code; e.g., (n=2, k=1). That is, each NAL unit would 3849 basically just be repeated. The disadvantage is obviously the 3850 bad code performance according to d), above, and the low 3851 flexibility, as only (n, k=1) codes can be used. 3853 2) The video encoder produces NAL units for which each video frame 3854 is encoded in one or more consecutive slices. Applying FEC, one 3855 could use a better code, e.g., (n=24, k=12), over a sequence of 3856 NAL units. Depending on the number of RTP packets per frame, a 3857 loss may introduce a significant delay, which is reduced when 3858 more RTP packets are used per frame. Packets of completely 3859 different length might also be connected, which decreases bit 3860 rate efficiency according to b), above. However, with some care 3861 and for slices of 1kb or larger, similar length (100-200 bytes 3862 difference) may be produced, which will not lower the bit 3863 efficiency catastrophically. 3865 3) The video encoder produces NAL units, for which a certain frame 3866 contains k slices of possibly almost equal length. Then, 3867 applying FEC, a better code, e.g., (n=24, k=12), can be used 3868 over the sequence of NAL units for each frame. The delay 3869 compared to that of 2), above, may be reduced, but several 3870 disadvantages are obvious. First, the coding efficiency of the 3871 encoded video is lowered significantly, as slice-structured 3872 coding reduces intra-frame prediction and additional slice 3873 overhead is necessary. Second, pre-encoded content or, when 3874 operating over a gateway, the video is usually not appropriately 3875 coded with k slices such that FEC can be applied. Finally, the 3876 encoding of video producing k slices of equal length is not 3877 straightforward and might require more than one encoding pass. 3879 Many of the mentioned disadvantages can be avoided by applying FUs 3880 in combination with FEC. Each NAL unit can be split into any 3881 number of FUs of basically equal length; therefore, FEC with a 3882 reasonable k and n can be applied, even if the encoder made no 3883 effort to produce slices of equal length. For example, a coded 3884 slice NAL unit containing an entire frame can be split to k FUs, 3885 and a parity check code (n=k+1, k) can be applied. However, this 3886 has the disadvantage that unless all created fragments can be 3887 recovered, the whole slice will be lost. Thus a larger section is 3888 lost than would be if the frame had been split into several slices. 3890 The presented technique makes it possible to achieve good 3891 transmission error tolerance, even if no additional source coding 3892 layer redundancy (such as periodic intra frames) is present. 3893 Consequently, the same coded video sequence can be used to achieve 3894 the maximum compression efficiency and quality over error-free 3895 transmission and for transmission over error-prone networks. 3896 Furthermore, the technique allows the application of FEC to pre- 3897 encoded sequences without adding delay. In this case, pre-encoded 3898 sequences that are not encoded for error-prone networks can still 3899 be transmitted almost reliably without adding extensive delays. In 3900 addition, FUs of equal length result in a bit rate efficient use of 3901 RFC 5109. 3903 If the error probability depends on the length of the transmitted 3904 packet (e.g., in case of mobile transmission [15]), the benefits of 3905 applying FUs with FEC are even more obvious. Basically, the 3906 flexibility of the size of FUs allows appropriate FEC to be applied 3907 for each NAL unit and unequal error protection of NAL units. 3909 When FUs and FEC are used, the incurred overhead is substantial but 3910 is in the same order of magnitude as the number of bits that have 3911 to be spent for intra-coded macroblocks if no FEC is applied. In 3912 [19], it was shown that the overall performance of the FEC-based 3913 approach enhanced quality when using the same error rate and same 3914 overall bit rate, including the overhead. 3916 12.6. Low Bit-Rate Streaming 3918 This scheme has been implemented with H.263 and non-standard RTP 3919 packetization and has given good results [20]. There is no 3920 technical reason why similarly good results could not be achievable 3921 with H.264. 3923 In today's Internet streaming, some of the offered bit rates are 3924 relatively low in order to allow terminals with dial-up modems to 3925 access the content. In wired IP networks, relatively large packets, 3926 say 500 - 1500 bytes, are preferred to smaller and more frequently 3927 occurring packets in order to reduce network congestion. Moreover, 3928 use of large packets decreases the amount of RTP/UDP/IP header 3929 overhead. For low bit-rate video, the use of large packets means 3930 that sometimes up to few pictures should be encapsulated in one 3931 packet. 3933 However, loss of a packet including many coded pictures would have 3934 drastic consequences for visual quality, as there is practically no 3935 other way to conceal a loss of an entire picture than to repeat the 3936 previous one. One way to construct relatively large packets and 3937 maintain possibilities for successful loss concealment is to 3938 construct MTAPs that contain interleaved slices from several 3939 pictures. An MTAP should not contain spatially adjacent slices 3940 from the same picture or spatially overlapping slices from any 3941 picture. If a packet is lost, it is likely that a lost slice is 3942 surrounded by spatially adjacent slices of the same picture and 3943 spatially corresponding slices of the temporally previous and 3944 succeeding pictures. Consequently, concealment of the lost slice 3945 is likely to be relatively successful. 3947 12.7. Robust Packet Scheduling in Video Streaming 3949 Robust packet scheduling has been implemented with MPEG-4 Part 2 3950 and simulated in a wireless streaming environment [21]. There is 3951 no technical reason why similar or better results could not be 3952 achievable with H.264. 3954 Streaming clients typically have a receiver buffer that is capable 3955 of storing a relatively large amount of data. Initially, when a 3956 streaming session is established, a client does not start playing 3957 the stream back immediately. Rather, it typically buffers the 3958 incoming data for a few seconds. This buffering helps maintain 3959 continuous playback, as, in case of occasional increased 3960 transmission delays or network throughput drops, the client can 3961 decode and play buffered data. Otherwise, without initial 3962 buffering, the client has to freeze the display, stop decoding, and 3963 wait for incoming data. The buffering is also necessary for either 3964 automatic or selective retransmission in any protocol level. If 3965 any part of a picture is lost, a retransmission mechanism may be 3966 used to resend the lost data. If the retransmitted data is 3967 received before its scheduled decoding or playback time, the loss 3968 is recovered perfectly. Coded pictures can be ranked according to 3969 their importance in the subjective quality of the decoded sequence. 3970 For example, non-reference pictures, such as conventional B 3971 pictures, are subjectively least important, as their absence does 3972 not affect decoding of any other pictures. In addition to non- 3973 reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard 3974 includes a temporal scalability method called sub-sequences [22]. 3975 Subjective ranking can also be made on coded slice data partition 3976 or slice group basis. Coded slices and coded slice data partitions 3977 that are subjectively the most important can be sent earlier than 3978 their decoding order indicates, whereas coded slices and coded 3979 slice data partitions that are subjectively the least important can 3980 be sent later than their natural coding order indicates. 3981 Consequently, any retransmitted parts of the most important slices 3982 and coded slice data partitions are more likely to be received 3983 before their scheduled decoding or playback time compared to the 3984 least important slices and slice data partitions. 3986 13. Informative Appendix: Rationale for Decoding Order Number 3988 13.1. Introduction 3990 The Decoding Order Number (DON) concept was introduced mainly to 3991 enable efficient multi-picture slice interleaving (see section 12.6) 3992 and robust packet scheduling (see section 12.7). In both of these 3993 applications, NAL units are transmitted out of decoding order. DON 3994 indicates the decoding order of NAL units and should be used in the 3995 receiver to recover the decoding order. Example use cases for 3996 efficient multi-picture slice interleaving and for robust packet 3997 scheduling are given in sections 13.2 and 13.3, respectively. 3998 Section 13.4 describes the benefits of the DON concept in error 3999 resiliency achieved by redundant coded pictures. Section 13.5 4000 summarizes considered alternatives to DON and justifies why DON was 4001 chosen to this RTP payload specification. 4003 13.2. Example of Multi-Picture Slice Interleaving 4005 An example of multi-picture slice interleaving follows. A subset 4006 of a coded video sequence is depicted below in output order. R 4007 denotes a reference picture, N denotes a non-reference picture, and 4008 the number indicates a relative output time. 4010 ... R1 N2 R3 N4 R5 ... 4012 The decoding order of these pictures from left to right is as 4013 follows: 4015 ... R1 R3 N2 R5 N4 ... 4017 The NAL units of pictures R1, R3, N2, R5, and N4 are marked with a 4018 DON equal to 1, 2, 3, 4, and 5, respectively. 4020 Each reference picture consists of three slice groups that are 4021 scattered as follows (a number denotes the slice group number for 4022 each macroblock in a QCIF frame): 4024 0 1 2 0 1 2 0 1 2 0 1 4025 2 0 1 2 0 1 2 0 1 2 0 4026 1 2 0 1 2 0 1 2 0 1 2 4027 0 1 2 0 1 2 0 1 2 0 1 4028 2 0 1 2 0 1 2 0 1 2 0 4029 1 2 0 1 2 0 1 2 0 1 2 4030 0 1 2 0 1 2 0 1 2 0 1 4031 2 0 1 2 0 1 2 0 1 2 0 4032 1 2 0 1 2 0 1 2 0 1 2 4034 For the sake of simplicity, we assume that all the macroblocks of a 4035 slice group are included in one slice. Three MTAPs are constructed 4036 from three consecutive reference pictures so that each MTAP 4037 contains three aggregation units, each of which contains all the 4038 macroblocks from one slice group. The first MTAP contains slice 4039 group 0 of picture R1, slice group 1 of picture R3, and slice group 4040 2 of picture R5. The second MTAP contains slice group 1 of picture 4041 R1, slice group 2 of picture R3, and slice group 0 of picture R5. 4042 The third MTAP contains slice group 2 of picture R1, slice group 0 4043 of picture R3, and slice group 1 of picture R5. Each non-reference 4044 picture is encapsulated into an STAP-B. 4046 Consequently, the transmission order of NAL units is the following: 4048 R1, slice group 0, DON 1, carried in MTAP,RTP SN: N 4049 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N 4050 R5, slice group 2, DON 4, carried in MTAP,RTP SN: N 4051 R1, slice group 1, DON 1, carried in MTAP,RTP SN: N+1 4052 R3, slice group 2, DON 2, carried in MTAP,RTP SN: N+1 4053 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+1 4054 R1, slice group 2, DON 1, carried in MTAP,RTP SN: N+2 4055 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N+2 4056 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+2 4057 N2, DON 3, carried in STAP-B, RTP SN: N+3 4058 N4, DON 5, carried in STAP-B, RTP SN: N+4 4060 The receiver is able to organize the NAL units back in decoding 4061 order based on the value of DON associated with each NAL unit. 4063 If one of the MTAPs is lost, the spatially adjacent and temporally 4064 co-located macroblocks are received and can be used to conceal the 4065 loss efficiently. If one of the STAPs is lost, the effect of the 4066 loss does not propagate temporally. 4068 13.3. Example of Robust Packet Scheduling 4070 An example of robust packet scheduling follows. The communication 4071 system used in the example consists of the following components in 4072 the order that the video is processed from source to sink: 4074 o camera and capturing 4075 o pre-encoding buffer 4076 o encoder 4077 o encoded picture buffer 4078 o transmitter 4079 o transmission channel 4080 o receiver 4081 o receiver buffer 4082 o decoder 4083 o decoded picture buffer 4084 o display 4086 The video communication system used in the example operates as 4087 follows. Note that processing of the video stream happens 4088 gradually and at the same time in all components of the system. 4089 The source video sequence is shot and captured to a pre-encoding 4090 buffer. The pre-encoding buffer can be used to order pictures from 4091 sampling order to encoding order or to analyze multiple 4092 uncompressed frames for bit rate control purposes, for example. In 4093 some cases, the pre-encoding buffer may not exist; instead, the 4094 sampled pictures are encoded right away. The encoder encodes 4095 pictures from the pre-encoding buffer and stores the output; i.e., 4096 coded pictures, to the encoded picture buffer. The transmitter 4097 encapsulates the coded pictures from the encoded picture buffer to 4098 transmission packets and sends them to a receiver through a 4099 transmission channel. The receiver stores the received packets to 4100 the receiver buffer. The receiver buffering process typically 4101 includes buffering for transmission delay jitter. The receiver 4102 buffer can also be used to recover correct decoding order of coded 4103 data. The decoder reads coded data from the receiver buffer and 4104 produces decoded pictures as output into the decoded picture buffer. 4105 The decoded picture buffer is used to recover the output (or 4106 display) order of pictures. Finally, pictures are displayed. 4108 In the following example figures, I denotes an IDR picture, R 4109 denotes a reference picture, N denotes a non-reference picture, and 4110 the number after I, R, or N indicates the sampling time relative to 4111 the previous IDR picture in decoding order. Values below the 4112 sequence of pictures indicate scaled system clock timestamps. The 4113 system clock is initialized arbitrarily in this example, and time 4114 runs from left to right. Each I, R, and N picture is mapped into 4115 the same timeline compared to the previous processing step, if any, 4116 assuming that encoding, transmission, and decoding take no time. 4117 Thus, events happening at the same time are located in the same 4118 column throughout all example figures. 4120 A subset of a sequence of coded pictures is depicted below in 4121 sampling order. 4123 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ... 4124 ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ... 4125 ... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ... 4127 Figure 16 Sequence of pictures in sampling order 4129 The sampled pictures are buffered in the pre-encoding buffer to 4130 arrange them in encoding order. In this example, we assume that 4131 the non-reference pictures are predicted from both the previous and 4132 the next reference picture in output order, except for the non- 4133 reference pictures immediately preceding an IDR picture, which are 4134 predicted only from the previous reference picture in output order. 4135 Thus, the pre-encoding buffer has to contain at least two pictures, 4136 and the buffering causes a delay of two picture intervals. The 4137 output of the pre-encoding buffering process and the encoding (and 4138 decoding) order of the pictures are as follows: 4140 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4141 ... -|---|---|---|---|---|---|---|---|- ... 4142 ... 60 61 62 63 64 65 66 67 68 ... 4144 Figure 17 Re-ordered pictures in the pre-encoding buffer 4146 The encoder or the transmitter can set the value of DON for each 4147 picture to a value of DON for the previous picture in decoding 4148 order plus one. 4150 For the sake of simplicity, let us assume that: 4152 o the frame rate of the sequence is constant, 4153 o each picture consists of only one slice, 4154 o each slice is encapsulated in a single NAL unit packet, 4155 o there is no transmission delay, and 4156 o pictures are transmitted at constant intervals (that is, 1 / 4157 (frame rate)). 4159 When pictures are transmitted in decoding order, they are received 4160 as follows: 4162 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4163 ... -|---|---|---|---|---|---|---|---|- ... 4164 ... 60 61 62 63 64 65 66 67 68 ... 4166 Figure 18 Received pictures in decoding order 4168 The OPTIONAL sprop-interleaving-depth media type parameter is set 4169 to 0, as the transmission (or reception) order is identical to the 4170 decoding order. 4172 The decoder has to buffer for one picture interval initially in its 4173 decoded picture buffer to organize pictures from decoding order to 4174 output order as depicted below: 4176 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... 4177 ... -|---|---|---|---|---|---|---|---|- ... 4178 ... 61 62 63 64 65 66 67 68 69 ... 4180 Figure 19 Output order 4182 The amount of required initial buffering in the decoded picture 4183 buffer can be signaled in the buffering period SEI message or with 4184 the num_reorder_frames syntax element of H.264 video usability 4185 information. num_reorder_frames indicates the maximum number of 4186 frames, complementary field pairs, or non-paired fields that 4187 precede any frame, complementary field pair, or non-paired field in 4188 the sequence in decoding order and that follow it in output order. 4189 For the sake of simplicity, we assume that num_reorder_frames is 4190 used to indicate the initial buffer in the decoded picture buffer. 4191 In this example, num_reorder_frames is equal to 1. 4193 It can be observed that if the IDR picture I00 is lost during 4194 transmission and a retransmission request is issued when the value 4195 of the system clock is 62, there is one picture interval of time 4196 (until the system clock reaches timestamp 63) to receive the 4197 retransmitted IDR picture I00. 4199 Let us then assume that IDR pictures are transmitted two frame 4200 intervals earlier than their decoding position; i.e., the pictures 4201 are transmitted as follows: 4203 ... I00 N58 N59 R03 N01 N02 R06 N04 N05 ... 4204 ... --|---|---|---|---|---|---|---|---|- ... 4205 ... 62 63 64 65 66 67 68 69 70 ... 4207 Figure 20 Interleaving: Early IDR pictures in sending order 4209 The OPTIONAL sprop-interleaving-depth media type parameter is set 4210 equal to 1 according to its definition. (The value of sprop- 4211 interleaving-depth in this example can be derived as follows: 4212 Picture I00 is the only picture preceding picture N58 or N59 in 4213 transmission order and following it in decoding order. Except for 4214 pictures I00, N58, and N59, the transmission order is the same as 4215 the decoding order of pictures. As a coded picture is encapsulated 4216 into exactly one NAL unit, the value of sprop-interleaving-depth is 4217 equal to the maximum number of pictures preceding any picture in 4218 transmission order and following the picture in decoding order.) 4220 The receiver buffering process contains two pictures at a time 4221 according to the value of the sprop-interleaving-depth parameter 4222 and orders pictures from the reception order to the correct 4223 decoding order based on the value of DON associated with each 4224 picture. The output of the receiver buffering process is as 4225 follows: 4227 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4228 ... -|---|---|---|---|---|---|---|---|- ... 4229 ... 63 64 65 66 67 68 69 70 71 ... 4231 Figure 21 Interleaving: Receiver buffer 4233 Again, an initial buffering delay of one picture interval is needed 4234 to organize pictures from decoding order to output order, as 4235 depicted below: 4237 ... N58 N59 I00 N01 N02 R03 N04 N05 ... 4238 ... -|---|---|---|---|---|---|---|- ... 4239 ... 64 65 66 67 68 69 70 71 ... 4241 Figure 22 Interleaving: Receiver buffer after reordering 4243 Note that the maximum delay that IDR pictures can undergo during 4244 transmission, including possible application, transport, or link 4245 layer retransmission, is equal to three picture intervals. Thus, 4246 the loss resiliency of IDR pictures is improved in systems 4247 supporting retransmission compared to the case in which pictures 4248 were transmitted in their decoding order. 4250 13.4. Robust Transmission Scheduling of Redundant Coded Slices 4252 A redundant coded picture is a coded representation of a picture or 4253 a part of a picture that is not used in the decoding process if the 4254 corresponding primary coded picture is correctly decoded. There 4255 should be no noticeable difference between any area of the decoded 4256 primary picture and a corresponding area that would result from 4257 application of the H.264 decoding process for any redundant picture 4258 in the same access unit. A redundant coded slice is a coded slice 4259 that is a part of a redundant coded picture. 4261 Redundant coded pictures can be used to provide unequal error 4262 protection in error-prone video transmission. If a primary coded 4263 representation of a picture is decoded incorrectly, a corresponding 4264 redundant coded picture can be decoded. Examples of applications 4265 and coding techniques using the redundant codec picture feature 4266 include the video redundancy coding [23] and the protection of "key 4267 pictures" in multicast streaming [24]. 4269 One property of many error-prone video communications systems is 4270 that transmission errors are often bursty. Therefore, they may 4271 affect more than one consecutive transmission packets in 4272 transmission order. In low bit-rate video communication, it is 4273 relatively common that an entire coded picture can be encapsulated 4274 into one transmission packet. Consequently, a primary coded 4275 picture and the corresponding redundant coded pictures may be 4276 transmitted in consecutive packets in transmission order. To make 4277 the transmission scheme more tolerant of bursty transmission errors, 4278 it is beneficial to transmit the primary coded picture and 4279 redundant coded picture separated by more than a single packet. 4280 The DON concept enables this. 4282 13.5. Remarks on Other Design Possibilities 4284 The slice header syntax structure of the H.264 coding standard 4285 contains the frame_num syntax element that can indicate the 4286 decoding order of coded frames. However, the usage of the 4287 frame_num syntax element is not feasible or desirable to recover 4288 the decoding order, due to the following reasons: 4290 o The receiver is required to parse at least one slice header per 4291 coded picture (before passing the coded data to the decoder). 4293 o Coded slices from multiple coded video sequences cannot be 4294 interleaved, as the frame number syntax element is reset to 0 in 4295 each IDR picture. 4297 o The coded fields of a complementary field pair share the same 4298 value of the frame_num syntax element. Thus, the decoding order 4299 of the coded fields of a complementary field pair cannot be 4300 recovered based on the frame_num syntax element or any other 4301 syntax element of the H.264 coding syntax. 4303 The RTP payload format for transport of MPEG-4 elementary streams 4304 [25] enables interleaving of access units and transmission of 4305 multiple access units in the same RTP packet. An access unit is 4306 specified in the H.264 coding standard to comprise all NAL units 4307 associated with a primary coded picture according to subclause 4308 7.4.1.2 of [1]. Consequently, slices of different pictures cannot 4309 be interleaved, and the multi-picture slice interleaving technique 4310 (see section 12.6) for improved error resilience cannot be used. 4312 14. Backward Compatibility to RFC 3984 4314 The current document is a revision of RFC 3984 and obsoletes it. 4315 This section addresses the backward compatibility issues. 4317 The technical changes are listed in section 15. 4319 Items 1), 2), 3), 7), 9), 10), 12) and 13) are bug-fix type of 4320 changes, and do not incur any backward compatibility issues. 4322 Item 4), addition of six new media type parameters, does not incur 4323 any backward compatibility issues for SDP Offer/Answer based 4324 applications, as legacy RFC 3984 receivers ignore these parameters, 4325 and it is fine for legacy RFC 3984 senders not to use these 4326 parameters as they are optional. However, there is a backward 4327 compatibility issue for SDP declarative usage based applications, 4328 e.g. those using RTSP and SAP, because the SDP receiver per RFC 4329 3984 cannot accept a session for which the SDP includes an 4330 unrecognized parameter. Therefore, the RTSP or SAP server may have 4331 to prepare two sets of streams, one for legacy RFC 3984 receivers 4332 and one for receivers according to this memo. 4334 Items 5), 6), and 11) are related to out-of-band transport of 4335 parameter sets. There are following backward compatibility issues. 4337 1) When a legacy sender per RFC 3984 includes parameter sets for a 4338 level different than the default level indicated by profile- 4339 level-id to sprop-parameter-sets, the parameter value of sprop- 4340 parameter-sets is invalid to the receiver per this memo and 4341 therefore the session may be rejected. 4343 2) In SDP Offer/Answer between a legacy offerer per RFC 3984 and an 4344 answerer per this memo, when the answerer includes in the answer 4345 parameter sets that are not a superset of the parameter sets 4346 included in the offer, the parameter value of sprop-parameter- 4347 sets is invalid to the offerer and the session may not be 4348 initiated properly (related to change item 11). 4350 3) When one endpoint A per this memo includes in-band-parameter- 4351 sets equal to 1, the other side B per RFC 3984 does not 4352 understand that it must transmit parameter sets in-band and B 4353 may still exclude parameter sets in the in-band stream it is 4354 sending. Consequently endpoint A cannot decode the stream it 4355 receives. 4357 Item7), allowance of conveying sprop-parameter-sets and sprop- 4358 level-parameter-sets using the "fmtp" source attribute as specified 4359 in section 6.3 of [9], is similar as item 4). It does not incur 4360 any backward compatibility issues for SDP Offer/Answer based 4361 applications, as legacy RFC 3984 receivers ignore the "fmtp" source 4362 attribute, and it is fine for legacy RFC 3984 senders not to use 4363 the "fmtp" source attribute as it is optional. However, there is a 4364 backward compatibility issue for SDP declarative usage based 4365 applications, e.g. those using RTSP and SAP, because the SDP 4366 receiver per RFC 3984 cannot accept a session for which the SDP 4367 includes an unrecognized parameter (i.e., the "fmtp" source 4368 attribute). Therefore, the RTSP or SAP server may have to prepare 4369 two sets of streams, one for legacy RFC 3984 receivers and one for 4370 receivers according to this memo. 4372 Item 14) removed that use of out-of-band transport of parameter 4373 sets is recommended. As out-of-band transport of parameter sets is 4374 still allowed, this change does not incur any backward 4375 compatibility issues. 4377 Item 15) does not incur any backward compatibility issues as the 4378 added subsection 8.5 is informative. 4380 Item 16) does not create any backward compatibility issues as the 4381 handling of default level is the same if either end is RFC 3984 4382 compliant, and furthermore, RFC 3984 compliant ends would simply 4383 ignore the new media type parameters, if present. 4385 15. Changes from RFC 3984 4387 Following is the list of technical changes (including bug fixes) 4388 from RFC 3984. Besides this list of technical changes, numerous 4389 editorial changes have been made, but not documented in this 4390 section. Note that section 8.2.2 is where much of the important 4391 changes in this memo occurs and deserves particular attention. 4393 1) In subsections 5.4, 5.5, 6.2, 6,3 and 6.4, removed that the 4394 packetization mode in use may be signaled by external means. 4396 2) In subsection 7.2.2, changed the sentence 4398 There are N VCL NAL units in the deinterleaving buffer. 4400 to 4402 There are N or more VCL NAL units in the de-interleaving buffer. 4404 3) In subsection 8.1, the semantics of sprop-init-buf-time, 4405 paragraph 2, changed the sentence 4407 The parameter is the maximum value of (transmission time of a 4408 NAL unit - decoding time of the NAL unit), assuming reliable and 4409 instantaneous transmission, the same timeline for transmission 4410 and decoding, and that decoding starts when the first packet 4411 arrives. 4413 to 4415 The parameter is the maximum value of (decoding time of the NAL 4416 unit - transmission time of a NAL unit), assuming reliable and 4417 instantaneous transmission, the same timeline for transmission 4418 and decoding, and that decoding starts when the first packet 4419 arrives. 4421 4) Added media type parameters max-smbps, sprop-level-parameter- 4422 sets, use-level-src-parameter-sets, in-band-parameter-sets, sar- 4423 understood and sar-supported. 4425 5) In subsection 8.1, removed the specification of parameter-add. 4426 Other descriptions of parameter-add (in subsections 8.2 and 8.4) 4427 are also removed. 4429 6) In subsection 8.1, added a constraint to sprop-parameter-sets 4430 such that it can only contain parameter sets for the same 4431 profile and level as indicated by profile-level-id. 4433 7) In subsection 8.2.1, added that sprop-parameter-sets and sprop- 4434 level-parameter-sets may be either included in the "a=fmtp" line 4435 of SDP or conveyed using the "fmtp" source attribute as 4436 specified in section 6.3 of [9]. 4438 8) In subsection 8.2.2, removed sprop-deint-buf-req from being part 4439 of the media format configuration in usage with the SDP 4440 Offer/Answer model. 4442 9) In subsection 8.2.2, made it clear that level is downgradable in 4443 the SDP Offer/Answer model, i.e. the use of the level part of 4444 "profile-level-id" does not need to be symmetric (the level 4445 included in the answer can be lower than or equal to the level 4446 included in the offer). 4448 10)In subsection 8.2.2, removed that the capability parameters may 4449 be used to declare encoding capabilities. 4451 11)In subsection 8.2.2, added rules on how to use sprop-parameter- 4452 sets and sprop-level-parameter-sets for out-of-band transport of 4453 parameter sets, with or without level downgrading. 4455 12)In subsection 8.2.2, clarified the rules of using the media type 4456 parameters with SDP Offer/Answer for multicast. 4458 13)In subsection 8.2.2, completed and corrected the list of how 4459 different media type parameters shall be interpreted in the 4460 different combinations of offer or answer and direction 4461 attribute. 4463 14)In subsection 8.4, changed the text such that both out-of-band 4464 and in-band transport of parameter sets are allowed and neither 4465 is recommended or required. 4467 15)Added subsection 8.5 (informative) providing example methods for 4468 decoder refresh to handle parameter set losses. 4470 16)Added media type parameters max-recv-level, and level-asymmetry- 4471 allowed, and adjusted associated text and examples for level 4472 upgrade and asymmetry. 4474 16. Acknowledgements 4476 Stephan Wenger, Miska Hannuksela, Thomas Stockhammer, Magnus 4477 Westerlund, and David Singer are thanked as the authors of RFC 3984. 4478 Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, 4479 Joerg Ott, and Colin Perkins are thanked for careful review during 4480 the development of RFC 3984. Stephen Botzko, Magnus Westerlund, 4481 Alex Eleftheriadis, Thomas Schierl, Tom Taylor, Ali Begen, Aaron 4482 Wells, Stuart Taylor, and Robert Sparks are thanked for their 4483 valuable comments and inputs during the development of this memo. 4485 This document was prepared using 2-Word-v2.0.template.dot. 4487 17. References 4489 17.1. Normative References 4491 [1] ITU-T Recommendation H.264, "Advanced video coding for 4492 generic audiovisual services", November 2007. 4494 [2] ISO/IEC International Standard 14496-10:2008. 4496 [3] ITU-T Recommendation H.241, "Extended video procedures and 4497 control signals for H.300 series terminals", May 2006. 4499 [4] Bradner, S., "Key words for use in RFCs to Indicate 4500 Requirement Levels", BCP 14, RFC 2119, March 1997. 4502 [5] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, 4503 "RTP: A Transport Protocol for Real-Time Applications", STD 4504 64, RFC 3550, July 2003. 4506 [6] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 4507 Description Protocol", RFC 4566, July 2006. 4509 [7] Josefsson, S., "The Base16, Base32, and Base64 Data 4510 Encodings", RFC 3548, July 2003. 4512 [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with 4513 Session Description Protocol (SDP)", RFC 3264, June 2002. 4515 [9] Lennox, J., Ott, J., and Schierl, T., "Source-Specific Media 4516 Attributes in the Session Description Protocol (SDP)", RFC 4517 5576, June 2009. 4519 17.2. Informative References 4521 [10] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special 4522 Issue on H.264/AVC. IEEE Transactions on Circuits and Systems 4523 on Video Technology, July 2003. 4525 [11] Ott, J., Bormann, C., Sullivan, G., Wenger, S., and R. Even 4526 (Ed.), "RTP Payload Format for ITU-T Rec. H.263 Video", RFC 4527 4629, January 2007. 4529 [12] ISO/IEC IS 14496-2. 4531 [13] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and 4532 Systems for Video technology, Vol. 13, No. 7, July 2003. 4534 [14] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", 4535 Proceedings Packet Video Workshop 02, April 2002. 4537 [15] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT 4538 Coding Network Abstraction Layer and IP-based Transport" in 4539 Proc. ICIP 2002, Rochester, NY, September 2002. 4541 [16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 4542 Video Conferences with Minimal Control", STD 65, RFC 3551, 4543 July 2003. 4545 [17] ITU-T Recommendation H.223, "Multiplexing protocol for low 4546 bit rate multimedia communication", July 2001. 4548 [18] Li, A., "RTP Payload Format for Generic Forward Error 4549 Correction", RFC 5109, December 2007. 4551 [19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, 4552 "Video Coding and Transport Layer Techniques for H.264/AVC- 4553 Based Transmission over Packet-Lossy Networks", IEEE 4554 International Conference on Image Processing (ICIP 2003), 4555 Barcelona, Spain, September 2003. 4557 [20] Varsa, V. and M. Karczewicz, "Slice interleaving in 4558 compressed video packetization", Packet Video Workshop 2000. 4560 [21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for 4561 wireless video streaming," International Packet Video 4562 Workshop 2002. 4564 [22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, 4565 available http://ftp3.itu.int/av-arch/video- 4566 site/0201_Gen/JVT-B042.doc, anuary 2002. 4568 [23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 4569 International Workshop on Audio-Visual Services over Packet 4570 Networks, September 1997. 4572 [24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error 4573 Resilient Video Coding Using Unequally Protected Key 4574 Pictures", in Proc. International Workshop VLBV03, September 4575 2003. 4577 [25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., 4578 and P. Gentric, "RTP Payload Format for Transport of MPEG-4 4579 Elementary Streams", RFC 3640, November 2003. 4581 [26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 4582 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 4583 RFC 3711, March 2004. 4585 [27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time 4586 Streaming Protocol (RTSP)", RFC 2326, April 1998. 4588 [28] Handley, M., Perkins, C., and E. Whelan, "Session 4589 Announcement Protocol", RFC 2974, October 2000. 4591 [29] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, 4592 January 2008. 4594 [30] Wenger, S., Chandra, U., and M. Westerlund, "Codec Control 4595 Messages in the RTP Audio-Visual Profile with Feedback 4596 (AVPF)", RFC 5104, February 2008. 4598 18. Authors' Addresses 4600 Ye-Kui Wang 4601 Huawei Technologies 4602 400 Somerset Corp Blvd, Suite 602 4603 Bridgewater, NJ 08807 4604 USA 4606 Phone: +1-908-541-3518 4607 EMail: yekuiwang@huawei.com 4609 Roni Even 4610 14 David Hamelech 4611 Tel Aviv 64953 4612 Israel 4614 Phone: +972-545481099 4615 Email: ron.even.tlv@gmail.com 4616 Tom Kristensen 4617 TANDBERG 4618 Philip Pedersens vei 22 4619 N-1366 Lysaker 4620 Norway 4622 Phone: +47 67125125 4623 Email: tom.kristensen@tandberg.com, tomkri@ifi.uio.no 4625 Randell Jesup 4626 WorldGate Communications 4627 3190 Tremont Ave 4628 Trevose, PA 19053 4629 USA 4631 Phone: +1-215-354-5166 4632 Email: rjesup@wgate.com