idnits 2.17.1 draft-ietf-avt-rtp-rfc3984bis-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The abstract seems to indicate that this document obsoletes RFC3984, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document seems to contain a disclaimer for pre-RFC5378 work, and may have content which was first submitted before 10 November 2008. The disclaimer is necessary when there are original authors that you have been unable to contact, or if some do not wish to grant the BCP78 rights to the IETF Trust. If you are able to get all authors (current and original) to grant those rights, you can and should remove the disclaimer; otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 30, 2009) is 5468 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' ** Obsolete normative reference: RFC 4566 (ref. '6') (Obsoleted by RFC 8866) ** Obsolete normative reference: RFC 3548 (ref. '7') (Obsoleted by RFC 4648) -- Obsolete informational reference (is this intentional?): RFC 2326 (ref. '27') (Obsoleted by RFC 7826) -- Obsolete informational reference (is this intentional?): RFC 5117 (ref. '29') (Obsoleted by RFC 7667) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Obsoletes RFC 3984 2 Audio/Video Transport WG Y.-K. Wang 3 Internet Draft Huawei Technologies 4 Intended status: Standards track R. Even 5 Expires: October 2009 Self-employed 6 T. Kristensen 7 Tandberg 8 April 30, 2009 10 RTP Payload Format for H.264 Video 11 draft-ietf-avt-rtp-rfc3984bis-06.txt 13 Status of this Memo 15 This Internet-Draft is submitted to IETF in full conformance with 16 the provisions of BCP 78 and BCP 79. This document may contain 17 material from IETF Documents or IETF Contributions published or 18 made publicly available before November 10, 2008. The person(s) 19 controlling the copyright in some of this material may not have 20 granted the IETF Trust the right to allow modifications of such 21 material outside the IETF Standards Process. Without obtaining an 22 adequate license from the person(s) controlling the copyright in 23 such materials, this document may not be modified outside the IETF 24 Standards Process, and derivative works of it may not be created 25 outside the IETF Standards Process, except to format it for 26 publication as an RFC or to translate it into languages other than 27 English. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF), its areas, and its working groups. Note that 31 other groups may also distribute working documents as Internet- 32 Drafts. 34 Internet-Drafts are draft documents valid for a maximum of six 35 months and may be updated, replaced, or obsoleted by other 36 documents at any time. It is inappropriate to use Internet-Drafts 37 as reference material or to cite them other than as "work in 38 progress". 40 The list of current Internet-Drafts can be accessed at 41 http://www.ietf.org/ietf/1id-abstracts.txt. 43 The list of Internet-Draft Shadow Directories can be accessed at 44 http://www.ietf.org/shadow.html. 46 This Internet-Draft will expire on October 30, 2009. 48 Copyright Notice 50 Copyright (c) 2009 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents in effect on the date of 55 publication of this document (http://trustee.ietf.org/license-info). 56 Please review these documents carefully, as they describe your 57 rights and restrictions with respect to this document. 59 Abstract 61 This memo describes an RTP Payload format for the ITU-T 62 Recommendation H.264 video codec and the technically identical 63 ISO/IEC International Standard 14496-10 video codec, excluding the 64 Scalable Video Coding (SVC) extension and the Multivew Video Coding 65 extension, for which the RTP payload formats are defined elsewhere. 66 The RTP payload format allows for packetization of one or more 67 Network Abstraction Layer Units (NALUs), produced by an H.264 video 68 encoder, in each RTP payload. The payload format has wide 69 applicability, as it supports applications from simple low bit-rate 70 conversational usage, to Internet video streaming with interleaved 71 transmission, to high bit-rate video-on-demand. 73 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 74 in section 18. Issues on backward compatibility to RFC 3984 are 75 discussed in section 17. 77 Table of Contents 79 1. Introduction..................................................4 80 1.1. The H.264 Codec..........................................4 81 1.2. Parameter Set Concept....................................6 82 1.3. Network Abstraction Layer Unit Types.....................6 83 2. Conventions...................................................7 84 3. Scope.........................................................8 85 4. Definitions and Abbreviations.................................8 86 4.1. Definitions..............................................8 87 4.2. Abbreviations...........................................10 88 5. RTP Payload Format...........................................11 89 5.1. RTP Header Usage........................................11 90 5.2. Payload Structures......................................13 91 5.3. NAL Unit Header Usage...................................15 92 5.4. Packetization Modes.....................................17 93 5.5. Decoding Order Number (DON).............................18 94 5.6. Single NAL Unit Packet..................................21 95 5.7. Aggregation Packets.....................................22 96 5.7.1. Single-Time Aggregation Packet.....................24 97 5.7.2. Multi-Time Aggregation Packets (MTAPs).............26 98 5.7.3. Fragmentation Units (FUs)..........................30 99 6. Packetization Rules..........................................34 100 6.1. Common Packetization Rules..............................34 101 6.2. Single NAL Unit Mode....................................35 102 6.3. Non-Interleaved Mode....................................35 103 6.4. Interleaved Mode........................................36 104 7. De-Packetization Process.....................................36 105 7.1. Single NAL Unit and Non-Interleaved Mode................36 106 7.2. Interleaved Mode........................................37 107 7.2.1. Size of the De-interleaving Buffer.................37 108 7.2.2. De-interleaving Process............................38 109 7.3. Additional De-Packetization Guidelines..................39 110 8. Payload Format Parameters....................................40 111 8.1. Media Type Registration.................................40 112 8.2. SDP Parameters..........................................58 113 8.2.1. Mapping of Payload Type Parameters to SDP..........58 114 8.2.2. Usage with the SDP Offer/Answer Model..............59 115 8.2.3. Usage in Declarative Session Descriptions..........66 116 8.3. Examples................................................67 117 8.4. Parameter Set Considerations............................74 118 8.5. Decoder Refresh Point Procedure using In-Band Transport of 119 Parameter Sets (Informative).................................76 120 8.5.1. IDR Procedure to Respond to a Request for a Decoder 121 Refresh Point.............................................77 122 8.5.2. Gradual Recovery Procedure to Respond to a Request for 123 a Decoder Refresh Point...................................77 124 9. Security Considerations......................................78 125 10. Congestion Control..........................................79 126 11. IANA Consideration..........................................80 127 12. Informative Appendix: Application Examples..................80 128 12.1. Video Telephony according to ITU-T Recommendation H.241 129 Annex A......................................................80 130 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 131 Aggregation..................................................80 132 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 133 Aggregation..................................................81 134 12.4. Video Telephony with Data Partitioning.................82 135 12.5. Video Telephony or Streaming with FUs and Forward Error 136 Correction...................................................82 137 12.6. Low Bit-Rate Streaming.................................85 138 12.7. Robust Packet Scheduling in Video Streaming............85 139 13. Informative Appendix: Rationale for Decoding Order Number...86 140 13.1. Introduction...........................................86 141 13.2. Example of Multi-Picture Slice Interleaving............86 142 13.3. Example of Robust Packet Scheduling....................88 143 13.4. Robust Transmission Scheduling of Redundant Coded Slices92 144 13.5. Remarks on Other Design Possibilities..................93 145 14. Acknowledgements............................................93 146 15. References..................................................94 147 15.1. Normative References...................................94 148 15.2. Informative References.................................94 149 16. Authors' Addresses..........................................96 150 17. Backward Compatibility to RFC 3984..........................97 151 18. Changes from RFC 3984.......................................98 153 1. Introduction 155 This memo specifies an RTP payload specification for the video 156 coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC 157 International Standard 14496 Part 10 [2] (both also known as 158 Advanced Video Coding, or AVC). In this memo the name H.264 is 159 used for the codec and the standard, but the memo is equally 160 applicable to the ISO/IEC counterpart of the coding standard. 162 This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized 163 in section 18. Issues on backward compatibility to RFC 3984 are 164 discussed in section 17. 166 1.1. The H.264 Codec 168 The H.264 video codec has a very broad application range that 169 covers all forms of digital compressed video, from low bit-rate 170 Internet streaming applications to HDTV broadcast and Digital 171 Cinema applications with nearly lossless coding. Compared to the 172 current state of technology, the overall performance of H.264 is 173 such that bit rate savings of 50% or more are reported. Digital 174 Satellite TV quality, for example, was reported to be achievable at 175 1.5 Mbit/s, compared to the current operation point of MPEG 2 video 176 at around 3.5 Mbit/s [10]. 178 The codec specification [1] itself distinguishes conceptually 179 between a video coding layer (VCL) and a network abstraction layer 180 (NAL). The VCL contains the signal processing functionality of the 181 codec; mechanisms such as transform, quantization, and motion 182 compensated prediction; and a loop filter. It follows the general 183 concept of most of today's video codecs, a macroblock-based coder 184 that uses inter picture prediction with motion compensation and 185 transform coding of the residual signal. The VCL encoder outputs 186 slices: a bit string that contains the macroblock data of an 187 integer number of macroblocks, and the information of the slice 188 header (containing the spatial address of the first macroblock in 189 the slice, the initial quantization parameter, and similar 190 information). Macroblocks in slices are arranged in scan order 191 unless a different macroblock allocation is specified, by using the 192 so-called Flexible Macroblock Ordering syntax. In-picture 193 prediction is used only within a slice. More information is 194 provided in [10]. 196 The Network Abstraction Layer (NAL) encoder encapsulates the slice 197 output of the VCL encoder into Network Abstraction Layer Units (NAL 198 units), which are suitable for transmission over packet networks or 199 use in packet oriented multiplex environments. Annex B of H.264 200 defines an encapsulation process to transmit such NAL units over 201 byte-stream oriented networks. In the scope of this memo, Annex B 202 is not relevant. 204 Internally, the NAL uses NAL units. A NAL unit consists of a one- 205 byte header and the payload byte string. The header indicates the 206 type of the NAL unit, the (potential) presence of bit errors or 207 syntax violations in the NAL unit payload, and information 208 regarding the relative importance of the NAL unit for the decoding 209 process. This RTP payload specification is designed to be unaware 210 of the bit string in the NAL unit payload. 212 One of the main properties of H.264 is the complete decoupling of 213 the transmission time, the decoding time, and the sampling or 214 presentation time of slices and pictures. The decoding process 215 specified in H.264 is unaware of time, and the H.264 syntax does 216 not carry information such as the number of skipped frames (as is 217 common in the form of the Temporal Reference in earlier video 218 compression standards). Also, there are NAL units that affect many 219 pictures and that are, therefore, inherently timeless. For this 220 reason, the handling of the RTP timestamp requires some special 221 considerations for NAL units for which the sampling or presentation 222 time is not defined or, at transmission time, unknown. 224 1.2. Parameter Set Concept 226 One very fundamental design concept of H.264 is to generate self- 227 contained packets, to make mechanisms such as the header 228 duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension 229 Code (HEC) [12] unnecessary. This was achieved by decoupling 230 information relevant to more than one slice from the media stream. 231 This higher layer meta information should be sent reliably, 232 asynchronously, and in advance from the RTP packet stream that 233 contains the slice packets. (Provisions for sending this 234 information in-band are also available for applications that do not 235 have an out-of-band transport channel appropriate for the purpose.) 236 The combination of the higher-level parameters is called a 237 parameter set. The H.264 specification includes two types of 238 parameter sets: sequence parameter set and picture parameter set. 239 An active sequence parameter set remains unchanged throughout a 240 coded video sequence, and an active picture parameter set remains 241 unchanged within a coded picture. The sequence and picture 242 parameter set structures contain information such as picture size, 243 optional coding modes employed, and macroblock to slice group map. 245 To be able to change picture parameters (such as the picture size) 246 without having to transmit parameter set updates synchronously to 247 the slice packet stream, the encoder and decoder can maintain a 248 list of more than one sequence and picture parameter set. Each 249 slice header contains a codeword that indicates the sequence and 250 picture parameter set to be used. 252 This mechanism allows the decoupling of the transmission of 253 parameter sets from the packet stream, and the transmission of them 254 by external means (e.g., as a side effect of the capability 255 exchange), or through a (reliable or unreliable) control protocol. 256 It may even be possible that they are never transmitted but are 257 fixed by an application design specification. 259 1.3. Network Abstraction Layer Unit Types 261 Tutorial information on the NAL design can be found in [13], [14], 262 and [15]. 264 All NAL units consist of a single NAL unit type octet, which also 265 co-serves as the payload header of this RTP payload format. The 266 payload of a NAL unit follows immediately. 268 The syntax and semantics of the NAL unit type octet are specified 269 in [1], but the essential properties of the NAL unit type octet are 270 summarized below. The NAL unit type octet has the following format: 272 +---------------+ 273 |0|1|2|3|4|5|6|7| 274 +-+-+-+-+-+-+-+-+ 275 |F|NRI| Type | 276 +---------------+ 278 The semantics of the components of the NAL unit type octet, as 279 specified in the H.264 specification, are described briefly below. 281 F: 1 bit 282 forbidden_zero_bit. The H.264 specification declares a value of 283 1 as a syntax violation. 285 NRI: 2 bits 286 nal_ref_idc. A value of 00 indicates that the content of the 287 NAL unit is not used to reconstruct reference pictures for inter 288 picture prediction. Such NAL units can be discarded without 289 risking the integrity of the reference pictures. Values greater 290 than 00 indicate that the decoding of the NAL unit is required 291 to maintain the integrity of the reference pictures. 293 Type: 5 bits 294 nal_unit_type. This component specifies the NAL unit payload 295 type as defined in Table 7-1 of [1], and later within this memo. 296 For a reference of all currently defined NAL unit types and 297 their semantics, please refer to section 7.4.1 in [1]. 299 This memo introduces new NAL unit types, which are presented in 300 section 5.2. The NAL unit types defined in this memo are marked as 301 unspecified in [1]. Moreover, this specification extends the 302 semantics of F and NRI as described in section 5.3. 304 2. Conventions 306 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 307 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 308 this document are to be interpreted as described in RFC 2119 [4]. 310 This specification uses the notion of setting and clearing a bit 311 when bit fields are handled. Setting a bit is the same as 312 assigning that bit the value of 1 (On). Clearing a bit is the same 313 as assigning that bit the value of 0 (Off). 315 3. Scope 317 This payload specification can only be used to carry the "naked" 318 H.264 NAL unit stream over RTP, and not the bitstream format 319 discussed in Annex B of H.264. Likely, the first applications of 320 this specification will be in the conversational multimedia field, 321 video telephony or video conferencing, but the payload format also 322 covers other applications, such as Internet streaming and TV over 323 IP. 325 4. Definitions and Abbreviations 327 4.1. Definitions 329 This document uses the definitions of [1]. The following terms, 330 defined in [1], are summed up for convenience: 332 access unit: A set of NAL units always containing a primary 333 coded picture. In addition to the primary coded picture, an 334 access unit may also contain one or more redundant coded 335 pictures or other NAL units not containing slices or slice data 336 partitions of a coded picture. The decoding of an access unit 337 always results in a decoded picture. 339 coded video sequence: A sequence of access units that consists, 340 in decoding order, of an instantaneous decoding refresh (IDR) 341 access unit followed by zero or more non-IDR access units 342 including all subsequent access units up to but not including 343 any subsequent IDR access unit. 345 IDR access unit: An access unit in which the primary coded 346 picture is an IDR picture. 348 IDR picture: A coded picture containing only slices with I or SI 349 slice types that causes a "reset" in the decoding process. 350 After the decoding of an IDR picture, all following coded 351 pictures in decoding order can be decoded without inter 352 prediction from any picture decoded prior to the IDR picture. 354 primary coded picture: The coded representation of a picture to 355 be used by the decoding process for a bitstream conforming to 356 H.264. The primary coded picture contains all macroblocks of 357 the picture. 359 redundant coded picture: A coded representation of a picture or 360 a part of a picture. The content of a redundant coded picture 361 shall not be used by the decoding process for a bitstream 362 conforming to H.264. The content of a redundant coded picture 363 may be used by the decoding process for a bitstream that 364 contains errors or losses. 366 VCL NAL unit: A collective term used to refer to coded slice and 367 coded data partition NAL units. 369 In addition, the following definitions apply: 371 decoding order number (DON): A field in the payload structure or 372 a derived variable indicating NAL unit decoding order. Values 373 of DON are in the range of 0 to 65535, inclusive. After 374 reaching the maximum value, the value of DON wraps around to 0. 376 NAL unit decoding order: A NAL unit order that conforms to the 377 constraints on NAL unit order given in section 7.4.1.2 in [1]. 379 NALU-time: The value that the RTP timestamp would have if the 380 NAL unit would be transported in its own RTP packet. 382 transmission order: The order of packets in ascending RTP 383 sequence number order (in modulo arithmetic). Within an 384 aggregation packet, the NAL unit transmission order is the same 385 as the order of appearance of NAL units in the packet. 387 media aware network element (MANE): A network element, such as a 388 middlebox or application layer gateway that is capable of 389 parsing certain aspects of the RTP payload headers or the RTP 390 payload and reacting to the contents. 392 Informative note: The concept of a MANE goes beyond normal 393 routers or gateways in that a MANE has to be aware of the 394 signaling (e.g., to learn about the payload type mappings of 395 the media streams), and in that it has to be trusted when 396 working with SRTP. The advantage of using MANEs is that they 397 allow packets to be dropped according to the needs of the 398 media coding. For example, if a MANE has to drop packets due 399 to congestion on a certain link, it can identify and remove 400 those packets whose elimination produces the least adverse 401 effect on the user experience. 403 static macroblock: A certain amount of macroblocks in the video 404 stream can be defined as static, as defined in section 8.3.2.8 405 in [3]. Static macroblocks free up additional processing 406 cycles for the handling of non-static macroblocks. Based on a 407 given amount of video processing resources and a given 408 resolution, a higher number of static macroblocks enables a 409 correspondingly higher frame rate. 411 default sub-profile: The subset of coding tools, which may be 412 all coding tools of one profile or the common subset of coding 413 tools of more than one profile, indicated by the profile-level- 414 id parameter. 416 default level: The level indicated by the profile-level-id 417 parameter, which consists of three octets, profile_idc, profile- 418 iop, and level_idc. The default level is indicated by level_idc 419 in most cases, and, in some cases, additionally by profile-iop. 421 4.2. Abbreviations 423 DON: Decoding Order Number 424 DONB: Decoding Order Number Base 425 DOND: Decoding Order Number Difference 426 FEC: Forward Error Correction 427 FU: Fragmentation Unit 428 IDR: Instantaneous Decoding Refresh 429 IEC: International Electrotechnical Commission 430 ISO: International Organization for Standardization 431 ITU-T: International Telecommunication Union, 432 Telecommunication Standardization Sector 433 MANE: Media Aware Network Element 434 MTAP: Multi-Time Aggregation Packet 435 MTAP16: MTAP with 16-bit timestamp offset 436 MTAP24: MTAP with 24-bit timestamp offset 437 NAL: Network Abstraction Layer 438 NALU: NAL Unit 439 SAR: Sample Aspect Ratio 440 SEI: Supplemental Enhancement Information 441 STAP: Single-Time Aggregation Packet 442 STAP-A: STAP type A 443 STAP-B: STAP type B 444 TS: Timestamp 445 VCL: Video Coding Layer 446 VUI: Video Usability Information 448 5. RTP Payload Format 450 5.1. RTP Header Usage 452 The format of the RTP header is specified in RFC 3550 [5] and 453 reprinted in Figure 1 for convenience. This payload format uses 454 the fields of the header in a manner consistent with that 455 specification. 457 When one NAL unit is encapsulated per RTP packet, the RECOMMENDED 458 RTP payload format is specified in section 5.6. The RTP payload 459 (and the settings for some RTP header bits) for aggregation packets 460 and fragmentation units are specified in sections 5.7 and 5.8, 461 respectively. 463 0 1 2 3 464 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 465 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 466 |V=2|P|X| CC |M| PT | sequence number | 467 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 468 | timestamp | 469 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 470 | synchronization source (SSRC) identifier | 471 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 472 | contributing source (CSRC) identifiers | 473 | .... | 474 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 476 Figure 1 RTP header according to RFC 3550 478 The RTP header information to be set according to this RTP payload 479 format is set as follows: 481 Marker bit (M): 1 bit 482 Set for the very last packet of the access unit indicated by the 483 RTP timestamp, in line with the normal use of the M bit in video 484 formats, to allow an efficient playout buffer handling. For 485 aggregation packets (STAP and MTAP), the marker bit in the RTP 486 header MUST be set to the value that the marker bit of the last 487 NAL unit of the aggregation packet would have been if it were 488 transported in its own RTP packet. Decoders MAY use this bit as 489 an early indication of the last packet of an access unit, but 490 MUST NOT rely on this property. 492 Informative note: Only one M bit is associated with an 493 aggregation packet carrying multiple NAL units. Thus, if a 494 gateway has re-packetized an aggregation packet into several 495 packets, it cannot reliably set the M bit of those packets. 497 Payload type (PT): 7 bits 498 The assignment of an RTP payload type for this new packet format 499 is outside the scope of this document and will not be specified 500 here. The assignment of a payload type has to be performed 501 either through the profile used or in a dynamic way. 503 Sequence number (SN): 16 bits 504 Set and used in accordance with RFC 3550. For the single NALU 505 and non-interleaved packetization mode, the sequence number is 506 used to determine decoding order for the NALU. 508 Timestamp: 32 bits 509 The RTP timestamp is set to the sampling timestamp of the 510 content. A 90 kHz clock rate MUST be used. 512 If the NAL unit has no timing properties of its own (e.g., 513 parameter set and SEI NAL units), the RTP timestamp is set to 514 the RTP timestamp of the primary coded picture of the access 515 unit in which the NAL unit is included, according to section 516 7.4.1.2 of [1]. 518 The setting of the RTP Timestamp for MTAPs is defined in section 519 5.7.2. 521 Receivers SHOULD ignore any picture timing SEI messages included 522 in access units that have only one display timestamp. Instead, 523 receivers SHOULD use the RTP timestamp for synchronizing the 524 display process. 526 RTP senders SHOULD NOT transmit picture timing SEI messages for 527 pictures that are not supposed to be displayed as multiple 528 fields. 530 If one access unit has more than one display timestamp carried 531 in a picture timing SEI message, then the information in the SEI 532 message SHOULD be treated as relative to the RTP timestamp, with 533 the earliest event occurring at the time given by the RTP 534 timestamp, and subsequent events later, as given by the 535 difference in SEI message picture timing values. Let tSEI1, 536 tSEI2, ..., tSEIn be the display timestamps carried in the SEI 537 message of an access unit, where tSEI1 is the earliest of all 538 such timestamps. Let tmadjst() be a function that adjusts the 539 SEI messages time scale to a 90-kHz time scale. Let TS be the 540 RTP timestamp. Then, the display time for the event associated 541 with tSEI1 is TS. The display time for the event with tSEIx, 542 where x is [2..n] is TS + tmadjst (tSEIx - tSEI1). 544 Informative note: Displaying coded frames as fields is needed 545 commonly in an operation known as 3:2 pulldown, in which film 546 content that consists of coded frames is displayed on a 547 display using interlaced scanning. The picture timing SEI 548 message enables carriage of multiple timestamps for the same 549 coded picture, and therefore the 3:2 pulldown process is 550 perfectly controlled. The picture timing SEI message 551 mechanism is necessary because only one timestamp per coded 552 frame can be conveyed in the RTP timestamp. 554 Informative note: Because H.264 allows the decoding order to 555 be different from the display order, values of RTP timestamps 556 may not be monotonically non-decreasing as a function of RTP 557 sequence numbers. Furthermore, the value for inter-arrival 558 jitter reported in the RTCP reports may not be a trustworthy 559 indication of the network performance, as the calculation 560 rules for inter-arrival jitter (section 6.4.1 of RFC 3550) 561 assume that the RTP timestamp of a packet is directly 562 proportional to its transmission time. 564 5.2. Payload Structures 566 The payload format defines three different basic payload structures. 567 A receiver can identify the payload structure by the first byte of 568 the RTP packet payload, which co-serves as the RTP payload header 569 and, in some cases, as the first byte of the payload. This byte is 570 always structured as a NAL unit header. The NAL unit type field 571 indicates which structure is present. The possible structures are 572 as follows: 574 Single NAL Unit Packet: Contains only a single NAL unit in the 575 payload. The NAL header type field will be equal to the original 576 NAL unit type; i.e., in the range of 1 to 23, inclusive. Specified 577 in section 5.6. 579 Aggregation Packet: Packet type used to aggregate multiple NAL 580 units into a single RTP payload. This packet exists in four 581 versions, the Single-Time Aggregation Packet type A (STAP-A), the 582 Single-Time Aggregation Packet type B (STAP-B), Multi-Time 583 Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and Multi- 584 Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The 585 NAL unit type numbers assigned for STAP-A, STAP-B, MTAP16, and 586 MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 587 5.7. 589 Fragmentation Unit: Used to fragment a single NAL unit over 590 multiple RTP packets. Exists with two versions, FU-A and FU-B, 591 identified with the NAL unit type numbers 28 and 29, respectively. 592 Specified in section 5.8. 594 Informative note: This specification does not limit the size of 595 NAL units encapsulated in single NAL unit packets and 596 fragmentation units. The maximum size of a NAL unit 597 encapsulated in any aggregation packet is 65535 bytes. 599 Table 1 summarizes NAL unit types and the corresponding RTP packet 600 types when each of these NAL units is directly used as a packet 601 payload, and where the types are described in this memo. 603 Table 1. Summary of NAL unit types and the corresponding packet 604 types 606 NAL Unit Packet Packet Type Name Section 607 Type Type 608 --------------------------------------------------------- 609 0 reserved - 610 1-23 NAL unit Single NAL unit packet 5.6 611 24 STAP-A Single-time aggregation packet 5.7.1 612 25 STAP-B Single-time aggregation packet 5.7.1 613 26 MTAP16 Multi-time aggregation packet 5.7.2 614 27 MTAP24 Multi-time aggregation packet 5.7.2 615 28 FU-A Fragmentation unit 5.8 616 29 FU-B Fragmentation unit 5.8 617 30-31 reserved - 619 5.3. NAL Unit Header Usage 621 The structure and semantics of the NAL unit header were introduced 622 in section 1.3. For convenience, the format of the NAL unit header 623 is reprinted below: 625 +---------------+ 626 |0|1|2|3|4|5|6|7| 627 +-+-+-+-+-+-+-+-+ 628 |F|NRI| Type | 629 +---------------+ 631 This section specifies the semantics of F and NRI according to this 632 specification. 634 F: 1 bit 635 forbidden_zero_bit. A value of 0 indicates that the NAL unit 636 type octet and payload should not contain bit errors or other 637 syntax violations. A value of 1 indicates that the NAL unit 638 type octet and payload may contain bit errors or other syntax 639 violations. 641 MANEs SHOULD set the F bit to indicate detected bit errors in 642 the NAL unit. The H.264 specification requires that the F bit 643 is equal to 0. When the F bit is set, the decoder is advised 644 that bit errors or any other syntax violations may be present in 645 the payload or in the NAL unit type octet. The simplest decoder 646 reaction to a NAL unit in which the F bit is equal to 1 is to 647 discard such a NAL unit and to conceal the lost data in the 648 discarded NAL unit. 650 NRI: 2 bits 651 nal_ref_idc. The semantics of value 00 and a non-zero value 652 remain unchanged from the H.264 specification. In other words, 653 a value of 00 indicates that the content of the NAL unit is not 654 used to reconstruct reference pictures for inter picture 655 prediction. Such NAL units can be discarded without risking the 656 integrity of the reference pictures. Values greater than 00 657 indicate that the decoding of the NAL unit is required to 658 maintain the integrity of the reference pictures. 660 In addition to the specification above, according to this RTP 661 payload specification, values of NRI indicate the relative 662 transport priority, as determined by the encoder. MANEs can use 663 this information to protect more important NAL units better than 664 they do less important NAL units. The highest transport 665 priority is 11, followed by 10, and then by 01; finally, 00 is 666 the lowest. 668 Informative note: Any non-zero value of NRI is handled 669 identically in H.264 decoders. Therefore, receivers need not 670 manipulate the value of NRI when passing NAL units to the 671 decoder. 673 An H.264 encoder MUST set the value of NRI according to the 674 H.264 specification (subclause 7.4.1) when the value of 675 nal_unit_type is in the range of 1 to 12, inclusive. In 676 particular, the H.264 specification requires that the value of 677 NRI SHALL be equal to 0 for all NAL units having nal_unit_type 678 equal to 6, 9, 10, 11, or 12. 680 For NAL units having nal_unit_type equal to 7 or 8 (indicating a 681 sequence parameter set or a picture parameter set, respectively), 682 an H.264 encoder SHOULD set the value of NRI to 11 (in binary 683 format). For coded slice NAL units of a primary coded picture 684 having nal_unit_type equal to 5 (indicating a coded slice 685 belonging to an IDR picture), an H.264 encoder SHOULD set the 686 value of NRI to 11 (in binary format). 688 For a mapping of the remaining nal_unit_types to NRI values, the 689 following example MAY be used and has been shown to be efficient 690 in a certain environment [14]. Other mappings MAY also be 691 desirable, depending on the application and the H.264/AVC Annex 692 A profile in use. 694 Informative note: Data Partitioning is not available in 695 certain profiles; e.g., in the Main or Baseline profiles. 696 Consequently, the NAL unit types 2, 3, and 4 can occur only 697 if the video bitstream conforms to a profile in which data 698 partitioning is allowed and not in streams that conform to 699 the Main or Baseline profiles. 701 Table 2. Example of NRI values for coded slices and coded slice 702 data partitions of primary coded reference pictures 704 NAL Unit Type Content of NAL unit NRI (binary) 705 ---------------------------------------------------------------- 706 1 non-IDR coded slice 10 707 2 Coded slice data partition A 10 708 3 Coded slice data partition B 01 709 4 Coded slice data partition C 01 711 Informative note: As mentioned before, the NRI value of non- 712 reference pictures is 00 as mandated by H.264/AVC. 714 An H.264 encoder SHOULD set the value of NRI for coded slice and 715 coded slice data partition NAL units of redundant coded 716 reference pictures equal to 01 (in binary format). 718 Definitions of the values for NRI for NAL unit types 24 to 29, 719 inclusive, are given in sections 5.7 and 5.8 of this memo. 721 No recommendation for the value of NRI is given for NAL units 722 having nal_unit_type in the range of 13 to 23, inclusive, 723 because these values are reserved for ITU-T and ISO/IEC. No 724 recommendation for the value of NRI is given for NAL units 725 having nal_unit_type equal to 0 or in the range of 30 to 31, 726 inclusive, as the semantics of these values are not specified in 727 this memo. 729 5.4. Packetization Modes 731 This memo specifies three cases of packetization modes: 733 o Single NAL unit mode 735 o Non-interleaved mode 737 o Interleaved mode 739 The single NAL unit mode is targeted for conversational systems 740 that comply with ITU-T Recommendation H.241 [3] (see section 12.1). 741 The non-interleaved mode is targeted for conversational systems 742 that may not comply with ITU-T Recommendation H.241. In the non- 743 interleaved mode, NAL units are transmitted in NAL unit decoding 744 order. The interleaved mode is targeted for systems that do not 745 require very low end-to-end latency. The interleaved mode allows 746 transmission of NAL units out of NAL unit decoding order. 748 The packetization mode in use MAY be signaled by the value of the 749 OPTIONAL packetization-mode media type parameter. The used 750 packetization mode governs which NAL unit types are allowed in RTP 751 payloads. Table 3 summarizes the allowed packet payload types for 752 each packetization mode. Packetization modes are explained in more 753 detail in section 6. 755 Table 3. Summary of allowed NAL unit types for each packetization 756 mode (yes = allowed, no = disallowed, ig = ignore) 758 Payload Packet Single NAL Non-Interleaved Interleaved 759 Type Type Unit Mode Mode Mode 760 ------------------------------------------------------------- 761 0 reserved ig ig ig 762 1-23 NAL unit yes yes no 763 24 STAP-A no yes no 764 25 STAP-B no no yes 765 26 MTAP16 no no yes 766 27 MTAP24 no no yes 767 28 FU-A no yes yes 768 29 FU-B no no yes 769 30-31 reserved ig ig ig 771 Some NAL unit or payload type values (indicated as reserved in 772 Table 3) are reserved for future extensions. NAL units of those 773 types SHOULD NOT be sent by a sender (direct as packet payloads, or 774 as aggregation units in aggregation packets, or as fragmented units 775 in FU packets) and MUST be ignored by a receiver. For example, the 776 payload types 1-23, with the associated packet type "NAL unit", are 777 allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", 778 but disallowed in "Interleaved Mode". However, NAL units of NAL 779 unit types 1-23 can be used in ''Interleaved Mode'' as aggregation 780 units in STAP-B, MTAP16 and MTAP14 packets as well as fragmented 781 units in FU-A and FU-B packets. Similarly, NAL units of NAL unit 782 types 1-23 can also be used in the "Non-Interleaved Mode" as 783 aggregation units in STAP-A packets or fragmented units in FU-A 784 packets, in addition to being directly used as packet payloads. 786 5.5. Decoding Order Number (DON) 788 In the interleaved packetization mode, the transmission order of 789 NAL units is allowed to differ from the decoding order of the NAL 790 units. Decoding order number (DON) is a field in the payload 791 structure or a derived variable that indicates the NAL unit 792 decoding order. Rationale and examples of use cases for 793 transmission out of decoding order and for the use of DON are given 794 in section 13. 796 The coupling of transmission and decoding order is controlled by 797 the OPTIONAL sprop-interleaving-depth media type parameter as 798 follows. When the value of the OPTIONAL sprop-interleaving-depth 799 media type parameter is equal to 0 (explicitly or per default), the 800 transmission order of NAL units MUST conform to the NAL unit 801 decoding order. When the value of the OPTIONAL sprop-interleaving- 802 depth media type parameter is greater than 0, 804 o the order of NAL units in an MTAP16 and an MTAP24 is not 805 required to be the NAL unit decoding order, and 807 o the order of NAL units generated by de-packetizing STAP-Bs, 808 MTAPs, and FUs in two consecutive packets is not required to be 809 the NAL unit decoding order. 811 The RTP payload structures for a single NAL unit packet, an STAP-A, 812 and an FU-A do not include DON. STAP-B and FU-B structures include 813 DON, and the structure of MTAPs enables derivation of DON as 814 specified in section 5.7.2. 816 Informative note: When an FU-A occurs in interleaved mode, it 817 always follows an FU-B, which sets its DON. 819 Informative note: If a transmitter wants to encapsulate a single 820 NAL unit per packet and transmit packets out of their decoding 821 order, STAP-B packet type can be used. 823 In the single NAL unit packetization mode, the transmission order 824 of NAL units, determined by the RTP sequence number, MUST be the 825 same as their NAL unit decoding order. In the non-interleaved 826 packetization mode, the transmission order of NAL units in single 827 NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL 828 unit decoding order. The NAL units within an STAP MUST appear in 829 the NAL unit decoding order. Thus, the decoding order is first 830 provided through the implicit order within a STAP, and second 831 provided through the RTP sequence number for the order between 832 STAPs, FUs, and single NAL unit packets. 834 Signaling of the value of DON for NAL units carried in STAP-B, MTAP, 835 and a series of fragmentation units starting with an FU-B is 836 specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON 837 value of the first NAL unit in transmission order MAY be set to any 838 value. Values of DON are in the range of 0 to 65535, inclusive. 839 After reaching the maximum value, the value of DON wraps around to 840 0. 842 The decoding order of two NAL units contained in any STAP-B, MTAP, 843 or a series of fragmentation units starting with an FU-B is 844 determined as follows. Let DON(i) be the decoding order number of 845 the NAL unit having index i in the transmission order. Function 846 don_diff(m,n) is specified as follows: 848 If DON(m) == DON(n), don_diff(m,n) = 0 850 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 851 don_diff(m,n) = DON(n) - DON(m) 853 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 854 don_diff(m,n) = 65536 - DON(m) + DON(n) 856 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 857 don_diff(m,n) = - (DON(m) + 65536 - DON(n)) 859 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 860 don_diff(m,n) = - (DON(m) - DON(n)) 862 A positive value of don_diff(m,n) indicates that the NAL unit 863 having transmission order index n follows, in decoding order, the 864 NAL unit having transmission order index m. When don_diff(m,n) is 865 equal to 0, then the NAL unit decoding order of the two NAL units 866 can be in either order. A negative value of don_diff(m,n) 867 indicates that the NAL unit having transmission order index n 868 precedes, in decoding order, the NAL unit having transmission order 869 index m. 871 Values of DON related fields (DON, DONB, and DOND; see section 5.7) 872 MUST be such that the decoding order determined by the values of 873 DON, as specified above, conforms to the NAL unit decoding order. 874 If the order of two NAL units in NAL unit decoding order is 875 switched and the new order does not conform to the NAL unit 876 decoding order, the NAL units MUST NOT have the same value of DON. 877 If the order of two consecutive NAL units in the NAL unit stream is 878 switched and the new order still conforms to the NAL unit decoding 879 order, the NAL units MAY have the same value of DON. For example, 880 when arbitrary slice order is allowed by the video coding profile 881 in use, all the coded slice NAL units of a coded picture are 882 allowed to have the same value of DON. Consequently, NAL units 883 having the same value of DON can be decoded in any order, and two 884 NAL units having a different value of DON should be passed to the 885 decoder in the order specified above. When two consecutive NAL 886 units in the NAL unit decoding order have a different value of DON, 887 the value of DON for the second NAL unit in decoding order SHOULD 888 be the value of DON for the first, incremented by one. 890 An example of the de-packetization process to recover the NAL unit 891 decoding order is given in section 7. 893 Informative note: Receivers should not expect that the absolute 894 difference of values of DON for two consecutive NAL units in the 895 NAL unit decoding order will be equal to one, even in error-free 896 transmission. An increment by one is not required, as at the 897 time of associating values of DON to NAL units, it may not be 898 known whether all NAL units are delivered to the receiver. For 899 example, a gateway may not forward coded slice NAL units of non- 900 reference pictures or SEI NAL units when there is a shortage of 901 bit rate in the network to which the packets are forwarded. In 902 another example, a live broadcast is interrupted by pre-encoded 903 content, such as commercials, from time to time. The first 904 intra picture of a pre-encoded clip is transmitted in advance to 905 ensure that it is readily available in the receiver. When 906 transmitting the first intra picture, the originator does not 907 exactly know how many NAL units will be encoded before the first 908 intra picture of the pre-encoded clip follows in decoding order. 909 Thus, the values of DON for the NAL units of the first intra 910 picture of the pre-encoded clip have to be estimated when they 911 are transmitted, and gaps in values of DON may occur. 913 5.6. Single NAL Unit Packet 915 The single NAL unit packet defined here MUST contain only one NAL 916 unit, of the types defined in [1]. This means that neither an 917 aggregation packet nor a fragmentation unit can be used within a 918 single NAL unit packet. A NAL unit stream composed by de- 919 packetizing single NAL unit packets in RTP sequence number order 920 MUST conform to the NAL unit decoding order. The structure of the 921 single NAL unit packet is shown in Figure 2. 923 Informative note: The first byte of a NAL unit co-serves as the 924 RTP payload header. 926 0 1 2 3 927 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 928 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 929 |F|NRI| Type | | 930 +-+-+-+-+-+-+-+-+ | 931 | | 932 | Bytes 2..n of a Single NAL unit | 933 | | 934 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 935 | :...OPTIONAL RTP padding | 936 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 938 Figure 2 RTP payload format for single NAL unit packet 940 5.7. Aggregation Packets 942 Aggregation packets are the NAL unit aggregation scheme of this 943 payload specification. The scheme is introduced to reflect the 944 dramatically different MTU sizes of two key target networks: 945 wireline IP networks (with an MTU size that is often limited by the 946 Ethernet MTU size; roughly 1500 bytes), and IP or non-IP (e.g., 947 ITU-T H.324/M) based wireless communication systems with preferred 948 transmission unit sizes of 254 bytes or less. To prevent media 949 transcoding between the two worlds, and to avoid undesirable 950 packetization overhead, a NAL unit aggregation scheme is introduced. 952 Two types of aggregation packets are defined by this specification: 954 o Single-time aggregation packet (STAP): aggregates NAL units with 955 identical NALU-time. Two types of STAPs are defined, one 956 without DON (STAP-A) and another including DON (STAP-B). 958 o Multi-time aggregation packet (MTAP): aggregates NAL units with 959 potentially differing NALU-time. Two different MTAPs are 960 defined, differing in the length of the NAL unit timestamp 961 offset. 963 Each NAL unit to be carried in an aggregation packet is 964 encapsulated in an aggregation unit. Please see below for the four 965 different aggregation units and their characteristics. 967 The structure of the RTP payload format for aggregation packets is 968 presented in Figure 3. 970 0 1 2 3 971 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 972 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 973 |F|NRI| Type | | 974 +-+-+-+-+-+-+-+-+ | 975 | | 976 | one or more aggregation units | 977 | | 978 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 979 | :...OPTIONAL RTP padding | 980 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 982 Figure 3 RTP payload format for aggregation packets 984 MTAPs and STAPs share the following packetization rules: The RTP 985 timestamp MUST be set to the earliest of the NALU-times of all the 986 NAL units to be aggregated. The type field of the NAL unit type 987 octet MUST be set to the appropriate value, as indicated in Table 4. 988 The F bit MUST be cleared if all F bits of the aggregated NAL units 989 are zero; otherwise, it MUST be set. The value of NRI MUST be the 990 maximum of all the NAL units carried in the aggregation packet. 992 Table 4. Type field for STAPs and MTAPs 994 Type Packet Timestamp offset DON related fields 995 field length (DON, DONB, DOND) 996 (in bits) present 997 -------------------------------------------------------- 998 24 STAP-A 0 no 999 25 STAP-B 0 yes 1000 26 MTAP16 16 yes 1001 27 MTAP24 24 yes 1003 The marker bit in the RTP header is set to the value that the 1004 marker bit of the last NAL unit of the aggregated packet would have 1005 if it were transported in its own RTP packet. 1007 The payload of an aggregation packet consists of one or more 1008 aggregation units. See sections 5.7.1 and 5.7.2 for the four 1009 different types of aggregation units. An aggregation packet can 1010 carry as many aggregation units as necessary; however, the total 1011 amount of data in an aggregation packet obviously MUST fit into an 1012 IP packet, and the size SHOULD be chosen so that the resulting IP 1013 packet is smaller than the MTU size. An aggregation packet MUST 1014 NOT contain fragmentation units specified in section 5.8. 1016 Aggregation packets MUST NOT be nested; i.e., an aggregation packet 1017 MUST NOT contain another aggregation packet. 1019 5.7.1. Single-Time Aggregation Packet 1021 Single-time aggregation packet (STAP) SHOULD be used whenever NAL 1022 units are aggregated that all share the same NALU-time. The 1023 payload of an STAP-A does not include DON and consists of at least 1024 one single-time aggregation unit, as presented in Figure 4. The 1025 payload of an STAP-B consists of a 16-bit unsigned decoding order 1026 number (DON) (in network byte order) followed by at least one 1027 single-time aggregation unit, as presented in Figure 5. 1029 0 1 2 3 1030 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1031 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1032 : | 1033 +-+-+-+-+-+-+-+-+ | 1034 | | 1035 | single-time aggregation units | 1036 | | 1037 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1038 | : 1039 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1041 Figure 4 Payload format for STAP-A 1043 0 1 2 3 1044 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1045 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1046 : decoding order number (DON) | | 1047 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1048 | | 1049 | single-time aggregation units | 1050 | | 1051 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1052 | : 1053 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1055 Figure 5 Payload format for STAP-B 1057 The DON field specifies the value of DON for the first NAL unit in 1058 an STAP-B in transmission order. For each successive NAL unit in 1059 appearance order in an STAP-B, the value of DON is equal to (the 1060 value of DON of the previous NAL unit in the STAP-B + 1) % 65536, 1061 in which '%' stands for the modulo operation. 1063 A single-time aggregation unit consists of 16-bit unsigned size 1064 information (in network byte order) that indicates the size of the 1065 following NAL unit in bytes (excluding these two octets, but 1066 including the NAL unit type octet of the NAL unit), followed by the 1067 NAL unit itself, including its NAL unit type byte. A single-time 1068 aggregation unit is byte aligned within the RTP payload, but it may 1069 not be aligned on a 32-bit word boundary. Figure 6 presents the 1070 structure of the single-time aggregation unit. 1072 0 1 2 3 1073 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 : NAL unit size | | 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1077 | | 1078 | NAL unit | 1079 | | 1080 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | : 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 Figure 6 Structure for single-time aggregation unit 1086 Figure 7 presents an example of an RTP packet that contains an 1087 STAP-A. The STAP contains two single-time aggregation units, 1088 labeled as 1 and 2 in the figure. 1090 0 1 2 3 1091 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1092 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1093 | RTP Header | 1094 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1095 |STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR | 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 | NALU 1 Data | 1098 : : 1099 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1100 | | NALU 2 Size | NALU 2 HDR | 1101 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 | NALU 2 Data | 1103 : : 1104 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1105 | :...OPTIONAL RTP padding | 1106 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1108 Figure 7 An example of an RTP packet including an STAP-A containing 1109 two single-time aggregation units 1111 Figure 8 presents an example of an RTP packet that contains an 1112 STAP-B. The STAP contains two single-time aggregation units, 1113 labeled as 1 and 2 in the figure. 1115 0 1 2 3 1116 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1117 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1118 | RTP Header | 1119 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1120 |STAP-B NAL HDR | DON | NALU 1 Size | 1121 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1122 | NALU 1 Size | NALU 1 HDR | NALU 1 Data | 1123 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1124 : : 1125 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1126 | | NALU 2 Size | NALU 2 HDR | 1127 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1128 | NALU 2 Data | 1129 : : 1130 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1131 | :...OPTIONAL RTP padding | 1132 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1134 Figure 8 An example of an RTP packet including an STAP-B containing 1135 two single-time aggregation units 1137 5.7.2. Multi-Time Aggregation Packets (MTAPs) 1139 The NAL unit payload of MTAPs consists of a 16-bit unsigned 1140 decoding order number base (DONB) (in network byte order) and one 1141 or more multi-time aggregation units, as presented in Figure 9. 1142 DONB MUST contain the value of DON for the first NAL unit in the 1143 NAL unit decoding order among the NAL units of the MTAP. 1145 Informative note: The first NAL unit in the NAL unit decoding 1146 order is not necessarily the first NAL unit in the order in 1147 which the NAL units are encapsulated in an MTAP. 1149 0 1 2 3 1150 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1151 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1152 : decoding order number base | | 1153 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1154 | | 1155 | multi-time aggregation units | 1156 | | 1157 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1158 | : 1159 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1161 Figure 9 NAL unit payload format for MTAPs 1163 Two different multi-time aggregation units are defined in this 1164 specification. Both of them consist of 16 bits unsigned size 1165 information of the following NAL unit (in network byte order), an 1166 8-bit unsigned decoding order number difference (DOND), and n bits 1167 (in network byte order) of timestamp offset (TS offset) for this 1168 NAL unit, whereby n can be 16 or 24. The choice between the 1169 different MTAP types (MTAP16 and MTAP24) is application dependent: 1170 the larger the timestamp offset is, the higher the flexibility of 1171 the MTAP, but the overhead is also higher. 1173 The structure of the multi-time aggregation units for MTAP16 and 1174 MTAP24 are presented in Figures 10 and 11, respectively. The 1175 starting or ending position of an aggregation unit within a packet 1176 is not required to be on a 32-bit word boundary. The DON of the 1177 NAL unit contained in a multi-time aggregation unit is equal to 1178 (DONB + DOND) % 65536, in which % denotes the modulo operation. 1179 This memo does not specify how the NAL units within an MTAP are 1180 ordered, but, in most cases, NAL unit decoding order SHOULD be used. 1182 The timestamp offset field MUST be set to a value equal to the 1183 value of the following formula: If the NALU-time is larger than or 1184 equal to the RTP timestamp of the packet, then the timestamp offset 1185 equals (the NALU-time of the NAL unit - the RTP timestamp of the 1186 packet). If the NALU-time is smaller than the RTP timestamp of the 1187 packet, then the timestamp offset is equal to the NALU-time + (2^32 1188 - the RTP timestamp of the packet). 1190 0 1 2 3 1191 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1192 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1193 : NAL unit size | DOND | TS offset | 1194 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1195 | TS offset | | 1196 +-+-+-+-+-+-+-+-+ NAL unit | 1197 | | 1198 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1199 | : 1200 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1202 Figure 10 Multi-time aggregation unit for MTAP16 1204 0 1 2 3 1205 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1206 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1207 : NAL unit size | DOND | TS offset | 1208 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1209 | TS offset | | 1210 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1211 | NAL unit | 1212 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1213 | : 1214 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1216 Figure 11 Multi-time aggregation unit for MTAP24 1218 For the "earliest" multi-time aggregation unit in an MTAP the 1219 timestamp offset MUST be zero. Hence, the RTP timestamp of the 1220 MTAP itself is identical to the earliest NALU-time. 1222 Informative note: The "earliest" multi-time aggregation unit is 1223 the one that would have the smallest extended RTP timestamp 1224 among all the aggregation units of an MTAP if the NAL units 1225 contained in the aggregation units were encapsulated in single 1226 NAL unit packets. An extended timestamp is a timestamp that has 1227 more than 32 bits and is capable of counting the wraparound of 1228 the timestamp field, thus enabling one to determine the smallest 1229 value if the timestamp wraps. Such an "earliest" aggregation 1230 unit may not be the first one in the order in which the 1231 aggregation units are encapsulated in an MTAP. The "earliest" 1232 NAL unit need not be the same as the first NAL unit in the NAL 1233 unit decoding order either. 1235 Figure 12 presents an example of an RTP packet that contains a 1236 multi-time aggregation packet of type MTAP16 that contains two 1237 multi-time aggregation units, labeled as 1 and 2 in the figure. 1239 0 1 2 3 1240 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1241 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1242 | RTP Header | 1243 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1244 |MTAP16 NAL HDR | decoding order number base | NALU 1 Size | 1245 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1246 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offset | 1247 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1248 | NALU 1 HDR | NALU 1 DATA | 1249 +-+-+-+-+-+-+-+-+ + 1250 : : 1251 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1252 | | NALU 2 SIZE | NALU 2 DOND | 1253 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1254 | NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA | 1255 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1256 : : 1257 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1258 | :...OPTIONAL RTP padding | 1259 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1261 Figure 12 An RTP packet including a multi-time aggregation packet 1262 of type MTAP16 containing two multi-time aggregation units 1264 Figure 13 presents an example of an RTP packet that contains a 1265 multi-time aggregation packet of type MTAP24 that contains two 1266 multi-time aggregation units, labeled as 1 and 2 in the figure. 1268 0 1 2 3 1269 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1270 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1271 | RTP Header | 1272 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1273 |MTAP24 NAL HDR | decoding order number base | NALU 1 Size | 1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 | NALU 1 Size | NALU 1 DOND | NALU 1 TS offs | 1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1277 |NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA | 1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1279 : : 1280 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1281 | | NALU 2 SIZE | NALU 2 DOND | 1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1283 | NALU 2 TS offset | NALU 2 HDR | 1284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1285 | NALU 2 DATA | 1286 : : 1287 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1288 | :...OPTIONAL RTP padding | 1289 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1291 Figure 13 An RTP packet including a multi-time aggregation packet 1292 of type MTAP24 containing two multi-time aggregation units 1294 5.7.3. Fragmentation Units (FUs) 1296 This payload type allows fragmenting a NAL unit into several RTP 1297 packets. Doing so on the application layer instead of relying on 1298 lower layer fragmentation (e.g., by IP) has the following 1299 advantages: 1301 o The payload format is capable of transporting NAL units bigger 1302 than 64 kbytes over an IPv4 network that may be present in pre- 1303 recorded video, particularly in High Definition formats (there 1304 is a limit of the number of slices per picture, which results in 1305 a limit of NAL units per picture, which may result in big NAL 1306 units). 1308 o The fragmentation mechanism allows fragmenting a single NAL unit 1309 and applying generic forward error correction as described in 1310 section 12.5. 1312 Fragmentation is defined only for a single NAL unit and not for any 1313 aggregation packets. A fragment of a NAL unit consists of an 1314 integer number of consecutive octets of that NAL unit. Each octet 1315 of the NAL unit MUST be part of exactly one fragment of that NAL 1316 unit. Fragments of the same NAL unit MUST be sent in consecutive 1317 order with ascending RTP sequence numbers (with no other RTP 1318 packets within the same RTP packet stream being sent between the 1319 first and last fragment). Similarly, a NAL unit MUST be 1320 reassembled in RTP sequence number order. 1322 When a NAL unit is fragmented and conveyed within fragmentation 1323 units (FUs), it is referred to as a fragmented NAL unit. STAPs and 1324 MTAPs MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU 1325 MUST NOT contain another FU. 1327 The RTP timestamp of an RTP packet carrying an FU is set to the 1328 NALU-time of the fragmented NAL unit. 1330 Figure 14 presents the RTP payload format for FU-As. An FU-A 1331 consists of a fragmentation unit indicator of one octet, a 1332 fragmentation unit header of one octet, and a fragmentation unit 1333 payload. 1335 0 1 2 3 1336 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1337 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1338 | FU indicator | FU header | | 1339 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1340 | | 1341 | FU payload | 1342 | | 1343 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1344 | :...OPTIONAL RTP padding | 1345 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1347 Figure 14 RTP payload format for FU-A 1349 Figure 15 presents the RTP payload format for FU-Bs. An FU-B 1350 consists of a fragmentation unit indicator of one octet, a 1351 fragmentation unit header of one octet, a decoding order number 1352 (DON) (in network byte order), and a fragmentation unit payload. 1353 In other words, the structure of FU-B is the same as the structure 1354 of FU-A, except for the additional DON field. 1356 0 1 2 3 1357 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1359 | FU indicator | FU header | DON | 1360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 1361 | | 1362 | FU payload | 1363 | | 1364 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1365 | :...OPTIONAL RTP padding | 1366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1368 Figure 15 RTP payload format for FU-B 1370 NAL unit type FU-B MUST be used in the interleaved packetization 1371 mode for the first fragmentation unit of a fragmented NAL unit. 1372 NAL unit type FU-B MUST NOT be used in any other case. In other 1373 words, in the interleaved packetization mode, each NALU that is 1374 fragmented has an FU-B as the first fragment, followed by one or 1375 more FU-A fragments. 1377 The FU indicator octet has the following format: 1379 +---------------+ 1380 |0|1|2|3|4|5|6|7| 1381 +-+-+-+-+-+-+-+-+ 1382 |F|NRI| Type | 1383 +---------------+ 1385 Values equal to 28 and 29 in the Type field of the FU indicator 1386 octet identify an FU-A and an FU-B, respectively. The use of the F 1387 bit is described in section 5.3. The value of the NRI field MUST 1388 be set according to the value of the NRI field in the fragmented 1389 NAL unit. 1391 The FU header has the following format: 1393 +---------------+ 1394 |0|1|2|3|4|5|6|7| 1395 +-+-+-+-+-+-+-+-+ 1396 |S|E|R| Type | 1397 +---------------+ 1399 S: 1 bit 1400 When set to one, the Start bit indicates the start of a 1401 fragmented NAL unit. When the following FU payload is not the 1402 start of a fragmented NAL unit payload, the Start bit is set to 1403 zero. 1405 E: 1 bit 1406 When set to one, the End bit indicates the end of a fragmented 1407 NAL unit, i.e., the last byte of the payload is also the last 1408 byte of the fragmented NAL unit. When the following FU payload 1409 is not the last fragment of a fragmented NAL unit, the End bit 1410 is set to zero. 1412 R: 1 bit 1413 The Reserved bit MUST be equal to 0 and MUST be ignored by the 1414 receiver. 1416 Type: 5 bits 1417 The NAL unit payload type as defined in Table 7-1 of [1]. 1419 The value of DON in FU-Bs is selected as described in section 5.5. 1421 Informative note: The DON field in FU-Bs allows gateways to 1422 fragment NAL units to FU-Bs without organizing the incoming NAL 1423 units to the NAL unit decoding order. 1425 A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the 1426 Start bit and End bit MUST NOT both be set to one in the same FU 1427 header. 1429 The FU payload consists of fragments of the payload of the 1430 fragmented NAL unit so that if the fragmentation unit payloads of 1431 consecutive FUs are sequentially concatenated, the payload of the 1432 fragmented NAL unit can be reconstructed. The NAL unit type octet 1433 of the fragmented NAL unit is not included as such in the 1434 fragmentation unit payload, but rather the information of the NAL 1435 unit type octet of the fragmented NAL unit is conveyed in F and NRI 1436 fields of the FU indicator octet of the fragmentation unit and in 1437 the type field of the FU header. An FU payload MAY have any number 1438 of octets and MAY be empty. 1440 Informative note: Empty FUs are allowed to reduce the latency of 1441 a certain class of senders in nearly lossless environments. 1442 These senders can be characterized in that they packetize NALU 1443 fragments before the NALU is completely generated and, hence, 1444 before the NALU size is known. If zero-length NALU fragments 1445 were not allowed, the sender would have to generate at least one 1446 bit of data of the following fragment before the current 1447 fragment could be sent. Due to the characteristics of H.264, 1448 where sometimes several macroblocks occupy zero bits, this is 1449 undesirable and can add delay. However, the (potential) use of 1450 zero-length NALU fragments should be carefully weighed against 1451 the increased risk of the loss of at least a part of the NALU 1452 because of the additional packets employed for its transmission. 1454 If a fragmentation unit is lost, the receiver SHOULD discard all 1455 following fragmentation units in transmission order corresponding 1456 to the same fragmented NAL unit. 1458 A receiver in an endpoint or in a MANE MAY aggregate the first n-1 1459 fragments of a NAL unit to an (incomplete) NAL unit, even if 1460 fragment n of that NAL unit is not received. In this case, the 1461 forbidden_zero_bit of the NAL unit MUST be set to one to indicate a 1462 syntax violation. 1464 6. Packetization Rules 1466 The packetization modes are introduced in section 5.2. The 1467 packetization rules common to more than one of the packetization 1468 modes are specified in section 6.1. The packetization rules for 1469 the single NAL unit mode, the non-interleaved mode, and the 1470 interleaved mode are specified in sections 6.2, 6.3, and 6.4, 1471 respectively. 1473 6.1. Common Packetization Rules 1475 All senders MUST enforce the following packetization rules 1476 regardless of the packetization mode in use: 1478 o Coded slice NAL units or coded slice data partition NAL units 1479 belonging to the same coded picture (and thus sharing the same 1480 RTP timestamp value) MAY be sent in any order; however, for 1481 delay-critical systems, they SHOULD be sent in their original 1482 decoding order to minimize the delay. Note that the decoding 1483 order is the order of the NAL units in the bitstream. 1485 o Parameter sets are handled in accordance with the rules and 1486 recommendations given in section 8.4. 1488 o MANEs MUST NOT duplicate any NAL unit except for sequence or 1489 picture parameter set NAL units, as neither this memo nor the 1490 H.264 specification provides means to identify duplicated NAL 1491 units. Sequence and picture parameter set NAL units MAY be 1492 duplicated to make their correct reception more probable, but 1493 any such duplication MUST NOT affect the contents of any active 1494 sequence or picture parameter set. Duplication SHOULD be 1495 performed on the application layer and not by duplicating RTP 1496 packets (with identical sequence numbers). 1498 Senders using the non-interleaved mode and the interleaved mode 1499 MUST enforce the following packetization rule: 1501 o MANEs MAY convert single NAL unit packets into one aggregation 1502 packet, convert an aggregation packet into several single NAL 1503 unit packets, or mix both concepts, in an RTP translator. The 1504 RTP translator SHOULD take into account at least the following 1505 parameters: path MTU size, unequal protection mechanisms (e.g., 1506 through packet-based FEC according to RFC 2733 [18], especially 1507 for sequence and picture parameter set NAL units and coded slice 1508 data partition A NAL units), bearable latency of the system, and 1509 buffering capabilities of the receiver. 1511 Informative note: An RTP translator is required to handle 1512 RTCP as per RFC 3550. 1514 6.2. Single NAL Unit Mode 1516 This mode is in use when the value of the OPTIONAL packetization- 1517 mode media type parameter is equal to 0 or the packetization-mode 1518 is not present. All receivers MUST support this mode. It is 1519 primarily intended for low-delay applications that are compatible 1520 with systems using ITU-T Recommendation H.241 [3] (see section 1521 12.1). Only single NAL unit packets MAY be used in this mode. 1522 STAPs, MTAPs, and FUs MUST NOT be used. The transmission order of 1523 single NAL unit packets MUST comply with the NAL unit decoding 1524 order. 1526 6.3. Non-Interleaved Mode 1528 This mode is in use when the value of the OPTIONAL packetization- 1529 mode media type parameter is equal to 1. This mode SHOULD be 1530 supported. It is primarily intended for low-delay applications. 1531 Only single NAL unit packets, STAP-As, and FU-As MAY be used in 1532 this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. The 1533 transmission order of NAL units MUST comply with the NAL unit 1534 decoding order. 1536 6.4. Interleaved Mode 1538 This mode is in use when the value of the OPTIONAL packetization- 1539 mode media type parameter is equal to 2. Some receivers MAY 1540 support this mode. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. 1541 STAP-As and single NAL unit packets MUST NOT be used. The 1542 transmission order of packets and NAL units is constrained as 1543 specified in section 5.5. 1545 7. De-Packetization Process 1547 The de-packetization process is implementation dependent. 1548 Therefore, the following description should be seen as an example 1549 of a suitable implementation. Other schemes may be used as well as 1550 long as the output for the same input is the same as the process 1551 described below. The same output means that the resulting NAL 1552 units, and their order, are identical. Optimizations relative to 1553 the described algorithms are likely possible. Section 7.1 presents 1554 the de-packetization process for the single NAL unit and non- 1555 interleaved packetization modes, whereas section 7.2 describes the 1556 process for the interleaved mode. Section 7.3 includes additional 1557 de-packetization guidelines for intelligent receivers. 1559 All normal RTP mechanisms related to buffer management apply. In 1560 particular, duplicated or outdated RTP packets (as indicated by the 1561 RTP sequence number and the RTP timestamp) are removed. To 1562 determine the exact time for decoding, factors such as a possible 1563 intentional delay to allow for proper inter-stream synchronization 1564 must be factored in. 1566 7.1. Single NAL Unit and Non-Interleaved Mode 1568 The receiver includes a receiver buffer to compensate for 1569 transmission delay jitter. The receiver stores incoming packets in 1570 reception order into the receiver buffer. Packets are de- 1571 packetized in RTP sequence number order. If a de-packetized packet 1572 is a single NAL unit packet, the NAL unit contained in the packet 1573 is passed directly to the decoder. If a de-packetized packet is an 1574 STAP-A, the NAL units contained in the packet are passed to the 1575 decoder in the order in which they are encapsulated in the packet. 1576 For all the FU-A packets containing fragments of a single NAL unit, 1577 the de-packetized fragments are concatenated in their sending order 1578 to recover the NAL unit, which is then passed to the decoder. 1580 Informative note: If the decoder supports Arbitrary Slice Order, 1581 coded slices of a picture can be passed to the decoder in any 1582 order regardless of their reception and transmission order. 1584 7.2. Interleaved Mode 1586 The general concept behind these de-packetization rules is to 1587 reorder NAL units from transmission order to the NAL unit decoding 1588 order. 1590 The receiver includes a receiver buffer, which is used to 1591 compensate for transmission delay jitter and to reorder NAL units 1592 from transmission order to the NAL unit decoding order. In this 1593 section, the receiver operation is described under the assumption 1594 that there is no transmission delay jitter. To make a difference 1595 from a practical receiver buffer that is also used for compensation 1596 of transmission delay jitter, the receiver buffer is here after 1597 called the de-interleaving buffer in this section. Receivers 1598 SHOULD also prepare for transmission delay jitter; i.e., either 1599 reserve separate buffers for transmission delay jitter buffering 1600 and de-interleaving buffering or use a receiver buffer for both 1601 transmission delay jitter and de-interleaving. Moreover, receivers 1602 SHOULD take transmission delay jitter into account in the buffering 1603 operation; e.g., by additional initial buffering before starting of 1604 decoding and playback. 1606 This section is organized as follows: subsection 7.2.1 presents how 1607 to calculate the size of the de-interleaving buffer. Subsection 1608 7.2.2 specifies the receiver process on how to organize received 1609 NAL units to the NAL unit decoding order. 1611 7.2.1. Size of the De-interleaving Buffer 1613 In either Offer/Answer or declarative SDP usage, the sprop-deint- 1614 buf-req media type parameter signals the requirement for the de- 1615 interleaving buffer size. It is therefore RECOMMENDED to set the 1616 de-interleaving buffer size, in terms of number of bytes, equal to 1617 or greater than the value of sprop-deint-buf-req media type 1618 parameter. 1620 When the SDP Offer/Answer model or any other capability exchange 1621 procedure is used in session setup, the properties of the received 1622 stream SHOULD be such that the receiver capabilities are not 1623 exceeded. In the SDP Offer/Answer model, the receiver can indicate 1624 its capabilities to allocate a de-interleaving buffer with the 1625 deint-buf-cap media type parameter. See section 8.1 for further 1626 information on deint-buf-cap and sprop-deint-buf-req media type 1627 parameters and section 8.2.2 for further information on their use 1628 in the SDP Offer/Answer model. 1630 7.2.2. De-interleaving Process 1632 There are two buffering states in the receiver: initial buffering 1633 and buffering while playing. Initial buffering occurs when the RTP 1634 session is initialized. After initial buffering, decoding and 1635 playback are started, and the buffering-while-playing mode is used. 1637 Regardless of the buffering state, the receiver stores incoming NAL 1638 units, in reception order, in the de-interleaving buffer as follows. 1639 NAL units of aggregation packets are stored in the de-interleaving 1640 buffer individually. The value of DON is calculated and stored for 1641 each NAL unit. 1643 The receiver operation is described below with the help of the 1644 following functions and constants: 1646 o Function AbsDON is specified in section 8.1. 1648 o Function don_diff is specified in section 5.5. 1650 o Constant N is the value of the OPTIONAL sprop-interleaving-depth 1651 media type parameter (see section 8.1) incremented by 1. 1653 Initial buffering lasts until one of the following conditions is 1654 fulfilled: 1656 o There are N or more VCL NAL units in the de-interleaving buffer. 1658 o If sprop-max-don-diff is present, don_diff(m,n) is greater than 1659 the value of sprop-max-don-diff, in which n corresponds to the 1660 NAL unit having the greatest value of AbsDON among the received 1661 NAL units and m corresponds to the NAL unit having the smallest 1662 value of AbsDON among the received NAL units. 1664 o Initial buffering has lasted for the duration equal to or 1665 greater than the value of the OPTIONAL sprop-init-buf-time media 1666 type parameter. 1668 The NAL units to be removed from the de-interleaving buffer are 1669 determined as follows: 1671 o If the de-interleaving buffer contains at least N VCL NAL units, 1672 NAL units are removed from the de-interleaving buffer and passed 1673 to the decoder in the order specified below until the buffer 1674 contains N-1 VCL NAL units. 1676 o If sprop-max-don-diff is present, all NAL units m for which 1677 don_diff(m,n) is greater than sprop-max-don-diff are removed 1678 from the de-interleaving buffer and passed to the decoder in the 1679 order specified below. Herein, n corresponds to the NAL unit 1680 having the greatest value of AbsDON among the NAL units in the 1681 de-interleaving buffer. 1683 The order in which NAL units are passed to the decoder is specified 1684 as follows: 1686 o Let PDON be a variable that is initialized to 0 at the beginning 1687 of the RTP session. 1689 o For each NAL unit associated with a value of DON, a DON distance 1690 is calculated as follows. If the value of DON of the NAL unit 1691 is larger than the value of PDON, the DON distance is equal to 1692 DON - PDON. Otherwise, the DON distance is equal to 65535 - 1693 PDON + DON + 1. 1695 o NAL units are delivered to the decoder in ascending order of DON 1696 distance. If several NAL units share the same value of DON 1697 distance, they can be passed to the decoder in any order. 1699 o When a desired number of NAL units have been passed to the 1700 decoder, the value of PDON is set to the value of DON for the 1701 last NAL unit passed to the decoder. 1703 7.3. Additional De-Packetization Guidelines 1705 The following additional de-packetization rules may be used to 1706 implement an operational H.264 de-packetizer: 1708 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1709 coded slice data partitions A (DPAs). If a lost DPA is detected, 1710 after taking into account possible retransmission and FEC, a 1711 gateway may decide not to send the corresponding coded slice 1712 data partitions B and C, as their information is meaningless for 1713 H.264 decoders. In this way a MANE can reduce network load by 1714 discarding useless packets without parsing a complex bitstream. 1716 o Intelligent RTP receivers (e.g., in gateways) may identify lost 1717 FUs. If a lost FU is found, a gateway may decide not to send 1718 the following FUs of the same fragmented NAL unit, as their 1719 information is meaningless for H.264 decoders. In this way a 1720 MANE can reduce network load by discarding useless packets 1721 without parsing a complex bitstream. 1723 o Intelligent receivers having to discard packets or NALUs should 1724 first discard all packets/NALUs in which the value of the NRI 1725 field of the NAL unit type octet is equal to 0. This will 1726 minimize the impact on user experience and keep the reference 1727 pictures intact. If more packets have to be discarded, then 1728 packets with a numerically lower NRI value should be discarded 1729 before packets with a numerically higher NRI value. However, 1730 discarding any packets with an NRI bigger than 0 very likely 1731 leads to decoder drift and SHOULD be avoided. 1733 8. Payload Format Parameters 1735 This section specifies the parameters that MAY be used to select 1736 optional features of the payload format and certain features of the 1737 bitstream. The parameters are specified here as part of the media 1738 subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. 1739 A mapping of the parameters into the Session Description Protocol 1740 (SDP) [6] is also provided for applications that use SDP. 1741 Equivalent parameters could be defined elsewhere for use with 1742 control protocols that do not use SDP. 1744 Some parameters provide a receiver with the properties of the 1745 stream that will be sent. The names of all these parameters start 1746 with "sprop" for stream properties. Some of these "sprop" 1747 parameters are limited by other payload or codec configuration 1748 parameters. For example, the sprop-parameter-sets parameter is 1749 constrained by the profile-level-id parameter. The media sender 1750 selects all "sprop" parameters rather than the receiver. This 1751 uncommon characteristic of the "sprop" parameters may not be 1752 compatible with some signaling protocol concepts, in which case the 1753 use of these parameters SHOULD be avoided. 1755 8.1. Media Type Registration 1757 The media subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is 1758 allocated from the IETF tree. 1760 The receiver MUST ignore any unspecified parameter. 1762 Media Type name: video 1764 Media subtype name: H264 1766 Required parameters: none 1768 OPTIONAL parameters: 1770 profile-level-id: 1771 A base16 [7] (hexadecimal) representation of the following 1772 three bytes in the sequence parameter set NAL unit specified 1773 in [1]: 1) profile_idc, 2) a byte herein referred to as 1774 profile-iop, composed of the values of constraint_set0_flag, 1775 constraint_set1_flag,constraint_set2_flag, 1776 constraint_set3_flag, and reserved_zero_4bits in bit- 1777 significance order, starting from the most significant bit, 1778 and 3) level_idc. Note that reserved_zero_4bits is required 1779 to be equal to 0 in [1], but other values for it may be 1780 specified in the future by ITU-T or ISO/IEC. 1782 The profile-level-id parameter indicates the default sub- 1783 profile, i.e. the subset of coding tools that may have been 1784 used to generate the stream or that the receiver supports, 1785 and the default level of the stream or the receiver supports. 1787 The default sub-profile is indicated collectively by the 1788 profile_idc byte and some fields in the profile-iop byte. 1789 Depending on the values of the fields in the profile-iop byte, 1790 the default sub-profile may be the set of coding tools 1791 supported by one profile, or a common subset of coding tools 1792 of multiple profiles, as specified in subsection 7.4.2.1.1 of 1793 [1]. The default level is indicated by the level_idc byte, 1794 and, when profile_idc is equal to 66, 77 or 88 (the Baseline, 1795 Main, or Extended profile) and level_idc is equal to 11, 1796 additionally by bit 4 (constraint_set3_flag) of the profile- 1797 iop byte. When profile_idc is equal to 66, 77 or 88 (the 1798 Baseline, Main, or Extended profile) and level_idc is equal 1799 to 11, and bit 4 (constraint_set3_flag) of the profile-iop 1800 byte is equal to 1, the default level is level 1b. 1802 Table 5 lists all profiles defined in Annex A of [1] and, for 1803 each of the profiles, the possible combinations of 1804 profile_idc and profile-iop that represent the same sub- 1805 profile. 1807 Table 5. Combinations of profile_idc and profile-iop 1808 representing the same sub-profile corresponding to the 1809 full set of coding tools supported by one profile. In 1810 the following, x may be either 0 or 1, while the profile 1811 names are indicated as follows. CB: Constrained Baseline 1812 profile, B: Baseline profile, M: Main profile, E: 1813 Extended profile, H: High profile, H10: High 10 profile, 1814 H42: High 4:2:2 profile, H44: High 4:4:4 Predictive 1815 profile, H10I: High 10 Intra profile, H42I: High 4:2:2 1816 Intra profile, H44I: High 4:4:4 Intra profile, and C44I: 1817 CAVLC 4:4:4 Intra profile. 1819 Profile profile_idc profile-iop 1820 (hexadecimal) (binary) 1822 CB 42 (B) x1xx0000 1823 same as: 4D (M) 1xxx0000 1824 same as: 58 (E) 11xx0000 1825 same as: 64 (H), 6E (H10), 1xx00000 1826 7A (H42), or F4 (H44) 1827 B 42 (B) x0xx0000 1828 same as: 58 (E) 10xx0000 1829 M 4D (M) 0x0x0000 1830 same as: 64 (H), 6E (H10), 01000000 1831 7A (H42), or F4 (H44) 1832 E 58 00xx0000 1833 H 64 00000000 1834 H10 6E 00000000 1835 H42 7A 00000000 1836 H44 F4 00000000 1837 H10I 64 00010000 1838 H42I 7A 00010000 1839 H44I F4 00010000 1840 C44I 2C 00010000 1842 For example, in the table above, profile_idc equal to 58 1843 (Extended) with profile-iop equal to 11xx0000 indicates the 1844 same sub-profile corresponding to profile_idc equal to 42 1845 (Baseline) with profile-iop equal to x1xx0000. Note that 1846 other combinations of profile_idc and profile-iop (not listed 1847 in Table 5) may represent a sub-profile equivalent to the 1848 common subset of coding tools for more than one profile. 1849 Note also that a decoder conforming to a certain profile may 1850 be able to decode bitstreams conforming to other profiles. 1851 For example, a decoder conforming to the High 4:4:4 profile 1852 at certain level must be able to decode bitstreams confirming 1853 to the Constrained Baseline, Main, High, High 10 or High 1854 4:2:2 profile at the same or a lower level. 1856 If the profile-level-id parameter is used to indicate 1857 properties of a NAL unit stream, it indicates that, to decode 1858 the stream, the minimum subset of coding tools a decoder has 1859 to support is the default sub-profile, and the lowest level 1860 the decoder has to support is the default level. 1862 If the profile-level-id parameter is used for capability 1863 exchange or session setup procedure, it indicates the subset 1864 of coding tools, which is equal to the default sub-profile, 1865 and the highest level, which is equal to the default level, 1866 that the codec supports. All levels lower than the default 1867 level are also supported by the codec. 1869 Informative note: Capability exchange and session setup 1870 procedures should provide means to list the capabilities 1871 for each supported sub-profile separately. For example, 1872 the one-of-N codec selection procedure of the SDP 1873 Offer/Answer model can be used (section 10.2 of [8]). 1874 The one-of-N codec selection procedure may also be used 1875 to provide different combinations of profile_idc and 1876 profile-iop that represent the same sub-profile. When 1877 there are many different combinations of profile_idc and 1878 profile-iop that represent the same sub-profile, using 1879 the one-of-N codec selection procedure may result into a 1880 fairly large SDP message. Therefore, a receiver should 1881 understand the different equivalent combinations of 1882 profile_idc and profile-iop that represent the same sub- 1883 profile, and be ready to accept an offer using any of the 1884 equivalent combinations. 1886 If no profile-level-id is present, the Baseline Profile 1887 without additional constraints at Level 1 MUST be inferred. 1889 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: 1890 These parameters MAY be used to signal the capabilities of a 1891 receiver implementation. These parameters MUST NOT be used 1892 for any other purpose. The profile-level-id parameter MUST 1893 be present in the same receiver capability description that 1894 contains any of these parameters. The level conveyed in the 1895 value of the profile-level-id parameter MUST be such that the 1896 receiver is fully capable of supporting. max-mbps, max-smbps, 1897 max-fs, max-cpb, max-dpb, and max-br MAY be used to indicate 1898 capabilities of the receiver that extend the required 1899 capabilities of the signaled level, as specified below. 1901 When more than one parameter from the set (max-mbps, max- 1902 smbps , max-fs, max-cpb, max-dpb, max-br) is present, the 1903 receiver MUST support all signaled capabilities 1904 simultaneously. For example, if both max-mbps and max-br are 1905 present, the signaled level with the extension of both the 1906 frame rate and bit rate is supported. That is, the receiver 1907 is able to decode NAL unit streams in which the macroblock 1908 processing rate is up to max-mbps (inclusive), the bit rate 1909 is up to max-br (inclusive), the coded picture buffer size is 1910 derived as specified in the semantics of the max-br parameter 1911 below, and other properties comply with the level specified 1912 in the value of the profile-level-id parameter. 1914 If a receiver can support all the properties of level A, the 1915 level specified in the value of the profile-level-id MUST be 1916 level A (i.e. MUST NOT be lower than level A). In other 1917 words, a sender or receiver MUST NOT signal values of max- 1918 mbps, max-fs, max-cpb, max-dpb, and max-br that taken 1919 together meet the requirements of a higher level compared to 1920 the level specified in the value of the profile-level-id 1921 parameter. 1923 Informative note: When the OPTIONAL media type parameters 1924 are used to signal the properties of a NAL unit stream, 1925 max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br 1926 are not present, and the value of profile-level-id must 1927 always be such that the NAL unit stream complies fully 1928 with the specified profile and level. 1930 max-mbps: The value of max-mbps is an integer indicating the 1931 maximum macroblock processing rate in units of macroblocks 1932 per second. The max-mbps parameter signals that the receiver 1933 is capable of decoding video at a higher rate than is 1934 required by the signaled level conveyed in the value of the 1935 profile-level-id parameter. When max-mbps is signaled, the 1936 receiver MUST be able to decode NAL unit streams that conform 1937 to the signaled level, with the exception that the MaxMBPS 1938 value in Table A-1 of [1] for the signaled level is replaced 1939 with the value of max-mbps. The value of max-mbps MUST be 1940 greater than or equal to the value of MaxMBPS for the level 1941 given in Table A-1 of [1]. Senders MAY use this knowledge to 1942 send pictures of a given size at a higher picture rate than 1943 is indicated in the signaled level. 1945 max-smbps: The value of max-smbps is an integer indicating the 1946 maximum static macroblock processing rate in units of static 1947 macroblocks per second, under the hypothetical assumption 1948 that all macroblocks are static macroblocks. When max-smbps 1949 is signalled the MaxMBPS value in Table A-1 of [1] should be 1950 replaced with the result of the following computation: 1952 o If the parameter max-mbps is signalled, set a variable 1953 MaxMacroblocksPerSecond to the value of max-mbps. 1954 Otherwise, set MaxMacroblocksPerSecond equal to the value 1955 of MaxMBPS for the level in Table A-1 [1]. 1957 o Set a variable P_non-static to the proportion of non- 1958 static macroblocks in picture n. 1960 o Set a variable P_static to the proportion of static 1961 macroblocks in picture n. 1963 o The value of MaxMBPS in Table A-1 of [1] should be 1964 considered by the encoder to be equal to: 1966 MaxMacroblocksPerSecond * max-smbps / ( P_non-static * 1967 max-smbps + P_static * MaxMacroblocksPerSecond) 1969 The encoder should recompute this value for each picture. The 1970 value of max-smbps MUST be greater than the value of MaxMBPS 1971 for the level given in Table A-1 of [1]. Senders MAY use 1972 this knowledge to send pictures of a given size at a higher 1973 picture rate than is indicated in the signalled level. 1975 max-fs: The value of max-fs is an integer indicating the maximum 1976 frame size in units of macroblocks. The max-fs parameter 1977 signals that the receiver is capable of decoding larger 1978 picture sizes than are required by the signaled level 1979 conveyed in the value of the profile-level-id parameter. 1980 When max-fs is signaled, the receiver MUST be able to decode 1981 NAL unit streams that conform to the signaled level, with the 1982 exception that the MaxFS value in Table A-1 of [1] for the 1983 signaled level is replaced with the value of max-fs. The 1984 value of max-fs MUST be greater than or equal to the value of 1985 MaxFS for the level given in Table A-1 of [1]. Senders MAY 1986 use this knowledge to send larger pictures at a 1987 proportionally lower frame rate than is indicated in the 1988 signaled level. 1990 max-cpb: The value of max-cpb is an integer indicating the 1991 maximum coded picture buffer size in units of 1000 bits for 1992 the VCL HRD parameters (see A.3.1 item i of [1]) and in units 1993 of 1200 bits for the NAL HRD parameters (see A.3.1 item j of 1994 [1]). The max-cpb parameter signals that the receiver has 1995 more memory than the minimum amount of coded picture buffer 1996 memory required by the signaled level conveyed in the value 1997 of the profile-level-id parameter. When max-cpb is signaled, 1998 the receiver MUST be able to decode NAL unit streams that 1999 conform to the signaled level, with the exception that the 2000 MaxCPB value in Table A-1 of [1] for the signaled level is 2001 replaced with the value of max-cpb. The value of max-cpb 2002 MUST be greater than or equal to the value of MaxCPB for the 2003 level given in Table A-1 of [1]. Senders MAY use this 2004 knowledge to construct coded video streams with greater 2005 variation of bit rate than can be achieved with the MaxCPB 2006 value in Table A-1 of [1]. 2008 Informative note: The coded picture buffer is used in the 2009 hypothetical reference decoder (Annex C) of H.264. The 2010 use of the hypothetical reference decoder is recommended 2011 in H.264 encoders to verify that the produced bitstream 2012 conforms to the standard and to control the output 2013 bitrate. Thus, the coded picture buffer is conceptually 2014 independent of any other potential buffers in the 2015 receiver, including de-interleaving and de-jitter buffers. 2016 The coded picture buffer need not be implemented in 2017 decoders as specified in Annex C of H.264, but rather 2018 standard-compliant decoders can have any buffering 2019 arrangements provided that they can decode standard- 2020 compliant bitstreams. Thus, in practice, the input 2021 buffer for video decoder can be integrated with de- 2022 interleaving and de-jitter buffers of the receiver. 2024 max-dpb: The value of max-dpb is an integer indicating the 2025 maximum decoded picture buffer size in units of 1024 bytes. 2026 The max-dpb parameter signals that the receiver has more 2027 memory than the minimum amount of decoded picture buffer 2028 memory required by the signaled level conveyed in the value 2029 of the profile-level-id parameter. When max-dpb is signaled, 2030 the receiver MUST be able to decode NAL unit streams that 2031 conform to the signaled level, with the exception that the 2032 MaxDPB value in Table A-1 of [1] for the signaled level is 2033 replaced with the value of max-dpb. Consequently, a receiver 2034 that signals max-dpb MUST be capable of storing the following 2035 number of decoded frames, complementary field pairs, and non- 2036 paired fields in its decoded picture buffer: 2038 Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs * 2039 256 * ChromaFormatFactor ), 16) 2041 PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are 2042 defined in [1]. 2044 The value of max-dpb MUST be greater than or equal to the 2045 value of MaxDPB for the level given in Table A-1 of [1]. 2046 Senders MAY use this knowledge to construct coded video 2047 streams with improved compression. 2049 Informative note: This parameter was added primarily to 2050 complement a similar codepoint in the ITU-T 2051 Recommendation H.245, so as to facilitate signaling 2052 gateway designs. The decoded picture buffer stores 2053 reconstructed samples. There is no relationship between 2054 the size of the decoded picture buffer and the buffers 2055 used in RTP, especially de-interleaving and de-jitter 2056 buffers. 2058 max-br: The value of max-br is an integer indicating the maximum 2059 video bit rate in units of 1000 bits per second for the VCL 2060 HRD parameters (see A.3.1 item i of [1]) and in units of 1200 2061 bits per second for the NAL HRD parameters (see A.3.1 item j 2062 of [1]). 2064 The max-br parameter signals that the video decoder of the 2065 receiver is capable of decoding video at a higher bit rate 2066 than is required by the signaled level conveyed in the value 2067 of the profile-level-id parameter. 2069 When max-br is signaled, the video codec of the receiver MUST 2070 be able to decode NAL unit streams that conform to the 2071 signaled level, conveyed in the profile-level-id parameter, 2072 with the following exceptions in the limits specified by the 2073 level: 2075 o The value of max-br replaces the MaxBR value of the 2076 signaled level (in Table A-1 of [1]). 2078 o When the max-cpb parameter is not present, the result of 2079 the following formula replaces the value of MaxCPB in 2080 Table A-1 of [1]: (MaxCPB of the signaled level) * max-br 2081 / (MaxBR of the signaled level). 2083 For example, if a receiver signals capability for Level 1.2 2084 with max-br equal to 1550, this indicates a maximum video 2085 bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum 2086 video bitrate of 1860 kbits/sec for NAL HRD parameters, and a 2087 CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000). 2089 The value of max-br MUST be greater than or equal to the 2090 value MaxBR for the signaled level given in Table A-1 of [1]. 2092 Senders MAY use this knowledge to send higher bitrate video 2093 as allowed in the level definition of Annex A of H.264, to 2094 achieve improved video quality. 2096 Informative note: This parameter was added primarily to 2097 complement a similar codepoint in the ITU-T 2098 Recommendation H.245, so as to facilitate signaling 2099 gateway designs. No assumption can be made from the 2100 value of this parameter that the network is capable of 2101 handling such bit rates at any given time. In particular, 2102 no conclusion can be drawn that the signaled bit rate is 2103 possible under congestion control constraints. 2105 redundant-pic-cap: 2106 This parameter signals the capabilities of a receiver 2107 implementation. When equal to 0, the parameter indicates 2108 that the receiver makes no attempt to use redundant coded 2109 pictures to correct incorrectly decoded primary coded 2110 pictures. When equal to 0, the receiver is not capable of 2111 using redundant slices; therefore, a sender SHOULD avoid 2112 sending redundant slices to save bandwidth. When equal to 1, 2113 the receiver is capable of decoding any such redundant slice 2114 that covers a corrupted area in a primary decoded picture (at 2115 least partly), and therefore a sender MAY send redundant 2116 slices. When the parameter is not present, then a value of 0 2117 MUST be used for redundant-pic-cap. When present, the value 2118 of redundant-pic-cap MUST be either 0 or 1. 2120 When the profile-level-id parameter is present in the same 2121 signaling as the redundant-pic-cap parameter, and the profile 2122 indicated in profile-level-id is such that it disallows the 2123 use of redundant coded pictures (e.g., Main Profile), the 2124 value of redundant-pic-cap MUST be equal to 0. When a 2125 receiver indicates redundant-pic-cap equal to 0, the received 2126 stream SHOULD NOT contain redundant coded pictures. 2128 Informative note: Even if redundant-pic-cap is equal to 0, 2129 the decoder is able to ignore redundant codec pictures 2130 provided that the decoder supports such a profile 2131 (Baseline, Extended) in which redundant coded pictures 2132 are allowed. 2134 Informative note: Even if redundant-pic-cap is equal to 1, 2135 the receiver may also choose other error concealment 2136 strategies to replace or complement decoding of redundant 2137 slices. 2139 sprop-parameter-sets: 2140 This parameter MAY be used to convey any sequence and picture 2141 parameter set NAL units (herein referred to as the initial 2142 parameter set NAL units) that can be placed in the NAL unit 2143 stream to precede any other NAL units in decoding order. The 2144 parameter MUST NOT be used to indicate codec capability in 2145 any capability exchange procedure. The value of the 2146 parameter is a comma (',') separated list of base64 [7] 2147 representations of parameter set NAL units as specified in 2148 sections 7.3.2.1 and 7.3.2.2 of [1]. Note that the number of 2149 bytes in a parameter set NAL unit is typically less than 10, 2150 but a picture parameter set NAL unit can contain several 2151 hundreds of bytes. 2153 Informative note: When several payload types are offered 2154 in the SDP Offer/Answer model, each with its own sprop- 2155 parameter-sets parameter, then the receiver cannot assume 2156 that those parameter sets do not use conflicting storage 2157 locations (i.e., identical values of parameter set 2158 identifiers). Therefore, a receiver should buffer all 2159 sprop-parameter-sets and make them available to the 2160 decoder instance that decodes a certain payload type. 2162 The "sprop-parameter-sets" parameter MUST only contain 2163 parameter sets that are conforming to the profile-level-id, 2164 i.e., the subset of coding tools indicated by any of the 2165 parameter sets MUST be equal to the default sub-profile, and 2166 the level indicated by any of the parameter sets MUST be 2167 equal to the default level. 2169 sprop-level-parameter-sets: 2170 This parameter MAY be used to convey any sequence and picture 2171 parameter set NAL units (herein referred to as the initial 2172 parameter set NAL units) that can be placed in the NAL unit 2173 stream to precede any other NAL units in decoding order and 2174 that are associated with one or more levels lower than the 2175 default level. The parameter MUST NOT be used to indicate 2176 codec capability in any capability exchange procedure. 2178 The sprop-level-parameter-sets parameter contains parameter 2179 sets for one or more levels which are lower than the default 2180 level. All parameter sets associated with one level are 2181 clustered and prefixed with a three-byte field which has the 2182 same syntax as profile-level-id. This enables the receiver 2183 to install the parameter sets for one level and discard the 2184 rest. The three-byte field is named PLId, and all parameter 2185 sets associated with one level are named PSL, which has the 2186 same syntax as sprop-parameter-sets. Parameter sets for each 2187 level are represented in the form of PLId:PSL, i.e., PLId 2188 followed by a colon (':') and the base64 [7] representation 2189 of the initial parameter set NAL units for the level. Each 2190 pair of PLId:PSL is also separated by a colon. Note that a 2191 PSL can contain multiple parameter sets for that level, 2192 separated with commas (','). 2194 The subset of coding tools indicated by each PLId field MUST 2195 be equal to the default sub-profile, and the level indicated 2196 by each PLId field MUST be lower than the default level. All 2197 sequence parameter sets contained in each PSL MUST have the 2198 three bytes from profile_idc to level_idc, inclusive, equal 2199 to the preceding PLId. 2201 Informative note: This parameter allows for efficient 2202 level downgrade in SDP Offer/Answer and out-of-band 2203 transport of parameter sets, simultaneously. 2205 use-level-src-parameter-sets: 2206 This parameter MAY be used to indicate a receiver capability. 2207 The value MAY be equal to either 0 or 1. When the parameter 2208 is not present, the value MUST be inferred to be equal to 0. 2209 The value 0 indicates that the receiver does not understand 2210 the sprop-level-parameter-sets parameter, and does not 2211 understand the "fmtp" source attribute as specified in 2212 section 6.3 of [9], and will ignore sprop-level-parameter- 2213 sets when present, and will ignore sprop-parameter-sets when 2214 conveyed using the "fmtp" source attribute. The value 1 2215 indicates that the receiver understands the sprop-level- 2216 parameter-sets parameter, and understands the "fmtp" source 2217 attribute as specified in section 6.3 of [9], and is capable 2218 of using parameter sets contained in the sprop-level- 2219 parameter-sets or contained in the sprop-parameter-sets that 2220 is conveyed using the "fmtp" source attribute. 2222 Informative note: An RFC 3984 receiver does not 2223 understand sprop-level-parameter-sets, use-level-src- 2224 parameter-sets, or the "fmtp" source attribute as 2225 specified in section 6.3 of [9]. Therefore, during SDP 2226 Offer/Answer, an RFC 3984 receiver as the answerer will 2227 simply ignore sprop-level-parameter-sets, when present in 2228 an offer, and sprop-parameter-sets, when conveyed using 2229 the "fmtp" source attribute as specified in section 6.3 2230 of [9]. Assume that the offered payload type was 2231 accepted at a level lower than the default level. If the 2232 offered payload type included sprop-level-parameter-sets 2233 or included sprop-parameter-sets conveyed using the 2234 "fmtp" source attribute, and the offerer sees that the 2235 answerer has not included use-level-src-parameter-sets 2236 equal to 1 in the answer, the offerer gets to know that 2237 in-band transport of parameter sets is needed. 2239 in-band-parameter-sets: 2240 This parameter MAY be used to indicate a receiver capability. 2241 The value MAY be equal to either 0 or 1. The value 1 2242 indicates that receiver discards out-of-band parameter sets 2243 in sprop-parameter-sets and sprop-level-parameter-sets, 2244 therefore the sender MUST transmit all parameter sets in-band. 2245 The value 0 indicates that the receiver utilizes out-of-band 2246 parameter sets included in sprop-parameter-sets and sprop- 2247 level-parameter-sets. However, in this case, the sender MAY 2248 still choose to send parameter sets in-band. When in-band- 2249 parameter-sets is equal to 1, use-level-src-parameter-sets 2250 MUST NOT be present or MUST be equal to 0. When the 2251 parameter is not present, this receiver capability is not 2252 specified, and therefore the sender MAY send out-of-band 2253 parameter sets only, or it MAY send in-band-parameter-sets 2254 only, or it MAY send both. 2256 packetization-mode: 2257 This parameter signals the properties of an RTP payload type 2258 or the capabilities of a receiver implementation. Only a 2259 single configuration point can be indicated; thus, when 2260 capabilities to support more than one packetization-mode are 2261 declared, multiple configuration points (RTP payload types) 2262 must be used. 2264 When the value of packetization-mode is equal to 0 or 2265 packetization-mode is not present, the single NAL mode, as 2266 defined in section 6.2 of RFC 3984, MUST be used. This mode 2267 is in use in standards using ITU-T Recommendation H.241 [3] 2268 (see section 12.1). When the value of packetization-mode is 2269 equal to 1, the non-interleaved mode, as defined in section 2270 6.3 of RFC 3984, MUST be used. When the value of 2271 packetization-mode is equal to 2, the interleaved mode, as 2272 defined in section 6.4 of RFC 3984, MUST be used. The value 2273 of packetization-mode MUST be an integer in the range of 0 to 2274 2, inclusive. 2276 sprop-interleaving-depth: 2277 This parameter MUST NOT be present when packetization-mode is 2278 not present or the value of packetization-mode is equal to 0 2279 or 1. This parameter MUST be present when the value of 2280 packetization-mode is equal to 2. 2282 This parameter signals the properties of an RTP packet stream. 2283 It specifies the maximum number of VCL NAL units that precede 2284 any VCL NAL unit in the RTP packet stream in transmission 2285 order and follow the VCL NAL unit in decoding order. 2286 Consequently, it is guaranteed that receivers can reconstruct 2287 NAL unit decoding order when the buffer size for NAL unit 2288 decoding order recovery is at least the value of sprop- 2289 interleaving-depth + 1 in terms of VCL NAL units. 2291 The value of sprop-interleaving-depth MUST be an integer in 2292 the range of 0 to 32767, inclusive. 2294 sprop-deint-buf-req: 2295 This parameter MUST NOT be present when packetization-mode is 2296 not present or the value of packetization-mode is equal to 0 2297 or 1. It MUST be present when the value of packetization- 2298 mode is equal to 2. 2300 sprop-deint-buf-req signals the required size of the de- 2301 interleaving buffer for the RTP packet stream. The value of 2302 the parameter MUST be greater than or equal to the maximum 2303 buffer occupancy (in units of bytes) required in such a de- 2304 interleaving buffer that is specified in section 7.2 of RFC 2305 3984. It is guaranteed that receivers can perform the de- 2306 interleaving of interleaved NAL units into NAL unit decoding 2307 order, when the de-interleaving buffer size is at least the 2308 value of sprop-deint-buf-req in terms of bytes. 2310 The value of sprop-deint-buf-req MUST be an integer in the 2311 range of 0 to 4294967295, inclusive. 2313 Informative note: sprop-deint-buf-req indicates the 2314 required size of the de-interleaving buffer only. When 2315 network jitter can occur, an appropriately sized jitter 2316 buffer has to be provisioned for as well. 2318 deint-buf-cap: 2319 This parameter signals the capabilities of a receiver 2320 implementation and indicates the amount of de-interleaving 2321 buffer space in units of bytes that the receiver has 2322 available for reconstructing the NAL unit decoding order. A 2323 receiver is able to handle any stream for which the value of 2324 the sprop-deint-buf-req parameter is smaller than or equal to 2325 this parameter. 2327 If the parameter is not present, then a value of 0 MUST be 2328 used for deint-buf-cap. The value of deint-buf-cap MUST be 2329 an integer in the range of 0 to 4294967295, inclusive. 2331 Informative note: deint-buf-cap indicates the maximum 2332 possible size of the de-interleaving buffer of the 2333 receiver only. When network jitter can occur, an 2334 appropriately sized jitter buffer has to be provisioned 2335 for as well. 2337 sprop-init-buf-time: 2338 This parameter MAY be used to signal the properties of an RTP 2339 packet stream. The parameter MUST NOT be present, if the 2340 value of packetization-mode is equal to 0 or 1. 2342 The parameter signals the initial buffering time that a 2343 receiver MUST wait before starting decoding to recover the 2344 NAL unit decoding order from the transmission order. The 2345 parameter is the maximum value of (decoding time of the NAL 2346 unit - transmission time of a NAL unit), assuming reliable 2347 and instantaneous transmission, the same timeline for 2348 transmission and decoding, and that decoding starts when the 2349 first packet arrives. 2351 An example of specifying the value of sprop-init-buf-time 2352 follows. A NAL unit stream is sent in the following 2353 interleaved order, in which the value corresponds to the 2354 decoding time and the transmission order is from left to 2355 right: 2357 0 2 1 3 5 4 6 8 7 ... 2359 Assuming a steady transmission rate of NAL units, the 2360 transmission times are: 2362 0 1 2 3 4 5 6 7 8 ... 2364 Subtracting the decoding time from the transmission time 2365 column-wise results in the following series: 2367 0 -1 1 0 -1 1 0 -1 1 ... 2369 Thus, in terms of intervals of NAL unit transmission times, 2370 the value of sprop-init-buf-time in this example is 1. The 2371 parameter is coded as a non-negative base10 integer 2372 representation in clock ticks of a 90-kHz clock. If the 2373 parameter is not present, then no initial buffering time 2374 value is defined. Otherwise the value of sprop-init-buf-time 2375 MUST be an integer in the range of 0 to 4294967295, inclusive. 2377 In addition to the signaled sprop-init-buf-time, receivers 2378 SHOULD take into account the transmission delay jitter 2379 buffering, including buffering for the delay jitter caused by 2380 mixers, translators, gateways, proxies, traffic-shapers, and 2381 other network elements. 2383 sprop-max-don-diff: 2384 This parameter MAY be used to signal the properties of an RTP 2385 packet stream. It MUST NOT be used to signal transmitter or 2386 receiver or codec capabilities. The parameter MUST NOT be 2387 present if the value of packetization-mode is equal to 0 or 1. 2388 sprop-max-don-diff is an integer in the range of 0 to 32767, 2389 inclusive. If sprop-max-don-diff is not present, the value 2390 of the parameter is unspecified. sprop-max-don-diff is 2391 calculated as follows: 2393 sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)}, 2394 for any i and any j>i, 2396 where i and j indicate the index of the NAL unit in the 2397 transmission order and AbsDON denotes a decoding order number 2398 of the NAL unit that does not wrap around to 0 after 65535. 2399 In other words, AbsDON is calculated as follows: Let m and n 2400 be consecutive NAL units in transmission order. For the very 2401 first NAL unit in transmission order (whose index is 0), 2402 AbsDON(0) = DON(0). For other NAL units, AbsDON is 2403 calculated as follows: 2405 If DON(m) == DON(n), AbsDON(n) = AbsDON(m) 2407 If (DON(m) < DON(n) and DON(n) - DON(m) < 32768), 2408 AbsDON(n) = AbsDON(m) + DON(n) - DON(m) 2410 If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768), 2411 AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n) 2413 If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768), 2414 AbsDON(n) = AbsDON(m) - (DON(m) + 65536 - DON(n)) 2416 If (DON(m) > DON(n) and DON(m) - DON(n) < 32768), 2417 AbsDON(n) = AbsDON(m) - (DON(m) - DON(n)) 2419 where DON(i) is the decoding order number of the NAL unit 2420 having index i in the transmission order. The decoding order 2421 number is specified in section 5.5 of RFC 3984. 2423 Informative note: Receivers may use sprop-max-don-diff to 2424 trigger which NAL units in the receiver buffer can be 2425 passed to the decoder. 2427 max-rcmd-nalu-size: 2428 This parameter MAY be used to signal the capabilities of a 2429 receiver. The parameter MUST NOT be used for any other 2430 purposes. The value of the parameter indicates the largest 2431 NALU size in bytes that the receiver can handle efficiently. 2432 The parameter value is a recommendation, not a strict upper 2433 boundary. The sender MAY create larger NALUs but must be 2434 aware that the handling of these may come at a higher cost 2435 than NALUs conforming to the limitation. 2437 The value of max-rcmd-nalu-size MUST be an integer in the 2438 range of 0 to 4294967295, inclusive. If this parameter is 2439 not specified, no known limitation to the NALU size exists. 2440 Senders still have to consider the MTU size available between 2441 the sender and the receiver and SHOULD run MTU discovery for 2442 this purpose. 2444 This parameter is motivated by, for example, an IP to H.223 2445 video telephony gateway, where NALUs smaller than the H.223 2446 transport data unit will be more efficient. A gateway may 2447 terminate IP; thus, MTU discovery will normally not work 2448 beyond the gateway. 2450 Informative note: Setting this parameter to a lower than 2451 necessary value may have a negative impact. 2453 sar-understood: 2454 This parameter MAY be used to indicate a receiver capability 2455 and not anything else. The parameter indicates the maximum 2456 value of aspect_ratio_idc (specified in [1]) smaller than 255 2457 that the receiver understands. Table E-1 of [1] specifies 2458 aspect_ratio_idc equal to 0 as "unspecified", 1 to 16, 2459 inclusive, as specific Sample Aspect Ratios (SARs), 17 to 254, 2460 inclusive, as "reserved", and 255 as the Extended SAR, for 2461 which SAR width and SAR height are explicitly signaled. 2462 Therefore, a receiver with a decoder according to [1] 2463 understands aspect_ratio_idc in the range of 1 to 16, 2464 inclusive and aspect_ratio_idc equal to 255, in the sense 2465 that the receiver knows what exactly the SAR is. For such a 2466 receiver, the value of sar-understood is 16. If in the 2467 future Table E-1 of [1] is extended, e.g., such that the SAR 2468 for aspect_ratio_idc equal to 17 is specified, then for a 2469 receiver with a decoder that understands the extension, the 2470 value of sar-understood is 17. For a receiver with a decoder 2471 according to the 2003 version of [1], the value of sar- 2472 understood is 13, as the minimum reserved aspect_ratio_idc 2473 therein is 14. 2475 When sar-understood is not present, the value MUST be 2476 inferred to be equal to 13. 2478 sar-supported: 2479 This parameter MAY be used to indicate a receiver capability 2480 and not anything else. The value of this parameter is an 2481 integer in the range of 1 to sar-understood, inclusive, equal 2482 to 255. The value of sar-supported equal to N smaller than 2483 255 indicates that the reciever supports all the SARs 2484 corresponding to H.264 aspect_ratio_idc values (see Table E-1 2485 of [1]) in the range from 1 to N, inclusive, without 2486 geometric distortion. The value of sar-supported equal to 2487 255 indicates that the receiver supports all sample aspect 2488 ratios which are expressible using two 16-bit integer values 2489 as the numerator and denominator, i.e., those that are 2490 expressible using the H.264 aspect_ratio_idc value of 255 2491 (Extended_SAR, see Table E-1 of [1]), without geometric 2492 distortion. 2494 H.264 compliant encoders SHOULD NOT send an aspect_ratio_idc 2495 equal to 0, or an aspect_ratio_idc larger than sar-understood 2496 and smaller than 255. H.264 compliant encoders SHOULD send 2497 an aspect_ratio_idc that the receiver is able to display 2498 without geometrical distortion. However, H.264 compliant 2499 encoders MAY choose to send pictures using any SAR. 2501 Note that the actual sample aspect ratio or extended sample 2502 aspect ratio, when present, of the stream is conveyed in the 2503 Video Usability Information (VUI) part of the sequence 2504 parameter set. 2506 Encoding considerations: 2507 This type is only defined for transfer via RTP (RFC 3550). 2509 Security considerations: 2510 See section 9 of RFC xxxx. 2512 Public specification: 2513 Please refer to RFC xxxx and its section 15. 2515 Additional information: 2516 None 2518 File extensions: none 2520 Macintosh file type code: none 2522 Object identifier or OID: none 2524 Person & email address to contact for further information: 2525 Ye-Kui Wang, yekuiwang@huawei.com 2527 Intended usage: COMMON 2529 Author: 2530 Ye-Kui Wang, yekuiwang@huawei.com 2532 Change controller: 2533 IETF Audio/Video Transport working group delegated from the 2534 IESG. 2536 8.2. SDP Parameters 2538 8.2.1. Mapping of Payload Type Parameters to SDP 2540 The media type video/H264 string is mapped to fields in the Session 2541 Description Protocol (SDP) [6] as follows: 2543 o The media name in the "m=" line of SDP MUST be video. 2545 o The encoding name in the "a=rtpmap" line of SDP MUST be H264 2546 (the media subtype). 2548 o The clock rate in the "a=rtpmap" line MUST be 90000. 2550 o The OPTIONAL parameters "profile-level-id", "max-mbps", "max- 2551 smbps", "max-fs", "max-cpb", "max-dpb", "max-br", "redundant- 2552 pic-cap", "use-level-src-parameter-sets", "in-band-parameter- 2553 sets", "packetization-mode", "sprop-interleaving-depth", "sprop- 2554 deint-buf-req", "deint-buf-cap", "sprop-init-buf-time", "sprop- 2555 max-don-diff", "max-rcmd-nalu-size", "sar-understood", and "sar- 2556 supported", when present, MUST be included in the "a=fmtp" line 2557 of SDP. These parameters are expressed as a media type string, 2558 in the form of a semicolon separated list of parameter=value 2559 pairs. 2561 o The OPTIONAL parameters "sprop-parameter-sets" and "sprop-level- 2562 parameter-sets", when present, MUST be included in the "a=fmtp" 2563 line of SDP or conveyed using the "fmtp" source attribute as 2564 specified in section 6.3 of [9]. For a particular media format 2565 (i.e., RTP payload type), a "sprop-parameter-sets" or "sprop- 2566 level-parameter-sets" MUST NOT be both included in the "a=fmtp" 2567 line of SDP and conveyed using the "fmtp" source attribute. 2568 When included in the "a=fmtp" line of SDP, these parameters are 2569 expressed as a media type string, in the form of a semicolon 2570 separated list of parameter=value pairs. When conveyed using 2571 the "fmtp" source attribute, these parameters are only 2572 associated with the given source and payload type as parts of 2573 the "fmtp" source attribute. 2575 Informative note: Conveyance of "sprop-parameter-sets" and 2576 "sprop-level-parameter-sets" using the "fmtp" source 2577 attribute allows for out-of-band transport of parameter sets 2578 in topologies like Topo-Video-switch-MCU [29]. 2580 An example of media representation in SDP is as follows (Baseline 2581 Profile, Level 3.0, some of the constraints of the Main profile may 2582 not be obeyed): 2584 m=video 49170 RTP/AVP 98 2585 a=rtpmap:98 H264/90000 2586 a=fmtp:98 profile-level-id=42A01E; 2587 packetization-mode=1; 2588 sprop-parameter-sets= 2590 8.2.2. Usage with the SDP Offer/Answer Model 2592 When H.264 is offered over RTP using SDP in an Offer/Answer model 2593 [8] for negotiation for unicast usage, the following limitations 2594 and rules apply: 2596 o The parameters identifying a media format configuration for 2597 H.264 are "profile-level-id" and "packetization-mode", when 2598 present. These media format configuration parameters (except 2599 for the level part of "profile-level-id") MUST be used 2600 symmetrically; i.e., the answerer MUST either maintain all 2601 configuration parameters or remove the media format (payload 2602 type) completely, if one or more of the parameter values are not 2603 supported. Note that the level part of "profile-level-id" 2604 includes level_idc, and, for indication of level 1b when 2605 profile_idc is equal to 66, 77 or 88, bit 4 2606 (constraint_set3_flag) of profile-iop. The level part of 2607 "profile-level-id" is downgradable, i.e. the answerer MUST 2608 maintain the same or a lower level or remove the media format 2609 (payload type) completely. 2611 Informative note: The requirement for symmetric use applies 2612 only for the above media format configuration parameters 2613 excluding the level part of "profile-level-id", and not for 2614 the other stream properties and capability parameters. 2616 Informative note: In H.264 [1], all the levels except for 2617 level 1b are equal to the value of level_idc divided by 10. 2618 Level 1b is a level higher than level 1.0 but lower than 2619 level 1.1, and is signaled in an ad-hoc manner, due to that 2620 the level was specified after level 1.0 and level 1.1. For 2621 the Baseline, Main and Extended profiles (with profile_idc 2622 equal to 66, 77 and 88, respectively), level 1b is indicated 2623 by level_idc equal to 11 (i.e. same as level 1.1) and 2624 constraint_set3_flag equal to 1. For other profiles, level 2625 1b is indicated by level_idc equal to 9 (but note that level 2626 1b for these profiles are still higher than level 1, which 2627 has level_idc equal to 10, and lower than level 1.1). In SDP 2628 Offer/Answer, an answer to an offer may indicate a level 2629 equal to or lower than the level indicated in the offer. Due 2630 to the ad-hoc indication of level 1b, offerers and answerers 2631 must check the value of bit 4 (constraint_set3_flag) of the 2632 middle octet of the parameter "profile-level-id", when 2633 profile_idc is equal to 66, 77 or 88 and level_idc is equal 2634 to 11. 2636 To simplify handling and matching of these configurations, the 2637 same RTP payload type number used in the offer SHOULD also be 2638 used in the answer, as specified in [8]. An answer MUST NOT 2639 contain a payload type number used in the offer unless the 2640 configuration is exactly the same as in the offer or the 2641 configuration in the answer only differs from that in the offer 2642 with a level lower than the default level offered. 2644 Informative note: When an offerer receives an answer, it has 2645 to compare payload types not declared in the offer based on 2646 the media type (i.e., video/H264) and the above media 2647 configuration parameters with any payload types it has 2648 already declared. This will enable it to determine whether 2649 the configuration in question is new or if it is equivalent 2650 to configuration already offered, since a different payload 2651 type number may be used in the answer. 2653 o The parameters "sprop-deint-buf-req", "sprop-interleaving-depth", 2654 "sprop-max-don-diff", and "sprop-init-buf-time" describe the 2655 properties of the RTP packet stream that the offerer or answerer 2656 is sending for the media format configuration. This differs 2657 from the normal usage of the Offer/Answer parameters: normally 2658 such parameters declare the properties of the stream that the 2659 offerer or the answerer is able to receive. When dealing with 2660 H.264, the offerer assumes that the answerer will be able to 2661 receive media encoded using the configuration being offered. 2663 Informative note: The above parameters apply for any stream 2664 sent by the declaring entity with the same configuration; 2665 i.e., they are dependent on their source. Rather than being 2666 bound to the payload type, the values may have to be applied 2667 to another payload type when being sent, as they apply for 2668 the configuration. 2670 o The capability parameters ("max-mbps", "max-smbps", "max-fs", 2671 "max-cpb", "max-dpb", "max-br", ,"redundant-pic-cap", "max-rcmd- 2672 nalu-size", "sar-understood", "sar-supported") MAY be used to 2673 declare further capabilities of the offerer or answerer for 2674 receiving. These parameters can only be present when the 2675 direction attribute is sendrecv or recvonly, and the parameters 2676 describe the limitations of what the offerer or answerer accepts 2677 for receiving streams. 2679 o An offerer has to include the size of the de-interleaving buffer, 2680 "sprop-deint-buf-req", in the offer for an interleaved H.264 2681 stream. To enable the offerer and answerer to inform each other 2682 about their capabilities for de-interleaving buffering in 2683 receiving streams, both parties are RECOMMENDED to include 2684 "deint-buf-cap". For interleaved streams, it is also 2685 RECOMMENDED to consider offering multiple payload types with 2686 different buffering requirements when the capabilities of the 2687 receiver are unknown. 2689 o The "sprop-parameter-sets" or "sprop-level-parameter-sets" 2690 parameter, when present (included in the "a=fmtp" line of SDP or 2691 conveyed using the "fmtp" source attribute as specified in 2692 section 6.3 of [9]), is used for out-of-band transport of 2693 parameter sets. However, when out-of-band transport of 2694 parameter sets is used, parameter sets MAY still be additionally 2695 transported in-band. If neither "sprop-parameter-sets" nor 2696 "sprop-level-parameter-sets" is present, then only in-band 2697 transport of parameter sets is used. 2699 An offer MAY include either or both of "sprop-parameter-sets" 2700 and "sprop-level-parameter-sets". An answer MAY include "sprop- 2701 parameter-sets", and MUST NOT include "sprop-level-parameter- 2702 sets". 2704 If the answer includes "in-band-parameter-sets" equal to 1, then 2705 the sender MUST transmit parameter sets in-band. 2707 Otherwise, the following applies. 2709 o When an offered payload type is accepted without level 2710 downgrade, i.e. the default level is accepted, the 2711 following applies. 2713 o When there is a "sprop-parameter-sets" included in the 2714 "a=fmtp" line of SDP, the answerer MUST be prepared to 2715 use the parameter sets included in "sprop-parameter- 2716 sets" for decoding the incoming NAL unit stream. 2718 o When there is a "sprop-parameter-sets" conveyed using 2719 the "fmtp" source attribute as specified in section 2720 6.3 of [9], and the answerer understands the "fmtp" 2721 source attribute, it MUST be prepared to use the 2722 parameter sets included in "sprop-parameter-sets" for 2723 decoding the incoming NAL unit stream, and it MUST 2724 include either "use-level-src-parameter-sets" equal to 2725 1 or the "fmtp" source attribute in the answer. 2727 o When there is a "sprop-parameter-sets" conveyed using 2728 the "fmtp" source attribute as specified in section 2729 6.3 of [9], and the answerer does not understand the 2730 "fmtp" source attribute, the sender MUST transmit 2731 parameter sets in-band, and the answerer MUST NOT 2732 include "use-level-src-parameter-sets" equal to 1 or 2733 the "fmtp" source attribute in the answer. 2735 o When "sprop-parameter-sets" is not present, the sender 2736 MUST transmit parameter sets in-band. 2738 o The answerer MUST ignore "sprop-level-parameter-sets", 2739 when present (either included in the "a=fmtp" line of 2740 SDP or conveyed using the "fmtp" source attribute). 2742 o When level downgrade is in use, i.e., a level lower than 2743 the default level offered is accepted, the following 2744 applies. 2746 o The answerer MUST ignore "sprop-parameter-sets", when 2747 present (either included in the "a=fmtp" line of SDP 2748 or conveyed using the "fmtp" source attribute). 2750 o When "use-level-src-parameter-sets" equal to 1 and the 2751 "fmtp" source attribute are not present in the answer 2752 for the accepted payload type, the answerer MUST 2753 ignore "sprop-level-parameter-sets", when present, and 2754 the sender MUST transmit parameter sets in-band. 2756 o When "use-level-src-parameter-sets" equal to 1 or the 2757 "fmtp" source attribute is present in the answer for 2758 the accepted payload type, the answerer MUST be 2759 prepared to use the parameter sets that are included 2760 in "sprop-level-parameter-sets" for the accepted level, 2761 when present, for decoding the incoming NAL unit 2762 stream, and ignore all other parameter sets included 2763 in "sprop-level-parameter-sets". 2765 o When no parameter sets for the accepted level are 2766 present in the "sprop-level-parameter-sets", the 2767 sender MUST transmit parameter sets in-band. 2769 The answerer MAY or MAY not include "sprop-parameter-sets", i.e., 2770 the answerer MAY use either out-of-band or in-band transport of 2771 parameter sets for the stream it is sending, regardless of 2772 whether out-of-band parameter sets transport has been used in 2773 the offerer-to-answerer direction. When the offer includes "in- 2774 band-parameter-sets" equal to 1, the answerer MUST NOT include 2775 "sprop-parameter-sets" and MUST transmit parameter sets in-band. 2776 All parameter sets included in the "sprop-parameter-sets", when 2777 present, for the accepted payload type in an answer MUST be 2778 associated with the accepted level, as indicated by the profile- 2779 level-id in the answer for the accepted payload type. 2781 Parameter sets included in "sprop-parameter-sets" in an answer 2782 are independent of those parameter sets included in the offer, 2783 as they are used for decoding two different video streams, one 2784 from the answerer to the offerer, and the other in the opposite 2785 direction. The offerer MUST be prepared to use the parameter 2786 sets included in the answer's "sprop-parameter-sets", when 2787 present, for decoding the incoming NAL unit stream. 2789 When "sprop-parameter-sets" or "sprop-level-parameter-sets" is 2790 conveyed using the "fmtp" source attribute in as specified in 2791 section 6.3 of [9], the receiver of the parameters MUST store 2792 the parameter sets included in the "sprop-parameter-sets" or 2793 "sprop-level-parameter-sets" for the accepted level and 2794 associate them to the source given as a part of the "fmtp" 2795 source attribute. Parameter sets associated with one source 2796 MUST only be used to decode NAL units conveyed in RTP packets 2797 from the same source. When this mechanism is in use, SSRC 2798 collision detection and resolution MUST be performed as 2799 specified in [9]. 2801 Informative note: Conveyance of "sprop-parameter-sets" and 2802 "sprop-level-parameter-sets" using the "fmtp" source 2803 attribute may be used in topologies like Topo-Video-switch- 2804 MCU [29] to enable out-of-band transport of parameter sets. 2806 For streams being delivered over multicast, the following rules 2807 apply: 2809 o The media format configuration is identified by the same 2810 parameters as above for unicast (i.e. "profile-level-id" and 2811 "packetization-mode", when present). These media format 2812 configuration parameters (including the level part of "profile- 2813 level-id") MUST be used symmetrically; i.e., the answerer MUST 2814 either maintain all configuration parameters or remove the media 2815 format (payload type) completely. Note that this implies that 2816 the level part of "profile-level-id" for Offer/Answer in 2817 multicast is not downgradable. 2819 To simplify handling and matching of these configurations, the 2820 same RTP payload type number used in the offer SHOULD also be 2821 used in the answer, as specified in [8]. An answer MUST NOT 2822 contain a payload type number used in the offer unless the 2823 configuration is the same as in the offer. 2825 o Parameter sets received MUST be associated with the originating 2826 source, and MUST be only used in decoding the incoming NAL unit 2827 stream from the same source. 2829 o The rules for other parameters are the same as above for unicast. 2831 Table 6 lists the interpretation of all the 20 media type 2832 parameters that MUST be used for the different direction attributes. 2834 Table 6. Interpretation of parameters for different direction 2835 attributes. 2837 sendonly --+ 2838 recvonly --+ | 2839 sendrecv --+ | | 2840 | | | 2841 profile-level-id C C P 2842 packetization-mode C C P 2843 sprop-deint-buf-req P - P 2844 sprop-interleaving-depth P - P 2845 sprop-max-don-diff P - P 2846 sprop-init-buf-time P - P 2847 max-mbps R R - 2848 max-smbps R R - 2849 max-fs R R - 2850 max-cpb R R - 2851 max-dpb R R - 2852 max-br R R - 2853 redundant-pic-cap R R - 2854 deint-buf-cap R R - 2855 max-rcmd-nalu-size R R - 2856 sar-understood R R - 2857 sar-supported R R - 2858 in-band-parameter-sets R R - 2859 use-level-src-parameter-sets R R - 2860 sprop-parameter-sets S - S 2861 sprop-level-parameter-sets S - S 2863 Legend: 2865 C: configuration for sending and receiving streams 2866 P: properties of the stream to be sent 2867 R: receiver capabilities 2868 S: out-of-band parameter sets 2869 -: not usable, when present SHOULD be ignored 2871 Parameters used for declaring receiver capabilities are in general 2872 downgradable; i.e., they express the upper limit for a sender's 2873 possible behavior. Thus a sender MAY select to set its encoder 2874 using only lower/less or equal values of these parameters. 2876 Parameters declaring a configuration point are not downgradable, 2877 with the exception of the level part of the "profile-level-id" 2878 parameter for unicast usage. This expresses values a receiver 2879 expects to be used and must be used verbatim on the sender side. 2881 When a sender's capabilities are declared, and non-downgradable 2882 parameters are used in this declaration, then these parameters 2883 express a configuration that is acceptable for the sender to 2884 receive streams. In order to achieve high interoperability levels, 2885 it is often advisable to offer multiple alternative configurations; 2886 e.g., for the packetization mode. It is impossible to offer 2887 multiple configurations in a single payload type. Thus, when 2888 multiple configuration offers are made, each offer requires its own 2889 RTP payload type associated with the offer. 2891 A receiver SHOULD understand all media type parameters, even if it 2892 only supports a subset of the payload format's functionality. This 2893 ensures that a receiver is capable of understanding when an offer 2894 to receive media can be downgraded to what is supported by the 2895 receiver of the offer. 2897 An answerer MAY extend the offer with additional media format 2898 configurations. However, to enable their usage, in most cases a 2899 second offer is required from the offerer to provide the stream 2900 property parameters that the media sender will use. This also has 2901 the effect that the offerer has to be able to receive this media 2902 format configuration, not only to send it. 2904 If an offerer wishes to have non-symmetric capabilities between 2905 sending and receiving, the offerer should offer different RTP 2906 sessions; i.e., different media lines declared as "recvonly" and 2907 "sendonly", respectively. This may have further implications on 2908 the system. 2910 8.2.3. Usage in Declarative Session Descriptions 2912 When H.264 over RTP is offered with SDP in a declarative style, as 2913 in RTSP [27] or SAP [28], the following considerations are 2914 necessary. 2916 o All parameters capable of indicating both stream properties and 2917 receiver capabilities are used to indicate only stream 2918 properties. For example, in this case, the parameter "profile- 2919 level-id" declares only the values used by the stream, not the 2920 capabilities for receiving streams. This results in that the 2921 following interpretation of the parameters MUST be used: 2923 Declaring actual configuration or stream properties: 2925 - profile-level-id 2926 - packetization-mode 2927 - sprop-interleaving-depth 2928 - sprop-deint-buf-req 2929 - sprop-max-don-diff 2930 - sprop-init-buf-time 2932 Out-of-band transporting of parameter sets: 2934 - sprop-parameter-sets 2935 - sprop-level-parameter-sets 2937 Not usable(when present, they SHOULD be ignored): 2939 - max-mbps 2940 - max-smbps 2941 - max-fs 2942 - max-cpb 2943 - max-dpb 2944 - max-br 2945 - redundant-pic-cap 2946 - max-rcmd-nalu-size 2947 - deint-buf-cap 2948 - sar-understood 2949 - sar-supported 2950 - in-band-parameter-sets 2951 - use-level-src-parameter-sets 2953 o A receiver of the SDP is required to support all parameters and 2954 values of the parameters provided; otherwise, the receiver MUST 2955 reject (RTSP) or not participate in (SAP) the session. It falls 2956 on the creator of the session to use values that are expected to 2957 be supported by the receiving application. 2959 8.3. Examples 2961 An SDP Offer/Answer exchange wherein both parties are expected to 2962 both send and receive could look like the following. Only the 2963 media codec specific parts of the SDP are shown. Some lines are 2964 wrapped due to text constraints. 2966 Offerer -> Answerer SDP message: 2968 m=video 49170 RTP/AVP 100 99 98 2969 a=rtpmap:98 H264/90000 2970 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; 2971 sprop-parameter-sets= 2972 a=rtpmap:99 H264/90000 2973 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 2974 sprop-parameter-sets= 2975 a=rtpmap:100 H264/90000 2976 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 2977 sprop-parameter-sets=; 2978 sprop-interleaving-depth=45; sprop-deint-buf-req=64000; 2979 sprop-init-buf-time=102478; deint-buf-cap=128000 2981 The above offer presents the same codec configuration in three 2982 different packetization formats. PT 98 represents single NALU mode, 2983 PT 99 represents non-interleaved mode, and PT 100 indicates the 2984 interleaved mode. In the interleaved mode case, the interleaving 2985 parameters that the offerer would use if the answer indicates 2986 support for PT 100 are also included. In all three cases the 2987 parameter "sprop-parameter-sets" conveys the initial parameter sets 2988 that are required by the answerer when receiving a stream from the 2989 offerer when this configuration is accepted. Note that the value 2990 for "sprop-parameter-sets" could be different for each payload type. 2992 Answerer -> Offerer SDP message: 2994 m=video 49170 RTP/AVP 100 99 97 2995 a=rtpmap:97 H264/90000 2996 a=fmtp:97 profile-level-id=42A01E; packetization-mode=0; 2997 sprop-parameter-sets= 2998 a=rtpmap:99 H264/90000 2999 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; 3000 sprop-parameter-sets=; 3001 max-rcmd-nalu-size=3980 3002 a=rtpmap:100 H264/90000 3003 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; 3004 sprop-parameter-sets=; 3005 sprop-interleaving-depth=60; 3006 sprop-deint-buf-req=86000; sprop-init-buf-time=156320; 3007 deint-buf-cap=128000; max-rcmd-nalu-size=3980 3009 As the Offer/Answer negotiation covers both sending and receiving 3010 streams, an offer indicates the exact parameters for what the 3011 offerer is willing to receive, whereas the answer indicates the 3012 same for what the answerer accepts to receive. In this case the 3013 offerer declared that it is willing to receive payload type 98. 3014 The answerer accepts this by declaring an equivalent payload type 3015 97; i.e., it has identical values for the two parameters "profile- 3016 level-id" and "packetization-mode" (since "packetization-mode" is 3017 equal to 0, "sprop-deint-buf-req" is not present). As the offered 3018 payload type 98 is accepted, the answerer needs to store parameter 3019 sets included in sprop-parameter-sets= in 3020 case the offer finally decides to use this configuration. In the 3021 answer, the answerer includes the parameter sets in sprop- 3022 parameter-sets= that the answerer would use 3023 in the stream sent from the answerer if this configuration is 3024 finally used. 3026 The answerer also accepts the reception of the two configurations 3027 that payload types 99 and 100 represent. Again, the answerer needs 3028 to store parameter sets included in sprop-parameter-sets= and sprop-parameter-sets= in 3030 case the offer finally decides to use either of these two 3031 configurations. The answerer provides the initial parameter sets 3032 for the answerer-to-offerer direction, i.e. the parameter sets in 3033 sprop-parameter-sets= and sprop-parameter- 3034 sets=, for payload types 99 and 100, 3035 respectively, that it will use to send the payload types. The 3036 answerer also provides the offerer with its memory limit for de- 3037 interleaving operations by providing a "deint-buf-cap" parameter. 3038 This is only useful if the offerer decides on making a second offer, 3039 where it can take the new value into account. The "max-rcmd-nalu- 3040 size" indicates that the answerer can efficiently process NALUs up 3041 to the size of 3980 bytes. However, there is no guarantee that the 3042 network supports this size. 3044 In the following example, the offer is accepted without level 3045 downgrading (i.e. the default level, 3.0, is accepted), and both 3046 "sprop-parameter-sets" and "sprop-level-parameter-sets" are present 3047 in the offer. The answerer must ignore sprop-level-parameter- 3048 sets= and store parameter sets in sprop- 3049 parameter-sets= for decoding the incoming 3050 NAL unit stream. The offerer must store the parameter sets in 3051 sprop-parameter-sets= in the answer for 3052 decoding the incoming NAL unit stream. Note that in this example, 3053 parameter sets in sprop-parameter-sets= must 3054 be associated with level 3.0. 3056 Offer SDP: 3058 m=video 49170 RTP/AVP 98 3059 a=rtpmap:98 H264/90000 3060 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3061 packetization-mode=1; 3062 sprop-parameter-sets=; 3063 sprop-level-parameter-sets= 3065 Answer SDP: 3067 m=video 49170 RTP/AVP 98 3068 a=rtpmap:98 H264/90000 3069 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3070 packetization-mode=1; 3071 sprop-parameter-sets= 3073 In the following example, the offer (Baseline profile, level 1.1) 3074 is accepted with level downgrading (the accepted level is 1b), and 3075 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3076 present in the offer. The answerer must ignore sprop-parameter- 3077 sets= and all parameter sets not for the 3078 accepted level (level 1b) in sprop-level-parameter-sets=, and must store parameter sets for the accepted level 3080 (level 1b) in sprop-level-parameter-sets= 3081 for decoding the incoming NAL unit stream. The offerer must store 3082 the parameter sets in sprop-parameter-sets= 3083 in the answer for decoding the incoming NAL unit stream. Note that 3084 in this example, parameter sets in sprop-parameter-sets= must be associated with level 1b. 3087 Offer SDP: 3089 m=video 49170 RTP/AVP 98 3090 a=rtpmap:98 H264/90000 3091 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3092 packetization-mode=1; 3093 sprop-parameter-sets=; 3094 sprop-level-parameter-sets= 3096 Answer SDP: 3098 m=video 49170 RTP/AVP 98 3099 a=rtpmap:98 H264/90000 3100 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3101 packetization-mode=1; 3102 sprop-parameter-sets=; 3103 use-level-src-parameter-sets=1 3105 In the following example, the offer (Baseline profile, level 1.1) 3106 is accepted with level downgrading (the accepted level is 1b), and 3107 both "sprop-parameter-sets" and "sprop-level-parameter-sets" are 3108 present in the offer. However, the answerer is a legacy RFC 3984 3109 implementation and does not understand "sprop-level-parameter-sets", 3110 hence it does not include "use-level-src-parameter-sets" (which the 3111 answerer does not understand, either) in the answer. Therefore, 3112 the answerer must ignore both sprop-parameter-sets= and sprop-level-parameter-sets=, and 3114 the offerer must transport parameter sets in-band. 3116 Offer SDP: 3118 m=video 49170 RTP/AVP 98 3119 a=rtpmap:98 H264/90000 3120 a=fmtp:98 profile-level-id=42A00B; //Baseline profile, Level 1.1 3121 packetization-mode=1; 3122 sprop-parameter-sets=; 3123 sprop-level-parameter-sets= 3125 Answer SDP: 3127 m=video 49170 RTP/AVP 98 3128 a=rtpmap:98 H264/90000 3129 a=fmtp:98 profile-level-id=42B00B; //Baseline profile, Level 1b 3130 packetization-mode=1 3132 In the following example, the offer is accepted without level 3133 downgrading, and "sprop-parameter-sets" is present in the offer. 3134 Parameter sets in sprop-parameter-sets= must 3135 be stored and used used by the encoder of the offerer and the 3136 decoder of the answerer, and parameter sets in sprop-parameter- 3137 sets=must be used by the encoder of the 3138 answerer and the decoder of the offerer. Note that sprop- 3139 parameter-sets= is basically independent of 3140 sprop-parameter-sets=. 3142 Offer SDP: 3144 m=video 49170 RTP/AVP 98 3145 a=rtpmap:98 H264/90000 3146 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3147 packetization-mode=1; 3148 sprop-parameter-sets= 3150 Answer SDP: 3152 m=video 49170 RTP/AVP 98 3153 a=rtpmap:98 H264/90000 3154 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3155 packetization-mode=1; 3156 sprop-parameter-sets= 3158 In the following example, the offer is accepted without level 3159 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3160 parameter-sets" is present in the offer, meaning that there is no 3161 out-of-band transmission of parameter sets, which then have to be 3162 transported in-band. 3164 Offer SDP: 3166 m=video 49170 RTP/AVP 98 3167 a=rtpmap:98 H264/90000 3168 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3169 packetization-mode=1 3171 Answer SDP: 3173 m=video 49170 RTP/AVP 98 3174 a=rtpmap:98 H264/90000 3175 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3176 packetization-mode=1 3178 In the following example, the offer is accepted with level 3179 downgrading and "sprop-parameter-sets" is present in the offer. As 3180 sprop-parameter-sets= contains level_idc 3181 indicating Level 3.0, therefore cannot be used as the answerer 3182 wants Level 2.0 and must be ignored by the answerer, and in-band 3183 parameter sets must be used. 3185 Offer SDP: 3187 m=video 49170 RTP/AVP 98 3188 a=rtpmap:98 H264/90000 3189 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3190 packetization-mode=1; 3191 sprop-parameter-sets= 3193 Answer SDP: 3195 m=video 49170 RTP/AVP 98 3196 a=rtpmap:98 H264/90000 3197 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3198 packetization-mode=1 3200 In the following example, the offer is also accepted with level 3201 downgrading, and neither "sprop-parameter-sets" nor "sprop-level- 3202 parameter-sets" is present in the offer, meaning that there is no 3203 out-of-band transmission of parameter sets, which then have to be 3204 transported in-band. 3206 Offer SDP: 3208 m=video 49170 RTP/AVP 98 3209 a=rtpmap:98 H264/90000 3210 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3211 packetization-mode=1 3213 Answer SDP: 3215 m=video 49170 RTP/AVP 98 3216 a=rtpmap:98 H264/90000 3217 a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0 3218 packetization-mode=1 3220 In the following example, the offerer is a Multipoint Control Unit 3221 (MCU) in a Topo-Video-switch-MCU like topology [29], offering 3222 parameter sets received (using out-of-band transport) from three 3223 other participants B, C, and D, and receiving parameter sets from 3224 the participant A, which is the answerer. The participants are 3225 identified by their values of CNAME, which are mapped to different 3226 SSRC values. The same codec configuration is used by all the four 3227 participants. The participant A stores and associates the 3228 parameter sets included in , , and to participants B, C, and D, 3230 respectively, and uses for decoding NAL 3231 units carried in RTP packets originated from participant B only, 3232 uses for decoding NAL units carried in RTP 3233 packets originated from participant C only, and uses for decoding NAL units carried in RTP packets 3235 originated from participant D only. 3237 Offer SDP: 3239 m=video 49170 RTP/AVP 98 3240 a=ssrc:SSRC-B cname:CNAME-B 3241 a=ssrc:SSRC-C cname:CNAME-C 3242 a=ssrc:SSRC-D cname:CNAME-D 3243 a=ssrc:SSRC-B fmtp:98 3244 sprop-parameter-sets= 3245 a=ssrc:SSRC-C fmtp:98 3246 sprop-parameter-sets= 3247 a=ssrc:SSRC-D fmtp:98 3248 sprop-parameter-sets= 3249 a=rtpmap:98 H264/90000 3250 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3251 packetization-mode=1 3253 Answer SDP: 3255 m=video 49170 RTP/AVP 98 3256 a=ssrc:SSRC-A cname:CNAME-A 3257 a=ssrc:SSRC-A fmtp:98 3258 sprop-parameter-sets= 3259 a=rtpmap:98 H264/90000 3260 a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0 3261 packetization-mode=1 3263 8.4. Parameter Set Considerations 3265 The H.264 parameter sets are a fundamental part of the video codec 3266 and vital to its operation; see section 1.2. Due to their 3267 characteristics and their importance for the decoding process, lost 3268 or erroneously transmitted parameter sets can hardly be concealed 3269 locally at the receiver. A reference to a corrupt parameter set 3270 has normally fatal results to the decoding process. Corruption 3271 could occur, for example, due to the erroneous transmission or loss 3272 of a parameter set NAL unit, but also due to the untimely 3273 transmission of a parameter set update. A parameter set update 3274 refers to a change of at least one parameter in a picture parameter 3275 set or sequence parameter set for which the picture parameter set 3276 or sequence parameter set identifier remains unchanged. Therefore, 3277 the following recommendations are provided as a guideline for the 3278 implementer of the RTP sender. 3280 Parameter set NALUs can be transported using three different 3281 principles: 3283 A. Using a session control protocol (out-of-band) prior to the 3284 actual RTP session. 3286 B. Using a session control protocol (out-of-band) during an ongoing 3287 RTP session. 3289 C. Within the RTP packet stream in the payload (in-band) during an 3290 ongoing RTP session. 3292 It is recommended to implement principles A and B within a session 3293 control protocol. SIP and SDP can be used as described in the SDP 3294 Offer/Answer model and in the previous sections of this memo. 3295 Section 8.2.2 includes a detailed discussion on transport of 3296 parameter sets in-band or out-of-band in SDP Offer/Answer using 3297 media type parameters "sprop-parameter-sets", "sprop-level- 3298 parameter-sets", "use-level-src-parameter-sets" and "in-band- 3299 parameter-sets". This section contains guidelines on how 3300 principles A and B should be implemented within session control 3301 protocols. It is independent of the particular protocol used. 3302 Principle C is supported by the RTP payload format defined in this 3303 specification. There are topologies like Topo-Video-switch-MCU [29] 3304 for which the use of principle C may be desirable. 3306 If in-band signaling of parameter sets is used, the picture and 3307 sequence parameter set NALUs SHOULD be transmitted in the RTP 3308 payload using a reliable method of delivering of RTP (see below), 3309 as a loss of a parameter set of either type will likely prevent 3310 decoding of a considerable portion of the corresponding RTP packet 3311 stream. 3313 If in-band signaling of parameter sets is used, the sender SHOULD 3314 take the error characteristics into account and use mechanisms to 3315 provide a high probability for delivering the parameter sets 3316 correctly. Mechanisms that increase the probability for a correct 3317 reception include packet repetition, FEC, and retransmission. The 3318 use of an unreliable, out-of-band control protocol has similar 3319 disadvantages as the in-band signaling (possible loss) and, in 3320 addition, may also lead to difficulties in the synchronization (see 3321 below). Therefore, it is NOT RECOMMENDED. 3323 Parameter sets MAY be added or updated during the lifetime of a 3324 session using principles B and C. It is required that parameter 3325 sets are present at the decoder prior to the NAL units that refer 3326 to them. Updating or adding of parameter sets can result in 3327 further problems, and therefore the following recommendations 3328 should be considered. 3330 - When parameter sets are added or updated, care SHOULD be taken 3331 to ensure that any parameter set is delivered prior to its usage. 3332 When new parameter sets are added, previously unused parameter 3333 set identifiers are used. It is common that no synchronization 3334 is present between out-of-band signaling and in-band traffic. 3335 If out-of-band signaling is used, it is RECOMMENDED that a 3336 sender does not start sending NALUs requiring the added or 3337 updated parameter sets prior to acknowledgement of delivery from 3338 the signaling protocol. 3340 - When parameter sets are updated, the following synchronization 3341 issue should be taken into account. When overwriting a 3342 parameter set at the receiver, the sender has to ensure that the 3343 parameter set in question is not needed by any NALU present in 3344 the network or receiver buffers. Otherwise, decoding with a 3345 wrong parameter set may occur. To lessen this problem, it is 3346 RECOMMENDED either to overwrite only those parameter sets that 3347 have not been used for a sufficiently long time (to ensure that 3348 all related NALUs have been consumed), or to add a new parameter 3349 set instead (which may have negative consequences for the 3350 efficiency of the video coding). 3352 Informative note: In some topologies like Topo-Video-switch- 3353 MCU [29] the origin of the whole set of parameter sets may 3354 come from multiple sources that may use non-unique parameter 3355 sets identifiers. In this case an offer may overwrite an 3356 existing parameter set if no other mechanism that enables 3357 uniqueness of the parameter sets in the out-of-band channel 3358 exists. 3360 - In a multiparty session, one participant MUST associate 3361 parameter sets coming from different sources with the source 3362 identification whenever possible, e.g. by conveying out-of-band 3363 transported parameter sets, as different sources typically use 3364 independent parameter set identifier value spaces. 3366 - Adding or modifying parameter sets by using both principles B 3367 and C in the same RTP session may lead to inconsistencies of the 3368 parameter sets because of the lack of synchronization between 3369 the control and the RTP channel. Therefore, principles B and C 3370 MUST NOT both be used in the same session unless sufficient 3371 synchronization can be provided. 3373 In some scenarios (e.g., when only the subset of this payload 3374 format specification corresponding to H.241 is used) or topologies, 3375 it is not possible to employ out-of-band parameter set transmission. 3376 In this case, parameter sets have to be transmitted in-band. Here, 3377 the synchronization with the non-parameter-set-data in the 3378 bitstream is implicit, but the possibility of a loss has to be 3379 taken into account. The loss probability should be reduced using 3380 the mechanisms discussed above. In case a loss of a parameter set 3381 is detected, recovery may be achieved by using a Decoder Refresh 3382 Point procedure, for example, using RTCP feedback Full Intra 3383 Request (FIR) [30]. Two example Decoder Refresh Point procedures 3384 are provided in the informative Section 8.5. 3386 - When parameter sets are initially provided using principle A and 3387 then later added or updated in-band (principle C), there is a 3388 risk associated with updating the parameter sets delivered out- 3389 of-band. If receivers miss some in-band updates (for example, 3390 because of a loss or a late tune-in), those receivers attempt to 3391 decode the bitstream using out-dated parameters. It is 3392 therefore RECOMMENDED that parameter set IDs be partitioned 3393 between the out-of-band and in-band parameter sets. 3395 8.5. Decoder Refresh Point Procedure using In-Band Transport of 3396 Parameter Sets (Informative) 3398 When a sender with a video encoder according to [1] receives a 3399 request for a decoder refresh point, the encoder shall enter the 3400 fast update mode by using one of the procedures specified 3401 in Section 8.5.1 or 8.5.2 below. The procedure in 8.5.1 is the 3402 preferred response in a lossless transmission environment. Both 3403 procedures satisfy the requirement to enter the fast update mode 3404 for H.264 video encoding. 3406 8.5.1. IDR Procedure to Respond to a Request for a Decoder Refresh 3407 Point 3409 This section gives one possible way to respond to a request for a 3410 decoder refresh point. 3412 The encoder shall, in the order presented here: 3414 1) Immediately prepare to send an IDR picture. 3416 2) Send a sequence parameter set to be used by the IDR picture to 3417 be sent. The encoder may optionally also send other sequence 3418 parameter sets. 3420 3) Send a picture parameter set to be used by the IDR picture to be 3421 sent. The encoder may optionally also send other picture 3422 parameter sets. 3424 4) Send the IDR picture. 3426 5) From this point forward in time, send any other sequence or 3427 picture parameter sets that have not yet been sent in this 3428 procedure, prior to their reference by any NAL unit, regardless 3429 of whether such parameter sets were previously sent prior to 3430 receiving the request for a decoder refresh point. As needed, 3431 such parameter sets may be sent in a batch, one at a time, or in 3432 any combination of these two methods. Parameter sets may be re- 3433 sent at any time for redundancy. Caution should be taken when 3434 parameter set updates are present, as described above in Section 3435 8.4. 3437 8.5.2. Gradual Recovery Procedure to Respond to a Request for a 3438 Decoder Refresh Point 3440 This section gives another possible way to respond to a request for 3441 a decoder refresh point. 3443 The encoder shall, in the order presented here: 3445 1) Send a recovery point SEI message (see Sections D.1.7 and D.2.7 3446 of [1]). 3448 2) Repeat any sequence and picture parameter sets that were sent 3449 before the recovery point SEI message, prior to their reference 3450 by a NAL unit. 3452 The encoder shall ensure that the decoder has access to all 3453 reference pictures for inter prediction of pictures at or after the 3454 recovery point, which is indicated by the recovery point SEI 3455 message, in output order, assuming that the transmission from now 3456 on is error-free. 3458 The value of the recovery_frame_cnt syntax element in the recovery 3459 point SEI message should be small enough to ensure a fast recovery. 3461 As needed, such parameter sets may be re-sent in a batch, one at a 3462 time, or in any combination of these two methods. Parameter sets 3463 may be re-sent at any time for redundancy. Caution should be taken 3464 when parameter set updates are present, as described above in 3465 Section 8.4. 3467 9. Security Considerations 3469 RTP packets using the payload format defined in this specification 3470 are subject to the security considerations discussed in the RTP 3471 specification [5], and in any appropriate RTP profile (for example, 3472 [16]). This implies that confidentiality of the media streams is 3473 achieved by encryption; for example, through the application of 3474 SRTP [26]. Because the data compression used with this payload 3475 format is applied end-to-end, any encryption needs to be performed 3476 after compression. A potential denial-of-service threat exists for 3477 data encodings using compression techniques that have non-uniform 3478 receiver-end computational load. The attacker can inject 3479 pathological datagrams into the stream that are complex to decode 3480 and that cause the receiver to be overloaded. H.264 is 3481 particularly vulnerable to such attacks, as it is extremely simple 3482 to generate datagrams containing NAL units that affect the decoding 3483 process of many future NAL units. Therefore, the usage of data 3484 origin authentication and data integrity protection of at least the 3485 RTP packet is RECOMMENDED; for example, with SRTP [26]. 3487 Note that the appropriate mechanism to ensure confidentiality and 3488 integrity of RTP packets and their payloads is very dependent on 3489 the application and on the transport and signaling protocols 3490 employed. Thus, although SRTP is given as an example above, other 3491 possible choices exist. 3493 Decoders MUST exercise caution with respect to the handling of user 3494 data SEI messages, particularly if they contain active elements, 3495 and MUST restrict their domain of applicability to the presentation 3496 containing the stream. 3498 End-to-End security with either authentication, integrity or 3499 confidentiality protection will prevent a MANE from performing 3500 media-aware operations other than discarding complete packets. And 3501 in the case of confidentiality protection it will even be prevented 3502 from performing discarding of packets in a media aware way. To 3503 allow any MANE to perform its operations, it will be required to be 3504 a trusted entity which is included in the security context 3505 establishment. 3507 10. Congestion Control 3509 Congestion control for RTP SHALL be used in accordance with RFC 3510 3550 [5], and with any applicable RTP profile; e.g., RFC 3551 [16]. 3511 An additional requirement if best-effort service is being used is: 3512 users of this payload format MUST monitor packet loss to ensure 3513 that the packet loss rate is within acceptable parameters. Packet 3514 loss is considered acceptable if a TCP flow across the same network 3515 path, and experiencing the same network conditions, would achieve 3516 an average throughput, measured on a reasonable timescale, that is 3517 not less than the RTP flow is achieving. This condition can be 3518 satisfied by implementing congestion control mechanisms to adapt 3519 the transmission rate (or the number of layers subscribed for a 3520 layered multicast session), or by arranging for a receiver to leave 3521 the session if the loss rate is unacceptably high. 3523 The bit rate adaptation necessary for obeying the congestion 3524 control principle is easily achievable when real-time encoding is 3525 used. However, when pre-encoded content is being transmitted, 3526 bandwidth adaptation requires the availability of more than one 3527 coded representation of the same content, at different bit rates, 3528 or the existence of non-reference pictures or sub-sequences [22] in 3529 the bitstream. The switching between the different representations 3530 can normally be performed in the same RTP session; e.g., by 3531 employing a concept known as SI/SP slices of the Extended Profile, 3532 or by switching streams at IDR picture boundaries. Only when non- 3533 downgradable parameters (such as the profile part of the 3534 profile/level ID) are required to be changed does it become 3535 necessary to terminate and re-start the media stream. This may be 3536 accomplished by using a different RTP payload type. 3538 MANEs MAY follow the suggestions outlined in section 7.3 and remove 3539 certain unusable packets from the packet stream when that stream 3540 was damaged due to previous packet losses. This can help reduce 3541 the network load in certain special cases. 3543 11. IANA Consideration 3545 The H264 media subtype name specified by RFC 3984 should be updated 3546 as defined in section 8.1 of this memo. 3548 12. Informative Appendix: Application Examples 3550 This payload specification is very flexible in its use, in order to 3551 cover the extremely wide application space anticipated for H.264. 3552 However, this great flexibility also makes it difficult for an 3553 implementer to decide on a reasonable packetization scheme. Some 3554 information on how to apply this specification to real-world 3555 scenarios is likely to appear in the form of academic publications 3556 and a test model software and description in the near future. 3557 However, some preliminary usage scenarios are described here as 3558 well. 3560 12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 3562 H.323-based video telephony systems that use H.264 as an optional 3563 video compression scheme are required to support H.241 Annex A [3] 3564 as a packetization scheme. The packetization mechanism defined in 3565 this Annex is technically identical with a small subset of this 3566 specification. 3568 When a system operates according to H.241 Annex A, parameter set 3569 NAL units are sent in-band. Only Single NAL unit packets are used. 3570 Many such systems are not sending IDR pictures regularly, but only 3571 when required by user interaction or by control protocol means; 3572 e.g., when switching between video channels in a Multipoint Control 3573 Unit or for error recovery requested by feedback. 3575 12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit 3576 Aggregation 3578 The RTP part of this scheme is implemented and tested (though not 3579 the control-protocol part; see below). 3581 In most real-world video telephony applications, picture parameters 3582 such as picture size or optional modes never change during the 3583 lifetime of a connection. Therefore, all necessary parameter sets 3584 (usually only one) are sent as a side effect of the capability 3585 exchange/announcement process, e.g., according to the SDP syntax 3586 specified in section 8.2 of this document. As all necessary 3587 parameter set information is established before the RTP session 3588 starts, there is no need for sending any parameter set NAL units. 3589 Slice data partitioning is not used, either. Thus, the RTP packet 3590 stream basically consists of NAL units that carry single coded 3591 slices. 3593 The encoder chooses the size of coded slice NAL units so that they 3594 offer the best performance. Often, this is done by adapting the 3595 coded slice size to the MTU size of the IP network. For small 3596 picture sizes, this may result in a one-picture-per-one-packet 3597 strategy. Intra refresh algorithms clean up the loss of packets 3598 and the resulting drift-related artifacts. 3600 12.3. Video Telephony, Interleaved Packetization Using NAL Unit 3601 Aggregation 3603 This scheme allows better error concealment and is used in H.263 3604 based designs using RFC 4629 packetization [11]. It has been 3605 implemented, and good results were reported [13]. 3607 The VCL encoder codes the source picture so that all macroblocks 3608 (MBs) of one MB line are assigned to one slice. All slices with 3609 even MB row addresses are combined into one STAP, and all slices 3610 with odd MB row addresses into another. Those STAPs are 3611 transmitted as RTP packets. The establishment of the parameter 3612 sets is performed as discussed above. 3614 Note that the use of STAPs is essential here, as the high number of 3615 individual slices (18 for a CIF picture) would lead to unacceptably 3616 high IP/UDP/RTP header overhead (unless the source coding tool FMO 3617 is used, which is not assumed in this scenario). Furthermore, some 3618 wireless video transmission systems, such as H.324M and the IP- 3619 based video telephony specified in 3GPP, are likely to use 3620 relatively small transport packet size. For example, a typical MTU 3621 size of H.223 AL3 SDU is around 100 bytes [17]. Coding individual 3622 slices according to this packetization scheme provides further 3623 advantage in communication between wired and wireless networks, as 3624 individual slices are likely to be smaller than the preferred 3625 maximum packet size of wireless systems. Consequently, a gateway 3626 can convert the STAPs used in a wired network into several RTP 3627 packets with only one NAL unit, which are preferred in a wireless 3628 network, and vice versa. 3630 12.4. Video Telephony with Data Partitioning 3632 This scheme has been implemented and has been shown to offer good 3633 performance, especially at higher packet loss rates [13]. 3635 Data Partitioning is known to be useful only when some form of 3636 unequal error protection is available. Normally, in single-session 3637 RTP environments, even error characteristics are assumed; i.e., the 3638 packet loss probability of all packets of the session is the same 3639 statistically. However, there are means to reduce the packet loss 3640 probability of individual packets in an RTP session. A FEC packet 3641 according to RFC 2733 [18], for example, specifies which media 3642 packets are associated with the FEC packet. 3644 In all cases, the incurred overhead is substantial but is in the 3645 same order of magnitude as the number of bits that have otherwise 3646 been spent for intra information. However, this mechanism does not 3647 add any delay to the system. 3649 Again, the complete parameter set establishment is performed 3650 through control protocol means. 3652 12.5. Video Telephony or Streaming with FUs and Forward Error 3653 Correction 3655 This scheme has been implemented and has been shown to provide good 3656 performance, especially at higher packet loss rates [19]. 3658 The most efficient means to combat packet losses for scenarios 3659 where retransmissions are not applicable is forward error 3660 correction (FEC). Although application layer, end-to-end use of 3661 FEC is often less efficient than an FEC-based protection of 3662 individual links (especially when links of different 3663 characteristics are in the transmission path), application layer, 3664 end-to-end FEC is unavoidable in some scenarios. RFC 5109 [18] 3665 provides means to use generic, application layer, end-to-end FEC in 3666 packet-loss environments. A binary forward error correcting code 3667 is generated by applying the XOR operation to the bits at the same 3668 bit position in different packets. The binary code can be 3669 specified by the parameters (n,k) in which k is the number of 3670 information packets used in the connection and n is the total 3671 number of packets generated for k information packets; i.e., n-k 3672 parity packets are generated for k information packets. 3674 When a code is used with parameters (n,k) within the RFC 5109 3675 framework, the following properties are well known: 3677 a) If applied over one RTP packet, RFC 5109 provides only packet 3678 repetition. 3680 b) RFC 5109 is most bit rate efficient if XOR-connected packets 3681 have equal length. 3683 c) At the same packet loss probability p and for a fixed k, the 3684 greater the value of n is, the smaller the residual error 3685 probability becomes. For example, for a packet loss probability 3686 of 10%, k=1, and n=2, the residual error probability is about 1%, 3687 whereas for n=3, the residual error probability is about 0.1%. 3689 d) At the same packet loss probability p and for a fixed code rate 3690 k/n, the greater the value of n is, the smaller the residual 3691 error probability becomes. For example, at a packet loss 3692 probability of p=10%, k=1 and n=2, the residual error rate is 3693 about 1%, whereas for an extended Golay code with k=12 and n=24, 3694 the residual error rate is about 0.01%. 3696 For applying RFC 5109 in combination with H.264 baseline coded 3697 video without using FUs, several options might be considered: 3699 1) The video encoder produces NAL units for which each video frame 3700 is coded in a single slice. Applying FEC, one could use a 3701 simple code; e.g., (n=2, k=1). That is, each NAL unit would 3702 basically just be repeated. The disadvantage is obviously the 3703 bad code performance according to d), above, and the low 3704 flexibility, as only (n, k=1) codes can be used. 3706 2) The video encoder produces NAL units for which each video frame 3707 is encoded in one or more consecutive slices. Applying FEC, one 3708 could use a better code, e.g., (n=24, k=12), over a sequence of 3709 NAL units. Depending on the number of RTP packets per frame, a 3710 loss may introduce a significant delay, which is reduced when 3711 more RTP packets are used per frame. Packets of completely 3712 different length might also be connected, which decreases bit 3713 rate efficiency according to b), above. However, with some care 3714 and for slices of 1kb or larger, similar length (100-200 bytes 3715 difference) may be produced, which will not lower the bit 3716 efficiency catastrophically. 3718 3) The video encoder produces NAL units, for which a certain frame 3719 contains k slices of possibly almost equal length. Then, 3720 applying FEC, a better code, e.g., (n=24, k=12), can be used 3721 over the sequence of NAL units for each frame. The delay 3722 compared to that of 2), above, may be reduced, but several 3723 disadvantages are obvious. First, the coding efficiency of the 3724 encoded video is lowered significantly, as slice-structured 3725 coding reduces intra-frame prediction and additional slice 3726 overhead is necessary. Second, pre-encoded content or, when 3727 operating over a gateway, the video is usually not appropriately 3728 coded with k slices such that FEC can be applied. Finally, the 3729 encoding of video producing k slices of equal length is not 3730 straightforward and might require more than one encoding pass. 3732 Many of the mentioned disadvantages can be avoided by applying FUs 3733 in combination with FEC. Each NAL unit can be split into any 3734 number of FUs of basically equal length; therefore, FEC with a 3735 reasonable k and n can be applied, even if the encoder made no 3736 effort to produce slices of equal length. For example, a coded 3737 slice NAL unit containing an entire frame can be split to k FUs, 3738 and a parity check code (n=k+1, k) can be applied. However, this 3739 has the disadvantage that unless all created fragments can be 3740 recovered, the whole slice will be lost. Thus a larger section is 3741 lost than would be if the frame had been split into several slices. 3743 The presented technique makes it possible to achieve good 3744 transmission error tolerance, even if no additional source coding 3745 layer redundancy (such as periodic intra frames) is present. 3746 Consequently, the same coded video sequence can be used to achieve 3747 the maximum compression efficiency and quality over error-free 3748 transmission and for transmission over error-prone networks. 3749 Furthermore, the technique allows the application of FEC to pre- 3750 encoded sequences without adding delay. In this case, pre-encoded 3751 sequences that are not encoded for error-prone networks can still 3752 be transmitted almost reliably without adding extensive delays. In 3753 addition, FUs of equal length result in a bit rate efficient use of 3754 RFC 5109. 3756 If the error probability depends on the length of the transmitted 3757 packet (e.g., in case of mobile transmission [15]), the benefits of 3758 applying FUs with FEC are even more obvious. Basically, the 3759 flexibility of the size of FUs allows appropriate FEC to be applied 3760 for each NAL unit and unequal error protection of NAL units. 3762 When FUs and FEC are used, the incurred overhead is substantial but 3763 is in the same order of magnitude as the number of bits that have 3764 to be spent for intra-coded macroblocks if no FEC is applied. In 3765 [19], it was shown that the overall performance of the FEC-based 3766 approach enhanced quality when using the same error rate and same 3767 overall bit rate, including the overhead. 3769 12.6. Low Bit-Rate Streaming 3771 This scheme has been implemented with H.263 and non-standard RTP 3772 packetization and has given good results [20]. There is no 3773 technical reason why similarly good results could not be achievable 3774 with H.264. 3776 In today's Internet streaming, some of the offered bit rates are 3777 relatively low in order to allow terminals with dial-up modems to 3778 access the content. In wired IP networks, relatively large packets, 3779 say 500 - 1500 bytes, are preferred to smaller and more frequently 3780 occurring packets in order to reduce network congestion. Moreover, 3781 use of large packets decreases the amount of RTP/UDP/IP header 3782 overhead. For low bit-rate video, the use of large packets means 3783 that sometimes up to few pictures should be encapsulated in one 3784 packet. 3786 However, loss of a packet including many coded pictures would have 3787 drastic consequences for visual quality, as there is practically no 3788 other way to conceal a loss of an entire picture than to repeat the 3789 previous one. One way to construct relatively large packets and 3790 maintain possibilities for successful loss concealment is to 3791 construct MTAPs that contain interleaved slices from several 3792 pictures. An MTAP should not contain spatially adjacent slices 3793 from the same picture or spatially overlapping slices from any 3794 picture. If a packet is lost, it is likely that a lost slice is 3795 surrounded by spatially adjacent slices of the same picture and 3796 spatially corresponding slices of the temporally previous and 3797 succeeding pictures. Consequently, concealment of the lost slice 3798 is likely to be relatively successful. 3800 12.7. Robust Packet Scheduling in Video Streaming 3802 Robust packet scheduling has been implemented with MPEG-4 Part 2 3803 and simulated in a wireless streaming environment [21]. There is 3804 no technical reason why similar or better results could not be 3805 achievable with H.264. 3807 Streaming clients typically have a receiver buffer that is capable 3808 of storing a relatively large amount of data. Initially, when a 3809 streaming session is established, a client does not start playing 3810 the stream back immediately. Rather, it typically buffers the 3811 incoming data for a few seconds. This buffering helps maintain 3812 continuous playback, as, in case of occasional increased 3813 transmission delays or network throughput drops, the client can 3814 decode and play buffered data. Otherwise, without initial 3815 buffering, the client has to freeze the display, stop decoding, and 3816 wait for incoming data. The buffering is also necessary for either 3817 automatic or selective retransmission in any protocol level. If 3818 any part of a picture is lost, a retransmission mechanism may be 3819 used to resend the lost data. If the retransmitted data is 3820 received before its scheduled decoding or playback time, the loss 3821 is recovered perfectly. Coded pictures can be ranked according to 3822 their importance in the subjective quality of the decoded sequence. 3823 For example, non-reference pictures, such as conventional B 3824 pictures, are subjectively least important, as their absence does 3825 not affect decoding of any other pictures. In addition to non- 3826 reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard 3827 includes a temporal scalability method called sub-sequences [22]. 3828 Subjective ranking can also be made on coded slice data partition 3829 or slice group basis. Coded slices and coded slice data partitions 3830 that are subjectively the most important can be sent earlier than 3831 their decoding order indicates, whereas coded slices and coded 3832 slice data partitions that are subjectively the least important can 3833 be sent later than their natural coding order indicates. 3834 Consequently, any retransmitted parts of the most important slices 3835 and coded slice data partitions are more likely to be received 3836 before their scheduled decoding or playback time compared to the 3837 least important slices and slice data partitions. 3839 13. Informative Appendix: Rationale for Decoding Order Number 3841 13.1. Introduction 3843 The Decoding Order Number (DON) concept was introduced mainly to 3844 enable efficient multi-picture slice interleaving (see section 12.6) 3845 and robust packet scheduling (see section 12.7). In both of these 3846 applications, NAL units are transmitted out of decoding order. DON 3847 indicates the decoding order of NAL units and should be used in the 3848 receiver to recover the decoding order. Example use cases for 3849 efficient multi-picture slice interleaving and for robust packet 3850 scheduling are given in sections 13.2 and 13.3, respectively. 3851 Section 13.4 describes the benefits of the DON concept in error 3852 resiliency achieved by redundant coded pictures. Section 13.5 3853 summarizes considered alternatives to DON and justifies why DON was 3854 chosen to this RTP payload specification. 3856 13.2. Example of Multi-Picture Slice Interleaving 3858 An example of multi-picture slice interleaving follows. A subset 3859 of a coded video sequence is depicted below in output order. R 3860 denotes a reference picture, N denotes a non-reference picture, and 3861 the number indicates a relative output time. 3863 ... R1 N2 R3 N4 R5 ... 3865 The decoding order of these pictures from left to right is as 3866 follows: 3868 ... R1 R3 N2 R5 N4 ... 3870 The NAL units of pictures R1, R3, N2, R5, and N4 are marked with a 3871 DON equal to 1, 2, 3, 4, and 5, respectively. 3873 Each reference picture consists of three slice groups that are 3874 scattered as follows (a number denotes the slice group number for 3875 each macroblock in a QCIF frame): 3877 0 1 2 0 1 2 0 1 2 0 1 3878 2 0 1 2 0 1 2 0 1 2 0 3879 1 2 0 1 2 0 1 2 0 1 2 3880 0 1 2 0 1 2 0 1 2 0 1 3881 2 0 1 2 0 1 2 0 1 2 0 3882 1 2 0 1 2 0 1 2 0 1 2 3883 0 1 2 0 1 2 0 1 2 0 1 3884 2 0 1 2 0 1 2 0 1 2 0 3885 1 2 0 1 2 0 1 2 0 1 2 3887 For the sake of simplicity, we assume that all the macroblocks of a 3888 slice group are included in one slice. Three MTAPs are constructed 3889 from three consecutive reference pictures so that each MTAP 3890 contains three aggregation units, each of which contains all the 3891 macroblocks from one slice group. The first MTAP contains slice 3892 group 0 of picture R1, slice group 1 of picture R3, and slice group 3893 2 of picture R5. The second MTAP contains slice group 1 of picture 3894 R1, slice group 2 of picture R3, and slice group 0 of picture R5. 3895 The third MTAP contains slice group 2 of picture R1, slice group 0 3896 of picture R3, and slice group 1 of picture R5. Each non-reference 3897 picture is encapsulated into an STAP-B. 3899 Consequently, the transmission order of NAL units is the following: 3901 R1, slice group 0, DON 1, carried in MTAP,RTP SN: N 3902 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N 3903 R5, slice group 2, DON 4, carried in MTAP,RTP SN: N 3904 R1, slice group 1, DON 1, carried in MTAP,RTP SN: N+1 3905 R3, slice group 2, DON 2, carried in MTAP,RTP SN: N+1 3906 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+1 3907 R1, slice group 2, DON 1, carried in MTAP,RTP SN: N+2 3908 R3, slice group 1, DON 2, carried in MTAP,RTP SN: N+2 3909 R5, slice group 0, DON 4, carried in MTAP,RTP SN: N+2 3910 N2, DON 3, carried in STAP-B, RTP SN: N+3 3911 N4, DON 5, carried in STAP-B, RTP SN: N+4 3913 The receiver is able to organize the NAL units back in decoding 3914 order based on the value of DON associated with each NAL unit. 3916 If one of the MTAPs is lost, the spatially adjacent and temporally 3917 co-located macroblocks are received and can be used to conceal the 3918 loss efficiently. If one of the STAPs is lost, the effect of the 3919 loss does not propagate temporally. 3921 13.3. Example of Robust Packet Scheduling 3923 An example of robust packet scheduling follows. The communication 3924 system used in the example consists of the following components in 3925 the order that the video is processed from source to sink: 3927 o camera and capturing 3928 o pre-encoding buffer 3929 o encoder 3930 o encoded picture buffer 3931 o transmitter 3932 o transmission channel 3933 o receiver 3934 o receiver buffer 3935 o decoder 3936 o decoded picture buffer 3937 o display 3939 The video communication system used in the example operates as 3940 follows. Note that processing of the video stream happens 3941 gradually and at the same time in all components of the system. 3942 The source video sequence is shot and captured to a pre-encoding 3943 buffer. The pre-encoding buffer can be used to order pictures from 3944 sampling order to encoding order or to analyze multiple 3945 uncompressed frames for bit rate control purposes, for example. In 3946 some cases, the pre-encoding buffer may not exist; instead, the 3947 sampled pictures are encoded right away. The encoder encodes 3948 pictures from the pre-encoding buffer and stores the output; i.e., 3949 coded pictures, to the encoded picture buffer. The transmitter 3950 encapsulates the coded pictures from the encoded picture buffer to 3951 transmission packets and sends them to a receiver through a 3952 transmission channel. The receiver stores the received packets to 3953 the receiver buffer. The receiver buffering process typically 3954 includes buffering for transmission delay jitter. The receiver 3955 buffer can also be used to recover correct decoding order of coded 3956 data. The decoder reads coded data from the receiver buffer and 3957 produces decoded pictures as output into the decoded picture buffer. 3958 The decoded picture buffer is used to recover the output (or 3959 display) order of pictures. Finally, pictures are displayed. 3961 In the following example figures, I denotes an IDR picture, R 3962 denotes a reference picture, N denotes a non-reference picture, and 3963 the number after I, R, or N indicates the sampling time relative to 3964 the previous IDR picture in decoding order. Values below the 3965 sequence of pictures indicate scaled system clock timestamps. The 3966 system clock is initialized arbitrarily in this example, and time 3967 runs from left to right. Each I, R, and N picture is mapped into 3968 the same timeline compared to the previous processing step, if any, 3969 assuming that encoding, transmission, and decoding take no time. 3970 Thus, events happening at the same time are located in the same 3971 column throughout all example figures. 3973 A subset of a sequence of coded pictures is depicted below in 3974 sampling order. 3976 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ... 3977 ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ... 3978 ... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ... 3980 Figure 16 Sequence of pictures in sampling order 3982 The sampled pictures are buffered in the pre-encoding buffer to 3983 arrange them in encoding order. In this example, we assume that 3984 the non-reference pictures are predicted from both the previous and 3985 the next reference picture in output order, except for the non- 3986 reference pictures immediately preceding an IDR picture, which are 3987 predicted only from the previous reference picture in output order. 3988 Thus, the pre-encoding buffer has to contain at least two pictures, 3989 and the buffering causes a delay of two picture intervals. The 3990 output of the pre-encoding buffering process and the encoding (and 3991 decoding) order of the pictures are as follows: 3993 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 3994 ... -|---|---|---|---|---|---|---|---|- ... 3995 ... 60 61 62 63 64 65 66 67 68 ... 3997 Figure 17 Re-ordered pictures in the pre-encoding buffer 3999 The encoder or the transmitter can set the value of DON for each 4000 picture to a value of DON for the previous picture in decoding 4001 order plus one. 4003 For the sake of simplicity, let us assume that: 4005 o the frame rate of the sequence is constant, 4006 o each picture consists of only one slice, 4007 o each slice is encapsulated in a single NAL unit packet, 4008 o there is no transmission delay, and 4009 o pictures are transmitted at constant intervals (that is, 1 / 4010 (frame rate)). 4012 When pictures are transmitted in decoding order, they are received 4013 as follows: 4015 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4016 ... -|---|---|---|---|---|---|---|---|- ... 4017 ... 60 61 62 63 64 65 66 67 68 ... 4019 Figure 18 Received pictures in decoding order 4021 The OPTIONAL sprop-interleaving-depth media type parameter is set 4022 to 0, as the transmission (or reception) order is identical to the 4023 decoding order. 4025 The decoder has to buffer for one picture interval initially in its 4026 decoded picture buffer to organize pictures from decoding order to 4027 output order as depicted below: 4029 ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... 4030 ... -|---|---|---|---|---|---|---|---|- ... 4031 ... 61 62 63 64 65 66 67 68 69 ... 4033 Figure 19 Output order 4035 The amount of required initial buffering in the decoded picture 4036 buffer can be signaled in the buffering period SEI message or with 4037 the num_reorder_frames syntax element of H.264 video usability 4038 information. num_reorder_frames indicates the maximum number of 4039 frames, complementary field pairs, or non-paired fields that 4040 precede any frame, complementary field pair, or non-paired field in 4041 the sequence in decoding order and that follow it in output order. 4042 For the sake of simplicity, we assume that num_reorder_frames is 4043 used to indicate the initial buffer in the decoded picture buffer. 4044 In this example, num_reorder_frames is equal to 1. 4046 It can be observed that if the IDR picture I00 is lost during 4047 transmission and a retransmission request is issued when the value 4048 of the system clock is 62, there is one picture interval of time 4049 (until the system clock reaches timestamp 63) to receive the 4050 retransmitted IDR picture I00. 4052 Let us then assume that IDR pictures are transmitted two frame 4053 intervals earlier than their decoding position; i.e., the pictures 4054 are transmitted as follows: 4056 ... I00 N58 N59 R03 N01 N02 R06 N04 N05 ... 4057 ... --|---|---|---|---|---|---|---|---|- ... 4058 ... 62 63 64 65 66 67 68 69 70 ... 4060 Figure 20 Interleaving: Early IDR pictures in sending order 4062 The OPTIONAL sprop-interleaving-depth media type parameter is set 4063 equal to 1 according to its definition. (The value of sprop- 4064 interleaving-depth in this example can be derived as follows: 4065 Picture I00 is the only picture preceding picture N58 or N59 in 4066 transmission order and following it in decoding order. Except for 4067 pictures I00, N58, and N59, the transmission order is the same as 4068 the decoding order of pictures. As a coded picture is encapsulated 4069 into exactly one NAL unit, the value of sprop-interleaving-depth is 4070 equal to the maximum number of pictures preceding any picture in 4071 transmission order and following the picture in decoding order.) 4073 The receiver buffering process contains two pictures at a time 4074 according to the value of the sprop-interleaving-depth parameter 4075 and orders pictures from the reception order to the correct 4076 decoding order based on the value of DON associated with each 4077 picture. The output of the receiver buffering process is as 4078 follows: 4080 ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ... 4081 ... -|---|---|---|---|---|---|---|---|- ... 4082 ... 63 64 65 66 67 68 69 70 71 ... 4084 Figure 21 Interleaving: Receiver buffer 4086 Again, an initial buffering delay of one picture interval is needed 4087 to organize pictures from decoding order to output order, as 4088 depicted below: 4090 ... N58 N59 I00 N01 N02 R03 N04 N05 ... 4091 ... -|---|---|---|---|---|---|---|- ... 4092 ... 64 65 66 67 68 69 70 71 ... 4094 Figure 22 Interleaving: Receiver buffer after reordering 4096 Note that the maximum delay that IDR pictures can undergo during 4097 transmission, including possible application, transport, or link 4098 layer retransmission, is equal to three picture intervals. Thus, 4099 the loss resiliency of IDR pictures is improved in systems 4100 supporting retransmission compared to the case in which pictures 4101 were transmitted in their decoding order. 4103 13.4. Robust Transmission Scheduling of Redundant Coded Slices 4105 A redundant coded picture is a coded representation of a picture or 4106 a part of a picture that is not used in the decoding process if the 4107 corresponding primary coded picture is correctly decoded. There 4108 should be no noticeable difference between any area of the decoded 4109 primary picture and a corresponding area that would result from 4110 application of the H.264 decoding process for any redundant picture 4111 in the same access unit. A redundant coded slice is a coded slice 4112 that is a part of a redundant coded picture. 4114 Redundant coded pictures can be used to provide unequal error 4115 protection in error-prone video transmission. If a primary coded 4116 representation of a picture is decoded incorrectly, a corresponding 4117 redundant coded picture can be decoded. Examples of applications 4118 and coding techniques using the redundant codec picture feature 4119 include the video redundancy coding [23] and the protection of "key 4120 pictures" in multicast streaming [24]. 4122 One property of many error-prone video communications systems is 4123 that transmission errors are often bursty. Therefore, they may 4124 affect more than one consecutive transmission packets in 4125 transmission order. In low bit-rate video communication, it is 4126 relatively common that an entire coded picture can be encapsulated 4127 into one transmission packet. Consequently, a primary coded 4128 picture and the corresponding redundant coded pictures may be 4129 transmitted in consecutive packets in transmission order. To make 4130 the transmission scheme more tolerant of bursty transmission errors, 4131 it is beneficial to transmit the primary coded picture and 4132 redundant coded picture separated by more than a single packet. 4133 The DON concept enables this. 4135 13.5. Remarks on Other Design Possibilities 4137 The slice header syntax structure of the H.264 coding standard 4138 contains the frame_num syntax element that can indicate the 4139 decoding order of coded frames. However, the usage of the 4140 frame_num syntax element is not feasible or desirable to recover 4141 the decoding order, due to the following reasons: 4143 o The receiver is required to parse at least one slice header per 4144 coded picture (before passing the coded data to the decoder). 4146 o Coded slices from multiple coded video sequences cannot be 4147 interleaved, as the frame number syntax element is reset to 0 in 4148 each IDR picture. 4150 o The coded fields of a complementary field pair share the same 4151 value of the frame_num syntax element. Thus, the decoding order 4152 of the coded fields of a complementary field pair cannot be 4153 recovered based on the frame_num syntax element or any other 4154 syntax element of the H.264 coding syntax. 4156 The RTP payload format for transport of MPEG-4 elementary streams 4157 [25] enables interleaving of access units and transmission of 4158 multiple access units in the same RTP packet. An access unit is 4159 specified in the H.264 coding standard to comprise all NAL units 4160 associated with a primary coded picture according to subclause 4161 7.4.1.2 of [1]. Consequently, slices of different pictures cannot 4162 be interleaved, and the multi-picture slice interleaving technique 4163 (see section 12.6) for improved error resilience cannot be used. 4165 14. Acknowledgements 4167 Stephan Wenger, Miska Hannuksela, Thomas Stockhammer, Magnus 4168 Westerlund, and David Singer are thanked as the authors of RFC 3984. 4169 Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, 4170 Joerg Ott, and Colin Perkins are thanked for careful review during 4171 the development of RFC 3984. Randell Jesup, Stephen Botzko, Magnus 4172 Westerlund, Alex Eleftheriadis, Thomas Schierl, and Tom Taylor are 4173 thanked for their valuable comments and inputs during the 4174 development of this memo. 4176 This document was prepared using 2-Word-v2.0.template.dot. 4178 15. References 4180 15.1. Normative References 4182 [1] ITU-T Recommendation H.264, "Advanced video coding for 4183 generic audiovisual services", November 2007. 4185 [2] ISO/IEC International Standard 14496-10:2008. 4187 [3] ITU-T Recommendation H.241, "Extended video procedures and 4188 control signals for H.300 series terminals", May 2006. 4190 [4] Bradner, S., "Key words for use in RFCs to Indicate 4191 Requirement Levels", BCP 14, RFC 2119, March 1997. 4193 [5] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, 4194 "RTP: A Transport Protocol for Real-Time Applications", STD 4195 64, RFC 3550, July 2003. 4197 [6] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session 4198 Description Protocol", RFC 4566, July 2006. 4200 [7] Josefsson, S., "The Base16, Base32, and Base64 Data 4201 Encodings", RFC 3548, July 2003. 4203 [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with 4204 Session Description Protocol (SDP)", RFC 3264, June 2002. 4206 [9] Lennox, J., Ott, J., and Schierl, T., "Source-Specific Media 4207 Attributes in the Session Description Protocol", draft-ietf- 4208 mmusic-sdp-source-attributes-02 (work in progress), October 4209 2008. 4211 15.2. Informative References 4213 [10] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special 4214 Issue on H.264/AVC. IEEE Transactions on Circuits and Systems 4215 on Video Technology, July 2003. 4217 [11] Ott, J., Bormann, C., Sullivan, G., Wenger, S., and R. Even 4218 (Ed.), "RTP Payload Format for ITU-T Rec. H.263 Video", RFC 4219 4629, January 2007. 4221 [12] ISO/IEC IS 14496-2. 4223 [13] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and 4224 Systems for Video technology, Vol. 13, No. 7, July 2003. 4226 [14] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", 4227 Proceedings Packet Video Workshop 02, April 2002. 4229 [15] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT 4230 Coding Network Abstraction Layer and IP-based Transport" in 4231 Proc. ICIP 2002, Rochester, NY, September 2002. 4233 [16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 4234 Video Conferences with Minimal Control", STD 65, RFC 3551, 4235 July 2003. 4237 [17] ITU-T Recommendation H.223, "Multiplexing protocol for low 4238 bit rate multimedia communication", July 2001. 4240 [18] Li, A., "RTP Payload Format for Generic Forward Error 4241 Correction", RFC 5109, December 2007. 4243 [19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, 4244 "Video Coding and Transport Layer Techniques for H.264/AVC- 4245 Based Transmission over Packet-Lossy Networks", IEEE 4246 International Conference on Image Processing (ICIP 2003), 4247 Barcelona, Spain, September 2003. 4249 [20] Varsa, V. and M. Karczewicz, "Slice interleaving in 4250 compressed video packetization", Packet Video Workshop 2000. 4252 [21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for 4253 wireless video streaming," International Packet Video 4254 Workshop 2002. 4256 [22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, 4257 available http://ftp3.itu.int/av-arch/video- 4258 site/0201_Gen/JVT-B042.doc, anuary 2002. 4260 [23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 4261 International Workshop on Audio-Visual Services over Packet 4262 Networks, September 1997. 4264 [24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error 4265 Resilient Video Coding Using Unequally Protected Key 4266 Pictures", in Proc. International Workshop VLBV03, September 4267 2003. 4269 [25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., 4270 and P. Gentric, "RTP Payload Format for Transport of MPEG-4 4271 Elementary Streams", RFC 3640, November 2003. 4273 [26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 4274 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 4275 RFC 3711, March 2004. 4277 [27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time 4278 Streaming Protocol (RTSP)", RFC 2326, April 1998. 4280 [28] Handley, M., Perkins, C., and E. Whelan, "Session 4281 Announcement Protocol", RFC 2974, October 2000. 4283 [29] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, 4284 January 2008. 4286 [30] Wenger, S., Chandra, U., and M. Westerlund, "Codec Control 4287 Messages in the RTP Audio-Visual Profile with Feedback 4288 (AVPF)", RFC 5104, February 2008. 4290 16. Authors' Addresses 4292 Ye-Kui Wang 4293 Huawei Technologies 4294 400 Somerset Corp Blvd, Suite 602 4295 Bridgewater, NJ 08807 4296 USA 4298 Phone: +1-908-541-3518 4299 EMail: yekuiwang@huawei.com 4301 Roni Even 4302 14 David Hamelech 4303 Tel Aviv 64953 4304 Israel 4306 Phone: +972-545481099 4307 Email: ron.even.tlv@gmail.com 4308 Tom Kristensen 4309 TANDBERG 4310 Philip Pedersens vei 22 4311 N-1366 Lysaker 4312 Norway 4314 Phone: +47 67125125 4315 Email: tom.kristensen@tandberg.com, tomkri@ifi.uio.no 4317 17. Backward Compatibility to RFC 3984 4319 The current document is a revision of RFC 3984 and intends to 4320 obsolete it. This section addresses the backward compatibility 4321 issues. 4323 The technical changes are listed in section 18. 4325 Items 1), 2), 3), 7), 9), 10), 12), 13) are bug-fix type of changes, 4326 and do not incur any backward compatibility issues. 4328 Item 4), addition of six new media type parameters, does not incur 4329 any backward compatibility issues for SDP Offer/Answer based 4330 applications, as legacy RFC 3984 receivers ignore these parameters, 4331 and it is fine for legacy RFC 3984 senders not to use these 4332 parameters as they are optional. However, there is a backward 4333 compatibility issue for SDP declarative usage based applications, 4334 e.g. those using RTSP and SAP, because the SDP receiver per RFC 4335 3984 cannot accept a session for which the SDP includes an 4336 unrecognized parameter. Therefore, the RTSP or SAP server may have 4337 to prepare two sets of streams, one for legacy RFC 3984 receivers 4338 and one for receivers according to this memo. 4340 Items 5), 6) and 11) are related to out-of-band transport of 4341 parameter sets. There are following backward compatibility issues. 4343 1) When a legacy sender per RFC 3984 includes parameter sets for a 4344 level different than the default level indicated by profile- 4345 level-id to sprop-parameter-sets, the parameter value of sprop- 4346 parameter-sets is invalid to the receiver per this memo and 4347 therefore the session may be rejected. 4349 2) In SDP Offer/Answer between a legacy offerer per RFC 3984 and an 4350 answerer per this memo, when the answerer includes in the answer 4351 parameter sets that are not a superset of the parameter sets 4352 included in the offer, the parameter value of sprop-parameter- 4353 sets is invalid to offerer and the session may not be initiated 4354 properly (related to change item 11)). 4356 3) When one endpoint A per this memo includes in-band-parameter- 4357 sets equal to 1, the other side B per RFC 3984 does not 4358 understand that it must transmit parameter sets in-band and B 4359 may still exclude parameter sets in the in-band stream it is 4360 sending. Consequently endpoint A cannot decode the stream it 4361 receives. 4363 Item 7), allowance of conveying sprop-parameter-sets and sprop- 4364 level-parameter-sets using the "fmtp" source attribute as specified 4365 in section 6.3 of [9], is similar as item 4). It does not incur 4366 any backward compatibility issues for SDP Offer/Answer based 4367 applications, as legacy RFC 3984 receivers ignore the "fmtp" source 4368 attribute, and it is fine for legacy RFC 3984 senders not to use 4369 the "fmtp" source attribute as it is optional. However, there is a 4370 backward compatibility issue for SDP declarative usage based 4371 applications, e.g. those using RTSP and SAP, because the SDP 4372 receiver per RFC 3984 cannot accept a session for which the SDP 4373 includes an unrecognized parameter (i.e., the "fmtp" source 4374 attribute). Therefore, the RTSP or SAP server may have to prepare 4375 two sets of streams, one for legacy RFC 3984 receivers and one for 4376 receivers according to this memo. 4378 Item 14) removed that use of out-of-band transport of parameter 4379 sets is recommended. As out-of-band transport of parameter sets is 4380 still allowed, this change does not incur any backward 4381 compatibility issues. 4383 Item 15) does not incur any backward compatibility issues as the 4384 added subsection 8.5 is informative. 4386 18. Changes from RFC 3984 4388 Following is the list of technical changes (including bug fixes) 4389 from RFC 3984. Besides this list of technical changes, numerous 4390 editorial changes have been made, but not documented in this memo. 4392 1) In subsections 5.4, 5.5, 6.2, 6,3 and 6.4, removed that the 4393 packetization mode in use may be signaled by external means. 4395 2) In subsection 7.2.2, changed the sentence 4397 There are N VCL NAL units in the deinterleaving buffer. 4399 to 4401 There are N or more VCL NAL units in the de-interleaving buffer. 4403 3) In subsection 8.1, the semantics of sprop-init-buf-time, 4404 paragraph 2, changed the sentence 4406 The parameter is the maximum value of (transmission time of a 4407 NAL unit - decoding time of the NAL unit), assuming reliable and 4408 instantaneous transmission, the same timeline for transmission 4409 and decoding, and that decoding starts when the first packet 4410 arrives. 4412 to 4414 The parameter is the maximum value of (decoding time of the NAL 4415 unit - transmission time of a NAL unit), assuming reliable and 4416 instantaneous transmission, the same timeline for transmission 4417 and decoding, and that decoding starts when the first packet 4418 arrives. 4420 4) Added six new media type parameters, namely max-smbps, sprop- 4421 level-parameter-sets, use-level-src-parameter-sets, in-band- 4422 parameter-sets, sar-understood and sar-supported. 4424 5) In subsection 8.1, removed the specification of parameter-add. 4425 Other descriptions of parameter-add (in subsections 8.2 and 8.4) 4426 are also removed. 4428 6) In subsection 8.1, added a constraint to sprop-parameter-sets 4429 such that it can only contain parameter sets for the same 4430 profile and level as indicated by profile-level-id. 4432 7) In subsection 8.2.1, added that sprop-parameter-sets and sprop- 4433 level-parameter-sets may be either included in the "a=fmtp" line 4434 of SDP or conveyed using the "fmtp" source attribute as 4435 specified in section 6.3 of [9]. 4437 8) In subsection 8.2.2, removed sprop-deint-buf-req from being part 4438 of the media format configuration in usage with the SDP 4439 Offer/Answer model. 4441 9) In subsection 8.2.2, made it clear that level is downgradable in 4442 the SDP Offer/Answer model, i.e. the use of the level part of 4443 "profile-level-id" does not need to be symmetric (the level 4444 included in the answer can be lower than or equal to the level 4445 included in the offer). 4447 10)In subsection 8.2.2, removed that the capability parameters may 4448 be used to declare encoding capabilities. 4450 11)In subsection 8.2.2, added rules on how to use sprop-parameter- 4451 sets and sprop-level-parameter-sets for out-of-band transport of 4452 parameter sets, with or without level downgrading. 4454 12)In subsection 8.2.2, clarified the rules of using the media type 4455 parameters with SDP Offer/Answer for multicast. 4457 13)In subsection 8.2.2, completed and corrected the list of how 4458 different media type parameters shall be interpreted in the 4459 different combinations of offer or answer and direction 4460 attribute. 4462 14)In subsection 8.4, changed the text such that both out-of-band 4463 and in-band transport of parameter sets are allowed and neither 4464 is recommended or required. 4466 15)Added subsection 8.5 (informative) providing example methods for 4467 decoder refresh to handle parameter set losses.