idnits 2.17.1 draft-ietf-avtext-avpf-ccm-layered-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC5104, but the abstract doesn't seem to directly say this. It does mention RFC5104 though, so this could be OK. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC5104, updated by this document, for RFC5378 checks: 2006-08-29) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (May 17, 2016) is 2901 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'I-D.ietf-mmusic-rid' is defined on line 391, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-mmusic-sdp-simulcast' is defined on line 397, but no explicit reference was found in the text == Outdated reference: A later version (-15) exists of draft-ietf-mmusic-rid-05 == Outdated reference: A later version (-14) exists of draft-ietf-mmusic-sdp-simulcast-04 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Wenger 3 Internet-Draft J. Lennox 4 Updates: 5104 (if approved) Vidyo, Inc. 5 Intended status: Standards Track B. Burman 6 Expires: November 18, 2016 M. Westerlund 7 Ericsson 8 May 17, 2016 10 Using Codec Control Messages in the RTP Audio-Visual Profile with 11 Feedback with Layered Codecs 12 draft-ietf-avtext-avpf-ccm-layered-01 14 Abstract 16 This document fixes a shortcoming in the specification language of 17 the Codec Control Message Full Intra Request (FIR) as defined in 18 RFC5104 when using with layered codecs. In particular, a Decoder 19 Refresh Point needs to be sent by a media sender when a FIR is 20 received on any layer of the layered bitstream, regardless on whether 21 those layers are being sent in a single or in multiple RTP flows. 22 The other payload-specific feedback messages defined in RFC 5104 and 23 RFC 4585 as updated by RFC 5506 have also been analyzed, and no 24 corresponding shortcomings have been found. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on November 18, 2016. 43 Copyright Notice 45 Copyright (c) 2016 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction and Problem Statement . . . . . . . . . . . . . 2 61 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4 62 3. Updated definition of Decoder Refresh Point . . . . . . . . . 4 63 4. Full Intra Request for Layered Codecs . . . . . . . . . . . . 4 64 5. Identifying the use of Layered Codecs (Informative) . . . . . 5 65 6. Layered Codecs and non-FIR codec control messages 66 (Informative) . . . . . . . . . . . . . . . . . . . . . . . . 6 67 6.1. Picture Loss Indication (PLI) . . . . . . . . . . . . . . 6 68 6.2. Slice Loss Indication (SLI) . . . . . . . . . . . . . . . 6 69 6.3. Reference Picture Selection Indication (RPSI) . . . . . . 6 70 6.4. Temporal-Spatial Trade-off Request and Notification 71 (TSTR/TSTN) . . . . . . . . . . . . . . . . . . . . . . . 7 72 6.5. H.271 Video Back Channel Message (VBCM) . . . . . . . . . 7 73 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 74 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 75 9. Security Considerations . . . . . . . . . . . . . . . . . . . 8 76 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 77 10.1. Normative References . . . . . . . . . . . . . . . . . . 8 78 10.2. Informative References . . . . . . . . . . . . . . . . . 8 79 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 10 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 82 1. Introduction and Problem Statement 84 RFC 4585 [RFC4585] and RFC 5104 [RFC5104] specify a number of 85 payload-specific feedback messages which a media receiver can use to 86 inform a media sender of certain conditions, or make certain 87 requests. The feedback messages are being sent as RTCP receiver 88 reports, and RFC 4585 specifies timing rules that make the use of 89 those messages practical for time-sensitive codec control. 91 Since the time those RFCs were developed, layered codecs have gained 92 in popularity and deployment. Layered codecs use multiple sub- 93 bitstreams called layers to represent the content in different 94 fidelities. Depending on the media codec and its RTP payload format 95 in use, single layers or groups of layers may be sent in their own 96 RTP streams (in MRST or MRMT mode as defined in RFC 7656 [RFC7656]), 97 or multiplexed (using media-codec specific multiplexing mechanisms) 98 in a single RTP stream (SRST mode as defined in RFC 7656 [RFC7656]). 99 The dependency relationship between layers forms a directed graph, 100 with the base layer at the root. Enhancement layers depend on the 101 base layer and potentially on other enhancement layers, and the 102 target layer and all layers it depends on have to be decoded jointly 103 in order to re-create the uncompressed media signal at the fidelity 104 of the target layer. 106 Implementation experience has shown that the Full Intra Request 107 command as defined in RFC 5104 [RFC5104] is underspecified when used 108 with layered codecs and when more than one RTP stream is used to 109 transport the layers of a layered bitstream at a given fidelity. In 110 particular, from the RFC 5104 [RFC5104] specification language it is 111 not clear whether an FIR received for only a single RTP stream of 112 multiple RTP streams covering the same layered bitstream necessarily 113 triggers the sending of a Decoder Refresh Point (as defined in 114 RFC 5104 [RFC5104] section 2.2) for all layers, or only for the layer 115 which is transported in the RTP stream which the FIR request is 116 associated with. 118 This document fixes this shortcoming by: 120 a. Updating the definition of the Decoder Refresh Point (as defined 121 in RFC 5104 [RFC5104] section 2.2) to cover layered codecs, in 122 line with the corresponding definitions used in a popular layered 123 codec format, namely H.264/SVC [H.264]. Specifically, a decoder 124 refresh point, in conjunction with layered codecs, resets the 125 state of the whole decoder, which implies that it includes hard 126 or gradual single-layer decoder refresh for all layers; 128 b. Requiring that, when a media sender receives a Full Intra Request 129 over the RTCP stream associated with any of the RTP streams over 130 which a part of the layered bitstream is transported, to send a 131 Decoder Refresh Point; 133 c. Require that a media receiver sends the FIR on the RTCP stream 134 associated with the base layer (the option of receiving FIR on 135 enhancement layer-associated RTCP stream as specified in point b) 136 above is kept for backward compatibility); and 138 d. Providing guidance on how to detect that a layered codec is in 139 use for which the above rules apply. 141 While, clearly, the reaction to FIR for layered codecs in RFC 5104 142 [RFC5104] and companion documents is underspecified, it appears that 143 this is not the case for any of the other payload-specific codec 144 control messages defined in any of RFC 4585 [RFC4585], RFC 5104 145 [RFC5104], or RFC 5506 [RFC5506]. A brief summary of the analysis 146 that led to this conclusion is also included in this document. 148 2. Requirements Language 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in RFC 2119 [RFC2119]. 154 3. Updated definition of Decoder Refresh Point 156 The text below updates the definition of Decoder Refresh Point in 157 section 2.2 of RFC 5104 [RFC5104]. 159 Decoder Refresh Point: A bit string, packetized in one or more RTP 160 packets, that completely resets the decoder to a known state. 162 Examples for "hard" single layer decoder refresh points are Intra 163 pictures in H.261 [H.261], H.263 [H.263], MPEG-1 [MPEG-1], MPEG-2 164 [MPEG-2], and MPEG-4 [MPEG-4]; Instantaneous Decoder Refresh (IDR) 165 pictures in H.264 [H.264], and H.265 [H.265]; and Keyframes in VP8 166 [RFC6386] and VP9 [I-D.grange-vp9-bitstream]. "Gradual" decoder 167 refresh points may also be used; see for example H.264 [H.264]. 168 While both "hard" and "gradual" decoder refresh points are acceptable 169 in the scope of this specification, in most cases the user experience 170 will benefit from using a "hard" decoder refresh point. 172 A decoder refresh point also contains all header information above 173 the syntactical level of the picture layer (or equivalent, depending 174 on the video compression standard) that is conveyed in-band. In 175 H.264 [H.264], for example, a decoder refresh point contains 176 parameter set Network Adaptation Layer (NAL) units that generate 177 parameter sets necessary for the decoding of the following slice/data 178 partition NAL units (and that are not conveyed out of band). 180 When a layered codec is in use, the above definition (and, in 181 particular, the requirement to COMPLETELY reset the decoder to a 182 known state) implies that the decoder refresh point includes hard or 183 gradual single layer decoder refresh points for all layers. 185 4. Full Intra Request for Layered Codecs 187 When a media receiver or middlebox has decided to send a FIR command 188 (based on the guidance provided in Section 4.3.1 of RFC 5104 189 [RFC5104], it MUST do so in the RTCP stream related to the forward 190 RTP stream that carries the base layer of the layered bitstream, and 191 the Feedback Control Information (FCI, and in particular the SSRC 192 field therein) MUST also refer to the forward RTP stream that carries 193 the base layer. 195 When a Full Intra Request Command is received by the designated media 196 sender in the RTCP stream associated with any of the RTP streams in 197 which any layer of a layered bitstream are sent, the designated media 198 sender MUST send a Decoder Refresh Point (Section 3) as defined above 199 at its earliest opportunity. The requirements related to congestion 200 control on the forward RTP streams as specified in sections 3.5.1.5 201 of RFC 5104 [RFC5104] apply for the RTP streams both in isolation and 202 combined. 204 Note: the requirement to react to FIR commands associated with 205 enhancement layers is included for robustness and backward 206 compatibility reasons. 208 5. Identifying the use of Layered Codecs (Informative) 210 The above modifications to RFC 5104 unambiguously define how to deal 211 with FIR when layered bitstreams are in use. However, it is 212 surprisingly difficult to identify this situation. In general, it is 213 expected that implementers know when layered coding (in its commonly 214 understood sense: with inter-layer prediction between pyramided- 215 arranged layers) is in use and when not, and can therefore implement 216 the above updates to RFC 5104 correctly. However, there are use 217 cases of the use of layered codecs that may be viewed as somewhat 218 exotic today but clearly are supported by the video coding syntax, in 219 which the above rules would lead to suboptimal system behavior. 220 Nothing would break, and there would not be an interop failure, but 221 the user experience may suffer through the sending or receiving of 222 Decoder Refresh Points at times or on parts of the bitstream that are 223 unnecessary from a user experience viewpoint. Therefore, this 224 informative section is included that provides the current 225 understanding of when a layered codec is in use and when not. 227 The key observation made here is that the RTP payload format 228 negotiated for the RTP streams, in isolation, is not necessarily an 229 indicator for the use of layering. Some layered codecs (including 230 H.264/SVC) can form decodable bitstreams including only (one or more) 231 enhancement layers, without the base layer, effectively creating 232 simulcastable sub-bitstreams in a scalable bitstream that does not 233 take advantage of inter-layer prediction. In such a scenario, it is 234 potentially (though not necessarily) unnecessary--or even counter- 235 productive--to send a decoder refresh point on all RTP streams using 236 that payload format and SSRC. 238 One good indication of the likely use of layering with interlayer 239 prediction is when the various RTP streams are "bound" together on 240 the signaling level. In an SDP environment, this would be the case 241 if they are marked as being dependent from each other using the 242 grouping framework RFC 4588 [RFC4588] and the layer dependency 243 RFC 5583 [RFC5583]. 245 6. Layered Codecs and non-FIR codec control messages (Informative) 247 Between them, RFC 4585 [RFC4585] (as updated by RFC 5506 [RFC5506]) 248 and RFC 5104 [RFC5104] define a total of seven Payload-specific 249 Feedback messages. For the FIR command message, guidance has been 250 provided above. In this section, some information is provided with 251 respect to the remaining six codec control messages. 253 6.1. Picture Loss Indication (PLI) 255 PLI is defined in RFC 4585 [RFC4585] section 6.3.1. The prudent 256 response to a PLI message received for an enhancement layer is to 257 "repair" (through whatever source-coding specific means) that 258 enhancement layer and all dependent enhancement layers, but not the 259 reference layer(s) used by the enhancement layer for which the PLI 260 was received. The encoder can figure out by itself what constitutes 261 a dependent enhancement layer and does not need help from the system 262 stack in doing so. Insofar, there is nothing that needs to be 263 specified herein. 265 6.2. Slice Loss Indication (SLI) 267 SLI is defined in RFC 4585 [RFC4585] section 6.3.2. The authors' 268 current understanding is that the prudent response to a SLI message 269 received for an enhancement layer is to "repair" (through whatever 270 source-coding specific means) the affected spatial area of that 271 enhancement layer and all dependent enhancement layers, but not the 272 reference layers used by the enhancement layer for which the SLI was 273 received. The encoder can figure out by itself what constitutes a 274 dependent enhancement layer and does not need help from the system 275 stack in doing so. Insofar, there is nothing that needs to be 276 specified herein. SLI has seen very little implementation and, as 277 far as it is known, none in conjunction with layered systems. 279 6.3. Reference Picture Selection Indication (RPSI) 281 RPSI is defined in RFC 4585 [RFC4585] section 6.3.3. While a 282 technical equivalent of RPSI has been in use with non-layered systems 283 for many years, no implementations are known in conjunction of 284 layered codecs. The authors' current understanding is that the 285 reception of an RPSI message on any layer forces the encoder to 286 "repair" the bitstream on that layer and all dependent layers without 287 the need of any system-provided guidance. Insofar, RPSI should work 288 without further need for specification language. 290 6.4. Temporal-Spatial Trade-off Request and Notification (TSTR/TSTN) 292 TSTN/TSTR are defined in RFC 5104 [RFC5104] section 4.3.2 and 4.3.3, 293 respectively. The TSTR request allows to communicate (typically 294 user-interface-obtained) guidance of the preferred trade-off between 295 spatial quality and frame rate. A technical equivalent of TSTN/TSTR 296 has seen deployment for many years in non-scalable systems. 298 The Temporal-Spatial Trade-off request and notification messages 299 include an SSRC target, which (similarly to FIR) may refer to an RTP 300 stream carrying a base layer, an enhancement layer, or multiple 301 layers. Therefore, the authors' current understanding is that the 302 semantics of the message applies to the layers present in the 303 targeted RTP stream. 305 It is noted that per-layer TSTR/TSTN is a mechanism that is, in some 306 ways, counterproductive in a system using layered codecs. Given a 307 sufficiently complex layered bitstream layout, a sending system has 308 flexibility in adjusting the spatio/temporal quality balance by 309 adding and removing temporal, spatial, or quality enhancement layers. 310 At present it is unclear whether an allowed (or even recommended) 311 option to the reception of a TSTR is to adjust the bit allocation 312 within the layer(s) present in the addressed RTP stream, or to adjust 313 the layering structure accordingly--which can involve more than just 314 the addressed RTP stream. 316 Until there is a sufficient critical mass of implementation practice, 317 it is probably prudent for an implementer not to assume either of the 318 two options (or any middleground that may exist between the two), be 319 liberal in accepting TSTR messages, perhaps responding in TSTN 320 indicating "no change," not sending TSTR messages except when 321 operating in SRST mode as defined in RFC 7656 [RFC7656], and 322 contribute to the IETF documentation of any implementation 323 requirements that make per-layer TSTR/TSTN useful. 325 6.5. H.271 Video Back Channel Message (VBCM) 327 VBCM is defined in RFC 5104 [RFC5104] section 4.3.4. What was said 328 above for RPSI (Section 6.3) applies here as well. 330 7. Acknowledgements 332 The authors want to thank Mo Zanaty for useful discussions. 334 8. IANA Considerations 336 This memo includes no request to IANA. 338 9. Security Considerations 340 The security considerations of RFC 4585 [RFC4585] (as updated by 341 RFC 5506 [RFC5506]) and RFC 5104 [RFC5104] apply. The clarified 342 response to FIR does not require any updates. 344 10. References 346 10.1. Normative References 348 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 349 Requirement Levels", BCP 14, RFC 2119, 350 DOI 10.17487/RFC2119, March 1997, 351 . 353 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 354 "Extended RTP Profile for Real-time Transport Control 355 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 356 DOI 10.17487/RFC4585, July 2006, 357 . 359 [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, 360 "Codec Control Messages in the RTP Audio-Visual Profile 361 with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, 362 February 2008, . 364 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 365 Real-Time Transport Control Protocol (RTCP): Opportunities 366 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 367 2009, . 369 10.2. Informative References 371 [H.261] ITU-T, "ITU-T Rec. H.261: Video codec for audiovisual 372 services at p x 64 kbit/s", 1993, 373 . 375 [H.263] ITU-T, "ITU-T Rec. H.263: Video coding for low bit rate 376 communication", 2005, 377 . 379 [H.264] ITU-T, "ITU-T Rec. H.264: Advanced video coding for 380 generic audiovisual services", 2014, 381 . 383 [H.265] ITU-T, "ITU-T Rec. H.265: High efficiency video coding", 384 2015, . 386 [I-D.grange-vp9-bitstream] 387 Grange, A. and H. Alvestrand, "A VP9 Bitstream Overview", 388 draft-grange-vp9-bitstream-00 (work in progress), February 389 2013. 391 [I-D.ietf-mmusic-rid] 392 Thatcher, P., Zanaty, M., Nandakumar, S., Burman, B., 393 Roach, A., and B. Campen, "RTP Payload Format 394 Constraints", draft-ietf-mmusic-rid-05 (work in progress), 395 March 2016. 397 [I-D.ietf-mmusic-sdp-simulcast] 398 Burman, B., Westerlund, M., Nandakumar, S., and M. Zanaty, 399 "Using Simulcast in SDP and RTP Sessions", draft-ietf- 400 mmusic-sdp-simulcast-04 (work in progress), February 2016. 402 [MPEG-1] ISO/IEC, "ISO/IEC 11172-2:1993 Information technology -- 403 Coding of moving pictures and associated audio for digital 404 storage media at up to about 1,5 Mbit/s -- Part 2: Video", 405 1993. 407 [MPEG-2] ISO/IEC, "ISO/IEC 13818-2:2013 Information technology -- 408 Generic coding of moving pictures and associated audio 409 information -- Part 2: Video", 2013. 411 [MPEG-4] ISO/IEC, "ISO/IEC 14496-2:2004 Information technology -- 412 Coding of audio-visual objects -- Part 2: Visual", 2004. 414 [RFC4588] Rey, J., Leon, D., Miyazaki, A., Varsa, V., and R. 415 Hakenberg, "RTP Retransmission Payload Format", RFC 4588, 416 DOI 10.17487/RFC4588, July 2006, 417 . 419 [RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding 420 Dependency in the Session Description Protocol (SDP)", 421 RFC 5583, DOI 10.17487/RFC5583, July 2009, 422 . 424 [RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., 425 Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding 426 Guide", RFC 6386, DOI 10.17487/RFC6386, November 2011, 427 . 429 [RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and 430 B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms 431 for Real-Time Transport Protocol (RTP) Sources", RFC 7656, 432 DOI 10.17487/RFC7656, November 2015, 433 . 435 Appendix A. Change Log 437 NOTE TO RFC EDITOR: Please remove this section prior to publication. 439 draft-wenger-avtext-avpf-ccm-layered-00-00: initial version 441 draft-ietf-avtext-avpf-ccm-layered-00: resubmit as avtext WG draft 442 per IETF95 and list confirmation by Rachel 4/25/2016 444 draft-ietf-avtext-avpf-ccm-layered-00: In section "Identifying the 445 use of Layered Codecs (Informative)", removed last sentence that 446 could be misread that the explicit signaling of simulcasting in 447 conjunction with payload formats supporting layered coding implies no 448 layering. 450 Authors' Addresses 452 Stephan Wenger 453 Vidyo, Inc. 455 Email: stewe@stewe.org 457 Jonathan Lennox 458 Vidyo, Inc. 460 Email: jonathan@vidyo.com 462 Bo Burman 463 Ericsson 464 Kistavagen 25 465 SE - 164 80 Kista 466 Sweden 468 Email: bo.burman@ericsson.com 469 Magnus Westerlund 470 Ericsson 471 Farogatan 6 472 SE- 164 80 Kista 473 Sweden 475 Phone: +46107148287 476 Email: magnus.westerlund@ericsson.com