idnits 2.17.1 draft-ietf-codec-ambisonics-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC7845, but the abstract doesn't seem to directly say this. It does mention RFC7845 though, so this could be OK. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 142 has weird spacing: '... order n = ...' == Line 143 has weird spacing: '... degree m = k...' -- The document date (June 19, 2018) is 2137 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 codec J. Skoglund 3 Internet-Draft Google LLC 4 Updates: 7845 (if approved) M. Graczyk 5 Intended status: Standards Track June 19, 2018 6 Expires: December 21, 2018 8 Ambisonics in an Ogg Opus Container 9 draft-ietf-codec-ambisonics-07 11 Abstract 13 This document defines an extension to the Opus audio codec to 14 encapsulate coded ambisonics using the Ogg format. It also contains 15 updates to RFC 7845 to reflect necessary changes in the description 16 of channel mapping families. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on December 21, 2018. 35 Copyright Notice 37 Copyright (c) 2018 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (https://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 54 3. Ambisonics With Ogg Opus . . . . . . . . . . . . . . . . . . 3 55 3.1. Channel Mapping Family 2 . . . . . . . . . . . . . . . . 3 56 3.2. Channel Mapping Family 3 . . . . . . . . . . . . . . . . 4 57 4. Downmixing . . . . . . . . . . . . . . . . . . . . . . . . . 6 58 5. Updates to RFC 7845 . . . . . . . . . . . . . . . . . . . . . 6 59 5.1. Format of the Channel Mapping Table . . . . . . . . . . . 6 60 5.2. Unknown Mapping Families . . . . . . . . . . . . . . . . 7 61 6. Experimental Mapping Families . . . . . . . . . . . . . . . . 8 62 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8 63 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 64 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 9 65 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 66 10.1. Normative References . . . . . . . . . . . . . . . . . . 9 67 10.2. Informative References . . . . . . . . . . . . . . . . . 9 68 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 70 1. Introduction 72 Ambisonics is a representation format for three dimensional sound 73 fields which can be used for surround sound and immersive virtual 74 reality playback. See [gerzon75] and [daniel04] for technical 75 details on the ambisonics format. For the purposes of the this 76 document, ambisonics can be considered a multichannel audio stream. 77 A separate stereo stream can be used alongside the ambisonics in a 78 head-tracked virtual reality experience to provide so-called non- 79 diegetic audio - audio which should remain unchanged by listener head 80 rotation; e.g., narration or stereo music. Ogg is a general purpose 81 container, supporting audio, video, and other media. It can be used 82 to encapsulate audio streams coded using the Opus codec. See 83 [RFC6716] and [RFC7845] for technical details on the Opus codec and 84 its encapsulation in the Ogg container respectively. 86 This document extends the Ogg Opus format by defining two new channel 87 mapping families for encoding ambisonics. The Ogg Opus format is 88 extended indirectly by adding an item with value 2 or 3 to the IANA 89 "Opus Channel Mapping Families" registry. When 2 or 3 are used as 90 the Channel Mapping Family Number in an Ogg stream, the semantic 91 meaning of the channels in the multichannel Opus stream is one of the 92 ambisonics layouts defined in this document. This mapping can also 93 be used in other contexts which make use of the channel mappings 94 defined by the Opus Channel Mapping Families registry. Furthermore, 95 mapping families 240 through 254 (inclusively) are reserved for 96 experimental use. 98 2. Terminology 100 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 101 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 102 "OPTIONAL" in this document are to be interpreted as described in BCP 103 14 [RFC2119] [RFC8174] when, and only when, they appear in all 104 capitals, as shown here. 106 3. Ambisonics With Ogg Opus 108 Ambisonics can be encapsulated in the Ogg format by encoding with the 109 Opus codec and setting the channel mapping family value to 2 or 3 in 110 the Ogg identification header (ID). A demuxer implementation 111 encountering Channel Mapping Family 2 or Family 3 MUST interpret the 112 Opus stream as containing ambisonics with the format described in 113 Section 3.1 or Section 3.2, respectively. 115 3.1. Channel Mapping Family 2 117 Allowed numbers of channels: (1 + n)^2 + 2j for n = 0...14 and j = 0 118 or 1, where n denotes the (highest) ambisonic order and j denotes 119 whether or not there is a separate non-diegetic stereo stream. This 120 corresponds to periphonic ambisonics from zeroth to fourteenth order 121 plus potentially two channels of non-diegetic stereo. Explicitly the 122 allowed number of channels are 1, 3, 4, 6, 9, 11, 16, 18, 25, 27, 36, 123 38, 49, 51, 64, 66, 81, 83, 100, 102, 121, 123, 144, 146, 169, 171, 124 196, 198, 225, and 227. 126 This channel mapping uses the same channel mapping table format used 127 by channel mapping family 1. The output channels are ambisonic 128 components ordered in Ambisonic Channel Number (ACN) order, defined 129 in Figure 1, followed by two optional channels of non-diegetic stereo 130 indexed (left, right). The terms order and degree are defined 131 according to [ambix]. 133 ACN = n * (n + 1) + m, 134 for order n and degree m. 136 Figure 1: Ambisonic Channel Number (ACN) 138 For the ambisonic channels the ACN component corresponds to channel 139 index as k = ACN. The reverse correspondence can also be computed 140 for an ambisonic channel with index k. 142 order n = floor(sqrt(k)), 143 degree m = k - n * (n + 1). 145 Figure 2: Ambisonic Degree and Order from ACN 147 Note that channel mapping family 2 allows for so-called mixed order 148 ambisonic representation where only a subset of the full ambisonic 149 order number of channels is encoded. By specifying the full number 150 in the channel count field, the inactive ACNs can then be indicated 151 in the channel mapping field using the index 255. 153 Ambisonic channels are normalized with Schmidt Semi-Normalization 154 (SN3D). The interpretation of the ambisonics signal as well as 155 detailed definitions of ACN channel ordering and SN3D normalization 156 are described in [ambix] Section 2.1. 158 3.2. Channel Mapping Family 3 160 Allowed numbers of channels: (1 + n)^2 + 2j for n = 0...14 and j = 0 161 or 1, where n denotes the (highest) ambisonic order and j denotes 162 whether or not there is a separate non-diegetic stereo stream. This 163 corresponds to periphonic ambisonics from zeroth to fourteenth order 164 plus potentially two channels of non-diegetic stereo. Explicitly the 165 allowed number of channels are 1, 3, 4, 6, 9, 11, 16, 18, 25, 27, 36, 166 38, 49, 51, 64, 66, 81, 83, 100, 102, 121, 123, 144, 146, 169, 171, 167 196, 198, 225, and 227. 169 In this mapping, C output channels (the channel count) are generated 170 at the decoder by multiplying K = N + M decoded channels with a 171 designated demixing matrix, D, having C rows and K columns. Here, N 172 denotes the number of streams encoded and M the number of these which 173 are coupled to produce two channels. As for channel mapping family 2 174 this mapping family also allows for encoding and decoding of full 175 order ambisonics, mixed order ambisonics, and for non-diegetic stereo 176 channels, but also has the added flexibility of mixing channels. Let 177 X denote a column vector containing K decoded channels X1, X2, ..., 178 XK (from N streams), and let S denote a column vector containing C 179 output streams S1, S2, ..., SC. Then S = D X, i.e., 181 / \ / \ / \ 182 | S1 | | D11 D12 ... D1K | | X1 | 183 | S2 | | D21 D22 ... D2K | | X2 | 184 | ... | = | ... ... ... ... | | ... | 185 | SC | | DC1 DC2 ... DCK | | XK | 186 \ / \ / \ / 188 Figure 3: Demixing in Channel Mapping Family 3 190 The matrix MUST be provided as side information and MUST be stored in 191 the channel mapping table part of the identification header, c.f. 192 section 5.1.1 in [RFC7845]. The matrix replaces the need for a 193 channel mapping field and for channel mapping family 3 the mapping 194 table has the following layout: 196 0 1 2 3 197 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 198 +-+-+-+-+-+-+-+-+ 199 | Stream Count | 200 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 201 | Coupled Count | Demixing Matrix : 202 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 204 Figure 4: Channel Mapping Table for Channel Mapping Family 3 206 The fields in the channel mapping table have the following meaning: 208 1. Stream Count 'N' (8 bits, unsigned): 210 This is the total number of streams encoded in each Ogg packet. 212 2. Coupled Stream Count 'M' (8 bits, unsigned): 214 This is the number of the N streams whose decoders are to be 215 configured to produce two channels (stereo). 217 3. Demixing Matrix (16*K*C bits, signed): 219 The coefficients of the demixing matrix stored column-wise as 220 16-bit, signed, two's complement fixed-point values with 15 221 fractional bits (Q15), little endian. If needed, the output gain 222 field can be used for a normalization scale. For mixed order 223 ambisonic representations, the silent ACN channels are indicated 224 by all zeros in the corresponding rows of the mixing matrix. 225 This allows also for mixed order with non-diegetic stereo as the 226 number of columns implies the presence of non-diegetic channels. 228 Note that [RFC7845] specifies that the identification header cannot 229 exceed one "page", which is 65,025 octets. This limits the ambisonic 230 order to be lower than 12, if full order is utilized and the number 231 of coded streams is the same as the ambisonic order plus the two non- 232 diegetic channels. The total output channel number, C, MUST be set 233 in the 3rd field of the identification header. 235 4. Downmixing 237 The downmixing matrices in this section are only examples known to 238 give acceptable results for stereo downmixing from ambisonics, but 239 other mixing strategies will be allowed, e.g., to emphasize a certain 240 panning. 242 An Ogg Opus player MAY use the matrix in Figure 5 to implement 243 downmixing from multichannel files using Channel Mapping Family 2 and 244 3, when there is no non-diegetic stereo. The first and second 245 ambisonic channels are known as "W" and "Y" respectively. 247 / \ / \ / \ 248 | L | | 0.5 0.5 0.0 ... | | W | 249 | R | = | 0.5 -0.5 0.0 ... | | Y | 250 \ / \ / | ... | 251 \ / 253 Figure 5: Stereo Downmixing Matrix for Channel Mapping Family 2 and 3 254 - only Ambisonic Channels 256 The first ambisonic channel (W) is a mono audio stream which 257 represents the average audio signal over all directions. Since W is 258 not directional, Ogg Opus players MAY use W directly for mono 259 playback. 261 If a non-diegetic stereo track is present, the player MAY use the 262 matrix in Figure 6 for downmixing. Ls and Rs denote the two non- 263 diegetic stereo channels. 265 / \ / \ / \ 266 | L | | 0.25 0.25 0.0 ... 0.5 0.0 | | W | 267 | R | = | 0.25 -0.25 0.0 ... 0.0 0.5 | | Y | 268 \ / \ / | ... | 269 | Ls | 270 | Rs | 271 \ / 273 Figure 6: Stereo Downmixing Matrix for Channel Mapping Family 2 and 3 274 - Ambisonic Channels Plus a Non-diegetic Stereo Stream 276 5. Updates to RFC 7845 278 5.1. Format of the Channel Mapping Table 280 The language in section 5.1.1 in [RFC7845] implies that the channel 281 mapping table, when present, has a fixed format for all channel 282 mapping families: 284 The order and meaning of these channels are defined by a channel 285 mapping, which consists of the 'channel mapping family' octet and, 286 for channel mapping families other than family 0, a 'channel 287 mapping table', as illustrated in Figure 3. 289 This document updates [RFC7845] to clarify that the format of the 290 channel mapping table may depend on the channel mapping family: 292 The order and meaning of these channels are defined by a channel 293 mapping, which consists of the 'channel mapping family' octet and 294 for channel mapping families other than family 0, a 'channel 295 mapping table'. 297 The format of the channel mapping table depends on the channel 298 mapping family. Unless the channel mapping family requires a 299 custom format for its channel mapping table, the RECOMMENDED 300 channel mapping table format for new mapping families is 301 illustrated in Figure 3. 303 The change above is not meant to change how families 1 and 255 304 currently work. To ensure that, the first paragraph of 305 Section 5.1.1.2 is changed from: 307 Allowed numbers of channels: 1...8. Vorbis channel order (see 308 below). 310 to 312 Allowed numbers of channels: 1...8, with the mapping specified 313 according to Figure 3. Vorbis channel order (see below). 315 Similary, the first paragraph of Section 5.1.1.4 is changed from: 317 Allowed numbers of channels: 1...255. No defined channel meaning. 319 to 321 Allowed numbers of channels: 1...255, with the mapping specified 322 according to Figure 3. No defined channel meaning. 324 5.2. Unknown Mapping Families 326 The treatment of unknown mapping families is changed slightly. 327 Section 5.1.1.4 of [RFC7845] states: 329 The remaining channel mapping families (2...254) are reserved. A 330 demuxer implementation encountering a reserved 'channel mapping 331 family' value SHOULD act as though the value is 255. 333 This is changed to: 335 The remaining channel mapping families (2...254) are reserved. A 336 demuxer implementation encountering a 'channel mapping family' 337 value that it does not recognize SHOULD NOT attempt to decode the 338 packets and SHOULD NOT use any information except for the first 19 339 octets of the ID header packet (Fig. 2) and the comment header 340 (Fig. 10). 342 6. Experimental Mapping Families 344 To make development of new mapping families easier while reducing the 345 risk of creating compatibility issues with non-final version of 346 mapping families, mapping families 240 through 254 (inclusively) are 347 now reserved for experiments and implementations of in-development 348 families. Implementers SHOULD attempt to use experimental family 349 numbers that have not recently been used and SHOULD advertise what 350 experimental numbers they use (e.g. for Internet-Drafts). 352 The ambisonics mapping experiments that led to this document used 353 experimental family 254 for family 2 and experimental family 253 for 354 family 3. 356 7. Security Considerations 358 Implementations of the Ogg container need to take appropriate 359 security considerations into account, as outlined in Section 10 of 360 [RFC7845]. The extension defined in this document requires that 361 semantic meaning be assigned to more channels than the existing Ogg 362 format requires. Since more allocations will be required to encode 363 and decode these semantically meaningful channels, care should be 364 taken in any new allocation paths. Implementations MUST NOT overrun 365 their allocated memory nor read from uninitialized memory when 366 managing the ambisonic channel mapping. 368 8. IANA Considerations 370 This document updates the IANA Media Types registry "Opus Channel 371 Mapping Families" to add 17 new assignments. 373 +---------+---------------------------+ 374 | Value | Reference | 375 +---------+---------------------------+ 376 | 2 | This Document Section 3.1 | 377 | | | 378 | 3 | This Document Section 3.2 | 379 | | | 380 | 240-254 | This Document Section 6 | 381 +---------+---------------------------+ 383 9. Acknowledgments 385 Thanks to Timothy Terriberry, Jean-Marc Valin, Mark Harris, Marcin 386 Gorzel, and Andrew Allen for their guidance and valuable 387 contributions to this document. 389 10. References 391 10.1. Normative References 393 [ambix] Nachbar, C., Zotter, F., Deleflie, E., and A. Sontacchi, 394 "AMBIX - A SUGGESTED AMBISONICS FORMAT", June 2011, 395 . 398 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 399 Requirement Levels", BCP 14, RFC 2119, 400 DOI 10.17487/RFC2119, March 1997, 401 . 403 [RFC6716] Valin, JM., Vos, K., and T. Terriberry, "Definition of the 404 Opus Audio Codec", RFC 6716, DOI 10.17487/RFC6716, 405 September 2012, . 407 [RFC7845] Terriberry, T., Lee, R., and R. Giles, "Ogg Encapsulation 408 for the Opus Audio Codec", RFC 7845, DOI 10.17487/RFC7845, 409 April 2016, . 411 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 412 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 413 May 2017, . 415 10.2. Informative References 417 [daniel04] 418 Daniel, J. and S. Moreau, "Further Study of Sound Field 419 Coding with Higher Order Ambisonics", May 2004, 420 . 423 [gerzon75] 424 Gerzon, M., "Ambisonics. Part one: General system 425 description", August 1975, 426 . 429 Authors' Addresses 431 Jan Skoglund 432 Google LLC 433 345 Spear Street 434 San Francisco, CA 94105 435 USA 437 Email: jks@google.com 439 Michael Graczyk 441 Email: michael@mgraczyk.com