idnits 2.17.1 draft-muthusamy-dispatch-haptics-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (15 November 2020) is 1252 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DISPATCH Y. K. Muthusamy 3 Internet-Draft C. Ullrich 4 Intended status: Standards Track Immersion Corporation 5 Expires: 19 May 2021 15 November 2020 7 The 'haptics' Top-level Media Type 8 draft-muthusamy-dispatch-haptics-01 10 Abstract 12 This memo serves to register and document the 'haptics' top-level 13 media type, under which subtypes for representation formats for 14 haptics may be registered. This document also serves as a 15 registration application for a set of intended subtypes, which are 16 representative of some existing subtypes already in use. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on 19 May 2021. 35 Copyright Notice 37 Copyright (c) 2020 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 42 license-info) in effect on the date of publication of this document. 43 Please review these documents carefully, as they describe your rights 44 and restrictions with respect to this document. Code Components 45 extracted from this document must include Simplified BSD License text 46 as described in Section 4.e of the Trust Legal Provisions and are 47 provided without warranty as described in the Simplified BSD License. 49 RFC XXXX The 'haptics' Top-level Media Type November 2020 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Background and Justification . . . . . . . . . . . . . . . . 3 56 2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3 57 2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 4 58 2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4 59 2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4 60 2.5. Haptic Data Formats in Use . . . . . . . . . . . . . . . 5 61 2.6. Haptic Subtypes (envisioned standards) . . . . . . . . . 6 62 2.7. 'application' top-level type not suitable . . . . . . . . 6 63 3. Security Considerations . . . . . . . . . . . . . . . . . . . 7 64 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 65 4.1. Definition and Encoding . . . . . . . . . . . . . . . . . 8 66 4.2. Registration Procedure . . . . . . . . . . . . . . . . . 8 67 4.3. Subtype Registrations . . . . . . . . . . . . . . . . . . 8 68 4.3.1. IVS Haptics Type . . . . . . . . . . . . . . . . . . 8 69 4.3.2. HAPT Haptics Type . . . . . . . . . . . . . . . . . . 9 70 5. Normative References . . . . . . . . . . . . . . . . . . . . 10 71 6. Informative References . . . . . . . . . . . . . . . . . . . 10 72 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 74 1. Introduction 76 The term 'haptics' refers to the generation of touch-related 77 sensations in a device or interface. Haptics is widely used in 78 consumer devices in order to provide touch-based feedback to users. 79 The most common use of haptics is in mobile devices, where it is used 80 to provide feedback to users interacting with the touchscreen, e.g., 81 typing on a virtual keyboard. Haptic technologies are unlike audio 82 and visual enabling technologies in the sense that they require some 83 form of actuation in order to create a tactile sensation. For mobile 84 phones and game controllers, these actuators are typically small 85 vibrating motors. For large touchscreens in vehicles, these 86 actuators can be specialized piezoelectric materials. Haptic 87 capabilities are found in nearly every modern smartphone and game and 88 virtual reality controller, making these devices an ideal target for 89 enhanced media experiences. 91 Internet Media Types [RFC6838] are used to label content carried over 92 Internet protocols. This document defines a new top-level type 93 'haptics' according to Section 4.2.7 of [RFC6838]. This top-level 94 type indicates that the content specifies haptic data. Under this 95 top-level type, different representation formats of haptics may be 96 registered. 98 RFC XXXX The 'haptics' Top-level Media Type November 2020 100 1.1. Terminology 102 The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, 103 SHOULD,SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear 104 in this document, are to be interpreted as described in [RFC2119]. 106 2. Background and Justification 108 Haptic signals provide an additional layer of entertainment and 109 sensory immersion for the user. Haptic tracks, in separate files, 110 can be combined with audio/video files and played back in sync to 111 provide an overall immersive media experience (audio, visual, 112 tactile) for the user. More recently, haptic tracks embedded in 113 standard file formats such as ISOBMFF (ISO Base Media File Format), 114 enable playback of the haptic signals over one or more actuators, 115 simultaneously with audio and video playback. 117 2.1. MPEG ISOBMFF 119 Historically, there has not been a registration of formats for 120 haptics. However, haptics has been proposed as a first-order media 121 type (at the same level as audio and video) in ISOBMFF. This 122 proposal was made to the MPEG Systems File Format sub-group in April 123 2020. The proposal was accepted and has progressed to DIS (Draft 124 International Standard) in October 2020 [ISOBMFF-AMD1][AMD1-Status]. 125 Once it completes its progression through the MPEG standardization 126 stages (expected in October 2021), haptics will become part of the 127 ISO/IEC 14496-12 (ISOBMFF) standard. Given this development, a 128 strong case can be made for haptics to be added to the list of top- 129 level media type recognized by the IETF. 131 We envision the following designations for haptics in mp4 files, once 132 the top-level type 'haptics' is registered: 134 1. 'haptics/mp4' - mp4 files with just haptic tracks in them (e.g., 135 streaming games, haptics files for haptic vests, belts, gloves, 136 etc.) 138 2. 'video/mp4' - mp4 files with video, audio, and haptics (to ensure 139 consistency with existing mp4 files with video content) 141 3. 'audio/mp4' - mp4 files with audio and haptics (to ensure 142 consistency with existing mp4 files with audio content without 143 any video) 145 RFC XXXX The 'haptics' Top-level Media Type November 2020 147 2.2. Haptic Sub-modalities 149 There are multiple sub-modalities of haptics: 151 * Vibrotactile (touch, vibration) 153 * Kinesthetic (force feedback) 155 * Surface (surface friction) 157 * Spatial, non-contact (ultrasound) 159 * Thermal (temperature) 161 Therefore, designating 'haptics' as a top-level media type would 162 enable the definition of data formats pertaining to these sub- 163 modalities in a more streamlined manner. This would not be possible 164 if 'haptics' were to be placed under other top-level types like 165 'audio', 'video', or 'application'. 167 2.3. Another Human Sense 169 The top-level media type 'audio' pertains to the human sense of 170 hearing, the top-level media type 'video' pertains to the human sense 171 of seeing, so it only makes sense for the (equally important) human 172 sense of touch to be represented by another top-level media type 173 'haptics'. Placing 'haptics' under 'audio' or 'video' is not 174 reflective of the kinds of files or use cases that would need haptics 175 but have nothing whatsoever to do with audio or video. 177 2.4. Commercial Uptake 179 Haptics is rapidly becoming a standard feature of consumer electronic 180 devices. For example: 182 * iPhone (191+ million units sold in 2019): native support for 183 haptic encoded data 185 * Android (1.18+ billion units sold in 2019): API support of haptic 186 buffers 188 * W3C (HTML vibration API): Optionally supported in mobile web 189 browsers 191 * Game consoles (39+ million units sold in 2019): MS Xbox, Sony 192 PlayStation, Nintendo Switch, etc. 194 * XR devices (9+ million units sold in 2019): OpenXR haptic API 196 RFC XXXX The 'haptics' Top-level Media Type November 2020 198 Haptic media is expected to be commonly exchanged between these 199 devices. Since they represent the majority of CE devices, a strong 200 case can be made for 'haptics' as a top-level media type. 202 2.5. Haptic Data Formats in Use 204 There are multiple instances of existing haptic data formats that 205 would live as sub-types under the proposed 'haptics' top-level media 206 type. While these subtypes have *not* been registered with IANA or 207 standardized (yet), the prevalence of these haptic data formats in a 208 large number of devices around the world, pre-dating the 209 standardization of haptic tracks in ISOBMFF, provides a compelling 210 argument for 'haptics' to be designated as a top-level media type: 212 * 'ahap': The AHAP haptic data format [AHAP] is currently the 213 standard encoding on all iOS devices + iOS connected game 214 peripherals. The format has seen usage and adoption beyond Apple 215 devices as well, with decoders available for Android and other XR 216 systems. 218 * 'ogg': Google has introduced a proprietary extension to the OGG 219 format in the latest version of Android 11. This encoding enables 220 haptic media to be stored in OGG files. 222 * 'ivs': The IVS haptic data format is currently a vendor-specific 223 format that is in use: 225 - In mobile phones from LG Electronics (specifically, the models 226 V30, V40, and the newest V50) that are sold worldwide 228 - In gaming phones from ASUS (specifically, models ROG, ROG Phone 229 II, ROG Phone 3) that are sold worldwide 231 * 'hapt': The HAPT haptic data format is currently a vendor-specific 232 format that is in use: 234 - In mobile haptic advertising (for W3C devices) 236 - The following Japanese game developers use the HAPT format as 237 part of Immersion's TouchSense SDK: 239 o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/) 241 o Craft&Meister: http://www.crafts-meister.co.jp/pc/ 242 company_en.html (http://www.crafts-meister.co.jp/pc/ 243 company_en.html) 245 RFC XXXX The 'haptics' Top-level Media Type November 2020 247 - Tencent is using the TouchSense SDK for their popular social 248 media application QQ and live streaming application NOW: 249 Immersion-Announces-Tencent-Licenses-TouchSense-Technology- 250 Deliver 251 (https://www.businesswire.com/news/home/20171026006443/en/ 252 Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE- 253 Technology-Deliver) 255 Given the widespread use of these subtypes, it makes sense for 256 'haptics' to be a top-level media type. 258 2.6. Haptic Subtypes (envisioned standards) 260 The MPEG ISOBMFF proposal included an informative annex of known 261 haptic coding formats with proposed FourCC codes for them. These 262 codes are not registered yet, but the plan is indeed to standardize 263 these haptic coding formats in the near future. Once standardized, 264 they will also live as subtypes under the proposed 'haptics' top- 265 level media type: 267 * 'hmpg': the selected coding format from the MPEG Call for 268 Proposals on the Coded Representation of Haptics 269 [MPEG-Haptics-CfP] 271 * 'hiee': IEEE P1918.1.1 vibrotactile coding standard being 272 developed under the IEEE Tactile Internet initiative as part of 273 the 5G URLL profile. 275 * 'henm': enumerated effects haptic coding format (based on MIDI) 277 * 'havc': audio-to-vibe haptic coding format (automatic audio to 278 vibration conversion algorithms) 280 2.7. 'application' top-level type not suitable 282 From the above arguments, it is clear that haptics does not really 283 belong under any other media type. To reiterate, there are three 284 main reasons why the 'haptics' media type does not fit under the 285 'application' top-level type: 287 * haptics connects to a sensory system, touch/motion, directly, and 288 is more specific than the abstract 'application' type, and 290 * 'application' has historically been used for applications, i.e., 291 code, which means it is viewed and treated with great care for 292 security. 'haptics' is not code, just as 'audio' and 'video' are 293 not code either. 295 RFC XXXX The 'haptics' Top-level Media Type November 2020 297 * haptics is a property of a media stream, it is not an application 298 under any normal definition. As such, it should be its own type. 300 3. Security Considerations 302 Haptics are interpreted data structures that represent collections of 303 different media rendering instructions intended to be decoded and 304 rendered on target device hardware. Haptic data can be represented 305 as collections of signal data and/or descriptive text in XML/JSON or 306 similar format. Signal data is typically not executed by endpoint 307 processors and represents minimal security risk. Descriptive text is 308 typically parsed and represented in memory using standard XML data 309 structures. This data is utilized to construct one or more signals 310 that are sent to the endpoint device hardware. 312 Because of the media/rendering nature of the data path for haptic 313 coded data the security profile of haptic data is expected to be 314 largely consistent with the security profile of visual and audio 315 media data. 317 As with any synthesized media data (audio, video, and haptics), there 318 is a security risk associated with execution of commands based on the 319 descriptive encoding either through its inherent extensibility or 320 through the insertion of arbitrary executable data in the descriptive 321 format itself. Indeed, media rendering systems are normally 322 implemented with a mix of user and kernel space execution since these 323 media must ultimately make their way to a hardware system. In 324 theory, malicious instructions present in descriptive haptic media 325 have the potential to execute arbitrary code in kernel space, 326 effectively bypassing system permissions structures and/or execution 327 sandboxes. 329 Haptics, audio, and video media have widespread use and careful 330 attention should be paid by operating system and device driver 331 implementors to ensure that synthesis and rendering signal paths do 332 not provide attack surfaces for malicious payloads. Ultimately, any 333 coded representation of haptic media is insufficient to implicitly 334 provide sufficient security and this protection should be enforced by 335 the operating system implementor. 337 4. IANA Considerations 339 This specification registers a new top-level type, 'haptics', in the 340 standards tree, adds it as an alternative value of "Type Name" in the 341 media types registration form [Media-Type-Registration], and 342 registers several subtypes for it. 344 RFC XXXX The 'haptics' Top-level Media Type November 2020 346 4.1. Definition and Encoding 348 'haptics' as the primary media content type indicates that the 349 content identified by it requires a certain haptics subsystem such as 350 low-level haptics APIs, which in turn will require hardware 351 capabilities such as one or more actuators to render the haptics 352 media. The 'haptics' media type does not provide any specific 353 information about the underlying data format and how the haptics 354 information should be interpreted -- the subtypes defined within a 355 'haptics' tree name the specific haptic formats. Unrecognized 356 subtypes of 'haptics' should be treated as 'application/octet- 357 stream'. Implementations may still pass unrecognized subtypes to the 358 haptics subsystem and associated rendering hardware. 360 4.2. Registration Procedure 362 New haptics formats should be registered using the online form 363 [Media-Type-Registration]. [RFC6838] should be consulted on 364 registration procedures. In particular, the haptics specification 365 should preferably be freely available. 367 Note that new parameter sub-values may be defined in the future. If 368 an implementation does not recognize a sub-value in the comma- 369 separated list, it should ignore the sub-value and continue 370 processing the other sub-values in the list. 372 4.3. Subtype Registrations 374 In this section, the initial entries under the top-level 'haptics' 375 media type are specified. They also serve as examples for future 376 registrations. 378 4.3.1. IVS Haptics Type 380 Type name: haptics 382 Subtype name: ivs 384 Required parameters: None 386 Optional parameters: None 388 Encoding considerations: Text/binary 390 Interoperability considerations: The IVS format is a device- 391 independent haptic effect coding. It is designed to enable 392 interoperability between distinct physical endpoints. Not all 393 devices may be able to render all effects present in an IVS file. 395 RFC XXXX The 'haptics' Top-level Media Type November 2020 397 Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 13 "Encoder Input 398 Format for Haptics" being developed by ISO/IEC JTC1/SC29 WG 2. 400 Applications that use this media type: All applications that are able 401 to create, edit, or display haptic media content. 403 Additional information: 405 * File extension(s): Haptic file extensions used for IVS files: .ivs 406 (xml) and .ivt (binary) 408 * Macintosh file type code(s): (no code specified) 410 * Macintosh Universal Type Identifier code: None 412 * Fragment Identifier: None 414 * Deprecated Alias: None 416 Person & email address to contact for further information: Yeshwant 417 Muthusamy(ymuthusamy@immersion.com) 419 Change controller: Immersion Corporation 421 4.3.2. HAPT Haptics Type 423 Type name: haptics 425 Subtype name: hapt 427 Required parameters: None 429 Optional parameters: None 431 Encoding considerations: Text/binary 433 Interoperability considerations: The HAPT format is a device- 434 dependent haptic effect coding based on the RIFF coding standard. It 435 is designed to enable efficient coding of a device-specific haptic 436 effect. 438 Published specification: HAPT is a logical extension of the RIFF 439 standard [RIFF] 441 Applications that use this media type: All applications that are able 442 to create, edit, or display haptic media content. 444 Additional information: 446 RFC XXXX The 'haptics' Top-level Media Type November 2020 448 * File extension(s): Haptic file extensions used for HAPT files: 449 .hapt 451 * Macintosh file type code(s): (no code specified) 453 * Macintosh Universal Type Identifier code: None 455 * Fragment Identifier: None 457 * Deprecated Alias: None 459 Person & email address to contact for further information: Yeshwant 460 Muthusamy(ymuthusamy@immersion.com) 462 Change controller: Immersion Corporation 464 5. Normative References 466 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 467 Requirement Levels", BCP 14, RFC 2119, 468 DOI 10.17487/RFC2119, March 1997, 469 . 471 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 472 Specifications and Registration Procedures", BCP 13, 473 RFC 6838, DOI 10.17487/RFC6838, January 2013, 474 . 476 6. Informative References 478 [ISOBMFF-AMD1] 479 "ISO/IEC 14496-12/CD AMD 1 Information technology - Coding 480 of audio-visual objects - Part 12: ISO base media file 481 format - Amendment 1: Support for new media types 482 (haptics, volumetric visual) and other improvements", 483 . 485 [AMD1-Status] 486 "Timeline of ISO/IEC 14496-12/CD Amd 1, Edition 6 - 487 Project ID: 81604, ISO/IEC JTC 1/SC 29/WG 3", 488 . 490 [AHAP] "Apple Haptic Audio Pattern", 491 . 494 RFC XXXX The 'haptics' Top-level Media Type November 2020 496 [MPEG-Haptics-CfP] 497 "Draft Call for Proposals on the Coded Representation of 498 Haptics", 499 . 501 [Media-Type-Registration] 502 "IANA, Application for a Media Type", 503 . 505 [RIFF] "Resource Interchange File Format", 506 . 509 Authors' Addresses 511 Yeshwant K. Muthusamy 512 Immersion Corporation 513 330 Townsend St. Suite 234 514 San Francisco, CA 94107 515 United States of America 517 Phone: +1 469-583-2171 518 Email: ymuthusamy@immersion.com 520 Chris Ullrich 521 Immersion Corporation 522 330 Townsend St. Suite 234 523 San Francisco, CA 94107 524 United States of America 526 Phone: +1 805-320-0774 527 Email: cullrich@immersion.com