idnits 2.17.1 draft-muthusamy-dispatch-haptics-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (14 May 2021) is 1071 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DISPATCH Y. K. Muthusamy 3 Internet-Draft C. Ullrich 4 Intended status: Standards Track Immersion Corporation 5 Expires: 15 November 2021 14 May 2021 7 The 'haptics' Top-level Media Type 8 draft-muthusamy-dispatch-haptics-02 10 Abstract 12 This memo serves to register and document the 'haptics' top-level 13 media type, under which subtypes for representation formats for 14 haptics may be registered. This document also serves as a 15 registration application for a set of intended subtypes, which are 16 representative of some existing subtypes already in use. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on 15 November 2021. 35 Copyright Notice 37 Copyright (c) 2021 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 42 license-info) in effect on the date of publication of this document. 43 Please review these documents carefully, as they describe your rights 44 and restrictions with respect to this document. Code Components 45 extracted from this document must include Simplified BSD License text 46 as described in Section 4.e of the Trust Legal Provisions and are 47 provided without warranty as described in the Simplified BSD License. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 52 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 53 2. Background and Justification . . . . . . . . . . . . . . . . 3 54 2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3 55 2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 4 56 2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4 57 2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4 58 2.5. Haptic Data Formats in Use . . . . . . . . . . . . . . . 5 59 2.6. Haptic Subtypes (envisioned standards) . . . . . . . . . 6 60 2.7. 'application' top-level type not suitable . . . . . . . . 6 61 3. Security Considerations . . . . . . . . . . . . . . . . . . . 7 62 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 63 4.1. Definition and Encoding . . . . . . . . . . . . . . . . . 8 64 4.2. Registration Procedure . . . . . . . . . . . . . . . . . 8 65 4.3. Subtype Registrations . . . . . . . . . . . . . . . . . . 8 66 4.3.1. IVS Haptics Type . . . . . . . . . . . . . . . . . . 8 67 4.3.2. HAPT Haptics Type . . . . . . . . . . . . . . . . . . 9 68 5. Normative References . . . . . . . . . . . . . . . . . . . . 10 69 6. Informative References . . . . . . . . . . . . . . . . . . . 10 70 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 72 1. Introduction 74 The term 'haptics' refers to the generation of touch-related 75 sensations in a device or interface. Haptics is widely used in 76 consumer devices in order to provide touch-based feedback to users. 77 The most common use of haptics is in mobile devices, where it is used 78 to provide feedback to users interacting with the touchscreen, e.g., 79 typing on a virtual keyboard. Haptic technologies are unlike audio 80 and visual enabling technologies in the sense that they require some 81 form of actuation in order to create a tactile sensation. For mobile 82 phones and game controllers, these actuators are typically small 83 vibrating motors. For large touchscreens in vehicles, these 84 actuators can be specialized piezoelectric materials. Haptic 85 capabilities are found in nearly every modern smartphone and game and 86 virtual reality controller, making these devices an ideal target for 87 enhanced media experiences. 89 Internet Media Types [RFC6838] are used to label content carried over 90 Internet protocols. This document defines a new top-level type 91 'haptics' according to Section 4.2.7 of [RFC6838]. This top-level 92 type indicates that the content specifies haptic data. Under this 93 top-level type, different representation formats of haptics may be 94 registered. 96 1.1. Terminology 98 The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, 99 SHOULD,SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear 100 in this document, are to be interpreted as described in [RFC2119]. 102 2. Background and Justification 104 Haptic signals provide an additional layer of entertainment and 105 sensory immersion for the user. Haptic tracks, in separate files, 106 can be combined with audio/video files and played back in sync to 107 provide an overall immersive media experience (audio, visual, 108 tactile) for the user. More recently, haptic tracks embedded in 109 standard file formats such as ISOBMFF (ISO Base Media File Format), 110 enable playback of the haptic signals over one or more actuators, 111 simultaneously with audio and video playback. 113 2.1. MPEG ISOBMFF 115 Historically, there has not been a registration of formats for 116 haptics. However, haptics has been proposed as a first-order media 117 type (at the same level as audio and video) in ISOBMFF. This 118 proposal was made to the MPEG Systems File Format sub-group in April 119 2020. The proposal was accepted and has progressed to DIS (Draft 120 International Standard) in October 2020 [ISOBMFF-AMD1][AMD1-Status]. 121 Once it completes its progression through the MPEG standardization 122 stages (expected in October 2021), haptics will become part of the 123 ISO/IEC 14496-12 (ISOBMFF) standard. Given this development, a 124 strong case can be made for haptics to be added to the list of top- 125 level media type recognized by the IETF. 127 We envision the following designations for haptics in mp4 files, once 128 the top-level type 'haptics' is registered: 130 1. 'haptics/mp4' - mp4 files with just haptic tracks in them (e.g., 131 streaming games, haptics files for haptic vests, belts, gloves, 132 etc.) 134 2. 'video/mp4' - mp4 files with video, audio, and haptics (to ensure 135 consistency with existing mp4 files with video content) 137 3. 'audio/mp4' - mp4 files with audio and haptics (to ensure 138 consistency with existing mp4 files with audio content without 139 any video) 141 2.2. Haptic Sub-modalities 143 There are multiple sub-modalities of haptics: 145 * Vibrotactile (touch, vibration) 147 * Kinesthetic (force feedback) 149 * Surface (surface friction) 151 * Spatial, non-contact (ultrasound) 153 * Thermal (temperature) 155 Therefore, designating 'haptics' as a top-level media type would 156 enable the definition of data formats pertaining to these sub- 157 modalities in a more streamlined manner. This would not be possible 158 if 'haptics' were to be placed under other top-level types like 159 'audio', 'video', or 'application'. 161 2.3. Another Human Sense 163 The top-level media type 'audio' pertains to the human sense of 164 hearing, the top-level media type 'video' pertains to the human sense 165 of seeing, so it only makes sense for the (equally important) human 166 sense of touch to be represented by another top-level media type 167 'haptics'. Placing 'haptics' under 'audio' or 'video' is not 168 reflective of the kinds of files or use cases that would need haptics 169 but have nothing whatsoever to do with audio or video. 171 2.4. Commercial Uptake 173 Haptics is rapidly becoming a standard feature of consumer electronic 174 devices. For example: 176 * iPhone (191+ million units sold in 2019): native support for 177 haptic encoded data 179 * Android (1.18+ billion units sold in 2019): API support of haptic 180 buffers 182 * W3C (HTML vibration API): Optionally supported in mobile web 183 browsers 185 * Game consoles (39+ million units sold in 2019): MS Xbox, Sony 186 PlayStation, Nintendo Switch, etc. 188 * XR devices (9+ million units sold in 2019): OpenXR haptic API 189 Haptic media is expected to be commonly exchanged between these 190 devices. Since they represent the majority of CE devices, a strong 191 case can be made for 'haptics' as a top-level media type. 193 2.5. Haptic Data Formats in Use 195 There are multiple instances of existing haptic data formats that 196 would live as sub-types under the proposed 'haptics' top-level media 197 type. While these subtypes have *not* been registered with IANA or 198 standardized (yet), the prevalence of these haptic data formats in a 199 large number of devices around the world, pre-dating the 200 standardization of haptic tracks in ISOBMFF, provides a compelling 201 argument for 'haptics' to be designated as a top-level media type: 203 * 'ahap': The AHAP haptic data format [AHAP] is currently the 204 standard encoding on all iOS devices + iOS connected game 205 peripherals. The format has seen usage and adoption beyond Apple 206 devices as well, with decoders available for Android and other XR 207 systems. 209 * 'ogg': Google has introduced a proprietary extension to the OGG 210 format in the latest version of Android 11. This encoding enables 211 haptic media to be stored in OGG files. 213 * 'ivs': The IVS haptic data format is currently a vendor-specific 214 format that is in use: 216 - In mobile phones from LG Electronics (specifically, the models 217 V30, V40, and the newest V50) that are sold worldwide 219 - In gaming phones from ASUS (specifically, models ROG, ROG Phone 220 II, ROG Phone 3) that are sold worldwide 222 * 'hapt': The HAPT haptic data format is currently a vendor-specific 223 format that is in use: 225 - In mobile haptic advertising (for W3C devices) 227 - The following Japanese game developers use the HAPT format as 228 part of Immersion's TouchSense SDK: 230 o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/) 232 o Craft&Meister: http://www.crafts-meister.co.jp/pc/ 233 company_en.html (http://www.crafts-meister.co.jp/pc/ 234 company_en.html) 236 - Tencent is using the TouchSense SDK for their popular social 237 media application QQ and live streaming application NOW: 238 Immersion-Announces-Tencent-Licenses-TouchSense-Technology- 239 Deliver 240 (https://www.businesswire.com/news/home/20171026006443/en/ 241 Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE- 242 Technology-Deliver) 244 Given the widespread use of these subtypes, it makes sense for 245 'haptics' to be a top-level media type. 247 2.6. Haptic Subtypes (envisioned standards) 249 The MPEG ISOBMFF proposal included an informative annex of known 250 haptic coding formats with proposed FourCC codes for them. These 251 codes are not registered yet, but the plan is indeed to standardize 252 these haptic coding formats in the near future. Once standardized, 253 they will also live as subtypes under the proposed 'haptics' top- 254 level media type: 256 * 'hmpg': the selected coding format from the MPEG Call for 257 Proposals on the Coded Representation of Haptics 258 [MPEG-Haptics-CfP] 260 * 'hiee': IEEE P1918.1.1 vibrotactile coding standard being 261 developed under the IEEE Tactile Internet initiative as part of 262 the 5G URLL profile. 264 * 'henm': enumerated effects haptic coding format (based on MIDI) 266 * 'havc': audio-to-vibe haptic coding format (automatic audio to 267 vibration conversion algorithms) 269 2.7. 'application' top-level type not suitable 271 From the above arguments, it is clear that haptics does not really 272 belong under any other media type. To reiterate, there are three 273 main reasons why the 'haptics' media type does not fit under the 274 'application' top-level type: 276 * haptics connects to a sensory system, touch/motion, directly, and 277 is more specific than the abstract 'application' type, and 279 * 'application' has historically been used for applications, i.e., 280 code, which means it is viewed and treated with great care for 281 security. 'haptics' is not code, just as 'audio' and 'video' are 282 not code either. 284 * haptics is a property of a media stream, it is not an application 285 under any normal definition. As such, it should be its own type. 287 3. Security Considerations 289 Haptics are interpreted data structures that represent collections of 290 different media rendering instructions intended to be decoded and 291 rendered on target device hardware. Haptic data can be represented 292 as collections of signal data and/or descriptive text in XML/JSON or 293 similar format. Signal data is typically not executed by endpoint 294 processors and represents minimal security risk. Descriptive text is 295 typically parsed and represented in memory using standard XML data 296 structures. This data is utilized to construct one or more signals 297 that are sent to the endpoint device hardware. 299 Because of the media/rendering nature of the data path for haptic 300 coded data the security profile of haptic data is expected to be 301 largely consistent with the security profile of visual and audio 302 media data. 304 As with any synthesized media data (audio, video, and haptics), there 305 is a security risk associated with execution of commands based on the 306 descriptive encoding either through its inherent extensibility or 307 through the insertion of arbitrary executable data in the descriptive 308 format itself. Indeed, media rendering systems are normally 309 implemented with a mix of user and kernel space execution since these 310 media must ultimately make their way to a hardware system. In 311 theory, malicious instructions present in descriptive haptic media 312 have the potential to execute arbitrary code in kernel space, 313 effectively bypassing system permissions structures and/or execution 314 sandboxes. 316 Haptics, audio, and video media have widespread use and careful 317 attention should be paid by operating system and device driver 318 implementors to ensure that synthesis and rendering signal paths do 319 not provide attack surfaces for malicious payloads. Ultimately, any 320 coded representation of haptic media is insufficient to implicitly 321 provide sufficient security and this protection should be enforced by 322 the operating system implementor. 324 4. IANA Considerations 326 This specification registers a new top-level type, 'haptics', in the 327 standards tree, adds it as an alternative value of "Type Name" in the 328 media types registration form [Media-Type-Registration], and 329 registers several subtypes for it. 331 4.1. Definition and Encoding 333 'haptics' as the primary media content type indicates that the 334 content identified by it requires a certain haptics subsystem such as 335 low-level haptics APIs, which in turn will require hardware 336 capabilities such as one or more actuators to render the haptics 337 media. The 'haptics' media type does not provide any specific 338 information about the underlying data format and how the haptics 339 information should be interpreted -- the subtypes defined within a 340 'haptics' tree name the specific haptic formats. Unrecognized 341 subtypes of 'haptics' should be treated as 'application/octet- 342 stream'. Implementations may still pass unrecognized subtypes to the 343 haptics subsystem and associated rendering hardware. 345 4.2. Registration Procedure 347 New haptics formats should be registered using the online form 348 [Media-Type-Registration]. [RFC6838] should be consulted on 349 registration procedures. In particular, the haptics specification 350 should preferably be freely available. 352 Note that new parameter sub-values may be defined in the future. If 353 an implementation does not recognize a sub-value in the comma- 354 separated list, it should ignore the sub-value and continue 355 processing the other sub-values in the list. 357 4.3. Subtype Registrations 359 In this section, the initial entries under the top-level 'haptics' 360 media type are specified. They also serve as examples for future 361 registrations. 363 4.3.1. IVS Haptics Type 365 Type name: haptics 367 Subtype name: ivs 369 Required parameters: None 371 Optional parameters: None 373 Encoding considerations: Text/binary 375 Interoperability considerations: The IVS format is a device- 376 independent haptic effect coding. It is designed to enable 377 interoperability between distinct physical endpoints. Not all 378 devices may be able to render all effects present in an IVS file. 380 Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 72 "Encoder Input 381 Format for Haptics" being developed by ISO/IEC JTC1/SC29 WG 2 382 [MPEG-Haptics-Encoder-Format]. 384 Applications that use this media type: All applications that are able 385 to create, edit, or display haptic media content. 387 Additional information: 389 * File extension(s): Haptic file extensions used for IVS files: .ivs 390 (xml) and .ivt (binary) 392 * Macintosh file type code(s): (no code specified) 394 * Macintosh Universal Type Identifier code: None 396 * Fragment Identifier: None 398 * Deprecated Alias: None 400 Person & email address to contact for further information: Yeshwant 401 Muthusamy(ymuthusamy@immersion.com) 403 Change controller: Immersion Corporation 405 4.3.2. HAPT Haptics Type 407 Type name: haptics 409 Subtype name: hapt 411 Required parameters: None 413 Optional parameters: None 415 Encoding considerations: Text/binary 417 Interoperability considerations: The HAPT format is a device- 418 dependent haptic effect coding based on the RIFF coding standard. It 419 is designed to enable efficient coding of a device-specific haptic 420 effect. 422 Published specification: HAPT is a logical extension of the RIFF 423 standard [RIFF] 425 Applications that use this media type: All applications that are able 426 to create, edit, or display haptic media content. 428 Additional information: 430 * File extension(s): Haptic file extensions used for HAPT files: 431 .hapt 433 * Macintosh file type code(s): (no code specified) 435 * Macintosh Universal Type Identifier code: None 437 * Fragment Identifier: None 439 * Deprecated Alias: None 441 Person & email address to contact for further information: Yeshwant 442 Muthusamy(ymuthusamy@immersion.com) 444 Change controller: Immersion Corporation 446 5. Normative References 448 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 449 Requirement Levels", BCP 14, RFC 2119, 450 DOI 10.17487/RFC2119, March 1997, 451 . 453 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 454 Specifications and Registration Procedures", BCP 13, 455 RFC 6838, DOI 10.17487/RFC6838, January 2013, 456 . 458 6. Informative References 460 [ISOBMFF-AMD1] 461 "ISO/IEC 14496-12/CD AMD 1 Information technology - Coding 462 of audio-visual objects - Part 12: ISO base media file 463 format - Amendment 1: Support for new media types 464 (haptics, volumetric visual) and other improvements", 465 . 467 [AMD1-Status] 468 "Timeline of ISO/IEC 14496-12/CD Amd 1, Edition 6 - 469 Project ID: 81604, ISO/IEC JTC 1/SC 29/WG 3", 470 . 472 [AHAP] "Apple Haptic Audio Pattern", 473 . 476 [MPEG-Haptics-CfP] 477 "Call for Proposals on the Coded Representation of Haptics 478 - Phase 1", . 481 [MPEG-Haptics-Encoder-Format] 482 "Encoder Input Format for Haptics", 483 . 486 [Media-Type-Registration] 487 "IANA, Application for a Media Type", 488 . 490 [RIFF] "Resource Interchange File Format", 491 . 494 Authors' Addresses 496 Yeshwant K. Muthusamy 497 Immersion Corporation 498 330 Townsend St. Suite 234 499 San Francisco, CA 94107 500 United States of America 502 Phone: +1 469-583-2171 503 Email: ymuthusamy@immersion.com 505 Chris Ullrich 506 Immersion Corporation 507 330 Townsend St. Suite 234 508 San Francisco, CA 94107 509 United States of America 511 Phone: +1 805-320-0774 512 Email: cullrich@immersion.com