idnits 2.17.1 draft-muthusamy-dispatch-haptics-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (12 October 2020) is 1292 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Media-Type-Registration' is mentioned on line 310, but not defined Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DISPATCH Y. K. Muthusamy 3 Internet-Draft C. Ullrich 4 Intended status: Standards Track Immersion Corporation 5 Expires: 15 April 2021 12 October 2020 7 The 'haptics' Top-level Media Type 8 draft-muthusamy-dispatch-haptics-00 10 Abstract 12 This memo serves to register and document the 'haptics' top-level 13 media type, under which subtypes for representation formats for 14 haptics may be registered. This document also serves as a 15 registration application for a set of intended subtypes, which are 16 representative of some existing subtypes already in use. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on 15 April 2021. 35 Copyright Notice 37 Copyright (c) 2020 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 42 license-info) in effect on the date of publication of this document. 43 Please review these documents carefully, as they describe your rights 44 and restrictions with respect to this document. Code Components 45 extracted from this document must include Simplified BSD License text 46 as described in Section 4.e of the Trust Legal Provisions and are 47 provided without warranty as described in the Simplified BSD License. 49 RFC XXXX The 'haptics' Top-level Media Type October 2020 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Background and Justification . . . . . . . . . . . . . . . . 3 56 2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3 57 2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 3 58 2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4 59 2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4 60 2.5. Haptic Sub-types (in use) . . . . . . . . . . . . . . . . 5 61 2.6. Haptic Sub-types (envisioned standards) . . . . . . . . . 6 62 2.7. 'application' top-level type not suitable . . . . . . . . 6 63 3. Definition and Encoding . . . . . . . . . . . . . . . . . . . 7 64 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 65 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 66 6. Normative References . . . . . . . . . . . . . . . . . . . . 8 67 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 69 1. Introduction 71 The term 'haptics' refers to the generation of touch related 72 sensations in a device or interface. Haptics is widely used in 73 consumer devices in order to provide touch-based feedback to users. 74 The most common use of haptics is in mobile devices, where it is used 75 to provide feedback to users interacting with the touchscreen, e.g., 76 typing on a virtual keyboard. Haptic technologies are unlike audio 77 and visual enabling technologies in the sense that they require some 78 form of actuation in order to create a tactile sensation. For mobile 79 phones and game controllers, these actuators are typically small 80 vibrating motors. For large touchscreens in vehicles, these 81 actuators can be specialized piezoelectric materials. Haptic 82 capabilities are found in nearly every modern smartphone and game and 83 virtual reality controller, making these devices an ideal target for 84 enhanced media experiences. 86 Internet Media Types [RFC6838] are used to label content carried over 87 Internet protocols. This document defines a new top-level type 88 'haptics' according to Section 4.2.7 of [RFC6838]. This top-level 89 type indicates that the content specifies haptic data. Under this 90 top-level type, different representation formats of haptics may be 91 registered. 93 1.1. Terminology 95 The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, 96 SHOULD,SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear 97 in this document, are to be interpreted as described in [RFC2119]. 99 RFC XXXX The 'haptics' Top-level Media Type October 2020 101 2. Background and Justification 103 Haptic signals provide an additional layer of entertainment and 104 sensory immersion for the user. Haptic tracks, in separate files, 105 can be combined with audio/video files and played back in sync to 106 provide an overall immersive media experience (audio, visual, 107 tactile) for the user. More recently, haptic tracks embedded in 108 standard file formats such as ISOBMFF (ISO Base Media File Format), 109 enable playback of the haptic signals over one or more actuators, 110 simultaneously with audio and video playback. 112 2.1. MPEG ISOBMFF 114 Historically, there has not been a registration of formats for 115 haptics. However, haptics has been proposed as a first-order media 116 type (at the same level as audio and video) in ISOBMFF. This 117 proposal was made to the MPEG Systems File Format sub-group in April 118 2020. The proposal was accepted and is scheduled to progress to DIS 119 (Draft International Standard) in late October 2020. Once it 120 completes its progression through the MPEG standardization stages 121 (expected in January 2022), haptics will become part of the ISO/IEC 122 14496-12 (ISOBMFF) standard. Given this development, a strong case 123 can be made for haptics to be added to the list of top-level media 124 type recognized by the IETF. 126 We envision the following designations for haptics in mp4 files: 128 1. 'haptics/mp4' - mp4 files with just haptic tracks in them (e.g., 129 streaming games, haptics files for haptic vests, belts, gloves, 130 etc.) 132 2. 'video/mp4' - mp4 files with video, audio, and haptics (to ensure 133 consistency with existing mp4 files with video content) 135 3. 'audio/mp4' - mp4 files with audio and haptics (to ensure 136 consistency with existing mp4 files with audio content without 137 any video) 139 2.2. Haptic Sub-modalities 141 There are multiple sub-modalities of haptics: 143 * Vibrotactile (touch, vibration) 145 * Kinesthetic (force feedback) 147 * Surface (surface friction) 149 RFC XXXX The 'haptics' Top-level Media Type October 2020 151 * Spatial, non-contact (ultrasound) 153 * Thermal (temperature) 155 Therefore, designating 'haptics' as a top-level media type would 156 enable the definition of data formats pertaining to these sub- 157 modalities in a more streamlined manner. This would not be possible 158 if 'haptics' were to be placed under other top-level types like 159 'audio', 'video', or 'application'. 161 2.3. Another Human Sense 163 The top-level media type 'audio' pertains to the human sense of 164 hearing, the top-level media type 'video' pertains to the human sense 165 of seeing, so it only makes sense for the (equally important) human 166 sense of touch to be represented by another top-level media type 167 'haptics'. Placing 'haptics' under 'audio' or 'video' is not 168 reflective of the kinds of files or use cases that would need haptics 169 but have nothing whatsoever to do with audio or video. 171 2.4. Commercial Uptake 173 Haptics is rapidly becoming a standard feature of consumer electronic 174 devices. For example: 176 * iPhone (191+ million units sold in 2019): native support for 177 haptic encoded data 179 * Android (1.18+ billion units sold in 2019): API support of haptic 180 buffers 182 * W3C (HTML vibration API): Optionally supported in mobile web 183 browsers 185 * Game consoles (39+ million units sold in 2019): MS Xbox, Sony 186 PlayStation, Nintendo Switch, etc. 188 * XR devices (9+ million units sold in 2019): OpenXR haptic API 190 Haptic media is expected to be commonly exchanged between these 191 devices. Since they represent the majority of CE devices, a strong 192 case can be made for 'haptics' as a top-level media type. 194 RFC XXXX The 'haptics' Top-level Media Type October 2020 196 2.5. Haptic Sub-types (in use) 198 There are multiple instances of existing haptic sub-types (data 199 formats) that would live under the proposed 'haptics' top-level media 200 type. While these sub-types have not been registered with IANA or 201 standardized (yet), the prevalence of these haptic data formats in a 202 large number of devices around the world, pre-dating the 203 standardization of haptic tracks in ISOBMFF, provides a compelling 204 argument for 'haptics' to be designated as a top-level media type: 206 * haptics/ahap: The AHAP haptic data format is currently the 207 standard encoding on all iOS devices + iOS connected game 208 peripherals. The format has seen usage and adoption beyond Apple 209 devices as well, with decoders available for Android and other XR 210 systems. 212 * haptics/ogg: Google has introduced a proprietary extension to the 213 OGG format in the latest version of Android 11. This encoding 214 enables haptic media to be stored in OGG files. 216 * haptics/ivt: The IVT haptic data format is currently an Immersion- 217 proprietary format that has been licensed and is in use: 219 - In mobile phones from LG Electronics (specifically, the models 220 V30, V40, and the newest V50) that are sold worldwide 222 - In gaming phones from ASUS (specifically, models ROG, ROG Phone 223 II, ROG Phone 3) that are sold worldwide 225 * haptics/hapt: The HAPT haptic data format is currently an 226 Immersion-proprietary format that has been licensed and is in use: 228 - In mobile haptic advertising (for W3C devices) 230 - The following Japanese game developers use the HAPT format as 231 part of Immersion's TouchSense SDK: 233 o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/) 235 o Craft&Meister: http://www.crafts-meister.co.jp/pc/ 236 company_en.html (http://www.crafts-meister.co.jp/pc/ 237 company_en.html) 239 RFC XXXX The 'haptics' Top-level Media Type October 2020 241 - Tencent is using the TouchSense SDK for their popular social 242 media application QQ and live streaming application NOW: 243 Immersion-Announces-Tencent-Licenses-TouchSense-Technology- 244 Deliver 245 (https://www.businesswire.com/news/home/20171026006443/en/ 246 Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE- 247 Technology-Deliver) 249 Given the widespread use of these sub-types, it makes sense for 250 'haptics' to be a top-level media type. 252 2.6. Haptic Sub-types (envisioned standards) 254 The MPEG ISOBMFF proposal included an informative annex of known 255 haptic coding formats with proposed FourCC codes for them. These 256 codes are not registered yet, but the plan is indeed to standardize 257 these haptic coding formats in the near future: 259 * haptics/hstr: would be the standards version of Immersion's HAPT 260 format 262 * haptics/hiee: IEEE P1918.1.1 vibrotactile coding standard being 263 developed under the IEEE Tactile Internet initiative as part of 264 the 5G URLL profile. 266 * haptics/henm: enumerated effects haptic coding format (based on 267 MIDI) 269 * haptics/havc: audio-to-vibe haptic coding format (automatic audio 270 to vibration conversion algorithms) 272 2.7. 'application' top-level type not suitable 274 From the above arguments, it is clear that haptics does not really 275 belong under any other media type. To reiterate, there are two main 276 reasons why the 'haptics' media type does not fit under the 277 'application' top-level type: 279 * haptics connects to a sensory system, touch/motion, directly, and 280 is more specific than the abstract 'application' type, and 282 * 'application' has historically been used for applications, i.e., 283 code, which means it is viewed and treated with great care for 284 security. 'haptics' is not code, just as 'audio' and 'video' are 285 not code either. 287 * haptics is a property of a media stream, it is not an application 288 under any normal definition. As such, it should be its own type. 290 RFC XXXX The 'haptics' Top-level Media Type October 2020 292 3. Definition and Encoding 294 'haptics' as the primary media content type indicates that the 295 content identified by it requires a certain haptics subsystem such as 296 low-level haptics APIs, which in turn will require hardware 297 capabilities such as one or more actuators to render the haptics 298 media. The 'haptics' media type does not provide any specific 299 information about the underlying data format and how the haptics 300 information should be interpreted -- the subtypes defined within a 301 'haptics' tree name the specific haptic formats. Unrecognized 302 subtypes of 'haptics' should be treated as 'application/octet- 303 stream'. Implementations may still pass unrecognized subtypes to the 304 haptics subsystem and associated rendering hardware. 306 4. IANA Considerations 308 This specification registers a new top-level type, 'haptics', in the 309 standards tree, adds it as an alternative value of "Type Name" in the 310 media types registration form [Media-Type-Registration], and 311 registers several subtypes for it. 313 5. Security Considerations 315 Haptics are interpreted data structures that represent collections of 316 different media rendering instructions intended to be decoded and 317 rendered on target device hardware. Haptic data can be represented 318 as collections of signal data and/or descriptive text in XML/JSON or 319 similar. Signal data is typically not executed by endpoint 320 processors and represents minimal security risk. Descriptive text is 321 typically parsed and represented in memory using standard XML data 322 structures. This data is utilized to construct one or more signals 323 that are sent to the endpoint device hardware. 325 Because of the media/rendering nature of the data path for haptic 326 coded data the security profile of haptic data is expected to be 327 largely consistent with the security profile of visual and audio 328 media data. 330 RFC XXXX The 'haptics' Top-level Media Type October 2020 332 As with any synthesized media data (audio, visual and haptic), there 333 is a security risk associated with execution of commands based on the 334 descriptive encoding either through its inherent extensibility or 335 through the insertion of arbitrary executable data in the descriptive 336 format itself. Indeed, media rendering systems are normally 337 implemented with a mix of user and kernel space execution since these 338 media must ultimately make their way to a hardware system. In 339 theory, malicious instructions present in descriptive haptic media 340 have the potential to execute arbitrary code in kernel space, 341 effectively bypassing system permissions structures and/or execution 342 sandboxes. 344 Haptics, audio and video media have widespread use and careful 345 attention should be paid by operating system and device driver 346 implementors to ensure that synthesis and rendering signal paths do 347 not provide attack surfaces for malicious payloads. Ultimately, any 348 coded representation of haptic media is insufficient to implicitly 349 provide sufficient security and this protection should be enforced by 350 the operating system implementor. 352 6. Normative References 354 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 355 Requirement Levels", BCP 14, RFC 2119, 356 DOI 10.17487/RFC2119, March 1997, 357 . 359 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 360 Specifications and Registration Procedures", BCP 13, 361 RFC 6838, DOI 10.17487/RFC6838, January 2013, 362 . 364 Authors' Addresses 366 Yeshwant K. Muthusamy 367 Immersion Corporation 368 330 Townsend St. Suite 234 369 San Francisco 371 Phone: +1 469-583-2171 372 Email: ymuthusamy@immersion.com 374 Chris Ullrich 375 Immersion Corporation 376 330 Townsend St. Suite 234 377 San Francisco 379 RFC XXXX The 'haptics' Top-level Media Type October 2020 381 Phone: +1 805-320-0774 382 Email: cullrich@immersion.com