Traditional View of the Web

• Someone sitting at a computer with a large colour display, full-sized keyboard and a high speed network connection.
• The Web provides access to remote information and services. The Web Browser identifies web pages via their URI and down loads the markup, graphics and other resources from web servers.

But ...

• Not everyone can see clearly or can operate the mouse etc.
  • Need to design for accessible web sites
• Many new kinds of devices with much smaller displays
  • Need to adapt to small displays
  • Need to design services for embedded or mobile context
• Traditional telephone with audio + phone key pad
  • VoiceXML: markup for speech dialogues
  • Corporate front desk
    • Telephone number for call centre
    • Web address (URI) for website

    Both can utilize W3C web technologies

The Architecture of the Web

The World Wide Web (WWW, or simply Web) is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI).

See http://www.w3.org/TR/webarch/

Web pages, browsers and Web sites

The role of scripting

The DOM

• Platform and language neutral API's for manipulating XML documents with a rich model of events
• Specialized API's for HTML and CSS
• Widely used for customizing Web pages
  • DHTML for dynamically configurable user interfaces
  • XMLHTTP and live pages (AJAX)
• Emerging opportunities for enhancing the DOM
  • User preferences, device capabilities, device configuration and environmental conditions
  • Device coordination and access to remote services
• Layering APIs on protocols
  • DOM property with get/set methods and events
  • Local manifestation of a remote property
  • Underlying implementation propagates changes
  • Implemented on top of HTTP, SIP or other protocol

Freedom of Choice

Multimodal Interaction

Giving users a choice of modes:

• Speech, keypad or pen for input from user
  • Speech recognition and recording
  • Keystrokes and pointer gestures
  • Handwriting recognition and trace capture
• Aural, visual or tactile output to user
  • Conventional display for visual output
  • Speech synthesis and pre-recorded audio
  • Phone vibrator and Haptic interfaces, e.g. BMW iDrive
• Text as the common denominator and basis for accessibility
• Need to dynamically adapt to the changing context
  • Context Awareness
    • Personal preferences
    • Noisy environment
    • Restaurants and Theatres
    • Low battery
    • Poor reception
    • Geographical location (GPS)

Dialogues

For speech and for small displays, the user interface has to be broken down into a sequence of small steps.

• Dialogue between the user and the application
  • Expressed via markup, client and server-side scripts
• Visual, aural, tactile or a combination of these
• Who is in the driving seat?
• Large displays offer users lots of choices
• Small displays don't ...
• Lead by the hand (Wizards)
• User initiative for changing task
  • Mouse click, key press or spoken command
• Translating user input into a form that is meaningful to the application

Extensible Multi-Modal Annotations (EMMA)

• XML format for data exchange between input processors and interaction management systems
• Enables processors to annotate application specific data with things like confidence scores, time stamps, mode of input, alternative recognition hypotheses, partial results, etc.
• Can be generated via combination of
  • Speech Recognition Grammar Specification (SRGS)
  • Semantic Interpretation for Speech Recognition (SISR)
• If the user says: I want to fly to New York next Tuesday
• The interpreted result can be represented in EMMA as follows:

  <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"> 
    <emma:interpretation id="int1" emma:confidence="0.75"> 
      <destination>New York</destination>
      <date>28062005</date>  
    </emma:interpretation>
  </emma:emma>
  

• Can also be generated from text obtained via keystrokes or handwriting recognition (SRGS and SISR are text based)

Authoring Multimodal Applications

• No complete standards based solution as yet
• Various approaches have been considered
• W3C Member contributions on SALT and X+V
• W3C MMI Working Group is developing a flexible architecture that would encompass a range of solutions
• Some other possible approaches
  • XHTML + scripting
  • XHTML + state transition networks
  • Plan and rule based approaches
  • Promise of XHTML + SVG
• Link to more details

Multimodal Interaction Architecture

• Modality components (MVC view)
  • Support interaction with the user
  • Can be simple or complex
  • Single or multiple modalities
  • Integrated or distributed across multiple devices/files
• Examples:
  • Speech component providing TTS and ASR services
  • XHTML web page providing visual interaction with a form
  • VoiceXML web page providing voice interaction with a form
  • Compound markup supporting multimodal interaction
  • Server based dialogue component encompassing multiple pages
• Modality components communicate via asynchronous messages
  • The DOM may hide the messaging layer/run-time framework
• Shared data model (MVC model)
• Optional Interaction Manager in coordination role (MVC controller)
  • e.g. event driven state transition network for synchronizing visual and aural modalities

Distributed DOM

We need to focus on enabling distributed DOM's

• Distributing DOM events
  • User interface events (interpreted user input)
  • Events signalling changes to user preferences, device configuration, device capabilities, environmental conditions, shared data model
  • Could be raised by clients or servers
• Synchronising DOM across multiple devices
  • Propagating changes to shared data model
  • Synchronising local copies of remote properties
    • ink levels in networked printer
    • geographical location of a mobile device
    • audio input and output levels and whether these are muted
  • Updating the DOM for a remote user interface
• Events and Updates can flow in both directions (in and out)
• Take place in the context of an application session
  • Both temporary and persistent sessions

Who does What?

• W3C is an effective forum for the authoring layer
  • Markup languages
  • DOM: interfaces for properties, methods and events
  • Application level view of sessions and resources
• IETF is a natural home for work on bindings to IETF protocols
  • Layering of distributed DOM on top of SIP, HTTP, etc.

Closing remarks

• W3C seeks to enable royalty free implementations of its Recommendations
• IETF work on protocol support for distributed DOM's would complement W3C's work on multimodal web applications
• Coordination through people participating in both efforts and through W3C/IETF liaison contacts
  • Dan Connolly, Philippe Le Hégaret, see also: public-ietf-w3c mailing list