QuicR - Media Delivery Protocol over QUIC

4.1. Typical Publish/Subscribe Message Exchange

A QuicR "Control Channel", based on QUIC streams, is used to setup properties for media delivery over a "Media Channel" which in turn can be over QUIC streams or QUIC datagrams. The Control channel is typically used to setup QuicR names to be used, control media properties during setup and mid-session, start/stop media delivery and so on. The media data itself is sent over the "Media Channel" and carry enough metadata to enable Relays and end-points to make a forward/drop decisions under congestion, for example. Further details on the messages can be found in Section 7.¶

Below is an high-level exchange capturing publish/subscribe flow between Alice, the publisher and Bob, Carl, the subscribers and the interactions that occur between Relays on-path and the origin server. The details on how the trust setup happens between these entities are skipped, however.¶

In the exchange depicted following sequence happen¶

• Alice sets up a control channel (QUIC Stream) to the relay indicating its intent to publish media with name (video1/1) as the representation id. It does so by sending publish_intent. video1/1 might represent video stream camera-1, quality-id 1 (HD), for example. QuicR manifests (Section 6) are used to setup and consume the names being published.¶

• On receiving the publish_intent from Alice, the Relay sets up another control channel to the authorized Origin server and forwards Alice's publish_intent message.¶

• Once publish_intent_ok is received from the Origin, Relay forwards the same to Alice to enable publishing the media over the media channel [QUIC Stream or QUIC Datagram]¶

• In the meanwhile, Bob and Carl subscribe to receiving media corresponding to the wildcard'ed name (video1/*). They each send subscribe messages to the Relay on the control channel and the same is forwarded by the Relay to the Origin. Successful subscribe responses are sent back to Bob and via the relay. Relay makes note of Bob and Carl's interest in the name (video1/*). The details of knowing the name via manifest is skipped in the callflow.¶

• Eventually, Alice publishes media on the name (video1/1) towards the relay on the media channel, which could be over QUIC Streams or QUIC Datagram as chosen by Alice.¶

• Media from Alice gets cached at the relay and is forwarded to the Origin server (optionally). On noting about interested subscribers, the media received from Alice is forwarded to both Bob and Carl from the local cache by the Relay.¶

┌───────┐     ┌───────┐    ┌───────┐        ┌───────┐         ┌───────┐
│ Alice │     │  Bob  │    │ Carl  │        │ Relay │         │Origin │
└───┬───┘     └───┬───┘    └───┬───┘        └───┬───┘         └───┬───┘
    │                          │                │                 │
    │          ctrl: pub_intent│                │                 │
    │             (video1/1)   │                │ ctrl:pub_intent │
    │                          │                │   (video1/1)    │
    ├─────────────┼────────────┼───────────────▶│                 │
    │             │            │                ├─────────────────▶
    │             │            │                │
    │                          │                │ ctrl:pub_intent_ok
    │           ctrl: pub_inten│_ok             │
    │                          │                │◀────────────────┤
    ◀─────────────┼────────────┼────────────────┤                 │
    │             │                             │                 │
    │             │  ctrl: subscribe (video1/*) │                 │
    │             │                             │                 │
    │             ├────────────┼────────────────▶ ctrl: subscribe │
    │             │            │                │   (video1/*)    │
    │             │            │                │─────────────────▶
    │             │            │                │                 │
    │             │            │                │     ctrl:       │
    │             │            │                │  subscribe_ok   │
    │             │            │                │   (video1/*)    │
    │             │            │                │                 │
    │             │            │                ◀─────────────────┤
    │             │            │                │                 │
    │             │                             │─────┐  add bob: │
    │             │ ctrl: subscribe_ok          │ ◀───┘  video1/* │
    │             │                             │                 │
    │             │◀───────────┼────────────────┤                 │
    │             │            │                │                 │
    │             │            │  ctrl:sub:     │                 │
    │             │            │  (video1/*)    │                 │
    │             │            │                │                 │
    │             │            ├────────────────▶ ctrl: subscribe │
    │             │            │                │   (video1/*)    │
    │             │            │                │─────────────────▶
    │             │            │                │     ctrl:       │
    │             │            │                │  subscribe_ok   │
    │             │            │                │   (video1/*)    │
    │             │            │                │                 │
    │             │            │                ◀─────────────────┤
    │             │            │                │                 │
    │             │                             ├────┐  add carl: │
    │             │   ctrl: subscribe_ok        │◀───┘  video1/*  │
    │             │                             │                 │
    │             │◀───────────┼────────────────┤                 │
    │                          │                │                 │
    │    media:pub:video1/1    │                │                 │
    │                          │                │     cache       │
    ├─────────────┼────────────┼───────────────▶│                 │
    │             │            │                │─────┐           │
    │             │            │                │ ◀───┘    [pub]  │
    │             │            │                │                 │
    │                          │  media:pub:    ├─────────────────▶
    │  media:pub:              │  (video1/1)    │                 │
    │  (video1/1)              │                │                 │
    │                          │◀───────────────┤                 │
    │◀────────────┼────────────┼────────────────┤                 │
    │             │            │                │                 │
    │             │            │                                  │

{: title="Pub/Sub flow between Alice(publisher), Bob, Carl (subscribers), Relay and Origin"}¶

4.2. Origin Server

The Origin serves as the authorization authority for the named resources, in the manner similar to an HTTP origin. QuicR names to be used under a given domain and the application are authorized by the Origin server. It is also responsible for establishing necessary trust relationship between the clients, the relay and itself.¶

4.3. Relays

The Relays play an important role within the QuicR architecture. They receive subscriptions and intent to publish and forwards them towards the origin. This may involve sending messages directly to the Origin Relay or possibly traverse another Relay on the path. Replies to theses message follow the reverse direction of the request and when the Origin gives the OK to a subscription or intent to publish, the Relay allows the subscription or future publishes to the Names in the request. In this role, the Relays perform the role of subscribers (consumers) as well as publishers (producers).¶

Subscriptions received are aggregated. When a relay receives a publish request with data, it will forward it both towards the Origin and to any clients or relays that have a matching subscriptions. This "short circuit" of distribution by a relay before the data has even reached the Origin servers provides significant latency reduction for clients closer to the relay.¶

The Relay keeps an outgoing queue of objects to be sent to the each subscriber and objects are sent in strict priority order. Relays MAY cache some of the information for short period of time and the time cached may depend on the origin and local cache policies.¶

Section 8 covers further details on the Relay functionality.¶

5.1. Named Objects

The names of each object in QuicR is composed of the following components:¶

1. Domain Identifier¶
2. Application Identifier¶
3. Data Identifier¶

Domain component uniquely identifies a given application domain. This is like a HTTP Origin or an standardized identifier that uniquely identifies the application and a root relay function.¶

Application component is scoped under a given Domain. This component identifies aspects specific to a given application instance hosted under a given domain (e.g. which movie or meeting identifier).¶

Data identifier identifies aspects of application, for example representation_id in a CMAF segment or video stream from a conference user. In cases where media being delivered is naturally grouped into independently consumable groups (video group of picture or audio synchronization points for example), this component is futher composed into set of such groups, which are in turn made up of set of objects (video frames idr, p-frame within a given gop). Each such group is identified by a monotonically increasing integer and objects within the group are also identified by another set of monotonically increasing integers. The groupID and objectID start at 0.¶

Example: In the example below the domain component identifies acme.meeting.com domain, the application component identifies an instance of a meeting under this domain, say "meeting123", and the data component captures high resolution camera stream from the user "alice" being published as object 17 under group 15.¶

Example 1
quicr://acme.meeting.com/meeting123/alice/cam5/HiRes/15/17

Example 2
quicr://twitch.com/channel-fluffy/video-quality-id/group12/object0

Once a named object is created, the content inside the named object can never be changed. Objects have an expiry time after which they should be discarded by caches. Objects have an priority that the relays and clients can use to make drop decisions or sequencing the sending order. The data inside an object is end-to-end encrypted whose keys are not available to Relay(s).¶

5.2. Wildcarding with Names

QuicR allows subscribers to request for media based on wildcard'ed names. Wildcarding enables subscribes/requests for media to be made as aggregates instead of at the object level granularity. Wildcard names are formed by skipping the right most segments of the "Data Identifier" component of the names.¶

For example, in an web conferencing use case, the client may subscribe to just the origin, meeting_id and one of the publishers so as to get all the media from that user in a particular. The example matches all the named objects published by the user alice in the meeting123.¶

quicr://acme.meeting.com/meeting123/alice/*¶

When subscribing, there is an option to tell the relay to one of:¶

A. Deliver any new objects it receives that match the name¶

B. Deliver any new objects it receives and in addition send any previous objects it has received that are in the same group that matches the name.¶

C. Wait until an object that has a objectId that matches the name is received then start sending any objects that match the name.¶

6.1. Scope of the manifest

The role of the manifest is to identify the names as well as aspects pertaining to the associated data in a given usage context of the application.¶

• Typically a manifest identifies the domain and application aspects for the set of names that can be published.¶

• The content of Manifest is application defined and end-to-end encrypted.¶

• The manifest is owned by the application's origin server and are accessed as a protected resources by the authorized QuicR clients.¶

• The QuicR protocol treats Manifests as a named object, thus allowing for clients to subscribe for the purposes of bootstrapping into the session as well as to follow manifest changes during a session [ new members joining a conference for example].¶

• The manifest has well known name on the Origin server.¶

Also to note, a given application might provide non QuicR mechanisms to retrieve the manifest.¶

Below is a sample manifest for streaming application where a media presentation server describes media streams available for distribution. For downstream distribution of media data to clients with varying requirements, the central server (along with the source) generate different quality media representations. Each such quality is represented with a unique name and subscribers are made know of the same via the Manifest.¶

Note: Below json representation is provided as an example and need not be the only way to encode the manifest.¶

{
  "liveSessionID" : "jon.doe.music.live.tv",
  "streams: [
      {
      "id": "1234",
      "codec": "av1",
      "quality": "1280x720_30fps",
      "bitrate": "1200kbps",
      "crypto": "aes128-gcm",
      },
      {
      "id": "5678",
      "codec": "av1",
      "quality": "3840x2160_30fps",
      "bitrate": "4000kbps",
      "crypto": "aes256-gcm",
      },
      {
      "id": "9999",
      "codec": "av1",
      "quality": "640x480_30fps",
      "crypto": "aes128-gcm",
      },
  ]
}

Given the above manifest, if a publisher wants to send a 4k stream, it does so by publishing objects following the below pattern¶

//4K Stream
publish: quicr://jon.doe.music.live.tv/video/5678/[groupN/objectM ...]

If at any point, the producer intends to move to a lower quality stream due to underlying network conditions, it can do so by switching the published stream as below¶

//480p Stream
publish: quicr://jon.doe.music.live.tv/video/9999/[groupN/objectM ...]

and subscriber who is capable of receiving any of the qualities can do so by¶

// Receiving any of the published streams and use its name
// to know more information about media properties via manifest and
// object data
subscribe: quicr://jon.doe.music.live.tv/video/*

7.1. Control Channel

When a client or relay begins a transaction with the relay/origin, the client starts by opening a new bilateral stream. This stream will act as the "control channel" for the exchange of data, carrying a series of control messages in both directions. The same channel can be used for carrying media "fragment" messages if the media data is sent in "stream" mode.¶

The control stream will remain open as long as the peers are still sending or receiving the media. If either peer closes the control stream, the other peer will close its end of the stream and discard the state associated with the media transfer.¶

Streams are "one way". If a peer both sends and receive media, there will be different control streams for sending and receiving.¶

7.2. QuicR Control Messages

The control channel carry series of messages, encoded as a length followed by a message value:¶

quicr_message {
    length(16),
    value(...)
}

The length is encoded as a 16 bit number in big endian network order.¶

enum subscribe_intent
{
  immediate(0),
  catch_up(1),
  wait_up(2),
}

quicr_subscribe_message {
 *     message_type(i),
 *     name_length(i),
 *     name(...),
 *     mask(7)
 *     subscribe_intent intent,
 *     [datagram_stream_id(i)]
 * }

enum response
{
  ok(0),
  expired(1)
  fail(2),
  redirect(2)
}

quicr_subscribe_reply
{
    Response response
    [Reason Phrase Length (i),
    [Reason Phrase (..)],
}

quicr_publish_intent_message {
 *     message_type(i),
 *     name_length(i),
 *     name(...)
 *     datagram_capable(i)
 * }

quicr_publish_intent_ok_message {
  *     message_type(i),
  *     use_datagram(i),
  *     [datagram_stream_id(i)]
}

 * quicr_start_point_message {
 *     message_type(i),
 *     start_group_id(i),
 *     start_object_id(i),
 *     flags(i)
 * }

quicr_fragment_message {
 *     message_type(i),
 *     group_id(i),
 *     object_id(i),
 *     best_before(i),
 *     flags(i),
 *     offset_and_fin(i),
 *     length(i),
 *     data(...)
 }

 flags := Reserved (3) | IsDiscardable (1) | Priority (3)

offset_and_fin = 2 * offset + is_last_fragment

 * quicr_fin_message {
 *     message_type(i),
 *     final_group_id(i),
 *     final_object_id(i)
 * }

 * quicr_subscribe_cancel_message {
 *     message_type(i),
 *     name_length(i),
 *     name(...)
 *     Reason Phrase Length (i),
 *     Reason Phrase (..),
 * }

quicr_relay_redirect_message
{
  relay_address_length(i),
  relay_address(...)
}

7.3. Sending Media as Datagrams

If transmission as datagram is negotiated, the media fragments are sent as QUIC Datagram frames.¶

datagram_frame_content {
    datagram_header,
    datagram_content
}

 * quicr_datagram_header {
 *     datagram_stream_id (i),
 *     group_id (i),
 *     object_id (i),
 *     offset_and_fin (i),
 *     best_before(i),
 *     flags (8),
 *     [nb_objects_previous_group (i)]
 * }

 flags := Reserved (3) | IsDiscardable (1) | Priority (3)

offset_and_fin = 2*offset + is_last_fragment

8.1. Cache and Relaying

The prototype relays maintain a separate cache of received fragments for each media stream that it is processing. If fragments are received in stream mode, they will arrive in order. If fragments are received in datagram mode, fragments may arrive out of order.¶

The cache is created the first time a client connection refers to the media URL. This might be:¶

• A client connection requesting the name, in which case the relay will ask a copy of the media from the origin or the next hop relay towards the origin.¶

• A client connection publishing named data, in which case the relay will post a copy of the media towards the origin.¶

Once the media is available, the relay will learn the starting group ID and object ID.¶

Fragments are received from the "incoming" connection. If fragments are received in stream mode, they will arrive in order. If fragments are received in datagram mode, fragments may arrive out of order. When receiving in datagram mode, the media order is used to remove incoming duplicate fragments. When a non duplicate fragment is received, it is added to the cache and posted to corresponding subscribers over streams or datagrams, when flow and congestion control allow transmissions¶

In stream mode, the transmission may be delayed until fragments are received in order. If the last fragment received "fills a hole", that fragment and the next available fragments in media order will be forwarded.¶

8.2. Out of order relaying

As noted in Section 8.1, fragments that arrive out of order are relayed immediately.¶

This design has better properties compared to the following alternatives:¶

• insisting on full order before relaying, as is done for stream mode; OR¶

• insisting on full reception of all fragments making an object.¶

Full order would introduce the same head-of-line blocking also visible in stream-based relays. In theory, relaying full objects without requiring that objects be ordered would avoid some of the head-of-line blocking, but in practice it is observed when some streams contain large and small objects, and that losses affecting fragments of large objects cause almost the same head of line blocking delays as full ordering. Moreover, if losses happen at several places in the relay graph, the delays will accumulate. Out of order relaying avoids these delays.¶

8.3. Relay or Cache or Drop Decisions

Relays makes use of priority, time-to-live, is_discardable metadata properties from the published data to make forward or drop decisions when reacting to congestion as indicated by the underlying QUIC stack. The same can be used to make caching decisions.¶

8.4. Cache cleanup

Relays store objects no more than best_before time associated with the object. Congestion/Rate control feedback can further influence what gets cached based on the relative priority and rate at which data can be delivered. Local cache policies can also limit the amount and duration of data that can be cached.¶

8.5. Relay fail over

A relay that wants to shutdown shall use the redirect message to move traffic to a new relay. If a relay has failed and restarted or been load balanced to a different relay, the client will need to resubscribe to the new relay after setting up the connection.¶

TODO: Cluster so high reliable relays should share subscription info and publication to minimize of loss of data during a full over.¶

8.6. Relay Discovery

Local relays can be discovered via MDNS query to TODO. Cloud relays are discovered via application defined ways that is out of scope of this document. A Relay can send a message to client with the address of new relay. Client moves to the new relay with all of its Subscriptions and then Client unsubscribes from old relay and closes connection to it.¶

This allows for make before break transfer from one relay to another so that no data is lost during transition. One of the uses of this is upgrade of the Relay software during operation.¶