[AVT] H.264/RFC 3984bis and SDP - levels, parameter-sets, etc.

[Randell Jesup <rjesup at wgate.com>]

To summarize the discussion today at the AVT meeting on RFC 3984bis and
open issue #7 - the interaction between "level downgrading" and
sprop-parameter-sets, and my suggestions:

1) The intent originally was to allow level-downgrading (per the 2-year-old
   clarification).  Note: this assumes that any decoder that can decode
   level N can decode levels 1.0 - N.  (Also note that level 1b is a
   confusion factor here.)  Also note this assumes that level-downgrading
   is due to inability (in some way) or unwillingness to handle levels
   higher than the response indicates.

2) Doing so interacts badly with how sprop-parameter-sets is handled in the
   spec, since a level downgrade may leave you with unusable parameter
   sets.  I'd contend it's close to impractical/impossible to implement as
   written.

3) Many existing implementations do not seem to have implemented the spec
   as written, especially when it comes to packetization-mode and 
   sprop-parameter-sets (and parameter-add).  (MASSIVE UNDERSTATEMENT)

If anyone disagrees with those assertions, please speak up.

Possible solutions/resolutions discussed:

1) Disallow level downgrading.  This was not the intent; existing
   implementations downgrade, and it would mean allowing downgrading
   requires a large number of payload types be added as lower-level
   alternatives.

2) Specify parameter-sets that include parameters for each level below the
   one offered.  This is smaller and requires less explosion of SDP lines
   than option 1, but still may require a LOT of parameter-sets, and some
   of them can be large.  The answerer can chose any lower level for which
   there's a parameter set.

3) Find some way that the answerer can modify the offered parameter-set to
   lower the level.  Since modifying a paramset you didn't generate is
   unlikely to be a workable idea (H.264 specialists, feel free to correct
   this impression!), perhaps the idea would be something like: "on a level
   downgrade, the parameter-sets offered in the response shall be used by
   the offerer if acceptable, otherwise the offerer shall re-negotiate the
   session using the lower level from the response".

   Downside: requires a possible re-INVITE/etc.

4) On level downgrade (or on downgrade when there was no parameter-set
   for the new level), re-INVITE/etc (re-negotiate).  (See previous answer
   for a way to avoid this in some cases.)

(Not really discussed in detail in the meeting):
5) Allow in-band parameter sets explicitly and allow a way to signal their
   use in SDP, either in general or for this case.

   This allows the actual parameter sets to be in the stream, so the
   offerer can choose the sets after seeing the level-downgrade.

   Downside: parameter-sets are now contained in unreliable transmission,
   and so can be lost.  An IDR request (FIR/etc) could be defined to (in
   this case at least) regenerate the parameter-sets in the stream, but
   this won't help cases with a non-existant or non-functional return-path.

   This could be something like:

      parameters-in-media=1

   or implicitly signalled by a lack of sprop-parameter-sets.  I'm not sure
   which one would be more compatible with existing implementations -
   comments?

   That also could be in an offer (or a lack of sprop-parameter-sets
   implicitly already does this).

   This has the (MAJOR) advantage of trying to standardize existing
   (mis)-use of RFC 3984.  This is done all the time; many implementations
   totally ignore sprop-parameter-sets, let alone deal with the subtleties
   of parameter-add, or the offer-answer requirement to include the offer's
   parameter-set, and make sure ones added don't conflict with it.

   This current incompatibility between implementations that try to follow
   the RFC as written and others that kinda-follow it is a REAL problem.  I
   know (confirmed by Roni) that most of the conferencing/H.323-derived
   H.264 devices implement in-media parameter-sets.

   Note that using in-media parameter sets may limit the ability to
   change them reliably in mid-stream, if that's an issue, and it may not be.


Some additional issues for 3984bis:

3984bis should include the clarification about how level 1b is signalled
(reserved bit in the middle byte is used for constraint_3, per the email
on AVT).

How does level downgrade to level 1b work?  Is it allowed?  (Note that
level 1b is signalled as "level 1.1 with constraint_3 set".)  What about
level downgrade *from* level 1b? (to 1.1 or 1.0)?  This was discussed, but
the answer/consensus wasn't clear (to me), and time was an issue at
the meeting due to SVC taking a lot....

How does level downgrade interact with the text on offer-matching, since
it's matched on profile-level-id, packetization-mode, and (for mode 2)
sprop-deint-buf-req?

How does all this interact with MCU/conferencing servers?

We *really* need to get H.264 users on at least the same "chapter" when it
comes to SDP negotiation; right now they're barely in the same book, let
alone chapter or page.  And really, unless we want to majorly rewrite the
3984 rules, everyone *needs* to support mode 1.  It's not that hard, guys,
and it cuts the overhead of in-media parameter-sets because you can use a
STAP-A packet.  I suspect it also helps compression (a little? more? at
all?), especially in smaller-MTU transports like cellphones, since
NAL/slice size isn't as constrained.

Right now, it seems like barely a majority (if that) even know mode 1
exists.


For reference, I'm including my previous email, though I'm not really
replying to it directly.

Randell Jesup <rjesup at wgate.com> writes:
>>Any comments on this clarification?
>
>Yes:
>
>There is another side-effect on the text of RFC 3984 of this clarification.
>
>In particular, this text (in description of Offer/Answer):
>
>  o  The "sprop-parameter-sets" parameter is used as described above.
>      In addition, an answerer MUST maintain all parameter sets received
>      in the offer in its answer.  Depending on the value of the
>      "parameter-add" parameter, different rules apply: If "parameter-
>      add" is false (0), the answer MUST NOT add any additional
>      parameter sets.  If "parameter-add" is true (1), the answerer, in
>      its answer, MAY add additional parameter sets to the "sprop-
>      parameter-sets" parameter.  The answerer MUST also, independent of
>      the value of "parameter-add", accept to receive a video stream
>      using the sprop-parameter-sets it declared in the answer.
>
>         Informative note: care must be taken when parameter sets are
>         added not to cause overwriting of already transmitted parameter
>         sets by using conflicting parameter set identifiers.
>
>is a real problem.
>
>If you offer level 3.0, and include a level 3.0 sprop-paramater-set (which
>you should), then per the above, I MUST maintain the 3.0 parameter-set in
>my response.  If I want to respond with level 1.0, Magnus says above
>that I can, but I'm going to have to include a level 3.0 parameter set.
>I can also (if allowed by parameter-add) add a level 1.0 set.  When you add
>this:
>
> o  The parameters "sprop-parameter-sets", "sprop-deint-buf-req",
>      "sprop-interleaving-depth", "sprop-max-don-diff", and "sprop-
>      init-buf-time" describe the properties of the NAL unit stream that
>      the offerer or answerer is sending for this media format
>      configuration.  This differs from the normal usage of the
>      Offer/Answer parameters: normally such parameters declare the
>      properties of the stream that the offerer or the answerer is able
>      to receive.  When dealing with H.264, the offerer assumes that the
>      answerer will be able to receive media encoded using the
>      configuration being offered.
>
>That says the offerer should assume the answerer can receive it, and the
>previous quote says that the answerer MUST be willing to receive any
>parameter-set in the answer, and it also says the answerer MUST maintain
>the original parameter sets in the answer.  Since in most cases, the reason
>for downgrading the level in a response is the inability to handle the
>level of the offer, the answerer won't be able to handle video encoded
>using the parameter sets from the offer.  
>
>Thus, I see no way an answerer can validly answer with a downgrade unless
>it internally decides to lie, and drop all video packets until the offerer
>renegotiates to remove the unusable parameter sets.
>
>Also, to verify: does this mean that offer-matching is done using the first
>4 digits of profile-level-id (and packetization-mode and
>sprop-deint-buf-req as needed)?  Should "is the same" (when applied to
>profile-level-id) always mean "first 4 digits the same, and last two digits
>be the same or lower"?
>
>
>This really, really, really needs a rewrite.  It's also so complex, it
>needs more examples, and if possible an informational(?) algorithm on
>how to respond. Right now, I'm not certain *anyone* has come even close to
>100% compliance.  (Maybe one or two have, but...)
>
>Existing examples of H.264 traces I've seen show huge variations:
>
>*) Many respond without a matching packetization-mode (i.e. respond to an
>   offer of packetization-mode=1 with no packetization-mode)
>*) Some respond with a level *higher* than the offer.
>*) None I've seen include the sprop-parameter-sets from the offerer.
>*) I haven't seen any that respect parameter-add, though given the previous
>   point it may not matter.
>*) Some don't even include profile-level-id at all.
>*) Some accept packetization-mode=1 in the SDP, but send packetization-mode=0.
>and there are more problems, that's just the start.  :-(

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
rjesup at wgate.com
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

_______________________________________________
Audio/Video Transport Working Group
avt at ietf.org
https://www.ietf.org/mailman/listinfo/avt