< draft-ietf-tictoc-multi-path-synchronization-05.txt   draft-ietf-tictoc-multi-path-synchronization-06.txt >
Network Working Group A. Shpiner Network Working Group A. Shpiner
Internet Draft Mellanox Internet Draft Mellanox
Intended status: Experimental R. Tse Intended status: Experimental R. Tse
Expires: February 2017 C. Schelp Expires: April 2017 PMC-Sierra
PMC-Sierra C. Schelp
Oracle
T. Mizrahi T. Mizrahi
Marvell Marvell
August 25, 2016 October 6, 2016
Multi-Path Time Synchronization Multi-Path Time Synchronization
draft-ietf-tictoc-multi-path-synchronization-05.txt draft-ietf-tictoc-multi-path-synchronization-06.txt
Abstract Abstract
Clock synchronization protocols are very widely used in IP-based Clock synchronization protocols are very widely used in IP-based
networks. The Network Time Protocol (NTP) has been commonly deployed networks. The Network Time Protocol (NTP) has been commonly deployed
for many years, and the last few years have seen an increasingly for many years, and the last few years have seen an increasingly
rapid deployment of the Precision Time Protocol (PTP). As time- rapid deployment of the Precision Time Protocol (PTP). As time-
sensitive applications evolve, clock accuracy requirements are sensitive applications evolve, clock accuracy requirements are
becoming increasingly stringent, requiring the time synchronization becoming increasingly stringent, requiring the time synchronization
protocols to provide high accuracy. Slave Diversity is a recently protocols to provide high accuracy. This memo describes a multi-path
introduced approach, where the master and slave clocks (also known as approach to PTP and NTP over IP networks, allowing the protocols to
server and client) are connected through multiple network paths, and run concurrently over multiple communication paths between the master
the slave combines the information received through all paths to and slave clocks, without modifying these protocols. The multi-path
obtain a higher clock accuracy compared to the conventional one-path approach can significantly contribute to clock accuracy, security and
approach. This document describes a multi-path approach to PTP and fault tolerance. The multi-path approach that is presented in this
NTP over IP networks, allowing the protocols to run concurrently over document enables backward compatibility with nodes that do not
multiple communication paths between the master and slave clocks. The support the multi-path functionality.
multi-path approach can significantly contribute to clock accuracy,
security and fault tolerance. The Multi-Path Precision Time Protocol
(MPPTP) and Multi-Path Network Time Protocol (MPNTP) define an
additional layer that extends the existing PTP and NTP without the
need to modify these protocols. MPPTP and MPNTP also allow backward
compatibility with nodes that do not support the multi-path
extension.
Status of this Memo Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 2, line 16 skipping to change at page 2, line 8
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on February 25, 2017. This Internet-Draft will expire on April 6, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................3 1. Introduction...................................................3
2. Conventions Used in this Document..............................5 2. Conventions Used in this Document..............................4
2.1. Abbreviations.............................................5 2.1. Abbreviations.............................................4
2.2. Terminology...............................................5 2.2. Terminology...............................................5
3. Multiple Paths in IP Networks..................................5 3. Multiple Paths in IP Networks..................................5
3.1. Load Balancing............................................5 3.1. Load Balancing............................................5
3.2. Using Multiple Paths Concurrently.........................6 3.2. Using Multiple Paths Concurrently.........................5
3.3. Two-Way Paths.............................................6 3.3. Two-Way Paths.............................................6
4. Solution Overview..............................................6 4. Solution Overview..............................................6
4.1. Path Configuration and Identification.....................6 4.1. Path Configuration and Identification.....................6
4.2. Combining.................................................7 4.2. Combining.................................................7
5. Multi-Path Time Synchronization Protocols over IP Networks.....7 5. Multi-Path Time Synchronization over IP Networks...............7
5.1. Overview..................................................7 5.1. Overview..................................................7
5.2. Single-Ended Multi-Path Synchronization...................9 5.2. Single-Ended Multi-Path Synchronization...................8
5.2.1. Single-Ended MPPTP Synchronization Message Exchange..9 5.2.1. Single-Ended MPPTP Synchronization Message Exchange..8
5.2.2. Single-Ended MPNTP Synchronization Message Exchange.10 5.2.2. Single-Ended MPNTP Synchronization Message Exchange..9
5.3. Dual-Ended Multi-Path Synchronization....................10 5.3. Dual-Ended Multi-Path Synchronization....................10
5.3.1. Dual-Ended MPPTP Synchronization Message Exchange...11 5.3.1. Dual-Ended MPPTP Synchronization Message Exchange...10
5.3.2. Dual-Ended MPNTP Synchronization Message Exchange...12 5.3.2. Dual-Ended MPNTP Synchronization Message Exchange...11
5.4. Using Traceroute for Path Discovery......................12 5.4. Using Traceroute for Path Discovery......................12
5.5. Using Unicast Discovery for MPPTP........................13 5.5. Using Unicast Discovery for MPPTP........................13
6. Combining Algorithm...........................................13 6. Combining Algorithm...........................................13
7. Security Considerations.......................................14 7. Security Considerations.......................................14
8. IANA Considerations...........................................14 8. IANA Considerations...........................................14
9. Acknowledgments...............................................14 9. Acknowledgments...............................................14
10. References...................................................14 10. References...................................................14
10.1. Normative References....................................14 10.1. Normative References....................................14
10.2. Informative References..................................14 10.2. Informative References..................................15
1. Introduction 1. Introduction
The two most common time synchronization protocols in IP networks are The two most common time synchronization protocols in IP networks are
the Network Time Protocol [NTP], and the Precision Time Protocol the Network Time Protocol [NTP], and the Precision Time Protocol
(PTP), defined in the IEEE 1588 standard [IEEE1588]. (PTP), defined in the IEEE 1588 standard [IEEE1588].
The accuracy of the time synchronization protocols directly depends The accuracy of the time synchronization protocols directly depends
on the stability and the symmetry of propagation delays on both on the stability and the symmetry of propagation delays on both
directions between the master and slave clocks. Depending on the directions between the master and slave clocks. Depending on the
nature of the underlying network, time synchronization protocol nature of the underlying network, time synchronization protocol
skipping to change at page 4, line 31 skipping to change at page 4, line 16
one path in the network, as shown in Figure 1, [SLAVEDIV] suggested one path in the network, as shown in Figure 1, [SLAVEDIV] suggested
that a time synchronization protocol can be run over multiple paths, that a time synchronization protocol can be run over multiple paths,
providing several advantages. First, it can significantly increase providing several advantages. First, it can significantly increase
the clock accuracy as shown in [SLAVEDIV]. Second, this approach the clock accuracy as shown in [SLAVEDIV]. Second, this approach
provides additional security, allowing to mitigate man-in-the-middle provides additional security, allowing to mitigate man-in-the-middle
attacks against the time synchronization protocol [DELAY-ATT]. Third, attacks against the time synchronization protocol [DELAY-ATT]. Third,
using multiple paths concurrently provides an inherent failure using multiple paths concurrently provides an inherent failure
protection mechanism. protection mechanism.
This document introduces Multi-Path PTP (MPPTP) and Multi-Path NTP This document introduces Multi-Path PTP (MPPTP) and Multi-Path NTP
(MPNTP), respectively. These extensions are defined at the network (MPNTP). The functionality of the multi-path approach is defined at
layer and do not require any changes in the PTP or in the NTP the network layer and does not require any changes in the PTP or in
protocols. the NTP protocols.
MPPTP and MPNTP are defined over IP networks. As IP networks MPPTP and MPNTP are defined over IP networks. As IP networks
typically combine ECMP routing, this property is leveraged for the typically combine ECMP routing, this property is leveraged for the
multiple paths used in MPPTP and MPNTP. The key property of the multiple paths used in MPPTP and MPNTP. The key property of the
multi-path extension is that clocks in the network can use more than multi-path approach is that clocks in the network can use more than
one IP address. Each {master IP, slave IP} address pair defines a one IP address. Each {master IP, slave IP} address pair defines a
path. Depending on the network topology and configuration, the IP path. Depending on the network topology and configuration, the IP
combination pairs can form multiple diverse paths used by the multi- combination pairs can form multiple diverse paths used by the multi-
path synchronization protocols. It has been shown [MULTI] that using path synchronization protocols. It has been shown [MULTI] that using
multiple IP addresses over the wide Internet indeed allows two end- multiple IP addresses over the wide Internet indeed allows two end-
points to attain multiple diverse paths. points to attain multiple diverse paths.
This document introduces two variants for each of the two multi-path This document introduces two variants of the multi-path approach; a
protocols; a variant that requires both master and slave nodes to variant that requires both master and slave nodes to support the
support the multi-path protocol, referred to as the dual-ended multi-path functionality, referred to as the dual-ended variant, and
variant, and a backward compatible variant that allows a multi-path a backward compatible variant that allows a multi-path clock to
clock to connect to a conventional single-path clock, referred to as connect to a conventional single-path clock, referred to as the
the single-ended variant. single-ended variant.
2. Conventions Used in this Document 2. Conventions Used in this Document
2.1. Abbreviations 2.1. Abbreviations
BMC Best Master Clock [IEEE1588] BMC Best Master Clock [IEEE1588]
ECMP Equal Cost Multiple Path ECMP Equal Cost Multiple Path
LAN Local Area Network LAN Local Area Network
skipping to change at page 7, line 29 skipping to change at page 7, line 12
IP address pairs, since some of the address pairs may share common IP address pairs, since some of the address pairs may share common
paths. paths.
4.2. Combining 4.2. Combining
Various methods can be used for combining the time information Various methods can be used for combining the time information
received from the different paths. The output of the combining received from the different paths. The output of the combining
algorithm is the accurate time offset. Combining methods are further algorithm is the accurate time offset. Combining methods are further
discussed in Section 6. discussed in Section 6.
5. Multi-Path Time Synchronization Protocols over IP Networks 5. Multi-Path Time Synchronization over IP Networks
5.1. Overview 5.1. Overview
This section presents two variants of MPPTP and MPNTP; single-ended This section presents two variants of MPPTP and MPNTP; single-ended
multi-path time synchronization and dual-ended multi-path time multi-path time synchronization and dual-ended multi-path time
synchronization. In the first variant, the multi-path protocol is run synchronization. In the first variant, the multi-path approach is
only by the slave and the master is not aware of its usage. In the only implemented by the slave and the master is not aware of its
second variant, all clocks must support the multi-path protocol. usage. In the second variant, all clocks use multiple paths.
The dual-ended protocol provides higher path diversity by using The dual-ended variant provides higher path diversity by using
multiple IP addresses at both ends, the master and slave, while the multiple IP addresses at both ends, the master and slave, while the
single-ended protocol only uses multiple addresses at the slave. On single-ended variant only uses multiple addresses at the slave.
the other hand, the dual-ended protocol can only be deployed when Consequently, the single-ended approach is can interoperate with
both the master and the slave support this protocol. Dual-ended and existing implementations, which do not use multiple paths. The dual-
single-ended protocols can co-exist in the same network. Each slave ended and single-ended approaches can co-exist in the same network;
selects the connection(s) it wants to make with the available each slave selects the connection(s) it wants to make with the
masters. A dual-ended slave could switch to single-ended mode if it available masters. A dual-ended slave could switch to single-ended
does not see any dual-ended masters available. A single-ended slave mode if it does not see any dual-ended masters available. A single-
could connect to a single IP address of a dual-ended master. ended slave could connect to a single IP address of a dual-ended
master.
Multi-path time synchronization, in both variants, requires clocks to Multi-path time synchronization, in both variants, requires clocks to
use multiple IP addresses. Using multiple IP addresses introduces a use multiple IP addresses. Using multiple IP addresses introduces a
tradeoff. A large number of IP addresses allows a large number of tradeoff. A large number of IP addresses allows a large number of
diverse paths, providing the advantages of slave diversity discussed diverse paths, providing the advantages of slave diversity discussed
in Section 1. On the other hand, a large number of IP addresses is in Section 1. On the other hand, a large number of IP addresses is
more costly, requires the network topology to be more redundant, and more costly, requires the network topology to be more redundant, and
exacts extra management overhead. exacts extra management overhead.
If possible, the set of IP addresses for each clock should be chosen If possible, the set of IP addresses for each clock should be chosen
skipping to change at page 8, line 39 skipping to change at page 8, line 20
discovery should be used to identify the possible paths between the discovery should be used to identify the possible paths between the
master and slave. Path discovery is further discussed in Section 5.4. master and slave. Path discovery is further discussed in Section 5.4.
The concept of using multiple IP addresses or multiple interfaces is The concept of using multiple IP addresses or multiple interfaces is
a well-established concept that is being used today by various a well-established concept that is being used today by various
applications and protocols, e.g., [MPTCP]. Using multiple interfaces applications and protocols, e.g., [MPTCP]. Using multiple interfaces
introduces some challenges and issues, which were presented and introduces some challenges and issues, which were presented and
discussed in [MIF]. discussed in [MIF].
The descriptions in this section refer to the end-to-end scheme of The descriptions in this section refer to the end-to-end scheme of
PTP, but are similarly applicable to the peer-to-peer scheme. The PTP, but are similarly applicable to the peer-to-peer scheme. MPNTP,
MPNTP protocol described in this document refers to the NTP client- as described in this document, refers to the NTP client-server mode,
server mode, although the concepts described here can be extended to although the concepts described here can be extended to include the
include the symmetric variant as well. symmetric variant as well.
Multi-path synchronization protocols by nature require protocol Multi-path synchronization by nature requires protocol messages to be
messages to be sent as unicast. Specifically in PTP, the following sent as unicast. Specifically in PTP, the following messages must be
messages must be sent as unicast in MPPTP: Sync, Delay_Req, sent as unicast in MPPTP: Sync, Delay_Req, Delay_Resp, PDelay_Req,
Delay_Resp, PDelay_Req, PDelay_Resp, Follow_Up, and PDelay_Resp, Follow_Up, and PDelay_Resp_Follow_Up. Note that
PDelay_Resp_Follow_Up. Note that [IEEE1588] allows these messages to [IEEE1588] allows these messages to be sent either as multicast or as
be sent either as multicast or as unicast. unicast.
5.2. Single-Ended Multi-Path Synchronization 5.2. Single-Ended Multi-Path Synchronization
In the single-ended approach, only the slave is aware of the fact In the single-ended approach, only the slave is aware of the fact
that multiple paths are used, while the master is agnostic to the that multiple paths are used, while the master is agnostic to the
usage of multiple paths. This approach allows a hybrid network, where usage of multiple paths. This approach allows a hybrid network, where
some of the clocks are multi-path clocks, and others are conventional some of the clocks are multi-path clocks, and others are conventional
one-path clocks. A single-ended multi-path clock presents itself to one-path clocks. A single-ended multi-path clock presents itself to
the network as N independent clocks, using N IP addresses, as well as the network as N independent clocks, using N IP addresses, as well as
N clock identity values (in PTP). Thus, the usage of multiple slave N clock identity values (in PTP). Thus, the usage of multiple slave
skipping to change at page 10, line 22 skipping to change at page 10, line 7
corresponding path delay and the offset from the master. corresponding path delay and the offset from the master.
o The slave combines the information from all negotiated paths. o The slave combines the information from all negotiated paths.
5.2.2. Single-Ended MPNTP Synchronization Message Exchange 5.2.2. Single-Ended MPNTP Synchronization Message Exchange
The single-ended MPNTP message exchange procedure is as follows. The single-ended MPNTP message exchange procedure is as follows.
o A single-ended MPNTP client has N separate identities, i.e., N IP o A single-ended MPNTP client has N separate identities, i.e., N IP
addresses. The assumption is that the server information, addresses. The assumption is that the server information,
including its IP address is known to the NTP clients. including its IP address is known to the NTP clients. This is a
fair assumption, as typically the address(es) of the NTP server(s)
are provided to the NTP client by configuration.
o A single-ended MPNTP client initiates the NTP protocol with an NTP o A single-ended MPNTP client initiates the NTP protocol with an NTP
server N times, using each of its N identities. server N times, using each of its N identities.
o The NTP protocol is maintained between the server and each of the o The NTP protocol is maintained between the server and each of the
N client identities. N client identities.
o The client sends NTP messages to the master using each of its N o The client sends NTP messages to the master using each of its N
identities. identities.
skipping to change at page 12, line 37 skipping to change at page 12, line 27
address combination from the received NTP packet. address combination from the received NTP packet.
o Using the {source, destination} IP address pair in the received o Using the {source, destination} IP address pair in the received
packets, the client identifies the path, and performs its packets, the client identifies the path, and performs its
computations for each of the paths accordingly. computations for each of the paths accordingly.
o The client combines the information from all paths. o The client combines the information from all paths.
5.4. Using Traceroute for Path Discovery 5.4. Using Traceroute for Path Discovery
The protocols presented above use multiple IP addresses in a single The approach described thus far uses multiple IP addresses in a
clock to create multiple paths. However, although each two-way path single clock to create multiple paths. However, although each two-way
is defined by a different {master, slave} address pair, some of the path is defined by a different {master, slave} address pair, some of
IP address pairs may traverse exactly the same network path, making the IP address pairs may traverse exactly the same network path,
them redundant. making them redundant.
Traceroute-based path discovery can be used for filtering only the IP Traceroute-based path discovery can be used for filtering only the IP
addresses that obtain diverse paths. 'Paris Traceroute' [PARIS] and addresses that obtain diverse paths. 'Paris Traceroute' [PARIS] and
'TraceFlow' [TRACEFLOW] are examples of tools that discover the paths 'TraceFlow' [TRACEFLOW] are examples of tools that discover the paths
between two points in the network. It should be noted that this between two points in the network. It should be noted that this
filtering approach is effective only if the Traceroute implementation filtering approach is effective only if the Traceroute implementation
uses the same IP addresses and UDP ports as the synchronization uses the same IP addresses and UDP ports as the synchronization
protocol packets. Since some Traceroute implementations vary the UDP protocol packets. Since some Traceroute implementations vary the UDP
ports, they may not be effective in identifying and filtering ports, they may not be effective in identifying and filtering
redundant paths in synchronization protocols. redundant paths in synchronization protocols.
skipping to change at page 14, line 4 skipping to change at page 13, line 41
6. Combining Algorithm 6. Combining Algorithm
Previous sections discussed the methods of creating the multiple Previous sections discussed the methods of creating the multiple
paths and obtaining the time information required by the slave paths and obtaining the time information required by the slave
algorithm. Once the time information is received through each of the algorithm. Once the time information is received through each of the
paths, the slave should use a combining algorithm, which consolidates paths, the slave should use a combining algorithm, which consolidates
the information from the different paths into a single clock. the information from the different paths into a single clock.
Various methods have been suggested for combining information from Various methods have been suggested for combining information from
different paths or from different clocks, e.g., [NTP], [SLAVEDIV], different paths or from different clocks, e.g., [NTP], [SLAVEDIV],
[HIGH-AVAI], [KALMAN]. The choice of the combining algorithm is local [HIGH-AVAI], [KALMAN]. The choice of the combining algorithm is local
to the slave, and does not affect the interoperability of the to the slave, and does not affect interoperability. Hence, this
protocol. Hence, this document does not define a specific method to document does not define a specific method to be used by the slave.
be used by the slave. The combining algorithm should be chosen carefully based on the
system properties, as different combining algorithms provide
different advantages. For example, some combining algorithms (e.g.,
[NTP], [DELAY-ATT]) are intended to be robust in the face of security
attacks, while other combining algorithms (e.g., [KALMAN]) are more
resilient to random delay variation.
7. Security Considerations 7. Security Considerations
The security aspects of time synchronization protocols are discussed The security aspects of time synchronization protocols are discussed
in detail in [TICTOCSEC]. The methods describe in this document in detail in [TICTOCSEC]. The methods describe in this document
propose to run a time synchronization protocol through redundant propose to run a time synchronization protocol through redundant
paths, and thus allow to detect and mitigate man-in-the-middle paths, and thus allow to detect and mitigate man-in-the-middle
attacks, as described in [DELAY-ATT]. attacks, as described in [DELAY-ATT]. It should be noted that when
using multiple paths, these paths may partially overlap, and thus an
attack that takes place in a common segment of these paths is not
mitigated by the redundancy. Moreover, an on-path attacker may in
some cases have access to more than one router, or may be able to
migrate from one router to another. Therefore, when using multiple
paths it is important for the paths to be as diverse and as
independent as possible, making the redundancy scheme more tolerant
to on-path attacks.
It should be noted that the multi-path approach requires the master
(or NTP server) to dedicate more resources to each slave (client)
than the conventional single-path approach. Hence, well-known
Distributed Denial-of-Service (DDoS) attacks may porentially be
amplified when the multi-path approach is enabled.
8. IANA Considerations 8. IANA Considerations
There are no IANA actions required by this document. There are no IANA actions required by this document.
RFC Editor: please delete this section before publication. RFC Editor: please delete this section before publication.
9. Acknowledgments 9. Acknowledgments
The authors gratefully acknowledge the useful comments provided by The authors gratefully acknowledge the useful comments provided by
Peter Meyer, Doug Arnold, Joe Abley, and Zhen Cao, as well as other Peter Meyer, Doug Arnold, Joe Abley, Zhen Cao, Watson Ladd, and
comments received from the TICTOC working group participants. Mirja Kuehlewind, as well as other comments received from the TICTOC
working group participants.
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
10. References 10. References
10.1. Normative References 10.1. Normative References
[IEEE1588] IEEE Instrumentation and Measurement Society, "IEEE [IEEE1588] IEEE Instrumentation and Measurement Society, "IEEE
Standard for a Precision Clock Synchronization Standard for a Precision Clock Synchronization
Protocol for Networked Measurement and Control Protocol for Networked Measurement and Control
skipping to change at page 16, line 25 skipping to change at page 16, line 36
Richard Tse Richard Tse
PMC-Sierra PMC-Sierra
8555 Baxter Place 8555 Baxter Place
Burnaby, BC Burnaby, BC
Canada Canada
V5A 4V7 V5A 4V7
Email: Richard.Tse@pmcs.com Email: Richard.Tse@pmcs.com
Craig Schelp Craig Schelp
PMC-Sierra Oracle
8555 Baxter Place
Burnaby, BC
Canada
V5A 4V7
Email: craig.schelp@pmcs.com
Email: craig.schelp@gmail.com
Tal Mizrahi Tal Mizrahi
Marvell Marvell
6 Hamada St. 6 Hamada St.
Yokneam, 20692, Israel Yokneam, 20692, Israel
Email: talmi@marvell.com Email: talmi@marvell.com
 End of changes. 28 change blocks. 
83 lines changed or deleted 95 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/