< draft-pwouters-multi-sa-performance-00.txt		draft-pwouters-multi-sa-performance-01.txt >
	Network A. Antony		Network A. Antony
	Internet-Draft S. Klassert		Internet-Draft S. Klassert
	Intended status: Standards Track secunet		Intended status: Standards Track secunet
	Expires: May 6, 2021 P. Wouters		Expires: August 26, 2021 P. Wouters
	Red Hat		Red Hat
	November 2, 2020		February 22, 2021
	IKEv2 support for per-queue Child SAs		IKEv2 support for per-queue Child SAs
	draft-pwouters-multi-sa-performance-00		draft-pwouters-multi-sa-performance-01
	Abstract		Abstract
	This document defines two Notification Payload (NUM_QUEUES and		This document defines two Notification Payloads for the Internet Key
	QUEUE_INFO) for the Internet Key Exchange Protocol Version 2 (IKEv2).		Exchange Protocol Version 2 (IKEv2): NUM_QUEUES and QUEUE_INFO.
	These payloads add support for negotiating multiple identical Child		These payloads add support for indicating that the negotiating of
	SAs that can be used to to optimize performance based on the number		multiple identical Child SAs are to be used to optimize performance
	of queues or CPUs, orcw to create multiple Child SAs for different		based on the number of queues or CPUs, or to create multiple Child
	Quality of Service (QoS) levels.		SAs for different Quality of Service (QoS) levels. It indicates that
			a newer idetnical Child SA should not be interpreted as a replacement
			Child SA.
	Using multiple identical Child Sa's has the additional benefit that		Using multiple identical Child Sa's has the benefit that each stream
	multiple streams have their own Sequence Number, ensuring that CPU's		has its own Sequence Number, ensuring that CPU's don't have to
	don't have to synchronize their crypto state or disable their replay		synchronize their crypto state or disable their packet replay
	window detection.		detection.
	Status of This Memo		Status of This Memo
	This Internet-Draft is submitted in full conformance with the		This Internet-Draft is submitted in full conformance with the
	provisions of BCP 78 and BCP 79.		provisions of BCP 78 and BCP 79.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on May 6, 2021.		This Internet-Draft will expire on August 26, 2021.
	Copyright Notice		Copyright Notice
	Copyright (c) 2020 IETF Trust and the persons identified as the		Copyright (c) 2021 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(https://trustee.ietf.org/license-info) in effect on the date of		(https://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
	1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3		1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
	2. Performance bottlenecks . . . . . . . . . . . . . . . . . . . 3		2. Performance bottlenecks . . . . . . . . . . . . . . . . . . . 3
	3. Negotiation of performance specific Child SAs . . . . . . . . 3		3. Negotiation of performance specific Child SAs . . . . . . . . 3
	4. Implementation specifics . . . . . . . . . . . . . . . . . . 4		4. Implementation specifics . . . . . . . . . . . . . . . . . . 4
	4.1. One Child per CPU . . . . . . . . . . . . . . . . . . . . 4		4.1. One CPU per Child . . . . . . . . . . . . . . . . . . . . 4
	4.2. QoS Child SA's . . . . . . . . . . . . . . . . . . . . . 5		4.2. QoS Child SA's . . . . . . . . . . . . . . . . . . . . . 6
	5. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 6		5. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 6
	5.1. NUM_QUEUES Notify Payload . . . . . . . . . . . . . . . . 6		5.1. NUM_QUEUES Notify Payload . . . . . . . . . . . . . . . . 7
	5.2. QUEUE_INFO Notify Payload . . . . . . . . . . . . . . . . 6		5.2. QUEUE_INFO Notify Payload . . . . . . . . . . . . . . . . 7
	6. Security Considerations . . . . . . . . . . . . . . . . . . . 7		6. Security Considerations . . . . . . . . . . . . . . . . . . . 8
	7. Implementation Status . . . . . . . . . . . . . . . . . . . . 7		7. Implementation Status . . . . . . . . . . . . . . . . . . . . 8
	7.1. Linux XFRM . . . . . . . . . . . . . . . . . . . . . . . 8		7.1. Linux XFRM . . . . . . . . . . . . . . . . . . . . . . . 8
	7.2. Libreswan . . . . . . . . . . . . . . . . . . . . . . . . 8		7.2. Libreswan . . . . . . . . . . . . . . . . . . . . . . . . 9
	7.3. strongSWAN . . . . . . . . . . . . . . . . . . . . . . . 9		7.3. strongSWAN . . . . . . . . . . . . . . . . . . . . . . . 9
	8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9		8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
	9. References . . . . . . . . . . . . . . . . . . . . . . . . . 9		9. References . . . . . . . . . . . . . . . . . . . . . . . . . 10
	9.1. Normative References . . . . . . . . . . . . . . . . . . 9		9.1. Normative References . . . . . . . . . . . . . . . . . . 10
	9.2. Informative References . . . . . . . . . . . . . . . . . 10		9.2. Informative References . . . . . . . . . . . . . . . . . 10
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10		Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
	1. Introduction		1. Introduction
	IPsec implementations are currently limited to using one queue or CPU		IPsec implementations are currently limited to using one queue or CPU
	per Child SA. The result is that a machine with many queues/CPUs is		per Child SA. The result is that a machine with many queues/CPUs is
	limited to only using one these per Child SA. This severely limits		limited to only using one these per Child SA. This severely limits
	the speeds that can be obtained. An unencrypted link of 10gbps or		the speeds that can be obtained. An unencrypted link of 10gbps or
	more is commonly reduced to 2-3gbps when IPsec is used to encrypt the		more is commonly reduced to 2-3gbps when IPsec is used to encrypt the
	link, for example when using AES-GCM.		link, for example when using AES-GCM.
	Furthermore IPsec implementations are currently limited to use the		Furthermore IPsec implementations are currently limited to use the
	same Child SA for all Quality of Service (QoS) types bacause the QoS		same Child SA for all Quality of Service (QoS) types because the QoS
	type is not a part of the TS. The result is that IPsec can't do		type is not a part of the TS. The result is that IPsec can't do
	active Quality of Service priorizing without disabling the anti		active Quality of Service prioritizing without disabling the anti
	replay detection.		replay detection.
			While this could be mitigated by setting up multiple narrowed Child
			SA's, for example using Populate From Packet (PFP) as specified in
			[RFC4301], this IPsec feature is not widely implemented.
	To make better use of multiple network queues and CPUs, it can be		To make better use of multiple network queues and CPUs, it can be
	beneficial to negotiate and install multiple identical Child SAs.		beneficial to negotiate and install multiple identical Child SAs.
	IKEv2 [RFC7296] already allows installing multiple identical Child		IKEv2 [RFC7296] already allows installing multiple identical Child
	SAs, but often implementations will assume the older Child SA is		SAs, but often implementations will assume the older Child SA is
	being replaced by the newer Child Sa, even when no INITIAL_CONTACT		being replaced by the newer Child Sa, even when no INITIAL_CONTACT
	notify payload was received.		notify payload was received.
	When two IKEv2 peers want to negotiate multiple Child SAs, it would		When two IKEv2 peers want to negotiate multiple Child SAs, it is
	be useful for them to convey how many of these are considered		useful to be able to convey how many Child SAs are required for
	acceptable to install. This avoids triggering CREATE_CHILD_SA		optimized traffic. This avoids triggering CREATE_CHILD_SA exchanges
	exchanges that will be rejected with TS_UNACCEPTABLE.		that will only be rejected by the peer.
	1.1. Requirements Language		1.1. Requirements Language
	The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",		The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
	"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and		"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
	"OPTIONAL" in this document are to be interpreted as described in BCP		"OPTIONAL" in this document are to be interpreted as described in BCP
	14 [RFC2119] [RFC8174] when, and only when, they appear in all		14 [RFC2119] [RFC8174] when, and only when, they appear in all
	capitals, as shown here.		capitals, as shown here.
	2. Performance bottlenecks		2. Performance bottlenecks
	Currently, most IPsec implementations are limited by using one CPU or		Currently, most IPsec implementations are limited by using one CPU or
	network queue per Child SA. There are a number of performance		network queue per Child SA. There are a number of practical reasons
	reasons for this, but a key limitation is that sharing the AEAD		for this, but a key limitation is that sharing the AEAD state,
	state, counters and sequence numbers between multiple CPUs is not		counters and sequence numbers between multiple CPUs is not feasible
	feasible without a significant performance penalty. There is a need		without a significant performance penalty. There is a need to
	to negotiate and establish multiple Child SA's with identical TSi/TSr		negotiate and establish multiple Child SA's with identical TSi/TSr on
	on a per-queue or per-CPU basis.		a per-queue or per-CPU basis.
	3. Negotiation of performance specific Child SAs		3. Negotiation of performance specific Child SAs
	The number of Child SA's notify payload refers to the number of		The number of Child SA's notify payload refers to the number of
	instances for this particular TSi/TSr combination. Both ends send		instances for this particular TSi/TSr combination beyond the initial
	their Preferred number of Child SAs and the maximum of Child SAs they		Child SA. Both peers send their minimum number of Child SAs they
	are willing to install. Both ends pick the highest preferred number		prefer to install. Both peers pick the maximum iof the two numbers
	up to the lowest maximum number. That is if one end prefers 16 but		(within reason). That is if one peer prefers 16 and the other peer
	accepts 32, and the other end prefers 48 and accepts 48, the number		prefers 48, then the number negotiated is 48. If a 49th Child SA is
	picked is 32. If a 33rd Child SA is attempted, the peer with the 32		attempted with QUEUE_INFO notify payload, it can be rejected using
	maximum SHOULD return TS_UNACCEPTABLE.		TS_UNACCEPTABLE.
	The NUM_QUEUES Notify is sent as part of the IKE_AUTH or		The NUM_QUEUES Notify payload is sent as part of the IKE_AUTH or as
	CREATE_CHILD_SA message that contains the Traffic Selector payload		part of an CREATE_CHILD_SA Exchange for an initial new Child SA
	for a new Child SA. If there are multiple IKE_AUTH exchanges, such		request. It identifies the initial Child SA of a set, and allows the
	as when using EAP, the TSi/TSr payloads and the Notify payloads		peers to ensure that the initial Child SA (or its rekeyed version)
	defines in this document only appear in the first IKE_AUTH message.		remains active for the lifetime of the IPsec connection. Further
	In CREATE_CHILD_SA, the NUM_QUEUES Notify MUST only be sent in		CREATE_CHILD_SA messages for subsequent copies of the original Child
	messages for new set of Child SA's (the message used to set up the		SA MUST NOT contain the NUM_QUEUES notify payload. This initial
	Head SA)		Child SA (or its REKEYed successor) MUST remain active for the
			lifetime of the IPsec session to ensure there is always a CHILD SA
			that can be selected to send traffic over. Subsequent Child SA's can
			be installed with an additional selector, such as CPU or queue, or
			ToS value.
	The QUEUE_INFO Notify MUST only be sent in CREATE_CHILD_SA for Sub		The QUEUE_INFO Notify MUST be sent in CREATE_CHILD_SA for subsequent
	SA's. During CREATE_CHILD_SA's sent for Child SA rekey, the		copies of the original Child SA. It is used to indicate the queue or
	QUEUE_INFO MAY be included. If it is included it MUST be the same as		CPU or QoS value of this specific copy of the initial Child SA.
	for the Child SA being rekeyed.		These additional Child SA's can be started on-demand or all at once
			and can also be deleted if a peer deems this specific queue or CPU or
			QoS value to be idle. During CREATE_CHILD_SA's sent for Child SA
			rekey, the QUEUE_INFO Notify MUST NOT be included. As with Traffic
			Selector payloads, the QUEUE_INFO may not be different from the Child
			SA being rekeyed.
			This implies a CREATE_CHILD_SA exchange can only have either a
			QUEUE_INFO or NUM_QUEUES notify. If both Notify types are received,
			NUM_QUEUES has precedence and QUEUE_INFO MUST be ignored.
			The NUM_QUEUES notify, even though it can be sent in IKE_AUTH
			exchange with TS, is not an attribute of the IKE peer. It is an
			attribute of the Child SA, similar as how the USE_TRANSPORT notify
			payload. This allows an IKE peer to have multiple Child SA's
			covering different traffic selectors and selectively decide whether
			or not to use multiple Child SA's for those different Child SA's.
	4. Implementation specifics		4. Implementation specifics
	There are various considerations that an implementation could use to		There are various considerations that an implementation can use to
	determine the best way to install the multiple Child SAs. Below are		determine the best way to install the multiple Child SAs. Below are
	examples of such strategies.		examples of such strategies.
	4.1. One Child per CPU		4.1. One CPU per Child
	A simple distribution could be to install one Child SA per CPU. Note		A simple distribution could be to install one Child SA on each CPU.
	that at least one of the Child SAs must be the "fallback" in case		Note that at least one of the Child SAs must be the "fallback" in
	there is no specific Child SA on a specific CPU. This is called the		case there is no specific Child SA on a specific CPU. This role is
	Head SA, where the per-CPU Child SA is called a Sub CA. The initial		performed by the initial Child SA of the set of identical Child SAs.
	Child SA negotiated with IKE becomes the Head SA. This ensures that		This ensures that any CPU generating traffic to be encrypted has an
	any CPU generating traffic to be encrypted has an available (if not		available (if not optimal) Child SA to use. Any subsequent Child
	optimal) Child SA to use. Any subsequent Child SA's with identical		SA's with identical TSi/TSr are installed in such a way to only be
	TSi/TSr are considered Sub SA's and installed to be used only by a		used by a single CPU.
	single CPU.
	Implementations supporting per-CPU SAs SHOULD extend their mechanism		Implementations supporting per-CPU SAs SHOULD extend their mechanism
	of on-demand negotiation that is triggered by traffic to include a		of on-demand negotiation that is triggered by traffic to include a
	CPU (or queue) identifier in their ACQUIRE message from the SPD to		CPU (or queue) identifier in their ACQUIRE message from the SPD to
	the IKE daemon (eg via NETLINK of PFKEYv2). If the kernel's ACQUIRE		the IKE daemon (eg via NETLINK of PFKEYv2). If the ACQUIRE message
	message does not support sending a per-CPU identifier, then the IKE		does not support sending a per-CPU identifier, then the IKE daemon
	daemon should initiate all its Child SAs immediately upon receiving		may initiate all its Child SAs immediately upon receiving an ACQUIRE.
	an ACQUIRE.
	Performing per-CPU Child SA negotiations can result in both peers		Performing per-CPU Child SA negotiations can result in both peers
	initiating Sub SAs at once. This is especially likely in the per-CPU		initiating additional Child SAs at once. This is especially likely
	acquire case. Responders should install the Sub SA on the CPU with		in the per-CPU acquire case. Responders should install the
	the least amount of Sub SA's for this TSi/TSr pair. It should count		additional Child SA on a CPU with the least amount of additional
	outstanding ACQUIREs as an assigned Sub SA. It is still possible		Child SA's for this TSi/TSr pair. It should count outstanding
			ACQUIREs as an assigned additional Child SA. It is still possible
	that when the peers only have one slot left to assign, that both		that when the peers only have one slot left to assign, that both
	peers send an ACQUIRE at the same time. The initiator that receives		peers send an ACQUIRE at the same time. The initiator that receives
	the CREATE_CHID_SA response last, eg the initiator of the slowest		the CREATE_CHID_SA response last, eg the initiator of the slowest
	duplicate MAY send a delete to delete the duplicate Child SA.		duplicate Child SA, MAY send a delete to delete the duplicate
			additional Child SA.
	As an optimization, Sub SA's that see little traffic MAY be deleted.		As an optimization, additional Child SAs that see little traffic MAY
	However, it MUST NOT delete an idle Head SA. This ensures both peers		be deleted. The initial Child SA that is not limited to a single CPU
	always have a Child SA that can be used by a CPU that does not have a		MUST NOT be deleted when idle, as it is likely to be idle if enough
	Sub SA (yet) and ensures encrypted traffic can always be exchanged,		per-CPU Child SA's are installed. However, if one of those per-CPU
	even when that traffic triggered a new per-CPU ACQUIRE.		child SA's is deleted because it was idle, and subsequently that CPU
			starts the generate traffic again, that traffic should be encrypted
			by the initial non-CPU specific Child SA while the IKE daemon
			processes the ACQUIRE to bring up a new per-CPU Child SA.
	When the number of queues or CPUs are different between the peers,		When the number of queues or CPUs are different between the peers,
	the peer with the least amount of queues or CPUs MAY decide to not		the peer with the least amount of queues or CPUs MAY decide to not
	install a second outbound Child SA as it will never use it to send		install a second outbound Child SA as it will never use that Child SA
	traffic. However, it MUST install all inbound Child SA's as it		to send traffic. However, it MUST install all inbound Child SA's as
	cannot predict which of these the other peer will use to send		it has commited to receiving traffic on these negotiated Child SAs.
	traffic. It MUST NOT generate an error when deleting the (missing)		It MUST NOT generate an error when deleting the (missing) outbound SA
	outbound SA component of the Child SA.		component of such a Child SA.
	A per-CPU ACQUIRE message SHOULD still send the Traffic Selector		A per-CPU ACQUIRE message SHOULD still send the Traffic Selector
	(TSi) information of the trigger packet. This information MAY be		(TSi) entry containing the information of the trigger packet . This
	used by the responder to select the most efficient target CPU to use.		information MAY be used by the peer to select the most optimal target
	For example, if the trigger packet was for TCP destination port 25		CPU to install the additional Child SA on. For example, if the
	(SMTP), it might be able to install the Child SA on the CPU that is		trigger packet was for a TCP destination to port 25 (SMTP), it might
	also running the mail server process. See [RFC7296] Section 2.9.		be able to install the Child SA on the CPU that is also running the
			mail server process. Trigger packet Traffic Selectors are documented
			in [RFC7296] Section 2.9.
	The QUEUE_INFO Notify payload MAY be sent in the CREATE_CHILD_SA		The QUEUE_INFO Notify payload MUST be sent in the CREATE_CHILD_SA
	request for the additional (subSA) Child SAs. It can be used to		request for the additional Child SAs. It is used to convey the QoS
	convey the QoS stream or CPUID.		stream or CPU id. Note that this ID value does not neccessarilly
			have to match any physical CPU IDs.
	[Clarify narrowing Traffic Selectors. Should it be allowed/forbidden		[Clarify narrowing Traffic Selectors. Should it be allowed/forbidden
	?]		?]
	[Clarify CP / INTERNAL_ADDRESS. Should it be allowed/forbidden ?]		[Clarify CP / INTERNAL_ADDRESS. Should it be allowed/forbidden ?]
	[UDP enacap Due to the nature handling of UDP encapsulated ESP at the		[UDP enacap Due to the nature handling of UDP encapsulated ESP at the
	receiver NIC queus and intermediate routers for parallel paths, UDP		receiver NIC queus and intermediate routers for parallel paths, UDP
	encapsulated ESP will used multiple source ports. NOTE: this is		encapsulated ESP may use multiple source ports. We need define a way
	implemented in libreswan on Linux XFRM.]		to select UDP source ports for the Sub SA while IKE SA and the Head
			remain on UDP port 4500 - 4500. NOTE: libreswan has an expirmental
			implementation for Linux XFRM.]
			[Add text about how this parallel SA use may inter operate with 6311?
			may be not?]
	4.2. QoS Child SA's		4.2. QoS Child SA's
	To install multiple Child SA's for different QoS levels, a method		To install multiple Child SA's for different QoS levels, a method
	similar to per-CPU is used. The initial Child SA is used for all QoS		similar to per-CPU is used. The initial Child SA is used for all QoS
	levels not matched by more specific Child SA's. Additional Child		levels not matched by more specific Child SA's. Additional Child
	SA's are installed per QoS level, which can be done on-demand if the		SA's are installed per QoS level, which can be done on-demand if the
	kernel's IPsec subsystem can send per-QoS level ACQUIREs to the IKE		kernel's IPsec subsystem can send per-QoS level ACQUIREs to the IKE
	daemon.		daemon.
	A request for a Child SA for a specific QoS value MUST include the		A request for a Child SA for a specific QoS value MUST include the
	QUEUE_INFO Notify payload set to the required QoS value so that both		QUEUE_INFO Notify payload set to the required QoS value so that both
	endpoints use the same Child SA for the same QoS level. If a certain		endpoints use the same Child SA for the same QoS level. If a certain
	QoS level proposed is not acceptable to the resonder, TS_UNACCEPTABLE		QoS level proposed is not acceptable to the responder,
	MUST be returned. During Child SA REKEY, the QUEUE_INFO Notify MAY		TS_UNACCEPTABLE MUST be returned. During Child SA REKEY, the
	be included but MUST contain the same value as the Child SA that is		QUEUE_INFO Notify MUST NOT be included and MUST be ignored when
	being rekeyed. [ This kind of suggests this should be a TS_TYPE and		received.
	not a Notify ]
	5. Payload Format		5. Payload Format
	All multi-octet fields representing integers are laid out in big		All multi-octet fields representing integers are laid out in big
	endian order (also known as "most significant byte first", or		endian order (also known as "most significant byte first", or
	"network byte order").		"network byte order").
	5.1. NUM_QUEUES Notify Payload		5.1. NUM_QUEUES Notify Payload
	The NUM_QUEUES Notify payload is related to a Child SA, and MAY be
	exchanged in IKE_AUTH or in a CREATE_CHILD_SA for new SA. It MUST
	NOT be sent in CREATE_CHILD_SA for REKEY. If received for a REKEY
	operation, it MUST be ignored. See [RFC7296] Section 1.3.1.
	1 2 3		1 2 3
	0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1		0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
	+-+-----------------------------+-------------------------------+		+-+-----------------------------+-------------------------------+
	! Next Payload !C! RESERVED ! Payload Length !		! Next Payload !C! RESERVED ! Payload Length !
	+---------------+---------------+-------------------------------+		+---------------+---------------+-------------------------------+
	! Protocol ID ! SPI Size ! Notify Message Type !		! Protocol ID ! SPI Size ! Notify Message Type !
	+---------------+---------------+-------------------------------+		+---------------+---------------+-------------------------------+
	! Preferred number of IPsec SAs | Max accepted number of SAs !		! Minimum number of IPsec SAs !
	+-------------------------------+-------------------------------+		+-------------------------------+-------------------------------+
	o Protocol ID (1 octet) - MUST be 0. MUST be ignored if not 0.		o Protocol ID (1 octet) - MUST be 0. MUST be ignored if not 0.
	o SPI Size (1 octet) - MUST be 0. MUST be ignored if not 0. by the		o SPI Size (1 octet) - MUST be 0. MUST be ignored if not 0.
	IPsec protocol ID
	o Notify Message Type (2 octets) - set to [TBD]		o Notify Message Type (2 octets) - set to [TBD]
	o Preferred number of per-CPU IPsec SAs (2 octets). Value MUST be		o Minimum number of per-CPU IPsec SAs (4 octets). initiator value
	greater than 0. If 0 is received, it MUST be interpreted as 1.		Value MUST be greater than 0. If 0 is received, it MUST be
			interpreted as 1.
	o Maximum accepted number of per-CPU IPsec SAs (2 octets). Value
	MUST be greater than 0. If 0 is received, it MUST be interpreted
	as 1.
	Note: The first Child SA that is not bound to a single CPU (Head SA)		Note: The first Child SA that is not bound to a single CPU is not
	is not counted as part of these numbers.		counted as part of these numbers.
	5.2. QUEUE_INFO Notify Payload		5.2. QUEUE_INFO Notify Payload
	The QUEUE_INFO Notify payload is an optional related to a Child SA,
	and MAY be exchanged in IKE_AUTH or in a CREATE_CHILD_SA for new SA.
	It MUST NOT be sent in CREATE_CHILD_SA for REKEY, see [RFC7296]
	Section 1.3.1.
	1 2 3		1 2 3
	0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1		0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
	+-+-----------------------------+-------------------------------+		+-+-----------------------------+-------------------------------+
	! Next Payload !C! RESERVED ! Payload Length !		! Next Payload !C! RESERVED ! Payload Length !
	+---------------+---------------+-------------------------------+		+---------------+---------------+-------------------------------+
	! Protocol ID ! SPI Size ! Notify Message Type !		! Protocol ID ! SPI Size ! Notify Message Type !
	+---------------+---------------+-------------------------------+		+---------------+---------------+-------------------------------+
	! !		! !
	~ Optional payload data ~		~ Optional payload data ~
	! !		! !
	+-------------------------------+-------------------------------+		+-------------------------------+-------------------------------+
	o Protocol ID (1 octet) - MUST be 0. MUST be ignored if not 0.		o Protocol ID (1 octet) - MUST be 0. MUST be ignored if not 0.
	o SPI Size (1 octet) - MUST be 0. MUST be ignored if not 0. by the		o SPI Size (1 octet) - MUST be 0. MUST be ignored if not 0.
	IPsec protocol ID
	o Notify Message Type (2 octets) - set to [TBD]		o Notify Message Type (2 octets) - set to [TBD]
	o Optional Payload Data. This can be to identify the QoS options or		o Optional Payload Data. This can be set to identify the QoS value
	CPU-ID [Probable needs to be specified by this document]		or the CPU ID. The interpretation of the value is left to local
			implementations? [Probable needs to be specified by this
			document]
	6. Security Considerations		6. Security Considerations
	[TO DO]		[TO DO]
	7. Implementation Status		7. Implementation Status
	[Note to RFC Editor: Please remove this section and the reference to		[Note to RFC Editor: Please remove this section and the reference to
	[RFC6982] before publication.]		[RFC6982] before publication.]
	skipping to change at page 8, line 28 ¶		skipping to change at page 9, line 4 ¶
	Organization: Linux kernel XFRM		Organization: Linux kernel XFRM
	Name: XFRM-PCPU-v1		Name: XFRM-PCPU-v1
	https://git.kernel.org/pub/scm/linux/kernel/git/klassert/linux-		https://git.kernel.org/pub/scm/linux/kernel/git/klassert/linux-
	stk.git/log/?h=xfrm-pcpu-v1		stk.git/log/?h=xfrm-pcpu-v1
	Description: An initial Kernel IPsec implementation of the per-CPU		Description: An initial Kernel IPsec implementation of the per-CPU
	method.		method.
	Level of maturity: Alpha		Level of maturity: Alpha
			Coverage: Implements Initial Child SA and per-CPU additional Child
	Coverage: Fully implements Head SA and per-CPU Sub SA's		SA's. Also implements per-CPU ACQUIRES using NETLINK. PFKEYv2 is
			not supported.
	Licensing: GPLv2		Licensing: GPLv2
	Implementation experience: TBD		Implementation experience: TBD
	Contact: Linux IPsec: members@linux-ipsec.org		Contact: Linux IPsec: members@linux-ipsec.org
	7.2. Libreswan		7.2. Libreswan
	Organization: The Libreswan Project		Organization: The Libreswan Project
	Name: pcpu-3 https://libreswan.org/wiki/XFRM_pCPU		Name: pcpu-3 https://libreswan.org/wiki/XFRM_pCPU
	Description: An initial IKE implementation of the per-CPU method.		Description: An initial IKE implementation of the per-CPU method.
	Level of maturity: Alpha		Level of maturity: Alpha
	Coverage: implements Head SA and per-CPU Sub SA's.		Coverage: implements Initial Child SA and per-CPU additional Child
			SA's
	Licensing: GPLv2		Licensing: GPLv2
	Implementation experience: TBD		Implementation experience: TBD
	Contact: Libreswan Development: swan-dev@libreswan.org		Contact: Libreswan Development: swan-dev@libreswan.org
	7.3. strongSWAN		7.3. strongSWAN
	Organization: Secunet		Organization: Secunet
	Name: XXXX https://secunet.com/somethingU		Name: StrongSWAN https://github.com/antonyantony/strongswan/
	Description: An initial IKE implementation of the per-CPU method.		Description: An initial IKE implementation of the per-CPU method.
	Level of maturity: Alpha		Level of maturity: Alpha
	Coverage: implements Head SA and per-CPU Sub SA's.		Coverage: implements Initial Child SA and per-CPU additional Child
			SA's
	Licensing: GPLv2		Licensing: GPLv2
	Implementation experience: TBD		Implementation experience: the Linux XFRM implemenation needs an
			addtional flag on the SPD entry, XFRM_POLICY_CPU_ACQUIRE. It
			should be set only on the "outgoing" policy. The flag should be
			disabled when the policy is a trap policy without SPD state.
			After a successfull negotation of NUM_QUEUES, the SPD policy is
			updated to enable the XFRM_POLICY_CPU_ACQUIRE flag. For the
			outgoing additional Child SAs, the u32 XFRMA_SA_PCPU attribute is
			set, starting from 0. The incoming SA do not need XFRMA_SA_PCPU.
			The kernel internally set the value 0xFFFFFF. The strongswan
			implentation uses private space values for NUM_QUEUES (40970) and
			QUEUE_INFO (40971). The iproute2 software that supporte these two
			attributes is available at
			https://github.com/antonyantony/iproute2/tree/pcpu-v1
	Contact: Antony Antony: antony.antony@secunet.com.		Contact: Antony Antony: antony.antony@secunet.com.
	8. IANA Considerations		8. IANA Considerations
	This document defines one new IKEv2 Notify Message for the IANA		This document defines two new IKEv2 Notify messages for the IANA
	"IKEv2 Notify Message Types - Status Types" registry.		"IKEv2 Notify Message Types - Status Types" registry.
	Value Notify Messages - Status Types Reference		Value Notify Messages - Status Types Reference
	----- ------------------------------ ---------------		----- ------------------------------ ---------------
	[TBD] NUM_QUEUES [this document]		[TBD] NUM_QUEUES [this document]
	[TBD] QUEUE_INFO [this document]		[TBD] QUEUE_INFO [this document]
	Figure 1		Figure 1
	9. References		9. References
	skipping to change at page 10, line 11 ¶		skipping to change at page 10, line 49 ¶
	Kivinen, "Internet Key Exchange Protocol Version 2		Kivinen, "Internet Key Exchange Protocol Version 2
	(IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October		(IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October
	2014, <https://www.rfc-editor.org/info/rfc7296>.		2014, <https://www.rfc-editor.org/info/rfc7296>.
	[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC		[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
	2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,		2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
	May 2017, <https://www.rfc-editor.org/info/rfc8174>.		May 2017, <https://www.rfc-editor.org/info/rfc8174>.
	9.2. Informative References		9.2. Informative References
			[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
			Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
			December 2005, <https://www.rfc-editor.org/info/rfc4301>.
	[RFC6982] Sheffer, Y. and A. Farrel, "Improving Awareness of Running		[RFC6982] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
	Code: The Implementation Status Section", RFC 6982,		Code: The Implementation Status Section", RFC 6982,
	DOI 10.17487/RFC6982, July 2013,		DOI 10.17487/RFC6982, July 2013,
	<https://www.rfc-editor.org/info/rfc6982>.		<https://www.rfc-editor.org/info/rfc6982>.
	[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running		[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
	Code: The Implementation Status Section", BCP 205,		Code: The Implementation Status Section", BCP 205,
	RFC 7942, DOI 10.17487/RFC7942, July 2016,		RFC 7942, DOI 10.17487/RFC7942, July 2016,
	<https://www.rfc-editor.org/info/rfc7942>.		<https://www.rfc-editor.org/info/rfc7942>.
End of changes. 48 change blocks.
	135 lines changed or deleted		178 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/