idnits 2.17.1 draft-ietf-anima-grasp-distribution-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([I-D.ietf-anima-grasp], [RFC7575]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 12, 2021) is 1017 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC5424' is mentioned on line 746, but not defined == Missing Reference: 'RFC6241' is mentioned on line 746, but not defined == Unused Reference: 'I-D.du-anima-an-intent' is defined on line 981, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-anima-bootstrapping-keyinfra' is defined on line 992, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-anima-reference-model' is defined on line 1004, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2629 (Obsoleted by RFC 7749) Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group X. Xiao (Ed.) 3 Internet-Draft Huawei Technologies 4 Intended status: Standards Track B. Liu (Ed.) 5 Expires: January 13, 2022 A. Hecker 6 MRC, Huawei Technologies 7 S. Jiang 8 Huawei Technologies 9 B. Carpenter 10 School of Computer Science, University of 11 Auckland 12 July 12, 2021 14 Information Distribution over GRASP 15 draft-ietf-anima-grasp-distribution-03 17 Abstract 19 Autonomic network infrastructure (ANI) is a generic platform for 20 tenant applications (i.e. AFs). As we will see in some examplery 21 use cases, AFs may not only require communication capability 22 supported from the infrastructure side, but also the capability the 23 infrastructure can hold and re-distribute information on-demand. 24 This document proposes a set of solutions for information 25 distribution in the ANI. Information distribution is categorized 26 into two different modes: 1) instantaneous distribution and 2) 27 publishing for retrieval. In the former case, the information is 28 sent, propagated and disposed of after reception. In the latter 29 case, information needs to be stored in the network; additionally, 30 conflict resolution is also needed when information stored in the 31 network is updated with proposals from two different AFs. 33 The capability of information distribution is a fundamental need for 34 an autonomous network ([RFC7575]). This document describes typical 35 use cases of information distribution in ANI and requirements to ANI, 36 such that abundant ways of information distribution can be natively 37 supported. This draft proposes a series of extensions to the 38 autonomic nodes and suggests an implementation based on GRASP 39 ([I-D.ietf-anima-grasp]) extensions as a protocol on the wire. 41 Status of This Memo 43 This Internet-Draft is submitted in full conformance with the 44 provisions of BCP 78 and BCP 79. 46 Internet-Drafts are working documents of the Internet Engineering 47 Task Force (IETF). Note that other groups may also distribute 48 working documents as Internet-Drafts. The list of current Internet- 49 Drafts is at https://datatracker.ietf.org/drafts/current/. 51 Internet-Drafts are draft documents valid for a maximum of six months 52 and may be updated, replaced, or obsoleted by other documents at any 53 time. It is inappropriate to use Internet-Drafts as reference 54 material or to cite them other than as "work in progress." 56 This Internet-Draft will expire on January 13, 2022. 58 Copyright Notice 60 Copyright (c) 2021 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents 65 (https://trustee.ietf.org/license-info) in effect on the date of 66 publication of this document. Please review these documents 67 carefully, as they describe your rights and restrictions with respect 68 to this document. Code Components extracted from this document must 69 include Simplified BSD License text as described in Section 4.e of 70 the Trust Legal Provisions and are provided without warranty as 71 described in the Simplified BSD License. 73 Table of Contents 75 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 76 2. Use Cases of Information Distribution . . . . . . . . . . . . 4 77 2.1. Vehicle-to-Everything (V2X) Communications . . . . . . . 4 78 2.2. Service-Based Architecture (SBA) in 3GPP . . . . . . . . 5 79 2.3. In-Network Computing (INC) . . . . . . . . . . . . . . . 7 80 3. General Requirements of Information Distribution in ANI . . . 8 81 4. Node Behaviors . . . . . . . . . . . . . . . . . . . . . . . 10 82 4.1. Instant Information Distribution (IID) Sub-module . . . . 10 83 4.1.1. Instant P2P Communication . . . . . . . . . . . . . . 11 84 4.1.2. Instant Flooding Communication . . . . . . . . . . . 11 85 4.2. Asynchronous Information Distribution (AID) Sub-module . 12 86 4.2.1. Information Storage . . . . . . . . . . . . . . . . . 12 87 4.2.2. Event Queue . . . . . . . . . . . . . . . . . . . . . 14 88 4.3. Bulk Information Transfer . . . . . . . . . . . . . . . . 16 89 4.4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 17 90 5. Extending GRASP for Information Distribution . . . . . . . . 18 91 5.1. Realizing Instant P2P Transmission . . . . . . . . . . . 18 92 5.2. Realizing Instant Selective Flooding . . . . . . . . . . 18 93 5.3. Realizing Bulk Information Transfer . . . . . . . . . . . 19 94 5.4. Realizing Subscription as An Event . . . . . . . . . . . 19 95 5.5. Un_Subscription Objective Option . . . . . . . . . . . . 19 96 5.6. Publishing Objective Option . . . . . . . . . . . . . . . 20 97 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 98 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 99 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 100 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 101 9.1. Normative References . . . . . . . . . . . . . . . . . . 21 102 9.2. Informative References . . . . . . . . . . . . . . . . . 21 103 Appendix A. Open Issues [RFC Editor: To Be removed before 104 becoming RFC] . . . . . . . . . . . . . . . . . . . 22 105 Appendix B. Closed Issues [RFC Editor: To Be removed before 106 becoming RFC] . . . . . . . . . . . . . . . . . . . 23 107 Appendix C. Change log [RFC Editor: To Be removed before 108 becoming RFC] . . . . . . . . . . . . . . . . . . . 23 109 Appendix D. Information Distribution Module in ANI . . . . . . . 23 110 Appendix E. Asynchronous ID Integrated with GRASP APIs . . . . . 24 111 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 113 1. Introduction 115 In an autonomic network, autonomic functions (AFs) running on 116 autonomic nodes constantly exchange information, e.g. AF control/ 117 management signaling or AF data exchange. This document discusses 118 the information distribution capability of such exchanges among AFs. 119 Many use cases can be abstracted to this model. In the following 120 sections, we will see that the information distribution capability 121 shall become a common denominator in future application scenarios. 123 In general, depending on the number of participants, the information 124 can be distributed in in the following scenarios: 126 1) Point-to-point (P2P) Communication: information is exchanged 127 between two AFs. 129 2) One-to-Many Communication: information exchanges involve one 130 source AF and multiple receiving AFs. 132 Approaches of infrmation distribution can be mainly categorized into 133 two basic modes: 135 1) An instantaneous mode (push): a source sends the actual content 136 (e.g. control/management signaling, synchronization data and so 137 on) to all interested receiver(s) immediately. Generally, some 138 preconfigurations are required, where nodes interested in this 139 information must be already known to all nodes because any source 140 AF must be able to decide, to which AFs the data is to be sent. 142 2) An asynchronous mode (delayed pull): here, a source AF publishes 143 the content in some forms in the network, which may later be 144 looked for, found and retrieved by some endpoints in the AN. 145 Here, depending on the size of the content, either the whole 146 content or only its metadata might be published into the AN. In 147 the latter case the metadata (e.g. a content descriptor, e.g. a 148 key, and a location in the ANI) may be used for the actual 149 retrieval. Importantly, the source, i.e., here as a publisher, 150 needs to be able to determine the location, where the information 151 (or its metadata) can be stored. 153 Note that in both cases, the total size of transferred information 154 can be larger than the payload size of a single message of a used 155 transport protocol (e.g., Synchronization and Flood messages in 156 GRASP). In this situation, this document also considers a case of 157 bulk data transfer. To avoid repetitive implementations by each AF 158 developer, this document opts for a common support for information 159 distribution implemented as a basic ANI capability. Therefore, it 160 will be available to all AFs. In fact, GRASP already provides part 161 of the capabilities. 163 Regardless, an AF may still define and implement its own information 164 distribution capability. Such a capability may then be advertised 165 using the common information distribution capability defined in this 166 document. Overall, ANI nodes and AFs may decide, which of the 167 information distribution mechanisms they want to use for which type 168 of information, according to their own preferences. 170 This document first analyzes requirements for information 171 distribution in autonomic networks (Section 3) and then discuss the 172 relevant node behaviors (Section 4). After that, the required GRASP 173 extensions are formally introduced (Section 5). 175 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 176 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 177 document are to be interpreted as described in RFC 2119 [RFC2119]. 179 2. Use Cases of Information Distribution 181 In this section, we present some important use cases where 182 information distribution is required and ACP's support is commanly 183 needed. 185 2.1. Vehicle-to-Everything (V2X) Communications 187 The connected Autonomous Driving (AD) vehicles market is driving the 188 evolution of the Internet of Vehicles (IoV) (or Vehicular IoT) and is 189 growing at a five-year compound annual growth rate of 45%, which is 190 10 times as fast as the overall car market. V2X communication is an 191 inevitable enabling technology that connects vehicles to networks, 192 where value-added services can be provided and enhance the 193 functionalities of a vehicle. In this section, we introduce some use 194 cases that will be closely relevant to information distribution in an 195 ANI. 197 1) Real-time and High Definition Maps (HDM): In the era of autonomous 198 driving, a digital map not only means for navigation, but real- 199 time and detailed information is required when driving a vehicle. 200 Real-time situational awareness is essential for autonomous 201 vehicles especially at critical road segments in cases of changing 202 road conditions (e.g. new traffic cone detected by another vehicle 203 some time ago). In addition, the relevant high definition local 204 maps have to be available with support from infrastructure side. 205 In this regards, a digital map should not be considered as static 206 information stored on the vehicle, which is spontaneously updated 207 in a periodical manner. Instead, it shall be considered as a 208 dynamic distribution based on information aggregated from the 209 local area and such a distribution shall consider latency 210 requirement. Clearly, the infrastructure side shall be able to 211 hold the information in the network sufficiently close to the 212 relevant area. 214 2) In-car Infotaiment: This is another popular use case where in-car 215 data demands will increase significantly in the near future. 216 Today, users their mobile phone to access Internet for retrieving 217 data for work or entertainment purposes. There is already a 218 concensus among OTTs, carriers and car manufacturers that vehicle 219 will become the center of information for passengers onboard. For 220 entertainment, typical scenarios can be stereo HD video streaming 221 and online gaming; for business purposes, examples can be mobile 222 conference. This therefore requires the infrastructure side to be 223 able to schedule and deliver requested information/data to the 224 users with quality-of-service (QoS) considerations. 226 3) Software Update: Software components of connected cars will be 227 remotely maintained in future. Therefore, software update has to 228 be supported by the infrastructure side. Although this can be 229 done by centralized solution where all vehicles access to a 230 central clouds, in terms of load balancing and efficiency, 231 prepared update components can be stored in the network and 232 delivered to endpoints in a distributed manner. 234 2.2. Service-Based Architecture (SBA) in 3GPP 236 In addition to Internet, carrier networks (i.e. wireless mobile 237 networks) is another world-wide networking system. The current 238 architecture of 5G mobile networks from 3GPP has been defined to 239 follow a service-based architecture (SBA) where any network function 240 (NF) can be dynamically associated with any other NF(s) when needed 241 to compose a network service. Note that one NF can simultaneously 242 associate with multiple other NFs, instead of being physically wired 243 as in the previous generations of mobile networks. NFs communicate 244 with each other over service-based interface (SBI), which is also 245 standardized by 3GPP [3GPP.23.501]. 247 To realize an SBA network system, detailed requirements are further 248 defined to specify how NFs should interact with each other with 249 information exchange over the SBI. We now list three requirements 250 that are related to information distribution here. 252 1) NF Pub/Sub: Any NF should be able to expose its service status to 253 the network and any NF should be able to subscribe the service 254 status of an NF and get notified if the status is available. A 255 concrete example is that a session management function (SMF) can 256 subscribe to the REGISTER notification from an access management 257 function (AMF) if there is a new user equipment trying to access 258 the mobile network [3GPP.23.502]. 260 2) Network Exposure Function (NEF): A particular network function 261 that is required to manage the event exposure and distributions. 262 Specifically, SBA requires such a functionality to register 263 network events from the other NFs (e.g. AMF, SMF and so on), 264 classify the events and properly handle event distributions 265 accordingly in terms of different criteria (e.g. priorities) 266 [3GPP.23.502]. 268 3) Network Repository Function (NRF): A particular network function 269 where all service status information is stored for the whole 270 network. An SBA network system requires all NFs to be stateless 271 so as to improve the resilience as well as agility of providing 272 network services. Therefore, the information of the available NFs 273 and the service status generated by those NFs will be globally 274 stored in NRF as a repository of the system. This clearly implies 275 storage capability that keeps the information in the network and 276 provides those information when needed. A concrete example is 277 that whenever a new NF comes up, it first of all registers itself 278 at NRF with its profile. When a network service requires a 279 certain NF, it first inquires NRF to retrieve the availability 280 information and decides whether or not there is an available NF or 281 a new NF must be instantiated [3GPP.23.502]. 283 (Note: 3GPP CT adopted HTTP2.0/JSON to be the protocol communicating 284 between NFs, but autonomic networks can also load HTTP2.0 with in 285 ACP.) 286 Note that the requirements of the information distribution for ANI 287 control plane are not mentioned here, rather only AF services that 288 have to be necessarily supported by additional information 289 distribution are discussed. 291 2.3. In-Network Computing (INC) 293 In-network computing recently gets a lot of attentions. INC improves 294 the utilization of the computing resources in the network; INC also 295 brings the processed results closer to the users, which may 296 potentially improves the QoS of network services. 298 Unlike existing network systems, INC deploys computing tasks directly 299 in the network rather than pushing the tasks to endpoints outside the 300 network. Therefore, a network device is not just a transport device, 301 but a mixture of forwarding, routing and computing. The requires an 302 INC-supported network device having storage by default. Furthermore, 303 computing agents deployed on network nodes will have to communicate 304 with each other by exchanging information. There are several typical 305 applications, where information distribution capability is required, 306 which are summarized below. 308 1) Data Backup: There can be multiple computing agents that are 309 created to serve the same purpose(s). In reality, the multiple 310 agents can run for service resilience, load balancing and so on. 311 This forms a service set. The instances in the service set can be 312 deployed at different locations in the network while they need to 313 keep synchronizing their local states for global consistency. In 314 this case, the computing agents will have to constantly send and 315 receive information across the network. 317 2) Data Aggregation: Multiple computing agents may process different 318 computing tasks but the derived results have to be aggregated or 319 combined. Then a collective result can be derived. In this case, 320 different computing agents collaborate with each other, where 321 information data are exchanged during the processing. A popular 322 example is distributed AI or federated learning applications, 323 where data are stored at different places and model training with 324 the local data is also done in a distributed way. After that, 325 trained models by distributed agents will have to be aggregated. 326 Information distribution will be utlized heavily, combining with 327 local storage. 329 Clearly, AFs running on network nodes in ANI are the abstraction of 330 the INC use case. AFs can be deployed for both scenarios above. 332 3. General Requirements of Information Distribution in ANI 334 According to the introduced use cases, the question of information 335 distribution in an autonomic network can be discussed through 336 particular use cases or more generally. Depending on the situation 337 it can be quite simple or might require more complex provisions. 339 Indeed, in the most general case, the information can be sent: 341 1) at once (in one or multiple packets, in one flow), 343 2) straightaway (send-and-forget), 345 3) to all nodes. 347 For the first scenario, presuming 1), 2) and 3) hold, information 348 distribution in smaller or scarce topologies can be implemented using 349 broadcast, i.e. unconstrained flooding. For reasons well-understood, 350 this approach has its limits in larger and denser networks. In this 351 case, a graph can be constructed such that it contains every node 352 exactly once (e.g. a spanning tree), still allowing to distribute any 353 information to all nodes straightaway. Multicast tree construction 354 protocols could be used in this case. There are reasonable use cases 355 for such scenarios, as presented in Section 2. 357 Secondly, a more complex scenario arises, if only 1) and 2) hold, but 358 the information only concerns a subset of nodes. Then, some kinds of 359 selection become required, to which nodes the given information 360 should be distributed. Here, a further distinction is necessary; 361 notably, if the selection of the target nodes is with respect to the 362 nature or position of the node, or whether it is with respect to the 363 information content. If the first, some knowledge about the node 364 types, its topological position, etc (e.g. the routing information 365 within ANI) can be used to distinguish nodes accordingly. For 366 instance, edge nodes and forwarding nodes can be distinguished in 367 this way. If the distribution scope is primarily to be defined by 368 the information elements, then a registration / join / subscription 369 or label distribution mechanism is unavoidable. This would be the 370 case, for instance, if the AFs can be dynamically deployed on nodes, 371 and the information is majorily destined to the AFs. Then, depending 372 on the current AF deployment, the distribution scope must be adjusted 373 as well. 375 Thirdly, if only 1) holds, but the information content might be 376 required again and again, or might not yet be fully available, then 377 more complex mechanisms might be required to store the information 378 within the network for later, for further redistribution, and for 379 notification of interested nodes. Examples for this include 380 distribution of reconfiguration information for different AF 381 instances, which might not require an immediate action, but only an 382 eventual update of the parameters. Also, in some situations, there 383 could be a significant delay between the occurrence of a new event 384 and the full content availability (e.g. if the processing requires a 385 lot of time). 387 Finally, none of the three might hold. Then, along with the 388 subscription and notification, the actual content might be different 389 from its metadata, i.e. some descriptions of the content and, 390 possibly, its location. The fetching can then be implemented in 391 different, appropriate ways, if necessary as a complex transport 392 session. 394 In essence, as flooding is usually not an option, and the interest of 395 nodes for particular information elements can change over time, ANI 396 should support autonomics also for the information distribution. 398 This calls for autonomic mechanisms in the ANI, allowing 399 participating nodes to 1) advertise/publish, 2) look for/subscribe to 400 3) store, 4) fetch/retrieve and 5) instantaneously push data 401 information. 403 In the following cases, situations depicting complicated ways of 404 information distribution are discussed. 406 1) Long Communication Intervals. The actual sending of the 407 information is not necessarily instantaneous with some events. 408 Sophisticated AFs may involve into longer jobs/tasks (e.g. 409 database lookup, validations, etc.) when processing requests, and 410 might not be able to reply immediately. Instead of actively 411 waiting for the reply, a better way for an interested AF might be 412 to get notified, when the reply is finally available. 414 2) Common Interest Distribution. AFs may share information that is a 415 common interest. For example, the network intent will be 416 distributed to network nodes enrolled, which is usually one-to- 417 many scenario. Intent distribution can also be performed by an 418 instant flooding (e.g. via GRASP) to every network node. However, 419 because of network changes, not every node can be just ready at 420 the moment when the network intent is broadcast. Also, a flooding 421 often does not cover all network nodes as there is usually a 422 limitation on the hop number. In fact, nodes may join in the 423 network sequentially. In this situation, an asynchronous 424 communication model could be a better choice where every (newly 425 joining) node can subscribe the intent information and will get 426 notified if it is ready (or updated). 428 3) Distributed Coordination. With computing and storage resources on 429 autonomic nodes, alive AFs not only consume but also generate data 430 information. An example is AFs coordinating with each other as 431 distributed schedulers, responding to service requests and 432 distributing tasks. It is critical for those AFs to make correct 433 decisions based on local information, which might be asymmetric as 434 well. AFs may also need synthetic/aggregated data information 435 (e.g. statistic info, like average values of several AFs, etc.) 436 to make decisions. In these situations, AFs will need an 437 efficient way to form a global view of the network (e.g. about 438 resource consumption, bandwidth and statistics). Obviously, 439 purely relying on instant communication model is inefficient, 440 while a scalable, common, yet distributed data layer, on which AFs 441 can store and share information in an asynchronous way, should be 442 a better choice. 444 4) Collision Update. Information data not only can be propagated and 445 stored on network nodes in the network, they have to be conflict- 446 free when information is updated especially when there is no 447 central authority available. For example, when two AFs try to 448 propose different updates for the same piece of information that 449 already exist in the network, a decision has to be made for how 450 the existing information shall be updated. Obviously, if this 451 duty has to be handled by individual AFs, the implematation of an 452 AF is too complicated. Therefore, information distribution should 453 consider conflict resultion and provides a set of general 454 solutions for AFs in order to keep information conflict free. 456 Therefore, for ANI, in order to support various communication 457 scenarios, an information distribution module is required, and both 458 instantaneous and asynchronous communication models should be 459 supported. Some real-world use cases are introduced in Section 2. 461 4. Node Behaviors 463 In this section, how a node should behave in order to support the two 464 identified modes of information distribution is discussed. An ANI is 465 a distributed system, so the information distribution module must be 466 implemented in a distributed way as well. 468 4.1. Instant Information Distribution (IID) Sub-module 470 In this case, an information sender directly specifies the 471 information receiver(s). The instant information distribution sub- 472 module will be the main element. 474 4.1.1. Instant P2P Communication 476 IID sub-module performs instant information transmission for ASAs 477 running in an ANI. In specific, IID sub-module will have to retrieve 478 the address of the information receiver specified by an ASA, then 479 deliver the information to the receiver. Such a delivery can be done 480 either in a connectionless or a connection-oriented way. 482 Current GRASP provides the capability to support instant P2P 483 synchronization for ASAs. A P2P synchronization is a use case of P2P 484 information transmission. However, as mentioned in Section 3, there 485 are some scenarios where one node needs to transmit some information 486 to another node(s). This is different to synchronization because 487 after transmitting the information, the local status of the 488 information does not have to be the same as the information sent to 489 the receiver. This is not directly support by existing GRASP. 491 4.1.2. Instant Flooding Communication 493 IID sub-module finishes instant flooding for ASAs in an ANI. Instant 494 flooding is for all ASAs in an ANI. An information sender has to 495 specify a special destination address of the information and 496 broadcast to all interfaces to its neighbors. When another IID sub- 497 module receives such a broadcast, after checking its TTL, it further 498 broadcast the message to the neighbors. In order to avoid flooding 499 storms in an ANI, usually a TTL number is specified, so that after a 500 pre-defined limit, the flooding message will not be further broadcast 501 again. 503 In order to avoid unnecessary flooding, a selective flooding can be 504 done where an information sender wants to send information to 505 multiple receivers at once. When doing this, sending information 506 needs to contain criteria to judge on which interfaces the 507 distributed information should and should not be sent. Specifically, 508 the criteria contain: 510 o Matching Condition: a set of matching rules such as addresses of 511 recipients, node features and so on. 513 o Action: what the node needs to do when the Matching Condition is 514 fulfilled. For example, the action could be forwarding or 515 discarding the distributed message. 517 Sent information must be included in the message distributed from the 518 sender. The receiving node reacts by first checking the carried 519 Matching Condition in the message to decide who should consume the 520 message, which could be either the node itself, some neighbors or 521 both. If the node itself is a recipient, Action field is followed; 522 if a neighbor is a recipient, the message is sent accordingly. 524 An exemplary extension to support selective flooding on GRASP is 525 described in Section 5. 527 4.2. Asynchronous Information Distribution (AID) Sub-module 529 In asynchronous information distribution, sender(s) and receiver(s) 530 are not immediately specified while they may appear in an 531 asynchronous way. Firstly, AID sub-module enables that the 532 information can be stored in the network; secondly, AID sub-module 533 provides an information publication and subscription (Pub/Sub) 534 mechanism for ASAs. 536 As sketched in the previous section, in general each node requires 537 two modules: 1) Information Storage (IS) module and 2) Event Queue 538 (EQ) module in the information distribution module. Details of the 539 two modules are described in the following sections. 541 4.2.1. Information Storage 543 IS module handles how to save and retrieve information for ASAs 544 across the network. The IS module uses a syntax to index 545 information, generating the hash index value (e.g. a hash value) of 546 the information and mapping the hash index to a certain node in ANI. 547 Note that, this mechanism can use existing solutions. Specifically, 548 storing information in an ANIMA network will be realized in the 549 following steps. 551 1) ASA-to-IS Negotiation. An ASA calls the API provided by 552 information distribution module (directly supported by IS sub- 553 module) to request to store the information somewhere in the 554 network. The IS module performs various checks of the request 555 (e.g. permitted information size). 557 2) Storing Peer Mapping. The information block will be handled by 558 the IS module in order to calculate/map to a peer node in the 559 network. Since ANIMA network is a peer-to-peer network, a typical 560 way is to use distributed hash table (DHT) to map information to a 561 unique index identifier. For example, if the size of the 562 information is reasonable, the information block itself can be 563 hashed, otherwise, some meta-data of the information block can be 564 used to generate the mapping. 566 3) Storing Peer Negotiation Request. Negotiation request of storing 567 the information will be sent from the IS module to the IS module 568 on the destination node. The negotiation request contains 569 parameters about the information block from the source IS module. 570 According to the parameters as well as the local available 571 resource, the requested storing peer will send feedback the source 572 IS module. 574 4) Storing Peer Negotiation Response. Negotiation response from the 575 storing peer is sent back to the source IS module. If the source 576 IS module gets confirmation that the information can be stored, 577 source IS module will prepare to transfer the information block; 578 otherwise, a new storing peer must be discovered (i.e. going to 579 step 7). 581 5) Information Block Transfer. Before sending the information block 582 to the storing peer that already accepts the request, the IS 583 module of the source node will check if the information block can 584 be afforded by one GRASP message. If so, the information block 585 will be directly sent by calling a GRASP API 586 ([I-D.ietf-anima-grasp-api]). Otherwise, a bulk data transmission 587 is needed. For that, there are multiple ways to do it. The first 588 option is to utilize one of existing protocols that is independent 589 of the GRASP stack. For example, a session connectivity can be 590 established to the storing peer, and over the connection the bulky 591 data can be transmitted part by part. In this case, the IS module 592 should support basic TCP-based session protocols such as HTTP(s) 593 or native TCP. The second option is to directly use GRASP itself 594 for bulky data transferring [I-D.carpenter-anima-grasp-bulk]. 596 6) Information Writing. Once the information block (or a smaller 597 block) is received, the IS module of the storing peer will store 598 the data block in the local storage is accessible. 600 7) (Optional) New Storing Peer Discovery. If the previously selected 601 storing peer is not available to store the information block, the 602 source IS module will have to identify a new destination node to 603 start a new negotiation. In this case, the discovery can be done 604 by using discovery GRASP API to identify a new candidate, or more 605 complex mechanisms can be introduced. 607 Similarly, Getting information from an ANI will be realized in the 608 following steps. 610 1) ASA-to-IS Request. An ASA accesses the IS module via the APIs 611 exposed by the information distribution module. The key/index of 612 the interested information will be sent to the IS module. An 613 assumption here is that the key/index should be known to an ASA 614 before an ASA can ask for the information. This relates to the 615 publishing/subscribing of the information, which are handled by 616 other modules (e.g. Event Queue with Pub/Sub supported by GRASP). 618 2) Storing Peer Mapping. IS module maps the key/index of the 619 requested information to a peer that stores the information, and 620 prepares the information request. The mapping here follows the 621 same mechanism when the information is stored. 623 3) Retrieval Negotiation Request. The source IS module sends a 624 request to the storing peer and asks if such an information object 625 is available. 627 4) Retrieval Negotiation Response. The storing peer checks the key/ 628 index of the information in the request, and replies to the source 629 IS module. If the information is found and the information block 630 can be afforded within one GRASP message, the information will be 631 sent together with the response to the source IS module. 633 5) (Optional) New Destination Request. If the information is not 634 found after the source IS module gets the response from the 635 originally identified storing peer, the source IS module will have 636 to discover the location of the requested information. 638 IS module can reuse distributed databases and key value stores like 639 NoSQL, Cassandra, DHT technologies. storage and retrieval of 640 information are all event-driven responsible by the EQ module. 642 4.2.2. Event Queue 644 The Event Queue (EQ) module is to help ASAs to publish information to 645 the network and subscribe to interested information in asynchronous 646 scenarios. In an ANI, information generated on network nodes is an 647 event labeled with an event ID, which is semantically related to the 648 topic of the information. Key features of EQ module are summarized 649 as follows. 651 1) Event Group: An EQ module provides isolated queues for different 652 event groups. If two groups of AFs could have completely 653 different purposes, the EQ module allows to create multiple queues 654 where only AFs interested in the same topic will be aware of the 655 corresponding event queue. 657 2) Event Prioritization: Events can have different priorities in ANI. 658 This corresponds to how much important or urgent the event 659 implies. Some of them are more urgent than regular ones. 660 Prioritization allows AFs to differentiate events (i.e. 661 information) they publish or subscribe to. 663 3) Event Matching: an information consumer has to be identified from 664 the queue in order to deliver the information from the provider. 665 Event matching keeps looking for the subscriptions in the queue to 666 see if there is an exact published event there. Whenever a match 667 is found, it will notify the upper layer to inform the 668 corresponding ASAs who are the information provider and 669 subscriber(s) respectively. 671 The EQ module on every network node operates as follows. 673 1) Event ID Generation: If information of an ASA is ready, an event 674 ID is generated according to the content of the information. This 675 is also related to how the information is stored/saved by the IS 676 module introduced before. Meanwhile, the type of the event is 677 also specified where it can be of control purpose or user plane 678 data. 680 2) Priority Specification: According to the type of the event, the 681 ASA may specify its priority to say how this event is to be 682 processed. By considering both aspects, the priority of the event 683 will be determined. 685 3) Event Enqueue: Given the event ID, event group and its priority, a 686 queue is identified locally if all criteria can be satisfied. If 687 there is such a queue, the event will be simply added into the 688 queue, otherwise a new queue will be created to accommodate such 689 an event. 691 4) Event Propagation: The published event will be propagated to the 692 other network nodes in the ANIMA domain. A propagation algorithm 693 can be employed to optimize the propagation efficiency of the 694 updated event queue states. 696 5) Event Match and Notification: While propagating updated event 697 states, EQ module in parallel keeps matching published events and 698 its interested consumers. Once a match is found, the provider and 699 subscriber(s) will be notified for final information retrieval. 701 The category of event priority is defined as the following. In 702 general, there are two event types: 704 1) Network Control Event: This type of events are defined by the ANI 705 for operational purposes on network control. A pre-defined 706 priority levels for required system messages is suggested. For 707 highest level to lowest level, the priority value ranges from 708 NC_PRIOR_HIGH to NC_PRIOR_LOW as integer values. The NC_PRIOR_* 709 values will be defined later according to the total number system 710 events required by the ANI. 712 2) Custom ASA Event: This type of events are defined by the ASAs of 713 users. This specifies the priority of the message within a group 714 of ASAs, therefore it is only effective among ASAs that join the 715 same message group. Within the message group, a group header/ 716 leader has to define a list of priority levels ranging from 717 CUST_PRIOR_HIGH to CUST_PRIOR_LOW. Such a definition completely 718 depends on the individual purposes of the message group. When a 719 system message is delivered, its event type and event priority 720 value have to be both specified. 722 Event contains the address where the information is stored, after a 723 subscriber is notified, it directly retrieves the information from 724 the given location. 726 4.3. Bulk Information Transfer 728 In both cases discussed previously, they are limited to distributing 729 GRASP Objective Options contained in messages that cannot exceed the 730 GRASP maximum message size of 2048 bytes. This places a limit on the 731 size of data that can be transferred directly in a GRASP message such 732 as a Synchronization or Flood operation for instantaneous information 733 distribution. 735 There are scenarios in autonomic networks where this restriction is a 736 problem. One example is the distribution of network policy in 737 lengthy formats such as YANG or JSON. Another case might be an 738 Autonomic Service Agent (ASA) uploading a log file to the Network 739 Operations Center (NOC). A third case might be a supervisory system 740 downloading a software upgrade to an autonomic node. A related case 741 might be installing the code of a new or updated ASA to a target 742 node. 744 Naturally, an existing solution such as a secure file transfer 745 protocol or secure HTTP might be used for this. Other management 746 protocols such as syslog [RFC5424] or NETCONF [RFC6241] might also be 747 used for related purposes, or might be mapped directly over GRASP. 748 The present document, however, applies to any scenario where it is 749 preferable to re-use the autonomic networking infrastructure itself 750 to transfer a significant amount of data, rather than install and 751 configure an additional mechanism. 753 The node behavior is to use the GRASP Negotiation process to transfer 754 and acknowledge multiple blocks of data in successive negotiation 755 steps, thereby overcoming the GRASP message size limitation. The 756 emphasis is placed on simplicity rather than efficiency, high 757 throughput, or advanced functionality. For example, if a transfer 758 gets out of step or data packets are lost, the strategy is to abort 759 the transfer and try again. In an enterprise network with low bit 760 error rates, and with GRASP running over TCP, this is not considered 761 a serious issue. Clearly, a more sophisticated approach could be 762 designed but if the application requires that, existing protocols 763 could be used, as indicated in the preceding paragraph. 765 As for any GRASP operation, the two participants are considered to be 766 Autonomic Service Agents (ASAs) and they communicate using a specific 767 GRASP Objective Option, containing its own name, some flag bits, a 768 loop count, and a value. In bulk transfer, we can model the ASA 769 acting as the source of the transfer as a download server, and the 770 destination as a download client. No changes or extensions are 771 required to GRASP itself, but compared to a normal GRASP negotiation, 772 the communication pattern is slightly asymmetric: 774 1) The client first discovers the server by the GRASP discovery 775 mechanism (M_DISCOVERY and M_RESPONSE messages). 777 2) The client then sends a GRASP negotiation request (M_REQ_NEG 778 message). The value of the objective expresses the requested item 779 (e.g., a file name - see the next section for a detailed example). 781 3) The server replies with a negotiation step (M_NEGOTIATE message). 782 The value of the objective is the first section of the requested 783 item (e.g., the first block of the requested file as a raw byte 784 string). 786 4) The client replies with a negotiation step (M_NEGOTIATE message). 787 The value of the objective is a simple acknowledgement (e.g., the 788 text string 'ACK'). 790 The last two steps repeat until the transfer is complete. The server 791 signals the end by transferring an empty byte string as the final 792 value. In this case the client responds with a normal end to the 793 negotiation (M_END message with an O_ACCEPT option). 795 Errors of any kind are handled with the normal GRASP mechanisms, in 796 particular by an M_END message with an O_DECLINE option in either 797 direction. In this case the GRASP session terminates. It is then 798 the client's choice whether to retry the operation from the start, as 799 a new GRASP session, or to abandon the transfer. The block size must 800 be chosen such that each step does not exceed the GRASP message size 801 limit of 2048 bits. 803 4.4. Summary 805 In summary, the general requirements for the information distribution 806 module on each autonomic node are realized by two sub-modules 807 handling instant communications and asynchronous communications, 808 respectively. For instantaneous mode, node requirements are simple, 809 calling for support for additional signaling. With minimum efforts, 810 reusing the existing GRASP is possible. 812 For asynchronous mode, information distribution module uses new 813 primitives on the wire, and implements an event queue and an 814 information storage mechanism. An architectural consideration on ANI 815 with the information distribution module is briefly discussed in 816 Appendix D. 818 In both cases, a scenario of bulk information transfer is considered 819 where the retrieved information cannot be fitted in one GRASP 820 message. Based on GRASP Negotiation operation, multiple 821 transmissions can be repeatedly done in order to transfer bulk 822 informtion piece by piece. 824 5. Extending GRASP for Information Distribution 826 5.1. Realizing Instant P2P Transmission 828 This could be a new message in GRASP. In fragmentary CDDL, an Un- 829 solicited Synchronization message follows the pattern: 831 unsolicited_synch-message = [M_UNSOLIDSYNCH, session-id, 832 objective] 834 A node MAY actively send a unicast Un-solicited Synchronization 835 message with the Synchronization data, to another node. This MAY be 836 sent to port GRASP_LISTEN_PORT at the destination address, which 837 might be obtained by GRASP Discovery or other possible ways. The 838 synchronization data are in the form of GRASP Option(s) for specific 839 synchronization objective(s). 841 5.2. Realizing Instant Selective Flooding 843 Since normal flooding is already supported by GRASP, this section 844 only defines the selective flooding extension. 846 In fragmentary CDDL, the selective flooding follows the pattern: 848 selective-flood-option = [O_SELECTIVE_FLOOD, +O_MATCH-CONDITION, 849 match-object, action] 851 O_MATCH-CONDITION = [O_MATCH-CONDITION, Obj1, match-rule, Obj2] 852 Obj1 = text 854 match-rule = GREATER / LESS / WITHIN / CONTAIN 856 Obj2 = text 857 match-object = NEIGHBOR / SELF 859 action = FORWARD / DROP 861 The option field encapsulates a match-condition option which 862 represents the conditions regarding to continue or discontinue flood 863 the current message. For the match-condition option, the Obj1 and 864 Obj2 are to objects that need to be compared. For example, the Obj1 865 could be the role of the device and Obj2 could be "RSG". The match 866 rules between the two objects could be greater, less than, within, or 867 contain. The match-object represents of which Obj1 belongs to, it 868 could be the device itself or the neighbor(s) intended to be flooded. 869 The action means, when the match rule applies, the current device 870 just continues flood or discontinues. 872 5.3. Realizing Bulk Information Transfer 874 5.4. Realizing Subscription as An Event 876 In fragmentary CDDL, a Subscription Objective Option follows the 877 pattern: 879 subscription-objection-option = [SUBSCRIPTION, 2, 2, subobj] 880 objective-name = SUBSCRIPTION 882 objective-flags = 2 884 loop-count = 2 886 subobj = text 888 This option MAY be included in GRASP M_Synchronization, when 889 included, it means this message is for a subscription to a specific 890 object. 892 5.5. Un_Subscription Objective Option 894 In fragmentary CDDL, a Un_Subscribe Objective Option follows the 895 pattern: 897 Unsubscribe-objection-option = [UNSUBSCRIB, 2, 2, unsubobj] 899 objective-name = SUBSCRIPTION 901 objective-flags = 2 903 loop-count = 2 904 unsubobj = text 906 This option MAY be included in GRASP M_Synchronization, when 907 included, it means this message is for a un-subscription to a 908 specific object. 910 5.6. Publishing Objective Option 912 In fragmentary CDDL, a Publish Objective Option follows the pattern: 914 publish-objection-option = [PUBLISH, 2, 2, pubobj] 916 objective-name = PUBLISH 918 objective-flags = 2 920 loop-count = 2 922 pubobj = text 924 This option MAY be included in GRASP M_Synchronization, when 925 included, it means this message is for a publish of a specific object 926 data. 928 6. Security Considerations 930 The distribution source authentication could be done at multiple 931 layers: 933 o Outer layer authentication: the GRASP communication is within ACP 934 ([I-D.ietf-anima-autonomic-control-plane]). This is the default 935 GRASP behavior. 937 o Inner layer authentication: the GRASP communication might not be 938 within a protected channel, then there should be embedded 939 protection in distribution information itself. Public key 940 infrastructure might be involved in this case. 942 7. IANA Considerations 944 TBD. 946 8. Acknowledgements 948 Valuable comments were received from Zoran Despotovic, Brian 949 Carpenter, Michael Richardson, Roland Bless, Mohamed Boucadair, Diego 950 Lopez, Toerless Eckert and other participants in the ANIMA working 951 group. 953 This document was produced using the xml2rfc tool [RFC2629]. 955 9. References 957 9.1. Normative References 959 [I-D.ietf-anima-grasp] 960 Bormann, C., Carpenter, B., and B. Liu, "A Generic 961 Autonomic Signaling Protocol (GRASP)", draft-ietf-anima- 962 grasp-15 (work in progress), July 2017. 964 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 965 Requirement Levels", BCP 14, RFC 2119, 966 DOI 10.17487/RFC2119, March 1997, 967 . 969 [RFC2629] Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629, 970 DOI 10.17487/RFC2629, June 1999, 971 . 973 9.2. Informative References 975 [I-D.carpenter-anima-grasp-bulk] 976 Carpenter, B., Jiang, S., and B. Liu, "Transferring Bulk 977 Data over the GeneRic Autonomic Signaling Protocol 978 (GRASP)", draft-carpenter-anima-grasp-bulk-05 (work in 979 progress), January 2020. 981 [I-D.du-anima-an-intent] 982 Du, Z., Jiang, S., Nobre, J. C., Ciavaglia, L., and M. 983 Behringer, "ANIMA Intent Policy and Format", draft-du- 984 anima-an-intent-05 (work in progress), February 2017. 986 [I-D.ietf-anima-autonomic-control-plane] 987 Eckert, T., Behringer, M. H., and S. Bjarnason, "An 988 Autonomic Control Plane (ACP)", draft-ietf-anima- 989 autonomic-control-plane-30 (work in progress), October 990 2020. 992 [I-D.ietf-anima-bootstrapping-keyinfra] 993 Pritikin, M., Richardson, M. C., Eckert, T., Behringer, M. 994 H., and K. Watsen, "Bootstrapping Remote Secure Key 995 Infrastructures (BRSKI)", draft-ietf-anima-bootstrapping- 996 keyinfra-45 (work in progress), November 2020. 998 [I-D.ietf-anima-grasp-api] 999 Carpenter, B., Liu, B., Wang, W., and X. Gong, "Generic 1000 Autonomic Signaling Protocol Application Program Interface 1001 (GRASP API)", draft-ietf-anima-grasp-api-10 (work in 1002 progress), January 2021. 1004 [I-D.ietf-anima-reference-model] 1005 Behringer, M. H., Carpenter, B., Eckert, T., Ciavaglia, 1006 L., and J. C. Nobre, "A Reference Model for Autonomic 1007 Networking", draft-ietf-anima-reference-model-10 (work in 1008 progress), November 2018. 1010 [RFC7575] Behringer, M., Pritikin, M., Bjarnason, S., Clemm, A., 1011 Carpenter, B., Jiang, S., and L. Ciavaglia, "Autonomic 1012 Networking: Definitions and Design Goals", RFC 7575, 1013 DOI 10.17487/RFC7575, June 2015, 1014 . 1016 Appendix A. Open Issues [RFC Editor: To Be removed before becoming RFC] 1018 1. More reference to the use cases in the introduction. 1020 2. Better explanation of the required context of the Connected-Car 1021 case: Not applicable unless the ACP will be extended to the car, 1022 which may not be desirable with the current ACP design, but maybe 1023 refocussing on an "autonomous fleet" use-case (e.g.: all cars 1024 operated by some taxi like service). 1026 3. Consider use-case/example of firmware update. By abstracting the 1027 location of the firmware from the name of the firmware, while 1028 providing a way to notify about it, this significantly supports 1029 distribution of firmware updates. References to SUIT would 1030 appropriate. 1032 4. Issues discussed in https://mailarchive.ietf.org/arch/msg/ 1033 anima/_0fYQPBcLPt8xzQee7P4dILsn3A 1035 5. Rethink/refine terminology, e.g.: "module" seems to be too 1036 prescriptive. Refine proposed extensions. 1038 6. Provide more protocol behavior description instead of only 1039 implementation / software module architecture description. 1040 Reduce the latter or provide better justification for their 1041 presence due to explained interoperability requirements. 1043 7. Better motivation in sections 1..4 of the proposed extensions 1044 8. Consider moving examples from appendices into core-text . Ideally 1045 craft a single use-case showing/applying all extensions (most 1046 simple use case that uses them all). 1048 9. Refine terminology to better match/reuse-the established 1049 terminology from the pre-existing ANIMA documents. 1051 Appendix B. Closed Issues [RFC Editor: To Be removed before becoming 1052 RFC] 1054 Appendix C. Change log [RFC Editor: To Be removed before becoming RFC] 1056 draft-ietf-anima-grasp-distribution-00, 2020-02-25: 1058 File name changed following WG adoption. 1060 __Added appendix A&B for open/closed issues. The open issues were 1061 comments received during the adoption call. 1063 Appendix D. Information Distribution Module in ANI 1065 This appendix describes how the information distribution module fits 1066 into the ANI and what extensions of GRASP are required. 1068 (preamble) 1070 +-------------------+ 1071 | ASAs | 1072 +-------------------+ 1073 ^ 1074 | 1075 v 1076 +-------------Info-Dist. APIs--------------+ 1077 | +---------------+ +--------------------+ | 1078 | | Instant Dist. | | Asynchronous Dist. | | 1079 | +---------------+ +--------------------+ | 1080 +------------------------------------------+ 1081 ^ 1082 | 1083 v 1084 +---GRASP APIs----+ 1085 | ACP | 1086 +-----------------+ 1088 Figure E.1 Information Distribution Module and GRASP Extension. 1090 As the Fig 1 shows, the information distribution module two sub- 1091 modules for instant and asynchronous information distributions, 1092 respectively, and provides APIs to ASAs. Specific Behaviors of 1093 modules are described in Section 5. 1095 Appendix E. Asynchronous ID Integrated with GRASP APIs 1097 Actions triggered to the information distribution module will 1098 eventually invoke underlying GRASP APIs. Moreover, EQ and IS modules 1099 are usually correlated. When an AF(ASA) publishes information, not 1100 only such an event is translated and sent to EQ module, but also the 1101 information is indexed and stored simultaneously. Similarly, when an 1102 AF(ASA) subscribes information, not only subscribing event is 1103 triggered and sent to EQ module, but also the information will be 1104 retrieved by IS module at the same time. 1106 o Storing and publishing information: This action involves both IS 1107 and EQ modules where a node that can store the information will be 1108 discovered first and related event will e published to the 1109 network. For this, GRASP APIs discover(), synchronize() and 1110 flood() are combined to compose such a procedure. In specific, 1111 discover() call will specific its objective being to "store_data" 1112 and the return parameters could be either an ASA_locator who will 1113 accept to store the data, or an error code indicating that no one 1114 could afford such data; after that, synchronize() call will send 1115 the data to the specified ASA_locator and the data will be stored 1116 at that node, with return of processing results like 1117 store_data_ack; meanwhile, such a successful event (i.e. data is 1118 stored successfully) will be flooded via a flood() call to 1119 interesting parties (such a multicast group existed). 1121 o Subscribing and getting information: This action involves both IS 1122 and EQ modules as well where a node that is interested in a topic 1123 will subscribe the topic by triggering EQ module and if the topic 1124 is ready IS module will retrieve the content of the topic (i.e. 1125 the data). GRASP APIs such as register_objective(), flood(), 1126 synchronize() are combined to compose the procedure. In specific, 1127 any subscription action received by EQ module will be translated 1128 to register_objective() call where the interested topic will be 1129 the parameter inside of the call; the registration will be 1130 (selectively) flooded to the network by an API call of flood() 1131 with the option we extended in this draft; once a matched topic is 1132 found (because of the previous procedure), the node finding such a 1133 match will call API synchronize() to send the stored data to the 1134 subscriber. 1136 Authors' Addresses 1138 Xun Xiao 1139 Huawei Technologies 1140 Q5, Huawei Campus 1141 No.156 Beiqing Road 1142 Hai-Dian District, Beijing 100095 1143 P.R. China 1145 Email: leo.liubing@huawei.com 1147 Bing Liu 1148 MRC, Huawei Technologies 1149 German Research Center 1150 Huawei Technologies 1151 Riesstr. 25 1152 Muenchen 80992 1153 Germany 1155 Email: xun.xiao@huawei.com 1157 Artur Hecker 1158 MRC, Huawei Technologies 1159 German Research Center 1160 Huawei Technologies 1161 Riesstr. 25 1162 Muenchen 80992 1163 Germany 1165 Email: artur.hecker@huawei.com 1167 Sheng Jiang 1168 Huawei Technologies 1169 Q27, Huawei Campus 1170 No.156 Beiqing Road 1171 Hai-Dian District, Beijing 100095 1172 P.R. China 1174 Email: jiangsheng@huawei.com 1175 Brian 1176 School of Computer Science, University of 1177 Auckland 1178 PB 92019 1179 Auckland 1142 1180 New Zealand 1182 Email: brian.e.carpenter@gmail.com