idnits 2.17.1 draft-irtf-t2trg-iot-edge-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (30 October 2020) is 1273 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-10) exists of draft-bernardos-sfc-fog-ran-08 == Outdated reference: A later version (-21) exists of draft-ietf-core-oscore-groupcomm-09 == Outdated reference: A later version (-05) exists of draft-mcbride-edge-data-discovery-overview-04 == Outdated reference: A later version (-04) exists of draft-sarathchandra-coin-appcentres-03 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Hong 3 Internet-Draft ETRI 4 Intended status: Informational Y-G. Hong 5 Expires: 3 May 2021 Tongmyong University 6 X. de Foy 7 InterDigital Communications, LLC 8 M. Kovatsch 9 Huawei Technologies Duesseldorf GmbH 10 E. Schooler 11 Intel 12 D. Kutscher 13 University of Applied Sciences Emden/Leer 14 30 October 2020 16 IoT Edge Challenges and Functions 17 draft-irtf-t2trg-iot-edge-01 19 Abstract 21 Many IoT applications have requirements that cannot be met by the 22 traditional Cloud (aka cloud computing). These include time 23 sensitivity, data volume, uplink cost, operation in the face of 24 intermittent services, privacy and security. As a result, the IoT is 25 driving the Internet toward Edge computing. This document outlines 26 the requirements of the emerging IoT Edge and its challenges. It 27 presents a general model, and major components of the IoT Edge, with 28 the goal to provide a common base for future discussions in T2TRG and 29 other IRTF and IETF groups. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on 3 May 2021. 48 Copyright Notice 50 Copyright (c) 2020 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 55 license-info) in effect on the date of publication of this document. 56 Please review these documents carefully, as they describe your rights 57 and restrictions with respect to this document. Code Components 58 extracted from this document must include Simplified BSD License text 59 as described in Section 4.e of the Trust Legal Provisions and are 60 provided without warranty as described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 3 66 2.1. Internet of Things (IoT) . . . . . . . . . . . . . . . . 3 67 2.2. Cloud Computing . . . . . . . . . . . . . . . . . . . . . 4 68 2.3. Edge Computing . . . . . . . . . . . . . . . . . . . . . 4 69 2.4. Examples of IoT Edge Computing Use Cases . . . . . . . . 6 70 3. IoT Challenges Leading Towards Edge Computing . . . . . . . . 9 71 3.1. Time Sensitivity . . . . . . . . . . . . . . . . . . . . 9 72 3.2. Uplink Cost . . . . . . . . . . . . . . . . . . . . . . . 10 73 3.3. Resilience to Intermittent Services . . . . . . . . . . . 10 74 3.4. Privacy and Security . . . . . . . . . . . . . . . . . . 10 75 4. IoT Edge Computing Functions . . . . . . . . . . . . . . . . 11 76 4.1. Overview of IoT Edge Computing Today . . . . . . . . . . 11 77 4.2. General Model . . . . . . . . . . . . . . . . . . . . . . 12 78 4.3. OAM Components . . . . . . . . . . . . . . . . . . . . . 16 79 4.3.1. Virtualization Management . . . . . . . . . . . . . . 16 80 4.3.2. Resource Discovery and Authentication . . . . . . . . 17 81 4.3.3. Edge Organization and Federation . . . . . . . . . . 17 82 4.4. Functional Components . . . . . . . . . . . . . . . . . . 18 83 4.4.1. External APIs . . . . . . . . . . . . . . . . . . . . 18 84 4.4.2. Communication Brokering . . . . . . . . . . . . . . . 18 85 4.4.3. In-Network Computation . . . . . . . . . . . . . . . 19 86 4.4.4. Edge Caching . . . . . . . . . . . . . . . . . . . . 20 87 4.4.5. Other Services . . . . . . . . . . . . . . . . . . . 21 88 4.5. Application Components . . . . . . . . . . . . . . . . . 21 89 4.5.1. IoT End Devices Management . . . . . . . . . . . . . 21 90 4.5.2. Data Management . . . . . . . . . . . . . . . . . . . 21 91 4.6. Simulation and Emulation Environments . . . . . . . . . . 22 92 5. Security Considerations . . . . . . . . . . . . . . . . . . . 23 93 6. Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . 23 94 7. Informative References . . . . . . . . . . . . . . . . . . . 23 95 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 97 1. Introduction 99 Currently, many IoT services leverage the Cloud, since it can provide 100 virtually unlimited storage and processing power. The reliance of 101 IoT on back-end cloud computing brings additional advantages such as 102 flexibility and efficiency. Today's IoT systems are fairly static 103 with respect to integrating and supporting computation. It's not 104 that there is no computation, but systems are often limited to static 105 configurations (edge gateways, cloud services). 107 However, IoT devices are creating vast amounts of data at the network 108 edge. To meet IoT use case requirements, that data increasingly is 109 being stored, processed, analyzed, and acted upon close to the data 110 producers. These requirements include time sensitivity, data volume, 111 uplink cost, resiliency in the face of intermittent connectivity, 112 privacy, and security, which cannot be addressed by today's 113 centralized cloud computing. These requirements suggest a more 114 flexible way to distribute computing (and storage) and to integrate 115 it in the edge-cloud continuum. We will refer to this integration of 116 edge computing and IoT as "IoT edge computing". Our draft describes 117 background, uses cases, challenges, and presents system models and 118 functional components. 120 Due to the dynamic nature of the IoT edge computing landscape, this 121 document does not list existing projects in this field. However, 122 Section 4.1 presents a high level overview of the field, based on a 123 limited review of standards, research, open-source and proprietary 124 products in [I-D-defoy-t2trg-iot-edge-computing-background]. 126 2. Background 128 2.1. Internet of Things (IoT) 130 Since the term "Internet of Things" (IoT) was coined by Kevin Ashton 131 in 1999 working on Radio-Frequency Identification (RFID) technology 132 [Ashton], the concept of IoT has evolved. It now reflects a vision 133 of connecting the physical world to the virtual world of computers 134 using (wireless) networks over which things can send and receive 135 information without human intervention. Recently, the term has 136 become more literal by actually connecting things to the Internet and 137 converging on Internet and Web technology. 139 Things are usually embedded systems of various kinds, such as home 140 appliances, mobile equipment, wearable devices, etc. Things are 141 widely distributed, but typically have limited storage and processing 142 power, which raise concerns regarding reliability, performance, 143 energy consumption, security, and privacy [Lin]. This limited 144 storage and processing power leads to complementing IoT with cloud 145 computing. 147 2.2. Cloud Computing 149 Cloud computing has been defined in [NIST]: "cloud computing is a 150 model for enabling ubiquitous, convenient, on-demand network access 151 to a shared pool of configurable computing resources (e.g., networks, 152 servers, storage, applications, and services) that can be rapidly 153 provisioned and released with minimal management effort or service 154 provider interaction". Low cost, massive availability of storage and 155 processing power enabled the realization of another computing model, 156 in which virtualized resources can be leased in an on-demand fashion, 157 being provided as general utilities. Companies like Amazon, Google, 158 Facebook, etc. widely adopted this paradigm for delivering services 159 over the Internet, gaining both economical and technical benefits 160 [Botta]. 162 Today, an unprecedented volume and variety of data is generated by 163 things, and applications deployed at the network edge consume this 164 data. In this context, cloud-based service models are not suitable 165 for some classes of applications, which for example need very short 166 response times, access to local personal data, or generate vast 167 amounts of data. Those applications may instead leverage edge 168 computing. 170 2.3. Edge Computing 172 Edge computing, in some settings also referred to as fog computing, 173 is a new paradigm in which substantial computing and storage 174 resources are placed at the edge of the Internet, that is, in close 175 proximity to mobile devices, sensors, actuators, or machines. Edge 176 computing happens near data sources [Mahadev], or closer 177 (topologically, physically, in term of latency, etc.) to where 178 decisions or interactions with the physical world are happening. It 179 processes both downstream data, e.g. originated from cloud services, 180 and upstream data, e.g. originated from end devices or network 181 elements. The term fog computing usually represents the notion of a 182 multi-tiered edge computing, that is, several layers of compute 183 infrastructure between the end devices and cloud services. 185 An edge device is any computing or networking resource residing 186 between data sources and cloud-based datacenters. In edge computing, 187 end devices not only consume data, but also produce data. And at the 188 network edge, devices not only request services and information from 189 the Cloud, but also handle computing tasks including processing, 190 storage, caching, and load balancing on data sent to and from the 191 Cloud [Shi]. This does not preclude end devices from hosting 192 computation themselves when possible, independently or as part of a 193 distributed edge computing platform (this is also referred to as Mist 194 Computing). 196 Several standards defining organization and industry forums have 197 provided definitions of edge and fog computing: 199 * ISO defines edge computing as a "form of distributed computing in 200 which significant processing and data storage takes place on nodes 201 which are at the edge of the network" [ISO_TR]. 203 * ETSI defines multi-access edge computing as a "system which 204 provides an IT service environment and cloud-computing 205 capabilities at the edge of an access network which contains one 206 or more type of access technology, and in close proximity to its 207 users" [ETSI_MEC_01]. 209 * The Industrial Internet Consortium (formerly OpenFog) defines fog 210 computing as "a horizontal, system-level architecture that 211 distributes computing, storage, control and networking functions 212 closer to the users along a cloud-to-thing continuum" [OpenFog]. 214 Based on these definitions, we can summarize a general philosophy of 215 edge computing as to distribute the required functions close to users 216 and data, while the difference to classic local systems is the usage 217 of management and orchestration features adopted from cloud 218 computing. 220 Actors from various industries approach edge computing using 221 different terms and reference models, although in practice these 222 approaches are not incompatible and may integrate with each other: 224 * The telecommunication industry tends to use a model where edge 225 computing services are deployed over NFV infrastructure at 226 aggregation points, or in proximity to the user equipment (e.g., 227 gNodeBs) [ETSI_MEC_03]. 229 * Enterprise and campus solutions often interpret edge computing as 230 an "edge cloud", that is, a smaller data center directly connected 231 to the local network (often referred to as "on-premise"). 233 * The automation industry defines the edge as the connection point 234 between IT from OT (Operational Technology). Hence, here edge 235 computing sometimes refers to applying IT solutions to OT problems 236 such as analytics, more flexible user interfaces, or simply having 237 more compute power than an automation controller. 239 2.4. Examples of IoT Edge Computing Use Cases 241 IoT edge computing can be used in home, industry, grid, healthcare, 242 city, transportation, agriculture, and/or education scenarios. We 243 discuss here only a few examples of such use cases, to point out 244 differentiating requirements. 246 *Smart Construction* 248 In traditional construction domain, heavy equipment and machinery 249 pose risks to humans and property. Thus, there have been many 250 attempts to deploy technology to protect lives and property in 251 construction sites. For example, measurements of noise, vibration, 252 and gas can be recorded and reported to an inspector. Today, data 253 produced by such measurements is collected by a local gateway and 254 transferred to a remote cloud server. This incurs transmission 255 costs, e.g., over a LTE connection, and storage costs, e.g., when 256 using Amazon Web Services. When an inspector needs to investigate an 257 incident, he checks the information stored on the cloud server. 259 To determine the exact cause of an incident, sensor data including 260 audio and video are transferred to a remote server. In this case, 261 audio and video data volume is typically very large and the cost of 262 transmission can be an issue. By leveraging IoT edge computing, 263 sensor data can be processed and analyzed on a gateway located within 264 or near a construction site. And with the help of statistical 265 analysis or machine learning technologies, we can predict future 266 incidents in advance and trigger an on-site alarm. Furthermore, 267 predicting the time of an incident can help reducing significantly 268 the volume and cost of transmitted data, by transmitting video at 269 high resolution during critical periods, while otherwise using a 270 lower resolution. 272 *Smart Grid* 274 In future smart city scenarios, the Smart Grid will be critical in 275 ensuring highly available/efficient energy control in city-wide 276 electricity management. Edge computing is expected to play a 277 significant role in those systems to improve transmission efficiency 278 of electricity; to react and restore power after a disturbance; to 279 reduce operation costs and reuse renewable energy effectively, since 280 these operations involve local decision making. In addition, edge 281 computing can help monitoring power generation and power demand, and 282 making local electrical energy storage decisions in the smart grid 283 system. 285 *Smart Water System* 287 The water system is one of the most important aspects of a city. 288 Effective use of water, and cost-effective and environment-friendly 289 water treatment are critical aspects of this system. Edge computing 290 can help with monitoring water consumption and transport, and with 291 predicting future water usage level. Examples of application 292 include: water harvesting, ground water monitoring, locally analyzing 293 collected information related to water control and management to 294 limit water losses. 296 *Smart Factory* 298 As part of the 4th industrial revolution, smart factories run real- 299 time processes based on IT technologies such as artificial 300 intelligence and big data. In a smart factory, even a very small 301 environmental change can lead to a situation in which production 302 efficiency decreases or product quality problems occur. Therefore, 303 simple but time sensitive processing can be performed at the edge: 304 for example, controlling temperature and humidity in the factory, or 305 operating machines based on real-time collection of operational 306 status of each machine. On the other hand, data requiring highly 307 precise analysis, such as machine lifecycle management or accident 308 risk prediction, can be transferred to a central data center for 309 processing. 311 The use of edge computing in a smart factory can reduce the cost of 312 network and storage resources by reducing the communication load to 313 the central data center or server. It is also possible to improve 314 process efficiency and facility asset productivity through real-time 315 prediction of facility failure, and to reduce the cost of failure 316 through preliminary measures. In the existing manufacturing field, 317 production facilities are manually run according to a program entered 318 in advance, but edge computing in a smart factory enables tailoring 319 solutions by analyzing data at each production facility and machine 320 level. 322 *Smart Agriculture* 323 Smart agriculture integrates information and communication technology 324 with farming technology. Intelligent farms use IoT technology to 325 measure and analyze temperature, humidity, sunlight, carbon dioxide, 326 soil, etc. in crop cultivation facilities. Depending on analysis 327 results, control devices are used to set environmental parameters to 328 an appropriate state. Remote management is also possible through 329 mobile devices such as smartphones. 331 In existing farms, simple systems such as management according to 332 temperature and humidity can easily and inexpensively be implemented 333 with IoT technology. Sensors in fields are gathering data on field 334 and crop condition. This data is then transmitted to cloud servers, 335 which process data and recommend actions. Usage of edge computing 336 can reduce by a large amount data transmitted up and down the 337 network, resulting in saving cost and bandwidth. Locally generated 338 data can be processed at the edge, and local computing and analytics 339 can drive local actions. With edge computing, it is also easy for 340 farmers to select large amounts of data for processing, and data can 341 be analyzed even in remote areas with poor access conditions. As the 342 number of people working on farming decreases over time, increasing 343 automation enabled by edge computing can be a driving force for 344 future smart agriculture. 346 *Self-Driving Car* 348 The self-driving car, with its focus on safety, is a system where 349 edge computing has an essential role. Autonomous vehicles are 350 equipped with high-resolution cameras, radars, laser scanners 351 (LIDAR), sonar sensors, and GPS systems. Edge computing nodes 352 collect and analyze vast amounts of data generated in real-time by 353 these sensors to keep track of distances between vehicles in front, 354 surrounding road conditions, vehicle flow, and to quickly respond to 355 unexpected situations. For example, if the speed of the car running 356 in front decreases, speed should be adjusted to maintain the distance 357 between the cars, and when a roadside signal changes, a self-driving 358 car should operate according to the new signal. If such a processing 359 is performed in a central data center, network delays or data 360 transmission errors can lead to accidents. Applying edge computing 361 can minimize these network delays and data transmission errors, 362 thereby improving safety. 364 *AR/VR* 366 Augmented Reality (AR) and Virtual Reality (VR) are likely to 367 strongly influence the Information and Communication Technology (ICT) 368 market in the future, since they can support innovative products in 369 most other use cases including smart factory, self-driving car, etc. 370 In AR/VR, due to large amounts of data generated at endpoints such as 371 mobile devices and PCs, user immersion can be significantly decreased 372 by a latency of only a few hundred milliseconds. Therefore, using an 373 edge computing infrastructure built close to endpoints can not only 374 reduce the cost and latency of data transmission, but also maximize 375 user immersion. For example, in AR using edge computing, streaming 376 video can be displayed realistically in higher quality, giving users 377 the best possible experience. 379 3. IoT Challenges Leading Towards Edge Computing 381 This section describes challenges met by IoT, that are motivating the 382 adoption of edge computing for IoT. Those are distinct from research 383 challenges applicable to IoT edge computing, some of which will be 384 mentioned in Section 4.3. 386 IoT technology is used with more and more demanding applications,e.g. 387 in industrial, automotive or healthcare domains, leading to new 388 challenges. For example, industrial machines such as laser cutters 389 already produce over 1 terabyte per hour, and similar amounts can be 390 generated in autonomous cars [NVIDIA]. 90% of IoT data is expected to 391 be stored, processed, analyzed, and acted upon close to the source 392 [Kelly], as cloud computing models alone cannot address the new 393 challenges [Chiang]. 395 Below we discuss IoT use case requirements that are moving cloud 396 capabilities to be more proximate and more distributed and 397 disaggregated. 399 3.1. Time Sensitivity 401 Many industrial control systems, such as manufacturing systems, smart 402 grids, oil and gas systems, etc., often require stringent end-to-end 403 latency between the sensor and control node. While some IoT 404 applications may require latency below a few tens of milliseconds 405 [Weiner], industrial robots and motion control systems have use cases 406 for cycle times in the order of microseconds [_60802]. In some cases 407 speed-of-light limitations may simply prevent a solution based on 408 remote cloud, however it is not the only challenge relative to time 409 sensitivity. An important aspect for real-time communications is not 410 only the latency, but also guarantees for jitter. This means control 411 packets need to arrive with as little variation as possible and 412 within a strict deadline. Given the best-effort characteristics of 413 the Internet this challenge is virtually impossible to address, 414 without using end-to-end guarantees for individual message delivery 415 and continuous data flows. 417 3.2. Uplink Cost 419 Many IoT deployments are not challenged by a constrained network 420 bandwidth to the Cloud. The fifth generation mobile networks (5G) 421 and Wi-Fi 6 both theoretically top out at 10 gigabits per second 422 (i.e., 4.5 terabyte per hour), which enables high-bandwidth uplinks. 423 However, the resulting cost for high-bandwidth connectivity to upload 424 all data to the Cloud is unjustifiable and impractical for most IoT 425 applications. In some settings, e.g. in aeronautical communication, 426 higher communication costs reduce the amount of data that can be 427 practically uploaded even further. 429 3.3. Resilience to Intermittent Services 431 Many IoT devices such as sensors, data collectors, actuators, 432 controllers, etc. have very limited hardware resources and cannot 433 rely solely on their limited resources to meet all their computing 434 and/or storage needs. They require reliable, uninterrupted or 435 resilient services to augment their capabilities in order to fulfill 436 their application tasks. This is hard and partly impossible to 437 achieve with cloud services for systems such as vehicles, drones, or 438 oil rigs that have intermittent network connectivity. The dual is 439 also true, a cloud back-end might want to have a reading of the 440 device even if it's currently asleep. 442 3.4. Privacy and Security 444 When IoT services are deployed at home, personal information can be 445 learned from detected usage data. For example, one can extract 446 information about employment, family status, age, and income by 447 analyzing smart meter data [ENERGY]. Policy makers started to 448 provide frameworks that limit the usage of personal data and put 449 strict requirements on data controllers and processors. However, 450 data stored indefinitely in the Cloud also increases the risk of data 451 leakage, for instance, through attacks on rich targets. 453 Industrial systems are often argued to not have privacy implications, 454 as no personal data is gathered. Yet data from such systems is often 455 highly sensitive, as one might be able to infer trade secrets such as 456 the setup of production lines. Hence, the owner of these systems are 457 generally reluctant to upload IoT data to the Cloud. 459 Furthermore, passive observers can perform traffic analysis on the 460 device-to-cloud path. Hiding traffic patterns associated with sensor 461 networks can therefore be another requirement for edge computing. 463 4. IoT Edge Computing Functions 465 In this section we first look at the current state of IoT edge 466 computing Section 4.1, and then define a general system model 467 Section 4.2. This provides context for IoT edge computing functions, 468 which are listed in Section 4.3. 470 4.1. Overview of IoT Edge Computing Today 472 This section provides an overview of today's IoT edge computing 473 field, based on a limited review of standards, research, open-source 474 and proprietary products in 475 [I-D-defoy-t2trg-iot-edge-computing-background]. 477 IoT gateways, both open-source (such as EdgeX Foundry or Home Edge) 478 and proprietary (such as Amazon Greengrass, Microsoft Azure IoT Edge, 479 Google Cloud IoT Core, and gateways from Bosh, Siemens), represent a 480 common class of IoT edge computing products, where the gateway is 481 providing a local service on customer premises, and is remotely 482 managed through a cloud service. IoT communication protocols are 483 typically used between IoT devices and the gateway, including CoAP, 484 MQTT and many specialized IoT protocols (such as OPC UA and DDS in 485 the Industrial IoT space), while the gateway communicates with the 486 distant cloud typically using HTTPS. Virtualization platforms enable 487 the deployment of virtual edge computing functions (as VMs, 488 application containers, etc.), including IoT gateway software, on 489 servers in the mobile network infrastructure (at base station and 490 concentration points), in edge datacenters (in central offices) or 491 regional datacenters located near central offices. End devices are 492 envisioned to become computing devices in forward looking projects, 493 but are not commonly used as such today. 495 Physical or virtual IoT gateways can host application programs, which 496 are typically built using an SDK to access local services through a 497 programmatic API. Edge cloud system operators host their customers' 498 applications VMs or containers on servers located in or near access 499 networks, which can implement local edge services. For example, 500 mobile networks can provide edge services for radio network 501 information, location and bandwidth management. 503 Life cycle management of services and applications on physical IoT 504 gateways is often cloud-based. Edge cloud management platforms and 505 products (such as StarlingX, Akraino Edge Stack, Mobile EdgeX) adapt 506 cloud management technologies (e.g., Kubernetes) to the edge cloud, 507 i.e., to smaller, distributed computing devices running outside a 508 controlled data center. Services and application life-cycle is 509 typically using a NFV-like management and orchestration model. 511 The platform typically includes services to advertise or consume APIs 512 (e.g., Mp1 interface in ETSI MEC supports service discovery and 513 communication), and enables communicating with local and remote 514 endpoints (e.g., message routing function in IoT gateways). The 515 service platform is typically extensible by edge applications, since 516 they can advertise an API that other edge applications can consume. 517 IoT communication services include protocols translation, analytics 518 and transcoding. Communication between edge computing devices is 519 enabled in tiered deployments or distributed deployments. 521 An edge cloud platform may enable pass-through without storage or 522 local storage (e.g., on IoT gateways). Some edge cloud platforms use 523 a distributed form of storage such as an ICN network (e.g., NFN nodes 524 can store data in NDN) or a distributed storage platform (e.g., IPFS, 525 Ceph). External storage, e.g., on databases in distant or local IT 526 cloud, is typically used for filtered data deemed worthy of long term 527 storage, although in some case it may be for all data, for example 528 when required for regulatory reasons. 530 Stateful computing is supported on platforms hosting native programs, 531 VMs or containers. Stateless computing is supported on platforms 532 providing a "serverless computing" service (a.k.a. function-as- 533 a-service), or on systems based on named function networking. 535 In many IoT use cases, a typical network usage pattern is high volume 536 uplink with some form of traffic reduction enabled by processing over 537 edge computing devices. Alternatives to traffic reduction include 538 deferred transmission (to off-peak hours or using physical shipping). 539 Downlink traffic includes application control and software updates. 540 Other, downlink-heavy traffic patterns are not excluded but are more 541 often associated with non-IoT usage (e.g., video CDNs). 543 4.2. General Model 545 Edge computing is expected to play an important role in deploying new 546 IoT services integrated with Big Data and AI, enabled by flexible in- 547 network computing platforms. Although there are lots of approaches 548 to edge computing, we attempt to lay out a general model and list 549 associated logical functions in this section. In practice, this 550 model can map to different architectures, such as: 552 * A single IoT gateway, or a hierarchy of IoT gateways, typically 553 connected to the cloud (e.g., to extend the traditionally cloud- 554 based management of IoT devices and data to the edge). A common 555 role of an IoT Gateway is to provide access to an heterogeneous 556 set of IoT devices/sensors; handle IoT data; and deliver IoT data 557 to its final destination in a cloud network. Whereas an IoT 558 gateway needs interactions with cloud like as conventional cloud 559 computing, it can also operate independently. 561 * A set of distributed computing nodes, e.g., embedded in switches, 562 routers, edge cloud servers or mobile devices. Some IoT end 563 devices can have enough computing capabilities to participate in 564 such distributed systems due to advances in hardware technology. 565 In this model, edge computing nodes can collaborate with each 566 other to share their resources. 568 +---------------------+ 569 | Remote network | +---------------+ 570 |(e.g., cloud network)| | Service | 571 +-----------+---------+ | Operator | 572 | +------+--------+ 573 | | 574 +--------------+-------------------+-----------+ 575 | Edge Computing Domain | 576 | | 577 | One or more Computing Nodes | 578 | (IoT gateway, end devices, switches, | 579 | routers, mini/micro-datacenters, etc.) | 580 | | 581 | OAM Components | 582 | - Virtualization Management | 583 | - Resource Discovery and Authentication | 584 | - Edge Organization and Federation | 585 | - ... | 586 | | 587 | Functional Components | 588 | - External APIs | 589 | - Communication Brokering | 590 | - In-Network Computation | 591 | - Edge Caching | 592 | - Other Services | 593 | - ... | 594 | | 595 | Application Components | 596 | - IoT End Devices Management | 597 | - Data Management | 598 | - ... | 599 | | 600 +------+--------------+-------- - - - -+- - - -+ 601 | | | | | 602 | | +-----+--+ 603 +----+---+ +-----+--+ | |compute | | 604 | End | | End | ... |node/end| 605 |Device 1| |Device 2| ...| |device n| | 606 +--------+ +--------+ +--------+ 607 + - - - - - - - -+ 609 Figure 1: Model of IoT Edge Computing 611 In the above model, the edge computing domain is interconnected with 612 IoT end devices (southbound connectivity) and possibly with a remote/ 613 cloud network (northbound connectivity), and with a service 614 operator's system. Edge computing nodes provide multiple logical 615 functions, or components, which may not all be present in a given 616 system. They may be implemented in a centralized or distributed 617 fashion, at the network edge, or through some interworking between 618 edge network and remote cloud network. 620 +----------------------------------------------+ 621 | Edge Computing Domain | 622 | | 623 | +--------+ +--------+ +--------+ | 624 | |Compute | |Compute | |Compute | | 625 | |node/End| |node/End| .... |node/End| | 626 | |device 1| |device 2| .... |device m| | 627 | +----+---+ +----+---+ +----+---+ | 628 | | | | | 629 | +---+-------------+-----------------+--+ | 630 | | IoT Edge Gateway | | 631 | +-----------+-------------------+------+ | 632 | | | | 633 +--------------+-------------------+-----------+ 634 | | 635 +-----------+---------+ +------+-------+ 636 | Remote network | | Service | 637 |(e.g., cloud network)| | Operator(s) | 638 +-----------+---------+ +------+-------+ 639 | | 640 +--------------+-------------------+-----------+ 641 | | | | 642 | +-----------+-------------------+------+ | 643 | | IoT Edge Gateway | | 644 | +---+-------------+-----------------+--+ | 645 | | | | | 646 | +----+---+ +----+---+ +----+---+ | 647 | |Compute | |Compute | |Compute | | 648 | |node/End| |node/End| .... |node/End| | 649 | |device 1| |device 2| .... |device n| | 650 | +--------+ +--------+ +--------+ | 651 | | 652 | Edge Computing Domain | 653 +----------------------------------------------+ 655 Figure 2: Example: Machine Learning over a Distributed IoT Edge 656 Computing System 658 In the above example of system, the edge computing domain is composed 659 of IoT edge gateways and IoT end devices which are also used as 660 computing nodes. Edge computing domains are connected with a remote/ 661 cloud network, and with their respective service operator's system. 662 IoT end devices/computing nodes provide logical functions, as part of 663 a distributed machine learning application. The processing 664 capabilities in IoT end devices being limited, they require the 665 support of other nodes: the training process for AI services is 666 executed at IoT edge gateways or cloud networks and the prediction 667 (inference) service is executed in the IoT end devices. 669 We now attempt to enumerate major edge computing domain components. 670 They are here loosely organized into OAM, functional and application 671 components, with the understanding that the distinction between these 672 classes may not always be clear, depending on actual system 673 architectures. Some representative research challenges are 674 associated with those functions. We used input from co-authors, IRTF 675 attendees and some comprehensive reviews of the field ([Yousefpour], 676 [Zhang2], [Khan]). 678 4.3. OAM Components 680 Edge computing OAM goes beyond the network-related OAM functions 681 listed in [RFC6291]. Besides infrastructure (network, storage and 682 computing resources), edge computing systems can also include 683 computing environments (for VMs, software containers, functions), IoT 684 end devices, data and code. 686 Operation related functions include performance monitoring for 687 service level agreement measurement; fault management and 688 provisioning for links, nodes, compute and storage resources, 689 platforms and services. Administration covers network/compute/ 690 storage resources, platforms and services discovery, configuration 691 and planning. Management covers monitoring and diagnostics of 692 failures, as well as means to minimize their occurrence and take 693 corrective actions. This may include software updates management, 694 high service availability through redundancy and multipath 695 communication. Centralized (e.g., SDN) and decentralized management 696 systems can be used. 698 We further detail a few OAM components. 700 4.3.1. Virtualization Management 702 Some IoT edge computing systems make use of virtualized (compute, 703 storage and networking) resources, which need to be allocated and 704 configured. This function is covered to a large extent by ETSI NFV 705 and MEC standards activities. Projects such as [LFEDGE-EVE] further 706 cover virtualization and its management into distributed edge 707 computing settings. 709 Related challenges include: 711 * Minimizing virtual function instantiation time and resource usage 712 * Integration of edge computing with virtualized radio networks (Fog 713 RAN) [I-D.bernardos-sfc-fog-ran] and with 5G access networks 715 * Handling of multi-tenancy with regards to limited resources at the 716 network edge 718 4.3.2. Resource Discovery and Authentication 720 Discovery and authentication may target platforms, infrastructure 721 resources, such as compute, network and storage, but also other 722 resources such as IoT end devices, sensors, data, code units, 723 services, applications or users interacting with the system. Broker- 724 based solutions can be used, e.g. using an IoT gateway as broker to 725 discover IoT resources. Today, centralized gateway-based systems 726 rely, for device authentication, on the installation of a secret on 727 IoT end devices and on computing devices (e.g., a device certificate 728 stored in a hardware security module). 730 Related challenges include: 732 * Discovery, authentication and trust establishment between end 733 devices, compute nodes and platforms, with regards to concerns 734 such as mobility, heterogeneity, scale, multiple trust domains, 735 constrained devices, anonymity and traceability 737 * Intermittent connectivity to the Internet, preventing relying on a 738 third-party authority [Echeverria] 740 * Resiliency to failures [Harchol], denial of service attacks, 741 easier physical access for attackers 743 4.3.3. Edge Organization and Federation 745 In a distributed system context, once edge devices have discovered 746 and authenticated each other, they can be organized, or self- 747 organize, into hierarchies or clusters. Organization may range from 748 centralized to peer-to-peer. Such groups can also form federations 749 with other edge or remote clouds. 751 Related challenges include: 753 * Sharing resources in multi-vendor/operator scenarios, with a goal 754 to optimize criteria such as profit [Anglano], resource usage, 755 latency or energy consumption 757 * Support for scaling, and enabling fault-tolerance or self-healing 758 [Jeong]. Besides using hierarchical organization to cope with 759 scaling, another available and possibly complementary mechanism is 760 multicast ([RFC7390] [I-D.ietf-core-oscore-groupcomm]) 762 * Capacity planning, placement of infrastructure nodes to minimize 763 delay [Fan], cost, energy, etc. 765 * Incentives for participation, e.g. in peer-to-peer federation 766 schemes 768 4.4. Functional Components 770 4.4.1. External APIs 772 An IoT edge cloud may provide a northbound data plane or management 773 plane interface to a remote network, e.g., a cloud, home or 774 enterprise network. This interface does not exist in standalone 775 (local-only) scenarios. To support such an interface when it exists, 776 an edge computing component needs to expose an API, deal with 777 authentication and authorization, support secure communication. 779 An IoT edge cloud may provide an API or interface to local or mobile 780 users, for example to provide access to services and applications, or 781 to manage data published by local/mobile devices. 783 Related challenges include: 785 * Defining edge computing abstractions suitable for users and cloud 786 systems to interact with edge computing systems. In one example, 787 this interaction can be based on the PaaS model [Yangui] 789 4.4.2. Communication Brokering 791 A typical function of IoT edge computing is to facilitate 792 communication with IoT end devices: for example, enable clients to 793 register as recipients for data from devices, as well as forwarding/ 794 routing of traffic to or from IoT end devices, enabling various data 795 discovery and redistribution patterns, e.g., north-south with clouds, 796 east-west with other edge devices 797 [I-D.mcbride-edge-data-discovery-overview]. Another related aspect 798 is dispatching of alerts and notifications to interested consumers 799 both inside and outside of the edge computing domain. Protocol 800 translation, analytics and transcoding may also be performed when 801 necessary. 803 Communication brokering may be centralized in some systems, e.g., 804 using a hub-and-spoke message broker, or distributed like with 805 message buses, possibly in a layered bus approach. Distributed 806 systems may leverage direct communication between end devices, over 807 device-to-device links. A broker can ensure communication 808 reliability, traceability, and in some cases transaction management. 810 Related challenges include: 812 * Enabling secure and resilient communication between IoT end 813 devices and remote cloud, e.g. through multipath support 815 4.4.3. In-Network Computation 817 A core function of IoT edge computing is to enable local computation 818 on a node at the network edge, e.g. processing input data from 819 sensors, making local decisions, preprocessing data, offloading 820 computation on behalf of a device, service or user. Related 821 functions include orchestrating computation (in a centralized or 822 distributed manner) and managing applications lifecycle. Support for 823 in-network computation may vary in term of capability, e.g., 824 computing nodes can host virtual machines, software containers, 825 software actors or unikernels able run stateful or stateless code, or 826 a rule engine providing an API to register actions in response to 827 conditions such as IoT device ID, sensor values to check, thresholds, 828 etc. 830 QoS can be provided in some systems through the combination of 831 network QoS (e.g., traffic engineering or wireless resource 832 scheduling) and compute/storage resource allocations. For example in 833 some systems a bandwidth manager service can be exposed to enable 834 allocation of bandwidth to/from an edge computing application 835 instance. 837 Related challenges include: 839 * (Computation placement) Selecting, in a centralized or 840 distributed/peer-to-peer manner, an appropriate compute device 841 based on available resources, location of data input and data 842 sinks, compute node properties, etc., and with varying goals 843 including for example end-to-end latency, privacy, high 844 availability, energy conservation, network efficiency (e.g. using 845 load balancing techniques to avoid congestion) 847 * Onboarding code on a platform or compute device, and invoking 848 remote code execution, possibly as part of a distributed 849 programming model and with respect to similar concerns of latency, 850 privacy, etc. These operations should deal with heterogeneous 851 compute nodes [Schafer], and may in some cases also support end 852 devices as compute nodes 854 * Adapting Quality of Results (QoR) for applications where a perfect 855 result is not necessary [Li] 857 * Assisted or automatic partitioning of code 858 [I-D.sarathchandra-coin-appcentres] 860 * Supporting computation across trust domains, e.g. verifying 861 computation results 863 * Relocating an instance from one compute node to another, while 864 maintaining a given service level. 866 * Session continuity when communicating with end devices that are 867 mobile, possibly at high speed (e.g. in vehicular scenarios) 869 * Defining, managing and verifying SLAs for edge computing systems. 870 Pricing is a related challenge 872 4.4.4. Edge Caching 874 A purpose of local caching may be to enable local data processing 875 (e.g., pre-processing or analysis), or to enable delayed virtual or 876 physical shipping. A responsibility of the edge caching component is 877 to manage data persistence, e.g., to schedule removal of data when it 878 is no longer needed. Another aspect of this component may be to 879 authenticate and encrypt data. It can for example take the form of a 880 distributed storage system. 882 Related challenges include 884 * (Cache and data placement) Using cache positioning and data 885 placement strategies to minimize data retrieval delay [Liu], 886 energy consumption. Caches may be positioned in the access 887 network infrastructure or may be on end devices using device-to- 888 device communication 890 * Maintaining data consistency, freshness and privacy in systems 891 that are distributed, constrained and dynamic (e.g. due to end 892 devices and computing nodes churn or mobility). For example, age 893 of information [Yates], a performance metric that captures the 894 timeliness of information from a sender (e.g. an IoT device), can 895 be exposed to networks to enable tradeoffs in this problem space 897 4.4.5. Other Services 899 Data generated by IoT devices and associated information obtained 900 from the access network may be used to provide high level services 901 such as end device location, radio network information and bandwidth 902 management. 904 4.5. Application Components 906 IoT edge computing can host applications such as the ones mentioned 907 in Section 2.4. While describing components of individual 908 applications is out of our scope, some of those applications share 909 similar functions, such as IoT end device management, data 910 management, described below. 912 4.5.1. IoT End Devices Management 914 IoT end device management includes managing information about the IoT 915 devices, including their sensors, how to communicate with them, etc. 916 Edge computing addresses the scalability challenges from the massive 917 number of IoT end devices by separating the scalability domain into 918 edge/local networks and remote network. 920 Challenges listed in Section 4.3.2 may be applicable to IoT end 921 devices management as well. 923 4.5.2. Data Management 925 Data storage and processing at the edge is a major aspect of IoT edge 926 computing, directly addressing high level IoT challenges listed in 927 Section 3. Data analysis such as performed in AI/ML tasks performed 928 at the edge may benefit from specialized hardware support on 929 computing nodes. 931 Related challenges include: 933 * Addressing concerns on resource usage, security and privacy when 934 sharing, discovering or managing data. For example by presenting 935 data in views composed of an aggregation of related data [Zhang], 936 protecting data communication between authenticated peers 937 [Basudan], classifying data (e.g., in terms of privacy, 938 importance, validity, etc.), compressing data 940 * Other concerns on edge data discovery (e.g., streaming data, 941 metadata, events) include siloization and lack of standard in edge 942 environments that can be dynamic (e.g. vehicular networks) and 943 heterogeneous [I-D.mcbride-edge-data-discovery-overview] 945 * Data driven programming models [Renart], e.g. event-based, 946 including handling of naming and data abstractions 948 * Addressing concerns such as limited resources, privacy, dynamic 949 and heterogeneous environment, to deploy machine learning at the 950 edge. For example, making machine learning more lightweight and 951 distributed, supporting shorter training time and simplified 952 models, and supporting models that can be compressed for efficient 953 communication [Murshed] 955 * While edge computing can support IoT services independently of 956 cloud computing, it can also be connected to cloud computing. 957 Thus, the relationship of IoT edge computing to cloud computing, 958 with regard to data management, is another potential challenge 959 [ISO_TR] 961 4.6. Simulation and Emulation Environments 963 IoT Edge Computing brings new challenges to simulation and emulation 964 tools used by researchers and developers. A varied set of 965 applications, network and computing technologies can coexist in a 966 distributed system, which make modelling difficult. Scale, mobility 967 and resource management are additional challenges [SimulatingFog]. 969 Tools include simulators, where simplified application logic runs on 970 top of a fog network model, and emulators, where actual applications 971 can be deployed, typically in software containers, over a cloud 972 infrastructure (e.g. Docker, Kubernetes) itself running over a 973 network emulating network edge conditions such as variable delays, 974 throughput and mobility events. To gain in scale, emulated and 975 simulated systems can be used together in hybrid federation-based 976 approaches [PseudoDynamicTesting], while to gain in realism physical 977 devices can be interconnected with emulated systems. Examples of 978 related work and platforms include the publicly accessible MEC 979 sandbox work recently initiated in ETSI [ETSI_Sandbox], and open 980 source simulators and emulators ([AdvantEDGE] emulator and tools 981 cited in [SimulatingFog]). 983 5. Security Considerations 985 As discussed in Section 4.3.2, authentication and trust (between 986 computing nodes, management nodes, end devices) can be challenging as 987 scale, mobility and heterogeneity increase. The sometimes 988 disconnected nature of edge resources can prevent relying on a third- 989 party authority. Distributed edge computing is exposed to issues 990 with reliability and denial of service attacks. Personal or 991 proprietary IoT data leakage is also a major threat, especially due 992 to the distributed nature of the systems (Section 4.5.2). 994 However, edge computing also brings solutions in the security space: 995 maintaining privacy by computing sensitive data closer to data 996 generators is a major use case for IoT edge computing. An edge cloud 997 can be used to take actions based on sensitive data, or anonymizing, 998 aggregating or compressing data prior to transmitting to a remote 999 cloud server. Edge computing communication brokering functions can 1000 also be used to secure communication between edge and cloud networks. 1002 6. Acknowledgment 1004 The authors would like to thank Joo-Sang Youn, Akbar Rahman, Michel 1005 Roy, Robert Gazda, Rute Sofia, Thomas Fossati and Chonggang Wang for 1006 their valuable comments and suggestions on this document. 1008 7. Informative References 1010 [AdvantEDGE] 1011 "Mobile Edge Emulation Platform", Source Code Repository , 1012 2020, . 1014 [Anglano] Anglano, C., Canonico, M., Castagno, P., Guazzone, M., and 1015 M. Sereno, "A game-theoretic approach to coalition 1016 formation in fog provider federations", IEEE Third 1017 International Conference on Fog and Mobile Edge Computing 1018 (FMEC), pages 123-130 , 2018. 1020 [Ashton] Ashton, K., "That Internet of Things thing", RFID J. vol. 1021 22, no. 7, pp. 97-114 , 2009. 1023 [Basudan] Basudan, S., Lin, X., and K. Sankaranarayanan, "A privacy- 1024 preserving vehicular crowdsensing-based road surface 1025 condition monitoring system using fog computing", IEEE 1026 Internet of Things Journal, 4(3):772-782 , 2017. 1028 [Botta] Botta, A., Donato, W., Persico, V., and A. Pescape, 1029 "Integration of Cloud Computing and Internet of Things: A 1030 survey", Future Gener. Comput. Syst., vol. 56, pp. 1031 684-700 , 2016. 1033 [Chiang] Chiang, M. and T. Zhang, "Fog and IoT: An overview of 1034 research opportunities", IEEE Internet Things J., vol. 3, 1035 no. 6, pp. 854-864 , 2016. 1037 [Echeverria] 1038 Echeverria, S., Klinedinst, D., Williams, K., and G. A 1039 Lewis, "Establishing trusted identities in disconnected 1040 edge environments", IEEE/ACM Symposium Edge Computing 1041 (SEC), pages 51-63. , 2016. 1043 [ENERGY] Beckel, C., Sadamori, L., Staake, T., and S. Santini, 1044 "Revealing Household Characteristics from Smart Meter 1045 Data", Energy, vol. 78, pp. 397-410 , 2014. 1047 [ETSI_MEC_01] 1048 ETSI, ., "Multi-access Edge Computing (MEC); Terminology", 1049 ETSI GS 001 , 2019, . 1052 [ETSI_MEC_03] 1053 ETSI, ., "Mobile Edge Computing (MEC); Framework and 1054 Reference Architecture", ETSI GS 003 , 2019, 1055 . 1058 [ETSI_Sandbox] 1059 "Multi-access Edge Computing (MEC) MEC Sandbox Work Item", 1060 Portal , 2020, 1061 . 1064 [Fan] Fan, Q. and N. Ansari, "Cost aware cloudlet placement for 1065 big data processing at the edge", IEEE International 1066 Conference on Communications (ICC), pages 1-6 , 2017. 1068 [Harchol] Harchol, Y., Mushtaq, A., McCauley, J., Panda, A., and S. 1069 Shenker, "Cessna: Resilient edge-computing", Workshop on 1070 Mobile Edge Communications, pages 1-6. ACM , 2018. 1072 [I-D-defoy-t2trg-iot-edge-computing-background] 1073 de Foy, X., Hong, J., Hong, Y., Kovatsch, M., Schooler, 1074 E., and D. Kutscher, "Machine learning at the network 1075 edge: A survey", draft-defoy-t2trg-iot-edge-computing- 1076 background-00 , 2020, . 1080 [I-D.bernardos-sfc-fog-ran] 1081 Bernardos, C., Rahman, A., and A. Mourad, "Service 1082 Function Chaining Use Cases in Fog RAN", Work in Progress, 1083 Internet-Draft, draft-bernardos-sfc-fog-ran-08, 17 1084 September 2020, . 1087 [I-D.ietf-core-oscore-groupcomm] 1088 Tiloca, M., Selander, G., Palombini, F., and J. Park, 1089 "Group OSCORE - Secure Group Communication for CoAP", Work 1090 in Progress, Internet-Draft, draft-ietf-core-oscore- 1091 groupcomm-09, 23 June 2020, . 1094 [I-D.mcbride-edge-data-discovery-overview] 1095 McBride, M., Kutscher, D., Schooler, E., Bernardos, C., 1096 and D. Lopez, "Edge Data Discovery for COIN", Work in 1097 Progress, Internet-Draft, draft-mcbride-edge-data- 1098 discovery-overview-04, 13 July 2020, . 1102 [I-D.sarathchandra-coin-appcentres] 1103 Trossen, D., Sarathchandra, C., and M. Boniface, "In- 1104 Network Computing for App-Centric Micro-Services", Work in 1105 Progress, Internet-Draft, draft-sarathchandra-coin- 1106 appcentres-03, 23 October 2020, . 1110 [ISO_TR] "Information Technology - Cloud Computing - Edge Computing 1111 Landscape", ISO/IEC TR 23188 , 2018. 1113 [Jeong] Jeong, T., Chung, J., Hong, J.W., and S. Ha, "Towards a 1114 distributed computing framework for fog", IEEE Fog World 1115 Congress (FWC), pages 1-6 , 2017. 1117 [Kelly] Kelly, R., "Internet of Things Data to Top 1.6 Zettabytes 1118 by 2022", 2015, 1119 . 1123 [Khan] Khan, L.U., Yaqoob, I., Tran, N.H., Kazmi, S.M.A., Dang, 1124 T.N., and C.S. Hong, "Edge Computing Enabled Smart Cities: 1125 A Comprehensive Survey", arXiv:1909.08747 , 2019. 1127 [LFEDGE-EVE] 1128 Linux Foundation, ., "Project Edge Virtualization Engine 1129 (EVE)", Portal , 2020, 1130 . 1132 [Li] Li, Y., Chen, Y., Lan, T., and G. Venkataramani, "Mobiqor: 1133 Pushing the envelope of mobile edge computing via quality- 1134 of-result optimization", IEEE 37th International 1135 Conference on Distributed Computing Systems (ICDCS), pages 1136 1261-1270 , 2017. 1138 [Lin] Lin, J., Yu, W., Zhang, N., Yang, X., Zhang, H., and W. 1139 Zhao, "A survey on Internet of Things: Architecture, 1140 enabling technologies, security and privacy, and 1141 applications", IEEE Internet of Things J., vol. 4, no. 5, 1142 pp. 1125-1142 , 2017. 1144 [Liu] Liu, J., Bai, B., Zhang, J., and K.B. Letaief, "Cache 1145 placement in fog-rans: From centralized to distributed 1146 algorithms", IEEE Transactions on Wireless Communications, 1147 16(11):7039-7051 , 2017. 1149 [Mahadev] Satyanarayanan, M., "The Emergence of Edge Computing", 1150 Computer, vol. 50, no. 1, pp. 30-39 , 2017. 1152 [Murshed] Murshed, M., Murphy, C., Hou, D., Khan, N., 1153 Ananthanarayanan, G., and F. Hussain, "Machine learning at 1154 the network edge: A survey", arXiv:1908.00080 , 2019. 1156 [NIST] Mell, P. and T. Grance, "The NIST definition of Cloud 1157 Computing", Natl. Inst. Stand. Technol, vol. 53, no. 6, p. 1158 50 , 2009. 1160 [NVIDIA] Grzywaczewski, A., "Training AI for Self-Driving Vehicles: 1161 the Challenge of Scale", NVIDIA Developer Blog , 2017, 1162 . 1165 [OpenFog] "OpenFog Reference Architecture for Fog Computing", 1166 OpenFog Consortium , 2017. 1168 [PseudoDynamicTesting] 1169 Ficco, M., Esposito, C., Xiang, Y., and F. Palmieri, 1170 "Pseudo-Dynamic Testing of Realistic Edge-Fog Cloud 1171 Ecosystems", IEEE Communications Magazine, Nov. 2017 , 1172 2017. 1174 [Renart] Renart, E.G., Diaz-Montes, J., and M. Parashar, "Data- 1175 driven stream processing at the edge", IEEE 1st 1176 International Conference on Fog and Edge Computing 1177 (ICFEC), pages 31-40 , 2017. 1179 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 1180 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 1181 Acronym in the IETF", BCP 161, RFC 6291, 1182 DOI 10.17487/RFC6291, June 2011, 1183 . 1185 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 1186 the Constrained Application Protocol (CoAP)", RFC 7390, 1187 DOI 10.17487/RFC7390, October 2014, 1188 . 1190 [Schafer] Schafer, D., Edinger, J., VanSyckel, S., Paluska, J.M., 1191 and C. Becker, "Tasklets: Overcoming Heterogeneity in 1192 Distributed Computing Systems", IEEE 36th International 1193 Conference on Distributed Computing Systems Workshops 1194 (ICDCSW), Nara, pp. 156-161 , 2016. 1196 [Shi] Shi, W., Cao, J., Zhang, Q., Li, Y., and L. Xu, "Edge 1197 computing: vision and challenges", IEEE Internet of Things 1198 J., vol. 3, no. 5, pp. 637-646 , 2016. 1200 [SimulatingFog] 1201 Svorobej, S. and . al, "Simulating Fog and Edge Computing 1202 Scenarios: An Overview and Research Challenges", MPDI 1203 Future Internet 2019 , 2019. 1205 [Weiner] Weiner, M., Jorgovanovic, M., Sahai, A., and B. Nikolie, 1206 "Design of a low-latency, high-reliability wireless 1207 communication system for control applications", IEEE Int. 1208 Conf. Commun. (ICC), Sydney, NSW, Australia, pp. 1209 3829-3835 , 2014. 1211 [Yangui] Yangui, S., Ravindran, P., Bibani, O., H Glitho, R., Ben 1212 Hadj-Alouane, N., Morrow, M.J., and P.A. Polakos, "A 1213 platform as-a-service for hybrid cloud/fog environments", 1214 IEEE International Symposium on Local and Metropolitan 1215 Area Networks (LANMAN), pages 1-7 , 2016. 1217 [Yates] Yates, R.D. and S.K. Kaul, "The Age of Information: Real- 1218 Time Status Updating by Multiple Sources", IEEE 1219 Transactions on Information Theory, vol. 65, no. 3, pp. 1220 1807-1827 , 2019. 1222 [Yousefpour] 1223 Yousefpour, A., Fung, C., Nguyen, T., Kadiyala, K., 1224 Jalali, F., Niakanlahiji, A., Kong, J., and J.P. Jue, "All 1225 one needs to know about fog computing and related edge 1226 computing paradigms: A complete survey", Journal of 1227 Systems Architecture, vol. 98, pp. 289-330 , 2019. 1229 [Zhang] Zhang, Q., Zhang, X., Zhang, Q., Shi, W., and H. Zhong, 1230 "Firework: Big data sharing and processing in 1231 collaborative edge environment", Fourth IEEE Workshop on 1232 Hot Topics in Web Systems and Technologies (HotWeb), pages 1233 20-25 , 2016. 1235 [Zhang2] Zhang, J., Chen, B., Zhao, Y., Cheng, X., and F. Hu, "Data 1236 Security and Privacy-Preserving in Edge Computing 1237 Paradigm: Survey and Open Issues", IEEE Access, vol. 6, 1238 pp. 18209-18237 , 2018. 1240 [_60802] IEC/IEEE, ., "Use Cases IEC/IEEE 60802 V1.3", IEC/IEEE 1241 60802 , 2018, . 1244 Authors' Addresses 1245 Jungha Hong 1246 ETRI 1247 218 Gajeong-ro Yuseung-Gu 1248 Daejeon 1249 34129 1250 Republic of Korea 1252 Email: jhong@etri.re.kr 1254 Yong-Geun Hong 1255 Tongmyong University 1256 428 Sinseon-ro Nam-gu 1257 Busan 1258 48520 1259 Republic of Korea 1261 Email: yonggeun.hong@gmail.com 1263 Xavier de Foy 1264 InterDigital Communications, LLC 1265 1000 Sherbrooke West 1266 Montreal H3A 3G4 1267 Canada 1269 Email: xavier.defoy@interdigital.com 1271 Matthias Kovatsch 1272 Huawei Technologies Duesseldorf GmbH 1273 Riesstr. 25 C // 3.OG 1274 80992 Munich 1275 Germany 1277 Email: ietf@kovatsch.net 1279 Eve Schooler 1280 Intel 1281 2200 Mission College Blvd. 1282 Santa Clara, CA, 95054-1537 1283 United States of America 1285 Email: eve.m.schooler@intel.com 1286 Dirk Kutscher 1287 University of Applied Sciences Emden/Leer 1288 Constantiaplatz 4 1289 26723 Emden 1290 Germany 1292 Email: ietf@dkutscher.net