idnits 2.17.1 draft-km-iotops-iiot-frwk-02.txt: -(797): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (5 March 2022) is 783 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Independent Submission K. Makhijani 3 Internet-Draft L. Dong 4 Intended status: Informational Futurewei 5 Expires: 6 September 2022 5 March 2022 7 Virtualization of PLC in Industrial Networks - Problem Statement 8 draft-km-iotops-iiot-frwk-02 10 Abstract 12 Conventional Programmable Logic Controllers (PLCs) impose several 13 challenges on factory floors as their numbers and size on the factory 14 floors/plants continues to grow. Virtualized PLCs can help overcome 15 many of those concerns. They can improve the automation in Industry 16 control networks by simplifying communication between higher-level 17 applications and low-level factory floor machine operations. Virtual 18 PLCs provide an opportunity to integrate a diverse set of non- 19 internet protocols supporting Industrial-IoT and IP connections to 20 improve coordination between applications and field devices. Besides 21 automation, virtual PLCs also enhance programmability in industry 22 process control systems by abstracting control functions from I/O 23 modules. However, to achieve desired outcome and benefits, both 24 operational and application networks should evolve. 26 This document introduces virtual PLC concept, describes the details 27 and benefits of virtualized PLCs, then focuses on the problem 28 statement and requirements. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on 6 September 2022. 47 Copyright Notice 49 Copyright (c) 2022 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 54 license-info) in effect on the date of publication of this document. 55 Please review these documents carefully, as they describe your rights 56 and restrictions with respect to this document. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 5 63 3. Virtualized PLCs . . . . . . . . . . . . . . . . . . . . . . 5 64 3.1. Definition . . . . . . . . . . . . . . . . . . . . . . . 6 65 3.2. Limitations with Physical PLCs . . . . . . . . . . . . . 6 66 3.2.1. Integrated Application Control Loop . . . . . . . . . 6 67 3.2.2. Single purpose to Multipurpose . . . . . . . . . . . 7 68 3.2.3. Simulation and Analytics . . . . . . . . . . . . . . 7 69 3.2.4. Managing Complexity . . . . . . . . . . . . . . . . . 7 70 3.3. Benefits and Opportunities . . . . . . . . . . . . . . . 8 71 3.3.1. Processing Capabilities . . . . . . . . . . . . . . . 8 72 3.3.2. Flexibility and Efficient Resource Use . . . . . . . 8 73 3.3.3. Interoperability and Optimization . . . . . . . . . . 8 74 3.3.4. Device Density on Factory Floor . . . . . . . . . . . 8 75 3.4. Incremental Realization Approaches . . . . . . . . . . . 9 76 3.4.1. Softwarized PLC . . . . . . . . . . . . . . . . . . . 9 77 3.4.2. Local Disaggregation of Control and I/O Modules . . . 9 78 3.4.3. Fully Virtualized PLC . . . . . . . . . . . . . . . . 10 79 4. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 10 80 4.1. Overview of Industrial Network Architecture . . . . . . . 10 81 4.2. Associating virtualized PLCs with IO Devices . . . . . . 11 82 4.3. Expectations from the Networks . . . . . . . . . . . . . 12 83 4.3.1. Hierarchical Structure . . . . . . . . . . . . . . . 12 84 4.3.2. Safety and Reliability of Operations . . . . . . . . 12 85 4.4. Multiprotocol Supporting PLCs . . . . . . . . . . . . . . 12 86 4.5. Identification of virtualized PLC . . . . . . . . . . . . 13 87 4.6. Security Aspects . . . . . . . . . . . . . . . . . . . . 13 88 5. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 13 89 5.1. Virtualized PLC Requirements . . . . . . . . . . . . . . 13 90 5.2. Key Performance Indicator Requirements . . . . . . . . . 15 91 5.3. Network Related Requirements . . . . . . . . . . . . . . 16 92 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 93 7. Security Considerations . . . . . . . . . . . . . . . . . . . 17 94 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 95 9. Informative References . . . . . . . . . . . . . . . . . . . 17 96 Appendix A. Appendix A. Purdue Model (ICA-95) . . . . . . . . . 19 97 A.1. Separation between Manufacturing and Enterprise 98 Networks . . . . . . . . . . . . . . . . . . . . . . . . 20 99 A.2. Collaborating with SDOs with Industry Network Focus . . . 21 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 102 1. Introduction 104 Programmable Logic Controllers (PLCs) have been instrumental to the 105 growth of automation in industrial process control. Industry 4.0 and 106 similar initiatives have put even more emphasis on automation of the 107 entire production process. For example, a typical workflow in the 108 Smart Factory to manufacture customized orders (reconfigurable 109 manufacturing [RECONF]) is executed autonomously, comprising several 110 related and inter-dependent processes. In this workflow, all the 111 dependencies and transitions occur seamlessly without human 112 intervention - such as requesting inventory before it becomes 113 unavailable, dispatching a request for specific maintenance, 114 performing quality control on the material, and adjusting operations 115 automatically. 117 This type of system-level automation requires close coordination 118 between PLCs (low-level machine controlling components) on the 119 factory floors and the high-level decision-making software. However, 120 in the current Industry control architecture, PLC operations are 121 isolated from higher-level components; they operate in an entirely 122 different proprietary hardware environment. Moreover, the number of 123 PLCs on a floor are growing along with their physical size to support 124 faster processors and more memory. This makes management of PLCs 125 with different type of hardware even more difficult. Although PLCs 126 can be customized, they are designed for limited set of controls, 127 therefore their extensibility is limited. To overcome above 128 mentioned challenges it should be possible to incorporate multiple 129 control functions in a hardware-agnostic platform. 131 Virtualization is a proven technique to abstract software logic from 132 the underlying hardware. Information Technology (IT) has proven that 133 virtualization benefits cost savings, flexibility, and efficient 134 resource usage. In the context of Industrial networks, 135 virtualization serves to integrate IT and OT software components, 136 which are essential for integrated automation. 138 This document describes the 'virtualized PLC' concept and its 139 realization. In Section 4 limitations in physical PLCs are covered 140 along with the benefits of virtualized PLC. Finally, Section 5 141 discusses requirements to support virtualized PLCs and their impact 142 on the network. 144 2. Terminology 146 Industrial Control Network: 147 Industrial control networks are the interconnection of equipment 148 used to operate, control, or monitor machines in the industry 149 environment. It involves different levels of communications - 150 between field bus devices, digital controllers, and software 151 applications. 153 Industry Automation: 154 Mechanisms that enable the machine to machine communication by use 155 of technologies that enable automatic control and operation of 156 industrial devices and processes leading to minimizing human 157 intervention. 159 Control Loop: 160 Control loops are part of process control systems in with desired 161 process response is provided as an input to the controller, which 162 performs the corresponding action (using actuators) and reads the 163 output values. Since no error correction is performed, these are 164 called open control loops. 166 Feedback Control Loop: 167 Feedback control loop is a system in which the output of a control 168 system is continuously measured and compared to the input 169 reference value. The controller uses any deviation from the input 170 value to adjust the output value for the desired response. Since 171 there is a feedback of error signal to the input, these are called 172 closed control loops. 174 Programmable logic controllers (PLC): 175 Industrial computers/servers to control manufacturing processes 176 such as assembly lines. 178 Supervisory Control and Data Acquisition (SCADA): 179 Software System to control industrial processes and collect and 180 manage data. 182 Distributed Control Systems (DCS): 183 Systems of sensors and controllers that are distributed throughout 184 a plant. 186 Manufacturing Execution System (MES): 187 Systems that connect production equipment across the factory floor 188 or multiple plants or sites. 190 Fieldbus Devices: 192 Operational Technology field devices include valves, transmitters, 193 switches, actuators, etc. 195 Virtualized PLC (vPLC): 196 A software component of PLC, in which the control part of factory 197 devices is decoupled from the I/O component. With vPLCs, the I/O 198 stays local to the machines (sensors, actuators, and drives), 199 while the controller logic lives as a software service implemented 200 over RT- hypervisors. 202 Scan-Cycle: 203 A scan cycle is the time to read the inputs, execute the program 204 (e.g., ladder logic), and update the outputs. The actual scan 205 time is affected by the processing speed of the PLC, the size of 206 the program, the type of instructions used in the program. In 207 virtualized PLCs, general-purpose processor speed and memory are 208 much higher than most physical PLCs. 210 2.1. Acronyms 212 * HMI: Human Machine Interface 214 * MES: Manufacturing Execution System 216 * CIN: Converged Industrial Network 218 * IIC: Industrial Internet Consortium 220 * IDMZ: Industrial Demilitarized Zone 222 * PLC: Programmable Logic Controller 224 * PDU: Protocol Data unit 226 * SCADA: Supervisory Control And Data Acquisition 228 * DCS: Distributed Control System 230 * OT: Operational Technology 232 * IT: Information Technology 234 3. Virtualized PLCs 235 3.1. Definition 237 Programmable Logic Controllers (PLCs) are specialized physical 238 devices (or computers) that are used to control the operation of 239 machines by coordinating the input sensors (temperature, pressure, 240 position, vibration, humidity, torque, etc. readings) to the output 241 actuators (such as motion control, voltage change, pressure valves, 242 etc.). PLC components include a control unit, memory (to store the 243 data, state, and process control instructions), and I/O modules to 244 communicate with Fieldbus devices (sensors and actuators) using 245 different standard or proprietary protocols. 247 Compared with commodity CPUS, most PLC control unit processing power 248 is extremely low, whereas new complex process control applications 249 require sophisticated and faster compute capabilities. Utilizing 250 commodity-grade CPUs for many PLC function blocks provides higher 251 compute and memory for PLC programs by separating its control unit 252 and memory from the physical PLCs. This will leave only I/O modules 253 connected to the devices. Thus, 255 Virtualized PLC is a hardware-agnostic abstraction of the 256 control unit and memory functions of a PLC. It is hardware- 257 independent and still needs an interface to communicate with 258 the I/O modules. 260 The concept has been discussed both in research [PLC-40] and industry 261 [VPLC-DRAGOS] [VPLC_IIC] [VPLC_CONV]. In the following section 262 motivation for virtualized-PLCs. 264 3.2. Limitations with Physical PLCs 266 3.2.1. Integrated Application Control Loop 268 Application performance is improved with better coordination between 269 applications and field devices. One way to achieve this is when 270 seamless sharing of both data and control operations, and it is 271 possible when both application and controller software use a common 272 language or interface. Today OPC-UA model is well-established and 273 provides a protocol-independent data model for the standard 274 representation of several Fieldbus protocols and requires a 275 translation layer. The use of software PLCs can unify the collection 276 of data and control processes even more efficiently since the 277 software PLCs are already hardware-independent. 279 Like IT, the manufacturing and process industry is evolving to a non- 280 monolithic mode of system operations. In a large-scale industrial 281 operation, several control processes run simultaneously and have 282 high-performance requirements. 284 3.2.2. Single purpose to Multipurpose 286 Currently, PLC controllers are designed for a single purpose long- 287 term use. There is an implicit expectation that PLC functions and 288 corresponding I/O devices will not be replaced for many years once 289 installed. This paradigm makes it difficult for industries to handle 290 changing requirements and can be prohibitive to adopting new 291 technologies and deploying new types of sensors that could provide 292 better monitoring. With virtualized PLC, re-programming control 293 logic to tweak the assembly line becomes a lot easier. 295 3.2.3. Simulation and Analytics 297 Physical PLCs are difficult to troubleshoot. Upon failures, 298 operators have to manually study the log files to generate traces 299 from historical data. Since Virtual PLCs are hardware-agnostic, they 300 are almost identical to their simulation counterparts. When replayed 301 with actual historical event data, the run-time state of a PLC at any 302 instance in the past can be recreated, which would help to 303 troubleshoot and root-cause failure events. It is difficult to do 304 this type of root-cause analysis with physical PLCs. 306 3.2.4. Managing Complexity 308 Complexity is a trait of overall system architecture. With Physical 309 PLCS, the plant-floors will continue to deploy proprietary protocols 310 and PLCs, leading to either managing solutions from different vendors 311 or being locked into one vendor-provided solution. While the former 312 adds to the complexity, the latter may not use innovation outside a 313 specific vendor. 315 Architecturally, PLCs require a lot of different types of 316 connections, such as PLC-PLC (peer to peer), PLC-SCADA, PLC-HMI, etc. 317 Depending on the interface and protocol, scaling PLCs would lead to a 318 higher number of gateways (and more wiring) that are difficult from a 319 maintenance perspective and can also cause poor performance. With 320 physical PLCs, heterogeneity of protocol interface will not go away. 322 Faults with PLC input/output (I/O) modules and field devices account 323 for 80 percent of system failures. Common causes of failure include 324 the rugged environment that devices are subjected to. In some cases, 325 consolidating different PLCs on a single powerful PC and protecting a 326 single node (hosting several PLCs) from failures of a power outage, 327 electromagnetic or radio frequency interference is a lot easier than 328 protecting a high number of PLCs. In other cases, PLCs can be placed 329 in the edge network, separated from the rugged environment. 331 3.3. Benefits and Opportunities 333 3.3.1. Processing Capabilities 335 Virtualization enables running software on commodity hardware. One 336 of the most important benefits is using more sophisticated processors 337 to perform complex computations beyond legacy PLCs (floating point, 338 arithmetic operations, counters, etc.). Currently, there are already 339 PLC control units supported on FPGAs [FPGA_PLC] indicating the need 340 for faster and parallel processing. Virtualization will enable 341 further integration of such different With the availability of high- 342 en. 344 3.3.2. Flexibility and Efficient Resource Use 346 Traditional PLCs are fixed-function controllers typically used for 347 specific jobs on the factory floor. Today, software-based PLCs are 348 available for general-purpose commercial hardware, but they have been 349 mainly used for simulation and training purposes. Now there is more 350 emphasis on customizations which will require PLCs to be programmed 351 every time a new custom product is requested, leading to longer 352 manufacturing cycles. Virtualization can enable running multiple 353 instances with its own set of allocated resources. Thus, it will be 354 possible to run different configurations for different customizations 355 simultaneously with efficient use of resources only on-demand. 357 Moreover, when virtualized PLCs and IT applications are on the same 358 platform, it is possible to have close coordination between the OT 359 and IT functions. Although it may not be compelling, virtualized 360 PLCs potentially eliminate the need for dedicated PLCs on the floor, 361 creating space and reducing the number of interconnections. 363 3.3.3. Interoperability and Optimization 365 Having abstracted PLC logic allows using a common communication 366 protocol, thus improving interoperability between different vendors 367 supplied I/O modules. Besides improving performance, this approach 368 also simplifies configuration, configuration, and monitoring. 370 3.3.4. Device Density on Factory Floor 372 With the innovations in IoT devices, it is anticipated that there 373 will be newer ways to measure, monitor, and collect various 374 environment-specific metrics; this signifies an even larger number of 375 devices and a corresponding increase in the number of controllers. 376 Virtualization can further simplify control of a considerably high 377 number of devices through a single PLC, thereby reducing some network 378 resource requirements. 380 While applications and services are beginning to get disaggregated, 381 PLCs' virtualization is very early stage. 383 3.4. Incremental Realization Approaches 385 Once virtualized, a PLC may be placed flexible anywhere in the 386 network and closer to the higher-level applications. However, 387 expanding beyond a factory site is a drastic change from the existing 388 isolated OT mindset. To address such concerns, the following 389 different approaches are possible: 391 3.4.1. Softwarized PLC 393 This is the basic approach with minimal change and minimal impact. A 394 PLC software is virtualized and runs on proprietary or commodity 395 hardware supporting legacy I/O modules. This type of change is 396 isolated to a specific PLC functionality, and the only benefit is 397 hardware independence. Potentially, there is a one-to-one 398 replacement of physical to software PLC. 400 3.4.2. Local Disaggregation of Control and I/O Modules 402 In addition to above approach, the software component of PLC (its 403 control unit) runs on commodity hardware; I/O modules are separated 404 from the PLC to provide a clear separation between I/O and 405 programmable components. It requires trivial I/0 interconnects to do 406 trivial Fieldbus frame forwarding to I/O modules which may not 407 require any memory or processing capability as shown in Figure 1. 409 .-,,-. fieldbus 410 +-+ .-( cite )-. IP _______ i/f 411 | | ---->( network )------->[_______] ------> |==| 412 +-+ '-( ).-' I/O I/O device 413 virtualized '-.-' inter-connects 414 PLC 416 Figure 1: virtualization of PLC and separation from I/O devices 418 Utilizing IT-style virtualization infrastructure, different instances 419 of virtualized PLC may run simultaneously on a single machine, or 420 even different types of PLCs may run together as a single instance of 421 virtualized PLC. A clean separation between PLC logic from I/O 422 module allows changes to PLC logic and I/O devices independently. 423 With this level of hardware independence, a virtualized PLC can be 424 instantiated on the same hardware and SCADA, HMI, or ICS components 425 providing close integration of these entities. 427 Since the location of virtualized PLC is within the manufacturing 428 zone, there is no impact on the security design. 430 3.4.3. Fully Virtualized PLC 432 Eventually, virtualized PLCs may be placed anywhere (in the cloud, 433 edge, or on-site) in a location-independent manner. All the benefits 434 considered in Section 3.4.2 apply with an advantage of leveraging 435 multi-tenant edge-compute infrastructure as a tenant. 437 However, the network will be required to provide more security and 438 safety mechanisms. 440 4. Problem Statement 442 The addition of PLC virtualization capabilities impacts the PLC 443 device and the network elements in the infrastructure. Design 444 considerations must be made to ensure that such impacts facilitate 445 automation by simplifying configurations, improving operations and 446 management, and reducing process-change overheads. Nevertheless, it 447 is a change from the current state of the Industrial Networks. 449 This section describes the challenges, starting with brief 450 information on the current architecture to set the context. 452 4.1. Overview of Industrial Network Architecture 454 The physical network architecture for process control, as shown in 455 Figure 2 is rigidly hierarchical. Note that the figure is over- 456 simplified, and in general, each level will have additional 457 hierarchies to extend the networks for scale. For example, a PLC 458 controlling a group of Fieldbus devices may, in turn, be controlled 459 by another PLC controller [networked-PLC] that runs ProfiNet protocol 460 because both sets of devices are interdependent. For such cases, 461 protocol translation gateways are required. Several network switches 462 are needed to interconnect gateways and numerous devices on the 463 factory floor. 465 The hierarchical architecture comprises security-oriented zones known 466 as ICA-95 model (or Purdue model see Appendix A) in which each zone 467 contains well-defined levels. Among the three zones (Manufacturing, 468 IDMZ, and Enterprise), the enterprise zone network is all IP, while 469 the manufacturing and IDMZ network on the factory floor is a 470 combination of IP and Industrial protocols. The communication across 471 the zone tends to get complex as each zone runs over different 472 network technologies. A large number of IP-based firewalls and 473 translation gateways are deployed in all the zones to control data 474 movement between IT and OT networks. 476 Industry control systems (SCADA, HMI, MES) perform complex 477 operations. They collect data from devices and simultaneously 478 administer several process control loop instances to handle complex 479 processes. Traditional best practices indirectly required data 480 delivery from L2 to L3 levels in reports, which caused a significant 481 time lag. 483 +-+-+-+-+-+-+ External 484 ^ | Data Apps | business logic network 485 : +-+-+-+-+-+-+ (L5) 486 : | | 487 v +-+-+ +-+-+ Translation 488 |IDS|..|FW | gateways and firewalls 489 ^ +-+-+ +-+-+ -----+ (L4) 490 : | | 491 v +-+-+-+-+-+-+ +-+-+-+-+--+ 492 | vendor A | |vendor B | Interconnection 493 | controller| |controller| of controllers 494 ^ +-+-+-+-+-+-+ +-+-+-+-+-+- (L2-L3) 495 : | | 496 : +-+-+-+-+ +-+-+-+-+ 497 v | PLCs-X | |PLCs-Y |--+ Device-controllers 498 ^ +-+-+-+-+ +-+--+-+| (L1) 499 : | | | 500 : +--+ +-+ +-+ 501 v | | | | | | Field level devices 502 +--+ +-+ +-+ (L0) 504 Figure 2: Hierarchy of Functions Industrial Control Networks 506 4.2. Associating virtualized PLCs with IO Devices 508 A physical PLC is generally associated with a few I/O devices and is 509 directly connected. The I/O modules are not required to authenticate 510 or verify the connection. A virtualized PLC is a software instance; 511 it may now be anywhere in the network; therefore, the system must 512 authenticate the virtualized PLC and I/O device connection pairing. 513 This is necessary to maintain the reliability and safety of the 514 system and prevent unauthenticated PLC from interacting with the 515 software. The association must be done under the constraint that I/O 516 modules are basic devices without any compute capability. Thus, the 517 network should provide these functions through gateways or 518 interconnecting devices. 520 4.3. Expectations from the Networks 522 The magnitude by which compute capability is improved allows a single 523 virtualized controller to handle more complex and faster scan cycles. 524 Then, the network to manage communication delays, packet formation, 525 processing, and forwarding overheads become critical to overall 526 system performance. Harnessing compute power at a lower cost from 527 edge-compute platforms is expected for several reasons. It is 528 anticipated that edge-networks will offer general purpose compute and 529 store capabilities for latency-sensitive applications. This piece of 530 infrastructure can serve many sites and needed not be owned but can 531 be leased, providing cloud-like services.It is a big change from the 532 traditional Purdue model or ICA architecture. 534 Thus, the plant-floor networks are now extended to edge networks 535 expanding the security zones creating 'new' requirements for multi- 536 tenancy support (isolation and network segmentation) in OT networks. 537 Note that in IT networks, these technologies are mature and already 538 standardized. 540 4.3.1. Hierarchical Structure 542 Virtualized PLCs and their flexible placement require flat structure 543 so that flow of information is context based and need not follow 544 strict hierarchy. Hierarchical flow of information is not always 545 efficient and is centralized. It does not inherently support 546 autonomous decision making which is central to Industry 4.0 type of 547 initiatives. In contrast, a distributed architecture with some form 548 of centralized view will be ideal since it combines both autonomous 549 operations and global view. 551 4.3.2. Safety and Reliability of Operations 553 The Fieldbus modules and PLCs are designed to perform for long period 554 of times. The commands or operations dispatched from virtual PLCs 555 must conform to same safety standards. Similarly, the communication 556 between PLC control unit and I/O module is highly reliable and such 557 data losses must be prevented. 559 4.4. Multiprotocol Supporting PLCs 561 A virtualized PLC can act as a single logical controller to 562 communicate with a different group of I/O devices over one or more 563 non-internet protocols such as Modbus, Profibus, CANbus, Profinet 564 [SURV], etc. Since each protocol specifies its packet format, 565 different translation gateways are generally needed. Thus, a multi- 566 protocol virtual PLC can reduce the number of gateways. 568 However, the challenge is to provide a standard communication format 569 for different I/O devices. Since it is not feasible to have a single 570 flat Fieldbus Fieldbus protocol due to address scale limitations 571 (limited address space up to 256 devices), an I/O interconnect is 572 required to perform format translation. Then the packet on the wire 573 should be multi-protocol aware. i.e., virtualized PLC needs to know 574 what type of Fieldbus device it is communicating with at the other 575 end. 577 4.5. Identification of virtualized PLC 579 The Fieldbus devices are serial buses and identify PLC as a device 580 with a specific bus address. It may be required for virtualized PLC 581 to support dual addresses, one exposed for the I/O module and the 582 other for IT applications. Converged IT/OT networks should leverage 583 specifics of factory floors designs and assign device ids based on 584 machine locations and context. As an example, a device with basic 585 address 0x14 may be defined as 'device 0x14, cell 'C1' and factory 586 floor 'F1', PLC bus address '0x1' in the communication path. The 587 reachability to a specific I/O module should have complete 588 information from virtualized PLC. 590 4.6. Security Aspects 592 The fundamental paradigm of security as described in ICA-95 593 architecture changes with virtualized PLC since those PLCs won't be 594 in the local manufacturing zone. The zone-aware security will not 595 apply. 597 Instead, the system will need a multi-dimensional security profile. 598 The first one encompasses both enterprise and manufacturing zones, 599 and the second is location-specific, i.e., using secure channels such 600 as VPN, IPSEC, etc. 602 5. Requirements 604 5.1. Virtualized PLC Requirements 606 A virtualized PLC's function and operation should be identical to 607 that of physical PLC. The following requirements relate to 608 virtualized PLC's reachability, identification, and discovery (or 609 attachment) in the network. 611 * Addresses scope 613 The virtualized PLC is expected to be an IP-addressed endpoint when 614 communicating with higher-level applications. However, southbound 615 communication may require some structured addressing scheme to reach 616 the Fieldbus device in the network (e.g., see [semantic-addressing] 617 and [asymmetric-addr]). There is no need to enforce IP addresses for 618 Fieldbus devices since they are constrained devices, and IP may not 619 be the most suitable address structure. A uniquely reachable address 620 space for all the Fieldbus I/O devices and PLCs is required such that 621 intermediate network elements know how to route (or switch) to those 622 addresses. Moreover, as the scale of the industry network grows, 623 there will be many 'same' types of devices with limited address space 624 (a Fieldbus or ModBus address limits up to 256) all across the floor. 625 It is maybe desirable to support variable-length identifiers to 626 handle both IT servers and I/O module-type devices. 628 * Converged Namespace 630 Addresses are resolved from namespaces. It should be possible to 631 associate all the endpoints (OT and IT) as part of their system- 632 defined namespace. The solution should not require different 633 operations and management schemes for industry I/O modules vs. IT 634 applications. It will improve security by verifying an endpoint 635 against a namespace. However, each vertical sector should be able to 636 choose its namespace. For example, In some cases, the classification 637 may be based on a level (PLCs, cell sites, type of application, 638 etc.), and the corresponding address is derived by concatenating them 639 together since factory devices do not change their location often in 640 the topology. 642 * Network Identifiers: 644 Virtualized PLC should be identifiable by what application it can 645 talk to or the service they are part of [semantic-addressing]. The 646 network identification is required for setting up security or 647 firewall policies. Note: legacy devices do not have network 648 identifiers, and deeper packet inspection will be required to 649 identify a specific PLC. Alternately [semantic-addressing] may be 650 useful in structuring the identifiers. 652 * Legacy support: 654 Virtualized PLCs and legacy PLCs must co-exist with support for 655 deployed protocol formats and their core capabilities. This is 656 needed to maintain non-disruptive operations. 658 * Auto-configuration: 660 Procedures should be efficient, i.e., comparable to the processing 661 capabilities of the I/O devices. On-boarding procedures (manual or 662 automatic) must have built-in or well-defined authentication. 664 * Controller and Fieldbus Pairing: 666 Virtualized PLCs must support a secure method of pairing 667 authenticating with their I/O devices. Virtualization allows 668 multiple PLCs to control (or at least monitor) the same device. This 669 can potentially lead to conflicts in device operation. Therefore, 670 careful access control mechanisms are required to prioritize 671 operation across the PLCs. 673 * Efficient Transport Protocol 675 Currently, factory-floor Fieldbus devices do not directly use any 676 transport protocols designed for the purpose, e.g., [MQTT_SPEC] and 677 [OPC_ARCH]. The data collected from sensors is encapsulated in TCP. 678 Alternate native transport based on principles of MQTT type of 679 protocols could help to improve the traffic efficiency in industrial 680 networks. 682 5.2. Key Performance Indicator Requirements 684 * Process Control 686 Performance depends on the deterministic behavior of devices. A 687 virtualized PLC must maintain all deterministic and low latency 688 attributes of physical PLC. 690 * Safety mechanisms 692 To keep a factory floor hazard and accident-free environment, the 693 virtualized PLC must implement mechanisms for proper operation of a 694 device, including commands sent from virtualized PLC that must not 695 exceed thresholds and are error-free and valid for the Fieldbus 696 operation. 698 * Deterministic or Time Sensitive Service Guarantees 700 Mechanisms should be implemented to assure time-sensitive delivery of 701 traffic. For this, [DETNET] or TSN technologies can be used. 703 * Security 705 Mechanisms should be implemented to protect against man-in-the-middle 706 attacks. Encryption overheads must be budgeted from virtualized PLC 707 to Fieldbus to maintain process control latency. Due to low 708 processing power, lightweight mechanisms should be devised. 710 5.3. Network Related Requirements 712 The topologies in the manufacturing zones do not change frequently, 713 and devices are designated in a zone or a cell for long-term use. 714 Such observations can help simplify network designs. Industry 715 networks could substantially benefit from a hybrid software-defined 716 networking and distributed routing approach. Former for initial 717 provisioning (or controlled bootstrapping), latter for reachability 718 and health of the fabric. Such hybrid techniques eliminate the need 719 for implementing complex routing protocol features. 721 * Backward Compatibility 723 Seamless integration of virtualized PLCs must be supported. The 724 network must support legacy traffic, and its performance should be no 725 worse than before the inclusion of virtualized PLCs. 727 * Efficiency of connections 729 Industrial networks have different connection endpoints, such as PLC- 730 PLC, PLC-SCADA, SCADA-IT-Systems, PLC-Firewalls, PLC-gateways, PLC-I/ 731 O modules. Without subscribing to a specific wire format, a flexible 732 packet format should be designed to address smooth connections 733 between any of the above endpoints. It implies that a variety of 734 endpoints interconnect in an identical fashion without requiring 735 device-specific translations. Efficient connections lead to less 736 processing or states in the network with improved resiliency and 737 performance. There may be opportunities to design packet formats 738 with minimal overheads by using in-band programmability paradigms 739 that carry embedded metadata and control information relating to 740 reachability, latency, jitter, reliability, and exceptions 741 characteristics. This approach is expected to reduce configurations 742 and the number of policies required for data steering through the 743 network. Existing methods that may be used, evaluated or extended 744 include IP with TSN, DETNET[DETNET], reachability headers SCHC, IPv6 745 compression schemes, or may be evaluated against newer schemes. 747 * Traffic segmentation support 749 As virtualized PLCs are spun off like VMs, connectivity with fieldbus 750 devices will be affected. It should not have adverse effect on 751 deterministic, low latency behavior on the other segmented traffic 752 (i.e., connectivity between another set of endpoints). Each 753 segmented traffic may be associated with a different protocol or 754 traffic profile, including legacy traffic format and profiles. The 755 methods to support segmentation include virtual network technologies 756 inside the fabric such as VxLAN, VPNs, etc. 758 * Resilient and Extensible Topologies 760 The industry network protocols must not limit to a constrained 761 physical topology. It must support a multi-path distributed 762 connectivity framework to prevent bottlenecks traffic concentration. 764 * Dynamic Bandwidth Management 766 Even industrial networks generate a high volume of data from the 767 sensors. Managing bandwidth for different types of data 768 (operational, control, statistics) should be supported through 769 existing QoS or in-band monitoring technologies. 771 6. IANA Considerations 773 This document requires no actions from IANA. 775 7. Security Considerations 777 The architecture at the very least must adhere to the security 778 guidance provided by ICS-95. 780 8. Acknowledgements 782 9. Informative References 784 [asymmetric-addr] 785 Makhijani, K. and L. Dong, "Requirements and Scenarios for 786 Industry Internet Addressing", Work in Progress, Internet- 787 Draft, draft-km-industrial-internet-requirements-00, 10 788 June 2021, . 791 [DETNET] Finn, N., Thubert, P., Varga, B., and J. Farkas, 792 "Deterministic Networking Architecture", RFC 8655, 793 DOI 10.17487/RFC8655, October 2019, 794 . 796 [FPGA_PLC] Huabing, Z., Benlei, L., Bolin, D., and F. Xiao, "Research 797 on FPGA-based Programmable Logic Controllers’ Technology", 798 TELKOMNIKA Indonesian Journal of Electrical 799 Engineering Vol. 11, DOI 10.11591/telkomnika.v11i12.3701, 800 December 2013, 801 . 803 [IIC] "Industry IoT Consortium", n.d., 804 . 806 [IIC_TALK] William Diab, W., "Overview of IIC – Building the IIoT 807 Ecosystem", 12 October 2021, . 811 [ISA95] "ANSI/ISA-95.00.01-2010 (IEC 62264-1 Mod) Enterprise- 812 Control System Integration - Part 1: Models and 813 Terminology", n.d., . 817 [MQTT_SPEC] 818 "MQTT Version 3.1.1 Plus Errata 01", December 2015, 819 . 822 [networked-PLC] 823 "Should PLCs be networked?", 4 October 2004, 824 . 827 [OPC] "Open Platform Communications", n.d., 828 . 830 [OPC_ARCH] "OPC 10000-1 - Part 1: Overview and Concepts", 2 November 831 2017, . 835 [OPC_INFO] "OPC-UA Information Model Specifications", n.d., 836 . 839 [PLC-40] Azarmipour, M., Elfaham, H., Gries, C., and U. Epple, "PLC 840 4.0: A Control System for Industry 4.0", IECON 2019 - 45th 841 Annual Conference of the IEEE Industrial 842 Electronics Society, DOI 10.1109/iecon.2019.8927026, 843 October 2019, 844 . 846 [RECONF] Koren, Y., "Reconfigurable Manufacturing System", CIRP 847 Encyclopedia of Production Engineering pp. 1417-1423, 848 DOI 10.1007/978-3-662-53120-4_6629, 2019, 849 . 851 [semantic-addressing] 852 Jia, Y., Trossen, D., Iannone, L., Shenoy, N., and P. 853 Mendes, "Gap Analysis in Internet Addressing", Work in 854 Progress, Internet-Draft, draft-jia-intarea-internet- 855 addressing-gap-analysis-01, 23 October 2021, 856 . 859 [SURV] Galloway, B. and G. Hancke, "Introduction to Industrial 860 Control Networks", IEEE Communications Surveys & 861 Tutorials Vol. 15, pp. 860-880, 862 DOI 10.1109/surv.2012.071812.00124, 2013, 863 . 865 [VPLC-DRAGOS] 866 Scott, A., "Programmable Logic Controller Virtualization", 867 8 February 2019, . 870 [VPLC_CONV] 871 Cruz, T., Simoes, P., and E. Monteiro, "Virtualizing 872 Programmable Logic Controllers: Toward a Convergent 873 Approach", IEEE Embedded Systems Letters Vol. 8, pp. 874 69-72, DOI 10.1109/les.2016.2608418, December 2016, 875 . 877 [VPLC_IIC] Lou, D., Graf, U., and M. Tseng, "Virtualized Programmable 878 Logic Controllers. An Industrial Internet Consortium Tech 879 Brief", 7 September 2021, 880 . 883 Appendix A. Appendix A. Purdue Model (ICA-95) 885 The International Society of Automation (ICA) has developed a model 886 [ISA95] to describe automated interfaces between enterprise and 887 control systems. In this widely deployed hierarchical model, five 888 levels are defined and they follow a strict ordering of interfaces 889 across the levels. At the lowest level 0, are the physical devices 890 while enterprise applications are at level 5. In between these two 891 levels, there are several supervisory, management, and intermediate 892 data collection applications that provide information to 893 | +-------------------------------+ Enterprise 894 | L5 | Enterprise applications | Security 895 +-- +-------------------------------+ Zone 896 | +-------------------------------+ 897 | L4 | Gateways, servers (ops, mgmt) | IDMZ 898 +-- +-------------------------------+ 899 | +-------------------------------+ 900 | L3 | Supervisory controls | Industry 901 | +-------------------------------+ Security 902 | L1 | Device control | Zone 903 | +-------------------------------+ 904 | L0 |Sensors, Actuators, Robots, etc| (cells or zones) 905 +-- +-------------------------------+ 907 Figure 3: ISA 95 or Purdue model of Automation Pyramid 909 A.1. Separation between Manufacturing and Enterprise Networks 911 The ICA-95 architecture recommends hierarchy, thereby a separation 912 between factory devices and applications through three different 913 security zones called Manufacturing, DMZ and enterprise zones as 914 shown in Figure 3 as below: 916 * Enterprise Security Zone: The IT applications reside in 917 enterprise networks and perform tasks necessary for business 918 operations such as inventory control, supply-chain logistics, 919 schedule and capacity planning. They need to collect data from 920 the OT systems in order to make those decisions. 922 * Industrial Demilitarized Zone: The OT and IT networks were 923 designed to prevent direct communication between them. The 924 IDMZ serves as an information sharing layer between the IT and 925 OT (L4 and L3) systems. This indicates that additional 926 security rules, inspection and protection of device identity 927 and access is necessary when transiting from L3 to L4. 929 * Manufacturing Zone: Consists of Levels 0 through 3 site wide 930 production system. Operations at level 3 (L3) Support site- 931 wide view of the production system. They also provide data to 932 L4. Area supervisory control (L2) performs operation and 933 control over a zone or smaller area in a production floor. 934 Each area has specific set of tasks or operations to perform. 935 Basic control at level 1 (L1) is for the actual control of the 936 equipment. The L1 components such include PLCs; they send 937 commands to L0 equipments to perform tasks (e.g. start motor, 938 alter pressure level, or reduce motor speed). Finally, actual 939 process takes place at level 0 (L0). At this level for the 940 process equipments performing actual operations are performed. 941 This include equipment and devices such as motors, pressure 942 valves, temperature, speed, etc sensors, etc. 944 The devices or controllers at level 1 are the ones of specific 945 interest for virtualization and the corresponding challenges are 946 covered in later section. 948 A.2. Collaborating with SDOs with Industry Network Focus 950 The paradigms of networking in OT are quite different than IP based 951 best-effort networking protocols. Yet, IETF protocols are 952 extensively used in OT applications. Often, it is not possible to 953 get contributors directly from the OT sectors, then it would make 954 more sense to coordinate with well-established consortia where OT 955 scenarios and requirements are is discussed may be utilized. Two 956 well established foundations are IIC [IIC] and OPC-UA [OPC]. For 957 example, a [IIC_TALK] provided overview of IIC activities. 959 Industrial IoT Consortium (IIC) provides use cases, scenarios, and 960 best-practice frameworks to solve specific problems and solution pain 961 points. It is a rich resources of case studies and demonstrations of 962 different test beds. The IIC itself is not involved in standards 963 development, but may help in formalizing requirements, further 964 insights into solutions developed in IETF, and potentially help 965 adoption of those solutions. 967 Open Platform Communications-Unified Architecture (OPC-UA) provides 968 interoperability across different hardware platforms using a standard 969 data model. It standardizes various information models, 970 corresponding client-server architecture and defines necessary access 971 mechanisms to those information models. The OPC-UA is an abstraction 972 layer to provide common interface to different data look-up and event 973 notifications. A number of information models are provided by OPC-UA 974 can be found here [OPC_INFO]. For example, OPC has a specification 975 on PLCs. It abstracts PLC specific protocols (such as Modbus, 976 Profibus, etc.) into a standardized interface allowing HMI/SCADA 977 systems to interface with a middleware that converts generic-OPC 978 read/write requests into device-specific requests and vice-versa. 980 Note: OPC-UA information model similar to YANG? 982 IETF solutions will focus on leveraging or extending IETF 983 technologies for IT and OT integration which is at the infrastructure 984 or communication layer. Thus, providing protocols that could 985 potentially benefit higher-level OPC-UA work. 987 Both IIC and OPC could provide guidance for the standards work. 989 Authors' Addresses 991 Kiran Makhijani 992 Futurewei 993 Santa Clara, CA 95050, 994 United States of America 995 Email: kiran.ietf@gmail.com 997 Lijun Dong 998 Futurewei 999 Santa Clara, CA 95050, 1000 United States of America 1001 Email: lijun.dong@futurewei.com