detnet P. Wetterwald Internet-Draft Cisco Intended status: Informational J. Raymond Expires: April 24, 2015 Hydro-Quebec October 23, 2014 Deterministic Networking Utilities requirements draft-wetterwald-detnet-utilities-reqs-00 Abstract This paper documents the needs in Smart Grid industry to establish multi-hop paths for characterized flows with deterministic properties . Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on April 24, 2015. Copyright Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 2 3. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 4. Communication Trends and General Communication Requirements . 3 Wetterwald & Raymond Expires April 24, 2015 [Page 1] Internet-DrafDeterministic Networking Utilities requirement October 2014 4.1. General communication Requirements . . . . . . . . . . . . 3 4.1.1. Migration to Packet-Switched Network . . . . . . . . . 4 4.2. Applications, Use cases and traffic patterns . . . . . . . 5 4.2.1. Transmission use cases . . . . . . . . . . . . . . . . 5 4.2.1.1. Tele Protection . . . . . . . . . . . . . . . . . 6 4.2.1.1.1. Latency Budget Consideration . . . . . . . . . 6 4.2.1.1.2. Asymetric delay . . . . . . . . . . . . . . . 7 4.2.1.1.2.1. Other traffic caracteristics . . . . . . . 7 4.2.1.1.2.2. Teleprotection network requirements . . . 8 4.2.1.2. Inter-Trip Protection scheme . . . . . . . . . . . 8 4.2.1.3. Current Differential Protection Scheme . . . . . . 9 4.2.1.4. Distance Protection Scheme . . . . . . . . . . . . 10 4.2.1.5. Inter-Substation Protection Signaling . . . . . . 11 4.2.1.6. Intra-Substation Process Bus Communication . . . . 11 4.2.1.7. Wide Area Monitoring and Control Systems . . . . . 12 4.2.2. Distribution use case . . . . . . . . . . . . . . . . 13 4.2.2.1. Fault Location Isolation and Service Restoration (FLISR) . . . . . . . . . . . . . . . . . . . . . 13 4.2.3. Generation use case . . . . . . . . . . . . . . . . . 15 4.2.3.1. Frequency Control / Automatic Generation Control (AGC) . . . . . . . . . . . . . . . . . . . . . . 15 4.3. Specific Network topologies of Smart Grid Applications . . 16 4.3.1. Precision Time Protocol . . . . . . . . . . . . . . . 17 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 6.1. Current Practices and Their Limitations . . . . . . . . . 18 6.2. Security Trends in Utility Networks . . . . . . . . . . . 19 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 8.1. Normative References . . . . . . . . . . . . . . . . . . . 21 8.2. Informative References . . . . . . . . . . . . . . . . . . 21 Appendix A. Additional Stuff . . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 21 1. Introduction [I-D.finn-detnet-problem-statement] discusses blah 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 3. Overview Evolution of Utility Telecom Networks Wetterwald & Raymond Expires April 24, 2015 [Page 2] Internet-DrafDeterministic Networking Utilities requirement October 2014 The business and technology trends that are sweeping the utility industry will drastically transform the utility business from the way it has been for many decades. At the core of many of these changes is a drive to modernize the electrical grid with an integrated communications infrastructure. However, interoperability, concerns, legacy networks, disparate tools, and stringent security requirements all add complexity to grid transformation. Given the range and diversity of the requirement that should be addressed by the next generation telecommunications infrastructure utilities need to adopt a holistic architectural approach to integrate the electrical grid with digital communication across the entire power delivery chain. Many utilities still rely on complex environments formed of multiple application-specific, proprietary networks. Information is siloed between operational areas. This prevents utility operations from realizing the operational efficiency benefits, visibility, and functional integration of operational information across grid applications and data networks. The key to modernizing grid communications is to provide a common, multi-service network infrastructure for the entire utility organization. Such a network serves as the platform for current capabilities while enabling future expansion of the network to accommodate new applications and services. To meet this diverse set of requirements, both today and in the future, the next generation utility telecom network will be based on open-standards-based IP architecture. An end-to-end IP architecture takes advantage of nearly three decades of IP technology development, facilitating interoperability across disparate networks and devices, as it has been already demonstrated in many mission-critical and highly secure networks. IEC and the different National Committees have mandated a specific adhoc group (AHG8) to define the migration strategy to IPv6 for all the IEC TC57 power automation standards. IPv6 is seen as the obvious future communication technology for the Smart Grid. The Adhoc Group will discose to the IEC coordination group, their conclusions end of 2014. It is imperative that utilities participate in standards development bodies to influence the development of future solutions and to benefit from shared experiences of other utilities and vendors. 4. Communication Trends and General Communication Requirements These general communication requirements are over and above the specific requirements of the use cases that have been addressed so far. These include both current and future communication related requirements that should be factored into the network architecture and design. 4.1. General communication Requirements Wetterwald & Raymond Expires April 24, 2015 [Page 3] Internet-DrafDeterministic Networking Utilities requirement October 2014 o IP Connectivity everywhere o Monitoring services everywhere and from different remote centers o Move services to a virtual data center o Unify access to applications / information from the corporate network o Unify services o Unified Communications Solutions o Mix of fiber and microwave technologies - obsolescence of SONET or TDM o Standardize grid communication protocol to opened standard to ensure interoperability o Reliable Communications for Transmission and Distribution Substations o IEEE 1588 time synchronization Client / Server Capabilities o Integration of Multicast Design o QoS Requirements Mapping o Enable Future Network Expansion o Substation Network Resilience o Fast Convergence Design o Scalable Headend Design o Define Service Level Agreements (SLA) and Enable SLA Monitoring o Integration of 3G/4G Technologies and future technologies o Ethernet Connectivity for Station Bus Architecture o Ethernet Connectivity for Process Bus Architecture o Protection and teleprotection on IP 4.1.1. Migration to Packet-Switched Network Wetterwald & Raymond Expires April 24, 2015 [Page 4] Internet-DrafDeterministic Networking Utilities requirement October 2014 Throughout the world, utilities are increasingly planning for a future based on smart grid applications requiring advanced telecommunications systems. Many of these applications utilize packet connectivity for communicating information and control signals across the utility's Wide Area Network (WAN), made possible by technologies such as multiprotocol label switching (MPLS). The data that traverses the utility WAN includes: o Grid monitoring, control, and protection data o Non-control grid data (e.g. asset data for condition-based monitoring) o Physical safety and security data (e.g. voice and video) o Remote worker access to corporate applications (voice, maps, schematics, etc.) o Field area network backhaul for smart metering, and distribution grid management o Enterprise traffic (email, collaboration tools, business applications) WANs support this wide variety of traffic to and from substations, the transmission and distribution grid, generation sites, between control centers, and between work locations and data centers. To maintain this rapidly expanding set of applications, many utilities are taking steps to evolve present time-division multiplexing (TDM)based and frame relay infrastructures to packet systems. Packet-based networks are designed to provide greater functionalities and higher levels of service for applications, while continuing to deliver reliability and deterministic (real-time) traffic support. 4.2. Applications, Use cases and traffic patterns Among the numerous applications and use cases that a utility deploys today, many rely on high availability and deterministic behaviour of the telecommunication networks. Protection use cases and generation control are the most demanding and can't rely on a best effort approach. 4.2.1. Transmission use cases Protection means not only the protection of the human operator but also the protection of the electric equipments and the preservation of the stability and frequency of the grid. If a default occurs on the transmission or the distribution of the electricity, important damages could occured to the human operator but also to very costly electrical equipments and perturb the grid leading to blackouts. The Wetterwald & Raymond Expires April 24, 2015 [Page 5] Internet-DrafDeterministic Networking Utilities requirement October 2014 time and reliability requirements are very strong to avoid dramatic impacts to the electrical infrastructure. 4.2.1.1. Tele Protection The key criteria for measuring Teleprotection performance are command transmission time, dependability and security. These criteria are defined by the IEC standard 60834 as follows: o Transmission time (Speed): The time between the moment where state changes at the transmitter input and the moment of the corresponding change at the receiver output, including propagation time. Overall operating time for a Teleprotection system includes the time for initiating the command at the transmitting end, the propagation time over the communications link and the selection and decision time at the receiving end, including any additional delay due to a noisy environment. o Dependability: The ability to issue and receive valid commands in the presence of interference and/or noise, by minimizing the probability of missing command (PMC). Dependability targets are typically set for a specific bit error rate (BER) level. o Security: The ability to prevent false tripping due to a noisy environment, by minimizing the probability of unwanted commands (PUC). Security targets are also set for a specific bit error rate (BER) level. Additional key elements that may impact Teleprotection performance include bandwidth rate of the Teleprotection system and its resiliency or failure recovery capacity. Transmission time, bandwidth utilization and resiliency are directly linked to the communications equipment and the connections that are used to transfer the commands between relays. 4.2.1.1.1. Latency Budget Consideration Delay requirements for utility networks may vary depending upon a number of parameters, such as the specific protection equipment used. Most power line equipment can tolerate short circuits or faults for up to approximately five power cycles before sustaining irreversible damage or affecting other segments in the network. This translates to total fault clearance time of 100ms. As a safety precaution, Wetterwald & Raymond Expires April 24, 2015 [Page 6] Internet-DrafDeterministic Networking Utilities requirement October 2014 however, actual operation time of protection systems is limited to 70- 80 percent of this period, including fault recognition time, command transmission time and line breaker switching time. Some system components, such as large electromechanical switches, require particularly long time to operate and take up the majority of the total clearance time, leaving only a 10ms window for the communications part of the protection scheme, independent of the distance to travel. Given the sensitivity of the issue, new networks impose requirements that are even more stringent: IEC standard 61850 limits the transfer time for protection messages to 1/4 - 1/2 cycle or 4 - 8ms (for 60Hz lines) for the most critical messages The following diagram shows the latency budget for a fault clearing time of a protection system, divided by the different actors involved. 4.2.1.1.2. Asymetric delay In addition to minimal transmission delay, a differential protection communication channel must be synchronous, i.e., experiencing symmetrical channel delay in transmit and receive paths. This requires special attention in jitter-prone packet networks. While optimally Teleprotection systems should support zero asymmetric delay, typical relays can tolerate discrepancies of up to 750us. The main tools available for lowering delay variation below this threshold are: o A jitter buffer at the multiplexers on each end of the line can be used to offset delay variation by queuing sent and received packets. The length of the queues must balance the need to regulate the rate of transmission with the need to limit overall delay, as larger buffers result in increased latency. This is the old TDM traditional way to fulfill this requirement. o Traffic management tools ensure that the Teleprotection signals receive the highest transmission priority and minimize the number of jitter addition during the path. This is one way to meet the requirement in IP networks. o Standard Packet-Based synchronization technologies, such as 1588-2008 Precision Time Protocol (PTP) and Synchronous Ethernet (Sync-E), can help maintain stable networks by keeping a highly accurate clock source on the different network devices involved. 4.2.1.1.2.1. Other traffic caracteristics o Redundancy: The existence in a system of more than one means of accomplishing a given function o Recovery time : it's the duration of time within which a business process must be restored after any type of disruption in order to avoid unacceptable consequences associated with a break in Wetterwald & Raymond Expires April 24, 2015 [Page 7] Internet-DrafDeterministic Networking Utilities requirement October 2014 business continuity. o performance management : In networking, a management function defined for controlling and analyzing different parameters/metrics such as the throughput, error rate o packet loss : one or more packets of data travelling across network fail to reach their destination 4.2.1.1.2.2. Teleprotection network requirements The following table captures the main network requirements (this is based on IEC 61850 standard) +---------------------------------+---------------------------------+ | Teleprotection Requirement | Attribute | +---------------------------------+---------------------------------+ | One way maximum delay | 4-10 ms | | Asymetric delay required | Yes | | Maximum jitter | less than 250 us | | Topology | Point to point, point to Multi- | | | point | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% to 1% | +---------------------------------+---------------------------------+ 4.2.1.2. Inter-Trip Protection scheme Inter-tripping is the controlled tripping of a circuit breaker to complete the isolation of a circuit or piece of apparatus in concert with the tripping of other circuit breakers. The main use of such schemes is to ensure that protection at both ends of a faulted circuit will operate to isolate the equipment concerned. Inter- tripping schemes use signaling to convey a trip command to remote circuit breakers to isolate circuits. Wetterwald & Raymond Expires April 24, 2015 [Page 8] Internet-DrafDeterministic Networking Utilities requirement October 2014 +---------------------------------+---------------------------------+ | Inter-Trip protection | Attribute | | Requirement | | +---------------------------------+---------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to Multi- | | | point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +---------------------------------+---------------------------------+ 4.2.1.3. Current Differential Protection Scheme Current differential protection is commonly used for line protection, and is typical for protecting parallel circuits. A main advantage for differential protection is that, compared to overcurrent protection, it allows only the faulted circuit to be de-energized in case of a fault. At both end of the lines, the current is measured by the differential relays, and based on Kirchhoff's law, both relays will trip the circuit breaker if the current going into the line does not equal the current going out of the line. This type of protection scheme assumes some form of communication being present between the relays at both end of the line, to allow both relays to compare measured current values. A fault in line 1, will cause overcurrent to be flowing in both lines, but because the current in line 2 is a through following current, this current is measured equal at both ends of the line, therefore the differential relays on line 2 will not trip line 2. Line 1 will be tripped, as the relays will not measure the same currents at both ends of the line. Line differential protection schemes assume a very low communications delay between both relays, often as low as 5ms. Moreover, as those systems are often not time-synchronized, they also assume symmetric communications paths with constant delay, which allows comparing current measurement values taken at the exact same time. Wetterwald & Raymond Expires April 24, 2015 [Page 9] Internet-DrafDeterministic Networking Utilities requirement October 2014 +------------------------------------+------------------------------+ | Current Differential protection | Attribute | | Requirement | | +------------------------------------+------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | Yes | | Maximum jitter | less than 250 us | | Topology | Point to point, point to | | | Multi-point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +------------------------------------+------------------------------+ 4.2.1.4. Distance Protection Scheme Distance (Impedance Relay) protection scheme is based on voltage and current measurements. A fault on a circuit will generally create a sag in the voltage level. If the ratio of voltage to current measured at the protection relay terminals, which equates to an impedance element, falls within a set threshold the circuit breaker will operate. The operating characteristics of this protection are based on the line characteristics. This means that when a fault appears on the line, the impedance setting in the relay is compared to the apparent impedance of the line from the relay terminals to the fault. If the relay setting is determined to be below the apparent impedance it is determined that the fault is within the zone of protection. When the transmission line length is under a minimum length distance protection becomes more difficult to coordinate. In these instances the best choice of protection is current differential protection. +---------------------------------+---------------------------------+ | Distance protection Requirement | Attribute | +---------------------------------+---------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to Multi- | | | point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | Wetterwald & Raymond Expires April 24, 2015 [Page 10] Internet-DrafDeterministic Networking Utilities requirement October 2014 | Redundancy | Yes | | Packet loss | 0.1% | +---------------------------------+---------------------------------+ 4.2.1.5. Inter-Substation Protection Signaling This use case describes the exchange of Sampled Value or GOOSE message between IEDs in two substations for protection and tripping coordination. The two IEDs are in a master-slave mode. The CT/VT in one substation sends the sampled analog voltage or current value to the Merging Unit (MU) over hard wire. The merging unit sends the time-synchronized 61850-9-2 sampled values to the slave IED. The slave IED forwards the information to the Master IED in the other substation. The master IED makes the determination (for example based on sampled value differentials) to send a trip command to the originating IED. Once the slave IED/Relay receives the GOOSE trip for breaker tripping, it opens the breaker. It then sends a confirmation message back to the master. All data exchanges between IEDs are either through Sampled Value or GOOSE messages. +------------------------------------+------------------------------+ | Inter-Substation protection | Attribute | | Requirement | | +------------------------------------+------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to | | | Multi-point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 1% | +------------------------------------+------------------------------+ 4.2.1.6. Intra-Substation Process Bus Communication This use case describes the data flow from the CT/VT to the IEDs in the substation via the merging unit (MU). The CT/VT in the substation send the sampled value (analog voltage or current) to the Merging Unit (MU) over hard wire. The merging unit sends the time- synchronized 61850-9-2 sampled values to the IEDs in the substation in GOOSE message format. The GPS Master Clock can send 1PPS or IRIG-B format to MU through serial port, or IEEE 1588 protocol via Wetterwald & Raymond Expires April 24, 2015 [Page 11] Internet-DrafDeterministic Networking Utilities requirement October 2014 network. Process bus communication using 61850 simplifies connectivity within the substation and removes the requirement for multiple serial connections and removes the slow serial bus architectures that are typically used. This also ensures increased flexibility and increased speed with the use of multicast messaging between multiple devices. +------------------------------------+------------------------------+ | Intra-Substation protection | Attribute | | Requirement | | +------------------------------------+------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to | | | Multi-point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on Node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes - No | | Packet loss | 0.1% | +------------------------------------+------------------------------+ 4.2.1.7. Wide Area Monitoring and Control Systems The application of synchrophasor measurement data from Phasor Measurement Units (PMU) to Wide Area Monitoring and Control Systems promises to provide important new capabilities for improving system stability. Access to PMU data enables more timely situational awareness over larger portions of the grid than what has been possible historically with normal SCADA data. Handling the volume and real-time nature of synchrophasor data presents unique challenges for existing application architectures. Wide Area management System (WAMS) makes it possible for the condition of the bulk power system to be observed and understood in real-time so that protective, preventative, or corrective action can be taken. Because of the very high sampling rate of measurements and the strict requirement for time synchronization of the samples, WAMS has stringent communication requirements in an IP network that are captured in the following table: Wetterwald & Raymond Expires April 24, 2015 [Page 12] Internet-DrafDeterministic Networking Utilities requirement October 2014 +--------------------------+----------------------------------------+ | WAMS Requirement | Attribute | +--------------------------+----------------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to Multi-point, | | | Multi-point to Multi-point | | Bandwidth | 100 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on Node | less than 50ms - hitless | | failure | | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 1% | +--------------------------+----------------------------------------+ 4.2.2. Distribution use case 4.2.2.1. Fault Location Isolation and Service Restoration (FLISR) As the name implies, Fault Location, Isolation, and Service Restoration (FLISR) refers to the ability to automatically locate the fault, isolate the fault, and restore service in the distribution network. It is a self-healing feature whose purpose is to minimize the impact of faults by serving portions of the loads on the affected circuit by switching to other circuits. It reduces the number of customers that experience a sustained power outage by reconfiguring distribution circuits. This will likely be the first wide spread application of distributed intelligence in the grid. Secondary substations can be connected to multiple primary substations. Normally, static power switch statuses (open/closed) in the network dictate the power flow to secondary substations. Reconfiguring the network in the event of a fault is typically done manually on site to operate switchgear to energize/de-energize alternate paths. Automating the operation of substation switchgear allows the utility to have a more dynamic network where the flow of power can be altered under fault conditions but also during times of peak load. It allows the utility to shift peak loads around the network. Or, to be more precise, alters the configuration of the network to move loads between different primary substations. The FLISR capability can be enabled in two modes: o Managed centrally from DMS, or o Executed locally through distributed control via intelligent switches and fault sensors. There are 3 distinct sub-functions that are performed: Wetterwald & Raymond Expires April 24, 2015 [Page 13] Internet-DrafDeterministic Networking Utilities requirement October 2014 1. Fault Location Identification This sub-function is initiated by SCADA inputs, such as lockouts, fault indications/location, and, also, by input from the Outage Management System (OMS), and in the future by inputs from fault-predicting devices. It determines the specific protective device, which has cleared the sustained fault, identifies the de-energized sections, and estimates the probable location of the actual or the expected fault. It distinguishes faults cleared by controllable protective devices from those cleared by fuses, and identifies momentary outages and inrush/cold load pick- up currents. This step is also referred to as Fault Detection Classification and Location (FDCL). This step helps to expedite the restoration of faulted sections through fast fault location identification and improved diagnostic information available for crew dispatch. Also provides visualization of fault information to design and implement a switching plan to isolate the fault. 2. Fault Type Determination I. Indicates faults cleared by controllable protective devices by distinguishing between: a. Faults cleared by fuses b. Momentary outages c. Inrush/cold load current II. Determines the faulted sections based on SCADA fault indications and protection lockout signals III. Increases the accuracy of the fault location estimation based on SCADA fault current measurements and real-time fault analysis 3. Fault Isolation and Service Restoration Once the location and type of the fault has been pinpointed the systems will attempt to isolate the fault and restore the non-faulted section of the network. This can have three modes of operation: I. Closed-loop mode : This is initiated by the Fault location sub-function. It generates a switching order (i.e., sequence of switching) for the remotely controlled switching devices to isolate the faulted section, and restore service to the non-faulted sections. The switching order is automatically executed via SCADA. II. Advisory mode : This is initiated by the Fault location sub- function. It generates a switching order for remotely and manually controlled switching devices to isolate the faulted section, and restore service to the non-faulted sections. The switching order is presented to operator for approval and execution Wetterwald & Raymond Expires April 24, 2015 [Page 14] Internet-DrafDeterministic Networking Utilities requirement October 2014 III. Study mode : the operator initiates this function. It analyzes a saved case modified by the operator, and generates a switching order under the operating conditions specified by the operator. With the increasing volume of data that are collected through fault sensors utilities will be to use Big Data query and analysis tools to study outage information to anticipate and prevent outages by detecting failure patterns and their correlation with asset age, type, load profiles, time of day, weather conditions, and other conditions to discover conditions that lead to faults and take the necessary preventive and corrective measures. +--------------------------+----------------------------------------+ | FLISR Requirement | Attribute | +--------------------------+----------------------------------------+ | One way maximum delay | 80 ms | | Asymetric delay Required | No | | Maximum jitter | 40 ms | | Topology | Point to point, point to Multi-point, | | | Multi-point to Multi-point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on Node | Depends on customer impact | | failure | | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +--------------------------+----------------------------------------+ 4.2.3. Generation use case 4.2.3.1. Frequency Control / Automatic Generation Control (AGC) The system frequency should be maintained within a very narrow band. Deviations from the acceptable frequency range are detected and forwarded to the Load Frequency Control (LFC) system so that required up or down generation increase / decrease pulses can be sent to the power plants for frequency regulation. The trend in system frequency is a measure of mismatch between demand and generation, and is a necessary parameter for load control in interconnected systems. Automatic generation control (AGC) is a system for adjusting the power output of generators at different power plants, in response to changes in the load. Since a power grid requires that generation and load closely balance moment by moment, frequent adjustments to the output of generators are necessary. The balance can be judged by measuring the system frequency; if it is increasing, more power is being generated than used, and all machines in the system are accelerating. If the system frequency is decreasing, more demand is Wetterwald & Raymond Expires April 24, 2015 [Page 15] Internet-DrafDeterministic Networking Utilities requirement October 2014 on the system than the instantaneous generation can provide, and all generators are slowing down. Where the grid has tie lines to adjacent control areas, automatic generation control helps maintain the power interchanges over the tie lines at the scheduled levels. The AGC takes into account various parameters including the most economical units to adjust, the coordination of thermal, hydroelectric, and other generation types, and even constraints related to the stability of the system and capacity of interconnections to other power grids. For the purpose of AGC we use static frequency measurements and averaging methods are used to get a more precise measure of system frequency in steady-state conditions. During disturbances, more real-time dynamic measurements of system frequency are taken using PMUs, especially when different areas of the system exhibit different frequencies. But that is outside the scope of this use case. +-------------------------------+----------------+ | FCAG Requirement | Attribute | +-------------------------------+----------------+ | One way maximum delay | 500 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point | | Bandwidth | 20 Kbps | | Availability | 99.999 | | precise timing required | Yes | | Recovery time on Node failure | N/A | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 1% | +-------------------------------+----------------+ 4.3. Specific Network topologies of Smart Grid Applications Utilities often have very large private telecommunications networks. It covers an entire territory / country. The main purpose of the network, until now, has been to support transmission network monitoring, control, and automation, remote control of generation sites, and providing FCAPS services from centralized network operation centers. Wetterwald & Raymond Expires April 24, 2015 [Page 16] Internet-DrafDeterministic Networking Utilities requirement October 2014 Going forward, one network will support operation and maintenance of electrical networks (generation, transmission, and distribution), voice and data services for ten of thousands of employees and for exchange with neighboring interconnections, and administrative services. To meet those requirements, utility may deploy several physical networks leveraging different technologies across the country: an optical network and a microwave network for instance. Each protection and automatism system between two points has two communication circuits, one on each network. Path diversity between two substations is key. Regardless of the event type (hurricane, ice storm, etc.), one path shall stay available so the SPS can still operate. This is to meet a mandatory NERC requirement. In the optical network, signals are transmitted over more than tens of thousands of circuits using fiber optic links, microwave and telephone cables. This network is the nervous system of the utility's power transmission operations. The optical network represents ten of thousands of km of cable deployed along the power lines. Due to vast distances between transmission substations (as far as 280km apart), the fiber signal is amplified to reach a distance of 280 km without attenuation. 4.3.1. Precision Time Protocol Some utilities do not use GPS clocks in generation substations. One of the main reasons is that some of the generation plants are 30 to 50 meters deep under ground and the GPS signal can be weak and unreliable. Instead, atomic clocks are used. Clocks are synchronized amongst each other. Rubidium clocks provide clock and 1ms timestamps for IRIG-B. Some companies plan to transition to the Precision Time Protocol (IEEE 1588), distributing the synchronization signal over the IP/MPLS network. The Precision Time Protocol (PTP) is defined in IEEE standard 1588. PTP is applicable to distributed systems consisting of one or more nodes, communicating over a network. Nodes are modeled as containing a real-time clock that may be used by applications within the node for various purposes such as generating time-stamps for data or ordering events managed by the node. The protocol provides a mechanism for synchronizing the clocks of participating nodes to a high degree of accuracy and precision. PTP operates based on the following assumptions : It is assumed that the network eliminates cyclic forwarding of PTP messages within each communication path (e.g., by using a spanning tree protocol). PTP eliminates cyclic forwarding of PTP messages between communication paths. Wetterwald & Raymond Expires April 24, 2015 [Page 17] Internet-DrafDeterministic Networking Utilities requirement October 2014 PTP is tolerant of an occasional missed message, duplicated message, or message that arrived out of order. However, PTP assumes that such impairments are relatively rare. PTP was designed assuming a multicast communication model. PTP also supports a unicast communication model as long as the behavior of the protocol is preserved. Like all message-based time transfer protocols, PTP time accuracy is degraded by asymmetry in the paths taken by event messages. Asymmetry is not detectable by PTP, however, if known, PTP corrects for asymmetry. A time-stamp event is generated at the time of transmission and reception of any event message. The time-stamp event occurs when the message?s timestamp point crosses the boundary between the node and the network. 5. IANA Considerations This memo includes no request to IANA. 6. Security Considerations 6.1. Current Practices and Their Limitations Grid monitoring and control devices are already targets for cyber attacks and legacy communication protocols have many intrinsic network related vulnerabilities. DNP3, Modbus, PROFIBUS/PROFINET, and other protocols are designed around a common paradigm of request and respond. Each protocol is designed for a master device such as an HMI system to send commands to subordinate slave devices to retrieve data (reading inputs) or control (writing to outputs). Because many of these protocols lack authentication, encryption, or other basic security measures, they are prone to network-based attacks, allowing a malicious actor or attacker to utilize the request-and-respond system as a mechanism for command-and-control like functionality. Specific security concerns common to most industrial control, including utility communication protocols include the following: o Network or transport errors (e.g. malformed packets or excessive latency) can cause protocol failure. o Protocol commands may be available that are capable of forcing slave devices into inoperable states, including powering-off devices, forcing them into a listen-only state, disabling alarming. o Protocol commands may be available that are capable of restarting communications and otherwise interrupting processes. Wetterwald & Raymond Expires April 24, 2015 [Page 18] Internet-DrafDeterministic Networking Utilities requirement October 2014 o Protocol commands may be available that are capable of clearing, erasing, or resetting diagnostic information such as counters and diagnostic registers. o Protocol commands may be available that are capable of requesting sensitive information about the controllers, their configurations, or other need-to-know information. o Most protocols are application layer protocols transported over TCP; therefore it is easy to transport commands over non-standard ports or inject commands into authorized traffic flows. o Protocol commands may be available that are capable of broadcasting messages to many devices at once (i.e. a potential DoS). o Protocol commands may be available to query the device network to obtain defined points and their values (i.e. a configuration scan). o Protocol commands may be available that will list all available function codes (i.e. a function scan). o Bump in the wire (BITW) solutions : A hardware device is added to provide IPSec services between two routers that are not capable of IPSec functions. This special IPsec device will intercept then intercept outgoing datagrams, add IPSec protection to them, and strip it off incoming datagrams. BITW can all IPSec to legacy hosts and can retrofit non-IPSec routers to provide security benefits. The disadvantages are complexity and cost. These inherent vulnerabilities, along with increasing connectivity between IT an OT networks, make network-based attacks very feasible. Simple injection of malicious protocol commands provides control over the target process. Altering legitimate protocol traffic can also alter information about a process and disrupt the legitimate controls that are in place over that process. A man- in-the-middle attack could provide both control over a process and misrepresentation of data back to operator consoles. 6.2. Security Trends in Utility Networks Although advanced telecommunication networks can assist in transforming the energy industry, playing a critical role in maintaining high levels of reliability, performance, and manageability, they also introduce the need for an integrated security infrastructure. Many of the technologies being deployed to support smart grid projects such as smart meters and sensors can increase the vulnerability of the grid to attack. Top security concerns for utilities migrating to an intelligent smart grid communications platform center on the following trends: o Integration of distributed energy resources Wetterwald & Raymond Expires April 24, 2015 [Page 19] Internet-DrafDeterministic Networking Utilities requirement October 2014 o Proliferation of digital devices to enable management, automation, protection, and control o Regulatory mandates to comply with standards for critical infrastructure protection o Migration to new systems for outage management, distribution automation, condition-based maintenance, load forecasting, and smart metering o Demand for new levels of customer service and energy management This development of a diverse set of networks to support the integration of microgrids, open-access energy competition, and the use of network-controlled devices is driving the need for a converged security infrastructure for all participants in the smart grid, including utilities, energy service providers, large commercial and industrial, as well as residential customers. Securing the assets of electric power delivery systems, from the control center to the substation, to the feeders and down to customer meters, requires an end-to-end security infrastructure that protects the myriad of communication assets used to operate, monitor, and control power flow and measurement. Cyber security refers to all the security issues in automation and communications that affect any functions related to the operation of the electric power systems. Specifically, it involves the concepts of: o Integrity : data cannot be altered undetectably o Authenticity : the communication parties involved must be validated as genuine o Authorization : only requests and commands from the authorized users can be accepted by the system o Confidentiality : data must not be accessible to any unauthenticated users Wetterwald & Raymond Expires April 24, 2015 [Page 20] Internet-DrafDeterministic Networking Utilities requirement October 2014 When designing and deploying new smart grid devices and communication systems, it's imperative to understand the various impacts of these new components under a variety of attack situations on the power grid. Consequences of a cyber attack on the grid telecommunication network can be catastrophic. This is why security for smart grid is not just an ad hoc feature or product, it's a complete framework integrating both physical and Cyber security requirements and covering the entire smart grid networks from generation to distribution. Security has therefore become one of the main foundations of the utility telecom network architecture and must be considered at every layer with a defense-in-depth approach. Migrating to IP based protocols is key to address these challenges for two reasons: 1. IP enables a rich set of features and capabilities to enhance the security posture 2. IP is based on open standards, which allows interoperability between different vendors and products, driving down the costs associated with implementing security solutions in OT networks. Securing OT communication over packet-switched IP networks follows the same principles that are foundational for securing the IT infrastructure, i.e., consideration must be given to enforcing electronic access control for both person- to-machine and machine-to-machine communications, and providing the appropriate levels of data privacy, device and platform integrity, and threat detection and mitigation. 7. Acknowledgements Faramarz Maghsoodlou, Ph. D. IoT Connected Industries and Energy Practice Cisco Pascal Thubert, CTAO Cisco 8. References 8.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [min_ref] authSurName, authInitials., "Minimal Reference", 2006. 8.2. Informative References [I-D.finn-detnet-problem-statement] Finn, N. and P. Thubert, "Deterministic Networking Problem Statement", Internet-Draft draft-finn-detnet-problem- statement-01, October 2014. Appendix A. Additional Stuff This becomes an Appendix. Authors' Addresses Wetterwald & Raymond Expires April 24, 2015 [Page 21] Internet-DrafDeterministic Networking Utilities requirement October 2014 Patrick Wetterwald Cisco Systems 45 Allees des Ormes Mougins, 06250 FRANCE Phone: +33 4 97 23 26 36 Email: pwetterw@cisco.com Jean Raymond Hydro-Quebec 1500 University Montreal, H3A3S7 Canada Phone: +1 514 840 3000 Email: raymond.jean@hydro.qc.ca Wetterwald & Raymond Expires April 24, 2015 [Page 22]