T2TRG Hong, Choong Seon Internet-Draft Kyung Hee University Intended status: Standards Track Munir, Md. Shirajum Expires: August 15, 2022 Kyung Hee University Kitae Kim Kyung Hee University Seok Won Kang Kyung Hee University Proactive energy management for smart city with edge computing using meta-reinforcement learning scheme draft-hongcs-t2trg-pem-00 Abstract Renewable energy enabled sustainable energy management ensures a high degree of reliability in order to fulfill the energy demand of a smart city. In such case, renewable energy generation is random over time and also energy consumption of smart city users’ is nondeterministic in nature. Therefore, to ensure sustainable energy management for smart city, a proactive energy management scheme should be integrated into smart city network. In which, edge node should be considered as local computational unit for each energy user and microgrid controller should be played the role of energy management decision aggregator. As a result, proactive energy management scheme not only overcomes the challenges of renewable energy-aware demand scheduling but also establishes a strong relationship for both energy generation and consumption over time. Therefore, a distributed mechanism is considered, where the edge node for executing local agent to determine an individual users’ policy with respect to energy consumption and renewable energy generation (users’ own sources). On the other hand, microgrid controller determines meta-policy through a meta-agent with Recurrent Neural Network (RNN). Since a meta-agent accepts local policy as an input with historical observations, which ensures fast and efficient execution of proactive energy management for the smart city. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." Hong, et al. Expires August 15, 2022 [Page 1] Internet-Draft Proactive energy management for smart city October 2020 This Internet-Draft will expire on August xx, 20xx. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . .. . . . . . 2 1.1. Terminology and Requirements Language . . . . . . . . . 2 2. Energy Data Flow Management . . . . . . . . . . . . . . . . . . 3 2.1. Energy Data Flow . . . . . . . . . . . . . .. . . . . . 4 2.2. Energy Data Format . . . . . . . . . . . . . . . . . . . 5 3. Proactive Energy Management for Smart City . . . . . . . . . . 6 3.1 Meta-Reinforcement Learning with Edge Computing . . . . 7 3.2. Process flow of proactive energy management. . . . . . . .8 4. IANA Considerations . . . . . . .. . . . . . . . . . . . . . . 8 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 6.1. Normative References . . . . . . . . . . . . . . . . . . . . . 9 6.2. Informative References . . . . . .. . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . .. . . . 10 Hong, et al. Expires August 15, 2022 [Page 2] Internet-Draft Proactive energy management for smart city October 2020 1. Introduction In the modern development arena, smart city, and renewable energy are indispensable toward the ecological growth of urban technology to enable sustainable smart services [a,b]. A microgrid is capable to fulfill that huge amount of energy demand by enabling the efficient demand scheduling of smart city energy consumptions. However, the challenges come with the unpredictable nature of both energy consumption and renewable generation, which also have a strong relationship over the history of energy consumption and generation [c,d]. Therefore, to overcome those challenges, a proactive energy management is essential such that both energy consumption and renewable energy generation can be considered. In order to do that, meta-reinforcement learning (Meta-RL)-based [e] energy scheduling model, in which this method is capable of handling both energy consumption and generation with the historical and current observations using Recurrent Neural Network (RNN). Proactive energy management for the smart city should be solved by distributed manner. . First, a local agent with an edge computing facility determines a local policy with respect to energy consumption and generation (user’s own renewable sources) for its nearby energy users’. . Second, by reusing the historical observations and local policy, the microgrid controller estimates the meta-policy for energy scheduling. Further, it takes necessary action based on meta-policy to enable sustainable energy management for the smart city. 1.1. Terminology and Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Hong, et al. Expires August 15, 2022 [Page 3] Internet-Draft Proactive energy management for smart city October 2020 2. Energy Data Flow Management The Microgrid-powered smart city includes both renewable and non-renewable energy sources, where each individual building has its own renewable (solar) energy sources. The city connected with edge computing enabled wireless networks to fulfill smart city services. Each energy user (i.e., home, building, school, commercial building, and so on) is associated with its nearby edge computing server. Energy demand, renewable generation, and required amount should send to its associated edge server. Based on each user data, edge server determine its own local energy management policy by applying Deep Q-learning. The output (reward(t), action(t) reward(t-1), action(t-1)) should send to microgrid controller. Microgrid controller decides the overall energy management policy for the smart city and send feed back to each user via edge server. The communication should be through the wireless communications protocol (LTE, 5G, LTU, Wifi, etc.) for exchanging the energy management data. Hong, et al. Expires August 15, 2022 [Page 4] Internet-Draft Proactive energy management for smart city October 2020 2.1. Energy Data Flow Energy data flow of proactive energy management for smart city as shown in Figure 1. . Energy user with renewable energy sources can send energy demand and generation raw data to MEC Server # in smart city network . Each energy user's observational data (reward and action) by executing reinforcement learning (Deep Q-learning) from MEC should send to microgrid controller . Other energy sources (except the energy sources that are associated with smart city user) should send to microgrid controller . Microgrid controller send the energy management executing command to each users’ Figure 1: Energy data flow of proactive energy management for smart city Hong, et al. Expires August 15, 2022 [Page 5] Internet-Draft Proactive energy management for smart city October 2020 2.2 Energy Data Format The data format complies with tuple. Figure 2 represent the data format of proactive energy management scheme. Figure 2: Energy data format of proactive energy management for smart city Hong, et al. Expires August 15, 2022 [Page 6] Internet-Draft Proactive energy management for smart city October 2020 3. Proactive Energy Management for Smart City Each edge server estimates the local policy for the associated energy users while Microgrid controller determines the meta policy using a little amount of information from local policy. Establishing a strong correlation between energy generation and consumption using Markovian properties for each energy user. Hong, et al. Expires August 15, 2022 [Page 7] Internet-Draft Proactive energy management for smart city October 2020 3.1 Meta-Reinforcement Learning with Edge Computing The proactive energy demand scheduling for smart city problem is solved distributively, where first, obtain the local policy by learning the local agent with respect to energy consumption and renewable energy generation through the nearby edge server. Second, in order to generate meta-policy, we send local policy information to the microgrid controller alone with previous policy observation, so that meta-agent can learn very fast with the optimal decision. This procedure is the same for every energy user and meta-RL model procedure is shown in Figure 3. Figure 3: Proactive energy management for smart city using Meta-RL Hong, et al. Expires August 15, 2022 [Page 8] Internet-Draft Proactive energy management for smart city October 2020 3.2. Process flow of proactive energy management Process flow of proactive energy management is illustrated in Figure 4. . Energy generation and demand data from all users at associated edge server . Local policy estimation process for each user’s at the edge server using DQN and observation data by local agent send to microgrid controller . Meta energy management policy estimation using local policy at microgrid controller and action command send to user through edge server . Apply energy management policy to smart city users by the edge server service Figure 4: Process flow of proactive energy management for smart city Hong, et al. Expires August 15, 2022 [Page 9] Internet-Draft Proactive energy management for smart city October 2020 4. IANA Considerations There are no IANA considerations related to this document. 5. Security Considerations This note touches communication security as in wireless communications protocol (LTE, 5G, LTU, Wifi, etc.). 6. References 6.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [a] M. S. Munir, S. F. Abedin, M. G. R. Alam, N. H. Tran and C. S. Hong, “Intelligent service fulfillment for software defined networks in smart city,” 2018 International Conference on Information Networking (ICOIN), pp. 516-521, 2018. (in Chiang Mai, Thailand). [b] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim and C. S. Hong, " Smart Agent based Dynamic Data Aggregation for Delay Sensitive Smart City Services," Journal of KIISE, vol. 45, no. 4, pp. 395-402, April 2018. [c] Y. Zhang, M. H. Hajiesmaili, S. Cai, M. Chen and Q. Zhu, "Peak-Aware Online Economic Dispatching for Microgrids," in IEEE Transactions on Smart Grid, vol. 9, no. 1, pp. 323-335, Jan. 2018. [D] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim, and C. S. Hong, “RNN based Energy Demand Prediction for Smart-Home in Smart-Grid Framework,” Korea Software Congress 2017, pp. 437-439, 2017 (in Korea). [E] J. X. Wang et al., “Learning to reinforcement learn,” CogSci , 2017. (In London, UK). 6.2. Informative References Hong, et al. Expires August 15, 2022 [Page 10] Internet-Draft Proactive energy management for smart city October 2020 Authors' Addresses Choong Seon Hong Computer Science and Engineering Department, Kyung Hee University Yongin, South Korea Phone: +82 (0)31 201 2532 Email: cshong@khu.ac.kr Md. Shirajum Munir Computer Science and Engineering Department, Kyung Hee University Yongin, South Korea Phone: +82 (0)31 201 2987 Email: munir@khu.ac.kr Ki Tae Kim Computer Science and Engineering Department, Kyung Hee University Yongin, South Korea Phone: +82 (0)31 201 2532 Email: glideslope@khu.ac.kr Seok Won Kang Computer Science and Engineering Department, Kyung Hee University Yongin, South Korea Phone: +82 (0)31 201 2532 Email: dudtntdud@khu.ac.kr