idnits 2.17.1 draft-hongcs-t2trg-pem-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 11 instances of lines with non-ascii characters in the document. == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 62 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 10 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 2 characters in excess of 72. ** There are 47 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'D' is defined on line 250, but no explicit reference was found in the text == Unused Reference: 'E' is defined on line 255, but no explicit reference was found in the text -- Possible downref: Non-RFC (?) normative reference: ref. 'D' -- Possible downref: Non-RFC (?) normative reference: ref. 'E' Summary: 2 errors (**), 0 flaws (~~), 7 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 T2TRG Hong, Choong Seon 2 Internet-Draft Kyung Hee University 3 Intended status: Standards Track Munir, Md. Shirajum 4 Expires: August 15, 2022 Kyung Hee University 5 Kitae Kim 6 Kyung Hee University 7 Seok Won Kang 8 Kyung Hee University 10 Proactive energy management for smart city with edge 11 computing using meta-reinforcement learning scheme 12 draft-hongcs-t2trg-pem-00 14 Abstract 16 Renewable energy enabled sustainable energy management ensures 17 a high degree of reliability in order to fulfill the energy 18 demand of a smart city. In such case, renewable energy 19 generation is random over time and also energy consumption of 20 smart city users’ is nondeterministic in nature. Therefore, to 21 ensure sustainable energy management for smart city, 22 a proactive energy management scheme should be integrated into 23 smart city network. In which, edge node should be considered as 24 local computational unit for each energy user and microgrid 25 controller should be played the role of energy management decision 26 aggregator. As a result, proactive energy management scheme 27 not only overcomes the challenges of renewable energy-aware 28 demand scheduling but also establishes a strong relationship 29 for both energy generation and consumption over time. 30 Therefore, a distributed mechanism is considered, where the edge 31 node for executing local agent to determine an individual 32 users’ policy with respect to energy consumption and renewable 33 energy generation (users’ own sources). On the other hand, 34 microgrid controller determines meta-policy through a meta-agent 35 with Recurrent Neural Network (RNN). Since a meta-agent accepts 36 local policy as an input with historical observations, which 37 ensures fast and efficient execution of proactive energy management 38 for the smart city. 40 Status of this Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at http://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six 51 months and may be updated, replaced, or obsoleted by other 52 documents at any time. It is inappropriate to use Internet-Drafts 53 as reference material or to cite them other than as 54 "work in progress." 55 This Internet-Draft will expire on August xx, 20xx. 57 Copyright Notice 59 Copyright (c) 2020 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (http://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . .. . . . . . 2 75 1.1. Terminology and Requirements Language . . . . . . . . . 2 76 2. Energy Data Flow Management . . . . . . . . . . . . . . . . . . 3 77 2.1. Energy Data Flow . . . . . . . . . . . . . .. . . . . . 4 78 2.2. Energy Data Format . . . . . . . . . . . . . . . . . . . 5 79 3. Proactive Energy Management for Smart City . . . . . . . . . . 6 80 3.1 Meta-Reinforcement Learning with Edge Computing . . . . 7 81 3.2. Process flow of proactive energy management. . . . . . . .8 82 4. IANA Considerations . . . . . . .. . . . . . . . . . . . . . . 8 83 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 84 6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 85 6.1. Normative References . . . . . . . . . . . . . . . . . . . . . 9 86 6.2. Informative References . . . . . .. . . . . . . . . . . . . . 9 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . .. . . . 10 89 1. Introduction 91 In the modern development arena, smart city, and renewable energy 92 are indispensable toward the ecological growth of urban technology 93 to enable sustainable smart services [a,b]. A microgrid is 94 capable to fulfill that huge amount of energy demand by enabling 95 the efficient demand scheduling of smart city energy consumptions. 96 However, the challenges come with the unpredictable nature of 97 both energy consumption and renewable generation, which also have 98 a strong relationship over the history of energy consumption and 99 generation [c,d]. 101 Therefore, to overcome those challenges, a proactive energy 102 management is essential such that both energy consumption 103 and renewable energy generation can be considered. In order to 104 do that, meta-reinforcement learning (Meta-RL)-based [e] 105 energy scheduling model, in which this method is capable of 106 handling both energy consumption and generation with the historical 107 and current observations using Recurrent Neural Network (RNN). 109 Proactive energy management for the smart city should be solved by 110 distributed manner. 111 . First, a local agent with an edge computing facility determines 112 a local policy with respect to energy consumption and generation 113 (user’s own renewable sources) for its nearby energy users’. 115 . Second, by reusing the historical observations and local policy, 116 the microgrid controller estimates the meta-policy for energy 117 scheduling. Further, it takes necessary action based on meta-policy 118 to enable sustainable energy management for the smart city. 120 1.1. Terminology and Requirements Language 122 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 123 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 124 document are to be interpreted as described in RFC 2119 [RFC2119]. 126 2. Energy Data Flow Management 128 The Microgrid-powered smart city includes both renewable and 129 non-renewable energy sources, where each individual building 130 has its own renewable (solar) energy sources. The city 131 connected with edge computing enabled wireless networks 132 to fulfill smart city services. Each energy user (i.e., 133 home, building, school, commercial building, and so on) 134 is associated with its nearby edge computing server. Energy 135 demand, renewable generation, and required amount should send 136 to its associated edge server. Based on each user data, edge 137 server determine its own local energy management policy by 138 applying Deep Q-learning. The output (reward(t), action(t) 139 reward(t-1), action(t-1)) should send to microgrid controller. 140 Microgrid controller decides the overall energy management 141 policy for the smart city and send feed back to each user 142 via edge server. The communication should be through the wireless 143 communications protocol (LTE, 5G, LTU, Wifi, etc.) for exchanging 144 the energy management data. 146 2.1. Energy Data Flow 148 Energy data flow of proactive energy management for smart 149 city as shown in Figure 1. 150 . Energy user with renewable energy sources can send energy 151 demand and generation raw data to MEC Server # in smart 152 city network 154 . Each energy user's observational data (reward and action) 155 by executing reinforcement learning (Deep Q-learning) from 156 MEC should send to microgrid controller 158 . Other energy sources (except the energy sources that are 159 associated with smart city user) should send to microgrid 160 controller 162 . Microgrid controller send the energy management executing 163 command to each users’ 165 Figure 1: Energy data flow of proactive energy management for 166 smart city 168 2.2 Energy Data Format 169 The data format complies with tuple. 170 Figure 2 represent the data format of proactive energy management 171 scheme. 173 Figure 2: Energy data format of proactive energy management 174 for smart city 175 3. Proactive Energy Management for Smart City 177 Each edge server estimates the local policy for the associated energy 178 users while Microgrid controller determines the meta policy using 179 a little amount of information from local policy. Establishing a 180 strong correlation between energy generation and consumption using 181 Markovian properties for each energy user. 183 3.1 Meta-Reinforcement Learning with Edge Computing 185 The proactive energy demand scheduling for smart city problem 186 is solved distributively, where first, obtain the local policy 187 by learning the local agent with respect to energy consumption 188 and renewable energy generation through the nearby edge server. 189 Second, in order to generate meta-policy, we send local policy 190 information to the microgrid controller alone with previous 191 policy observation, so that meta-agent can learn very fast 192 with the optimal decision. This procedure is the same for 193 every energy user and meta-RL model procedure is shown in 194 Figure 3. 196 Figure 3: Proactive energy management for smart city using 197 Meta-RL 198 3.2. Process flow of proactive energy management 200 Process flow of proactive energy management is illustrated in 201 Figure 4. 202 . Energy generation and demand data from all users at associated 203 edge server 205 . Local policy estimation process for each user’s at the edge server 206 using DQN and observation data by local agent send to microgrid 207 controller 209 . Meta energy management policy estimation using local policy at 210 microgrid controller and action command send to user through 211 edge server 213 . Apply energy management policy to smart city users by the edge 214 server service 216 Figure 4: Process flow of proactive energy management for smart city 217 4. IANA Considerations 219 There are no IANA considerations related to this document. 221 5. Security Considerations 223 This note touches communication security as in wireless communications 224 protocol (LTE, 5G, LTU, Wifi, etc.). 226 6. References 228 6.1. Normative References 230 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 231 Requirement Levels", BCP 14, RFC 2119, March 1997. 233 [a] M. S. Munir, S. F. Abedin, M. G. R. Alam, N. H. Tran 234 and C. S. Hong, “Intelligent service fulfillment 235 for software defined networks in smart city,” 2018 236 International Conference on Information Networking 237 (ICOIN), pp. 516-521, 2018. (in Chiang Mai, Thailand). 239 [b] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim 240 and C. S. Hong, " Smart Agent based Dynamic Data 241 Aggregation for Delay Sensitive Smart City Services," 242 Journal of KIISE, vol. 45, no. 4, pp. 395-402, April 243 2018. 245 [c] Y. Zhang, M. H. Hajiesmaili, S. Cai, M. Chen and Q. Zhu, 246 "Peak-Aware Online Economic Dispatching for Microgrids," 247 in IEEE Transactions on Smart Grid, vol. 9, no. 1, pp. 248 323-335, Jan. 2018. 250 [D] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim, 251 and C. S. Hong, “RNN based Energy Demand Prediction for 252 Smart-Home in Smart-Grid Framework,” Korea Software 253 Congress 2017, pp. 437-439, 2017 (in Korea). 255 [E] J. X. Wang et al., “Learning to reinforcement learn,” 256 CogSci , 2017. (In London, UK). 258 6.2. Informative References 259 Authors' Addresses 261 Choong Seon Hong 262 Computer Science and Engineering Department, Kyung Hee University 263 Yongin, South Korea 264 Phone: +82 (0)31 201 2532 265 Email: cshong@khu.ac.kr 266 Md. Shirajum Munir 267 Computer Science and Engineering Department, Kyung Hee University 268 Yongin, South Korea 269 Phone: +82 (0)31 201 2987 270 Email: munir@khu.ac.kr 271 Ki Tae Kim 272 Computer Science and Engineering Department, Kyung Hee University 273 Yongin, South Korea 274 Phone: +82 (0)31 201 2532 275 Email: glideslope@khu.ac.kr 276 Seok Won Kang 277 Computer Science and Engineering Department, Kyung Hee University 278 Yongin, South Korea 279 Phone: +82 (0)31 201 2532 280 Email: dudtntdud@khu.ac.kr