idnits 2.17.1 draft-ylonen-sshkeybcp-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to use 'NOT RECOMMENDED' as an RFC 2119 keyword, but does not include the phrase in its RFC 2119 key words list. -- The document date (April 04, 2013) is 4011 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '1' on line 1051 -- Looks like a reference, but probably isn't: '2' on line 2629 == Unused Reference: 'FIPS200' is defined on line 2679, but no explicit reference was found in the text == Unused Reference: 'NIST800-53' is defined on line 2683, but no explicit reference was found in the text == Unused Reference: 'KENT' is defined on line 2689, but no explicit reference was found in the text -- Possible downref: Non-RFC (?) normative reference: ref. 'FIPS199' -- Possible downref: Non-RFC (?) normative reference: ref. 'FIPS200' -- Possible downref: Non-RFC (?) normative reference: ref. 'NIST800-53' -- Possible downref: Non-RFC (?) normative reference: ref. 'KENT' -- Duplicate reference: RFC4251, mentioned in 'RFC4120', was also mentioned in 'RFC3280'. -- Duplicate reference: RFC4251, mentioned in 'RFC4251', was also mentioned in 'RFC4120'. -- Possible downref: Normative reference to a draft: ref. 'SFTP' Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 10 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 draft-ylonen-sshkeybcp-01.txt 3 Network Working Group T. Ylonen 4 Internet-Draft SSH Communications Security 5 Expires: October 06, 2013 G. Kent 6 SecureIT 7 M. Souppaya 8 NIST 9 April 04, 2013 11 Managing SSH Keys for Automated Access - Current Recommended Practice 13 Abstract 15 This document presents current recommended practice for managing SSH 16 user keys for automated access. It provides guidelines for 17 discovering, remediating, and continuously managing SSH user keys and 18 other authentication credentials. 20 Various threats from poorly managed SSH keys are identified, 21 including virus spread, unaudited backdoors, illegitimate access 22 using leaked keys, lack of proper termination of access, use of 23 legitimate access for unintended purposes, and accidental human 24 errors. 26 Hundreds of thousands, even over a million SSH keys authorizing 27 access have been found from the IT environments of many large 28 organizations. This is many times more than they have interactive 29 users. These access-granting credentials have largely been ignored 30 in identity and access management, and present a real risk to 31 information security. 33 A process is presented for discovering who has access to what, 34 bringing an existing IT environment under control with respect to 35 automated access and SSH keys. The process includes moving 36 authorized keys to protected locations, removing unused keys, 37 associating authorized keys with a business process or application 38 and removing keys for which no valid purpose can be found, rotating 39 existing keys, restricting what can be done with each authorized key, 40 and establishing an approval process for new authorized keys. A 41 process is also presented for continuous monitoring and controlled 42 authorized key setup. 44 Finally, recommendations are made for security policy makers for 45 ensuring that automated access and SSH keys are properly addressed in 46 an organization's security policy. 48 Specific requirements are presented that address the security issues 49 while keeping costs reasonable. 51 Guidance is also provided on how to reduce operational cost while 52 addressing the threats and how to use tools to automate the 53 management process. 55 Status of This Memo 57 This Internet-Draft is submitted in full conformance with the 58 provisions of BCP 78 and BCP 79. 60 Internet-Drafts are working documents of the Internet Engineering 61 Task Force (IETF). Note that other groups may also distribute 62 working documents as Internet-Drafts. The list of current Internet- 63 Drafts is at http://datatracker.ietf.org/drafts/current/. 65 Internet-Drafts are draft documents valid for a maximum of six months 66 and may be updated, replaced, or obsoleted by other documents at any 67 time. It is inappropriate to use Internet-Drafts as reference 68 material or to cite them other than as "work in progress." 70 This Internet-Draft will expire on October 06, 2013. 72 Copyright Notice 74 Copyright (c) 2013 IETF Trust and the persons identified as the 75 document authors. All rights reserved. 77 This document is subject to BCP 78 and the IETF Trust's Legal 78 Provisions Relating to IETF Documents 79 (http://trustee.ietf.org/license-info) in effect on the date of 80 publication of this document. Please review these documents 81 carefully, as they describe your rights and restrictions with respect 82 to this document. Code Components extracted from this document must 83 include Simplified BSD License text as described in Section 4.e of 84 the Trust Legal Provisions and are provided without warranty as 85 described in the Simplified BSD License. 87 Table of Contents 89 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 90 1.1. Purpose and Scope . . . . . . . . . . . . . . . . . . . . 4 91 1.2. Audience . . . . . . . . . . . . . . . . . . . . . . . . 4 92 1.3. Structure of This Document . . . . . . . . . . . . . . . 4 93 1.4. Words Signifying Level of Requirement . . . . . . . . . . 5 94 1.5. Impact Levels for Information Systems . . . . . . . . . . 5 95 2. The Basics of SSH Protocol and Implementations . . . . . . . 8 96 2.1. The SSH Protocol . . . . . . . . . . . . . . . . . . . . 8 97 2.2. User Authentication in SSH . . . . . . . . . . . . . . . 8 98 2.2.1. Password Authentication . . . . . . . . . . . . . . . 9 99 2.2.2. Public Key Authentication . . . . . . . . . . . . . . 9 100 2.2.3. Kerberos Authentication . . . . . . . . . . . . . . . 9 101 2.2.4. Host-Based Authentication . . . . . . . . . . . . . . 10 102 2.2.5. Comparison of the Authentication Methods . . . . . . 10 103 2.2.6. Dangers of Unverified and Shared Host Keys . . . . . 10 104 2.3. Common Use Cases . . . . . . . . . . . . . . . . . . . . 11 105 2.3.1. Interactive Use . . . . . . . . . . . . . . . . . . . 11 106 2.3.2. Automated Access . . . . . . . . . . . . . . . . . . 11 107 2.3.3. File Transfers . . . . . . . . . . . . . . . . . . . 12 108 3. Threats Arising from Poorly Managed Automated Access and SSH 109 Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 110 3.1. Virus Spread Threat . . . . . . . . . . . . . . . . . . . 13 111 3.2. Unaudited Backdoor Threat . . . . . . . . . . . . . . . . 14 112 3.3. Leaked Keys May Provide Access for Extended Periods . . . 15 113 3.4. Lack of Proper Termination of Access . . . . . . . . . . 16 114 3.5. Use for Unintended Purpose . . . . . . . . . . . . . . . 17 115 3.6. Accidental Data Transfers and Human Errors . . . . . . . 18 116 3.7. Problem Under Radar . . . . . . . . . . . . . . . . . . . 19 117 4. Assessing the SSH Key Management Situation and Risks . . . . 20 118 5. Key Remediation Solution Planning and Deployment . . . . . . 23 119 5.1. Discovering SSH Keys and Trust Relationships . . . . . . 24 120 5.2. Moving Authorized Keys to Protected Locations . . . . . . 28 121 5.3. Monitoring Use of Trust Relationships and Keys . . . . . 29 122 5.4. Removing Trust Relationships That Are No Longer Used or 123 Otherwise Inappropriate . . . . . . . . . . . . . . . . . 31 124 5.5. Associating Trust Relationships with Application and/or 125 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . 32 126 5.6. Implementing Approval Process for Setting Up New Trust 127 Relationships . . . . . . . . . . . . . . . . . . . . . . 33 128 5.7. Rotating Existing SSH User Keys . . . . . . . . . . . . . 35 129 5.8. Configuring Command Restrictions on Authorized Keys . . . 36 130 5.9. Configuring IP Address Restrictions on Authorized Keys . 37 131 6. Continuous Monitoring and Management of SSH Keys and 132 Automated Access . . . . . . . . . . . . . . . . . . . . . . 38 133 6.1. Continuous Monitoring of Changes to Trust Relationships . 38 134 6.2. Removal of Trust Relationships . . . . . . . . . . . . . 41 135 6.3. Periodic Rotation of Trust Relationships . . . . . . . . 41 136 7. Policy Recommendations . . . . . . . . . . . . . . . . . . . 41 137 8. Considerations for Software Tools . . . . . . . . . . . . . . 45 138 8.1. Reducing Cost and Improving Security by Automation . . . 46 139 9. Security Considerations . . . . . . . . . . . . . . . . . . . 47 140 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 49 141 11. Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 49 142 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 57 143 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 58 145 1. Introduction 146 1.1. Purpose and Scope 148 This document focuses on risks related to poorly managed automated 149 access in information systems and particularly SSH user keys, and how 150 to reduce the risks. It documents current best practice of managing 151 SSH keys for cost-effectively minimizing the risks, and provides 152 security policy recommendations and auditing guidelines relating to 153 SSH keys and other automated access. 155 1.2. Audience 157 This document is intended for information security policy makers, 158 auditors, security managers, IT operations managers, system 159 administrators, and others who are responsible for specifying, 160 acquiring, testing, implementing, maintaining, and auditing identity 161 and access management and SSH solutions. Portions of the document 162 may be of interest to technically advanced end users and systems 163 programmers involved with SSH and other automated access 164 technologies. 166 1.3. Structure of This Document 168 Section 1.4 specifies what certain words indicating level of 169 requirement for compliance with this standard mean. 171 Section 1.5 defines impact levels for information systems, including 172 some important definitions relating to information systems having low 173 impact themselves but having automated access to higher-impact 174 information systems. 176 Section 2 summarizes the basics of the SSH protocol and 177 implementations, with particular emphasis on authentication methods 178 for automated access and typical use cases for automated access. 180 Section 3 describes threats arising from poorly managed SSH user 181 keys. Most of the threats are also relevant for other kinds of 182 automated access. However, because of the ubiquity of SSH keys and 183 the acuteness of addressing them the discussion focuses on SSH keys. 185 Section 4 introduces simple metrics and questions that are useful in 186 scoping the risks related to SSH user keys and gaining basic 187 understanding of the size of the problem in an organization. The 188 approach is suitable for both IT auditors responsible for assessing 189 compliance with security policies as well as government and other 190 policy makers wanting to measure the overall state of compliance and 191 security across agencies. 193 Section 5 introduces the process for detailed analysis of existing 194 automated trust relationships and risks (with an emphasis on SSH user 195 keys), as well as recommended steps for remediating the risks. This 196 section also discusses the specific threats addressed by each 197 remediation step and risks involved in not implementing a particular 198 step. 200 Section 6 provides recommendations for continuous monitoring and 201 management of SSH user keys and other automated trust relationships, 202 as well as for auditing steps to be taken for ensuring that an 203 organization keeps automated access under control after an initial 204 remediation phase. 206 Section 7 provides recommendations for an organization's security 207 policy for properly addressing SSH user keys and automated access. 209 Section 8 summarizes issues to consider when planning use of 210 automated software tools for managing automated access with SSH and 211 particularly SSH user keys. It also illustrates how to achieve cost 212 savings in existing operational processes. 214 Section 9 summarizes security considerations. Most of this document 215 is about security and managing elements of security and access, and 216 this section serves as a conclusion and summary of this document. 218 Section 11 provides a glossary of the technical terms used in this 219 document. 221 1.4. Words Signifying Level of Requirement 223 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 224 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 225 document are to be interpreted as described in RFC 2119. 227 1.5. Impact Levels for Information Systems 229 The appropriate level of security and effort expended on security 230 often depends on the level of impact from a failure or compromise of 231 an information system. FIPS Publication 199 [FIPS199] provides 232 designations for impact levels on organizational information systems 233 and a process for categorizing information systems. 235 This document makes reference to the impact levels described FIPS 199 236 (please see original document for exact definitions, this is just a 237 simplifying summary): 239 Low impact: Unauthorized disclosure, modification, destruction, or 240 disruption of access have limited adverse effect on organizational 241 operations, organizational assets, or individuals. 243 Moderate impact: Unauthorized disclosure, modification, destruction, 244 or disruption of access could be expected to have a serious 245 adverse effect on organizational operations, organizational 246 assets, or individuals. 248 High impact: Unauthorized disclosure, modification, destruction, or 249 disruption of access could be expected to have a severe or 250 catastrophic adverse effect on organizational operations, 251 organizational assets, or individuals. 253 FIPS Publication 199 analyzes impact levels separately for 254 confidentiality, integrity, and availability. For considerations 255 around automated access, the impact level of access to an account on 256 an information system is taken to be the highest level of these three 257 principles for the information system, since this specification 258 primarily relates to operating system level access to information 259 systems, and operating system level access can often be used to 260 breach all three objectives simultaneously. Furthermore, experience 261 has shown that once an attacker has operating-system level access to 262 one user account on a computer, various attack vectors (including 263 bugs in system software and misconfigurations) can often be utilized 264 to escalate the access to high-level administrative access. That 265 definitely compromises all three principles of information security. 267 Configured trust relationships for automated access (e.g., using SSH 268 user keys) may permit access from low-impact information systems to 269 high-impact information systems without providing a password or other 270 authentication credential from a user. This is particularly relevant 271 if the authentication credential or authorized key permits access on 272 the high-impact information system without restrictions on the 273 commands that can be executed on the high-impact information system. 274 In this case, access to the low-impact information system implies 275 access to the high-impact information system. The information system 276 owner inherently accepted this risk by allowing a low-impact system 277 access to a high-impact system with providing compensation controls. 278 There may also be situations where the high-impact system owner may 279 not know the key has been copied to a low-impact system, or from one 280 low-impact system to another. 282 Therefore, whenever a low-impact information system has a configured 283 trust relationship permitting it to access a high-impact information 284 system without a restriction on the commands that can be executed on 285 the high-impact information system, the low-impact information system 286 MUST be treated as having the impact level of the highest-impact 287 information system that it can access using automated trust 288 relationships. 290 This implies that to avoid treating low-impact information systems as 291 high-impact systems, there must be a well-defined boundary in the IT 292 environment that trust relationships can only cross in the direction 293 allowing access from higher-impact systems into lower-impact systems, 294 but not vice versa. If such boundary is relied on, it MUST be 295 audited and continuously monitored to enforce its existence. Such a 296 boundary could exist, for example, between development and production 297 systems. 299 Sometimes otherwise low-impact systems are used for producing code, 300 software distributions, or data sets that will be used on higher- 301 impact systems. Such systems SHOULD be treated as higher-impact 302 systems in view of Advanced Persistent Threat (APT) scenarios where 303 an attacker could insert a backdoor in software that eventually gets 304 copied to production systems. 306 It should be noted that several current SSH implementations 307 (including OpenSSH) only permit configuring command restrictions for 308 access based on SSH user keys. It is currently not possible to 309 configure command restrictions for Kerberos-based authentication, 310 host-based authentication, hard-coded passwords, or passwords coming 311 from password vaults, which has implications for the above 312 requirement. 314 Command restrictions are a compensation control that can be leveraged 315 to minimize the exposure to the additional risks exposed to a high- 316 impact system from allowing limited access to the hosted resources 317 from a low-impact system. Command restrictions used for this purpose 318 MUST be designed to be effective in limiting what actually can be 319 done with the access, and MUST prevent interactive access and port 320 forwarding. For example, a command restriction permitting arbitrary 321 commands or interactive shell is not effective. 323 Furthermore, if a trust relationship has a command restriction that 324 limits its use to file transfers but the directories from which files 325 can be read or modified using it have not been restricted, it exposes 326 the server to a more limited risk. The trust relationship may be 327 used to read any file or directory on the server accessible to the 328 user account used for file transfers, even those outside the intended 329 directories. It may also be used to write any file that is writable 330 by the user account; it is fairly common to have configuration files 331 on servers that are inappropriately world-writable, in which case 332 these files could be modified. 334 If a trust relation is restricted to file transfers but does not 335 limit the directories that can be accessed, the originating 336 information system SHOULD be considered as having at least the impact 337 level of the highest-impact information system to which it has such 338 access. 340 2. The Basics of SSH Protocol and Implementations 342 SSH (Secure Shell) is a protocol and software tool for logging into a 343 remote machine, executing commands remotely, and transferring files 344 with a remote machine over a computer network. SSH can also be used 345 for implementing a protected tunnel for delivering other services. 347 SSH is very widely used for administering Linux and Unix systems, 348 virtual machines, routers, firewalls, and other network devices. It 349 is also embedded in many leading file transfer solutions, systems 350 management solutions, identity management solutions, and privileged 351 access management solutions. It is widely used for integrating IT 352 systems and automating their operation. 354 2.1. The SSH Protocol 356 The SSH protocol is an IETF standards track protocol and is described 357 in RFC 4251 [RFC4251], RFC 4252 [RFC4252], RFC 4253 [RFC4253], and 358 RFC 4254 [RFC4254]. 360 Many independent commercial and open source implementations are 361 available, including Tectia SSH, OpenSSH, and many others. SSH is 362 available for nearly all platforms, including Linux/Unix, Windows, 363 mainframes, routers, telephone exchanges, mobile devices such as 364 smartphones and tablets, various embedded devices, and many 365 industrial automation systems. 367 In the SSH protocol, an SSH client application initiates a TCP/IP 368 connection over a network to a destination server, negotiates 369 encryption, authenticates the remote server, and then sends a 370 destination user name and authentication credentials to the server. 371 Server authentication is done using host keys, and its primary 372 purpose is to prevent man-in-the-middle attacks; however server 373 authentication is beyond the scope of this document. 375 2.2. User Authentication in SSH 377 The SSH protocol supports several mechanisms for authenticating 378 users, including passwords, public key authentication, Kerberos, and 379 host-based authentication. 381 2.2.1. Password Authentication 383 There are two kinds of password authentication mechanisms in SSH: 384 basic password authentication and keyboard-interactive 385 authentication. Keyboard-interactive authentication can support 386 various types of challenge-response systems and various other 387 authentication mechanisms. 389 Password authentication is commonly used for interactive users, but 390 less commonly for automated access (through it is sometimes seen with 391 hard-coded passwords in scripts and management systems, especially 392 for managing routers and file transfers). 394 2.2.2. Public Key Authentication 396 Public key authentication in SSH uses user keys or certificates to 397 authenticate/authorize a connection. An SSH client has an identity 398 key, typically an RSA or DSA private key, and the server must have 399 the corresponding public key configured as an authorized key for the 400 destination user. The private key may be protected by a passphrase, 401 in which case it is encrypted by a key derived from the passphrase 402 (passphrases are common for interactive users but rarely used for 403 automated access). 405 Many widely used SSH implementations support configuring restrictions 406 for SSH user keys. These may be used for limiting what can be done 407 on the server using the key (command restrictions) and for limiting 408 the IP addresses from which the key can be used (source 409 restrictions). 411 Public key authentication is by far the most frequently used method 412 of configuring automated access using SSH at the time of this writing 413 and represents best current practice. 415 2.2.3. Kerberos Authentication 417 Many organizations use Kerberos or Active Directory authentication 418 with SSH. Kerberos (usually together with LDAP) implements single 419 sign-on and allows user accounts to be stored in a centralized 420 directory. 422 In practice, Kerberos is rarely used for non-interactive accounts. 423 While it can be configured to use keytab files or cached tickets for 424 functional accounts, these approaches rely on having long-term 425 credentials stored on the host or at least accessible to the process 426 on the host that is obtaining tickets. These credentials can be 427 exploited by an attacker largely in the same way as SSH user keys, 428 e.g., by using them to obtain a ticket granting ticket (TGT) for the 429 functional account and using the ticket to gain access to other hosts 430 or accounts that the functional account can access (see virus spread 431 threat below). 433 One problem with Kerberos for automated access is that the single 434 sign-on feature implies that once access has been gained to one 435 account using Kerberos, it is usually possible to log in to any other 436 server that has the same account and is in the same domain without 437 further authentication. This can very easily create lots of unwanted 438 implicit trust relationships. Existing implementations also do not 439 support command restrictions for Kerberos. 441 2.2.4. Host-Based Authentication 443 Host-based authentication uses the source host's host key to 444 authenticate the source host and to vouch for the identity of the 445 user on the client side. It is rarely used and does not permit 446 configuring command restrictions. Therefore its use for automated 447 access is NOT RECOMMENDED. 449 2.2.5. Comparison of the Authentication Methods 451 All these authentication methods fundamentally rely on some secret 452 information, and when used for automated access, this secret 453 information must be stored on or accessible to the source host. 455 A major advantage of public key authentication over the other methods 456 is that it allows configuring a command restriction. The command 457 restriction can be used for preventing virus spread and other 458 attacks, as described in Section 3. It also does not create any 459 implicit trust relationships and the permitted access can be reliably 460 determined by inspecting the destination host (except for OpenSSH's 461 proprietary certificate authentication, which SHOULD NOT be used 462 because it cannot be reliably audited). For these reasons, public 463 key authentication is the RECOMMENDED authentication mechanism for 464 automated access with SSH. 466 Password authentication SHOULD NOT be used for automated access, 467 because hard-coded passwords may be obtained by attackers and 468 password vaults are also not particularly secure against local 469 attacks on the client software. If password authentication is used 470 for automated access, the passwords MUST be rotated every three 471 months. 473 2.2.6. Dangers of Unverified and Shared Host Keys 475 Many file transfer applications, privileged access management 476 systems, and systems management applications do not check host keys 477 for hosts that they connect to. This permits a man-in-the-middle 478 attack to be performed in the network. Many tools are available for 479 this and any device connected to a network through which the 480 connection goes can be used for the attack - including, e.g., 481 reprogrammed smart switches. 483 Man-in-the-middle attacks are a risk regardless of the authentication 484 method if hosts keys are not properly verified. The attack permits 485 injection of arbitrary commands into the session, and reading and 486 modifying any transferred files (including injection of bogus file 487 transfers). A successful man-in-the-middle attack from the network 488 gives the same power as being able to use a trust relation leading to 489 the destination host. 491 Such man-in-the-middle attacks are practical, and are exploited in 492 freely available attack tools and malware, as well as security 493 software from multiple vendors for co-operative auditing purposes. 495 Besides applications that do not check host keys, there are also some 496 large enterprise that share the same host key on thousands of 497 machines (for example, one Fortune 500 company is known to use the 498 same host key on 14000 computers at the time of this writing). If 499 any of the computers is compromised, they all become vulnerable to 500 man-in-the-middle attacks. 502 Therefore, while this document is not really about host keys, the 503 destination host MUST be properly authenticated by the client for all 504 automated access and a unique host key MUST be used for each host. 505 An exception may be made for the very first connection to a server to 506 simplify system administration. 508 2.3. Common Use Cases 510 2.3.1. Interactive Use 512 SSH has become the standard used by system administrators for 513 configuring and managing Unix and Linux computers, routers, and 514 various other equipment remotely. It is also widely used by software 515 developers, and in some organizations by ordinary end users for 516 running applications remotely (particularly text-based legacy 517 applications). Public key authentication is often used by advanced 518 end users for single sign-in. Sometimes it is also used on jump 519 servers. 521 2.3.2. Automated Access 523 SSH is very frequently used for automated access between systems. 524 Such automated access is necessary for cost-efficiently managing 525 large IT environments, for integrating applications, and for cost- 526 effectively provisioning virtual machines in cloud services. 528 Automated access refers to accessing a computer from another computer 529 in an automated fashion. Automated access may be unrestricted, 530 allowing any commands to be executed, or may be limited to specific 531 commands or operations, such as file transfers (perhaps limited to a 532 specific directory). 534 Automated access is most commonly used with functional accounts, 535 system accounts, service accounts, and other non-interactive user 536 accounts (sometimes also called non-user accounts). Such accounts 537 are used by operating systems, middleware, databases, and 538 applications for running processes. System or service accounts are 539 likely to have sensitive levels of access to system resources (in 540 that case they are often called privileged accounts). 542 Automated access using SSH is common also in Windows and mainframe 543 environments, especially for file transfer applications. There are 544 also various native mechanisms on Windows that can be used for 545 automated access, but such mechanisms are beyond the scope of this 546 document. 548 Automated access has been largely ignored in Identity and Access 549 Management. Hundreds of thousands to over a million authorized SSH 550 keys have been found from the IT systems of several large 551 enterprises. This means that they have many times more entry points 552 for automated access configured on their servers than they have 553 interactive users in the organization! It is clear that such entry 554 points cannot be ignored. 556 2.3.3. File Transfers 558 SSH is frequently used as a file transfer tool in itself, using the 559 "scp" and "sftp" tools. The SFTP [SFTP] protocol is gaining 560 popularity in commercial and open source file transfer products, and 561 a substantial fraction of the world's file transfers now use the SFTP 562 protocol. Automated file transfers using SSH typically use public 563 key authentication or hard-coded passwords, and they are often also 564 used between organizations. 566 3. Threats Arising from Poorly Managed Automated Access and SSH Keys 568 This section outlines some of the threats that have been identified 569 with poorly managed SSH keys. The guidelines and recommendations of 570 this document are intended to address these while minimizing the 571 administrative burden from managing the keys. 573 Several of the problems described below are present with many 574 technologies for automated access besides SSH keys. The issues must 575 be addressed regardless of technology. 577 3.1. Virus Spread Threat 579 Malware can be engineered to use SSH keys to spread to most servers 580 within an organization once it has penetrated one server. Experience 581 has shown that viruses frequently manage to penetrate individual 582 servers in an organization. Malware often uses multiple attack 583 vectors to penetrate an organization and could use SSH user keys (or 584 other trust relationships such as Kerberos) to spread within the 585 organization's server infrastructure in minutes after penetrating the 586 first server, thereby defeating layered security defenses. 588 The Morris worm in 1988 utilized automated access trust relationships 589 to spread in a similar manner (at that time based on ".rhosts" 590 authentication). This attack vector can be very powerful, and its 591 importance is increasing as systems management becomes more 592 automated. Many computer forensics experts are aware of cases were 593 SSH keys have been used to spread an attack from one server to 594 another, and several high profile incidents in the last year have 595 used SSH keys as an attack vector. 597 Experience has shown that most large organizations have 3 to 100+ SSH 598 keys configured granting access to each Unix/Linux server. Some keys 599 grant high-level administrative access or access to sensitive 600 accounts, such as those storing database files or critical software. 601 In practice it has been found that in many organizations 602 approximately 10% of SSH keys grant access to root accounts or other 603 privileged accounts. 605 The mesh of automated access relationships is so dense in many cases 606 that it is likely that an attack can spread to most servers in an 607 organization after penetrating the first few servers, especially if 608 other attack vectors are used to escalate privileges. 610 Implementing SSH keys as an attack vector in malware is quite easy, 611 requiring only a few hundred lines of code. Once the malware has 612 penetrated a server, it may use the server to further the attack and/ 613 or, e.g., leave a backdoor, steal, alter, or corrupt data, subvert 614 encryption systems or databases, or outright destroy the server. 616 The virus spread threat can be reduced by combining several 617 approaches: 619 Mandating forced command restrictions for as many trust 620 relationships as possible. 622 Removing trust relationships that are no longer needed. 624 Minimizing the number of trust relationships leading to root 625 accounts (directly or indirectly). 627 Minimizing implicit trust relationships arising from privilege 628 escalation (e.g., "sudo"), single-sign-on (e.g., Kerberos), and 629 host equivalence. 631 3.2. Unaudited Backdoor Threat 633 Many large organizations mandate that all privileged access to their 634 servers take place through a privileged access management system that 635 records any actions performed. Key-based access (and other automated 636 trust relationships) can be used for creating backdoors that bypasses 637 such privileged access management systems. 639 System administrators and production support personnel regularly gain 640 access to various accounts in the course of legitimate work. An 641 administrator or production support person may add a new authorized 642 key to an account with a single command (e.g., "echo ...keydata... 643 >>~/.ssh/authorized_keys"). As of this writing, most organizations 644 never audit authorized keys for their user accounts, and thus the 645 added key may remain unnoticed for years. Such a key can then be 646 used to log into the account using any SSH client, bypassing the 647 privileged access management system. It thus provides a relatively 648 permanent unaudited backdoor. 650 Key-based backdoors can also circumvent password vaults and systems 651 that change root (or other privileged account) passwords regularly. 653 Experience has shown that many organizations have no control or 654 tracking of trust relationship creation. Any system administrator or 655 production support personnel can create and install a user key pair 656 as needed without any reporting, logging, or authorization. Such 657 practice undermines traditional controls for privileges access. 659 The unaudited backdoor threat can be reduced by the following: 661 Prevent non-root/non-Administrator users from granting automated 662 access to accounts without proper approval. For example, move all 663 authorized keys files to root-owned directories that are not 664 writable by normal users. 666 Continuously monitor the environment to detect unauthorized trust 667 relationships configured by someone having root access. 669 Require proper explanation and valid purpose for the existence of 670 every trust relationship and remove any unused trust 671 relationships. 673 Use privileged access management systems that capture SSH and 674 other protocols using automated access on the network level or at 675 the server (transparent access auditing). Enforce privileged 676 access management for connections using automated trust 677 relationships. 679 3.3. Leaked Keys May Provide Access for Extended Periods 681 Most security policies and regulations mandate that all passwords 682 must be changed regularly, e.g., every three months. Some security 683 standards mandate that encryption keys must also be changed 684 regularly. However, very few security policies at this time make it 685 explicit that authentication/authorization keys should also be 686 regularly changed. In a sense, authentication keys are even more 687 critical than encryption keys, because once access to a user account 688 has been gained, it is generally possible to access and modify any 689 data for that user account - including reading and modifying memory 690 of processes running under that user account and/or modifying any 691 executable programs owned by that user account, thus subverting 692 encryption systems and other critical applications. 694 At the time of this writing, most organizations do not track which 695 SSH keys their users, administrators, backup operators, and janitors 696 may have had access to and copied over the years. In addition, they 697 never change their SSH keys. Most environments do not use source 698 restrictions on authorized keys. Therefore, a leaked key may be used 699 from any computer or network within the organization (unless limited 700 by internal firewalls). 702 This means that anyone who may have obtained a copy of a key (e.g., 703 by copying it from a host, accessing a backup, or having acquired 704 some decommissioned equipment that was not securely wiped) may gain 705 access to production servers in the organization. 707 No audit or discovery process can ever guarantee finding all copies 708 of identity keys, as they are small files that could be hidden 709 anywhere, and there could be copies on laptops, tablets, USB sticks, 710 offline, and even on paper (they are small enough to be typed in). 711 Identity keys can easily be taken out from an organization on a 712 single piece of paper, or by taking a photograph of a screen using a 713 cellphone. 715 There have also been many instances where private keys have been 716 uploaded to, e.g., "github" (a repository used by many open source 717 software development projects). In January 2013, hundreds of private 718 keys and passwords were found from the repository, some of which were 719 being used for attacks. Obviously, identity keys MUST NOT be 720 uploaded into any public repository. 722 The problems created by leaked or unchanged keys can be reduced by: 724 Rotating all keys regularly to guarantee the eventual termination 725 of access. 727 Configuring source restrictions for authorized keys, making it 728 more difficult to exploit copied identity keys. 730 Using certificate-based authentication, which can provide 731 revocation and expiration, but is cumbersome to manage (and still 732 does not protect from some of the other threats). 734 Using Kerberos authentication, which allows terminating access to 735 the account; however, Kerberos authentication does not in itself 736 prevent leaked keys that have not been changed from being used. 738 Besides rotating keys at regular intervals to avoid their leakage and 739 to limit the duration of the exploitation window should they leak, 740 periodic rotation also applies to credentials used for obtaining 741 actual authentication credentials; for example, it is not enough to 742 periodically obtain a new Kerberos ticket - one must also regularly 743 change the authentication credentials used for obtaining an initial 744 ticket. It is also not enough to issue a new certificate for the 745 same private key - the private key must also be replaced by a newly 746 generated private key. 748 3.4. Lack of Proper Termination of Access 750 Most security standards mandate proper termination of access when an 751 employee leaves or changes roles. If the user remains in possession 752 of identity keys that continue to have access to the organization's 753 information system, access is not being properly terminated. 755 Since administrators can quite easily copy identity keys (and may 756 have legitimately configured key-based access from their personal 757 account to various accounts they used in their previous role), the 758 only practical way to guarantee proper termination of access is to 759 remove or rotate any keys that the employee may have had access to. 761 At the time of this writing, most organizations do not know what each 762 key is used for. Without this knowledge, they seldom remove or 763 rotate keys, because something could break if they accidentally 764 remove a key that is needed for some important business process or 765 omit some authorized keys corresponding to a public key being 766 rotated. 768 Some organizations have used manual spreadsheets for tracking key 769 usage. However, it has turned out they are usually out of date, 770 inaccurate, and have not been maintained throughout the organization. 771 Many organizations have no monitoring whatsoever of automated access 772 or new user key setups. 774 Proper termination of access can be ensured by: 776 Moving all authorized keys files to root-owned directories that 777 are not writeable by non-privileged users. 779 Regularly rotating keys (ensures termination of access by copied 780 keys latest at next key change). 782 Triggering immediate key rotation for private keys on accounts 783 accessible by the person whose access is being terminated. 785 3.5. Use for Unintended Purpose 787 Firewalls commonly have rules that permit specific communications for 788 file transfer purposes. When the file transfer is using SSH (or 789 SFTP), it is important that a forced command be used on the server to 790 ensure that the access permitted through the firewall cannot be used 791 for other purposes, such as executing shell commands on the server. 793 Another related use case is employees creating their own backdoors 794 into the enterprise to circumvent corporate policies against 795 uncontrolled remote access by opening an SSH connection from the 796 office to their home machine with a port forwarding from the home 797 machine back to the office machine. Such backdoors may provide 798 hackers an entry point into the company intranet, especially if the 799 home machine is compromised and the user's password is obtained 800 using, e.g., key logger malware. 802 Various commercial products are available for auditing SSH 803 connections at a firewall to enforce that opened ports are not used 804 for unintended purposes regardless of server configuration. 806 Various SSH implementations permit port forwarding even when forced 807 commands are used. Therefore, a trust relationship that is intended 808 only for file transfer may actually be used to obtain a connection to 809 any port at any host on the internal network (or external network, 810 for hiding the source of an attack). 812 The threat of unintended use can be minimized by: 814 Allowing SSH/SFTP through a firewall only when required and 815 restricting sources and destinations to fewest systems required. 817 Configuring forced commands for authorized keys used by external 818 parties. 820 Implementing tools to audit SSH connections at the firewall and 821 monitoring use to ensure that access was not misused. 823 Avoiding trust relationships that cross security boundaries or 824 allow connections from low-impact systems to higher-impact 825 systems. 827 3.6. Accidental Data Transfers and Human Errors 829 Not all risks associated with unmanaged automated access arise from 830 malicious behavior. If there is automated access from non-production 831 systems to production systems, data may accidentally be copied from 832 non-production systems into production systems, where the incorrect 833 data may cause substantial loss of money. Alternatively, data may be 834 inadvertently copied from production systems to non-production 835 systems, where the data may be exposed due to looser security 836 controls. 838 People are also known to make human errors when manually setting up 839 new trust relationships. For example, it is fairly easy for a 840 security administrator to accidentally copy an authorized key to the 841 root account on a host instead of some other account that was 842 intended. Such errors can go undetected for years. 844 Some key setups involve thousands of hosts. It is easy to miss one 845 or more hosts when copying an authorized key to so many hosts 846 manually. Debugging such errors can be very time consuming. Also, 847 when manually fixing such problems, security administrators are 848 likely to just copy the missing keys to the proper accounts, without, 849 for example, checking whether they have accidentally been copied to 850 the root account. 852 The threat of accidents and human errors can be minimized by: 854 Automating key provisioning to implement the authorized keys 855 exactly as they were requested and approved. 857 Configuring source and command restrictions for authorized keys. 859 Enforcing policies for preventing trust relationships between 860 systems that cross security zone boundaries. 862 3.7. Problem Under Radar 864 The SSH key management problem has been recognized in various circles 865 for some time. The scope of the problem, and its relation to 866 automated access overall, however, has not been widely understood. 868 The problems have remained under the radar because they are deeply 869 technical and obscure, within the domain of system administrators. 870 Each system administrator typically only sees a small corner of the 871 IT environment and does not have a full picture. Although 872 administrators and their managers may recognize that there is a 873 problem, they simply have not had time to analyze the scope or 874 possible implications of the problem. There have also been no 875 guidelines or training materials on how to address it. 877 Most IT auditors do not have SSH key management or automated access 878 more generally on their checklists or audit programs, yet it is 879 central to identity and access governance given the prevalence of 880 automated access entry points to systems. 882 SSH keys, or control of credentials for automated access more 883 generally, has not been sufficiently highlighted in security control 884 frameworks and auditing guidelines for FFIEC, SOX, PCI, FISMA, HIPAA, 885 NERC, or COBIT. Even many CISOs are only vaguely aware of the 886 problem, and many CIOs and IT risk management professionals have 887 never heard of it. 889 Training, books, and systems in the identity and access management 890 space have largely only dealt with actual human users and control of 891 interactive access by people. Automated access by machines has been 892 largely ignored, despite many organizations now having many times 893 more credentials for automated access to their systems than they have 894 interactive accounts. 896 Despite the risk, the problem will likely not be addressed until IT 897 security auditors, IT operations managers, security architects, 898 CISOs, and IT risk management professionals understand the issue. It 899 must be addressed in security regulations, guidelines, standards, and 900 internal security policies. Education and training will also be 901 needed to ensure that the SSH key management problem and remediation 902 actions are understood. It must be evaluated during audits to ensure 903 that action is taken. 905 4. Assessing the SSH Key Management Situation and Risks 907 Addressing threats related to automated access and SSH keys begins 908 with understanding the extent to which automated access and SSH keys 909 are used in an organization, understanding how they are managed, and 910 identifying areas likely to require further attention. 912 Risks associated with SSH key management are generally relevant for 913 organizations where at least one of the following is true (the list 914 is not exhaustive, and other automated access technologies affect 915 other organizations): 917 significant number of Unix or Linux systems; 919 significant use of SSH or SFTP on Windows or Mainframe; 921 virtualization or cloud services that are internally managed using 922 SSH (possibly inside automated scripts/tools/frameworks); 924 web server farms that are managed over SSH; 926 network devices (e.g., routers, switches, xDSL models, firewalls) 927 configured and managed using SSH and/or automated management 928 systems; 930 significant number of automated file transfers using SFTP; 932 password management or privileged access management tools using 933 SSH to connect to end servers; or 935 systems management tools using SSH to communicate with managed 936 systems. 938 Results of the scoping phase help estimate risk exposure and the 939 probability of non-compliance with mandatory regulations. This 940 information also helps auditors and decision-makers determine whether 941 a more detailed discovery and remediation project is warranted. 943 Certain types of relatively easily obtainable information are useful 944 in understanding the scope of the problem in an organization. This 945 information may easily be obtained as part of an audit or regular 946 review. 948 Some preliminary indicators about the level of risk can be obtained 949 by reviewing the sshd_config file for a sample of SSH servers. The 950 AuthorizedKeyFile parameter indicates the location of files that 951 store user's authorized key files. If the AuthorizedKeyFile is 952 located within the user's home directory (which is the default), then 953 it is likely that a significant problem exists because users are able 954 to provision new trust relationships. On the other hand, if the 955 AuthorizedKeyFile is defined within the /etc directory, for example, 956 then the risk of inappropriate trust relationships is significantly 957 lessened. Another significant configuration parameter is 958 PermitRootLogin. A value of "yes" or "without-password" indicates 959 that SSH keys can be used for root access, which significantly 960 increases the potential impact of poorly managed keys. However, if 961 this parameter is set to "no" or "forced-commands-only", then the 962 potential impact is substantially lessened since interactive root 963 access is disallowed. 965 Examining authorized keys provides a meaningful indication of the 966 level of risk. The following metrics generally give insight into 967 whether an organization is affected by the issue: 969 total number of authorized keys in the environment; 971 total number of authorized keys in the environment that grant 972 access to a root account (any account with user id 0); 974 total number of authorized keys in the environment that grant 975 access to a system account or service account; 977 total number of authorized keys without a command restriction; and 979 total number of non-root service accounts or system accounts that 980 have access to add new authorized keys at will. 982 There are also a number of scoping questions that can give insight 983 into the severity of the problem. These questions are well-suited 984 for a questionnaire or interview of knowledgeable personnel. An 985 affirmative answer indicates that SSH keys are being managed; a 986 ``no'' indicates that risk exists. The questions are provided below: 988 Does installing a new authorized key require approval from a 989 system resource owner or authorized manager 1) for keys granting 990 root access, 2) for keys granting access to non-root service 991 accounts or system accounts, or 3) for keys granting access to 992 interactive user accounts? 994 Are such approvals enforced through the provisioning process? 996 Are non-root users technically prevented from installing new 997 authorized keys, e.g., by moving the authorized keys files to 998 root-owned locations? 999 Has this configuration been verified to be the case 1) across all 1000 high-risk systems and 2) all moderate-impact systems? 1002 Is a continuous monitoring process in place for detecting 1003 authorized keys that are added outside of the provisioning process 1004 and without proper approvals 1) for root accounts, 2) for service 1005 and system accounts, and 3) for interactive user accounts? 1007 Is a policy in place for rotating SSH user keys? Are monitoring 1008 procedures in place to verify that all user keys have actually 1009 been rotated in accordance with the policy (and the private keys 1010 actually changed)? 1012 For all authorized keys for the root account (and critical system 1013 and service accounts), can the organization easily identify who is 1014 using the keys to connect? 1016 Has the reason of existence for every authorized key, including 1017 the application or business process it relates to, been documented 1018 1) for high-impact systems, 2) moderate-impact systems, and 3) 1019 low-impact systems? 1021 Are SSH keys systematically removed when they are no longer 1022 needed? 1024 Do all authorized keys used for external SFTP/SCP file transfers 1025 with other organizations have a command restriction? 1027 Are command restrictions enforced for trust relationships leading 1028 to moderate-impact and high-impact systems? 1030 Preliminary scoping information can be obtained relatively easily and 1031 scoping questions can be answered without having to install new 1032 software on servers. However, the answers are only approximate. 1033 Experience has shown that many organizations do not know clear or 1034 definitive answers to the questions, and sometimes management 1035 perceptions do not match reality. Therefore the answers are best 1036 obtained and analyzed as part of a regular audit that actually 1037 verifies the answers, at least by representative sampling. 1039 Another interesting diagnostic exercise to gauge the level of risk is 1040 to obtain listings from a few servers of the public keys (or 1041 signatures) from the authorized keys file of root and other sensitive 1042 accounts (such a system or service accounts). If the organization 1043 can readily identify who is using those keys (or could use the keys) 1044 to connect and why, then it is likely that the organization is 1045 effectively managing SSH users key for access control. If the 1046 organization is unable to identify who can use keys to obtain 1047 sensitive access to systems, then the access control problem and risk 1048 is self-evident. 1050 Freely available scripts and tools for doing a proper scoping 1051 analysis are available at http://www.ssh.com/auditing [1]. The 1052 scripts and tools will be useful for internal security professionals, 1053 system administrators, and auditors working with customers. 1055 5. Key Remediation Solution Planning and Deployment 1057 Once it has been determined that further analysis of automated access 1058 and/or SSH keys in an organization is warranted, the organization 1059 typically engages in a multi-step process consisting of additional 1060 discovery and remediation of existing trust relationships, 1061 establishment of appropriate policies, and continuous monitoring to 1062 ensure that automated access is only enabled in accordance with 1063 policy. 1065 A typical key remediation process consists of: 1067 discovering all existing trust relationships based on SSH keys 1068 (and other trust relationships, if applicable); 1070 moving authorized keys files to protected locations to prevent 1071 non-root users from adding new authorized keys; 1073 monitoring use of trust relationships and authorized keys in the 1074 existing environment; 1076 removing trust relationships that are no longer in use; 1078 associating each trust relationship with an application or some 1079 other valid purpose; 1081 implementing an approval process for setting up new trust 1082 relationships; 1084 rotating existing SSH user keys; 1086 configuring forced command restrictions on authorized keys; and 1088 configuring IP address restrictions on authorized keys. 1090 While it is possible to perform all the remediation steps manually, 1091 in a larger environment the use of software tools to assist in the 1092 process can save a huge amount of work. Parts of the process can be 1093 fairly labor-intensive, for example, associating each trust 1094 relationship with an application or valid purpose may require a 1095 substantial amount of manual work, and removing of unused trust 1096 relationships needs to be done with care to avoid any problems with 1097 critical business applications. (See Section 8 for more 1098 information.) 1100 Automating management of SSH keys and other trust relationships can 1101 also bring substantial cost savings. Many organizations spend a 1102 substantial amount of administrator time setting up and maintaining 1103 trust relationships, and the cost of such manual key management can 1104 often be eliminated by automating the process. Ideally, new trust 1105 relationships are approved in the organization's normal security 1106 entitlement approval system and automatically implemented throughout 1107 the IT environment by software for managing SSH authorized keys and/ 1108 or other trust relationships. Automation can also reduce human 1109 errors and radically reduce the number of administrators requiring 1110 root access on servers. (See Section 8.1.) 1112 In some environments it may be desirable to prohibit public key 1113 authentication for interactive logins to ordinary user accounts. 1114 This can help enforce ordinary interactive logins to go through a 1115 privileged access management system (unless some administrators have 1116 copied private keys, which is generally possible). 1118 5.1. Discovering SSH Keys and Trust Relationships 1120 The purpose of the discovery phase is to obtain reliable and 1121 reasonably complete information about configured SSH keys and trust 1122 relationships throughout an IT environment. Discovery should ideally 1123 include all Unix/Linux systems, Windows systems (at least those 1124 running SSH servers, SSH clients, or file transfer solutions running 1125 SFTP), Mac servers, workstations, laptops, mainframes, and other 1126 systems using the SSH protocol, including file transfer solutions 1127 using SFTP, virtualization platforms, and privileged access 1128 management gateways. 1130 Since it is not possible to know what trust relationships exist in an 1131 IT environment without scanning all systems for SSH authorized keys 1132 and identity keys, all organizations SHOULD perform initial discovery 1133 of SSH user keys on all systems that use the SSH protocol. 1135 Although some organizations may want to focus only on high-impact 1136 systems, a limited scope discovery process provides only limited 1137 visibility into the security risk of the current environment. It is 1138 important to identify which low-impact systems can access high-impact 1139 systems, and this can only be known by scanning low-impact systems 1140 too. An identity key intended to allow access between two high- 1141 impact systems could be copied onto a low-impact system. Unless 1142 source restrictions are defined for the authorized key, the identity 1143 key can be used from the low-impact system to access the high-impact 1144 system. Therefore, the scope of the discovery process SHOULD include 1145 all systems in the network, including even low-impact systems. 1147 Ideally, routers, BIOS management ports, and other specialized 1148 computing devices should also be included, but at the time of this 1149 writing, software is not yet available for full SSH key discovery for 1150 these devices. This is expected to change in the future. It is also 1151 be possible to audit them manually. 1153 The following MUST be determined during discovery: 1155 Configured authorized keys for all user accounts on all servers of 1156 interest (accounts may be local, in LDAP, in Active Directory, in 1157 NIS, or any combination) 1159 Configured identity keys on all user accounts on all servers and 1160 clients (workstations, laptops, etc) of interest. It should be 1161 understood, however, that one can never be certain that all 1162 identity keys have been found, because some could be copied into 1163 non-standard directories, stored offline, or even printed on 1164 paper. Thus not finding an identity key on a particular system 1165 does not guarantee that the key will not eventually be used from 1166 that system later. On a broader scale, even if the discovery 1167 process fails to find a particular identity key anywhere on the 1168 network, this does not necessarily mean that the key cannot be 1169 used later. 1171 Configured restrictions for each authorized key, such as command 1172 restrictions and source restrictions 1174 Established trust relationships (source host, source account, 1175 destination host, destination account, and restrictions) 1177 The following SHOULD be determined during discovery (these may become 1178 MUST in the future): 1180 Kerberos-based trust relationships for automated access, 1181 particularly access using keytab files, cached tickets, or service 1182 processes holding tickets. 1184 Implicit trust relationships arising from Kerberos single sign-on. 1186 Implicit trust relationships arising from sharing the same home 1187 directory across multiple accounts. Many organizations share the 1188 same home directory for multiple accounts. Adding an authorized 1189 key for any of the accounts implies adding it to all of the 1190 accounts. Thus, any accounts that can write to the shared home 1191 directory effectively have access to all accounts sharing the home 1192 directory. 1194 Trust relationships configured using host-based authentication 1195 (".shosts", ".rhosts", "hosts.equiv", or "shosts.equiv"). 1197 The following MAY be determined during discovery (some of these may 1198 become SHOULD or MUST in the future): 1200 Trust relationships configured using password authentication 1201 (whether hard-coded in scripts or in password vaults). (In 1202 practice it may be difficult to do this reliably. However, it may 1203 be possible to, e.g., recognize certain syntactic patterns and 1204 commands from scripts as using hard-coded password 1205 authentication.) 1207 Implicit trust relationships configured using "sudo" or some other 1208 privilege escalation tool. 1210 Implicit trust relationships arising from user accounts that have 1211 NFS-mounted home directories. NFS is usually not configured to 1212 provide security against network-level attacks, and an attacker 1213 who gains access to a network segment may be able to read and 1214 modify the NFS traffic of any host on the network and impersonate 1215 any other host or user on the network (including reading and 1216 modifying any file on NFS file systems). When home directories 1217 are stored in NFS, a sophisticated attacker with root-level access 1218 to any device on the network (e.g., any unconfigured smart switch 1219 where the attacker can replace the firmware) may add new 1220 authorized keys to any home directory in an NFS file system. 1222 Implicit trust relationships arising from user accounts that have 1223 home directories on a Windows share. Windows file sharing (CIFS, 1224 Common Internet File System) may suffer from same issues as NFS, 1225 and thus the same considerations may also apply to it. 1227 It is important to understand that even though the discovery process 1228 finds keys, and in the short term most trust relationships are 1229 configured using SSH keys, the primary concern is not with the keys 1230 themselves or the underlying cryptography. Rather, the primary 1231 purpose of discovery is to determine who (or what) can access what 1232 and how such access is restricted. In terms of cryptography, all SSH 1233 keys created with default settings since version 1.0 use the 1234 equivalent of at least 1024 bit RSA key, which is still relatively 1235 safe, especially since servers never disclose authorized keys 1236 (cryptographic attacks on the key would first require access to an 1237 authorized keys file, which usually already means access to the 1238 host). Thus verifying key sizes or algorithms is not critical 1239 (though may be required by policy); however, knowing who (or what) 1240 can access what is critical and addresses a real security concern. 1241 SSH user keys grant access and are fundamentally authentication 1242 credentials, rather than encryption keys. The whole issue is 1243 fundamentally not an encryption key problem, but an identity and 1244 access management problem! 1246 Doing discovery properly is complicated. At least the following 1247 aspects need to be properly considered when planning discovery: 1249 SSH user keys cannot be discovered by a network scan because the 1250 SSH protocol was designed not to reveal authorized keys. It is 1251 possible to query whether an already known key is acceptable for a 1252 particular user, but an SSH server will not reveal an authorized 1253 key that is otherwise unknown. This means that the discovery 1254 process will need to connect to each host and access the 1255 authorized keys through the file system. (Host key discovery, on 1256 the other hand, is possible over a network, but host keys are 1257 beyond the scope of this document.) 1259 Different SSH versions are commonly deployed. Many large 1260 organizations have some combination of OpenSSH, Tectia SSH, 1261 SunSSH, Reflection for Secure IT, and various other products. Not 1262 all SSH implementations use the same key formats, store SSH keys 1263 in the same locations, or use the same key fingerprint format. 1264 Furthermore, OpenSSH comes in many flavors and patch set 1265 combinations, and some vendors pack a version of OpenSSH with 1266 another product - sometimes without providing a proper way of 1267 identifying the particular version. The discovery process (and 1268 tools) should be able to properly analyze keys and trust 1269 relationships for any SSH version that is deployed in the 1270 environment. 1272 Many organizations use the "root_squash" option for NFS exports, 1273 which converts file system accesses by root to accesses by an 1274 unprivileged account "nobody". A side effect of this is that a 1275 key discovery process running as root may not be able to read SSH 1276 keys in NFS home directories. 1278 Systems using SELinux may not allow reading SSH authorized key 1279 files or identity key files by ordinary processes. Reading such 1280 files may require special authorization or the use of special 1281 programs, such as "ssh-keycat". 1283 In a large environment, some servers are down for maintenance at 1284 any time, and in a geographically dispersed organization some 1285 networks may not be reachable at the time the discovery is 1286 initially run. Thus, discovery cannot be assumed to succeed on 1287 all servers the first time. This problem is compounded for 1288 discovery of SSH clients since laptops may be disconnected from 1289 the network and therefore be unreachable for scanning. 1291 5.2. Moving Authorized Keys to Protected Locations 1293 Moving authorized keys to protected locations, or locking down 1294 authorized keys, MUST be performed on all moderate-impact and high- 1295 impact information systems (including systems having automated access 1296 to such systems). It MAY be performed on low-impact information 1297 systems. 1299 Moving authorized keys to a protected location may be performed, 1300 e.g., by copying authorized keys for each user to a root-owned 1301 directory, and modifying the system-wide SSH server configuration 1302 file to specify the authorized keys file path (typically using a 1303 pattern that refers to a user name). 1305 Failure to move authorized keys to protected locations allows system 1306 administrators and other users with legitimate access to create new 1307 trust relationships as unaudited backdoors and makes ensuring 1308 termination of access very difficult. Locking down authorized keys 1309 files helps to enforce the requirement that new trust relationships 1310 be properly approved. (See Section 5.3 for discussion of using an 1311 approval process for setting up new trust relationships.) 1313 It is important to lock down authorized keys files early in the 1314 remediation process to create a stable environment for discovery. 1315 Otherwise, inappropriate authorized keys that continue to be added 1316 may not identified during the discovery process. 1318 Similarly, authorized keys MUST be moved away from home directories 1319 susceptible to active network-level attacks (e.g., unencrypted NFS 1320 and CIFS home directories - in practice this includes most NFS home 1321 directories today) on all moderate-impact and high-impact systems 1322 (including systems having unrestricted automated access to such 1323 systems). It is RECOMMENDED that the same be performed on low-impact 1324 information systems. 1326 Failure to move authorized keys away from NFS and CIFS home 1327 directories may allow a network-level attacker (whether human or 1328 automated) to add new authorized keys to any account that accepts 1329 authorized keys from such a directory, permitting unrestricted access 1330 to such accounts. Such attacks are known to have been performed by 1331 some penetration testers and are certainly within the capabilities of 1332 Advanced Persistent Threat (APT) groups. 1334 Furthermore, identity key files SHOULD be moved away from home 1335 directories susceptible to network-level attacks (e.g., unencrypted 1336 NFS and CIFS home directories) on all moderate-impact and high-impact 1337 systems. Otherwise, an attacker who gains privileged access one one 1338 host on the network may be able to read identity keys from user's 1339 home directories and use them for attack (this technique is known to 1340 have been used by attackers). It is RECOMMENDED that the same be 1341 performed on low-impact systems. 1343 Failure to move identity keys away from NFS and CIFS home directories 1344 may allow a network-level attacker (whether human or automated) to 1345 obtain copies of identity keys for later use or for immediately 1346 furthering the attack to other systems. It substantially increases 1347 the risks associated with leaked keys and substantially expands the 1348 group of people who may be able to obtain copies of identity keys. 1349 Some penetation testers are known to use this technique for attacks. 1351 5.3. Monitoring Use of Trust Relationships and Keys 1353 After the initial discovery phase, the environment SHOULD be 1354 monitored for some time (preferably several months) to collect data 1355 on how authorized keys are actually used. While the process can be 1356 performed manually in a small environment, use of scripts or 1357 commercial tools is highly recommended in larger environments. 1358 Software tools can gather and correlate log data from many hosts to 1359 determine the following types of information: which keys are 1360 currently being used, which source hosts they are used from, which 1361 keys are external keys, and what commands they are used with. This 1362 information about the use of trust relationships will help the 1363 organization in later phases of remediation, such as deciding which 1364 authorized keys will be removed for non-use (see Section 5.4) and 1365 which keys can be easily configured with command and source 1366 restrictions (see Section 5.8 and Section 5.9). In addition, 1367 information gathered during monitoring will help the organization 1368 understand automated access patterns in the existing environment and 1369 further evaluate the concrete risks. 1371 Monitoring use of trust relationships may involve configuring SSH 1372 servers and clients to use a higher level of logging and collecting 1373 and analyzing log data. To determine which authorized keys are still 1374 being used, for example, an organization can configure SSH servers 1375 with a log level that causes the fingerprints of keys used for public 1376 key authentication to be logged, collect such log data for an 1377 extended period (several weeks to an year), and analyze the data to 1378 determine which authorized keys were actually used during the log 1379 collection period. 1381 On SSH clients, identity keys that have not been accessed for a long 1382 time (according to file system's file access timestamps) are also 1383 good candidates for removal. However, many programs and commands may 1384 access identity key files, and a recent access time does not 1385 necessarily mean that the key was actually recently used for 1386 authentication. Furthermore, the identity key file timestamp 1387 provides no information regarding the destination host for the last 1388 connection. An identity key may still be in use for some destination 1389 host where it is authorized but not for another. 1391 It should be noted that orphaned keys (authorized keys without a 1392 corresponding identity key) may be either unused or external keys. 1393 Thus they SHOULD NOT be removed without monitoring, as if they are 1394 external keys, trust relationships with hosts outside the managed 1395 environment could be inadvertently broken. 1397 From a project management perspective, the monitoring period can well 1398 be used for assigning impact levels to systems, defining internal 1399 boundaries, and defining host groupings. In practice in large 1400 environments, different parts of the IT environment may be at 1401 different stages of remediation during the project. However, some of 1402 the remediation steps require a reasonably complete picture from 1403 longer-term monitoring before they can be safely performed. 1405 Identifying trust relationships crossing certain boundaries, such as 1406 access from test and development systems into high-impact production 1407 systems, is of high interest to auditors and security managers. 1408 Detecting and controlling such unwanted access is an important audit 1409 objective. This information generally becomes available with 1410 reasonable certainty during the monitoring phase, after internal 1411 boundaries have been configured and impact levels for information 1412 systems determined. 1414 Information collected during the monitoring stage will be helpful in 1415 later stages of a remediation project. The information gathering 1416 takes some time to get a reliable picture. It is RECOMMENDEED that a 1417 sufficient period of time be given for monitoring use of keys: at 1418 least 3-6 months or even a year. The remediation project should not 1419 be rushed, as it increases risk of having incomplete information 1420 about existing trust relationships or external keys, which in turn 1421 increases the risk that remediation activities may disrupt 1422 operations. 1424 Failure to perform the monitoring step properly increases risk of 1425 disruption when unused keys are removed (see Section 5.4), and may 1426 even make removing unused keys impossible (one cannot remove unused 1427 keys without knowing with a high degree of certainty which keys are 1428 unused). Relying solely on the knowledge of application teams and 1429 managers to identify unused keys is generally insufficient due to the 1430 large number of legacy trust relationships, personnel changes, and 1431 poor documentation of trust relationships. 1433 Failure to perform the monitoring step properly also risks missing 1434 some external trust relationships and external keys. This may cause 1435 key rotation (see Section 5.7) to break external connections with 1436 systems outside the managed environment, such as data transfers with 1437 suppliers, contractors, distributors, or regional offices. 1439 5.4. Removing Trust Relationships That Are No Longer Used or Otherwise 1440 Inappropriate 1442 Various security standards and prudent information security require 1443 that access to information systems must be properly terminated when 1444 it is no longer needed. If trust relationships for automated access 1445 are left enabled on systems when no longer needed, they accumulate. 1446 Real-world experience has shown they sometimes accumulate at the rate 1447 of dozens of incoming trust relationships per year per system, even 1448 in security-sensitive environments. 1450 Therefore, all organizations MUST remove trust relationships leading 1451 to moderate-impact or high-impact information systems (including low- 1452 impact systems having automated access to such systems) that are no 1453 longer needed as part of the initial remediation process. Unused 1454 trust relations leading to low-impact information systems SHOULD be 1455 removed. 1457 Unused keys SHOULD NOT be removed until it is known with reasonable 1458 certainty which keys are really unused. This is usually accomplished 1459 by monitoring key usage over a period of time (see Section 5.3). It 1460 is further RECOMMENDED that unused trust relationships be reviewed by 1461 respective application owners to reduce the possibility of disruption 1462 from removal of a trust relationship that is actually needed. It is 1463 important to test infrequently used functionality, such as disaster 1464 recovery systems, reasonably soon after removing unused trust 1465 relationships. 1467 It is also likely that the remediation process will identify many 1468 trust relationships that are still being used but that have no 1469 legitimate business purpose (see Section 5.5), that cross configured 1470 boundaries in inappropriate ways, or that lead to accounts (e.g., 1471 root) that should not be accessible. Such inappropriate trust 1472 relationships should also be removed (and alternate steps taken to 1473 implement any functionality they support in a more appropriate way, 1474 if required). Some of such trust relationships may have been created 1475 by attackers and may warrant further forensics investigation, such as 1476 identifying when they were created and by whom. 1478 Failure to remove authorized keys for unused trust relationships 1479 increases the risk that key-based attacks for unauthorized access may 1480 succeed and spread throughout the network, allows previously created 1481 unaudited backdoors (using keys that are not regularly used) to 1482 remain in existence, and allows leaked keys (that are not regularly 1483 used) to remain usable indefinitely (if not rotated). 1485 5.5. Associating Trust Relationships with Application and/or Purpose 1487 Because authorized keys provide access to systems, their existence 1488 should be understood, justified, and controlled just like any other 1489 form of access control. After trust relationships that are not used 1490 have been removed, it is important to analyze the remaining active 1491 trust relationships to distinguish those authorized keys that support 1492 a valid application or business purpose and those keys that do not. 1493 Just because a trust relationship is being used does not actually 1494 guarantee that it is needed or legitimate. It may be used, e.g., by 1495 an attacker, by a user who created an inappropriate authorized key as 1496 a backdoor for access, or by a stale cron job relating to a 1497 decommissioned application. Determining the purpose of trust 1498 relationships is important for detecting such illegitimate trust 1499 relationships so that they can removed (see Section 5.4). 1501 After having removed unused authorized keys, the existence of every 1502 remaining incoming trust relationship MUST be justified for moderate- 1503 impact and high-impact information systems (including low-impact 1504 systems having access to such systems). 1506 This is an area where the justification can be more lax on low-impact 1507 systems. However, many low-impact systems, such as those used for 1508 internal software development or packaging, may generate binaries or 1509 distributions that later get installed on production systems. An 1510 attacker could use such systems to gain access to production servers, 1511 especially in an Advanced Persistent Threat (APT) scenario. It is 1512 thus RECOMMENDED that even access to low-impact systems be prudently 1513 justified, or alternatively systems with data/code paths to 1514 production be treated as moderate-impact or high-impact systems. 1516 One practical approach for low-impact systems may be to review 1517 discovered incoming trust relationships in bulk (perhaps for a group 1518 of hosts belonging to the same business process), and label them as 1519 legacy trust relationships relating to the associated business 1520 process. While not ideal, such an approach may make a reasonable 1521 compromise between cost and security in many environments. 1523 It is RECOMMENDED that each trust relationship be associated with the 1524 business process and/or application that it supports, and that the 1525 application/business purpose be documented for future reference. 1527 This will help with removing trust relationships when applications 1528 are replaced and business processes re-engineered, and serves to 1529 assign responsibility for each trust relationship to somebody (the 1530 business process or application owner). Assigning trust 1531 relationships to applications or business processes effectively 1532 assigns ownership for the trust relationships (and related authorized 1533 keys) to the application or business process owner (or group). This 1534 owner MAY be permitted to approve new trust relationships leading to 1535 the same account on the same host, MAY schedule rotation of keys, and 1536 MAY be asked to periodically validate the existence of each trust 1537 relationship relating to the application or business process. 1539 Failure to associate trust relationships with a purpose, business 1540 process, or application means that there remains access to 1541 information systems without reason or justification. Illegitimate 1542 backdoors may remain unnoticed and unnecessary trust relationships 1543 may remain in place that can be used by attackers, especially if keys 1544 are leaked (and not rotated). 1546 5.6. Implementing Approval Process for Setting Up New Trust 1547 Relationships 1549 Real-world experience has shown that many enterprises do not have a 1550 well-defined process for approving new trust relationships for 1551 automated access, and almost no enterprise today systematically 1552 enforces or audits approvals for automated access. 1554 Organizations MUST implement an approval process for ensuring the 1555 validity of new trust relationships granting access to moderate- 1556 impact and high-impact information systems. It is further 1557 RECOMMENDED that organizations implement an approval process for 1558 trust relationships granting access into low-impact systems. This 1559 reduces risk of low-impact systems being used for staging attacks 1560 into high-impact systems. See also Section 8.1 for ideas on how 1561 automated setup of approved trust relationships can reduce costs. 1563 New trust relationships SHOULD NOT be approved without a proper 1564 justification and association with a business process, application, 1565 or other valid purpose. In addition, the approval for new trust 1566 relationships MUST specify any command or source restrictions that 1567 should be implemented to limit security exposure. (See Section 5.8 1568 and Section 5.9) 1570 The approval process SHOULD carefully review whether trust 1571 relationships violate internal boundaries, such as allowing access 1572 from test or development systems into production or allowing access 1573 from low-impact systems into high-impact systems. Documented 1574 justification for crossing boundaries and secondary approval by 1575 higher-level management SHOULD be required for such trust 1576 relationships. Organizations MAY also use the approval process to 1577 enforce other internal boundaries, such as those between business 1578 units or functions, or "Chinese walls" between, e.g., retail banking 1579 and investment banking. 1581 Special approvals SHOULD also be required for trust relationships 1582 leading to root accounts or other highly privileged accounts on 1583 moderate-impact and high-impact systems. 1585 The approval for each such trust relationship MUST be documented and 1586 MUST be retained for later audit. Approvals MUST be organized so 1587 that it is possible find the approval for each new authorized key. 1589 Required approvals for new authorized keys MUST be enforced so that 1590 users cannot bypass the approval process. Enforcement typically 1591 involves both continuous monitoring and securing authorized keys 1592 files: 1594 Continuous monitoring (as discussed in Section 6) is needed to 1595 detect trust relationships that were implemented without approval. 1596 Existing trust relationships must be regularly audited against 1597 approved trust relationships. This requires periodically re- 1598 performing discovery to find all existing trust relationships so 1599 that they can be compared against a database of approved trust 1600 relationships. If software tools are used to perform this 1601 auditing, enforcement may be performed in real time or very 1602 frequently, e.g., once an hour or once per day. 1604 Another way to help enforcement is to move all authorized keys to 1605 protected locations (as discussed in Section 5.2) and tightly 1606 control access to root accounts using a privileged access 1607 management system (preferably one that also logs key-based access 1608 to root accounts for accountability). However, regular audits 1609 should still be performed, e.g., annually, to catch any trust 1610 relationships that may have been missed in the normal process. 1612 Although the focus of this section has been on approving new trust 1613 relationships, existing legacy trust relationships should also be 1614 approved, or at least associated with a business process, 1615 application, or proper purpose. This was discussed in Section 5.5. 1617 Failure to implement or enforce approvals means it is impossible to 1618 ensure that new trust relationships are valid and appropriately 1619 restricted. Not knowing what each trust relationship is used for 1620 makes it very difficult to know which trust relationships can be 1621 removed without substantial risk of disruption to business processes. 1622 Lack of up-to-date documentation of trust relationships, including 1623 lack of knowledge of which application or business process they 1624 relate to, is one of the main causes of the current poor situation 1625 with SSH user keys in many organization. Failure to implement and 1626 enforce approvals for trust relationships also implies that system 1627 administrators can continue to create unaudited backdoors to 1628 production systems, bypassing most existing privileged access 1629 management systems. 1631 5.7. Rotating Existing SSH User Keys 1633 SSH user keys are authentication credentials, like passwords. They 1634 should be rotated (i.e., changed) regularly. 1636 Rotating an SSH user key for a trust relationship means generating a 1637 new identity key (key pair), storing (and configuring, if applicable) 1638 the identity key for the source account of the trust relationship, 1639 configuring the corresponding public key as an authorized key for the 1640 destination account (with the same restrictions as the old key for 1641 the trust relationship), and finally removing the old authorized key 1642 from the destination account and the old identity key from the source 1643 account. If the same identity key can access more than one 1644 destination account (i.e., is used for more than one trust 1645 relationship), then it the authorized key must be copied to (and the 1646 old key removed from) all such destination accounts. 1648 Rotating external keys (i.e., keys used with hosts outside the 1649 managed environment) require special care and coordination between 1650 the organizations responsible for the respective hosts. The basic 1651 principle is that the new key should be added as an authorized key on 1652 all destination hosts where the old identity key is used before the 1653 old identity key is removed. 1655 Authentication credentials for all trust relationships leading to 1656 moderate-impact and high-impact systems MUST be rotated every 12 1657 months, and it is RECOMMENDED that trust relationships leading to 1658 low-impact systems be rotated every 12 months. It is recommended 1659 that all keys be rotated as part of a remediation process to ensure 1660 that any previously leaked keys cease to be usable. 1662 If two or more users have had access to a shared account that has 1663 access to an identity key, the identity key and any trust 1664 relationships using it and leading to moderate-impact or high-impact 1665 systems MUST be rotated during the remediation process and thereafter 1666 every three months. 1668 If an employee leaves or changes roles, immediate rotation for all 1669 identity keys the employee is known to have accesss to and leading to 1670 moderate-impact or high-impact systems SHOULD be triggered. 1672 If a security breach is suspected, all identity keys stored on 1673 affected servers SHOULD be immediately rotated. 1675 If certificates are used for access, such certificates MUST be 1676 renewed (with new private keys) annually if they can be used for 1677 accessing moderate-impact or high-impact systems. If Kerberos is 1678 used for configuring trust relationships, then the Kerberos 1679 credentials used for authentication MUST be rotated annually if they 1680 can be used for accessing moderate-impact or high-impact systems. 1682 Failure to rotate keys allows leaked keys to continue working 1683 forever. 1685 Failure to rotate keys in response to an employee leaving or changing 1686 roles means that there is no proper termination of access. Many 1687 industries must comply with mandatory regulations that require proper 1688 termination of access. 1690 Failure to rotate keys in response to a suspected breach means that 1691 keys copied by the attacker may be used to attack the systems again, 1692 and there can be no guarantee that the system has been properly 1693 cleaned up after the attack. 1695 5.8. Configuring Command Restrictions on Authorized Keys 1697 Command restrictions limit what can be done with a trust relationship 1698 on the destination host. Typically, a command restriction (also 1699 called "forced command") specifies the only command that can be 1700 executed on the server using that key. If any other command is 1701 attempted, the configured command will be executed instead or the 1702 attempt is rejected. 1704 A command restriction may further limit directories that can be used 1705 for file transfers (if supported by the SSH implementation) and 1706 whether writing files is allowed. 1708 On some implementations, it may be necessary to prevent pseudo-tty 1709 allocation for command restrictions to be effective. 1711 It is usually desirable to prevent TCP/IP forwarding for all 1712 authorized keys. Otherwise such keys could be used to mask the 1713 source of attacks by redirecting them using port forwarding. 1715 All non-interactive trust relationships leading to moderate-impact or 1716 high-impact information systems MUST be configured with a command 1717 restriction, unless an exemption has been approved as specified in 1718 the organization's security process and based on a valid reason for 1719 not having a forced command restriction (the relatively small effort 1720 of configuring the command restriction not being a valid reason). 1721 The specific command MUST be part of the approval, and a new approval 1722 MUST be required if the command is later changed. 1724 Trust relationships that are used for interactive access SHOULD NOT 1725 have a command restriction (command restrictions that permit running 1726 a shell and then arbitrary commands SHOULD NOT be used, because they 1727 may be mistaken as real command restriction; if they are detected in 1728 an audit, they SHOULD be flagged). 1730 Regardless of impact level of the destination system, all trust 1731 relationships intended for use with the SFTP protocol by external 1732 parties or by lower-impact information systems MUST have a command 1733 restriction that limits the use of the trust relationship to SFTP and 1734 prevents interactive use. 1736 Failure to configure command restrictions for keys increases virus 1737 spread risk and can be used for other attacks. It also increases 1738 risk from leaked keys. 1740 Failure to configure command restrictions for trust relationships 1741 used with external parties may allow a virus or attack to enter the 1742 organization. 1744 5.9. Configuring IP Address Restrictions on Authorized Keys 1746 Source restrictions (also called "from" option in authorized keys 1747 files) specify from which IP addresses an authorized key can be used. 1749 Trust relationships permitting interactive access to moderate-impact 1750 and high-impact systems SHOULD specify a source restriction to 1751 hardened jump servers (privileged access management systems) or a 1752 transparent access auditing solution SHOULD be used to ensure such 1753 access is properly controlled and audited. If any such trust 1754 relationships have been approved, they MUST be listed in an annual 1755 audit report and their existence rejustified annually. 1757 Source restrictions SHOULD be used for all trust relationships 1758 leading to high-impact systems. Otherwise, the use of source 1759 restrictions is OPTIONAL. They are laborious to configure manually 1760 and make, e.g., IP renumbering and IPv6 transition painful. It is 1761 also easy to make mistakes where, e.g., a secondary server for some 1762 critical service is not permitted by a source restriction, which 1763 could increase risk of outages under unusual operating conditions. 1764 On the other hand, they can significantly reduce the exploitation 1765 potential of leaked keys. 1767 Failure to configure source restrictions has only mild security 1768 impact if other recommendations are followed. It is in part 1769 compensated by regular key rotation that also reduces the potential 1770 for exploitation of leaked keys, and thus a reasonable balance may be 1771 to not implement source restrictions for most trust relationships. 1772 However, source restrictions can completely prevent the exploitation 1773 of leaked keys (without sophisticated active network-level IP 1774 spoofing attacks), and thus is warranted for high-impact systems. 1776 6. Continuous Monitoring and Management of SSH Keys and Automated 1777 Access 1779 The remediation process (as discussed in Section 5) addresses both 1780 the one-time analysis and clean-up of existing legacy SSH trust 1781 relationships and the implementation of an ongoing approval process 1782 for validating, documenting, and restricting new trust relationships 1783 that are added to the environment. Following the approval process 1784 (discussed in Section 5.6) for all new authorized keys added to the 1785 environment serves as a preventive control. Continuous monitoring of 1786 trust relationships is needed to provide ongoing detection of non- 1787 compliance, including instances where the approval process was too 1788 lenient or was bypassed altogether. Continuous monitoring is also 1789 important for identifying trust relationships that violate policy, 1790 that can be removed because they have become unused or otherwise not 1791 needed, or that require keys to rotated. 1793 6.1. Continuous Monitoring of Changes to Trust Relationships 1795 Proper management of automated access requires continuous monitoring 1796 of the IT environment because system administrators operating as root 1797 may add new trust relationships for any user account. Continuous 1798 monitoring is also prudent for detecting keys that are no longer 1799 used, identifying external keys, and identifying changes in the 1800 patterns of usage of automated access. 1802 The main rationale for the continuous monitoring of the environment 1803 and annual audits and requiring reporting and revalidation of certain 1804 aspects of automated access annually is to enforce proper policy 1805 (policies usually do not get implemented if their implementation is 1806 not enforced or if waivers are too easily available). However, IT 1807 environments are complex and sometimes there is a need to have 1808 automated access relationships for special purposes that would not 1809 otherwise be advisable. Special waivers and corresponding approvals 1810 can be used for implementing such special cases, but they MUST be 1811 revisited annually and MUST NOT be used to circumvent remediating the 1812 existing environment. 1814 Ideally, continuous monitoring should be a real-time or near-realtime 1815 process. For some areas, hourly or daily analysis would generally be 1816 perfectly sufficient. Using automated tools allows monitoring to be 1817 performed more frequently, cost-effectively, and more thoroughly. On 1818 the other hand, if implemented manually using audits, cost 1819 constraints may limit continuous monitoring to annual audits. Even 1820 when continuous monitoring is performed using software tools, 1821 auditors SHOULD do some random sampling and testing annually to 1822 verify that the continuous monitoring tools are working properly. 1824 In some respects, continuous monitoring resembles re-performing 1825 discovery on an ongoing basis. Configured SSH user keys and trust 1826 relationships throughout the environment need to be discovered, and 1827 checked for validity. Alerts, audit findings, or reports may be 1828 produced based on the results of the checks. As in the discovery 1829 phase, the continuous monitoring process MUST identify every trust 1830 relationship and authorized key throughout the managed IT environment 1831 so that they can be compared against a database of approved trust 1832 relationships. 1834 For each found authorized key, the trust relationship should be 1835 analyzed to identify possible instances of non-compliance or 1836 excessive security risk. Trust relationships leading to moderate- 1837 impact or high-impact hosts with the following attributes MUST be 1838 reported for further investigation: 1840 Trust relationships without proper approval 1842 Trust relationships without proper justification and an associated 1843 application/business process 1845 Trust relationships that have no command restriction configured 1847 Trust relationships with command restrictions that do not match 1848 the command restrictions specified during approval 1850 Trust relationships from low-impact hosts with no command 1851 restrictions 1853 Trust relationships that cross defined internal boundaries 1855 Trust relationships that have not been used in the last 12 months 1856 (or other time period specified by policy) 1858 Trust relationships whose keys have not been rotated in the last 1859 12 months (or other time period specified by policy) 1861 Trust relationships leading to low-impact hosts with the following 1862 attributes SHOULD be reported for further investigation: 1864 Trust relationships without proper approval 1866 Trust relationships without proper justification and an associated 1867 application/business process 1869 Trust relationships leading to privileged accounts that have no 1870 command restriction configured 1872 Trust relationships that have not been used in the last 12 months 1873 (or other time period specified by policy) 1875 Trust relationships whose keys have not been rotated in the last 1876 12 months (or other time period specified by policy) 1878 If trust relationships have existing waivers (e.g., for having no 1879 command restrictions, crossing boundaries, or not being used or 1880 rotated), then special approval of the waiver MUST be verified and 1881 waivers SHOULD be re-justified and approved annually. Trust 1882 relationships that are flagged by continuous monitoring MUST be 1883 investigated and resolved. Possible resolution activities consist of 1884 the following: 1886 Obtaining approvals and justifications (including the associated 1887 application/business process) for trust relationships that are 1888 valid, including getting secondary approval by higher-level 1889 management for trust relationships that cross boundaries. This 1890 would retroactively apply the approval process described in 1891 Section 5.6. 1893 Adding command restrictions to the authorized keys file to limit 1894 access according to policy. (See Section 5.8) 1896 Removing trust relationships that are unused, not needed, or 1897 otherwise invalid. (See Section 5.4) 1899 Rotating private keys. (See Section 5.7) 1901 Obtaining waivers with appropriate levels of approval. 1903 Even if waivers are obtained, the resulting risk needs to be 1904 considered. For example, if the trust relationship from a low-impact 1905 host to a medium-impact or high-impact host has inadequate command 1906 restrictions, then the low-impact host MUST be reclassified as having 1907 the impact level of the higher-impact host, even if a waiver is 1908 obtained. 1910 Failure to monitor SSH trust relationships prevents the organization 1911 from enforcing policies related to SSH user keys. Policy enforcement 1912 and detection of non-compliant trust relationships is needed to 1913 prevent new keys from re-creating the same type of problems that 1914 existed in the legacy population of user keys. Failure to enforce 1915 approvals for newly-added trust relationships allows users to create 1916 unaudited backdoors or trust relationships that cross boundaries or 1917 are unrestricted. If there is no continuous monitoring for 1918 unapproved or inappropriate trust relationships, such trust 1919 relationships will be essentially permanent. 1921 6.2. Removal of Trust Relationships 1923 Trust relationships MUST be removed when they are no longer needed. 1924 Ideally, the business or application owner of a trust relationship 1925 SHOULD expressly request that it be removed as soon as it is no 1926 longer needed. In addition, the owner MAY periodically recertify and 1927 validate the continuing need for each trust relationship. 1929 Sometimes a trust relationship may be removed by express request, 1930 e.g., when a business process is changed so that it is no longer 1931 needed. 1933 Sometimes a trust relationship may be removed because the application 1934 or business process it relates to is decommissioned or replaced by 1935 another application. 1937 Sometimes a trust relationship may be removed because continuous 1938 monitoring detects that it is no longer being used. This basically 1939 implies that something changed in the environment, but the trust 1940 relationship was inadvertently not removed at that time. (This 1941 scenario appears to be very common in practice). In addition, some 1942 trust relationships may be removed because continuous monitoring 1943 detected an unapproved or otherwise invalid trust relationship. 1945 When trust relationships are removed, the associated authorized key 1946 (if it is key-based) MUST be removed from the authorized keys file of 1947 the destination server. 1949 When there are no trust relationships remaining using a particular 1950 identity key, the identity key SHOULD be removed. 1952 6.3. Periodic Rotation of Trust Relationships 1954 Keys must be regularly rotated as specified in Section 5.7. 1956 7. Policy Recommendations 1957 Effective security policies are important for defining expectations 1958 for controls and acceptable user behavior. Well-defined policies are 1959 no less important for governing SSH user keys than for other elements 1960 of an organization's security program. In fact, because few people 1961 understand the problem and poor SSH user key management practices are 1962 so pervasive, policies are essential to the success of any SSH key 1963 management remediation process. 1965 To support the key remediation and continuous monitoring steps 1966 outlined elsewhere in this document, there is a common core set of 1967 policy statements that should be adopted by all organizations. The 1968 following policy statements are RECOMMENDED (with limited 1969 organizational-specific customization and optionally limited to apply 1970 to moderate-impact and high-impact systems) to provide the governance 1971 framework for controlling SSH user keys: 1973 All SSH servers shall be configured to store authorized keys in a 1974 root-owned /etc directory (or other suitable directory not 1975 writable by normal users). 1977 Users shall not create new identity keys or authorized keys, shall 1978 not share identity keys with other users, and shall not copy or 1979 move identity keys to other SSH client systems. 1981 SSH identity keys and authorized keys shall be provisioned only by 1982 the access management group. 1984 SSH user key requests shall follow the standard provisioning 1985 process. All requests for SSH authorized keys shall be 1986 provisioned only when required by a valid business need and 1987 approved by the destination account's owner. 1989 Trust relationships shall not cross security zone boundaries. If 1990 this is a requirement for a trust relationship, then the new user 1991 key request shall provide a rationale and require a waiver 1992 approved by the server operations director and information 1993 security director. 1995 Trust relationships shall not allow access from low-impact systems 1996 to higher-impact systems. If such trust relationships are 1997 required, then those low-impact systems shall be reclassified as 1998 higher-impact systems and shall be subject to the higher security 1999 requirements of higher-impact systems, unless command restrictions 2000 prevent obtaining an interactive shell and writing arbitrary files 2001 using such trust relationships. 2003 Trust relationships for non-interactive access shall be configured 2004 with command restrictions. If commands cannot be restricted, then 2005 the new user key request shall provide a rationale and require a 2006 waiver approved by the server operations director and information 2007 security director. 2009 Trust relationships permitting interactive access (especially to 2010 privileged accounts) shall enforce source restrictions to 2011 authorized, hardened jump servers or transparent access auditing 2012 solutions are used that ensure such access is properly controlled. 2014 A registry of SSH user keys shall be maintained for tracking trust 2015 relationships (including their owner, purpose, approval, 2016 restrictions, and business purpose) throughout the environment. 2018 SSH user keys and corresponding trust relationships shall be 2019 removed when no longer needed or no longer used. 2021 Usage of SSH user keys shall be tracked so that unused authorized 2022 keys can be identified. 2024 All SSH user keys shall be rotated annually. 2026 When a user terminates employment or transfers to new job 2027 responsibilities, all keys assigned to that user shall be rotated, 2028 or the corresponding authorized key relationships shall be 2029 removed. 2031 If a key is compromised or shared by two or more users, then the 2032 key shall immediately be rotated, or the corresponding authorized 2033 key relationships shall be removed. 2035 SSH authorized keys shall be revalidated annually by the 2036 destination account owner to ensure that trust relationships 2037 continue to be valid and proper. 2039 Authorized keys for privileged accounts such as root shall be 2040 revalidated annually and approved by the server operations 2041 director and information security director. 2043 Trust relationships throughout the network shall be monitored at 2044 least annually to enforce compliance with this policy. At a 2045 minimum, monitoring activities shall be in place to detect the 2046 following types of non-compliance for immediate resolution: 2048 SSH user key trust relationships that bypassed the formal 2049 provisioning process and were not authorized and configured by 2050 the access management group 2051 SSH user key trust relationships that cross security zone 2052 boundaries 2054 SSH user keys that have been not rotated in over a year 2056 Dormant trust relationships that have not been used 2058 Other policy statements are highly dependent on the risk tolerance 2059 and context of each organization. Depending on the unique 2060 circumstances of the organization, these policy statements may or may 2061 not be applicable. During the remediation process, organizations 2062 often make risk-based decisions about how to cost-effectively control 2063 and manage their SSH keys in their own context. It is critical that 2064 these decisions be properly reflected in security policy in order to 2065 influence user behavior and provide a framework for organization- 2066 specific controls. Examples of these types of policy statements are 2067 provided below: 2069 All SSH user keys assigned to human users for interactive logins 2070 shall be assigned a passphrase that is at least 15 characters 2071 long. (The reason for this policy is self-evident.) 2073 SSH trust relationships for human accounts shall be limited to 2074 other human accounts. Human accounts shall never have trust 2075 relationships to system accounts or service accounts. (This 2076 policy makes sense for organizations with lots of keys and 2077 transitive trust relationships that are too difficult to manage. 2078 Eliminating human-to-system account trust relationships can help 2079 simplify the mesh of trust and therefore minimize the risk of 2080 inadvertently allowing unneeded access.) 2082 SSH user keys shall be used only for automated access and shall 2083 not be used for interactive logins by human users. (An 2084 organization may decide to do this to reduce the number of keys in 2085 the environment and lighten the load on the provisioning process, 2086 for example, if no automation is available and provisioning is 2087 done manually.) 2089 SSH servers shall be configured to deny connections to the root 2090 account. (If key-based connections to root are not required, then 2091 setting "PermitRootLogin no" can significantly contain the damage 2092 that can be done through unauthorized use of keys). 2094 Unique SSH host keys shall be created for every system. (This is 2095 essential when SSH host-based authentication is used and for 2096 protecting against man-in-the-middle attacks.) 2098 8. Considerations for Software Tools 2100 All requirements specified in this document can be implemented 2101 manually and with regular audits, without using software tools. Use 2102 of software tools is OPTIONAL. However, automated software tools for 2103 managing SSH keys are commercially available from multiple vendors 2104 and their use is RECOMMENDED in large environments, as they can 2105 substantially reduce the time, cost, and effort needed for 2106 remediating existing SSH user keys and provide substantial ongoing 2107 cost savings for continuously managing and monitoring SSH keys in an 2108 organization. 2110 Here are certain key things to consider in planning an SSH key 2111 management remediation solution and its deployment: 2113 Does the solution support all required operating systems where SSH 2114 keys need to be managed (including mainframe, if applicable)? 2116 Does the solution support all SSH implementations and versions 2117 that are use in the environment, including their key formats and 2118 fingerprint formats? 2120 Does the solution support keys moved to protected, root-only- 2121 writable locations? Can it help move keys to such locations? How 2122 does it determine where the keys are stored on each host? 2124 Can trust relationships that are not actually used be 2125 automatically detected and proposed for removal (with selective 2126 approval)? 2128 Can the solution associate trust relationships and keys with an 2129 application, business process, or other purpose? Can it enforce 2130 that all authorized keys have a documented purpose? How is this 2131 implemented for legacy trust relationships (from time before 2132 deployment of the solution)? Can it distinguish legacy keys from 2133 those that are set up afterwards? 2135 How does the solution implement approvals for new keys? How does 2136 it integrate to existing workflows and tools? Does it support an 2137 approval workflow which integrates into external systems? 2139 Can creation of new keys and trust relationships be automated 2140 based on approvals done in an existing IT change control system? 2141 If no existing IT change control system is in use in the 2142 organization, does the solution provide one to enforce approvals? 2144 Does the solution support grouping systems based on the impact of 2145 their disruption or compromise? 2146 Does the solution support rotating SSH user keys? 2148 How is key rotation implemented for external trust relationships/ 2149 external keys? Can it automatically recognize external keys? 2151 Does the solution support configuring command restrictions for 2152 authorized keys/trust relationships? Does it support requiring 2153 special approvals for trust relationships that do not have a 2154 command restriction? 2156 Does the solution support configuring source restrictions for 2157 authorized keys/trust relationships? 2159 Does the solution provide continuous monitoring capabilities as 2160 specified in Section 6? 2162 If the management system is unavailable for some reason, will 2163 normal operation of managed hosts be disrupted (other than not 2164 being able to create/modify trust relationships)? 2166 Will the solution run as root on managed hosts, or can it use a 2167 non-root account and "sudo" (or equivalent) to perform limited 2168 operations as root? 2170 Is the solution able to retry discovery, key setups, etc. on 2171 hosts that are down or unreachable at the time of the initial 2172 attempt? How does the solution cope with poor network 2173 connectivity? 2175 Does the solution support user accounts stored in LDAP or Active 2176 Directory? How does it prevent crashing LDAP or Active Directory 2177 servers by reading directory contents from all servers 2178 simultaneously? 2180 Can the solution discover keys from directories that are not 2181 readable by root (e.g., NFS directories using the "root_squash" 2182 option)? 2184 Does the solution work with SELinux, if such support is needed? 2186 How can the solution save operational costs in SSH user key 2187 management in the organization? Have existing user key management 2188 costs been estimated on an annual level? 2190 8.1. Reducing Cost and Improving Security by Automation 2192 Some large organizations are seeing over a hundred thousand new 2193 authorized keys being configured every year. Some trust relationship 2194 setups may involve installing the same authorized key on thousands of 2195 servers. Given that setting up and a manual trust relationship can 2196 easily take fifteen minutes or more, the cost can be millions of 2197 dollars per year. 2199 Some software tools allow integration into existing security 2200 entitlement approval systems, and can implement a suitably formatted 2201 trust relationship setup request automatically, without manual 2202 intervention. 2204 Such automation provides several benefits: 2206 Substantial cost savings by eliminating the manual work associated 2207 with trust relationship setups. 2209 Substantial reduction in outages due to errors in manual key 2210 setups. 2212 Need for root access is significantly reduced, as SSH user key 2213 setups no longer require root access. 2215 Substantial security improvements from eliminating root access (or 2216 the need for being able to install new trust relationships) from 2217 most system administrators (having five people with access to the 2218 software tool system is much more secure than having two hundred 2219 administrators able to manually install keys). 2221 9. Security Considerations 2223 This document is all about security, including how to evaluate the 2224 impact of disruption or compromise of information systems, how to 2225 reduce the risk to information system from automated access, how to 2226 remediate current unmanaged base of SSH user key based trust 2227 relationships for automated access, and how to manage and 2228 continuously monitor automated access as an ongoing process. 2230 Section 1.5 defined information system impact levels based on FIPS 2231 199, but expanding on it by considering information systems having 2232 automated access to higher-impact information systems as also having 2233 the impact level of the higher-impact information system. 2235 Section 2.2.6 briefly discussed unmanaged host keys and how they can 2236 be used to compromise authentication and integrity protection using 2237 active network-level man-in-the-middle attacks. 2239 Section 3 discussed various threats arising from poorly managed 2240 automated access and SSH user keys, including virus spread threat, 2241 unaudited backdoor threat, leaked keys granting near-permanent 2242 access, and lack of proper termination of access when an employee 2243 leaves or changes roles. It also discussed how ports opened in 2244 firewalls may be used for unintended purposes, including command 2245 execution, access to internal services, or for hiding source of 2246 attacks, if not properly controlled. 2248 Section 4 discussed assessing the threats and exposure of an 2249 organization to them as a quick precheck during audit, before 2250 engaging in a full discovery and remediation project. 2252 Section 5 provided recommendations on how to bring existing trust 2253 relationships for automated access, particularly SSH user keys, under 2254 control. 2256 Section 6 provided recommendations for continuous monitoring and 2257 management of automated access and SSH user keys. 2259 Section 7 provided recommendations for organizational security 2260 policy. 2262 As a summary, automated access between systems MUST NOT be overlooked 2263 in identity and access management. It has become so prevalent that 2264 many organizations have many times more credentials for automated 2265 access to their information systems that they have user accounts for 2266 employees. 2268 Management of SSH keys is about managing access, with strong ties to 2269 identity and access management, security architecture, privileged 2270 access management, IT change control, and security audits. 2271 Cryptographic properties of the keys are in practice of little 2272 importance, as all keys generated with default settings by most 2273 commonly used SSH implementations are still cryptographically 2274 reasonably strong. 2276 Virus spread threat using automated trust relationships may remove 2277 defense in depth against attacks and malware. Automated access may 2278 provide pathways for bypassing existing privileged access management 2279 systems. Rogue administrators may use SSH user keys to create near- 2280 permanent unaudited backdoors, and leaked keys may be used for 2281 breaking into production servers. Even accidental access using 2282 poorly configured trust relationships has in the past caused 2283 substantial financial losses. 2285 Risks of unmanaged, unaudited automated access are sufficiently high 2286 and the state of their management in some of the largest 2287 organizations in the world so appalling that all organizations should 2288 evaluate to what extent they use automated access within and between 2289 their information systems, how it is managed and audited, and whether 2290 they are exposed to the risks. 2292 IT security auditors, policy makers, and security architects are 2293 urged to take automated access and SSH keys on their agenda. 2295 10. Acknowledgements 2297 The authors thank and acknowledge the contribution of the following 2298 people to the development of this document and/or the underlying 2299 ideas: Bruno Canamasas, Roman Hernandez, Jan Hlinovski, Kalle 2300 Jaaskelainen, Mitch Klein, Sami Lehtinen, Sami Marttinen, Matthew 2301 McKenna, S. Moonesamy, Tim Polk, Joe Scaff. We also wish to thank 2302 anyone else who has helped by providing comments or input. 2304 11. Glossary 2306 account: A user account on a computer. An account may belong to an 2307 actual person (interactive user) or may be used internally in a 2308 system (in which case it is sometimes called a functional account, 2309 process account, system account, or non-user account). 2311 Active Directory: A directory service created by Microsoft for 2312 Windows domain networks, providing a central repository for user 2313 information, user groups, and various other kinds of configuration 2314 information. Active Directory makes use of the LDAP and Kerberos 2315 protocols, among others, and can serve as an LDAP directory and 2316 Kerberos Key Distribution Center (KDC). 2318 Advanced Persistent Threat (APT): A group, such as a government, 2319 with the capability and intent to persistently target an entity 2320 using a variety of cyberwarfare techniques, such as espionage, 2321 social engineering, custom malware, and sophisticated hacking and 2322 evasion techniques. 2324 authorized key: A public key that has been configured as authorizing 2325 access to an account by anyone capable of using the corresponding 2326 private key (identity key) in the SSH protocol. An authorized key 2327 may be configured with certain restrictions, most notably a forced 2328 command and a source restriction. 2330 automated access: Access to a computer without an interactive user, 2331 generally machine-to-machine access. Automated access is often 2332 triggered from scripts or schedulers, e.g., by executing an SSH 2333 client or a file transfer application. Many programs may also use 2334 automated access using SSH internally, including many privileged 2335 access management systems and systems management tools. 2337 automated trust relationship: A trust relationship for automated 2338 access. 2340 command restriction: See forced command. 2342 certificate: A public key signed by a certification authority (CA) 2343 key, together with additional information about the public key. 2344 X.509 [RFC3280] is a widely used standard for certificates. 2345 OpenSSH also implements its own proprietary certificate format; 2346 however, use of the proprietary format is NOT RECOMMENDED (in part 2347 because OpenSSH's authorization model does not permit reliably 2348 determining which trust relationships exist granting access to a 2349 server). 2351 CIFS: Common Internet File System, a protocol used on Windows for 2352 file sharing. The protocol is unencrypted and may be read and 2353 subverted by a network-level attacker. The protocol is extremely 2354 widely used in Windows environments, less frequently with Unix/ 2355 Linux. 2357 CISO: Chief Information Security Officer. A person responsible for 2358 IT security in an organization. 2360 COBIT: Control Objectives for Information and Related Technology, a 2361 framework created by ISACA (Information Systems Audit and Control 2362 Association) for information technology (IT) management and IT 2363 governance. 2365 CryptoAuditor: A product from SSH Communications Security for 2366 controlling and auditing content of SSH sessions and other 2367 encrypted communications, including file transfers. Can also be 2368 used for auditing use of SSH/SFTP connections at a firewall and 2369 for privileged access auditing for key-based access. 2371 destination account: In an SSH connection or trust relationship, the 2372 user account for which authentication is provided and under which 2373 any commands or other operations performed by the connection are 2374 executed (acknowledging that some commands, such as "sudo", may 2375 further escalate privileges). 2377 destination host: In an SSH connection or trust relationship, the 2378 destination host of the connection. A destination host would 2379 typically run an SSH server. 2381 DSA: Digital Signature Algorithm. An algorithm for public-key 2382 cryptography, particularly digital signatures. It is a United 2383 States government standard, specified in FIPS 186-3. 2385 external key: An authorized key that is used from outside the 2386 organization (or outside the environment considered for SSH user 2387 key management purposes), or an identity key that is used for 2388 authenticating to outside the organization (or outside the 2389 environment considered for SSH user key management purposes). Key 2390 rotation can break external keys, and therefore it must be ensured 2391 that the other side of trust relationships involving external keys 2392 is also properly updated as part of rotation. Alternatively, 2393 rotation of external keys may be prevented, but that is not a 2394 sustainable solution long-term. 2396 fingerprint: A hash value of a (public) key encoded into a string 2397 (e.g., into hexadecimal). Several fingerprint formats are in use 2398 by different SSH implementations. 2400 FISMA: Federal Information Security Management Act of 2002, a United 2401 States law that mandates how US government agencies must implement 2402 their it security. 2404 forced command: A restriction configured for an authorized key that 2405 prevents executing commands other than the specified command when 2406 logging in using the key. In Tectia SSH and OpenSSH, forced 2407 command can be configured by using a "command=" restriction in an 2408 authorized keys file. 2410 functional account: An account used for running applications or 2411 other processes, as opposed to an interactive account normally 2412 used by a person. Functional accounts are sometimes also called 2413 process accounts, system accounts, or non-user accounts (with 2414 slight nuances in meaning). 2416 host: A computer or other device on a network. A host may be a 2417 physical computer, a virtual machine, or any other logical or 2418 physical device that can communicate on a network, typically using 2419 one or more IP addresses. Some hosts may be multi-homed, meaning 2420 that they have more than one IP address. 2422 host certificate: A certificate for a host key for host 2423 authentication in the SSH protocol (typically an X.509v3 2424 certificate). Host certificates can eliminate the need for 2425 distributing host keys to all communicating hosts, greatly 2426 simplifying management and rotation of host keys. Widely used 2427 with Tectia SSH to avoid copying host keys and to make rotating 2428 them easier. 2430 host credential: A Kerberos credential that is used for 2431 authenticating to a Kerberos KDC as a host principal. 2433 host key: A public key used for authenticating a host in the SSH 2434 protocol to hosts that want to communicate with it (each host also 2435 generally has its own private host key). Some hosts may have more 2436 than one host key (e.g., one for each algorithm). Host keys are 2437 used for authenticating hosts (machines) themselves, not users or 2438 accounts, whereas identity keys and authorized keys relate to 2439 authenticating users/accounts and authorizing access to accounts 2440 on hosts. See also Section 2.2.6. 2442 identity key: A private key that is used for authentication in the 2443 SSH protocol; grants access to the accounts for which the 2444 corresponding public key has been configured as an authorized key. 2446 indirect trust relationship: A sequence of trust relationships that 2447 indirectly leads to another account. For example, account A may 2448 be able to log into account B, which may be able to log into 2449 account C; then, account C indirectly trusts account A (and B 2450 directly trusts A and C directly trusts B). Indirect trust 2451 relationships may involve many kinds of trust relationships (e.g., 2452 SSH keys, Kerberos and privilege escalation). 2454 interactive user: A person (human) that uses a computer (and may 2455 type passwords or provide other authentication credentials as 2456 needed), as opposed to a computer that performs operations on 2457 another computer in an automated fashion. 2459 jump host: A server that a user logs into for the purpose of logging 2460 infrom there to another server. They are used for privileged 2461 access management, centralizing configuration of access to a large 2462 number of servers (e.g., at retail locations), and for accessing 2463 restricted subnets that do not have normal routing from the rest 2464 of the organization. 2466 KDC: Key Distribution Center, a component of Kerberos. 2468 Kerberos: A centralized authentication and single-sign on system. 2469 Also used as part of Active Directory. See RFC 4120 [RFC4120]. 2471 key: A cryptographic key. In this document, keys generally refer to 2472 public key cryptography key pairs used for authentication of users 2473 and/or machines (using digital signatures). Examples include 2474 identity key and authorized keys. The SSH protocol also uses host 2475 keys that are used for authenticating SSH servers to SSH clients 2476 connecting them. 2478 Key Distribution Center: A component of Kerberos and Active 2479 Directory infrastructure that verifies credentials and issues 2480 tickets to principals (e.g., users and hosts). An Active 2481 Directory server includes a KDC. Frequently multiple KDCs 2482 synchronize information for redundancy. 2484 known host: A host whose host key is known (to a particular SSH 2485 client). 2487 LDAP: Lightweight Directory Access Protocol, a protocol for 2488 accessing and maintaining distributed directory information 2489 services. See RFC 4511 [RFC4511]. 2491 locking down keys: This refers to moving authorized keys to root- 2492 owned (or otherwise protected) locations, so that non-root users 2493 cannot add new authorized keys to themselves. This helps prevent 2494 system administrators and users from creating key-based backdoors 2495 that may survive the termination of their account and bypass 2496 privileged access management systems. See Section 5.2 for more 2497 information. 2499 NERC: North American Electric Reliability Corporation, an 2500 organization that, among other things, maintains the Critical 2501 Infrastructure Protection (CIP) standards that set minimum 2502 security requirements for protecting power generation and 2503 distribution infrastructure. 2505 NFS: Network File System, a file sharing protocol widely used in 2506 Unix/Linux environments in enterprises and universities. The 2507 protocol is unencrypted and may be subverted by a network-level 2508 attacker, permitting modification of any file. (NFS4 adds some 2509 security but is rarely used at the time of this writing, or is 2510 used with the security features disabled.) 2512 OpenSSH: An open source implementation of SSH based on Tatu Ylonen's 2513 original free version of SSH from 1995 and further developed by 2514 the OpenBSD group. 2516 orphaned key: An authorized key for which no corresponding public 2517 key can be found. An orphaned key may be currently unused, or the 2518 identity key might just be on a server that was not part of the 2519 discovery process (it could be an external key). Therefore 2520 orphaned keys should not be removed without first monitoring 2521 whether they are actually used. 2523 password logger: A software or hardware module for recording 2524 keystrokes, especially user names and passwords, typed by an 2525 interactive user. Password loggers are nowadays commonly included 2526 in various malware and used as part of Advanced Persistent Threat 2527 (APT) attacks. Hardware-based key loggers may used in conjunction 2528 with physical access to a desktop or laptop (perhaps using a 2529 social engineering attack, such as getting hired as a janitor) to 2530 obtain passwords for entry into information systems. 2532 PCI DSS: A set of Data Security Standards defined by the Payment 2533 Card Industry Security Standards Council, an organization 2534 originally formed by major credit card companies. 2536 PKI: See Public Key Infrastructure. 2538 privilege escalation mechanism: A means for escalating a user's (or 2539 processes) privileges from those of one account to those of 2540 another account (particularly a root or Administrator account). 2541 Examples of privilege escalation mechanisms include intentional 2542 provilege escalation tools such as "sudo" and unintentional 2543 privilege escalation possibilities based on vulnerabilities and 2544 configuration errors (experience has shown that it is very often 2545 possible to find vulnerabilities or misconfigurations on that 2546 enable privilege escalation once inside a host). An attacker 2547 having access to an account can generally change the configuration 2548 of the account to cause the user to unknowingly run the attacker's 2549 programs that may, e.g., steal the user's password and then use 2550 the password to spread the attack. Also, having high-level access 2551 on one host on a network may effectively imply access to every 2552 user account on every host whose home directory is in networked 2553 storage accessible through the same network as the compromised 2554 host. Against advanced attackers, even vulnerable embedded 2555 devices such as switches, printers and copiers can be used to 2556 perform network-level active attacks against other hosts. Some 2557 limit will have to be put on what theoretical possiblities are 2558 considered, however. Privilege escalation possibilities 2559 effectively imply additional trust relationships that may in turn 2560 imply a multitude of indirect trust relationships. 2562 Public Key Infrastructure: An arrangement that binds public keys 2563 with respective user identities using digital signatures issued by 2564 a certificate authority (CA). See RFC 3280 [RFC3280]. 2566 Putty: An Open Source SSH client for Windows. 2568 Reflection for Secure IT: A commercial version of SSH from 2569 Attachmate. 2571 root account: In Linux/Unix, a privileged account that is usually 2572 able to do anything in a computer (including reading any files and 2573 modifying any programs). In Windows, Local Administrator and 2574 Domain Administrator have similar or even broader power. (This 2575 document mostly talks about root access as SSH is mostly used on 2576 Linux/Unix and embedded devices, but the same issues often also 2577 apply to other technologies and the Windows environment.) 2579 rotating a key: Key rotation means changing the key, i.e., replacing 2580 it by a new key. The places that use the key or keys derived from 2581 it (e.g., authorized keys derived from an identity key, legitimate 2582 copies of the identity key, or certificates granted for a key) 2583 typically need to be correspondingly updated. With SSH user keys, 2584 it means replacing an identity key by a newly generated key and 2585 updating authorized keys correspondingly. See also external key. 2587 RSA: An algorithm for public-key cryptography based on the 2588 difficulty of factoring large integers, invented by Ron Rivest, 2589 Adi Shamir and Leonard Adleman. 2591 SELinux: Security-Enhanced Linux, a Linux feature that provides 2592 mechanisms for supporting access control security policies. 2593 SELinux is enabled by default on several Linux distributions (at 2594 least in what is called "targeted" mode, where it protects 2595 selected services). 2597 SFTP: SSH File Transfer Protocol, a file transfer and file sharing 2598 protocol typically used with the SSH protocol and originally 2599 developed by Tatu Ylonen for ssh-2.0. The protocol is 2600 unofficially described in SFTP [SFTP]; there is no normative 2601 reference available at the time of this writing. 2603 source account: In an SSH connection or trust relationship, a source 2604 account is the user account on the host initiating the connection, 2605 typically the user account under which an SSH client runs. 2607 source host: In an SSH connection or trust relationship, a source 2608 host is the host initiating the connection (typically by running 2609 an SSH client). 2611 source restriction: A restriction configured for an authorized key 2612 that limits the IP addresses or host names from which login using 2613 the key may take place. In OpenSSH, source restrictions can be 2614 configured by using a "from=" restriction in an authorized keys 2615 file. 2617 SOX: Sarbanes-Oxley Act of 2002, also known as the Public Company 2618 Accounting Reform and Investor Protection Act, a United States law 2619 that, among other things, sets requirements for protecting certain 2620 sensitive information in listed companies. 2622 SSH: SSH (Secure Shell) is a protocol and tool for remote system 2623 administration, file transfers, and for tunneling TCP/IP 2624 communications securely, originally developed by Tatu Ylonen. 2626 SSH Communications Security: A company founded by Tatu Ylonen, the 2627 inventor of SSH, with products improving security and operational 2628 efficiency of large IT environments, particularly for large SSH 2629 environments. See http://www.ssh.com [2]. 2631 sudo: A privilege escalation mechanism/tool on Unix/Linux that can 2632 be used for executing commands as root from a non-root account. 2633 The operation of "sudo" depends on its configuration. In some 2634 configurations, certain accounts may perform any command as root 2635 using "sudo". In some other systems, certain users, such as 2636 members of a "wheel" group can perform commands as root by 2637 confirming the operation with the user's password. Several 2638 commercial tools also exist for the same purpose. 2640 Tectia Manager: A product for managing SSH host keys and 2641 configurations, from SSH Communications Security. 2643 Tectia SSH: A commercial version of SSH servers and clients for 2644 Windows, z/OS (IBM mainframes), Unix, and Linux from SSH 2645 Communications Security. 2647 transparent access auditing: A method of doing privileged access 2648 management and auditing on the network (using a co-operative man- 2649 in-the-middle attack to transparently gain access to the 2650 connection) or at an SSH server (by having auditing code built 2651 into the server). See, e.g., the CryptoAuditor solution. 2653 trust relationship: Something that permits a source account to log 2654 in to a destination account (possibly on a different computer). 2655 In a sense, the destination account trusts the source account, and 2656 if the source account is compromised, so is the destination 2657 account. An example is an authorized key (and corresponding 2658 identity key) configured for public key authentication in SSH. 2659 See also indirect trust relationship and privilege escalation. 2661 Universal SSH Key Manager: A product from SSH Communications 2662 Security for managing and monitoring SSH keys and other trust 2663 relationships for automated access. 2665 user key: A key that is used for granting access to a user account 2666 in the SSH protocol (as opposed to a host key, which does not 2667 grant access to anything but serves to authenticate a host). Both 2668 authorized keys and identity keys are user keys. 2670 X.509: A standardized widely used certificate format for public key 2671 infrastructure (PKI). See RFC 3280 [RFC3280]. 2673 12. References 2675 [FIPS199] Evans, D. L., Bond, P. J., and A. L. Bement, "Standards 2676 for Security Categorization of Federal Information and 2677 Information Systems", FIPS Publication 199, February 2004. 2679 [FIPS200] Gutierrez, C. M. and W. Jeffrey, "Minimum Security 2680 Requirements for Federal Information and Information 2681 Systems", FIPS Publication 200, March 2006. 2683 [NIST800-53] 2684 Locke, G. and P. D. Gallagher, "Recommended Security 2685 Controls for Federal Information Systems and 2686 Organizations", NIST Special Publication 800-53 (revision 2687 3 with updates as of 05-01-2010), August 2009. 2689 [KENT] Kent, G. and B. Shrestha, "Unsecured SSH - The Challenge 2690 of Managing SSH Keys and Associations", SecureIT White 2691 Paper, 2010. 2693 [RFC3280] Housley, R., Polk, W., Ford, W., and D. Solo, "Internet 2694 X.509 Public Key Infrastructure Certificate and 2695 Certificate Revocation List (CRL) Profile", RFC 4251, 2696 April 2002. 2698 [RFC4120] Neuman, C., Yu, T., Hartman, S., and K. Raeburn, "The 2699 Kerberos Network Authentication Service (V5)", RFC 4251, 2700 July 2005. 2702 [RFC4251] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) 2703 Protocol Architecture", RFC 4251, January 2006. 2705 [RFC4252] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) 2706 Authentication Protocol", RFC 4252, January 2006. 2708 [RFC4253] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) 2709 Transport Layer Protocol", RFC 4253, January 2006. 2711 [RFC4254] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) 2712 Connection Protocol", RFC 4254, January 2006. 2714 [RFC4511] Sermersheim, J., "Lightweight Directory Access Protocol 2715 (LDAP): The Protocol", RFC 4511, June 2006. 2717 [SFTP] Galbraith, J. and O. Saarenmaa, "SSH File Transfer 2718 Protocol", draft-ietf-secsh-filexfer-13.txt (work in 2719 progress), June 2006. 2721 Authors' Addresses 2723 Tatu Ylonen 2724 SSH Communications Security 2726 Email: ylo@ssh.com 2727 URI: http://www.ssh.com 2729 Greg Kent 2730 SecureIT 2732 Email: gkent@secureit.com 2734 Murugiah Soyppaya 2735 NIST 2737 Email: soyppaya@nist.gov