By John Shaw, GarrettCom
The cyber security compliance challenge is like that of the Y2K event in enterprise computing nearly a decade ago. FERC and NERC timetables are forcing significant changes in utility networking and computing infrastructure. Utilities can look at cyber security as merely an intrusive requirement, forcing investment in security-specific technologies, or, alternatively, utilities can view this as a forced upgrade to their infrastructure that creates opportunity for much broader improvements and benefits.
The power utility industry is increasingly preoccupied with cyber security standard compliance. Developed by NERC and having impetus from federal legislation and teeth from FERC oversight, cyber security regulations cannot be put off. While details of the standards continue to evolve, much is now clear, and further developments will likely only raise the bar on required technologies and procedures. Compliance efforts are uneven. Different companies are taking different approaches, and the levels of intensity in their preparations differ according to attitudes, existing capabilities and utility size, which can influence the perceived scope of the undertaking. For all, full compliance can be expected to be challenging technologically and, even more so, administratively.
The overarching reasons for cyber security investment are the real and perceived threats from current and former employees and contractors and from direct and indirect (i.e., non-utility specific) activities of cyber terrorists or “hackers.” A more definite and less ambiguous driver is the legislated mandates taking shape in the NERC Critical Infrastructure Protection (CIP) standards (CIP 002 through 009), and especially the threatened fines for non-compliance. A third major driver for cyber security investment is the opportunity it presents: The compliance burden can become a boon to network infrastructure modernization.
Management has viewed many automation initiatives at utilities as highly discretionary-often taking a back seat to more fundamental generation, transmission and distribution investments. Many substations today operate at a relatively low level of automation; however, smart grid initiatives and variations on the “intelligent utility” promise exciting technology investments to improve reliability, reduce costs, and enhance customer choice and service. When retooling for NERC/CIP compliance, utilities can achieve new levels of substation networking relatively painlessly-and for little additional life cycle cost.
The new mandates offer an opportunity to create more cost-effective and flexible networks by consolidating communications onto an integrated network architecture. This makes it easier to define and administer CIP-required electronic security perimeters (ESPs), as well as making it easier to deploy additional automation applications over a common, flexible infrastructure. A second opportunity is to facilitate remote engineering, provisioning and administrative access to devices at substation. Utilities can improve operational responsiveness and reduce travel time by using updated remote access networking that is both easy to use and security compliant. Third, end-to-end accountability forces deployment of intelligent network elements that can be optimized for network resiliency, thus improving data network reliability and contributing to overall grid reliability.
Electronic Perimeter Security via Network Integration
The NERC CIP requirements most directly applicable to network security are CIP-005, Electronic Security Perimeter, and CIP-007, Systems Management Security.
CIP-005 mandates that an ESP be established at control centers, critical substations and any other locations having critical cyber assets (CCAs). The utility must identify all CCAs within a physical location and then define an electronic perimeter such that all connections to this collection of devices and software systems are secured. All network connections across this defined perimeter must have, at a minimum, a “firewall” that permits only authorized connections and traffic to enter the secured zone. In addition, all physical and software-defined ports to all devices and applications within the electronic perimeter must be identified and secured, and all unused ports must be disabled (see Figure 1, above).
Today, many utilities have a variety of network types connecting to substations. In many cases this means multiple network connections to the same substation, which can complicate establishment of an effective security perimeter. Connections may include multiple leased analog circuits implemented for different projects over a period of years, as well as dial-up connections used by remote engineering and administrative personnel to access specific IEDs. Many utilities have begun to deploy Internet protocol (IP) networks with leased digital lines that are dedicated to IP traffic, but may not yet have consolidated legacy applications onto this new network.
A collection of mixed networks is not cost-effective in terms of telecommunications expense; in addition, there is little flexibility for adding incremental applications or devices. Most importantly, without consolidation, these diverse networks will require numerous, separate solutions for network perimeter defense.
The need to establish a security perimeter can be used as a catalyst for deploying a modern integrated network, thus converting a mandate into an opportunity. Newer-generation substation networks are more flexible and can support multiple applications simultaneously. Integrated networks typically using IP can converge serial and Ethernet devices and operational and engineering applications onto a single infrastructure. Figure 2 (below) contrasts a single IP-centric substation wide area network (WAN) with more historical, non-integrated networks.
The requirements for developing an integrated architecture will differ from one utility to another. In some cases, all applications may run on a single IP-based infrastructure. In other cases, technologies such as frame relay permit IP services and legacy services to efficiently merge and share a common digital WAN. Integrated networks consolidate the substation’s electronic perimeter entry points while minimizing telecomm costs and facilitating additional application deployment.
Within the substation, multiple functionalities are required to consolidate data traffic and to provide cyber security functions. Figure 3 (right) shows some of the functions that must be present, either in discrete networking elements or integrated in a single multi-function device. Ethernet switching and serial terminal servers provide connectivity for Ethernet and serial-based IEDs, respectively. An IP firewall is often integrated into an IP router. The router may connect to WAN facilities directly or via another transport network element.
The firewall function at a minimum must provide IP address filtering and upper-level protocol ports for TCP, UDP and more specific protocols such as Telnet, HTTP, and HTTPS. In some cases, especially with public or shared IP services, the router may also provide encrypted virtual private network (VPN) connections to the central control network. Within the substation, the Ethernet switch and serial device servers must also provide port security, disabling unused ports and securely linking used ports to defined distant end points.
In some cases, these security perimeter functions may be integrated into a single box that contains all necessary WAN, Ethernet, serial and firewall functions for a small substation. For larger substations, an integrated architecture will be distributed over several devices with an Ethernet core switching network, distributed Ethernet and serial edge devices, and a secure gateway router/firewall to the WAN. Such networks may be provided by a single vendor or built from a diversity of products sharing well-established Ethernet-based standards.
Secure Access Control Can Enhance Productivity
Secure access control enables on-demand interactive access to devices within the ESP on a remote basis, but only to rigorously authorized and authenticated users and applications. Access control plays a major role in several of the CIP standards, including CIP-003, -004, -005, -006 and -007. Primary network requirements are in CIP-005 and CIP-007, where access requirements are defined and imposed on specific applications and critical devices. Ultimately, utilities must clearly define who can access what and when, and implement controls that enable appropriate access, prevent inappropriate access and consistently track and audit activities in both categories.
Implemented poorly, access controls can create a frustrating layer of overhead for engineers and administrators who need to utilize remote technologies. Implemented well, access controls facilitate remote access to substation assets, making access simpler, more consistent and more finely tuned to the task at hand. Remote access can increase productivity by reducing travel time, by allowing similar or correlated tasks to be executed as a group across many substations in a short time, and by reducing reaction time to both initial events and ongoing interventions.
One effective implementation of access controls involves establishing a centralized access management system (AMS) combined with an integrated substation network (see Figure 4, above). While interactive users perceive that they are directly connecting to remote IEDs, they, in fact, connect to an AMS. The AMS authenticates the user, checks specific access authorizations and then establishes its own connection to the requested remote IED. It links the user and IED via this “proxied” gateway connection. The AMS also fulfills CIP requirements by logging all activities, optionally including every key stroke or transaction the user executes. An AMS may provide some CIP compliance reports itself or it may provide logs to other compliance management systems. Similarly, the AMS may provide strong authentication itself, but typically integrates with an RSA server (e.g., RADIUS or SecureID) for centralized administration of personal profiles and two-factor user authentication.
Examples of the AMS approach include the MyIED element of the Substation Suite from Subnet Solutions and IED Anywhere from Bow Networks. These AMS applications are deployed on a central server where user profiles and system profiles are defined. A user accesses the AMS via a web browser using two-factor authentication and SSL encryption to secure communications; users cannot connect directly to the remote IEDs, rather only via the AMS gateway.
Many of the critical security features of the AMS operate behind the scenes, making the user’s experience as productive and non-intrusive as practical while meeting all regulatory requirements. The AMS must be linked to corporate servers for authentication. The AMS must have tightly secured connection to the remote elements. Techniques for securing AMS-to-IED links include firewalls, IP/port filtering, secure socket layer (SSL) connections and/or IPsec tunnels. The AMS must also generate user activity logs including “who,” “when” and “what” details.
By providing an easy-to-use interface designed for productivity enhancement and by hiding most security functions behind the scenes, the implementation of access controls can be an opportunity for productivity rather than a major imposition on a busy engineering staff.
End-to-end Accountability and Reliability
NERC requirements stress accountability and auditability at multiple layers of security and on an end-to-end basis. Especially wherever IP and Ethernet protocols are used, end-to-end accountability includes being able to monitor and assure secure communications with no weak links. All IP-aware elements must provide logging and alerting on events and assist in real-time analysis and periodic audits that can correlate events and find consistencies and exceptions.
Network elements themselves must meet many of the CIP-007 requirements for system management security. Typically, this requires secure protocols such as HTTPS for web management interfaces, SSH for command line consoles, SNMPv3 for system management interfaces, syslog and SNMP traps for alerting, and RADIUS and strong-form passwords for user authentication.
The same network intelligence that provides security for network elements also typically provides the resources to enable resilient network architectures. Among control centers and substations, there are often multiple mirrored master systems for critical SCADA and other operational functions. Within substations and control centers, Ethernet rings and mesh architectures can also provide redundant, reliable networking with nearly instantaneous recovery from failures.
Resilient architectures that are designed to support the activity of different routes at different times may complicate the task of auditing end-to-end security. A primary strategy for assuring that end-to-end communication paths remain secure, independent of physical network route, is to utilize authentication and encryption protocols directly between master systems or user PCs and the ultimate IED or other remote critical asset. One technology that is particularly effective is the secure socket layer (SSL) protocol. SSL is supported in most server technologies, in AMS, in some remote IEDs natively, and in some hardened serial device servers. SSL does not take the place of other perimeter security mechanisms, but it supplies an end-to-end authentication mechanism that provides auditability between end points, independent of network topology (see Figure 5, page 36). This approach permits rigorous end-to-end accountability to be implemented effectively in concert with end-to-end network resiliency technologies, at the server, the WAN and the substation LAN.
NERC CIP requirements and deadlines offer a unique opportunity to define and implement a more streamlined, easier to use, more cost-effective communications architecture for power utilities and substations. Each utility has its own set of expectations, history, legacies and requirements, but whatever the current state of connectivity, each will need to comply with CIP requirements soon. Electronic security perimeters and access controls are critical to meeting these requirements, and an integrated, IP and Ethernet-centric network offers an extensible and future-proof base upon which to build.
Taking into account the life-cycle costs of NERC CIP compliance, including design, deployment, training, and maintenance, deployment of a new integrated network will likely have similar or even lower costs than attempting to bring a collection of legacy communications systems into compliance. The operational ability to easily accommodate future expansion, ensure network reliability and assure end-to-end accountability lies with the integrated substation network. NERC mandates are a major disruptive event in the history of substation networking. There will never be a time like the present to position substation networks to serve the rapidly evolving needs of the intelligent-and compliant-utility.
John M. Shaw is executive vice president of GarrettCom Inc., a supplier of substation-hardened networking products. Contact him at firstname.lastname@example.org.