Virtualisation in Electrical Substations
Background
Virtualisation is commonly found in Information Technology (IT), where it has provided improvements in performance, cost-effectiveness and user-friendliness for many years. However, in operational technology (OT), virtualisation has taken longer to establish itself. As large-scale industrial applications such as manufacturing execution systems (MES) have migrated to the datacentre, they too have adopted virtualisation techniques. Virtualisation is also now regularly seen on supervisory control systems (SCADA) if hard real time response is not a priority. Even when real time responses are required, virtualisation is increasingly used to run real-time control, with a real-time operating system (OS) and a standard Windows or Linux environment running on the same hardware, showing how practical this technology can be. One application area that has avoided virtualisation so far, but is rapidly changing, however, is the usually safe and conservative world of substation engineering.
Virtualisation
The word virtual does not imply a lack of physical objects; rather, it is a logical representation of physical tasks. In computer virtualisation, logical devices defined in software, such as computers, switches, routers and storage disks, behave as if they were physical devices. A physical device will have physical characteristics, such as a certain number of cores on a CPU, a specific amount of storage on a disk, and certain connectivity options; however, in the virtual world, a virtual computer can possess any number of cores, a virtual disk can store far more than the capacity of a single physical disk, and connectivity between the different elements can be made up by software.
As long as there really is the physical equipment behind the virtual front end, developers can deploy their applications and users can operate them without having to worry about the hardware. As virtualisation takes place at a software level, it allows for easy reconfiguration. The physical hardware can remain the same, but the virtual hardware can change with time, different customers/users or new applications.
Equally, the application can remain the same, but the hardware can benefit from improvements in technology. Upgrades can provide vast increases in performance over the life of an application, adding more users, more customers or more data without having to change the application.
Virtualisation works by abstracting the hardware from the software. A hypervisor is installed, and, above this, multiple virtual machines run their own operating system. Each VM can then behave as a computer. Between VMs, virtual communication devices such as switches and routers control the transfer of data between them and the outside world.
Containerisation is the latest development. Instead of supporting multiple VMs, each necessarily including its own operating system, containers leverage the common operating system instance on the underlying hardware. Each container will have individual access privileges to functions outside of the container, and will have defined resource limits on processor cores, memory & storage. Furthermore, this is achieved using techniques which abstract the underlying operating system such that an application designed to run in a container can actually run without modification, for example, on a Linux or Windows physical machine.
Why Not Yet in Substations
Virtualisation presents commercial advantages and has spawned the entire cloud as-a-service business model, but users have been nervous of adoption in the OT world. Computer systems are relatively easy to manage while every aspect of it is written in software and operates in a single CPU. However, when that application has to interact with the real world, with machines, sensors, human inputs, actuators, then there are a number of complexities. The first is that computers are not homogenous devices. They are made up of a large number of complex components, some even behaving as mini computers in their own right. Computers also carry a lot of historical baggage left over from the days of mainframe computers and the first IBM PCs. Modern laptops, workstations and servers may no longer have serial ports, but in operational technology (OT) environments, this is still a requirement. Industrial equipment still uses COM port technology, and even older proprietary systems as well as more modern digital inputs and outputs, industrial fieldbuses and networks. These have mechanical and electrical properties which are difficult to model and simulate, and therefore difficult to replicate in a logical software world.
As well as dealing with real world data streams, industrial systems also need to deal with real time responses. IT systems in an insurance company may benefit from a high-speed process to deal with claims, but the claim handling process will not break down if an OCR scan takes a second longer than usual. In industrial processes, however, a 1 second time delay before ordering an actuator to move a barrier can cause a break in production, damage to equipment or even threaten the safety of operators. As virtualised systems have little direct control over the hardware that they run on, the prediction of response times, latency and jitter is difficult
Another problem facing administrators of industrial computer systems is that these are often installed in unmanned remote locations. While an admin in a datacentre can always go to the server and connect to the console port directly, restart the computer or simply unplug it and plug it back in, this may not be practical for many OT admins.
The computers used in industry are often old, are kept because they have specific features needed to operate the machinery they connect to, and cannot be upgraded to a virtualised environment. Many such devices look nothing like computers, and are usually called embedded as they have very tightly integrated hardware and software.
Finally, there are always concerns about safety, reliability and security. This is especially true in critical infrastructure, but also in high-value production. Security, safety and reliability are all easier to measure and therefore to manage, when each function is represented by a single box. Although mathematically it makes no sense, there is a touching faith in the idea that if a box fails, it is only that function that dies with it. The reality is, that with modern applications, the single box will just as likely take down the entire network. The other major concern is to do with cybersecurity. There is much substation engineering associated with identifying and managing the risk, and that workload will initially be greater with a new technology. However, as with most such systems, the ability to modify systems remotely and at a moment's notice will automatically add greater security, and a centralised orchestrator provides a much easier method to observe and record intrusions.
Why now ?
The migration to IP technologies in substations with the adoption of IEC-61850 brings virtualisation much closer. A new digital substation designed around IEC-61850 will use all IP technologies, which are easy to simulate in software, so virtualised elements can easily transfer data from one to another. Even older substations can benefit with the new generation of merging units (MU) which convert legacy equipment to IP technologies.
If enabling technologies are important to encourage the adoption of virtualisation in the substation, the demands for new applications are also important. New applications would otherwise require additional hardware, with the space that it occupies, a power supply, cooling, heating and ventilation requirements, maintenance, a supply chain, and a whole lifecycle management. Each new application would require its own set of resources, resources which are in increasingly greater demand and lesser supply. In Europe, substations are built increasingly underground, using gas insulated switchgear to minimise space; lower voltage substations are prefabricated for easy transportation and installation, with minimal space for components or to retrofit new equipment later.
The substation has many administration functions that could run on virtualised computers. Security services such as authentication and access control, engineering workstations and operator terminals, historians, and the basic functions of a communications network, both wide area and local area, can all run on virtualised systems. SCADA functions such as substation automation and gateways also commonly operate on the same computer, although virtualisation may not be necessary if they both run on the same operating system. Modern applications, such as demand response, fault finding, and the operation of some traditional protection functions can also be envisaged now in virtual machines. The applications themselves are increasingly available to run on VMs in both open source and proprietary versions.
Advantech’s experience in Virtualised Substations
Advantech was one of the first enterprises to show a virtualised environment for electrical substations, when it participated at CIGRÉ 2018. The system Advantech showed combined both virtual communications and virtual automation. The communications infrastructure included virtual switches (vSwitch), virtual routers (vRouter), and firewalls that included deep-packet inspection of substation protocols like IEC-60870-5-104. The communication operated on Linux virtual machines (VM), but automation traditionally uses Windows as the operating system. SCADA software therefore had a separate VM running Windows 7. Overall, the system benefited from an orchestrator, which supports the easy replication of the architecture when new substations are built and added to the wide area network.
The substation installation was simulated with Advantech’s ECU-4784. This is a substation-hardened server certified to IEC-61850-3 for operation in the harsh environment found in electrical infrastructure, with much higher levels of protection against shock, vibration and electro-magnetic interference. The ECU-4784 is now available as the 4th generation of Advantech’s substation computers, with all the experience of its global customers to ensure a reliable and long service.
Later the same year, Advantech’s partner Inorks, from Columbia, showed its Quantum SDN solution at the IEC-61850 Global Forum in Germany. Large substation networks could now be created in a software-defined way, and managed from another virtualised application.
Substation engineers also require hard real-time control and Advantech developed a customised version which can provide guaranteed response times to time-critical interrupts. This enabled the use of the ECU-4784 for real-time protection applications which can also be virtualised.
The ECU-4784 is a fanless solution which utilises a single CPU to minimise power consumption and heat production, and permit operation in extreme environmental conditions up to 70°C, necessary in outdoor locations if air conditioning is not available or breaks down. Although it comes in a variety of 4-core CPUs, the low power nature of these items means that there is a limit to the processing power and the number of VMs that it can support. However, virtualisation promises such great benefits in modern substations that utilities and substation engineers have long demanded greater processing power while insisting on keeping the substation and environmental protection, the real-time processing capability and the wide range of hardware options.
Virtual Substation Server
This year, Advantech launches its next generation series of substation computer, starting with the ECU-579, the most powerful substation-certified computer in the world. Capable of handling all the virtualised applications in a substation, in both Windows and Linux, including real-time applications, the ECU-579 builds on Advantech’s unrivalled knowledge of substation computing while adapting the techniques used in telecommunications base station servers to build powerful but reliable equipment.
The ECU-579 is based around server-grade CPUs running up to 24 Intel Xeon cores, providing up to ten times the processing capacity of the last ECU-4784, in a single 2U 19” rack unit. This processing capacity, typically installed in a redundant architecture, is sufficient to operate a virtualisation environment, with real-time operating system (RTOS) for mission-critical functions like protection relay VMs, and a standard OS for SCADA and admin functions for an entire substation.
A fully digital substation, with the flexibility to add in new functions as they are developed at any time during the lifetime of substation, is now a possibility with a virtualised environment. Hot-swap hardware and software upgrades can be managed remotely and for vast numbers of substations, without the mechanical and physical cost of individually visiting substations for the installation of new applications, and without having to worry about the software when upgrading the hardware.
Virtualised substations – their time has come.