Hard realtime control and Windows NT

A number of papers have been written on the inability of Windows NT alone to perform the hard real-time control required to replace a PLC. This report from Entivity examines the available solutions

Microsoft defines a hard realtime operating system as one which "must, without fail, provide a response to some kind of event within a specified time window. The response must be predictable and independent of other activities undertaken by the operating system." Microsoft goes on to explain that "under this definition, Microsoft Windows NT Workstation is not a hard realtime operating system."

There are several alternatives being introduced to the market that combine hard realtime control and Windows NT on the same platform. Each of these approaches has benefits and drawbacks which are important for the control engineer to understand when selecting a PC-based control system.

For reasons of safety, a machine control program must survive a crash of Windows NT and not be affected by any problems with the hard disk. Control must be maintained during one of these events, and an indication must be given to the control program that NT is no longer functioning. Though other recently announced implementations claim hard realtime performance, this paper argues that the only approach that currently meets the requirements and is a proven solution for machine control applications is the iRMX solution offered by Entivity.

Entivity has used iRMX as the control kernel for its hard realtime product since it introduced the Visual Logic Controller (VLC) running with Windows 3.1 in several years ago. With the shift in the market towards Windows NT, Entivity chose this same, well-tested engine for the NT version of the Visual Logic Controller.

There are two types of PC-based control systems - Soft Logic and Hard Realtime Control (HRTC). With most soft-logic control systems, the control engine is run as a high priority "realtime" task under Windows NT. "Realtime" tasks can be interrupted by deferred procedure calls (DPCs) which are used to service NT system functions such as disk access, network communications, and mouse interrupts. DPCs can be triggered by lower priority user tasks such as opening files or starting application programs. The process in which a user task can take operational precedence over higher priority task is known as "priority inversion."

Deterministic control

Soft Logic cannot provide the same level of deterministic control as a PLC because higher priority system functions can interrupt and delay the realtime control engines. In high-speed discrete applications, this lack of determinism is unsafe and unacceptable. In contrast, hard realtime control systems provide the same level of deterministic control as a PLC (or better).

How does the PLC accomplish hard realtime control? A PLC is really nothing more that a microprocessor with a realtime operating system (RTOS) on a proprietary hardware platform. The RTOS is the kernel of code that controls all operations and tasks which run on the microprocessor. PLC manufacturers achieve deterministic control by building the control engine around the RTOS. It is the RTOS that controls the PLC I/O scan and logic control functions, and gives the PLC its fast, deterministic response. Several options have recently been announced to combine hard realtime control and Windows NT in the same platform. The Windows NT environment provides a number of benefits over traditional PLC systems including network communications, graphical user interface, and information processing. The combination of Windows NT with the hard realtime characteristics of a PLC yields a very powerful control solution.

Critical requirements

Several critical requirements must be met when a PC is used to replace a PLC in any control application:

• Protection from poorly behaved Windows applications and drivers: As there are thousands of available Windows applications, it's difficult to control and test all applications which could be running on the PC being used as a control platform. It is imperative that the control system is not impacted in the event the system crashes or GPFs from a poorly written Windows NT driver.
• Protection from Windows System failures: When Windows NT crashes, a blue screen is displayed and the entire system comes to a halt. This condition is commonly referred to as the "Blue Screen of Death." Poorly behaved Windows NT drivers or HAL extensions are among the items which can cause this condition. Control is the most important function of the system, and it is imperative that the control program and engine continue to run even after the Blue screen.
• Hard-Disk Crash Survival: Because they are mechanical assemblies, hard disks have the highest likelihood of failure of any component in a PC-based control system. It is essential that deterministic control is uninterrupted in the event that a hard disk crashes or is removed from a PC-based control system.
• Robustness in the realtime engine: The control engine is one of the most critical sections of software running on the PC. Most PLCs run with realtime operating systems that have millions of hours of operating history. Quantifiable measures of code reliability include: the size of the code; the number of installed nodes; and its time in service.

The larger the code, the more likely it is that undiscovered bugs may surface during operation. Windows NT has 7MB of core code space, resides in over 150MB of disk space, and was developed over the last several years. Even though Microsoft has done extensive testing, it is not unusual to see Windows NT lock-up during normal use. On the other hand, some hard realtime operating systems use less than 100K of code space, and have been installed in millions of applications worldwide over several decades. It is fair to conclude that these hard realtime operating systems are several orders of magnitude more robust than Windows NT for replacing traditional PLC solutions.

There are two fundamental approaches to applying hard realtime performance characteristics to Windows NT:

• Modification of the Hardware Abstraction Layer (HAL) of Windows NT
• Combining Windows NT with a proven hard realtime operating system.

Several companies have announced realtime extensions of Windows NT by making modifications to the Hardware Abstraction Layer (HAL) (see Fig 1). The HAL is the lowest level of Windows NT that abstracts, or defines a common software interface, to the hardware on which NT is running. Each microprocessor type, be it an Intel, DEC Alpha, or Power PC device, requires a unique HAL in order for Windows NT to run. In fact, some types of Intel-compatible PCs may require different HAL implementations in order for NT to function properly. The HAL layer is responsible for all of the hardware related functions such as interrupt handling, video control, hard disk access, and (important for this discussion) the hardware timer. By modifying the HAL, it is possible to write code at the lowest level of Windows NT that makes special use of the hardware timer to provide realtime response. Several vendors have provided realtime extensions to Windows NT by writing code that provides "realtime" services through a driver that makes use of changes to the HAL (see Fig 2).

Major drawbacks

This approach has several drawbacks that make it less than acceptable for realtime manufacturing control systems. These drawbacks include:

• Lack of protection from Windows NT systems failures (Blue Screen of Death): Because the realtime extensions are part of Windows NT, PLC control functions cannot be protected from an NT crash. There is no way to annunciate an NT crash and no way to bring your machine to a safe state by executing an orderly shutdown. When the NT crashes or locks up, logic processing simply stops.
• No protection from poorly behaved device drivers: Another unfortunate downside in clipping realtime extensions to Windows NT is that there is no protection from poorly behaved driver and system level functions. If a driver or system level function faults, control is lost leaving a machine or process in a potentially dangerous condition.
• Reliance on presence of functional hard disk drive: If the hard disk fails during operation, NT can become completely crippled, again killing control of your process.
• A general lack of robustness in mission critical applications: HAL modification as a viable realtime solution has not yet been proven. At the time of writing, no realtime modified HAL solution has been delivered to the market beyond an alpha stage release.

Given its short history and intense interaction with complex NT code, it is difficult to determine the robustness of this solution.

Another approach to achieving hard realtime control with Windows NT is to combine a hard realtime operating system with Windows NT on the same platform (see Fig. 3). An example of this is the hard realtime NT approach taken by Radisys and adopted by Entivity. In this approach, Windows NT is loaded as the lowest priority task in the iRMX hard realtime operating system.

In this case, the user only works within Windows NT and never sees or touches the hard realtime operating system. All of the control functions are run as higher priority tasks in the realtime operating system and are actually isolated in memory from Windows NT applications and drivers. By using the memory protection functions inside the Intel processor, Windows NT is prohibited from accessing any of the memory or CPU cycles dedicated to the realtime engine.

Advantages of combining a realtime operating system with Windows NT include:

• Survives the Blue Screen of Death: The main reason this approach is essential for control applications is that Windows NT can crash without impacting hard realtime control. Or simply, the iRMX/NT implementation survives the Blue Screen of Death. A poorly behaved driver or any application program anomaly can crash and the control system will continue running. Because of the careful integration between iRMX and NT, this solution has the unique ability to inform the control program that NT has experienced complete system failure. This allows the control program to continue running as normal, or execute an orderly shutdown to bring the machine into a safe state.
• Protection from poorly behaved drivers: Unlike the modified HAL approach, the iRMX/NT solution offers complete isolation from unstable drivers and errors in system level functions. Driver or system level function faults have no impact on the operation of the control program.
• Survives hard disk failure: By loading a separate hard realtime operating system into the PC, dependency on a functional hard disk is eliminated. Because the entire RTOS is loaded and active in memory, failure of the hard disk, regardless of its impact on Windows NT, has no influence on hard realtime control activities.
• Robust, proven solution: Another reason iRMX is a natural for combination with NT is its history as a hard realtime operating system. iRMX was invented by Intel in the mid-1970s with a core kernel of less than 100k of code. It has been in use for more than twenty years, and has an installed base of over 2 million applications - more than Windows NT. With its history of advanced process control, defense, and avionics installations, iRMX has a proven track record in mission critical applications.

Entivity

Reply number s122