This article was automatically translated from the original Turkish version.
Real-time systems are systems that must not only produce correct results but also complete those results within a specified time limit (deadline). In these systems, timing determinism is critical, and the system must exhibit predictable behavior. We interact with such systems across a wide range of technologies, from everyday consumer devices to critical safety systems in the aviation industry.
The most fundamental characteristic of real-time systems is that time is an inseparable part of system performance. In traditional computer systems, how long a task takes to complete is generally a secondary concern; however, in real-time systems, violation of time constraints can lead to system failure.
Real-time systems are divided into three main categories based on the criticality of their time constraints, each with its own distinct characteristics.
1. Soft real-time systems are systems that can tolerate occasional violations of time limits. Delays in these systems do not cause system failure but result in degraded performance. Video players are among the most common examples; a delayed video frame degrades the viewer experience but the system continues to operate. Online games can also tolerate delays, though the player experience suffers. Multimedia applications in general fall under this category because minor imperfections in user experience are acceptable.
2. Firm real-time systems treat results produced after a time limit has been exceeded as completely invalid and reject them. Financial transaction systems provide a perfect example; a transaction not completed within a specified time is canceled and must be restarted. A similar situation occurs in data acquisition systems; in sensor systems requiring data readings at regular intervals, delayed data becomes meaningless and is discarded. Timeout mechanisms in network communication protocols also reflect this principle.
3. Hard real-time systems constitute the most critical category. In these systems, missing a time deadline is unacceptable and can lead to severe consequences. Airbag control systems must respond within milliseconds during a crash; any delay can result in loss of life. Nuclear reactor control systems must intervene immediately in abnormal conditions. Aircraft control systems must respond to pilot commands at the microsecond level; otherwise, flight safety is compromised.
The traditional Linux kernel was designed as a general-purpose operating system (GPOS), and this design philosophy is insufficient to meet the requirements of real-time systems. Linux’s core design principles focus on fair resource sharing and high throughput, but these approaches conflict with the need for deterministic timing.
1. Kernel locking mechanisms are among the biggest obstacles Linux faces in real-time applications. Critical sections remain locked for extended periods, causing even high-priority tasks to be delayed. Large kernel locks and spinlocks introduce unpredictable delays across the system, while the priority inversion problem causes low-priority tasks to block high-priority ones. This contradicts the fundamental principles of real-time systems.
2. Interrupt handling issues also prevent Linux from exhibiting deterministic behavior. Interrupt handlers introduce non-deterministic delays and can run for unpredictable durations, especially under heavy network traffic. In interrupt storm conditions, the system may become completely unresponsive, which is unacceptable for real-time applications.
3. Memory management delays are another critical issue. Long-running atomic operations in the memory management subsystem negatively impact system response time. Delays caused by page table updates and memory allocation can reach unacceptable levels for real-time applications. Garbage collection and memory compaction operations introduce unpredictable delays, disrupting the system’s deterministic behavior.
4. Timer limitations are another reason Linux falls short in real-time applications. The traditional Linux timer’s resolution of 100Hz to 1000Hz is insufficient for many real-time applications. The jiffy-based fixed-interval interrupt system cannot provide microsecond-level precision, negatively affecting the performance of applications requiring precise timing.
PREEMPT_RT is a comprehensive patch developed to adapt the Linux kernel for real-time applications. Its development process exemplifies the collaborative power of the open-source community and has matured over two decades into its current form. Below are the project’s milestones listed chronologically.
1. Full preemption is one of PREEMPT_RT’s most fundamental features. Nearly all kernel code has been made preemptible, with only raw spinlocks and the top halves of interrupt handlers left non-preemptible. Mutexes have replaced spinlocks, allowing tasks to sleep, significantly improving system response time.
2. Interrupt threads convert interrupt handlers into kernel threads, enabling IRQ threads to be scheduled like regular tasks with priority. This change brings interrupt processing under control and ensures deterministic behavior.
3. Priority inheritance mechanism resolves the priority inversion problem by temporarily granting low-priority tasks the priority of higher-priority tasks they block. The RT-Mutex structure provides an enhanced locking mechanism, ensuring more predictable system behavior overall.
4. High-resolution timers with the dynamic tick mechanism replace fixed-interval jiffy-based interrupts with on-demand triggering. This system provides microsecond-level timing precision and supports nanosecond-resolution timers while also improving energy efficiency.
The real-world success stories of PREEMPT_RT demonstrate the effectiveness of this technology.
1. Industrial automation widely uses PREEMPT_RT in PLC systems, robotic control systems, SCADA systems, and factory automation applications.
2. Telecommunications and networking sectors rely on PREEMPT_RT for 5G base station control systems, SDN applications, low-latency network communication in routers, and edge computing platforms. Consistency in packet processing times is vital for service quality in these applications.
3. Multimedia and audio processing benefit greatly from PREEMPT_RT in professional audio recording and mixing systems, live broadcasting applications, music production software, and video processing systems. Audio buffer underruns and latency issues are minimized through this technology.
4. Financial technology high-frequency trading systems, algorithmic trading platforms, and risk management systems leverage the deterministic behavior provided by PREEMPT_RT. In this sector, microsecond-level delays can have multi-million-dollar impacts, making PREEMPT_RT critical.
Although PREEMPT_RT is a powerful solution, it does not deliver perfect performance in all scenarios. For example, in applications requiring hard real-time guarantees such as aircraft control systems and nuclear reactor control, its use is limited. The inability of COTS hardware to guarantee fully deterministic behavior and unpredictable delays caused by hardware interrupts and SMI sources are the main reasons for these limitations.
1. Performance overhead must also be considered. Context switching costs have increased, resulting in a 5–15% reduction in system efficiency and higher memory usage due to additional threads. While these costs are acceptable for most applications, they require attention in high-performance scenarios.
2. Multi-core system challenges are significant issues in modern hardware. Delays from cache coherency protocols, memory access uncertainty in NUMA architectures, and increased resource contention are primary concerns. Lock contention problems become especially pronounced in systems with a high number of cores.
3. Hardware dependencies are also important factors. Network cards and storage devices that do not exhibit real-time behavior, GPU and specialized hardware driver compatibility issues with PREEMPT_RT, and the impact of power management features on real-time performance fall under this category.
When PREEMPT_RT is insufficient, alternative solutions are available. These can be categorized into three groups: specialized real-time operating systems, hypervisor-based solutions, and hardware-based solutions.
1. Specialized real-time operating systems include QNX, which has achieved success in industrial systems and the automotive sector; VxWorks, with its long-standing use in aerospace and defense; RTEMS, effective in experimental physics and space applications; and FreeRTOS, widely used in microcontroller applications.
2. Hypervisor-based solutions are gaining increasing importance in modern systems. This approach uses virtualization to partition hardware resources, allowing real-time and non-real-time workloads to coexist on the same platform while preserving scheduling guarantees. The QNX Hypervisor’s ability to run infotainment systems, vehicle control panels, and safety-critical ADAS functions on the same hardware, and Wind River Helix’s isolation of safety-critical life-support functions from user interfaces in medical applications, are successful examples of this approach.
3. Hardware-based solutions are preferred in applications demanding the highest performance and deterministic behavior, particularly those using FPGA implementations and specialized DSPs. These solutions eliminate software-layer uncertainties by providing real-time behavior at the hardware level.
Classification of Real-Time Systems
Linux and Real-Time Challenges
Linux PREEMPT_RT Patch: A Comprehensive Solution
Key Improvements
Industrial Applications
Limits and Considerations
Alternative Solutions