This OS quietly powers all AI - and most future IT jobs, too

Elyse Betters Picaro / ZDNET

Follow ZDNET: Add us as a preferred source on Google.

ZDNET's key takeaways

• AI runs on Linux. Period. There are no substitutes.
• Canonical and Red Hat are building Nvidia Vera Rubin-specific Linux distros.
• The Linux kernel is being tuned for AI and ML workloads.

Modern AI began with open source, and it ran on Linux. Today, Linux isn't just important for artificial intelligence; it's the foundation upon which today's entire modern AI stack runs. From hyperscale training clusters down to edge inference boxes, it's all Linux from top to bottom.

Also: I tried a Linux distro that promises free, built-in AI - and things got weird

AI's magic tricks are really the aggregate output of very prosaic infrastructure: supercomputers, GPU farms, and cloud clusters that almost all run some flavor of Linux. The core machine-learning frameworks -- TensorFlow, PyTorch, scikit-learn, and friends -- were all developed and tuned first on Linux. Tooling around these tools, from Jupyter and Anaconda to Docker and Kubernetes, is similarly optimized for Linux. 

Why IT jobs will live and die on Linux

Why? Because it's on Linux where researchers and production engineers actually deploy AI. Future IT jobs will live and die on Linux. 

You see, AI runs on Linux because it's the most flexible, powerful, and scalable environment for the GPU‑heavy, distributed workloads modern AI requires. In addition, the entire tooling and cloud ecosystem has standardized on Linux. 

Yes, every AI platform, whether it's OpenAI, Copilot, Perplexity, Anthropic, or your favorite AI chatbot, is built on Linux, plus drivers, libraries, and orchestration, all glued together in different ways. The proprietary bits may grab the branding, but without Linux, they're nowhere.

That translates into more Linux jobs. 

As the Linux Foundation's 2025 State of Tech Talent Report noted, AI is driving a net increase in tech jobs, particularly Linux jobs. What this looks like comes down to "AI [is] reshaping roles rather than eliminating them," according to the report, "leading to shifts in skill demand and new opportunities for workforce growth." 

Besides increasing Linux system and network administration jobs, the site Linux Careers sees "a rapidly emerging trend involving professionals who combine Linux expertise with artificial intelligence and machine learning operations." Such new AI/Linux jobs include AI Operations Specialist, MLOps Engineer, ML Engineer, and DevOps/AI Engineer.

Of course, Linux distributors know all this, which is why, when new Linux distros are released, their makers emphasize AI features.

For example, Canonical and Red Hat are racing to plant their Linux flags on Nvidia's new Vera Rubin AI supercomputer platform. The race is on to see who will own the operating system layer of "gigascale AI factories."

For its part, Red Hat is introducing Red Hat Enterprise Linux (RHEL) for Nvidia. This curated edition of RHEL is optimized specifically for Nvidia's Rubin platform, including the Vera Rubin NVL72 rack-scale systems. 

Also: How AI and Rust are rewriting Linux and Windows programming

The company says this variant will ship with Day 0 support for the Vera CPU, Rubin GPUs, and Nvidia's CUDA X stack, with validated OpenRM drivers and toolkits delivered directly through Red Hat repositories.

The Linux kernel and AI

Canonical is also rolling out official Ubuntu support for the Nvidia Rubin platform, also targeting the Vera Rubin NVL72. The London-headquartered company is anchoring its story around making the custom Arm-based Vera CPU a "first-class citizen," with x86 parity in its forthcoming Ubuntu 26.04 release. 

So, unlike Red Hat, which has a RHEL just for Nvidia's processors, the new Ubuntu will support Nvidia. This version will also upstream features such as Nested Virtualization and ARM Memory Partitioning and Monitoring (MPAM) to better partition memory bandwidth and cache for multi-tenant AI workloads.

Also: Even Linus Torvalds is vibe coding now

What runs all this is a Linux kernel that has been steadily modified to keep up with AI's voracious appetite for hardware acceleration. Modern kernels juggle GPU and specialized accelerator drivers, sophisticated memory management for moving tensors around quickly, and schedulers tuned for massively parallel batch jobs. 

In short, the kernel has been rewired over the last decade to become an operating system for AI hardware accelerators.

Memory: putting data where the GPUs are

Specifically, one of the most important enablers has been Heterogeneous Memory Management. This enables device memory, such as Graphics Processing Unit/Video Random Access Memory (GPU VRAM), to be integrated into Linux's virtual memory subsystem. 

That, combined with Direct Memory Access Buffering (DMA-BUF) and Non-Uniform Memory Access (NUMA) optimization, enables AI runtimes to keep tensors close to the accelerator and cut back on data copying, which tends to slow down performance.

Also: Here's how I finally cracked a tricky Linux problem with this AI terminal app

Recent kernels also treat advanced CPU-GPU combinations, such as tightly coupled NUMA-style CPU/GPU nodes, as first-class citizens. With this, memory can be migrated between CPU-attached RAM and high-bandwidth GPU memory on demand. 

This, as Nvidia explained, "enables the CPU and GPU to share a single per-process page table, enabling all CPU and GPU threads to access all system-allocated memory."

Accelerators: a real subsystem, not an add-on

Linux now has a dedicated compute accelerators subsystem that's designed to expose GPUs, Tensor Processing Units (TPUs), and custom AI application-specific integrated circuits (ASICs) to your AI and machine learning (ML) programs. 

On top of that, GPU support has matured from graphics-first to compute-heavy, via the Direct Rendering Manager (DRM), open stacks like ROCm and OpenCL, and Nvidia's Compute Unified Device Architecture (CUDA) drivers.

Also: AI is already part of Linux's plumbing - whether developers like it or not

Kernel work has expanded to cover newer AI accelerators such as Intel's Habana Gaudi, Google's Edge TPU, and FPGA/ASIC boards, with drivers and bus abstractions. This enables AI programs such as PyTorch or TensorFlow to see and use them as just another device. Thus, anyone making new AI silicon today rightly assumes that Linux will be running on it.

Scheduling: feeding hungry accelerators

Linux's default scheduler, the Earliest Eligible Virtual Deadline First (EEVDF), real-time scheduler, and NUMA balancing have all been tuned to enable AI workloads to pin CPUs, isolate noisy neighbors, and feed accelerators without jitter. Work on raising the default kernel timer frequency from 250 Hz to 1000 Hz is already showing measurable boosts in Large Language Model (LLM) acceleration with negligible power cost. 

Also: I found 7 essential Linux apps for students - including a local AI

While not a Linux default setting, some distros, like the Ubuntu low-latency kernels, now come with this as a standard setting.

Direct paths: cutting out the CPU middleman

Modern kernels allow GPUs to access memory, storage, and even peer devices directly, using technologies such as Nvidia's GPUDirect and peer-to-peer DMA. Combined with Compute Express Link (CXL) and improved Input/Output Memory Management Unit (IOMMU) handling, it enables accelerators to bypass the CPU when moving data. This eliminates bottlenecks that previously stalled ML training runs. This invisible plumbing is why AI clusters can scale out without collapsing under their own I/O.

Also: My two favorite AI apps on Linux - and how I use them to get more done

What all this adds up to is that, when executives talk about "AI strategy," what they're not saying is that the unglamorous reality is that AI strategy depends on managing Linux at scale. It's all about patching kernels, hardening containers, and securing opaque workloads. AI may get the headlines, but Linux remains the operating system doing the actual work.