Yuan Tang

Production-Grade LLM Inference at Scale with KServe, llm-d, and vLLM

Tue, 21 Apr 2026 00:00:00 +0000

Authors:

Yuan Tang: Project Lead, KServe; Senior Principal Software Engineer, Red Hat
Scott Cabrinha: Staff Site Reliability Engineer, Tesla
Robert Shaw: Director of Engineering, Red Hat
Sai Krishna: Staff Software Engineer, Tesla

Originally posted on llm-d Blog and on KServe Blog.

The Problem with “Simple” LLM Deployments

Deploying large language models is becoming common, but the real difficulty lies in managing hundreds of LLMs efficiently at scale. Our initial approach—a straightforward vLLM deployment wrapped in Kubernetes StatefulSet—created three major operational challenges:

Storage Performance Issues: Large models like Llama 3 exceed hundreds of gigabytes. Network storage solutions proved impractical due to sluggish performance with massive safetensors files.
Infrastructure Constraints: Local LVM persistent volumes resolved speed issues but introduced rigid node-to-pod affinity requirements. Hardware failures necessitated manual intervention to delete Persistent Volume Claims and reschedule pods—an unacceptable operational burden.
Inefficient Load Balancing: Simple round-robin strategies fail to leverage the KV-cache on GPUs, a critical vLLM feature that maximizes throughput. Given GPU cost sensitivity, optimizing efficiency is non-negotiable.

What We Needed from an Operator

The team identified specific requirements for a Kubernetes Operator designed for AI/ML workloads:

Full specification-level customization beyond typical Custom Resource capabilities
Flexible deployment patterns rather than rigid prefill/decode architectures
Integration with standard Kubernetes APIs rather than novel abstraction layers

The Winning Combination: KServe + llm-d + vLLM

The solution combined three technologies: llm-d (powered by KServe and its Inference Gateway Extension), Envoy, and Envoy AI Gateway.

Key advantages delivered:

Deep Customization: LLMInferenceService and LLMInferenceConfig objects expose standard Kubernetes APIs, enabling precise specification overrides for specialized hardware and rapid flag implementation.
Intelligent Routing: Integration of Envoy, Envoy AI Gateway, and Gateway API Inference Extension enables “prefix-cache aware routing.” This directs requests to appropriate vLLM instances, maximizing KV-cache utilization and GPU efficiency.

Performance Results:

Testing with Llama 3.1 70B on 4 MI300X AMD GPUs (tensor-parallel-size=4, gpu-memory-utilization=0.90, max-model-len=65536) showed:

3x improvement in output tokens/second
2x reduction in time to first token (TTFT)

These improvements occurred after enabling prefix-cache aware routing.

Community Contributions and Collaboration

Production deployment surfaced issues fixed upstream in KServe, benefiting the broader community:

New feature requests: Issues #4901, #4900, #4898, #4899
Made storageInitializer optional (kserve#4970) — allowing RunAI Model Streamer as an alternative to default storage initialization
Added support for latest Gateway API Inference Extension (kserve#4886)

These contributions emerged directly from encountering production edge cases, with validation of KServe and llm-d at scale helping strengthen the platform for all Kubernetes LLM workloads.

Acknowledgement

We’d like to thank the following community contributors from Red Hat and Tesla:

Red Hat: Sergey Bekkerman, Nati Fridman, Killian Golds, Andres Llausas, Bartosz Majsak, Greg Pereira, Pierangelo Di Pilato, Ran Pollak, Vivek Karunai Kiri Ragavan, Robert Shaw, Yuan Tang

Tesla: Scott Cabrinha, Sai Krishna

Get Involved with llm-d

Explore code at the GitHub organization
Join the Slack community for direct maintainer contact
Attend open community calls (Wednesdays 12:30pm ET)
Follow project updates on Twitter/X, Bluesky, and LinkedIn
Watch demos on the llm-d YouTube channel
Review documentation on the community page

Best of Both Worlds: Cloud-Native AI Inference at Scale using KServe and llm-d

Thu, 05 Mar 2026 00:00:00 +0000

Authors:

Yuan Tang: Project Lead, KServe; Senior Principal Software Engineer
Ran Pollak: Manager, AI Catalyst at Red Hat

Originally posted on KServe Blog. Also available on Red Hat Developers.

Enterprises today seek to integrate generative AI (GenAI) capabilities into their applications. However, scaling large AI models introduces complexity: managing high-volume traffic from large language models (LLMs), optimizing inference performance, maintaining predictable latency, and controlling infrastructure costs.

Platform engineering leaders require more than just model deployment capabilities. They need a robust, Kubernetes-native infrastructure that supports:

Efficient GPU utilization
Intelligent request routing
Distributed inference patterns
Cost-aware autoscaling
Production-grade governance

This article demonstrates how two open-source solutions, KServe and llm-d, can be combined to address these challenges.

We explore the role of each solution, illustrate their integration architecture, and provide practical guidance for AI platform teams, with deeper focus on KServe’s LLMInferenceService, available since KServe v0.16.

KServe: Simplified Deployment of AI Models on Kubernetes

KServe is a Kubernetes-based model serving platform that simplifies deploying and managing ML models, including LLMs, at scale.

For platform engineers, KServe acts as the model serving control plane: the layer responsible for lifecycle, scaling, and operational governance.

Inference as a Service

InferenceService serves as KServe’s core abstraction for model deployment, encapsulating the full serving lifecycle, including:

Automatic deployment creation and reconciliation
Request-based autoscaling with scale-to-zero and autoscaling based on custom metrics
Revision management and canary rollouts
Endpoint exposure and traffic routing
Runtime abstraction across serving backends for both predictive and generative AI
Optional pre-processing/post-processing, inference pipelines, and ensembles

ML engineers provide trained models. Platform engineers retain operational control without writing custom deployment code.

LLMInferenceService in KServe

KServe v0.16 introduces stronger generative AI capabilities, including LLMInferenceService, designed specifically for large language model workloads.

Unlike traditional stateless predictors, LLM workloads require:

Long-running streaming responses
GPU-heavy memory footprints
Prefix KV-cache management
High-concurrency token streaming
OpenAI-compatible APIs

LLMInferenceService shares common foundations with InferenceService but introduces additional capabilities tailored for large language models, including:

Unlocking Generative AI Serving with LLMInferenceService: From Pod-Level Speed to Cluster-Wide Intelligence

Imagine you want to bring the power of generative AI directly into your applications, but without rewriting your entire stack. It offers OpenAI-compatible endpoints like /v1/chat/completions, complete with streaming token responses and multi-turn support. With prompt templating built in, developers can integrate seamlessly with existing tools—whether it’s the OpenAI SDKs, LangChain, LlamaIndex, Llama Stack, RAG frameworks, or even enterprise GenAI gateways.

Under the hood, KServe connects to LLM-optimized runtimes such as vLLM, Hugging Face TGI, or other GPU-native backends. These engines bring advanced capabilities like continuous batching, memory-efficient paged attention, and KV-cache reuse, delivering high throughput per GPU.

Yet, while these runtime-level optimizations make each pod lightning fast, true cluster-wide efficiency needs more. That’s exactly the role of llm-d: adding an extra layer of intelligence that orchestrates resources and maximizes performance across the entire deployment.

Distributed & Multi-Node Model Support

LLMInferenceService supports advanced parallelism strategies implemented by runtimes, including tensor parallelism, pipeline parallelism, and multi-GPU sharding.

This enables hosting 70B+ parameter models, partitioning models across nodes, and serving models larger than single-GPU memory.

KServe orchestrates the deployment topology, while the runtime manages execution parallelism.

Advanced Autoscaling & Networking (Including Scale-to-Zero)

KServe integrates deeply with Kubernetes to support request- and concurrency-based autoscaling via Knative, GPU-backed scaling, and scale-to-zero for cost control.

It also integrates with the Kubernetes Gateway API for TLS termination, traffic splitting, and advanced routing.

This makes it suitable for development environments, internal copilots, and large-scale production workloads.

Kubernetes Gateway API Integration

KServe integrates with Kubernetes Gateway API for:

Enterprise-grade routing
TLS termination
Traffic splitting
Multi-model routing

This enables integration with modern Kubernetes networking stacks.

Where KServe Alone Is Not Enough

Even with LLMInferenceService and optimized runtimes, KServe does not inherently:

Route requests based on KV-cache locality across replicas
Separate prefill and decode cluster-wide
Perform SLA-aware routing decisions
Optimize GPU utilization across multiple pods

To address these, we introduce llm-d.

llm-d: Distributed Intelligence for LLM Inference

llm-d is a Kubernetes-native distributed inference framework designed to enhance performance and efficiency of LLM workloads.

If KServe is the control plane for models, llm-d is the distributed intelligence scheduling layer.

KV-Cache Aware Scheduling and Disaggregated Inference with llm-d

As LLM deployments mature, scaling is no longer just about adding GPUs. It’s about using them intelligently. Modern runtimes such as vLLM introduced prefix (KV) caching to reduce redundant computation, but without smart scheduling, much of that benefit is lost.

This is where llm-d changes the game.

Disaggregated Inference (Prefill / Decode Separation)

LLM inference consists of two distinct phases: prefill and decode. The prefill phase is compute-heavy, processing the full prompt and building the model’s attention context. The decode phase is latency-sensitive, generating tokens step by step where responsiveness directly impacts user experience.

llm-d separates these phases across different GPU groups, assigning compute-optimized resources to prefill and latency-optimized resources to decode. With intelligent scheduling between them, workloads are aligned to the right hardware profile.

This phase-aware architecture increases GPU utilization, reduces tail latency, and lowers cost per token by eliminating resource contention between fundamentally different workloads.

Intelligent Inference Scheduler

llm-d’s inference scheduler evaluates the following metrics:

GPU utilization
Queue depth
Cache residency
SLA constraints
Load distribution

It enhances load balancing with an intelligent scheduler to decrease serving latency and increase throughput with prefix-cache aware routing, utilization-based load balancing, fairness and prioritization for multi-tenant serving, and predicted latency balancing.

KServe LLMInferenceService and llm-d

Responsibility Separation

This layered design ensures composability and specialization, providing a complete, production-ready solution for generative AI. KServe acts as the control plane and LLMInferenceService delivers the generative API abstraction, while llm-d provides the cluster-wide optimization.

Layer	Responsibility
KServe	Model lifecycle, scaling, governance
LLMInferenceService	Generative API abstraction
vLLM	Efficient execution inside runtime
llm-d	Cross-runtime routing & cache awareness
Kubernetes	Resource orchestration

Together, KServe and llm-d enable a production-ready, Kubernetes-native inference platform that balances scalability, performance, and cost efficiency, providing the best of both worlds for cloud-native AI inference at scale.

Cost Efficiency Comparison: Naive vs Optimized

Serving LLMs at scale is no longer just a model problem. It is a distributed systems problem where naive load balancing leads to significant inefficiencies and wasted resources — lost cache locality, GPU imbalance, redundant prefill processing, high tail latency, and overprovisioned GPUs.

Naive Problems:

Cache locality loss
GPU imbalance
Redundant prefill processing
High tail latency
Overprovisioned GPUs

Optimized Architecture with KServe + llm-d

The combined KServe and llm-d solution introduces distributed intelligence to solve the problems of naive architectures, delivering superior performance, scalability, and cost control. This optimized architecture is pluggable and extensible to work well with many AI and cloud-native technologies.

Benefits:

Cache reuse preserved
Balanced GPU utilization
Reduced recomputation
Lower cost per token
Controlled autoscaling via LLMInferenceService

Benchmark Results: Why Cluster-Level Intelligence Matters

By integrating llm-d’s cache-aware routing, prefill and decode disaggregation, and SLA-based scheduling with KServe’s enterprise-grade generative serving and autoscaling, the system achieves cluster-wide GPU optimization.

Note: The following results are based on benchmarks published by the llm-d project

Optimization Area	Naive Architecture (Round Robin LB)	Optimized (KServe + llm-d)	Source
Cache Locality	Requests routed randomly → KV cache frequently missed	Cache-aware routing preserves prefix locality	llm-d blog
Time to First Token (P90)	Baseline latency under cache-blind scheduling	Up to ~57× faster P90 TTFT in benchmark	llm-d blog
Token Throughput	~4,400 tokens/sec (baseline test cluster)	~8,730 tokens/sec (~2× improvement)	llm-d blog
Throughput at Scale	Degrades under multi-tenant load	Sustained 4.5k–11k tokens/sec	llm-d blog
Tail Latency (P95/P99)	Higher tail latency due to stragglers & imbalance	~50% tail latency reduction (reported tests)	Red Hat Developers
GPU Utilization	Uneven utilization, idle GPUs possible	Improved effective utilization via routing intelligence	llm-d docs
Autoscaling Control	Scale reacts to load only	Works with KServe autoscaling + routing intelligence	KServe docs

Modern GenAI platforms require cache locality awareness, phase-aware scheduling, distributed intelligence, and composable Kubernetes-native design. This combination ensures a production-ready system that meets the demands of large-scale production workloads.

Next Steps

Explore detailed project documentation:

KServe
llm-d

Engage with community resources and Slack channels to stay updated and contribute to ongoing developments:

Kubernetes Serving Working Group Has Succeeded and Will Be Disbanded

Fri, 13 Feb 2026 00:00:00 +0000

Originally posted on Kubernetes dev mailing list and on CNCF Blog.

We’d like to announce that Kubernetes WG Serving has succeeded and will be disbanded! Thank you everyone who have participated and contributed to the discussions and initiatives!

The Kubernetes Working Group Serving was created to support development of AI Inference stack on Kubernetes. The goal of this working group is to ensure that the Kubernetes is an orchestration platform of choice for Inference workload. This goal was accomplished and we are disbanding the working group.

The WG Serving formed workstreams to collect requirements from various model servers, hardware providers, and inference vendors. This work resulted in a common understanding of inference workload specifics and trends and laid the foundation for improvements across many SIGs in Kubernetes.

The working group oversaw several key evolutions to the role of load balancing and workloads - the inference gateway was adopted as a request scheduler, multiple groups have worked to standardize AI gateway functionality, and early inference gateway participants went on to seed agent networking in SIG Network. The use cases and problem statements informed the design of AIBrix. And many of the unresolved problems in distributed inference - especially benchmarking and recommended best practices - have been picked up by the llm-d project which hybridizes the infrastructure and ML ecosystems and is better able to steer model server co-evolution.

In particular, we believe llm-d and AIBrix represent more appropriate forums for driving requirements to Kubernetes SIGs than this working group. llm-d’s goal is to provide well-lit paths for achieving state-of-the-art inference and aims to provide recommendations that can compose into existing inference user platforms. AIBrix provides a complete platform solution for cost efficient LLM inference.

WG Serving helped with Kubernetes AI Conformance requirements and llm-d leveraging multiple components from the profile and making recommendations to end users consistent with Kubernetes direction (Kueue, inference gateway, LWS, DRA, etc.). Widely adopted patterns and solutions are expected to go into the conformance program.

All the efforts currently running inside the WG Serving can be migrated to other WGs or to SIGs directly, requirements for them will be discussed in SIGs and llm-d community. Specifically:

Autoscaling questions - mostly related to fast bootstrap - will be either SIG Node or SIG Scheduling.
Multi-host, multi-node work can continue as part of the SIG Apps (e.g. for LWS project) and DRA requirements discussed in WG Device Management.
Orchestration will be covered by SIG Scheduling and SIG Node
Requirements for DRA will be discussed in WG Device Management.

The Gateway API Inference Extension project is already sponsored by SIG Network and it will stay this way.

The Serving Catalog work can be moved to the Inference Perf project. Originally it was designed for a larger scope, but was used mostly for Inference perf since.

The Inference Perf project is sponsored by SIG Scalability and no change of ownership is needed.

Cheers,
Yuan Tang On behalf of Kubernetes WG Serving Co-Chairs

Feeling Thankful Today and Reflecting on Two Incredible Years at Red Hat

Fri, 28 Nov 2025 00:00:00 +0000

Originally posted on my Substack.

While my official two-year anniversary is just a few days away, Thanksgiving feels like the perfect time to express my gratitude for this journey.

I still remember coming across a job posting shared by Jeremy Eder in the Kubeflow Slack channel. It described everything I hoped to do next. That led to interview panels with amazing colleagues (Daniele Zonca, Edson Tirelli, Jessica Forrester, Antonin Stefanutti) and my manager Taneem Ibrahim for the first year.

I’m deeply grateful to the Red Hat leadership, John Graham, Sherard Griffin, Steven Huels, Taneem Ibrahim, Jennifer Albertson, Chris Wright, and Matt Hicks, for their incredible support and trust, and for giving me the opportunity to take on technical leadership role working so closely with the open source community.

A heartfelt thank you to my mentors: Daniele Zonca, Jessica Forrester, Jason Greene, and Adam Miller. My journey wouldn’t have been this fruitful without your support, guidance, and encouragement along the way.

Earlier this year, I was promoted to Senior Principal Engineer - a milestone made possible by the incredible mentors and colleagues who supported me along the way. I’m especially grateful to those who took the time to write thoughtful reference letters. Jeremy Eder, Carlos Costa, Mrunal Patel, Huamin Chen, Andrew Block, Vincent Caldeira, Adam Tetelman, Josh Bottum, Andrey Velichkevich, Carlos Eduardo Arango Gutierrez, James Wu, Johnu George

None of this would have been possible without my wife and our three wonderful kids. Thank you for being my foundation and inspiration.

To all the collaborators and partners in the open source community: thank you. Every conversation and collaboration has meant a lot.

The past two years have been the most fulfilling and accomplished in my career. Some highlights:

Delivered sessions and keynotes in front of up to 9,000 audience; participated and organized panels; and served as co-chairs at major industry conferences.
Co-founded and co-chaired Kubernetes WG Serving, WG AI Conformance, and Cloud Native Computing Foundation (CNCF) TAG Workloads Foundation.
Represented Red Hat in Kubeflow Steering Committee and the LF AI & Data Foundation Technical Advisory Council.
Became a maintainer and project lead of KServe and led successful transition to CNCF.
Contributed to Llama Stack in collaboration with Meta, which led to organization-wide adoption.
Served as a Staff Engineer in the technical leadership team, helping shape our technical strategy and product alignment.
Published the State of Model Serving Communities newsletter, which fostered collaborations across internal teams. Now with over 1k subscribers externally.
Filed 5 patent applications and published 10 blog posts.
Received organization-wide awards such as the IBM Tech Leadership and Red Hat AI Engineering Jedi Award.

Time flies! Looking forward to many more years to come!

KubeCon North America 2025: Red Hat AI Model Serving Highlights

Tue, 18 Nov 2025 00:00:00 +0000

Originally posted on InferenceOps Substack newsletter.

Time: Nov 9th - 13th, 2025
Location: Atlanta, Georgia, USA

Executive Summary

Red Hat AI had a great presence at KubeCon. Highlights of what we’ve accomplished this week:
- Delivered the day 1 sponsored keynote that focuses on security and AI.
- Got invited to the opening keynotes to talk about K8s AI Conformance Program. Red Hat has been leading this effort and is among first vendors that are certified.
- Co-chaired the two main AI related co-located events: Cloud Native AI Day and Kubeflow Summit.
- Showcased our model serving related projects in many demos, breakout sessions, and media interviews.
- KServe joined CNCF as an incubating project and many people joined our session and at the booth for questions.
AI on K8s
- The community started collaborating and actively addressing pain points to improve AI workloads on K8s.
- Many K8s vendors have adopted the new K8s AI Conformance Program that we’ve been organizing, which was announced at the opening keynote.
- Many model serving related initiatives and projects exist but more collaborations would be useful to share solutions to common LLM inference challenges.
Hot topics
- Security and observability for AI agents and workloads are increasingly important.
- Kubernetes AI Conformance Program has been announced and many vendors have already adopted it.
- Dynamic Resource Allocation (DRA) is now GA and delivers 50-70% GPU efficiency gains.
- The EU Cyber Resilience Act and AI sovereign drive compliance-first infrastructure decisions.
Event statistics
- 9,000+ attendees
- 19+ co-located events
- 270 vendor booths
- 67 project booths
- 34 Red Hat sessions

Red Hat Booth

Red Hat AI team members and Kubeflow + KServe community members

Keynotes

Opening Keynote (Kubernetes AI Conformance)

The CNCF leadership invited the co-chairs of the Kubernetes WG AI Conformance, including Yuan Tang (Red Hat), Rita Zhang (Microsoft), Janet Kuo (Google), and Mario Fahlandt (Kubermatic), to share their experiences and vision of the Kubernetes AI Conformance Program.

Special thanks to Laura Santamaria, Derek Carr, and Mrunal Patel for offering insights from both community and technical perspectives as well as Jessica Forrester and Jason Greene for their support throughout the journey.

Red Hat has been leading this effort and is among first vendors that are certified:

Yuan on the keynote stage, together with other program co-chairs of K8s WG AI Conformance:

Other sessions

llm-d

Sponsored demo on llm-d by Christopher Nuland at the demo theater located in solutions showcase area:

llm-d: Multi-Accelerator LLM Inference on Kubernetes by Erwan Gallen: https://sched.co/27Fee

Intelligent routing

Intelligent LLM Routing: A New Paradigm for Multi-Model AI Orchestration in Kubernetes by Chen Wang (IBM Research) and Huamin Chen (Red Hat)

You got a match! LLM Prefix Aware Routing with Kubernetes by Ricardo Noriega (Red Hat) and Cong Liu (Google)

Routing Stateful AI Workloads in Kubernetes by Maroon Ayoub (IBM) and Michey Mehta (Red Hat)

Benchmarking

Samuel Monson (Red Hat) and Jing Chen (IBM Research) delivered a tutorial on distributed LLM inference benchmarking with Junchen Jiang (University of Chicago), Ganesh Kudleppanavar (NVIDIA), and Jason Kramberger (Google).

Session: https://sched.co/27FXL

KServe

Yuan Tang (Red Hat) and Dan Sun (Co-founder of KServe, Bloomberg) shared the project’s current state and future direction.

Session: https://sched.co/28D4J

K8s WG Serving

Yuan Tang (Red Hat) shared the current state of model inference on Kubernetes with Rita Zhang (Microsoft), Jiaxin Shan (Bytedance), and Sergey Kanzhelev (Google) as well as efforts and initiatives from K8s WG Serving.

Session: https://sched.co/27Nlv

Media interview (KServe and vLLM Semantic Router)

Huamin Chen and Yuan Tang joined a media interview to discuss KServe joining CNCF and the new vLLM semantic router project.

Other relevant sessions

Full list of sessions from Red Hat: https://kccncna2025.sched.com/?searchstring=red+hat

Relevant events

Cloud Native & Kubernetes AI Day

300+ attendees
Yuan Tang co-chaired the event to ensure the event ran smoothly from start to finish and delivered the closing remarks

Kubernetes SIGs Lunch and Learn

Representation for AI related WGs: WG Serving and WG AI Conformance

Red Hat Time Machine

Invited community members and collaborators to join this event and had great discussions

Kubeflow Summit and project booth

Valentina Rodriguez Sosa Co-chaired Kubeflow Summit and helped coordinate booth schedules for the Kubeflow project and ensured our presence at the booth through the week

Upcoming KubeCons

Dates and locations for upcoming KubeCons in 2026 were announced during the opening keynote:

KubeCon Europe in Amsterdam, Netherlands (March 23–26)
KubeCon India in Mumbai (June 18 - 19)
KubeCon Japan in Yokohama (July 29 - 30)
KubeCon China in Shanghai (September 8 - 9)
KubeCon North America in Salt Lake City, Utah (November 9 - 12)

See you in 2026!

KServe Joins CNCF as an Incubating Project

Tue, 11 Nov 2025 00:00:00 +0000

Originally posted on Red Hat Blog. In addition, check out the complementary announcement blog from CNCF.

We are excited to share that KServe, the leading standardized AI inference platform on Kubernetes, has been accepted as an incubating project by the Cloud Native Computing Foundation (CNCF).

This milestone validates KServe’s maturity, stability and role as the foundation for scalable, multi-framework model serving in production environments. By moving into the CNCF’s neutral governance, KServe’s development will be driven purely by community needs, accelerating its standardization for serving AI models on Kubernetes.

For Red Hat this is a validation of our commitment to delivering open, reliable and standardized AI solutions for the hybrid cloud.

The critical engine behind Red Hat OpenShift AI

At Red Hat, we believe the best AI infrastructure is built on open standards and Kubernetes. KServe is the critical model serving component that powers Red Hat OpenShift AI, helping ensure our customers can transition from model experimentation to production inference seamlessly and at scale.

OpenShift AI leverages KServe’s features to solve the biggest enterprise AI challenges, helping enterprises realize:

High-performance LLM optimization - KServe is optimized for large language models (LLMs), providing high-performance features like KV cache offloading, distributed inference with vLLM, as well as disaggregated serving, pre-fix caching, intelligent scheduling and variant autoscaling via the integration with llm-d.
Advanced autoscaling - In addition to the horizontal pod autoscaling capability from Kubernetes, KServe also supports autoscaling with KEDA (Kubernetes Event-driven Autoscaler), which enables event-driven scaling based on external metrics such as vLLM metrics.
Both predictive and generative AI model inference - KServe supports pluggable, reusable, extensible runtimes, ranging from scikit-learn and XGBoost for predictive AI to Hugging Face and vLLM for generative AI model inference. This helps ensure that enterprises can switch to the best runtime for specific use cases.

Unlocking enterprise AI value

The journey of AI from the lab to the bottom line requires production infrastructure that can handle exponential growth, especially as enterprise usage shifts to widespread generative applications.

Now bolstered by the full resources and neutral governance of the CNCF, KServe directly addresses these core operational challenges - from tackling complexity with a unified API to controlling cloud costs through its scale-to-zero capabilities.

This move offers enterprises confidence in the longevity, security and open future of their AI infrastructure investment. KServe is now positioned to be the open-source standard for cloud-native model serving, empowering enterprises to confidently build, deploy and scale the next generation of intelligent applications on a foundation of open standards and community-driven innovation.

Join the Movement!

We invite the community to join us in congratulating the KServe maintainers and contributors on this achievement.

Want to get involved and shape the future of AI inference? Here’s how you can join the journey:

Check out KServe GitHub repo: https://github.com/kserve/kserve
Join KServe community: https://github.com/kserve/community

Additionally, join us at our KubeCon+CloudNativeCon NA sessions to hear more about KServe:

Anchoring Trust in the Age of AI: Identities Across Humans, Machines, and Models - Tuesday November 11, 2025 10:08 am ET

The Best Choice for AI Inference: vLLM

Thu, 09 Oct 2025 00:00:00 +0000

Originally posted on Medium and on Red Hat Blog.

Authors: Fatih E. Nar, Greg Pereira, Yuan Tang, Robert Shaw, Anish Asthana

As organizations move from Large language model (LLM) experimentation to production deployment, the choice of inference platform becomes a critical business decision. This choice impacts not just operational performance, and also flexibility, cost optimization, and the ability to adapt to rapidly evolving business needs.

For technologists & solution architects evaluating LLM inference platforms, three key considerations should drive the decision:

Architectural Flexibility: The ability to deploy across diverse hardware accelerators and hybrid cloud environments without vendor lock-in.
Operational Scalability: Support for advanced deployment patterns that can scale from single-GPU deployments to distributed, multi-node architectures.
Ecosystem Openness: Ecosystem Openness: Compatibility with the broadest range of models and kernel support as well as integration with a wide range & variety of enterprise-software ecosystem.

vLLM, uniquely addresses these considerations through its open-source foundation, advanced memory management capabilities, and upcoming distributed deployment blueprints. Unlike proprietary or hardware-specific solutions, this combination provides the flexibility to optimize for cost, performance, and operational requirements as they evolve.

This article examines why vLLM’s technical architecture & abilities, particularly its KV-Cache management, parallelization strategies, and with the upcoming llm-d distributed capabilities provides the most sustainable path for production LLM deployment.

The Open-Source Advantage

Community-Driven Innovation at Scale

The evolution of LLM inference has been fundamentally shaped by open-source innovation. vLLM has achieved remarkable success in supporting diverse models, features, and hardware backends over the past 1.5 years vLLM V1: A Major Upgrade to vLLM’s Core Architecture - vLLM Blog, growing from a UC Berkeley research project to become the de facto serving solution for the open-source AI ecosystem vLLM 2024 Retrospective and 2025 Vision - vLLM Blog.

This transformation illustrates a critical advantage; open-source projects can iterate and adapt faster than proprietary solutions. vLLM is now a hosted project under PyTorch Foundation GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs, ensuring long-term sustainability and governance that enterprises require.

Enterprise Support Meets Open Innovation

Red Hat’s approach to vLLM mirrors its successful Linux/Openstack/Kubernetes strategy, taking community-driven innovation and adding enterprise-grade support, security, and operational tooling. This model provides:

No Vendor Lock-in: Organizations can deploy on-premises, in public clouds, or hybrid environments.
Transparent Development: Security vulnerabilities and bugs are publicly tracked and rapidly addressed.
Community Contributions: Features developed by one organization benefit the entire ecosystem.
Flexibility to Customize: Source code access enables modifications for specific requirements.

On top, vLLM additionally provides several strategic advantages:

Hardware Independence: Unlike TensorRT-LLM (NVIDIA-specific), vLLM supports NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs,GPUs and XPUs, PowerPC CPUs, and TPU GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs.
Rapid Feature Adoption: vLLM V1 introduces a comprehensive re-architecture of its core components, including the scheduler, KV cache manager, worker, sampler, and API server vLLM V1: A Major Upgrade to vLLM’s Core Architecture - vLLM Blog.
Ecosystem Integration: Native compatibility with Hugging Face, OpenAI APIs, and Kubernetes ecosystems.
Cost Optimization: Freedom to choose the most cost-effective hardware for specific workloads.

Architectural Flexibility and Parallelization Strategies

Understanding Parallelization

LLMs present unique scaling challenges, for example; a 70B parameter model requires approximately 140GB of memory just for weights in FP16 precision, far exceeding single accelerator capacities. Red Hat Openshift AI (with vLLM inside) addresses such challenges through four complementary parallelization strategies, each solving different scaling challenges.

Data Parallelism (DP): Scaling Across Models

Data parallelism represents the simplest scaling pattern, running complete model replicas across multiple servers, with each processing different batches of requests. In this approach:

Maintains Full Models: Each accelerator/server holds the complete model weights.
Distributes Requests: Load balancers distribute incoming requests across replicas.
Enables Linear Scaling: Adding servers proportionally increases throughput.
Simplifies Deployment: No model sharding complexity.

       [Load Balancer]
          	    |
    -----------------------------------------
    |      	   |      	    |
[Server1] 	[Server2] 	[Server3]
[Model A] 	[Model A] 	[Model A]
    |      	   |      	    |
[Batch 1] 	[Batch 2] 	[Batch 3]

Key: Each server has complete model copy
     Different requests processed in parallel

This pattern works exceptionally well with Red Hat OpenShift AI (RHOAI)’s Model Serving capabilities based on KServe (with Replica-Sets for Model Copy Serving), enabling automatic scaling based on request load while maintaining model serving simplicity.

Pipeline Parallelism (PP): Layer-wise Distribution

Pipeline parallelism divides the model by layers, with different accelerators handling different neural network layers:

Sequential Processing: Requests flow through GPUs in sequence.
Memory Balance: Distributes memory requirements evenly.
Flexible Deployment: Can span multiple nodes without high-speed interconnects.
Micro-batching: Maintains GPU utilization through careful scheduling.

Request → [GPU1:NN-Layers 1-8] → [GPU2:NN-Layers 9-16] → [GPU3:NN-Layers 17-24] → Response
             ↓                    	↓                     	↓
          [Memory]             	[Memory]              	[Memory]
          
Timeline: Request flows sequentially through GPUs

While pipeline parallelism remains useful for certain architectures, it faces challenges with modern LLMs:

Pipeline Bubble Inefficiency: GPUs idle while waiting for data from previous stages.
Communication Overhead: Inter-stage data transfer becomes a bottleneck.
Latency Sensitivity: Each stage adds latency, impacting time-to-first-token.
Poor Fit for MoE Models: Mixture of Experts architectures with selective activation don’t map cleanly to sequential pipeline stages.

These limitations have led many production deployments to favor tensor parallelism for intra-node scaling and data parallelism for inter-node scaling, though PP still has value in specific scenarios like memory-constrained environments or when combined with other parallelization strategies.

Tensor Parallelism (TP): Distributing Model Weights

For models too large for a single accelerator, tensor parallelism splits model weights across multiple GPUs:

Horizontal Layer Splitting: Each matrix multiplication is distributed across GPUs.
Synchronized Computation: GPUs communicate via high-speed interconnects (NVLink, infinity fabric).
Memory Efficiency: Enables serving models 4-8x larger than single GPU capacity.
Low Latency: Minimal communication overhead with proper hardware.

Original Matrix Operation:
[Large Weight Matrix] × [Input] = [Output]

With Tensor Parallelism:
[Weight Part 1] → GPU1 ↘
[Weight Part 2] → GPU2 → [Combine] → [Output]
[Weight Part 3] → GPU3 ↗

Key: Single matrix operation split across GPUs
     High-speed interconnect (NVLink) required

vLLM’s tensor parallel implementation is hardware-agnostic, supporting various interconnect technologies across different accelerator types.

Expert Parallelism (EP): Distributing MoE Experts across nodes

For Mixture-of-Experts (MoE) architectures, Expert Parallelism distributes individual experts across multiple GPUs or nodes. Instead of every GPU holding all experts, each device stores only a subset, and a router layer dynamically dispatches tokens to the appropriate expert(s).

Distributed Expert Sharding: Experts are partitioned across GPUs and nodes, allowing models with hundreds of experts to scale far beyond single-device memory limits.
Dynamic Token Routing: Each token is sent only to its assigned expert(s), reducing compute overhead compared to dense model execution.
Expert Parallel Load Balancing (EPLB): Prevents “hot” experts from overloading by dynamically replicating or redistributing popular experts.
Hierarchical Scheduling: In multi-node clusters, routing and replication are coordinated first across nodes and then across GPUs within each node, ensuring even utilization and minimal inter-node traffic.
Performance Gains: Enables higher throughput and efficiency for large-scale MoE models, maintaining near-linear scaling on high-speed interconnects (NVLink, InfiniBand).

   [Router Layer]
                    		|
        ---------------------------------------------------------------
        |      		|        		|       		|
      GPU1		GPU2     		GPU3    		GPU4
    [Exp 1,2] 		[Exp 3,4] 	[Exp 5,6] 	[Exp 7,8]
    
Token Flow: Router → Selected Expert(s) → Output

vLLM’s Unified Approach

What distinguishes vLLM is its ability to combine these strategies seamlessly. vLLM supports tensor, pipeline, data and expert parallelism for distributed inference, allowing organizations to:

Mix Strategies: Use tensor parallelism within nodes and pipeline parallelism across nodes.
Adapt to Hardware: Optimize based on available interconnects and GPU memory.
Scale Dynamically: Start with single-node deployment and scale as needed.
Maintain Compatibility: Same API regardless of parallelization strategy.

This flexibility becomes critical when deploying across hybrid cloud environments where hardware configurations vary between on-premises and cloud deployments.

Hybrid TP+PP+EP+DP Deployment (MoE Model):

Pipeline Stage 1:              	Pipeline Stage 2:              Pipeline Stage 3:
Node 1 (Attention Layers)     	 Node 2 (MoE FFN Layers)       Node 3 (Output Layers)
[Attn Part A] ←→ [Attn Part B] 	[Router + Experts Subset]      [Out Part A] ←→ [Out Part B]
    GPU1           GPU2         	GPU3: Experts 1-4                  GPU5            GPU6
      ↕ NVLink ↕               	GPU4: Experts 5-8                       ↕ NVLink ↕
                                    	    ↕ NVLink ↕

Expert Distribution Detail (Node 2):
            [Router]
         GPU3  |  GPU4
        -------|-------
        Exp1   |  Exp5
        Exp2   |  Exp6  
        Exp3   |  Exp7
        Exp4   |  Exp8

Request Flow: → [Attention] → [MoE Layer/Router] → [Output] → Response
                              ↓
                    [Route to specific experts]

Parallelization Breakdown:
- Tensor Parallel (TP): Attention and Output layers split within nodes.
- Pipeline Parallel (PP): Different model stages across nodes .
- Expert Parallel (EP): MoE experts distributed across GPUs in Node 2.
- Data Parallel (DP): Entire pipeline can be replicated for more throughput.

Better Memory Management with KV-Cache

Understanding KV-Cache in LLM Inference

The KV-Cache (Key-Value Cache) represents one of the most critical features in LLM inference optimization. During attention computation, models must access previous token representations -> a process that becomes memory-intensive as sequence lengths grow.

Efficient KV-Cache management can mean the difference between serving 10 concurrent users or 100 on the same hardware at the same time, since cache reuse trades computation for memory efficiency.

(Q * K^T) * V computation process with caching

Step 1 (Prefill Phase)
                    Keys Transpose
                          ↓
    Queries              Keys              Values            Results
       |                  |                  |                  |
    [  Q  ]      ×     [  K  ]      →    [Scores]     ×    [  V  ]    →    [Output]
       |                  |                  |                  |
      64                 64                                    64              64
                          ↓                                     ↓
                    [Storing K]                           [Storing V]
                          ↓                                     ↓
                    ←── Cache memory (stores both K and V) ──→

Step N (Decode Phase)
                    Keys Transpose
                          ↓
    Query              Cached K            Cached V          Results
       |                  |                  |                  |
    [  q  ]      ×   [K₁...Kₙ]      →    [Scores]     ×   [V₁...Vₙ]   →    [output]
    (new)             (from cache)                         (from cache)       (new)
       |                  |                  |                  |
       1                 64                                    64               1
                          ↑                                     ↑
                    ←── Reading from cache (both K and V) ──→

Key Points:
- Prefill: Computes K and V for all input tokens, stores both in cache
- Decode: For each new token, reads ALL previous K and V values from cache
- Both K and V matrices must be cached (not just V)
- Cache size grows with sequence length

Prefill vs. Decode: Two Distinct Phases

LLM inference consists of two fundamentally different phases:

Prefill Phase (Prompt Processing)

Processes all input tokens in parallel.
Compute-intensive with high GPU utilization.
Generates initial KV-Cache entries for all prompt tokens.
Latency proportional to prompt length.
Benefits from larger batch sizes.

Decode Phase (Token Generation)

Generates one token at a time sequentially.
Memory-bandwidth bound operation.
Reads the entire KV-Cache for each new token.
Latency proportional to number of output tokens.
Benefits from efficient cache management.

PagedAttention: vLLM’s Memory Revolution

vLLM introduced PagedAttention, a breakthrough in KV-Cache management that treats GPU memory like virtual memory in operating systems:

Non-contiguous Storage: KV-Cache blocks can be stored anywhere in GPU memory.
Dynamic Allocation: Memory allocated only as sequences grow.
Memory Sharing: Identical prompt prefixes share KV-Cache blocks.
Near-Zero Waste: Eliminates internal fragmentation common in static allocation.

This design allows vLLM to sustain much larger batch sizes, higher concurrency, and better GPU utilization than systems that rely on static, monolithic KV-cache buffers.

Continuous Batching: Maximizing GPU Utilization

Traditional static batching waits for all sequences in a batch to complete before processing new requests. vLLM’s continuous batching:

Dynamic Request Addition: New requests join running batches between decoding steps.
Early Completion Handling: Finished sequences free resources instantly.
Optimal GPU Usage: Maintains high utilization by mixing prefill and decode operations.
Preemption Support: Can pause low-priority requests for urgent ones.

Practical Implications for Deployment

These memory management innovations translate to concrete operational benefits:

Higher Concurrency: Serve more users with the same hardware.
Better Cost Efficiency: Reduce infrastructure requirements significantly.
Improved Latency: Faster time-to-first-token through efficient scheduling.
Flexibility: Handle varying sequence lengths without reconfiguration.

The KV-Cache optimizations become even more critical with the upcoming llm-d distributed architecture, where efficient memory usage enables new deployment patterns previously impossible with traditional approaches.

Scaling with llm-d: Kubernetes-Native Distributed Inference

Beyond Single-Server Deployment

While vLLM excels as a high-performance inference engine, production deployments at scale require sophisticated orchestration and intelligent request routing. The llm-d project, launched in May 2025 by Red Hat, Google Cloud, IBM Research, NVIDIA, and CoreWeave, addresses this by providing a Kubernetes-native distributed serving stack built on top of vLLM.

llm-d is not a feature of vLLM, it’s a complementary orchestration layer. Think of it like the relationship between Linux and Kubernetes: vLLM provides the inference engine, while llm-d provides distributed orchestration and intelligent scheduling across multiple vLLM instances. The stack integrates three open-source technologies: vLLM as the high-performance inference engine, Inference Gateway for AI-aware routing, and Kubernetes for industry-standard orchestration.

In addition, KServe has added llm-d integration via a new LLMInference CRD in KServe to have a single and coherent API that unifies the serving experience across use cases and maturity levels, supporting a smoother journey into generative AI for enterprise users.

llm-d integrates three foundational open-source technologies into a unified serving stack:

vLLM - The high-performance inference engine that executes model inference.
Inference Gateway (IGW) - An official Kubernetes project extending Gateway API with AI-aware routing.
Kubernetes - The industry-standard orchestration platform for deployment and scaling.

By combining these technologies, llm-d enables organizations to deploy LLM inference at scale across hybrid cloud environments with the fastest time-to-value and competitive performance per dollar.

Three Key Innovations

Intelligent Inference Scheduling

Traditional load balancing uses simple round-robin routing, treating all servers equally. llm-d’s vLLM-aware scheduler makes intelligent decisions by routing requests to instances with matching cached prefixes, distributing load based on whether instances are handling compute-intensive prefill or memory-bound decode operations, and using real-time telemetry from vLLM to avoid overloaded instances while prioritizing low-latency paths. This intelligent routing reduces infrastructure costs by 30-50% while maintaining latency SLOs.

Disaggregated Serving

llm-d orchestrates vLLM’s native disaggregated serving (via KVConnector API) at production scale, separating prefill and decode across specialized workers:

Prefill Workers: Handle compute-intensive prompt processing on high-performance GPUs (H100s, MI300X) and scale independently based on demand.
Decode Workers: Focus on memory-bound token generation using cost-effective GPUs (A100s, L40S) and scale based on concurrent sessions.
KV-Cache Transfer: Provides efficient cache movement using NVIDIA NIXL over UCX, support for offloading to storage backends (Future Delivery), and global cache awareness across the cluster.

This allows right-sizing infrastructure: expensive GPUs only for prefill, cost-optimized hardware for serving thousands of concurrent users.

Distributed Prefix Caching

llm-d extends vLLM’s prefix caching across multiple instances with two approaches:

Local Caching: Offloading to memory/disk on each instance with zero operational cost
Shared Caching (Planned): KV transfer between instances with global indexing for cluster-wide cache awareness

Deployment Patterns

llm-d enables several advanced enterprise patterns:

Heterogeneous Hardware: Mix GPU vendors and generations based on workload—high-end GPUs for prefill, cost-optimized GPUs for decode, or CPU clusters for low-frequency requests.
Dynamic Scaling: Independently adjust prefill capacity during peak hours while maintaining steady decode capacity for active sessions, with automatic resource allocation and failover.
Geographic Distribution (Roadmap): Deploy centralized prefill workers in primary data centers with edge decode workers near users for low-latency responses.

Integration with Red Hat OpenShift AI

OpenShift AI provides enterprise packaging for llm-d with unified deployment via KServe for all components, service mesh routing between workers, full observability with pre-built dashboards, and GitOps configuration management. Enterprise security features include consistent RBAC policies, encrypted communication between workers, audit logging for distributed flows, and network policy enforcement.

Operational Benefits

Cost Optimization: 2-3x better GPU utilization, 40-60% less over-provisioning
Scalability: Independent scaling of components, proven to 100+ node deployments
Resilience: Failure isolation between phases, automatic failover, graceful degradation

Multi-Accelerator and Hybrid Cloud Support

Breaking Free from Hardware Lock-in

The rapid evolution of AI accelerators has created a diverse hardware landscape. While specialized solutions like TensorRT-LLM deliver special optimizations for NVIDIA GPUs, they create vendor lock-in that limits deployment flexibility. vLLM’s hardware-agnostic design provides freedom to choose the optimal accelerator for each use case.

Comprehensive Hardware Support

vLLM supports NVIDIA GPUs (first-class optimizations for H100, with support for every NVIDIA GPU from V100 and newer), AMD GPUs (MI200, MI300, and Radeon RX 7900 series), Google TPUs (v4, v5p, v5e, and the latest v6e), AWS Inferentia and Trainium (trn1/inf2 instances), Intel Gaudi (HPU) and GPU (XPU), and CPUs featuring support for x86, ARM, and PowerPC vLLM 2024 Retrospective and 2025 Vision - vLLM Blog.

This broad support enables several strategic advantages:

Cost Optimization

Choose AMD MI300X for price/performance on certain workloads.
Use AWS Inferentia for cost-effective inference on AWS.
Deploy on existing CPU infrastructure for low-throughput use cases.

Supply Chain Resilience

Avoid dependency on single GPU vendor availability.
Negotiate better pricing with multiple options.
Adapt to regional hardware availability.

Workload Matching

NVIDIA H100s for maximum performance.
AMD GPUs for open-source aligned deployments.
TPUs for Google Cloud deployments.
Intel Gaudi for specific enterprise agreements.

Hybrid Cloud Deployment Patterns

vLLM on Red Hat OpenShift AI enables true hybrid cloud flexibility:

On-Premises Core

Sensitive data processing on local infrastructure.
Compliance-required workloads.
Predictable capacity for baseline load.

Cloud Burst Scaling

Handle peak loads with cloud resources.
Experiment with new hardware (H100s, TPU v6e).
Geographic expansion without infrastructure investment.

Edge Inference

Deploy on edge-appropriate hardware.
CPU or smaller GPU inference.
Integrated with central management.

Unified Operations Across Environments

Red Hat OpenShift AI provides consistent operations regardless of deployment location (public vs private cloud):

Single Control Plane: Manage all deployments from a unified interface.
Consistent APIs: Same application integration across environments.
Unified Monitoring: Aggregated metrics across hybrid deployments.
Policy Enforcement: Consistent security and compliance policies.

Real-World Flexibility Examples

Financial Services Scenario

On-premises NVIDIA GPUs for sensitive data processing.
AWS Inferentia for public-facing chatbots.
CPU inference for branch edge deployments.

Healthcare Provider

AMD MI300X in private cloud for cost optimization.
Google TPUs for research workloads.
Intel CPUs for clinical decision support.

Retail Organization

Centralized GPU clusters for training and complex inference.
Edge CPU deployment in stores.
Cloud scaling for seasonal peaks.

This hardware and deployment flexibility ensures that architectural decisions made today won’t constrain options tomorrow, a critical consideration as the AI hardware landscape continues to evolve rapidly.

Model Ecosystem and Compatibility

Supporting the Entire Open-Source Model Landscape

As AI gold-run increases it’s pace, vLLM had evolved to support performant inference for > 100 model architectures: spanning nearly every prominent open-source large language model (LLM), multimodal (image, audio, video), encoder-decoder, speculative decoding, classification, embedding, and reward models vLLM 2024 Retrospective and 2025 Vision - vLLM Blog. This comprehensive support represents a fundamental advantage over specialized solutions that focus on limited model families.

Beyond Traditional LLMs

vLLM’s architecture supports diverse model types:

Language Models

Llama family (including Llama 3.1 405B).
Mistral and Mixtral MoE models.
Google’s Gemma models.
IBM’s Granite series.
Alibaba’s Qwen models.

Multimodal Models

Vision-language models (LLaVA, Qwen-VL).
Document understanding models.
Audio-language models.
Video understanding capabilities.

Specialized Architectures

Production support for state-space language models, exploring the future of non-transformer language models vLLM 2024 Retrospective and 2025 Vision - vLLM Blog.
Mixture of Experts (MoE) models.
Encoder-decoder architectures.
Embedding and reranking models.

Ease of Model Integration

Adding new models to vLLM follows a standardized process:

Model Architecture Definition: Implement using familiar PyTorch patterns.
Attention Backend Integration: Leverage existing optimized kernels.
Tokenizer Support: Direct Hugging Face compatibility.
Configuration Mapping: Standard YAML-based configuration.

This standardization means new models can often be added timely, which is critical for organizations wanting to experiment with latest models. A great example for vLLM’s agility to adapt to the changing model landscape would be introducing a support for gpt-oss, which was released August 5, 2025, and shortly after, vLLM v0.11.0 release included support for serving it in production environments.

Hugging Face Ecosystem Integration

vLLM’s native Hugging Face compatibility provides:

Direct Model Loading

Load models directly from Hugging Face Hub and/or from S3 compatible object storage backends (ex; MinIO).
Support for private model repositories.
Automatic weight conversion handling.

Tokenizer Compatibility

Use existing tokenizer implementations.
Custom tokenizer support.
Fast tokenizer optimizations.

Configuration Preservation

Respect model-specific configurations.
Support for custom model parameters.
Compatibility with fine-tuned variants.

Production Model Management

Red Hat OpenShift AI adds enterprise features for model lifecycle:

Model Registry Integration

Version control for deployed models.
A/B testing capabilities.
Rollback mechanisms.

Model Monitoring

Performance tracking per model.
Usage analytics and cost attribution.
Drift detection capabilities.

Future-Proofing Model Support

The rapid pace of model innovation requires an inference platform that can adapt:

Community Contributions

Active community adding new models.
Vendor-neutral development process.
Rapid integration of breakthrough architectures.

Flexible Architecture

Modular design supports new model paradigms.
Not tied to specific model assumptions.
Ready for post-transformer architectures.

Enterprise Validation

Red Hat’s testing and certification process.
Security scanning for model artifacts.
Performance validation across hardware.

This comprehensive model support ensures organizations can adopt new models as they emerge, without platform migrations or architectural changes a critical capability as the AI landscape continues its rapid evolution.

Enterprise Deployment with Red Hat OpenShift AI

Production-Ready from Day One

Red Hat OpenShift AI is a flexible, scalable MLOps platform with tools to build, deploy, and manage AI-enabled applications. Built using open source technologies, it provides trusted, operationally consistent capabilities for teams to experiment, serve models, and deliver innovative apps. Read more about how RHOAI is solving this challenge in: Accelerating generative AI adoption: Red Hat OpenShift AI achieves impressive results in MLPerf inference benchmarks with vLLM runtime.

KServe Integration: Intelligent Model Serving

The integration between vLLM and KServe within OpenShift AI provides enterprise-grade serving capabilities:

GenAI Features

Multi-node/multi-GPU inference with vLLM serving runtime.
Key-Value cache offloading with vLLM + LMCache integrations.
Efficient model reuse via Model Cache.
KEDA integration to allow autoscaling based on external metrics.
Rate-limiting and request routing via integration with Envoy AI Gateway
Access to llm-d capabilities via the LLMInferenceService CRD

Advanced Autoscaling

Request-based scaling for optimal resource usage.
Scale-to-zero capabilities for cost optimization.
Predictive scaling based on traffic patterns.
Multi-metric scaling (GPU utilization, queue depth, latency).

Traffic Management

Canary deployments for safe model updates.
Blue-green deployments for instant rollback.
A/B testing for model comparison.
Shadow traffic for validation.

Service Mesh Integration

End-to-end encryption with Istio.
Advanced routing and load balancing.
Circuit breaking and retry logic.
Distributed tracing for debugging.

Automated Operations

Health checking and automatic recovery.
Resource optimization recommendations.
Automated certificate management.
Log aggregation and analysis.

Security and Compliance Features

Enterprise deployments require robust security:

Access Control

RBAC integration with enterprise identity providers.
Model-level access permissions.
API key management.
Audit logging for all operations.

Data Protection

Encryption at rest and in transit.
Private endpoint options.
Network policy enforcement.
Compliance reporting tools.

Supply Chain Security

Signed container images.
SBOM (Software Bill of Materials) generation.
Vulnerability scanning.
Policy-based deployment controls.

MLPerf-Validated Performance

Red Hat, in collaboration with Supermicro, has made significant strides in addressing this challenge through the publication of impressive MLPerf inference results using Red Hat OpenShift AI with NVIDIA GPUs and the vLLM inference runtime Accelerating generative AI adoption: Red Hat OpenShift AI achieves impressive results in MLPerf inference benchmarks with vLLM runtime. These results validate:

Production-grade performance at scale.
Efficient resource utilization.
Consistent latency under load.
Multi-instance coordination capabilities.

Integrated Observability

Comprehensive monitoring without additional tooling:

Metrics and Dashboards

Pre-built Grafana dashboards for vLLM metrics.
Pre-built Grafana dashboard for request scheduler metrics driving routing decisions.
Token generation rates and latencies.
GPU utilization and memory usage.
Queue depths and rejection rates.

Alerting and Response

Automated alerts for SLA violations.
Integration with enterprise monitoring systems.
Runbook automation capabilities.
Capacity planning insights.

Cost Management and Optimization (Future Delivery)

Features designed for enterprise cost control:

Chargeback/Showback: Track usage by team or project.
Resource Quotas: Prevent runaway costs.
Spot Instance Support: Reduce costs for batch workloads.
Idle Detection: Automatically scale down unused resources.

This enterprise-grade platform transforms vLLM from a high-performance inference engine into a complete production solution, ready for mission-critical deployments.

Feature Comparison: vLLM vs TGI vs TensorRT-LLM

✓ = Full Support ◐ = Partial/Limited Support ✗ = No Support

Key Differentiators for RHOAI with vLLM:

Hardware Flexibility: Broadest accelerator support including AMD, Intel, Google TPUs, and CPUs.
Model Ecosystem: Supports >100 model architectures vs 25-40 for alternatives.
Distributed Architecture: Upcoming llm-d enables disaggregated prefill/decode and system wide kv-cache routing for distributed optimal scaling.
Enterprise Integration: Native Red Hat OpenShift AI support with KServe autoscaling.
Memory Efficiency: Advanced PagedAttention and KV-Cache management.
Open Development: PyTorch Foundation project with rapid community innovation.

Making the Right Choice

Count your “Yes” responses across all categories:

0-2 “Yes” responses: Consider specialized solutions if they meet your specific needs.
3-5 “Yes” responses: vLLM provides significant advantages for your use case.
6+ “Yes” responses: vLLM on Red Hat OpenShift AI is the strategic choice.

Each decision point addresses fundamental architectural constraints:

Flexibility Requirements determine if you need a hardware-agnostic solution.
Operational Complexity evaluates if simplified operations justify open-source adoption.
Model & Innovation assesses if rapid evolution demands an adaptable platform.
Long-term Sustainability considers total cost of ownership and strategic risk.

Strategic Recommendation

For organizations answering “Yes” to multiple questions above, vLLM on Red Hat OpenShift AI delivers:

Maximum Flexibility: Deploy anywhere, on any supported hardware.
Operational Excellence: Enterprise-grade platform with minimal complexity.
Future-Readiness: Support for emerging models and architectures.
Cost Optimization: Choose optimal hardware for each workload.
Risk Mitigation: Open-source foundation with enterprise support.

The convergence of these factors makes vLLM the strategic choice for organizations building sustainable AI infrastructure.

Action Steps for Adoption

Immediate

Deploy vLLM on Red Hat OpenShift AI in the development environment.
Test with your specific models and workloads.
Validate hardware flexibility with available accelerators.

Short-term

Implement Autoscaling with KServe.
Establish monitoring and observability.
Train team on operational procedures.

Medium-term

Plan production deployment architecture.
Implement hybrid cloud patterns if needed.
Prepare for llm-d distributed architecture.

Long-term

Optimize costs through hardware selection.
Implement advanced deployment patterns.
Contribute improvements back to the community.

Conclusion

The choice of LLM inference platform represents a strategic commitment that will impact your organization’s AI capabilities for years to come.

Our analysis demonstrates that vLLM on Red Hat OpenShift AI uniquely addresses the three critical requirements for enterprise LLM deployment:

Flexibility: Deploy on any hardware (NVIDIA, AMD, Intel, TPUs) across hybrid clouds.
Scalability: Advanced memory management and upcoming llm-d architecture enable 10-100x better resource utilization.
Sustainability: Open-source foundation with enterprise support eliminates vendor lock-in.

While TensorRT-LLM offers NVIDIA-specific optimizations and TGI provides Hugging Face integration, only vLLM delivers the architectural flexibility required for a rapidly evolving AI landscape. With support for > 100 model architectures, hardware-agnostic design, and the backing of both PyTorch Foundation -> RHOAI with vLLM provides the most robust foundation for long-term success.

Eight Lessons on Open Source Leadership and Community

Sat, 20 Sep 2025 00:00:00 +0000

Originally posted on Substack.

Over the past months, I’ve been sharing thoughts on open source, leadership, and community building on social media. This post provides a quick roundup of the themes so far.

I recently started my journey here on Substack and you are welcome to follow along! You can also find me on LinkedIn, X, Bluesky, Mastodon, and GitHub.

🔑 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗹𝗲𝗮𝗱𝗲𝗿𝘀𝗵𝗶𝗽 𝗶𝗻 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗶𝘀𝗻’𝘁 𝗮𝗯𝗼𝘂𝘁 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 - 𝗶𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁 𝘀𝘁𝗲𝘄𝗮𝗿𝗱𝘀𝗵𝗶𝗽

In OSS, you don’t manage people. You manage trust.
You don’t assign work. You inspire contribution.
You don’t build teams. You cultivate communities.

As a technical leader in open source, your role shifts:
- From directing code → to curating vision
- From fixing bugs → to empowering contributors
- From owning decisions → to navigating consensus

Strong OSS leaders do more than merge PRs. They:
✅ Set clear technical direction
✅ Write accessible, empathetic documentation
✅ Welcome newcomers without gatekeeping
✅ Review with kindness, not ego
✅ Know when to say “no” — and when to say “not yet”
Most importantly, they recognize that leadership in OSS is earned, not appointed.

Whether you’re a maintainer, contributor, or just starting out — remember: open source runs on shared effort and mutual respect. Technical excellence matters. But how you lead matters more.

🌱 𝗪𝗮𝗻𝘁 𝘁𝗼 𝗴𝗲𝘁 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 𝗶𝗻 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲? 𝗦𝘁𝗮𝗿𝘁 𝘀𝗺𝗮𝗹𝗹 - 𝗯𝘂𝘁 𝘀𝘁𝗮𝗿𝘁

When people ask how I got involved in open source, they’re often surprised by the answer:

📌 It started with 𝗳𝗶𝗹𝗶𝗻𝗴 𝗮 𝗯𝘂𝗴 𝗼𝗻 𝗚𝗶𝘁𝗛𝘂𝗯.

Early in my career, I was building ML models and hit a blocker in a popular Python library. I filed a bug and waited… After two days, I realized the fix wouldn’t come sooner unless I made it — so I dug into the unfamiliar codebase and fixed it.

That small act turned into a habit:
➡️ Fixing small bugs
➡️ Improving documentation
➡️ Asking questions
➡️ Following community discussions
➡️ Submitting thoughtful PRs

One contribution led to another. Over time, I became a 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻𝗲𝗿 of many open source projects like Kubeflow, Argo Project, and KServe, XGBoost, and TensorFlow.

The pattern repeated itself:
🛠️ I saw a need → made contributions → joined community calls → helped others.

Today, I contribute by:
✅ Reviewing code & triaging issues
✅ Organizing contributor meetings
✅ Giving conference talks
✅ Writing books & blog posts
✅ Mentoring contributors

💡 𝗠𝘆 𝗮𝗱𝘃𝗶𝗰𝗲?
You don’t need permission to contribute.
Start with curiosity, stay consistent, and participate with empathy.

💙 𝗦𝘂𝘀𝘁𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝗻 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗶𝘀𝗻’𝘁 𝗮𝗯𝗼𝘂𝘁 𝗰𝗼𝗱𝗲. 𝗜𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁 𝗰𝗮𝗿𝗲

Open source projects don’t fail because of bad technology.
They fail because contributors burn out, communities fracture, or energy fades.

Technical debt matters. But so does 𝗲𝗺𝗼𝘁𝗶𝗼𝗻𝗮𝗹 𝗱𝗲𝗯𝘁:

PRs without response → contributors feel invisible.
Harsh reviews → contributors stop showing up.
No clear roadmap → contributors drift away.

As a maintainer or leader, your job isn’t just writing great software. It’s creating conditions where people 𝘸𝘢𝘯𝘵 to keep showing up.

That means:
✅ Balancing vision with flexibility
✅ Saying thank you as often as saying “LGTM”
✅ Mentoring the next wave of maintainers
✅ Building processes that survive you
✅ Remembering that community health is a feature, not an afterthought

The hardest part of OSS leadership isn’t scaling code. It’s scaling trust, empathy, and continuity.

If we care for the people behind the code, the code will take care of itself.

👥 𝗢𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗶𝘀 𝗽𝗼𝘄𝗲𝗿𝗲𝗱 𝗯𝘆 𝗽𝗲𝗼𝗽𝗹𝗲 - 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝗰𝗼𝗱𝗲

Behind every release, every merged PR, every answered issue…
There’s someone who took the time to care.

🧑‍💻 A maintainer reviewing PRs late at night
📚 A contributor improving documentation so others can follow
🛠️ A developer fixing a tricky bug that wasn’t even theirs
🙋‍♀️ A community member helping someone in a Slack thread

These quiet acts of contribution keep open source alive.

Today, let’s do something simple but meaningful:

💬 𝗧𝗮𝗴 𝗼𝗻𝗲 𝗽𝗲𝗿𝘀𝗼𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗰𝗼𝗺𝗺𝗲𝗻𝘁𝘀 𝘄𝗵𝗼’𝘀 𝗺𝗮𝗱𝗲 𝘆𝗼𝘂𝗿 𝗢𝗦𝗦 𝗷𝗼𝘂𝗿𝗻𝗲𝘆 𝗯𝗲𝘁𝘁𝗲𝗿.
Maybe they reviewed your first PR, mentored you, or just modeled what good community looks like.

A little gratitude goes a long way.

🏆 𝗜𝗻 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲, 𝘀𝗺𝗮𝗹𝗹 𝘄𝗶𝗻𝘀 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗯𝗶𝗴 𝘄𝗶𝗻𝘀

It’s easy to focus only on major milestones — big releases, new features, or keynote talks.

But the real strength of an OSS project lies in the 𝘭𝘪𝘵𝘵𝘭𝘦 𝘮𝘰𝘮𝘦𝘯𝘵𝘴 that often go unnoticed:

✅ A first-time contributor submitting their first PR
✅ A bug fix that unblocks someone
✅ A community member getting promoted to reviewer or maintainer
✅ A shout-out in a conference session
✅ Someone asking (or answering) a good question in Slack or GitHub

These are the moments that build trust.
That turn users into 𝘤𝘰𝘯𝘵𝘳𝘪𝘣𝘶𝘵𝘰𝘳𝘴.
Contributors into 𝘭𝘦𝘢𝘥𝘦𝘳𝘴.
And projects into 𝘤𝘰𝘮𝘮𝘶𝘯𝘪𝘵𝘪𝘦𝘴.

If we want OSS to thrive, we have to make celebration part of the culture — not just for the code, but for the people.

💡 A kind comment, a public “thank you,” or a quick emoji reaction might be what keeps someone coming back.

Let’s celebrate the small wins — because in open source, they’re never really small.

🌟 𝗢𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗺𝗲𝗻𝘁𝗼𝗿𝘀𝗵𝗶𝗽 𝗶𝘀𝗻’𝘁 𝗮𝗯𝗼𝘂𝘁 𝗵𝗮𝘃𝗶𝗻𝗴 𝗮𝗹𝗹 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 - 𝗶𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁 𝗰𝗿𝗲𝗮𝘁𝗶𝗻𝗴 𝘀𝗽𝗮𝗰𝗲 𝗳𝗼𝗿 𝗴𝗿𝗼𝘄𝘁𝗵

I learned about writing good unit tests from a PR review in the pandas Python library. Thank you Jeff Reback for your guidance many years ago.

Some of the best developers I know didn’t learn from a course.
They learned from a kind comment on an issue.
From someone taking the time to say:
“𝘏𝘦𝘺, 𝘸𝘢𝘯𝘵 𝘵𝘰 𝘱𝘢𝘪𝘳 𝘰𝘯 𝘵𝘩𝘪𝘴?”

That’s mentorship. And open source needs more of it.

But here’s the thing — mentorship in OSS isn’t structured.
It’s not assigned. It’s organic and takes 𝘪𝘯𝘵𝘦𝘯𝘵𝘪𝘰𝘯𝘢𝘭𝘪𝘵𝘺.

If you’re a maintainer or experienced contributor, consider this:
✅ Label a few good first issues
✅ Offer context, not just code comments
✅ Invite questions without judgment
✅ Share why something works — not just that it does
✅ Remember what it felt like to be new

You won’t just grow contributors — you’ll grow leaders.

And if you’re earlier in your journey:
💬 Ask. Offer. Show up. OSS has a place for you.

Mentorship in open source is one of the most impactful — and most overlooked — forms of technical leadership.

🔥 𝗜𝗻 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲, 𝘆𝗼𝘂 𝗱𝗼𝗻’𝘁 𝗮𝘀𝘀𝗶𝗴𝗻 𝘁𝗮𝘀𝗸𝘀 - 𝘆𝗼𝘂 𝗶𝗻𝘀𝗽𝗶𝗿𝗲 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻

𝗜 𝗺𝗲𝗻𝘁𝗶𝗼𝗻𝗲𝗱 𝘁𝗵𝗶𝘀 𝗶𝗻 𝗺𝘆 𝗽𝗿𝗲𝘃𝗶𝗼𝘂𝘀 𝗽𝗼𝘀𝘁, 𝗯𝘂𝘁 𝗵𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝗶𝗻𝘀𝗽𝗶𝗿𝗲 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻?

✅ Share vision, not just code
Make the vision publicly visible. Projects like vLLM, KServe, Kubeflow, Llama Stack all have published their roadmaps.

✅ Be open to ideas you didn’t come up with
OSS thrives when people feel ownership, not just direction. llm-d is inviting the community to contribute ideas through a survey: https://lnkd.in/gh3ceWYD

✅ Lower the barrier to entry
Clear docs, labeled issues, and kind reviews are magnets for first-time contributors.

✅ Celebrate contributions publicly
Recognition drives retention. Gratitude is free and incredibly effective. Andrey showed this perfectly through promoting Shao and Antonin to Kubeflow maintainers: https://lnkd.in/gNu4YXiv

This is how open source projects grow — and how technical leaders earn trust in public.

🎓 𝗔𝗱𝘃𝗶𝗰𝗲 𝗳𝗼𝗿 𝗻𝗲𝘄 𝗖𝗦 𝗴𝗿𝗮𝗱𝘂𝗮𝘁𝗲𝘀 𝗲𝗻𝘁𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝗷𝗼𝗯 𝗺𝗮𝗿𝗸𝗲𝘁

It’s no secret that this year has been especially tough for new grads seeking software engineering roles. Many fall into the trap of endlessly polishing resumes or grinding LeetCode. But here’s the truth:

🚫 If you’re still only “learning tricks” to improve your resume - you’re doing it wrong.
✅ What will actually make you stand out? 𝗖𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗻𝗴 𝘁𝗼 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲.

Open source contribution gives you:
- Real-world experience collaborating with global teams
- Hands-on exposure to tools, frameworks, and workflows used in industry
- Code reviews and mentorship from experienced maintainers
- A public portfolio that recruiters and hiring managers can see

Hours spent on LeetCode help you with interviews.
Hours spent on OSS help you with your career.

That’s why so many of us in the industry actively look for OSS contributions on resumes. It shows initiative, collaboration, and practical impact.

🌍 And here’s the best part: 𝘄𝗲’𝗿𝗲 𝗵𝗶𝗿𝗶𝗻𝗴!

If you’re passionate about cloud, AI/ML, or open source , we’d love to connect. Contributing to OSS isn’t just great prep, it’s also one of the best ways to get noticed by teams like ours.

I recently started my journey here on Substack and you are welcome to follow along! You can also find me on LinkedIn, X, Bluesky, Mastodon, and GitHub.

PyTorch on Kubernetes: Kubeflow Trainer Joins PyTorch Ecosystem

Mon, 28 Jul 2025 00:00:00 +0000

Authors: Andrey Velichkevich, Apple; Yuki Iwai, CyberAgent, Inc.; Yuan Tang, Red Hat; Antonin Stefanutti, Red Hat; Johnu George, Nutanix

Originally posted on PyTorch blog.

We’re thrilled to announce that the Kubeflow Trainer project has been integrated into the PyTorch ecosystem! This integration ensures that Kubeflow Trainer aligns with PyTorch’s standards and practices, giving developers a reliable, scalable, and community-backed solution to run PyTorch on Kubernetes.

To view the PyTorch Ecosystem, see the PyTorch Landscape. Learn more about how projects can join the PyTorch Ecosystem.

About Kubeflow Trainer

Kubeflow Trainer is a Kubernetes-native project enabling scalable, distributed training of AI models and purpose-built for fine-tuning large language models (LLMs). It simplifies the scale-out of training workloads on multiple nodes, managing large datasets efficiently and ensuring fault-tolerance.

The core features include:

Simplify Kubernetes complexity: Kubeflow Trainer APIs are designed for two primary user personas - AI practitioners - ML engineers and data scientists who develop AI models using the Kubeflow Python SDK and TrainJob APIs, platform admins - administrators and DevOps engineers responsible for managing Kubernetes clusters and Kubeflow Trainer runtimes APIs. AI practitioners can focus on the application code in PyTorch without worrying about infrastructure details. Meanwhile, platform admins can flexibly schedule workload resources for maximum cluster utilization and cost efficiency. To support these roles, Kubeflow Trainer specifies purpose-built Kubernetes Custom Resource Definitions (CRDs) that streamline model training and infrastructure management.

Python SDK: A Pythonic interface designed for AI practitioners, abstract the details of interacting directly with Kubernetes APIs. It enables users to focus on developing PyTorch models without worrying about Kubernetes YAML configurations.
Blueprints for LLMs fine-tuning on Kubernetes: With built-in trainers, Kubeflow Trainer enables AI practitioners to seamlessly fine-tune their favorite LLMs using the desired configuration for datasets, LoRA parameters, learning rate, etc. In the first release, it implements recipes to support various fine-tuning strategies, including Supervised Fine-Tuning (SFT), Knowledge Distillation, DPO, PPO, GRPO, and Quantization-Aware Training. Community is working towards adding more builtin trainers powered by LLaMA-Factory, Unsloth, HuggingFace TRL to enable efficient LLMs fine-tuning.
Optimized GPU utilization: Kubeflow Trainer maximizes GPU efficiency by streaming large-scale data directly to distributed GPUs using an in-memory distributed data cache powered by Apache Arrow and Apache DataFusion
Advanced scheduling capabilities: Kubeflow Trainer supports gang scheduling through the PodGroupPolicy API, enabling coordinated scheduling of pods across nodes. It also integrates with Kubernetes schedulers such as Kueue, Coscheduling, Volcano, and KAI Scheduler to ensure all required resources are allocated before training jobs start.
Accelerate MPI workloads on Kubernetes: Kubeflow Trainer supports MPI-based runtimes such as DeepSpeed and MLX. It handles all necessary orchestration of MPI workloads with SSH-based optimization to boost MPI performance.
Improved resilience and fault-tolerance: By leveraging Kubernetes-native APIs like Jobs and JobSets, Kubeflow Trainer improves reliability and efficiency of AI workloads. With support for the PodFailurePolicy API, users can reduce cost by avoiding unnecessary restarts. Additionally, the SuccessPolicy API allows training jobs to complete early once the target objective is achieved.

Background and Evolution

This project was originally started as a distributed training operator for TensorFlow (e.g. TFJob), and later we merged efforts from other Kubeflow Training Operators (e.g. PyTorchJob, MPIJob) to provide a unified and simplified experience for both users and developers. We are very grateful to all who filed issues or helped resolve them, asked and answered questions, and were part of inspiring discussions. We’d also like to thank everyone who’s contributed to and maintained the original operators.

By joining the PyTorch Ecosystem, we strive to apply best practices of deploying distributed PyTorch applications on Kubernetes and bring first-class PyTorch support in Kubeflow Trainer.

Integrations with PyTorch Ecosystem

Kubeflow Trainer is deeply integrated with the PyTorch ecosystem, supporting a broad range of tools and libraries—including torch, DeepSpeed, HuggingFace, Horovod, and more.

It empowers PyTorch users to implement advanced distributed training strategies such as Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP & FSDP2), and Tensor Parallelism, enabling efficient large-scale model training on Kubernetes.

Additionally, Kubeflow Trainer supports data parallelism using PyTorch IterableDatasets, streaming data directly from distributed in-memory data cache nodes. This allows scalable training even with massive datasets that exceed local memory capacity.

Quick Start

Follow the steps below to quickly deploy Kubeflow Trainer and run your first training job.

Prerequisites

Installed kubectl
Installed kind

Install Kubeflow Trainer

Deploy Kubeflow Trainer control plane on your local kind cluster:

$ kind create cluster

$ kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/manager?ref=v2.0.0"


# Ensure that JobSet and Trainer controller manager are running.
$ kubectl get pods -n kubeflow-system

NAME                                                  READY   STATUS    RESTARTS   AGE
jobset-controller-manager-54968bd57b-88dk4            2/2     Running   0          65s
kubeflow-trainer-controller-manager-cc6468559-dblnw   1/1     Running   0          65s


# Deploy the Kubeflow Trainer runtimes.
$ kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=v2.0.0"

# Install Kubeflow SDK
$ pip install git+https://github.com/kubeflow/sdk.git@64d74db2b6c9a0854e39450d8d1c0201e1e9b3f7#subdirectory=python

Define PyTorch Training Function

After installing the Kubeflow Trainer, define your PyTorch training function that contains end-to-end training script:

def train_pytorch():
    import os
    import torch
    import torch.distributed as dist
    from torch.utils.data import DataLoader, DistributedSampler
    from torchvision import datasets, transforms, models

    # [1] Configure CPU/GPU device and distributed backend.
    device, backend = ("cuda", "nccl") if torch.cuda.is_available() else ("cpu", "gloo")
    dist.init_process_group(backend=backend)
    local_rank = int(os.getenv("LOCAL_RANK", 0))
    device = torch.device(f"{device}:{local_rank}")
    
    # [2] Get the pre-defined model.
    model = models.shufflenet_v2_x0_5(num_classes=10)
    model.conv1 = torch.nn.Conv2d(1, 24, kernel_size=3, stride=2, padding=1, bias=False)
    model = torch.nn.parallel.DistributedDataParallel(model.to(device))
    optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
   
    # [3] Get the FashionMNIST dataset and distribute it across all available devices.
    if local_rank == 0: # Download dataset only on local_rank=0 process.
        dataset = datasets.FashionMNIST("./data", train=True, download=True, transform=transforms.Compose([transforms.ToTensor()]))
    dist.barrier()
    dataset = datasets.FashionMNIST("./data", train=True, download=False, transform=transforms.Compose([transforms.ToTensor()]))

    train_loader = DataLoader(dataset, batch_size=100, sampler=DistributedSampler(dataset))

    # [4] Define the PyTorch training loop.
    for epoch in range(3):
        for batch_idx, (inputs, labels) in enumerate(train_loader):
            inputs, labels = inputs.to(device), labels.to(device)
            # Forward and Backward pass
            outputs = model(inputs)
            loss = torch.nn.functional.cross_entropy(outputs, labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            if batch_idx % 10 == 0 and dist.get_rank() == 0:
                print(f"Epoch {epoch} [{batch_idx * len(inputs)}/{len(train_loader.dataset)}] "
                    f"Loss: {loss.item():.4f}"
                )

Run PyTorch on Kubernetes with TrainJob

After defining the training function, use the Kubeflow SDK to create TrainJob:

from kubeflow.trainer import TrainerClient, CustomTrainer

job_id = TrainerClient().train(
    trainer=CustomTrainer(
        func=train_pytorch,
        num_nodes=2,
        resources_per_node={
            "cpu": 3,
            "memory": "3Gi",
            # "gpu": 2, # Uncomment this line if you have GPUs.
        },
    ),
    runtime=TrainerClient().get_runtime("torch-distributed"),
)

Get the TrainJob Results

After creating the TrainJob, you should be able to list it:

for job in TrainerClient().list_jobs():
    print(f"TrainJob: {job.name}, Status: {job.status}")

TrainJob: q33a18f65635, Status: Created

It may take a few minutes for the TrainJob to pull the PyTorch image the first time. Once the image is pulled, the TrainJob’s steps should transition to Running status. Each step represents a training node, and the number of devices per step corresponds to the number of devices on that node:

for s in TrainerClient().get_job(name=job_id).steps:
    print(f"Step: {s.name}, Status: {s.status}, Devices: {s.device} x {s.device_count}")

Step: node-0, Status: Running, Devices: cpu x 3
Step: node-1, Status: Running, Devices: cpu x 3

After steps are running, you can check the TrainJob logs. The dataset of 60,000 samples has been evenly distributed across 6 CPUs, with each device processing 10,000 samples: 60,000 / 6 = 10,000

print(TrainerClient().get_job_logs(name=job_id)["node-0"])

...
Epoch 0 [8000/60000] Loss: 0.4476
Epoch 0 [9000/60000] Loss: 0.4784
Epoch 1 [0/60000] Loss: 0.3909
Epoch 1 [1000/60000] Loss: 0.4888
Epoch 1 [2000/60000] Loss: 0.4100
...

Congratulations, you created your first distributed training job with PyTorch and Kubeflow Trainer!

What’s next

Kubeflow Trainer has exciting roadmap including the following items:

Local TrainJob Execution - run Kubeflow Trainer jobs locally without Kubernetes.
Distributed Data Cache - stream in-memory distributed data powered by Apache Arrow and Apache DataFusion.
Advanced scheduling capabilities - improve resources management and gang-scheduling capabilities by integrating with Kueue, KAI Scheduler, Volcano.
Support for JAX runtime.
Automate Checkpointing for GPU-accelerated workloads.

Call to Action

We are excited to welcome Kubeflow Trainer to the PyTorch ecosystem! Kubeflow Trainer democratizes AI model training on Kubernetes and significantly improves the development experience for AI practitioners. We invite you to explore the following resources to learn more about the project:

Read the Kubeflow Trainer v2 announcement blog post and release notes.
Explore the official Kubeflow Trainer documentation.
Join the conversations in the #kubeflow-trainer Slack channel.
Attend our bi-weekly Kubeflow Trainer community calls every Wednesday.
Share your use cases or feature proposals by opening an issue on the GitHub repository.
Tell your store by writing a Kubeflow blog post or speaking at upcoming Kubeflow Events.
Explore the Kubeflow Python SDK for AI practitioners.

We can’t wait to see what you’ll build with Kubeflow Trainer!

Announcing KServe v0.15: Advancing Generative AI Model Serving

Tue, 27 May 2025 00:00:00 +0000

Authors: Alexa Griffith, Dan Sun, Yuan Tang, Johnu George, Lize Cai

Originally posted on KServe blog.

We are thrilled to announce the release of KServe v0.15, marking a significant leap forward in serving both predictive and generative AI models. This release introduces enhanced support for generative AI workloads, including advanced features for serving large language models (LLMs), improved model and KV caching mechanisms, and integration with Envoy AI Gateway.

Embracing Generative AI Workloads

KServe v0.15 brings first-class support for generative AI workloads, marking a key evolution beyond traditional predictive AI. Unlike predictive models that infer outcomes from existing data, generative models like large language models (LLMs) create new content from prompts. This fundamental difference introduces new serving challenges. KServe now provides the infrastructure and optimizations needed to serve these models efficiently at scale.

To support these workloads, we’ve introduced a dedicated Generative AI section in our documentation, detailing the new capabilities and configurations tailored for generative models.

KServe now offers a lightweight installation for hosting LLMs on Kubernetes, please follow generative inference installation guide to get started. KEDA is an optional component for scaling based on LLM specific metrics and Envoy AI gateway is integrated for advanced traffic management capabilities with token rate limiting, unified API and intelligent routing.

Key Generative AI Features in v0.15

Envoy AI Gateway Integration
Multi Node Inference
LLM Autoscaler with KEDA
Distributed KV Cache with LMCache

Envoy AI Gateway Support

KServe v0.15 adds initial support for Envoy AI Gateway, a CNCF open source project built on top of Envoy Gateway and designed specifically for managing generative AI traffic at scale.

Envoy Gateway is also now supported in KServe along with Kubernetes Gateway API. Unlike traditional gateway solutions, Envoy AI Gateway provides advanced capabilities tailored to AI serving, including:

Dynamic model routing based on request content, model metadata, or user context.
Built-in support for multi-tenant inference, with fine-grained access controls and authentication.
Unified API for routing and managing LLM/AI traffic easily.
Integrated observability for model-level performance insights.
Extensibility for inference-specific policies like rate-limiting by token, and model lifecycle management.
Automatic failover mechanisms to ensure service reliability.

This integration enables a unified, intelligent entrypoint for both predictive and generative workloads—scaling from traditional models to complex LLMs—all while abstracting infrastructure complexity from the user. Please refer to Envoy AI Gateway integration doc for more details.

Multi-Node Inference

To support LLMs too large for a single node (e.g., Llama 3.1 405B), KServe v0.15 introduces multi-node inference across distributed GPUs, unlocking large model serving at scale. As models continue to increase in size, multi-node inference capabilities are increasingly important for production deployments that require real-time user experience. Please refer to the Multi Node inference doc for more details.

The community is also working on a new distributed inference API to allow scaling Multi Node Inference and support Disaggregatd Prefilling which is targeted for large LLM deployments.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-llama3
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: pvc://llama-3-8b-pvc/hf/8b_instruction_tuned
    workerSpec:
      pipelineParallelSize: 2
      tensorParallelSize: 1

LLM Autoscaler with KEDA (Kubernetes Event-driven Autoscaling)

Autoscaling LLMs is challenging due to their high resource demands and variable inference traffic patterns. The dynamic nature of LLM inference, with varying input lengths and token generation speeds, further complicates the prediction of resource needs, demanding sophisticated and adaptive autoscaling solutions. KServe now integrates with KEDA (Kubernetes Event-Driven Autoscaling) offers a powerful solution to many of the challenges associated with LLM autoscaling by extending Kubernetes’ native Horizontal Pod Autoscaler (HPA) capabilities. KEDA can monitor custom metrics which means you can expose LLM metrics from your LLM inference servers and use KEDA to scale based on these precise indicators.

This empowers users to efficiently manage LLM workloads with more intelligent scaling decisions based on workload characteristics for improved performance and cost optimization. Please follow the tutorial doc for how to autoscale based on vLLM metrics.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-llama3-keda
  annotations:
    serving.kserve.io/autoscalerClass: "keda"
    sidecar.opentelemetry.io/inject: "huggingface-llama3-keda"
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=llama3
        - --model_id=meta-llama/meta-llama-3-70b
    minReplicas: 1
    maxReplicas: 5
    autoScaling:
      metrics:
        - type: PodMetric
          podmetric:
            metric:
              backend: "opentelemetry"
              metricNames:
                - vllm:num_requests_running
              query: "vllm:num_requests_running"
            target:
              type: Value
              value: "4"

Distributed KV Cache with LMCache

Key-Value (KV) cache offloading is a technique used in large language model (LLM) serving to store and reuse the intermediate key and value tensors generated during model inference. In transformer-based models, these KV caches represent the context for each token processed, and reusing them allows the model to avoid redundant computations for repeated or similar prompts.

Enabling KV cache offloading across multiple requests and serving instances can achieve reduced Time To First Token(TTFT), improve scalability for shared cache across replicas, and improve user experience for multi-turn QA or RAG.

KServe integrates LMCache, the-state-of-art KV cache layer library developed by LMCache Lab to reduce inference costs and ensure SLOs for both latency and throughput at scale. Please follow the LMCache integration doc to optimize your GenAI inference workload.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-llama3-lmcache
spec:
  predictor:
    minReplicas: 2
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=llama3
        - --model_id=meta-llama/meta-llama-3-70b
        - --kv-transfer-config
        - '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
        - --enable-chunked-prefill

Advanced Model Caching Mechanisms

To reduce model loading times and improve overall efficiency of serving large models, KServe v0.15 introduces advanced model caching features:

LocalModelCache Enhancements: Improved the LocalModelCache custom resource to support multiple node groups, providing greater flexibility in model placement and caching strategies.
Node Agent Improvements: Enhanced the local model node agent for better performance and reliability.

Enhanced vLLM Backend Support

The vLLM backend has been significantly upgraded to better serve generative AI models:

Version Upgrade: Updated to vLLM 0.8.5, bringing performance improvements with v1 backend and new features.
Qwen3 & Llama4: Added support for Qwen3 and Llama4 models.
Reranking Support: Added support for reranking models.
Embedding Support: Added support for OpenAI-compatible embeddings API, enabling a broader range of applications.

Additional Improvements

This release also includes several other enhancements:

Support Deep Health Checks #3348
Collocated Transformer & Predictor Feature #4255
Kubernetes Gateway API support #3952
Security Updates

For a complete list of changes and updates, please refer to the official release notes.

Join the Community

We invite you to explore the new features in KServe v0.15 and contribute to the ongoing development of the project:

Visit KServe Website
Explore KServe GitHub Repository
Join the conversation on CNCF Slack #kserve
Attend our community meetings by subscribing to the KServe calendar

We extend our gratitude to all the contributors who made this release possible. Your efforts continue to drive the advancement of KServe as a leading platform for serving machine learning models.

Happy serving!