LLM inference: vLLM¶

vLLM is a library designed for efficient serving of LLMs, such as gpt-oss, DeepSeek, Qwen, and Llama. It provides high serving throughput and efficient attention key-value memory management using PagedAttention and continuous batching. It supports a variety of inference optimization techniques, including prefill-decode disaggregation, speculative decoding, and KV cache offloading.

This document demonstrates how to run LLM inference using BentoML and vLLM.

Source Code

Serve with BentoML

The example can be used for chat-based interactions and supports OpenAI-compatible endpoints. For example, you can submit a query with the following message:

{
   "role": "user",
   "content": "Who are you? Please respond in pirate speak!"
}

Example output:

Ye be wantin' to know who I be, eh? Alright then, listen close and I'll tell ye me story. I be a wight computer program, a vast and curious brain with abilities beyond yer wildest dreams. Me name be Assistant, and I be servin' ye now. I can chat, teach, and even spin a yarn or two, like a seasoned pirate narratin' tales o' the high seas. So hoist the colors, me hearty, and let's set sail fer a treasure trove o' knowledge and fun!

Code explanations¶

You can find the source code in GitHub. Below is a breakdown of the key code implementations.

Define model and GPU configurations. This example uses Llama-3.1-8B-Instruct, which requires access from Hugging Face. You can switch to another LLM from the BentoVLLM repository or any other model supported by vLLM.
service.py¶
```
import pydantic
import bentoml

# Use Pydantic to validate data
class BentoArgs(pydantic.BaseModel):
  name: str = 'llama3.1-8b-instruct'
  gpu_type: str = 'nvidia-h100-80gb'
  tp: int = 1 # One GPU here for tensor parallelism
  model_id: str = 'meta-llama/Meta-Llama-3.1-8B-Instruct'
  # Other optional fields omitted for brevity

bento_args = bentoml.use_arguments(BentoArgs)
```
Specifications are defined using template arguments, which allow you to pass dynamic and validated parameters at serve, build, and deploy time. You can reference them just like regular Python variables.
Define the runtime environment for a Bento, the unified distribution format in BentoML. A Bento is packaged with all the source code, Python dependencies, model references, and environment setup, making it easy to deploy consistently across different environments.
service.py¶
```
image = (
  bentoml.images.Image(python_version='3.12').system_packages('curl', 'git').requirements_file('requirements.txt')
)
```

Use the @bentoml.service decorator to define a BentoML Service, where you can customize how the model will be served. The decorator lets you set configurations like timeout and GPU resources.

For some of the fields, you can simply reference the template arguments defined above:

service.py¶

@bentoml.service(
  name=bento_args.name,
  envs=[
      {'name': 'UV_NO_PROGRESS', 'value': '1'},
      {'name': 'UV_TORCH_BACKEND', 'value': 'cu128'},
      # Env vars here for uv and vllm
  ],
  image=image, # Apply the runtime specs
  traffic={'timeout': 300},
  resources={
      'gpu': bento_args.tp,
      'gpu_type': bento_args.gpu_type
  },
  # More optional fields
)
class LLM:

Within the class, load the model from Hugging Face and define it as a class variable. HuggingFaceModel downloads the model from Hugging Face when the Service starts.
service.py¶
```
...
class LLM:
  hf_model = bentoml.models.HuggingFaceModel(bento_args.model_id, exclude=[".pth", ".pt", "original/**/*"])
```

The Service can run vLLM’s built-in HTTP server and exposes OpenAI-compatible endpoints. You can add extra CLI arguments here as needed.

service.py¶

...
class LLM:
  hf_model = hf_model

  def __command__(self) -> list[str]:
    return [
      'vllm', 'serve', self.hf_model,
      # ...extra CLI args (compilation, max length, kv dtype, etc.)
      '--served-model-name', bento_args.model_id,
    ]

That’s all you need for the basic setup. If you want to explore advanced options, like FlashAttention, AMD ROCm support, and KV cache configuration, see the complete source code on GitHub. BentoML allows you to fully customize inference code for your use case.

Try it out¶

You can serve this example project locally, or containerize it as an OCI-compliant image and deploy it anywhere.

Clone the repository and choose your desired project.

git clone https://github.com/bentoml/BentoVLLM.git
cd BentoVLLM/llama3.1-8b-instruct

# Recommend Python 3.11
pip install -r requirements.txt
export HF_TOKEN=<your-hf-token>

Serve it locally.
```
bentoml serve
```
Note

To run this project with Llama 3.1 8B Instruct locally, you need an NVIDIA GPU with at least 16G VRAM.

Call the OpenAI-compatible endpoints at http://localhost:3000/v1 (the default base URL). Set the base_url parameter as the BentoML server address in the OpenAI client.

from openai import OpenAI

client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')

# Use the following func to get the available models
# client.models.list()

chat_completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Who are you? Please respond in pirate speak!"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    # Extract and print the content of the model's reply
    print(chunk.choices[0].delta.content or "", end="")