BentoML Documentation¶

BentoML is a Unified Model Serving Framework for building model inference APIs and multi-model serving systems with any open-source or custom AI models.

Featured examples¶

Deploy an open-source LLM endpoint

Serve large language models with OpenAI-compatible APIs and vLLM inference backend.

LLM inference: vLLM

Serve a LangGraph agent

Serve a LangGraph agent that searches the web and calls an external or open-source LLM.

Agent: LangGraph

Serve MLflow models

Import an MLflow model into BentoML and serve it as an inference API.

MLflow

Serve XGBoost models

Serve a gradient boosting model for tabular predictions, such as breast cancer classification.

XGBoost

What is BentoML¶

BentoML is an open-source Python library for building online serving systems optimized for AI apps and model inference. With BentoML, you can:

Build APIs for any AI/ML model. Turn any model inference script into a REST API server with a few lines of code and standard Python type hints.
Package and containerize with ease. Manage environments, dependencies and model versions in one place. BentoML generates reproducible Docker images that you can deploy to any environment.
Maximize CPU/GPU utilization. Use built-in serving optimizations like adaptive batching, model parallelism, multi-stage pipelines and multi-model composition.
Customize everything. Implement your own APIs or task queues with custom business logic, model inference and multi-model composition. BentoML supports any ML framework, modality and inference runtime.
Run in production on your own infrastructure. Develop, run and debug locally, then deploy with Docker containers or Kubernetes.

To get started, use pip to install BentoML, which is distributed as a Python package on PyPI.

# Recommend Python 3.9+
pip install bentoml

How-tos¶

Create online API Services

Build your custom AI APIs with BentoML.

Create online API Services

Packaging for deployment

Package your AI application as a Bento and containerize it for deployment.

Packaging for deployment

Agent skills

Let a coding agent deploy your BentoML Service to Kubernetes or EC2.

Agent skills

Work with GPUs

Run model inference on GPUs with BentoML.

Work with GPUs

Adaptive batching

Batch requests dynamically for higher throughput and better resource utilization.

Adaptive batching

Load and manage models

Load and serve your custom models with BentoML.

Load and manage models

Stay informed¶

The BentoML team uses the following channels to announce important updates like major product releases and share tutorials, case studies, as well as community news.

To receive release notifications, star and watch the BentoML project on GitHub. For release notes and detailed changelogs, see the Releases page.