Serve a language model with vLLM

Before you start

Select a vLLM template certified for the GPU/provider combination. Model architecture, precision, and memory requirements must be supported by that runtime.

Steps

1. Review the template’s model, startup command, and memory defaults before deployment.
2. Provide enough disk for weights and caches and use the persistent mount configured by the template.
3. Watch logs until model loading completes and the server reports its ready listener.
4. Use the published application endpoint and rental credentials to make a small inference request with the model name served by that runtime.
5. Increase context size or concurrency gradually after a successful small request. Monitor GPU memory and latency.

Check the result

The inference endpoint accepts an authenticated request and returns output from the loaded model.

Troubleshooting

An incompatible architecture or oversized context can prevent startup. Check logs and supported models; downloading weights or reaching a TCP port is not sufficient to prove inference works.