Self-hosted inference
Published Updated
What is Self-hosted inference?
The practice of running and managing AI models on your own servers or local hardware, rather than sending data to external providers like OpenAI.
Self-hosted inference involves downloading open-weight models and running them using engines like vLLM. Organizations do this to maintain strict data sovereignty, avoid vendor lock-in, and bypass the unpredictable rate limits and latency fluctuations of public AI APIs.
How it relates to Orbit
Orbit can route a tier to a local or custom OpenAI-compatible endpoint when you host the model yourself. See Complexity Tiers & Routing for details.
When you would use it
You mandate self-hosted inference when handling highly sensitive medical, financial, or defense data that legally cannot be transmitted to third-party cloud AI providers.
Common operations
- Deploying a proprietary Llama model securely within a corporate firewall.
- Ensuring zero customer data leaves the internal network.
Related terms
Dashed terms are named in the source but do not have a page here yet.
Where this is taught
No learning path uses this term yet. Browse the Learning Atlas for guided sequences through related ideas.