Docker Model Runner

Docker Model Runner standardizes local LLM deployment by wrapping high-performance inference engines like llama.cpp into portable, OCI-compliant containers.

Docker Model Runner eliminates the 'works on my machine' headache for AI engineering. It packages model weights and inference runtimes into a unified Docker image, allowing developers to spin up local APIs (compatible with OpenAI's schema) using a single command. By leveraging Docker Desktop's GPU passthrough for NVIDIA and Apple Silicon, it delivers near-native performance for models like Llama 3 or Mistral 7B without requiring complex local dependency management. This approach ensures that every member of a dev team runs the exact same model version and environment, accelerating the transition from local prototyping to production-ready microservices.

https://github.com/docker/labs-ai-tools-for-devs