DeployMachine — Open-Weight LLMs on GPUs in Your Own Cloud

The customer's cloud, without the customer's keys

A customer connects their own AWS account through a CloudFormation quick-create role with an external id, not long-lived access keys. Before the connection is accepted, verification checks that assuming the role works with the external id and fails without it.

Every cloud sits behind one provider interface with an adapter written in-house (DigitalOcean first, then AWS) rather than an outside provisioning project, so core infrastructure does not depend on someone else's roadmap.

One Go codebase holds the control plane and the node agent; Postgres holds both the state and the job queue. There is no broker and no second datastore.

2 cloudsNVIDIA + AMD1 Go codebase

Our own production traffic runs on it

One DeployMachine node, a single AMD MI325X on ROCm, co-hosts three models with vLLM: a 27B FP8 LLM (60% of GPU memory) for chat and extraction, a 9B LLM (20%) for small fast tasks, and a 0.6B embedding model (5%). They serve a production CRM and sales-assistant backend. With an FP8 KV cache, the 27B's share holds about 29 concurrent requests at the full 32k context. The 9B also exposes vLLM's decision endpoint, which returns probabilities over a fixed answer set with no generated tokens; in testing through the DeployMachine proxy, a topic classification put 0.99 on the correct label. The 256 GiB card was chosen so the whole model set fits on one GPU instead of several rented ones, which is why ROCm support was built. vLLM is pinned in its own container because the cloud's own inference image shipped vLLM 0.9.0. Since 26 October the backend's LLM gateway tries these self-hosted models first and falls back to Claude, then OpenAI; any server or network error turns the self-hosted route off for 60 seconds, so users never see an outage. The page builder deliberately stays on a frontier model: self-host where it is good enough, keep frontier models where quality is the product. Assistant chat, pitch writing, signal classification and relationship summaries all run on them first.

Embeddings for three production services moved from OpenAI to self-hosted Qwen embeddings (1024 dimensions) served by vLLM's pooling runner, with the OpenAI collections kept for rollback.

The model choice was measured, not assumed: on 30 real contacts the 9B was about 4x faster at structured extraction but broke the output schema 3 times and gave looser labels, so extraction stayed on the 27B.

Background work shares the GPU with chat. vLLM priority scheduling (user-facing calls at priority 0, background at 10), plus a back-off that drops the background worker to one request at a time for 60 seconds after any user-facing call, keeps chat responsive while the queue drains; after queue fixes, relationship summaries went from about 350 to about 3,900 an hour. In an evening spot check every one of about 40 model calls ran on our own models, at no per-call cost.

Several models share one machine, each behind its own route, and editing one model's settings leaves the others serving. Instances can also take a custom name, <name>.nodes.deploymachine.com, served over TLS by the node agent.

3 models on 1 MI325X~11x summaries per hour$0 per call

Refuse misfits, clean up failures

A model that does not fit used to crash-loop for 45 minutes on a paid H100 before failing. Now a fit check reads the model's file sizes and config from Hugging Face, adds weights, overhead and the KV cache for one max-length request against one GPU's share, and refuses a clear misfit before any machine is created.

The same numbers feed the console's deploy wizard, so the user sees what fits before clicking Deploy.

A machine whose bootstrap fails is torn down automatically. A model that fails to load keeps its machine for 15 minutes so it can be fixed, then the machine is deleted.

Multi-GPU sizes spread each model across every GPU with tensor parallelism, and models can be edited on a live instance: the node agent restarts only the models whose settings changed.

Measured on real hardware: an H200 went from request to serving Qwen3.5-9B on vLLM in 5 min 30 s, answering 32 tokens in 0.30 s with the first byte in 80 ms; an AMD MI325X went from create to serving a 27B model and back to deleted in 7.5 min.

On AWS, a fresh g6.xlarge (one L4) first took 17.5 minutes to serve a model, about 12 of them pulling and unpacking the vLLM image and starting vLLM. Putting Docker images and weights on the instance's local NVMe, and timing every step, brought it to 10 minutes; what remains is the ~6 minute image download and vLLM's own startup.

0 GPUs for a misfit5 min 30 s to servingAWS 17.5 → 10 min