Development¶

Prerequisites¶

Docker
NVIDIA Container Toolkit (optional — only required for GPU development)
VS Code with the Dev Containers extension

Quick start (Dev Container)¶

The recommended way to develop Modelship is with VS Code Dev Containers. The configuration in .devcontainer/ builds the dev image, mounts the repo, forwards ports, and installs all required extensions automatically.

Set required environment variables on your host:

export HF_TOKEN=your_token_here

Open the repo in VS Code and run Dev Containers: Reopen in Container from the command palette (Ctrl+Shift+P / Cmd+Shift+P).
Once inside the container, sync dependencies:

uv sync --extra dev

Start the server. It starts its own Ray head, auto-detecting CPUs/GPUs unless MSHIP_NODE_NUM_CPUS / MSHIP_NODE_NUM_GPUS are set:

uv run python -m modelship.launcher start

Why not mship start? That console script ships only in the separate bootstrap/ package (the mship installer), never synced into this repo's venv; python -m modelship.launcher <command> is its equivalent here.

Why the extra steps? The dev target is built for source work: it syncs dependencies from uv.lock into /.venv and never installs the project, so you sync and start it yourself. The published images instead run mship bootstrap at build time and start at mship start — see Docker install.

The Dev Container automatically: - Builds the dev image from Dockerfile (target: dev), with redis-server - Bind-mounts the repo to /modelship for live editing - Forwards ports 8000 (API), 8265 (Ray Dashboard), and 8079 (Metrics) - Installs extensions: Ruff, Python, Pyright, and Claude Code - Configures the Python interpreter and linting to use the container's venv at /.venv

Environment variables¶

The following environment variables are set in the dev image with sensible defaults. Override them in your host shell or in .devcontainer/devcontainer.json under remoteEnv as needed:

Variable	Default	Description
`MSHIP_NODE_NUM_CPUS`	(unset)	Optional override: CPUs this node reserves (`--node-num-cpus` flag). Node-scoped, not head-only — the same knob a future worker node sizes itself from. If unset, auto-detects.
`MSHIP_NODE_NUM_GPUS`	(unset)	Optional override: GPUs this node reserves (`--node-num-gpus` flag). If unset, auto-detects.
`MSHIP_NODE_MEMORY`	(unset)	Optional override: this node's total memory budget, e.g. `8Gi` (`--node-memory` flag). Split into Ray's `object_store_memory` and schedulable `memory` resource via Ray's own object-store proportion. If unset, auto-detects from actually-free host RAM (not Ray's own uncapped-cgroup estimate, which can miss non-Docker consumers) — still double/triple-counts when co-locating multiple modelship containers on one host without per-container cgroup memory limits, see the sharp-edge note in `AGENTS.md`.
`MSHIP_RAY_DASHBOARD_HOST`	`127.0.0.1`	Ray dashboard bind host, `start` only (`--ray-dashboard-host` flag). The dashboard always starts; this sets where it binds — `0.0.0.0` exposes it beyond the container (ShadowRay/CVE-2023-48022 exposure vector; only do this on a trusted/private network, with `--enable-ray-auth` — `start` warns without it).
`MSHIP_RAY_AUTH`	`false`	Ray cluster authentication (`--enable-ray-auth` flag), `start` only. `true` makes `start` require a bearer token for the dashboard and cluster-internal RPC. Never gates the OpenAI API or Prometheus metrics.
`MSHIP_GCS_PORT`	`6380`	Ray GCS server port, `start` only (`--gcs-port` flag). Pinned by default (not `6379`, which collides with the recommended same-host Redis state store under `--network=host`) so `mship join --gcs-address` has a stable target across head restarts.
`MSHIP_RAY_DASHBOARD_PORT`	`8265`	Optional override: Ray dashboard port, `start` only (`--ray-dashboard-port` flag). Only needed to run multiple modelship heads on one host under `--network=host`, where Ray's fixed default would otherwise collide between them.
`MSHIP_GCS_ADDRESS`	(unset)	The head's GCS address as `host[:port]` (port `6380` by default) that `mship join` joins (`--gcs-address` flag). See Multi-node without Kubernetes.
`MSHIP_RAY_AUTH_TOKEN`	(unset)	Optional: the Ray auth token (env only, no flag). `join`/`deploy` send it to a cluster started with `--enable-ray-auth`; `start --enable-ray-auth` uses it instead of generating one.
`MSHIP_CACHE_DIR`	`/.cache`	Model cache directory (`--cache-dir` flag). May be shared storage.
`MSHIP_NODE_CACHE_DIR`	`/opt/mship/node-cache`	Node-local vLLM/Triton/FlashInfer compile caches (`--node-cache-dir` flag). Must not be shared storage.
`MSHIP_STATE_STORE`	`memory://`	State-store URI for the gateways' deploy versions + `/v1/responses` conversations: `memory://` or `redis://host:port/db`, with any password in `MSHIP_REDIS_PASSWORD` on every node. See model-configuration.md. The chart always sets `redis://` for k8s.
`MSHIP_GATEWAY_MIN_REPLICAS`	`1`	Fewest replicas each API gateway autoscales down to (`--gateway-min-replicas`). 2 or more gives routing/ingress HA; each replica copies its routing table from the gateway coordinator.
`MSHIP_GATEWAY_MAX_REPLICAS`	`4`	Most replicas each API gateway autoscales up to (`--gateway-max-replicas`).
`MSHIP_GATEWAY_TARGET_ONGOING_REQUESTS`	`64`	Ongoing requests per gateway replica that autoscaling aims for (`--gateway-target-ongoing-requests`). A streamed response counts until it ends.
`MSHIP_GATEWAY_MAX_ONGOING_REQUESTS`	`1024`	Per-replica Ray Serve concurrency cap for the gateway (`--gateway-max-ongoing-requests`). The gateway holds a slot for the whole lifetime of each streamed response, so a low cap throttles before the engine does.
`MSHIP_LLAMA_SERVER_BIN`	`/opt/mship/bin/llama-server.sh` in the Dev Container (unset otherwise)	Unified `llama` executable used by the `llama_server` loader and its preflight; see llama-server binary.

Manual setup (without Dev Containers)¶

If you prefer not to use Dev Containers, you can build and run the dev image directly.

Building the dev image¶

CUDA (GPU):

docker build -t modelship_dev_cuda --target dev .

CPU:

docker build -t modelship_dev_cpu --target dev --build-arg MSHIP_VARIANT=cpu .

Thin (control/coordinator, no torch/vllm):

docker build -t modelship_dev_thin --target dev --build-arg MSHIP_VARIANT=thin .

Running with live source mounting¶

The dev image does not bake in source files. Mount the repo root so changes take effect without rebuilding:

CUDA:

docker run -it --rm --shm-size=8g --gpus all \
  -e HF_TOKEN=your_token_here \
  --mount type=bind,src=./,dst=/modelship \
  -v ./models-cache:/.cache \
  -p 8000:8000 modelship_dev_cuda

CPU:

docker run -it --rm --shm-size=8g \
  -e HF_TOKEN=your_token_here \
  --mount type=bind,src=./,dst=/modelship \
  -v ./models-cache:/.cache \
  -p 8000:8000 modelship_dev_cpu

The dev image drops into a shell. Start the server — it starts its own Ray head, auto-detecting resources:

uv run python -m modelship.launcher start

llama-server binary (`llama_server` loader)¶

The llama_server loader and its preflight both invoke the unified llama binary (llama serve / llama fit-params) found via MSHIP_LLAMA_SERVER_BIN — the env var name predates the binary's unification and stuck. mship bootstrap provisions it (downloaded, sha256-verified, cached under $MSHIP_HOME/builds/<variant>/llama.cpp/) and mship deploy points the env var at its wrapper script — see bootstrap/mship_bootstrap/llama_cpp.py and its _ASSETS table. These are modelship's own builds of one llama.cpp tag for every platform it ships (modelship-ai/llama-cpp-builds), so every node in a cluster runs the identical binary. The cuda variant adds libggml-cuda.so beside it. The thin variant ships no binary at all — it never loads a model.

The published images run that same bootstrap at build time, so they arrive preconfigured. The dev target does not run it, but the Dev Container fetches it for you: .devcontainer/provision-llama-server.sh runs from postCreateCommand, importing bootstrap/mship_bootstrap/llama_cpp.py straight from source (it's stdlib-only) to reuse its fetch/verify logic against the cuda variant, and MSHIP_LLAMA_SERVER_BIN is preset (via remoteEnv) to the stable symlink it leaves at $MSHIP_HOME/bin/llama-server.sh. Building and running the dev image manually still skips this — set MSHIP_LLAMA_SERVER_BIN yourself, run that script, or run mship bootstrap inside it, to exercise the llama_server loader there.

For a platform outside that table, or to use a different llama.cpp build than the pinned one, download a build from llama.cpp releases and extract it. The raw llama binary does not run standalone — it dynamically links the libggml*/libllama* libraries that ship as sibling files in the same archive, so point MSHIP_LLAMA_SERVER_BIN at a small wrapper that puts the extracted directory on the loader path:

#!/bin/sh
# llama-server.sh — use DYLD_LIBRARY_PATH instead on macOS
export LD_LIBRARY_PATH="/path/to/extracted${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
exec /path/to/extracted/llama "$@"

export MSHIP_LLAMA_SERVER_BIN=/path/to/llama-server.sh

Production Builds¶

Three images are published — a thin control/coordinator image (no torch/vllm) plus two accelerator node images:

Variant	Tag suffix	Platforms
Thin (control/coordinator)	(none) — default	amd64, arm64
CUDA (GPU node)	`-cuda`	amd64
CPU (CPU node)	`-cpu`	amd64, arm64

All three share the prod target; MSHIP_VARIANT is what differs.

Floating tags (:latest, :latest-cuda, :latest-cpu) are single-node only — Ray refuses to form a cluster across mismatched versions, so any multi-node deployment must pin every node to the same :X.Y.Z (or -cuda/-cpu) tag.

The production images install the release wheels rather than the source tree, so build those first — release.yml does the same, from a wheels job the image builds depend on:

uv build -o wheels .
uv build -o wheels bootstrap

Then, per variant (thin shown; swap in cpu or cuda):

docker build -t modelship:dev-thin \
  --target prod \
  --build-arg MSHIP_VARIANT=thin \
  --build-arg MSHIP_VERSION="$(uv version --short)" \
  --build-context wheels=./wheels \
  .

MSHIP_VERSION must match the wheels in ./wheels — the build resolves mship==<version> against that directory, so a stale wheels/ from an earlier version fails to resolve rather than silently installing the wrong one. Rebuild the wheels after a version bump.

Ports¶

Port	Service
`8000`	API
`8265`	Ray dashboard
`8079`	Metrics (`MSHIP_METRICS=false` to disable)