Skip to content

Native install

One install command on every platform, then one bootstrap to pick the node's role:

uv tool install mship       # or: pipx install mship / pip install mship
Bootstrap command Node role
mship bootstrap --cuda NVIDIA GPU node (vLLM, Diffusers, llama.cpp GPU offload)
mship bootstrap --cpu CPU node (vLLM CPU, llama.cpp, whisper.cpp, sherpa-onnx, stable-diffusion.cpp)
mship bootstrap --metal Apple Silicon (Metal offload)
mship bootstrap --thin Coordinator/head only — serves nothing itself

Then, with no variant flag:

mship deploy --config models.yaml

Bootstrapping with no variant is an error that lists these options; there is no default and no auto-detection, so a node's role is always something you stated. MSHIP_VARIANT is the environment-variable equivalent, for systemd units and CI, and overrides the recorded role for a single command.

What mship bootstrap does

The mship package is a small installer that runs on any Python 3.10+. Bootstrapping:

  1. Refuses unsupported platforms (Windows, musl/Alpine) before downloading anything.
  2. Checks the variant's build prerequisites — --cuda needs nvcc and ninja.
  3. Provisions ~/.modelship/envs/<variant>/ on CPython 3.12.10, installing from a hash-pinned dependency list shipped inside the package.
  4. Fetches the pinned llama-server build for the platform.
  5. Records the variant in ~/.modelship/env.

Every node therefore lands on an identical interpreter and dependency set, which is what lets a native node join a cluster of Docker nodes — Ray refuses to form a cluster across mismatched Python versions. The Docker images run these same steps at image-build time.

A copy of the exact pinned list that built each environment is left at ~/.modelship/envs/<variant>/pins.txt. mship info reports what is provisioned.

deploy itself installs nothing. It stops if the environment is missing or was built by a different mship version:

error: the cuda environment was built for mship 0.7.12, but this is 0.7.13.

Run: mship bootstrap --cuda

So upgrading is uv tool upgrade mship followed by mship bootstrap.

To bootstrap more than one variant on a host, run bootstrap for each — the last one is the recorded default, and mship deploy --thin selects another explicitly.

Bootstrapping never checks for the accelerator itself, so --cuda provisions on a host with no GPU — a golden image, or a node whose driver is not up yet. The variant flag decides what to install; deploy is what refuses to run without the hardware.

Platform prerequisites

These are not installed for you.

# macOS — compiles stable-diffusion-cpp-python on first install
xcode-select --install

# Linux, all variants
sudo apt-get install -y build-essential cmake

# Linux --cpu, additionally (lscpu feeds vLLM's CPU NUMA detection; its absence
# surfaces as an opaque `Engine core initialization failed`)
sudo apt-get install -y libnuma1 openssl util-linux

# Linux --cuda, additionally: flashinfer JIT-compiles kernels at model-load time
sudo apt-get install -y ninja-build gnupg
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
  | sudo gpg --dearmor -o /usr/share/keyrings/cuda-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/cuda-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /" \
  | sudo tee /etc/apt/sources.list.d/cuda.list
sudo apt-get update && sudo apt-get install -y \
  cuda-nvcc-13-0 cuda-cuobjdump-13-0 libcurand-dev-13-0

nvcc does not need to be on PATH — those packages create the /usr/local/cuda symlink flashinfer looks for. The first vLLM deploy is slow while kernels compile; they are cached per GPU architecture under ~/.modelship/cache/flashinfer.

mship bootstrap --cuda refuses to run without nvcc and ninja, rather than letting the first vLLM deploy fail with Could not find nvcc after several GB have downloaded. The --cuda variant ships vLLM and Diffusers, so both are required even though llama_server GGUF offload needs neither. Being software, they are checkable wherever you provision — unlike the GPU, which bootstrap does not look for at all. mship info reports the same check afterwards.

Loader coverage on --cuda. vllm, diffusers, and llama_server all get full GPU. llama_server gets a CUDA ggml backend beside the same binary every other platform runs. ggml skips a backend it cannot load without saying so, so if a GGUF deploy seems slow, check "$MSHIP_LLAMA_SERVER_BIN" --list-devices — it prints (none) instead of a CUDA0: line. stable_diffusion_cpp, whispercpp, and sherpa_onnx are CPU-only here, same as in the image.

Files on disk

~/.modelship/
  env                       the variant bootstrap recorded (MSHIP_VARIANT=…)
  cache/                    models and other downloads (MSHIP_CACHE_DIR)
  envs/<variant>/           one environment per variant
  builds/<variant>/         llama-server binaries
  bin/uv                    only if uv was not already installed

~/.modelship/env is written read-only by bootstrap and holds a single MSHIP_VARIANT= line. Being an ordinary env file, it drops straight into a systemd unit:

[Service]
EnvironmentFile=/home/youruser/.modelship/env
ExecStart=/home/youruser/.local/bin/mship deploy --config /etc/modelship/models.yaml

Change it with mship bootstrap, not by hand.

MSHIP_CACHE_DIR may point at shared storage — model weights are identical on every node. MSHIP_HOME (default ~/.modelship) must stay node-local: environments and binaries are platform- and variant-specific. To reset a variant, delete ~/.modelship/envs/<variant>/ and bootstrap it again; your models are untouched.

Scaling beyond one node

To join multiple hosts into one Ray cluster, see Multi-node without Kubernetes — the --address/--token flags work the same for a native node as for a container.