Skip to content

Model Configuration

Reference for models.yaml (default: config/models.yaml). Each entry under models: defines one deployment. Unknown keys are rejected: a typo like n_ctxx is a validation error, not a silently ignored setting — in the file and in the CLI flags alike.

CLI Options

mship deploy accepts these arguments (env vars work as fallbacks; CLI wins over env):

Argument Env Var Default Description
--config — config/models.yaml Path to models config file. An explicit path that doesn't exist is a hard error
--gateway-name MSHIP_GATEWAY_NAME modelship Name for the API gateway app. Multiple gateways can coexist on one cluster, each mounted at /<slugified-name> (e.g. modelship → /modelship/v1/...)
--gateway-replicas MSHIP_GATEWAY_REPLICAS 1 Number of API gateway replicas (routing/ingress HA; replicas sync routing via the deploy coordinator)
--openai-api-port MSHIP_OPENAI_API_PORT 8000 Port for the OpenAI-compatible API
--use-existing-ray-cluster MSHIP_USE_EXISTING_RAY_CLUSTER false Connect to a Ray cluster you manage (driver must run on a cluster node) instead of starting one. Implies deploy-and-exit, no teardown
--address MSHIP_ADDRESS — Join an existing Ray cluster as an additional compute node, given its head's GCS address as host:port (e.g. mship-head:6380). Mutually exclusive with --use-existing-ray-cluster. See Multi-node without Kubernetes
--token MSHIP_RAY_AUTH_TOKEN — Cluster auth token for joining a head running --ray-auth=token. Only meaningful with --address; retrieve via docker exec <head> cat ~/.ray/auth_token
--ray-auth MSHIP_RAY_AUTH none Ray cluster authentication, own-head only. token requires a bearer token (generated by Ray itself) for the dashboard and cluster-internal RPC
--ray-port MSHIP_RAY_PORT 6380 Ray GCS server port, own-head only — what a joiner's --address points at
--dashboard-port MSHIP_RAY_DASHBOARD_PORT 8265 Ray dashboard port, own-head only. Only needed to run multiple modelship heads on one host under --network=host
--node-num-cpus MSHIP_NODE_NUM_CPUS auto-detect CPUs this node reserves
--node-num-gpus MSHIP_NODE_NUM_GPUS auto-detect GPUs this node reserves. Refused at startup if it exceeds what the container can actually see
--node-memory MSHIP_NODE_MEMORY auto-detect This node's total memory budget, e.g. 8Gi. Set explicitly when co-locating multiple modelship containers on one host without per-container cgroup memory limits
--prune-ray-sessions MSHIP_PRUNE_RAY_SESSIONS true When starting its own Ray head, delete stale session_* dirs left under the Ray temp root by previous, no-longer-running heads. A live head's session is always kept
--reconcile — false Make the cluster match the config: add new models, remove dropped ones, replace changed ones (vs. the default additive union). With no --config, reconciles to this gateway's persisted effective config (self-heal)
--replace-strategy — blue_green How to replace a changed model: blue_green (deploy new before dropping old, no request loss) or stop_start (drop old first, brief unavailability)
--cache-dir MSHIP_CACHE_DIR /.cache Base cache directory
--state-store MSHIP_STATE_STORE memory:// Connection URI for the effective config + deploy coordinator + /v1/responses state (see State store)
— MSHIP_LOG_LEVEL INFO Log level (env-var-only: must be set before import ray so library loggers latch the right level)
--log-format MSHIP_LOG_FORMAT text text or json
--log-target MSHIP_LOG_TARGET console console or syslog URI (e.g. syslog://host:514, syslog+tcp://host:514)
--otel-endpoint OTEL_EXPORTER_OTLP_ENDPOINT — OpenTelemetry OTLP endpoint (e.g. http://collector:4317)
--no-metrics MSHIP_METRICS enabled Disable Prometheus metrics (port 8079)
--no-preflight MSHIP_PREFLIGHT enabled Disable preflight hardware auto-sizing; models run on loader/library defaults plus explicit config. Useful for benchmarking
--api-keys MSHIP_API_KEYS — Comma-separated API keys
--trusted-identity-header MSHIP_TRUSTED_IDENTITY_HEADER — Header name (e.g. X-Consumer-Id) a fronting credentials layer sets with a caller identity it already resolved and authorized. See Trusted identity header
--max-request-body-bytes MSHIP_MAX_REQUEST_BODY_BYTES 52428800 Max request body size in bytes
--responses-ttl-s MSHIP_RESPONSES_TTL_S 2592000 TTL in seconds for stored /v1/responses conversation state; <=0 disables expiry
--state-sweep-interval-s MSHIP_STATE_SWEEP_INTERVAL_S 300 Interval in seconds between expired-key sweeps in the in-memory state store

Single-model deploys (no config file)

--model deploys one model straight from the command line, no models.yaml needed:

mship deploy --model lmstudio-community/Qwen3-8B-GGUF:'*Q4_K_M.gguf' \
             --loader llama_server --usecase generate --num-cpus 4

These flags mirror the root-level fields of a models: entry one for one, and are validated by the same schema — anything rejected in the file is rejected here, with the same message:

Flag Field
--model model
--name name (inferred from --model when omitted)
--usecase usecase
--loader loader
--num-gpus num_gpus
--num-cpus num_cpus
--num-replicas num_replicas
--max-ongoing-requests max_ongoing_requests

Tuning blocks

The nested blocks are generated from the same schema, one flag per key, named for its config path — --<block>.<key>, hyphenated:

mship deploy --model lmstudio-community/Qwen3-8B-GGUF:'*Q4_K_M.gguf' \
             --loader llama_server --usecase generate \
             --llama-server-config.n-ctx 8192 \
             --llama-server-config.parallel 4

So --vllm-engine-kwargs.max-model-len 8192 is vllm_engine_kwargs: {max_model_len: 8192}, --autoscaling-config.max-replicas 3 is autoscaling_config: {max_replicas: 3}, and so on for diffusers_config, stable_diffusion_cpp_config and whispercpp_config. chat_template_kwargs is free-form, so it takes one whole map. mship deploy --help lists every flag with its type and default.

Values are read as YAML: the same text you would write after the colon in the file.

Flag Value
--llama-server-config.n-ctx 8192 8192
--vllm-engine-kwargs.trust-remote-code true true
--vllm-engine-kwargs.limit-mm-per-prompt '{image: 2}' a map
--llama-server-config.tensor-split '[3, 1]' a list
--llama-server-config.threads null null — an explicit value, not "left unset"
--chat-template-kwargs '{enable_thinking: false}' a map

That includes YAML's own quirks: bare on, off, yes and no are booleans, so quote them — --llama-server-config.chat-template '"on"' — when the string is what you mean.

Limits:

  • One model per invocation. Use --config for several. A second mship deploy --model ... against a running cluster adds to it (the default additive merge), so models can also be added one at a time.
  • --model and --config are mutually exclusive. With --model, the default config/models.yaml is ignored entirely. The tuning flags configure the model --model deploys, so they need it too.
  • A flag can set a key, not unset one. Omitting it leaves the schema default; pass null for the fields that accept it.

Inferred names

With no --name, the name clients call the model by comes from the model reference — its basename (or the filename stem for a local weight file), minus GGUF and quantization decoration. The selector is ignored: it picks a quant, it doesn't identify the model.

--model Inferred name
lmstudio-community/Qwen3-8B-GGUF:*Q4_K_M.gguf qwen3-8b
bartowski/Llama-3.3-70B-Instruct-GGUF:*Q8_0*-of-*.gguf llama-3.3-70b-instruct
Qwen/Qwen3-8B qwen3-8b
/models/qwen3-8b-instruct.Q4_K_M.gguf qwen3-8b-instruct
~/models/Qwen3-8B/ qwen3-8b
base.en base.en

Inference is deterministic, so re-running the same command is idempotent. It also means two different references can infer the same name — deploying Qwen/Qwen3-8B on vllm and then a Qwen3-8B GGUF on llama_server both resolve to qwen3-8b, and the second replaces the first, since a name maps to exactly one deployment. That is what you want when swapping a model's loader; pass --name to run both side by side.

Cache directory structure

Under MSHIP_CACHE_DIR (default /.cache):

Subdir Contents Env var
huggingface HF models and tokenizers HF_HOME
vllm vLLM compiled artifacts VLLM_CACHE_ROOT
flashinfer FlashInfer kernels FLASHINFER_CACHE_DIR
whispercpp pywhispercpp built-in model downloads MSHIP_WHISPERCPP_CACHE_DIR
sherpa_onnx/<name> sherpa-onnx registry tarballs —

Additive vs. reconcile deploys

By default, deploys add models to a running cluster without touching existing deployments — run repeatedly against different config files to compose a cluster incrementally. --reconcile instead makes the cluster match the config exactly (add/remove/replace, draining in-flight requests on replace); it never tears the cluster down.

mship deploy --config config/llm.yaml            # deploy
mship deploy --config config/tts.yaml             # add, doesn't touch the LLMs
mship deploy --config config/models.yaml --reconcile   # make the cluster match exactly

Trusted Identity Header

modelship never authenticates callers itself — that's MSHIP_API_KEYS' job, and it stops at "is this caller allowed at all." There is no login, permissions, or per-model access control, and none is planned; that belongs to whatever sits in front (nginx, Kong, LiteLLM, a custom credentials layer). MSHIP_TRUSTED_IDENTITY_HEADER lets that layer forward a caller identity it already resolved (a consumer/tenant id), used for log correlation and for scoping server-side state (see Stateful responses) — never for authorization.

The header's value is trusted unconditionally — no signature check. Safe only if both hold:

  1. The fronting layer unconditionally overwrites the header, stripping any client-supplied copy. A layer that only sets it when absent lets a client send X-Consumer-Id: someone-elses-id and impersonate them.
  2. modelship is reachable only from that fronting layer (network policy, private subnet, or same pod) — never directly from any client, who could otherwise set the header themselves.

For stronger guarantees than network isolation, add mTLS on that internal hop (service-mesh sidecar, Kong, a local proxy) so the peer's certificate — not just placement — proves the request's origin. modelship does not implement or verify certificates itself.

If unset (the default), modelship falls back to hashing the matched MSHIP_API_KEYS entry, and further to a single shared bucket if no key matches (or auth is disabled).

Fields

Field Type Description
name string Model identifier used in API requests. Maps to exactly one deployment — a repeated name with different settings is a validation error
model string HuggingFace repo ID, local path, or repo:filename (see Model source)
usecase string generate, embed, transcription, translation, tts, or image. Defaults to image and is otherwise rejected for diffusers/stable_diffusion_cpp; must be tts for sherpa_onnx
loader string vllm, diffusers, llama_server, stable_diffusion_cpp, whispercpp, sherpa_onnx
num_gpus float | int Default 0. Fractional < 1 shares one GPU — supported by vllm, diffusers, llama_server, and (Metal-only) whispercpp; forced to 0 for sherpa_onnx and off-Darwin stable_diffusion_cpp/whispercpp. Integer ≥ 1 requests that many whole GPUs (for vllm, auto-sets tensor_parallel_size = num_gpus unless tp/pp is already specified). llama_server/whispercpp reject a non-integer ≥ 1. See Sharing one GPU
num_cpus float CPU units to allocate. Default 0.1
num_replicas int Fixed Ray Serve replica count. Default 1. Mutually exclusive with autoscaling_config
autoscaling_config object Autoscale replicas with load instead of a fixed num_replicas (see Autoscaling)
max_ongoing_requests int Per-replica Ray Serve concurrency cap (default: Ray Serve's own default). Streaming requests hold a slot for the whole generation, so a low cap throttles upstream of the engine
vllm_engine_kwargs object vLLM engine options (see vLLM Loader)
diffusers_config object Diffusers pipeline options (see Diffusers Loader)
llama_server_config object llama-server loader options (see llama_server Loader)
stable_diffusion_cpp_config object stable-diffusion.cpp loader options (see stable-diffusion.cpp Loader)
whispercpp_config object whisper.cpp loader options (see whispercpp Loader)
chat_template_kwargs object Extra variables forwarded into the chat-template render — vllm only. E.g. enable_thinking: false for Qwen3. Only has an effect if the model's template branches on the key. A per-request chat_template_kwargs overrides the model default

sherpa_onnx has no sherpa_onnx_config: provider is always cpu and thread count is derived from num_cpus, neither user-supplied.

Sharing one GPU

A fractional num_gpus (0 < n < 1) lets two or more deploys share one physical GPU via Ray's own fractional scheduling — e.g. a vllm model at 0.7 and a llama_server model at 0.3 on the same card. Keep fractions on one GPU summing to at most 0.9: the remaining headroom covers each engine's fixed pre-allocation overhead (CUDA context, etc.) outside the declared share.

Each GPU-capable loader sizes and enforces its share differently:

Loader Enforcement
vllm Hard — allocates total VRAM × gpu_memory_utilization (a fractional num_gpus sets this directly) and fails at startup if it doesn't fit
diffusers Hard — torch.cuda.set_per_process_memory_fraction caps the process; allocating past it OOMs
llama_server Static — preflight sizes n_ctx/n_gpu_layers to fit the declared share; llama.cpp allocates upfront and never grows
whispercpp None (Metal only) — num_gpus > 0 is a plain on/off switch, no VRAM knob to size
sherpa_onnx, stable_diffusion_cpp (off-Darwin) N/A — never touch CUDA; num_gpus is ignored (forced to 0)

Preflight sizes a fractional deploy from its declared share of the GPU's total capacity, not free VRAM at that moment; if free VRAM falls short, it logs a warning naming both numbers.

Sharp edges: - No SM/compute isolation between co-tenants — they time-slice like any two CUDA processes sharing a device (fine for one large + one small model; not a MIG/MPS substitute if both are large). - On a multi-GPU node you can't pin which physical GPU a fractional deploy lands on — Ray picks. - If a co-tenant frees VRAM while vLLM is mid-startup (profiling its own footprint), vLLM's internal consistency check can abort; Ray Serve's replica restart recovers automatically.

Model source

The model: field accepts three forms. For built-in loaders, modelship validates the source on the driver before any Ray actor spins up — auth failures, missing repos, bad selectors, and (on vllm) GGUF files all surface at startup, not inside a stuck deployment. Weight download happens per-replica, on whichever node hosts it (see Multi-node clusters) — the driver only resolves metadata and pins the revision every node fetches.

Form Example Notes
HF repo ID Qwen/Qwen3-7B Downloaded with a universal filter (prefers *.safetensors, skips *.bin when both exist)
Local path /mnt/nfs/models/qwen-7b A directory of HF-format files, or a single file for llama.cpp/vllm GGUF
repo:filename lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf Glob selector; must match exactly one file (or a sharded set, e.g. *-of-*.gguf)

The :filename selector also works against a local directory: if model: is a directory path containing :, the selector matches files inside it; the loader gets the resolved file's full path.

Multi-node clusters

Each node downloads its own copy of whatever gets scheduled onto it — not the driver's HF_HOME shared out to workers. A thin/control-only node that never hosts a replica downloads nothing; a node that does host one pulls exactly that model, pinned to the revision the driver validated (byte-identical weights across nodes even if the upstream repo changes between deploys).

  • Shared storage (NFS/EFS) for MSHIP_CACHE_DIR is optional, not required — mount it to dedupe across nodes; without it, each node downloads its own, correctly.
  • Every node that can host a model needs its own disk and egress (HF rate limits apply per node).
  • A local-path model: is resolved on whichever node hosts the replica — the path must exist on every node that could host it; there's no cross-node copying.

A model also only schedules onto a node whose image has that loader's backend installed (cpu has no diffusers, for instance): every node advertises mship_<loader> Ray custom resources for what it can run, and every deploy requests its loader's resource. See Architecture: Capability-aware scheduling and MSHIP_NODE_CAPABILITIES to override the probe.

See Multi-node without Kubernetes for the full non-k8s cluster setup (auth, ports, co-location).

Multi-variant GGUF repos

If model: points at an HF repo with more than one .gguf file and no :filename selector, modelship raises at startup listing the variants:

HF repo 'lmstudio-community/Qwen2.5-7B-Instruct-GGUF' contains 5 GGUF variants — pick one with the `:filename` syntax (glob supported, must match exactly one file):
  - Qwen2.5-7B-Instruct-Q2_K.gguf
  - Qwen2.5-7B-Instruct-Q4_K_M.gguf
  - Qwen2.5-7B-Instruct-Q5_K_M.gguf
  - Qwen2.5-7B-Instruct-Q8_0.gguf
  - Qwen2.5-7B-Instruct-fp16.gguf
Example: model: lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf

vLLM Loader

Chat/generation, embeddings, transcription, and translation. Configured via vllm_engine_kwargs.

Two vLLM settings are not keys here, because modelship derives them and setting them is a config error: model (the engine always loads the resolved top-level model:) and gpu_memory_utilization (a fractional num_gpus for a shared GPU, else a preflight recommendation, else 0.9 — 0.4 on CPU, where vLLM reads it as a host RAM fraction instead of VRAM).

Field Type Default Description
tensor_parallel_size int 1 GPUs for tensor parallelism
pipeline_parallel_size int 1 GPUs for pipeline parallelism
max_model_len int auto Max sequence length; must be positive if set. Left unset on GPU, vLLM fits the largest context its own post-profiling memory allows (the min across workers, so TP/PP are covered). On num_gpus: 0, preflight sizes it from host RAM instead, falling back to that same auto-fit when it declines (missing config.json, unreadable KV-cache geometry, etc.). Setting it explicitly overrides both
dtype string auto auto, float16, bfloat16
tokenizer string model default Custom tokenizer path
trust_remote_code bool false Allow remote code execution
quantization string — e.g. awq, gptq
enable_auto_tool_choice bool — Enable automatic tool/function calling
tool_call_parser string — e.g. llama3_json, hermes
enforce_eager bool — Disable CUDA graph capture
kv_cache_dtype string — e.g. fp8
enable_prefix_caching bool vLLM default: on Disable to turn off vLLM's automatic prefix caching entirely. Not usually needed — every request is already cache-salted per caller identity (see Identity-scoped prefix caching); this is for an operator who wants caching off regardless of identity
max_num_batched_tokens int — vLLM scheduler batch cap
max_num_seqs int — Max concurrent sequences
limit_mm_per_prompt dict — Per-modality multimodal item cap, e.g. {"image": 4}
mm_processor_kwargs dict — Forwarded to the HF processor (e.g. min_pixels/max_pixels for Qwen2.5-VL)

GGUF is not supported on the vllm loader — vLLM 0.24 dropped in-tree GGUF, so a .gguf source is rejected at driver preflight, unconditionally regardless of GPU vs. CPU. Use loader: llama_server for GGUF; vllm takes safetensors, or AWQ/GPTQ/FP8 quants.

CPU (no GPU required)

Installable via the vllm-cpu extra (paired with num_gpus: 0) for quantized chat with no GPU — safetensors, or AWQ/GPTQ/compressed-tensors quants (CPU backend supports AWQ/GPTQ on x86 plus INT8 W8A8); GGUF is rejected here too.

gpu_memory_utilization means host RAM fraction on CPU, not VRAM — modelship's default drops to 0.4 (vLLM's own 0.9 would try to reserve 90% of node RAM and fail at worker init). Since it isn't a settable key, preflight's recommendation (from actual free RAM and the model's weight footprint) always applies when it can compute one; the 0.4 fallback only kicks in when preflight declines (e.g. unreadable config.json). Set max_model_len explicitly for finer control. vLLM also reads VLLM_CPU_KVCACHE_SPACE (fixed GiB budget) and VLLM_CPU_OMP_THREADS_BIND (thread pinning) directly from the environment — vLLM-native, not modelship config.

Minimum config — preflight fills in the rest:

models:
  - name: qwen-cpu
    model: Qwen/Qwen3-8B-AWQ
    usecase: generate
    loader: vllm
    num_gpus: 0

See config/examples/vllm-cpu.yaml for a complete example with tool calling.

Examples

models:
  - name: qwen                              # chat
    model: Qwen/Qwen3-0.6B
    usecase: generate
    loader: vllm
    num_gpus: 0.30
    vllm_engine_kwargs:
      max_model_len: 8192

  - name: llama                             # tool calling
    model: meta-llama/Llama-3.1-8B-Instruct
    usecase: generate
    loader: vllm
    num_gpus: 0.70
    vllm_engine_kwargs:
      enable_auto_tool_choice: true
      tool_call_parser: llama3_json

  - name: nomic-embed                       # embeddings
    model: nomic-ai/nomic-embed-text-v1.5
    usecase: embed
    loader: vllm
    num_gpus: 0.15
    vllm_engine_kwargs:
      trust_remote_code: true

  - name: whisper                           # transcription
    model: openai/whisper-large-v3-turbo
    usecase: transcription
    loader: vllm
    num_gpus: 0.15
    vllm_engine_kwargs:
      trust_remote_code: true

Multi-GPU with tensor parallelism

num_gpus: 2 is shorthand for "2 whole GPUs" — tensor_parallel_size is auto-derived. Setting both is redundant (a warning is logged); setting only tensor_parallel_size/pipeline_parallel_size is fine too. Each slot always owns one whole GPU.

models:
  - name: llama-70b
    model: meta-llama/Llama-3.1-70B-Instruct
    usecase: generate
    loader: vllm
    num_gpus: 2

Multi-slot deploys always use vLLM's ray distributed executor: each TP/PP slot is its own Ray worker actor inside a Ray Serve placement group (STRICT_PACK, one whole-GPU bundle per slot, all on one node for NVLink).

Fractional num_gpus (< 1) is single-GPU only. Combining it with tensor_parallel_size > 1 or pipeline_parallel_size > 1 is rejected at config time — Ray packs fractional placement-group bundles onto the same physical GPU, which breaks tensor parallelism. Share a GPU with num_gpus: 0.x and tp: 1; use whole-GPU integer num_gpus for TP.

Diffusers Loader

HuggingFace Diffusers for image generation; any AutoPipelineForText2Image-compatible model works out of the box. Configured via diffusers_config:

Field Type Default Description
torch_dtype string float16 float16, bfloat16, float32
num_inference_steps int 30 Default denoising steps (overridable per request)
guidance_scale float 7.5 Default classifier-free guidance scale (overridable per request) 
models:
  - name: sdxl-turbo
    model: stabilityai/sdxl-turbo
    usecase: image
    loader: diffusers
    num_gpus: 0.35
    diffusers_config:
      torch_dtype: "float16"
      num_inference_steps: 4
      guidance_scale: 0.0

llama_server Loader

Runs GGUF models by launching a llama-server subprocess and proxying its native OpenAI-compatible HTTP API. Chat templating, tool-call parsing, and reasoning parsing are all llama-server's own (--jinja --reasoning-format auto), not modelship's. --parallel request slots let concurrent requests actually overlap instead of serializing behind a single lock. Requires the unified llama binary discoverable via MSHIP_LLAMA_SERVER_BIN (see development.md); Docker images ship a pinned build at /opt/llama.cpp.

num_gpus accepts 0 (CPU-only), a fraction < 1 (shares one physical GPU), or a whole integer — see Sharing one GPU. Configured via llama_server_config:

Field Type Default Description
n_ctx int auto (preflight); 2048 when preflight declines Per-slot context length. The launch command multiplies this by parallel for llama-server's total -c. Preflight shells out to llama fit-params, which builds the real KV cache and compute buffers to solve n_ctx/n_gpu_layers/tensor_split together
n_batch int 512 Batch size for prompt processing
n_gpu_layers int auto (preflight); -1 when preflight declines Layers offloaded to GPU when num_gpus > 0; forced to 0 when num_gpus is 0. -1 hits llama-server's own auto-fit-to-free-memory path — verified for any negative value against b9859, unconfirmed on the current b10375 pin (--help's documented 'auto'/'all' string tokens aren't reachable through this int field)
threads int None (llama-server default: all cores) Compute thread count (--threads). Preflight recommends num_cpus when the deploy reserves ≥1 whole CPU and it wouldn't undercut parallel
parallel int 1 Concurrent request slots (--parallel). Also becomes max_ongoing_requests's default when that's unset, so overflow queues in Ray Serve rather than inside llama-server
chat_template string — Built-in template name (e.g. chatml) or a Jinja file path. Omit to use the GGUF's embedded template
mmproj string — Multimodal projector file/repo ref for vision models — see Vision
cache_reuse int 0 Min chunk size (tokens) for fuzzy KV-cache reuse via position-shifting (--cache-reuse). 0 means exact-prefix reuse only; raise it to also reuse chunks after a mid-prompt divergence (changed system prompt, swapped RAG doc)
context_shift bool false Evict oldest tokens and keep generating when a slot's context fills, instead of erroring (--context-shift)
cache_ram_mib int None (llama-server default: 8192) In-RAM prompt-cache cap in MiB (-cram). -1 = no limit, 0 disables the cache
ubatch_size int 512 Physical max batch size (-ub) — the largest single memory lever after context itself
flash_attn on/off/auto auto Flash Attention use (-fa)
cache_type_k / cache_type_v string f16 KV cache quantization (-ctk/-ctv); also f32, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
tensor_split list[float] None Proportional offload split across GPUs (-ts). Preflight recommends an uneven split on heterogeneous cards

llama-server caches prompts in RAM by default (--cache-prompt, always on, 8 GiB idle-slot cache) — exact-prefix reuse works out of the box for ordinary append-only chat. There is no persistent on-disk prompt cache; caching is in-memory only and doesn't survive a process restart, unlike modelship's disk cache for other loaders.

MLA models (DeepSeek, MiniCPM3)

llama.cpp caches the compressed latent for MLA architectures only when the GGUF ships split attn_k_b/attn_v_b projections. Conversions predating that support carry a fused attn_kv_b and fall back to full per-head K/V — for DeepSeek-V2-Lite that is 276,480 B/token instead of 31,104, so the same VRAM buys ~8.9x less context. llama fit-params builds the real KV cache either way, so n_ctx is sized correctly for whichever layout the file has — but the only way to recover the missing context is to use a newer conversion. Nothing in the model's metadata reveals this, so check the tensor names if a GGUF sizes far smaller than expected.

Minimum config — preflight fills in n_ctx, n_gpu_layers, threads:

models:
  - name: "qwen-llama-server"
    model: "lmstudio-community/Qwen3-8B-GGUF:*Q4_K_M.gguf"
    usecase: "generate"
    loader: "llama_server"
    num_gpus: 1

Tool calling and reasoning gaps vs. the OpenAI spec

Both parsers are auto-detected from the model's chat template — no modelship-level override. Gaps are per-model-family, not per-loader, so test against the specific model in use:

  • Named-function forcing is unsupported. tool_choice: {"type": "function", "function": {"name": "X"}} silently falls back to auto (llama-server logs a warning; modelship doesn't surface it as an error).
  • tool_choice: required depends on the template family. Grammar-enforced for harmony-style templates (e.g. gpt-oss); a silent no-op for hermes-style templates (e.g. Qwen3) — the model may answer in free text with no error.
  • Bare response_format: {"type": "json_object"} (no schema) is not enforced, despite llama-server's docs describing it as supported — verified against b9859 and b10200, unconfirmed on the current b10375 pin. type: json_schema requests (what modelship sends whenever a schema is given) are unaffected and correctly constrained.

response_format/json_schema can combine with reasoning in the same request; logprobs/top_logprobs are forwarded and returned.

Vision (GGUF)

Set mmproj to a multimodal projector file (local path or repo:filename) to enable image input. Requests with image_url/input_image content parts are rejected at the gateway when mmproj isn't configured.

models:
  - name: "qwen-vl-llama-server"
    model: "lmstudio-community/Qwen3-VL-8B-Instruct-GGUF:*Q4_K_M.gguf"
    usecase: "generate"
    loader: "llama_server"
    llama_server_config:
      mmproj: "lmstudio-community/Qwen3-VL-8B-Instruct-GGUF:mmproj-Qwen3-VL-8B-Instruct-F16.gguf"

Embeddings (GGUF)

models:
  - name: nomic-embed-server
    model: "nomic-ai/nomic-embed-text-v1.5-GGUF:nomic-embed-text-v1.5.Q4_K_M.gguf"
    usecase: embed
    loader: llama_server

stable-diffusion.cpp Loader

Runs stable-diffusion.cpp (via stable-diffusion-cpp-python) for GGUF-quantized single-file diffusion checkpoints (SD1.5, SDXL, SD-Turbo, all-in-one Flux) in a few GB of RAM. CPU-only everywhere except Apple Silicon, where ggml's runtime backend registry picks up Metal automatically — no config needed. Off Darwin, any num_gpus is ignored (warning logged, actor allocated num_gpus: 0); on Darwin it's honored normally. usecase defaults to image; serves /v1/images/generations, /v1/images/edits, /v1/images/variations.

Configured via stable_diffusion_cpp_config:

Field Type Default Description
sample_steps int 20 Denoising steps
cfg_scale float 7.0 Classifier-free guidance scale
sample_method string default Sampler; default lets sd.cpp pick per architecture
scheduler string default Denoiser sigma scheduler
wtype string default On-the-fly weight quantization (e.g. q4_0, q8_0, f16); default auto-detects
n_threads int -1 CPU threads; -1 uses half the cores
vae_tiling bool false Tile the VAE decode to cut peak RAM (auto-recommended by preflight on low-RAM hosts)
diffusion_model_path / clip_l_path / clip_g_path / t5xxl_path / vae_path string — Standalone component paths for split checkpoints (pre-placed local paths; single-file models are the v1 focus)
model_kwargs object {} Extra keyword arguments passed to the StableDiffusion constructor

MSHIP_LOG_LEVEL=TRACE enables verbose mode in the underlying engine. GGUF variants in an HF repo use the same :filename syntax as the llama_server loader (see Model source).

models:
  - name: sd21-turbo
    model: "gpustack/stable-diffusion-v2-1-turbo-GGUF:*Q4_1.gguf"
    usecase: image
    loader: stable_diffusion_cpp
    num_cpus: 4
    stable_diffusion_cpp_config:
      sample_steps: 4
      cfg_scale: 1.0

whispercpp Loader

Runs whisper.cpp speech-to-text in-process via pywhispercpp bindings — no subprocess, unlike llama_server. CPU-only on Linux (num_gpus ignored, forced to 0). On Apple Silicon, num_gpus > 0 (including a fraction, to share the GPU) enables Metal offload — no separate VRAM knob, just a plain on/off switch, so the model's own footprint determines whether it fits its share.

Configured via whispercpp_config:

Field Type Default Description
n_threads int pywhispercpp's own default (min(4, cores)) Compute thread count
flash_attn bool false ggml flash attention
models_dir string <cache_root>/whispercpp Only used when model: is a bare pywhispercpp built-in name

model: accepts: - A pywhispercpp built-in name (e.g. base.en, large-v3-turbo — see pywhispercpp's model list); pywhispercpp resolves/downloads these itself, modelship doesn't pre-validate. - A local path or HF repo:filename pointing at a ggml-*.bin file, resolved like every other loader. Both GGUF and safetensors are rejected with a pointer at the right form.

models:
  - name: "whisper-cpp-base"
    model: "base.en"
    usecase: "transcription"
    loader: "whispercpp"
    num_cpus: 2

See config/examples/whispercpp.yaml for every model: form and a Metal example.

sherpa_onnx Loader

Runs sherpa-onnx in-process via its Python bindings. Current scope: TTS only, kokoro family only, CPU only — never touches CUDA or CoreML, so num_gpus is ignored entirely (any value accepted at config time, forced to 0 at deploy with a warning if nonzero) and reserves no GPU capacity another deploy could use.

model: is not an HF repo or path — it's a name from a curated built-in registry (or a local directory whose basename matches one). Each name maps to a GitHub release tarball with a pinned sha256, downloaded and cached under <cache_root>/sherpa_onnx/<name>/ on first use.

Name Speakers Notes
kokoro-en-v0_19 11 English only. Includes af_bella
kokoro-multi-lang-v1_0 53 English, Mandarin, Japanese, and more. Includes af_bella and af_heart 
models:
  - name: "kokoro"
    model: "kokoro-en-v0_19"
    usecase: "tts"
    loader: "sherpa_onnx"
    num_cpus: 2

See config/examples/sherpa-onnx.yaml for the local-directory form.

Scaling a Deployment

Use num_replicas to run several identical copies; Ray Serve load-balances across them.

models:
  - name: "kokoro"
    model: "kokoro-en-v0_19"
    usecase: "tts"
    loader: "sherpa_onnx"
    num_cpus: 2
    num_replicas: 2

Autoscaling

Set autoscaling_config instead of a fixed num_replicas to let Ray Serve grow/shrink replica count with load. The two are mutually exclusive — setting both is a config error.

models:
  - name: "bursty-llm"
    model: "Qwen/Qwen3-0.6B"
    usecase: "generate"
    loader: "vllm"
    num_gpus: 0.3
    autoscaling_config:
      min_replicas: 1            # floor; 0 enables scale-to-zero (cold-start on first request)
      max_replicas: 4            # ceiling
      target_ongoing_requests: 8 # autoscaler setpoint: in-flight requests per replica (lower = scales out sooner)
      initial_replicas: 1        # seed count on first deploy, before load signal (default: min_replicas)
      upscale_delay_s: 10        # debounce before scaling out
      downscale_delay_s: 300     # debounce before scaling in (longer avoids thrashing GPU warm-up)
Field Type Description
min_replicas int Lower bound. Default 1. 0 enables scale-to-zero
max_replicas int Upper bound. Default 1. Must be ≥ min_replicas
initial_replicas int Seed count before the autoscaler has a load signal. Default: min_replicas
target_ongoing_requests float Desired in-flight requests per replica — the autoscaler's setpoint. Lower scales out sooner. Default: Ray Serve's own
upscale_delay_s float Seconds of sustained over-load before adding replicas. Default: Ray Serve's own
downscale_delay_s float Seconds of sustained under-load before removing replicas. Default: Ray Serve's own. Raise it to avoid thrashing on models with slow GPU warm-up

Autoscaling bounds are changed in place on mship deploy --reconcile (excluded from the config fingerprint) — tuning them doesn't tear down and rebuild the deployment.

Environment Variables

Variable Description Default
HF_TOKEN HuggingFace access token —
MSHIP_CACHE_DIR Model cache directory (HuggingFace, vLLM, sherpa_onnx, etc.) /.cache
MSHIP_STATE_STORE State-store connection URI for the effective config, deploy coordinator + /v1/responses conversations (see State store) memory://
MSHIP_GATEWAY_NAME Name for the API gateway app modelship
MSHIP_GATEWAY_REPLICAS Number of API gateway replicas 1
MSHIP_OPENAI_API_PORT Port for the OpenAI-compatible API 8000
MSHIP_MAX_REQUEST_BODY_BYTES Maximum allowed request body size in bytes 52428800 (50 MB)
MSHIP_LOG_TARGET Log target: console or syslog URI console
OTEL_EXPORTER_OTLP_ENDPOINT OpenTelemetry OTLP endpoint for log export. Requires uv sync --extra otel —
CUDA_DEVICE_ORDER GPU enumeration order; set to PCI_BUS_ID for deterministic ordering in multi-GPU systems PCI_BUS_ID
MSHIP_RAY_DASHBOARD Ray dashboard bind host, own-head only. Dashboard always starts; this sets where it binds — 0.0.0.0 exposes it beyond the container (the ShadowRay/CVE-2023-48022 exposure vector), so only do this on a trusted network 127.0.0.1
MSHIP_RAY_AUTH Ray cluster authentication, own-head only. token requires a bearer token (generated by Ray at ~/.ray/auth_token) for the dashboard and all cluster-internal RPC. The OpenAI API and Prometheus metrics are never gated either way none
MSHIP_RAY_PORT Ray GCS server port, own-head only. Pinned so a joiner's --address has a stable target; not 6379 since that collides with the recommended same-host Redis state store under --network=host 6380
MSHIP_RAY_DASHBOARD_PORT Ray dashboard port, own-head only. Needed only when running multiple modelship heads on one host under --network=host 8265
MSHIP_ADDRESS Join an existing Ray cluster as an additional node, given the head's GCS address as host:port —
MSHIP_RAY_AUTH_TOKEN Cluster auth token for joining a head running --ray-auth=token. Only meaningful with MSHIP_ADDRESS —
MSHIP_NODE_NUM_CPUS Override: CPUs this node reserves auto-detect
MSHIP_NODE_NUM_GPUS Override: GPUs this node reserves. Refused at startup if it exceeds what the container can actually see auto-detect
MSHIP_NODE_MEMORY Override: this node's total memory budget, e.g. 8Gi. Split into Ray's object_store_memory (30%) and schedulable memory (70%), matching Ray's own auto-detect proportion. Set when co-locating multiple modelship containers on one host without per-container cgroup memory limits auto-detect
MSHIP_NODE_CAPABILITIES Override: this node's advertised mship_<loader> capability resources, as JSON (e.g. {"mship_vllm": 1}) — replaces the find_spec()/binary probe wholesale. See Architecture: Capability-aware scheduling auto-probed
RAY_OBJECT_STORE_SHM_SIZE Shared memory for Ray object store 8g
VLLM_USE_V1 Use vLLM v1 API 1
ONNX_PROVIDER ONNX Runtime execution provider CUDAExecutionProvider
NVIDIA_CUDA_VERSION CUDA toolkit version 12.8.1

State store (MSHIP_STATE_STORE)

Three pieces of state share one pluggable store: this gateway's effective config (its desired model set, replayed by --reconcile with no --config to self-heal after cluster loss), the deploy coordinator's routing registry (which gateway owns which model + the expected set), and /v1/responses conversations (see Stateful responses). A single connection URI picks the backend and carries its connection:

URI Backend Durability
memory:// (default) dict shared cluster-wide by a detached Ray actor survives a deploy re-run, coordinator restart, gateway-replica restart — not cluster death
redis://[:pw@]host:6379/0 (rediss:// = TLS) one JSON value per key in Redis survives head/coordinator death and cluster loss; password parsed from the URL by redis.from_url

memory:// is cluster-scoped, not process-local — every gateway replica and model actor shares one detached Ray actor, so it's correct at any replica count. Sized for small-traffic single-node deployments: every operation is a Ray RPC through that one actor, and large values spill to the object store.

The Helm chart always sets redis://… in Kubernetes; the same Redis also backs Ray GCS fault tolerance (chart's Head-node HA section) and is what lets the gateway self-heal routing after a head restart instead of needing a redeploy.

A file:// backend existed before v0.7.0 and was removed: a poor fit for per-turn conversation snapshots (one JSON file each, no native TTL, last-writer-wins across replicas). Migrate --state-store file://…/MSHIP_STATE_DIR to redis://, or drop to memory:// if you don't need to survive cluster loss.

Stateful responses

/v1/responses keeps conversations server-side, so a follow-up turn sends only the new input instead of replaying the whole history:

curl localhost:8000/modelship/v1/responses -H 'Content-Type: application/json' \
  -d '{"model": "qwen", "input": "my name is Alex"}'
# response id comes back in `id`

curl localhost:8000/modelship/v1/responses -H 'Content-Type: application/json' \
  -d '{"model": "qwen", "input": "what is my name?", "previous_response_id": "resp_…"}'
Route Purpose
POST /v1/responses store (default true) persists the response; previous_response_id continues from one; background: true queues instead of blocking
GET /v1/responses/{id} Fetch a stored response — for a background response, poll until status is terminal
DELETE /v1/responses/{id} Drop a stored response; on an in-flight background response this implies cancel
POST /v1/responses/{id}/cancel Cancel an in-flight background response
GET /v1/responses/{id}/input_items The input a stored response was produced from

Send "store": false to opt out of storage — no id to continue from. An unknown, expired, or already-deleted previous_response_id is 404; an unreachable state store is 503 (never a silent stateless fallback).

"background": true returns status: "queued" immediately instead of blocking; poll GET until status reaches a terminal value (completed/incomplete/failed/cancelled). Requires store. Combined with "stream": true, the initial call instead streams live, and a disconnected client can resume with GET /v1/responses/{id}?stream=true&starting_after=<last sequence_number seen> — the replay buffer is short-lived (MSHIP_RESPONSES_STREAM_BUFFER_TTL_S, default 600s). Use redis:// for production background use, since a background response must outlive the request that created it.

Conversations are scoped to the caller's identity, so one caller can never read or continue another's. With no auth configured, every caller shares the single unscoped identity — one conversation pool. Set MSHIP_API_KEYS or MSHIP_TRUSTED_IDENTITY_HEADER (see Trusted identity header) before serving more than one user.

Variable Description Default
MSHIP_RESPONSES_TTL_S How long a stored conversation lives. Each turn rewrites a fresh TTL, so an active conversation stays alive while superseded snapshots age out. 0 disables expiry 2592000 (30 days)
MSHIP_STATE_SWEEP_INTERVAL_S How often the memory:// store reclaims expired keys. 0 disables sweeping 300

Sizing: a snapshot holds the whole conversation as of that turn, so an n-turn conversation costs O(n²) storage total — the price of continuing in a single read. On the default memory:// that all sits in one Ray actor's RAM for up to the TTL, so for sustained multi-user traffic lower MSHIP_RESPONSES_TTL_S or move to redis://.