Skip to content

Production Readiness

What the cluster is already proven to do, and what is still on the hardening list.

Verified cluster behaviour

Every row below is asserted end-to-end against a live cluster with real models over real HTTP — no mocks, no stubs. Run them with pytest tests/ -m integration, or a single area via its own marker (e.g. -m blue_green).

Capability What the cluster guarantees Proven by
Zero-downtime model cutover Change a live model's config and the swap is atomic: traffic is hammered continuously through the cutover with zero failed requests, and the old deployment is fully torn down rather than left as a zombie test_blue_green_integration.py
Rescale without a reload Change a live model's num_replicas or autoscaling_config and its running replicas keep serving, with zero failed requests while replicas are added or stopped; a cancelled scale-up goes back to the committed count on the same replicas. A scale-down runs before the same deploy's new models, which then use the room it frees, and sending the config again puts back a replica count that no longer matches it test_rescale_integration.py
Load-driven autoscaling Replicas scale out under sustained concurrency (bounded by max_replicas) and scale back to min_replicas once load stops — no errors on the way up or down test_autoscaling_integration.py
Multi-replica gateway convergence With multiple gateway replicas, a deploy or removal converges on every replica — a removed model 404s on all of them instead of routing into a torn-down deployment test_gateway_ha_integration.py
Gateway autoscaling Sustained load past its target scales the gateway out with no failed requests, and the new replica serves the deployed models test_gateway_autoscaling_integration.py
Engine crash recovery SIGKILL the vLLM engine-core subprocess and the replica respawns and serves again on its own — no operator action, no cluster restart test_crash_recovery_integration.py
Authenticated node join / clean leave A node joining with a missing or wrong token is rejected; with the right token it joins, and SIGTERM removes only that node — the head and its other nodes are untouched test_cluster_join.py
Fractional GPU multi-tenancy Two different engines (vLLM at num_gpus: 0.6, llama-server at 0.3) share one physical GPU and serve concurrent traffic without starving each other test_fractional_gpu_sharing.py
Cross-process disconnect cancellation A dropped client socket cancels in-flight inference across the Ray process boundary and frees the GPU; other requests on the same replica are unaffected test_disconnect_integration.py
Per-identity prefix-cache isolation An identical prompt from a different identity never hits another identity's KV-cache entry — the cross-tenant timing side channel is closed by construction test_vllm_integration.py
Resumable background runs A client can disconnect mid-stream; the run continues server-side and the client reconnects with starting_after to replay from where it left off through to completion test_responses_integration.py
Durable conversation state Stored /v1/responses chains survive deletion of a parent turn, fan out into independent concurrent branches from one parent, and a cancel stays cancelled test_responses_integration.py
Server-side MCP tool loop Real MCP server, discovered and called by the model — including the require_approval round trip, background mode, mid-flight cancel that sticks, and an unreachable server surfacing as a failed response rather than a 5xx test_mcp_integration.py
Real in-replica concurrency llama-server parallel slots serve concurrent requests genuinely in parallel — N at once complete in well under N× the single-request time, not serialized behind a lock test_llama_server_integration.py
Hot reconcile of the model set The entire integration suite swaps the deployed model set repeatedly via --reconcile against one long-lived gateway process that is never restarted conftest.py
Cross-loader pipelines TTS audio generated by one deployment is fed straight back in as STT input to another, through the same gateway test_audio_integration.py

Alongside these, replicas sharing a node load their models one at a time (a per-node deploy lease), so their memory spikes never overlap, and /v1/responses passes the independent Open Responses compliance suite 17/17 — on both the vllm and llama_server loaders, which agreed on every test when re-audited 2026-09-10.

Hardening roadmap

Remaining work, organized by severity and area. The Kubernetes/KubeRay path is the current focus — a Helm chart ships in helm/modelship, with GPU-aware probes and gateway-level rate limiting next.

Critical (Must Have Before Production)

Security

  • [x] API authentication layer — delegated to the layer in front of the gateway (nginx, Kong, LiteLLM); modelship takes the caller identity it resolved from MSHIP_TRUSTED_IDENTITY_HEADER
  • [ ] Rate limiting — per-user/IP/model throttling to prevent GPU resource monopolization
  • [x] Input size limits — coarse payload size limit at the gateway (MSHIP_MAX_REQUEST_BODY_BYTES, default 50 MB)
  • [ ] Per-model context validation — no loader validates a request's prompt length against the model's context before inference
  • [ ] Lock down CORS — replace wildcard * origins with environment-specific allowed origins

Health & Readiness

  • [x] Detailed readiness probe — /readyz returns 200 only when every expected model is routed by the gateway; 503 with loaded/pending lists while loading. /health stays as a cheap liveness endpoint. Per-model load times and total time-to-ready are exposed via /readyz for observability.
  • [ ] Model-specific health checks — per-model liveness status (vLLM engine, Ray actor state)
  • [ ] GPU memory checks — detect and report memory pressure before OOM

Testing

  • [x] API endpoint tests — HTTP-level tests for all /v1/ endpoints (via tests/test_*_integration.py)
  • [x] Integration tests — actual model loading and inference (using Qwen-0.5B/0.6B)
  • [x] Streaming tests — SSE streaming correctness and error handling
  • [ ] Error recovery tests — simulate failures and verify behavior

High Priority (Should Have)

Deployment & Infrastructure

  • [x] Kubernetes manifests — KubeRay RayCluster, a deploy Job, gateway Service, cache PVC, secrets, optional PodMonitor (via the Helm chart in helm/modelship), with resource requests/limits, GPU scheduling, node affinity, and tolerations per worker group
  • [x] Helm chart — parameterized deployment in helm/modelship (see its README)
  • [x] Simpler non-K8s deployment — reframed from "Docker Compose", which orchestrates a single host and can't form a cluster across VMs. Supported path is mship start / mship join in plain docker run containers (see docs/multi-node-docker.md): a few VMs, no orchestrator, joined into one Ray cluster via mship join --gcs-address with MSHIP_RAY_AUTH_TOKEN. Compose remains a possible single-host wrapper around single-container mode, not planned work.
  • [x] Liveness/readiness probes in container spec — KubeRay's health checks (/api/healthz) gate each Ray pod's readiness on its raylet (and GCS on the head); /readyz returns 503 until all models load, suitable for an external LB/Ingress health check

Alerting & Observability

  • [x] Prometheus alerting rules — error rate thresholds, latency P99 breaches, Serve app health, GPU memory pressure (see docs/prometheus-alerts.yml)
  • [ ] SLO/SLI definitions — define target availability and latency for each endpoint type
  • [x] Structured logging (JSON) — MSHIP_LOG_FORMAT=json for log aggregation (ELK/Loki/Splunk)
  • [x] Request-ID correlation — trace a request from gateway through Ray actor boundaries via contextvars
  • [x] Log level configuration — MSHIP_LOG_LEVEL controls app logs; DEBUG and TRACE enable library debug logs
  • [x] Syslog support — --log-target syslog://host:port ships logs to a remote syslog server (UDP or TCP)
  • [x] OpenTelemetry log export — --otel-endpoint ships logs (and enables Ray traces) via OTLP to any OTel collector

Resilience

  • [x] Head-node HA (GCS fault tolerance) — the chart backs Ray's GCS with Redis (gcsFaultToleranceOptions, redis.address required): a restarted head pod recovers cluster state, workers + model actors survive. The same Redis backs the modelship state store (redis://), so each gateway's committed version survives head death, and the gateway coordinator (max_restarts=-1) recomputes routing from Serve's state on recovery — no redeploy
  • [ ] Ray actor restart policies — auto-restart crashed model actors
  • [ ] Circuit breaker — stop routing to a failing model after N consecutive errors
  • [ ] Backpressure / queue depth limits — reject requests when queue is saturated instead of unbounded queuing
  • [ ] Graceful shutdown timeout — add timeout wrapper around serve.shutdown() to prevent hanging
  • [ ] GPU OOM recovery — detect and recover from GPU memory exhaustion

Update Strategy

  • [ ] Rolling update support — configure Ray Serve's built-in rolling updates for zero-downtime deploys
  • [x] Per-model autoscaling — autoscaling_config (min/max replicas, target ongoing requests, up/downscale delays; scale-to-zero supported) scales replica count with load instead of a fixed num_replicas
  • [x] Gateway HA and autoscaling — every gateway autoscales between --gateway-min-replicas and --gateway-max-replicas on its ongoing requests per replica; a minimum of 2 or more keeps it HA. Every replica copies its routing table from the gateway coordinator, long-polling for changes, and a Serve proxy on the head and on every worker hosting a replica, each in the gateway Service only while it answers, lets the Service survive single-pod loss
  • [x] Self-heal after cluster loss — each successful deploy commits this gateway's model set to the configured state store (MSHIP_STATE_STORE: redis://, which the chart always sets; the memory:// default is cluster-scoped but dies with the cluster). With Redis a restarted head brings its models back by itself: the ones in its --config, or without one, the committed set; after a full cluster loss mship deploy --reconcile (no --config, run via helm upgrade) redeploys the committed set
  • [x] Model hot-reload — allow models.yaml changes without full server restart (via mship deploy --reconcile)
  • [x] Changelog — track breaking changes between versions
  • [x] Migration guide — document config format changes between versions

Medium Priority (Nice to Have)

CI/CD Hardening

  • [ ] Security scanning — Trivy for Docker images, dependency vulnerability checks
  • [ ] SBOM generation — Software Bill of Materials for supply chain visibility
  • [ ] Multi-arch builds — ARM64 support alongside AMD64
  • [ ] Performance benchmarks in CI — detect throughput/latency regressions

Operations

  • [ ] Secrets management integration — document Vault / K8s Secrets / sealed-secrets usage for HF_TOKEN, the Ray auth token and the Redis password
  • [ ] Troubleshooting runbook — common failure modes and resolution steps for on-call
  • [ ] Capacity planning guide — estimate concurrent users per GPU setup per model mix
  • [ ] GPU memory budgeting guide — model co-location recommendations to avoid fragmentation
  • [x] Multi-node Ray cluster setup docs — head + worker topology, multi-node ingress, cache PVC access modes, and self-heal documented in the Helm chart README (helm/modelship/README.md)
  • [ ] Request audit trail — persistent log of requests for compliance/debugging

Documentation

  • [ ] OpenAPI/Swagger spec — formal API reference for consumers
  • [ ] Performance tuning guide — vLLM engine kwargs, batch sizes, KV cache sizing
  • [ ] Blue-green / canary deployment patterns — documented strategies for safe rollouts
  • [ ] Model pre-warming — mechanism to pre-download and cache models before deploy, reducing cold start from minutes to seconds

Current Scorecard

Area Current Target
Architecture & Design 9/10 9/10
Monitoring (metrics) 9/10 9/10
Monitoring (alerting + logs) 9/10 9/10
Security 4/10 8/10
Resilience 8/10 8/10
Testing 8/10 9/10
DevOps Experience 8/10 8/10
Update/Deploy Strategy 7/10 7/10