Production Readiness Plan¶

Future development priorities for making Modelship production-ready, organized by severity and area.

Critical (Must Have Before Production)¶

[x] API authentication layer — API key auth at the gateway level via MSHIP_API_KEYS env var; OpenAI-compatible Authorization: Bearer <key> header
[ ] Rate limiting — per-user/IP/model throttling to prevent GPU resource monopolization
[x] Input size limits — coarse payload size limit at the gateway (MSHIP_MAX_REQUEST_BODY_BYTES, default 50 MB); per-model max_context_length validation in every loader before inference
[ ] Lock down CORS — replace wildcard * origins with environment-specific allowed origins
[ ] Plugin sandboxing — plugins run with full server privileges; add signature verification or sandboxing

[x] Detailed readiness probe — /readyz returns 200 only when every expected model is registered with the gateway; 503 with loaded/pending lists while loading. /health stays as a cheap liveness endpoint. Per-model load times and total time-to-ready are exposed via /readyz for observability.
[ ] Model-specific health checks — per-model liveness status (vLLM engine, Ray actor state)
[ ] GPU memory checks — detect and report memory pressure before OOM

[x] API endpoint tests — HTTP-level tests for all /v1/ endpoints (via tests/test_integration.py)
[x] Integration tests — actual model loading and inference (using Qwen-0.5B/0.6B)
[x] Streaming tests — SSE streaming correctness and error handling
[x] Plugin loading tests — verify plugin lifecycle
[ ] Error recovery tests — simulate failures and verify behavior

[x] Kubernetes manifests — KubeRay RayCluster + RayJob, gateway Service, models ConfigMap, cache PVC, secrets, optional PodMonitor (via the Helm chart in helm/modelship), with resource requests/limits, GPU scheduling, node affinity, and tolerations per worker group
[x] Helm chart — parameterized deployment in helm/modelship (see its README)
[x] Simpler non-K8s deployment — reframed from "Docker Compose": Compose orchestrates a single host and can't form a cluster across VMs, so it was never going to satisfy this literally. The supported path is own-head/join docker run (see docs/multi-node-docker.md) — a few VMs, no orchestrator, joined into one Ray cluster via --address/--token. Compose remains a possible single-host convenience wrapper around the existing single-container mode, not planned work.
[x] Liveness/readiness probes in container spec — KubeRay gates each Ray pod on a Serve proxy /-/healthz check (via the named serve port); /readyz returns 503 until all models load, suitable for an external LB/Ingress health check

[x] Prometheus alerting rules — error rate thresholds, latency P99 breaches, model load failures, GPU memory pressure (see docs/prometheus-alerts.yml)
[ ] SLO/SLI definitions — define target availability and latency for each endpoint type
[x] Structured logging (JSON) — MSHIP_LOG_FORMAT=json for log aggregation (ELK/Loki/Splunk)
[x] Request-ID correlation — trace a request from gateway through Ray actor boundaries via contextvars
[x] Log level configuration — MSHIP_LOG_LEVEL controls app logs; TRACE enables library debug logs
[x] Syslog support — --log-target syslog://host:port ships logs to a remote syslog server (UDP or TCP)
[x] OpenTelemetry log export — --otel-endpoint ships logs (and enables Ray traces) via OTLP to any OTel collector

[x] Head-node HA (GCS fault tolerance) — the chart backs Ray's GCS with Redis (gcsFaultToleranceOptions, redis.address required), so a restarted head pod recovers cluster state and workers + model actors survive instead of the whole cluster going down. The same Redis backs the modelship state store (redis://), so the deploy coordinator's routing registry survives head/coordinator death and the gateway self-heals its routing on recovery (the coordinator runs max_restarts=-1)
[ ] Ray actor restart policies — auto-restart crashed model actors
[ ] Circuit breaker — stop routing to a failing model after N consecutive errors
[ ] Backpressure / queue depth limits — reject requests when queue is saturated instead of unbounded queuing
[ ] Graceful shutdown timeout — add timeout wrapper around serve.shutdown() to prevent hanging
[ ] GPU OOM recovery — detect and recover from GPU memory exhaustion

[ ] Rolling update support — configure Ray Serve's built-in rolling updates for zero-downtime deploys
[x] Per-model autoscaling — autoscaling_config (min/max replicas, target ongoing requests, up/downscale delays; scale-to-zero supported) scales replica count with load instead of a fixed num_replicas
[x] Gateway HA — MSHIP_GATEWAY_REPLICAS > 1 runs multiple gateway replicas; routing tables stay consistent via the deploy coordinator's watch loop, and a Serve proxy on every node lets the gateway Service survive single-pod loss
[x] Self-heal after cluster loss — each deploy persists this gateway's effective config + routing registry to the configured state store (MSHIP_STATE_STORE: redis://, which the chart always sets; the memory:// default is cluster-scoped but dies with the cluster). With Redis the gateway self-heals automatically on a head restart; after a full cluster loss mship_deploy --reconcile (no --config, run via helm upgrade) replays the recorded set
[x] Model hot-reload — allow models.yaml changes without full server restart (via mship_deploy --reconcile)
[x] Changelog — track breaking changes between versions
[x] Migration guide — document config format changes between versions

[ ] Security scanning — Trivy for Docker images, dependency vulnerability checks
[ ] SBOM generation — Software Bill of Materials for supply chain visibility
[ ] Multi-arch builds — ARM64 support alongside AMD64
[ ] Performance benchmarks in CI — detect throughput/latency regressions

[ ] Secrets management integration — document Vault / K8s Secrets / sealed-secrets usage for HF_TOKEN and future API keys
[ ] Troubleshooting runbook — common failure modes and resolution steps for on-call
[ ] Capacity planning guide — estimate concurrent users per GPU setup per model mix
[ ] GPU memory budgeting guide — model co-location recommendations to avoid fragmentation
[x] Multi-node Ray cluster setup docs — head + worker topology, multi-node ingress, cache PVC access modes, and self-heal documented in the Helm chart README (helm/modelship/README.md)
[ ] Request audit trail — persistent log of requests for compliance/debugging

[ ] OpenAPI/Swagger spec — formal API reference for consumers
[ ] Performance tuning guide — vLLM engine kwargs, batch sizes, KV cache sizing
[ ] Blue-green / canary deployment patterns — documented strategies for safe rollouts
[ ] Model pre-warming — mechanism to pre-download and cache models before deploy, reducing cold start from minutes to seconds