Monitoring & Logging

Modelship exposes Prometheus metrics through a single port via Ray's metrics agent. When enabled, all metrics — Ray cluster, Ray Serve, vLLM engine, and custom Modelship metrics — are available on one scrape endpoint.

Logging

Modelship uses a centralized logging system with structured output and request correlation. All application logs go through the modelship.* logger hierarchy, separate from library logs (Ray, vLLM, etc.).

Configuration

Env Var	Default	Description
MSHIP_LOG_LEVEL	INFO	App log level. Set to TRACE for request/response payloads, DEBUG for detailed diagnostics. Each level sets library logs to the next level up (e.g. DEBUG app → INFO libs).
MSHIP_LOG_FORMAT	text	text for human-readable output, json for structured JSON lines (for log aggregation with ELK/Loki/Splunk).
MSHIP_LOG_TARGET	console	Log target. console writes to stderr; syslog URIs ship logs to a remote syslog server (see below).
OTEL_EXPORTER_OTLP_ENDPOINT	—	When set, logs are also exported to an OpenTelemetry collector via OTLP (see below).

Log Levels

Each level sets library logs (Ray, vLLM, transformers) to the next level up:

Level	App logs (modelship.*)	Library logs
TRACE	Request/response payloads (audio bytes, transcription text, chat messages, etc.)	DEBUG
DEBUG	Detailed diagnostics, per-chunk details	INFO
INFO (default)	Startup, deployment, request summaries	WARNING
WARNING	Warnings only	ERROR

Request Correlation

Every API request is assigned a unique request ID that appears in all log lines for that request — both in the API gateway process and in the model deployment actor. This allows tracing a request end-to-end across Ray actor boundaries.

Text format example:

[2025-04-09 14:06:54] INFO     modelship.api [a1b2c3d4] | chat_completion model=llama messages=3 stream=True max_tokens=512

JSON format example:

{"timestamp": "2025-04-09T14:06:54", "level": "INFO", "logger": "modelship.api", "message": "chat_completion model=llama messages=3 stream=True max_tokens=512", "request_id": "a1b2c3d4", "pid": 12345}

Logger Names

Logger	Scope
modelship.startup	Application initialization and shutdown
modelship.api	API gateway endpoints
modelship.api.auth	Authentication middleware
modelship.infer	Base inference layer
modelship.infer.deployment	Ray Serve model deployment actor
modelship.infer.vllm	vLLM inference backend
modelship.infer.diffusers	Diffusers inference backend
modelship.infer.diffusers.image	Diffusers image generation
modelship.infer.custom	Custom/plugin inference backend
modelship.plugin.<name>	Individual plugins (kokoroonnx, orpheus, whispercpp)

Syslog

Ship logs to a remote syslog server instead of stderr. Useful for centralized logging on bare-metal or Unraid setups without extra infrastructure.

# UDP (default)
python mship_deploy.py --log-target syslog://192.168.1.50:514

# TCP (reliable delivery)
python mship_deploy.py --log-target syslog+tcp://192.168.1.50:514

# Via environment variable
MSHIP_LOG_TARGET=syslog://192.168.1.50:514 python mship_deploy.py

Supported URI formats:

URI	Protocol	Notes
syslog://host:port	UDP	Default, fire-and-forget
syslog+tcp://host:port	TCP	Reliable delivery
syslog://host	UDP	Port defaults to 514

The syslog target replaces the console handler — logs go to the syslog server only. The --log-format setting still applies (text or JSON).

OpenTelemetry

Export logs to an OpenTelemetry collector via OTLP. Unlike syslog, OTel is additive — logs still go to the console (or syslog) handler, and are also shipped to the collector.

First, install the optional dependencies:

uv sync --extra otel

Then configure the endpoint:

# Via CLI
python mship_deploy.py --otel-endpoint http://collector:4317

# Via environment variable
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317 python mship_deploy.py

When OTel is enabled: - Logs are exported via BatchLogRecordProcessor with OTLPLogExporter (gRPC) - The service name is set to modelship - RAY_TRACING_ENABLED=1 is set automatically so Ray workers also export traces - HTTPS endpoints are detected from the URI scheme; all others use insecure connections

If the opentelemetry-sdk and opentelemetry-exporter-otlp packages are not installed, a warning is logged and OTel is skipped.

Architecture

Prometheus  ──scrape──>  Ray Metrics Agent (:8079)
                              |
                              |-- ray_node_*          Ray cluster: GPU, CPU, memory
                              |-- ray_serve_*         Ray Serve: HTTP requests, latency, replicas
                              |-- ray_vllm_*          vLLM engine: KV cache, TTFT, tokens, queue
                              |-- ray_modelship_*         Custom: per-model latency, errors, load time

Note: All metrics are prefixed with ray_ by Ray's metrics agent. vLLM metric names are also sanitized (: → _), so e.g. the vLLM-native vllm:kv_cache_usage_perc becomes ray_vllm_kv_cache_usage_perc.

Enabling Metrics

Metrics are enabled by default. Set MSHIP_METRICS=false to disable:

docker run --rm --shm-size=8g --gpus all \
  -e HF_TOKEN=your_token \
  -e MSHIP_METRICS=true \
  -e MSHIP_RAY_DASHBOARD=0.0.0.0 \
  -v ./models.yaml:/modelship/config/models.yaml \
  -v ./models-cache:/.cache \
  -p 8000:8000 -p 8079:8079 -p 8265:8265 \
  ghcr.io/alez007/modelship:latest-cuda

The dashboard always starts and binds 127.0.0.1 by default — MSHIP_RAY_DASHBOARD=0.0.0.0 above is what makes the published 8265 port actually reachable. Only do this on a trusted/private network (see troubleshooting.md). Pair it with --ray-auth=token to require a bearer token for the now-reachable dashboard — retrieve it with docker exec <container> cat /home/modelship/.ray/auth_token.

Env Var	Default	Description
MSHIP_METRICS	true	Master toggle. Enables all metrics and the Ray metrics export port.
RAY_METRICS_EXPORT_PORT	8079	Port for the Ray metrics agent (only active when MSHIP_METRICS=true).

Set MSHIP_METRICS=false to disable all metrics collection. When disabled, port 8079 is not exposed and there is zero overhead.

Connecting to Prometheus

Add Modelship as a scrape target in your prometheus.yml:

scrape_configs:
  - job_name: modelship
    scrape_interval: 15s
    static_configs:
      - targets: ["<modelship-host>:8079"]

For multi-node Ray clusters, use Ray's auto-generated service discovery file instead of static targets:

scrape_configs:
  - job_name: modelship
    file_sd_configs:
      - files: ["/tmp/ray/prom_metrics_service_discovery.json"]

Connecting to Grafana

A pre-built Grafana dashboard is included at docs/grafana-dashboard.json.

To import it:

1. Open Grafana and go to Dashboards > Import
2. Upload grafana-dashboard.json or paste its contents
3. Select your Prometheus datasource when prompted

At the top the dashboard has Gateway, Model, and Node dropdowns. The Gateway dropdown filters every ray_modelship_* panel to a single gateway (or All); the vLLM/Ray-Serve/GPU rows stay cluster-wide since those metrics aren't per-gateway (see The gateway dimension).

The dashboard has 9 rows:

Row	What it shows	Metric sources
Overview	Request rate, error rate, in-flight requests, models loaded, client disconnects, auth failures	ray_modelship_*
Latency	Gateway P50/P95/P99, per-model latency, per-usecase latency (generate, TTS, image, STT, embed)	ray_modelship_*
vLLM Engine	KV cache usage, TTFT, inter-token latency, token throughput, queue depth, preemptions, prefix cache hit rate	ray_vllm_*
GPU & System	GPU utilization, GPU memory, CPU, system memory (cluster aggregate)	ray_node_*
Ray Serve	Health check latency, request count, deployment processing latency, HTTP request latency	ray_serve_*
Operational	Model load time, load failures, resource cleanup errors, streaming chunks/s	ray_modelship_*
Alerts	Error rate %, KV cache usage, queue depth, TTFT P99, client disconnects, preemptions, GPU memory	ray_modelship_*, ray_vllm_*, ray_node_*
Cluster / HA	Deploy lock, reservation outcomes, operator force-releases, coordinator-vs-replica routing generation, gateway watch errors, state-store latency/errors, deploy duration & model changes	ray_modelship_*
Per-Node Resources	GPU utilization/memory, CPU, system memory broken out by node (filtered by the Node dropdown)	ray_node_*

Deploying via the Helm chart: set grafanaDashboard.enabled=true to ship this dashboard as a ConfigMap the Grafana sidecar auto-imports (and prometheusRule.enabled=true for the alert rules below) — no manual import needed. See the chart README.

Alerting

A standalone Prometheus alerting rules file is included at docs/prometheus-alerts.yml. The Grafana dashboard also has a dedicated Alerts row with threshold lines on the key panels.

Importing Alert Rules

Add the rules file to your Prometheus config:

rule_files:
  - /path/to/prometheus-alerts.yml

Then reload Prometheus (kill -HUP <pid> or POST /-/reload if --web.enable-lifecycle is set).

Alert Reference

Critical (page-worthy)

Alert	Condition	For	Description
ModelshipHighErrorRate	Error rate > 5% of traffic	5m	Significant portion of requests are failing
ModelshipNoModelsLoaded	models_loaded == 0	2m	Server is running but cannot serve requests
ModelshipModelLoadFailure	Any increase in model_load_failures_total	0m	A model failed to initialize
ModelshipKVCacheExhausted	KV cache usage > 95%	5m	Requests will queue or be preempted
ModelshipStateStoreErrors	Any state-store op errors	5m	Durable HA state failing — self-heal at risk
ModelshipDeployJobFailed	kube_job_status_failed > 0	0m	Deploy RayJob failed; last upgrade may not be applied (needs kube-state-metrics)

Warning (investigate)

Alert	Condition	For	Description
ModelshipHighP99Latency	Gateway P99 > 30s	5m	End-to-end latency is very high
ModelshipHighQueueDepth	Waiting requests > 10	5m	vLLM engine is falling behind
ModelshipPreemptions	Preemption rate > 0	5m	GPU memory pressure causing request eviction
ModelshipClientDisconnects	Disconnect rate > 1/min	5m	Clients timing out or dropping connections
ModelshipGPUMemoryPressure	Available GPU memory < 1 GB	5m	GPU is nearly out of memory
ModelshipHighTTFT	TTFT P99 > 5s	5m	Users waiting too long for first token
ModelshipDeployLockStuck	deploy_lock_held == 1 for 10m	0m	A deploy is hung holding the cluster-wide deploy lock
ModelshipOperatorForceReleased	Any operator force-release	0m	A deploy operator died ungracefully; lock reclaimed
ModelshipGatewayRoutingDivergence	Replica generation < coordinator for 10m	10m	A gateway replica is routing from a stale table
ModelshipRayWorkerNotReady	Ray worker pod not ready	5m	Cluster capacity degraded (needs kube-state-metrics)

Tuning Thresholds

All thresholds are starting points. Adjust based on your deployment:

• Error rate: 5% is aggressive — if you run small models that occasionally OOM, raise to 10%.
• P99 latency: 30s works for chat completions with long outputs. For embeddings or TTS, consider lowering to 5-10s by adding per-endpoint rules.
• Queue depth: 10 assumes a single vLLM instance. Scale proportionally with replicas.
• KV cache: 95% is the danger zone. If you use prefix caching heavily, 90% may be more appropriate.
• TTFT: 5s is generous. For interactive chat, consider 2-3s.
• GPU memory: 1 GB threshold assumes you're not running anything else on the GPU. Raise if you have shared workloads.

Health and readiness

Two endpoints are always available regardless of the metrics toggle.

/health — cheap liveness probe. Returns 200 as soon as the gateway is up (before model deployments finish):

curl http://localhost:8000/health
# {"status": "ok", "uptime_s": 12.3}

/readyz — readiness + timing. Returns 200 when every expected model has a registered deployment; 503 with the same JSON body while any model is still pending. Bodies carry full state so a single poll tells you what's loaded, what's outstanding, and how long each model took to come up:

curl http://localhost:8000/readyz
# 200 when ready:
# {
#   "status": "ok",
#   "ready": true,
#   "uptime_s": 420.5,
#   "time_to_ready_s": 215.3,
#   "models_loaded":   ["kokoro", "llm", "nomic-embed", "whisper"],
#   "models_expected": ["llm", "kokoro", "whisper", "nomic-embed"],
#   "models_pending":  [],
#   "model_load_times_s": {"llm": 50.2, "kokoro": 8.1, "whisper": 120.5, "nomic-embed": 36.5}
# }

Per-model timings are gateway-measured: the gap between one model registering and the next (models deploy sequentially in mship_deploy.py), so the first model's entry includes any framework-level setup time preceding it.

Use /health for Kubernetes liveness probes and /readyz for readiness probes — /readyz returning 503 prevents a service from flipping traffic onto the pod before models are loaded.

Modelship Metrics Reference

Custom metrics are exported through Ray's metrics agent with a ray_ prefix. Per-model/per-gateway metrics use ray.serve.metrics (they're emitted inside Serve replicas); the HA control-plane metrics use ray.util.metrics (emitted by the deploy coordinator, state store, and deploy driver, which are not Serve replicas).

The gateway dimension

Every per-model and per-gateway metric below carries a gateway tag identifying which gateway emitted it, so multiple gateways sharing one Ray cluster stay distinguishable (the Grafana dashboard exposes a Gateway dropdown built on it). The tag is stamped automatically from MSHIP_GATEWAY_NAME — the deploy driver sets it and forwards it to every replica via runtime_env, so no call site passes it explicitly.

Cluster-scoped metrics are not per-gateway, because the thing they measure is shared cluster-wide: the deploy lock and reservations (one mutex per cluster), the state store (one shared backend), and all inherited ray_vllm_* / ray_serve_* / ray_node_* metrics (engine/cluster level). The dashboard's Gateway dropdown therefore filters the ray_modelship_* per-model panels but leaves the vLLM/GPU/Ray-Serve panels cluster-wide.

Gateway

Metric	Type	Tags	Description
ray_modelship_request_total	Counter	model, endpoint, status, gateway	Total requests by model and API method
ray_modelship_request_duration_seconds	Histogram	model, endpoint, gateway	End-to-end request latency
ray_modelship_request_errors_total	Counter	model, endpoint, error_type, gateway	Errors: validation_error, inference_error, stream_error, unhandled
ray_modelship_request_in_progress	Gauge	model, endpoint, gateway	Currently processing requests
ray_modelship_client_disconnects_total	Counter	model, endpoint, gateway	Client disconnected before response completed
ray_modelship_stream_chunks_total	Counter	model, gateway	Streaming chunks emitted
ray_modelship_auth_failures_total	Counter	reason, gateway	Requests rejected for invalid/missing API key (reason: missing, invalid)

Model Deployment

Metric	Type	Tags	Description
ray_modelship_model_load_duration_seconds	Histogram	model, loader, gateway	Time to initialize a model
ray_modelship_model_load_failures_total	Counter	model, loader, gateway	Failed model initializations
ray_modelship_models_loaded	Gauge	gateway	Number of loaded and ready models

Inference Timing

Metric	Type	Tags	Description
ray_modelship_generation_duration_seconds	Histogram	model, gateway	Chat/text generation latency
ray_modelship_tts_generation_duration_seconds	Histogram	model, gateway	Text-to-speech latency
ray_modelship_image_generation_duration_seconds	Histogram	model, gateway	Image generation latency
ray_modelship_transcription_duration_seconds	Histogram	model, gateway	Speech-to-text latency
ray_modelship_embedding_duration_seconds	Histogram	model, gateway	Embedding latency

Resource Cleanup

Metric	Type	Tags	Description
ray_modelship_resource_cleanup_errors_total	Counter	model, component, gateway	Errors during engine/model cleanup

HA Control Plane

These cover the multi-node / HA machinery deployed by the Helm chart (deploy coordinator, pluggable state store, gateway watch loop). The first six carry a gateway tag; the rest are cluster-scoped.

Metric	Type	Tags	Description
ray_modelship_gateway_reconciles_total	Counter	gateway	Routing reconciles a gateway replica applied from the coordinator
ray_modelship_gateway_watch_errors_total	Counter	gateway	Errors in a gateway replica's coordinator watch loop (retried)
ray_modelship_gateway_routing_generation	Gauge	gateway	Routing generation a gateway replica has reconciled to
ray_modelship_coordinator_generation	Gauge	gateway	Coordinator's current routing generation (compare to the replica gauge to spot lag)
ray_modelship_deploy_duration_seconds	Histogram	gateway	Wall-clock time for a deploy run to settle
ray_modelship_deploy_models_changed_total	Counter	gateway, action	Models changed by a deploy (action: add, remove, evict)
ray_modelship_deploy_lock_held	Gauge		1 while the cluster-wide deploy lock is held, else 0
ray_modelship_deploy_reservations_total	Counter	result	Deploy-lock reservation attempts (result: granted, locked, insufficient_gpu, insufficient_cpu)
ray_modelship_operator_force_release_total	Counter	reason	Locks force-released after ungraceful operator death (reason: probe_gone, unresponsive)
ray_modelship_state_store_operations_total	Counter	backend, op, result	State-store ops (op: get/set/delete; result: ok/error)
ray_modelship_state_store_operation_duration_seconds	Histogram	backend, op	State-store operation latency

Built-in Metrics from vLLM and Ray

These are automatically available when MSHIP_METRICS=true — no additional configuration needed.

vLLM (ray_vllm_*)

vLLM metrics are routed through Ray's metrics agent via RayPrometheusStatLogger. The native vllm: prefix is sanitized to ray_vllm_.

• ray_vllm_num_requests_running / ray_vllm_num_requests_waiting — queue depth
• ray_vllm_kv_cache_usage_perc — KV cache utilization (0-1)
• ray_vllm_time_to_first_token_seconds — TTFT histogram
• ray_vllm_inter_token_latency_seconds — ITL histogram
• ray_vllm_e2e_request_latency_seconds — end-to-end latency histogram
• ray_vllm_request_queue_time_seconds — time spent waiting in queue
• ray_vllm_prompt_tokens_total / ray_vllm_generation_tokens_total — token throughput counters
• ray_vllm_num_preemptions_total — memory pressure signal
• ray_vllm_prefix_cache_hits_total / ray_vllm_prefix_cache_queries_total — cache efficiency

Full reference: vLLM Metrics Documentation

Ray Serve (ray_serve_*)

• ray_serve_num_http_requests_total — request count by route, method, status
• ray_serve_http_request_latency_ms — request latency histogram
• ray_serve_handle_request_counter_total — request count by deployment
• ray_serve_deployment_processing_latency_ms — per-replica processing time
• ray_serve_health_check_latency_ms — health check latency histogram

Full reference: Ray Serve Monitoring

Ray Cluster (ray_*)

• ray_node_gpus_utilization — GPU utilization by device
• ray_node_gram_used / ray_node_gram_available — GPU memory
• ray_node_cpu_utilization — CPU usage
• ray_node_mem_used / ray_node_mem_total — system memory

Full reference: Ray Metrics