Observability#

vime’s default observability path is intentionally small: training metrics still go to W&B / TensorBoard; high-frequency vLLM Prometheus metrics are no longer uploaded to W&B; request timings from vLLM response metrics are stored in sample traces and aggregated once per rollout step as compact perf/... metrics.

W&B / TensorBoard Metrics#

W&B and TensorBoard still receive reward, loss, KL, entropy, eval, and other training metrics. vLLM request timing summaries are logged under perf/, for example:

perf/request/e2e_latency/mean
perf/request/queue_time/median
perf/request/count
perf/request/profiled_count
perf/decode/throughput/mean
perf/prefill/queue_duration/mean
perf/prefill/ttft_duration/max
perf/prefill/transfer_speed_gb_s/mean
perf/decode/remote_kv_wait_duration/mean
perf/decode/allocation_wait_duration/mean
perf/decode/transfer_worker_duration/max
perf/decode/handshake_wait_worker_duration/mean
perf/decode/transfer_post_worker_duration/mean

These metrics are aggregated once per rollout step, not emitted once per request, so they should not slow W&B like uploading raw Prometheus metrics would.

Without PD, common perf/request/... metrics and available perf/decode/throughput/... metrics still exist. Detailed perf/prefill/... and perf/decode/...duration metrics only appear when vLLM returns the corresponding timing fields, which vime translates into pd_* trace attributes.

Where Prometheus Data Is Stored#

vime does not store per-second Prometheus data. vLLM exposes /metrics on each engine’s HTTP port; the router exposes its own metrics on a separate Prometheus port. Prometheus scrapes those endpoints periodically and stores the time series in Prometheus’s own TSDB.

That means:

  • If Prometheus is not running, serving metrics are only available from the current vLLM process memory and endpoint output, with no historical storage.

  • If Prometheus is running, history is stored under Prometheus’s --storage.tsdb.path.

  • vime does not upload these high-frequency metrics to W&B.

Useful vLLM metrics include:

vllm:num_requests_waiting
vllm:num_requests_running
vllm:num_requests_waiting_by_reason
vllm:nixl_xfer_time_seconds_bucket
vllm:nixl_bytes_transferred_bucket
vllm:mooncake_xfer_time_seconds_bucket
vllm:mooncake_bytes_transferred_bucket

Connector metrics depend on the selected backend: nixl_* for NIXL, mooncake_* for Mooncake. These are useful in Prometheus / Grafana for live queue buildup, transfer volume, latency histograms, and other serving-side symptoms.

Starting Prometheus#

Prometheus must run while training is running because it can only scrape endpoints that are currently alive. It does not need to run inside the training Python process; run it as a side process in the same machine or job.

A minimal config is:

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: vime-vllm
    metrics_path: /metrics
    static_configs:
      - targets:
          - "ENGINE_IP:ENGINE_PORT"
  - job_name: vime-router
    metrics_path: /metrics
    static_configs:
      - targets:
          - "ROUTER_IP:ROUTER_PROMETHEUS_PORT"

List each engine’s HTTP address under vime-vllm. For vime-router, use the router address and the separate Prometheus port printed by vime, not --vllm-router-port.

Start Prometheus with its TSDB path on persistent storage:

prometheus \
  --config.file=/path/to/prometheus.yml \
  --storage.tsdb.path=/path/to/prometheus-data \
  --storage.tsdb.retention.time=7d \
  --web.listen-address=0.0.0.0:9090

The vime image includes the prometheus binary, so this command can run directly inside the container. You can also start a side container from the same image as long as it can reach the configured engine and router scrape targets and mounts /path/to/prometheus-data on persistent storage.

If --storage.tsdb.path points to container-local disk, the data is lost when the container is removed. If it points to NFS, a persistent volume, or a job output directory, you can restart Prometheus with the same TSDB directory after training and query the historical time range in the Prometheus UI or Grafana. This is time-series replay, not full per-request trace replay; per-sample request timings still come from sample traces / debug rollout data.

Trace Viewer#

Debug rollout dumps saved with --save-debug-rollout-data include sample traces. The trace viewer reads vLLM timing attrs directly from those traces and uses pd_* fields to render synthetic [P] / [D] lanes.

python tools/trace_timeline_viewer.py /path/to/debug/rollout_0.pt

The default path does not require separate ReqTimeStats(...) logs, Loki, or a compaction tool.