xxiaoxiong/dsh-prometheus

dsh-prometheus

Prometheus metrics and a ready-to-import Grafana dashboard for DeepSeek Harness. The plugin observes public DSH lifecycle seams, keeps metric series bounded, and avoids exporting conversation or tool payloads.

Release status: 0.1.0 is the first developer-preview release, tested against DSH 0.1.0-rc.6.

What it monitors

• Active sessions; agent turns, steps, errors, and duration histograms
• LLM request rate, terminal status, latency, and disjoint input/output/reasoning/cache token counters
• Tool call rate, normalized success/error status, and latency
• Approval request and fixed outcome counters
• Subagent starts, outcomes, and duration
• Active/started/completed/killed/failed background jobs and duration
• Process start time and cardinality-overflow diagnostics

The exact names, labels, and meanings are in docs/metrics.md.

Compatibility

DeepSeek Harness is a developer preview. Treat every DSH upgrade as a compatibility event and run the full test/release checklist.

Install

Install the package into a DSH profile and verify its bundle layer:

dsh plugin --profile monitoring add dsh-prometheus
dsh --profile monitoring --dump-config

The bundle inserts one row with id prometheus. Verify the dump includes the dsh-prometheus layer before booting the profile.

Endpoint selection

The default mode: auto makes one activation-time decision:

1. If a public DSH webServer is already active and safely bound, register the exact /metrics route on it.
2. Otherwise, start a standalone endpoint on 127.0.0.1:9464.

For the bundled Docker example, force standalone mode so Prometheus has a stable target. Add this row to the profile’s cordis.patch.yml (later layers replace the whole config value):

- id: prometheus
  config:
    enabled: true
    mode: standalone
    host: 127.0.0.1
    port: 9464
    path: /metrics
    allowRemote: false
    maxLabelValues: 64
    maxLabelValueLength: 80

Configuration

Remote exposure has no built-in authentication or TLS. Read SECURITY.md before enabling it.

Quick scrape

curl --fail --silent http://127.0.0.1:9464/metrics

The endpoint accepts GET and HEAD, returns 405 for other methods, disables caching, and uses the Prometheus/OpenMetrics-compatible prom-client content type.

Prometheus and Grafana

The repository includes a full local stack:

docker compose -f examples/docker-compose.yml up -d

Then open Prometheus at http://127.0.0.1:9090 and Grafana at http://127.0.0.1:3000 (admin / admin; change it outside local development). The datasource and DeepSeek Harness / Overview dashboard are provisioned automatically.

The standalone assets are examples/prometheus.yml and grafana/dsh-overview.json. Security, compatibility, and verification details live in docs/security.md, docs/compatibility.md, and docs/verification-report.md.

Useful queries:

# Turn throughput
sum(rate(dsh_agent_turns_total[5m]))

# Turn failure ratio
sum(rate(dsh_agent_turns_total{status=~"error|blocked|aborted"}[5m]))
/
clamp_min(sum(rate(dsh_agent_turns_total[5m])), 0.000001)

# P95 LLM latency by provider/model
histogram_quantile(0.95,
  sum by (le, provider, model) (rate(dsh_llm_request_duration_seconds_bucket[5m])))

# Tool errors by tool
sum by (tool) (rate(dsh_tool_calls_total{status!="success"}[5m]))

See examples/prometheus/alerts.yml for starter alerts. Tune thresholds to the workload; quiet agent processes legitimately have zero throughput.

Privacy and cardinality

The collector never exports prompts, user/assistant messages, system prompts, tool arguments or results, error text, file paths, working directories, arbitrary metadata, or session/agent/call/job/subagent ids.

Only four runtime-controlled label keys exist: provider, model, tool, and job kind. Values must match [A-Za-z0-9][A-Za-z0-9_.:/-]*, remain within the configured length, and fit the per-key vocabulary cap. Rejected or excess values become __other__; the event is counted by dsh_metrics_label_overflow_total without logging the original value.

Provider/model/tool/kind names are still operational metadata. Do not put customer identifiers or secrets in those names.

Lifecycle behavior

All listeners, job observers, routes, sockets, and registries belong to the plugin’s Cordis fiber. On hot reload or uninstall, the route is removed or the standalone server is closed and remaining sockets are destroyed. The registry is instance-local, so reload does not collide with process-global collectors.

Existing live sessions and jobs seed gauges only. Historical events are not replayed into counters. Counter resets across process restart/hot reload are normal Prometheus behavior.

Upgrade, downgrade, remove

# Upgrade after obtaining a newer tarball/version
dsh plugin --profile monitoring add ./dsh-prometheus-NEW.tgz
dsh --profile monitoring --dump-config

# Remove the dependency and bundle layer
dsh plugin --profile monitoring remove dsh-prometheus

Before a downgrade, compare metric and config contracts in CHANGELOG.md. Removing the plugin does not delete data already stored by Prometheus.

Development

pnpm install --frozen-lockfile
pnpm check
pnpm test:coverage
pnpm pack

The suite covers config/cardinality, privacy, LLM/tool waterfalls, endpoint methods, remote-exposure gates, WebServer route teardown, standalone socket teardown, job/subagent lifecycles, and a real rc.6 AgentLoop turn. See CONTRIBUTING.md, docs/architecture.md, and docs/release-checklist.md.

Troubleshooting and limitations

See docs/troubleshooting.md and docs/known-limitations.md.

License

MIT