pengxuding/dsh-plugin-judge

dsh-plugin-judge · Plugin Value Auditor for DeepSeek Harness

Judge whether a DeepSeek Harness plugin is worth using: pre-install review, post-install audit, and re-audit reminders when the model changes.

中文文档

The problem this plugin solves

Plugins fall into two natures:

• Capability plugins give the model new abilities (new tools, skills, MCP servers, services, commands) — like giving the model “an extra hand”.
• Constraint plugins impose rules and guardrails on the model (injecting system prompt sections, personas, AGENTS.md rules, intercepting agent/pre-step / tools/pre-execute, restrict / guard on tools, or complete: true wholesale prompt replacement) — like putting the model “in a cage”.

Constraint plugins carry a hidden cost: the model will follow the rules faithfully (compliance looks great), but it performs worse on the task you actually care about — rules crowd the context and suppress the model’s own judgment, putting a low ceiling over a capable model. Worse, as models get upgraded, many “patch rules for the old model” turn from patches into ceilings, and the plugin becomes obsolete or even harmful.

So a plugin’s value is not a property of the plugin alone — it is a binary relation of “plugin × current model”. That is exactly what this plugin evaluates.

Features

How it judges (two layers combined)

1. Rule heuristics (free, deterministic): scans the plugin source and its injected content for signals, weights them into a 0–100 constraint-risk score, and classifies the plugin as capability / constraint / cosmetic / hybrid.
2. LLM judge (fine-grained, on demand): hands the scan result, the injected content summary, and the current model’s identity to a model and asks whether these rules help or suppress this specific model, returning a verdict with reasons.

Usage

# Install (requires an existing web profile)
dsh plugin --profile web add dsh-plugin-judge

# Or install from source
dsh plugin --profile web add <path-to-this-repo>

After installing and restarting the harness:

• Type /plugin-audit github:some/repo in chat for a pre-install review;
• Or let the model call the judge_plugin tool directly;
• The settings panel gains a Plugin Judge page: scan reports for every installed plugin, a pre-install review input, and audit history;
• After a model switch, if any plugins need re-auditing, a floating reminder pops up on the page.

Design

Data flow

Installed plugins: profile manifest (~/.dsh/profiles/<profile>/package.json → dsh.profile.bundles)
                   + each package's package.json / lib source (read locally via node:fs)
Candidate plugins: package.json + entry source fetched from npm registry / jsDelivr / GitHub raw (Node fetch)
Heuristic scan:   capability signals (+) vs constraint signals (−), weighted → class + risk score
LLM judge:        llm.stream (generative verdict, carrying the current model identity) → verdict + reasons
Model watch:      agentDefaultModel.currentSelection() polling (fallback: agent/request waterfall observation)
Persistence:      ~/.dsh/plugin-judge/audits.json (audit history + model fingerprint)
Bridge:           Host ctx.connection.rpc.handle('/plugin-judge', …) ← Client rpc.call polling
Reminders:        Host produces pending-reminders → Client polls → shell.overlay popup
Reports:          Client settings.section "Plugin Judge" page + judge_plugin tool card

Signal table (heuristics)

Capability signals: tools.register / defineTool / skills / MCP / webServer.register / registerFetchProvider / commands.register / provide( …

Constraint signals: systemPrompt.section / ctx.systemPrompt / persona / writing AGENTS.md / agent/pre-step / tools/pre-execute / tools/execute waterfall listeners / restrict( / guard( / complete: true wholesale replacement / strong-directive density (must, never, always, MUST, NEVER, 必须、不得、禁止) / fixed-template instructions (strictly follow, exact format, 严格按照、按以下格式) …

Layout

judgePlugin/
├── package.json          # dsh.bundle.patch + dsh.client.inject + exports (Node >= 18)
├── cordis.patch.yml      # insert: plugin-judge
├── lib/
│   └── index.js          # Host half: inventory / source fetching / heuristic scan / LLM judge /
│                         # model watch & reminders / tool & command / connection RPC bridge / persistence
└── client/
    └── client.js         # Client half (__ModuleLoader__ format, no build step):
                          # settings.section report panel + shell.overlay model-switch reminder popup

Disclaimer

Verdicts are advisory opinions produced by heuristics plus a model judge — not a security audit. Installing any plugin runs third-party code on your machine with your own permissions; read the source before you install.