Anionex/dsh-computer-use

DSH Computer Use

Native macOS control for DeepSeek Harness that keeps your real cursor and foreground application alone by default; the Bundle may bring the target app forward before keyboard input for reliable typing.

DSH Computer Use gives an Agent fresh Accessibility observations, exact process/window targeting, stale-state rejection, scoped application access, and verified post-action state. Semantic Accessibility comes first; mouse, drag, wheel, and keyboard fallback are routed to the selected process instead of the global desktop.

Why it is different

Accessibility permission lets a process inspect and operate macOS UI elements, but the permission itself does not prevent focus stealing or cursor movement. Those behaviors depend on the input route.

The default DSH Computer Use route is deliberately non-interfering:

• No system-cursor movement: the helper contains no cursor-warp path.
• No global pointer injection: click, scroll, and drag fallback use a pid/window-targeted SkyLight route, not the global HID event stream.
• No pointer-triggered activation: semantic Accessibility, process-targeted pointer input, and keyboardPolicy: preserve run without activation; keyboardPolicy: activate (Bundle default) brings the target app forward before keyboard fallback, matching Codex Computer Use.
• A separate Agent cursor: click, scroll, and drag actions animate a click-through, nonactivating software cursor while the macOS system cursor remains untouched. It is visible by default and stays at the action position until the bound window changes or a hide command; cursorAutoHideMs can opt into timed auto-hide.
• No blind replay: every action is tied to an exact, unexpired observation and returns fresh state.

The result is a native action layer that can operate many background applications while the user continues working in the current foreground application.

What it adds

• Observe before acting. Return a bounded Accessibility tree, indexed elements, exact app/process/window metadata, permission state, and an optional screenshot Artifact.
• Bind actions to state. Every element exposes an observation-local index and opaque targetHandle; exact lookup remains compatible, while explicitly allowed rebinding accepts only a unique native or semantic identity inside the same process and window.
• Prefer semantic input. Use AXPress, editable values, selected-text assignment, and advertised Accessibility actions before pointer fallback.
• Route fallback to the target. Keyboard input goes to the selected pid; pointer input goes to the selected pid and CGWindowID with window-local coordinates, resolving the app window under the point so arbitrary screen coordinates work.
• Return fresh evidence. Every successful action settles for a bounded interval and returns a new full or diff observation.
• Scope application access. Read and control leases are separated by Agent, Session, turn, and exact bundle id; high-impact actions require one-use confirmation.
• Keep the model surface focused. Execution Tools appear only after the current Agent loads the Computer Use Skill.

Proof: a never-active background fixture

The repository includes a deterministic AppKit fixture and a universal native helper. Release tests start the fixture with open -g in background mode, then use the same protocol exposed to the Agent.

observe exact bundle id + pid
-> element: "Targeted pointer probe", no AXPress action
-> computer_click with observationId + element index + allowCoordinateFallback
-> fresh observation
-> activation "not-requested"; pointerRouting "target-process"
-> status "Status: pointer click"

The fixture records every applicationDidBecomeActive callback. An independent native monitor also samples the system cursor and frontmost pid every millisecond throughout click, scroll, and drag. The default release path must not increase activationCount; it also requires unchanged cursor coordinates, an unchanged frontmost pid, exact click/scroll counts, and one complete down/up drag gesture.

See Foreground-safe input policy for the requirements, architecture, decisions, evidence, and compatibility limits.

Scope

dsh-computer-use is the native action layer. It does not replace narrower interfaces:

• browser tasks should use browser automation and DOM/CDP state;
• APIs, CLIs, and purpose-built application plugins remain preferable when available;
• OCR, visual grounding, and pixel interpretation should use the separately installed dsh-vision-toolkit: load the vision-tools Skill and pass the exact screenshot Artifact path to vision_glance, vision_ground, vision_detect, vision_crop, or vision_long_screenshot_ocr; do not replace those tools with shell-driven tesseract, screencapture, or ad hoc Swift/Python OCR;
• domain bundles such as dsh-design can compose Computer Use when a workflow crosses into a native application.

Quick start

Prerequisites

• macOS 14 or newer.
• DeepSeek Harness with a Web or Headless Profile and the Skill Tool mounted.
• macOS Accessibility permission for observation and native actions.
• macOS Screen Recording permission only when a screenshot is requested.
• Node.js ^22.19.0 or >=24.0.0 when building this repository.

Install the Web and Headless bundles directly from npm:

dsh plugin --profile web add @anionex/dsh-computer-use
dsh plugin --profile headless add @anionex/dsh-computer-use

dsh --profile web --dump-config | grep computer-use
dsh --profile headless --dump-config | grep computer-use

For local development, replace the package name with an absolute checkout path.

Restart a running dsh web host after changing the installed plugin, then start a new Session so the host reloads the Bundle and Skill catalog.

Load the Skill in that Session:

/computer-use

Then try:

Use Computer Use to inspect the running DSH Computer Use Fixture, enable its deterministic option, and report the fresh status. Prefer Accessibility elements and do not reuse an old observation.

How it works

flowchart LR
    A["Select exact bundle id and pid"] --> B["Acquire scoped read access"]
    B --> C["Observe AX tree and optional screenshot"]
    C --> D["Choose target handle, index, or window-relative point"]
    D --> E["Acquire control and optional one-use confirmation"]
    E --> F["Re-observe and validate exact target"]
    F --> G{"Input route"}
    G -->|"Semantic"| H["Accessibility action or value"]
    G -->|"Keyboard"| I["Post to target pid"]
    G -->|"Pointer"| J["Post to target pid + window"]
    H --> K["Wait for settlement"]
    I --> K
    J --> K
    K --> L["Return fresh full or diff observation"]

Every observed element has an observation-local compatibility index and an opaque targetHandle. Index-only actions retain exact locator behavior. A low-risk element action may pass targetHandle with allowRebind: true; immediately before input, the provider-independent resolver obtains fresh Accessibility state and checks, in order, the original locator, a unique provider-native identifier such as macOS AXIdentifier, then one unique semantic match over role, accessible name, advertised actions, and stable ancestor fingerprint. The resolver keeps the exact bundle id, pid, and selected-window identity, and fails closed with COMPUTER_TARGET_AMBIGUOUS or COMPUTER_TARGET_LOW_CONFIDENCE instead of guessing. Coordinate actions still require the complete referenced window state to remain current.

Successful element actions report resolution.mode, confidence, candidateCount, and targetChanged. A sensitive target that needs rebinding invalidates the prior one-use confirmation and returns COMPUTER_TARGET_REBIND_REQUIRES_CONFIRMATION; the caller must observe the current UI and confirm the newly selected handle. Visual coordinates are not target handles and never authorize sensitive rebinding. Provider-native visual hit-testing is not part of this foundation release and remains follow-up work.

The default interaction policy is:

interaction:
  focusPolicy: preserve
  keyboardPolicy: activate
  pointerInputPolicy: targeted
  cursorVisualization: visible
  cursorMotionMs: 180
  cursorAutoHideMs: 0

cursorVisualization: visible displays the Agent’s own non-interactive cursor for click, scroll, and drag. It never replaces or moves the macOS system cursor. Set it to hidden when visual feedback is unwanted. pointerInputPolicy: deny disables coordinate click/fallback, scroll, and drag. keyboardPolicy: activate (Bundle default) makes type-text keyboard fallback and press-key reliable by activating the target app first; focusPolicy: activate is the broader compatibility mode that also activates before pointer input. After activation, the helper re-observes and revalidates the exact target before input.

The cursor is a 28x28 transparent whole-image cursor (Cursor arrow plus DeepSeek whale, assets/cursor.png) with the hotspot at the image’s top-left corner. It is a separate process, click-through, nonactivating, and bound to the exact observed pid, window, and frame so it disappears if the target window closes, moves, resizes, or is minimized.

The helper executable is an internal DSH transport rather than a public authorization API. It requires an isolated process group plus parent-owned standard transports, so ordinary shell redirection fails closed before command parsing. This is defense in depth, not authentication against arbitrary code running as the same macOS user: a deliberately constructed detached parent can reproduce that transport topology. Use the registered Tools so application leases, sensitive-action confirmation, and host policy checks remain in force; danger-full-access must not be treated as protection against direct native invocation.

Successful action results include:

activation: 'not-requested' | 'already-frontmost' | 'activated'
pointerInput: boolean
pointerRouting: 'none' | 'target-process'
resolution?: {
  mode: 'exact-locator' | 'native-identifier' | 'semantic-rebind'
  confidence: number
  candidateCount: number
  targetChanged: boolean
}

The model cannot override these host policies through Tool arguments.

Model Tools

The Bundle initially contributes only computer_use_activate. Loading the Skill exposes the focused execution vocabulary for that Agent.