htcqp802/dsh-knowledge-base

dsh-knowledge-base turns documents into a searchable knowledge base for DeepSeek Harness agents. Drop in a PDF, Word, or Markdown file — it is parsed, chunked by section, and instantly searchable by your agent with FTS5 (BM25) ranked retrieval. A built-in Web UI lets you manage the knowledge base like a file manager: create/rename/delete folders, move files, and browse content.

✨ Features

• 📥 Import anything — md / txt / json / yml / docx / pdf, no size limit; PDF is parsed in-process with pdfjs-dist (cross-platform, zero system dependencies)
• 🔪 Auto-chunking — documents are split by headings/paragraphs; adjacent small chunks are merged to reduce fragmentation
• 🔁 Upsert — re-importing the same file overwrites it; no duplicate accumulation
• 🗂 Folder management — categories are folders: create / rename / delete (empty) / move files, like an OS file manager
• 🔍 FTS5 full-text search — SQLite FTS5 (trigram tokenizer, no Chinese segmentation needed) + BM25 relevance ranking; falls back to LIKE for short queries or anomalies
• 🤖 Agent-native tools — kb_query / kb_import / kb_list / kb_update / kb_delete, usable directly by the model in conversation
• 🖥 Web management UI — a “Knowledge Base” tab in the conversation view: drag-and-drop import, directory browsing, category management, search
• ⚙️ Configurable categories — no preset categories by default (fits any domain); create folders at runtime from the UI

🚀 Install

dsh plugin --profile web add dsh-knowledge-base

Dependencies: DeepSeek Harness (dsh) provides the @deepseek-ai/* runtime; Node ≥ 22.5 (built-in node:sqlite).

🏃 Quick Start

Via the Web UI (recommended)

1. Start dsh Web: dsh web (or dsh --profile web)
2. Open your browser, create a session, and switch to the Knowledge Base tab at the top of the session view
3. Drag and drop files to import → auto-chunked → browse / rename / move in the directory view
4. Ask your agent in chat: “Use kb_query to search for ISO9001”

Via agent tools (headless / any profile)

import:  Use kb_import to import /path/to/manual.pdf, category "Documents", tags ["manual"]
search:  Use kb_query to search for "transformer"
list:    Use kb_list / kb_list category=Documents
update:  Use kb_update to change id=3's category to "Documents"
delete:  Use kb_delete to delete id=3

⚙️ Configuration

# In the profile's cordis.patch.yml or when installing the bundle
# No categories are preset (fits any domain); entries without a category go to "Uncategorized".
# You can also create folders directly in the UI (persisted, no config edit needed).
- id: knowledge-base
  name: 'dsh-knowledge-base'
  config:
    categories:            # configure as needed, e.g.:
      - Documents
      - Manuals

🗂 Web API (for the UI and third-party integrations)

🏗 Architecture

dsh-knowledge-base (one npm package, three plugin rows)
├── dsh-knowledge-base          host tools: kb_query / kb_import / kb_list / kb_update / kb_delete
├── dsh-knowledge-base/web      Web endpoints: /api/kb/* (web composition only)
└── (client half)               "Knowledge Base" conversation view tab + directory browser UI

Data storage (default):

$DSH_HOME/knowledge-base/kb.sqlite    # entries + FTS5 index + meta (dynamic categories)
$DSH_HOME/knowledge-base/inbox/       # upload temp dir (cleaned up after import)

Tables: kb(id, category, name, summary, payload, tags, source, updated_at) + kb_fts (FTS5 external-content table) + meta (dynamic categories).

🔧 Development

npm run build                     # tsc type-check + tsdown bundle (host half + client bundle)

# Local verification (use a workspace-local test home, never touch ~/.dsh)
DSH_HOME=$PWD/.dsh-home DSH_TELEMETRY_DISABLED=1 \
  dsh --profile headless --patch dev-headless.cordis.yml \
  "Use kb_import to import /tmp/test.md, then kb_query to search for 'keyword'"

The standalone repo ships dev.cordis.yml / dev-headless.cordis.yml for local verification. During development the @deepseek-ai/* dependencies are symlinked from an official checkout via scripts/link-official-deps.mjs — see AGENTS.md → “Dependencies”.

🗺 Roadmap

• File import (md/txt/json/yml/docx/pdf)
• Folder management (create/rename/delete/move)
• FTS5 full-text search (BM25 + Chinese trigram)
• Knowledge base management Web UI
• AI auto-classification (direct ctx.llm structured calls on import)
• Entry detail view/edit
• OCR (scanned PDFs)
• Better Chinese tokenization (custom tokenizer instead of trigram)

🤝 Contributing

PRs welcome! Please read AGENTS.md (agent development guide) first. Before submitting:

1. npm run build passes
2. The headless tool chain is self-tested
3. No local data is committed (.dsh-home/, .test-workspace/, etc. — see .gitignore)

📄 License

MIT © dsh-knowledge-base contributors

🔗 Related

• DeepSeek Harness — Everything is a Plugin
• awesome-dsh-plugin — curated community plugin list
• pdfjs-dist — PDF parsing engine