Rebuild knowledge index

Product language

Rebuild index with AI means: under the current agent’s knowledge base, from existing Markdown how-tos, automatically create or refresh two-layer retrieval index files, so later conversations can lock the related documents first, then read the original to answer or act. Users are not asked to care about index.json, knowledge/, and similar implementation names; ops and integrators can see Implementation mapping.

Mechanism goals

1. Few missed recalls: themes, summaries, and tags in the index should cover how users usually speak, so the first step picks the right theme or file.
2. Paths you can trust: document paths in entries must match .md files that actually exist on disk; do not rely on the model “inventing” paths that are not there.
3. Detail lives in the body: the index is navigation and coarse match only; clauses, numbers, and steps come from the opened Markdown body.
4. Evolvable: if block-level vector retrieval is added later, file-level paths and index structure should stay stable; block data hangs as an enhancement layer and does not break the two-layer retrieval contract.

Two-layer structure and file convention

Aligned with the “indexed how-to / skill documents” method (see 索引式文档与反馈闭环.md), the user-agent knowledge tree uses:

Path rules (implementation contract):

• In the root index, Markdown directly under the root is in documents; path is the filename only, with no /.
• In the root index, themes dir is a first-level theme directory name (one segment, no /).
• In a sub-index, documents[].path is relative to that first-level directory (may include nested folders) and must match scan results exactly.

JSON fields and Go structs: backend/internal/skilldocs/index.go (Index, ThemeItem, DocItem).

Scan and input (before generation)

1. Scope: Recursively walk the whole tree from that agent’s knowledge root (filepath.WalkDir semantics).
2. Include: *.md only; path and content must be valid UTF-8.
3. Exclude: files whose names start with . (same as common hidden/config convention).
4. Excerpt: Before a file enters the model, take the first N characters (counted in runes) so a single request is not huge; when assembling a subdirectory index prompt, excerpts may be shortened again (token control).
5. Grouping:

- .md one level under the root → participate in the root index documents. - Remaining .md files go to the theme named by the first path segment (first-level directory) for that theme’s sub-index; root index themes only cover first-level directories that actually contain Markdown.

Model call order and duties

1. Root first, then subdirectories (merge per article): generate and write the root index.json first, then for each first-level directory call the model per article to produce a DocItem, and the program merges into {dir}/index.json (avoids one huge prompt for a whole directory timing out).
2. Root index prompt duty: from root .md excerpts + the first-level directory list, produce version, themes, documents; each given first-level directory appears in themes at most once, and dir matches the list.
3. Subdirectory index (one article): each article outputs one DocItem object; after merge, themes is a fixed empty array; path in documents must come from the closed list given by the implementation; summaries and tags serve retrieval.

Temperature and similar hyperparameters follow the implementation (root and per-article generation currently use a low temperature for stable JSON).

Output and validation

1. Accept one top-level JSON object only; if the model appends natural language after the JSON, parsing takes only the first balanced { ... } object (quotes and escapes inside strings must be handled), to avoid parse failure.
2. Root index allowlist:

- documents: keep only scanned root .md filenames; drop items that contain .., /, or are not .md. - themes: keep only items whose dir is in the scanned first-level directory set; normalize defaults such as index_file. - If a first-level directory that should exist is missing a theme item, the implementation may fill a default theme item (so the retrieval path does not break).

1. Sub-index allowlist: path in documents must be in the list given when generating that file; otherwise drop or fail (implementation decides).
2. Write to disk: writing index.json must not let readers see a half-written file (e.g. write a temp file then replace; see the implementation).

How retrieval uses it (consistent with the index rules)

Retrieval reads the root index first → matches themes to the question → reads the matching subdirectory index → picks several documents → opens Markdown originals by path to build context (see skilldocs.BuildContext and similar).

So summary / tags in the index should serve “which article does this sentence look like”, not retell the whole text.

Optional paths (conditional load): documents / themes may add a paths string array (globs, e.g. ["/*.tsx"]). Inject that article only when the user message or attachment path hits a glob; do not fill this on generic onboarding / overview docs. With no path context, entries that have paths do not** enter conversation (see skilldocs/itemMatchesScope).

Chinese questions: retrieval splits consecutive Han characters into 1-grams / 2-grams (so a whole sentence with no spaces can still hit); if nothing hits, it falls back to injecting priority articles such as 导读.md, 00-总则与功能导航.md (see skilldocs/tokenize, fallbackDocs).

Evolution (not required now)

• Huge theme directories: if a single prompt is still too long, generate by subfolder then merge (merge can be deterministic rules + an optional light model); you need not force “one call per leaf directory”.
• Block-level vectors: later, each index entry or block metadata may add source_path and chunk_id; the file-level index can still be first-layer filter. Vector-store choice (LanceDB and similar) is a deploy/implementation decision and does not change the product meaning of two-layer index + read the original.

Stepwise generation and progress (implementation)

The product UI may request scan preview → root index → each theme sub-index in order, so it can show current step / total steps:

Preview GET .../reindex/preview steps_total = 1 (root) + Markdown article count under each theme directory; top_dir_docs lists relative paths still to index under each theme.

Implementation mapping (engineering and contracts)

Related documents

• 索引式文档与反馈闭环.md (two-layer indexes and the closed-loop method)
• 智能体调用知识文档的方式.md (conversation injection and three layers of knowledge)
• ../产品规格.md (product language and layers for the “knowledge base”)