Tune and troubleshoot

Improve Hexabot RAG quality, evaluate retrieval, and diagnose helper, indexing, and content problems.

RAG quality depends on four independent layers:

1. source-content quality;

2. retrieval configuration and indexing;

3. ranking quality;

4. generation instructions and model behavior.

Troubleshoot them separately. A fluent model answer does not prove that the correct content was retrieved.

Start with an evaluation set

Create a small repeatable dataset before tuning:

Column	Example
Query	“I lost my password; how do I get back in?”
Expected Content ID/title	“Reset a password”
Must retrieve in top K	Yes
Must not retrieve	“Change billing address”
Expected answer facts	Reset link is sent by email

Use representative wording from real users, including abbreviations, spelling variants, and multiple supported languages. Run the same set after every helper, model, chunking, or content change.

Measure retrieval independently from generation. At minimum, track whether the expected Content entry appears in the first K hits.

Improve the source content first

Use one primary topic per entry

A focused entry gives both lexical and vector helpers a clearer signal. Split a long page that mixes several unrelated procedures.

Write descriptive titles

Titles are part of the canonical search text. Prefer Reset a customer password over Account help.

Put searchable facts in string fields

Use Text and Text Area fields for facts that must be retrieved. Numeric, boolean, attachment, and relation values are not expanded into prose automatically.

Include important terminology

For lexical retrieval, include common aliases, acronyms, plan names, error codes, and product names in natural text.

Keep publication status accurate

Active content is returned by default. An inactive entry may be correctly excluded rather than missing from the index.

Do not rely on uploaded files alone

Media Library files are not automatically parsed into the Content corpus. Extract their text into entries or build an ingestion helper.

Tune retrieval settings

Result limit

The default is 3 hits. Increase it only when relevant evidence is consistently ranked just below the cutoff.

A higher limit can:

• improve recall;

• increase model input cost;

• introduce conflicting or weak context;

• make the final answer less focused.

Common starting values are 3 to 5. Validate larger values with your evaluation set.

Content Type filter

A Content Type filter is often the highest-value precision improvement. Use it whenever the workflow knows the domain.

Chunk size for vector helpers

The official vector packages, hexabot-helper-sqlite-vector and hexabot-helper-pgvector, default to 2000 characters with 200 characters of overlap. Chunking is character-based and deterministic; paragraph and line boundaries are preferred when possible.

• Smaller chunks can improve precision for short, independent facts but create more vectors and may lose surrounding context.

• Larger chunks preserve context but can mix topics and dilute the matching signal.

• Overlap helps preserve facts that cross a boundary but increases embedding volume and duplicate context.

Keep overlap smaller than chunk size. Change one parameter at a time and re-run the same evaluation queries.

Embedding model and dimensions

Use an embedding model appropriate for your languages and domain. The Embedding dimensions value is a request; models or providers that do not support dimension reduction return their native vector size.

Changing provider, model, dimensions, chunk size, or overlap creates a new vector profile and requires re-embedding.

Scores

score is optional and helper-specific. Do not display it as a universal confidence percentage or apply the same threshold across fulltext-search, the sqlite-vector and pgvector runtime helpers, and custom helpers unless you have calibrated those scores.

Diagnose common symptoms

Symptom	Likely causes	Checks and actions
RAG helper is missing from Settings	Package not installed, API not restarted, wrong database type, compiled helper missing	Verify package.json, reinstall dependencies, restart the API, confirm SQLite vs PostgreSQL, inspect startup logs.
warning says the helper is unavailable	Default helper points to an incompatible or unregistered helper	Select a helper compatible with the active database.
warning says settings or credential are invalid	Missing credential, empty secret, unsupported provider, invalid base URL/model/chunk values	Reopen the helper settings and credential; verify an embedding model, not a chat model.
Exact terms are not found by full-text search	Entry inactive, facts not stored as strings, terminology absent, Content Type filter excludes it	Inspect the entry status and canonical text inputs; add searchable wording; verify filters.
Semantic helper returns no results	Reindex incomplete, provider unreachable, no active indexed content, wrong profile, extension unavailable	Check API logs, provider quota/network, selected helper, status policy, and database extension. Schedule reindex.
New PostgreSQL content is temporarily missing	Durable indexing job has not completed or is retrying	Check API logs and provider availability; allow the queue to reconcile.
Results are broadly related but not precise	Entries or chunks mix topics; no Content Type filter; limit too high	Split content, reduce chunk size, add filter, lower limit.
Correct hit is ranked below irrelevant hits	Weak title/text, embedding model mismatch, multilingual/domain limitations	Improve source wording, evaluate another embedding model, add metadata filtering or a reranker in a custom helper.
Model ignores correct retrieved text	Prompt does not delimit context, instructions permit model memory, too much context, prompt injection	Strengthen grounding instructions, separate question/context, reduce weak hits, treat retrieved text as untrusted data.
Direct search API fails but workflow action shows a warning	Different error surfaces	The action wraps known configuration/availability errors; the direct endpoint can return an HTTP error. Inspect both response and logs.

Inspect retrieval before the LLM

During debugging, temporarily stop the workflow after retrieve_rag_content and inspect:

• hits.length;

• each hit's contentId and title;

• text returned to the model;

• source;

• optional scores;

• warning.

Only tune generation after retrieval reliably returns the intended evidence.

Security checklist

• Treat retrieved content as untrusted data and tell the model not to follow instructions inside it.

• Keep system and developer instructions separate from retrieved text.

• Do not index secrets or content that the workflow's users are not allowed to see.

• Remember that the official helper filters are status and Content Type, not per-user authorization.

• Review embedding-provider retention, regional processing, and incident-response terms.

• Keep Index only active content enabled unless there is a reviewed reason not to.

• Log content IDs and helper names for audits without unnecessarily logging full confidential text.