Troubleshooting¶

Common failure modes and how to root-cause them. Issues are grouped by the layer they originate in: infrastructure → ingest → MCP → agent.

Infrastructure¶

Weaviate pod is CrashLoopBackOff¶

kubectl -n ai-workloads logs deploy/weaviate --tail=80

Look for:

• failed to load schema: PVC state is incompatible with a new image version. If you just upgraded from 1.27 to 1.35 and don't care about old data, delete the PVC and reapply:

  kubectl -n ai-workloads delete pvc weaviate-pvc
  kubectl apply -f k3s-deployment/base/weaviate-pvc.yaml
  kubectl -n ai-workloads rollout restart deploy/weaviate
  

• module text2vec-transformers not enabled: the env var didn't land. Double check the deployment manifest has both ENABLE_MODULES=text2vec-transformers and DEFAULT_VECTORIZER_MODULE.

Weaviate pod stays 0/1 Running and exits every ten minutes¶

kubectl -n ai-workloads logs deploy/weaviate --previous ends with could not join raft join list: context deadline exceeded. Weaviate detected this node to have state stored after RAFT_BOOTSTRAP_TIMEOUT (600 s), and the readiness probe returns 503 the whole time. Weaviate names its Raft node after the pod unless CLUSTER_HOSTNAME is set, so a re-created pod (image change, node reboot, rollout restart) cannot rejoin the single-voter cluster its predecessor left on the volume. The hint in the message about raising the timeout does not apply: the database is not loading, the node identity changed.

The base manifest pins CLUSTER_HOSTNAME=node1 and sets RAFT_ENABLE_ONE_NODE_RECOVERY=true, which lets a node whose name changed rewrite the stored configuration to itself. Apply it and watch for the recovery line:

kubectl apply -f k3s-deployment/base/weaviate-deployment.yaml
kubectl -n ai-workloads logs deploy/weaviate | grep -m1 'perform cluster recovery'
# ... "old_single_cluster_node_name":"weaviate-<old pod>","new_single_cluster_node_name":"node1" ...

On 2026-10-25 the pod was ready eleven seconds after the apply; the data on the volume is untouched by the recovery.

Keyword (BM25) scores are 0 and hybrid ranking looks arbitrary¶

Exact-title pages fall out of the top ten, every search_artifacts hit carries the same score, and a direct probe returns matches whose score is exactly 0 on the word-tokenized properties while the explanation still shows a term frequency:

python -c "
from fractal_rag.client import client_session
from weaviate.classes.query import MetadataQuery
with client_session() as c:
    res = c.collections.get('FractalArtifact').query.bm25(
        query='lacunarity', query_properties=['content'], limit=3,
        return_metadata=MetadataQuery(score=True, explain_score=True))
    for o in res.objects: print(o.metadata.score, o.metadata.explain_score)
"

Weaviate before 1.35.3 writes a NaN average property length for a segment flush that contains only deletions (weaviate/weaviate#10102); compaction spreads it through the collection, and on arm64 the NaN becomes an average of 0, so the BM25 term weight is 0 for every match in title, summary, content and authors (tags, source_type and the other FIELD-tokenized properties keep scoring). A delete_many of 97,762 objects on 2026-10-25 triggered it: the keyword leg of every hybrid query became a flat bonus and the Hurst Exponent concept page ranked 26^th for its own title. The base manifest pins 1.35.23: applying it restarted Weaviate on the patched release, whose loader recomputes the averages from the stored per-document lengths, and the scores were back without a rebuild (the concept page ranks first). On a version that cannot be upgraded, schema bootstrap --drop plus a full re-ingest also clears the NaN segments.

t2v-transformers pod is Pending forever¶

kubectl -n ai-workloads describe pod -l app=t2v-transformers | tail -40

If you see 0/1 nodes are available: 1 Insufficient nvidia.com/gpu:

• Confirm the GPU plugin is running: kubectl -n kube-system get pods | grep nvidia-device-plugin
• Confirm GPUs are visible: kubectl describe node | grep nvidia.com/gpu should show capacity ≥1.
• Check that the manifest's resources.requests.nvidia.com/gpu isn't asking for more GPUs than exist.

t2v-transformers runs but inference is slow¶

kubectl -n ai-workloads exec deploy/t2v-transformers -- nvidia-smi — if no GPU is listed, the container fell back to CPU. Usual cause: the NVIDIA runtime isn't the default. Either set spec.runtimeClassName: nvidia in the deployment or fix the K3s NVIDIA runtime config (see nvidia-device-plugin-daemonset.yaml).

Weaviate is up but curl localhost:30080/v1/.well-known/ready fails¶

If you used the docker-compose stack instead of K3s, the port is 8080, not 30080. Either fix the URL or set the env override:

export FRACTAL_RAG_WEAVIATE_HTTP_PORT=8080
export FRACTAL_RAG_WEAVIATE_GRPC_PORT=50051

Schema¶

schema bootstrap returns Unsupported vectorizer module¶

config.py defaults to text2vec-transformers. If your Weaviate doesn't have that module enabled, the bootstrap fails fast. Re-check the ENABLE_MODULES env var on the Weaviate pod / container.

Second schema bootstrap is not idempotent¶

That means the prior run partially failed and left only one of the two collections. Inspect:

fractal-rag status        # one of the collections will show null

Either delete the orphan and rerun, or use the destructive flag:

fractal-rag schema bootstrap --drop

Ingest¶

fractal-rag pull fails with gocmd: not found¶

gocmd isn't on PATH for the user running the CLI. Install per GoCommands docs and make sure ~/.irods/ is initialized (gocmd ls /iplant/home/<you>).

pull succeeds but ingest --source json reports parsed_docs: 0¶

The corpus path doesn't match what ingest.discover_jsonl is looking for — specifically, any *.jsonl whose path contains embeddings/. Either move your file under embeddings/ or override the default corpus_root to point at the JSONL's parent:

export FRACTAL_RAG_CORPUS_ROOT=/where/you/actually/put/it

ingest --source docs (or notebooks) reports parsed_docs: 0¶

The markdown source walks <repo_root>/docs/**/*.md, and repo_root defaults to the K3s host path /opt/tswetnam/github/fractal-notebooks (config.py). On a laptop, a CI runner or any other checkout location that directory does not exist, and rglob on a missing path is not an error — it silently yields zero documents. Point the setting at your checkout:

export FRACTAL_RAG_REPO_ROOT=/path/to/your/fractal-notebooks   # or $PWD from the repo root
fractal-rag ingest --source docs

The notebooks source scans <repo_root>/docs/**/*.ipynb as well (skipping the generated per-site 3DEP notebooks), so the same FRACTAL_RAG_REPO_ROOT covers it; FRACTAL_RAG_CORPUS_ROOT (the iRODS mirror) only adds the mirrored copies and may be absent. Do not point CORPUS_ROOT at the checkout: that tree is scanned without the site's exclusion rules. To make the setting stick, put it in a gitignored .env at the repo root together with any port overrides; Settings reads .env from the directory you run fractal-rag in, so run the CLI from the repo root.

Re-index after a reorganisation¶

Object UUIDs are derived from file paths and ingest upserts without ever pruning, so moving, renaming or deleting pages (as the docs reorganisation into Foundations, Research, Notebooks, Apps and Knowledge tabs did) leaves stale objects under the old paths. Rebuild from scratch:

export FRACTAL_RAG_REPO_ROOT=$PWD        # the checkout whose docs/ you want indexed
fractal-rag schema bootstrap --drop      # destructive: drops and recreates both collections
fractal-rag ingest --source docs
fractal-rag ingest --source notebooks
fractal-rag status

Expect the Concept count to be unchanged and the FractalArtifact count to drop to the current page set. Searching for text that only existed on a deleted page should then return no hits.

Ingest is dropping a lot of objects¶

Watch failed_chunks per source in the JSON report that fractal-rag ingest prints (the last run is also saved to ~/.cache/fractal-rag/state.json; fractal-rag status only prints collection counts) and tail the failures file:

tail -f ~/.cache/fractal-rag/failed_objects.jsonl

Common causes:

• Embedding sidecar saturated: GPU OOM or 429 backoff in the sidecar log. The ingest sends one chunk at a time, so this points at an undersized pod memory limit rather than at load (FRACTAL_RAG_BATCH_CONCURRENCY is read but not used by the current loop).
• Property too large: a single chunk above ~64 KiB will fail with a Weaviate validation error. Reduce chunk_max_tokens or fix the upstream parser that's emitting a giant blob.
• Concept reference unknown: an artifact references a Concept UUID that wasn't created in pass 1. This shouldn't happen with the deterministic uuid5(...) scheme; if it does, rerun pass 1 (fractal-rag schema bootstrap + reseed).

status shows fewer json objects than inserted_chunks¶

Object UUIDs derive from source_uri plus the chunk index, so two records with the same source_uri overwrite each other while the loop counts both as inserted. The json parser keys records on the Constellate id (a Portico ark:// identifier or a JSTOR stable URL), which is unique across the export; before 2026-10-25 it keyed them on the DOI, and DOIs are shared by book chapters, reprints and corrigenda: 1,125 of the 50,000 records repeated an earlier DOI (48,875 distinct), so that run reported 100,012 inserted chunks and status showed 97,762 json objects. An index built with the old parser must be rebuilt, because every json UUID changes:

fractal-rag schema bootstrap --drop      # destructive: drops and recreates both collections
fractal-rag ingest --source all

Deleting only the json objects (delete_many filtered on source_type == json_corpus, repeated until matches is 0 because the server deletes at most 10,000 objects per call) and re-running ingest --source json saves the ten minutes the docs and notebooks take, but only on Weaviate 1.35.3 or newer: on 1.35.0 the bulk delete zeroed every keyword score (see Keyword scores are 0). On the DGX Spark the json re-ingest took two hours and left exactly 100,012 json objects for 100,012 inserted chunks: an anchor chunk and one content chunk for each of the 50,000 records (12 long records get a second content chunk).

Ingest takes forever on the JSONL corpus¶

The 50K-article JSONL is ~17 GB and dominates wall clock. Options:

• Ingest a slice first: fractal-rag ingest --source json --limit 5000.
• Run two sidecar replicas: kubectl -n ai-workloads scale deploy/t2v-transformers --replicas=2 took a DGX Spark from 8 to 14 chunks per second (the full corpus then takes about two hours). FRACTAL_RAG_BATCH_SIZE is not consulted by the current one-chunk-at-a-time loop, so raising it changes nothing.
• Run the ingest in screen/tmux; it's resumable because the deterministic UUID scheme upserts rather than appends.

MCP / Claude Code¶

/mcp doesn't list fractal-rag¶

• You're not in the repo root, or the repo's .mcp.json isn't being picked up. Run cat .mcp.json to confirm.
• Or fractal-rag isn't on PATH for the Claude Code process. In a fresh terminal: which fractal-rag. If empty, re-run pip install -e . and reopen Claude Code so it picks up your shell's PATH.

MCP server starts but returns weaviate_error: connection refused¶

• Weaviate isn't running on the host/port the server expects. Check FRACTAL_RAG_WEAVIATE_HTTP_PORT — defaults to 30080 (K3s NodePort). Local docker-compose users need 8080.
• The gRPC port is configured separately (FRACTAL_RAG_WEAVIATE_GRPC_PORT). Weaviate v4 SDK uses both REST and gRPC.

MCP tool returns not_found for a concept you "know" is there¶

Concept names are case-sensitive in artifacts_by_concept and get_concept. Use list_concepts() or search_concepts(query) first to find the canonical name. The agent already does this — if you're hitting this manually from the CLI, the same rule applies.

SSH-tunneled MCP server (remote VM) drops connection¶

• Use keys-only auth; a password prompt corrupts the stdio stream.
• Add -o ServerAliveInterval=30 to the SSH args in .mcp.json to keep long retrievals alive.
• Pre-warm the venv: SSH connection setup + Python import can take ~3 s on cold start, which Claude Code may flag as a startup timeout. Bumping the MCP server start timeout (in Claude Code settings) helps.

Agent quality¶

Agent says "I couldn't find anything" but the topic obviously exists¶

1. Check the index actually has content for that source type:

   fractal-rag status
   

2. Run the same query directly against the index:

   fractal-rag query "<the question>" --limit 10
   

3. If the CLI returns hits but the agent doesn't, the agent's tool selection is wrong. Re-prompt with an explicit nudge: "Use search_artifacts with target_vector='content_vec' and alpha=0.3 for that question." If that fixes it, codify the rubric in .claude/agents/fractal-explorer.md.

Agent invents citations¶

The agent shouldn't do this — the citation discipline section of the subagent file forbids it. If it slips, two fixes:

• Tighten the prompt: add an explicit "If a tool returned no objects, say so plainly and STOP — do not synthesize." line.
• Verify the returned source_uri actually resolves (Read it from disk). The agent has the Read tool for exactly this.

Re-embedding after a model swap takes too long¶

After swapping the embedding model image, you need to re-vectorize every object. The reusable trick is to scope the schema drop to artifacts only, keep concepts (they're tiny), and rerun ingest:

python -c "
from fractal_rag.client import client_session
with client_session() as c: c.collections.delete('FractalArtifact')"
fractal-rag schema bootstrap
fractal-rag ingest --source all

Still stuck?¶

• Check kubectl -n ai-workloads get events --sort-by=.lastTimestamp | tail for cluster-level signals.
• Read the design spec — the "Risks / mitigations" section calls out known weak spots.
• File an issue with the failing CLI command, the relevant log snippet, and the output of fractal-rag status.