https://gitlab.synchro.net/main/sbbs/-/commit/a3550af9d37165b25bacc92d
Modified Files:
exec/llm_index.js
Log Message:
llm_index.js: cap index size below SpiderMonkey's max string length
The index is written with one JSON.stringify() and read back by
chat_llm.js with one File.read() + JSON.parse(), so it can never exceed SpiderMonkey 1.8.5's 2^28-1 char string limit. A large enough crawl (e.g.
a raised index_max_chunks over big DOVE-Net subs) died at write time with
a bare "InternalError: allocation size overflow", which reads like a heap problem; raising the JS heap (see #1291) only lets the crawl get further.
build_index() now keeps a running estimate of the serialized size and
stops adding chunks before 240 MiB, warning how many were dropped and
what to change. Chunks arrive in source priority order (msgs
newest-first), so the tail is the least valuable. write_index() also
catches a JSON.stringify() failure, removes the .tmp file and reports an actionable error.
Below the cap the output is byte-identical to before.
Co-Authored-By: Claude Opus 5.5 <
noreply@anthropic.com>
---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net