• exec/llm_index.js

    From Rob Swindell (on Debian Linux)@VERT to Git commit to main/sbbs/master on Sun Oct 18 22:28:23 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/a3550af9d37165b25bacc92d
    Modified Files:
    exec/llm_index.js
    Log Message:
    llm_index.js: cap index size below SpiderMonkey's max string length

    The index is written with one JSON.stringify() and read back by
    chat_llm.js with one File.read() + JSON.parse(), so it can never exceed SpiderMonkey 1.8.5's 2^28-1 char string limit. A large enough crawl (e.g.
    a raised index_max_chunks over big DOVE-Net subs) died at write time with
    a bare "InternalError: allocation size overflow", which reads like a heap problem; raising the JS heap (see #1291) only lets the crawl get further.

    build_index() now keeps a running estimate of the serialized size and
    stops adding chunks before 240 MiB, warning how many were dropped and
    what to change. Chunks arrive in source priority order (msgs
    newest-first), so the tail is the least valuable. write_index() also
    catches a JSON.stringify() failure, removes the .tmp file and reports an actionable error.

    Below the cap the output is byte-identical to before.

    Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net