Documents & retrieval

The obvious design is: chunk, embed, store vectors, search by cosine similarity. This app doesn't. Upload extracts the text and stores it whole; every query then chunks it on the spot, shortlists with a cheap lexical pass, and lets a reranker pick the winners.

Why: there is no index to keep warm or migrate, rerankers beat embeddings on relevance at this size (per-user sets are hundreds of chunks, not millions), and reranking runs through the AI Gateway key that is already required.

The pipeline

The browser uploads to Vercel Blob directly, then posts the resulting URL to /api/documents, which writes metadata to the document table. The bytes never pass through a route handler — see The upload path for why that matters. Then, at upload:

1. Extract — lib/document-text.ts. attachmentKind classifies the file, pdfText handles PDFs, extractDocumentText is the entry point. Scanned PDFs raise ScannedPdfError rather than returning an empty string. The result is cached whole on document.content. The bytes are fetched back from the store for this, since the upload didn't bring them here.

And at query time, all three in agent/tools/search_documents.ts:

2. Chunk — chunkText from lib/rag.ts, run over document.content on every search. It splits on blank lines, so chunk quality is really the extractor's quality: unpdf returned one undifferentiated paragraph, leaving chunkText nothing to split on but a hard slice every 1400 characters, and 29 of 34 chunks began mid-word. That is why extraction moved to pdf-inspector's Markdown — 13 of 41 on the same paper.
3. Shortlist — lexicalShortlist in lib/rag.ts, a BM25-shaped scorer (IDF weighting, saturating term frequency). When it cannot discriminate, it returns everything rather than an arbitrary slice.
4. Rerank — the shortlist goes to cohere/rerank-v4-fast, which returns the top chunks.

The upload path

The browser uploads straight to the blob store and only then tells the server about it. Files used to be posted to /api/documents as multipart and forwarded on, which put them in a Vercel Function's request body — capped at 4.5 MB. The route claimed 12 MB, so anything between the two failed with a platform 413 before the handler ran, and because that response isn't JSON the composer showed a bare Upload failed (413).

Three pieces, in order:

1. /api/documents/upload-token issues a short-lived upload token. Every check that used to sit next to put() happens here, before the token exists: session, file type, size, and whether the chat or project is yours. It is deliberately not wrapped in authed — the store calls the same route back on onUploadCompleted with no session cookie.
2. The browser uploads, using upload() from @vercel/blob/client. Over 8 MB it goes multipart: parts in parallel, retried individually.
3. /api/documents takes the resulting URL and creates the row.

The ceiling is now 50 MB, and what bounds it is extraction rather than the request body: step 1 of the pipeline fetches the file back into a function and parses it there.

When extraction fails

A file with no recoverable text — a scanned PDF, an empty upload — is stored with status = "error" and the reason on document.error. Search skips it, via an isNotNull(document.content) filter, so it can never contribute an empty chunk to a shortlist.

That exclusion is silent by nature, so the project panel labels the file Not searchable with the reason on hover. Without that the only symptom is an agent that never cites a document sitting in plain sight, which is indistinguishable from a retrieval bug. The fix is to remove the file and upload a version with a text layer; no script involved.

The two constants

const SHORTLIST_KEEP = 100
const MAX_CANDIDATES = 200

Both are measured. Against a 216-passage corpus, comparing each setting's top-6 against the top-6 from reranking everything, SHORTLIST_KEEP = 60 returned only 4 of the 6; 100 is the smallest value that loses nothing. If you change the chunker, re-run that sweep. The number is a property of the corpus, not a universal constant.