Six new open-source tools to put your own documents in front of an LLM
Six OSS Hunter listings cover the path from PDF to answer: opendataloader-pdf extracts, chonkie chunks, cocoindex keeps the index fresh, PageIndex retrieves without a vector DB, txtai does it in one process, and llm-d serves the model.

Start at the PDF. Every document AI stack begins with getting text out of the file, and the failures you blame on the model later usually started there.
That is the reader need behind this list. You have documents you cannot hand to someone else's API: contracts, clinical notes, board packs, internal manuals. You want an AI that can answer questions about them, running on software and hardware you control. Six projects added to the OSS Hunter directory on 18 November 2026 cover that path end to end, from parsing a PDF to serving the model that writes the answer.
All six are MIT or Apache-2.0 licensed, all six have live OSS Hunter listings linked below, and the factual claims come from each project's own repository, release notes and changelog, checked on 18 November 2026. Star counts are a snapshot from that date. Prices in the directory are 0 for all six, since they are open source, with one exception worth reading about further down.
The shape of the stack
Six projects, one job. The pipeline has five stages, and you may need one, two or all of them.
- Extract text from the source files.
- Split that text into units a model can use.
- Keep the index current as sources change.
- Retrieve the right parts at question time.
- Serve the model that turns retrieved context into an answer.
opendataloader-pdf and chonkie handle steps one and two. cocoindex handles step three. PageIndex replaces step four entirely with a different retrieval model. txtai collapses steps two, four and five into one process. llm-d handles step five when you have outgrown one machine.
opendataloader-pdf: get the text out first
OSS Hunter listing | GitHub | Java | Apache-2.0 | 30,331 stars
opendataloader-pdf is a PDF parser aimed at AI pipelines. It turns a PDF into Markdown, JSON with bounding boxes, or HTML, with reading order and table detection. There are two modes. A deterministic local mode runs on your own machine, and an AI hybrid mode sends only the hard pages to an AI backend. The project states that simple pages stay local and take about 0.02 seconds, while complex pages route to AI for higher table accuracy. OCR is built in for scanned documents, and hybrid mode also handles LaTeX formulas and generates descriptions for pictures and charts.
Use it when the PDF is the front door of your pipeline. If a contract, an invoice or a scanned report has to become text an LLM can read, this is the step that has to be right, because every later stage inherits its mistakes.
There is a second job here that is unusual to find in an open source tool. The project auto-tags untagged PDFs into screen-reader-ready Tagged PDFs, built against the PDF Association specification and validated with veraPDF, the validator maintained by Dual Lab. The license is Apache-2.0 from v2.0 onward, and versions before 2.0 were MPL-2.0. Export to PDF/UA-1 or PDF/UA-2 is an enterprise add-on, so auto-tagging is the open source half of that job and not the whole of it.
The accuracy number on the listing, 0.907 overall and 0.928 for tables across 200 real-world PDFs, is the project's own benchmark. The listing does not link the benchmark data, so read it as a project claim rather than an independently checked result. That is normal for parser leaderboards, and it is worth running five of your own worst documents through it before you plan around it.
It shipped v2.5.22 on 12 November 2026, with fixes for CLI stdout handling, column reading order, the PDF 2.0 tagged-PDF namespace and Node CLI exit codes.
Skip it when your documents are already HTML or Markdown, or when PDF/UA export is the requirement, because that part is the paid tier.

chonkie: split the text without adopting a platform
OSS Hunter listing | GitHub | Python | MIT | 4,840 stars
chonkie is a small Python library for splitting text into chunks. The range is wider than the usual token and recursive chunkers: there is a SIMD-accelerated fast chunker, semantic chunking, late chunking, code chunking, a table chunker and an LLM-driven chunker, plus refineries that post-process chunks and porters that write them straight into a vector store. The docs live at docs.chonkie.ai.
Use it when you have decided to go with chunk and embed, and you want the chunking step to be a dependency you can read rather than a subsystem you have to adopt. It installs as a library, and it also runs as a local API server with interactive documentation at /docs and workflow configurations stored in a local SQLite database.
One caveat on numbers. The project README says 32+ integrations across tokenizers, embedding providers, LLMs, refineries, porters and vector databases, while the OSS Hunter listing copy says 45+ integrations in one line and 10+ vector stores in another. The exact count is not settled between the sources I could read. Treat "dozens of integrations" as the safe summary and check the README for the specific one you need.
No tagged release landed between 11 and 18 November 2026. The latest is v1.7.0 from July 2026, which added LiteParse integration and exposed a component registry. The directory snapshot records a push on 18 November 2026, so main is still moving even though nothing shipped this week.
Skip it if you want no chunking stage at all, or if you need a pipeline engine rather than a library you call yourself.

cocoindex: stop re-indexing everything every night
OSS Hunter listing | GitHub | Rust | Apache-2.0 | 11,908 stars
cocoindex is an incremental indexing engine, a Python API over a Rust core. You declare what the target should look like as a function of the source, and it recomputes only the difference when a source changes. For a document pipeline that means re-embedding and re-indexing the one PDF that changed rather than the whole corpus.
Use it when your corpus moves. Codebases, meeting notes, inboxes, Slack channels, PDFs and videos are all listed as sources. The trade-off is that you are adopting a data framework with its own concepts, its own configuration and a Rust core, rather than calling a library inside a script you already have. That pays off when re-indexing cost dominates your bill. It is overkill for a fixed set of documents that you load once and keep.
The project released v1.2.0 on 9 November 2026, just before this window, adding a Discord source connector and experimental data dependencies for memoized functions. No new release landed between 11 and 18 November, but the project published a post on 17 November on recursive tasks, a concurrency cap, and avoiding deadlock in large scale agentic systems.
Skip it when your corpus is static, or when you would rather own the refresh logic in the scheduler you already run.

PageIndex: retrieve by reasoning through the document, not by similarity
OSS Hunter listing | GitHub | Python | MIT | 40,461 stars
PageIndex takes the opposite route from the other five. Instead of embedding chunks, it builds a hierarchical tree index of each document and lets an LLM reason through that tree to find the pages that answer a question. No vector database, no chunking, and the retrieval trail is a path through the document that you can show a user.
Use it when the documents are long and structured: financial reports, legal contracts, regulatory filings, technical manuals, medical literature. The project's argument is that similarity search returns what is similar rather than what is relevant, and that reasoning over a tree is closer to how a person reads a long report and finds the right section.
It was the busiest of the six this week. v0.6.0 landed on 14 November with the PageIndex Agent, the Page Search and Page Extract APIs, collections with metadata, citation formats and JS SDK 0.9.0. v0.7.0 followed on 15 November, and v0.7.1 on 17 November, adding a unified local and cloud client, the Flash engine as the default for local index generation, agent tool integration for MCP and for OpenAI and Anthropic tool contracts, typed config shapes and JS SDK 0.10.0.
On cost, the SDK runs locally with your own LLM API key, and the hosted service is priced separately. The changelog records a pricing change on 18 November 2026 where outline-free uploads cost $0.10 per 1,000 pages. So the "free" reading of this listing applies to the code, not to the hosted service.
The headline claim on the listing, that the Page Search API cuts token cost on document question answering by 97%, is the project's own benchmark. The project says the full results, data and reproduction instructions are in its benchmark repository, so you can check the number yourself.
Skip it if your corpus is a few thousand short pages, where chunk and embed is simpler and cheaper to operate, or if you need retrieval to work with no LLM call at query time. This is a reasoning-based design, so a model is still in the loop when a question arrives.

txtai: one process instead of four services
OSS Hunter listing | GitHub | Python | Apache-2.0 | 13,610 stars
txtai is the do-not-assemble-the-pieces-yourself option. Its core is an embeddings database that is a union of sparse and dense vector indexes, graph networks and relational databases. On top of that sit workflows, agents, RAG pipelines, and web and MCP APIs, with bindings for JavaScript, Java, Rust and Go.
Use it when you want one deployable process instead of four. It is Python 3.11 and up, built on Hugging Face Transformers, Sentence Transformers and FastAPI. Agents are built on the smolagents framework, and the project supports Hugging Face, llama.cpp, and OpenAI, Claude or AWS Bedrock through LiteLLM. That last point is what makes it fit a document AI stack specifically: the same code can point at a local llama.cpp model or a hosted API without a rewrite.
No release landed in the window. The latest is v9.14.0 from 6 October 2026, but the commit log shows active work on 11, 12 and 13 November, mostly SQL parser fixes, workflow fixes and HTML extraction fixes.
Skip it when you want control over each stage, or when a Python-only stack is a constraint for part of your team. The breadth is the point here, and breadth means a larger framework to learn.

llm-d: the layer for when one box is not enough
OSS Hunter listing | GitHub | Python | Apache-2.0 | 5,295 stars
llm-d is a different layer from the other five. It is a distributed inference serving stack for Kubernetes, and it sits above model servers such as vLLM and SGLang rather than replacing them. What it adds is orchestration: prefix-cache aware and load-aware routing, KV-cache management with tiered offload to CPU or disk, prefill and decode disaggregation, agent-aware serving that reuses context across turns, multi-tenant quality of service, zero-downtime upgrades and SLO-aware autoscaling.
Use it when a document AI has stopped being one box. If you have moved from a script to a team, from one model to several, or from occasional questions to real traffic, this is the layer that stops every request queueing behind a single GPU. It is also the layer that makes a reasoning-based retriever like PageIndex affordable, because that design spends an LLM call on each retrieval.
It is a CNCF Sandbox project, accepted on 12 March 2026, founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. No release landed in the window. The latest is v1.0.0 from 20 October 2026. The commit log shows work on 11 and 13 November, including GKE overlays and nightly end-to-end lanes for fast model actuation and an AMD CI kernel-config fix.
The listing also notes a Google Cloud integration of llm-d 1.0 into Google Kubernetes Engine. I could not confirm that on a primary source in this run, so treat it as the listing's claim until you see it on a Google Cloud page.
Skip it unless you already run Kubernetes and already run model servers. On a single machine this is the wrong tool, and a local llama.cpp server is the honest answer.

The rest of the field moved too
These six are today's OSS Hunter batch, but they are not alone in the category. Elsewhere on the open web in the same week: Marker shipped v2.2.0 on 12 November with an in-process ONNX layout and OCR engine, a web viewer and an agent-ready CLI, and reports 88.9% on OmniParseBench in balanced mode, a project benchmark. Docling released v2.150.1 on 14 November. LlamaIndex v0.14.26 shipped on 13 November with a fix for langchain-community breaking the langchain bridge imports. LangChain 1.5.0 and langchain-core 1.8.0 landed on 11 November and dropped Python 3.10 support.
If you are choosing a parser, that list is the field you are choosing between. None of these are in today's OSS Hunter batch, so check the directory before you plan around them.
Three ways to assemble this
- Smallest thing that works. opendataloader-pdf to get the text, chonkie to split it, your own vector store. Add txtai if you want the store, the retrieval API and the server in one process.
- No chunking at all. opendataloader-pdf, or whatever parser you already run, into the PageIndex SDK in local mode with your own model key.
- Production. cocoindex for continuous indexing, PageIndex or txtai for retrieval, llm-d for serving once the traffic justifies it.
Before you commit
Test the parser on your own worst document, not on the vendor's sample. Parser accuracy is the ceiling on everything downstream, and a leaderboard number from the project itself is a starting point, not a decision.
Decide where the model call happens. PageIndex reasons with an LLM at query time. If that has to happen offline or on a restricted network, that rules it out, and chunk-and-embed with a local model is the design that still works.
Check release cadence, not just stars. Two of the six shipped nothing between 11 and 18 November (chonkie and llm-d), and PageIndex shipped three releases in four days. Both patterns are fine, but they mean different things about what you are depending on.
The takeaway
The pipeline is short and each part has one job. Parse well, split well, keep the index fresh, retrieve with a method that fits the shape of your documents, and serve the model only once you need to. opendataloader-pdf and chonkie are the two to install today if you want text you can trust. PageIndex is the one to watch if your documents are long reports rather than web pages. txtai is the one to pick if you would rather not run five services. llm-d is the one you will need later, and probably not this week.
Pick the narrowest thing that solves the actual problem, and measure it on your own documents.