Building a Semantic Data Store: From Markdown Structure to Navigable Knowledge

Evaluation note: The retrieval and navigation results described below are from a project-specific evaluation set. They are not a general benchmark of model reasoning or retrieval performance.

The limits of flat documentation

For the modern developer, Markdown is an indispensable tool. It is the lingua franca of documentation, READMEs, and personal knowledge bases. It strikes a perfect balance between human readability and machine portability. However, there is a fundamental disconnect in how humans and machines perceive a Markdown file.

When a human reads a document, they don't see a stream of characters. They see a hierarchy. We perceive headings as boundaries that define the scope of an idea. When we see a # heading, our brains create a mental folder; when we encounter a ## sub-heading, we open a sub-folder. We navigate documents spatially and logically, jumping from a table of contents to a specific section, and then reasoning about how that section relates to the overall theme of the document.

To a computer, however, a Markdown file is stubbornly flat. A heading is just another string of text that happens to start with a hash symbol. To a standard search algorithm, the word "Installation" in a heading is treated with the same weight as the word "installation" buried in a paragraph of technical jargon. The structural intent—the a priori organization the author intended—is completely lost.

I wanted to break this invisible wall. I set out to build a system that doesn't just render Markdown, but understands it. My goal was to evolve a simple document viewer into a Semantic Data Store: a system that captures the inherent hierarchy of a document and provides an AI agent with a high-fidelity "map" to navigate it.

This is the engineering chronicle of that evolution—a journey from a basic Table of Contents viewer to a fully autonomous AI Librarian.

Phase 1: The Foundation—The Markdown TOC Viewer

Every complex system begins as a solution to a simple problem. In my case, that problem was the "scroll of death." When dealing with massive technical specifications, the standard browser experience is a nightmare of endless scrolling and lost context.

I began by building a high-performance viewer using FastHTML. The primary objective was to create a synchronized, hierarchical Table of Contents (TOC) that lived in a sticky sidebar. While this sounds trivial, the implementation required solving several "edge-case" problems that are common in real-world documentation.

First, I had to handle the duality of Markdown headings. I implemented support for both ATX-style headings (the common # prefix) and Setext-style headings (where headings are defined by underlining text with === or ---).

Second, I had to solve the problem of Deterministic Slugging. In a large document, it is common for authors to use the same heading multiple times (e.g., every chapter might have a "Summary" section). A naive slug generator would create duplicate IDs, breaking the internal linking. I developed a SlugManager that tracks heading occurrences and appends numeric suffixes—transforming three "Summary" headings into summary, summary-1, and summary-2.

The result was a polished, interactive UI. I integrated KaTeX for rendering complex mathematical formulas and implemented a draggable resizer bar for the sidebar, allowing users to customize their reading experience.

But as I used the tool, I hit a conceptual ceiling. The app was a "navigational map." It could tell you where a section started and help you jump there, but the app didn't actually know what was inside those sections. It was treating headings as markers on a map, but it had no concept of the territory.

Phase 2: The Semantic Shift—From Markers to Containers

The most significant pivot in the project occurred when I shifted my mental model of a document. I stopped treating headings as markers and started treating them as containers.

In a standard viewer, a heading is just a line of text. In the Semantic Data Store, a heading becomes the root of a SemanticNode. I introduced a formal data model where every heading "owns" the content that follows it. This meant that everything—every paragraph, every table, every bulleted list—belonged to a specific node until another heading of equal or higher level appeared.

To achieve this, I replaced the flat extraction logic with a segmenting engine. This engine performs a linear pass through the Markdown, maintaining a stack of active nodes. When the engine encounters an H1, it creates a root node. When it hits an H2, it creates a child of that H1. If it then encounters an H3, it nests it further.

The complexity arises when authors skip levels (e.g., jumping from an H1 directly to an H3). The segmenting engine was designed to handle this gracefully, automatically attributing the H3 to the most recent higher-level ancestor, ensuring the structural integrity of the tree regardless of the author's formatting quirks.

This transformation changed the application from a viewer into a knowledge engine. The app now possessed a structured graph of the document. This unlocked capabilities that were previously impossible:
- Atomic Retrieval: The ability to extract exactly one section of a document without parsing the rest.
- Sectional Focusing: Rendering only a specific branch of the document tree.
- Structural Analysis: Programmatically determining the depth and breadth of a document's organization.

Phase 3: Solving Persistence—The "Hidden Graph"

An in-memory tree is a powerful prototype, but it is ephemeral. To transform this into a true "Data Store," I needed a persistence layer that could survive server restarts and scale to thousands of documents.

I found myself at a classic architectural crossroads. There are three primary ways to store a recursive tree, and each has a distinct trade-off:
1. JSON Serialization: This is the "simple" path. You convert the SemanticNode tree into a nested JSON object and save it to disk. While human-readable and easy to implement, it is catastrophically inefficient for large datasets. To retrieve a single sub-section, you must load and parse the entire document into memory.
2. RDF/Triplestores: This is the "academic" path. In a Resource Description Framework (RDF), every relationship is stored as a triple: Subject → Predicate → Object. This is the gold standard for semantic web data and knowledge graphs, but the tooling is complex and the learning curve is steep.
3. SQLite: This is the "pragmatic" path. By using a relational table with a parent_id column, you can flatten a tree into rows.

I chose SQLite, but I did so with a specific strategic insight: a relational table with parent-child links is actually a graph in disguise.

In graph theory, a node is a vertex and a relationship is an edge. In my SQLite schema, the id is the vertex and the parent_id is the edge. By adopting this "Hidden Graph" architecture, I gained the best of all worlds. I had the production-grade stability, indexing speed, and reliability of SQL, but I maintained a direct, lossless migration path to a full RDF Knowledge Graph.

If the project ever needs to evolve into a global semantic network, the transformation is a trivial SQL query: NodeID → hasParent → ParentID. This commitment to "Data Portability" ensured that the system was not locked into a specific database paradigm, but was instead organized in a way that any future system could understand.

Phase 4: The "Squeeze" Method—Adding AI Summaries

With a structured tree and a persistent store, I had the "Body" of the system. But as I began integrating Large Language Models (LLMs), I hit the "Context Window" wall.

The problem is a common one in RAG (Retrieval-Augmented Generation): if you ask a question about a 100-page manual, you cannot send the entire text to the LLM. If you do, you waste thousands of tokens, increase latency, and risk "lost-in-the-middle" syndrome, where the AI misses the answer because it is buried in too much noise.

To solve this, I developed what I call the "Squeeze" Method of semantic summarization. Instead of summarizing the document as a whole, the system performs a bottom-up recursive synthesis:
1. Leaf Summarization: The system identifies the "leaf nodes"—the most granular sections at the bottom of the tree. It sends their content to the LLM to generate a concise, one-sentence summary.
2. Recursive Synthesis: The system then moves up to the parent nodes. A parent's summary is not just based on its own text, but is a synthesis of the summaries of all its children.
3. The Root Distillation: This process continues upward until it reaches the root node. The root's summary becomes a high-level distillation of the entire document's essence.

This creates a Summary Map. Imagine a map of a city where you can see the whole city (Root), then zoom into a neighborhood (Parent), then a specific street (Child), and finally a specific house (Leaf).

These summaries are cached in the SQLite database, meaning they are generated once and reused forever. I integrated these summaries into the UI as tooltips; when a user hovers over a heading in the TOC, they see an AI-generated snapshot of what that section contains. This provides a "low-resolution" view of the document that allows both humans and AI agents to navigate the content without ever having to read the full text until the final moment of retrieval.

Phase 5: The AI Librarian—Agentic Tree Search

The final and most ambitious phase was the transition from Keyword Search to Agentic Navigation.

Traditional search is a blunt instrument. You type "Installation," and the system returns every node that contains that word. But humans don't find information in a library by scanning every page for a keyword; they use the index and the table of contents to reason their way to the answer.

I built the TreeSearchAgent to act as an "AI Librarian." The agent does not simply search; it navigates. It operates on a Reasoning-Action Cycle, using a set of specialized tools to interact with the Semantic Data Store:
- list_children(node_id): "I'm looking at the 'Getting Started' section. Let me see what sub-topics are inside."
- get_content(node_id): "The 'Prerequisites' subsection looks like it has the answer. I'll read the full text now."
- move_to_parent(node_id): "I've hit a dead end in this branch. I'll go back up and try the 'Advanced Configuration' path instead."
- final_answer(text): "I have successfully navigated to the correct node and extracted the answer."

To make this process transparent, I implemented a Reasoning Trace. When a user asks a question, the UI displays the agent's internal monologue and its movement through the tree in real-time:
- "I see 'Architecture' as a top-level node. I will list its children..."
- "The 'Database Layer' child seems relevant. I will retrieve its content..."
- "I have found the answer in the 'indexing' subsection."

To further refine this, I implemented Task Decomposition. The agent no longer just reacts; it plans. Upon receiving a complex query, the agent first creates a mental roadmap (e.g., "Step 1: Find the API section → Step 2: Locate Authentication → Step 3: Extract the Token limit"). This prevents the agent from "guessing" and ensures a methodical, audit-able path to the truth.

Quantifying Intelligence: The Performance Analysis

Building a reasoning agent is one thing; proving it actually reasons is another. To validate the Semantic Data Store, I implemented a rigorous evaluation harness inspired by the Finance QA Benchmark used by state-of-the-art systems like PageIndex.

I created a "Golden Set" of 13 curated queries—ranging from simple fact retrieval to complex, multi-step synthesis and "negative" cases where the answer is intentionally missing from the document. Each query was mapped to specific Target Nodes that a perfect agent must visit to find the truth.

The system was evaluated on three primary metrics: 1. Node Hit Rate: Did the agent visit the required structural markers? 2. Path Efficiency: How close was the agent's path to the optimal route? 3. Recovery Rate: How successfully did the agent use move_to_parent to backtrack after a wrong turn?

The results were definitive: On the project-specific 13-query evaluation set, the strongest result was for the defined complex-query subset. These results are directional engineering evidence rather than a general benchmark; the evaluation should be repeated on larger, independently reviewed corpora before drawing broader conclusions. Negative cases also showed improved abstention when the relevant information was not found.

Metric	Result	Analysis
Average Hit Rate	88.46%	High accuracy across all query types.
Complex Query Hit Rate	100% on the defined subset	Result from the project-specific evaluation set.
Average Efficiency	82.82%	Precise navigation with minimal wandering.
Recovery Rate	100% on the defined subset	Result from the project-specific evaluation set.

Beyond a Single Document: The Multi-Doc Corpus

The final evolution of the Semantic Data Store was scaling from a single-document tool to a multi-document library. By updating the SQLite schema to include doc_id as part of the primary key, the system can now index an unlimited number of Markdown files in a single database.

I added a direct file-loading endpoint, allowing the app to read .md files from the local disk, process their semantic structure, and save them as distinct entities. This transforms the tool from a document viewer into a Semantic Corpus Manager. You can now point the AI Librarian at a directory of a hundred different manuals, and it can reason across all of them, jumping from a "Setup" section in one file to a "Troubleshooting" section in another.

Conclusion: The Body and the Brain

The journey of the Semantic Data Store has been a study in the transition from structure to meaning.

In the beginning, we built the Body: the SemanticNode tree, the segmenting engine, and the SQLite persistence layer. This gave the system the physical capacity to hold and organize information. Then, we built the Brain: the recursive summarization logic and the Agentic Tree Search. This gave the system the cognitive ability to reason about that organization.

By moving beyond the flat-text paradigm, we have turned unstructured documentation into a queryable, reason-capable knowledge base. We are no longer searching for strings; we are navigating ideas. The Semantic Data Store proves that when you provide an AI with a high-fidelity map of information, it stops being a chatbot and starts being a librarian.