TraceGraph AI: Building a Knowledge Graph for LLM Telemetry Diagnostics

The Problem: Disconnected Telemetry

Debugging a failing LLM system in production means correlating three completely disconnected information sources:

1. Product documentation — what the system is supposed to do
2. Application runtime behavior — what it's actually doing
3. Recent code changes — what changed before it broke

Typically these live in a wiki, a staging environment, and a GitHub PR — with no unified view connecting them. Root-cause analysis is manual, slow, and inconsistent.

TraceGraph AI automates this by building a Neo4j knowledge graph from all three sources, then letting you query causal paths across them.

The Evidence-First Design Principle

The key design decision: every node in the graph must trace back to a real, bounded input. No synthetic data, no hallucinated relationships.

This means:

• Document nodes come only from an explicitly selected, publicly accessible product document
• Application behavior nodes come only from a Playwright crawl of an allowlisted domain
• Code change nodes come only from a real GitHub PR with verified diffs

This isn't just a technical constraint — it's an epistemic one. A diagnostic tool that synthesizes data is no better than asking an LLM to guess.

Source 1: Product Document Ingestion

def ingest_document(url: str) -> list[DocumentNode]:
    """Parse a public product document into entity nodes."""
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    
    sections = []
    for heading in soup.find_all(["h1", "h2", "h3"]):
        section_text = extract_section_text(heading)
        sections.append(DocumentNode(
            id=f"doc:{slugify(heading.text)}",
            label=heading.text,
            content=section_text,
            source_url=url,
        ))
    return sections

Each section becomes a graph node with its content, source URL, and structural position.

Source 2: Playwright Bounded Crawl

The crawl is deliberately bounded — only allowlisted URLs are visited, with a depth limit and page count ceiling.

async def bounded_crawl(
    start_url: str,
    allowlist: list[str],
    max_pages: int = 20,
) -> list[PageNode]:
    """Crawl an allowlisted application and extract behavior nodes."""
    visited = set()
    queue = [start_url]
    pages = []
    
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        
        while queue and len(visited) < max_pages:
            url = queue.pop(0)
            if url in visited or not any(url.startswith(a) for a in allowlist):
                continue
            
            await page.goto(url)
            content = await page.content()
            pages.append(PageNode(url=url, content=content))
            visited.add(url)
            
            # Extract internal links within allowlist
            links = await page.eval_on_selector_all("a[href]", "els => els.map(e => e.href)")
            queue.extend([l for l in links if any(l.startswith(a) for a in allowlist)])
    
    return pages

Source 3: GitHub PR Diffs

def fetch_pr_diffs(repo: str, pr_number: int, token: str) -> list[DiffNode]:
    """Extract file-level diffs from a real GitHub PR."""
    headers = {"Authorization": f"Bearer {token}"}
    files = requests.get(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}/files",
        headers=headers,
    ).json()
    
    return [
        DiffNode(
            filename=f["filename"],
            additions=f["additions"],
            deletions=f["deletions"],
            patch=f["patch"],
            status=f["status"],
        )
        for f in files
    ]

Building the Neo4j Graph

Once all three sources are ingested, they're linked by entity matching — function names that appear in both a document and a diff, UI elements that appear in both a crawled page and a code change.

// Link document sections to code changes by shared entity names
MATCH (doc:DocumentSection), (diff:FileDiff)
WHERE any(entity IN doc.entities WHERE entity IN diff.changed_symbols)
CREATE (doc)-[:RELATES_TO {confidence: 0.85}]->(diff)

Diagnostic Queries

With the graph built, you can ask questions like:

// What documentation sections relate to recently changed files?
MATCH (doc:DocumentSection)-[:RELATES_TO]->(diff:FileDiff)
RETURN doc.label, diff.filename, diff.additions, diff.deletions
ORDER BY diff.additions + diff.deletions DESC

This surfaces the highest-change files alongside their documented behavior — the starting point for root-cause analysis.

What I'd Build Next

The current prototype handles three sources well. Natural extensions:

• LLM trace logs as a fourth source — connecting actual inference traces to the graph
• Automated anomaly detection — flag when a diff changes a function that's central to documented behavior
• Temporal analysis — track how the graph changes across multiple PRs

GitHub: github.com/karthikrshet/tracegraph-ai