karthik.dev
Back to projects
AI

TraceGraph AI

AI-powered trace graph analysis & telemetry diagnostics for LLM architectures

GitHub
PythonNeo4jGraph AnalyticsPlaywrightGitHub APILLM TelemetryTrace Analysis

Overview

TraceGraph AI is an evidence-first AI engineering platform that builds a Neo4j knowledge graph from three real inputs: an explicitly selected public product document, a bounded Playwright crawl of an allowlisted public application, and a real GitHub pull request with every file delta. Designed for AI Engineer diagnostic scenarios where LLM telemetry and trace graphs expose root causes in production systems.

Problem

Debugging LLM architectures in production requires connecting trace data, repository changes, and product documentation — but these are stored in completely separate systems with no unified graph view. Root-cause analysis across these sources is manual and time-consuming.

Solution

A graph construction pipeline that ingests three bounded, real-world sources: a product document (structured knowledge), a Playwright-crawled application (runtime behavior), and a GitHub PR diff (code changes). Neo4j stores the entity graph; analysis layers surface causal relationships between traces, code deltas, and documented behavior.

Architecture

  1. 1Product document ingestion — structured parsing into entity nodes
  2. 2Playwright bounded crawl — allowlisted application behavior capture
  3. 3GitHub API integration — real PR diffs and file delta extraction
  4. 4Neo4j graph construction — entity linking across all three sources
  5. 5Trace analysis layer — LLM telemetry correlation with graph nodes
  6. 6Diagnostic query engine — surfaces root cause paths from failure traces

Technical Challenges

  • Bounding the Playwright crawl to only allowlisted URLs to prevent scope creep in evidence collection.
  • Entity linking across three heterogeneous sources (documents, crawled pages, code diffs) without false relationship edges.
  • Keeping the graph construction deterministic and reproducible for auditable AI engineer diagnostic workflows.

Results

  • Functional AI engineer take-home prototype demonstrating production-grade graph telemetry architecture.
  • Evidence-first design ensures all graph nodes trace back to real, bounded inputs — no hallucinated relationships.