GTM Engineering: Why I Ship an llms.txt Next to My Sitemap
2026-01-10 · 4 min read
Traditional technical SEO — sitemaps, JSON-LD structured data, OpenGraph tags, Lighthouse scores — is still table stakes. But a growing share of discovery now happens through AI systems that summarize or cite a site rather than sending a click.
What llms.txt actually does
An llms.txt file gives AI crawlers a clean, curated map of a site's most important content in plain language, separate from the messier HTML a traditional crawler parses. It's not a replacement for good structured data — it's a companion for a different kind of consumer.
Where this fits in a GTM engineering stack
On CodeMyFYP's production domains, this sits alongside the usual pipeline: dynamic sitemap generation in Next.js, Organization, BreadcrumbList, and FAQPage JSON-LD, and Lighthouse/PageSpeed budgets enforced in CI. The llms.txt file is one more artifact generated from the same content source of truth, so it never drifts out of sync with the actual site.
The measurable part
The honest answer is that attribution for "AI crawler discoverability" is still immature compared to classic SEO analytics. What is measurable today: faster indexing after sitemap changes, cleaner structured-data validation in Search Console, and Lighthouse scores that stay green as the codebase grows — the fundamentals that make any future discovery channel, human or AI, easier to build on top of.