llms.txt Implementation on Astro + Cloudflare Workers: What We Actually Shipped

August 21, 2026 ai-search

Most of what’s written about llms.txt is agencies paraphrasing one arXiv proposal at each other. Almost none of it shows a real file on a real server with real response headers. This post is the other thing: exactly what we added to atiqlabs.com — an Astro 5 static site served from Cloudflare Workers — what we shipped, why we made the choices we made, and how to verify yours works.

Tested on: Astro 5.18.2 (static output), Cloudflare Workers assets runtime with compatibility_date = "2026-08-01". Headers below captured 2026-08-22 via curl against production.

What llms.txt actually is

llms.txt is a proposed convention: a markdown file at /llms.txt that gives language models a curated, link-rich map of your site. It sits alongside robots.txt and sitemap.xml but serves a different job — those files tell crawlers what they may fetch and where pages live; llms.txt tells a model what your site means, in a format it can ingest without fighting nav chrome, scripts, and layout markup.

The spec defines two companion files: /llms-full.txt, which contains the full markdown content expanded inline so a model doesn’t have to follow links at all, and the convention that any page be available as clean markdown by swapping its extension to .md. Adoption is early and voluntary — no major crawler is documented as prioritizing it yet — which is precisely why it’s cheap to implement now and why verification matters more than ceremony.

File structure that works

We settled on this shape after reading the spec and looking at what survives an actual model context window:

/
├── llms.txt        # curated index: blockquote summary + H2 sections + links
└── (llms-full.txt) # optional: full content inline — see note below

llms.txt itself has a required skeleton:

An # H1 with the site/project name — mandatory, first line.
A > blockquote with a one-to-three sentence summary of what the site is.
Zero or more ## H2 sections, each a flat list of [title](url): optional description links.

That’s it. Anything else — deep nesting, tables, prose walls — dilutes the signal. Here’s our complete production file, all 16 lines:

# ATIQ Labs

> Independent AI consulting: on-premises LLM deployment, hybrid AI infrastructure, model fine-tuning, workflow automation, and managed maintenance. Own your token pipeline end to end.

## Services
- [On-Premises LLM Deployment](https://atiqlabs.com/services/on-premises-llm-deployment/): private, self-hosted AI on your own hardware
- [Hybrid & Multi-Provider AI Infrastructure](https://atiqlabs.com/services/hybrid-ai-infrastructure/): unified gateways across local + cloud + APIs
- [Model Fine-Tuning & Customization](https://atiqlabs.com/services/model-fine-tuning/): LoRA training, dataset prep, evaluation
- [AI Workflow & Agent Development](https://atiqlabs.com/services/ai-workflow-development/): production automations on infrastructure you own
- [Managed Maintenance & Support](https://atiqlabs.com/services/managed-ai-maintenance/): updates, monitoring, model refreshes

## Blog
Guides and news on local AI, self-hosted inference, and owning your stack: https://atiqlabs.com/blog/

## Contact
Contact form only: https://atiqlabs.com/#contact

On llms.md vs llms.txt vs llms-full.txt: the spec names .txt; some sites serve the same content as .md. We use .txt only, because Cloudflare will serve either with identical bytes and there’s no evidence models prefer one. We have not shipped llms-full.txt yet — our site is five service pages plus a blog, and every linked target is already fast, clean, server-rendered HTML that parses trivially. A full-content variant earns its keep when you have docs-scale content where following ten links costs a model more than reading one big file. Ship the index first; add -full when your link graph gets expensive.

One requirement worth stating plainly: the body must be markdown, not HTML. No <div>, no entities, no minified anything. If your pipeline emits HTML into llms.txt you’ve shipped a worse version of your homepage.

The Astro implementation

Honest deviation from the usual tutorial script: we do not generate llms.txt from content collections at build time. There’s no integration, no endpoint, no script. Our implementation is a single static file checked into public/:

public/
├── llms.txt      ← hand-maintained, ships verbatim into dist/
├── robots.txt
└── ...

Astro copies public/* into dist/ untouched during astro build, and the Workers assets config ({"assets": {"directory": "./dist"}} in wrangler.jsonc) serves it as-is. That’s the whole mechanism. You can confirm it survived the build with:

astro build && diff public/llms.txt dist/llms.txt && echo ok

Why not generate it? Because our site is small enough that a generator would be indirection without payoff: the service list lives in src/data/services.ts, but the summaries we want in llms.txt are hand-written distillations of that data, not fields we could mechanically extract. Generation starts paying for itself when (a) content collections grow past a screenful or (b) URLs drift out of sync with the file — at that point a small build-time integration that maps collection entries to markdown links is the right move, and it’s maybe forty lines. If you’re there, hook it into astro.config.mjs as an integration writing to dist/llms.txt in astro:build:done. Until then, a static file with a line-item in your deploy checklist is more honest than a script nobody maintains.

What we did wire deliberately: the file’s links point at canonical trailing-slash URLs matching the routes Astro generates (/services/[slug].astro over SERVICES), so nothing 404s under redirect normalization.

Hosting specifics on Cloudflare Workers/Pages

Serving llms.txt from Workers assets mostly just works, but three details are worth knowing before someone asks you why it looks wrong:

Content-type. Cloudflare serves .txt as text/plain. That’s correct per the spec — llms.txt is plain text/markdown, and there is no special MIME type. Real headers from our production file:

$ curl -sSI https://atiqlabs.com/llms.txt
HTTP/2 200
content-type: text/plain
cache-control: public, max-age=0, must-revalidate
etag: "32cf60933f29138924350eadf81e8cf9"
cf-cache-status: MISS
server: cloudflare

If you see text/html here — common when an SPA fallback route or a catch-all _redirects rule intercepts unknown paths — models and validators will treat the file as a page, and some parsers will choke on it. Check for a not_found_handling setting routing to HTML (ours uses "404-page" for genuine misses) and make sure your fallback excludes real files.

Cache behavior. Note max-age=0, must-revalidate with an etag: every request revalidates, unchanged files serve from edge cache (cf-cache-status: HIT on subsequent fetches), and deploys invalidate cleanly because the asset hash changes. For a file this small that’s ideal — stale builds self-correct on the next fetch rather than serving old content for a TTL you’d have to reason about.

Robots.txt conflicts. Our robots.txt is Allow: / with a sitemap pointer, so nothing to reconcile. But if you run a stricter policy, remember that some AI crawlers respect robots.txt before they ever look at llms.txt — a disallow on /*.txt patterns or on the crawler’s user-agent silently cuts off discovery regardless of how good your file is. Audit both together.

Verification

Verification has two tiers, and only one is available on day one.

Tier 1 — mechanical checks (do these now):

# 200 + text/plain, not HTML
curl -sSI https://yoursite.com/llms.txt | grep -iE 'HTTP|content-type'

# body matches what you deployed (catches stale builds)
diff public/llms.txt <(curl -sS https://yoursite.com/llms.txt)

# every link resolves — no 404s hiding in your map
grep -oE 'https://[^)]+' public/llms.txt | while read u; do
  code=$(curl -s -o /dev/null -w '%{http_code}' "$u"); echo "$code $u"
done

All three pass against atiqlabs.com as of 2026-08-22, including the live-vs-repo byte comparison.

Tier 2 — crawler pickup (pending). Whether GPTBot, ClaudeBot, PerplexityBot, and friends actually fetch /llms.txt is an empirical question answered by access logs, and ours haven’t accrued enough post-deploy history to report a number. We’ll update this post with observed fetch counts by user-agent once the log window is meaningful — that’s the honest state of it. Anyone telling you their llms.txt “boosted AI visibility” without UA-level log evidence is selling you something.

Common mistakes

Wrong path. It must be at the domain root: https://example.com/llms.txt. Not /blog/llms.txt, not case-varied (LLMS.txt is a different URL).
HTML instead of markdown. Usually caused by SPA fallback intercepting the path (see above), or by exporting a rendered page instead of source markdown.
Stale builds. The file drifts out of sync with real URLs after a redesign. The diff check against production belongs in your deploy routine.
Link rot. Every URL in the file is a promise. Relative links are fine per the spec, but absolute canonical URLs remove one class of resolution failure.
robots.txt conflicts. A blanket disallow or an aggressive crawl-delay applied to AI user-agents makes the whole exercise moot. Decide the policy for both files in the same sitting.
Treating it as SEO. There is currently no documented ranking effect in any traditional search engine from llms.txt. Its value proposition is machine readability, full stop. Claiming more than that is speculation.

If you want help auditing how machines read your site — llms.txt, structured data, and the infrastructure underneath — that’s most of what we do; start at our services or get in touch.

Building your own AI infrastructure?

Talk to us