Transparency for LLM crawlers

SeoRend is built to improve content legibility for AI systems without sacrificing editorial control. We expose machine-readable signals that allow large language models to verify what they ingest.

Transparency feed

• Generated on each successful render and exposed at /__seorend/transparency.
• Contains checksum, canonical URL, locale, release tag, and rendering engine version.
• Supports HTTP caching headers so crawlers can poll efficiently.

{
  "url": "https://docs.seorend.com/setup",
  "locale": "en",
  "checksum": "sha256-d5f8a8...",
  "rendered_at": "2024-05-21T09:15:33.000Z",
  "release": "docs@2024.05.21",
  "llm_notes": "Prerequisites updated with Ubuntu 22.04 support"
}

Signature validation

Each transparency item is signed with an Ed25519 keypair managed by SeoRend. LLM crawlers can call the /__seorend/keys endpoint to fetch the current public keys and validate the signature field. Rotations are announced via the same endpoint with a 14-day overlap period.

Human-readable audit trail

The Observatory dashboard diff-checks HTML snapshots to highlight changes that may affect generated answers. Editors can annotate renders with intent labels ("typo fix", "policy update"), giving LLM providers more context about the change.

Integration tips

1. Respect rate limits: the transparency feed supports conditional requests with If-None-Match headers.
2. Preserve locale info: use the locale attribute to route content to the correct language model bucket.
3. Report confidence: send checksum and signature data along with your embeddings to maintain provenance.

SeoRend’s mission is to make the web friendlier for both people and intelligent crawlers. Let us know how we can extend the transparency pipeline for your use cases.