Cloudflare Workers Integration

This guide will help you set up prerendering for your site via Cloudflare Workers. After setup, all bot requests will be automatically proxied through SeoRend.

Requirements

• Cloudflare account
• Domain connected to Cloudflare
• SeoRend account

Important: Proxied mode

Your domain in Cloudflare must be in Proxied mode (orange cloud). DNS Only mode doesn’t allow Workers to function.

Step-by-Step Guide

1. Create an API Key in SeoRend

   Log into the SeoRend dashboard and go to the API Keys section.

   1. Click the Create API Key button
   2. Enter a key name (e.g., production or my-site.com)
   3. Click Create
   4. Copy the secret key — you’ll need it in the next step

   The key looks something like: sk_live_abc123def456...

   Tip

   Save the key in a secure location. You can always view it in the dashboard, but it’s better to have a copy.

2. Add Secret to Cloudflare

   Go to Cloudflare Dashboard and select your domain.

   1. In the left menu, select Workers & Pages
   2. Click Manage account-level secrets (or Settings → Variables)
   3. In the Environment Variables section, click Add variable
   4. Fill in the fields:
      • Variable name: SEOREND_API_KEY
      • Value: paste your API key from step 1
   5. Check the Encrypt box (for security)
   6. Click Save

   Note

   The SEOREND_API_KEY secret will be available to all Workers in your Cloudflare account. This is convenient if you have multiple domains.

3. Create a Worker

   1. In Cloudflare Dashboard, go to Workers & Pages
   2. Click Create application → Create Worker
   3. Enter a Worker name (e.g., seorend-prerender)
   4. Click Deploy (creates an empty Worker)
   5. After creation, click Edit code

4. Upload Worker Code

   Replace the contents of worker.js with the SeoRend Worker code.

   Download the ready file:

   Download cloudflare_worker.js

   Or copy the code below. After pasting, click Save and Deploy.

worker.js

// Seorend Cloudflare Worker
// Detects bots and serves pre-rendered content, falls back to origin on errors

// Bot signatures — matched as lowercase substrings of User-Agent
const BOT_SIGNATURES = [
  // Google
  "googlebot", "googleother", "storebot-google", "google-inspectiontool",
  "google-extended", "google-cloudvertexbot", "google-favicon",
  "apis-google", "adsbot-google", "mediapartners-google",
  "feedfetcher-google", "google page speed",

  // Bing / Microsoft
  "bingbot", "bingpreview", "msnbot", "adidxbot", "microsoftpreview",

  // Yandex
  "yandexbot", "yandeximages", "yandexvideo", "yandexmedia",
  "yandexmobilebot", "yandexmetrika", "yandexaccessibilitybot", "yabrowser",

  // Baidu (all variants: image, video, news, mobile, render)
  "baiduspider",

  // Other search engines
  "duckduckbot", "duckassistbot", "applebot", "naver",
  "sogouspider", "360spider", "coccoc", "seznambot", "qwantbot",
  "ecosia", "yahoo", "exabot", "petalbot", "bravebot",

  // AI / LLM crawlers
  "gptbot", "oai-searchbot", "chatgpt-user", "chatgpt",
  "claudebot", "claude-web", "claude-user", "claude-searchbot", "anthropic-ai",
  "perplexitybot", "perplexity-user",
  "amazonbot", "ccbot",
  "meta-externalagent", "meta-externalfetcher",
  "mistralai-user", "cohere-ai", "ai2bot",
  "diffbot", "deepseekbot", "firecrawlagent", "kagi-fetcher",
  "bytespider", "tiktokspider",
  "xai-crawler", "timpibot", "anchor browser", "novellum ai crawl", "proratainc",

  // Social media / messaging link previews
  "facebookexternalhit", "facebookcatalog", "facebookbot", "facebookplatform",
  "instagram", "twitterbot", "linkedinbot",
  "pinterestbot", "pinterest",
  "slackbot", "slack-imgproxy",
  "discordbot", "telegrambot", "whatsapp",
  "redditbot", "snapchat", "vkshare",
  "tumblr", "flipboard", "outbrain", "embedly", "bitlybot",
  "quora link preview", "line", "viber",

  // SEO tools
  "semrushbot", "splitsignalbot",
  "ahrefsbot", "ahrefssiteaudit",
  "mj12bot", "rogerbot", "dotbot",
  "chrome-lighthouse", "screaming frog",
  "oncrawlbot", "botifybot", "deepcrawl", "lumar",
  "dataforseobot", "serpstatbot", "w3c_validator",

  // Internet Archive
  "ia_archiver", "archive.org_bot",
];

// Extensions that should never be prerendered
const STATIC_EXTENSIONS = new Set([
  ".js", ".css", ".xml", ".less", ".json",
  ".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico", ".webp", ".avif",
  ".pdf", ".doc", ".txt", ".rss", ".atom",
  ".zip", ".rar", ".7z", ".gz", ".tar",
  ".mp3", ".mp4", ".m4a", ".wav", ".mov", ".m4v", ".flv", ".webm", ".ogg",
  ".exe", ".dmg", ".iso", ".torrent",
  ".woff", ".woff2", ".ttf", ".eot", ".otf",
  ".webmanifest", ".map", ".wasm",
]);

// Paths that should never be prerendered
const SKIP_PATHS = [
  "/api/", "/graphql", "/_next/", "/__", "/ws", "/socket.io",
];

function getFileExtension(pathname) {
  const p = (pathname || "").toLowerCase();
  const lastSlash = p.lastIndexOf("/");
  const file = lastSlash >= 0 ? p.slice(lastSlash + 1) : p;
  const dot = file.lastIndexOf(".");
  if (dot <= 0) return "";
  return file.slice(dot);
}

function isBotUserAgent(ua) {
  if (!ua) return false;
  const uaLower = ua.toLowerCase();
  for (const sig of BOT_SIGNATURES) {
    if (uaLower.includes(sig)) return true;
  }
  return false;
}

function shouldSkip(url) {
  // Skip static files
  const ext = getFileExtension(url.pathname);
  if (ext && STATIC_EXTENSIONS.has(ext)) return true;

  // Skip API and internal paths
  const pathLower = url.pathname.toLowerCase();
  for (const skip of SKIP_PATHS) {
    if (pathLower.startsWith(skip)) return true;
  }

  return false;
}

async function fetchFromOrigin(request) {
  return fetch(request);
}

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    const ua = request.headers.get("user-agent") || "";

    // Loop protection
    if (request.headers.get("x-seorend-processed") === "1") {
      return fetchFromOrigin(request);
    }

    // Skip non-GET requests
    if (request.method !== "GET") {
      return fetchFromOrigin(request);
    }

    // Skip static assets and API paths
    if (shouldSkip(url)) {
      return fetchFromOrigin(request);
    }

    // Only bots get prerender
    if (!isBotUserAgent(ua)) {
      return fetchFromOrigin(request);
    }

    // Config
    const RENDER_API_URL = "https://render.seorend.com/v1/render";
    const apiKey = env.SEOREND_API_KEY;

    if (!apiKey) {
      console.error("SEOREND_API_KEY not configured");
      return fetchFromOrigin(request);
    }

    try {
      // Call Seorend render API
      const renderResponse = await fetch(RENDER_API_URL, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "Authorization": `Bearer ${apiKey}`,
        },
        body: JSON.stringify({
          url: request.url,
          userAgent: ua,
        }),
      });

      // Handle specific error codes - fallback to origin
      if (renderResponse.status === 401 || renderResponse.status === 403) {
        // Auth error - misconfiguration, log and fallback
        console.error("Seorend auth error:", renderResponse.status);
        return fetchFromOrigin(request);
      }

      if (renderResponse.status === 402 || renderResponse.status === 429) {
        // Payment required or rate limit - fallback to origin gracefully
        // Site continues to work, just without prerendering
        return fetchFromOrigin(request);
      }

      if (renderResponse.status === 504) {
        // Render timeout - fallback to origin
        return fetchFromOrigin(request);
      }

      if (!renderResponse.ok) {
        // Any other error - fallback to origin
        console.error("Seorend error:", renderResponse.status);
        return fetchFromOrigin(request);
      }

      // Parse successful response
      const data = await renderResponse.json();

      // Build response headers
      const headers = new Headers();
      headers.set("Content-Type", "text/html; charset=utf-8");
      headers.set("X-Seorend-Cache", data.cache?.hit ? "HIT" : "MISS");
      headers.set("X-Seorend-Render-Time", String(data.timings?.total_ms || 0));

      // Return pre-rendered HTML
      return new Response(data.html, {
        status: data.status || 200,
        headers,
      });

    } catch (err) {
      // Network error or other exception - fallback to origin
      console.error("Seorend fetch error:", err.message || err);
      return fetchFromOrigin(request);
    }
  },
};

5. Connect Worker to Domain

   1. In Worker settings, go to the Triggers tab
   2. In the Routes section, click Add route
   3. Fill in the fields:
      • Route: example.com/* (replace with your domain)
      • Zone: select your domain
   4. Click Save

   Tip

   For the Worker to also work on www, add a second route: www.example.com/*

6. Verify Proxied Mode

   Make sure your domain’s DNS records go through Cloudflare:

   1. Go to DNS → Records
   2. Find the A or CNAME record for your domain
   3. Make sure the status shows an orange cloud (Proxied)

   If the cloud is gray (DNS Only), click it to switch to Proxied mode.

Verification

After setup, verify everything works correctly.

Test with curl

Make a request with a bot User-Agent:

curl -A "Googlebot/2.1" -I https://example.com/

If everything is set up correctly, you’ll see headers:

X-Seorend-Cache: HIT  (or MISS on first request)

Browser Test

1. Open DevTools (F12)
2. Go to the Network tab
3. Open your site
4. Verify that the response doesn’t have X-Seorend-* headers (regular users shouldn’t be proxied)

Troubleshooting

Worker Not Triggering

• Verify the Route is correct (with /* at the end)
• Make sure DNS is in Proxied mode
• Check Worker logs in the Logs section

401/403 Error

• Verify the SEOREND_API_KEY secret is created correctly
• Make sure the API key is active in the SeoRend dashboard

429 Error

Render limit exceeded. Options:

• Wait for the new billing period
• Upgrade to a higher plan
• Enable Warm Bot to cache popular pages

Next Steps

• Configure caching — optimize TTL for your content
• Enable Warm Bot — automatic warming of popular pages