Skip to content
SeoRend

NGINX Integration

This guide shows how to integrate SeoRend with NGINX so that bot requests are automatically sent through the prerendering service while regular user traffic is served normally.

How It Works

Section titled “How It Works”

NGINX inspects the User-Agent header on every incoming request. If the request comes from a known bot, it’s forwarded to a lightweight Node.js proxy that calls the SeoRend render API. Regular users receive your app directly.

Bot → NGINX → Node.js proxy → SeoRend API → Rendered HTML
User → NGINX → Your app

Requirements

Section titled “Requirements”
  • NGINX installed on your server
  • Node.js 18+ (for the proxy script)
  • A SeoRend API key from app.seorend.com

Step-by-Step Setup

Section titled “Step-by-Step Setup”

  1. Get Your API Key

    Section titled “Get Your API Key”

    Log into app.seorend.com, go to API Keys, and create a new key. Copy the key — you’ll need it in step 3.

  2. Create the Proxy Script

    Section titled “Create the Proxy Script”

    Create a file /opt/seorend-proxy/server.js:

    /opt/seorend-proxy/server.js
    const http = require('http');
    const https = require('https');
    

    const API_KEY = process.env.SEOREND_API_KEY;
    const PORT = process.env.PORT || 3099;
    const RENDER_URL = 'https://render.seorend.com/v1/render';
    

    if (!API_KEY) {
      console.error('SEOREND_API_KEY environment variable is required');
      process.exit(1);
    }
    

    const server = http.createServer((req, res) => {
      // Reconstruct the original URL from headers set by NGINX
      const originalUrl = req.headers['x-original-url'];
      const userAgent = req.headers['x-original-ua'] || '';
    

      if (!originalUrl) {
        res.writeHead(400);
        res.end('Missing x-original-url header');
        return;
      }
    

      const body = JSON.stringify({ url: originalUrl, userAgent });
      const options = {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${API_KEY}`,
          'Content-Length': Buffer.byteLength(body),
        },
      };
    

      const proxyReq = https.request(RENDER_URL, options, (proxyRes) => {
        let data = '';
        proxyRes.on('data', chunk => data += chunk);
        proxyRes.on('end', () => {
          try {
            const json = JSON.parse(data);
            res.writeHead(json.status || 200, {
              'Content-Type': 'text/html; charset=utf-8',
              'X-Seorend-Cache': json.cache?.hit ? 'HIT' : 'MISS',
            });
            res.end(json.html || '');
          } catch {
            res.writeHead(502);
            res.end('Proxy error');
          }
        });
      });
    

      proxyReq.on('error', (err) => {
        console.error('Seorend proxy error:', err.message);
        res.writeHead(502);
        res.end('Proxy error');
      });
    

      proxyReq.write(body);
      proxyReq.end();
    });
    

    server.listen(PORT, '127.0.0.1', () => {
      console.log(`Seorend proxy listening on port ${PORT}`);
    });

  3. Run the Proxy as a Service

    Section titled “Run the Proxy as a Service”

    Create a systemd service:

    Terminal window
    sudo tee /etc/systemd/system/seorend-proxy.service << 'EOF'
    [Unit]
    Description=SeoRend NGINX Proxy
    After=network.target
    

    [Service]
    Environment=SEOREND_API_KEY=your_api_key_here
    Environment=PORT=3099
    ExecStart=/usr/bin/node /opt/seorend-proxy/server.js
    Restart=always
    User=www-data
    

    [Install]
    WantedBy=multi-user.target
    EOF
    

    sudo systemctl daemon-reload
    sudo systemctl enable --now seorend-proxy

  4. Configure NGINX

    Section titled “Configure NGINX”

    Add the bot detection map and proxy block to your NGINX config:

    /etc/nginx/sites-available/example.com
    # Bot detection map — add this at the http{} level (outside server{})
    map $http_user_agent $seorend_bot {
      default                      0;
      # Google
      ~*googlebot                  1;
      ~*googleother                1;
      ~*storebot-google            1;
      ~*google-inspectiontool      1;
      ~*google-extended            1;
      ~*google-cloudvertexbot      1;
      ~*google-favicon             1;
      ~*apis-google                1;
      ~*adsbot-google              1;
      ~*mediapartners-google       1;
      ~*feedfetcher-google         1;
      # Bing / Microsoft
      ~*bingbot                    1;
      ~*bingpreview                1;
      ~*msnbot                     1;
      ~*adidxbot                   1;
      ~*microsoftpreview           1;
      # Yandex
      ~*yandexbot                  1;
      ~*yandeximages               1;
      ~*yandexvideo                1;
      ~*yandexmobilebot            1;
      ~*yandexmetrika              1;
      ~*yandexaccessibilitybot     1;
      ~*yabrowser                  1;
      # Baidu & other search
      ~*baiduspider                1;
      ~*duckduckbot                1;
      ~*duckassistbot              1;
      ~*applebot                   1;
      ~*naver                      1;
      ~*sogouspider                1;
      ~*360spider                  1;
      ~*coccoc                     1;
      ~*seznambot                  1;
      ~*qwantbot                   1;
      ~*ecosia                     1;
      ~*yahoo                      1;
      ~*exabot                     1;
      ~*petalbot                   1;
      ~*bravebot                   1;
      # AI / LLM
      ~*gptbot                     1;
      ~*oai-searchbot              1;
      ~*chatgpt                    1;
      ~*claudebot                  1;
      ~*claude-web                 1;
      ~*anthropic-ai               1;
      ~*perplexitybot              1;
      ~*perplexity-user            1;
      ~*amazonbot                  1;
      ~*ccbot                      1;
      ~*meta-externalagent         1;
      ~*meta-externalfetcher       1;
      ~*mistralai-user             1;
      ~*cohere-ai                  1;
      ~*ai2bot                     1;
      ~*diffbot                    1;
      ~*deepseekbot                1;
      ~*firecrawlagent             1;
      ~*kagi-fetcher               1;
      ~*xai-crawler               1;
      ~*bytespider                 1;
      ~*tiktokspider               1;
      # Social media / link previews
      ~*facebookexternalhit        1;
      ~*facebookbot                1;
      ~*facebookplatform           1;
      ~*instagram                  1;
      ~*twitterbot                 1;
      ~*linkedinbot                1;
      ~*pinterestbot               1;
      ~*slackbot                   1;
      ~*discordbot                 1;
      ~*telegrambot                1;
      ~*whatsapp                   1;
      ~*redditbot                  1;
      ~*snapchat                   1;
      ~*vkshare                    1;
      ~*viber                      1;
      # SEO tools
      ~*semrushbot                 1;
      ~*ahrefsbot                  1;
      ~*ahrefssiteaudit            1;
      ~*mj12bot                    1;
      ~*rogerbot                   1;
      ~*dotbot                     1;
      ~*oncrawlbot                 1;
      ~*botifybot                  1;
      ~*deepcrawl                  1;
      ~*lumar                      1;
      ~*dataforseobot              1;
      ~*serpstatbot                1;
      # Internet Archive
      ~*ia_archiver                1;
      ~*archive.org_bot            1;
    }
    

    server {
      listen 443 ssl;
      server_name example.com www.example.com;
    

      # ... your SSL config ...
    

      location / {
        # Skip static files — never prerender assets
        location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|webp|woff2?|ttf|eot|pdf|zip)$ {
          try_files $uri =404;
        }
    

        # Bots → SeoRend proxy
        if ($seorend_bot = 1) {
          proxy_pass http://127.0.0.1:3099;
          break;
        }
    

        # Regular users → your app
        proxy_pass http://127.0.0.1:3000;
      }
    }

    Then configure the proxy headers. Add this location for the bot proxy:

    location @seorend {
      proxy_pass http://127.0.0.1:3099;
      proxy_set_header X-Original-URL $scheme://$host$request_uri;
      proxy_set_header X-Original-UA $http_user_agent;
      proxy_read_timeout 30s;
    }

    A cleaner approach using try_files with a named location:

    /etc/nginx/sites-available/example.com (recommended)
    map $http_user_agent $seorend_bot {
      # ... (same map as above)
    }
    

    server {
      listen 443 ssl;
      server_name example.com;
    

      location / {
        # Skip static assets
        if ($request_uri ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|webp|woff|woff2|ttf|eot|pdf)$) {
          break;
        }
    

        set $proxy_target http://127.0.0.1:3000;
        if ($seorend_bot = 1) {
          set $proxy_target http://127.0.0.1:3099;
        }
    

        proxy_pass $proxy_target;
        proxy_set_header Host $host;
        proxy_set_header X-Original-URL $scheme://$host$request_uri;
        proxy_set_header X-Original-UA $http_user_agent;
        proxy_read_timeout 30s;
      }
    }

  5. Reload NGINX

    Section titled “Reload NGINX”
    Terminal window
    sudo nginx -t && sudo systemctl reload nginx

Verification

Section titled “Verification”

Test with a bot User-Agent:

Terminal window
curl -s -I -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/

You should see:

X-Seorend-Cache: MISS  ← first request (or HIT if cached)

Test that regular users are NOT proxied:

Terminal window
curl -s -I -A "Mozilla/5.0 Chrome/120" https://example.com/
# No X-Seorend-* headers

Troubleshooting

Section titled “Troubleshooting”

Proxy script not starting: check logs with journalctl -u seorend-proxy -f

NGINX if warnings: NGINX discourages if in location blocks for proxy_pass — use set $proxy_target approach shown above

Timeout errors: increase proxy_read_timeout 30s if your pages are complex

Next Steps

Section titled “Next Steps”