Reverse-proxy setup

In production you do not expose headless-prerender directly. Instead, your web server or CDN inspects each request and, only for crawler traffic, forwards it to the prerender service. Everyone else gets your normal SPA.

What counts as a crawler

Two common signals:

• User-Agent matches a known bot (googlebot, bingbot, facebookexternalhit, twitterbot, slackbot, linkedinbot, whatsapp, discordbot, ...).
• The request is not for a static asset (skip .js, .css, images, fonts, .json, .xml, .txt).

The rewrite

For a bot request to:

https://www.example.com/products/42?ref=x

forward to:

http://prerender:3000/render?url=https://www.example.com/products/42?ref=x

The prerender service returns the rendered HTML with the original page's status code.

Nginx

map $http_user_agent $is_bot {
    default                                 0;
    "~*googlebot|bingbot|yandex|duckduckbot" 1;
    "~*facebookexternalhit|twitterbot"       1;
    "~*slackbot|linkedinbot|embedly"         1;
    "~*whatsapp|telegrambot|discordbot"      1;
}

map $uri $is_asset {
    default                                  0;
    "~*\.(js|css|json|xml|txt|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|map)$" 1;
}

server {
    listen 80;
    server_name www.example.com;
    root /var/www/example/dist;

    location / {
        set $prerender 0;
        if ($is_bot = 1) { set $prerender 1; }
        if ($is_asset = 1) { set $prerender 0; }

        if ($prerender = 1) {
            rewrite ^ /_prerender last;
        }

        try_files $uri /index.html;
    }

    location /_prerender {
        internal;
        proxy_set_header X-Prerender-Token "$PRERENDER_TOKEN";
        proxy_pass http://127.0.0.1:3000/render?url=$scheme://$host$request_uri&token=$PRERENDER_TOKEN;
        proxy_read_timeout 30s;
    }
}

Cloudflare Workers

const BOT = /googlebot|bingbot|yandex|duckduckbot|facebookexternalhit|twitterbot|slackbot|linkedinbot|whatsapp|telegrambot|discordbot|embedly/i;
const ASSET = /\.(js|css|json|xml|txt|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|map)$/i;

const PRERENDER_ORIGIN = "https://prerender.example.com";
const PRERENDER_TOKEN = "REPLACE_ME";

export default {
  async fetch(request) {
    const url = new URL(request.url);
    const ua = request.headers.get("user-agent") || "";

    if (BOT.test(ua) && !ASSET.test(url.pathname)) {
      const target = new URL("/render", PRERENDER_ORIGIN);
      target.searchParams.set("url", url.toString());
      target.searchParams.set("token", PRERENDER_TOKEN);

      const rendered = await fetch(target, {
        headers: { "x-prerender-token": PRERENDER_TOKEN },
      });
      return new Response(rendered.body, {
        status: rendered.status,
        headers: { "content-type": "text/html; charset=utf-8" },
      });
    }

    return fetch(request);
  },
};

Apache (mod_rewrite + mod_proxy)

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|facebookexternalhit|twitterbot|slackbot|linkedinbot|whatsapp|discordbot) [NC]
RewriteCond %{REQUEST_URI} !\.(js|css|json|xml|txt|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|map)$ [NC]
RewriteRule ^(.*)$ http://127.0.0.1:3000/render?url=%{REQUEST_SCHEME}://%{HTTP_HOST}%{REQUEST_URI} [P,L]

Checklist

• [ ] Bot list covers the engines and social scrapers you care about.
• [ ] Static assets are excluded from the rewrite.
• [ ] The proxy passes the full original URL (scheme, host, path, query).
• [ ] A --token is set and forwarded, so the prerender service is not open.
• [ ] --allow-origin is set to your site's origin(s).
• [ ] Proxy read timeout is longer than your render --timeout.
• [ ] Caching is enabled for repeated bot hits.