Here is what I learned building that scanner, including the two places where the obvious approach confidently returns wrong answers.

Why fetching the HTML is not enough

My first prototype fetched the page server-side and grepped for known script URLs. It took an hour to build and it was wrong on almost every real website.

Modern sites load trackers through tag managers, so the HTML contains one googletagmanager.com script and nothing else, while the browser ends up talking to five ad networks. Consent tools gate half of the requests behind a click. Bundlers hide vendor SDKs inside chunk-4f2a.js. And some services never appear in any markup because another script injects them at runtime.

The only reliable source of truth is the network traffic of a real browser. So the scanner became a headless Chromium instance that opens the page and records every request and every cookie.

const requests = []
page.on('request', r => requests.push(new URL(r.url()).hostname))
await page.goto(url)
await page.waitForLoadState('networkidle')

(Simplified. The production version also waits for late scripts, scrolls a bit to trigger lazy loading, and visits the contact and booking pages, because that is where booking widgets and captchas live.)

The consent diff is the most valuable part

Under EU rules, non-essential trackers may only load after the visitor consents. In Austria that is § 165 of the telecom act, and it is the single most violated rule I see in the wild. The site has a consent banner, looks compliant, and Google Analytics fires on the first page load anyway.

A network-recording browser detects this almost for free. You record in two phases.

1. Load the page and record everything that happens before anyone touches the banner.

2. Find the accept button, click it, wait, and record again.

Everything from phase two is a service that belongs in the privacy policy. Everything trackery from phase one is a service that loads without consent, which the generator flags with a warning instead of silently writing a paragraph that claims consent is obtained first.

Clicking the banner is the messy bit. There is no standard, so it is a pile of heuristics. Known consent-platform selectors first, then buttons and links matched by text in German and English, scored so that "accept all" beats "settings". It fails on exotic custom banners, and that is fine, the scan then simply reports what loads without any interaction.

From hostnames to services

Raw hostnames are useless to normal people, so the scanner maps them against a hand-maintained catalog of services. Each entry has match patterns plus the metadata the legal text needs, meaning provider, purpose, legal basis, retention and whether data leaves the EU. Cookie names act as a second signal, which catches services that route traffic through the first-party domain.

I resisted the temptation to auto-generate this catalog. Sixty-odd services curated by hand beat six hundred scraped ones, because every entry directly produces legal text someone will publish.

Hosting detection, or why nameservers lie

The policy also needs a section about the hosting provider, so the scanner tries to detect it. My first version looked up the domain's nameservers. That felt reasonable and was wrong embarrassingly often.

Nameservers tell you who manages DNS, which is usually the registrar, not the hoster. In Austria, tons of domains sit on the DNS of a big telco while the site itself runs on Hetzner or wherever. The signals that actually work are the reverse DNS entry of the site's IP address and the ASN it belongs to, in that order. The PTR record of a Hetzner IP says Hetzner no matter who sold the domain, and the ASN catches providers with unhelpful PTR records.

const [ip] = await dns.promises.resolve4(hostname)
const [ptr] = await dns.promises.reverse(ip).catch(() => [null])

One more trap. Scan the final URL after redirects, not the one the user typed. example.at redirecting to www.example.at or to a completely different domain is common, and scanning the redirect source gives you an empty page and a wrong hoster.

Where it ended up

The scanner powers the free privacy policy generator I built for Austrian websites. It pre-selects the detected services, fills in the hosting provider, pulls the operator data from the site's legal notice, and warns about trackers that fire before consent. The generated policy cites the Austrian laws instead of the German ones that most templates copy. If you run an .at website or are just curious what the scan finds on your site, it lives at https://webgaudi.at/datenschutzerklaerung-generator-oesterreich/

The lesson that stuck with me is an old one. For anything compliance-adjacent, do not trust declarations, not even your own HTML. Measure the behavior.

Two well-documented cases to set the stakes. Cloudflare went down globally for 27 minutes on 2 July 2019 because a WAF rule contained the sub-pattern .*.*=.* and it pinned CPU across the fleet. Stack Overflow was down for 34 minutes on 20 July 2016 because a post-trimming regex, roughly ^[\s‌]+|[\s‌]+$, met a post with about 20,000 consecutive spaces.

Neither was an exotic pattern. Both were validation code that ran fine for years.

Here is a pass you can do this afternoon on your own service.

Step 1: find every regex, including the ones you forgot

rg -n --type js -e 'new RegExp\(' -e '/(?:[^/\n\\]|\\.){8,}/[gimsuyd]*'

Then the ones that are not in your code:

rg -n -e 'pattern:' -e 'matches:' --type yaml --type json

That second one catches JSON Schema pattern keywords, OpenAPI specs, Joi and Zod .regex() calls, and validation rules that live in config. In most codebases I have looked at, that is where the risky ones actually are, because a schema is data and never gets code review.

Sort what you find into two piles: regexes that ever touch a request body, a query string, a header or an uploaded file, and everything else. Only the first pile matters. A regex over a hardcoded constant cannot be attacked.

Step 2: triage by shape, not by feeling

There are three shapes worth knowing. If a pattern has none of them, move on.

Nested quantifier. A repetition inside a repetition, where the inner one can match the same character the outer one can.

/^(a+)+$/
/^([a-zA-Z0-9_.\-])+@(([a-zA-Z0-9\-])+\.)+([a-zA-Z0-9]{2,4})+$/

That second one is a real email validator that has been copied into thousands of projects. The + after the final group is the bug. Feed it 30 valid-looking characters followed by one that breaks the match and the engine walks an exponential number of ways to split them.

Overlapping alternation under a quantifier.

/^(\w|\d|_)*$/
/(a|ab)+c/

\w already contains \d and _. Every character has three ways to match, so the engine has three paths per character to back through.

Two quantifiers that can eat the same characters.

/.*.*=.*/          // the Cloudflare one, reduced
/^\s*(.*?)\s*$/
/(\s*)+$/

This one is only quadratic, not exponential, which sounds reassuring and is not. Quadratic at 100,000 characters is still ten billion steps, and a 100 KB request body is nothing.

The rule of thumb underneath all three: ambiguity plus repetition plus a failing match. The blowup needs the match to eventually fail, which is why "I tested it with valid input and it was instant" tells you nothing at all. Valid input takes the first path. Attackers do not send valid input.

Step 3: measure it, do not reason about it

Reasoning about backtracking is a good way to convince yourself a pattern is safe when it is not. Time it instead.

const re = /^([a-zA-Z0-9_.\-])+@(([a-zA-Z0-9\-])+\.)+([a-zA-Z0-9]{2,4})+$/;

for (let n = 10; n <= 40; n += 2) {
  const attack = 'a'.repeat(n) + '@';   // matches the local part, then fails
  const t = process.hrtime.bigint();
  re.test(attack);
  const ms = Number(process.hrtime.bigint() - t) / 1e6;
  console.log(n, ms.toFixed(2) + 'ms');
  if (ms > 1000) break;
}

You are not looking at the absolute numbers. You are looking at the shape of the curve. If each step of 2 roughly quadruples the time, it is exponential and you are done deliberating. If it roughly quadruples every time you double n, it is quadratic and you need to know your input size limit. If it stays flat, the pattern is fine.

The number to write in the ticket is the input length at which the match costs one full second of CPU. If that number is smaller than your body-size limit, it is a live denial of service, not a code smell.

Step 4: know what your tooling does and does not cover

npm audit finds published ReDoS advisories in your dependencies. It has nothing to say about the regex you wrote in validators.js, which is where yours is.

safe-regex is the package people reach for and it is old, static-only, and has real false negatives. The current state of the art is recheck, which does a hybrid static and fuzzing analysis. It is packaged as eslint-plugin-redos, so you can put it in CI:

npm i -D eslint-plugin-redos

That covers regex literals in JS. It does not cover patterns in JSON Schema, OpenAPI or config, which is why step 1 is a separate step.

Step 5: fixes, in order of how much they buy you

1. Cap the input before the regex sees it. if (value.length > 254) return false; in front of an email validator removes an entire class of attack for one line, and 254 is the actual RFC limit anyway. Do this first, always, even after you fix the pattern.

2. Remove the ambiguity. (\w|\d|_)* is just \w*. ([a-zA-Z0-9\-]+\.)+ should have a bounded repetition. Most nested quantifiers are a redundant + that someone added while debugging and never removed.

3. Anchor and be specific. An unanchored pattern gets retried at every start position, which multiplies whatever the per-position cost is by the input length.

4. Stop using a regex. Email, URLs, dates and CSV are all better handled by a parser or a library. new URL() cannot be made to backtrack.

5. Swap the engine where the pattern must stay. The re2 npm package binds Google's RE2, which is linear time because it does not backtrack at all. The trade is no backreferences and no lookbehind. For validation patterns that is almost never a real constraint. V8 also has an experimental linear-time engine behind --enable-experimental-regexp-engine, which is worth knowing about and not worth shipping on yet.