CrawlGate

1. One-liner

Shows publishers which AI bots their site actually blocks, and what each block costs in lost citations.

2. Trend signal — why now?

Three things happened in the last ninety days, and together they turned a config file nobody read into a revenue decision nobody is equipped to make.

Cloudflare changed the default. From 15 September 2026, Cloudflare blocks “mixed-use” crawlers — bots that blend search, agent use, and training — from any page hosting ads, by default. It applies to new customers, new sites on existing accounts, and all existing free customers. That last clause is the whole business. Hundreds of thousands of small ad-supported sites sit on Cloudflare’s free tier. On September 15 their bot policy changes underneath them and nobody sends them an invoice or a summary. They find out from their traffic graph, three months late.

The crawler taxonomy stopped being academic. Training bots (GPTBot, Google-Extended, ClaudeBot, CCBot) take content and return nothing. Grounding bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) are the ones that produce citations and referral clicks. Conflating them is described in the trade press as the single most common mistake in AI crawler policy — you block GPTBot to protect your work, accidentally catch OAI-SearchBot in the same rule, and vanish from ChatGPT search while your Google rankings sit there looking fine. The failure is silent and lagging. Nothing in your analytics says “you did this to yourself in March.”

The scale of the misconfiguration got measured. Otterly’s AI Citations Report 2026, built on over a million AI citations, found 73% of sites had technical barriers blocking AI crawler access. A separate figure making the rounds: 17.6% of top sites return 403 to bots their own robots.txt explicitly allows — the firewall and the text file disagree, and the firewall wins. That’s not an opinion gap. That’s a measurable defect on a specific URL that I can show you in ten seconds.

Meanwhile the ground is burning. Search referrals to small publishers (1K–10K daily pageviews) fell 60% over two years; medium publishers 47%; large publishers 22%. For queries where AI Overviews appear, organic CTR crashed 61%. When your old traffic source is down 60%, the new one — being cited in the answer — stops being a nice-to-have and becomes the business. And most site owners cannot currently tell you whether they’re eligible for it.

Provenance:

• Signal 1 (demand): Otterly AI Citations Report 2026 — 73% of sites have technical barriers blocking AI crawler access; Cloudflare default bot config means small businesses may block every AI crawler without deliberately deciding to — https://mediabusmarketing.com/ai-crawlers-in-2026-every-bot-you-should-optimize-for/ — 2026-08-03
• Signal 2 (feasibility/platform): Cloudflare blocks mixed-use crawlers from ad-hosting pages by default from 15 Sept 2026, applying to all existing free customers — https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ — 2026-08-03
• Signal 3 (economic): Search referrals down 60% for small publishers, 47% medium, 22% large; AI Overview queries show 61% organic CTR collapse; publishers forecast a further 43% decline within three years — https://almcorp.com/blog/search-traffic-decline-small-publishers-chartbeat-data/ — 2026-08-03
• Signal 4 (market gap): Existing tooling is enterprise-priced — Scrunch AI $250/mo with crawler analytics Enterprise-only, Cloudflare Bot Management from $200/mo Business tier, TollBit demo-gated — https://geoptie.com/blog/scrunch-ai-review — 2026-08-03 Category: Platform shift

3. The opportunity

There are two products in this market and neither serves the person with the problem.

At the top, enterprise AI-visibility suites — Scrunch at $250/mo for four engines with crawler analytics locked behind Enterprise, TollBit doing real licensing deals for TIME-scale publishers, Cloudflare Bot Management starting at the $200/mo Business plan. These are built for a media company with a revenue ops team. They are priced and sold accordingly.

At the bottom, free raw analytics — Cloudflare’s AI Crawl Control will happily show you a list of bot hits. It tells you what happened. It does not tell you whether what happened was what you wanted, and it certainly doesn’t tell you what it cost.

Between those sits the actual customer: someone running a recipe site, a hobbyist gear-review blog, a regional news outlet, a niche B2B content property. Ad revenue between $2K and $80K a month. One to four people, often one. They are on Cloudflare’s free plan. On September 15 their defaults change. They do not know which of the forty-odd AI user-agents matter, they do not know that their firewall is overriding their robots.txt, and they have no way to connect “I blocked something in March” to “Perplexity stopped citing me in April.”

The wedge is that this is provable, not advisory. Every competitor in the AI-visibility space sells a dashboard of scores and recommendations — soft stuff, arguable, easy to ignore. CrawlGate does something narrower and much harder to wave away: it fetches your pages as each named bot, from outside, and shows you the actual HTTP response. Your robots.txt says OAI-SearchBot is allowed. Your edge returns 403. Here is the request, here is the response, here is the timestamp. That’s not a recommendation. That’s a receipt. And the fix is a one-line change the owner can make in four minutes.

The 10× isn’t intelligence, it’s resolution and price. A $39/mo tool that gives a solo publisher the same per-bot ground truth a $250/mo enterprise suite gates behind a sales call.

4. Target market

• Primary customer: Owner-operators of ad-supported content sites — 1–4 people, 10K–2M monthly pageviews, revenue predominantly from display ads (Mediavine, Raptive, AdThrive, Ezoic) or affiliate. Global English-language first; the problem is geography-agnostic because the crawlers are. Secondary: small SEO/GEO agencies managing 10–60 client sites who need this per-client.

• Why they buy: Because their traffic fell off a cliff and they cannot tell which cliff. In their world the question is no longer “how do I rank” but “am I even readable by the things that now answer the questions.” The specific purchase trigger is the September 15 default change — a dated, named event with their hosting provider’s name on it. When someone’s platform changes policy on a specific date and they don’t understand the consequences, they buy the thing that explains it. That’s the most reliable purchase impulse in all of SMB software.

• Rough TAM reasoning: Ad-network publisher programs give the shape. Raptive and Mediavine each work with tens of thousands of creator-run sites; Ezoic serves a much larger long tail at lower traffic thresholds. Add non-network independent publishers and niche B2B content properties and the addressable pool of sites where ad revenue is material and staffing is 1–4 people is comfortably in the low hundreds of thousands globally. I need roughly 2,100 of them paying $39/mo to hit $1M ARR. That’s a fraction of a percent. The math is not the risk here.

• Why now for them: September 15, 2026. Before that date this is a good-hygiene product and hygiene products don’t sell. After that date their bot policy has already changed without their consent and the product becomes an incident response. The window where this sells hardest is roughly August 2026 through Q1 2027 — pre-deadline anxiety, then post-deadline damage assessment.

5. Product sketch (MVP)

• Bot’s-eye fetch. Request a sample of the site’s pages from outside as each named agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, Googlebot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent and the rest of the current list — and record the literal status code, redirect chain, and whether a challenge page was served.

• The contradiction report. The headline artifact: a table of every bot where stated policy and actual behaviour disagree. robots.txt says allow, edge says 403. Or robots.txt is silent, Cloudflare’s managed rule blocks anyway. Each row shows the rule believed to be responsible and the exact fix.

• Training vs. grounding split. Every bot is classified by what it gives back. Blocking training bots is a legitimate business choice; blocking grounding bots is almost always an accident. The report separates the deliberate from the self-inflicted and refuses to lump them together.

• September 15 exposure check. For Cloudflare-fronted sites: flag whether the account is in the population hit by the new mixed-use default, which ad-bearing page templates are in scope, and what the policy will look like on the 16th if nothing is done.

• Continuous re-check with drift alerts. Weekly re-fetch. When a bot that was reachable last week starts returning 403, that’s an email the same day — because the whole failure mode of this category is that changes are silent and consequences arrive a quarter later.

• Citation cross-reference. Track a set of the site’s target queries across ChatGPT, Perplexity, and Google AI surfaces, and line up citation presence against crawl access over time. The purpose is not a vanity visibility score — it’s to close the loop: you unblocked OAI-SearchBot on the 3rd, citations resumed on the 19th.

• The fix, generated. Output a corrected robots.txt and the specific Cloudflare rule changes, reflecting the owner’s stated intent — “I want training blocked and grounding open” — rather than a generic template.

• Client-portfolio view. For agencies: all managed sites in one table, sorted by severity, with per-client exportable PDF.

6. AI angle — what’s load-bearing

Honest answer first: the fetch-and-compare core is not AI. It’s disciplined HTTP work, and I’d rather have that be boring and correct than clever.

Where the model genuinely carries weight:

Rule attribution. A site returns 403 to PerplexityBot. Why? Could be robots.txt, a Cloudflare managed rule, a WAF custom rule, a plugin, an origin-level nginx block, or the CDN’s bot-score threshold. Reasoning backward from observed responses plus available config to the probable cause — and expressing it as “your Cloudflare Bot Fight Mode is doing this, here’s where to turn it off” — is a genuine inference task over messy, partial evidence. Getting this right is the difference between a report that gets acted on and one that gets ignored.

Citation attribution. Determining whether a given AI answer actually drew on this site — cited by name, paraphrased without link, or absent — across engines that all format their outputs differently and change formats without warning. Brittle to do with parsers, tractable with a model.

Intent translation. The owner says “I don’t want my recipes used for training but I want to show up in ChatGPT.” Turning that sentence into a correct, current, per-agent policy across a bot list that shifts every few weeks — and re-deriving it when a new agent appears — is the recurring work the subscription pays for.

Strip the AI out and you have a status-code checker: still useful, worth maybe $9/mo, and a weekend project for anyone. The model is what makes it a diagnosis instead of a log.

7. Localization angle (if any)

N/A — this is a global play. AI crawlers do not respect borders and the user-agent list is identical whether the site is in Ohio or Odisha. English-first for launch because the publisher communities where distribution happens (ad-network forums, SEO Twitter, niche subreddits) operate in English.

One genuine regional wrinkle worth noting rather than forcing: EU publishers have an adjacent set of questions about AI Act transparency and text-and-data-mining reservations under the DSM Directive. That’s a plausible content angle and a later feature, not a launch localization.

8. Business model — path to $1M–$5M ARR

• Pricing:

  • Free scan — one site, one-time contradiction report, no monitoring. This is the marketing, not a tier.
  • Solo — $39/mo — one site, weekly re-checks, drift alerts, generated fixes.
  • Publisher — $99/mo — up to 5 sites, daily re-checks, citation tracking on 50 queries.
  • Agency — $299/mo — up to 40 client sites, portfolio view, white-labelled client PDFs.

• ACV: Blended around $560/yr assuming a mix weighted toward Solo with meaningful agency contribution.

• Rough math to $1M ARR: 1,400 Solo ($655K) + 300 Publisher ($356K) + 30 Agency ($108K) ≈ $1.12M. Or read plainly: about 2,100 paying accounts across tiers.

• Rough math to $5M ARR: Needs the agency tier to carry more — roughly 250 agencies at $299 plus 5,000 direct publishers, or the same revenue via an upmarket tier for ad networks and small media groups who want this monitored across their entire portfolio of sites. The network-level deal is the interesting one: a mid-size ad network with 8,000 publisher sites has a direct financial stake in every one of them being AI-readable, since blocked grounding bots eventually mean less traffic and less inventory. One such deal is worth several hundred direct accounts. I would not build the plan around it, but I’d take the meeting.

• Expansion path: Site count, then check frequency, then queries tracked. The natural upsell is from diagnosis to enforcement — once you know which bots you want where, managing that policy across a portfolio (and eventually negotiating paid access) is a bigger job than reporting on it.

• Cost note: Gross margin is comfortable. Costs are outbound HTTP requests, modest model calls per report, and query checks against AI engines. The last item is the one that scales with usage and needs a cap per tier — 50 tracked queries on the $99 plan is a deliberate ceiling, not an arbitrary one.

9. Go-to-market wedge — first 100 customers

The advantage here is that I can prove the problem before asking for anything, and the proof is site-specific. That makes cold outreach unusually strong — this isn’t “here’s my product,” it’s “here’s a defect on your property.”

• The 403 list. Assemble 3,000 ad-supported niche sites from public sources — ad-network publisher showcases, niche directories, RSS aggregators in food/travel/finance/parenting/gear. Run the bot’s-eye fetch across all of them. Keep only the ones with an actual contradiction: robots.txt allows a grounding bot, edge returns 403. On the published rates something like half to three-quarters of the list will have a real finding. Email each: subject line is their domain and the bot name, body is three lines and the raw request/response, link to the full free report. This is not a spray — it is a personalized technical bug report, and the reply rates on those are unlike anything in normal cold email. Even 3% converting to free scan and 15% of those to paid is over 100 customers from one pass, with list to spare.

• Ride the September 15 date. Publish one genuinely definitive piece: “What changes for your site on September 15, and how to check yours in 60 seconds,” with the free scanner embedded. Then place it where these people actually congregate — the Mediavine and Raptive publisher Facebook groups, r/juststart, r/blogging, r/SEO, Indie Hackers, the Ezoic community forum. Dated platform changes reliably generate a scramble for a definitive explainer; being the tool inside that explainer is worth more than any ad spend. Time the push for late August, when anxiety peaks and the deadline is still actionable.

• Agency back-door. Small SEO and GEO agencies are already fielding “why did our AI traffic drop” from clients and have no clean answer. Run the scan across a target agency’s visible client roster, send the partner one email containing findings across their own portfolio. An agency signing up brings 15–40 sites at once and does the client education for me. Twenty agencies is a fifth of the way to the first hundred.

• Ad-network partnership. Networks like Ezoic and the mid-tier ad managers have aligned incentives — their revenue is downstream of their publishers’ traffic. A co-marketed free scan for their publisher base costs them nothing and makes them look useful. One placement in a network’s publisher newsletter reaches more qualified prospects than a quarter of content marketing.

10. Build complexity — justification

Low. The core is a fetch harness that requests URLs with specific user-agents from clean IPs, plus robots.txt parsing, plus a diff engine and a scheduler. All standard. The model work sits on top of collected evidence and uses off-the-shelf APIs — no training, no custom infrastructure, no proprietary dataset required to start.

Two things need real care rather than real complexity. First, the bot taxonomy has to stay current — agents appear and get renamed every few weeks, and a stale list makes the product wrong rather than merely incomplete. That’s ongoing maintenance and it’s also part of the moat. Second, fetching honestly: requests must come from IPs that aren’t already flagged, and the tool must identify itself truthfully rather than impersonating a real crawler in a way that gets it banned or creates a legitimate complaint. Impersonating Googlebot to test a firewall is a line I won’t cross — the tool declares itself and tests policy behaviour, which is sufficient for the diagnosis and keeps it clean.

A competent solo builder ships a credible v1 in 5–7 weeks. Free scanner in two.

11. Gating checklist

Gate	Pass?	Note
Legal in target market	✅	Fetching public pages with a truthfully-identified agent. No impersonation of real crawlers, no auth bypass, no scraping of gated content.
Ethical — no harm / dark patterns	✅	Helps publishers regain control over their own content policy. The free scan is a genuine free scan, not a crippled teaser.
Market exists (evidence above)	✅	73% of sites with technical barriers; enterprise tools charging $250/mo for a subset of this; a dated platform change forcing the issue.
1–5 person team can build this	✅	Solo-buildable. Standard web stack plus off-the-shelf model APIs.
Launchable with <$50K / ₹40L	✅	Realistically under $6K to first revenue — infrastructure, proxies, model credits, domain.

12. Feasibility score

Axis	Weight	Score	Notes
Problem intensity	20	17/20	Direct revenue consequence — invisible in AI answers means lost referrals on top of a 60% search decline. Not quite hair-on-fire because the pain is lagging and attributable only in hindsight, which is exactly why they need the tool but also why some won’t feel urgency until after the damage.
Demand evidence	15	14/15	Unusually strong. A hard measured statistic (73% misconfigured), enterprise competitors charging real money for a slice of it, a dated platform change, and a documented traffic collapse. A skeptic nods.
Build feasibility	15	13/15	Solo, 5–7 weeks, off-the-shelf everything. Docked for the ongoing accuracy burden of a taxonomy that shifts weekly.
Distribution clarity	15	12/15	The personalized-defect cold email is genuinely strong and the channels are named and reachable. Docked because publisher communities are noisy and somewhat tool-fatigued, and free-scan-to-paid conversion is the untested number.
Revenue mechanics	15	11/15	Pricing is benchmarked below visible competitors and 2,100 accounts for $1M is achievable. Docked because $39/mo prosumer SaaS churns, and this product has a “fixed it, cancelling” failure mode that monitoring must overcome.
Time to first revenue	10	8/10	Free scanner live in two weeks, paid conversions within 4–6. Slightly short of top marks because the first cohort will want to see a second month of monitoring before believing it’s recurring.
Defensibility	10	6/10	Soft. The fetch logic is copyable in a weekend. What isn’t: an accurate, continuously-maintained bot taxonomy, accumulated cross-site data on which edge configs cause which blocks, and the historical record per customer that makes switching lossy. Twelve months of longitudinal data on a customer’s own site is a real retention hook.
Total	100	77/100

13. Qualitative modifiers

Founder-fit tags

technical-heavy · content-heavy

Technical because credibility here is entirely a function of the diagnosis being correct — a false positive telling someone they’re blocked when they aren’t destroys trust permanently. Content because the distribution is authority-led: the person who writes the definitive explainer of the September 15 change owns this category’s attention for a quarter.

Key assumptions to validate (3–5)

1. Assumption: A meaningful share of small ad-supported sites have a genuine, demonstrable contradiction between robots.txt and edge behaviour. How to test: Run the fetch across 500 sites this week. Measure the real contradiction rate. If it’s under 20%, the cold-email wedge collapses and the whole GTM needs rebuilding. This is the first thing I’d do and it costs nothing but a few days.

2. Assumption: Publishers will pay for ongoing monitoring rather than treating this as a one-time fix. How to test: Track month-2 retention in the first paid cohort. Below 70% and this is a one-shot audit product — which means either an annual-scan price point or a pivot to agencies who have a continuous portfolio need.

3. Assumption: Fixing crawl access produces observable citation recovery within a reasonable window. How to test: Instrument 20 early customers — record fix date, track citation presence weekly across engines for eight weeks. This is the product’s core promise. If unblocking changes nothing measurable, I’m selling hygiene, and hygiene doesn’t hold a $39/mo subscription.

4. Assumption: The September 15 change actually bites the population I think it does. How to test: Sample 200 small ad-supported sites, determine Cloudflare free-tier fronting, re-scan the same set on September 16 and measure what actually changed. This also produces the single best piece of content I could publish that week.

Risk flags

1. Platform dependency (severe). Cloudflare could ship this as a free feature inside AI Crawl Control at any moment — they already have the traffic data, and the diagnosis is a natural extension. This is the biggest risk on the board. The mitigations are being multi-CDN from day one (Cloudflare-only is a feature, cross-platform is a product), and pushing fast toward the citation-outcome loop, which Cloudflare has no visibility into.

2. Market timing (double-edged). The September 15 date creates a sharp demand spike and I’m arriving right ahead of it, which is good. The risk is that it’s a spike — if the entire market buys in Q4 and fixes their config, the Q2 2027 cohort is much smaller. Answer: the taxonomy churns constantly and new agents keep appearing, so drift monitoring should have durable value. But that’s a thesis, not yet a fact, and assumption #2 is what tests it.

3. Verification fragility. Citation tracking depends on scraping or querying AI engines whose interfaces change without notice and whose terms may restrict automated querying. That feature could degrade or need rebuilding on short notice. The contradiction report — the core value — doesn’t depend on it, which is why it must stay the headline.

4. Commoditization. “Check if AI bots can see your site” is an obvious free-tool idea and several SEO vendors will bolt on a lite version. The defence is depth of diagnosis and longitudinal record, not the check itself.

14. Structured verdict

Score:                  77/100
Verdict:                GO
Confidence:             Medium
Best-fit builder:       Technical solo founder comfortable with HTTP/edge-infrastructure
                        detail, who can also write authoritatively for a publisher audience
Time to revenue:        4–6 weeks from start of build
Capital to launch:      $5–6K (₹5L) — infra, clean-IP proxies, model credits, domain
Top 3 assumptions to validate first:
  1. Contradiction rate ≥20% across 500 real ad-supported sites — run the scan this week
  2. Month-2 retention ≥70% in first paid cohort — proves monitoring, not one-shot audit
  3. Citation recovery observable within 8 weeks of fix across 20 instrumented customers
Kill criteria:
  - Abandon if the contradiction rate across 500 sampled sites is under 20% — the
    personalized-defect wedge is the entire GTM and it doesn't survive a weak hit rate
  - Abandon if month-2 retention is below 50% after two cohorts — that's a one-time
    audit, not a subscription, and the ARR math doesn't work at audit pricing
  - Abandon if Cloudflare ships equivalent per-bot contradiction diagnosis free inside
    AI Crawl Control before v1 launches

15. Next step — 1-week validation sprint

• Day 1–2: Build only the fetch harness. Assemble 500 ad-supported niche sites from ad-network showcases and directories. Fetch each as eight named agents. Compute the contradiction rate — sites whose robots.txt permits a grounding bot but whose edge returns 403 or a challenge. This single number decides everything downstream.

• Day 3–4: Take the 50 worst offenders. Send each owner a three-line email: their domain, the bot being blocked, the raw request and response, and one sentence on what it costs them. No product, no link to a pricing page, no pitch — just the finding and an offer to explain. Measure reply rate and, more importantly, the content of replies: do they already know, do they care, do they ask how to fix it.

• Day 5: Decide. Go if the contradiction rate is ≥20% and ≥10 of 50 emails get a reply and ≥3 replies contain some version of “how do I fix this” or “can you do this for my other sites.” No-go if the contradiction rate is under 20% (the defect isn’t widespread enough to fuel outreach) or replies are dominated by “yes, that block is deliberate” (they already know and it’s intentional, meaning I’ve mistaken a choice for an accident).

The falsifiable core: either a fifth of small ad-supported sites are silently blocking the bots that would cite them, or they aren’t. One week and 500 HTTP fetches settles it, and no amount of market reasoning substitutes for that number.