# Fastest Freelancers — https://fastestfreelancers.com
#
# Served as a real static file. The SPA catch-all in netlify.toml is a non-forced
# 200 rewrite, so existing files in frontend/public/ win over it; without this file
# /robots.txt was answered with the index.html shell (HTTP 200, text/html), which
# reads as a misconfigured site to crawlers and compliance reviewers.
#
# AI POLICY (owner, 10/24/2026). AI assistants fetching a page for a person, and AI search
# indexing, are welcome everywhere public, because being found by them is how clients reach
# us. Client profiles are kept out of AI agents' reach: a client is not selling anything,
# so exposing them would cost their privacy for nothing. Model training is DISCOURAGED,
# not blocked, through the Content-Signal line (contentsignals.org): search=yes,
# ai-input=yes (answering a person's question), ai-train=no.
#
# ⚠ A crawler obeys only the MOST SPECIFIC group that names it (RFC 9309), so the AI group
# below repeats every rule of the `*` group. A rule added to one must be added to both.
#
# A user agent here is a request, not an identity: anything can claim to be GPTBot. Claims
# are checked server-side against each operator's published IP ranges or Web Bot Auth
# signature (backend lib/aiBotVerify.js), and failed claims are counted as impersonation.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
# The documented Agent API (/agent-api.json, /llms.txt) is the one part of /api/ meant for
# automated callers. It is the more specific rule, so it wins under RFC 9309's longest
# match, and it comes FIRST so a first-match parser agrees. Everything not disallowed is
# allowed, so there is no `Allow: /` (which a first-match parser would let win over all).
Allow: /api/agent/
# Authenticated and transactional surfaces — nothing here is useful to a crawler.
Disallow: /api/
Disallow: /admin
Disallow: /settings
Disallow: /notifications
# An AI agent's handoff link is a private, single-use key for one person: never crawled.
Disallow: /agent-handoff/

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: Meta-ExternalAgent
User-agent: cohere-ai
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /api/agent/
Disallow: /api/
Disallow: /admin
Disallow: /settings
Disallow: /notifications
Disallow: /agent-handoff/
Disallow: /client-profile

Sitemap: https://fastestfreelancers.com/sitemap.xml