# Robots.txt for Malaga Directory
# https://malaga-directory.com
User-agent: *
Allow: /
# Private areas
Disallow: /admin/
Disallow: /business-owner/dashboard.php
Disallow: /business-owner/edit-*.php
Disallow: /includes/
Disallow: /config/
Disallow: /api/
# NOTE: the rule above matches ONLY paths starting /api/. It does NOT cover
# /business-owner/api/, which is a separate directory - that gap let Googlebot
# crawl four authenticated endpoints (connect-place, fetch-rating,
# search-claimable, set-featured-review) that correctly answer 401 to anonymous
# requests, producing GSC's "Blocked due to unauthorised request (401)" alert on
# 2026-08-12. The 401s are right; Google reaching them was not. Blocking is the
# correct tool here (unlike the facet case below): these are JSON/401 responses
# with no meta robots and no X-Robots-Tag, so there is no noindex to cancel.
Disallow: /business-owner/api/
# Search results stay crawlable so Google can read noindex,follow and remove
# historical URL-only entries. GSC 2026-11-03 still reports 503 indexed-but-
# blocked search URLs. Do not re-add /search disallows: search.php supplies the
# indexing exclusion, and internal search/facet links remain nofollow.
# Faceted category filters - BLOCK ADDED 2026-07-14, REMOVED 2026-08-08.
#
# DO NOT RE-ADD THESE. The 2026-07-14 block was:
# Disallow: /category/*cuisine= (+ vibe, good_for, features, type, style,
# price_band, pickup, min_rating)
# Its stated rationale was that blocking was "safe here precisely because
# they're already noindex'd + not indexed (no 'indexed though blocked' risk)".
# That was exactly backwards, and GSC proved it within two weeks.
#
# WHY IT BACKFIRED: a robots.txt block stops Googlebot DOWNLOADING the page, so
# Google can never read the noindex tag on it. Faced with a URL it may not fetch
# but knows exists, Google indexes the URL with no content. The block therefore
# converted ~49,745 URLs sitting cleanly in "Excluded by noindex" into a pile of
# URL-only index entries that reached **76,201 "Indexed, though blocked by
# robots.txt"** by 2026-08-08 - 78% of the site's entire reported 97.6k indexed
# count, against a real sitemap of ~10,600 URLs. robots.txt and noindex must
# never be applied to the same URL: they cancel each other out.
#
# (This is the same lesson as the dirty-URL section further down, learned
# 2026-07-03 - five days before the opposite conclusion was drawn here.)
#
# THE FILTER URLS ARE STILL FULLY HANDLED, just correctly now:
# * category.php emits on any filtered URL
# * every filtered URL canonicals to the clean /category/
# * all 25 facet links in the filter rail carry rel="nofollow" (verified live
# 2026-08-08), so unblocking does NOT reopen the discovery floodgates the
# way it would have in July - that nofollow is what makes this safe.
# Google can now crawl them, finally SEE the noindex, and drop them for good.
# Expect a temporary crawl-budget bump while it works through the backlog, and
# the "Indexed, though blocked" bucket to drain over the following weeks.
#
# If facet crawling ever needs limiting again, the answer is nofollow / fewer
# linked combinations / GSC parameter handling - NOT a robots.txt Disallow on a
# noindex'd URL.
# User-personal page, no SEO value
Disallow: /favorites.php
# Don't waste crawl budget on the legacy generator path (rewrites point Google at clean URLs)
Disallow: /sitemap-generator.php
# Don't index the business-owner area beyond register/login
Disallow: /business-owner/add-business.php
Disallow: /business-owner/claim-business.php
Disallow: /business-owner/manage-images.php
Disallow: /business-owner/account-settings.php
# Allow important pages
Allow: /business-owner/register.php
Allow: /business-owner/login.php
# Note: legacy /business.php and /category.php URLs 301 to clean /business/
# and /category/, so Googlebot must be allowed to crawl them to follow the
# redirects.
# Dirty-URL disallows REMOVED 2026-07-03 (acting on the standing plan below).
# These previously blocked clean-path query-string dupes like
# /business/foo?slug=foo&lang=es, produced by an old header.php language-switcher
# bug that echoed $_GET back into hrefs. That source bug was fixed 2026-05-23, so
# NO new dirty URLs are generated - the rules were pure legacy cruft.
#
# Why removed: GSC's "Indexed, though blocked by robots.txt" climbed to ~3 (the
# documented trigger was ~1). For a dirty URL Google has ALREADY indexed, the
# block is counterproductive - Google can't recrawl to see the page's
# self-canonical (all these paths canonical to the clean URL), so the dupe stays
# frozen in the index instead of consolidating. Unblocking lets Google crawl ->
# read the canonical -> merge onto the clean URL. (Blocking never accelerated
# deindexing anyway; only crawl-and-see-canonical / noindex / GSC removal does.)
# If a fresh wave of dirty URLs ever appears, fix the SOURCE (don't re-add these).
#
# Legacy /business.php /category.php /business_details.php stay crawlable (no rule
# blocks them) so their 301s to clean URLs keep firing.
# LLM / AI crawlers - explicit welcome (default is Allow already, but being
# explicit signals intent and future-proofs against any default-deny changes).
# Crawlers listed here power answer engines that cite their sources, sending
# referral traffic when they recommend our pages.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: Bytespider
Allow: /
Sitemap: https://malaga-directory.com/sitemap.xml