# Robots.txt for Malaga Directory # https://malaga-directory.com User-agent: * Allow: / # Private areas Disallow: /admin/ Disallow: /business-owner/dashboard.php Disallow: /business-owner/edit-*.php Disallow: /includes/ Disallow: /config/ Disallow: /api/ # NOTE: the rule above matches ONLY paths starting /api/. It does NOT cover # /business-owner/api/, which is a separate directory - that gap let Googlebot # crawl four authenticated endpoints (connect-place, fetch-rating, # search-claimable, set-featured-review) that correctly answer 401 to anonymous # requests, producing GSC's "Blocked due to unauthorised request (401)" alert on # 2026-08-12. The 401s are right; Google reaching them was not. Blocking is the # correct tool here (unlike the facet case below): these are JSON/401 responses # with no meta robots and no X-Robots-Tag, so there is no noindex to cancel. Disallow: /business-owner/api/ # Search results stay crawlable so Google can read noindex,follow and remove # historical URL-only entries. GSC 2026-11-03 still reports 503 indexed-but- # blocked search URLs. Do not re-add /search disallows: search.php supplies the # indexing exclusion, and internal search/facet links remain nofollow. # Faceted category filters - BLOCK ADDED 2026-07-14, REMOVED 2026-08-08. # # DO NOT RE-ADD THESE. The 2026-07-14 block was: # Disallow: /category/*cuisine= (+ vibe, good_for, features, type, style, # price_band, pickup, min_rating) # Its stated rationale was that blocking was "safe here precisely because # they're already noindex'd + not indexed (no 'indexed though blocked' risk)". # That was exactly backwards, and GSC proved it within two weeks. # # WHY IT BACKFIRED: a robots.txt block stops Googlebot DOWNLOADING the page, so # Google can never read the noindex tag on it. Faced with a URL it may not fetch # but knows exists, Google indexes the URL with no content. The block therefore # converted ~49,745 URLs sitting cleanly in "Excluded by noindex" into a pile of # URL-only index entries that reached **76,201 "Indexed, though blocked by # robots.txt"** by 2026-08-08 - 78% of the site's entire reported 97.6k indexed # count, against a real sitemap of ~10,600 URLs. robots.txt and noindex must # never be applied to the same URL: they cancel each other out. # # (This is the same lesson as the dirty-URL section further down, learned # 2026-07-03 - five days before the opposite conclusion was drawn here.) # # THE FILTER URLS ARE STILL FULLY HANDLED, just correctly now: # * category.php emits on any filtered URL # * every filtered URL canonicals to the clean /category/ # * all 25 facet links in the filter rail carry rel="nofollow" (verified live # 2026-08-08), so unblocking does NOT reopen the discovery floodgates the # way it would have in July - that nofollow is what makes this safe. # Google can now crawl them, finally SEE the noindex, and drop them for good. # Expect a temporary crawl-budget bump while it works through the backlog, and # the "Indexed, though blocked" bucket to drain over the following weeks. # # If facet crawling ever needs limiting again, the answer is nofollow / fewer # linked combinations / GSC parameter handling - NOT a robots.txt Disallow on a # noindex'd URL. # User-personal page, no SEO value Disallow: /favorites.php # Don't waste crawl budget on the legacy generator path (rewrites point Google at clean URLs) Disallow: /sitemap-generator.php # Don't index the business-owner area beyond register/login Disallow: /business-owner/add-business.php Disallow: /business-owner/claim-business.php Disallow: /business-owner/manage-images.php Disallow: /business-owner/account-settings.php # Allow important pages Allow: /business-owner/register.php Allow: /business-owner/login.php # Note: legacy /business.php and /category.php URLs 301 to clean /business/ # and /category/, so Googlebot must be allowed to crawl them to follow the # redirects. # Dirty-URL disallows REMOVED 2026-07-03 (acting on the standing plan below). # These previously blocked clean-path query-string dupes like # /business/foo?slug=foo&lang=es, produced by an old header.php language-switcher # bug that echoed $_GET back into hrefs. That source bug was fixed 2026-05-23, so # NO new dirty URLs are generated - the rules were pure legacy cruft. # # Why removed: GSC's "Indexed, though blocked by robots.txt" climbed to ~3 (the # documented trigger was ~1). For a dirty URL Google has ALREADY indexed, the # block is counterproductive - Google can't recrawl to see the page's # self-canonical (all these paths canonical to the clean URL), so the dupe stays # frozen in the index instead of consolidating. Unblocking lets Google crawl -> # read the canonical -> merge onto the clean URL. (Blocking never accelerated # deindexing anyway; only crawl-and-see-canonical / noindex / GSC removal does.) # If a fresh wave of dirty URLs ever appears, fix the SOURCE (don't re-add these). # # Legacy /business.php /category.php /business_details.php stay crawlable (no rule # blocks them) so their 301s to clean URLs keep firing. # LLM / AI crawlers - explicit welcome (default is Allow already, but being # explicit signals intent and future-proofs against any default-deny changes). # Crawlers listed here power answer engines that cite their sources, sending # referral traffic when they recommend our pages. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: cohere-ai Allow: / User-agent: Bytespider Allow: / Sitemap: https://malaga-directory.com/sitemap.xml