From 6544f0247cb4cab499f43a87b24f5d04d4791a5c Mon Sep 17 00:00:00 2001 From: songtianlun Date: Thu, 23 Jan 2025 19:03:00 +0800 Subject: [PATCH] chore: update robots.txt for better indexing - Add disallow rules for /admin/ to prevent indexing - Update sitemap URL to point to the correct domain These changes improve site indexing by ensuring that sensitive admin pages are not accessible to search engine crawlers and updating the sitemap to reflect the accurate URL, helping discovery and SEO efforts. --- public/robots.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/public/robots.txt b/public/robots.txt index e9e0b4c..5c230b6 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,5 +1,7 @@ # See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Allow: / +Disallow: /admin/ +Disallow: /admin -Sitemap: https://your-domain.com/sitemap.xml.gz +Sitemap: https://todayaiweather.com/sitemap.xml.gz