Reviewing Indexing & Crawling: SEO Tools for Google and AI Crawlers
Free individual checks and generators · without registration
Check whether search engines are allowed to retrieve your URLs, which indexing signals a page sends, and whether your sitemaps match the current website status.
Allow crawling and test changes
- Robots.txt CheckerChecks Robots.txt files for accessibility, syntax, rules, locks and hints on sitemaps.
- Robots.txt GeneratorCreates a clean robots.txt with user-agent, Allow, Disallow and optional sitemap entry – including preview, copy and download.
- robots.txt control testerTest inserted robots.txt rules against specific URLs and show the decisive Allow or Disallow rule with row number.
- robots.txt change simulatorCompare before and after: Find newly blocked or shared URLs from lists and sitemaps.
Clarify indexing and preferred URLs
- Indexability CheckerCheck if a page is technically indexable – including status code, canonical, robot signals and HTML basics.
- Meta Robots CheckerChecks meta-robots and X-robots signals for index, noindex, follow, nofollow and other crawler clues.
- Meta Robots GeneratorCreates clean meta-robot tags for index, noindex, follow, nofollow and other robots directives – including preview and copy & paste code.
- Canonical CheckerCheck canonical tags, redirects, and destination URLs – to find conflicting signals on tracking parameters or URL variants.
- Canon URL GeneratorCreates clean rel="canonical" tags for pages, variant URLs and duplicate content scenarios – including preview and copy.
- SEO URL CheckerChecks a URL for status code, redirects, canonical, meta robots, X-Robots tag, title, H1, hreflang, and technical indexability.
Create, check and compare sitemaps
- XML Sitemap CheckerChecks existing sitemaps for error-freeness and lists contained URLs incl. Last-modified.
- Sitemap Audit ToolCheck up to 50 Sitemap entries for final status, Meta Robots and Canonical. The sample helps with targeted sitemap corrections.
- XML Sitemap GeneratorCreates a clean sitemap.xml from a URL list – including preview, download and optional lastmod, changefreq and priority.
- XML Sitemap Diff CheckerCompare two sitemap states: new, removed and modified URLs with lastmod and other metadata. With file import, filters and CSV export.
- HTML sitemap generatorCreates an HTML sitemap for visitors from XML or URL lists. Edit link texts, exclude pages and export finished HTML.
- Image Sitemap GeneratorCreates XML image sitemaps from page and image URLs. Groups images per page and removes duplicate assignments – locally in the browser.
- Video Sitemap GeneratorCreates XML video sitemaps from video pages, thumbnails and player URLs – with form, CSV import and download.
- Google News Sitemap GeneratorCreates news sitemaps for current articles, checks publication dates, and displays older, omitted entries.
- RSS/Atom feed checkerCheck RSS and Atom for XML errors, duplicate entries, dates and accessible article links.
Check accessibility and language versions additionally
- Orphan Page CheckerCompares sitemap URLs to a crawl of internal links. Shows undiscovered candidates, source pages, and actual crawl coverage.
- Hreflang CheckerChecks language codes, references and canonical conflicts – to find errors in the hreflang linking of international sites.
- HTTP status code checkerChecks HTTP status codes for URL, URL list or sitemap and shows 200 OK, 301/302, 404/410, 5xx and final destination URL.
Then they join the exams.
Start with a specific URL
First, check the HTTP status, indexing rules, and canonical of an important page. A reachable URL can still be excluded from indexing by noindex.
Indexability CheckerTest a rule change in advance
Before publishing, compare which example URLs a modified robots.txt rule set allows or blocks. A crawl lock does not replace a noindex statement.
robots.txt change simulatorCrosscheck URL inventory
Compare the sitemap before and after a changeover and then examine new or removed URLs. A sitemap does not guarantee indexing.
XML Sitemap Diff Checker
Frequent questions
Is an XML sitemap sufficient for indexing?
No. A sitemap supports the discovery of URLs. Accessibility, crawling, indexing rules and content must additionally fit; the inclusion in the index is decided by the search engine.
Which tool is reviewing a planned robots.txt change?
The change simulator compares two sets of rules using example URLs. The robots.txt checker, on the other hand, checks the published file.
Checking the website in connection
The free SEO check examines up to 20 URLs and bundles findings and prioritized measures.