Submit a tool
Freemium

crawler.sh

crawler.sh runs SEO audits and Markdown content extraction from a local Rust-based crawler instead of a cloud crawling service.

3.9 SEO
3.9 Good 3.9
Running locally avoids the recurring cloud costs typical crawlers charge Built and maintained by one developer, a continuity risk versus funded competitors
Reviewed by Challenging Voice Editorial · Updated Jul 2026 How we rate
PricingFrom $8.25/mo
Free planYes
Companycrawler.sh
PlatformsMac, Windows
CategorySEO
Founded2026
Visits5
Last reviewedJul 2026
UpdatedJul 2026
Ask AI about crawler.sh ChatGPT Claude Perplexity

Overview

crawler.sh is a website-crawling and SEO-audit tool built by independent developer Mehmet Kose, launched on Product Hunt on March 2, 2026. Kose built it after running into the same problem repeatedly: needing to choose between bloated enterprise crawling suites, slow cloud services, or a pile of custom scripts to audit a site or extract its content.

The tool runs locally from a terminal or a native desktop app built with Rust and Tauri, crawling a site, running 24 automated SEO checks covering issues like missing titles, duplicate descriptions, and thin content, and extracting page content as clean Markdown for AI training or retrieval-augmented generation use. Because it runs on the user's own machine, there is no per-page API fee and no account requirement for the free tier.

Developers, SEO practitioners, and content teams who want fast, private site audits without cloud costs are the target users. A Free tier crawls up to 400 pages per session for signed-in users, and a Pro plan at $99 a year raises that to 10,000 pages and unlocks Content Archive export plus NDJSON output.

Key features

  • Local-first crawling with no cloud service or per-page fees
  • 24 automated SEO checks including thin content and duplicate description detection
  • Clean Markdown extraction built for AI training and RAG pipelines
  • Native desktop app plus terminal and CLI access built on Rust and Tauri
  • JavaScript rendering through a custom engine, no headless Chrome dependency
  • Export to JSON, NDJSON, CSV, and Sitemap XML formats

Screenshots & demo

crawler.sh screenshot 1

Pricing

Free
$0
  • Crawl up to 50 pages per session
  • SEO analysis with CSV/TXT export
  • JSON and Sitemap XML export
  • No account required
CLI Pro
$99/year
  • Crawl up to 10,000 pages per session
  • NDJSON, JSON, Sitemap XML output formats
  • Markdown content extraction
  • 16-category SEO analysis
Desktop Pro
$99/year
  • Visual dashboard with 8 interactive cards
  • Real-time crawl feed with status badges
  • SEO issues panel with per-URL details
  • Content archive export as Markdown

Is crawler.sh expensive?

crawler.sh starts at $8.25, which is 81% below the SEO median of $43.8 a month across the 68 priced tools we list in that category.

The cheapest paid option in SEO starts at $7 and the most expensive at $390.

39% of SEO tools in the directory offer a free tier, and this is one of them.

Compared against every priced listing in SEO, recalculated as the catalogue changes. Method and the full market breakdown are in our AI tool pricing study.

Pros & cons

Pros

  • Running locally avoids the recurring cloud costs typical crawlers charge
  • Free tier's 400-pages-per-session limit covers most small-to-mid site audits
  • Markdown output is purpose-built for feeding AI training and RAG pipelines
  • One-time $99 annual Pro fee undercuts subscription-based crawling tools

Cons

  • Built and maintained by one developer, a continuity risk versus funded competitors
  • New product launched March 2026, limited independent long-term reviews
  • Local-only architecture means no built-in team collaboration or shared dashboards
  • JavaScript-heavy sites may render differently than they would under real Chrome

How it compares

ToolRatingFreeFromBest known for
crawler.sh (this tool)3.9Yes$8.25/moLocal-first crawling with no cloud service or per-page fees
Screaming Frog SEO Spider4.6Yes$23.25/moDesktop software crawling a site locally, not a cloud SaaS dashboard
Sitebulb4.5No$15/moDual access: desktop application and cloud-based platform
Ecommerce Booster4.0Yes$29.99/moFull Shopify store crawl scoring text, images, speed, and accessibility

Alternatives to crawler.sh

4 tools matched to crawler.sh on what they do, their category and their price.

Frequently asked questions

What is crawler.sh?
crawler.sh is a website-crawling and SEO-audit tool built by independent developer Mehmet Kose, launched on Product Hunt on March 2, 2026.
Is crawler.sh free?
Yes, crawler.sh offers a free plan. Paid plans unlock more features and higher usage limits, starting at $8.25/month.
How much does crawler.sh cost?
crawler.sh plans: Free $0; CLI Pro $99/year; Desktop Pro $99/year.
What are the best crawler.sh alternatives?
The closest matches in the directory are Screaming Frog SEO Spider, Sitebulb, and Ecommerce Booster, compared side by side above.

Reviews

3.9 No reviews yet

Write a review

More tools like crawler.sh

Copywritely

3.7

Audits existing pages for the content problems holding their rankings back, then walks you through rewriting them, with a Chrome extension and API.

Free Trial SEO
View details Visit

RankNow.ai

3.4

13 AI tools for keyword research, rank tracking, and SEO audits – plus an AEO Analyzer that shows how ChatGPT, Claude, Gemini, and Perplexity actually see your content.

Freemium SEO
View details Visit