crawler.sh is a website-crawling and SEO-audit tool built by independent developer Mehmet Kose, launched on Product Hunt on March 2, 2026. Kose built it after running into the same problem repeatedly: needing to choose between bloated enterprise crawling suites, slow cloud services, or a pile of custom scripts to audit a site or extract its content.
The tool runs locally from a terminal or a native desktop app built with Rust and Tauri, crawling a site, running 24 automated SEO checks covering issues like missing titles, duplicate descriptions, and thin content, and extracting page content as clean Markdown for AI training or retrieval-augmented generation use. Because it runs on the user's own machine, there is no per-page API fee and no account requirement for the free tier.
Developers, SEO practitioners, and content teams who want fast, private site audits without cloud costs are the target users. A Free tier crawls up to 400 pages per session for signed-in users, and a Pro plan at $99 a year raises that to 10,000 pages and unlocks Content Archive export plus NDJSON output.
