Skip to content
TechIndexSubmit
Back to directory

markdown-crawler

A multithreaded πŸ•ΈοΈ web crawler that recursively crawls a website and creates a πŸ”½ markdown file for each page, designed for LLM RAG

Similar tools

  • Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

    Models
  • Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data β€” all from Rust. CLI, REST API, and MCP server.

    MCP Servers
  • πŸ”₯ Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

    MCP Servers
  • Open-source multi-server Discord channel scraper. Configure many servers in one TOML file, pick channels by name or ID, export to JSONL for offline search and LLM pipelines.

    Datasets
  • A GenAI-powered Website Q&A system built using RAG (Retrieval-Augmented Generation). Users can input any URL and ask questions about its content. The system uses web scraping, embeddings, vector search (ChromaDB), and Groq Llama 3 models to generate context-aware answers.

    AI Apps
  • XCrawl gives developers one API for web data. Scrape URLs into Markdown, JSON, HTML or screenshots; crawl sites; map domains; and fetch structured SERP results. Rendering, proxies and retries run server-side. Python/Node SDKs, CLI, MCP server and n8n node are available; new accounts get 1,000 free credits.

    Dev Tools