AI Plugin Index
Polite Crawler
Verified · npm:polite-crawlerMCP server by mark1russell7 · updated · ★ 0 · 842 downloads a week
A polite web crawler for research and AI agents. It fetches a source once, keeps it, and reads it many times: robots.txt obeyed, one request per site, full-text search. The MCP server is the package polite-crawler-mcp. Polite Crawler is published by npm:polite-crawler from its verified domain in the official MCP Registry.
Last commit in the past day
MIT: permissive, fine for commercial use
Clear description, published package or server URL, original repository
Not checked yet. What we check
Install Polite Crawler
Built from the published package, never by AIclaude mcp add polite-crawler -- npx -y polite-crawlerRun in your terminal. Add --scope user to use it in every project.
{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler"
]
}
}
}Settings, Developer, Edit config. Paste into claude_desktop_config.json and restart.
{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler"
]
}
}
}Add to ~/.cursor/mcp.json (all projects) or .cursor/mcp.json (this project).
{
"servers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler"
]
}
}
}Add to .vscode/mcp.json. VS Code uses "servers", not "mcpServers".
codex mcp add polite-crawler -- npx -y polite-crawler{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler"
]
}
}
}Add to ~/.gemini/config/mcp_config.json (all projects) or .agents/mcp_config.json (this project).
gemini mcp add polite-crawler npx -y polite-crawlerGemini CLI was replaced by Antigravity CLI in June 2026 and remains only for enterprise licences and paid API keys.
MCP servers run with your permissions and can read what you give them access to. Some need an API key: check the README and add it under env in the polite-crawler block.
Alternatives to Polite Crawler
All alternatives →
Control a real Chrome browser to complete any task: fill forms, extract data, book flights.
Let your AI agent operate any website through a real, logged-in browser session.
Dark web OSINT over Tor: search onion engines, scrape pages, report with your own model.
Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.
draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand
AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.
Polite Crawler: common questions
How do I install Polite Crawler?
In Claude Code run claude mcp add polite-crawler -- npx -y polite-crawler. For Claude Desktop, Cursor and VS Code, paste the config shown above and restart. See the install guides.
Is Polite Crawler the official server?
Polite Crawler is published by npm:polite-crawler from its verified domain in the official MCP Registry.
Is Polite Crawler free?
The code is published under the MIT licence. The service it connects to may have its own pricing.
Is Polite Crawler safe to use?
It scores 81 of 100. Grades: maintenance A, licence A, quality C. The security check has not run for it yet. Review the code before you install. How grades work.
What are alternatives to Polite Crawler?
Similar MCP servers: browser-use, Actionbook, Robin: AI-Powered Dark Web OSINT, Scrapling MCP Server.
Does Polite Crawler work with ChatGPT?
Not directly. It runs locally, and ChatGPT only connects to remote servers with a URL. Use it in Claude Code, Claude Desktop, Cursor, VS Code or Codex.