Python · Projects and Interview · Advanced · 50 min read
Project: Build a Polite Async Web Crawler with Rate Limiting, robots.txt, Graceful Shutdown and Persistence
Build a concurrent crawler with asyncio and httpx. Covers requirements, architecture with a frontier queue and worker pool, URL normalisation and deduplication, robots.txt and per-host politeness, rate limiting and concurrency limits, retries and timeouts, HTML parsing and link extraction, persistence and resumability with SQLite, graceful shutdown on signals, metrics and logging, tests with a fake server, and ethical and legal considerations.
Create an account to start
Create a free account to see what this lesson covers
- Every free lesson in every section, in full
- Track your progress and pick up where you left off
- Free practice questions and an incident simulation
- Takes under a minute: name, email and a password
No card needed. Full access to Python is a paid upgrade you can choose later.