Scrapy Proxy Middleware
Rotate HTTP proxies in Scrapy with middleware, failed proxy tracking, and download slot tuning.
Rotate HTTP proxies in Scrapy with middleware, failed proxy tracking, and download slot tuning.
Middleware pattern
Set request.meta proxy URL in downloader middleware. Rotate from Redis or file on each request.
Ban detection
Drop proxies that return repeated 403 or captcha responses.
Related: rotation strategies.
Concurrency tuning
Limit concurrent requests per proxy IP to avoid burning slots.
Related: connection limits.
Need proxies at scale?
proxies.st offers health-checked HTTP and SOCKS pools with dashboard access, API keys, and plain-text bulk feeds for pipelines.
Related guides
Using Proxies with Python requests and httpx
Configure HTTP and SOCKS proxies in Python with requests and httpx — rotation, error handling, and production patterns.
Rotating Proxies for Web Scraping at Scale
Rotation strategies, session stickiness, anti-bot considerations, and scraper architecture for high-volume crawling.