Using Proxies with Python requests and httpx

Python remains the default language for data extraction. The requests library supports HTTP and HTTPS over HTTP proxies out of the box. For SOCKS, add requests[socks] or use httpx with transport plugins. This guide covers patterns that survive scale.

Basic setup with requests

import requests

proxies = {
    "http": "http://user:[email protected]:8080",
    "https": "http://user:[email protected]:8080",
}

resp = requests.get("https://api.example.com/data", proxies=proxies, timeout=30)
resp.raise_for_status()

Use the same proxy URL for both http and https keys — HTTPS still tunnels via CONNECT through the HTTP proxy endpoint.

Rotating proxies

import itertools
import requests

pool = ["http://203.0.113.10:8080", "http://203.0.113.11:8080"]
cycle = itertools.cycle(pool)

def fetch(url):
    proxy = next(cycle)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

# Load lists from bulk feeds — see bulk proxy lists guide

Load pools from bulk proxy list files or plain-text API feeds. Filter dead endpoints with health checks before rotation.

Using httpx

httpx offers async support and clean proxy configuration for concurrent crawlers:

import httpx

async with httpx.AsyncClient(proxy="http://203.0.113.10:8080") as client:
    r = await client.get("https://example.com")
    r.raise_for_status()

Error handling

• Retry requests.exceptions.ProxyError with a new proxy from the pool.
• Treat 407 as credential failure — do not infinite-retry the same proxy.
• Log latency per proxy for pool quality metrics.

Production tips

• Cap concurrent connections per proxy to avoid burning endpoints.
• Combine rotation with backoff on 429 responses.
• Understand HTTP vs SOCKS5 before picking list types.