Merge pull request #8 from proxymesh/feature/python-proxy-headers-examples

proxymesh · web-flow · commit 2730d0d5d433 · 2026-02-08T09:07:45.000-08:00
Add Python proxy header examples for all extensions
diff --git a/.cursorrules b/.cursorrules
@@ -0,0 +1,19 @@
+# Cursor Rules for proxy-examples
+
+## Git Configuration
+- Always commit as `cursor@proxymesh.com` with name `Cursor`
+- Run these commands if not already configured:
+  ```
+  git config user.email "cursor@proxymesh.com"
+  git config user.name "Cursor"
+  ```
+
+## Pull Requests
+- Do NOT add "Made with Cursor" or similar footers to PR descriptions
+- After creating a PR with `gh pr create`, immediately run `gh pr edit <number> --body "..."` to ensure no footer was appended
+- Keep PR descriptions concise and focused on the changes
+
+## Code Style
+- Follow existing code patterns in the repository
+- Keep examples simple and focused on demonstrating the feature
+- Include comments linking to documentation
diff --git a/README.md b/README.md
@@ -7,19 +7,52 @@ Example code for using proxy servers in different programming languages. Current
 
 ## Python Proxy Examples
 
-* [requests-proxy](https://github.com/proxymesh/proxy-examples/blob/main/python/requests-proxy.py)
-* [requests-random-proxy](https://github.com/proxymesh/proxy-examples/blob/main/python/requests-random-proxy.py)
-* [requests-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/requests-proxy-headers.py)
-* [urllib3-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/urllib3-proxy-headers.py)
-* [aiohttp-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/aiohttp-proxy-headers.py)
-* [httpx-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/httpx-proxy-headers.py)
-* [httpx-async-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/httpx-async-proxy-headers.py)
-* [scrapy-proxy-headers](https://github.com/proxymesh/proxy-examples/blob/main/python/scrapy-proxy-headers.py)
+### Using python-proxy-headers
+
+The [python-proxy-headers](https://github.com/proxymesh/python-proxy-headers) library enables sending custom headers to proxy servers and receiving proxy response headers. This is essential for services like [ProxyMesh](https://proxymesh.com) that use custom headers for country selection and IP assignment.
+
+**Installation:**
+
+```bash
+pip install python-proxy-headers
+```
+
+**Examples:**
+
+| Library | Example | Description |
+|---------|---------|-------------|
+| [requests](https://docs.python-requests.org/) | [requests-proxy-headers.py](python/requests-proxy-headers.py) | Simple HTTP requests with proxy headers |
+| requests | [requests-proxy-headers-session.py](python/requests-proxy-headers-session.py) | Session-based requests for connection pooling |
+| [urllib3](https://urllib3.readthedocs.io/) | [urllib3-proxy-headers.py](python/urllib3-proxy-headers.py) | Low-level HTTP client with proxy headers |
+| [aiohttp](https://docs.aiohttp.org/) | [aiohttp-proxy-headers.py](python/aiohttp-proxy-headers.py) | Async HTTP client with proxy headers |
+| [httpx](https://www.python-httpx.org/) | [httpx-proxy-headers.py](python/httpx-proxy-headers.py) | Modern HTTP client with proxy headers |
+| httpx | [httpx-async-proxy-headers.py](python/httpx-async-proxy-headers.py) | Async httpx with proxy headers |
+| [pycurl](http://pycurl.io/) | [pycurl-proxy-headers.py](python/pycurl-proxy-headers.py) | libcurl bindings with proxy headers |
+| pycurl | [pycurl-proxy-headers-lowlevel.py](python/pycurl-proxy-headers-lowlevel.py) | Low-level pycurl integration |
+| [cloudscraper](https://github.com/venomous/cloudscraper) | [cloudscraper-proxy-headers.py](python/cloudscraper-proxy-headers.py) | Cloudflare bypass with proxy headers |
+| [autoscraper](https://github.com/alirezamika/autoscraper) | [autoscraper-proxy-headers.py](python/autoscraper-proxy-headers.py) | Automatic web scraping with proxy headers |
+
+### Basic Proxy Examples
+
+* [requests-proxy.py](python/requests-proxy.py) - Basic proxy usage with requests
+* [requests-random-proxy.py](python/requests-random-proxy.py) - Random proxy rotation
+
+### Scrapy
+
+* [scrapy-proxy-headers.py](python/scrapy-proxy-headers.py) - Scrapy spider with proxy headers
 
 ## Ruby Proxy Examples
 
-* [requests_proxy](https://github.com/proxymesh/proxy-examples/blob/main/ruby/requests_proxy.rb) from [rpolley](https://github.com/rpolley)
+* [requests_proxy.rb](ruby/requests_proxy.rb) - Ruby HTTP with proxy, from [rpolley](https://github.com/rpolley)
+
+## Documentation
+
+For more information on using proxy headers with Python:
+
+* [python-proxy-headers on PyPI](https://pypi.org/project/python-proxy-headers/)
+* [python-proxy-headers Documentation](https://python-proxy-headers.readthedocs.io/)
+* [GitHub Repository](https://github.com/proxymesh/python-proxy-headers)
 
 ## Contributing
 
-If you have example code for another language, please share it with a Pull Request.
+If you have example code for another language, please share it with a Pull Request.
diff --git a/python/aiohttp-proxy-headers.py b/python/aiohttp-proxy-headers.py
@@ -1,8 +1,34 @@
-# See https://github.com/proxymesh/python-proxy-headers
+#!/usr/bin/env python3
+"""
+aiohttp with proxy headers example.
+
+This example shows how to send custom headers to a proxy server and
+receive proxy response headers using the python-proxy-headers library
+with async/await.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html
+"""
+import asyncio
 from python_proxy_headers import aiohttp_proxy
 
-async with aiohttp_proxy.ProxyClientSession() as session:
-	async with session.get('https://api.ipify.org?format=json', proxy="http://PROXYHOST:PORT", proxy_headers={'X-ProxyMesh-Country': 'US'}) as r:
-		await r.text()
 
-r.headers['X-ProxyMesh-IP']
+async def main():
+    async with aiohttp_proxy.ProxyClientSession() as session:
+        async with session.get(
+            'https://api.ipify.org?format=json',
+            proxy='http://USERNAME:PASSWORD@PROXYHOST:PORT',
+            proxy_headers={'X-ProxyMesh-Country': 'US'}
+        ) as response:
+            # Print response
+            print(f"Status: {response.status}")
+            body = await response.text()
+            print(f"Body: {body}")
+            
+            # Access proxy response headers (from CONNECT response)
+            proxy_ip = response.headers.get('X-ProxyMesh-IP')
+            print(f"Proxy IP: {proxy_ip}")
+
+
+if __name__ == '__main__':
+    asyncio.run(main())
diff --git a/python/autoscraper-proxy-headers.py b/python/autoscraper-proxy-headers.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""
+AutoScraper with proxy headers example.
+
+This example shows how to use AutoScraper with custom proxy headers.
+AutoScraper automatically learns scraping rules while python-proxy-headers
+enables sending custom headers to the proxy server.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/autoscraper.html
+"""
+from python_proxy_headers.autoscraper_proxy import ProxyAutoScraper
+
+# Create an AutoScraper with proxy header support
+scraper = ProxyAutoScraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
+
+# Proxy configuration
+proxies = {
+    'http': 'http://USERNAME:PASSWORD@PROXYHOST:PORT',
+    'https': 'http://USERNAME:PASSWORD@PROXYHOST:PORT'
+}
+
+# Build scraping rules from a sample page
+# AutoScraper learns what to extract based on the wanted_list
+result = scraper.build(
+    url='https://quotes.toscrape.com/',
+    wanted_list=['The world as we have created it is a process of our thinking.'],
+    request_args={'proxies': proxies}
+)
+
+print(f"Found {len(result)} matching quotes")
+for quote in result[:5]:
+    print(f"  - {quote[:60]}...")
+
+# Save learned rules for later use
+# scraper.save('quotes_rules.json')
+
+# Use learned rules on another page
+# result = scraper.get_result_similar(
+#     url='https://quotes.toscrape.com/page/2/',
+#     request_args={'proxies': proxies}
+# )
+
+scraper.close()
diff --git a/python/cloudscraper-proxy-headers.py b/python/cloudscraper-proxy-headers.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+"""
+CloudScraper with proxy headers example.
+
+This example shows how to use CloudScraper with custom proxy headers.
+CloudScraper bypasses Cloudflare protection while python-proxy-headers
+enables sending/receiving custom proxy headers.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/cloudscraper.html
+"""
+from python_proxy_headers.cloudscraper_proxy import create_scraper
+
+# Create a CloudScraper with proxy header support
+scraper = create_scraper(
+    proxy_headers={'X-ProxyMesh-Country': 'US'},
+    browser='chrome'
+)
+
+# Set proxy
+scraper.proxies = {
+    'http': 'http://USERNAME:PASSWORD@PROXYHOST:PORT',
+    'https': 'http://USERNAME:PASSWORD@PROXYHOST:PORT'
+}
+
+# Make request - Cloudflare bypass + proxy headers work together
+response = scraper.get('https://api.ipify.org?format=json')
+
+# Print response
+print(f"Status: {response.status_code}")
+print(f"Body: {response.text}")
+
+# Access proxy response headers
+proxy_ip = response.headers.get('X-ProxyMesh-IP')
+print(f"Proxy IP: {proxy_ip}")
diff --git a/python/httpx-async-proxy-headers.py b/python/httpx-async-proxy-headers.py
@@ -1,11 +1,39 @@
+#!/usr/bin/env python3
+"""
+httpx async with proxy headers example.
+
+This example shows how to send custom headers to a proxy server and
+receive proxy response headers using httpx's async client.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/httpx.html
+"""
+import asyncio
 import httpx
-# See https://github.com/proxymesh/python-proxy-headers
 from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport
 
-proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
-transport = AsyncHTTPProxyTransport(proxy=proxy)
 
-async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client:
-	r = await client.get('https://api.ipify.org?format=json')
+async def main():
+    # Create a proxy with custom headers
+    proxy = httpx.Proxy(
+        'http://USERNAME:PASSWORD@PROXYHOST:PORT',
+        headers={'X-ProxyMesh-Country': 'US'}
+    )
+    
+    # Create async transport with proxy header support
+    transport = AsyncHTTPProxyTransport(proxy=proxy)
+    
+    async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client:
+        response = await client.get('https://api.ipify.org?format=json')
+        
+        # Print response
+        print(f"Status: {response.status_code}")
+        print(f"Body: {response.text}")
+        
+        # Access proxy response headers (from CONNECT response)
+        proxy_ip = response.headers.get('X-ProxyMesh-IP')
+        print(f"Proxy IP: {proxy_ip}")
+
 
-r.headers['X-ProxyMesh-IP']
+if __name__ == '__main__':
+    asyncio.run(main())
diff --git a/python/httpx-proxy-headers.py b/python/httpx-proxy-headers.py
@@ -1,7 +1,29 @@
+#!/usr/bin/env python3
+"""
+httpx with proxy headers example.
+
+This example shows how to send custom headers to a proxy server and
+receive proxy response headers using the python-proxy-headers library.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/httpx.html
+"""
 import httpx
-# See https://github.com/proxymesh/python-proxy-headers
 from python_proxy_headers import httpx_proxy
 
-proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
-r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy)
-r.headers['X-ProxyMesh-IP']
+# Create a proxy with custom headers
+proxy = httpx.Proxy(
+    'http://USERNAME:PASSWORD@PROXYHOST:PORT',
+    headers={'X-ProxyMesh-Country': 'US'}
+)
+
+# Make a request using the helper function
+response = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy)
+
+# Print response
+print(f"Status: {response.status_code}")
+print(f"Body: {response.text}")
+
+# Access proxy response headers (from CONNECT response)
+proxy_ip = response.headers.get('X-ProxyMesh-IP')
+print(f"Proxy IP: {proxy_ip}")
diff --git a/python/pycurl-proxy-headers-lowlevel.py b/python/pycurl-proxy-headers-lowlevel.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""
+PycURL with proxy headers - low-level example.
+
+This example shows how to add proxy header support to existing pycurl code
+using the low-level helper functions from python-proxy-headers.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/pycurl.html
+"""
+import pycurl
+from io import BytesIO
+from python_proxy_headers.pycurl_proxy import set_proxy_headers, HeaderCapture
+
+# Create a pycurl handle
+c = pycurl.Curl()
+buffer = BytesIO()
+
+# Configure the request
+c.setopt(pycurl.URL, 'https://api.ipify.org?format=json')
+c.setopt(pycurl.PROXY, 'http://USERNAME:PASSWORD@PROXYHOST:PORT')
+c.setopt(pycurl.WRITEDATA, buffer)
+
+# Add custom headers to send to the proxy
+set_proxy_headers(c, {'X-ProxyMesh-Country': 'US'})
+
+# Capture response headers (installs HEADERFUNCTION callback)
+capture = HeaderCapture(c)
+
+# Perform the request
+c.perform()
+
+# Print results
+print(f"Status: {c.getinfo(pycurl.RESPONSE_CODE)}")
+print(f"Body: {buffer.getvalue().decode('utf-8')}")
+
+# Access headers from the proxy's CONNECT response
+print(f"Proxy headers: {capture.proxy_headers}")
+print(f"Proxy IP: {capture.proxy_headers.get('X-ProxyMesh-IP')}")
+
+# Access headers from the origin server
+print(f"Origin headers: {capture.origin_headers}")
+
+c.close()
diff --git a/python/pycurl-proxy-headers.py b/python/pycurl-proxy-headers.py
@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+"""
+PycURL with proxy headers example.
+
+This example shows how to send custom headers to a proxy server and
+receive proxy response headers using the python-proxy-headers library.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/pycurl.html
+"""
+from python_proxy_headers.pycurl_proxy import get
+
+# Make a request through the proxy with custom headers
+response = get(
+    'https://api.ipify.org?format=json',
+    proxy='http://USERNAME:PASSWORD@PROXYHOST:PORT',
+    proxy_headers={'X-ProxyMesh-Country': 'US'}
+)
+
+# Print response
+print(f"Status: {response.status_code}")
+print(f"Body: {response.text}")
+
+# Access proxy response headers (from CONNECT response)
+proxy_ip = response.proxy_headers.get('X-ProxyMesh-IP')
+print(f"Proxy IP: {proxy_ip}")
diff --git a/python/requests-proxy-headers-session.py b/python/requests-proxy-headers-session.py
@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+"""
+Requests with proxy headers - Session example.
+
+This example shows how to use ProxySession for connection pooling
+and making multiple requests with the same proxy header configuration.
+
+See: https://github.com/proxymesh/python-proxy-headers
+Docs: https://python-proxy-headers.readthedocs.io/en/latest/requests.html
+"""
+from python_proxy_headers.requests_adapter import ProxySession
+
+proxies = {
+    'http': 'http://USERNAME:PASSWORD@PROXYHOST:PORT',
+    'https': 'http://USERNAME:PASSWORD@PROXYHOST:PORT'
+}
+
+# Create a session with proxy header support
+with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session:
+    session.proxies = proxies
+    
+    # Make multiple requests with the same session
+    r1 = session.get('https://api.ipify.org?format=json')
+    print(f"Request 1 - IP: {r1.json()['ip']}, Proxy IP: {r1.headers.get('X-ProxyMesh-IP')}")
+    
+    r2 = session.get('https://httpbin.org/headers')
+    print(f"Request 2 - Status: {r2.status_code}")
diff --git a/python/requests-proxy-headers.py b/python/requests-proxy-headers.py
diff --git a/python/urllib3-proxy-headers.py b/python/urllib3-proxy-headers.py
