Merge pull request #1 from davidteather/v0.2.0-add-proxy-picking-algorithm-support

V0.2.0 - Add Algos for picking proxies, Add library support for formatting proxies (and formatter)

Author: davidteather

.github/workflows/ci.yml

@@ -32,9 +32,9 @@ jobs:
       - name: Type check with mypy
         run: poetry run mypy proxyproviders
 
-      - name: Run unit tests with coverage
+      - name: Run unit tests with coverage (excluding integration tests)
         run: |
-          poetry run coverage run -m pytest
+          poetry run coverage run -m pytest tests/ --ignore=tests/integration -v
           poetry run coverage report --fail-under=95
           poetry run coverage xml
 

.pre-commit-config.yaml

@@ -0,0 +1,49 @@
+repos:
+  # Black code formatter
+  - repo: https://github.com/psf/black
+    rev: 23.12.1
+    hooks:
+      - id: black
+        language_version: python3
+        args: [--line-length=88]
+
+  # isort import sorting
+  - repo: https://github.com/pycqa/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        args: [--profile=black]
+
+  # flake8 linting
+  - repo: https://github.com/pycqa/flake8
+    rev: 7.0.0
+    hooks:
+      - id: flake8
+        args: ["--max-line-length=88", "--extend-ignore=E203,W503,E501,F401,F541"]
+
+  - repo: local
+    hooks:
+      # Unit tests
+      - id: pytest-unit
+        name: Run unit tests
+        entry: python3 -m pytest tests/ -v --ignore=tests/integration
+        language: system
+        pass_filenames: false
+        always_run: true
+        stages: [commit]
+
+      # Integration tests
+      - id: pytest-integration
+        name: Run integration tests
+        entry: bash -c 'if [ "$RUN_INTEGRATION_TESTS" = "1" ]; then python3 -m pytest tests/integration/ -v; else echo "Skipping integration tests (env vars not set)"; fi'
+        language: system
+        pass_filenames: false
+        always_run: true
+        stages: [commit]
+
+default_language_version:
+  python: python3
+
+fail_fast: true
+
+default_stages: [commit]

.sphinx/conf.py

@@ -6,10 +6,11 @@
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
-import toml
 import os
 import sys
 
+import toml
+
 sys.path.insert(0, os.path.abspath("../"))
 pyproject = toml.load("../pyproject.toml")
 

CONTRIBUTING.md

@@ -0,0 +1,102 @@
+# Contributing to ProxyProviders
+
+This guide shows you how to add new proxy providers and test them properly.
+
+## Quick Start
+
+1. **Fork and clone**: `git clone https://github.com/your-username/proxyproviders.git`
+2. **Install**: `pip install -e . && pip install pre-commit && pre-commit install`
+3. **Test**: `python -m pytest tests/ --ignore=tests/integration -v`
+
+## Development Setup
+
+### Prerequisites
+
+- Python 3.9 or higher
+- Git
+
+## Running Tests
+
+```bash
+# Unit tests (fast)
+python -m pytest tests/ --ignore=tests/integration -v
+
+# Integration tests (need API keys set for each provider)
+export RUN_INTEGRATION_TESTS=1 WEBSHARE_API_KEY="your-key" BRIGHTDATA_API_KEY="your-key"
+python -m pytest tests/integration/ -v
+
+# Check coverage (should be >95%)
+python -m coverage run -m pytest tests/ --ignore=tests/integration
+python -m coverage report --show-missing
+```
+
+## Adding a New Proxy Provider
+
+If you're adding a new provider that isn't already in the library, here's a short guide to help you get started.
+
+### 1. Create the Provider
+
+Consider looking at `proxyproviders/providers/webshare.py` as an example. Your implemented provider will need to implement the `ProxyProvider` from `proxyproviders/proxy_provider.py`.
+
+Here's a small scaffold that might be useful, however it may be outdated so check the `ProxyProvider` object for the latest required methods.
+
+```python
+class YourProvider(ProxyProvider):
+    def __init__(self, api_key: str, config: Optional[ProxyConfig] = None):
+        super().__init__(config)
+        self.api_key = api_key
+
+    def _fetch_proxies(self) -> List[Proxy]:
+        # Call your provider's API and return List[Proxy]
+        pass
+
+    def _convert_to_proxy(self, data: Dict) -> Proxy:
+        # Convert API response to Proxy object
+        return Proxy(
+            id=str(data["id"]),
+            username=data["username"],
+            password=data["password"],
+            proxy_address=data["host"],
+            port=int(data["port"]),
+            # ... other fields
+        )
+```
+
+### 2. Add to Package
+
+Update `proxyproviders/__init__.py`:
+```python
+from .providers.your_provider import YourProvider
+# Add "YourProvider" to __all__ list
+```
+
+## Writing Tests
+
+### 3. Unit Tests
+
+Create `tests/providers/test_your_provider.py` - feel free tocopy from `test_webshare.py` and modify it:
+
+**Required tests:**
+- Mock API success/error responses
+- Test algorithm integration (`get_proxy()` works)
+- Integration test with real API (if `YOUR_PROVIDER_API_KEY` env var set)
+
+### 4. Integration Test
+
+Create `tests/integration/test_your_provider_integration.py` - copy from existing integration tests.
+
+## Submitting Changes
+
+1. **Run tests**: `python -m pytest tests/ -v`
+2. **Check coverage**: `python -m coverage run -m pytest tests/ --ignore=tests/integration && python -m coverage report`
+3. **Pre-commit passes**: `pre-commit run --all-files`
+4. **Create PR** with:
+   - What provider you're adding
+   - How to get API keys
+   - Test coverage proof
+
+**Requirements for PR approval:**
+- Follows existing code patterns
+- \>95% test coverage
+- Unit + integration tests
+- Clear documentation

README.md

@@ -62,6 +62,24 @@ proxies = proxy_provider.list_proxies()
 print(proxies)
 ```
 
+For simple usage, you can get a proxy and use it immediately:
+```py
+from proxyproviders import Webshare
+import requests
+
+provider = Webshare(api_key="your-api-key")
+
+# Get a proxy (uses RoundRobin by default) and use it with requests
+from proxyproviders.models.proxy import ProxyFormat
+
+proxy = provider.get_proxy()
+response = requests.get("https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS))
+print(response.json())
+
+# Or in one-line
+response = requests.get("https://httpbin.org/ip", proxies=provider.get_proxy().format(ProxyFormat.REQUESTS))
+```
+
 Each provider has their own custom options, the `Webshare` class lets you specify url params according to their [api spec](https://apidocs.webshare.io/proxy-list/list#parameters), here's an example which will only return proxies that are based in the US.
 
 ```py
@@ -107,32 +125,86 @@ some_function(brightdata)
 
 Here's a more meaningful example that takes the `Proxy` class and uses it to create a python requests http proxy.
 
+#### Simple Usage
 ```py
-from proxyproviders import Webshare, BrightData, ProxyProvider
+from proxyproviders import Webshare
+from proxyproviders.algorithms import Random, RoundRobin
+from proxyproviders.models.proxy import ProxyFormat
 import requests
-import os
 
+provider = Webshare(api_key="your_api_key")
 
-def request_with_proxy(provider: ProxyProvider):
-    requests_proxy = None
+# Get proxy using default RoundRobin and make request
+proxy = provider.get_proxy()
+response = requests.get("https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS))
 
-    if provider:
-        proxies = provider.list_proxies()
+# Or in one-line
+response = requests.get("https://httpbin.org/ip", proxies=provider.get_proxy().format(ProxyFormat.REQUESTS))
+```
 
-        requests_proxy = {
-            "http": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
-            "https": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
-        }
+#### Built-in Algorithms
+```py
+from proxyproviders import Webshare
+from proxyproviders.algorithms import Random, RoundRobin, First
 
-    r = requests.get("https://httpbin.org/ip", proxies=requests_proxy)
-    return r.json()
+provider = Webshare(api_key="your_api_key")
 
-webshare = Webshare(api_key="your_api_key")
-brightdata = BrightData(api_key="your_api_key", zone="your_zone")
+# Default: RoundRobin (cycles through proxies for load balancing)
+proxy = provider.get_proxy()
 
-print(f"Your IP: {request_with_proxy(None)}")
-print(f"Webshare: {request_with_proxy(webshare)}")
-print(f"BrightData: {request_with_proxy(brightdata)}")
+# Random selection
+proxy = provider.get_proxy(Random())
+
+# Always first proxy (deterministic)
+proxy = provider.get_proxy(First())
+
+# Algorithm can maintain state when reused
+round_robin = RoundRobin()
+proxy1 = provider.get_proxy(round_robin)
+proxy2 = provider.get_proxy(round_robin)  # Next in sequence
+```
+
+#### Algorithm State Management
+```py
+from proxyproviders import Webshare
+from proxyproviders.algorithms import RoundRobin, Random
+
+provider = Webshare(api_key="your_api_key")
+
+# Create reusable algorithm for state management
+round_robin = RoundRobin()  # Maintains state across calls
+random_algo = Random()      # Stateless but reusable
+
+# Each call to round_robin will cycle to next proxy
+for i in range(3):
+    proxy = provider.get_proxy(round_robin)
+    print(f"RoundRobin {i}: {proxy.proxy_address}")
+
+# Provider also maintains its own default RoundRobin state when not specified
+for i in range(3):
+    proxy = provider.get_proxy()  # Uses provider's default RoundRobin
+    print(f"Default {i}: {proxy.proxy_address}")
+```
+
+#### Custom Algorithms
+```py
+from proxyproviders.algorithms import Algorithm
+from typing import List
+from proxyproviders.models.proxy import Proxy
+
+class USProxyAlgorithm(Algorithm):
+    """Prefers US proxies, falls back to first available."""
+
+    def select(self, proxies: List[Proxy]) -> Proxy:
+        # Try to find a US proxy
+        for proxy in proxies:
+            if proxy.country_code == "US":
+                return proxy
+        # Fall back to first proxy
+        return proxies[0]
+
+# Use your custom algorithm
+proxy = provider.get_proxy(USProxyAlgorithm())
 ```
 
 ### Making Your Own Proxy Provider

examples/request_with_proxy_provider.py

@@ -1,26 +1,72 @@
-from proxyproviders import Webshare, BrightData, ProxyProvider
+"""Examples showing provider-agnostic proxy usage."""
+
 import requests
-import os
 
+from proxyproviders import BrightData, ProxyProvider, Webshare
+from proxyproviders.algorithms import First, Random, RoundRobin
+from proxyproviders.models.proxy import ProxyFormat
+
+
+def make_request_with_proxy(provider: ProxyProvider):
+    """Make HTTP request through any proxy provider."""
+    proxy = provider.get_proxy()
+    response = requests.get(
+        "https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
+    )
+    return response.json()
+
+
+def algorithm_examples():
+    """Show different algorithms with same provider."""
+    provider = Webshare(api_key="your_api_key")
+
+    # Default RoundRobin
+    provider.get_proxy()
+
+    # Random selection
+    provider.get_proxy(Random())
+
+    # Always first proxy
+    provider.get_proxy(First())
+
+    # Reusable RoundRobin (maintains state)
+    round_robin = RoundRobin()
+    provider.get_proxy(round_robin)
+    provider.get_proxy(round_robin)  # Next in sequence
+
+
+def one_liner_examples():
+    """One-liner HTTP requests through proxy."""
+    provider = Webshare(api_key="your_api_key")
+
+    # Default algorithm
+    requests.get(
+        "https://httpbin.org/ip",
+        proxies=provider.get_proxy().format(ProxyFormat.REQUESTS),
+    )
+
+    # With specific algorithm
+    requests.get(
+        "https://httpbin.org/ip",
+        proxies=provider.get_proxy(Random()).format(ProxyFormat.REQUESTS),
+    )
 
-def request_with_proxy(provider: ProxyProvider):
-    requests_proxy = None
 
-    if provider:
-        proxies = provider.list_proxies()
+def main():
+    # Works with any provider
+    webshare = Webshare(api_key="your_api_key")
+    brightdata = BrightData(api_key="your_api_key", zone="your_zone")
 
-        requests_proxy = {
-            "http": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
-            "https": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
-        }
+    # Same function works with different providers
+    make_request_with_proxy(webshare)
+    make_request_with_proxy(brightdata)
 
-    r = requests.get("https://httpbin.org/ip", proxies=requests_proxy)
-    return r.json()
+    # Algorithm examples
+    algorithm_examples()
 
+    # One-liner usage
+    one_liner_examples()
 
-webshare = Webshare(api_key="your_api_key")
-brightdata = BrightData(api_key="your_api_key", zone="your_zone")
 
-print(f"Your IP: {request_with_proxy(None)}")
-print(f"Webshare: {request_with_proxy(webshare)}")
-print(f"BrightData: {request_with_proxy(brightdata)}")
+if __name__ == "__main__":
+    main()