Initial Release : Leadhunter_Pro

Author: FAAQJAVED

.env.example

@@ -0,0 +1,6 @@
+# LeadHunter Pro — environment variables
+# Copy to .env and fill in your values. This file is safe to commit; .env is not.
+
+# Residential proxy URL for Bing geo-unlock (optional — leave blank to skip Bing)
+# Format: http://user:pass@host:port  or  socks5://user:pass@host:port
+BING_PROXY=

.github/workflows/ci.yml

@@ -0,0 +1,26 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt -r requirements-dev.txt
+
+      - name: Lint with ruff
+        run: ruff check .
+
+      - name: Run tests
+        run: pytest tests/ --tb=short

.gitignore

@@ -0,0 +1,44 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+.venv/
+venv/
+env/
+*.egg-info/
+dist/
+build/
+
+# Project outputs (never commit scraped data)
+outputs/
+logs/
+checkpoints/
+debug_html/
+
+# Enricher state
+enrich_checkpoint.json
+found_contacts_*.xlsx
+found_contacts_*.csv
+command.txt
+
+# Config (user must copy from example)
+config.yaml
+
+# OS
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Coverage
+.coverage
+htmlcov/
+.pytest_cache/
+
+# Environment variables (never commit)
+.env

BLUEPRINT.md

@@ -0,0 +1,39 @@
+# Changelog
+
+## v1.1.0 — 2026-05-02
+
+### Bug Fixes
+
+| Ref | File | What changed |
+|-----|------|--------------|
+| BUG-1 | `core/controls.py`, `main.py` | Added `interruptible_sleep()` to `controls.py`. Phase 1 inter-engine and inter-query delays now use it instead of bare `time.sleep()`. `ControlListener` is now instantiated in `main.py` at startup so P/Q/R/S/W keys work instantly during any sleep. |
+| BUG-2 | `pipeline/data_cleaner.py` | Company name is now derived from the domain (`derive_name_from_domain()`) instead of the search-engine page title. Uses 2-pass CamelCase splitting (before lowercasing) + hyphen/underscore splitting. Example: `alpha-block-management.co.uk` → `Alpha Block Management`. |
+| BUG-3 | `core/email_utils.py` | Added `_PLACEHOLDER_DOMAINS` and `_PLACEHOLDER_LOCALS` blocklists. Addresses like `user@domain.com`, `john@doe.com`, and `filler@godaddy.com` are now rejected before they reach scoring. |
+| BUG-4 | `core/email_utils.py` | Mailto query strings (`?subject=…`) and URL fragments (`#…`) are now stripped from every extracted email address before validation. |
+| BUG-5 | `core/email_utils.py` | HTML entities in phone strings are decoded with `html.unescape()` before extraction. Phone candidates containing a decimal point (`412 132.305`) or three+ consecutive zeros are rejected as prices/placeholder numbers. |
+| BUG-6 | `pipeline/data_cleaner.py`, `enricher.py` | Directory domains are now hard-excluded in `DataCleaner.process()` — no `CleanRecord` is created. `enricher.py` additionally filters any row where `flagged=YES` and `flag_reason=directory` from its input. `threebestrated.co.uk`, `trovit.co.uk`, `idobusiness.co.uk`, `servicevista.co.uk` moved to `_ALWAYS_EXCLUDED`. |
+| BUG-7 | `pipeline/data_cleaner.py` | `_normalise_to_root()` threshold changed from `>= 2` segments to `>= 1`. Any URL with a path (`/property-management-london`) is collapsed to root (`/`). Cuts ~30% of Phase 2 HTTP requests. |
+
+### Improvements
+
+| Ref | File | What changed |
+|-----|------|--------------|
+| IMP-1 | `enricher.py` | Phase 2 Pass 1 (HTTP) now runs concurrently via `ThreadPoolExecutor`. Worker count controlled by `enricher_workers` in `config.yaml` (default 5). Thread-safe writes via `threading.Lock`. Playwright Pass 2 remains sequential. |
+| IMP-2 | `pipeline/data_cleaner.py`, `config.py` | `GEO_SUSPECT_TLDS` list added to `config.py` (default empty). Domains whose TLD matches get `flagged=True, flag_reason='geo-suspect'` and a `-2` score penalty. |
+| IMP-3 | Multiple files | All UK-specific hardcoding removed: city lists (`_UK_CITIES`, `_US_CITIES`), `.co.uk` scoring bonus, `.gov.uk`/`.org.uk` auto-flag, `en-GB` Accept-Language header, industry-specific `generic_email_keywords`. `SCORE_BOOST_KEYWORDS` added to `config.py` (default empty). Tool is now general-purpose. |
+| IMP-4 | `enricher.py` | Email list deduplicated with `list(set(emails))` before `best_email()` selection. `junk_email_domains` expanded to match `_PLACEHOLDER_DOMAINS` in `email_utils.py`. |
+| IMP-5 | `main.py` | Per-engine stats table printed in Phase 1 completion summary (engine, leads found, pages completed). |
+| IMP-6 | `engines/bing.py` | Bing loop detection added. If page 2 returns a domain set that is a subset of page 1's domains, engine logs `"Results are looping (geo-block confirmed)"` and sets `is_banned = True` immediately rather than running all 20 pages. |
+| CF-FIX | `core/email_utils.py` | Fixed Cloudflare regex typo: closing quote was inside the capture group (`([a-f0-9]+"`), causing zero matches. Correct pattern: `([a-f0-9]+)"`. |
+| CAMEL-FIX | `pipeline/data_cleaner.py` | `derive_name_from_domain()` now does CamelCase detection **before** lowercasing the domain string. Fixes `JPropertyManagement.com` → `J Property Management` (was `Jpropertymanagement`). |
+
+### Test updates
+
+- `tests/test_cleaner.py`: Updated `test_derive_name_from_domain` expectations; added `test_directory_domain_produces_none`, `test_normalise_to_root_single_segment`, `test_geo_suspect_flag`, `test_irrelevance_flag`.
+- `tests/test_email_utils.py`: Added `test_mailto_query_string_stripped`, `test_placeholder_domain_rejected`, `test_placeholder_local_rejected`, `test_html_entity_phone_decoded`, `test_decimal_phone_rejected`, `test_zero_loop_phone_rejected`.
+
+---
+
+## v1.0.0 — 2026-04-01
+
+Initial release.

CHANGELOG.md

@@ -0,0 +1,43 @@
+# Contributing to LeadHunter Pro
+
+Contributions are welcome. Please read the guidelines below before opening a pull request — they keep the codebase consistent and reviews fast.
+
+## Running the test suite
+
+```bash
+pytest
+```
+
+All tests live under `tests/`. Run the full suite before pushing; CI will also run it automatically on every PR.
+
+## Adding a new search engine
+
+1. Create `engines/<engine_name>.py` and subclass `engine_base.EngineBase`.
+2. Implement the `search(query: str, pages: int) -> list[SearchResult]` method.
+3. Register the engine in `engines/__init__.py` by adding it to `ENGINE_MAP`:
+
+   ```python
+   from engines.myengine import MyEngine
+   ENGINE_MAP["myengine"] = MyEngine
+   ```
+
+4. Add `"myengine"` to `ENGINES_PRIORITY` in `config.py` if it should run by default.
+5. Add at least one HTML-parsing test in `tests/test_engines.py` (see existing tests for the pattern).
+
+## Code style
+
+The project uses **ruff** for linting and formatting (configured in `pyproject.toml`). Line length is **100 characters**.
+
+```bash
+ruff check .
+ruff format .
+```
+
+Fix all ruff warnings before submitting. Do not suppress rules without a comment explaining why.
+
+## Pull request guidelines
+
+- **One PR per feature or fix.** Mixed-concern PRs are hard to review and harder to revert.
+- **Reference the relevant issue** in the PR description (e.g. `Closes #42`).
+- Keep commit messages short and imperative: `Add Ecosia engine`, `Fix Yahoo warmup retry logic`.
+- If your change touches scraping logic, include a note on which engine/site was tested and what the result looked like.

CONTRIBUTING.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 LeadHunter Pro Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.