|
| 1 | +# 404 Analysis Summary |
| 2 | + |
| 3 | +## Overview |
| 4 | +Analyzing Google Search Console 404 report from `~/Downloads/404s - Table (1).csv` to identify legitimate redirects vs noise. |
| 5 | + |
| 6 | +## Files Generated |
| 7 | +- `_redirects_from_suggestions` - Valid redirects found (360 out of 1000 tested) |
| 8 | +- `failed_redirects.csv` - URLs that failed validation (640 out of 1000 tested) |
| 9 | +- `.cache/redirect_sanitizer.sqlite` - Cache database from the main sanitize script |
| 10 | + |
| 11 | +## Key Findings |
| 12 | + |
| 13 | +### Main Sanitize Script Results |
| 14 | +- **Very conservative approach** - marked all URLs as "no_evidence" |
| 15 | +- Script requires Wayback Machine evidence or git history to create redirects |
| 16 | +- Processed 50 URLs: 0 redirects, 48 for review, 2 filtered out |
| 17 | +- This suggests the script is being overly cautious for this dataset |
| 18 | + |
| 19 | +### Direct CSV Processing Results |
| 20 | +- **Much more productive** - found 360 valid redirects out of 1000 URLs tested |
| 21 | +- Uses suggested URLs from CSV as starting points |
| 22 | +- Validates that suggested targets actually work (return 200 status) |
| 23 | +- Success rate: 36% (360/1000) |
| 24 | + |
| 25 | +### Data Quality Observations |
| 26 | +1. **Mixed quality data** - some legitimate URLs, some spam (FIFA coins, etc.) |
| 27 | +2. **Suggested URLs work** - many of the suggested redirects in the CSV are valid |
| 28 | +3. **Language variations** - many URLs are for Japanese (ja) and Korean (ko) versions |
| 29 | +4. **Common patterns** - many redirect to `/support/`, `/guides/`, or language-specific guides |
| 30 | + |
| 31 | +## Next Steps |
| 32 | +1. Review the 360 valid redirects in `_redirects_from_suggestions` |
| 33 | +2. Check the failed URLs in `failed_redirects.csv` for any false negatives |
| 34 | +3. Consider running the full dataset (1171 URLs total) |
| 35 | +4. Potentially modify the main sanitize script to be less conservative |
| 36 | + |
| 37 | +## Commands Used |
| 38 | +```bash |
| 39 | +# Set up environment |
| 40 | +python3 -m venv venv |
| 41 | +source venv/bin/activate |
| 42 | +pip install requests |
| 43 | + |
| 44 | +# Run main sanitize script (conservative) |
| 45 | +python scripts/sanitize_404s.py --input "/Users/matt.linville/Downloads/404s - Table (1).csv" --out-redirects _redirects_test --out-review needs_review_test.csv --owned-hosts "docs.wandb.ai" --dry-run-limit 5 |
| 46 | + |
| 47 | +# Run direct CSV processing (more productive) |
| 48 | +python process_404s_direct.py |
| 49 | +``` |
| 50 | + |
| 51 | +## Files to Review |
| 52 | +- `_redirects_from_suggestions` - The good redirects found |
| 53 | +- `failed_redirects.csv` - URLs that need manual review |
| 54 | +- `needs_review_test.csv` - Results from conservative script |
| 55 | + |
| 56 | +## Environment |
| 57 | +- Working directory: `/Users/matt.linville/docs` |
| 58 | +- Virtual environment: `venv/` (activated) |
| 59 | +- Python script: `process_404s_direct.py` (custom script for direct CSV processing) |
0 commit comments