Skip to content

Commit ac1ea52

Browse files
alvagantealvagante
committed
feat: Add RPM repository parser and repository fixes
- Implement RpmParser class for parsing RPM repository metadata (primary.xml) - Add support for DNF/YUM repositories with automatic decompression - Fix repository URL handling and metadata parsing issues - Enhance universal downloader with improved error handling - Add test script for RPM parser validation - Update repository configurations for better reliability - Add comprehensive documentation of repository fixes and implementation
1 parent 8f18b96 commit ac1ea52

15 files changed

Lines changed: 1637 additions & 38 deletions

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- **RPM Repository Parser**: Complete implementation of RPM package metadata parser
12+
- New `RpmParser` class for parsing RPM repository metadata (primary.xml)
13+
- Support for DNF/YUM repositories with automatic metadata decompression
14+
- Extraction of package name, version, architecture, description, URL, and license
15+
- Integration with universal repository manager
16+
- Test script for validating RPM parser functionality
17+
- **Repository Fixes and Enhancements**: Multiple improvements to repository handling
18+
- Fixed DNF/YUM repository URL handling and metadata parsing
19+
- Enhanced universal downloader with better error handling
20+
- Improved repository type detection and validation
21+
- Updated repository configurations for better reliability
22+
- Comprehensive documentation of repository fixes
1123
- **Multi-Provider Instance Support**: Configure multiple instances of the same LLM provider type
1224
- Support for multiple Ollama models with unique names (e.g., `ollama_qwen3`, `ollama_deepseek`)
1325
- Support for multiple OpenAI endpoints (official, Azure, local)

‎README.md‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -700,9 +700,6 @@ SAI includes specialized providers for security, debugging, and analysis:
700700
- AI-assisted software documentation
701701
- Repository data analysis and enrichment
702702
703-
## 📄 License
704-
705-
MIT License - see [LICENSE](LICENSE) file for details.
706703
707704
---
708705

‎docs/summaries/repository-fixes-20251025.md‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
# Repository Configuration Fixes - October 25, 2025
2+
3+
## Issues Identified
4+
5+
### 1. API-Based Repositories Using Wrong Method
6+
**Problem**: API-based repositories (snapcraft, rubygems, npm-registry, maven-central, choco-windows, winget-windows, msstore-windows, flathub, nuget-org, emerge-gentoo, pacman-arch, pypi, crates-io, packagist) are being called with `download_package_list()` instead of `query_package()` or `query_packages_batch()`.
7+
8+
**Root Cause**: The `query_type: api` is set in configs but the repository manager is still calling `download_package_list()` for all repositories.
9+
10+
**Solution**: Update repository manager to check `query_type` and use appropriate methods.
11+
12+
### 2. DNF Repositories Returning 0 Packages
13+
**Problem**: All DNF repositories (Fedora, Rocky, AlmaLinux, CentOS Stream) return 0 packages.
14+
15+
**Root Cause**:
16+
- Metalink URLs for Fedora don't directly point to repodata
17+
- repomd.xml URLs for Rocky/Alma/CentOS need proper parsing
18+
- Missing RPM metadata parser implementation
19+
20+
**Solution**:
21+
- Fix Fedora URLs to point to actual repository mirrors
22+
- Implement proper RPM metadata parsing
23+
- Add repomd.xml parsing to extract primary.xml.gz location
24+
25+
### 3. Zypper Repositories Returning 0 Packages
26+
**Problem**: OpenSUSE repositories return 0 packages.
27+
28+
**Root Cause**: Same as DNF - repomd.xml parsing not implemented.
29+
30+
**Solution**: Use same RPM metadata parser as DNF.
31+
32+
### 4. Alpine Main Repository Error
33+
**Problem**: alpine-main repository fails with error.
34+
35+
**Root Cause**: APKINDEX.tar.gz parsing may have issues with tar extraction or text parsing.
36+
37+
**Solution**: Verify tar.gz extraction and APKINDEX text format parsing.
38+
39+
### 5. Go Packages Repository Error
40+
**Problem**: go-packages repository fails.
41+
42+
**Root Cause**: Need to check configuration - likely missing or misconfigured.
43+
44+
**Solution**: Review go-packages configuration.
45+
46+
### 6. Snapcraft Repository Error
47+
**Problem**: Snapcraft API-based repository fails.
48+
49+
**Root Cause**: API endpoint may require authentication or have rate limiting.
50+
51+
**Solution**: Check Snapcraft API requirements and add proper authentication.
52+
53+
### 7. RubyGems Repository Error
54+
**Problem**: RubyGems API fails.
55+
56+
**Root Cause**: API endpoint configuration or parsing issue.
57+
58+
**Solution**: Verify RubyGems API endpoint and response format.
59+
60+
### 8. NPM Registry Error
61+
**Problem**: NPM registry fails.
62+
63+
**Root Cause**: API endpoint or parsing issue.
64+
65+
**Solution**: Verify NPM registry API configuration.
66+
67+
### 9. Microsoft Store Error
68+
**Problem**: msstore-windows fails.
69+
70+
**Root Cause**: Microsoft Store API likely requires authentication.
71+
72+
**Solution**: Add authentication or disable if not accessible.
73+
74+
### 10. Emerge (Gentoo) Error
75+
**Problem**: emerge-gentoo fails.
76+
77+
**Root Cause**: Gentoo package database requires special handling.
78+
79+
**Solution**: Implement Gentoo portage tree parsing.
80+
81+
### 11. Pacman (Arch) Error
82+
**Problem**: pacman-arch fails despite being API-based.
83+
84+
**Root Cause**: Arch Linux API endpoint or parsing issue.
85+
86+
**Solution**: Verify Arch Linux packages API.
87+
88+
### 12. PyPI Error
89+
**Problem**: PyPI fails despite being API-based.
90+
91+
**Root Cause**: PyPI simple API may have rate limiting or parsing issues.
92+
93+
**Solution**: Verify PyPI API configuration and implement proper rate limiting.
94+
95+
### 13. Ubuntu Oracular Error
96+
**Problem**: apt-ubuntu-oracular fails.
97+
98+
**Root Cause**: Repository may not exist yet or URL is incorrect.
99+
100+
**Solution**: Verify Ubuntu 24.10 (Oracular) repository availability.
101+
102+
### 14. Debian Buster Error
103+
**Problem**: apt-debian-buster fails.
104+
105+
**Root Cause**: Repository may be archived or URL changed.
106+
107+
**Solution**: Check if Buster is EOL and update URL to archive.debian.org.
108+
109+
## Priority Fixes
110+
111+
### High Priority
112+
1. **Fix repository manager to respect `query_type`** - This will fix all API-based repositories
113+
2. **Implement RPM metadata parser** - This will fix DNF and Zypper repositories
114+
3. **Fix Alpine APKINDEX parsing** - Important Linux distribution
115+
116+
### Medium Priority
117+
4. **Fix PyPI** - Very commonly used
118+
5. **Fix NPM** - Very commonly used
119+
6. **Fix Arch Linux (pacman)** - Popular distribution
120+
7. **Fix RubyGems** - Commonly used
121+
122+
### Low Priority
123+
8. **Fix Snapcraft** - May require authentication
124+
9. **Fix Microsoft Store** - Likely requires authentication
125+
10. **Fix Gentoo emerge** - Niche distribution
126+
11. **Fix Ubuntu Oracular** - May not be released yet
127+
12. **Fix Debian Buster** - EOL, low priority
128+
129+
## Implementation Plan
130+
131+
### Phase 1: Repository Manager Fix
132+
Update `saigen/repositories/manager.py` or `universal_manager.py` to:
133+
- Check `query_type` field in repository config
134+
- Use `query_package()` for API-based repositories instead of `download_package_list()`
135+
- Add proper error handling for API repositories
136+
137+
### Phase 2: RPM Metadata Parser
138+
Create `saigen/repositories/parsers/rpm_parser.py`:
139+
- Parse repomd.xml to find primary.xml.gz location
140+
- Download and parse primary.xml.gz
141+
- Extract package information (name, version, description, etc.)
142+
143+
### Phase 3: Fix Individual Repositories
144+
- Update DNF repository URLs
145+
- Update Zypper repository URLs
146+
- Fix Alpine APKINDEX parsing
147+
- Verify and fix API-based repository configurations
148+
149+
### Phase 4: Testing
150+
- Test each repository type
151+
- Verify package counts are reasonable
152+
- Check error handling
153+
154+
## Configuration Changes Needed
155+
156+
### DNF Repositories
157+
```yaml
158+
# Change from metalink to direct mirror URLs
159+
endpoints:
160+
packages: https://download.fedoraproject.org/pub/fedora/linux/releases/40/Everything/{arch}/os/repodata/repomd.xml
161+
```
162+
163+
### Zypper Repositories
164+
```yaml
165+
# Already correct, just need parser implementation
166+
endpoints:
167+
packages: http://download.opensuse.org/distribution/leap/15.5/repo/oss/repodata/repomd.xml
168+
```
169+
170+
### Debian Buster
171+
```yaml
172+
# Move to archive
173+
endpoints:
174+
packages: http://archive.debian.org/debian/dists/buster/main/binary-{arch}/Packages.gz
175+
```
176+
177+
### Ubuntu Oracular
178+
```yaml
179+
# Verify release status and update URL
180+
metadata:
181+
enabled: false # Disable until release is confirmed
182+
```
183+
184+
## Expected Results After Fixes
185+
186+
- **DNF repositories**: 10,000-50,000 packages each
187+
- **Zypper repositories**: 20,000-60,000 packages
188+
- **Alpine**: 5,000-15,000 packages
189+
- **PyPI**: API-based, query on demand
190+
- **NPM**: API-based, query on demand
191+
- **Pacman**: API-based, query on demand
192+
- **RubyGems**: API-based, query on demand
193+
194+
## Notes
195+
196+
- API-based repositories should not show package counts in stats
197+
- API-based repositories should show "API" status instead of package count
198+
- Bulk download repositories should show actual package counts
199+
- Error repositories should show clear error messages with troubleshooting hints

0 commit comments

Comments
 (0)