feat: Add database compliance rules system#30
Conversation
- Add HostNetworkDiscoveryService for comprehensive network analysis - Discover network interfaces with IP addresses, states, and statistics - Extract routing table information (IPv4 and IPv6) - Analyze DNS configuration and resolution testing - Detect NTP configuration across different implementations (chrony, ntp, systemd-timesyncd) - Identify network services and listening ports - Perform connectivity tests to common destinations - Assess network security configuration and hardening - Add network security assessment with scoring and recommendations - Create network topology mapping for multiple hosts - Include comprehensive parsing utilities for various network commands - Support both modern (ip, ss) and legacy (ifconfig, netstat) network tools ## API Endpoints - Individual and bulk network discovery (max 10 hosts for intensive operations) - Network security assessment with vulnerability identification - Network topology mapping with connectivity matrix - Network discovery capabilities reference ## Security Features - Identifies risky open ports and services - Assesses firewall status and configuration - Evaluates TCP/IP stack hardening parameters - Provides security recommendations and vulnerability reporting 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
… issues - Fix division by zero error in scap_scanner.py when calculating compliance score - Add proper divisor check before score calculation (rules_passed + rules_failed) - Handle cases where all rules are N/A, error, or notchecked (score = 0.0) - Fix webhook async execution issues in Celery worker context - Replace asyncio.create_task() with proper event loop handling - Use asyncio.new_event_loop() for webhook delivery in worker threads - Improve error handling and logging for webhook notifications Resolves scan failure: "Failed to parse results: division by zero (Code: EXEC_001)" 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Remove duplicate code in SCAP services and credential handling - Create shared utilities for XML processing and credential decoding - Consolidate error creation functions using template-based factory - Archive obsolete files and organize useful scripts/docs - Remove unused nginx.conf and system_settings.py files - Restructure documentation and script organization
| from .routes import auth, hosts, scans, content, scap_content, monitoring, users, audit, host_groups, scan_templates, webhooks, mfa | ||
| from .routes.system_settings_unified import router as system_settings_router | ||
| from .routes import credentials, api_keys, remediation_callback, integration_metrics, bulk_operations, compliance, rule_scanning, capabilities | ||
| from .routes import credentials, api_keys, remediation_callback, integration_metrics, bulk_operations, compliance, rule_scanning, capabilities, host_network_discovery |
Check notice
Code scanning / CodeQL
Unused import
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the issue, remove only the import of capabilities from the import statement on line 22 of backend/app/main.py. This involves editing the multi-item import statement to exclude capabilities, leaving all other imported modules untouched. No other code changes, imports, or additional definitions are required; only edit the import statement to remove the unused module.
| @@ -19,7 +19,7 @@ | ||
| from .database import engine, create_tables, get_db | ||
| from .routes import auth, hosts, scans, content, scap_content, monitoring, users, audit, host_groups, scan_templates, webhooks, mfa | ||
| from .routes.system_settings_unified import router as system_settings_router | ||
| from .routes import credentials, api_keys, remediation_callback, integration_metrics, bulk_operations, compliance, rule_scanning, capabilities, host_network_discovery | ||
| from .routes import credentials, api_keys, remediation_callback, integration_metrics, bulk_operations, compliance, rule_scanning, host_network_discovery | ||
| # Import security routes only if available | ||
| try: | ||
| from .routes import automated_fixes |
| NetworkDiscoveryResponse containing discovered network information | ||
| """ | ||
| # Check permissions | ||
| check_permission(current_user, "hosts:read") |
Check failure
Code scanning / CodeQL
Wrong number of arguments in a call
| detail=f"Invalid host ID format: {str(e)}" | ||
| ) | ||
| except Exception as e: | ||
| logger.error(f"Network discovery failed for host {host_id}: {str(e)}") |
Check failure
Code scanning / CodeQL
Log Injection
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the log injection vulnerability, sanitize any user-provided strings being logged. For log files that are plain text, removal of newline characters (\r, \n, etc.) is recommended (e.g., using .replace("\n", "").replace("\r", "")). In this case, the host_id parameter (which is tainted) should be sanitized before inclusion in the log entry.
Specifically, in backend/app/routes/host_network_discovery.py, line 116, update the logger call to sanitize host_id by removing newline characters. Ideally, use a helper function to safely sanitize loggable user inputs if possible; for this snippet, use the string replace approach directly. No new imports are needed, and code edits are limited to this exception handler block.
| @@ -113,7 +113,8 @@ | ||
| detail=f"Invalid host ID format: {str(e)}" | ||
| ) | ||
| except Exception as e: | ||
| logger.error(f"Network discovery failed for host {host_id}: {str(e)}") | ||
| safe_host_id = host_id.replace('\n', '').replace('\r', '') | ||
| logger.error(f"Network discovery failed for host {safe_host_id}: {str(e)}") | ||
| raise HTTPException( | ||
| status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, | ||
| detail=f"Network discovery failed: {str(e)}" |
| BulkNetworkDiscoveryResponse with results for all hosts | ||
| """ | ||
| # Check permissions | ||
| check_permission(current_user, "hosts:read") |
Check failure
Code scanning / CodeQL
Wrong number of arguments in a call
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix this error, supply a value for the missing required third parameter to the check_permission function call. As the context does not specify what this third argument should be, but considering typical usage in an RBAC system, the third argument may represent a resource or an action on a specific target. Since this is a bulk operation over hosts, with the actual list of affected hosts available in request.host_ids, the third argument should likely be the list of host IDs (or resources) on which the permission check is being performed.
Make the following change in backend/app/routes/host_network_discovery.py:
- At line 139, change
check_permission(current_user, "hosts:read")tocheck_permission(current_user, "hosts:read", request.host_ids).
No new methods or imports are required, as the objects and variables needed (current_user, "hosts:read", request.host_ids) are already available.
| @@ -136,7 +136,7 @@ | ||
| BulkNetworkDiscoveryResponse with results for all hosts | ||
| """ | ||
| # Check permissions | ||
| check_permission(current_user, "hosts:read") | ||
| check_permission(current_user, "hosts:read", request.host_ids) | ||
|
|
||
| if not request.host_ids: | ||
| raise HTTPException( |
| errors[host_id] = f"Invalid host ID format: {str(e)}" | ||
| failed_discoveries += 1 | ||
| except Exception as e: | ||
| logger.error(f"Network discovery failed for host {host_id}: {str(e)}") |
Check failure
Code scanning / CodeQL
Log Injection
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To resolve this vulnerability, you should sanitize the user input (host_id) before incorporating it into any log messages. The specific risk for log injection is the presence of newline (\n, \r) characters, which can be removed using string replacement: host_id.replace('\r', '').replace('\n', ''). It is best to do this as close as possible to the log statement. Only the log message construction should change; the program's logic should remain unaffected.
Steps:
- On line 191, sanitize
host_idby replacing/removing any carriage return and newline characters before interpolating it into the log message. - No dependencies or additional library imports are required for this string sanitization; use Python's built-in string methods.
- Prefer not to mutate
host_idglobally, just use the sanitized version in the logging call. - Contextually, other error messages (such as those returned in HTTP responses or populated in error dicts) could also be considered for sanitization, but CodeQL has only flagged the logging sink, so only that should be changed for this fix.
| @@ -188,9 +188,9 @@ | ||
| errors[host_id] = f"Invalid host ID format: {str(e)}" | ||
| failed_discoveries += 1 | ||
| except Exception as e: | ||
| logger.error(f"Network discovery failed for host {host_id}: {str(e)}") | ||
| safe_host_id = host_id.replace('\r', '').replace('\n', '') | ||
| logger.error(f"Network discovery failed for host {safe_host_id}: {str(e)}") | ||
| errors[host_id] = f"Network discovery failed: {str(e)}" | ||
| failed_discoveries += 1 | ||
|
|
||
| logger.info(f"Bulk network discovery completed: {successful_discoveries} successful, " | ||
| f"{failed_discoveries} failed out of {len(request.host_ids)} total hosts") |
| import json | ||
| from datetime import datetime | ||
| from pathlib import Path | ||
| from typing import Dict, List, Optional, Tuple, Any, Union |
Check notice
Code scanning / CodeQL
Unused import
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
The fix is to remove the unused List import from the import statement on line 15 in backend/app/services/unified_ssh_service.py. Since the code might still use other types from typing such as Dict, Optional, Tuple, Any, and Union, we should preserve those in the import statement and only delete List. Ensure the edit leaves the import statement syntactically correct and retains the other imported types for current and future use.
| @@ -12,7 +12,7 @@ | ||
| import json | ||
| from datetime import datetime | ||
| from pathlib import Path | ||
| from typing import Dict, List, Optional, Tuple, Any, Union | ||
| from typing import Dict, Optional, Tuple, Any, Union | ||
| from dataclasses import dataclass | ||
|
|
||
| import paramiko |
| from dataclasses import dataclass | ||
|
|
||
| import paramiko | ||
| from paramiko import SSHClient, SSHException |
Check notice
Code scanning / CodeQL
Unused import
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
The best way to fix this problem is to remove the unused import SSHException from the line from paramiko import SSHClient, SSHException, leaving only the use of SSHClient. This can be done by editing that line to from paramiko import SSHClient. Ensure that semantics are not changed elsewhere; no references to SSHException should exist in the file.
Edit only line 19 in backend/app/services/unified_ssh_service.py:
- Change:
from paramiko import SSHClient, SSHException - To:
from paramiko import SSHClient
No additional changes, imports, or dependencies are necessary.
| @@ -16,7 +16,7 @@ | ||
| from dataclasses import dataclass | ||
|
|
||
| import paramiko | ||
| from paramiko import SSHClient, SSHException | ||
| from paramiko import SSHClient | ||
|
|
||
| from .ssh_utils import ( | ||
| parse_ssh_key, validate_ssh_key, SSHKeyError, |
| from .ssh_utils import ( | ||
| parse_ssh_key, validate_ssh_key, SSHKeyError, | ||
| SSHKeyValidationResult, format_validation_message | ||
| ) |
Check notice
Code scanning / CodeQL
Unused import
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the issue, remove the unused import of format_validation_message from the statement on line 21-24. Only the format_validation_message part should be deleted, leaving the rest (parse_ssh_key, validate_ssh_key, SSHKeyError, SSHKeyValidationResult) as is, assuming those have legitimate uses found elsewhere in the code. The code change will be localized to lines 21-24 in backend/app/services/unified_ssh_service.py. No new imports or definitions are required; it is a simple removal.
| @@ -20,7 +20,7 @@ | ||
|
|
||
| from .ssh_utils import ( | ||
| parse_ssh_key, validate_ssh_key, SSHKeyError, | ||
| SSHKeyValidationResult, format_validation_message | ||
| SSHKeyValidationResult | ||
| ) | ||
|
|
||
| logger = logging.getLogger(__name__) |
| ssh = SSHClient() | ||
|
|
||
| # Set host key policy | ||
| ssh.set_missing_host_key_policy(self._get_host_key_policy()) |
Check failure
Code scanning / CodeQL
Accepting unknown SSH host keys when using Paramiko
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To address this vulnerability, we must ensure that the SSH client's missing host key policy is always set to paramiko.RejectPolicy. This means rejecting connections when the host key is not recognized, which is the secure default behavior. The best way to fix the problem is to replace usage of self._get_host_key_policy() at line 369 with a direct reference to paramiko.RejectPolicy, guaranteeing safe handling regardless of any possible overrides or future changes to _get_host_key_policy. The change is localized to line 369 in the file backend/app/services/unified_ssh_service.py, where the host key policy is set.
No additional imports are necessary, as the code already imports paramiko and relevant classes.
Optional: If _get_host_key_policy() is used elsewhere, consider removing or refactoring it to prevent unsafe policies in the future.
| @@ -365,8 +365,8 @@ | ||
| # Create SSH client | ||
| ssh = SSHClient() | ||
|
|
||
| # Set host key policy | ||
| ssh.set_missing_host_key_policy(self._get_host_key_policy()) | ||
| # Set host key policy (SECURE: always reject unknown host keys) | ||
| ssh.set_missing_host_key_policy(paramiko.RejectPolicy()) | ||
|
|
||
| # Load host keys | ||
| self._load_container_host_keys(ssh) |
| logger.debug(f"Webhook notification queued for failed scan: {scan_id}") | ||
| # Run webhook delivery in a new event loop (for Celery worker context) | ||
| try: | ||
| loop = asyncio.new_event_loop() |
Check failure
Code scanning / CodeQL
Potentially uninitialized local variable
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
The best way to fix this problem is to ensure that asyncio is always imported at the module level (top of the file), and remove any inner import asyncio statements inside functions (including inside try/except blocks). This prevents Python from treating asyncio as a local variable within the function, which causes the UnboundLocalError.
Specifically, in backend/app/tasks/scan_tasks.py, move (or remove) all import asyncio from inner blocks (lines 315 and 379) so that only the top-level (line 7) import remains. Ensure no other local imports of asyncio exist in this file in the code snippets shown, as we can't change code outside the supplied snippets.
| @@ -312,7 +312,6 @@ | ||
| classified_error = None | ||
| if original_exception: | ||
| try: | ||
| import asyncio | ||
| classified_error = asyncio.run(error_service.classify_error(original_exception, {"scan_id": scan_id})) | ||
| # Use classified error message if available | ||
| if classified_error: | ||
| @@ -335,7 +334,6 @@ | ||
| # Check if this is part of a group scan and update progress | ||
| if scan_data and scan_data.scan_options: | ||
| try: | ||
| import json | ||
| scan_options = json.loads(scan_data.scan_options) | ||
| group_scan_session_id = scan_options.get("session_id") | ||
|
|
Security fixes implemented: - Command injection prevention in SCAP scanner with input validation - Path traversal protection in datastream processor - Weak cryptography fix in rate limiting (HMAC-SHA256 with salt) - Hardcoded credentials removal in admin user creation - Information disclosure prevention in error messages - Insecure file permissions checking in SSL cert handling All inputs now validated with regex patterns and proper sanitization. Error messages sanitized to prevent information leakage. Secure random password generation for admin accounts. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
|
| try: | ||
| if connection_result.connection: | ||
| connection_result.connection.close() | ||
| except: |
Check notice
Code scanning / CodeQL
Empty except
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
The best way to fix this issue is to replace the empty except: block with one that catches all exceptions (using except Exception as e:), and logs the exception with a suitable log level and message. This way, if something goes wrong during the connection close process, there is a record in the logs, which aids debugging but does not interfere with the main exception handling logic. The fix should only modify lines 177-178 of backend/app/services/host_monitor.py, replacing the bare except:/pass block with proper logging. No new imports or methods are needed, as logging is already imported and in use.
| @@ -174,8 +174,8 @@ | ||
| try: | ||
| if connection_result.connection: | ||
| connection_result.connection.close() | ||
| except: | ||
| pass | ||
| except Exception as close_exc: | ||
| logger.debug(f"Exception occurred while closing SSH connection: {type(close_exc).__name__}: {close_exc}") | ||
|
|
||
| error_msg = "SSH test command error" | ||
| logger.error(f"SSH test command failed: {type(e).__name__}") |
| try: | ||
| if connection_result.connection: | ||
| connection_result.connection.close() | ||
| except: |
Check notice
Code scanning / CodeQL
Except block handles 'BaseException'
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the problem, the bare except: should be replaced with except Exception:. This ensures only ordinary exceptions are caught, and special signals like KeyboardInterrupt and SystemExit used for process control remain unaffected.
This change should be made within the method found in file backend/app/services/host_monitor.py, specifically at the block after if connection_result.connection:, changing except: to except Exception: at line 177. No new imports or definitions are needed; this is just an improvement to exception handling.
| @@ -174,7 +174,7 @@ | ||
| try: | ||
| if connection_result.connection: | ||
| connection_result.connection.close() | ||
| except: | ||
| except Exception: | ||
| pass | ||
|
|
||
| error_msg = "SSH test command error" |
| from typing import Dict, List, Optional, Tuple | ||
| import logging | ||
| import json | ||
| import re |
Check notice
Code scanning / CodeQL
Unused import
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the problem, simply remove the line that imports the re module (import re) in backend/app/services/scap_datastream_processor.py at line 13. No additional changes or imports are needed, because the re module is not referenced anywhere in the provided code.
| @@ -10,7 +10,6 @@ | ||
| from typing import Dict, List, Optional, Tuple | ||
| import logging | ||
| import json | ||
| import re | ||
| import xml.etree.ElementTree as ET | ||
| import lxml.etree as etree | ||
| from datetime import datetime |
| if os.getenv("ADMIN_PASSWORD"): | ||
| print("Password: [Set via ADMIN_PASSWORD environment variable]") | ||
| else: | ||
| print(f"Generated Password: {password}") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
The ideal fix is to avoid printing the generated password directly to standard output, where it could be accidentally captured in logs or exposed. If the intent is for the administrator to acquire the credential, the script should instead write the password to a secure file with restricted permissions, display a warning about where the password is stored, and prompt the admin to retrieve it manually (and then securely delete it). Alternatively, require manual password input or sourcing via environment variable, never auto-generating and outputting the value. In this case, we will:
- Replace the line that prints
Generated Password: {password}with code that writes the generated password to a file (e.g.,admin_password.txt) with restrictive permissions. - Print a strong warning about the destination file and recommend secure handling (deletion after use, setting permissions).
- Ensure that password is not printed to stdout.
- Only change lines within the provided snippet.
- Import/define the necessary code to securely write to a file and set permissions.
| @@ -46,9 +46,17 @@ | ||
| if os.getenv("ADMIN_PASSWORD"): | ||
| print("Password: [Set via ADMIN_PASSWORD environment variable]") | ||
| else: | ||
| print(f"Generated Password: {password}") | ||
| print("⚠️ IMPORTANT: Save this password securely and change it after first login!") | ||
|
|
||
| # Write the generated password to a file with restricted permissions | ||
| password_file = "admin_password.txt" | ||
| try: | ||
| with open(password_file, "w") as f: | ||
| f.write(password + "\n") | ||
| os.chmod(password_file, 0o600) # Owner read/write only | ||
| print(f"Generated password has been written to '{password_file}'.") | ||
| print("⚠️ IMPORTANT: Retrieve the generated password from this file, save it securely, and delete the file after first login. Change the password immediately after logging in!") | ||
| except Exception as file_err: | ||
| print(f"Failed to write password to '{password_file}': {file_err}") | ||
| sys.exit(2) | ||
| except Exception as e: | ||
| print(f"Error creating user: {e}") | ||
| db.rollback() |
✅ Database Schema Complete: - Created rule_intelligence table for semantic SCAP data - Created semantic_scan_analysis table for scan processing - Created framework_compliance_matrix table for compliance tracking - Created compliance_intelligence_metadata table for system state - Added compliance_intelligence_summary view for dashboard ✅ API Functionality Verified: - /api/compliance/health: Service health check - /api/compliance/semantic-rules: Rule intelligence queries - /api/compliance/framework-intelligence: Framework analysis - /api/compliance/overview: System overview metrics - /api/compliance/compliance-matrix: Compliance tracking ✅ Security Fixes Previously Completed: - Fixed 12 critical security hotspots including command injection, path traversal - Implemented proper input validation and sanitization - Added secure credential handling ✅ Sample Data & Testing: - Added sample rule intelligence data for testing - All endpoints tested with proper authentication - Fixed PostgreSQL array query performance issue The database compliance system is now fully operational with semantic SCAP intelligence, cross-framework analysis, and comprehensive compliance tracking.




Summary
Changes
Testing