Merge pull request #1871 from stratosphereips/alya/immune/improve_logging

Improve logging and fix log rotation

Author: AlyaGomaa

.gitignore

@@ -18,6 +18,10 @@ slack_bot_token_secret
 # Ignore daemon output files
 daemon/
 
+# Ignore private AGENTS.md, may contain private paths
+private/AGENTS.md
+
+
 # Ignore the results folders
 2019-
 2020-

.secrets.baseline

@@ -149,7 +149,7 @@
         "filename": "config/slips.yaml",
         "hashed_secret": "4cac50cee3ad8e462728e711eac3e670753d5016",
         "is_verified": false,
-        "line_number": 278
+        "line_number": 295
       }
     ],
     "dataset/test14-malicious-zeek-dir/http.log": [
@@ -7185,5 +7185,5 @@
       }
     ]
   },
-  "generated_at": "2026-03-27T14:25:16Z"
+  "generated_at": "2026-04-08T14:13:03Z"
 }

AGENTS.md

@@ -1,42 +1,136 @@
 # AGENTS.md
 
-## Project overview
-- Entry point: `slips.py` (starts the main process, spawns modules, runs in interactive/daemon modes).
-- Core framework code lives in `slips/`, `slips_files/`, and `managers/`.
-- Detection/analysis modules are in `modules/` (implement the `IModule` interface).
-- Configuration is in `config/` (main config: `config/slips.yaml`).
-- Tests live under `tests/` (unit + integration suites).
-- Documentation is in `docs/` (see `docs/contributing.md` for contribution workflow, branching, and PR expectations).
-- UIs/tools: `SlipsWeb/`, `webinterface/`, `webinterface.sh`, and `kalipso.sh`.
-
-## Build and test commands
-- Run locally (no build step):
-  - `./slips.py -e 1 -f dataset/test7-malicious.pcap -o output_dir`
-- Build the Docker image (from `docs/installation.md`):
-  - `docker build --no-cache -t slips -f docker/Dockerfile .`
-  - If build networking fails: `docker build --network=host --no-cache -t slips -f docker/Dockerfile .`
-- Run the Docker image:
-  - `docker run -it --rm --net=host slips`
-
-## Code style guidelines
-- Python formatting is enforced via pre-commit:
-  - Black with `--line-length 79` (see `.pre-commit-config.yaml`).
-  - Ruff is used for linting and autofixes.
-- Keep docstrings at the top of files where present (pre-commit `check-docstring-first`).
-- Maintain clean whitespace (no trailing whitespace, final newline).
-- Follow existing module patterns (`IModule` in `slips_files/common/abstracts/module.py`).
-
-## Testing instructions
-- The canonical test runner is `tests/run_all_tests.sh` (runs unit tests then integration tests).
-- Equivalent manual sequence (from `tests/run_all_tests.sh`):
-  - `./slips.py -cc`
-  - `printf "0" | ./slips.py -k`
-  - `python3 -m pytest tests/ --ignore="tests/integration_tests" -n 7 -p no:warnings -vvvv -s`
-  - `python3 tests/destrctor.py`
-  - `./slips.py -cc`
-  - `printf "0" | ./slips.py -k`
-  - `python3 -m pytest -s tests/integration_tests/test_portscans.py -p no:warnings -vv`
-  - `python3 -m pytest -s tests/integration_tests/test_dataset.py -p no:warnings -vv`
-  - `python3 -m pytest -s tests/integration_tests/test_config_files.py -p no:warnings -vv`
-  - `printf "0" | ./slips.py -k`
-  - `./slips.py -cc`
+## 1. Project Overview
+
+- Entry point: `slips.py`
+  - Starts the main process
+  - Spawns modules
+  - Supports interactive and daemon modes
+
+- Core code directories:
+  - `slips/`
+  - `slips_files/`
+  - `managers/`
+
+- Detection modules:
+  - Located in `modules/`
+  - Must implement `IModule` from:
+    `slips_files/common/abstracts/module.py`
+
+- Configuration:
+  - Main file: `config/slips.yaml`
+
+- Tests:
+  - Located in `tests/`
+  - Includes unit and integration tests
+
+- Documentation:
+  - Located in `docs/`
+  - Contribution guide: `docs/contributing.md`
+
+- UI / tools:
+  - `SlipsWeb/`
+  - `webinterface/`
+  - `webinterface.sh`
+  - `kalipso.sh`
+
+- Repository root:
+  - All commands MUST be executed from `StratosphereLinuxIPS/`
+
+---
+
+## 2. Build and Run
+
+### to run slips locally
+./slips.py -e 1 -f dataset/test7-malicious.pcap -o output_dir
+
+### Build Docker image
+docker build --no-cache -t slips -f docker/Dockerfile .
+
+- If networking fails:
+
+docker build --network=host --no-cache -t slips -f docker/Dockerfile .
+
+### Run Docker container
+docker run -it --rm --net=host slips
+
+## 3. Code Style Rules
+
+These rules MUST be followed:
+
+- No trailing whitespace
+- File must end with a newline
+- Docstring must be the first statement in a file (if present)
+- Avoid using environment variables, use variables from slips/config.yaml instead.
+
+### Paths:
+- NEVER use absolute paths
+- ALWAYS use relative paths
+### Files:
+- If a non-debug file is created → MUST be added with git add
+### Documentation:
+If a feature is added → MUST update relevant docs in docs/
+### Functions:
+- Every new function MUST include a docstring
+Docstrings MUST include:
+- Short description
+- Parameters (if applicable)
+- Return value (if applicable)
+
+## 4. Testing
+- Canonical test runner
+tests/run_all_tests.sh
+## 5. Unit Test Update Workflow
+
+When instructed to "update unit tests", follow EXACTLY:
+
+Step 1 — Run tests
+python3 -m pytest tests/unit/ \
+  --ignore="tests/integration_tests" \
+  -n 7 -p no:warnings -vvvv -s
+
+Step 2 — Identify failures
+Collect ALL failing tests
+
+Step 3 — Fix tests
+Update failing tests ONE BY ONE
+Do NOT batch fixes
+
+Step 4 — Add missing tests for new files
+For every new source file in the branch:
+
+- Mirror its directory under tests/unit/
+
+- C/reate file:
+test_<filename>.py
+- Add unit tests for that file
+
+Step 5 — Add tests for new functions
+- Identify functions added in this branch (not in origin/develop)
+- Add unit tests for each new function
+
+Step 6 — Test structure rules
+- MUST use @pytest.mark.parametrize when applicable
+
+EACH test MUST:
+Start with object creation using module_factory
+
+Step 7 — Re-run tests
+Run the same pytest command again
+Ensure ALL tests pass
+
+Step 8 — Git tracking
+If new test files were created → run:
+git add <files>
+
+Step 9 — Failure fallback
+If tests are still failing and cannot be fixed:
+STOP
+Report the issue
+
+## 6. Custom Instructions
+ALSO apply rules from:
+private/AGENTS.md
+
+If conflicts occur:
+Prefer private/AGENTS.md

config/slips.yaml

@@ -1,14 +1,13 @@
 # This configuration file controls several aspects of the working of Slips.
 
 ---
-modes:
+output:
   # Define the file names for the default output.
   stdout: slips.log
   stderr: errors.log
-  logsfile: slips.log
+  logs: slips.log
 
 #############################
-# Parameters that can be also specified with modifiers in the command line
 
 parameters:
   # The verbosity is related to how much data you want to see about the
@@ -77,6 +76,11 @@ parameters:
   # whitelist file, current commit and date
   metadata_dir: true
 
+  # This directory stores databases and runtime-generated files that must
+  # persist across different slips runs and should not be overwritten.
+  # this dir is inside slips root dir.
+  permanent_dir: permanent
+
   # Default pcap packet filter. Used with zeek pcapfilter : 'ip or not ip'
   # If you want more important traffic and forget the multicast and broadcast
   # traffic, you can use
@@ -112,7 +116,7 @@ parameters:
   # label: malicious
   # label: unknown
   label: normal
-  # If Zeek files are rotated or not to avoid running out of disk.
+  # determines if Zeek files are rotated or not to avoid running out of disk.
   # Zeek rotation is enabled by default when using an interface,
   # which means Slips will delete all Zeek log files after 1 day
   # of running
@@ -126,17 +130,16 @@ parameters:
   # Whitespace between the numeric constant and time unit is optional.
   # Appending the letter s to the time unit in order to
   # pluralize it is also optional
-  # rotation_period = 30min
-  # rotation_period = 2hr
-  # rotation_period = 30sec
-  rotation_period: 1day
+  # default_rotation_interval: 30min
+  # default_rotation_interval: 2hr
+  default_rotation_interval: 30sec
+  #default_rotation_interval: 1day
 
   # How many days Slips keeps the rotated Zeek files before deleting them.
   # Value should be in days
   # set it to 0 day if you want to delete them immediately
-  # keep_rotated_files_for : 1 day
+  # keep_rotated_files_for : 0 day
   # keep_rotated_files_for : 2 day
-  # keep_rotated_files_for : 3 day
   keep_rotated_files_for: 1 day
 
   # How many minutes to wait for all modules to finish before killing them
@@ -201,9 +204,9 @@ modules:
   # List of modules to ignore. By default we always ignore the template,
   # do not remove it from the list
   # Add the names of other modules that you want to disable
-  # (they all should be lowercase with no special characters). Example,
-  # threatintelligence, blocking, networkdiscovery, timeline, virustotal,
-  # rnnccdetection, flowmldetection, updatemanager
+  # (use module snake_case names). Example,
+  # threat_intelligence, blocking, network_discovery, timeline, virustotal,
+  # rnn_cc_detection, flow_ml_detection, update_manager
   disable: [template]
 
   # For each line in timeline file there is a timestamp.
@@ -212,10 +215,10 @@ modules:
   timeline_human_timestamp: true
 
 #############################
-flowmldetection:
+flow_ml_detection:
   # This is a module that uses machine learning for detection.
   # It can be used in train mode or test mode.
-  # The mode 'train' should be used to tell the flowmldetection module
+  # The mode 'train' should be used to tell the flow_ml_detection module
   # that the flows received are all for training.
   # A label should be provided in the [Parameters] section
   # mode : train
@@ -226,7 +229,7 @@ flowmldetection:
   mode: test
 
 #############################
-bruteforcing:
+brute_force_detector:
   # Minimum number of SSH attempts from one source to one destination
   # before Slips considers it brute forcing.
   ssh_attempt_threshold: 9
@@ -299,7 +302,7 @@ virustotal:
 #############################
 threatintelligence:
 
-  # By default, slips starts without the TI files, and runs the Update Manager
+  # By default, slips starts without the TI files, and runs the update_manager
   # in the background. If this option is set to true, slips will not start
   # analyzing the flows until the update manager finished and all TI files are
   # loaded successfully.
@@ -325,7 +328,7 @@ threatintelligence:
   # The remote TI files will be temporaly stored in this directory
   download_path_for_remote_threat_intelligence: modules/threat_intelligence/remote_data_files/
 
-  # Update period of Threat Intelligence files. How often should Slips update
+  # Update period of threat_intelligence files. How often should Slips update
   # the IoCs.
   # The expected value is in seconds.
   # 1 day = 86400 seconds
@@ -389,8 +392,8 @@ whitelists:
   local_whitelist_path: config/whitelist.conf
 
 #############################
-flowalerts:
-  # For the flowalerts module
+flow_alerts:
+  # For the flow_alerts module
   # We need a thrshold to determine a long connection in seconds.
   # In Slips by default is 25 minutes
   long_connection_threshold: 1500
@@ -487,7 +490,7 @@ exporting_alerts:
   taxii_timeout: 10
 
 #############################
-CESNET:
+cesnet:
   # Slips also supports exporting and importing evidence in the IDEA format to/from
   # warden servers of CESNET organization in Czech Republic.
   send_alerts: false
@@ -597,7 +600,7 @@ global_p2p:
    use_global_p2p: False
    iris_conf: config/iris_config.yaml
    bootstrapping_node: False
-   bootstrapping_modules: ["fidesModule", "irisModule"]
+   bootstrapping_modules: ["fides", "iris"]
 
 #############################
 local_p2p:

conftest.py

@@ -23,7 +23,6 @@
 parent_dir = os.path.dirname(current_dir)
 sys.path.insert(0, parent_dir)
 
-
 # Suppress TensorFlow logs from C++ backend
 os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"  # 3 = ERROR
 # TensorFlow logs oneDNN messages even with TF_CPP_MIN_LOG_LEVEL=3.

docs/FAQ.md

@@ -12,7 +12,7 @@ If the tensorflow version you're using isn't compatible with your architecture,
 you will get the "Illegal instruction" error and slips will terminate.
 
 To fix this you can disable the modules that use tensorflow by adding
-```rnn-cc-detection, flowmldetection``` to the ```disable``` key in ```config/slips.yaml```
+```rnn-cc-detection, flow_ml_detection``` to the ```disable``` key in ```config/slips.yaml```
 
 
 ## Docker time is not in sync with that of the host

docs/P2P.md

@@ -112,7 +112,7 @@ development of new trust models and modelling behavior of the P2P network.
 
 To use the experiments, clone
 the https://github.com/stratosphereips/p2p4slips-experiments repository into
-`modules/p2ptrust/testing/experiments`.
+`modules/p2p_trust/testing/experiments`.
 
 The experiments run independently (outside of Slips) and start all processes that are needed, including relevant parts
 of Slips.
@@ -148,6 +148,8 @@ The network then replies with a score and confidence for the IP. The higher the
 Once we get the score of the IP, we store it in the database,
 and we alert if the score of this IP is more than 0 (threat level=info).
 
+The persistent local P2P runtime directory is stored under the directory configured by ```parameters.permanent_dir``` in ```config/slips.yaml```. By default, this is ```permanent/p2p_trust_runtime/```.
+
 
 ### Answering the network's request about an IP