Merge pull request #1634 from netalertx/next_release

Next release

Author: jokob-sk

.github/skills/code-standards/SKILL.md

@@ -8,12 +8,22 @@ description: NetAlertX coding standards and conventions. Use this when writing c
 - ask me to review before going to each next step (mention n step out of x)  (AI only)
 - before starting, prepare implementation plan  (AI only)
 - ask me to review it and ask any clarifying questions first
-- add test creation as last step - follow repo architecture patterns - do not place in the root of /test
+- add test creation as last step - follow repo architecture patterns - do not place in the root of `/test`
 - code has to be maintainable, no duplicate code
 - follow DRY principle - maintainability of code is more important than speed of implementation
 - code files should be less than 500 LOC for better maintainability
 - DB columns must not contain underscores, use camelCase instead (e.g., deviceInstanceId, not device_instance_id)
-- treat DB as temporary storage for stats, long term configuration should be stored in the /config folder, the /config folder should allow you to restore most of your functionality (excluding historical data)
+- treat DB as temporary storage for stats, long-term configuration should be stored in the `/config` folder, the `/config` folder should allow you to restore most of your functionality (excluding historical data)
+- never access DB directly from application layers, always use helper functions in `server/db/db_helper.py` and implement new functionality in handlers (e.g., `DeviceInstance` in `server/models/device_instance.py`)
+- always validate and normalize MAC addresses before writing to DB (use `normalize_mac` from `plugin_helper.py`)
+- all subprocess calls must set explicit timeouts
+- use `timeNowUTC` from `utils.datetime_utils` for all time-related operations and DB timestamps (store all timestamps in UTC)
+- use sanitizers from `server/helper.py` for user input before storing in DB
+- reuse shared mocks and factories from `test/db_test_helpers.py` for tests, never redefine them locally
+- use environment variables for runtime paths, never hardcode paths or use relative paths
+- follow existing code style and structure, and ensure backward compatibility with existing installations when submitting PRs
+- all code needs to be scalable to handle large networks with thousands of devices (10k+) without performance degradation
+
 
 ## File Length
 

README.md

@@ -157,7 +157,7 @@ Check the [GitHub Issues](https://github.com/netalertx/NetAlertX/issues) for the
 ## Everything else
 <!--- --------------------------------------------------------------------- --->
 
-<a href="https://trendshift.io/repositories/12670" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12670" alt="jokob-sk%2FNetAlertX | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
+<a href="https://trendshift.io/repositories/19712" target="_blank"><img src="https://trendshift.io/api/badge/repositories/19712" alt="jokob-sk%2FNetAlertX | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 
 ### 📧 Get notified what's new
 
@@ -170,6 +170,7 @@ Get notified about a new release, what new functionality you can use and about b
 - [Fing](https://www.fing.com/) - Network scanner app for your Internet security (Commercial, Phone App, Proprietary hardware)
 - [NetBox](https://netboxlabs.com/) - The gold standard for Network Source of Truth (NSoT) and IPAM.
 - [Zabbix](https://www.zabbix.com/) or [Nagios](https://www.nagios.org/) - Strong focus on infrastructure monitoring.
+- [Domotz](https://www.domotz.com/) - Commercial network monitoring and remote management platform aimed at MSPs, IT teams, and multi-site environments.
 - [NetAlertX](https://netalertx.com) - The streamlined, discovery-focused choice for real-time asset intelligence and noise-free alerting.
 
 ### 💙 Donations

back/app.conf

@@ -30,6 +30,7 @@ REPORT_DASHBOARD_URL='update_REPORT_DASHBOARD_URL_setting'
 INTRNT_RUN='schedule'
 ARPSCAN_RUN='schedule'
 NSLOOKUP_RUN='before_name_updates'
+DIGSCAN_RUN='before_name_updates'
 AVAHISCAN_RUN='before_name_updates'
 NBTSCAN_RUN='before_name_updates'
 

docs/ADVISORY_EYES_ON_GLASS.md

@@ -4,6 +4,9 @@ For Managed Service Providers (MSPs) and Network Operations Centers (NOC), "Eyes
 
 ![filters](./img/ADVISORIES/filters.png)
 
+> [!TIP]
+> If you are using Grafana check the `/metrics` endpoint that exposes **Prometheus-compatible metrics** for NetAlertX, including aggregate device counts and per-device status. See the [Metrics API endpoint](./API_METRICS.md) documentation for details.
+
 ---
 
 ### 1. Configure Auto-Refresh for Live Monitoring

docs/ADVISORY_MULTI_NETWORK.md

@@ -12,20 +12,25 @@ Effective multi-network monitoring starts with understanding how NetAlertX "sees
 * **Manual Entry:** For static assets where only ICMP (ping) status is needed.
 
 > [!TIP]
-> Explore the [remote networks](./REMOTE_NETWORKS.md) documentation for more details on how to set up the approaches menationed above.
+> Explore the [remote networks](./REMOTE_NETWORKS.md) documentation for more details on how to set up the approaches mentioned above.
 
 ---
 
 ### 2. Automating IT Asset Inventory with Workflows
 
-[Workflows](./WORKFLOWS.md) are the "engine" of NetAlertX, reducing manual overhead as your device list grows.
+[Workflows](./WORKFLOWS.md) are the "engine" of NetAlertX, reducing manual overhead as your device list grows. See some examples below:
+
+#### A. Logical Ownership & VLAN Tagging
+
+Create a workflow triggered on **Device Creation** to:
 
-* **A. Logical Ownership & VLAN Tagging:** Create a workflow triggered on **Device Creation** to:
 1. Inspect the IP/Subnet.
 2. Set `devVlan` or `devOwner` custom fields automatically.
 
+#### B. Auto-Grouping:
+
+Use conditional logic to categorize devices.
 
-* **B. Auto-Grouping:** Use conditional logic to categorize devices.
 * *Example:* If `devLastIP == 10.10.20.*`, then `Set devLocation = "BranchOffice"`.
 
 ```json
@@ -56,9 +61,9 @@ Effective multi-network monitoring starts with understanding how NetAlertX "sees
   ]
 }
 ```
+#### C. Sync Node Tracking
 
-
-* **C. Sync Node Tracking:** When using multiple instances, ensure all synchub nodes have a descriptive `SYNC_node_name` name to distinguish between sites.
+When using multiple instances, ensure all sync hub nodes have a descriptive `SYNC_node_name` name to distinguish between sites.
 
 > [!TIP]
 > Always test new workflows in a "Staging" instance. A misconfigured workflow can trigger thousands of unintended updates across your database.
@@ -107,6 +112,7 @@ As your environment grows, tuning the underlying engine is vital to maintain a s
 * **Plugin Scheduling:** Avoid "Scan Storms" by staggering plugin execution. Running intensive tasks like `NMAP` or `MASS_DNS` simultaneously can spike CPU and cause database locks.
 * **Database Health:** Large-scale monitoring generates massive event logs. Use the **[DBCLNP (Database Cleanup)](https://www.google.com/search?q=https://docs.netalertx.com/PLUGINS/%23dbclnp)** plugin to prune old records and keep the SQLite database performant.
 * **Resource Management:** For high-device counts, consider increasing the memory limit for the container and utilizing `tmpfs` for temporary files to reduce SD card/disk I/O bottlenecks.
+* Enable the `DEEP_SLEEP` setting.
 
 > [!IMPORTANT]
 > For a deep dive into hardware requirements, database vacuuming, and specific environment variables for high-load instances, refer to the full **[Performance Optimization Guide](https://docs.netalertx.com/PERFORMANCE/)**.

docs/DEVICE_FIELD_LOCK.md

@@ -1,5 +1,7 @@
 # Quick Reference Guide - Device Field Lock/Unlock System
 
+> For how scan overwrite rules (`SET_ALWAYS`, `SET_EMPTY`) and source tracking work under the hood, see [Device Source Fields](./DEVICE_SOURCE_FIELDS.md).
+
 ## Overview
 
 ![Field source and locks](./img/DEVICE_MANAGEMENT/field_sources_and_locks.png)

docs/DEVICE_SOURCE_FIELDS.md

@@ -1,5 +1,7 @@
 # Understanding Device Source Fields and Field Updates
 
+> For the UI guide on locking and unlocking individual fields, see [Device Field Lock/Unlock](./DEVICE_FIELD_LOCK.md).
+
 When the system scans a network, it finds various details about devices (like names, IP addresses, and manufacturers). To ensure the data remains accurate without accidentally overwriting manual changes, the system uses a set of "Source Rules."
 
 ![Field source and locks](./img/DEVICE_MANAGEMENT/field_sources_and_locks.png)
@@ -17,6 +19,8 @@ Every piece of information for a device has a **Source**. This source determines
 | **NEWDEV** | This value was initialized from `NEWDEV` plugin settings. | **Always** |
 | **(Plugin Name)** | The value was found by a specific scanner (e.g., `NBTSCAN`). | **Only if specific rules are met** |
 
+> For how `USER` and `LOCKED` sources are set through the UI (lock/unlock buttons), see [Device Field Lock/Unlock](./DEVICE_FIELD_LOCK.md).
+
 ---
 
 ## How Scans Update Information
@@ -34,6 +38,8 @@ Some plugins are configured to be "authoritative." If a field is in the **SET_AL
 * The scanner will **always** overwrite the current value with the new one.
 * *Note: It will still never overwrite a `USER` or `LOCKED` field.*
 
+> On large networks, enabling `SET_ALWAYS` on name-resolution fields (e.g., `devName`) widens the resolution scope and increases DNS query volume. See [Performance — Plugin Field Authority](./PERFORMANCE.md#plugin-field-authority-set_always-and-set_empty) for details.
+
 ### 3. SET_EMPTY
 
 If a field is in the **SET_EMPTY** list:

docs/PERFORMANCE.md

@@ -95,6 +95,41 @@ For example, the **ICMP plugin** allows scanning only IPs that match a specific
 
 ---
 
+## Plugin Field Authority: `SET_ALWAYS` and `SET_EMPTY`
+
+Plugins can be configured to control how aggressively they overwrite existing device field values via two settings:
+
+| Setting | Behaviour |
+|---|---|
+| `<PLUGIN>_SET_ALWAYS` | Plugin always overwrites the field, as long as it can resolve a value and the field is not `USER`/`LOCKED` |
+| `<PLUGIN>_SET_EMPTY` | Plugin only writes when the field is currently empty |
+
+Both settings accept a list of field names (e.g., `devName`, `devFQDN`). See [Name resolution](./NAME_RESOLUTION.md) and [Field locking](./DEVICE_SOURCE_FIELDS.md) docs for details.
+
+### Performance Impact of `SET_ALWAYS` on Name Resolution
+
+By default, name resolution (DIGSCAN, NBTSCAN, NSLOOKUP, AVAHISCAN) only runs against **devices that have no name yet**. This keeps DNS query volume proportional to new devices discovered, not the total inventory.
+
+When **any** name-resolution plugin has `devName` in its `SET_ALWAYS` list, the system additionally runs a second resolution pass against **all devices whose name is not `USER`/`LOCKED` protected**. This allows a higher-priority plugin (e.g., DIGSCAN) to overwrite names previously set by a lower-priority one (e.g., NBTSCAN).
+
+**Cost:** one DNS query per unprotected, already-named device per name-resolution cycle.
+
+| Scenario | Devices resolved per cycle |
+|---|---|
+| No `SET_ALWAYS` on `devName` | Only new/unknown devices |
+| `SET_ALWAYS: devName` on any plugin | New/unknown devices **+** all unprotected named devices |
+
+> [!WARNING]
+> On large installations (thousands of devices), enabling `SET_ALWAYS: devName` significantly increases DNS query volume and cycle duration. To mitigate:
+>
+> * Increase the scan interval of name-resolution plugins (`DIGSCAN_RUN_SCHD`, `NBTSCAN_RUN_SCHD`, etc.).
+> * Mark devices whose name should never change as `USER` or `LOCKED` — they are excluded from the re-resolve pass entirely.
+> * Use `SET_ALWAYS` only on the highest-priority plugin; leave lower-priority plugins without it.
+
+The actual number of DB rows updated is logged at `verbose` level under `[Update Device Name] SET_ALWAYS re-resolve - DB rows updated`.
+
+---
+
 ## Storing Temporary Files in Memory
 
 On devices with slower I/O, you can improve performance by storing temporary files (and optionally the database) in memory using `tmpfs`.

docs/PLUGINS_DEV_UI_COMPONENTS.md

@@ -7,6 +7,7 @@ Configure how your plugin's data is displayed in the NetAlertX web interface.
 Plugin results are displayed in the UI via the **Plugins page** and **Device details tabs**. You control the appearance and functionality of these displays by defining `database_column_definitions` in your plugin's `config.json`.
 
 Each column definition specifies:
+
 - Which data field to display
 - How to render it (label, link, color-coded badge, etc.)
 - What CSS classes to apply
@@ -275,6 +276,7 @@ Replaces specific values with display strings or HTML.
 ```
 
 **Output Examples:**
+
 - `"online"` → 🟢 Online
 - `"offline"` → 🔴 Offline
 - `"idle"` → 🟡 Idle

docs/WORKFLOW_EXAMPLES.md

@@ -45,22 +45,22 @@ Sometimes devices are manually archived (e.g., no longer expected on the network
   ],
   "enabled": "Yes"
 }
-```
+````
 
 ### 🔍 Explanation
 
-    - Trigger: Listens for updates to device records.
-    - Conditions:
-        - `devIsArchived` is `1` (archived).
-        - `devPresentLastScan` is `1` (device was detected in the latest scan).
-    - Action: Updates the device to set `devIsArchived` to `0` (unarchived).
+* **Trigger**: Listens for updates to device records.
+* **Conditions**:
 
-### ✅ Result
+  * `devIsArchived` is `1` (archived).
+  * `devPresentLastScan` is `1` (device was detected in the latest scan).
+* **Action**:
 
-Whenever a previously archived device shows up during a network scan, it will be automatically unarchived — allowing it to reappear in your device lists and dashboards.
+  * Updates the device to set `devIsArchived` to `0` (unarchived).
 
+### ✅ Result
 
-Here is your updated version of **Example 2** and **Example 3**, fully aligned with the format and structure of **Example 1** for consistency and professionalism:
+Whenever a previously archived device shows up during a network scan, it will be automatically unarchived — allowing it to reappear in your device lists and dashboards.
 
 ---
 
@@ -107,7 +107,7 @@ When new devices join your network, assigning them to the correct network node i
 ### 🔍 Explanation
 
 * **Trigger**: Activates when a new device is added.
-* **Condition**:
+* **Conditions**:
 
   * `devLastIP` contains `192.168.1.` (matches subnet).
 * **Action**:
@@ -173,12 +173,12 @@ You may want to automatically clear out newly detected Google devices (such as C
 * **Trigger**: Runs on device updates.
 * **Conditions**:
 
-  * Vendor contains `Google`.
-  * Device is marked as new (`devIsNew` is `1`).
+  * `devVendor` contains `Google`.
+  * `devIsNew` is `1` (device marked as new).
 * **Actions**:
 
-  1. Set `devIsNew` to `0` (mark as not new).
-  2. Delete the device.
+  1. Sets `devIsNew` to `0` (mark as not new).
+  2. Deletes the device.
 
 ### ✅ Result