fix(misp): eliminate N+1 HTTP requests in MISP connector. Closes #3571 by jagapathi20 · Pull Request #3579 · intelowlproject/IntelOwl

jagapathi20 · 2026-03-31T05:06:17Z

Description

The MISP.run() method called misp_instance.add_attribute() in a loop
after add_event(), issuing one HTTP request per attribute (N+1 pattern).

Fixed by pre-attaching all attributes to the local MISPEvent object
before calling add_event(), so MISP persists the event and all its
attributes in a single request.

Closes #3571

Type of change

Bug fix (non-breaking change which fixes an issue).

Checklist

I have read and understood the rules about how to Contribute to this project
The pull request is for the branch develop
I have inserted the copyright banner at the start of the file
Please avoid adding new libraries as requirements whenever it is possible. Use new libraries only if strictly needed to solve the issue you are working for.
Linters (Ruff) gave 0 errors.

Copilot

Pull request overview

This PR optimizes the MISP connector to avoid an N+1 HTTP request pattern when creating events with multiple attributes, reducing connector runtime/network overhead and addressing issue #3571.

Changes:

Pre-attaches all generated attributes to the local pymisp.MISPEvent before calling add_event(), so event + attributes are sent in a single request.
Avoids recomputing the base attribute by caching it once in run().
Fixes hash type normalization by correctly applying replace("-", "") when mapping hash classifications.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-31T05:10:00Z

+        # bulk: attach all attributes to the event object before sending
        for attr in attributes:
-            misp_instance.add_attribute(misp_event.id, attr)
+            event.add_attribute(
+                attr.type,
+                attr.value,
+                **{k: v for k, v in attr.to_dict().items() if k not in ("type", "value", "uuid")},
+            )
+
+        # add event + all attributes in a single request (bulk optimization)
+        misp_event = misp_instance.add_event(event, pythonify=True)
+
+        result = misp_instance.get_event(misp_event.id)



Optional: add a regression test for this bulk optimization (e.g., patch pymisp.PyMISP and assert add_event is called once and add_attribute is not invoked per-attribute) to help prevent the N+1 pattern from being reintroduced later.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-31T05:53:23Z

+        # bulk: attach all attributes to the event object before sending
        for attr in attributes:
-            misp_instance.add_attribute(misp_event.id, attr)
+            event.add_attribute(
+                attr.type,
+                attr.value,
+                **{k: v for k, v in attr.to_dict().items() if k not in ("type", "value", "uuid")},
+            )
+
+        # single request — event + all attributes sent together
+        misp_event = misp_instance.add_event(event, pythonify=True)


The refactor changes the connector’s behavior from multiple PyMISP.add_attribute() calls to a single add_event() payload. There are unit tests for other connectors under tests/api_app/connectors_manager/, but none covering the MISP connector’s request pattern. Please add a test that mocks pymisp.PyMISP and asserts add_event() is called once and add_attribute() is not called, so the N+1 regression is prevented.

jagapathi20 · 2026-03-31T06:22:17Z

hi @mlodic, as proposed in issue #3571, I have made changes. can you review them, when you get a chance?

mlodic

without unittests this won't be merged. I also need proof that this work with a MISP instance otherwise this is all theoretical only

jagapathi20 · 2026-03-31T17:08:44Z

without unittests this won't be merged. I also need proof that this work with a MISP instance otherwise this is all theoretical only

okay. Understood. I will update this PR shortly

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-10T10:45:53Z

+        # bulk: attach all attributes to the event object before sending
        for attr in attributes:
-            misp_instance.add_attribute(misp_event.id, attr)
+            event.add_attribute(
+                attr.type,
+                attr.value,


This change materially alters how attributes are sent to MISP (bundled into the initial add_event call). There don’t appear to be unit tests covering the MISP connector; please add tests to assert that PyMISP.add_event is called once with the event containing all expected attributes and that PyMISP.add_attribute is no longer called (preventing regressions back to N+1 requests).

Copilot · 2026-04-10T10:45:53Z

        # append attribute name to event info
        event.info += f": {self._base_attr_obj.value}"

-        # add event to MISP Instance
-        misp_event = misp_instance.add_event(event, pythonify=True)
-        # add attributes to event on MISP Instance
+        # bulk: attach all attributes to the event object before sending


event.info += f": {self._base_attr_obj.value}" recomputes _base_attr_obj even though it was already computed when building attributes above. Since _base_attr_obj performs a DB query to build the analyzers list, consider computing it once (e.g., base_attr = self._base_attr_obj) and reusing it for both event.info and the attributes list to avoid duplicate queries and keep values consistent.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-10T12:43:59Z

+from api_app.analyzables_manager.models import Analyzable
+from api_app.choices import Classification
+from api_app.connectors_manager.connectors.misp import MISP
+from api_app.connectors_manager.models import ConnectorConfig, ConnectorReport
+from api_app.models import Job, Parameter, PluginConfig
+from tests import CustomTestCase


PluginConfig is imported but never used in this test module. Removing unused imports avoids lint noise and keeps the test focused.

jagapathi20 · 2026-04-10T13:39:04Z

Proof

Jobs created in IntelOwl

Jobs got in MISP

Number of requests made by IntelOwl to MISP

Unittests

jagapathi20 · 2026-04-10T13:47:46Z

Hi @mlodic, sorry for the delay, was busy in college work.As asked I implemented unittests, and attached proof that this works with MISP Instance, can you review them, when you get a chance.

Copilot AI review requested due to automatic review settings March 31, 2026 05:06

Copilot started reviewing on behalf of jagapathi20 March 31, 2026 05:06 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

jagapathi20 force-pushed the fix/misp-n-plus-one-add-attribute branch from fb35091 to 5ab8e7c Compare March 31, 2026 05:27

Copilot AI review requested due to automatic review settings March 31, 2026 05:50

Copilot started reviewing on behalf of jagapathi20 March 31, 2026 05:50 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

jagapathi20 force-pushed the fix/misp-n-plus-one-add-attribute branch from 4eb3f22 to 34da343 Compare March 31, 2026 06:06

mlodic requested changes Mar 31, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 10, 2026 10:42

Copilot AI reviewed Apr 10, 2026

View reviewed changes

fix(connectors): eliminate N+1 HTTP requests in MISP connector

327e86f

jagapathi20 force-pushed the fix/misp-n-plus-one-add-attribute branch from a374c35 to 416b018 Compare April 10, 2026 11:02

Copilot AI review requested due to automatic review settings April 10, 2026 12:40

jagapathi20 force-pushed the fix/misp-n-plus-one-add-attribute branch from 416b018 to 8f8b906 Compare April 10, 2026 12:40

Copilot started reviewing on behalf of jagapathi20 April 10, 2026 12:40 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

test(connectors): add unit tests for MISP bulk add_event

d74da6f

jagapathi20 force-pushed the fix/misp-n-plus-one-add-attribute branch from 8f8b906 to d74da6f Compare April 10, 2026 13:20

mlodic merged commit a6a7246 into intelowlproject:develop Apr 14, 2026
9 checks passed

Uh oh!

Conversation

jagapathi20 commented Mar 31, 2026

Description

Type of change

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

jagapathi20 commented Mar 31, 2026

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

jagapathi20 commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

jagapathi20 commented Apr 10, 2026

Proof

Uh oh!

jagapathi20 commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants