fix/python upgrade 3.11->3.13 by IshaanXCoder · Pull Request #3320 · intelowlproject/IntelOwl

IshaanXCoder · 2026-02-15T20:44:01Z

Description

This PR resolved #3306 where we're upgrading python version from 3.11 to 3.13
I have updated the necessary configuration files, CI workflows, Docker files and test files for compatibility (a few test files are still breaking)

Minor changes were made due to some depreceated functions like assertDictContainsSubset()
pydeep has been replaced with ppdeep
removal of PosixPath
in api_app/analyzers_manager/observable_analyzers/maxmind.py , i noticed the test was failing with cls._api_key_name, changed it to auth_token (need a feedback on this)
made some changes in docker/test.override.yml for my local , adding for reference here.

Type of change

New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected -> tried my best to fix these, still might be a few, will fix with discussion).

Checklist

I have read and understood the rules about how to Contribute to this project
The pull request is for the branch develop
I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.
Please avoid adding new libraries as requirements whenever it is possible. Use new libraries only if strictly needed to solve the issue you are working for. In case of doubt, ask a maintainer permission to use a specific library.
If external libraries/packages with restrictive licenses were added, they were added in the Legal Notice section.
Linters (Ruff) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review by using GitHub's reviewing system detailed here.

IshaanXCoder · 2026-02-15T20:45:37Z

@mlodic ptal at this draft pr, still need to make many changes on failing tests, i just want to get some reviews on the work done till now

Copilot

Pull request overview

This PR upgrades the project from Python 3.11 to Python 3.13, addressing compatibility issues with deprecated functions and updating dependencies. The linked issue mentions Python 3.14, but correctly targets Python 3.13 (the latest stable version).

Changes:

Updated Python base images to 3.13 in Dockerfiles and CI workflows
Replaced deprecated testing methods (assertDictContainsSubset) with explicit key-value assertions
Replaced pydeep library with ppdeep and updated API calls accordingly
Upgraded major dependencies (Django 5.2, DRF 3.16.1, Celery 5.6.0, etc.)
Fixed bug in maxmind.py where incorrect variable was passed to update_all_dbs
Removed unnecessary PosixPath usage from TextChoices enum
Updated typing.Generator syntax to simplified Python 3.9+ format

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docker/Dockerfile	Updated base image from python:3.11.7 to python:3.13-slim
integrations/tor_analyzers/Dockerfile	Fixed python:3.14-slim (non-existent) to python:3.13-slim
integrations/malware_tools_analyzers/Dockerfile	Updated base image to python:3.13-slim
.github/workflows/pull_request_automation.yml	Updated CI Python version to 3.13
.github/workflows/codeql-analysis.yml	Updated CI Python version to 3.13
pyproject.toml	Updated Ruff target-version to py313
requirements/project-requirements.txt	Upgraded Django, DRF, Celery, and other dependencies for Python 3.13 compatibility; replaced pydeep with ppdeep
api_app/choices.py	Removed PosixPath wrapper from TextChoices enum values; simplified typing.Generator syntax
api_app/analyzers_manager/file_analyzers/file_info.py	Updated to use ppdeep.hash_from_file instead of pydeep.hash_file
api_app/analyzers_manager/observable_analyzers/maxmind.py	Fixed bug: pass auth_token instead of cls._api_key_name to update_all_dbs
tests/test_crons.py	Removed skip import (unused); updated test_maxmind_updater with proper mocking; changed retention days from 10 to 30
tests/api_app/test_views.py	Replaced deprecated assertDictContainsSubset with explicit for-loop assertions
tests/api_app/test_api.py	Replaced deprecated assertDictContainsSubset with assertIn and assertEqual
tests/api_app/analyzers_manager/unit_tests/observable_analyzers/test_greynoise_labs.py	Changed report mock from dict to SimpleNamespace with save method
tests/api_app/analyzers_manager/unit_tests/file_analyzers/test_file_info.py	Updated mock from pydeep.hash_file to ppdeep.hash_from_file
tests/api_app/analyzers_manager/unit_tests/file_analyzers/test_blint_scan.py	Fixed mock patch to target import location in module (best practice)
tests/api_app/analyzers_manager/unit_tests/file_analyzers/base_test_class.py	Added missing test file mapping for application/x-elf
docker/test.override.yml	Added MOCK_CONNECTIONS and CELERY_TASK_SOFT_TIME_LIMIT environment variables

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

IshaanXCoder · 2026-02-17T06:07:03Z

Hey @mlodic PTAL at this, i've made the required changes, few tests are still failing on my local, but they seem to be unrelated with python migration.

mlodic

there are still linter errors.

Also, ppdeep is not compliant to IntelOwl basic scorecard checks. I checked that project and there is a single maintainer keeping it so it can be quite unreliable on the long term. Isn't there any alternatives?

IshaanXCoder · 2026-02-17T10:58:57Z

there are still linter errors.

alright i've fixed this, there was an erorr due to modern style of python (eg, Generator[Signature, None, None] to the modern equivalent Generator[Signature])

Also, ppdeep is not compliant to IntelOwl basic scorecard checks. I checked that project and there is a single maintainer keeping it so it can be quite unreliable on the long term. Isn't there any alternatives?

an alternative which we've is https://github.com/DinoTools/python-ssdeep , but it doesnt have any commits since past 5 years.

IshaanXCoder · 2026-02-17T12:29:57Z

@mlodic should i proceed with the above approaches?

mlodic · 2026-02-17T13:55:53Z

that's still another very niche and old repository but it is older like the other one that we are already using as of now. You can try it and see whether it passes the tests and the scorecard test

IshaanXCoder · 2026-02-17T14:31:08Z

that's still another very niche and old repository but it is older like the other one that we are already using as of now. You can try it and see whether it passes the tests and the scorecard test

Alright, agreed.
Sure, i'll check and let you know about the same at the earliest.

IshaanXCoder · 2026-02-17T17:14:23Z

Nope, ssdeep didnt work due to python version issue

mlodic · 2026-02-17T19:54:06Z

we can use the ppdeep you propose first but we have to find a way to circumvent the dependency check with a special exclusion. See: https://github.com/intelowlproject/IntelOwl/blob/master/.github/workflows/dependency_review.yml
https://github.com/actions/dependency-review-action

IshaanXCoder · 2026-02-17T20:11:14Z

find a way to circumvent the dependency check with a special exclusion

this can be done using allow-dependencies-licenses (mentioned here - https://github.com/actions/dependency-review-action/blob/main/README.md#configuration-options)

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-18T20:10:03Z

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.13.12


Several integration Dockerfiles still use older Python versions and have not been updated to Python 3.13. For consistency and to complete the Python upgrade, the following Dockerfiles should also be updated: integrations/bbot/Dockerfile (uses python:3.12-slim), integrations/phishing_analyzers/Dockerfile (uses python:3.12.3), and integrations/phunter/Dockerfile (uses python:3.12-slim). Note that integrations/pcap_analyzers, integrations/thug, integrations/nuclei_analyzer, and integrations/cyberchef use base images that don't directly specify Python versions.

Copilot · 2026-02-18T20:10:03Z

                return_value="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            ),
-            patch("pydeep.hash_file", return_value=b"3:AOn4:An"),
            patch("tlsh.hash", return_value="T1234567890ABCDEF"),


Missing mock for ppdeep in the test. The old pydeep mock was removed, but no mock for ppdeep.hash_from_file was added. This will cause the test to fail if ppdeep is not installed or if the test file path doesn't exist, as ppdeep.hash_from_file will be called with an actual file path. Consider adding a mock for ppdeep similar to the other hash function mocks.

Suggested change

patch("tlsh.hash", return_value="T1234567890ABCDEF"),

patch("tlsh.hash", return_value="T1234567890ABCDEF"),

patch("ppdeep.hash_from_file", return_value="PPDEEP_HASH_VALUE"),

IshaanXCoder · 2026-02-18T20:22:05Z

A lot of commits have been added, i'll make a clean pr later after all the changes/reviews.

mlodic · 2026-02-19T12:14:42Z

sure, thanks

IshaanXCoder · 2026-02-19T12:16:22Z

sure, thanks

H3y @mlodic ! do we need any more changes or are we good to go? i'll make a new pr in that case.

mlodic · 2026-02-19T12:21:40Z

it seems there are still errors in the CI

IshaanXCoder · 2026-02-20T10:06:55Z

Hey @mlodic i checked out the test failures, some seemed to be independent of the python upgrade, but i've made a clean fix for all the backend tests. Have used patch and override_settings in some cases, hope that's not an issue.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

IshaanXCoder · 2026-02-20T11:12:15Z

working on this fail

IshaanXCoder · 2026-02-20T11:20:11Z

there's an issue with pillow dependency,
https://github.com/intelowlproject/IntelOwl/actions/runs/22221253027/job/64276951563?pr=3320
Checked in the codebase, it's not being used anywhere, should we remove this?

mlodic · 2026-02-20T11:28:03Z

It's a subdependency IIRC. Anyway I don't see the error, where it is?

IshaanXCoder · 2026-02-20T11:30:12Z

under vulnerabilities

Copilot

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

IshaanXCoder · 2026-02-21T13:52:11Z

Opened a clean PR at #3352

fix/python upgrade 3.11->3.13

8e00415

Copilot AI review requested due to automatic review settings February 15, 2026 20:44

IshaanXCoder marked this pull request as draft February 15, 2026 20:44

Copilot started reviewing on behalf of IshaanXCoder February 15, 2026 20:44 View session

Copilot AI reviewed Feb 15, 2026

View reviewed changes

Comment thread tests/test_crons.py Outdated

Comment thread docker/test.override.yml Outdated

mlodic requested changes Feb 16, 2026

View reviewed changes

fix/maintainer feedback, 3.13-slim -> 3.13.12

b19c638

mlodic requested changes Feb 17, 2026

View reviewed changes

mlodic reviewed Feb 17, 2026

View reviewed changes

Comment thread requirements/project-requirements.txt Outdated

fix/linter-error

b09018d

fix/ignore-dependency-check

7beaf2d

IshaanXCoder marked this pull request as ready for review February 18, 2026 19:51

Copilot AI review requested due to automatic review settings February 18, 2026 19:51

Copilot started reviewing on behalf of IshaanXCoder February 18, 2026 19:52 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

Comment thread pyproject.toml

Comment thread requirements/project-requirements.txt Outdated

Comment thread api_app/analyzers_manager/file_analyzers/file_info.py

Comment thread docker/test.override.yml Outdated

Update docker/test.override.yml

1c58384

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 18, 2026 20:04

Copilot started reviewing on behalf of IshaanXCoder February 18, 2026 20:04 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

fix/suggestions

4933917

IshaanXCoder requested a review from mlodic February 18, 2026 22:13

fix/failing backend tests

81f3a9e

Copilot AI review requested due to automatic review settings February 20, 2026 10:52

Copilot AI reviewed Feb 20, 2026

View reviewed changes

fix/cicd errors

9fd0ae1

IshaanXCoder commented Feb 20, 2026

View reviewed changes

Comment thread requirements/project-requirements.txt Outdated

Apply suggestion from @IshaanXCoder

67c9c3e

Copilot AI review requested due to automatic review settings February 20, 2026 12:46

Copilot started reviewing on behalf of IshaanXCoder February 20, 2026 12:46 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

Comment thread .github/workflows/dependency_review.yml

Comment thread tests/api_app/test_classes.py

Comment thread docker/Dockerfile Outdated

Comment thread api_app/serializers/job.py

Comment thread api_app/analyzers_manager/classes.py

Comment thread tests/test_crons.py

IshaanXCoder added 2 commits February 20, 2026 19:02

fix/copilot-suggestions

ccccfa8

fix

880ffed

Copilot AI review requested due to automatic review settings February 20, 2026 13:38

Copilot started reviewing on behalf of IshaanXCoder February 20, 2026 13:39 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

fix/typing-errors

87e4bc7

IshaanXCoder mentioned this pull request Feb 21, 2026

Python and Django upgrade #3352

Open

6 tasks

IshaanXCoder closed this Feb 21, 2026

	patch("tlsh.hash", return_value="T1234567890ABCDEF"),
	patch("tlsh.hash", return_value="T1234567890ABCDEF"),
	patch("ppdeep.hash_from_file", return_value="PPDEEP_HASH_VALUE"),

Uh oh!

Conversation

IshaanXCoder commented Feb 15, 2026

Description

Type of change

Checklist

Important Rules

Uh oh!

IshaanXCoder commented Feb 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IshaanXCoder commented Feb 17, 2026

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

IshaanXCoder commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IshaanXCoder commented Feb 17, 2026

Uh oh!

mlodic commented Feb 17, 2026

Uh oh!

IshaanXCoder commented Feb 17, 2026

Uh oh!

IshaanXCoder commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlodic commented Feb 17, 2026

Uh oh!

IshaanXCoder commented Feb 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

IshaanXCoder commented Feb 18, 2026

Uh oh!

mlodic commented Feb 19, 2026

Uh oh!

IshaanXCoder commented Feb 19, 2026

Uh oh!

mlodic commented Feb 19, 2026

Uh oh!

IshaanXCoder commented Feb 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

IshaanXCoder commented Feb 20, 2026

Uh oh!

IshaanXCoder commented Feb 20, 2026

IshaanXCoder commented Feb 17, 2026 •

edited

Loading

IshaanXCoder commented Feb 17, 2026 •

edited

Loading