Skip to content

Commit ed08ca6

Browse files
authored
Merge pull request GreedyBear-Project#1025 from intelowlproject/develop
3.3.0
2 parents 5d209b3 + ec99a5b commit ed08ca6

178 files changed

Lines changed: 14687 additions & 3561 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.dockerignore

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
.vscode
44
.lgtm.yml
55
__pycache__
6+
.venv/
67
venv/
78
**/build
89
.env
@@ -15,4 +16,6 @@ frontend/dist
1516
frontend/build
1617
docker-compose*
1718
.pre-commit-config.yaml
18-
.ipython/
19+
.ipython/
20+
.github/
21+
tests/

.github/dependabot.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
version: 2
22

33
updates:
4-
- package-ecosystem: "pip"
5-
directory: "/requirements"
4+
- package-ecosystem: "uv"
5+
directory: "/"
66
schedule:
77
interval: "weekly"
88
day: "tuesday"

.github/release_template.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Checklist for creating a new release
22

3-
- [ ] Change version number in `docker/.version` and in `.env_template`
3+
- [ ] Change version number in `pyproject.toml`
44
- [ ] Verify CI Tests
55
- [ ] Verify that the PR is named with a correct version number like x.x.x
66
- [ ] Merge the PR to the `main` branch. The release will be done automatically by the CI

.github/workflows/_python.yml

Lines changed: 10 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ on:
1717
type: string
1818
required: true
1919
requirements_path:
20-
description: Path to the requirements.txt file
20+
description: Path to the requirements.txt file (deprecated, unused with uv)
2121
type: string
22-
required: true
22+
required: false
2323
project_dev_requirements_file:
2424
description: Path to an additional project dev requirements file
2525
type: string
@@ -264,7 +264,7 @@ jobs:
264264
id: setup_python
265265
uses: actions/setup-python@v5
266266
with:
267-
python-version: ${{ matrix.python_version }}
267+
python-version-file: "pyproject.toml"
268268

269269
- name: Inject stuff to environment
270270
run: |
@@ -312,128 +312,19 @@ jobs:
312312
with:
313313
apt_requirements_file_path: ${{ inputs.packages_path }}
314314

315-
- name: Create linter requirements file
316-
uses: ./.github/actions/python_requirements/create_linter_requirements_file
317-
with:
318-
install_from: ${{ inputs.install_from }}
319-
django_settings_module: ${{ inputs.django_settings_module }}
320-
use_autoflake: ${{ inputs.use_autoflake }}
321-
use_bandit: ${{ inputs.use_bandit }}
322-
use_black: ${{ inputs.use_black }}
323-
use_flake8: ${{ inputs.use_flake8 }}
324-
use_isort: ${{ inputs.use_isort }}
325-
use_pylint: ${{ inputs.use_pylint }}
326-
use_ruff_formatter: ${{ inputs.use_ruff_formatter }}
327-
use_ruff_linter: ${{ inputs.use_ruff_linter }}
328-
329-
- name: Create dev requirements file
330-
uses: ./.github/actions/python_requirements/create_dev_requirements_file
331-
with:
332-
install_from: ${{ inputs.install_from }}
333-
project_dev_requirements_file: ${{ inputs.project_dev_requirements_file }}
334-
335-
- name: Create docs requirements file
336-
uses: ./.github/actions/python_requirements/create_docs_requirements_file
315+
- name: Install uv
316+
uses: astral-sh/setup-uv@v7
337317
with:
338-
install_from: ${{ inputs.install_from }}
339-
check_docs_directory: ${{ inputs.check_docs_directory }}
340-
django_settings_module: ${{ inputs.django_settings_module }}
318+
enable-cache: true
319+
version: "0.11.1"
341320

342-
- name: Restore Python virtual environment related to PR event
343-
id: restore_python_virtual_environment_pr
344-
uses: ./.github/actions/python_requirements/restore_virtualenv/
345-
with:
346-
requirements_paths: "${{ inputs.requirements_path }} requirements-linters.txt requirements-dev.txt requirements-docs.txt"
347-
python_version: ${{ steps.setup_python.outputs.python-version }}
348-
349-
- name: Restore Python virtual environment related to target branch
350-
id: restore_python_virtual_environment_target_branch
351-
if: steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true'
352-
uses: ./.github/actions/python_requirements/restore_virtualenv/
353-
with:
354-
requirements_paths: ${{ inputs.requirements_path }}
355-
git_reference: ${{ github.base_ref }}
356-
python_version: ${{ steps.setup_python.outputs.python-version }}
357-
358-
- name: Create Python virtual environment
359-
if: >
360-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true' &&
361-
steps.restore_python_virtual_environment_target_branch.outputs.cache-hit != 'true'
362-
uses: ./.github/actions/python_requirements/create_virtualenv
363-
364-
- name: Restore pip cache related to PR event
365-
id: restore_pip_cache_pr
366-
if: >
367-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true' &&
368-
steps.restore_python_virtual_environment_target_branch.outputs.cache-hit != 'true'
369-
uses: ./.github/actions/python_requirements/restore_pip_cache
370-
371-
- name: Restore pip cache related to target branch
372-
id: restore_pip_cache_target_branch
373-
if: >
374-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true' &&
375-
steps.restore_python_virtual_environment_target_branch.outputs.cache-hit != 'true' &&
376-
steps.restore_pip_cache_pr.outputs.cache-hit != 'true'
377-
uses: ./.github/actions/python_requirements/restore_pip_cache
378-
with:
379-
git_reference: ${{ github.base_ref }}
380-
381-
- name: Install project requirements
382-
if: >
383-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true' &&
384-
steps.restore_python_virtual_environment_target_branch.outputs.cache-hit != 'true'
385-
run: pip install -r ${{ inputs.requirements_path }}
386-
shell: bash
387-
working-directory: ${{ inputs.install_from }}
388-
389-
- name: Install other requirements
390-
if: >
391-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true'
321+
- name: Install dependencies
392322
run: |
393-
pip install -r requirements-dev.txt
394-
pip install -r requirements-linters.txt
395-
pip install -r requirements-docs.txt
323+
uv sync --frozen --all-groups
324+
echo "${{ github.workspace }}/${{ inputs.install_from }}/.venv/bin" >> $GITHUB_PATH
396325
shell: bash
397326
working-directory: ${{ inputs.install_from }}
398327

399-
- name: Check requirements licenses
400-
if: >
401-
inputs.check_requirements_licenses &&
402-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true'
403-
id: license_check_report
404-
continue-on-error: true
405-
uses: pilosus/action-pip-license-checker@v2
406-
with:
407-
requirements: ${{ inputs.install_from }}/${{ inputs.requirements_path }}
408-
exclude: ${{ inputs.ignore_requirements_licenses_regex }}
409-
headers: true
410-
fail: 'StrongCopyleft,NetworkCopyleft,Error'
411-
fails-only: true
412-
413-
- name: Print wrong licenses
414-
if: steps.license_check_report.outcome == 'failure'
415-
run: |
416-
echo "License check failed"
417-
echo "===================="
418-
echo "${{ steps.license_check_report.outputs.report }}"
419-
echo "===================="
420-
exit 1
421-
shell: bash
422-
423-
- name: Save Python virtual environment related to PR event
424-
if: >
425-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true'
426-
uses: ./.github/actions/python_requirements/save_virtualenv
427-
with:
428-
requirements_paths: "${{ inputs.requirements_path }} requirements-linters.txt requirements-dev.txt requirements-docs.txt"
429-
python_version: ${{ steps.setup_python.outputs.python-version }}
430-
431-
- name: Save pip cache related to PR event
432-
if: >
433-
steps.restore_python_virtual_environment_pr.outputs.cache-hit != 'true' &&
434-
steps.restore_pip_cache_pr.outputs.cache-hit != 'true'
435-
uses: ./.github/actions/python_requirements/save_pip_cache
436-
437328
- name: Run linters
438329
uses: ./.github/actions/python_linter
439330
if: >

.github/workflows/dependency_review.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,4 @@ jobs:
1313
- name: 'Checkout Repository'
1414
uses: actions/checkout@v3
1515
- name: 'Dependency Review'
16-
uses: actions/dependency-review-action@v3
16+
uses: actions/dependency-review-action@v4

.github/workflows/pull_request_automation.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,6 @@ jobs:
6262
use_ruff_formatter: true
6363
use_ruff_linter: true
6464

65-
requirements_path: requirements/project-requirements.txt
66-
project_dev_requirements_file: requirements/dev-requirements.txt
6765
packages_path: packages.txt
6866
django_settings_module: greedybear.settings
6967

api/serializers.py

Lines changed: 63 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@
55
from django.core.exceptions import FieldDoesNotExist
66
from rest_framework import serializers
77

8-
from greedybear.consts import REGEX_DOMAIN, REGEX_IP
9-
from greedybear.models import IOC, GeneralHoneypot
8+
from greedybear.consts import REGEX_DOMAIN
9+
from greedybear.models import IOC, GeneralHoneypot, Sensor, Tag
10+
from greedybear.utils import is_ip_address
1011

1112
logger = logging.getLogger(__name__)
1213

@@ -19,8 +20,22 @@ def to_representation(self, value):
1920
return value.name
2021

2122

23+
class TagSerializer(serializers.ModelSerializer):
24+
class Meta:
25+
model = Tag
26+
fields = ["key", "value", "source"]
27+
28+
29+
class SensorSerializer(serializers.ModelSerializer):
30+
class Meta:
31+
model = Sensor
32+
fields = ["address", "label"]
33+
34+
2235
class IOCSerializer(serializers.ModelSerializer):
2336
general_honeypot = GeneralHoneypotSerializer(many=True, read_only=True)
37+
tags = TagSerializer(many=True, read_only=True)
38+
sensors = SensorSerializer(many=True, read_only=True)
2439

2540
class Meta:
2641
model = IOC
@@ -36,22 +51,38 @@ class EnrichmentSerializer(serializers.Serializer):
3651

3752
def validate(self, data):
3853
"""
39-
Check a given observable against regex expression
54+
Validate that the query is a valid IP address (IPv4/IPv6) or domain.
4055
"""
41-
observable = data["query"]
42-
if re.match(r"^[\d\.]+$", observable) and not re.match(REGEX_IP, observable):
43-
raise serializers.ValidationError("Observable is not a valid IP")
44-
if not re.match(REGEX_IP, observable) and not re.match(REGEX_DOMAIN, observable):
45-
raise serializers.ValidationError("Observable is not a valid IP or domain")
56+
observable = data["query"].strip()
57+
data["query"] = observable
58+
59+
# A valid domain must match the domain regex AND contain at least one alphabetic character
60+
is_domain = bool(re.match(REGEX_DOMAIN, observable)) and any(c.isalpha() for c in observable)
61+
62+
if not is_ip_address(observable) and not is_domain:
63+
raise serializers.ValidationError("Observable is not a valid IP address or domain")
64+
4665
try:
47-
required_object = IOC.objects.get(name=observable)
66+
required_object = IOC.objects.prefetch_related("tags", "sensors").get(name=observable)
4867
data["found"] = True
4968
data["ioc"] = required_object
5069
except IOC.DoesNotExist:
5170
data["found"] = False
5271
return data
5372

5473

74+
def parse_feed_types(feed_type_str: str) -> list:
75+
"""Split a comma-separated feed type string into a stripped list of individual feed types.
76+
77+
Args:
78+
feed_type_str (str): Comma-separated feed type string (e.g. "cowrie,adbhoney").
79+
80+
Returns:
81+
list[str]: List of non-empty, stripped feed type tokens.
82+
"""
83+
return [ft.strip() for ft in feed_type_str.split(",") if ft.strip()]
84+
85+
5586
def feed_type_validation(feed_type: str, valid_feed_types: frozenset) -> str:
5687
"""Validates that a given feed type exists in the set of valid feed types.
5788
@@ -96,7 +127,7 @@ def ordering_validation(ordering: str) -> str:
96127

97128

98129
class FeedsRequestSerializer(serializers.Serializer):
99-
feed_type = serializers.CharField(max_length=120)
130+
feed_type = serializers.CharField()
100131
attack_type = serializers.ChoiceField(choices=["scanner", "payload_request", "all"])
101132
ioc_type = serializers.ChoiceField(choices=["ip", "domain", "all"])
102133
max_age = serializers.IntegerField(min_value=1)
@@ -107,11 +138,28 @@ class FeedsRequestSerializer(serializers.Serializer):
107138
ordering = serializers.CharField(max_length=120)
108139
verbose = serializers.ChoiceField(choices=["true", "false"])
109140
paginate = serializers.ChoiceField(choices=["true", "false"])
110-
format = serializers.ChoiceField(choices=["csv", "json", "txt"])
141+
format = serializers.ChoiceField(choices=["csv", "json", "txt", "stix21"])
142+
asn = serializers.IntegerField(min_value=1, required=False, allow_null=True)
143+
min_score = serializers.FloatField(min_value=0, max_value=1, required=False, allow_null=True)
144+
port = serializers.IntegerField(min_value=1, max_value=65535, required=False, allow_null=True)
145+
start_date = serializers.DateField(format="%Y-%m-%d", required=False, allow_null=True)
146+
end_date = serializers.DateField(format="%Y-%m-%d", required=False, allow_null=True)
147+
tag_key = serializers.CharField(max_length=128, required=False, allow_blank=True)
148+
tag_value = serializers.CharField(max_length=256, required=False, allow_blank=True)
111149

112150
def validate_feed_type(self, feed_type):
113151
logger.debug(f"FeedsRequestSerializer - validation feed_type: '{feed_type}'")
114-
return feed_type_validation(feed_type, self.context["valid_feed_types"])
152+
feed_types = parse_feed_types(feed_type)
153+
if not feed_types:
154+
raise serializers.ValidationError("Invalid feed_type: must not be empty")
155+
valid_feed_types = self.context["valid_feed_types"]
156+
if len(feed_types) > len(valid_feed_types):
157+
raise serializers.ValidationError(f"Invalid feed_type: too many types specified (max {len(valid_feed_types)})")
158+
if "all" in feed_types and len(feed_types) > 1:
159+
raise serializers.ValidationError("Invalid feed_type: 'all' cannot be combined with other feed types")
160+
for ft in feed_types:
161+
feed_type_validation(ft, valid_feed_types)
162+
return feed_type
115163

116164
def validate_ordering(self, ordering):
117165
logger.debug(f"FeedsRequestSerializer - validation ordering: '{ordering}'")
@@ -122,6 +170,7 @@ class ASNFeedsOrderingSerializer(FeedsRequestSerializer):
122170
ALLOWED_ORDERING_FIELDS = frozenset(
123171
{
124172
"asn",
173+
"as_name",
125174
"ioc_count",
126175
"total_attack_count",
127176
"total_interaction_count",
@@ -183,6 +232,8 @@ class FeedsResponseSerializer(serializers.Serializer):
183232
login_attempts = serializers.IntegerField(min_value=0)
184233
recurrence_probability = serializers.FloatField(min_value=0, max_value=1)
185234
expected_interactions = serializers.FloatField(min_value=0)
235+
attacker_country = serializers.CharField(allow_null=True, allow_blank=True, max_length=120)
236+
tags = TagSerializer(many=True, required=False, default=list)
186237

187238
def validate_feed_type(self, feed_type):
188239
logger.debug(f"FeedsResponseSerializer - validation feed_type: '{feed_type}'")

api/throttles.py

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# This file is a part of GreedyBear https://github.com/honeynet/GreedyBear
2+
# See the file 'LICENSE' for copying permission.
3+
from rest_framework.throttling import SimpleRateThrottle
4+
5+
6+
class FeedsThrottle(SimpleRateThrottle):
7+
"""Rate-limit for public (unauthenticated) feeds endpoints."""
8+
9+
scope = "feeds"
10+
11+
def get_cache_key(self, request, view):
12+
return self.cache_format % {
13+
"scope": self.scope,
14+
"ident": self.get_ident(request),
15+
}
16+
17+
18+
class FeedsAdvancedThrottle(SimpleRateThrottle):
19+
"""Rate-limit for authenticated feeds endpoints (advanced, asn)."""
20+
21+
scope = "feeds_advanced"
22+
23+
def get_cache_key(self, request, view):
24+
if request.user and request.user.is_authenticated:
25+
ident = request.user.pk
26+
else:
27+
ident = self.get_ident(request)
28+
29+
return self.cache_format % {
30+
"scope": self.scope,
31+
"ident": ident,
32+
}
33+
34+
35+
class SharedFeedRateThrottle(SimpleRateThrottle):
36+
"""
37+
Rate throttle for the public shared feed consume endpoint.
38+
39+
Limits unauthenticated access to prevent abuse.
40+
Rate is configurable via the ``FEEDS_SHARED_THROTTLE_RATE`` environment variable
41+
(key: ``feeds_shared`` in DEFAULT_THROTTLE_RATES). Default: 10/minute.
42+
"""
43+
44+
scope = "feeds_shared"
45+
46+
def get_cache_key(self, request, view):
47+
return self.cache_format % {"scope": self.scope, "ident": self.get_ident(request)}

0 commit comments

Comments
 (0)