Merge branch 'master' of github.com:codex-team/hawk.api.nodejs into feat/timestamp-index-repetition

Author: slaveeks

.env.sample

@@ -1,6 +1,9 @@
 # API server port
 PORT=4000
 
+# Metrics server port
+METRICS_PORT=9090
+
 # Hawk API database URL
 MONGO_HAWK_DB_URL=mongodb://mongodb:27017/hawk
 
@@ -17,6 +20,8 @@ JWT_SECRET_REFRESH_TOKEN=abacaba
 # JWT secret for user's access token
 JWT_SECRET_ACCESS_TOKEN=belarus
 
+# max document could for one read request to the database
+MAX_DB_READ_BATCH_SIZE=100000
 
 # JWT secret for signing tokens for processing billing requests
 JWT_SECRET_BILLING_CHECKSUM=checksum_secret

.github/workflows/build-and-push-docker-image.yml

@@ -3,7 +3,7 @@ name: Build and push docker image
 on:
   push:
     branches:
-      - '*'
+      - '**'
     tags:
       - 'v*'
 

.github/workflows/integration-tests.yml

@@ -10,5 +10,10 @@ jobs:
     steps:
       - uses: actions/checkout@v2
 
+      - name: Use Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version-file: '.nvmrc'
+
       - name: Run tests
         run: yarn test:integration

.github/workflows/tests.yml

@@ -9,6 +9,10 @@ jobs:
     runs-on: ubuntu-22.04
     steps:
       - uses: actions/checkout@v2
+      - name: Use Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version-file: '.nvmrc'
       - name: Install modules
         run: yarn
       - name: Run tests

.gitignore

@@ -7,3 +7,4 @@ uploads
 globalConfig.json
 coverage
 tls
+package-lock.json

LICENSE

@@ -0,0 +1,96 @@
+Business Source License 1.1
+
+Parameters
+
+Licensor:             CodeX (Hawk)
+Licensed Work:        hawk.api.nodejs (https://github.com/codex-team/hawk.api.nodejs)
+                      The Licensed Work is © 2025 CodeX
+Additional Use Grant: Self-hosted use for own/internal needs and research/evaluation is permitted.
+                      Using this code to provide a competing error-tracking SaaS or commercial
+                      hosted service without the Licensor’s prior permission is prohibited.
+
+Change Date:          2030-01-01
+
+Change License:       AGPL-3.0
+
+-------------------------------------------------------------------------------
+
+License text copyright © 2024 MariaDB plc, All Rights Reserved.
+“Business Source License” is a trademark of MariaDB plc.
+
+Terms
+
+The Licensor hereby grants you the right to copy, modify, create derivative
+works, redistribute, and make non-production use of the Licensed Work. The
+Licensor may make an Additional Use Grant, above, permitting limited
+production use.
+
+Effective on the Change Date, or the fourth anniversary of the first publicly
+available distribution of a specific version of the Licensed Work under this
+License, whichever comes first, the Licensor hereby grants you rights under
+the terms of the Change License, and the rights granted in the paragraph
+above terminate.
+
+If your use of the Licensed Work does not comply with the requirements
+currently in effect as described in this License, you must purchase a
+commercial license from the Licensor, its affiliated entities, or authorized
+resellers, or you must refrain from using the Licensed Work.
+
+All copies of the original and modified Licensed Work, and derivative works
+of the Licensed Work, are subject to this License. This License applies
+separately for each version of the Licensed Work and the Change Date may vary
+for each version of the Licensed Work released by Licensor.
+
+You must conspicuously display this License on each original or modified copy
+of the Licensed Work. If you receive the Licensed Work in original or
+modified form from a third party, the terms and conditions set forth in this
+License apply to your use of that work.
+
+Any use of the Licensed Work in violation of this License will automatically
+terminate your rights under this License for the current and all other
+versions of the Licensed Work.
+
+This License does not grant you any right in any trademark or logo of
+Licensor or its affiliates (provided that you may use a trademark or logo of
+Licensor as expressly required by this License). TO THE EXTENT PERMITTED BY
+APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON AN “AS IS” BASIS. LICENSOR
+HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, EXPRESS OR IMPLIED, INCLUDING
+(WITHOUT LIMITATION) WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE, NON-INFRINGEMENT, AND TITLE. MariaDB hereby grants you permission to
+use this License’s text to license your works, and to refer to it using the
+trademark “Business Source License”, as long as you comply with the Covenants
+of Licensor below.
+
+Covenants of Licensor
+
+In consideration of the right to use this License’s text and the “Business
+Source License” name and trademark, Licensor covenants to MariaDB, and to all
+other recipients of the licensed work to be provided by Licensor:
+
+1. To specify as the Change License the GPL Version 2.0 or any later version,
+   or a license that is compatible with GPL Version 2.0 or a later version,
+   where “compatible” means that software provided under the Change License can
+   be included in a program with software provided under GPL Version 2.0 or a
+   later version. Licensor may specify additional Change Licenses without
+   limitation.
+
+2. To either: (a) specify an additional grant of rights to use that does not
+   impose any additional restriction on the right granted in this License, as
+   the Additional Use Grant; or (b) insert the text “None”.
+
+3. To specify a Change Date not later than the fourth anniversary of the first
+   publicly available distribution of a specific version of the Licensed Work
+   under this License.
+
+4. Not to modify this License in any other way.
+
+Notice
+
+The Business Source License (this document, or the “License”) is not an Open
+Source license. However, the Licensed Work will eventually be made available
+under an Open Source License, as stated in this License.
+
+For more information on the use of the Business Source License for MariaDB
+products, please visit the MariaDB Business Source License FAQ.
+For more information on the use of the Business Source License generally,
+please visit the Adopting and Developing Business Source License FAQ.

README.md

@@ -1,5 +1,7 @@
 # Hawk API
 
+![License](https://img.shields.io/badge/license-BSL--1.1-orange)
+
 ## Start API
 For deployment (both in production and in development), you can use Docker.
 See Docker instructions [here](DOCKER.md).
@@ -23,9 +25,15 @@ To execute the request, enter it in the input field on the left and click on the
 On the right side you will see the result of the query.
 
 ## GraphQL Voyager
-You can view API Schema visualization in `/voyager` page in your browser. To see current production schema go to [here](https://api.beta.hawk.so/voyager)
+You can view API Schema visualization in `/voyager` page in your browser.
+To see current production schema go to [here](https://api.beta.hawk.so/voyager)
 
 ## Migrations
 
 Run `yarn migrations:up` command to apply migration revisions or
 `yarn migrations:down` to rollback the last revision.
+
+## License
+
+Source code is available under **Business Source License 1.1 (BSL 1.1)**.
+See [`LICENSE`](./LICENSE) for terms, including:

docker/Dockerfile.dev

@@ -1,13 +1,13 @@
-FROM node:14.17.0-alpine as builder
+FROM node:16-alpine as builder
 
 WORKDIR /usr/src/app
-RUN apk add --no-cache git gcc g++ python make musl-dev
+RUN apk add --no-cache git gcc g++ python3 make musl-dev
 
 COPY package.json yarn.lock ./
 
 RUN yarn install
 
-FROM node:14.17.0-alpine
+FROM node:16-alpine
 
 WORKDIR /usr/src/app
 

docs/METRICS.md

@@ -0,0 +1,205 @@
+# Prometheus Metrics
+
+This application exposes Prometheus-compatible metrics on a separate port from the main API server.
+
+## Configuration
+
+The metrics server runs on a separate port configured via the `METRICS_PORT` environment variable:
+
+```bash
+# Default: 9090
+METRICS_PORT=9090
+```
+
+Add this to your `.env` file. See `.env.sample` for reference.
+
+## Metrics Endpoint
+
+The metrics are served at:
+
+```
+http://localhost:9090/metrics
+```
+
+(Replace `9090` with your configured `METRICS_PORT` if different)
+
+## Available Metrics
+
+### Default Node.js Metrics
+
+The following default Node.js metrics are automatically collected:
+
+- **nodejs_version_info** - Node.js version information
+- **process_cpu_user_seconds_total** - Total user CPU time spent in seconds
+- **process_cpu_system_seconds_total** - Total system CPU time spent in seconds
+- **nodejs_heap_size_total_bytes** - Total heap size in bytes
+- **nodejs_heap_size_used_bytes** - Used heap size in bytes
+- **nodejs_external_memory_bytes** - External memory in bytes
+- **nodejs_heap_space_size_total_bytes** - Total heap space size in bytes
+- **nodejs_heap_space_size_used_bytes** - Used heap space size in bytes
+- **nodejs_eventloop_lag_seconds** - Event loop lag in seconds
+- **nodejs_eventloop_lag_min_seconds** - Minimum event loop lag
+- **nodejs_eventloop_lag_max_seconds** - Maximum event loop lag
+- **nodejs_eventloop_lag_mean_seconds** - Mean event loop lag
+- **nodejs_eventloop_lag_stddev_seconds** - Standard deviation of event loop lag
+- **nodejs_eventloop_lag_p50_seconds** - 50th percentile event loop lag
+- **nodejs_eventloop_lag_p90_seconds** - 90th percentile event loop lag
+- **nodejs_eventloop_lag_p99_seconds** - 99th percentile event loop lag
+
+### Custom HTTP Metrics
+
+#### http_request_duration_seconds (Histogram)
+
+Duration of HTTP requests in seconds, labeled by:
+- `method` - HTTP method (GET, POST, etc.)
+- `route` - Request route/path
+- `status_code` - HTTP status code
+
+Buckets: 0.01, 0.05, 0.1, 0.5, 1, 5, 10 seconds
+
+#### http_requests_total (Counter)
+
+Total number of HTTP requests, labeled by:
+- `method` - HTTP method (GET, POST, etc.)
+- `route` - Request route/path
+- `status_code` - HTTP status code
+
+### GraphQL Metrics
+
+#### hawk_gql_operation_duration_seconds (Histogram)
+
+Histogram of total GraphQL operation duration by operation name and type.
+
+Labels:
+- `operation_name` - Name of the GraphQL operation
+- `operation_type` - Type of operation (query, mutation, subscription)
+
+Buckets: 0.01, 0.05, 0.1, 0.5, 1, 5, 10 seconds
+
+**Purpose**: Identify slow API operations (P95/P99 latency).
+
+#### hawk_gql_operation_errors_total (Counter)
+
+Counter of failed GraphQL operations grouped by operation name and error class.
+
+Labels:
+- `operation_name` - Name of the GraphQL operation
+- `error_type` - Type/class of the error
+
+**Purpose**: Detect increased error rates and failing operations.
+
+#### hawk_gql_resolver_duration_seconds (Histogram)
+
+Histogram of resolver execution time per type, field, and operation.
+
+Labels:
+- `type_name` - GraphQL type name
+- `field_name` - Field name being resolved
+- `operation_name` - Name of the GraphQL operation
+
+Buckets: 0.01, 0.05, 0.1, 0.5, 1, 5 seconds
+
+**Purpose**: Find slow or CPU-intensive resolvers that degrade overall performance.
+
+### MongoDB Metrics
+
+#### hawk_mongo_command_duration_seconds (Histogram)
+
+Histogram of MongoDB command duration by command, collection family, and database.
+
+Labels:
+- `command` - MongoDB command name (find, insert, update, etc.)
+- `collection_family` - Collection family name (extracted from dynamic collection names to reduce cardinality)
+- `db` - Database name
+
+Buckets: 0.01, 0.05, 0.1, 0.5, 1, 5, 10 seconds
+
+**Purpose**: Detect slow queries and high-latency collections.
+
+**Note on Collection Families**: To reduce metric cardinality, dynamic collection names are grouped into families. For example:
+- `events:projectId` → `events`
+- `dailyEvents:projectId` → `dailyEvents`
+- `repetitions:projectId` → `repetitions`
+- `membership:userId` → `membership`
+- `team:workspaceId` → `team`
+
+This prevents metric explosion when dealing with thousands of projects, users, or workspaces, while still providing meaningful insights into collection performance patterns.
+
+#### hawk_mongo_command_errors_total (Counter)
+
+Counter of failed MongoDB commands grouped by command and error code.
+
+Labels:
+- `command` - MongoDB command name
+- `error_code` - MongoDB error code
+
+**Purpose**: Track transient or persistent database errors.
+
+## Testing
+
+### Manual Testing
+
+You can test the metrics endpoint using curl:
+
+```bash
+curl http://localhost:9090/metrics
+```
+
+Or run the provided test script:
+
+```bash
+./test-metrics.sh
+```
+
+### Integration Tests
+
+Integration tests for metrics are located in `test/integration/cases/metrics.test.ts`.
+
+Run them with:
+
+```bash
+npm run test:integration
+```
+
+## Implementation Details
+
+The metrics implementation uses the `prom-client` library and consists of:
+
+1. **Metrics Module** (`src/metrics/index.ts`):
+   - Initializes a Prometheus registry
+   - Configures default Node.js metrics collection
+   - Defines custom HTTP metrics (duration histogram and request counter)
+   - Registers GraphQL and MongoDB metrics
+   - Provides middleware for tracking HTTP requests
+   - Creates a separate Express app for serving metrics
+
+2. **GraphQL Metrics** (`src/metrics/graphql.ts`):
+   - Implements Apollo Server plugin for tracking GraphQL operations
+   - Tracks operation duration, errors, and resolver execution time
+   - Automatically captures operation name, type, and field information
+
+3. **MongoDB Metrics** (`src/metrics/mongodb.ts`):
+   - Implements MongoDB command monitoring
+   - Tracks command duration and errors
+   - Uses MongoDB's command monitoring events
+   - Extracts collection families from dynamic collection names to reduce cardinality
+
+4. **Integration** (`src/index.ts`, `src/mongo.ts`):
+   - Adds GraphQL metrics plugin to Apollo Server
+   - Adds metrics middleware to the main Express app
+   - Enables MongoDB command monitoring on database clients
+   - Starts metrics server on a separate port
+   - Keeps metrics server isolated from main API traffic
+
+## Prometheus Configuration
+
+To scrape these metrics with Prometheus, add the following to your `prometheus.yml`:
+
+```yaml
+scrape_configs:
+  - job_name: 'hawk-api'
+    static_configs:
+      - targets: ['localhost:9090']
+```
+
+Adjust the target host and port according to your deployment.