Skip to content

Commit 895dba6

Browse files
ajaskolskiajaskolski
committed
chore(tests): convert remote datastore remote tests to testcontainers approach using real service; add ci tests with real catalog service
1 parent d35d8de commit 895dba6

6 files changed

Lines changed: 481 additions & 4 deletions

File tree

‎.github/workflows/pull-request-main.yml‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ jobs:
4545
# disable the checkptr runtime check due a false positive in github.com/xssnick/tonutils-go
4646
# causing tests in ci to fail "fatal error: checkptr: pointer arithmetic result points to invalid allocation"
4747
# https://github.com/xssnick/tonutils-go/issues/310
48-
# Exclude provider packages which use Docker containers
49-
go-test-cmd: go test -race -gcflags=all=-d=checkptr=0 -coverprofile=coverage.txt $(go list ./... | grep -v '/provider')
48+
# Exclude provider packages which use Docker containers and remote catalog tests
49+
go-test-cmd: go test -race -gcflags=all=-d=checkptr=0 -coverprofile=coverage.txt $(go list ./... | grep -v '/provider' | grep -v '/remote')
5050
use-go-cache: true
5151
artifact-name: unit-tests
5252

@@ -71,11 +71,40 @@ jobs:
7171
use-go-cache: true
7272
artifact-name: provider-tests
7373

74+
ci-test-catalog-remote:
75+
name: Catalog Remote Tests
76+
runs-on: ubuntu-latest
77+
timeout-minutes: 15
78+
permissions:
79+
id-token: write
80+
contents: read
81+
actions: read
82+
env:
83+
# Temporary set the Catalog Service image for testing
84+
CATALOG_SERVICE_IMAGE: ${{ secrets.AWS_ACCOUNT_NUMBER_PROD }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/op-catalog-service:v0.0.1
85+
steps:
86+
- name: Pull Catalog Service ECR Image
87+
uses: smartcontractkit/.github/actions/pull-private-ecr-image@2f8f0baf38e46140c6a119eb551a56eaaabcc09e # pull-private-ecr-image@1.0.0
88+
with:
89+
aws-account-number: ${{ secrets.AWS_ACCOUNT_NUMBER_PROD }}
90+
aws-region: ${{ secrets.AWS_REGION }}
91+
aws-role-arn: ${{ secrets.ECR_READ_ROLE_ARN }}
92+
ecr-repository: "op-catalog-service"
93+
image-tag: "v0.0.1"
94+
95+
- name: Run Catalog Remote Integration Tests
96+
uses: smartcontractkit/.github/actions/ci-test-go@dfcba48f05933158428bce867d790e3d5a9baa6b # ci-test-go@1.1.0
97+
with:
98+
# Must cd into datastore/catalog/remote because TestMain only runs from package directory
99+
go-test-cmd: cd datastore/catalog/remote && go test -v -race -timeout 10m -gcflags=all=-d=checkptr=0 -coverprofile=../../../coverage.txt
100+
use-go-cache: true
101+
artifact-name: catalog-remote-tests
102+
74103
sonarqube:
75104
name: Sonar Scan
76105
if: github.event_name == 'pull_request'
77106
runs-on: ubuntu-24.04
78-
needs: [ci-test, ci-test-provider, ci-lint-misc, ci-lint]
107+
needs: [ci-test, ci-test-provider, ci-test-catalog-remote, ci-lint-misc, ci-lint]
79108
permissions:
80109
contents: read
81110
actions: read

‎datastore/catalog/remote/README.md‎

Lines changed: 84 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,87 @@ state to be maintained through that stream connection. If the stream closes,
66
normal cleanup will rollback the transaction. The [Catalog service APIs] include
77
message to begin, commit, and roll-back a transaction.
88

9-
[Catalog service APIs]: http://github.com/smartcontractkit/chainlink-catalog
9+
[Catalog service APIs]: http://github.com/smartcontractkit/op-catalog
10+
11+
## Running Tests
12+
13+
The tests in this package use **testcontainers** to automatically start the required services:
14+
- PostgreSQL database
15+
- Catalog service (from ECR or local image)
16+
- Database migration
17+
18+
**Important:** Testcontainers uses images from your local Docker daemon. You must either:
19+
1. Build the image locally: `docker build -t op-catalog-service:latest .`
20+
2. Pull from ECR with a specific tag: `docker pull 123123123123.dkr.ecr.us-east-1.amazonaws.com/chainlink-catalog-service:TAG`
21+
- Use `aws ecr describe-images` to find available tags
22+
- Common patterns: version tags (v0.0.1)
23+
24+
### Quick Start (Local Build)
25+
26+
```bash
27+
# Build the catalog service image first
28+
cd op-catalog
29+
docker build -t op-catalog-service:latest .
30+
31+
# Run the tests (must be in the remote directory)
32+
cd ../chainlink-deployments-framework/datastore/catalog/remote
33+
go test -v
34+
```
35+
36+
## Configuration Options
37+
38+
### Environment Variables
39+
40+
- `CATALOG_SERVICE_IMAGE`: The Docker image to use for the catalog service
41+
- Default: `op-catalog-service:latest`
42+
- CI: Set to the ECR image URL
43+
- Example: `export CATALOG_SERVICE_IMAGE="123456789.dkr.ecr.us-east-1.amazonaws.com/op-catalog-service:latest"`
44+
45+
- `CATALOG_GRPC_ADDRESS`: Connect to an existing catalog service instead of starting containers
46+
- Example: `export CATALOG_GRPC_ADDRESS="localhost:8080"`
47+
48+
## Running Without Testcontainers
49+
50+
If you prefer to manage the services manually:
51+
52+
### Option 1: Docker Compose
53+
54+
```bash
55+
cd op-catalog
56+
docker-compose up -d
57+
58+
# Wait for the service to be ready
59+
sleep 5
60+
61+
# Run tests pointing to the local service
62+
cd ../chainlink-deployments-framework/datastore/catalog/remote
63+
export CATALOG_GRPC_ADDRESS="localhost:8080"
64+
go test -v
65+
```
66+
67+
## CI/CD Integration
68+
69+
The CI workflow automatically:
70+
1. Downloads the latest Catalog service image from ECR
71+
2. Runs the tests with testcontainers using the downloaded image
72+
3. Generates coverage reports
73+
74+
See `.github/workflows/pull-request-main.yml` for the full configuration.
75+
76+
## Test Workflow
77+
78+
The test setup follows this flow:
79+
80+
1. **TestMain** (`main_test.go`) - Entry point that:
81+
- Checks for `CATALOG_GRPC_ADDRESS` environment variable
82+
- If not set, starts testcontainers automatically
83+
- Sets up PostgreSQL and Catalog service
84+
- Runs all tests
85+
- Cleans up containers
86+
87+
2. **TestContainerSetup** (`testcontainer_setup.go`) - Manages:
88+
- Docker network creation
89+
- PostgreSQL container startup
90+
- Database migration via catalog service
91+
- Catalog service container startup
92+
- Port mapping and connection details

‎datastore/catalog/remote/main_test.go‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
package remote
2+
3+
import (
4+
"context"
5+
"log"
6+
"os"
7+
"testing"
8+
"time"
9+
)
10+
11+
var (
12+
// globalTestSetup is the shared testcontainer setup for all remote tests
13+
globalTestSetup *TestContainerSetup
14+
// catalogGRPCAddress is the address of the catalog service for tests
15+
catalogGRPCAddress string
16+
)
17+
18+
// TestMain is the entry point for all tests in this package
19+
// It sets up testcontainers once for all tests and tears them down at the end
20+
func TestMain(m *testing.M) {
21+
// Check if we should use an existing catalog service instead of testcontainers
22+
existingAddr := os.Getenv("CATALOG_GRPC_ADDRESS")
23+
if existingAddr != "" {
24+
log.Printf("Using existing catalog service at: %s", existingAddr)
25+
catalogGRPCAddress = existingAddr
26+
// Run tests and exit
27+
os.Exit(m.Run())
28+
}
29+
30+
// Setup context with timeout for initialization
31+
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
32+
defer cancel()
33+
34+
log.Println("========================================")
35+
log.Println("Setting up testcontainers for catalog remote tests...")
36+
log.Println("========================================")
37+
38+
// Setup testcontainers
39+
setup, err := SetupCatalogTestContainers(ctx)
40+
if err != nil {
41+
log.Fatalf("Failed to setup testcontainers: %v", err)
42+
}
43+
globalTestSetup = setup
44+
catalogGRPCAddress = setup.GetCatalogGRPCAddress()
45+
46+
log.Println("========================================")
47+
log.Println("Testcontainers setup complete!")
48+
log.Printf("PostgreSQL DSN: %s", setup.PostgresDSN)
49+
log.Printf("Catalog gRPC Address: %s", catalogGRPCAddress)
50+
log.Println("========================================")
51+
52+
// Set the environment variable so tests can find the service
53+
os.Setenv("CATALOG_GRPC_ADDRESS", catalogGRPCAddress)
54+
55+
// Run all tests
56+
exitCode := m.Run()
57+
58+
// Cleanup
59+
log.Println("========================================")
60+
log.Println("Cleaning up testcontainers...")
61+
log.Println("========================================")
62+
63+
// Use a fresh context for cleanup to avoid timeout issues
64+
cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 2*time.Minute)
65+
defer cleanupCancel()
66+
67+
if err := globalTestSetup.Teardown(cleanupCtx); err != nil {
68+
log.Printf("Warning: Failed to teardown testcontainers: %v", err)
69+
}
70+
71+
log.Println("Cleanup complete!")
72+
os.Exit(exitCode)
73+
}

0 commit comments

Comments
 (0)