Add certificate renewal infrastructure for Collectors by bernd · Pull Request #25537 · Graylog2/graylog2-server

bernd · 2026-04-02T05:21:56Z

Warning

We added SKI/AKI values to the certificates and the database. Testing this PR requires a full reset of the Collector CA! We can't write a migration for this.

Add dynamic TLS certificate management so the Collector mTLS endpoint supports certificate renewal without server restarts. Custom X509KeyManager and X509TrustManager implementations fetch certificates from a Caffeine cache that is invalidated via cluster events when the CA config changes.
Use X.509 SKI/AKI extensions for trust chain resolution, allowing multiple signing certs to coexist during renewal (old collector certs remain trusted via AKI to SKI lookup in the database).
Implement CollectorCaService#renewCertificates which checks signing and OTLP server cert lifetimes and renews them when less than 20% of their lifetime remains, with cascading re-issue of the server cert when the signing cert is renewed.
Cap collector cert lifetime in CertificateBuilder#signCsr to the issuer's remaining validity so a signed certificate never outlives its signing cert.
Add CollectorCaTrustManager checks for client certificate EKU (clientAuth) and basic constraints (rejects CA certs used as client identities).
Add MongoDB index on subject_key_identifier for efficient SKI-based lookups.

Fixes #25231
/nocl extending unreleased feature

…ewal

Also add optional authority_key_identifier field.

CertificateBuilder#signCsr now ensures a signed certificate never outlives its issuer. When the requested lifetime exceeds the signing cert's remaining validity, it is automatically capped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

CollectorCaKeyManager now extends X509ExtendedKeyManager instead of implementing X509KeyManager. Netty uses SSLEngine for TLS handshakes, and the JDK wraps plain X509KeyManager in an adapter that adds endpoint identification checks. X509ExtendedKeyManager is used directly, bypassing the wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ewal

Since we lookup the issuer cert by SKI, we need to ensure we don't accept any issuer cert.

…ewal

Copilot

Pull request overview

Introduces certificate renewal support for the Collector mTLS endpoint by adding dynamic TLS certificate loading (key/trust managers backed by a cache) plus X.509 SKI/AKI plumbing to support overlapping signing certs during renewal.

Changes:

Add SKI/AKI persistence + lookup (including Mongo index + service lookup method) and cap signed cert lifetimes to issuer remaining validity.
Add Collector CA renewal logic (leader-only periodical) and dynamic Netty TLS wiring via cache-backed X509ExtendedKeyManager/X509ExtendedTrustManager.
Add extensive unit/integration tests covering renewal behavior, key/trust manager behavior, and SKI-based issuer resolution.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
graylog2-server/src/main/java/org/graylog/security/pki/CertificateBuilder.java	Adds serial RNG, issuer-lifetime capping, and SKI/AKI handling for CSR-signed certs.
graylog2-server/src/main/java/org/graylog/security/pki/CertificateEntry.java	Adds `authority_key_identifier` field to persisted certificate records.
graylog2-server/src/main/java/org/graylog/security/pki/CertificateService.java	Adds SKI index + SKI lookup method; preserves AKI in DN enrichment.
graylog2-server/src/main/java/org/graylog/collectors/CollectorCaService.java	Implements renewal logic + clock injection; refactors cert creation helpers.
graylog2-server/src/main/java/org/graylog/collectors/CollectorCaCache.java	Introduces Caffeine-backed cache for CA/signing/server certs + SKI lookups and event invalidation.
graylog2-server/src/main/java/org/graylog/collectors/CollectorCaKeyManager.java	Provides dynamic server key/cert selection for TLS without restarts.
graylog2-server/src/main/java/org/graylog/collectors/CollectorCaTrustManager.java	Adds SKI/AKI-based issuer resolution + EKU/basic-constraints enforcement for client certs.
graylog2-server/src/main/java/org/graylog/collectors/CollectorTLSUtils.java	Centralizes OTLP server `SslContextBuilder` creation using dynamic key/trust managers.
graylog2-server/src/main/java/org/graylog/collectors/CollectorsConfigService.java	Posts a cluster event when cert IDs change to trigger cache invalidation.
graylog2-server/src/main/java/org/graylog/collectors/events/CollectorCaConfigUpdated.java	New cluster event type for CA config changes.
graylog2-server/src/main/java/org/graylog/collectors/CollectorsModule.java	Wires new cache/key/trust/TLS utils singletons and initializer.
graylog2-server/src/main/java/org/graylog/collectors/input/transport/CollectorIngestHttpTransport.java	Switches OTLP server TLS setup to `CollectorTLSUtils`.
graylog2-server/src/main/java/org/graylog/collectors/periodical/CollectorCaRenewalPeriodical.java	Makes renewal leader-only, more frequent, and exception-safe.
graylog2-server/src/main/java/org/graylog/collectors/CollectorsConfig.java	Adds `toBuilder()` to support updating cert IDs during renewal.
graylog2-server/src/test/java/org/graylog/security/pki/CertificateBuilderTest.java	Adds tests for lifetime capping + CSR signing extensions/behavior.
graylog2-server/src/test/java/org/graylog/security/pki/CertificateEntryTest.java	Updates record/JSON/Mongo roundtrip tests for new AKI field.
graylog2-server/src/test/java/org/graylog/security/pki/CertificateServiceTest.java	Adds SKI lookup tests and updates fixtures for AKI.
graylog2-server/src/test/java/org/graylog/collectors/CollectorCaServiceTest.java	Adds renewal-threshold and renewal-cascade tests; updates TLS builder test.
graylog2-server/src/test/java/org/graylog/collectors/CollectorCaCacheTest.java	New tests for cache behavior, invalidation, and SKI lookups.
graylog2-server/src/test/java/org/graylog/collectors/CollectorCaKeyManagerTest.java	New tests validating key manager alias/chain/key behavior.
graylog2-server/src/test/java/org/graylog/collectors/CollectorCaTrustManagerTest.java	New tests for issuer resolution, EKU/basic-constraints, and validation failures.
graylog2-server/src/test/java/org/graylog/collectors/CollectorTLSUtilsIT.java	New Netty-based integration test for end-to-end mTLS handshake with dynamic managers.
graylog2-server/src/test/java/org/graylog/collectors/CollectorsConfigServiceTest.java	New tests asserting config changes emit CA cache invalidation events.
graylog2-server/src/test/java/org/graylog/collectors/opamp/auth/AgentTokenServiceTest.java	Updates construction for new `CollectorsConfigService`/`CollectorCaService` signatures.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

graylog2-server/src/main/java/org/graylog/collectors/CollectorCaCache.java

graylog2-server/src/test/java/org/graylog/collectors/CollectorCaCacheTest.java

graylog2-server/src/test/java/org/graylog/collectors/CollectorTLSUtilsIT.java

graylog2-server/src/main/java/org/graylog/security/pki/CertificateEntry.java

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

graylog2-server/src/main/java/org/graylog/collectors/CollectorCaCache.java

graylog2-server/src/main/java/org/graylog/security/pki/CertificateBuilder.java

…numbers

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

graylog2-server/src/main/java/org/graylog/collectors/CollectorCaKeyManager.java

The JDK can report Ed25519 keys as either "EdDSA" or "Ed25519" depending on the provider. chooseServerAlias now accepts both names, consistent with CertificateBuilder and PemUtils.

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

graylog2-server/src/test/java/org/graylog/collectors/CollectorTLSUtilsIT.java

…ewal

kroepke

Overall 👍, one minor nitpick and one question about how we want to deal with failures during renewal, but I won't block merge for that.

...og2-server/src/main/java/org/graylog/collectors/periodical/CollectorCaRenewalPeriodical.java

graylog2-server/src/main/java/org/graylog/collectors/CollectorCaService.java

bernd force-pushed the collectors/cert-renewal branch 2 times, most recently from 78a4679 to 31fbe77 Compare April 7, 2026 09:19

Add SKI and AKI to the Collector certificates

5741f26

bernd force-pushed the collectors/cert-renewal branch from 31fbe77 to 5741f26 Compare April 7, 2026 09:50

bernd and others added 11 commits April 7, 2026 14:35

Add CollectorCaCache class

ce01cdc

Add CollectorCaCache#getBySubjectKeyIdentifier method

ce02ab9

Add custom Key and Trust manager implementations

16a9d58

Add CollectorTLSUtils

e4277be

Use CollectorTLSUtils in CollectorIngestHttpTransport

cb18a4f

Merge remote-tracking branch 'origin/master' into collectors/cert-ren…

e5cf5a4

…ewal

Make subject_key_identifier field required

47e597e

Also add optional authority_key_identifier field.

Address review comments

65a3601

Implement CollectorCaService#renewCertificates

2d22f7c

Remove redundant javadoc

0802960

bernd changed the title ~~Start Collector key manager~~ Add certificate renewal infrastructure for Collectors Apr 7, 2026

bernd and others added 14 commits April 7, 2026 18:53

Only run the renewal periodical on leader nodes

1dc9e0f

Merge branch 'master' into collectors/cert-renewal

148ab93

Merge remote-tracking branch 'origin/master' into collectors/cert-ren…

a1e7939

…ewal

Validate the issuer certificate in CollectorCaTrustManager

93a6090

Since we lookup the issuer cert by SKI, we need to ensure we don't accept any issuer cert.

Load config as late as possible to minimize concurrent write issues

7d5e8c5

Cap lifetimes for all certs but root CA certs

0c26f90

Fix CollectorTLSUtilsIT

352edd7

Merge remote-tracking branch 'origin/master' into collectors/cert-ren…

9d29748

…ewal

Add more tests to CertificateBuilderTest

beee72c

Use _static_ prefix for cache keys

ce5e37d

Bind CollectorTLSUtils as singleton

6db494a

Use SecureRandom to generate unique cert serial numbers

3a15cc1

Fix test names in CollectorCaCacheTest

194b751

bernd added 3 commits April 8, 2026 14:56

Add tests for CollectorsConfigService

0258627

Handle errors in CollectorCaRenewalPeriodical

a1a520d

Extend X509ExtendedTrustManager in CollectorCaTrustManager

8faddb4

bernd requested a review from Copilot April 8, 2026 13:21

Copilot started reviewing on behalf of bernd April 8, 2026 13:22 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Address review comments

283f81d

bernd requested a review from Copilot April 8, 2026 15:33

Copilot started reviewing on behalf of bernd April 8, 2026 15:34 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Accept EdDSA algorithm name in signCsr and guarantee positive serial …

a82cdb8

…numbers

bernd requested a review from Copilot April 8, 2026 15:53

Copilot started reviewing on behalf of bernd April 8, 2026 15:54 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

graylog2-server/src/main/java/org/graylog/collectors/CollectorCaKeyManager.java Show resolved Hide resolved

Accept both EdDSA and Ed25519 key types in CollectorCaKeyManager

af8bbc0

The JDK can report Ed25519 keys as either "EdDSA" or "Ed25519" depending on the provider. chooseServerAlias now accepts both names, consistent with CertificateBuilder and PemUtils.

bernd requested a review from Copilot April 8, 2026 16:22

Copilot started reviewing on behalf of bernd April 8, 2026 16:23 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

graylog2-server/src/test/java/org/graylog/collectors/CollectorTLSUtilsIT.java Outdated Show resolved Hide resolved

Remove unused import

3e45935

bernd marked this pull request as ready for review April 8, 2026 16:31

bernd requested a review from a team April 8, 2026 16:32

Merge remote-tracking branch 'origin/master' into collectors/cert-ren…

1fc04cc

…ewal

kroepke approved these changes Apr 10, 2026

View reviewed changes

bernd merged commit 924d742 into master Apr 10, 2026
23 checks passed

bernd deleted the collectors/cert-renewal branch April 10, 2026 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add certificate renewal infrastructure for Collectors#25537

Add certificate renewal infrastructure for Collectors#25537
bernd merged 34 commits intomasterfrom
collectors/cert-renewal

bernd commented Apr 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

kroepke left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bernd commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

kroepke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bernd commented Apr 2, 2026 •

edited

Loading