DRAFT - keycloak upgrade test

keycloak-upgrade/README.md

@@ -0,0 +1,45 @@
+# Keycloak Upgrade Integration Tests
+
+This test verifies that data created with the previous Keycloak image still works after installing a newer `folio-keycloak` image.
+
+The test is split into two runners because the Keycloak container must be replaced between phases while preserving the Keycloak database.
+
+## Run
+
+When no state file exists, the seed phase generates a random tenant name and id, then writes them to `target/keycloak-upgrade-tenant.properties`. The verify phase reads the same file.
+
+If the seed phase fails after tenant creation, remove `target/keycloak-upgrade-tenant.properties` to generate a new tenant, or clean up the failed tenant before rerunning the seed phase.
+
+Start the environment with the old Keycloak image, then seed upgrade data:
+
+```shell
+mvn clean test -pl common,testrail-integration,keycloak-upgrade \
+  -DfailIfNoTests=false \
+  -Dsurefire.failIfNoSpecifiedTests=false \
+  -Dtest=KeycloakUpgradeSeedTests \
+  -DargLine="-Dkarate.env=eureka2"
+```
+
+Restart Keycloak with the upgraded image against the same database, wait for Keycloak and Kong to become ready, then verify:
+
+```shell
+mvn test -pl common,testrail-integration,keycloak-upgrade \
+  -DfailIfNoTests=false \
+  -Dsurefire.failIfNoSpecifiedTests=false \
+  -Dtest=KeycloakUpgradeVerifyTests \
+  -DargLine="-Dkarate.env=eureka2"
+```
+
+To use an explicit state file location, add `-Dkeycloak.upgrade.stateFile=/path/to/keycloak-upgrade-tenant.properties` to both `argLine` values.
+
+To use an explicit tenant instead of the generated one, add both `-DtestTenant=<tenant>` and `-DtestTenantId=<uuid>` to the seed command. The seed phase still writes them to the state file for the verify command.
+
+## What It Covers
+
+- Tenant authentication state survives the Keycloak restart.
+- Existing FOLIO user and credentials still work.
+- Login, refresh, user lookup, and effective permission resolution still work through FOLIO APIs.
+- FOLIO-issued tokens still contain the expected `sub`, `user_id`, and client claims.
+- Tenant metadata can still be updated through the FOLIO tenant manager API.
+- Role/capability assignments created before the upgrade still resolve after the upgrade.
+- The verify phase cleans up the generated tenant after a successful run.

keycloak-upgrade/pom.xml

@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+    <parent>
+        <artifactId>folio-integration-tests</artifactId>
+        <groupId>org.folio</groupId>
+        <version>1.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>keycloak-upgrade</artifactId>
+    <version>1.0-SNAPSHOT</version>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.folio</groupId>
+            <artifactId>common</artifactId>
+            <version>1.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.folio</groupId>
+            <artifactId>testrail-integration</artifactId>
+            <version>1.0-SNAPSHOT</version>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-surefire-plugin</artifactId>
+                <configuration>
+                    <testFailureIgnore>false</testFailureIgnore>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+</project>

keycloak-upgrade/src/main/resources/eureka/keycloak-upgrade/features/destroy-upgrade-data.feature

@@ -0,0 +1,8 @@
+Feature: destroy Keycloak upgrade test data
+
+  Background:
+    * url baseUrl
+    * configure readTimeout = 3000000
+
+  Scenario: delete upgrade test tenant
+    * call read('classpath:common/eureka/destroy-data.feature') { testTenantId: '#(testTenantId)' }

keycloak-upgrade/src/main/resources/eureka/keycloak-upgrade/features/seed-before-upgrade.feature

@@ -0,0 +1,124 @@
+Feature: seed data before Keycloak upgrade
+
+  Background:
+    * url baseUrl
+    * configure cookies = null
+    * configure retry = { count: 20, interval: 5000 }
+
+    * table modules
+      | name                  |
+      | 'mod-users-keycloak'  |
+      | 'mod-login-keycloak'  |
+      | 'mod-roles-keycloak'  |
+
+    * table userPermissions
+      | name                                          |
+      | 'login.item.post'                             |
+      | 'auth.token.post'                             |
+      | 'auth.refreshtoken.post'                      |
+      | 'auth.token.sign.post'                        |
+      | 'auth.token.refresh.post'                     |
+      | 'users.item.get'                              |
+      | 'users-keycloak.item.get'                     |
+      | 'roles.item.get'                              |
+      | 'roles.item.delete'                           |
+      | 'roles.item.post'                             |
+      | 'roles.collection.get'                        |
+      | 'roles.users.item.get'                        |
+      | 'roles.users.item.post'                       |
+      | 'roles.users.collection.get'                  |
+      | 'capabilities.item.get'                       |
+      | 'capabilities.collection.get'                 |
+      | 'role-capabilities.collection.post'           |
+      | 'role-capabilities.collection.get'            |
+      | 'permissions.users.item.get'                  |
+
+  Scenario: create tenant, user credentials, and role data before upgrade
+    # Create the tenant and enable required applications before requesting tenant-scoped Keycloak tokens.
+    * call read('classpath:common/eureka/setup-users.feature@createTenant')
+    * call read('classpath:common/eureka/setup-users.feature@createEntitlement')
+
+    # Wait until the tenant realm is visible before retrieving the tenant-scoped module token.
+    * def keycloakResponse = call read('classpath:common/eureka/keycloak.feature@getKeycloakMasterToken')
+    * def keycloakMasterToken = keycloakResponse.response.access_token
+    Given url baseKeycloakUrl
+    And path 'admin', 'realms', testTenant
+    And header Authorization = 'Bearer ' + keycloakMasterToken
+    And retry until responseStatus == 200
+    When method get
+    Then status 200
+
+    # Create the test user, assign credentials, and grant the permissions needed for verification.
+    * configure headers = null
+    * configure cookies = null
+    * call read('classpath:common/eureka/setup-users.feature@getAuthorizationToken')
+    * call read('classpath:common/eureka/setup-users.feature@createTestUser')
+    * call read('classpath:common/eureka/setup-users.feature@specifyUserCredentials')
+    * call read('classpath:common/eureka/setup-users.feature@addUserCapabilities')
+    * def testUserId = karate.get('userId')
+
+    # Verify the user can authenticate before the Keycloak upgrade.
+    Given url baseUrl
+    * configure cookies = null
+    Given path 'authn', 'login'
+    And headers { 'Content-Type': 'application/json', 'x-okapi-tenant': '#(testTenant)' }
+    And request { username: '#(testUser.name)', password: '#(testUser.password)' }
+    When method post
+    Then status 201
+    And match response.okapiToken == '#string'
+    And match response.refreshToken == '#string'
+    And match responseCookies contains { folioAccessToken: '#present', folioRefreshToken: '#present' }
+    * def userToken = response.okapiToken
+    * def jwt = decodeJwtPayload(userToken)
+    And match jwt.sub == testUser.name
+    And match jwt.user_id == testUserId
+
+    # Resolve a known capability that will be attached to a role before the upgrade.
+    * configure headers = { 'Content-Type': 'application/json', 'x-okapi-token': '#(userToken)', 'Accept': '*/*', 'x-okapi-tenant': '#(testTenant)' }
+    Given path 'capabilities'
+    And param query = 'permission=="' + upgradeRolePermission + '"'
+    When method get
+    Then status 200
+    And match response.capabilities == '#[1]'
+    * def upgradeCapability = response.capabilities[0]
+
+    # Create a durable role fixture that should survive the Keycloak upgrade.
+    Given path 'roles'
+    And request { name: '#(upgradeRoleName)', description: '#(upgradeRoleDescription)', type: 'REGULAR' }
+    When method post
+    Then status 201
+    And match response.name == upgradeRoleName
+    * def upgradeRoleId = response.id
+
+    # Attach the capability to the role so post-upgrade access resolution can be verified.
+    Given path 'roles', 'capabilities'
+    And request { roleId: '#(upgradeRoleId)', capabilityIds: ['#(upgradeCapability.id)'] }
+    When method post
+    Then status 201
+    And match response.roleCapabilities[*].capabilityId contains upgradeCapability.id
+
+    # Assign the role to the test user before the upgrade.
+    Given path 'roles', 'users'
+    And request { userId: '#(testUserId)', roleIds: ['#(upgradeRoleId)'] }
+    When method post
+    Then status 201
+    And match response.userRoles[*].roleId contains upgradeRoleId
+
+    # Confirm effective permissions include the role capability before the upgrade.
+    Given path 'permissions', 'users', testUserId
+    When method get
+    Then status 200
+    And match response.permissions contains upgradeRolePermission
+
+    # Verify module-to-module authentication can read the seeded user before the upgrade.
+    * configure headers = null
+    * configure cookies = null
+    * call read('classpath:common/eureka/setup-users.feature@getAuthorizationToken')
+    * def m2mToken = karate.get('accessToken')
+    Given url baseUrl
+    And path 'users-keycloak', 'users', testUserId
+    And headers { 'x-okapi-tenant': '#(testTenant)', 'x-okapi-token': '#(m2mToken)' }
+    When method get
+    Then status 200
+    And match response.id == testUserId
+    And match response.username == testUser.name

keycloak-upgrade/src/main/resources/eureka/keycloak-upgrade/features/verify-after-upgrade.feature

@@ -0,0 +1,118 @@
+Feature: verify data after Keycloak upgrade
+
+  Background:
+    * url baseUrl
+    * configure cookies = null
+    * configure retry = { count: 20, interval: 5000 }
+
+  Scenario: verify pre-upgrade tenant data through FOLIO APIs after restart
+    # Verify the pre-upgrade user can still authenticate through the FOLIO login API.
+    * configure cookies = null
+    Given path 'authn', 'login'
+    And headers { 'Content-Type': 'application/json', 'x-okapi-tenant': '#(testTenant)' }
+    And request { username: '#(testUser.name)', password: '#(testUser.password)' }
+    When method post
+    Then status 201
+    And match response contains { okapiToken: '#string', refreshToken: '#string' }
+    And match responseCookies contains { folioAccessToken: '#present', folioRefreshToken: '#present' }
+    * def loginToken = response.okapiToken
+    * def refreshToken = responseCookies.folioRefreshToken.value
+    * def jwt = decodeJwtPayload(loginToken)
+    And match jwt.sub == testUser.name
+    And match jwt.user_id == '#uuid'
+    And match jwt.azp == '#string'
+    * def testUserId = jwt.user_id
+
+    # Verify cross-module access resolution through sidecars:
+    # users-keycloak returns user data and effective permissions resolved from roles-keycloak.
+    * configure headers = { 'Content-Type': 'application/json', 'x-okapi-token': '#(loginToken)', 'Accept': '*/*', 'x-okapi-tenant': '#(testTenant)' }
+    Given path 'users-keycloak', '_self'
+    When method get
+    Then status 200
+    And match response.user.id == testUserId
+    And match response.user.username == testUser.name
+    And match response.permissions.permissions contains upgradeRolePermission
+
+    # Verify the pre-upgrade user can still be read by id.
+    Given path 'users-keycloak', 'users', testUserId
+    When method get
+    Then status 200
+    And match response.id == testUserId
+    And match response.username == testUser.name
+
+    # Verify the pre-upgrade role fixture still exists.
+    Given path 'roles'
+    And param query = 'name=="' + upgradeRoleName + '"'
+    When method get
+    Then status 200
+    And match response.roles == '#[1]'
+    * def upgradeRoleId = response.roles[0].id
+
+    # Verify the role still has the capability assigned before the upgrade.
+    Given path 'roles', upgradeRoleId, 'capabilities'
+    When method get
+    Then status 200
+    And match response.capabilities[*].permission contains upgradeRolePermission
+
+    # Verify effective permissions still include access contributed by the pre-upgrade role.
+    Given path 'permissions', 'users', testUserId
+    When method get
+    Then status 200
+    And match response.permissions contains upgradeRolePermission
+
+    # Verify refresh-token rotation still works after the upgrade.
+    * configure headers = null
+    * configure cookies = null
+    Given url baseUrl
+    And path 'authn', 'refresh'
+    And header x-okapi-tenant = testTenant
+    And cookie folioRefreshToken = refreshToken
+    When method post
+    Then status 201
+    And match responseCookies.folioAccessToken == '#present'
+    And match responseCookies.folioRefreshToken == '#present'
+    * def refreshedToken = responseCookies.folioAccessToken.value
+    * def refreshedJwt = decodeJwtPayload(refreshedToken)
+    And match refreshedJwt.sub == testUser.name
+    And match refreshedJwt.user_id == testUserId
+
+    # Verify the refreshed access token can be used against FOLIO APIs.
+    * configure headers = { 'Content-Type': 'application/json', 'x-okapi-token': '#(refreshedToken)', 'Accept': '*/*', 'x-okapi-tenant': '#(testTenant)' }
+    Given path 'users-keycloak', 'users', testUserId
+    When method get
+    Then status 200
+    And match response.username == testUser.name
+
+    # Verify tenant metadata can still be updated through the FOLIO tenant manager API.
+    * configure headers = null
+    * configure cookies = null
+    # Use the manager token only to authorize the FOLIO tenant manager request.
+    * def managerTokenResponse = call read('classpath:common/eureka/keycloak.feature@getKeycloakMasterToken')
+    * def managerToken = managerTokenResponse.response.access_token
+
+    # Read the existing tenant before updating it.
+    Given path 'tenants', testTenantId
+    And header Authorization = 'Bearer ' + managerToken
+    When method get
+    Then status 200
+    And match response.id == testTenantId
+    And match response.name == testTenant
+    * def tenant = response
+    * def updatedDescription = 'Updated by Keycloak upgrade IT ' + uuid()
+    * set tenant.description = updatedDescription
+
+    # Update only the tenant description and verify the update response.
+    Given path 'tenants', testTenantId
+    And header Authorization = 'Bearer ' + managerToken
+    And header Content-Type = 'application/json'
+    And request tenant
+    When method put
+    Then status 200
+    And match response.description == updatedDescription
+
+    # Read the tenant again to confirm the description was persisted.
+    Given path 'tenants', testTenantId
+    And header Authorization = 'Bearer ' + managerToken
+    When method get
+    Then status 200
+    And match response.description == updatedDescription