Schema registry: Adding required field on JSON schema is allowed on BACKWARD compatibility mode

[theo303]

Redpanda version: v26.1.6

I'm using the docker-compose file from this page: https://docs.redpanda.com/redpanda-labs/docker-compose/single-broker/#run-the-lab
I can reproduce this behavior using the web-ui or directly with the schema registry API.

I think there's an issue with the JSON schema validation, adding a required field is allowed when the compatibility is set on BACKWARD.

What happened:

For example if I add this schema to a subject:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        }
    },
    "required": [
        "name"
    ],
    "additionalProperties": false
}

I'm allowed to add this one afterwards:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "id": {
            "type": "integer"
        }
    },
    "required": [
        "name",
        "id"
    ],
    "additionalProperties": false
}

What should have happened instead?

The confluent doc indicates that this should not be possible and the official schema registry is refusing the second version of my schema. I can only add it if I remove the id field from the required fields. I tested with confluentinc/cp-schema-registry:7.7.8

How to reproduce the issue?

1. Start redpanda and create a topic (I named it test-json), by default the global configuration of the schema registry is BACKWARD.
2. Upload the first version of the schema

curl -X POST http://localhost:18081/subjects/test-json-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schemaType": "JSON",
    "schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"}},\"required\":[\"name\"],\"additionalProperties\":false}"
  }'

3. Test the compatibility of the new schema

curl -X POST http://localhost:18081/compatibility/subjects/test-json-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schemaType": "JSON",
    "schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"id\":{\"type\":\"integer\"}},\"required\":[\"name\",\"id\"],\"additionalProperties\":false}"
  }'

response is:

{"is_compatible":true}

With the same curl on the confluent schema registry, I get false

Additional information

I've tried with BACKWARD and BACKWARD_TRANSITIVE. I've also tried to set the compatibility on the subject directly and not keeping DEFAULT which fallback on the global configuration.
I've tried with avro and the behavior is not the same, adding a non-nullable field is not allowed.

[dotnwat]

@pgellert

[pgellert]

Hi @theo303, thanks for the bug report. I've added this to our backlog, but would welcome a contribution if you'd like to take a shot at fixing the bug.