Feature: Foreign key check with null safe support

[IvannKurchenko]

Changes

Added null_safe parameter to foreign_key check to explicitly enable null foreign key comparison. To distinguish cases when null foreign key is matched vs foreign key is not found completely when null_safe=True wrapping even single columns comparison into Struct.

Special behaviour

Enabling null_safe=True on a previously non-null-safe single-column FK changes the message and auto created check name:

• the rule name from a_not_exists_in_ref_b → struct_a_as_a_not_exists_in_ref_struct_b_as_a
• the violation message from Value '...' in column 'a' not found in reference column 'b' → 'struct(a AS a)' ... 'struct(b AS a)'

Linked issues

#1069

Resolves #1069

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[mwojtyczka]

@IvannKurchenko thank you for the PR. We require all commits to be signed. Please follow the instructions: https://databrickslabs.github.io/dqx/docs/dev/contributing/#first-contribution

[mwojtyczka]

Thanks for the contribution — the idea of wrapping single columns in a struct to preserve NULL-aware base_condition is a sound approach. However, the join condition inside the apply closure is not updated for null-safe matching, which means the null-safe tests should fail. See inline comments for details.

Summary:

Severity	Item
Blocker	Join condition uses == which cannot match NULL struct fields — null-safe matching won't work
High	Missing validation for negate + null_safe despite being documented as raising an error
Low	Docstring grammar

[mwojtyczka]

@IvannKurchenko you need to repush all commits, the first commit is still not signed

[IvannKurchenko]

@mwojtyczka Thank you for the review and feedback. I have added the tests you requested, along with a case for negate enabled.

[mwojtyczka]

LGTM

[mwojtyczka]

@mwojtyczka I apologise for this, there was also merge commits from GitHub, now everything is singed.

No worries, thanks for your awesome contribution! will merge as soon as all tests pass