Skip to content

test: add Gherkin/testbed compliance tests to Java, Python, and Go #75

Description

@aepfli

Problem

Only Rust runs the official flagd Gherkin testbed specifications (testbed/gherkin/*.feature + testbed/flags/*.json). Java, Python, and Go all use ad-hoc inline flag configurations, meaning specification compliance is only verified in one language.

This creates a risk that host-language implementations silently deviate from the flagd spec, especially around edge cases covered by the testbed (zero-value flags, context enrichment, evaluator reuse via $ref, custom operators).

Shared Test Fixtures Available

testbed/gherkin/
├── evaluation.feature       # Basic flag types, zero values, defaults
├── targeting.feature        # $ref, fractional, sem_ver, starts/ends_with, edge cases
├── contextEnrichment.feature # $flagd.flagKey, $flagd.timestamp
└── metadata.feature         # Flag and flag-set metadata

testbed/flags/
├── testing-flags.json, zero-flags.json, custom-ops.json
├── evaluator-refs.json, edge-case-flags.json, metadata-flags.json

Proposal

Add test suites to each language that load the shared JSON fixtures and verify the same scenarios:

  • Java: JUnit tests loading testbed/flags/*.json via updateState() and asserting evaluation results match expected values from the feature files
  • Python: pytest tests doing the same via PyO3 FlagEvaluator
  • Go: Go tests doing the same via wazero FlagEvaluator

This would immediately close the majority of cross-language test gaps identified in the gap analysis.

Priority

HIGH — this is the single biggest test coverage gap across the project.

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: lowLow priority - backlogtestingTest coverage gaps and new test suites

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions