[Feature] [e2e-infra] Support duplicate data verification (#12253)

[Navadeep0007]

Summary

This PR adds support for verifying duplicate data in list-type checks within the e2e infrastructure framework.

Changes

• Added internal/verify/duplicate_check.go — implements CheckForDuplicates() utility to detect duplicate items in a list.
• Added internal/verify/duplicate_check_test.go — unit tests to validate duplicate detection logic.

Behavior

The CheckForDuplicates() function returns an error when duplicate entries are found in a given list, helping ensure data integrity during verification.

Related Issue

Fixes apache/skywalking#12253

Checklist

• Implemented duplicate data verification utility
• Added corresponding unit tests
• Passed all Go tests locally

Notes

Future improvements may integrate this utility into the existing list verification logic for automated duplicate checking during e2e validation.

[Navadeep0007]

Hi @wu-sheng ,
I’ve implemented the duplicate data verification feature as described in issue #12253.
The PR adds a CheckForDuplicates() utility along with corresponding unit tests.

It would be great if you could please review it when you have a moment.
I’d also be happy to extend this in a follow-up to integrate with the existing list verification flow if needed.

This PR is ready for review and merge.
It adds the duplicate data verification utility (CheckForDuplicates) with unit tests as discussed in issue #12253.

All tests have passed locally, and the change is backward-compatible.

[kezhenxu94]

you’ll need to expose this to the yaml so users are able to use it

[Navadeep0007]

Hi @kezhenxu94, thanks for the feedback!

I'd be happy to expose this to YAML. Could you clarify the preferred integration approach?

Option A:-Add a noDuplicates: true field to existing list verifications in YAML
Option B:- Create a new verification type specifically for duplicate checking

I'm leaning towards Option A for better integration with existing workflows. Would that work, or do you have a different pattern in mind?

Happy to implement whichever approach fits the project best!

[kezhenxu94]

@Navadeep0007 option A looks good to me, please go ahead, thanks!

[Navadeep0007]

Hi @kezhenxu94,
I’ve implemented the YAML integration for Option A (noDuplicates: true) as discussed.
The flag is now supported in list verifications and calls CheckForDuplicates().
Tests and examples have been updated.

[Navadeep0007]

Added missing Apache License headers to the following files to fix the CI failure reported by license-eye:

examples/list_verification.yaml

internal/verify/config.go

internal/verify/runner.go

The build should now pass successfully. ✅

[wu-sheng]

@Navadeep0007 Any update on this?

[wu-sheng]

This has been addressed differently in #145 using a noDuplicates pipe function integrated with the existing template engine.

Instead of a standalone utility, noDuplicates works as a standard Go template pipe that composes naturally with contains, containsOnce, and range:

{{- contains (.metrics | noDuplicates) }}
- name: {{ notEmpty .name }}
{{- end }}

No new packages or config fields needed — just one function registered in funcs.go. Closing in favor of #145.