Skip to content

Flaky store tests: CreateDevice test helper generates colliding MACs under the accepted-mac unique index #6617

Description

@gustavosbarreto

Summary

TestPgStore/StatsStore (and any store test that creates more than one accepted
device in the same namespace) is flaky. It fails intermittently with:

--- FAIL: TestPgStore/StatsStore/succeeds_with_specific_tenantID
    stats_tests.go:49:
        Error Trace: api/store/storetest/helpers.go:226
                     api/store/storetest/stats_tests.go:49
        Error: Received unexpected error:
               document duplicate

Seen on CI: https://github.com/shellhub-io/shellhub/actions/runs/28873799116/job/85643304102
A plain re-run of the same commit goes green, which is the tell-tale of a flake.

Root cause

CreateDevice in api/store/storetest/helpers.go builds the default device MAC
from the wall clock:

Identity: &models.DeviceIdentity{MAC: fmt.Sprintf(
    "%02x:%02x:%02x:%02x:%02x:%02x",
    time.Now().UnixNano()%256, time.Now().UnixNano()%256, ...)},

On a runner whose clock resolution is coarser than the six time.Now() calls,
all six octets collapse to the same value, so the whole MAC becomes
VV:VV:VV:VV:VV:VV. Two devices created back to back then collide on their MAC
with probability ~1/256 per pair.

That collision was harmless until migration 007_devices_accepted_mac_unique
(#6600) added a unique index on (namespace_id, mac) WHERE status = 'accepted'.
Now a colliding MAC across two accepted devices in the same namespace turns into
a document duplicate on insert. StatsStore/succeeds_with_specific_tenantID
creates three accepted devices in tenant1, so it is the most frequent victim.

The default UID (fmt.Sprintf("%064x", time.Now().UnixNano())) has the same
weakness: two devices created within one clock tick collide on the primary key.

Suggested fix

Give the test helper collision-free defaults instead of deriving them from the
clock. A package-level atomic.Uint64 counter that feeds both the UID and the
MAC is enough:

var deviceSeq atomic.Uint64

n := deviceSeq.Add(1)
uid := fmt.Sprintf("%064x", n)
mac := fmt.Sprintf("00:00:%02x:%02x:%02x:%02x", byte(n>>24), byte(n>>16), byte(n>>8), byte(n))

Callers that need a specific MAC already pass one through a DeviceOption, so
only the default changes. The other clock-derived defaults in this file
(namespace name, user email, session UID) are candidates for the same treatment
if they start colliding under their own unique constraints.

Notes

This is a pre-existing issue in the test helper on master, exposed by the
#6600 unique index. It is not tied to any one feature branch; it can fail any PR
whose run happens to hit the collision.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions