Summary
TestPgStore/StatsStore (and any store test that creates more than one accepted
device in the same namespace) is flaky. It fails intermittently with:
--- FAIL: TestPgStore/StatsStore/succeeds_with_specific_tenantID
stats_tests.go:49:
Error Trace: api/store/storetest/helpers.go:226
api/store/storetest/stats_tests.go:49
Error: Received unexpected error:
document duplicate
Seen on CI: https://github.com/shellhub-io/shellhub/actions/runs/28873799116/job/85643304102
A plain re-run of the same commit goes green, which is the tell-tale of a flake.
Root cause
CreateDevice in api/store/storetest/helpers.go builds the default device MAC
from the wall clock:
Identity: &models.DeviceIdentity{MAC: fmt.Sprintf(
"%02x:%02x:%02x:%02x:%02x:%02x",
time.Now().UnixNano()%256, time.Now().UnixNano()%256, ...)},
On a runner whose clock resolution is coarser than the six time.Now() calls,
all six octets collapse to the same value, so the whole MAC becomes
VV:VV:VV:VV:VV:VV. Two devices created back to back then collide on their MAC
with probability ~1/256 per pair.
That collision was harmless until migration 007_devices_accepted_mac_unique
(#6600) added a unique index on (namespace_id, mac) WHERE status = 'accepted'.
Now a colliding MAC across two accepted devices in the same namespace turns into
a document duplicate on insert. StatsStore/succeeds_with_specific_tenantID
creates three accepted devices in tenant1, so it is the most frequent victim.
The default UID (fmt.Sprintf("%064x", time.Now().UnixNano())) has the same
weakness: two devices created within one clock tick collide on the primary key.
Suggested fix
Give the test helper collision-free defaults instead of deriving them from the
clock. A package-level atomic.Uint64 counter that feeds both the UID and the
MAC is enough:
var deviceSeq atomic.Uint64
n := deviceSeq.Add(1)
uid := fmt.Sprintf("%064x", n)
mac := fmt.Sprintf("00:00:%02x:%02x:%02x:%02x", byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
Callers that need a specific MAC already pass one through a DeviceOption, so
only the default changes. The other clock-derived defaults in this file
(namespace name, user email, session UID) are candidates for the same treatment
if they start colliding under their own unique constraints.
Notes
This is a pre-existing issue in the test helper on master, exposed by the
#6600 unique index. It is not tied to any one feature branch; it can fail any PR
whose run happens to hit the collision.
Summary
TestPgStore/StatsStore(and any store test that creates more than one accepteddevice in the same namespace) is flaky. It fails intermittently with:
Seen on CI: https://github.com/shellhub-io/shellhub/actions/runs/28873799116/job/85643304102
A plain re-run of the same commit goes green, which is the tell-tale of a flake.
Root cause
CreateDeviceinapi/store/storetest/helpers.gobuilds the default device MACfrom the wall clock:
On a runner whose clock resolution is coarser than the six
time.Now()calls,all six octets collapse to the same value, so the whole MAC becomes
VV:VV:VV:VV:VV:VV. Two devices created back to back then collide on their MACwith probability ~1/256 per pair.
That collision was harmless until migration
007_devices_accepted_mac_unique(#6600) added a unique index on
(namespace_id, mac) WHERE status = 'accepted'.Now a colliding MAC across two accepted devices in the same namespace turns into
a
document duplicateon insert.StatsStore/succeeds_with_specific_tenantIDcreates three accepted devices in
tenant1, so it is the most frequent victim.The default UID (
fmt.Sprintf("%064x", time.Now().UnixNano())) has the sameweakness: two devices created within one clock tick collide on the primary key.
Suggested fix
Give the test helper collision-free defaults instead of deriving them from the
clock. A package-level
atomic.Uint64counter that feeds both the UID and theMAC is enough:
Callers that need a specific MAC already pass one through a
DeviceOption, soonly the default changes. The other clock-derived defaults in this file
(namespace name, user email, session UID) are candidates for the same treatment
if they start colliding under their own unique constraints.
Notes
This is a pre-existing issue in the test helper on
master, exposed by the#6600 unique index. It is not tied to any one feature branch; it can fail any PR
whose run happens to hit the collision.