Skip to content

Scattered collection PoC#427

Closed
mmastrac wants to merge 4 commits into
masterfrom
scattered-collect
Closed

Scattered collection PoC#427
mmastrac wants to merge 4 commits into
masterfrom
scattered-collect

Conversation

@mmastrac
Copy link
Copy Markdown
Owner

@mmastrac mmastrac commented May 5, 2026

This is the PoC for a new crate built on top of ctor and link-section: scattered-collect. This won't land directly -- it will get carved up into smaller PRs and polished.

The idea is:

#[gather]
pub static COLLECTION: ScatteredSlice<T>;

#[scatter(COLLECTION)]
pub static ITEM: T = { ... };

Why?

Distributed slice is awesome and nearly perfect but it's a static snapshot out of the linker and downstream code occasionally needs more functionality on top of it, ie:

  • Sorting/ordering
  • Lookups by something other than ordinal index

Also, it doesn't support WASM which is a tougher to fix.

Inventory supports pretty much everything, but it's just a #[ctor] with a linked list and doesn't offer easy random access.

TODO

I'll spin some PRs off of this one and slowly land some pieces:

  • PR for placeholder crate (& publish a 0.0.0)
  • Land the top-level scatter/gather macro implementations
  • Implement and land the slice/referenced slice trivial types
  • Implement and land sorted versions of slices
  • Implement and land map
  • Implement and land iterable

Data types

  • (TODO) Iterable: collected items, only available via iterator (analogous to inventory crate)
  • Slice: collected items, not available at original definition site
  • Referenced slice: collection items, available at original definition site (analogous to distributed slice crate)
  • Sorted slice: collected and sorted items, not available at original definition site
  • Sorted referenced slice: collected and sorted items, available at original definition site
  • Map: similar to hashmap's swisstable implementation, but space is allocated at link time and hashes are precomputed to make construction much cheaper

There are "referenced" and "unreferenced" versions of the slices - the unreferenced versions may be faster as the code is free to re-organize the data more easily.

Const hashing

  • New trait: ConstHasher (should be able to derive this for custom structs as well)

Platform challenge notes

  • WASM: link sections are pretty much useless unless we do some sort of post-processing on the binary, which might be an option. We can, however, at least count the number of items using link-section and then place all the data via alloc + ctor copies where needed (the scattered iterator won't need this, and the map needs at least one iteration anyways, so we might be able to have a custom impl there).
  • All other platforms: generally will work via slices as expected. Similar single-threaded init with a late ctor to ctor's static will allow for guaranteed init before main without ordering issues.

@mmastrac mmastrac force-pushed the scattered-collect branch from 2a79cd3 to a330cf8 Compare May 5, 2026 13:57
mmastrac added a commit that referenced this pull request May 5, 2026
Broken off from #427 - this is the basic crate structure and supporting
macros.
@mmastrac mmastrac force-pushed the scattered-collect branch 4 times, most recently from 0e9e658 to 825fa5e Compare May 11, 2026 17:39
mmastrac added a commit that referenced this pull request May 15, 2026
Implement two of the collections from #427: slice and sorted-slice, both
of which are pretty trivial.

As part of this work, fixing some link-section issues re: mutable
slices: windows requires them to be in `UnsafeCell`.
@mmastrac mmastrac force-pushed the scattered-collect branch 2 times, most recently from 763bc64 to 3a435cd Compare May 17, 2026 19:49
@mmastrac mmastrac changed the title Early PoC: scattered-collect Scattered collection PoC May 17, 2026
@mmastrac mmastrac force-pushed the scattered-collect branch from 68d983a to 47f0360 Compare May 19, 2026 00:52
@mmastrac mmastrac mentioned this pull request May 19, 2026
@mmastrac
Copy link
Copy Markdown
Owner Author

Closing - PoC completed.

@mmastrac mmastrac closed this May 19, 2026
mmastrac added a commit that referenced this pull request May 19, 2026
Adds a scattered iterable, completing the set of collections for #427
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant