Skip to content

Memory grows unbounded during long pagination #7

Description

@sujito00

Bug

The paginated methods (searchAll, getUserTweetsAll, getBookmarksAll) use a seen Set to deduplicate tweets. This Set accumulates all tweet IDs indefinitely.

File: src/api/client.ts (lines ~514-623)

For getBookmarksAll() with maxPages = Infinity, this could grow to millions of entries on accounts with large bookmark collections.

Expected behavior

Either:

  • Use a bounded data structure (e.g., last N IDs) since duplicates typically only appear in adjacent pages
  • Stream results instead of accumulating them (yield tweets as they're fetched)
  • Add a hard cap with a warning

Impact

Low for typical use. High for power users exporting large datasets (bookmarks --all on accounts with 10k+ bookmarks).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions