Skip to content

Low priority: parse_ep_bid_table's fixed 1500-char slice can silently drop bids on a large table #151

Description

@alexwolson

Problem

parse_ep_bid_table (scrapers/toronto_bids/sources/ep_board.py:158) scopes the bid table to a fixed 1,500-character window after the "Table 1: Tender Price Submission" header:

tail = text[head.end():head.end() + 1500]

For the corpus captured so far this is ample (the 3- and 5-bidder fixtures fit comfortably), but it is a latent, silent data-loss boundary: a tender with many bidders — or wide, wrapped name/price cells — can push later rows past 1,500 chars, and those bids are dropped with no error and no log. Bid completeness is a core value of this archive ("was it competitive?"), so a silent under-count is exactly the failure mode #94/#116 worked to eliminate elsewhere.

Fix direction

Bound the region by structure, not a byte count — read from the header to the next blank-line gap / section boundary (the parser already looks for that), rather than a +1500 slice. Alternatively, if a hard cap is kept, log when the window is hit so truncation is never silent.

Acceptance

  • A synthetic Table 1 with enough bidders to exceed 1,500 chars extracts every row.
  • Existing EP bid-table tests still pass.

Surfaced by the #130 whole-branch review (deferrable Minor). Low priority — no known real report hits it yet, but it will bite a large tender without warning.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions