Cinco de mayo (#312)

* Finish removing IBasicCSVParser

* Create csv::internals::speculative

* Make CSVParserCore templated

* Fix -Werror=reorder

* Simplified RowCollection. Removed unsync access methods.

* Add append_rows() to RowCollection

* Refactor TSD to accept batches

* 1gb/s

C:\CSV Datasets\metadata.tsv>csv_bench metadata.tsv
Parsing took (including disk IO): 11.4867
Dimensions: 9074875 rows x 58 columns
Parser worker threads: 12
Speculative chunks: 1210 ambiguous=0 probability_model=0 size_heuristic=0 repairs=0 assumed_quoted=0 assumed_unquoted=1210
Columns:  strain virus gisaid_epi_isl genbank_accession genbank_accession_rev sra_accession date region country division location region_exposure country_exposure division_exposure segment length host age sex Nextstrain_clade pango_lineage GISAID_clade originating_lab submitting_lab authors url title paper_url date_submitted date_updated sampling_strategy database clade_nextstrain clade_who Nextclade_pango immune_escape ace2_binding missing_data divergence nonACGTN coverage rare_mutations reversion_mutations potential_contaminants QC_missing_data QC_mixed_sites QC_rare_mutations QC_snp_clusters QC_frame_shifts QC_stop_codons QC_overall_score QC_overall_status frame_shifts deletions insertions substitutions aaSubstitutions clock_deviation

* Remove CSVRowOutput

* Clean-up BOM stripping logic

* Get rid of tuple stuff

* Restore previous parser hot loop

* Reorganize code

* Make threading fully runtime customizable

* Update csv_read_scheduler.hpp

* Reorganize namespaces

* Update driver.cpp

* Code deduplication

* Test strengthening

* Added more tests

* Update test_read_csv_file.cpp

* Fix poor performance with quoted values

* Split up memory optimization parts

* Fix TSan issue

* More cleanup

* Improve Python benchmarks

* Update benchmarks

* Updated docs

Removed speculative_parsing() method. It now automatically turns on for a specific threshold.

* Up timeout for TSD stress test

Author: vincentlaucsb

.claude/rules/csv_reader_rules.md

@@ -63,7 +63,7 @@ jobs:
         python-version: '3.x'
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-${{ matrix.c_compiler }}-std${{ matrix.cxx_standard }}-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -127,7 +127,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-single-threaded-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -167,7 +167,7 @@ jobs:
         python-version: '3.x'
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-emscripten-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -228,7 +228,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-simd-${{ matrix.simd_mode }}-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}

.github/workflows/cmake-multi-platform.yml

@@ -43,7 +43,7 @@ jobs:
         python-version: '3.x'
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-msvc-no-zc-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -95,7 +95,7 @@ jobs:
           mingw-w64-x86_64-ninja
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-mingw-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}

.github/workflows/compat-edge-cases.yml

@@ -24,7 +24,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-coverage-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}

.github/workflows/coverage.yml

@@ -49,7 +49,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-sanitizer-${{ matrix.sanitizer }}-std${{ matrix.cxx_standard }}-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -81,7 +81,7 @@ jobs:
 
     - name: Upload sanitizer logs
       if: failure()
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: linux-sanitizer-logs-${{ matrix.sanitizer }}-std${{ matrix.cxx_standard }}
         path: build/Testing/
@@ -103,7 +103,7 @@ jobs:
       uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-msvc-asan-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -130,7 +130,7 @@ jobs:
 
     - name: Upload MSVC AddressSanitizer logs
       if: failure()
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: windows-msvc-asan-logs-std20
         path: build/msvc-asan/Testing/
@@ -145,7 +145,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-valgrind-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}
@@ -177,7 +177,7 @@ jobs:
 
     - name: Upload Valgrind results
       if: failure()
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: valgrind-results
         path: build/Testing/
@@ -194,7 +194,7 @@ jobs:
         submodules: recursive
 
     - name: Cache CMake FetchContent dependencies
-      uses: actions/cache@v4
+      uses: actions/cache@v5
       with:
         path: ${{ env.CSV_FETCHCONTENT_BASE_DIR }}
         key: fetchcontent-${{ runner.os }}-o3-coverage-${{ hashFiles('CMakeLists.txt', 'tests/CMakeLists.txt') }}

.github/workflows/sanitizers.yml

@@ -7,6 +7,7 @@ bin/
 build/
 
 # Build: Python
+__pycache__/
 *.pyc
 *.pyd
 

.gitignore

@@ -1,6 +1,3 @@
 [submodule "tests/data"]
 	path = tests/data
 	url = https://github.com/vincentlaucsb/csv-data.git
-[submodule "python/pybind11"]
-	path = python/pybind11
-	url = https://github.com/pybind/pybind11.git

.gitmodules

@@ -2,7 +2,7 @@
 
 Architectural overview for AI assistants working with this codebase.
 
-> **Maintenance rule:** Whenever this file is changed, update both `CLAUDE.md` and `ARCHITECTURE.md` in the same directory to reflect relevant changes. `CLAUDE.md` is a bullet-point summary and `ARCHITECTURE.md` is the top-level architecture index; both must stay in sync with this guidance.
+> **Maintenance rule:** `AGENTS.md` is the canonical AI-agent guidance. When this file changes, update `ARCHITECTURE.md` in the same directory if the architecture index or durable engineering guidance needs to reflect the change. `CLAUDE.md` is only a compatibility shim that points here.
 
 ## Critical: single_include/csv.hpp Is A Shim
 
@@ -52,6 +52,7 @@ For detailed file mapping, parser data flow, and component relationships, see `A
 8.  **Opportunistic rewrites/refactors are allowed when they are safe and justified.** Keep them separated from build-fix urgency where possible, and avoid bundling unrelated churn with compiler triage unless explicitly requested.
 9. **When proposing changes that affect compile-time behavior, explain the tradeoff clearly.** Call out any impact to codegen, performance, portability, and readability before applying the change.
 10. **If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.** Provide a short justification before expanding further.
+11. **`CSVReader::iterator` is intentionally single-pass.** Do not cache all `RawCSVDataPtr` chunks to make it behave like a forward iterator; that defeats bounded-memory streaming for large CSV files. Algorithms that need multi-pass access should first materialize rows into a container such as `std::vector<CSVRow>`.
 
 See `tests/AGENTS.md` for test strategy, checklist, and conventions.
 
@@ -71,6 +72,9 @@ See `tests/AGENTS.md` for test strategy, checklist, and conventions.
    - **Consolidation:** If a `.cpp` would be under ~100 lines *and* the split causes excessive comment duplication between the two files, prefer a single `.hpp` with definitions marked `inline` (free functions and methods alike). Do not use `CSV_INLINE` for consolidated definitions — `CSV_INLINE` expands to empty in multi-header mode, which would produce ODR violations across TUs. Do not consolidate just for brevity — only when duplication is the dominant cost.
 7. **Prefer LF (`\n`) line endings for tracked source, test, CMake, and Markdown files.** When you touch a file with mixed line endings, normalize the edited file to LF unless there is a file-specific reason not to. Avoid introducing mixed CRLF/LF endings in the same file.
 8. **Keep preprocessor directives flush left.** `#define`, `#if`, `#ifdef`, `#else`, and `#endif` should start at column 0. Code inside multi-line macros should be indented exactly as the equivalent non-macro code would be; do not add extra indentation just because it lives inside a macro body.
+9. **Keep constructor initializer lists in declaration order.** C++ initializes bases and members in declaration order, not initializer-list order. When adding or editing a constructor, order its initializer list to match the class declaration exactly so GCC/Clang `-Wreorder` stays clean and readers do not infer a false initialization dependency.
+10. **Internal folder namespaces should match folder structure.** When adding or moving files under `include/internal/`, place their contents in the matching nested namespace when practical. For example, `include/internal/speculative/` maps to `csv::internals::speculative`, and `include/internal/parser/` maps to `csv::internals::parser`. Do not churn existing files solely for this rule unless the namespace move is part of an intentional architecture cleanup.
+11. **Do not accidentally pass large objects by value.** Use `const&` for observation, `&` for mutation, and `&&` / by-value-with-an-explicit-`std::move` for ownership transfer. If passing a large object by value is intentional, make the consuming semantics obvious at the call site or add a brief comment.
 
 ### Rules for Comments
 1. **Always update or remove incorrect comments.**

.travis.yml

@@ -7,6 +7,7 @@ Primary architecture document:
 
 Subsystem deep-dive:
 - include/internal/THREADSAFE_DEQUE_DESIGN.md
+- BOMStrippingRefactor.md
 
 Operational/testing guidance:
 - AGENTS.md
@@ -23,7 +24,9 @@ Notes:
 - Private member naming should prefer trailing underscores; when editing mixed-style code, normalize the touched region toward that convention.
 - Prefer LF (`\n`) line endings for tracked source, test, CMake, and Markdown files; when touching a file with mixed endings, normalize it to LF unless there is a file-specific reason not to.
 - Keep preprocessor directives flush left; `#define`, `#if`, `#ifdef`, `#else`, and `#endif` should start at column 0, and code inside multi-line macros should be indented as if the macro wrapper were not present.
-- Compatibility macros defined in `common.hpp` must only be referenced after including `common.hpp`. See AGENTS.md and CLAUDE.md for details.
+- Keep constructor initializer lists in the same order as base/member declarations so GCC/Clang `-Wreorder` remains clean and initialization dependencies stay obvious.
+- Internal folder namespaces should match folder structure when practical; for example `include/internal/parser/` maps to `csv::internals::parser`.
+- Compatibility macros defined in `common.hpp` must only be referenced after including `common.hpp`. See AGENTS.md for details.
 - API constraints should be user-friendly: do not over-constrain templates unless needed for correctness, safety, or a measured performance win.
 - `CSVReader` is intentionally non-copyable and move-enabled; use explicit ownership transfer patterns (`std::move`, `std::unique_ptr`) at API boundaries.
 - Respect existing compile-time compatibility macros (`IF_CONSTEXPR`, `CONSTEXPR_VALUE`, etc.) unless correctness requires change.
@@ -32,4 +35,3 @@ Notes:
 - When changing compile-time behavior, explicitly document tradeoffs (codegen, performance, portability, readability).
 - If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.
 - Apply the 5/2 anti-duplication rule: if equivalent behavior exists in 2+ code paths and each copy is ~5+ meaningful lines, extract a shared helper; if duplication remains, document why and keep regression coverage for each path.
-