Linecount for `limit` and double-quotes

[luke-kiernan]

I'm using limit and skipto to parse part of a larger file. I get correct results for small files, but for a large file, the read starts one line later than it should:

big_file_kwargs = Dict(:skipto => 15, :limit => 2000)
small_file_kwargs = Dict(:skipto => 21, :limit => 5)
common_kwargs = Dict(:header => false, :ntasks => 1, :delim => '\t')
small_filepath = "case5.m"
big_filepath = "ACTIVSg2000.m"

using CSV
using DataFrames

# correct
df1 = DataFrame(CSV.File(small_filepath;
                        common_kwargs...,
                        small_file_kwargs...))
display(df1[1, :])
for (i, line) in enumerate(eachline(small_filepath))
    if i == small_file_kwargs[:skipto]
        println("first row of dataframe should be: $line")
    end
end
# incorrect: starts at line 16, not line 15.
df2 = DataFrame(CSV.File(big_filepath;
                        common_kwargs...,
                        big_file_kwargs...))
display(df2[1, :])
for (i, line) in enumerate(eachline(big_filepath))
    if i == big_file_kwargs[:skipto]
        println("first row of dataframe should be: $line")
    end
end

The inputs I'm using can be found here.

[ahcyip]

This looks serious!

[luke-kiernan]

Pinpointed the problem! In lines 5-6 of ACTIVSg2000.m, there's a citation:

% Reference: A.B. Birchfield, T. Xu, K.M. Gegner, K.S. Shetye, and T.J. Overbye, "Grid Structural Characteristics as Validation
% Criteria for Synthetic Networks," IEEE Transactions on Power Systems, vol. 32, no. 4, pp. 3258-3265, July 2017.

The parser sees the stuff enclosed by the double-quotes Grid ... \n ... Networks, as a potential string datum in the CSV row. So lines 5 and 6 are counted as a single line, and the parse starts one line later expected.

Seems to me like double quotes should only be treated this way if they're preceeded by the CSV deliminator:

• ,"long\nstring",: a single CSV string entry, long\nstring
• ,a"\n"b,: a pair of CSV string entries, a" on one row and "b on the next.