compatibility with mariadb connector 3.2.8

[hhaensel]

Currently usage of the latest mariadb odbc connector fails due to two reasons

• the termination character is not stripped off
• the max value for characters in a row is returned as 65535, which leads to a Column length too big error

Both root causes are addressed with this PR.

[codecov]

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.82%. Comparing base (a6e4fb7) to head (7e57b9b).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/utils.jl	71.42%	2 Missing ⚠️
src/load.jl	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #392      +/-   ##
==========================================
- Coverage   80.32%   78.82%   -1.50%     
==========================================
  Files           7        7              
  Lines         910      940      +30     
==========================================
+ Hits          731      741      +10     
- Misses        179      199      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[quinnj]

Could this be narrowed to the specific MariaDB text type case? Replacing every 65535 in the SQL type string can silently shrink unrelated columns, e.g. VARBINARY(65535) or DECIMAL(65535,6), to 255. [findings by codex; reviewed by quinnj]

[hhaensel]

VARBINARY(65535) is also not possible, so that also needs to be reduced. I searched a bit how to determine which connector is selected, but I couldn't find a way to distinguish between mariadb 3.1 and 3.2.
Maybe a more reasonable approach would be to pass a dict with type => size correlation or name => size correlation?

[hhaensel]

along these lines ...

function createtable(
    conn::Connection, 
    nm::AbstractString, 
    sch::Tables.Schema; 
    debug::Bool=false, 
    quoteidentifiers::Bool=true, 
    createtableclause::AbstractString="CREATE TABLE", 
    columnsuffix=Dict(),
    columntypes=Dict()
)
    names = sch.names
    checkdupnames(names)
    types = [haskey(columntypes, names[i]) ? 
             string(columntypes[names[i]]) : 
             sqltype(conn, sch.types[i]) 
             for i = 1:length(names)]

    columns = (string(quoteidentifiers ? quoteid(conn, String(names[i])) : names[i], ' ', types[i], ' ', get(columnsuffix, names[i], "")) for i = 1:length(names))
    debug && @info "executing create table statement: `$createtableclause $nm ($(join(columns, ", ")))`"
    return DBInterface.execute(conn, "$createtableclause $nm ($(join(columns, ", ")))")
end

or these

function sqltype(conn, T, columnsize::Union{Integer, Nothing} = nothing)
    if isempty(conn.types)
        types = Tables.columntable(Cursor(API.gettypes(conn.dbc)))
        conn.alltypes = types
        i = findfirst(==(API.SQL_VARCHAR), types.DATA_TYPE)
        if i === nothing
            defaultT = "VARCHAR(255)"
        else
            cs = something(columnsize, types.COLUMN_SIZE[i])
            defaultT = types.TYPE_NAME[i] * "($cs)"
        end
        for jlT in BINDTYPES
            _, sqlT = bindtypes(jlT)
            i = findfirst(==(sqlT), types.DATA_TYPE)
            nm = i !== nothing ? types.TYPE_NAME[i] : defaultT
            if i !== nothing && types.CREATE_PARAMS[i] !== missing && nm != "DOUBLE" && nm != "FLOAT"
                cs = something(columnsize, types.COLUMN_SIZE[i])
                nm *= occursin(',', types.CREATE_PARAMS[i]) ? "($(typeprecision(jlT)),$(typescale(jlT)))" : "($cs)"
            end
            conn.types[jlT] = nm
        end
    end
    return conn.types[T]
end

checkdupnames(names) = length(unique(map(x->lowercase(String(x)), names))) == length(names) || error("duplicate case-insensitive column names detected; sqlite doesn't allow duplicate column names and treats them case insensitive")

function createtable(
    conn::Connection, 
    nm::AbstractString, 
    sch::Tables.Schema; 
    debug::Bool=false, 
    quoteidentifiers::Bool=true, 
    createtableclause::AbstractString="CREATE TABLE", 
    columnsuffix=Dict(),
    columnsizes=Dict()
)
    names = sch.names
    checkdupnames(names)
    types = [sqltype(conn, T, get(columnsizes, names[i], nothing)) for (i, T) in enumerate(sch.types)]
    columns = (string(quoteidentifiers ? quoteid(conn, String(names[i])) : names[i], ' ', types[i], ' ', get(columnsuffix, names[i], "")) for i = 1:length(names))
    debug && @info "executing create table statement: `$createtableclause $nm ($(join(columns, ", ")))`"
    return DBInterface.execute(conn, "$createtableclause $nm ($(join(columns, ", ")))")
end

[hhaensel]

Or rather provide a conntypes to be merged with conn.types
Dict{Type, String} with 19 entries:
Dec64 => "DECIMAL(16,6)"
Vector{UInt8} => "VARBINARY(65535)"
Float32 => "VARCHAR(65535)"
Int64 => "BIGINT"
Dec128 => "DECIMAL(35,6)"
Bool => "BIT"
Time => "TIME"
UInt16 => "SMALLINT"
Int8 => "TINYINT"
UUID => "VARCHAR(65535)"
Date => "DATE"
UInt32 => "INTEGER"
DateTime => "DATETIME"
Int32 => "INTEGER"
UInt64 => "BIGINT"
Float64 => "DOUBLE"
String => "VARCHAR(65535)"
Int16 => "SMALLINT"
UInt8 => "TINYINT"

[hhaensel]

After some nightly reflection I think none of these proposals is a good one.
We should rather provide better type detection in sqltype(). I'll come up with a proposal.

[hhaensel]

@quinnj what do you think about this approach. It tests whether default sizes have the typical excessive values of the mariadb driver 3.2 that don't make sense for default types.

function sqltype(conn, T)
    if isempty(conn.types)
        types = Tables.columntable(Cursor(API.gettypes(conn.dbc)))
        conn.alltypes = types
        i = findfirst(==(API.SQL_VARCHAR), types.DATA_TYPE)
        if i === nothing
            defaultT = "VARCHAR(255)"
        else
            # Safeguard: Any driver returning exactly the maximum tiny/medium text range 
            # for default VARCHAR triggers a standard 255 fallback, e.g. mariadb version 3.2
            colsize = types.COLUMN_SIZE[i]
            (colsize == 65535 || types.COLUMN_SIZE[i] == 16777215) && (colsize = 255)
            defaultT = types.TYPE_NAME[i] * "($colsize)"
        end
        for jlT in BINDTYPES
            _, sqlT = bindtypes(jlT)
            i = findfirst(==(sqlT), types.DATA_TYPE)
            nm = i !== nothing ? types.TYPE_NAME[i] : defaultT
            if i !== nothing && types.CREATE_PARAMS[i] !== missing && nm != "DOUBLE" && nm != "FLOAT"
                
                colsize = types.COLUMN_SIZE[i]
                # Step down the explicit type bindings if they hit MariaDB max sizes
                if (sqlT in (API.SQL_VARCHAR, API.SQL_VARBINARY, API.SQL_WVARCHAR)) && 
                   (colsize == 65535 || colsize == 16777215)
                    colsize = 255
                end
                
                nm *= occursin(',', types.CREATE_PARAMS[i]) ? "($(typeprecision(jlT)),$(typescale(jlT)))" : "($(colsize))"
            end
            conn.types[jlT] = nm
        end
    end
    return conn.types[T]
end

[hhaensel]

Moreover, the README for testing is outdated, e.g. we need a mariadb for testing, an mysql will throw upon

result = DBInterface.execute(
    conn, "SELECT a, b, a/b AS c FROM (VALUES (2,1),(1,0),(2,1)) AS t(a,b)"
) |> columntable

and setting TRAVIS_DIR is deprecated, if I'm right.
Should I update that section in this PR?

[quinnj]

Thanks, this makes sense. I agree that adding columntypes/columnsizes/conntypes feels too broad for this PR. The sqltype() approach is the direction I’d prefer: normalize the suspicious MariaDB max sizes while interpreting the driver metadata, and only for the variable string/binary SQL types, rather than doing a blanket string replace in createtable.

Could you update the branch with that approach and remove the replace(sqltype(...), "65535" => "255") call? I’d keep the README testing cleanup separate from this compatibility fix.

[hhaensel]

@quinnj done