utf8 conversion errors on old firebird 2.5 - 5 charset NONE databases

[rpijnacker]

We are coming from delphi and are converting our software from unidac to rust using rfsbclient.
Works great after we convert database to charset WIN1252 for utf8 compabilty.
To convert from charset NONE to WIN1252 we need to do metadata extract and insert all data from old to new database
We have such hhuge database that we cannot achieve this in operation downtime is to big
Backup and restore can be done but complete pump of data takes too long

We have written our own flexible serde implementation to create json dynamically from sql/tables statements.
Our problem is that hen we setup driver with

rsfbclient::builder_native()
// .with_dyn_link()
.with_dyn_load(
firebird_driver.clone()
)
.with_remote()
.host(host)
.port(port)
.db_name(db_name)
.charset(Charset { on_firebird: ("NONE"), on_rust: None })
.to_owned()

and do our own conversion , in the rfsbclient already an utf8 conversion occurs cause it does not covers the case of real NONE connection. Can this be supported in the drivers or advise us how to write our own implementation so we can use firebird 2.5 to firebird 5 type 1 charset NONE connections. I think this applies for a lot of users that want to convert from delphi and never consider
anything about charsets

fn sql_type_to_val(v: SqlType) -> Option {
match v {
// ==================== TEXT (VARCHAR, CHAR, TEXT) ====================
SqlType::Text(text) => {
if text.contains("Array") {
// Your original array logic
if let Ok(json_val) = serde_json::from_str(&text[5..]) {
Some(json_val)
} else {
Some(Value::Null)
}
} else if text.contains("JSON") {
// Your original JSON logic
if let Ok(json_val) = serde_json::from_str(&text[4..]) {
Some(json_val)
} else {
Some(Value::Null)
}
} else {
// Normal text coming from Firebird with charset=NONE
let bytes = text.into_bytes();

            // Try UTF-8 first (in case some data is clean)
            if let Ok(valid_utf8) = String::from_utf8(bytes.clone()) {
                Some(Value::String(valid_utf8))
            } else {
                // Decode using Windows-1252 (most common for Delphi + UniDAC + charset=NONE)
                // This handles Latin-1 characters properly
                let (cow, _, _) = WINDOWS_1252.decode(&bytes);
                Some(Value::String(cow.into_owned()))
            }
        }
    }

    // ==================== BINARY / BLOB ====================
    SqlType::Binary(bytes) => {
        // Distinguish between BLOB SUB_TYPE 0 (binary) and SUB_TYPE 1 (text)
        if is_likely_text_blob(&bytes) {
            // BLOB SUB_TYPE 1 → treat as text (Delphi memos, long text)
            // Use Windows-1252 for proper Latin-1 character handling
            let (cow, _, _) = WINDOWS_1252.decode(&bytes);
            Some(Value::String(cow.into_owned()))
        } else {
            // BLOB SUB_TYPE 0 → real binary (images, files, etc.)
            // Return as JSON array of numbers (safe and standard)
            let byte_array: Vec<Value> = bytes.iter().map(|&b| Value::from(b as u64)).collect();
            Some(Value::Array(byte_array))
        }
    }

    // ==================== OTHER TYPES ====================
    SqlType::Integer(v) => Some(Value::from(v)),
    SqlType::Floating(v) => Some(Value::from(v)),

    SqlType::Timestamp(v) => {
        let val = format!("{}", v.format("%d/%m/%Y %H:%M:%S"));
        Some(Value::String(val))
    }

    SqlType::Null => Some(Value::Null),

    // Fallback for any other types (arrays, boolean, etc.)
    _ => None,
}

}

[fernandobatels]

Have you tried make a converter with two connection instances?

1. Read with WIN1252

   rsfbclient/src/tests/charset.rs

   Lines 65 to 74 in 1f29f88

   	// Hmm, I need use the same charset of inserted content
   	let mut conn = cbuilder().charset(ISO_8859_1)
   	.connect()?;
   	let (pao,): (String,) = conn.query_first(&format!("select * from {}", table), ())?
   	.unwrap();
   	assert_eq!("pão de queijo", pao);
   	Ok(())

2. Write on new DB wit UTF8