Serialize/Deserialize for Table and Array

[kpiwonski]

Up to my knowledge, currently there is no way to serialize/deserialize a Table or Array. Even if you don't need to serialize data, in a lot of cases you'd like to serialize a data model - data types, column names, and for categorical columns categories. For now, the only option is to reproduce all structures. For some types, like ArrowType it's ugly. Hence, I believe option to serialize/deserialize would be extremely useful.

I see 2 potential options:

• Using serde derive. I believe this might be an overkill, especially if you'd like to place it under a feature flag.
• Implement to_bytes -> Vec<u8>/from_bytes(data: &[u8]). I think it would be cleaner. Moreover, it looks like Arrow exchange structures (eg. Arrow Stream) might be close to it. C structs should be quite close to raw bytes, so it seems plausible there is an option to dump them. At least, if there is not too many pointers.

[pbower]

Hi @kpiwonski ,

Good call. I do have a solution available on this, in a related crate. I'll check about pulling it in.

Keeping with the minimal ethos I'm keen to avoid serde as it can be a little heavyweight.

I'll have a think on this and come back on it.

Regards,
Peter

[kpiwonski]

Hey, in the end I ended up using my own structs for serialization. As I do not need to serialize data I think it's a bit better for my use case. That being said, might be in the future someone will come up with this problem.

[pbower]

Hi - an update on this - I have been working on it with as part of the separate lightstream library which has a significant upgrade coming. It includes serialisation mechanisms that add the capability via trait handling. It is similarly opt-up in terms of features so it should not pull too much in.