Is it possible to add a multiprocessing argument?

[myrainbowandsky]

We could split a csv into pieces of csvs and use twarc-csv for each of them. Or maybe we could use joblib's n_jobs to read CPU-thread number of blocks of Json lines. A server usually has multi-cores. Is it possible to add such a parameter?

[igorbrigadir]

Yes, it's not implemented but possible, and welcome (with an added requirement of keeping things in the same order as the json, as this helps downstream tools).

The process of generating CSVs is already "embarrassingly parallel" as it splits the data into chunks here: https://github.com/DocNow/twarc-csv/blob/main/csv_writer.py#L82 so that would be the best place for it.

Joblib would be nice. Will accept good PRs for this.