Possible new feature - query chunking

[neilgilmour]

I've been hacking around on this package for the last few weeks at work - I had some tables that were tens of millions of rows to anonymise. I ended up adding a method to the TableDefinition which lets me set a queryChunkSize property for the table. Then I changed the dumpTableData() method to use the queryChunkSize, collecting the results until the insert statement's chunkSize was reached.

I also added an option to export to CSV instead of SQL, but that may be beyond the desired scope for this package.

Is there any appetite from BeyondCode (maybe @mechelon or @mpociot) for a PR for any of this? I haven't really contributed anything to an open source package before so I'm not sure on the etiquette. I'd need to tidy it up before submitting, so I wanted to check first.

[mechelon]

That would be a great addition! 🙂 Would be great if you could provide tests as well as the documentation for it as well.
We just have to make sure it doesn't get confusing with the existing INSERT statement chunking: https://expose.dev/docs/server/server/server-configuration So I'm not sure about the name yet 🤔 Any ideas?

[neilgilmour]

I used queryChunkSize for the property, and queryInChunksOf() for the public method. The public method for the insert chunks is nicely named outputInChunksOf(), but the property is chunkSize. I don't know about the implications of renaming an existing property, but something like outputChunkSize would be nice. My thoughts are that it probably isn't necessary - the public methods are clearly named, and hopefully there's enough distinction in the method params between ->queryInChunksOf(queryChunkSize: 200) and ->outputInChunksOf(chunkSize: 500).

Of course I'm very happy to rename my bits - I may be missing some wider context/experience!

[mechelon]

I think you're right - as long as we are consistent with the new feature being queryChunkSize it should be comprehensible! 👍