New plugin to generate data schemas in build/compile time

[Jolanrensen]

DataFrame used to have a Gradle/KSP plugin, able to generate data schemas from (a sample of) data before it was deprecated in 1.0.0-Beta5.
https://kotlin.github.io/dataframe/gradle-plugin.html
https://kotlin.github.io/dataframe/schemasimportsqlgradle.html

This is very useful to have, as the initial data schema provides information to the compiler plugin about which columns will be present in the dataframe. Then, after each operation, the compiler plugin can correctly track any changes to the schema and update the accessors accordingly.

We have a couple of possible options here:

Generate Data Schemas manually inside the project (The current recommended solution)

This is explained here, but in short:
you create a DataFrame instance using (a sample of) your data, and then call .generateDataClasses() or .generateInterfaces() on that dataframe or its data schema. The printed output you then paste into your codebase manually.

This can also be done for sources that provide types (only), like JDBC or OpenAPI.
For JDBC, there are the DataSchema.readSqlTable(...) functions, where you can read just the types of the database (well, sort of, #1681).
For OpenAPI, creating a DataSchema is not enough, you need enums and typealiases as well, so there's OpenApi().readCodeForGeneration() and readOpenApi() from the openapi-generator which produce copy-pastable code: https://kotlin.github.io/dataframe/openapi.html.

The benefit of this approach is that it works in any environment or build system. The downside is that your code needs to be able to run; the project needs to compile, or you need a separate project or notebook with access to the same data to generate the initial data schemas with. Any changes to your data will also require manual intervention and potential refactoring.

(I also need to mention requireColumn {} #1808 here, as it could be a good approach to avoid having to generate a data schema altogether, if you just need a couple columns)

Generate Data Schemas manually from Gradle

This is the same idea as above, but instead of generating code inside the project and copy-pasting it into the file, you let Gradle handle it. You'll need to let your build scripts depend on DataFrame and write a custom task which reads (a sample of) your data. Instead of just printing the generated code, you save it to a file in the build directory, included in your sources.

(I remember one of our users use this approach, but I cannot seem to find it anymore. Please respond if you find it!)

New KSP plugin, Annotating something like @DataSource/@DataSchemaSource

We could create a new KSP plugin which can generate new data schema files automatically from (a sample of) data in compile/build time.

The approach would be fundamentally different from the previous KSP/Gradle plugin to avoid the same pitfalls:
The old KSP plugin used @file:ImportDataSchema() annotations with many arguments to support all IO sources, yet it seemed to lack a lot of options that were needed to successfully read the data. The same applied to the dataframes { schema {} } notation for the Gradle plugin. The new approach should give the user the flexibility to use all DataFrame operations freely before generating the data schemas. And not only should it allow generating just data schemas, but it should also be able to generate additional types, like enums and typealiases. (Also, it should support maven: #721)

Some possible approaches:

• From Explore the possiblity to define schema and constunct object in Spring (DI framework) style #1321

@DataSource(csvFile = "data.csv")
val df: DataFrame<MyRowType>

• From (internal MVP) Prototype new iteration of ImportDataSchema annotation #1416

public class JsonSchemaReader : SchemaReader {
    override fun accepts(path: String, qualifier: String): Boolean =
        qualifier == SchemaReader.DEFAULT_QUALIFIER && path.endsWith(".json")

    override fun read(path: String): DataFrame<*> = DataFrame.readJson(path)
}

@DataSchemaSource(source = "data_sample.json")
interface MyRowType {
    // filled by the compiler plugin
}

fun main() {
    val df: DataFrame<MyRowType> = MyRowType.read("data.json")
}

Though, any approach would need a way to provide additional arguments and potential preprocessing steps to generate the data schema classes (and potentially any additional typealiases and enums). We would also need to be careful when and how often the compiler plugin would execute and generate this data schema code, as it could potentially contain a server-call.

Hopefully, we can create a helpful solution after the DataFrame 1.0 release. Please leave any feedback or ideas you might have for this use case!