moving and fixing kotlin-spark example to /projects

Author: Jolanrensen

docs/StardustDocs/topics/concepts/concepts.md

@@ -44,7 +44,7 @@ This is why it was designed to be hierarchical and allows nesting of columns and
 * [**Interoperable**](collectionsInterop.md) — convertable with Kotlin data classes and collections.
   This also means conversion to/from other libraries' data structures is usually quite straightforward!
   See our [examples](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources) 
-  for some conversions between DataFrame and [Apache Spark](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/spark), [Multik](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/multik), and [JetBrains Exposed](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/exposed).
+  for some conversions between DataFrame and [Apache Spark](https://github.com/Kotlin/dataframe/tree/master/examples/projects/spark-parquet-dataframe), [Multik](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/multik), and [JetBrains Exposed](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/exposed).
 * **Generic** — can store objects of any type, not only numbers or strings.
 * **Typesafe** — the Kotlin DataFrame library provides a mechanism of on-the-fly [**generation of extension properties**](extensionPropertiesApi.md) 
 that correspond to the columns of a dataframe. 

docs/StardustDocs/topics/dataSources/Integrations.md

@@ -19,7 +19,7 @@ Below is a list of example integrations with other data frameworks.
 These examples demonstrate how to bridge Kotlin DataFrame with external libraries or APIs.
 
 - [Kotlin Exposed](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/exposed)
-- [Apache Spark (with/without Kotlin Spark API)](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/spark)
+- [Apache Spark (with/without Kotlin Spark API)](https://github.com/Kotlin/dataframe/tree/master/examples/projects/spark-parquet-dataframe)
 - [Multik](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/multik)
 
 You can use these examples as templates to create your own integrations

docs/StardustDocs/topics/guides/Guides-And-Examples.md

@@ -58,7 +58,7 @@ and make working with your data both convenient and type-safe.
 * [Using Unsupported Data Sources](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/examples):
   — A guide by examples. While these might one day become proper integrations of DataFrame, for now,
   we provide them as examples for how to make such integrations yourself.
-    * [Apache Spark Interop (With and Without Kotlin Spark API)](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/spark)
+    * [Apache Spark Interop (With and Without Kotlin Spark API)](https://github.com/Kotlin/dataframe/tree/master/examples/projects/spark-parquet-dataframe)
     * [Multik Interop](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/multik)
     * [JetBrains Exposed Interop](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/exposed)
     * [Hibernate ORM](https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/hibernate)

examples/README.md

@@ -21,7 +21,7 @@ They show how to convert to and from Kotlin DataFrame and their respective table
     for an example of using Kotlin DataFrame with [Exposed](https://github.com/JetBrains/Exposed).
   * **Hibernate**: See the [hibernate folder](./idea-examples/unsupported-data-sources/hibernate)
     for an example of using Kotlin DataFrame with [Hibernate](https://hibernate.org/orm/).
-  * **Apache Spark**: See the [spark folder](./idea-examples/unsupported-data-sources/spark)
+  * **Apache Spark**: See the [spark folder](./projects/kotlin-spark)
     for an example of using Kotlin DataFrame with [Spark](https://spark.apache.org/) and with the [Kotlin Spark API](https://github.com/JetBrains/kotlin-spark-api).
   * **Multik**: See the [multik folder](./idea-examples/unsupported-data-sources/multik)
     for an example of using Kotlin DataFrame with [Multik](https://github.com/Kotlin/multik).

examples/projects/dev/kotlin-spark/.editorconfig

@@ -0,0 +1,41 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+indent_style = space
+indent_size = 4
+max_line_length = 120
+
+[*.json]
+indent_size = 2
+
+[{*.yaml,*.yml}]
+indent_size = 2
+
+[*.ipynb]
+insert_final_newline = false
+
+[*.{kt,kts}]
+ij_kotlin_code_style_defaults = KOTLIN_OFFICIAL
+
+#  Disable wildcard imports entirely
+ij_kotlin_name_count_to_use_star_import = 2147483647
+ij_kotlin_name_count_to_use_star_import_for_members = 2147483647
+ij_kotlin_packages_to_use_import_on_demand = unset
+
+ktlint_code_style = ktlint_official
+ktlint_experimental = enabled
+ktlint_standard_filename = disabled
+ktlint_standard_no-empty-first-line-in-class-body = disabled
+ktlint_class_signature_rule_force_multiline_when_parameter_count_greater_or_equal_than = 4
+ktlint_function_signature_rule_force_multiline_when_parameter_count_greater_or_equal_than = 4
+ktlint_standard_chain-method-continuation = disabled
+ktlint_ignore_back_ticked_identifier = true
+ktlint_standard_multiline-expression-wrapping = disabled
+ktlint_standard_when-entry-bracing = disabled
+ktlint_standard_expression-operand-wrapping = disabled
+
+[{*/build/**/*,**/*keywords*/**,**/*.Generated.kt,**/*$Extensions.kt,**/BuildConfig.kt}]
+ktlint = disabled

examples/projects/dev/kotlin-spark/README.md

@@ -0,0 +1,18 @@
+# Apache Spark
+
+Showcase of how to use DataFrame [Apache Spark](https://spark.apache.org/) and
+the [Kotlin Spark API](https://github.com/JetBrains/kotlin-spark-api).
+
+Even though Spark is not officially supported as a data source in DataFrame,
+this project shows how to convert from and to Spark tables.
+
+This project uses the
+[Kotlin DataFrame Compiler Plugin](https://kotlin.github.io/dataframe/compiler-plugin.html).
+
+We recommend using an up-to-date IntelliJ IDEA for the best experience,
+as well as the latest Kotlin plugin version.
+
+> [!WARNING]
+> For proper functionality in IntelliJ IDEA requires version 2025.2 or newer.
+
+[Download this Example](https://github.com/Kotlin/dataframe/raw/example-projects-archives/kotlin-spark.zip)

…rted-data-sources/spark/build.gradle.kts …ojects/dev/kotlin-spark/build.gradle.ktsexamples/idea-examples/unsupported-data-sources/spark/build.gradle.kts renamed to examples/projects/dev/kotlin-spark/build.gradle.kts examples/idea-examples/unsupported-data-sources/spark/build.gradle.kts renamed to examples/projects/dev/kotlin-spark/build.gradle.kts

@@ -1,28 +1,25 @@
 import org.jetbrains.kotlin.gradle.dsl.JvmTarget
 
 plugins {
-    application
-    kotlin("jvm")
-
-    // uses the 'old' Gradle plugin instead of the compiler plugin for now
-    id("org.jetbrains.kotlinx.dataframe")
+    alias(libs.plugins.kotlin.jvm)
+    alias(libs.plugins.kotlin.dataframe)
+    alias(libs.plugins.ktlint.gradle)
 
-    // only mandatory if `kotlin.dataframe.add.ksp=false` in gradle.properties
-    id("com.google.devtools.ksp")
+    application
 }
 
 repositories {
-    mavenLocal() // in case of local dataframe development
     mavenCentral()
 }
 
 dependencies {
-    // implementation("org.jetbrains.kotlinx:dataframe:X.Y.Z")
-    implementation(project(":"))
+    implementation(libs.dataframe)
 
-    // (kotlin) spark support
+    // (Kotlin) Spark SQL (Spark 3.3.2)
     implementation(libs.kotlin.spark)
-    compileOnly(libs.spark)
+    compileOnly(libs.spark.sql)
+
+    // Logging to keep Spark quiet
     implementation(libs.log4j.core)
     implementation(libs.log4j.api)
 }
@@ -64,6 +61,7 @@ val runSparkUntypedDataset by tasks.registering(JavaExec::class) {
 }
 
 kotlin {
+    jvmToolchain(11)
     compilerOptions {
         jvmTarget = JvmTarget.JVM_11
         freeCompilerArgs.add("-Xjdk-release=11")

examples/projects/dev/kotlin-spark/gradle.properties

@@ -0,0 +1,5 @@
+org.gradle.jvmargs=-Xmx1g -Dfile.encoding=UTF-8
+kotlin.code.style=official
+# Disabling incremental compilation will no longer be necessary
+# when https://youtrack.jetbrains.com/issue/KT-66735 is resolved.
+kotlin.incremental=false

examples/projects/dev/kotlin-spark/gradle/libs.versions.toml

@@ -0,0 +1,24 @@
+[versions]
+kotlin = "2.3.21"
+dataframe = "1.0.0-Beta5"
+ktlint-gradle = "14.0.1"
+ktlint = "1.8.0"
+log4j = "2.25.4"
+
+# check the versions down in the [libraries] section too!
+kotlin-spark = "1.2.4"
+spark3 = "3.3.2"
+
+[libraries]
+dataframe = { module = "org.jetbrains.kotlinx:dataframe", version.ref = "dataframe" }
+log4j-core = { group = "org.apache.logging.log4j", name = "log4j-core", version.ref = "log4j" }
+log4j-api = { group = "org.apache.logging.log4j", name = "log4j-api", version.ref = "log4j" }
+kotlin-spark = { group = "org.jetbrains.kotlinx.spark", name = "kotlin-spark-api_3.3.2_2.13", version.ref = "kotlin-spark" }
+spark-sql = { group = "org.apache.spark", name = "spark-sql_2.13", version.ref = "spark3" }
+
+[plugins]
+kotlin-jvm = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }
+ktlint-gradle = { id = "org.jlleitschuh.gradle.ktlint", version.ref = "ktlint-gradle" }
+
+# The Kotlin DataFrame Compiler plugin is the same version as the Kotlin plugin.
+kotlin-dataframe = { id = "org.jetbrains.kotlin.plugin.dataframe", version.ref = "kotlin" }