Skip to content

Commit 6f64d02

Browse files
committed
docs: update user guide with instructions for publishing local snapshots and inspecting Spark dependency trees
- Added steps for publishing snapshot artifacts to Maven local using Gradle tasks. - Documented usage of `dependencies` and `dependencyInsight` tasks for analyzing Spark dependency configurations.
1 parent fdee68c commit 6f64d02

1 file changed

Lines changed: 60 additions & 0 deletions

File tree

doc/user-guide.adoc

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,36 @@ The `extensions` module keeps Hadoop, Kafka, Avro, Schema Registry, and serializ
7373

7474
If an application needs to force its own runtime versions, use normal Gradle dependency constraints, platforms, or exclusions in the consuming build. Future extension modules can split these integrations further by feature so users can depend only on the extension families they need.
7575

76+
== Publishing Local Snapshots
77+
78+
Publish all snapshot artifacts to Maven local when another local project needs to consume the current checkout:
79+
80+
[source,bash]
81+
----
82+
GRADLE_USER_HOME=/data/.gradle ./gradlew publishToMavenLocal
83+
----
84+
85+
Publish only selected modules when you do not need the full repository:
86+
87+
[source,bash]
88+
----
89+
GRADLE_USER_HOME=/data/.gradle ./gradlew :core:publishToMavenLocal
90+
GRADLE_USER_HOME=/data/.gradle ./gradlew :junit5:publishToMavenLocal
91+
GRADLE_USER_HOME=/data/.gradle ./gradlew :extensions:publishToMavenLocal
92+
----
93+
94+
The consuming build must include `mavenLocal()` before remote repositories if it should prefer the local snapshot:
95+
96+
[source,kotlin]
97+
----
98+
repositories {
99+
mavenLocal()
100+
mavenCentral()
101+
}
102+
----
103+
104+
Use the current project version from `gradle.properties` or the root build. For example, if this checkout is `0.1.1-SNAPSHOT`, consume `org.openprojectx.bigdata.test:core:0.1.1-SNAPSHOT`, `org.openprojectx.bigdata.test:junit5:0.1.1-SNAPSHOT`, or `org.openprojectx.bigdata.test:extensions:0.1.1-SNAPSHOT`.
105+
76106
== Programmatic Usage
77107

78108
Use `BigDataTestKit.builder()` when you want direct lifecycle control.
@@ -1104,6 +1134,36 @@ The Kerberos axis currently enables the shared KDC and Kafka Kerberos path used
11041134

11051135
`sparkClouderaDepsApacheHmsTest` and `sparkClouderaDepsApacheHmsKerberosTest` are compatibility task names only; both use Cloudera HMS.
11061136

1137+
To inspect the resolved dependency tree for each Spark dependency line, use Gradle's `dependencies` task with the resolvable classpath configurations from `example/spark/build.gradle.kts`:
1138+
1139+
[source,bash]
1140+
----
1141+
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencies \
1142+
--configuration apacheSparkRuntimeClasspath
1143+
1144+
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencies \
1145+
--configuration clouderaSparkRuntimeClasspath
1146+
----
1147+
1148+
Use `dependencyInsight` when checking why a specific module or version is present:
1149+
1150+
[source,bash]
1151+
----
1152+
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
1153+
--configuration apacheSparkRuntimeClasspath \
1154+
--dependency org.apache.spark:spark-sql_2.12
1155+
1156+
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
1157+
--configuration clouderaSparkRuntimeClasspath \
1158+
--dependency org.apache.hadoop:hadoop-common
1159+
1160+
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
1161+
--configuration clouderaSparkRuntimeClasspath \
1162+
--dependency org.apache.iceberg
1163+
----
1164+
1165+
`testRuntimeClasspath` is not the right configuration for comparing the matrix lines because the Spark matrix overrides each test task's runtime classpath. Use `apacheSparkRuntimeClasspath` for the Apache Spark/Hadoop line and `clouderaSparkRuntimeClasspath` for the Cloudera line.
1166+
11071167
Legacy HMS-only task aliases are still present:
11081168

11091169
[source,bash]

0 commit comments

Comments
 (0)