Skip to content

Commit 64ecf45

Browse files
authored
Explicit dependencies in pxf-hdfs (#60)
`parquet-pig` transitively pulls couple compressors to testRuntimeClasspath and hides the fact that pxf-hdfs doesn't have snappy, zstd as direct runtime dependency. ``` +--- org.apache.parquet:parquet-pig -> 1.15.2 +--- org.apache.parquet:parquet-hadoop -> 1.15.2 | +--- org.apache.parquet:parquet-column:1.15.2 (*) | +--- org.apache.parquet:parquet-format-structures:1.15.2 (*) | +--- org.apache.parquet:parquet-common:1.15.2 (*) | +--- org.apache.parquet:parquet-jackson:1.15.2 | +--- org.xerial.snappy:snappy-java:1.1.10.7 | +--- io.airlift:aircompressor:2.0.2 | +--- commons-pool:commons-pool:1.6 | +--- com.github.luben:zstd-jni:1.5.6-6 -> 1.5.7-6 | \--- org.slf4j:slf4j-api:1.7.33 -> 1.7.36 ``` So, this PR removes transitive test-dependencies and explicitly adds runtime dependencies
1 parent b987ba1 commit 64ecf45

1 file changed

Lines changed: 4 additions & 2 deletions

File tree

server/pxf-hdfs/build.gradle

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ dependencies {
5555
// Dependencies for writing Avro files with compression
5656
implementation("org.apache.commons:commons-compress") { transitive = false }
5757
implementation("org.tukaani:xz") { transitive = false }
58+
implementation("com.github.luben:zstd-jni") { transitive = false }
59+
implementation("org.xerial.snappy:snappy-java") { transitive = false }
60+
implementation("io.airlift:aircompressor") { transitive = false }
5861

5962
// Hadoop 2.10.1 uses log4j:1.2.17, but since we are using slf4j
6063
// we bring log4j-over-slf4j which provides compatibility for log4j
@@ -96,8 +99,7 @@ dependencies {
9699

97100
testCompileOnly("org.apache.hadoop:hadoop-annotations")
98101

99-
testImplementation("io.airlift:aircompressor") // for HcfsTypeTest for testing write URI with LzopCodec
100-
testImplementation("org.apache.parquet:parquet-pig") // for parquet tests
102+
testImplementation("org.apache.parquet:parquet-pig") { transitive = false } // for parquet tests
101103
testImplementation('org.springframework.boot:spring-boot-starter-test')
102104
}
103105

0 commit comments

Comments
 (0)