Skip to content
This repository was archived by the owner on Oct 8, 2020. It is now read-only.

Commit 278f6ef

Browse files
AklakanAklakan
committed
Merge branch 'release/0.4.0'
2 parents 88a207f + 585ba33 commit 278f6ef

5,355 files changed

Lines changed: 6456 additions & 508097 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,6 @@ hs_err_pid*
5151
stat*.txt
5252

5353
.idea
54+
55+
scalastyle-output.xml
56+

‎.travis.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
language: scala
2+
sudo: false
3+
cache:
4+
directories:
5+
- $HOME/.m2
6+
scala:
7+
- 2.11.11
8+
script:
9+
- mvn scalastyle:check
10+
- mvn test

‎README.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,30 @@ SANSA Query is a library to perform queries directly into [Spark](https://spark.
1010
SANSA uses vertical partitioning (VP) approach and is designed to support extensible partitioning of RDF data. Instead of dealing with a single three-column table (s, p, o), data is partitioned into multiple tables based on the used RDF predicates, RDF term types and literal datatypes. The first column of these tables is always a string representing the subject. The second column always represents the literal value as a Scala/Java datatype. Tables for storing literals with language tags have an additional third string column for the language tag. Its uses [Sparqlify](https://github.com/AKSW/Sparqlify) as a scalable SPARQL-SQL rewriter.
1111

1212
### SANSA Query Spark
13-
On SANSA Query Spark the method for partitioning an `RDD[Triple]` is located in [RdfPartitionUtilsSpark](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-spark-parent/sansa-rdf-spark-core/src/main/scala/net/sansa_stack/rdf/spark/partition/core/RdfPartitionUtilsSpark.scala). It uses an [RdfPartitioner](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-partition-parent/sansa-rdf-partition-core/src/main/scala/net/sansa_stack/rdf/partition/core/RdfPartitioner.scala) which maps a Triple to a single [RdfPartition](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-partition-parent/sansa-rdf-partition-core/src/main/scala/net/sansa_stack/rdf/partition/core/RdfPartition.scala) instance.
13+
On SANSA Query Spark the method for partitioning an `RDD[Triple]` is located in [RdfPartitionUtilsSpark](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-spark/src/main/scala/net/sansa_stack/rdf/spark/partition/core/RdfPartitionUtilsSpark.scala). It uses an [RdfPartitioner](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-common/src/main/scala/net/sansa_stack/rdf/common/partition/core/RdfPartitioner.scala) which maps a Triple to a single [RdfPartition](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-common/src/main/scala/net/sansa_stack/rdf/common/partition/core/RdfPartition.scala) instance.
1414

15-
* [RdfPartition](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-partition-parent/sansa-rdf-partition-core/src/main/scala/net/sansa_stack/rdf/partition/core/RdfPartition.scala) - as the name suggests, represents a partition of the RDF data and defines two methods:
15+
* [RdfPartition](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-common/src/main/scala/net/sansa_stack/rdf/common/partition/core/RdfPartition.scala) - as the name suggests, represents a partition of the RDF data and defines two methods:
1616
* `matches(Triple): Boolean`: This method is used to test whether a triple fits into a partition.
17-
* `layout: TripleLayout`: This method returns the [TripleLayout](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-partition-parent/sansa-rdf-partition-core/src/main/scala/net/sansa_stack/rdf/partition/layout/TripleLayout.scala) associated with the partition, as explained below.
17+
* `layout: TripleLayout`: This method returns the [TripleLayout](https://github.com/SANSA-Stack/SANSA-RDF/blob/develop/sansa-rdf-common/src/main/scala/net/sansa_stack/rdf/common/partition/layout/TripleLayout.scala) associated with the partition, as explained below.
1818
* Furthermore, RdfPartitions are expected to be serializable, and to define equals and hash code.
1919
* TripleLayout instances are used to obtain framework-agnostic compact tabular representations of triples according to a partition. For this purpose it defines the two methods:
2020
* `fromTriple(triple: Triple): Product`: This method must, for a given triple, return its representation as a [Product](https://www.scala-lang.org/files/archive/api/2.11.8/index.html#scala.Product) (this is the super class of all Scala tuples)
2121
* `schema: Type`: This method must return the exact Scala type of the objects returned by `fromTriple`, such as `typeOf[Tuple2[String,Double]]`. Hence, layouts are expected to only yield instances of one specific type.
2222

23-
See the [available layouts](https://github.com/SANSA-Stack/SANSA-RDF/tree/develop/sansa-rdf-partition-parent/sansa-rdf-partition-core/src/main/scala/net/sansa_stack/rdf/partition/layout) for details.
23+
See the [available layouts](https://github.com/SANSA-Stack/SANSA-RDF/tree/develop/sansa-rdf-common/src/main/scala/net/sansa_stack/rdf/common/partition/layout) for details.
2424

2525
## Usage
2626

2727
The following Scala code shows how to query an RDF file SPARQL syntax (be it a local file or a file residing in HDFS):
2828
```scala
2929

30-
val graphRdd = NTripleReader.load(spark, new File("path/to/rdf.nt"))
30+
val spark: SparkSession = ...
3131

32-
val partitions = RdfPartitionUtilsSpark.partitionGraph(graphRdd)
32+
val lang = Lang.NTRIPLES
33+
val triples = spark.rdf(lang)("path/to/rdf.nt")
34+
35+
36+
val partitions = RdfPartitionUtilsSpark.partitionGraph(triples)
3337
val rewriter = SparqlifyUtils3.createSparqlSqlRewriter(spark, partitions)
3438

3539
val qef = new QueryExecutionFactorySparqlifySpark(spark, rewriter)
@@ -38,6 +42,8 @@ val port = 7531
3842
val server = FactoryBeanSparqlServer.newInstance.setSparqlServiceFactory(qef).setPort(port).create()
3943
server.join()
4044

41-
4245
```
4346
An overview is given in the [FAQ section of the SANSA project page](http://sansa-stack.net/faq/#sparql-queries). Further documentation about the builder objects can also be found on the [ScalaDoc page](http://sansa-stack.net/scaladocs/).
47+
48+
## How to Contribute
49+
We always welcome new contributors to the project! Please see [our contribution guide](http://sansa-stack.net/contributing-to-sansa/) for more details on how to get started contributing to SANSA.

‎bundle-scaladocs.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
targetFolder="target/scaladocs-bundle"
4+
mkdir -p "$targetFolder"
5+
6+
for srcFolder in `find . -type d -name scaladocs`; do
7+
moduleName=$(basename $(dirname $(dirname $(dirname "$srcFolder"))))
8+
9+
if [[ "$moduleName" == "." ]]; then
10+
moduleName="sansa-parent"
11+
fi
12+
13+
cp -rf "$srcFolder" "$targetFolder/$moduleName"
14+
# echo "$ --- $moduleName";
15+
done
16+

‎pom.xml‎

Lines changed: 138 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
<groupId>net.sansa-stack</groupId>
66
<artifactId>sansa-query-parent_2.11</artifactId>
7-
<version>0.3.0</version>
7+
<version>0.4.0</version>
88
<packaging>pom</packaging>
99

1010
<name>SANSA Stack - Query Layer - Parent</name>
@@ -17,32 +17,28 @@
1717
<url>http://sda.tech</url>
1818
</organization>
1919

20-
<modules>
21-
<module>sansa-query-spark-parent</module>
22-
<module>sansa-query-flink-parent</module>
23-
</modules>
2420

2521
<properties>
2622
<maven.compiler.source>1.8</maven.compiler.source>
2723
<maven.compiler.target>1.8</maven.compiler.target>
2824
<encoding>UTF-8</encoding>
2925

30-
<sansa.version>0.3.0</sansa.version>
26+
<sansa.version>0.4.0</sansa.version>
3127

3228
<scala.version>2.11.11</scala.version>
3329
<scala.binary.version>2.11</scala.binary.version>
3430
<scala.classifier>${scala.binary.version}</scala.classifier>
3531

3632
<scala.version.suffix>_${scala.binary.version}</scala.version.suffix>
3733

38-
<spark.version>2.2.1</spark.version>
39-
<flink.version>1.4.0</flink.version>
34+
<spark.version>2.3.1</spark.version>
35+
<flink.version>1.5.0</flink.version>
4036

41-
<jena.version>3.5.0</jena.version>
42-
<jsa.subversion>2</jsa.subversion>
37+
<jena.version>3.7.0</jena.version>
38+
<jsa.subversion>3</jsa.subversion>
4339

4440
<jsa.version>${jena.version}-${jsa.subversion}</jsa.version>
45-
41+
<scalastyle.config.path>${project.basedir}/scalastyle-config.xml</scalastyle.config.path>
4642

4743
<httpcomponents.version>4.5.3</httpcomponents.version>
4844
</properties>
@@ -91,6 +87,7 @@
9187
</developer>
9288
</developers>
9389

90+
9491
<profiles>
9592
<profile>
9693
<id>doclint-java8-disable</id>
@@ -146,6 +143,20 @@
146143
</plugins>
147144
</build>
148145
</profile>
146+
147+
<!-- profile necessary for Scalastyle plugin to find the conf file -->
148+
<profile>
149+
<id>root-dir</id>
150+
<activation>
151+
<file>
152+
<exists>${project.basedir}/../../scalastyle-config.xml</exists>
153+
</file>
154+
</activation>
155+
<properties>
156+
<scalastyle.config.path>${project.basedir}/../scalastyle-config.xml</scalastyle.config.path>
157+
</properties>
158+
</profile>
159+
149160
</profiles>
150161

151162
<repositories>
@@ -196,6 +207,39 @@
196207
<dependencyManagement>
197208
<dependencies>
198209

210+
211+
<dependency>
212+
<groupId>${project.groupId}</groupId>
213+
<artifactId>sansa-rdf-common${scala.version.suffix}</artifactId>
214+
<version>${sansa.version}</version>
215+
</dependency>
216+
217+
<dependency>
218+
<groupId>${project.groupId}</groupId>
219+
<artifactId>sansa-rdf-spark${scala.version.suffix}</artifactId>
220+
<version>${sansa.version}</version>
221+
</dependency>
222+
223+
<dependency>
224+
<groupId>${project.groupId}</groupId>
225+
<artifactId>sansa-rdf-flink${scala.version.suffix}</artifactId>
226+
<version>${sansa.version}</version>
227+
</dependency>
228+
229+
230+
<dependency>
231+
<groupId>${project.groupId}</groupId>
232+
<artifactId>sansa-query-spark${scala.version.suffix}</artifactId>
233+
<version>${project.version}</version>
234+
</dependency>
235+
236+
<dependency>
237+
<groupId>org.apache.spark</groupId>
238+
<artifactId>spark-graphx_${scala.binary.version}</artifactId>
239+
<version>${spark.version}</version>
240+
</dependency>
241+
242+
199243
<!-- http components -->
200244
<dependency>
201245
<groupId>org.apache.httpcomponents</groupId>
@@ -235,12 +279,21 @@
235279
<version>${scala.version}</version>
236280
</dependency>
237281

282+
283+
<!-- Benchmarking bsbm and visualization of the results -->
238284
<dependency>
239285
<groupId>org.aksw.bsbm</groupId>
240286
<artifactId>bsbm-jsa</artifactId>
241-
<version>3.1.1</version>
287+
<version>3.1.2</version>
288+
</dependency>
289+
290+
<dependency>
291+
<groupId>org.aksw.beast</groupId>
292+
<artifactId>beast-bundle</artifactId>
293+
<version>1.0.0</version>
242294
</dependency>
243295

296+
244297
<dependency>
245298
<groupId>com.google.guava</groupId>
246299
<artifactId>guava</artifactId>
@@ -258,7 +311,7 @@
258311
<dependency>
259312
<groupId>org.aksw.sparqlify</groupId>
260313
<artifactId>sparqlify-core</artifactId>
261-
<version>0.8.3</version>
314+
<version>0.8.5</version>
262315
<exclusions>
263316
<exclusion>
264317
<groupId>org.aksw.sparqlify</groupId>
@@ -275,25 +328,6 @@
275328
</exclusions>
276329
</dependency>
277330

278-
<dependency>
279-
<groupId>net.sansa-stack</groupId>
280-
<artifactId>sansa-rdf-common-partition${scala.version.suffix}</artifactId>
281-
<version>${sansa.version}</version>
282-
</dependency>
283-
284-
<dependency>
285-
<groupId>net.sansa-stack</groupId>
286-
<artifactId>sansa-rdf-test-resources${scala.version.suffix}</artifactId>
287-
<version>${sansa.version}</version>
288-
</dependency>
289-
290-
<dependency>
291-
<groupId>${project.groupId}</groupId>
292-
<artifactId>sansa-rdf-partition-sparqlify${scala.version.suffix}</artifactId>
293-
<version>${sansa.version}</version>
294-
</dependency>
295-
296-
297331
<dependency>
298332
<groupId>org.aksw.jena-sparql-api</groupId>
299333
<artifactId>jena-sparql-api-server-standalone</artifactId>
@@ -313,6 +347,13 @@
313347
<version>3.0.3</version>
314348
</dependency>
315349

350+
<dependency>
351+
<groupId>com.holdenkarau</groupId>
352+
<artifactId>spark-testing-base_${scala.binary.version}</artifactId>
353+
<version>2.1.0_0.6.0</version>
354+
<scope>test</scope>
355+
</dependency>
356+
316357
<dependency>
317358
<groupId>junit</groupId>
318359
<artifactId>junit</artifactId>
@@ -569,6 +610,66 @@
569610
</configuration>
570611
</plugin>
571612

613+
<!--This plugin's configuration is used to store Eclipse m2e settings
614+
only. It has no influence on the Maven build itself. -->
615+
<plugin>
616+
<groupId>org.eclipse.m2e</groupId>
617+
<artifactId>lifecycle-mapping</artifactId>
618+
<version>1.0.0</version>
619+
<configuration>
620+
<lifecycleMappingMetadata>
621+
<pluginExecutions>
622+
<pluginExecution>
623+
<pluginExecutionFilter>
624+
<groupId>
625+
net.alchim31.maven
626+
</groupId>
627+
<artifactId>
628+
scala-maven-plugin
629+
</artifactId>
630+
<versionRange>
631+
[3.3.1,)
632+
</versionRange>
633+
<goals>
634+
<goal>testCompile</goal>
635+
<goal>compile</goal>
636+
<goal>add-source</goal>
637+
</goals>
638+
</pluginExecutionFilter>
639+
<action>
640+
<ignore></ignore>
641+
</action>
642+
</pluginExecution>
643+
</pluginExecutions>
644+
</lifecycleMappingMetadata>
645+
</configuration>
646+
</plugin>
647+
648+
<!-- Scalastyle -->
649+
<plugin>
650+
<groupId>org.scalastyle</groupId>
651+
<artifactId>scalastyle-maven-plugin</artifactId>
652+
<version>1.0.0</version>
653+
<configuration>
654+
<verbose>false</verbose>
655+
<failOnViolation>true</failOnViolation>
656+
<includeTestSourceDirectory>true</includeTestSourceDirectory>
657+
<failOnWarning>false</failOnWarning>
658+
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
659+
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
660+
<!-- we use a central config located in the root directory -->
661+
<configLocation>${scalastyle.config.path}</configLocation>
662+
<outputFile>${project.basedir}/scalastyle-output.xml</outputFile>
663+
<outputEncoding>UTF-8</outputEncoding>
664+
</configuration>
665+
<executions>
666+
<execution>
667+
<goals>
668+
<goal>check</goal>
669+
</goals>
670+
</execution>
671+
</executions>
672+
</plugin>
572673
</plugins>
573674
</pluginManagement>
574675
</build>
@@ -592,4 +693,9 @@
592693
</snapshotRepository>
593694
</distributionManagement>
594695

696+
<modules>
697+
<module>sansa-query-common</module>
698+
<module>sansa-query-flink</module>
699+
<module>sansa-query-spark</module>
700+
</modules>
595701
</project>

0 commit comments

Comments
 (0)