Skip to content

Commit cc37f86

Browse files
JohannesJanderJohannesJander
committed
Work on foreign keys, API changes in regards to iterators (necessary b/c return types were partly wrong), more robust BeanSchema/BeanIterator
1 parent 3ef4c29 commit cc37f86

50 files changed

Lines changed: 1128 additions & 635 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/table-reading.md‎

Lines changed: 16 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,10 @@ And with that, we can read the data as `SimpleDataBean` instances:
5353
URL url = new URL("https://raw.githubusercontent.com/frictionlessdata/tableschema-java/master" +
5454
"/src/test/resources/fixtures/data/simple_data.csv");
5555
// Load the data from URL without a schema.
56-
Table simpleTable = Table.fromSource(url, (Schema)null, DataSourceFormat.getDefaultCsvFormat());
56+
Table table = Table.fromSource(url, (Schema)null, DataSourceFormat.getDefaultCsvFormat());
5757

5858
List<SimpleDataBean> data = new ArrayList<>();
59-
Iterator<SimpleDataBean> bit = new BeanIterator<>(simpleTable, SimpleDataBean.class, false);
59+
Iterator<SimpleDataBean> bit = new BeanIterator<>(table, SimpleDataBean.class, false);
6060
while (bit.hasNext()) {
6161
SimpleDataBean record = bit.next();
6262
data.add(record);
@@ -96,40 +96,34 @@ while(iter.hasNext()){
9696
List<Object[]> allData = table.read();
9797
```
9898

99-
### Reading tabular data using a Schema
10099

101-
To assure format integrity, we can create a Schema that describes the data we expect.
102-
Each row of the table will be returned as an Object array. Values in each column are parsed and
103-
converted to Java objects according to the field definition in the Schema. If data type and field
104-
definition do not match, an exception would be thrown.
100+
### Reading tabular data returning rows as Maps
105101

106-
See https://specs.frictionlessdata.io/table-schema/ for more details on Schemas
102+
The Table object has flexible iterators that can return table rows as `Map<String, Object>` instead
103+
of `Object`. In this case, the column header is the key and the row value is the value in the map:
107104

108105
```java
109-
// Let's start by defining and building the schema of a table that contains data about employees:
110106
Schema schema = new Schema();
107+
111108
schema.addField(new IntegerField("id"));
112109
schema.addField(new StringField("title"));
113-
114110
URL url = new URL("https://raw.githubusercontent.com/frictionlessdata/tableschema-java/master" +
115111
"/src/test/resources/fixtures/data/simple_data.csv");
116-
// Load the data from URL with the schema.
117-
Table table = Table.fromSource(url, schema, DataSourceFormat.getDefaultCsvFormat());
112+
Table table = Table.fromSource(url);
118113

119114
// Iterate through rows
120-
Iterator<Object[]> iter = table.iterator();
115+
Iterator<Map<String, Object>> iter = table.keyedIterator();
121116
while(iter.hasNext()){
122-
Object[] row = iter.next();
123-
System.out.println(Arrays.toString(row));
117+
Map<String, Object> row = iter.next();
118+
System.out.println(row);
124119
}
125120

126-
List<Object[]> allData = table.read();
127-
128-
// [1, foo]
129-
// [2, bar]
130-
// [3, baz]
121+
// {id=1, title=foo}
122+
// {id=2, title=bar}
123+
// {id=3, title=baz}
131124
```
132125

126+
133127
### Parse a CSV with a Schema, extended version
134128

135129
If you have a schema, you can input it as parameter when creating the `Table` instance so that the [data](https://raw.githubusercontent.com/frictionlessdata/tableschema-java/master/src/test/resources/fixtures/employee_data.csv) from the CSV will be cast into the field types defined in the schema:
@@ -165,12 +159,12 @@ URL url = new URL("https://raw.githubusercontent.com/frictionlessdata/tableschem
165159
"/src/test/resources/fixtures/data/employee_data.csv");
166160
Table table = Table.fromSource(url, schema, DataSourceFormat.getDefaultCsvFormat());
167161

168-
Iterator<Object[]> iter = table.iterator(false, false, true, false);
162+
Iterator<Object> iter = table.iterator(false, false, true, false);
169163
while(iter.hasNext()){
170164

171165
// The fetched array will contain row values that have been cast into their
172166
// appropriate types as per field definitions in the schema.
173-
Object[] row = iter.next();
167+
Object[] row = (Object[])iter.next();
174168

175169
BigInteger id = (BigInteger)row[0];
176170
String name = (String)row[1];

‎pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<modelVersion>4.0.0</modelVersion>
44
<groupId>io.frictionlessdata</groupId>
55
<artifactId>tableschema-java</artifactId>
6-
<version>0.4.0-SNAPSHOT</version>
6+
<version>0.5.0-SNAPSHOT</version>
77
<packaging>jar</packaging>
88
<issueManagement>
99
<url>https://github.com/frictionlessdata/tableschema-java/issues</url>

‎src/main/java/io/frictionlessdata/tableschema/Table.java‎

Lines changed: 210 additions & 177 deletions
Large diffs are not rendered by default.

‎src/main/java/io/frictionlessdata/tableschema/exception/TableIOException.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
package io.frictionlessdata.tableschema.exception;
2+
3+
public class TableIOException extends TableSchemaException{
4+
5+
public TableIOException() {}
6+
7+
public TableIOException(String msg) {
8+
super(msg);
9+
}
10+
11+
public TableIOException(Throwable t) {
12+
super(t);
13+
}
14+
}

‎src/main/java/io/frictionlessdata/tableschema/field/ArrayField.java‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,15 @@
44
import io.frictionlessdata.tableschema.exception.InvalidCastException;
55
import io.frictionlessdata.tableschema.exception.JsonParsingException;
66
import io.frictionlessdata.tableschema.exception.TypeInferringException;
7+
import io.frictionlessdata.tableschema.schema.TypeInferrer;
78
import io.frictionlessdata.tableschema.util.JsonUtil;
89

910
import java.net.URI;
11+
import java.util.ArrayList;
12+
import java.util.Arrays;
13+
import java.util.List;
1014
import java.util.Map;
15+
import java.util.stream.Collectors;
1116

1217

1318
public class ArrayField extends Field<Object[]> {
@@ -36,8 +41,20 @@ public Object[] parseValue(String value, String format, Map<String, Object> opti
3641
}
3742

3843
@Override
39-
public String formatValueAsString(Object[] value, String format, Map<String, Object> options) throws InvalidCastException, ConstraintsException {
40-
return JsonUtil.getInstance().serialize(value);
44+
public String formatValueAsString(Object[] value, String format, Map<String, Object> options)
45+
throws InvalidCastException, ConstraintsException {
46+
List<String> vals = new ArrayList<>();
47+
String val;
48+
for (Object o : value) {
49+
if (o instanceof String) {
50+
val = "\""+o+"\"";
51+
} else {
52+
Field f = FieldInferrer.infer(o);
53+
val = f.formatValueAsString(o);
54+
}
55+
vals.add(val);
56+
}
57+
return "[" + vals.stream().collect(Collectors.joining(",")) +"]";
4158
}
4259

4360

‎src/main/java/io/frictionlessdata/tableschema/field/DateField.java‎

Lines changed: 23 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ public LocalDate parseValue(String value, String format, Map<String, Object> opt
3737
Pattern pattern = Pattern.compile(REGEX_DATE);
3838
Matcher matcher = pattern.matcher(value);
3939

40-
if(matcher.matches()){
40+
if (matcher.matches() && ((null == format) || format.equals("default"))) {
4141
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd");
4242
TemporalAccessor dt = formatter.parse(value);
4343

@@ -52,39 +52,35 @@ public LocalDate parseValue(String value, String format, Map<String, Object> opt
5252
by Python / C standard strptime using <PATTERN>). Example for "format": "%d/%m/%y" which
5353
would correspond to dates like: 30/11/14
5454
*/
55-
String regex = format
56-
.replaceAll("%d", "dd")
57-
.replaceAll("%m", "MM")
58-
.replaceAll("%y", "yy");
55+
String regex = parseDateFormat(format);
5956
DateTimeFormatter formatter = DateTimeFormatter.ofPattern(regex);
60-
try {
61-
return LocalDate.from(formatter.parse(value));
62-
} catch (DateTimeParseException ex) {
63-
regex = format
64-
.replaceAll("%d", "dd")
65-
.replaceAll("%m", "MM")
66-
.replaceAll("%y", "yyyy");
67-
68-
formatter = DateTimeFormatter.ofPattern(regex);
69-
return LocalDate.from(formatter.parse(value));
70-
}
57+
return LocalDate.from(formatter.parse(value));
7158
}
7259
throw new TypeInferringException();
7360
}
7461
}
7562

63+
7664
@Override
7765
public Object formatValueForJson(LocalDate value) throws InvalidCastException, ConstraintsException {
7866
if (null == value)
7967
return null;
80-
return value.toString();
68+
if ((null == format) || format.equals("default")) {
69+
return DateTimeFormatter.ISO_DATE.format(value);
70+
}
71+
String translatedFormat = parseDateFormat(format);
72+
return value.format(DateTimeFormatter.ofPattern(translatedFormat));
8173
}
8274

8375
@Override
8476
public String formatValueAsString(LocalDate value, String format, Map<String, Object> options) throws InvalidCastException, ConstraintsException {
8577
if (null == value)
86-
return null;
87-
return value.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"));
78+
return null;
79+
if ((null == format) || format.equals("default")) {
80+
return DateTimeFormatter.ISO_DATE.format(value);
81+
}
82+
String translatedFormat = parseDateFormat(format);
83+
return value.format(DateTimeFormatter.ofPattern(translatedFormat));
8884
}
8985

9086

@@ -102,5 +98,13 @@ LocalDate checkMinimumContraintViolated(LocalDate value) {
10298
return null;
10399
}
104100

101+
private String parseDateFormat(String cString) {
102+
String retVal = cString;
103+
retVal = retVal.replaceAll("%d", "dd");
104+
retVal = retVal.replaceAll("%m", "MM");
105+
retVal = retVal.replaceAll("%y", "yy");
106+
retVal = retVal.replaceAll("%Y", "yyyy");
105107

108+
return retVal;
109+
}
106110
}

‎src/main/java/io/frictionlessdata/tableschema/field/Field.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -612,7 +612,7 @@ T castValue(String value, boolean enforceConstraints, Map<String, Object> option
612612
if(enforceConstraints && this.constraints != null){
613613
Map<String, Object> violatedConstraints = checkConstraintViolations(castValue);
614614
if(!violatedConstraints.isEmpty()){
615-
throw new ConstraintsException("Violated "+ violatedConstraints.size()+" contstraints");
615+
throw new ConstraintsException("Violated "+ violatedConstraints.size()+" constraints");
616616
}
617617
}
618618

‎src/main/java/io/frictionlessdata/tableschema/field/FieldInferrer.java‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
package io.frictionlessdata.tableschema.field;
2+
3+
import com.fasterxml.jackson.databind.JsonNode;
4+
import com.google.common.util.concurrent.AtomicDouble;
5+
import io.frictionlessdata.tableschema.util.JsonUtil;
6+
import org.locationtech.jts.geom.Coordinate;
7+
8+
import java.lang.reflect.Array;
9+
import java.math.BigDecimal;
10+
import java.math.BigInteger;
11+
import java.time.LocalDate;
12+
import java.time.LocalTime;
13+
import java.util.Arrays;
14+
import java.util.Base64;
15+
import java.util.Collection;
16+
import java.util.UUID;
17+
import java.util.concurrent.atomic.AtomicInteger;
18+
import java.util.concurrent.atomic.AtomicLong;
19+
20+
public class FieldInferrer {
21+
22+
public static Field<?> infer(Object o) {
23+
if (null == o) {
24+
return new AnyField("Any");
25+
}
26+
Class<?> oClass = o.getClass();
27+
Field<?> f = generateNumberField(oClass, oClass.getName());
28+
if (null != f)
29+
return f;
30+
f = generateStringField(oClass, oClass.getName());
31+
if (null != f)
32+
return f;
33+
if (LocalDate.class.equals(oClass)){
34+
f = new DateField(oClass.getName());
35+
} else if (LocalTime.class.equals(oClass)){
36+
f = new TimeField(oClass.getName());
37+
} else if (Boolean.class.equals(oClass)){
38+
f = new BooleanField(oClass.getName());
39+
} else if (Coordinate.class.isAssignableFrom(oClass)) {
40+
f = new GeopointField(oClass.getName());
41+
} else if (Collection.class.isAssignableFrom(oClass)
42+
|| (o instanceof Array)) {
43+
f = new ArrayField(oClass.getName());
44+
}
45+
if (oClass.equals(JsonNode.class)) {
46+
f = new ObjectField(oClass.getName());
47+
} else if (oClass.equals(Object.class)) {
48+
f = new ObjectField(oClass.getName());
49+
}
50+
return f;
51+
}
52+
53+
private static Field<?> generateStringField(Class<?> declaredClass, String name) {
54+
Field<?> f = null;
55+
if (UUID.class.equals(declaredClass)){
56+
f = new StringField(name);
57+
f.setFormat("uuid");
58+
} else if (byte[].class.equals(declaredClass)){
59+
f = new StringField(name);
60+
f.setFormat("binary");
61+
} else if (String.class.equals(declaredClass)) {
62+
f = new StringField(name);
63+
}
64+
return f;
65+
}
66+
67+
private static Field<?> generateNumberField(Class<?> declaredClass, String name) {
68+
Field<?> field = null;
69+
if ((declaredClass.equals(Integer.class))
70+
|| (declaredClass.equals(int.class))
71+
|| (declaredClass.equals(Long.class))
72+
|| (declaredClass.equals(long.class))
73+
|| (declaredClass.equals(Short.class))
74+
|| (declaredClass.equals(short.class))
75+
|| (declaredClass.equals(Byte.class))
76+
|| (declaredClass.equals(byte.class))
77+
|| (declaredClass.equals(BigInteger.class))
78+
|| (declaredClass.equals(AtomicInteger.class))
79+
|| (declaredClass.equals(AtomicLong.class))) {
80+
field = new IntegerField(name);
81+
} else {
82+
if ((declaredClass.equals(Float.class))
83+
|| (declaredClass.equals(float.class))
84+
|| (declaredClass.equals(Double.class))
85+
|| (declaredClass.equals(double.class))
86+
|| (declaredClass.equals(BigDecimal.class))
87+
|| (declaredClass.equals(AtomicDouble.class))) {
88+
field = new NumberField(name);
89+
}
90+
}
91+
return field;
92+
}
93+
}

‎src/main/java/io/frictionlessdata/tableschema/field/GeopointField.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,9 @@ public double[] parseValue(String value, String format, Map<String, Object> opti
8181
}
8282

8383
}catch(Exception e){
84+
if (e instanceof TypeInferringException) {
85+
throw e;
86+
}
8487
throw new TypeInferringException(e);
8588
}
8689
}

‎src/main/java/io/frictionlessdata/tableschema/field/ObjectField.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,7 @@ public ObjectField(String name, String format, String title, String description,
2929
public Map<String, Object> parseValue(String value, String format, Map<String, Object> options)
3030
throws TypeInferringException {
3131
try {
32-
return JsonUtil.getInstance().deserialize(value, new TypeReference<Map<String, Object>>() {
33-
});
32+
return JsonUtil.getInstance().deserialize(value, new TypeReference<Map<String, Object>>() {});
3433
} catch (JsonParsingException ex) {
3534
throw new TypeInferringException(ex);
3635
}

0 commit comments

Comments
 (0)