You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: app/docs/importing-data.md
+51-2Lines changed: 51 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,7 @@ If you know how to import data already and need a reference of the whole exchang
11
11
-[`ConvertArrayToStream`](#convertarraytostream)
12
12
-[`ConvertToUTF8`](#converttoutf8)
13
13
-[`DebugLog`](#debuglog)
14
+
-[`Filter`](#filter)
14
15
-[`HTTPDownload`](#httpdownload)
15
16
-[`Limit`](#limit)
16
17
-[`MultiHTTPDownload`](#multihttpdownload)
@@ -21,6 +22,8 @@ If you know how to import data already and need a reference of the whole exchang
21
22
-[`Split`](#split)
22
23
-[`TransformData`](#transformdata)
23
24
-[`TransformScript`](#transformscript)
25
+
-[`UploadRemoteFileToS3`](#uploadremotefiletos3)
26
+
-[`UpsertImage`](#upsertimage)
24
27
-[`UpsertPlace`](#upsertplace)
25
28
-[`UpsertEquipment`](#upsertequipment)
26
29
-[`UpsertDisruption`](#upsertdisruption)
@@ -99,7 +102,11 @@ We support the following stream processing units:
99
102
100
103
### `ConvertArrayToStream`
101
104
102
-
Reads Array objects on the input, takes their elements and outputs them as single objects.
105
+
Gets arrays from the input, takes the array elements and outputs them as single objects.
106
+
107
+
#### Parameters
108
+
109
+
-`path` (optional): Set this to a key path to allow processing JavaScript objects that wrap arrays, instead of processing arrays directly. If you use this parameter, you must supply a key path (in dot-notation) where to find the array in each processed object. If you omit this parameter, input chunks must be arrays, otherwise objects.
103
110
104
111
### `ConvertToUTF8`
105
112
@@ -113,6 +120,16 @@ Converts downloaded data into UTF8 (the format of the database). [Input formats]
113
120
114
121
Displays the first and the last chunk read from input in the UI.
115
122
123
+
### `Filter`
124
+
125
+
Filters incoming JavaScript objects by truthiness of a property, or by equality of a property to a given value. Outputs matching objects, discards all other objects.
126
+
127
+
#### Parameters
128
+
129
+
-`path`: key path of the property in incoming objects to match when filtering, in dot-notation.
130
+
-`negate` (optional): Inverts the filtering: discards matching objects, and outputs only objects that do *not* match the filter.
131
+
-`expectedValue` (optional): If given, the stream compares this value with the property value at the given `path` in each input object. The filter counts the object as a match if both values are equal.
132
+
116
133
### `HTTPDownload`
117
134
118
135
Starts a single HTTP GET request to downloads data from a given URL.
@@ -234,16 +251,40 @@ Transforms each input chunk or object into something else, using a JavaScript ex
234
251
235
252
-`javascript`: JavaScript expression like `Array.from({length: d}, (v, k) => k + 1)`. Supports ES2015. `d` is predefined as the input chunk/object. Evaluates the expression and writes its result as new chunk/object to the output.
236
253
254
+
### `UploadRemoteFileToS3`
255
+
256
+
For each given input object, stream a file from a remote URL to our Amazon S3 storage. An admin must enable this feature for the data source before you can use this stream unit.
257
+
258
+
#### Parameters
259
+
260
+
-`remotePathProperty`: Key path (in dot-notation) where to find the remote path on our S3 in any given input object.
261
+
-`mimeTypeProperty`: Key path (in dot-notation) where to find the mime type to set on S3 in any given input object.
262
+
-`isUploadedFlagProperty`: Key path (in dot-notation) where to find a flag that the object is already uploaded to S3. If the value of this property is `true`, the server will skip the upload for the input object.
263
+
-`sourceUrlTemplateDataProperty`: Key path (in dot-notation) where to find the property to use as input for string template-based URLs.
264
+
-`sourceUrlProperty`: Key path (in dot-notation) where to find the URL of the source file to stream.
265
+
266
+
### `UpsertImage`
267
+
268
+
Inserts an `Image` document into the database. Upserting will link the image with another target document, e.g. a `Place`.
269
+
270
+
#### Parameters
271
+
272
+
-`refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
273
+
237
274
### `UpsertPlace`
238
275
239
-
Inserts an object into the database as place. Upserting will link the place with the source you have created in the accessibility.cloud web UI, and will make it available to API users when…
276
+
Inserts a `PlaceInfo` document into the database. Upserting will link the place with the source you have created in the accessibility.cloud web UI, and will make it available to API users when…
240
277
241
278
- you have accepted the terms and conditions for the data source's organization
242
279
- your data source is not in draft state anymore (see the source's 'Settings' tab in the web UI)
243
280
- the app token used to query the API belongs to an app by an organization that is allowed to read your source's data (you can either allow everyone or specific organizations to use your data source in the 'Settings' tab).
244
281
245
282
`UpsertPlace` is usually the last unit in the stream chain, but it outputs status data for each processed chunk that you can use for debugging.
246
283
284
+
#### Parameters
285
+
286
+
-`refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
287
+
247
288
### `UpsertEquipment`
248
289
249
290
Inserts a record into the database that describes an equipment or facility of a place, for example an elevator or escalator.
@@ -259,6 +300,10 @@ This works like `UpsertPlace`, but using it marks the source as a data source fo
259
300
- accessibility.cloud will show the imported equipment/facilities on the overview page of associated place data source.
260
301
- The accessibility.cloud `/placeInfos` API will include equipment/facility data as `properties.equipmentInfos` property.
261
302
303
+
#### Parameters
304
+
305
+
-`refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
306
+
262
307
### `UpsertDisruption`
263
308
264
309
Inserts a record into the database that describes a (possibly timed) disruption of an equipment object.
@@ -287,6 +332,10 @@ This works like `UpsertPlace`, but using it marks the source as a data source fo
287
332
-`/placeInfos` API responses will include disruption data if you supply a `includeRelated=equipmentInfos.disruptions` query parameter.
288
333
-`/equipmentInfos` API responses will include disruption data if you supply a `includeRelated=disruptions` query parameter.
289
334
335
+
#### Parameters
336
+
337
+
-`refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
338
+
290
339
## Convenience functions and libraries
291
340
292
341
You can use the following popular libraries for your transform code:
0 commit comments