Skip to content

Commit b028145

Browse files
committed
Add documentation for new import functions
1 parent 3f331a7 commit b028145

1 file changed

Lines changed: 51 additions & 2 deletions

File tree

app/docs/importing-data.md

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ If you know how to import data already and need a reference of the whole exchang
1111
- [`ConvertArrayToStream`](#convertarraytostream)
1212
- [`ConvertToUTF8`](#converttoutf8)
1313
- [`DebugLog`](#debuglog)
14+
- [`Filter`](#filter)
1415
- [`HTTPDownload`](#httpdownload)
1516
- [`Limit`](#limit)
1617
- [`MultiHTTPDownload`](#multihttpdownload)
@@ -21,6 +22,8 @@ If you know how to import data already and need a reference of the whole exchang
2122
- [`Split`](#split)
2223
- [`TransformData`](#transformdata)
2324
- [`TransformScript`](#transformscript)
25+
- [`UploadRemoteFileToS3`](#uploadremotefiletos3)
26+
- [`UpsertImage`](#upsertimage)
2427
- [`UpsertPlace`](#upsertplace)
2528
- [`UpsertEquipment`](#upsertequipment)
2629
- [`UpsertDisruption`](#upsertdisruption)
@@ -99,7 +102,11 @@ We support the following stream processing units:
99102

100103
### `ConvertArrayToStream`
101104

102-
Reads Array objects on the input, takes their elements and outputs them as single objects.
105+
Gets arrays from the input, takes the array elements and outputs them as single objects.
106+
107+
#### Parameters
108+
109+
- `path` (optional): Set this to a key path to allow processing JavaScript objects that wrap arrays, instead of processing arrays directly. If you use this parameter, you must supply a key path (in dot-notation) where to find the array in each processed object. If you omit this parameter, input chunks must be arrays, otherwise objects.
103110

104111
### `ConvertToUTF8`
105112

@@ -113,6 +120,16 @@ Converts downloaded data into UTF8 (the format of the database). [Input formats]
113120

114121
Displays the first and the last chunk read from input in the UI.
115122

123+
### `Filter`
124+
125+
Filters incoming JavaScript objects by truthiness of a property, or by equality of a property to a given value. Outputs matching objects, discards all other objects.
126+
127+
#### Parameters
128+
129+
- `path`: key path of the property in incoming objects to match when filtering, in dot-notation.
130+
- `negate` (optional): Inverts the filtering: discards matching objects, and outputs only objects that do *not* match the filter.
131+
- `expectedValue` (optional): If given, the stream compares this value with the property value at the given `path` in each input object. The filter counts the object as a match if both values are equal.
132+
116133
### `HTTPDownload`
117134

118135
Starts a single HTTP GET request to downloads data from a given URL.
@@ -234,16 +251,40 @@ Transforms each input chunk or object into something else, using a JavaScript ex
234251

235252
- `javascript`: JavaScript expression like `Array.from({length: d}, (v, k) => k + 1)`. Supports ES2015. `d` is predefined as the input chunk/object. Evaluates the expression and writes its result as new chunk/object to the output.
236253

254+
### `UploadRemoteFileToS3`
255+
256+
For each given input object, stream a file from a remote URL to our Amazon S3 storage. An admin must enable this feature for the data source before you can use this stream unit.
257+
258+
#### Parameters
259+
260+
- `remotePathProperty`: Key path (in dot-notation) where to find the remote path on our S3 in any given input object.
261+
- `mimeTypeProperty`: Key path (in dot-notation) where to find the mime type to set on S3 in any given input object.
262+
- `isUploadedFlagProperty`: Key path (in dot-notation) where to find a flag that the object is already uploaded to S3. If the value of this property is `true`, the server will skip the upload for the input object.
263+
- `sourceUrlTemplateDataProperty`: Key path (in dot-notation) where to find the property to use as input for string template-based URLs.
264+
- `sourceUrlProperty`: Key path (in dot-notation) where to find the URL of the source file to stream.
265+
266+
### `UpsertImage`
267+
268+
Inserts an `Image` document into the database. Upserting will link the image with another target document, e.g. a `Place`.
269+
270+
#### Parameters
271+
272+
- `refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
273+
237274
### `UpsertPlace`
238275

239-
Inserts an object into the database as place. Upserting will link the place with the source you have created in the accessibility.cloud web UI, and will make it available to API users when…
276+
Inserts a `PlaceInfo` document into the database. Upserting will link the place with the source you have created in the accessibility.cloud web UI, and will make it available to API users when…
240277

241278
- you have accepted the terms and conditions for the data source's organization
242279
- your data source is not in draft state anymore (see the source's 'Settings' tab in the web UI)
243280
- the app token used to query the API belongs to an app by an organization that is allowed to read your source's data (you can either allow everyone or specific organizations to use your data source in the 'Settings' tab).
244281

245282
`UpsertPlace` is usually the last unit in the stream chain, but it outputs status data for each processed chunk that you can use for debugging.
246283

284+
#### Parameters
285+
286+
- `refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
287+
247288
### `UpsertEquipment`
248289

249290
Inserts a record into the database that describes an equipment or facility of a place, for example an elevator or escalator.
@@ -259,6 +300,10 @@ This works like `UpsertPlace`, but using it marks the source as a data source fo
259300
- accessibility.cloud will show the imported equipment/facilities on the overview page of associated place data source.
260301
- The accessibility.cloud `/placeInfos` API will include equipment/facility data as `properties.equipmentInfos` property.
261302

303+
#### Parameters
304+
305+
- `refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
306+
262307
### `UpsertDisruption`
263308

264309
Inserts a record into the database that describes a (possibly timed) disruption of an equipment object.
@@ -287,6 +332,10 @@ This works like `UpsertPlace`, but using it marks the source as a data source fo
287332
- `/placeInfos` API responses will include disruption data if you supply a `includeRelated=equipmentInfos.disruptions` query parameter.
288333
- `/equipmentInfos` API responses will include disruption data if you supply a `includeRelated=disruptions` query parameter.
289334

335+
#### Parameters
336+
337+
- `refetchAfterUpsert` (optional): Set this to `true` to output a refetched document after each upsert. This document will contain all properties in the DB, even the ones that you have not included in the upsert modifier.
338+
290339
## Convenience functions and libraries
291340

292341
You can use the following popular libraries for your transform code:

0 commit comments

Comments
 (0)