Skip to content

Commit 1c41761

Browse files
thatbudakguythatbudakguy
committed
Update README; add note about OGM_SKIP_RESTRICTED
1 parent 256bd03 commit 1c41761

1 file changed

Lines changed: 114 additions & 91 deletions

File tree

‎README.md‎

Lines changed: 114 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,20 @@
11
# GeoCombine
22

33
![CI](https://github.com/OpenGeoMetadata/GeoCombine/actions/workflows/ruby.yml/badge.svg)
4-
| [![Coverage Status](https://img.shields.io/badge/coverage-95%25-brightgreen)]()
5-
| [![Gem Version](https://img.shields.io/gem/v/geo_combine.svg)](https://github.com/OpenGeoMetadata/GeoCombine/releases)
4+
[![Coverage Status](https://img.shields.io/badge/coverage-95%25-brightgreen)]()
5+
[![Gem Version](https://img.shields.io/gem/v/geo_combine.svg)](https://github.com/OpenGeoMetadata/GeoCombine/releases)
66

77
A Ruby toolkit for managing geospatial metadata, including:
88

9-
- tasks for cloning, updating, and indexing OpenGeoMetadata metadata
10-
- library for converting metadata between standards
9+
- harvesting and indexing records from OpenGeoMetadata
10+
- harvesting records directly from a GeoBlacklight instance
11+
- converting geospatial metadata records between standards
1112

1213
## Installation
1314

15+
> [!TIP]
16+
> If you're using GeoBlacklight, GeoCombine is already bundled with your installation – you can skip this section.
17+
1418
Add this line to your application's `Gemfile`:
1519

1620
```ruby
@@ -20,175 +24,131 @@ gem 'geo_combine'
2024
And then execute:
2125

2226
```sh
23-
$ bundle install
27+
bundle install
2428
```
2529

2630
Or install it yourself as:
2731

2832
```sh
29-
$ gem install geo_combine
33+
gem install geo_combine
3034
```
3135

3236
## Usage
3337

34-
### Converting metadata
35-
36-
#### Converting metadata into GeoBlacklight JSON
37-
38-
GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
39-
40-
```ruby
41-
GeoCombine::Iso19139 # ISO 19139 XML
42-
GeoCombine::OGP # OpenGeoPortal JSON
43-
GeoCombine::Fgdc # FGDC XML
44-
GeoCombine::EsriOpenData # Esri Open Data Portal JSON
45-
GeoCombine::CkanMetadata # CKAN JSON
46-
```
47-
48-
An example for converting an ISO 19139 XML record:
49-
50-
```ruby
51-
# Create a new ISO19139 object
52-
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
53-
54-
# Convert to GeoBlacklight's metadata format
55-
> iso_metadata.to_geoblacklight
56-
57-
# Output it as JSON instead of a Ruby hash
58-
> iso_metadata.to_geoblacklight.to_json
59-
```
60-
61-
Some formats also support conversion into HTML for display in a web browser:
62-
63-
```ruby
64-
# Create a new ISO19139 object
65-
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
66-
67-
# Convert ISO to HTML
68-
> iso_metadata.to_html
69-
```
70-
71-
### Migrating metadata
72-
73-
You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
74-
75-
Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
76-
77-
```ruby
78-
# Load a record in geoblacklight v1 schema
79-
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
80-
81-
# Migrate it to Aardvark schema
82-
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
83-
```
84-
85-
Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
86-
87-
```ruby
88-
# You can store this mapping as a JSON or CSV file and load it into a hash
89-
id_map = {
90-
'My Collection 1' => 'institution:my-collection-1',
91-
'My Collection 2' => 'institution:my-collection-2'
92-
}
93-
94-
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
95-
```
96-
9738
### Downloading metadata from OpenGeoMetadata
9839

99-
#### Logging
40+
#### Schema version
10041

101-
Some of the tools and scripts in this gem use Ruby's `Logger` class to print information to `$stderr`. By default, the log level is set to `Logger::INFO`. For more verbose information, you can set the `LOG_LEVEL` environment variable to `DEBUG`:
42+
By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the `SCHEMA_VERSION` environment variable:
10243

10344
```sh
104-
$ export LOG_LEVEL=DEBUG
45+
# Only fetch and index schema version 1.0 records
46+
export SCHEMA_VERSION=1.0
10547
```
10648

107-
#### Schema version
49+
#### Restricted records
10850

109-
By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the `SCHEMA_VERSION` environment variable:
51+
By default, GeoCombine will only fetch and index records that are not marked as restricted. If you want to include restricted records, you can set the `OGM_SKIP_RESTRICTED` environment variable to `false`:
11052

11153
```sh
112-
# Only fetch and index schema version 1.0 records
113-
$ export SCHEMA_VERSION=1.0
54+
export OGM_SKIP_RESTRICTED=false
11455
```
11556

11657
#### Clone OpenGeoMetadata repositories locally
11758

11859
```sh
119-
$ bundle exec rake geocombine:clone
60+
bundle exec rake geocombine:clone
12061
```
12162

12263
Will clone all OpenGeoMetadata repositories containing metadata matching the `SCHEMA_VERSION` into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
12364

12465
```sh
125-
$ OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
66+
OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
12667
```
12768

12869
You can also specify a single repository:
12970

13071
```sh
131-
$ bundle exec rake geocombine:clone[edu.stanford.purl]
72+
bundle exec rake geocombine:clone[edu.stanford.purl]
13273
```
13374

13475
_Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
13576

13677
```sh
137-
$ bundle exec rake geocombine:clone\[edu.stanford.purl\]
78+
bundle exec rake geocombine:clone\[edu.stanford.purl\]
13879
```
13980

14081
#### Update local OpenGeoMetadata repositories
14182

14283
```sh
143-
$ bundle exec rake geocombine:pull
84+
bundle exec rake geocombine:pull
14485
```
14586

14687
Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`).
14788

14889
You can also specify a single repository:
14990

15091
```sh
151-
$ bundle exec rake geocombine:pull[edu.stanford.purl]
92+
bundle exec rake geocombine:pull[edu.stanford.purl]
15293
```
15394

15495
_Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
15596

15697
```sh
157-
$ bundle exec rake geocombine:pull\[edu.stanford.purl\]
98+
bundle exec rake geocombine:pull\[edu.stanford.purl\]
15899
```
159100

160101
#### Index GeoBlacklight documents
161102

103+
> [!TIP]
104+
> If you're using GeoBlacklight, GeoCombine will automatically detect the path to your Solr index if you run it in the directory where GeoBlacklight is installed.
105+
162106
To index into Solr, GeoCombine requires a Solr instance that is running the
163107
[GeoBlacklight schema](https://github.com/geoblacklight/geoblacklight):
164108

165109
```sh
166-
$ bundle exec rake geocombine:index
110+
bundle exec rake geocombine:index
167111
```
168112

169113
If Blacklight is installed in the ruby environment and a solr index is configured, the rake task will use the solr index configured in the Blacklight application (this is the case when invoking GeoCombine from your GeoBlacklight installation). If Blacklight is unavailable, the rake task will try to find a Solr instance running at `http://localhost:8983/solr/blacklight-core`.
170114

171115
You can also set the Solr instance URL using `SOLR_URL`:
172116

173117
```sh
174-
$ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
118+
SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
119+
```
120+
121+
#### Logging
122+
123+
Some of the tools and scripts in this gem use Ruby's `Logger` class to print information to `$stderr`. By default, the log level is set to `Logger::INFO`. For more verbose information, you can set the `LOG_LEVEL` environment variable to `DEBUG`:
124+
125+
```sh
126+
export LOG_LEVEL=DEBUG
175127
```
176128

177129
### Indexing local documents
178130

179-
To index an arbitrary collection of records in a custom directory, run one of the following:
131+
To index a collection of records in a local directory, you can pass the directory as a `rake` argument:
132+
133+
```sh
134+
bundle exec rake geocombine:index\[/path/to/your/files\]
135+
```
136+
137+
Or you can provide it as `OGM_PATH`:
180138

181139
```sh
182-
rake geocombine:index\[/path/to/your/files\]
183140
OGM_PATH=/path/to/your/files rake geocombine:index
184141
```
185142

186-
### Indexing a single document
143+
For a single file, it works the same way:
187144

188-
To index a single arbitrary record, run one of the following:
145+
```sh
146+
bundle exec rake geocombine:index\[/path/to/your/file.json\]
147+
```
148+
149+
Or:
189150

190151
```sh
191-
rake geocombine:index\[/path/to/your/file.json\]
192152
OGM_PATH=/path/to/your/file.json rake geocombine:index
193153
```
194154

@@ -253,12 +213,75 @@ GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
253213
end
254214
```
255215

216+
### Converting metadata
217+
218+
#### Converting metadata into GeoBlacklight JSON
219+
220+
GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
221+
222+
```ruby
223+
GeoCombine::Iso19139 # ISO 19139 XML
224+
GeoCombine::OGP # OpenGeoPortal JSON
225+
GeoCombine::Fgdc # FGDC XML
226+
GeoCombine::EsriOpenData # Esri Open Data Portal JSON
227+
GeoCombine::CkanMetadata # CKAN JSON
228+
```
229+
230+
An example for converting an ISO 19139 XML record:
231+
232+
```ruby
233+
# Create a new ISO19139 object
234+
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
235+
236+
# Convert to GeoBlacklight's metadata format
237+
> iso_metadata.to_geoblacklight
238+
239+
# Output it as JSON instead of a Ruby hash
240+
> iso_metadata.to_geoblacklight.to_json
241+
```
242+
243+
Some formats also support conversion into HTML for display in a web browser:
244+
245+
```ruby
246+
# Create a new ISO19139 object
247+
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
248+
249+
# Convert ISO to HTML
250+
> iso_metadata.to_html
251+
```
252+
253+
### Migrating metadata
254+
255+
You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
256+
257+
Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
258+
259+
```ruby
260+
# Load a record in geoblacklight v1 schema
261+
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
262+
263+
# Migrate it to Aardvark schema
264+
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
265+
```
266+
267+
Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
268+
269+
```ruby
270+
# You can store this mapping as a JSON or CSV file and load it into a hash
271+
id_map = {
272+
'My Collection 1' => 'institution:my-collection-1',
273+
'My Collection 2' => 'institution:my-collection-2'
274+
}
275+
276+
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
277+
```
278+
256279
## Tests
257280

258281
To run the tests, use:
259282

260283
```sh
261-
$ bundle exec rake spec
284+
bundle exec rake spec
262285
```
263286

264287
## Contributing

0 commit comments

Comments
 (0)