Skip to content

Commit 0aa88a1

Browse files
author
R. S. Doiel
committed
commiting to main
1 parent 94acbca commit 0aa88a1

9 files changed

Lines changed: 32 additions & 34 deletions

README.md

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,10 @@
22
datatools
33
=========
44

5-
_datatools_ is a rich collection of command line programs targetting
5+
_datatools_ is a rich collection of command line programs targeting
66
data conversion, cleanup and analysis directly from your favorite
7-
POSIX shell. It has proven useful for data collaborations where
8-
individual members of a project may prefer different toolsets in their
7+
POSIX shell or Powershell. It has proven useful for data collaborations where
8+
individual members of a project may prefer different tool sets in their
99
analysis (e.g. Julia, R, Python) but want to work from a common baseline.
1010
It also has been used intensively for internal reporting from various
1111
Caltech Library metadata sources.
@@ -18,10 +18,10 @@ The tools fall into three broad categories
1818

1919
See [user manual](user-manual.md) for a complete list of the command line
2020
programs. The data transformation tools include support for formats such as
21-
Excel XML, csv, tab delimited files, json, yaml and toml.
21+
Excel XML, csv, tab delimited files, json, yaml, toml and url encoding/decoding.
2222

23-
Compiled versions of the datatools collection are provided for Linux
24-
(amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7).
23+
Compiled versions of the datatools collection are provided for Linux (aarch64/amd64),
24+
Mac OS X (aarch64/amd64), Windows 10 (aarch64/amd64) and Raspberry Pi OS (aarch64).
2525
See https://github.com/caltechlibrary/datatools/releases.
2626

2727
Use "-help" option for a full list of options for each utility (e.g. `csv2json -help`).
@@ -30,8 +30,8 @@ Data transformation
3030
-------------------
3131

3232
The tooling around transformation includes data conversion. These
33-
include tools that work with CSV, tab delimited, JSON, TOML, YAML
34-
and Excel XML.
33+
include tools that work with CSV, tab delimited, JSON, TOML, YAML,
34+
Excel XML, and url encoded text.
3535

3636
There is also tooling to change data shapes using JSON as the
3737
intermediate data format.
@@ -41,22 +41,21 @@ For the shell
4141

4242
Various utilities for simplifying work on the command line.
4343

44-
+ [findfile](docs/findfile/) - find files based on prefix, suffix or contained string
45-
+ [finddir](docs/finddir/) - find directories based on prefix, suffix or contained string
46-
+ [mergepath](docs/mergepath/) - prefix, append, clip path variables
47-
+ [range](docs/range/) - emit a range of integers (useful for numbered loops in Bash)
48-
+ [reldate](docs/reldate/) - display a relative date in YYYY-MM-DD format
49-
+ [reltime](docs/reltime/) - display a relative time in 24 hour notation, HH:MM:SS format
50-
+ [timefmt](docs/timefmt/) - format a time value based on Golang's time format language
51-
+ [urlparse](docs/urlparse/) - split a URL into parts
44+
- [mergepath](docs/mergepath/) - prefix, append, clip path variables
45+
- [reldocpath](reldocpath.1.md) - calculates relative paths given do paths
46+
- [range](docs/range/) - emit a range of integers (useful for numbered loops in Bash)
47+
- [reldate](docs/reldate/) - display a relative date in YYYY-MM-DD format
48+
- [reltime](docs/reltime/) - display a relative time in 24 hour notation, HH:MM:SS format
49+
- [timefmt](docs/timefmt/) - format a time value based on Golang's time format language
50+
- [urlparse](docs/urlparse/) - split a URL into parts
5251

5352
For strings
5453
-----------
5554

5655
_datatools_ provides the [string](docs/string/) command for working with
5756
text strings (limited to memory available). This is commonly needed when
5857
cleanup data for analysis. The _string_ command was created for when the
59-
old Unix standbys- grep, awk, sed, tr are unwieldly or inconvenient.
58+
old Unix standbys- grep, awk, sed, tr are unwieldy or inconvenient.
6059
_string_ provides operations are common in most language like, trimming,
6160
splitting, and transforming letter case. The _string_ command also makes
6261
it easy to join JSON string arrays into single a string using a delimiter

index.html

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,12 @@
2626
<section>
2727
<h1 id="datatools">datatools</h1>
2828
<p><em>datatools</em> is a rich collection of command line programs
29-
targetting data conversion, cleanup and analysis directly from your
30-
favorite POSIX shell. It has proven useful for data collaborations where
31-
individual members of a project may prefer different toolsets in their
32-
analysis (e.g. Julia, R, Python) but want to work from a common
33-
baseline. It also has been used intensively for internal reporting from
34-
various Caltech Library metadata sources.</p>
29+
targeting data conversion, cleanup and analysis directly from your
30+
favorite POSIX shell or Powershell. It has proven useful for data
31+
collaborations where individual members of a project may prefer
32+
different tool sets in their analysis (e.g. Julia, R, Python) but want
33+
to work from a common baseline. It also has been used intensively for
34+
internal reporting from various Caltech Library metadata sources.</p>
3535
<p>The tools fall into three broad categories</p>
3636
<ul>
3737
<li>data transformation and conversion</li>
@@ -41,28 +41,27 @@ <h1 id="datatools">datatools</h1>
4141
</ul>
4242
<p>See <a href="user-manual.html">user manual</a> for a complete list of
4343
the command line programs. The data transformation tools include support
44-
for formats such as Excel XML, csv, tab delimited files, json, yaml and
45-
toml.</p>
44+
for formats such as Excel XML, csv, tab delimited files, json, yaml,
45+
toml and url encoding/decoding.</p>
4646
<p>Compiled versions of the datatools collection are provided for Linux
47-
(amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7). See
47+
(aarch64/amd64), Mac OS X (aarch64/amd64), Windows 10 (aarch64/amd64)
48+
and Raspberry Pi OS (aarch64). See
4849
https://github.com/caltechlibrary/datatools/releases.</p>
4950
<p>Use “-help” option for a full list of options for each utility
5051
(e.g. <code>csv2json -help</code>).</p>
5152
<h2 id="data-transformation">Data transformation</h2>
5253
<p>The tooling around transformation includes data conversion. These
53-
include tools that work with CSV, tab delimited, JSON, TOML, YAML and
54-
Excel XML.</p>
54+
include tools that work with CSV, tab delimited, JSON, TOML, YAML, Excel
55+
XML, and url encoded text.</p>
5556
<p>There is also tooling to change data shapes using JSON as the
5657
intermediate data format.</p>
5758
<h2 id="for-the-shell">For the shell</h2>
5859
<p>Various utilities for simplifying work on the command line.</p>
5960
<ul>
60-
<li><a href="docs/findfile/">findfile</a> - find files based on prefix,
61-
suffix or contained string</li>
62-
<li><a href="docs/finddir/">finddir</a> - find directories based on
63-
prefix, suffix or contained string</li>
6461
<li><a href="docs/mergepath/">mergepath</a> - prefix, append, clip path
6562
variables</li>
63+
<li><a href="reldocpath.1.html">reldocpath</a> - calculates relative
64+
paths given do paths</li>
6665
<li><a href="docs/range/">range</a> - emit a range of integers (useful
6766
for numbered loops in Bash)</li>
6867
<li><a href="docs/reldate/">reldate</a> - display a relative date in
@@ -78,7 +77,7 @@ <h2 id="for-strings">For strings</h2>
7877
command for working with text strings (limited to memory available).
7978
This is commonly needed when cleanup data for analysis. The
8079
<em>string</em> command was created for when the old Unix standbys-
81-
grep, awk, sed, tr are unwieldly or inconvenient. <em>string</em>
80+
grep, awk, sed, tr are unwieldy or inconvenient. <em>string</em>
8281
provides operations are common in most language like, trimming,
8382
splitting, and transforming letter case. The <em>string</em> command
8483
also makes it easy to join JSON string arrays into single a string using
1.59 KB
Binary file not shown.
14.2 KB
Binary file not shown.
25.1 KB
Binary file not shown.
25.4 KB
Binary file not shown.
24.9 KB
Binary file not shown.

pagefind/pagefind-entry.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{"version":"1.4.0","languages":{"en-us":{"hash":"en-us_bda35fd0becf3","wasm":"en-us","page_count":207}},"include_characters":["_","","","","","","","","","_"]}
1+
{"version":"1.4.0","languages":{"en-us":{"hash":"en-us_93a871e52da1c","wasm":"en-us","page_count":207}},"include_characters":["_","","","","","","","","","_"]}
1.56 KB
Binary file not shown.

0 commit comments

Comments
 (0)