dataset/docs/index.html at main · caltechlibrary/dataset · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
<!DOCTYPE html>
<html>
<head>
    <title>Dataset Project</title>
    <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
    <link rel="stylesheet" href="https://caltechlibrary.github.io/css/site.css">
</head>
<body>
<header>
<a href="http://library.caltech.edu" title="link to Caltech Library Homepage"><img src="https://caltechlibrary.github.io/assets/liblogo.gif" alt="Caltech Library logo"></a>
</header>
<nav>
<ul>
    <li><a href="/">Home</a></li>
    <li><a href="../">README</a></li>
    <li><a href="../LICENSE">LICENSE</a></li>
    <li><a href="../INSTALL.html">INSTALL</a></li>
    <li><a href="../user_manual.html">User Manual</a></li>
    <li><a href="../about.html">About</a></li>
	<li><a href="../search.html">Search</a></li>
    <li><a href="https://github.com/caltechlibrary/dataset">GitHub</a></li>
</ul>
</nav>

<section>
<h1 id="dataset">dataset</h1>
<p>The documentation is organized around the command line options and as
a series of “how to” style examples.</p>
<ul>
<li><a href="../how-to/getting-started-with-dataset.html">getting
started with dataset</a> (covers both Bash and Python)</li>
<li>Explore additional other <a href="../how-to/">tutorials</a></li>
</ul>
<h2 id="command-line-program-documentation">Command line program
documentation</h2>
<ul>
<li><a href="dataset.html">dataset</a> - usage page for managing
collections with <em>dataset</em></li>
</ul>
<h2 id="internal-project-concepts">Internal project concepts</h2>
<ul>
<li><a href="../how-to/upgrading-a-collection.html">upgrading a
collection</a> - Describes how to upgrade a collection from a previous
version of dataset to a new one</li>
<li><a href="../how-to/how-attachments-work.html">how attachments
work</a> - Detailed description of attachments and their metadata</li>
</ul>
<h2 id="dataset-operations"><strong>dataset</strong> Operations</h2>
<p>The basic operations support by <em>dataset</em> are listed below
organized by collection and JSON document level.</p>
<h2 id="a-word-about-keys">A word about keys</h2>
<p><strong>dataset</strong> is based around the concept of key/value
pairs where the key is the unique identifier for an object stored
(i.e. the value) in the collection. Each storage option supported by
dataset and its own issues around what things can be called.
<strong>Keys should be lower case alpha numeric or underscore
only.</strong> E.g. the pairtree storage relies on the file system to
store the JSON objects. Some file systems are not case sensitive, others
face challenges with non-alpha numeric filenames.</p>
<h2 id="collection-level">Collection Level</h2>
<ul>
<li><a href="init.html">init</a> creates a collection</li>
<li><a href="keys.html">keys</a> list keys of JSON documents in a
collection, supports filtering and sorting</li>
<li><a href="haskey.html">has-key</a> returns true if key is found in
collection, false otherwise</li>
<li><a href="count.html">count</a> returns the number of documents in a
collection, supports filtering for subsets</li>
<li><a href="dump.html">dump</a> export a collection to a JSON lines
file</li>
<li><a href="load.html">load</a> import an collection from a JSON files
file.</li>
</ul>
<h2 id="json-document-level">JSON Document level</h2>
<ul>
<li><a href="create.html">create</a> a JSON document in a
collection</li>
<li><a href="read.html">read</a> back a JSON document in a
collection</li>
<li><a href="update.html">update</a> a JSON document in a
collection</li>
<li><a href="delete.html">delete</a> a JSON document in a
collection</li>
</ul>
<h2 id="json-document-attachments">JSON Document Attachments</h2>
<ul>
<li><a href="attach.html">attach</a> a file to a JSON document in a
collection</li>
<li><a href="attachments.html">attachments</a> lists the files attached
to a JSON document in a collection</li>
<li><a href="retrieve.html">retrieve</a> retrieve an attached file
associated with a JSON document in a collection</li>
<li><a href="prune.html">prune</a> delete one or more attached files of
a JSON document in a collection</li>
</ul>
<h1 id="datasetd-as-a-web-service">datasetd as a web service</h1>
<p>New as of version v2 is a web service providing access to dataset
collections. This is described in the <a
href="datasetd.html">datasetd</a> documentation page.</p>
<p><a href="datasetd.html">datasetd</a> supports the following end
points.</p>
<h1 id="storage-engines">Storage engines</h1>
<p>In v2 dataset is starting to suport storing your JSON document in a
SQL database. Currently three SQL databases can be used to store the
JSON documents, SQLite 3 (default engine, used in dataset’s test
suites), MySQL 8 (minimally tested), Postgres &gt;= 12 (well tested).
See <a href="storage-engines.html">storage engines</a> for more
details.</p>
<h1 id="compatibity">Compatibity</h1>
<p>Migrating dataset collections between major versions or just
different collections can be done using the “dump” and “load” feature.
This replaces the old process in early v2 that required you to run a
“repair” operation to convert a collection to the current version of
dataset.</p>
<p>Example migrating from dataset “data_v2.ds” from v2 to v3 as
“data_v3.ds”.</p>
<pre class="shell"><code>dataset3 init data_v3.ds
dataset dump data_v2.ds | dataset3 load data_v3.ds</code></pre>
</section>

<footer>
<span>&copy; 2022 <a href="https://www.library.caltech.edu/copyright">Caltech Library</a></span>
<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address>
<span><a href="mailto:library@caltech.edu">Email Us</a></span>
<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
</footer>
</body>
</html>