dataset/docs/storage-engines.html at main · caltechlibrary/dataset · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
<!DOCTYPE html>
<html>
<head>
    <title>Dataset Project</title>
    <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
    <link rel="stylesheet" href="https://caltechlibrary.github.io/css/site.css">
</head>
<body>
<header>
<a href="http://library.caltech.edu" title="link to Caltech Library Homepage"><img src="https://caltechlibrary.github.io/assets/liblogo.gif" alt="Caltech Library logo"></a>
</header>
<nav>
<ul>
    <li><a href="/">Home</a></li>
    <li><a href="../">README</a></li>
    <li><a href="../LICENSE">LICENSE</a></li>
    <li><a href="../INSTALL.html">INSTALL</a></li>
    <li><a href="../user_manual.html">User Manual</a></li>
    <li><a href="../about.html">About</a></li>
	<li><a href="../search.html">Search</a></li>
    <li><a href="https://github.com/caltechlibrary/dataset">GitHub</a></li>
</ul>
</nav>

<section>
<h1 id="storage-engines">Storage Engines</h1>
<p>With the introduction of v2 of dataset you now have a choice of
storage engines. In v2.2 SQLite3 became the default storage engine.
Current supported storage engines are.</p>
<ul>
<li>SQL Storage (via MySQL 8, Postgres &gt;= 12 and SQLite3 &gt;= 3.4)
<ul>
<li>SQLite3 is the default storage engine since it requires no
configuration</li>
</ul></li>
<li>pairtree (the original storage engine of v1, depricated)</li>
</ul>
<p>With the introduction of SQL Storage dataset can be used in a
multi-process/multi-user mode via a RESTful API. The SQL storage is
experimental and as it gets you more various considerations are coming
to the surface</p>
<ul>
<li>SQLite3 works fine across single and multie process scenarios. It
has the advantage of requiring no configuration and avoiding the need to
run a database manage system such as MySQL or Postgres.</li>
<li>Postgres 14.5 is preferred and most test implementation when a
database management system is needed.</li>
<li>MySQL 8 works well when you need multi-user and multi-process
access. (depreciated)</li>
</ul>
<h2 id="cautions">Cautions</h2>
<p>The pairtree storage engine is stable. The primary limitations are
the file system (where it stores the JSON documents) case limitations
and lack of record/field locking. Pairtree work fine for batch
operations, single user/single process operations with less than 100k
documents. It does not appropriate for concurrent access involving
writes. I remains in v2.2 as a historical artifact. Will be removed in
v3.</p>
<p>MySQL is used less and less in the software I work with. MySQL
support maybe dropped in version 3 of dataset.</p>
</section>

<footer>
<span>&copy; 2022 <a href="https://www.library.caltech.edu/copyright">Caltech Library</a></span>
<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address>
<span><a href="mailto:library@caltech.edu">Email Us</a></span>
<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
</footer>
</body>
</html>