Skip to content

Commit 8c8abb5

Browse files
Fokkobitsondatadev
andauthored
Add SQLite Catalog support (#178)
Co-authored-by: Brian "bits" Olsen <bits@bitsondata.dev>
1 parent e92e10a commit 8c8abb5

5 files changed

Lines changed: 433 additions & 170 deletions

File tree

mkdocs/docs/configuration.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,9 @@ catalog:
140140

141141
## SQL Catalog
142142

143-
The SQL catalog requires a database for its backend. As of now, pyiceberg only supports PostgreSQL through psycopg2.
144-
The database connection has to be configured using the `uri` property (see SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls)):
143+
The SQL catalog requires a database for its backend. PyIceberg supports PostgreSQL and SQLite through psycopg2. The database connection has to be configured using the `uri` property. See SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls):
144+
145+
For PostgreSQL:
145146

146147
```yaml
147148
catalog:
@@ -150,6 +151,22 @@ catalog:
150151
uri: postgresql+psycopg2://username:password@localhost/mydatabase
151152
```
152153

154+
In the case of SQLite:
155+
156+
<!-- prettier-ignore-start -->
157+
158+
!!! warning inline end "Development only"
159+
SQLite is not built for concurrency, you should use this catalog for exploratory or development purposes.
160+
161+
<!-- prettier-ignore-end -->
162+
163+
```yaml
164+
catalog:
165+
default:
166+
type: sql
167+
uri: sqlite:////tmp/pyiceberg.db
168+
```
169+
153170
## Hive Catalog
154171

155172
```yaml

poetry.lock

Lines changed: 15 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pyiceberg/catalog/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ def infer_catalog_type(name: str, catalog_properties: RecursiveDict) -> Optional
153153
return CatalogType.REST
154154
elif uri.startswith("thrift"):
155155
return CatalogType.HIVE
156-
elif uri.startswith("postgresql"):
156+
elif uri.startswith(("sqlite", "postgresql")):
157157
return CatalogType.SQL
158158
else:
159159
raise ValueError(f"Could not infer the catalog type from the uri: {uri}")

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ sqlalchemy = { version = "^2.0.18", optional = true }
7474
[tool.poetry.dev-dependencies]
7575
pytest = "7.4.3"
7676
pytest-checkdocs = "2.10.1"
77+
pytest-lazy-fixture = "0.6.3"
7778
pre-commit = "3.5.0"
7879
fastavro = "1.9.1"
7980
coverage = { version = "^7.3.2", extras = ["toml"] }

0 commit comments

Comments
 (0)