|
| 1 | +# Contributing to Crux Console |
| 2 | + |
| 3 | +Crux Console is a web UI for querying [Crux](https://github.com/juxt/crux) |
| 4 | +(bitemporal database). It consists of: |
| 5 | + |
| 6 | +| Path | What it is | |
| 7 | +|-----------------------|-------------------------------------------------------------------| |
| 8 | +| `src/crux_ui/` | Frontend SPA — ClojureScript, re-frame + reagent, built by shadow-cljs | |
| 9 | +| `src/crux_ui_lib/` | Async HTTP client for the Crux REST API (used by the frontend) | |
| 10 | +| `src/crux_ui_server/` | Backend — Clojure, aleph HTTP server that serves the SPA and can optionally embed a Crux node | |
| 11 | +| `resources/static/` | Static assets; compiled JS lands in `resources/static/crux-ui/compiled/` | |
| 12 | +| `test/` | ClojureScript tests (browser test runner) | |
| 13 | + |
| 14 | +The browser talks to a Crux node **directly** over its HTTP API — the console |
| 15 | +server only serves the frontend (and optionally starts the Crux node for you). |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +- **JDK** 8+ (the release build targets Java 8; Java 11/17 work for local dev) |
| 20 | +- **Leiningen** ≥ 2.9.1 |
| 21 | +- **Node.js** and **Yarn** (`yarn install` fetches npm deps, including shadow-cljs) |
| 22 | + |
| 23 | +## Running in development |
| 24 | + |
| 25 | +You need two processes: the frontend compiler (hot reload) and the server. |
| 26 | + |
| 27 | +### 1. Frontend — shadow-cljs watch |
| 28 | + |
| 29 | +```sh |
| 30 | +yarn install # once |
| 31 | +dev/shadow-dev.sh # = node_modules/.bin/shadow-cljs watch dev |
| 32 | +``` |
| 33 | + |
| 34 | +The `dev` build compiles without optimizations to the same output dir the |
| 35 | +server serves from, hot-swaps code on save, and includes |
| 36 | +[re-frame-10x](https://github.com/day8/re-frame-10x) for inspecting app state. |
| 37 | + |
| 38 | +(`lein cljs-dev` also works but watches the `app` build, which compiles with |
| 39 | +`:advanced` optimizations — much slower feedback; prefer the `dev` build.) |
| 40 | + |
| 41 | +Note: shadow-cljs prints many |
| 42 | +`failed to inspect resource ".../es5-ext/... #/..." ` warnings — these are |
| 43 | +harmless (npm files with `#` in their paths) and can be ignored. |
| 44 | + |
| 45 | +### 2. Server — with an embedded Crux node |
| 46 | + |
| 47 | +The easiest way to get a fully working console is to let the server start a |
| 48 | +Crux node for you. The Crux dependencies live in the `crux-jars` lein profile, |
| 49 | +so it must be active: |
| 50 | + |
| 51 | +```sh |
| 52 | +lein with-profile +crux-jars run -m crux-ui-server.main |
| 53 | +``` |
| 54 | + |
| 55 | +Configuration is read from `crux-console-conf.edn` in the working directory |
| 56 | +(see `crux-console-conf.sample.edn`). For local dev with an embedded node use: |
| 57 | + |
| 58 | +```edn |
| 59 | +{:console/frontend-port 5000 |
| 60 | + :console/embed-crux true |
| 61 | + :console/routes-prefix "/console" |
| 62 | + :console/hide-features #{:features/attribute-history} |
| 63 | + :console/crux-node-url-base "localhost:8080" |
| 64 | + :console/crux-http-port 8080} |
| 65 | +``` |
| 66 | + |
| 67 | +Then open <http://localhost:5000/console>. Smoke-test with a transaction |
| 68 | + |
| 69 | +```clojure |
| 70 | +[[:crux.tx/put {:crux.db/id :hello :title "world"}]] |
| 71 | +``` |
| 72 | + |
| 73 | +followed by a query |
| 74 | + |
| 75 | +```clojure |
| 76 | +{:find [e] :where [[e :crux.db/id _]]} |
| 77 | +``` |
| 78 | + |
| 79 | +The embedded node persists to `./data/` (RocksDB) — delete that directory for |
| 80 | +a fresh database. |
| 81 | + |
| 82 | +All config keys can also be passed as CLI flags after `--` |
| 83 | +(e.g. `-- --embed-crux true --frontend-port 5001`); values are parsed with |
| 84 | +`read-string`, so quote strings: `--crux-node-url-base '"localhost:8080"'`. |
| 85 | + |
| 86 | +### Gotchas |
| 87 | + |
| 88 | +- **`:crux-node-url-base` must NOT have the `/crux` suffix for local dev.** |
| 89 | + The default (`localhost:8080/crux`) is for the nginx reverse-proxy |
| 90 | + deployment described in the readme, where nginx rewrites `/crux` away. |
| 91 | + Against an embedded node (which serves its API at the root of port 8080) |
| 92 | + the `/crux` prefix produces `400 Bad Request` on every frontend request. |
| 93 | +- **Service worker caching.** The console registers a service worker that |
| 94 | + caches the HTML, and the Crux node URL is baked into that HTML as a |
| 95 | + `data-crux-base-url` attribute. After changing config, hard-reload; if the |
| 96 | + browser still hits stale URLs, unregister the service worker |
| 97 | + (DevTools → Application → Service Workers) and reload. |
| 98 | +- Pointing the console at an external Crux node? The URL must be reachable |
| 99 | + from the **browser** (not just the server) and the node must send CORS |
| 100 | + headers (the embedded node is started with permissive CORS — |
| 101 | + see `src/crux_ui_server/crux_auto_start.clj`). |
| 102 | + |
| 103 | +## Sample data & querying |
| 104 | + |
| 105 | +The database starts empty — the console ships with example **generators** |
| 106 | +(`src/crux_ui/logic/example_queries.cljs`), not preloaded data. The built-in |
| 107 | +dataset is a stock-market domain you load from the examples strip in the UI: |
| 108 | + |
| 109 | +- **`:crux.tx/put`** — inserts 16 stock exchanges (NYSE, NASDAQ + its European |
| 110 | + subsidiaries, LSE, Euronext, …) and 50 random tickers with `:ticker/price` |
| 111 | + and `:ticker/market` references. Click it, then hit **Run Query**. |
| 112 | +- **`put with valid time`** — writes ~500 days of price history for one ticker |
| 113 | + (derived from real Amazon closing prices, see `example_txes_amzn.cljs`). |
| 114 | + Load this to make the bitemporal features (time controls, history charts) |
| 115 | + interesting. |
| 116 | + |
| 117 | +With data in place, try the query presets: |
| 118 | + |
| 119 | +| Preset | Demonstrates | |
| 120 | +|-----------------------|-----------------------------------------------------------| |
| 121 | +| simple query | basic Datalog: `{:find [e p] :where [[e :ticker/price p]]}` | |
| 122 | +| join | joins across entities, `:order-by`, `:limit`, `:offset` | |
| 123 | +| rules and args | parameterized queries (`:args`) and Datalog `:rules` | |
| 124 | +| full-results | `:full-results? true` — return whole documents | |
| 125 | +| full-results with refresh | `:ui/poll-interval-seconds?` — console re-runs the query on an interval | |
| 126 | +| delete / evict | `:crux.tx/delete` (tombstone) vs `:crux.tx/evict` (erase) | |
| 127 | + |
| 128 | +The editor auto-detects input type: an EDN **map** is a query, a **vector** is |
| 129 | +a transaction. Use the time controls (datepickers / sliders) to run a query at |
| 130 | +a different valid time / transaction time against the history data. |
| 131 | + |
| 132 | +Tips: |
| 133 | + |
| 134 | +- Entity-history charts are behind the `:features/attribute-history` feature, |
| 135 | + which the sample config hides — remove it from `:console/hide-features` and |
| 136 | + restart to enable them. |
| 137 | +- You can load your own preset list with the URL param |
| 138 | + `?examples-gist=<raw-gist-url>` pointing to an EDN file of |
| 139 | + `{:title ... :query ...}` maps (format in the readme). |
| 140 | + |
| 141 | +## HTTP API |
| 142 | + |
| 143 | +The console server exposes one JSON-free, EDN-in/EDN-out endpoint of its own |
| 144 | +(everything else is the Crux node's REST API, which the browser calls |
| 145 | +directly): |
| 146 | + |
| 147 | +### `POST <routes-prefix>/api/query` |
| 148 | + |
| 149 | +Runs a Datalog query against the Crux node (server-side, via |
| 150 | +`:console/crux-http-port`) and returns the results as an EDN **list** `(...)` |
| 151 | +— unlike the Crux node's own `/query`, which returns a set `#{...}` for |
| 152 | +unordered queries and a vector `[...]` for `:order-by` queries. |
| 153 | + |
| 154 | +The body may be either the bare query map or the `{:query {...}}` wrapping |
| 155 | +that the Crux node's `/query` endpoint expects — both are accepted: |
| 156 | + |
| 157 | +```sh |
| 158 | +curl -X POST http://localhost:5000/console/api/query \ |
| 159 | + -H "Content-Type: application/edn" \ |
| 160 | + -d '{:find [e p] :where [[e :ticker/price p]] :order-by [[p :asc]] :limit 3}' |
| 161 | +# => ([:ids/fashion-ticker-21 2] [:ids/industry-ticker-48 5] ...) |
| 162 | +``` |
| 163 | + |
| 164 | +Errors come back as `400` with an EDN body `{:error "..."}`. |
| 165 | +Handler: `::api-query` in `src/crux_ui_server/main.clj`. |
| 166 | + |
| 167 | +## Tests |
| 168 | + |
| 169 | +Frontend tests live in `test/` (`*_test.cljs`, run by |
| 170 | +`crux-console.test-runner` via shadow-cljs' `:browser-test` target): |
| 171 | + |
| 172 | +```sh |
| 173 | +node_modules/.bin/shadow-cljs watch test |
| 174 | +``` |
| 175 | + |
| 176 | +then open <http://localhost:4001> — results render in the browser and re-run |
| 177 | +on save. |
| 178 | + |
| 179 | +There is no backend (clj) test suite at the moment. |
| 180 | + |
| 181 | +## Production build |
| 182 | + |
| 183 | +```sh |
| 184 | +lein build # yarn install + shadow-cljs release app |
| 185 | +lein with-profile base:crux-jars uberjar # builds both jars |
| 186 | +``` |
| 187 | + |
| 188 | +The uberjar step produces: |
| 189 | + |
| 190 | +- `target/crux-console-skimmed.jar` — console only; expects an external Crux |
| 191 | + node (`:console/embed-crux false`) |
| 192 | +- `target/crux-console.jar` — bundles Crux; can run with `--embed-crux true` |
| 193 | + |
| 194 | +Run with: |
| 195 | + |
| 196 | +```sh |
| 197 | +java -jar target/crux-console.jar --embed-crux true |
| 198 | +``` |
| 199 | + |
| 200 | +`make all` is an equivalent shortcut (clean, install, release build, uberjar), |
| 201 | +and `lein build-ebs` additionally packs an AWS Elastic Beanstalk bundle |
| 202 | +(see `dev/build-ebs.sh`). |
| 203 | + |
| 204 | +## Code style / architecture notes |
| 205 | + |
| 206 | +- Frontend state management is re-frame: events and subscriptions are under |
| 207 | + `src/crux_ui/events/` and `src/crux_ui/subs.cljs`; views under |
| 208 | + `src/crux_ui/views/` (the query screen is `views/query/`). |
| 209 | +- Query parsing/classification (query vs transaction) lives in |
| 210 | + `src/crux_ui/logic/query_analysis.cljs` — a good place to start reading. |
| 211 | +- Server routing is bidi, defined in `src/crux_ui_server/main.clj`; HTML is |
| 212 | + generated server-side by `src/crux_ui_server/pages.clj` (page-renderer), |
| 213 | + which is also where frontend runtime config is injected as `data-*` |
| 214 | + attributes on `<html>`. |
0 commit comments