|
| 1 | +// SPDX-License-Identifier: CC-BY-SA-4.0 |
| 2 | +// SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 3 | += julianiser |
| 4 | +:toc: macro |
| 5 | +:toclevels: 2 |
| 6 | + |
| 7 | +Auto-generate Julia modules from existing Python/R data-science code. |
| 8 | + |
| 9 | +toc::[] |
| 10 | + |
| 11 | +== What Is This? |
| 12 | + |
| 13 | +Julianiser analyses existing Python and R data science code, identifies common |
| 14 | +library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and generates a |
| 15 | +corresponding Julia module using known source-to-Julia function mappings. The |
| 16 | +goal is drop-in replacement modules that let you run the equivalent logic under |
| 17 | +Julia's LLVM JIT compilation. |
| 18 | + |
| 19 | +*Current status (honest, as of this writing):* the working, tested pipeline is |
| 20 | +the *Rust codegen* in `src/codegen/` (line-based Python/R parsing + |
| 21 | +template-based Julia generation), invoked by the Rust CLI (`src/main.rs`). It |
| 22 | +does not (yet) measure or claim any actual speedup — the generated |
| 23 | +`benchmarks/results.toml` ships with `speedup = 0.0 # TODO: fill in`, to be |
| 24 | +measured by the user after running the generated Julia code against the |
| 25 | +original. Any "10-100x" figures elsewhere in this repository's docs describe the |
| 26 | +_general, well-documented characteristics of Julia's JIT_ on this class of |
| 27 | +workload, not a number julianiser itself has measured. |
| 28 | + |
| 29 | +The *Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)* layer |
| 30 | +described in earlier drafts of this README is *scaffolding*: the Idris2 proofs |
| 31 | +and the Zig C-ABI bridge exist as source files, but the Zig implementation is |
| 32 | +TODO-stubbed (parsing is a no-op, codegen returns a hardcoded placeholder |
| 33 | +module, and the benchmark hook returns a hardcoded `speedup = 1.0`), and the |
| 34 | +Rust CLI never links, dlopens, or otherwise calls into it. It is tracked for |
| 35 | +future wiring, not part of the operational translation path today. |
| 36 | + |
| 37 | +Scientists keep their Python notebooks. Julia runs underneath, once generated. |
| 38 | + |
| 39 | +== How It Works (today) |
| 40 | + |
| 41 | +Describe your data pipeline in a `julianiser.toml` manifest. The Rust CLI, via |
| 42 | +`src/codegen/`: |
| 43 | + |
| 44 | +. *Parses* your Python or R source with a line-based scanner |
| 45 | + (`src/codegen/parser.rs`) that detects import statements and library-qualified |
| 46 | + calls. |
| 47 | +. *Looks up* each detected call in a table of known source-language → Julia |
| 48 | + translations (`src/codegen/julia_gen.rs`). |
| 49 | +. *Generates* a Julia module with `using` statements and translated calls (or a |
| 50 | + `# TODO` comment where no mapping exists). |
| 51 | +. *Generates* benchmark scaffolding (`benchmarks/benchmark.jl`, |
| 52 | + `benchmark_runner.sh`, `results.toml`) so _you_ can measure and record the |
| 53 | + speedup after running both versions — julianiser does not measure it for you |
| 54 | + yet. |
| 55 | +. *Falls back* gracefully — if Julia is not installed, your Python/R code still |
| 56 | + runs; the generated Julia is optional, additive output. |
| 57 | + |
| 58 | +The Idris2-ABI / Zig-FFI seam described below is planned future infrastructure |
| 59 | +for a compiled, zero-copy bridge; it is not required for (and is not currently |
| 60 | +used by) the codegen path above. |
| 61 | + |
| 62 | +== Key Value |
| 63 | + |
| 64 | +* *No Julia knowledge required* — Julianiser handles the translation for the |
| 65 | + library calls it recognises. |
| 66 | +* *Julia's JIT is well-documented to deliver large speedups* on numeric, array, |
| 67 | + and dataframe code — julianiser generates the scripts to measure that for |
| 68 | + _your_ workload; it does not yet ship a measured number of its own. |
| 69 | +* *Gradual adoption* — julianise one function at a time, benchmark each. |
| 70 | +* *Drop-in modules* — generated Julia code exposes a `run_pipeline()` entry |
| 71 | + point mirroring the detected calls in your Python/R script. |
| 72 | +* *Idris2 ABI + Zig FFI scaffolding exists* (`src/interface/`) for a future |
| 73 | + formally-verified, compiled bridge — not yet wired into the translation path |
| 74 | + (see Architecture, below). |
| 75 | + |
| 76 | +== Supported Patterns |
| 77 | + |
| 78 | +Julianiser recognises and translates these common data science patterns: |
| 79 | + |
| 80 | +[cols="1,1,2",options="header"] |
| 81 | +|=== |
| 82 | +| Python | Julia Equivalent | Notes |
| 83 | + |
| 84 | +| `pandas.DataFrame` | `DataFrames.jl` | Column operations, groupby, joins |
| 85 | +| `numpy` arrays | Native Julia arrays | Broadcasting, slicing, linear algebra |
| 86 | +| `scipy.optimize` | `Optim.jl` / Julia stdlib | Minimisation, root-finding, curve fitting |
| 87 | +| `matplotlib` / `seaborn` | `Plots.jl` / `Makie.jl` | Static and interactive plotting |
| 88 | +| `scikit-learn` pipelines | `MLJ.jl` | ⚠ *Planned* — sklearn imports are _detected_ (the `sk` alias is recognised in the parser), but no sklearn→MLJ function conversions are mapped yet in the codegen |
| 89 | +| R `data.frame` / `tibble` | `DataFrames.jl` | dplyr-style verbs mapped to Julia |
| 90 | +| R `apply` / `sapply` / `lapply` | Julia broadcasting / `map` | Vectorised equivalents |
| 91 | +|=== |
| 92 | + |
| 93 | +== Why Julia? |
| 94 | + |
| 95 | +Julia achieves C-like performance through: |
| 96 | + |
| 97 | +* *LLVM JIT compilation* — code is compiled to native machine code on first call. |
| 98 | +* *Multiple dispatch* — the type system enables aggressive specialisation. |
| 99 | +* *Type inference* — the compiler infers concrete types for fast code paths. |
| 100 | +* *Broadcasting* — element-wise operations fuse into single loops (no temporary |
| 101 | + arrays). |
| 102 | +* *In-place operations* — `mul!`, `ldiv!`, and friends avoid allocation. |
| 103 | + |
| 104 | +The "two-language problem" (prototype in Python, rewrite in C) disappears. Julia |
| 105 | +is both the prototyping language and the production language. |
| 106 | + |
| 107 | +== Architecture |
| 108 | + |
| 109 | +Follows the hyperpolymath -iser pattern (same as |
| 110 | +https://github.com/hyperpolymath/chapeliser[Chapeliser]) in file layout, but — |
| 111 | +to be explicit about what is operational today versus what is scaffolding for |
| 112 | +later: |
| 113 | + |
| 114 | +*Operational (this is what `julianiser generate` actually runs):* |
| 115 | + |
| 116 | +* *Manifest* (`julianiser.toml`) — describe WHAT you want julianised. |
| 117 | +* *Source Parser* (`src/codegen/parser.rs`) — line-based Python/R scanner that |
| 118 | + detects import statements and known library-qualified calls. |
| 119 | +* *Julia Codegen* (`src/codegen/julia_gen.rs`) — looks up each detected call in |
| 120 | + a source-to-Julia translation table and emits a Julia module; unmapped calls |
| 121 | + become a `# TODO` comment, not a runtime error. |
| 122 | +* *Benchmark scaffold generator* (`src/codegen/benchmark.rs`) — generates |
| 123 | + `benchmark.jl` / `benchmark_runner.sh` / `results.toml` so the user can run |
| 124 | + and record original-vs-Julia timings themselves. |
| 125 | +* *Rust CLI* (`src/main.rs`) — orchestrates manifest validation, parsing, and |
| 126 | + generation (`init`, `validate`, `generate`, `build`, `run`). |
| 127 | + |
| 128 | +*Scaffolding — present in the tree, not yet wired into the CLI:* |
| 129 | + |
| 130 | +* *Idris2 ABI* (`src/interface/abi/`) — formal type/layout declarations intended |
| 131 | + to eventually prove equivalence between source and generated code. Exists as |
| 132 | + `.idr` source; not built or checked by `cargo build`/`cargo test`. |
| 133 | +* *Zig FFI* (`src/interface/ffi/`) — a C-ABI bridge skeleton. The exported |
| 134 | + functions are TODO-stubbed (parsing is a no-op that only records the source |
| 135 | + language; codegen returns a fixed placeholder module string; the benchmark |
| 136 | + hook returns a hardcoded `speedup = 1.0`). The Rust CLI never calls into this |
| 137 | + library — no `dlopen`, no linking. It is tracked for future wiring, not part |
| 138 | + of today's translation pipeline. |
| 139 | + |
| 140 | +Today, the user writes zero Julia code and the Rust codegen generates the |
| 141 | +translated module; the Idris2/Zig seam is where a future, formally-verified, |
| 142 | +compiled bridge is planned to replace the current pure-codegen approach. |
| 143 | + |
| 144 | +Part of the https://github.com/hyperpolymath/iseriser[-iser family] of |
| 145 | +acceleration frameworks. |
| 146 | + |
| 147 | +== Use Cases |
| 148 | + |
| 149 | +* *Data science acceleration* — speed up pandas/numpy pipelines without rewriting. |
| 150 | +* *Batch processing* — convert overnight Python ETL jobs to minutes with Julia. |
| 151 | +* *Scientific computing migration* — gradually move research code from Python/R |
| 152 | + to Julia. |
| 153 | +* *HPC preparation* — Julia code can target GPU (CUDA.jl) and distributed |
| 154 | + (Distributed.jl) backends. |
| 155 | + |
| 156 | +== Quick Start |
| 157 | + |
| 158 | +[source,bash] |
| 159 | +---- |
| 160 | +# Install julianiser |
| 161 | +cargo install julianiser |
| 162 | +
|
| 163 | +# Initialise a manifest in your project |
| 164 | +julianiser init |
| 165 | +
|
| 166 | +# Edit julianiser.toml to point at your Python/R code |
| 167 | +# Then generate Julia replacements |
| 168 | +julianiser generate |
| 169 | +
|
| 170 | +# Benchmark original vs. generated |
| 171 | +julianiser run --benchmark |
| 172 | +---- |
| 173 | + |
| 174 | +== Status |
| 175 | + |
| 176 | +*Codebase in progress.* The Rust CLI and codegen pipeline (`src/codegen/`) are |
| 177 | +implemented and covered by an automated test suite (`cargo test`). The |
| 178 | +Idris2-ABI / Zig-FFI layer (`src/interface/`) is scaffolding — present in the |
| 179 | +tree but not yet built, checked, or called by the Rust CLI. No speedup number |
| 180 | +has been measured by julianiser itself; the generated benchmark scripts exist |
| 181 | +for the user to measure their own workload. |
| 182 | + |
| 183 | +== License |
| 184 | + |
| 185 | +Code: `MPL-2.0`. Documentation (including this README): `CC-BY-SA-4.0`. Full |
| 186 | +texts are in `LICENSES/`. |
0 commit comments