Command-line interface to Turing

[penelopeysm]

Shifting this to an issue to be able to track thoughts / discuss. The central point is: developing a CmdStan-like interface for Turing.jl using Julia Pkg’s App feature

[penelopeysm]

My 2p is that a CLI to Turing is definitely worth pursuing, especially since it allows us to break out of Julia’s REPL-heavy workflow that makes it really hard to do interop. There are two technical things that need to be figured out, IMO:

1. Precompilation for a given input MUST be able to be cached or else TTFX will make the whole experience quite unsavoury. That is, running a model for the first time via the CLI can be slow, but running it again should not incur TTFX again. (That is of course how CmdStan behaves — the first time it has to perform compilation, but the compiled outputs are saved to disk and reused.) Now, unfortunately, with PackageCompiler.jl and sysimages this is definitely not possible. You can generate and bundle a static precompilation cache with it, but that is not enough for us as we need to precompile users’ models on demand so that the next time they run their specific model it is fast — it’s not enough to precompile some generic model. I don’t know if this is the same with the new Pkg apps. Ideally, we would have a precompilation cache that would be stored in a fixed file on disk, updated with each invocation of the app, and allowed to grow (endlessly?). But I wouldn’t be surprised if this approach will run into has similar limitations.*

   • An alternative to this, which I believe might actually be easier, is to have a long-running Julia daemon / background worker — this is a technique used in e.g. QuartoNotebookRunner.jl (with some really impressive results), and I am almost certain that we can copy a lot of its implementation. Then as long as that background process stays alive, we don’t need to worry about precompilation at all. It would still be easy to expose a CLI, but the CLI app instead of running actual Turing models will communicate with the worker process instead (just like Quarto).

2. Presumably this interface will take a string as an input (i.e. file contents). For Cmdstan this is a Stan model specification. However for Turing, should this string be Julia syntax, or do we want to use a different input format that might be more constrained but might be easier to programmatically generate? How do we pass in data and other arguments to the model (probably JSON)?

---

* There was one time I briefly investigated compiling Turing code to WASM (so that you could run Turing in the browser), using https://tshort.github.io/WebAssemblyCompiler.jl/stable/, but ran into the same problem: every function you want to use has to be compiled ahead of time. That means users cannot define their own models and run them in the browser (which is rather underwhelming). What you really need to do is to compile Julia itself (i.e., the entire runtime) to WASM, which is a far more ambitious project (that's essentially what Pyodide does for Python).

[nsiccha]

Now, unfortunately, with PackageCompiler.jl and sysimages this is definitely not possible. You can generate and bundle a static precompilation cache with it, but that is not enough for us as we need to precompile users’ models on demand so that the next time they run their specific model it is fast — it’s not enough to precompile some generic model.

Can't we generate a package for every user specified model under the hood and precompile that using the standard tools?

[penelopeysm]

Maybe but I think it's difficult since it's not just the model, it's also what functions they call on the model (MCMC/VI/etc.), what sampler they want to use, AD backends and things like that. Unless you say that they have to pick one sampler/adtype in advance and that's all they can ever use without incurring TTFX (unless they recompile).