Pre-parse sentences and save in DB

[XanderVertegaal]

Currently, LangPro will both parse and prove each problem it receives. Especially the parsing takes a lot of time, so we stand to gain a lot of time by preparsing every sentence in the corpus and storing the raw Prolog results for the CCG Tree (the other parse trees can be derived very quickly) in the DB.

When the user hits 'Parse and Proof' in the annotator, the backend will first retrieve the parse results from the DB (if they are available) and send them to LangPro, which will simply have to derive the other three parse trees and do the proving.

This requires

• a new endpoint on LangPro Container (e.g. /api/parse) that takes a problem as its input and returns just the parse output.
• a new endpoint on LangPro Container or an update to the existing /api/prove route: it should be able to receive parse results and simply run the prover (bypassing the parser). If we implement a new endpoint, the output should have the same format as /api/prove.
• an asynchronous worker that can be triggered to go over all problems, send them to LangPro one by one to be parsed and then store the results in the DB.
• DB models for the stored parse results.

For the async worker, we can use a Celery/RabbitMQ setup. If we're comfortable upgrading to Django 6.0, there is also the new 'Tasks' module to consider.

[kovvalsky]

DB models for the stored parse results.

Is this about the format of a parse that will be used for saving in a BD?
Anyway, the format of the parse needs some deliberation.

1. Prolog format is an option as it is very compact (can be later useful for manually editing wrong parses in the annotator; see examples), and also directly consumable by LangPro. But this will require running LangPro every time to convert prolog to the format suitable for display. Also, for a data release for research purposes, prolog is not a common format.
2. json, or other common format (maybe yaml?). This will be good for instant display or release, but needs to be converted to feed LangPro.
3. Record both formats per tree. (1) for LangPro and (2) for display and release.