Update readme

Author: themoenen

generator/README.md

@@ -6,8 +6,8 @@ This script generates and updates a number of markdown pages for the documentati
 
 ### How to regenerate documentation
 
-Generate all files → save to [/_output](_output)  
-More details for each script can be found in the docstrings.
+Generate files → save to [_output](_output):  
+_More details for each script can be found in the docstrings._
 
 ```
 python3 generator/generate_docs.py
@@ -44,127 +44,13 @@ Files generated or updated for the [openad-toolkit] repo:
 Other files:
 - [commands.csv](generator/_output/other/commands.csv) overview file (not used, for record only)
 
-
-### What this does
-
-1. Use the descriptions from the [`/source`](./source) folder to update:
-   - The `README.md` with OpenAD description
-   - The `README/plugins.md` with plugins description
-   - The [openad_intro] text (/openad/helpers/output_content.py) that is displayed when running the `intro` command
-   - The informational paragraphs that are displayed when running the `? workspace`, `? mws`, `? tookit`, `? context` and `? run` commands (see _output_content_ in [main.py](/openad/app/main.py))
-   - The [Base Concepts](https://acceleratedscience.github.io/openad-docs/base-concepts.html) page on the documentation website
-1. Duplicate all the README markdown pages, prepare them for ***Just the Docs*** consumption, then copy them over to the documentation repository if it's available
-1. Generate two additional bespoke pages for the documentation website:
-   - [Base Concepts](https://acceleratedscience.github.io/openad-docs/base-concepts.html)
-   - [Commands](https://openad.accelerate.science/docs/commands)
-2. Update the [llm_description.txt](/openad/user_toolkits/DS4SD/llm_description.txt) for each individual plugin with the correct commands, used for LLM training.
-
 <br>
 
 ### How it works
 
--   The `/source` folder is the "single source of truth" and contains a number of text files with descriptions for OpenAD and some of its core concepts.
--   The `/input` folder contains the markdown templates that are used to generate the markdown pages for the [openad-docs] repository. The header info in the input files defines the order of pages in the side navigation.
--   The `/output` folder contains all the files that are exported by the script:
-    -   `/markdown`<br>The markdown files for the [openad-docs] repository:
-        - [index.md](output/markdown/index.md)
-        - [installation.md](output/markdown/installation.md)
-        - [getting-started.md](output/markdown/getting-started.md)
-        - [base-concepts.md](output/markdown/base-concepts.md)
-        - [commands.md](output/markdown/commands.md)
-        - [models-service.md](output/markdown/models-service.md)
-        - [plugins.md](output/markdown/plugins.md)
-        - [ai-assistant.md](output/markdown/ai-assistant.md)
-        - [developers.md](output/markdown/developers.md)
-    -   `/csv`
-        -   [csv/commands.csv](output/csv/commands.csv) - A CSV file with all available commands that is not used anywhere but comes in handy to have a clean overview of available commands
-
-<br>
-
-### Link mapping
-
-A key part of the translation from **GitHub** markdown to **Just the Docs** and **PyPI** markdown files, is to update internal links.
-
-- **GitHub:** Links and relative and point to other README files.
-- **[Documentation](https://acceleratedscience.github.io/openad-docs):** Links and relative and point to the other pages on the site.
-- **[PyPI](https://pypi.org/project/openad/):** Links are absolute and point to the documentation website.
-    
-**Source: GitHub**
-
-    [Link](README.md)
-    [Link](README.md#quick-install)
-    [Link](README/developers.md)
-    [Link](README/developers.md#testing-a-branch)
-    [Link]: README.md
-    [Link]: README.md#quick-install
-    [Link]: README/developers.md
-    [Link]: README/developers.md#testing-a-branch
-    
-    [Link](/README.md)
-    [Link](/README.md#quick-install)
-    [Link](/README/developers.md)
-    [Link](/README/developers.md#testing-a-branch)
-    [Link]: /README.md
-    [Link]: /README.md#quick-install
-    [Link]: /README/developers.md
-    [Link]: /README/developers.md#testing-a-branch
-
-**Translation: Just the Docs**
-
-    [Link](index.html)
-    [Link](index.html#quick-install)
-    [Link](developers.html)
-    [Link](developers.html#testing-a-branch)
-    [Link]: index.html
-    [Link]: index.html#quick-install
-    [Link]: developers.html
-    [Link]: developers.html#testing-a-branch
-
-    [Link](index.html)
-    [Link](index.html#quick-install)
-    [Link](developers.html)
-    [Link](developers.html#testing-a-branch)
-    [Link]: index.html
-    [Link]: index.html#quick-install
-    [Link]: developers.html
-    [Link]: developers.html#testing-a-branch
-
-**Translation: PyPI**
-
-    [Link](https://acceleratedscience.github.io/openad-docs/)
-    [Link](https://acceleratedscience.github.io/openad-docs/#quick-install)
-    [Link](https://acceleratedscience.github.io/openad-docs/developers.html)
-    [Link](https://acceleratedscience.github.io/openad-docs/developers.html#testing-a-branch)
-    [Link]: https://acceleratedscience.github.io/openad-docs/
-    [Link]: https://acceleratedscience.github.io/openad-docs/#quick-install
-    [Link]: https://acceleratedscience.github.io/openad-docs/developers.html
-    [Link]: https://acceleratedscience.github.io/openad-docs/developers.html#testing-a-branch
-
-    [Link](https://acceleratedscience.github.io/openad-docs/)
-    [Link](https://acceleratedscience.github.io/openad-docs/#quick-install)
-    [Link](https://acceleratedscience.github.io/openad-docs/developers.html)
-    [Link](https://acceleratedscience.github.io/openad-docs/developers.html#testing-a-branch)
-    [Link]: https://acceleratedscience.github.io/openad-docs/
-    [Link]: https://acceleratedscience.github.io/openad-docs/#quick-install
-    [Link]: https://acceleratedscience.github.io/openad-docs/developers.html
-    [Link]: https://acceleratedscience.github.io/openad-docs/developers.html#testing-a-branch
-
-[openad-docs]: https://github.com/acceleratedscience/openad-docs
-
-
-
-Documentation pages are generated by script:
-- [Publicly Available Models](https://openad.accelerate.science/docs/model-service/deploying-models)
-- [OpenAD Commands](https://openad.accelerate.science/docs/commands)
-- [Base Concepts](https://openad.accelerate.science/docs/base-concepts)
-
-Documentation pages that have descriptions that are updated by script:
-- [OpenAD Plugins](https://openad.accelerate.science/docs/plugins)
-
-Other files that are generated or updated by the script:
-- [README.md](https://github.com/acceleratedscience/openad-toolkit?tab=readme-ov-file#readme) for the openad-toolkit repo
-- [commands.csv](generator/_output/other/commands.csv) overview file
-
+-   The `/source` folder is the "single source of truth" and contains a number of text files with descriptions for OpenAD and some of its core concepts, as well as the list of model services.
+-   The `/_templates` folder contains the markdown templates that are used to generate some of the markdown pages for the [openad-docs] repository.
+-   The `/output` folder is where all files are stored before they're distributed to their final destination
 
 
 [openad-toolkit]: https://github.com/acceleratedscience/openad-toolkit

generator/_input/commands.md generator/_templates/commands.mdgenerator/_input/commands.md renamed to generator/_templates/commands.md

@@ -3,14 +3,14 @@
 from openad.helpers.output import output_text
 from openad.app.main import RUNCMD as cmd_pointer
 from openad.core.help import organize_commands
-from .shared import DO_NOT_EDIT, read_input_file, write_output_file
+from .shared import DO_NOT_EDIT, read_template_file, write_output_file
 
 
 def generate_commands_md(filename="commands.md"):
     """
     Generate the commands.md page for the documentation.
 
-    - Start from the _input/commands.md template
+    - Start from the _templates/commands.md template
     - Loop through all OpenAD commands & generate markdown
     - Store result in _output/docs/commands.md
     """
@@ -43,7 +43,7 @@ def generate_commands_md(filename="commands.md"):
     #
 
     # Read commands.md input content
-    commands_md = read_input_file(filename)
+    commands_md = read_template_file(filename)
 
     # Insert sections
     # fmt: off

…_input/model-service~available-models.md …plates/model-service~available-models.mdgenerator/_input/model-service~available-models.md renamed to generator/_templates/model-service~available-models.md generator/_input/model-service~available-models.md renamed to generator/_templates/model-service~available-models.md

@@ -1,4 +1,5 @@
 # -- NO LONGER IN USE --
+# Kept here for reference only
 # Todo: strategy on how to integrate LLM training for plugins
 
 import os

…erator/_input/openad~helpers~concepts.py …or/_templates/openad~helpers~concepts.pygenerator/_input/openad~helpers~concepts.py renamed to generator/_templates/openad~helpers~concepts.py generator/_input/openad~helpers~concepts.py renamed to generator/_templates/openad~helpers~concepts.py

@@ -8,7 +8,7 @@
     DO_NOT_EDIT,
     FLAG_ERROR,
     source_dir,
-    read_input_file,
+    read_template_file,
     write_output_file,
 )
 
@@ -21,8 +21,8 @@ def generate_model_docs(filename="model-service~available-models.md"):
     """
     Generate the available-models.md page for the documentation.
 
-    - Start from the _input/model-service~available-models.md template
-    - Load the models.csv file from _input/models.csv
+    - Start from the _templates/model-service~available-models.md template
+    - Load the models.csv file from _templates/models.csv
     - Scrape the README.md files from the GitHub repositories listed in models.csv
     - Generate overview markdown for each model
     - Store result in _output/docs/model-service~available-models.md
@@ -37,7 +37,7 @@ def generate_model_docs(filename="model-service~available-models.md"):
     )
 
     # Read commands.md input content
-    available_models_md = read_input_file(filename)
+    available_models_md = read_template_file(filename)
 
     # Insert DO NOT EDIT comment
     available_models_md = re.sub(

generator/methods/generate_commands_md.py

@@ -1,12 +1,12 @@
 from openad.helpers.output import output_text
-from .shared import read_input_file, read_source_file, write_output_file
+from .shared import read_template_file, read_source_file, write_output_file
 
 
 def oad_generate_concepts(filename="openad~helpers~concepts.py"):
     """
     Generate the concept.py file for the openad-toolkit repo.
 
-    - Start from the _input/openad~helpers~concepts.py template
+    - Start from the _templates/openad~helpers~concepts.py template
     - Fill in all descriptions from the _source directory
     - Store result in _output/openad-toolkit/openad~helpers~concepts.py
 
@@ -17,7 +17,7 @@ def oad_generate_concepts(filename="openad~helpers~concepts.py"):
     output_text(f"<h1>Generating <yellow>{filename}</yellow></h1>", pad_top=2)
 
     # Read concepts.py template content
-    concepts_py = read_input_file(filename)
+    concepts_py = read_template_file(filename)
 
     # Read source file content
     description_txt = read_source_file("description.txt")

generator/methods/generate_llm_descriptions_text.py

@@ -6,7 +6,7 @@
 this_dir = os.path.dirname(os.path.abspath(__file__))
 docs_dir = os.path.normpath(os.path.join(this_dir, "../../main/docs"))
 source_dir = os.path.normpath(os.path.join(this_dir, "../_source"))
-input_dir = os.path.normpath(os.path.join(this_dir, "../_input"))
+template_dir = os.path.normpath(os.path.join(this_dir, "../_templates"))
 output_dir = os.path.normpath(os.path.join(this_dir, "../_output"))
 openad_toolkit_dir = os.path.normpath(os.path.join(this_dir, "../../../openad-toolkit"))
 
@@ -63,9 +63,9 @@ def read_source_file(filename):
     return _read_file_content(path)
 
 
-def read_input_file(filename):
+def read_template_file(filename):
     """Return the content of a file from the input dir."""
-    path = os.path.join(input_dir, filename)
+    path = os.path.join(template_dir, filename)
     return _read_file_content(path)