Skypilot upgrade to Main (#56)

* Basic persistent GUI server prototype in separate thread

* Added install and launch gui commands

* gui api

* gui api updates

* Added GUI api endpoints

* Remove openad-gui files from git

* stash

* stash laptop

* demo prototype

* stash

* switch workspace GUI API

* Added hydrogen atoms to 3d coordinates data

* Websocket experiment

* created filesystem worker and refactored 'list files' command to use the same method as the GUI api, plus added optional path parameter

* Refactored fs_get_file and fs_get_workspace files to follow similar data structure and moved fileType logic to backend

* stash

* Before removing index_only functionality from get_molset()

* Indexing done right wip

* Stash

* stash before refactoring molset fs/cache reading

* sdf processing + api refactor

* added 'enriched' parameter to molecule object

* fixed list files

* Removed old messy molecule API functions

* implemented reverse sort

* Support for invalid file paths and smiles in .smi files

* stash

* various

* cleanup & debugging

* Mega commit regarding persistent GUI updates

* RXN login error message update

* updated grammar in models to allow auth_group and service name without quotes

* Merged conflicts #2

* fix for unspported PDF viewing

* changed host for flask apps to 0.0.0.0

* changed host for flask apps to 0.0.0.0 port 5005 starter

* cleaned up molecules

* adding auth group to catalog model service

* adding auth group to catalog model service

* fix opena api session management

* fine tune jupyter install

* install update

* install update

* install update

* fix openad rxn  api session management

* fix openad rxn  api session management

* Added command to show working set molecules

* stash

* Added openad-gui build folder

* stash show_molset

* Moved openad-gui folder inside openad folder so it is part of pip install, renamed to gui-build

* Added gui-build folder

* Move to port 0.0.0.0

* New place for the gui-build dir, updated to latest version

* show molset command + updated gui launch/shutdown messages

* Centralized duplicate normalize_mol_df and _smiles_to_iupac functions into mol_functions

* mymols fix

* debugged dataviewer display and updated notebooks

* debugged dataviewer display and updated notebooks

* enabled append for loading molecules

* First step for  command

* Remove icloud duplicate files

* formatting and fox new_molecule bug

* merge properties bug

* gui notebook and readme

* markdown test

* readme & gui demo

* gui readme update

* stash

* Support for display molecules from a dataframe

* stash

* prox server major upgrade and debug visuals

* prox server major upgrade and debug visuals

* prox server major upgrade and debug visuals

* stash

* Undid renaming gui-build to gui-build-proxy

* paths for routes

* acheived build

* merge conflicts mergibg with gui_api_moe #2

* Merge conflicts #3

* Merge wrapup

* Simplified/cleaned up JL_PROXY proxy code in gui_launcher

* Demo Notebook design

* patch launcher

* notebook updates

* stash

* gui-build & gui-build-proxy wip

* Updated gui build folders

* Updated gui build folders

* New gui build

* notebook upgrade

* fine tuning anf testing

* fine tuning anf testing

* added warning for parameters in service call order

* added warning for parameters in service call order

* linter

* readme fix

---------

Co-authored-by: Moenen Erbuer <moenen@shapish.com>
Co-authored-by: Moenen Erbuer <moenen@arthur.io>
Co-authored-by: Phil Downey <phildowney@pd-work-mac.local>
Co-authored-by: Moenen Erbuer <moenen.erbuer@ibm.com>

Author: 4 people

README.md

@@ -23,13 +23,16 @@ for (var i=0; i< urls.length; i++) {
 
 -->
 
+
 # OpenAD Beta
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/openad)](https://pypi.org/project/openad/)
 [![PyPI version](https://img.shields.io/pypi/v/openad)](https://pypi.org/project/openad/)
 [![License MIT](https://img.shields.io/github/license/acceleratedscience/open-ad-toolkit)](https://opensource.org/licenses/MIT)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Docs](https://img.shields.io/badge/website-live-brightgreen)](https://acceleratedscience.github.io/openad-docs/)
 
+
+
 **Open Accelerated Discovery Client**<br>
 [Documentation](https://acceleratedscience.github.io/openad-docs/)
 
@@ -39,17 +42,20 @@ The goal of openAD is to provide a common language for scientists to interact wi
 
 ---
 > **Pre-install Note:** 
-For updating to 0.3.0 or above first remove toolkits `remove toolkit DS4SD` and `remove toolkit RXN` prior to updating
+For updating to 0.4.0 or above first remove toolkits `remove toolkit DS4SD` and `remove toolkit RXN` prior to updating
 
 > **Whats New ?**
 -  `%Openadd` has been added to the magic commands to provide pure data type results for data returning commands
-- Upgraded IBM BAM model support for latest IBm generative AI embeddings and Langchain 
+- Upgraded skypilot to 0.6.0 
+- Support for deploying in OpenSHift AI / Open Data hub workbench or podman/docker image `https://github.com/acceleratedscience/openad_workbench`
+- Support for Application API
 - Property and Data Set Generation Services
    We support the following Model Services
      - GT4SD Generation Services  `git@github.com:acceleratedscience/generation_inference_service.git`
      - GT4SD Property Services `git@github.com:acceleratedscience/property_inference_service.git`
      - GT4SD MoleR Generation `git@github.com:acceleratedscience/moler_inference_service.git`
      - GT4SD Molformer `git@github.com:acceleratedscience/molformer_inference_service.git`
+
     Pre-Requisite is that you have a AWS Account and can launch your own EC2 Instances Or someone else can launch them for you and you can catalog a Remote Service via URL.
 
         **Example:**

_for_testing/Demonstration.ipynb

@@ -257,9 +257,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "ad-venv",
+   "display_name": "venv",
    "language": "python",
-   "name": "ad-venv"
+   "name": "venv"
   },
   "language_info": {
    "codemirror_mode": {
@@ -271,7 +271,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

_for_testing/test_everything.ipynb

@@ -34,7 +34,7 @@
 from openad.plugins.style_parser import tags_to_markdown
 from openad.helpers.output import output_error, output_text, output_success
 from openad.helpers.output_msgs import msg
-from openad.helpers.general import open_file, write_file
+from openad.helpers.files import open_file, write_file
 
 # Get the repo path, this python file's parent folder.
 REPO_PATH = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
@@ -202,9 +202,9 @@ def _parse_description(description):
     # be treated as part of the blockquote.
     description = re.sub(
         r"(\*\*Note:\*\*.+?)(\n{1,})",
-        lambda match: f"  > {match.group(1)}\n\n"
-        if len(match.group(2)) == 1
-        else f"  > {match.group(1)}{match.group(2)}",
+        lambda match: (
+            f"  > {match.group(1)}\n\n" if len(match.group(2)) == 1 else f"  > {match.group(1)}{match.group(2)}"
+        ),
         description,
         flags=re.MULTILINE,
     )

docs/generate_docs.py

@@ -6,7 +6,7 @@ get workspace [ <workspace_name> ];Workspaces
 create workspace <workspace_name> [ description('<description>') on path '<path>' ];Workspaces
 remove workspace <workspace_name> ;Workspaces
 list workspaces;Workspaces
-add molecule <name> | <smiles> | <inchi> | <inchikey> | <cid> [ as '<name>' ] [ basic ] [force ];Molecules
+add molecule <name> | <smiles> | <inchi> | <inchikey> | <cid> [ as '<name>' ] [ basic ] [ force ];Molecules
 display molecule <name> | <smiles> | <inchi> | <inchikey> | <cid>;Molecules
 display sources <name> | <smiles> | <inchi> | <inchikey> | <cid>;Molecules
 rename molecule <molecule_identifer_string> as <molecule_name>;Molecules

docs/output/csv/commands.csv

@@ -30,7 +30,7 @@ To update it, see openad/docs/generate_docs.py
   - [Search Collections](#search-collections)
   - [Collections](#collections)
 - [RXN](#rxn)
-  - [General](#general)
+  - [General](#general-1)
   - [Retrosynthesis](#retrosynthesis)
   - [Prediction](#prediction)
 - [ST4SD](#st4sd)
@@ -77,8 +77,8 @@ Lists all your workspaces. <br><br>
 
 ### Molecules
 
-`add molecule <name> | <smiles> | <inchi> | <inchikey> | <cid> [ as '<name>' ] [ basic ] [force ]`{: .cmd }
-This command is how you add a molecule to a current working list of molecules in memory. When adding a molecule by name, this name will become the molecule's identifying string.  <br> 
+`add molecule <name> | <smiles> | <inchi> | <inchikey> | <cid> [ as '<name>' ] [ basic ] [ force ]`{: .cmd }
+This command is how you add a molecule to a current working list of molecules in memory. When adding a molecule by name, this name will become the molecule's identifying string. <br>
 
 It will take any molecules identifier from the following categories: <br> 
     -`smiles ` <br> 

docs/output/markdown/commands.md

@@ -1,4 +1,4 @@
-from openad.app.magic.openad import AD
+from openad.app.magic.openad_magic import AD
 
 
 def load_ipython_extension(ipython):

openad/app/magic/__init__.py

@@ -11,11 +11,14 @@
 import uuid
 from cmd import Cmd
 from pandas import DataFrame
+import atexit
 
 # Main
 from openad.app.main_lib import lang_parse, initialise, set_context, unset_context
 from openad.toolkit.toolkit_main import load_toolkit
 from openad.app import login_manager
+from openad.gui.gui_launcher import gui_init, GUI_SERVER, gui_shutdown
+from openad.gui.ws_server import ws_server  # Web socket server for gui - experimental
 from openad.helpers.output import output_table
 
 # Core
@@ -41,6 +44,7 @@
 from openad.helpers.output_msgs import msg
 from openad.helpers.general import refresh_prompt
 from openad.helpers.splash import splash
+from openad.helpers.files import empty_trash
 from openad.helpers.output_content import info_workspaces, info_toolkits, info_runs, info_context
 
 # Globals
@@ -57,6 +61,7 @@
 import inspect
 import importlib
 
+
 MAGIC_PROMPT = None
 PLUGIN_CLASS_LIST = []
 installed_packages = pkg_resources.working_set
@@ -138,19 +143,6 @@ class RUNCMD(Cmd):
     molecule_list = []
     last_external_molecule = None
 
-    def workspace_path(self, workspace: str):
-        """Returns the default workspace directory path"""
-        try:
-            x = os.path.expanduser(self.settings["paths"][workspace.upper()] + "/" + workspace.upper())
-            return x
-        except Exception:  # pylint: disable=broad-exception-caught
-            # various exceptions can cause this... Any error results in same outcome
-            return os.path.expanduser(_meta_workspaces + "/" + workspace.upper())
-
-    def set_workspace_path(self, workspace: str, path: str):
-        """Sets the current workspace path in the settings dictionary"""
-        self.settings["paths"][workspace.upper()] = os.path.expanduser(path)
-
     # Initialises the class for Run command.
     def __init__(self, completekey="Tab", api=False):
         super().__init__()
@@ -215,6 +207,20 @@ def __init__(self, completekey="Tab", api=False):
 
         output_train_statements(self)
 
+    def workspace_path(self, workspace: str = None):
+        """Returns the default workspace directory path"""
+        workspace = workspace.upper() if workspace else self.settings["workspace"].upper()
+        try:
+            x = os.path.expanduser(self.settings["paths"][workspace.upper()] + "/" + workspace.upper())
+            return x
+        except Exception:  # pylint: disable=broad-exception-caught
+            # various exceptions can cause this... Any error results in same outcome
+            return os.path.expanduser(_meta_workspaces + "/" + workspace.upper())
+
+    def set_workspace_path(self, workspace: str, path: str):
+        """Sets the current workspace path in the settings dictionary"""
+        self.settings["paths"][workspace.upper()] = os.path.expanduser(path)
+
     def do_help(self, inp, display_info=True, starts_with_only=False, **kwargs):
         """CMD class called function:
         Display help about a command, for example 'list'.
@@ -605,8 +611,13 @@ def complete(self, text, state):
     # Catches the exit command
     def do_exit(self, dummy_inp_do_not_remove):
         """CMD Funcion: called on exit command"""
+        try:
+            cleanup()
+        except:
+            pass
         write_registry(self.settings, self, True)
         delete_session_registry(self.session_id)
+
         # exiting the application. Shorthand: x q.
         return True
 
@@ -821,7 +832,7 @@ def error_first_word_grabber(error):
     return str(word)
 
 
-# Main execution application
+# Main execution application.
 # If the application is called with parameters, it executes the parameters.
 # If called without parameters, the command line enters the shell environment.
 # History is only kept for commands executed once in the shell.
@@ -844,6 +855,7 @@ def api_remote(
     - It is deliberate that the whole RUNCMD class object is not kept alive as there is no logical
       exit point for magic commands, unlike a command line.
     """
+
     global MAGIC_PROMPT
     # GLOBAL_SETTINGS["display"] = "notebook"
 
@@ -859,6 +871,7 @@ def api_remote(
         MAGIC_PROMPT = magic_prompt
     else:
         magic_prompt = MAGIC_PROMPT
+
     if api_context["workspace"] is None:
         api_context["workspace"] = magic_prompt.settings["workspace"]
     else:
@@ -938,6 +951,9 @@ def cmd_line():
         a_space = " "
     try:
         command_line = RUNCMD()
+        # Launch the GUI if it is installed.
+        # gui_init(command_line)
+        # ws_server(command_line) # Experimental
     except KeyboardInterrupt:
         output_error(msg("err_key_exit_before_init"))
         return
@@ -1000,15 +1016,25 @@ def cmd_line():
                 # The cmdloop parameter controls the startup screen, it overrides self.intro.
                 command_line.cmdloop(splash(command_line.settings["context"], command_line, startup=True))
                 lets_exit = True
+
             except KeyboardInterrupt:
                 command_line.postloop()
+
                 if confirm_prompt("Are you sure you wish to exit?", default=True):
+                    # from openad.gui.gui_launcher import gui_shutdown
+                    empty_trash(command_line)
                     lets_exit = True
                     command_line.do_exit("dummy do not remove")
+
             except Exception as err:  # pylint: disable=broad-exception-caught
                 # We do not know what the error could be, so no point in being more specific.
                 output_error(msg("err_invalid_cmd", err), command_line)
 
 
+def cleanup():
+    gui_shutdown(ignore_warning=True)
+
+
+atexit.register(cleanup)
 if __name__ == "__main__":
     cmd_line()