Make ICortex work on Google Colab, doc improvements (#29)

* Improve history format

* Wip

* Improved docs, config init

* Updated dependencies to work with Python 3.7 (used by google colab)

* Undid output capturing for now

Author: osolmaz

README.md

@@ -15,24 +15,23 @@
 </p>
 <hr />
 
-ICortex is a [Jupyter kernel](https://jupyter-client.readthedocs.io/en/latest/kernels.html) that lets you develop **soft programs**:
+tl;dr in goes English, out comes Python:
 
-- sets of instructions (i.e. prompts) [written in natural language](https://en.wikipedia.org/wiki/Natural-language_programming) (such as English)
-- processed by language models that generate Python code
-- to perform useful work in various contexts
-- more flexibly than regular software.
+https://user-images.githubusercontent.com/2453968/199964302-0dbe1d7d-81c9-4244-a9f2-9d959775e471.mp4
 
-To put it simply—in goes English, out comes Python:
+ICortex enables you to develop **soft programs**:
 
-https://user-images.githubusercontent.com/2453968/199964302-0dbe1d7d-81c9-4244-a9f2-9d959775e471.mp4
+> *Soft program:* a set of instructions (i.e. prompts) [written in natural language](https://en.wikipedia.org/wiki/Natural-language_programming) (e.g. English), processed by a language model that generates code at a lower layer of abstraction (e.g. Python), to perform work more flexibly than regular software.
+
+In other words, ICortex is a **natural language programming** (NLP) framework that enables you to write code in English, and then run it in Python. It aims to make programming more accessible to non-programmers.
 
-ICortex is ...
+ICortex is designed to be …
 
-- a drop-in replacement for the IPython kernel. Prompts can be executed with the [magic commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html) `%prompt` or `%p` for short.
-- interactive—install missing packages directly, decide whether to execute the generated code or not, and so on, directly in the Jupyter Notebook cell.
-- open source and fully extensible—if you think we are missing a model or an API, you can request it by creating an issue, or implement it yourself by subclassing `ServiceBase` under [`icortex/services`](icortex/services).
+- a drop-in replacement for the [IPython kernel](https://ipython.org/). Prompts are executed via [magic commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html) such as `%prompt`.
+- interactive—automatically install missing packages, decide whether to execute the generated code or not, and so on, directly in the Jupyter Notebook cell.
+- open source and fully extensible—ICortex introduces a [domain-specific language](https://en.wikipedia.org/wiki/Domain-specific_language) for orchestrating various code generation services. If you think we are missing a model or an API, you can request it by creating an issue, or implement it yourself by subclassing `ServiceBase` under [`icortex/services`](icortex/services).
 
-It is similar to [Github Copilot](https://github.com/features/copilot) but with certain differences that make it stand out:
+ICortex is similar to [Github Copilot](https://github.com/features/copilot) but with certain differences that make it stand out:
 
 | Feature | GitHub Copilot | ICortex |
 |---|:---:|:---:|
@@ -48,6 +47,8 @@ ICortex is currently in alpha, so expect breaking changes. We are giving free cr
 
 ## Installation
 
+### Locally
+
 Install directly from PyPI:
 
 ```sh
@@ -56,6 +57,15 @@ pip install icortex
 python -m icortex.kernel.install
 ```
 
+### On Google Colab
+
+[Google Colab](https://colab.research.google.com/) is a restricted computing environment that does not allow installing new Jupyter kernels. However, you can still use ICortex by running the following code in a Colab notebook:
+
+```
+!pip install icortex
+import icortex.init
+```
+
 ## Quickstart
 
 [Click here to visit the docs and get started using ICortex](https://icortex.readthedocs.io/en/latest/quickstart.html).

docs/source/conf.py

@@ -25,6 +25,7 @@
     "sphinx.ext.coverage",
     "sphinx.ext.napoleon",
     "sphinx_copybutton",
+    "sphinx.ext.autosectionlabel",
 ]
 
 templates_path = ["_templates"]

docs/source/index.rst

@@ -6,36 +6,27 @@
 ICortex
 =======
 
+tl;dr in goes English, out comes Python:
 
-ICortex is a `Jupyter
-kernel <https://jupyter-client.readthedocs.io/en/latest/kernels.html>`__
-that lets you develop **soft programs**:
+.. video:: https://user-images.githubusercontent.com/2453968/199964302-0dbe1d7d-81c9-4244-a9f2-9d959775e471.mp4
+   :width: 100%
 
--  sets of instructions (i.e. prompts) `written in natural
-   language <https://en.wikipedia.org/wiki/Natural-language_programming>`__
-   (such as English)
--  processed by language models that generate Python code
--  to perform useful work in various contexts
--  more flexibly than regular software.
+ICortex enables you to develop **soft programs**:
 
-To put it simply—in goes English, out comes Python:
+   *Soft program:* a set of instructions (i.e. prompts) `written in natural language <https://en.wikipedia.org/wiki/Natural-language_programming>`__ (e.g. English), processed by a language model that generates code at a lower layer of abstraction (e.g. Python), to perform work more flexibly than regular software.
 
-.. video:: https://user-images.githubusercontent.com/2453968/199964302-0dbe1d7d-81c9-4244-a9f2-9d959775e471.mp4
-   :width: 100%
+In other words, ICortex is a **natural language programming** (NLP) framework that enables you to write code in English, and then run it in Python. It aims to make programming more accessible to non-programmers.
 
-ICortex is …
+ICortex is designed to be …
 
--  a drop-in replacement for the IPython kernel. Prompts can be executed
-   with the `magic
+-  a drop-in replacement for the `IPython kernel <https://ipython.org/>`__. Prompts can be executed
+   via `magic
    commands <https://ipython.readthedocs.io/en/stable/interactive/magics.html>`__
-   ``%prompt`` or ``%p`` for short.
--  interactive—install missing packages directly, decide whether to
+   such as ``%prompt``.
+-  interactive—automatically install missing packages, decide whether to
    execute the generated code or not, and so on, directly in the Jupyter
    Notebook cell.
--  open source and fully extensible—if you think we are missing a model
-   or an API, you can request it by creating an issue, or implement it
-   yourself by subclassing ``ServiceBase`` under
-   `icortex/services <https://github.com/textcortex/icortex/tree/main/icortex/services>`__.
+-  open source and fully extensible—ICortex introduces a `domain-specific language <https://en.wikipedia.org/wiki/Domain-specific_language>`__ for orchestrating various code generation services. If you think we are missing a model or an API, you can request it by creating an issue, or implement it yourself by subclassing `ServiceBase` under `icortex/services <https://github.com/textcortex/icortex/tree/main/icortex/services>`__.
 
 Get started
 -----------

docs/source/quickstart.rst

@@ -5,7 +5,10 @@ Quickstart
 Installation
 ------------
 
-Install ICortex from PyPI, along with JupyterLab:
+Locally
+~~~~~~~
+
+Install ICortex on your local machine, along with JupyterLab:
 
 .. code:: sh
 
@@ -18,15 +21,37 @@ The second line is needed to install the kernel spec into Jupyter, otherwise, yo
 
    jupyter kernelspec list
 
-ICortex should be visible in the list of available kernels.
+ICortex needs to be visible in the list of available kernels.
+
+On Google Colab
+~~~~~~~~~~~~~~~
+
+`Google Colab <https://colab.research.google.com/>`__ is a restricted computing environment that does not allow installing new Jupyter kernels. However, you can still use ICortex by running the following code in a Colab notebook:
+
+.. code:: text
+
+   !pip install icortex
+   import icortex.init
 
 Using ICortex
 -------------
 
+Create a TextCortex account
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. tip::
+    **This does not require you to enter any payment information.** Your account will receive free credits to try out the service. If you already have an account, you can skip this step.
+
+
+You need a TextCortex account to connect to TextCortex's code generation API. If you are planning to use a different code generation service, you can skip to :ref:`Start JupyterLab`.
+
+|signup_link|
+
+Once you have an account, proceed to the next section.
+
 Start JupyterLab
 ~~~~~~~~~~~~~~~~
 
-
 Create a directory for your new project, and start JupyterLab:
 
 ::
@@ -35,7 +60,7 @@ Create a directory for your new project, and start JupyterLab:
    cd new-icortex-project/
    jupyter lab
 
-Once JupyterLab is up and running, create a new notebook that uses ICortex.
+Once JupyterLab is up and running, create a new notebook with ICortex kernel. If you are using Google Colab, you can skip this step.
 
 .. important::
     If you don't see ICortex in the list of available kernels, you may have skipped kernel installation above—run ``python -m icortex.kernel.install`` and restart JupyterLab. If you still don't see ICortex there, `create a new installation issue on GitHub <https://github.com/textcortex/icortex/issues/new>`__.
@@ -60,11 +85,11 @@ After running ``%icortex init`` you should see the following message:
 ::
 
     Which code generation service would you like to use?
-    Variables: textcortex, huggingface, openai
-    Default [textcortex]
+    Available services: textcortex, huggingface, openai
+    Default: textcortex  (Press enter to select the default option)
 
 
-ICortex supports different code generation services such as the TextCortex API, OpenAI Codex API, local HuggingFace transformers, and so on. We recommend that you start with TextCortex. It is already selected by default—**press Enter once** to move on to the next step.
+ICortex supports different code generation services. We recommend that you start with TextCortex. It is selected by default—**press Enter once** to move on to the next step.
 
 In the next step, the dialog will ask whether to use the default parameters for TextCortex's code generation service:
 
@@ -80,18 +105,6 @@ In the next step, the dialog will ask whether to use the default parameters for
    https://app.textcortex.com/user/dashboard/settings/api-key)
 
 
-Do not type anything yet, and proceed to the next section.
-
-Create a TextCortex account
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You need a TextCortex account to connect to TextCortex's code generation API.
-
-|signup_link|.
-
-.. note::
-    This does not require you to enter any payment information and your account will receive free credits to try out the service. If you already have an account, you can skip this step.
-
 Next, |api_key_link|.
 
 Copy your API key from the dashboard, go back to the Jupyter notebook where you initially ran ``%icortex init``, and paste it in the dialog where it was asked for. Press Enter to continue.
@@ -192,7 +205,7 @@ ICortex printed the code generated by the API and is now asking whether it shoul
 
 .. |signup_link| raw:: html
 
-   <a href="https://app.textcortex.com/user/signup" target="_blank">Click here sign up on the website</a>
+   <blockquote><div style="text-align: center"><p><a href="https://app.textcortex.com/user/signup?registration_source=icortex" target="_blank">Click here sign up at TextCortex</a></p></div></blockquote>
 
 .. |api_key_link| raw:: html
 

icortex/__init__.py

@@ -1,6 +1,6 @@
-from icortex.kernel import ICortexKernel, print_service_help, get_icortex_kernel
+from icortex.kernel import ICortexKernel, print_service_help, get_icortex
 
 import icortex.services
-import importlib.metadata
+import importlib_metadata
 
-__version__ = importlib.metadata.version("icortex")
+__version__ = importlib_metadata.version("icortex")

icortex/cli.py

@@ -184,10 +184,10 @@ def main(argv=None, prog=None, kernel=None):
     elif args.command == "help":
         parser.print_help()
     elif args.command == "shell" or args.command is None:
-        from icortex.kernel import get_icortex_kernel
+        from icortex.kernel import get_icortex
         from icortex.kernel.app import ZMQTerminalICortexApp
 
-        kernel = get_icortex_kernel()
+        kernel = get_icortex()
         if kernel is None:
             ZMQTerminalICortexApp.launch_instance()
         else:

icortex/config.py

@@ -50,7 +50,7 @@ def set_service_var(self, var_name: str, var_value) -> bool:
         success = self.write_config()
         if success:
             print(f"Set variable {var_name} to {cast_value}.")
-            # kernel = get_icortex_kernel()
+            # kernel = get_icortex()
             if self.kernel is not None:
                 self.set_service()
             return True
@@ -78,7 +78,7 @@ def set_service_config(self, service_name: str, hard_init=False) -> bool:
         success = self.write_config()
         if success:
             print(f"Set service to {service_name} successfully.")
-            # kernel = get_icortex_kernel()
+            # kernel = get_icortex()
             if self.kernel is not None:
                 self.set_service()
             return True
@@ -87,7 +87,7 @@ def set_service_config(self, service_name: str, hard_init=False) -> bool:
 
     def set_service(self):
         # TODO: pass the --config flag from icortex somehow
-        # kernel = get_icortex_kernel()
+        # kernel = get_icortex()
         if self.kernel is None:
             return False
 
@@ -113,11 +113,12 @@ def set_service(self):
     def ask_which_service(self) -> str:
         sorted_services = get_available_services()
         service_name = prompt_input(
-            "Which code generation service would you like to use?\nVariables: "
+            "Which code generation service would you like to use?\nAvailable services: "
             + ", ".join(sorted_services)
-            + "\nDefault",
+            + "\nDefault:",
             type=str,
             default=DEFAULT_SERVICE,
+            press_enter=True,
         )
         return service_name
 

icortex/context.py

@@ -1,10 +1,10 @@
 import typing as t
-import importlib.metadata
+import importlib_metadata
 from copy import deepcopy
 import platform
 from icortex.defaults import DEFAULT_HISTORY_VAR
 
-icortex_version = importlib.metadata.version("icortex")
+icortex_version = importlib_metadata.version("icortex")
 
 INITIAL_HISTORY_VAL = {
     "metadata": {

icortex/helper.py

@@ -47,9 +47,11 @@ def yes_no_input(message: str, default=True) -> bool:
     return (user_input == "" and default) or user_input.strip().lower() == "y"
 
 
-def prompt_input(message: str, type=str, default=None):
+def prompt_input(message: str, type=str, default=None, press_enter=False):
     if default:
-        message += f" [{default}]"
+        message += f" {default}"
+        if press_enter:
+            message += "  (Press enter to select the default option)"
 
     print(message)
     user_input = input()

icortex/init.py

@@ -0,0 +1,15 @@
+import logging
+from icortex.kernel import ICortexShell
+import __main__
+
+# import ipdb; ipdb.set_trace()
+# If the kernel is IPython, load the magics.
+if "get_icortex" in __main__.__dict__:
+    logging.warning("ICortex is already initialized, skipping.")
+elif "get_ipython" in __main__.__dict__ and "get_icortex" not in __main__.__dict__:
+    # from icortex.magics import load_ipython_extension
+    # ipython = globals()["get_ipython"]()
+    ICortexShell._init_icortex_shell(__main__.get_ipython())
+    # load_ipython_extension(get_ipython())
+else:
+    raise Exception("IPython is not available, cannot initialize ICortex.")