[ENH] Speed up first completion (#399)

* Remove python 2 compat junk

* WIP

* Cut as many corners as possible while still remaining with python

* Fix type annotation

* cpp extension boilerplate

* meson build is now working

* WIP

* wip

* Implement a compiled extension which finds modules

* Compiled extension tested, working

* Don't use `python setup.py <whatever>`

* Don't use editable installs for testing

* Don't build/test with build isolation

* Switch order of args

* Correct a bad docstring

* Add pybind11 dep to CI...

* Enforce c++17 standard

* Fix the doc build configuration

* Use dependency groups for docs and lint; don't build docs in test

* Add 'build' as test dependency

* Make tests run in editable mode

* Add hypothesis; follow meson-python docs around editable mode

* Make epydoc and wheel direct dependencies of pyflyby

* Add setuptools and wheel, required by epydoc

* Make the install `pure: false`; prevent site-packages pollution

* Set lower bound on meson-python to 0.18.0 to enforce PEP639

* Add caching; progress bars aren't quite right yet

* Don't bother with progress bars

* Add a function to force the cache to be rebuilt

* Export rebuild_import_cache at top level

* Cleanup

* Expand import caching test

* Check that updating the mtime of an importer path regenerates the cache

* Add `appdirs` to autodoc_mock_imports; remove unused import

* Add `prompt-toolkit` as dependency

* Fix tests broken by new log messages

* Disable test broken due to log messages

* Remove patch_stdout calls; that's a problem for another time

* Try using two scenarios

* Try stripping out log lines; can't do anything else since subprocess...

* Add env variable for suppressing pyflyby cache rebuild log messages

Author: peytondmurray

.github/workflows/docs.yml

@@ -12,10 +12,8 @@ jobs:
       - uses: actions/setup-python@v5
       - name: Install dependencies
         run: |
-          pip install sphinx sphinx_rtd_theme sphinx-autodoc-typehints
-          python -m pip install --upgrade pip setuptools wheel
-          python -m pip install --upgrade rlipython ipykernel==5.4.3 requests jupyter flaky 'notebook<6.1' 'prompt_toolkit<3.0.15' wheel 'jupyter_console>=6.2' 'pytest-cov<3' ipython 'coverage<6.3' pytest-json-report
-          pip install -e .
+          pip install -U pip
+          pip install -v --group docs
       - name: Build docs
         run: |
           make html

.github/workflows/lint.yml

@@ -3,25 +3,22 @@ name: Lint
 on: [push, pull_request]
 
 jobs:
-  test:
+  lint:
     runs-on: "ubuntu-latest"
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.13"]
-
     steps:
     - uses: actions/checkout@v5
-    - name: Set up Python ${{ matrix.python-version }}
+    - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install and update Python dependencies on Python 3
+        python-version: 3.13
+    - name: Install pyflyby
       run: |
-        python -m pip install --upgrade pip setuptools wheel
-        python -m pip install --upgrade pyflakes flake8 mypy
-        python -m pip install types-six
-        pip install -e .
+        # Include build dependencies for run time; see
+        # https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#build-dependencies
+        # for details.
+        pip install meson-python meson ninja pybind11>=2.10.4
+        pip install setuptools wheel # needed for epydoc
+        pip install --no-build-isolation -ve . --group lint
     - name: Mypy
       run: |
         mypy lib/python --ignore-missing-imports
@@ -32,4 +29,3 @@ jobs:
     - name: Self-tidy-import
       run: |
         ./bin/tidy-imports -d lib/python/ tests/
-        

.github/workflows/test.yml

@@ -30,14 +30,17 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install and update Python dependencies on Python 3
+    - name: Install pyflyby
       run: |
-        python -m pip install --upgrade pip setuptools wheel
-        python -m pip install --upgrade "pexpect>=3.3" 'pytest<=8' rlipython 'ipykernel>=5.4.3' requests jupyter flaky 'notebook<6.1' wheel 'jupyter_console>=6.2' pytest-cov ipython coverage pytest-json-report hypothesis
-        pip install -e .
+        # Include build dependencies for run time; see
+        # https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#build-dependencies
+        # for details.
+        pip install meson-python meson ninja pybind11>=2.10.4
+        pip install setuptools wheel # needed for epydoc
+        pip install --no-build-isolation -ve .[test]
     - name: test release build
       run: |
-        python setup.py sdist bdist_wheel
+        python -m build
     - name: compileall
       run: |
         python -We:invalid -m compileall -f -q lib/ etc/;
@@ -62,10 +65,3 @@ jobs:
         name: pytest-timing-${{ matrix.os }}-${{ matrix.python-version }}
         path: ./report-*.json
     - uses: codecov/codecov-action@v5
-    - name: Build docs
-      if: ${{ matrix.python-version == '3.11'}}
-      run: |
-        pip install sphinx sphinx_rtd_theme sphinx-autodoc-typehints
-        cd doc
-        make html
-        cd ..

README.rst

@@ -459,7 +459,7 @@ Emacs support
 
 * To get a ``M-x tidy-imports`` command in GNU Emacs, add to your ``~/.emacs``::
 
-    (load "/path/to/pyflyby/lib/emacs/pyflyby.el")
+    (load "/<site-packages>/pyflyby/share/emacs/site-lisp/pyflyby.el")
 
 
 - Pyflyby.el doesn't yet work with XEmacs; patches welcome.
@@ -546,5 +546,3 @@ Release
 
 8. Check/update https://github.com/conda-forge/pyflyby-feedstock for new pyflyby
    release on conda-forge
-
-

doc/api/idents.rst

@@ -2,4 +2,3 @@ _idents module
 ==============
 .. automodule:: pyflyby._idents
    :members:
-   :exclude-members: _my_iskeyword

doc/conf.py

@@ -2,6 +2,8 @@
 
 # -- Path setup --------------------------------------------------------------
 import os
+import pathlib
+import re
 import sys
 sys.path.insert(0, os.path.abspath('../lib/python'))
 sys.path.insert(0, os.path.abspath('..'))
@@ -11,9 +13,21 @@
 copyright = '2019, Karl Chen'
 author = 'Karl Chen'
 # The full version, including alpha/beta/rc tags
-import pyflyby
 
-release = pyflyby.__version__
+def find_version():
+    # Extract version information via regex to avoid importing
+    project_root = pathlib.Path(__file__).parent.parent
+    with open(project_root / "lib" / "python" / "pyflyby" / "_version.py") as f:
+        version_match = re.search(
+            r"^__version__ = ['\"](?P<version>.*)['\"]$",
+            f.read(),
+            re.M,
+        )
+        if version_match:
+            return version_match.group("version")
+        raise RuntimeError("Unable to find version string.")
+
+release = find_version()
 
 
 # -- General configuration ---------------------------------------------------
@@ -29,6 +43,12 @@
     'private-members': True
 }
 
+autodoc_mock_imports = [
+    "pyflyby._fast_iter_modules",
+    "appdirs",
+    "prompt_toolkit",
+]
+
 html_theme_options = {
     'collapse_navigation': False,
     'navigation_depth': -1,

lib/python/pyflyby/__init__.py

@@ -31,6 +31,7 @@
                                         unload_ipython_extension)
 from   pyflyby._livepatch       import livepatch, xreload
 from   pyflyby._log             import logger
+from   pyflyby._modules         import rebuild_import_cache
 from   pyflyby._parse           import PythonBlock, PythonStatement
 from   pyflyby._saveframe       import saveframe
 from   pyflyby._saveframe_reader \

lib/python/pyflyby/_idents.py

@@ -5,20 +5,14 @@
 
 
 from   functools                import total_ordering
-from   keyword                  import kwlist
+from   keyword                  import iskeyword
 import re
 
 from   pyflyby._util            import cached_attribute, cmp
 
 from typing import Optional, Tuple, Dict
 
 
-# Don't consider "print" a keyword, in order to be compatible with user code
-# that uses "from __future__ import print_function".
-_my_kwlist = list(kwlist)
-_my_iskeyword = frozenset(_my_kwlist).__contains__
-
-
 # TODO: use DottedIdentifier.prefixes
 def dotted_prefixes(dotted_name, reverse=False):
     """
@@ -114,7 +108,7 @@ def is_identifier(s: str, dotted: bool = False, prefix: bool = False):
         return is_identifier(s + '_', dotted=dotted, prefix=False)
     if dotted:
         return all(is_identifier(w, dotted=False) for w in s.split('.'))
-    return s.isidentifier() and not _my_iskeyword(s)
+    return s.isidentifier() and not iskeyword(s)
 
 
 def brace_identifiers(text):