Start to check type hints gradually (#2714)

* Start to gradually add type hints

Use mypy-baseline to ignore all existing errors for now while avoiding adding new ones in new code.

* Remove Python 2 fallback imports

* Remove py2compat layer

* Avoid redefining a tube helper function

* Remove explicit new-style class (object) syntax

old-style classes were removed in Python 3 so there is no distinction anymore.

* Ignore type error in shellcraft helper script

* Fix repeatedly changing `linux_dirent.d_name`

* Avoid typechecking windows_termcap on Linux

We run mypy on Linux in CI, ignore Windows related import errors

* Run mypy-baseline in CI

* Add type hint plan to CONTRIBUTING

* Sort the mypy baseline

Run with `--sort-baseline` for better diffs.

`mypy | mypy-baseline sync --sort-baseline`

* Switch dev dependencies to dependency-groups

The dependency groups aren't baked into build metadata and won't appear on Pypi.

Author: peace-maker

.github/ISSUE_TEMPLATE/bug_report.md

@@ -39,7 +39,7 @@ You should see `[DEBUG]` statements that show what's happening behind the scenes
 
 ## Verify on Ubuntu
 
-If possible, please verify that your issue occurs on 64-bit Ubuntu 22.04.  We provide a Dockerfile based on Ubuntu 22.04 via `docker.io` to make this super simple, no VM required!
+If possible, please verify that your issue occurs on 64-bit Ubuntu 24.04.  We provide a Dockerfile based on Ubuntu 24.04 via `docker.io` to make this super simple, no VM required!
 
 ```sh
 # Download the Docker image

.github/PULL_REQUEST_TEMPLATE.md

@@ -24,7 +24,7 @@ You can always [change the branch][change] after you create the PR if it's again
 
 [contributing]: https://github.com/Gallopsled/pwntools/blob/dev/CONTRIBUTING.md
 [testing]: https://github.com/Gallopsled/pwntools/blob/dev/TESTING.md
-[change]: https://github.com/blog/2224-change-the-base-branch-of-a-pull-request
+[change]: https://github.blog/news-insights/product-news/change-the-base-branch-of-a-pull-request/
 
 ## Changelog
 

.github/workflows/lint.yml

@@ -36,3 +36,8 @@ jobs:
       run: |
         pip install vermin
         vermin -vvv --no-tips -t=3.10- --violations  ./pwnlib ./pwn
+
+    - name: MyPy type hint baseline
+      run: |
+        pip install -U . --group dev
+        mypy | mypy-baseline filter

CONTRIBUTING.md

@@ -2,32 +2,57 @@
 
 Github has a great guide for contributing to open source projects:
 
-- [Contributing to a project](https://guides.github.com/activities/forking/)
-- [Fork the repository](https://guides.github.com/activities/forking/#fork)
-- [Clone your fork](https://guides.github.com/activities/forking/#clone)
-- [Making and pushing changes](https://guides.github.com/activities/forking/#making-changes)
-- [Making a Pull Request](https://guides.github.com/activities/forking/#making-a-pull-request)
-- [Huzzah!](https://guides.github.com/activities/forking/#huzzah)
+- [Contributing to a project](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project)
+- [Fork the repository](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project#creating-your-own-copy-of-a-project)
+- [Clone your fork](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project#cloning-a-fork-to-your-computer)
+- [Creating a branch to work on](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project#creating-a-branch-to-work-on)
+- [Making and pushing changes](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project#making-and-pushing-changes)
+- [Making a Pull Request](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project#making-a-pull-request)
 
 ## pwntools Specifics
 
 In general, we like to keep things documented.  You should add documentation to any new functionality, and update it for any changed functionality.  Our docstrings use the [Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html#example-google).
 
-After you have documentation, you should add a [doctest](https://docs.python.org/2/library/doctest.html).
+After you have documentation, you should add a [doctest](https://docs.python.org/3/library/doctest.html).
 
 Finally, it is probably a good idea to run the test suite locally before doing
 the pull-request to make sure everything works, however this is not a
-requirement.
+requirement. See the [TESTING](TESTING.md) documentation on how to do that.
 
 Once you are ready to do a pull-request, you should figure out if your changes
 constitutes a new feature or a bugfix in stable or beta. If it is a bugfix in
 stable or beta, you should do the pull-request against the branch in question,
 and otherwise your pull-request should be against the dev branch.
 
-Once you do the pull-request Travis CI will run the test-suite on it. Once it
+Once you do the pull-request Github Actions will run the test-suite on it. Once it
 passes one of the core developers will look at your pull request, possibly
 comment on it and then hopefully merge it into the branch in question.
 
+## Python Type Hints
+
+Since Pwntools 5 dropped support for Python 2, type hints are added gradually to the existing code base.
+[Type hints in Python](https://typing.python.org/en/latest/spec/index.html) allow static analysis tools and IDEs to help catch errors and provide a better developer
+experience without running the code.
+
+New code should include type hints where applicable and shouldn't add new type warnings.
+You can use [this cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) as a reference for the type hint syntax.
+
+Pwntools uses [mypy](https://mypy.readthedocs.io/en/stable/index.html) as a static type checker. Instead of trying to
+introduce type hints everywhere at once, Pwntools uses [mypy-baseline](https://mypy-baseline.orsinium.dev/usage) to gradually reduce the number of problems over time.
+
+To run the type checker locally, you can install Pwntools with the `dev` extra and check for new errors (`--group dev` requires pip 25.1+):
+
+```shell
+$ pip install -U . --group dev
+$ mypy | mypy-baseline filter
+```
+
+If you fixed some problems, you can update the list of known errors in `mypy-baseline.txt` using
+
+```shell
+$ mypy | mypy-baseline sync --sort-baseline
+```
+
 ## Automated Testing
 
 Pull requests against Pwntools require at a minimum that no tests have been broken, and ideally each pull request will include new tests to ensure that all of the functionality works as intended.

TESTING.md

@@ -10,12 +10,19 @@ To run the test suite, it is best to use Ubuntu 22.04 or 24.04, and run the foll
 bash travis/install.sh
 bash travis/ssh_setup.sh
 pip install --upgrade --editable .
+pip install --upgrade -r docs/requirements.txt
 PWNLIB_NOTERM=1 make -C docs doctest
 ```
 
+You can run doctests for a single file for faster iteration:
+
+```sh
+PWNLIB_NOTERM=1 python -m sphinx -b doctest docs/source docs/build/doctest docs/source/elf/elf.rst
+```
+
 ## Testing in Docker
 
-A `Dockerfile` has been provided which has a clean testing environment with Ubuntu Jammy.  It is very similar to the online Github Actions CI testing environment, but uses a more modern version of Ubuntu.
+A `Dockerfile` has been provided which has a clean testing environment with Ubuntu Noble.  It is very similar to the online Github Actions CI testing environment.
 
 See `travis/docker/README.md` for more information.
 

docs/source/conf.py

@@ -85,7 +85,7 @@ def filter(self, record):
 # Sphinx modifies sys.stdout, and context.log_terminal has
 # a reference to the original instance.  We need to update
 # it for logging to be captured.
-class stdout(object):
+class stdout:
     def __getattr__(self, name):
         return getattr(sys.stdout, name)
     def __setattr__(self, name, value):

mypy-baseline.txt

@@ -0,0 +1,107 @@
+pwn/__init__.py:0: error: Incompatible types in assignment (expression has type "PwnlibArgs", variable has type Module)  [assignment]
+pwn/__init__.py:0: error: Module has no attribute "update"  [attr-defined]
+pwn/toplevel.py:0: error: Incompatible import of "size" (imported name has type "Callable[[Any, Any, Any], Any]", local name has type "str")  [assignment]
+pwn/toplevel.py:0: error: Incompatible import of "size" (imported name has type "Callable[[Any, Any, Any], Any]", local name has type "str")  [assignment]
+pwn/toplevel.py:0: error: Incompatible import of "size" (imported name has type "int", local name has type "str")  [assignment]
+pwnlib/abi.py:0: error: Need type annotation for "register_arguments" (hint: "register_arguments: list[<type>] = ...")  [var-annotated]
+pwnlib/asm.py:0: error: Module "pwnlib.util.hashes" has no attribute "sha1sumhex"  [attr-defined]
+pwnlib/asm.py:0: error: Need type annotation for "util_versions"  [var-annotated]
+pwnlib/atexception.py:0: error: Function "_run_handlers" could always be true in boolean context  [truthy-function]
+pwnlib/commandline/cyclic.py:0: error: "Callable[[Any, Any, Any, Any], Any]" has no attribute "add_argument"  [attr-defined]
+pwnlib/commandline/cyclic.py:0: error: "Callable[[Any, Any, Any, Any], Any]" has no attribute "add_argument"  [attr-defined]
+pwnlib/commandline/cyclic.py:0: error: Incompatible types in assignment (expression has type "_MutuallyExclusiveGroup", variable has type "Callable[[Any, Any, Any, Any], Any]")  [assignment]
+pwnlib/commandline/template.py:0: error: Module has no attribute "data"  [attr-defined]
+pwnlib/context/__init__.py:0: error: Argument 1 to "int" has incompatible type "str | None"; expected "str | Buffer | SupportsInt | SupportsIndex | SupportsTrunc"  [arg-type]
+pwnlib/context/__init__.py:0: error: Need type annotation for "defaults"  [var-annotated]
+pwnlib/elf/__init__.py:0: error: Name "datatypes" is not defined  [name-defined]
+pwnlib/elf/datatypes.py:0: error: "type[constants]" has no attribute "DT_GNU_HASH"  [attr-defined]
+pwnlib/elf/datatypes.py:0: error: "type[constants]" has no attribute "STN_UNDEF"  [attr-defined]
+pwnlib/elf/datatypes.py:0: error: Unsupported dynamic base class "generate_siginfo"  [misc]
+pwnlib/elf/datatypes.py:0: error: Unsupported dynamic base class "generate_siginfo"  [misc]
+pwnlib/elf/elf.py:0: error: Name "address" already defined on line 0  [no-redef]
+pwnlib/elf/elf.py:0: error: Need type annotation for "functions" (hint: "functions: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/elf/elf.py:0: error: Need type annotation for "got" (hint: "got: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/elf/elf.py:0: error: Need type annotation for "plt" (hint: "plt: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/elf/elf.py:0: error: Need type annotation for "symbols" (hint: "symbols: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/encoders/arm/alphanumeric/__init__.py:0: error: Incompatible types in assignment (expression has type "str", base class "Encoder" defined the type as "None")  [assignment]
+pwnlib/encoders/arm/xor.py:0: error: Incompatible types in assignment (expression has type "str", base class "Encoder" defined the type as "None")  [assignment]
+pwnlib/encoders/encoder.py:0: error: Need type annotation for "_encoders"  [var-annotated]
+pwnlib/encoders/encoder.py:0: error: Need type annotation for "blacklist" (hint: "blacklist: set[<type>] = ...")  [var-annotated]
+pwnlib/encoders/i386/delta.py:0: error: Incompatible types in assignment (expression has type "str", base class "Encoder" defined the type as "None")  [assignment]
+pwnlib/encoders/i386/xor.py:0: error: Incompatible types in assignment (expression has type "str", base class "Encoder" defined the type as "None")  [assignment]
+pwnlib/encoders/mips/xor.py:0: error: Incompatible types in assignment (expression has type "str", base class "Encoder" defined the type as "None")  [assignment]
+pwnlib/filepointer.py:0: error: Need type annotation for "length" (hint: "length: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/filepointer.py:0: error: Need type annotation for "vars_" (hint: "vars_: list[<type>] = ...")  [var-annotated]
+pwnlib/filesystem/path.py:0: error: "classmethod" used with a non-method  [misc]
+pwnlib/filesystem/path.py:0: error: "classmethod" used with a non-method  [misc]
+pwnlib/filesystem/path.py:0: error: "type[Path]" has no attribute "mkdtemp"  [attr-defined]
+pwnlib/filesystem/path.py:0: error: "type[Path]" has no attribute "mktemp"  [attr-defined]
+pwnlib/gdb_api_bridge.py:0: error: Name "socket_path" is not defined  [name-defined]
+pwnlib/libcdb.py:0: error: Module "pwnlib.util.hashes" has no attribute "md5filehex"  [attr-defined]
+pwnlib/libcdb.py:0: error: Module "pwnlib.util.hashes" has no attribute "sha1filehex"  [attr-defined]
+pwnlib/libcdb.py:0: error: Module "pwnlib.util.hashes" has no attribute "sha256filehex"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_blue"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_blue"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_blue"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_green"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_red"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_red"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_yellow"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "bold_yellow"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "magenta"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "on_red"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "on_red"  [attr-defined]
+pwnlib/log.py:0: error: Module has no attribute "on_red"  [attr-defined]
+pwnlib/log.py:0: error: Need type annotation for "_one_time_infos" (hint: "_one_time_infos: set[<type>] = ...")  [var-annotated]
+pwnlib/log.py:0: error: Need type annotation for "_one_time_warnings" (hint: "_one_time_warnings: set[<type>] = ...")  [var-annotated]
+pwnlib/memleak.py:0: error: Module "pwnlib.util.packing" has no attribute "_p8lu"  [attr-defined]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "Callable[[AdbClient, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "Callable[[AdbClient, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_autoclose" has incompatible type "def __exit__(AdbClient, /, *Any, **Any) -> Any"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_sync" has incompatible type "Callable[[AdbClient, Any, Any, Any, Any, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_sync" has incompatible type "Callable[[AdbClient, Any, Any, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_sync" has incompatible type "Callable[[AdbClient, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_sync" has incompatible type "Callable[[AdbClient, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient, Any, Any, Any, Any, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient, Any], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/protocols/adb/__init__.py:0: error: Argument 1 to "_with_transport" has incompatible type "Callable[[AdbClient], Any]"; expected "AdbClient"  [arg-type]
+pwnlib/rop/call.py:0: error: Name "address" already defined on line 0  [no-redef]
+pwnlib/rop/call.py:0: error: Need type annotation for "args" (hint: "args: list[<type>] = ...")  [var-annotated]
+pwnlib/rop/call.py:0: error: Need type annotation for "values" (hint: "values: list[<type>] = ...")  [var-annotated]
+pwnlib/rop/gadgets.py:0: error: Need type annotation for "insns" (hint: "insns: list[<type>] = ...")  [var-annotated]
+pwnlib/rop/gadgets.py:0: error: Need type annotation for "regs" (hint: "regs: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/rop/rop.py:0: error: Need type annotation for "descriptions" (hint: "descriptions: dict[<type>, <type>] = ...")  [var-annotated]
+pwnlib/rop/srop.py:0: error: Need type annotation for "_regs" (hint: "_regs: list[<type>] = ...")  [var-annotated]
+pwnlib/shellcraft/__init__.py:0: error: Argument 1 to "append" of "list" has incompatible type "LazyImporter"; expected "MetaPathFinderProtocol"  [arg-type]
+pwnlib/shellcraft/__init__.py:0: error: Need type annotation for "_templates" (hint: "_templates: list[<type>] = ...")  [var-annotated]
+pwnlib/term/__init__.py:0: error: Module has no attribute "height"  [attr-defined]
+pwnlib/term/__init__.py:0: error: Module has no attribute "width"  [attr-defined]
+pwnlib/term/key.py:0: error: Need type annotation for "_kbuf" (hint: "_kbuf: list[<type>] = ...")  [var-annotated]
+pwnlib/term/readline.py:0: error: Module has no attribute "reverse"  [attr-defined]
+pwnlib/term/readline.py:0: error: Need type annotation for "history" (hint: "history: list[<type>] = ...")  [var-annotated]
+pwnlib/term/readline.py:0: error: Need type annotation for "search_results" (hint: "search_results: list[<type>] = ...")  [var-annotated]
+pwnlib/term/term.py:0: error: Need type annotation for "on_winch" (hint: "on_winch: list[<type>] = ...")  [var-annotated]
+pwnlib/tubes/process.py:0: error: Cannot assign to a type  [misc]
+pwnlib/tubes/process.py:0: error: Incompatible types in assignment (expression has type "PTY", variable has type "type[PTY]")  [assignment]
+pwnlib/tubes/process.py:0: error: Module "pwnlib.util.hashes" has no attribute "sha256file"  [attr-defined]
+pwnlib/tubes/tube.py:0: error: Argument 1 to "make_wrapper" has incompatible type "function"; expected "tube"  [arg-type]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "blue"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "blue"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "gray"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "gray"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "green"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "has_gray"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "has_gray"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "red"  [attr-defined]
+pwnlib/util/fiddling.py:0: error: Module has no attribute "red"  [attr-defined]
+pwnlib/util/iters.py:0: error: Name "pairwise" already defined (possibly by an import)  [no-redef]

pwnlib/abi.py

@@ -2,7 +2,7 @@
 from pwnlib.context import context
 
 
-class ABI(object):
+class ABI:
     """
     Encapsulates information about a calling convention.
     """