Skip to content

Commit beacfd8

Browse files
sinonsinon
committed
First pass of draft post on typing conformance
1 parent 45bc1b3 commit beacfd8

2 files changed

Lines changed: 116 additions & 0 deletions

File tree

‎content/future-python-type-checkers.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
+++
2+
title = "Ty versus Pyrefly versus Zuban: Evaluating the future of Python type checkers"
3+
date = 2025-08-29
4+
description = "A comprehensive comparison of three new Rust-based Python type checkers: Astral's ty, Meta's pyrefly, and David Halter's zuban"
5+
draft = true
6+
7+
[taxonomies]
8+
tags = ["rust", "python", "type-checkers", "typing"]
9+
+++
10+
11+
# Table of Contents
12+
13+
- [Introduction](#introduction)
14+
- [`ty` from Astral](#ty-from-astral)
15+
- [`pyrefly` from Meta](#pyrefly-from-meta)
16+
- [`zuban` from David Halter](#zuban-from-david-halter)
17+
- [Typing Conformance Suite Analysis](#typing-conformance-suite-analysis)
18+
- [Summary](#summary)
19+
- [Review of progress](#review-of-progress)
20+
- [Other resources to learn more](#other-resources-to-learn-more)
21+
- [Footnotes](#footnotes)
22+
23+
# Introduction
24+
25+
The Python type checking landscape is experiencing a particularly active phase of innovation in 2025. This year has witnessed the emergence of not one, not two, but **three** new Python type checking tools, each backed by significant pedigree and resources. While these tools have slightly different goals and philosophies, they share a common foundation: all are built in Rust with performance as a core design principle.
26+
27+
This continued shift toward Rust-based implementations represents a significant evolution in the Python tooling ecosystem, promising faster type checking, better IDE integration, and improved developer experience for large codebases.
28+
29+
30+
## `ty` from Astral
31+
32+
**Repository:** <https://github.com/astral-sh/ty/>
33+
**Development Repository:** <https://github.com/astral-sh/ruff/>
34+
35+
**Key Highlights:**
36+
- Strong focus on gradual guarantee principles
37+
- Planned tight integration with Astral's existing linting tool `ruff`, with the aim to support type based linting rules.[^1]
38+
- Backing from a team combining Python core developers and very experienced Rust tooling developers
39+
- Incremental computation at the function level leveraging the [salsa](https://github.com/salsa-rs/salsa) library used by [rust-analyzer](https://github.com/rust-lang/rust-analyzer/). This could be particularly important on the LSP that `ty` provides to give very quick/responsible feedback loops in code editors.
40+
41+
**Philosophy:** Astral's approach emphasizes reliability and gradual adoption, making it easier for teams to incrementally add type checking to existing codebases without overwhelming developers with false positives.
42+
43+
## `pyrefly` from Meta
44+
45+
**Repository:** <https://github.com/facebook/pyrefly>
46+
47+
**Key Highlights:**
48+
- Successor to Meta's existing [pyre](https://github.com/facebook/pyre-check) type checker, designed to eventually replace it
49+
- Enhanced type inference capabilities by default
50+
- Potentially higher upfront adoption cost due to aggressive inference which might flag issues with correct code
51+
- Backed by Meta's substantial engineering resources and real-world usage at scale
52+
53+
**Philosophy:** Meta prioritizes powerful inference and catching more potential issues out of the box, even if this means a steeper learning curve for teams new to type checking.
54+
55+
## `zuban` from David Halter
56+
57+
**Homepage:** <https://zubanls.com/>
58+
**Documentation:** <https://docs.zubanls.com/en/latest/>
59+
60+
**Key Highlights:**
61+
- Created by the author of the popular Python LSP tool `jedi`
62+
- Aims for high-degree of compatibility with `mypy` to make adoption in large existing codebases seamless.
63+
- Not FOSS[^2], will require a license for codebases above 1.5 MB (~50,000 lines of code)[^3].
64+
- Currently maintained by a single author seems a potential risk to long-term sustainability as Python typing does not stand still.
65+
66+
**Philosophy:** Zuban aims to provide the smoothest possible migration path from existing type checkers, particularly `mypy`, making it attractive for organizations with substantial existing typed codebases.
67+
68+
69+
# Typing Conformance Suite Analysis
70+
71+
The Python Typing Council maintains a [Conformance test suite](https://github.com/python/typing/tree/main/conformance) which validates the behavior of static type checkers against the expectations defined in the [Python typing specification](https://typing.python.org/en/latest/spec/index.html).
72+
73+
`ty` and `pyrefly` have not yet been added to the conformance suite, so it's harder to establish a baseline for their progress on this front.
74+
75+
To help resolve this gap I have [expanded the current test harness to support both of them](https://github.com/sinon/typing/pull/1)[^4], this also adds some additional debugging information to the html report to show the split between false negatives and false positives that the suite has detected in each testcase.
76+
77+
I've also included a local build of `ty` for two main reasons:
78+
79+
* I've wanted to investigate contributing to `ty` and/or `ruff` for a while, so this was good impetus to get things set up.
80+
* `ty` releases are cut relatively infrequently and I am impatient.
81+
82+
## Summary
83+
84+
| Type Checker | Test Case Passes | Test Case Fails | Test Case Unknown status | False Positives | False Negatives |
85+
|-------------------------------------------------|------------------|-----------------|--------------------------|-----------------|-----------------|
86+
| zuban 0.0.20 | 96 | 38 | 5 | 149 | 92 |
87+
| ty 0.0.1-alpha.19 (e9cb838b3 2025-08-19) | 20 | 0 | 119 | 371 | 603 |
88+
| Local:ty ruff/0.12.11+27 (0bf5d2a20 2025-08-29) | 20 | 0 | 119 | 370 | 590 |
89+
| pyrefly 0.30.0 | 81 | 0 | 58 | 100 | 187 |
90+
91+
## Review of progress
92+
93+
`zuban` has a lead, having full `Pass` on ~69% of testcase, compared with ~15% for `ty` and ~58% for `pyrefly`. Which makes sense as though it's released in a similar time period to `ty` and `pyrefly` it has been in active development in private for several years.
94+
95+
The thing that surprised me more was how much progress `pyrefly` has made when compared to `ty`. This can maybe be partially explained from a point raised in [Edward Li's blog excellent post on the Typing Summit at PyCon 2025][1] which mentions that the `pyrefly` team devoted a lot of up front time into solving some of the hard problems, such as generics.
96+
97+
98+
# Other resources to learn more
99+
100+
* [Edward Li's excellent blog post comparing `ty` and `pyrefly`][1] which also contains the videos recorded at the PyCon typing summit which both `ty` and `pyrefly` gave presentations at.
101+
* [Happy Path Programming - 114 ty: Fast Python Type Checking with Carl Meyer](https://www.youtube.com/watch?v=V1OmqEYoSz4)
102+
* [Happy Path Programming - 115 More Python Type Checking! Pyrefly with Aaron Pollack & Steven Troxler](https://www.youtube.com/watch?v=huHF0Rv8L14)
103+
* [Talk Python - ty: Astral's New Type Checker (Formerly Red-Knot)](https://talkpython.fm/episodes/show/506/ty-astrals-new-type-checker-formerly-red-knot)
104+
105+
106+
# Footnotes
107+
108+
[^1]: Which can be demonstrated in the [open issues](https://github.com/astral-sh/ruff/issues?q=is%3Aissue%20state%3Aopen%20label%3Atype-inference ) on ruff tagged with `type-inference` which are bugs or new features that can only be resolved with `ruff` having access to deeper type inference data that `ty` can supply.
109+
[^2]: David has indicated a plan to make [source available in the future](https://github.com/python/typing/pull/2067#issuecomment-3177937964) when adding Zuban to the Python typing conformance suite.
110+
[^3]: Full pricing information at: <https://zubanls.com/pricing/>
111+
[^4]: This is just for this blog post, no plans to seek merging this.
112+
113+
114+
<!-- Reference links --->
115+
[1]: https://blog.edward-li.com/tech/comparing-pyrefly-vs-ty/

‎content/pages/_index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
+++
2+
render = false
23
+++

0 commit comments

Comments
 (0)