Currently for each fixture which depends on another fixture, a "finalizer" is added to the list of the dependent fixture.
For example:
tmpdir, as we know, depends on tmpdir_path to create the temporary directory. Each tmpdir invocation ends up adding its finalization to the list of finalizers of tmpdir_path. This is the mechanism used to finalize fixtures in the correct order (as thought we still have bugs in this area, as #1895 shows for example), and ensures that every tmpdir will be destroyed before the requested tmpdir_path fixture.
This then means that every high-scoped fixture might contain dozens, hundreds or thousands of "finalizers" attached to them. Fixture finalizers can be called multiple times without problems, but this consumes memory: each finalizer keeps its SubRequest object alive, containing a number of small variables:
|
class FixtureRequest(FuncargnamesCompatAttr): |
|
""" A request for a fixture from a test or fixture function. |
|
|
|
A request object gives access to the requesting test context |
|
and has an optional ``param`` attribute in case |
|
the fixture is parametrized indirectly. |
|
""" |
|
|
|
def __init__(self, pyfuncitem): |
|
self._pyfuncitem = pyfuncitem |
|
#: fixture for which this request is being performed |
|
self.fixturename = None |
|
#: Scope string, one of "function", "class", "module", "session" |
|
self.scope = "function" |
|
self._fixture_defs = {} # argname -> FixtureDef |
|
fixtureinfo = pyfuncitem._fixtureinfo |
|
self._arg2fixturedefs = fixtureinfo.name2fixturedefs.copy() |
|
self._arg2index = {} |
|
self._fixturemanager = pyfuncitem.session._fixturemanager |
This can easily be demonstrated by applying this patch:
diff --git a/src/_pytest/runner.py b/src/_pytest/runner.py
index 55dcd805..b3a94bc6 100644
--- a/src/_pytest/runner.py
+++ b/src/_pytest/runner.py
@@ -393,6 +393,8 @@ class SetupState(object):
for col in self.stack:
if hasattr(col, "_prepare_exc"):
six.reraise(*col._prepare_exc)
+ if self.stack:
+ print(len(self._finalizers.get(self.stack[0])))
for col in needed_collectors[len(self.stack) :]:
self.stack.append(col)
try:(this prints the finalizers attached to the "Session" node, where the session fixtures attach their finalization to)
And running this test:
import pytest
@pytest.mark.parametrize('i', range(10))
def test(i, tmpdir):
passλ pytest .tmp\test-foo.py -qs
.1
.2
.3
.4
.5
.6
.7
.8
.9
.
10 passed in 0.16 seconds
I believe we can think of ways to refactor the fixture teardown mechanism to avoid this accumulation of finalizers on the objects. Ideally we should build a proper DAG of fixture dependencies which should be destroyed in the proper order. This would also make things more explicit and easier to follow IMHO.
Currently for each fixture which depends on another fixture, a "finalizer" is added to the list of the dependent fixture.
For example:
tmpdir, as we know, depends ontmpdir_pathto create the temporary directory. Eachtmpdirinvocation ends up adding its finalization to the list of finalizers oftmpdir_path. This is the mechanism used to finalize fixtures in the correct order (as thought we still have bugs in this area, as #1895 shows for example), and ensures that everytmpdirwill be destroyed before the requestedtmpdir_pathfixture.This then means that every high-scoped fixture might contain dozens, hundreds or thousands of "finalizers" attached to them. Fixture finalizers can be called multiple times without problems, but this consumes memory: each finalizer keeps its
SubRequestobject alive, containing a number of small variables:pytest/src/_pytest/fixtures.py
Lines 341 to 359 in ed68fcf
This can easily be demonstrated by applying this patch:
(this prints the finalizers attached to the "Session" node, where the session fixtures attach their finalization to)
And running this test:
I believe we can think of ways to refactor the fixture teardown mechanism to avoid this accumulation of finalizers on the objects. Ideally we should build a proper DAG of fixture dependencies which should be destroyed in the proper order. This would also make things more explicit and easier to follow IMHO.