pytest_generate_tests() created fixture, together with pytest.mark.parametrize() created fixture (of different name) -- results in messed up fixture scope - and inappropriate re-instantiation of fixtures

[nebbish]

I've looked at the older but similar issue #4017 and the newer (very similar but not the same) #6962.
This problem cannot be related to #6962 because I'm in the 4.6.x branch of Pytest. It could be related to #4017, maybe caused by the fix put in at that time. I'm not sure. I have not looked backward to see when this might have worked.

I'm working on Windows (7 actually) and I'm using Pytest to test C++ built binaries - because Pytest is so clear and expressive at the site of the test case. I build my c++ component in both 32 and 64-bit versions and both MSVCRT versions (eckk I know).

Anyhow I have a test suite that is appropriate to run against every version of the built binary - so I started trying to use pytest_generate_tests() to create a session-scoped fixture that is parameterized based on cmd-line arguments. The command line can specify either/or or both of either choice. When the cmd-line says to test both - the session-scoped fixture will have multiple parameters, so I'm expecting the whole test suite should run multiple times, and the session fixture should create & finalize only once for each parameter of this new fixture..

Here's the smallest example I could build that easily demonstrates both behaviors.

#!/usr/bin/env python2
# conftest.py
from __future__ import print_function
import pytest

def pytest_addoption(parser):
    parser.addoption('--arch', choices=['all','32','64'], default='64')
    parser.addoption('--crt', choices=['all','static','dynamic'], default='static')

##################################################################
# Use this to control WHICH session-scoped parameterized fixture to use
if True:
    # Dynamically created session-scoped parameterized fixture
    def pytest_generate_tests(metafunc):
        if 'platstr' in metafunc.fixturenames:
            (arch, crt) = [metafunc.config.getoption(o) for o in ['--arch','--crt']]
            archs = [v for v in ['32','64'] if arch in ['all'] + [v]]
            crts = [v for v in ['static','dynamic'] if crt in ['all'] + [v]]
            plats = ['win{}-{}'.format(a,c) for a in archs for c in crts]
            metafunc.parametrize('platstr', plats, scope='session')
else:
    # Hard coded session-scoped parameterized fixture
    @pytest.fixture(scope='session', params=['win64-static'])
    def platstr(request):
        return request.param
##################################################################

@pytest.fixture(scope='session', autouse=True)
def env(request, platstr):
    print('\nCreating:   TestEnv({})'.format(platstr), end='')
    def finalize():
        print('\n\tFinalizing TestEnv', end='')
    request.addfinalizer(finalize)
    return platstr

Any test case with parameterization shows the trouble:

#!/usr/bin/env python2
# demo.py
import pytest
@pytest.mark.parametrize('raw_data', ['asdf','qwer'])
def test_param_case(raw_data):
    assert raw_data

Here's the output of pytest demo.py -s as pasted above:

(venv) D:\pytest-testing>pytest demo.py -s
============================= test session starts =============================
platform win32 -- Python 2.7.14, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
rootdir: D:\pytest-testing
collected 2 items

demo.py
Creating:   TestEnv(win64-static).
        Finalizing TestEnv
Creating:   TestEnv(win64-static).
        Finalizing TestEnv

========================== 2 passed in 0.06 seconds ===========================

(venv) D:\pytest-testing>

In that output - the session fixture is torn down when it should not have been.

Here's the output when the if True: above is changed to if False: - and the behavior is as expected:

(venv) D:\pytest-testing>pytest demo.py -s
============================= test session starts =============================
platform win32 -- Python 2.7.14, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
rootdir: D:\pytest-testing
collected 2 items

demo.py
Creating:   TestEnv(win64-static)..
        Finalizing TestEnv

========================== 2 passed in 0.05 seconds ===========================

(venv) D:\pytest-testing>pytest demo.py -s

Pip list:

Package Version

---

atomicwrites 1.3.0
attrs 19.3.0
colorama 0.4.3
configparser 4.0.2
contextlib2 0.6.0.post1
funcsigs 1.0.2
importlib-metadata 1.5.1
more-itertools 5.0.0
packaging 20.3
pathlib2 2.3.5
pip 20.0.2
pluggy 0.13.1
py 1.8.1
pyparsing 2.4.6
pytest 4.6.9
scandir 1.10.0
setuptools 44.1.0
six 1.14.0
wcwidth 0.1.9
wheel 0.34.2
zipp 1.2.0

• [x ] a detailed description of the bug or suggestion
• [x ] output of pip list from the virtual environment you are using
• [x ] pytest and operating system versions
• [x ] minimal example if possible

[SalmonMode]

@nebbish could be related to an issue I found a while back with things not being torn down properly. There's a PR for the (potential) fix here. Would you mind checking out that branch and giving it a shot?