Tests always pass no matter what.

[ntoll]

Hi,

Thank you for pytest-pyodide.

Context: I'm trying to run some "normal" tests inside Pyodide with PyTest and Chrome.

Problem: no matter what I put into the test functions (e.g. assert False), they always pass.

Example: I've created a repository with a bare minimum reproduction of the problem: https://github.com/ntoll/borked Just follow the instructions in the README and make test.

When I run the tests without --run-in-pyodide they fail as expected. Help?!?! cc/ @hoodmane

[ntoll]

I've spent some more time digging into this problem and focussed on this function.

As far as I can tell there are several problems:

• My code to be tested (not the actual test code itself), is not on Pyodide's filesystem in the browser. I was expecting to see (using my example reproduction of the problem), my_module.py somewhere in the filesystem so it can be imported in the test module.
• The exit code returned from pytest.main is 4 - i.e. pytest command line usage error
• I think the exit code is because node_tree_id simply doesn't appear valid given the context in which pytest is run. There appears to be (as mentioned in this function's comments) a test_files base folder which means that the specified test is never found and....
• The output XML looks like this:

<?xml version="1.0" encoding="utf-8"?>
  <testsuites>
    <testsuite name="pytest" errors="0" failures="0" skipped="0" tests="0" time="0.021" timestamp="2023-03-09T13:02:59.376000" hostname="emscripten" />
</testsuites>

• If I hard code a replacement for node_tree_id to the absolute path to the test directory (i.e. testfiles/tests) then I get this XML result, which includes the ModuleNotFoundError because the code-to-be-tested hasn't been copied over:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
	<testsuite name="pytest" errors="1" failures="0" skipped="0" tests="1" time="0.197" timestamp="2023-03-09T14:08:39.361000" hostname="emscripten">
		<testcase classname="" name="test_files.tests.test_my_module" time="0.000">
			<error message="collection failure">ImportError while importing test module \'/home/pyodide/test_files/tests/test_my_module.py\'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/lib/python3.10/importlib/__init__.py:126: in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\ntest_files/tests/test_my_module.py:1: in &lt;module&gt;\n    import my_module\nE   ModuleNotFoundError: No module named \'my_module\'</error>
		</testcase>
	</testsuite>
</testsuites>

Not sure what to suggest - but something is definitely amiss: only the test module is copied over, and it's not referenced properly when calling pytest.main is the TL;DR.

[hoodmane]

I guess the solution for now is to make a wheel.

[ntoll]

🤣 yeah ☸️ 👍

All part of the fun.

[ntoll]

Make a wheel (and then document / script the process)

[hoodmane]

It would be great to fix pytest-pyodide so that it is not required to build a wheel for pure Python libs though.