Importing dataFileToViz can break other Viz related functionality on newer SWIG

[sassy-asjp]

Describe the bug
When test_scenarioDataToViz is run alongside other tests that use Viz, the test fails, and causes other tests to fail. It is also not a test ordering dependency, but rather a test collection dependency: just being in the same test instance as other tests that use viz, causes those other tests to fail, even if those tests run first.

A typical error is usually in the form of

>       del viz.settings.instrumentGuiSettingsList[:]  # clear settings list to replace it with updated list
E       TypeError: 'SwigPyObject' object does not support item deletion

It seems like the offending line is line 75 in the scenario itself. Just importing dataFileToViz causes problems for other Viz usage. If the line is commented out, tests other than test_scenarioDataToViz will all pass.

from Basilisk.simulation import dataFileToViz

It seems like importing dataFileToViz messes with the SWIG proxy stuff? Adding a breakpoint and looking at when dataFileToViz was or was not imported, we can observe

# BAD case PDB output
-> del viz.settings.instrumentGuiSettingsList[:]  # clear settings list to replace it with updated list
(Pdb) viz.settings.instrumentGuiSettingsList
<Swig Object of type 'std::vector< InstrumentGuiSettings,std::allocator< InstrumentGuiSettings > > *' at 0x146f08540>
# continuing to try to interact with viz.settings.instrumentGuiSettingsList will show weird behavior, like not having the std vector functions like size()
(Pdb) viz.settings.instrumentGuiSettingsList.empty()
*** AttributeError: 'SwigPyObject' object has no attribute 'empty'
(Pdb) viz.settings.instrumentGuiSettingsList.size()
*** AttributeError: 'SwigPyObject' object has no attribute 'size'

# GOOD case PDB output
-> del viz.settings.instrumentGuiSettingsList[:]  # clear settings list to replace it with updated list
(Pdb)  viz.settings.instrumentGuiSettingsList
<Basilisk.simulation.vizInterface.InstrumentGuiSettingsConfig; proxy of <Swig Object of type 'std::vector< InstrumentGuiSettings > *' at 0x148c78b10> >
(Pdb) viz.settings.instrumentGuiSettingsList.empty()
True
(Pdb) viz.settings.instrumentGuiSettingsList.size()
0

To reproduce

pytest -vv src/tests # many tests fail
find src/tests -name \*.py | sort | grep -v DataToViz | xargs pytest -vv # all tests pass

Expected behavior
All tests to pass when run together

Screenshots
N/A

Desktop (please complete the following information):

Platform 1 (bug present):

• OS: macOS
• Version: 0c1bdcc
• Python version: 3.11.6
• SWIG version: 4.2.1

Platform 2 (bug present):

• OS: Linux
• Version: 0c1bdcc
• Python version: 3.10.12
• SWIG version: 4.2.1

Platform 3 (bug present):

• OS: Linux
• Version: 0c1bdcc
• Python version: 3.10.12
• SWIG version: 4.2.0

Platform 4 (behaves as expected):

• OS: Linux
• Version: 0c1bdcc
• Python version: 3.10.12
• SWIG version: 4.0.1

Platform 5 (behaves as expected):

• OS: macOS
• Version: 0c1bdcc
• Python version: 3.11.6
• SWIG version: 4.1.1

Additional context
The bug was tested for existence on all the listed platforms, however all other debugging/troubleshooting happened on macOS Platform 1.

No detailed analysis of what SWIG, Basilisk, or Python versions have the bug was done, but based on the testing done, I suspect it is specific to SWIG 4.2.0+.

This seems related to #579

[schaubh]

If your run pytest from the src folder, do these tests still fail? I encountered similar odd behavior just last week. All tests pass on my system with swig 4.2 if I run python run_all_test.py. Don't know what is causing this odd behavior.

[sassy-asjp]

If a run pytest from the src folder first (after a fresh git clean -fdx then setting everything up again) it works, and continues working even when I run from the repository root. If I run first from the repository root, it still won't work even if I run from src folder. The bug is somehow frozen into the python bytecode cache probably.

I don't think it's really fixable unless someone can figure out how to reproduce it with a very simple example, then have SWIG developers look at it.

Probably won't look into this issue much further. For future reference here are some commands that can be run from fresh ubuntu:22.04 docker image to show the issue.

apt update
apt install -y git git-lfs cmake build-essential python3 python3-dev python3.10-venv libpcre2-dev wget
git clone https://github.com/AVSLab/basilisk.git
cd basilisk/
python3 -m venv .venv
. .venv/bin/activate
pip install conan==1.60 pytest
wget http://prdownloads.sourceforge.net/swig/swig-4.2.1.tar.gz
tar -xzvf swig-4.2.1.tar.gz 
cd swig-4.2.1
./configure 
make
make install
cd ..
python3 conanfile.py
pytest -vv src/tests # tons of tests fail
cd src
pytest -vv tests # tests keep failing

apt update
apt install -y git git-lfs cmake build-essential python3 python3-dev python3.10-venv libpcre2-dev wget
git clone https://github.com/AVSLab/basilisk.git
cd basilisk/
python3 -m venv .venv
. .venv/bin/activate
pip install conan==1.60 pytest
wget http://prdownloads.sourceforge.net/swig/swig-4.2.1.tar.gz
tar -xzvf swig-4.2.1.tar.gz 
cd swig-4.2.1
./configure 
make
make install
cd ..
python3 conanfile.py
cd src
pytest -vv tests # tests pass
cd ..
pytest -vv src/tests # tests continue to pass

[sassy-asjp]

I took a fresh look at this and I think I've solved it

[sassy-asjp]

Ah, actually this fix I came up with doesn't actually work for everything.

At least I think the root cause is clear. I'll think a bit more about the correct fix.

[sassy-asjp]

Ok, I pushed a new change that fixes it without breaking anything else.