Jmafoster1/marie kondo

[jmafoster1]

Closes #398 #394 #392

This PR gets rid of a load of relic infrastructure classes that no longer spark joy. The result is a much more streamlined testing workflow. Notable removals include:

• the MetamorphicRelation class, in favour of directly generating test cases from a causal DAG
• The Scenario, Variable, and BaseTestCase classes, in favour of expressing variables as strings, and modelling scenarios as pandas query strings.
• The CubicSplineEstimator class, in favour of specifying cubic splines as formulas with LinearRegressionEstimator.
• Treatment and outcome variables are now specified as part of the estimator only.

It also adds a new command "evaluate" to evaluate a DAG with respect to a given dataset (with confidence intervals for the number of passing test cases calculated by resampling, similar to data adequacy).

[github-actions]

🦙 MegaLinter status: ⚠️ WARNING

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
⚠️ PYTHON	black	30		1	1.23s
✅ PYTHON	pylint	30		0	6.24s

See detailed report in MegaLinter reports

MegaLinter is graciously provided by

[jmafoster1]

OK, so looking at the CausalTestCase class, it seems that the treatment_variable and outcome_variable properties aren't ever actually used (except in serialising to/from JSON), so we could get rid of them altogether and just store them in the associated Estimator. Programmatically this makes sense, but intuitively I'm not keen. The vibes are off - it puts too much responsibility on the "estimator". However, since the estimator also stores the adjustment set and (in the case of regression), the equational relationship, I think it might make sense to rename the Estimator family to something like CausalRelationship or something from the world of structural equational models. What are other people's thoughts?

For reference, instantiating a causal test case currently looks like this:

estimation_model = LinearRegressionEstimator(
            treatment_variable="A",
            outcome_variable="C",
            treatment_value=self.treatment_value,
            control_value=self.control_value,
            adjustment_set=self.minimal_adjustment_set,
        )
        causal_test_case = CausalTestCase(
            treatment_variable="A",
            outcome_variable="C",
            expected_causal_effect=self.expected_causal_effect,
            estimator=estimation_model,
            effect_measure="ate",
        )

In the new system, it'd look like this, which would be a lot cleaner:

causal_relationship = LinearRegressionRelationship(
            treatment_variable="A",
            outcome_variable="C",
            treatment_value=self.treatment_value,
            control_value=self.control_value,
            adjustment_set=self.minimal_adjustment_set,
        )
        causal_test_case = CausalTestCase(
            expected_causal_effect=self.expected_causal_effect,
            causal_relationship=causal_relationship,
            effect_measure="ate",
        )

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.54%. Comparing base (3e6fd99) to head (e753b1e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #402      +/-   ##
==========================================
+ Coverage   98.18%   98.54%   +0.36%     
==========================================
  Files          32       25       -7     
  Lines        1704     1514     -190     
==========================================
- Hits         1673     1492     -181     
+ Misses         31       22       -9     

Files with missing lines	Coverage Δ
causal_testing/__main__.py	99.15% <100.00%> (+3.41%)	⬆️
causal_testing/causal_testing_framework.py	100.00% <100.00%> (ø)
causal_testing/discovery/abstract_discovery.py	100.00% <100.00%> (ø)
causal_testing/discovery/hill_climber_discovery.py	100.00% <100.00%> (+1.78%)	⬆️
causal_testing/estimation/abstract_estimator.py	100.00% <100.00%> (ø)
...esting/estimation/abstract_regression_estimator.py	100.00% <100.00%> (+3.70%)	⬆️
...ausal_testing/estimation/experimental_estimator.py	100.00% <100.00%> (ø)
...ting/estimation/instrumental_variable_estimator.py	100.00% <100.00%> (ø)
causal_testing/estimation/ipcw_estimator.py	99.25% <ø> (-0.02%)	⬇️
..._testing/estimation/linear_regression_estimator.py	100.00% <100.00%> (ø)
... and 5 more

... and 3 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 355f04d...e753b1e. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jmafoster1]

Update: Turns out removing treatment_variable and outcome_variable from CausalTestCase was really easy as nothing depended on it! I've added in treatment_variable and outcome_variable property methods for backwards compatibility and easy access to the variables, but the specification of test cases is a lot cleaner now. I think I will rename the Estimator classes to CausalRelationship to create a better intuition of what they do. This will essentially allow us to create full structural causal models!

[jwhogg]

looks good, pending that set change