Overview
For a given causal test case, we cannot confirm whether the causal estimate is correct or not. This is because the correctness of the causal estimate depends on a number of identifiability and modelling assumptions. The best we can do is to make clear the assumptions on which the estimate is founded. For this reason, we will implement a causal inference contract.
A causal inference contract should be attached to every test result and it should state all of the assumptions on which a causal inference is made. That way, another modeller or tester has the ability to accept or dispute test results on the the basis of their identifiability or modelling assumptions. Therefore, if the Causal DAG is poorly specified, the modeller/tester can reject the results, modify the DAG and repeat the analysis to observe the difference.
Identifiability assumptions
The identifiability assumptions are assumptions that must hold for the identified estimand to be considered causal. The main identifiability assumptions we care about are:
- Consistency: (Informally) Does the observed outcome for every treated/untreated individual correspond to the actual outcome? This requires a precise definition of the treatment and corresponding outcome and for this to be reflected in the data.
- Positivity: (Informally) Do we have sufficient data to infer the causal effect of a treatment X on an outcome Y?
- Exchangeabiliy: (Informally) Does our Causal DAG contain all confounding variables and can we block their influence?
If all of these assumptions hold, we know that our identified adjustment set is capable of isolating the causal effect of X on Y. However, we can only check if the positivity assumptions holds; we cannot check consistency and exchangeability (to my knowledge).
To check positivity, we can confirm that there is data within each stratum of the estimand. If not, the user can only continue if they wish to use a parametric estimator to extrapolate this missing values. However, this parametric estimator must be correctly specified for this to be considered valid. Alternatively, we can point the user towards the positivity violation, providing guidance as to the minimal set of executions which will enable causal inference. In some cases, the positivity violation cannot be fixed because it is not possible to observe a particular individual (known as a structural positivity violation). This requires the user to restrict the causal question to individuals outside of the positivity violation.
We cannot check consistency and exchangeability because these depend on domain expertise that cannot be learned from the data. For example, we cannot tell from the data whether all confounders have been considered and we cannot tell whether the data is consistent with the question. The best we can do here is to make the assumptions clear.
For exchangeability, the user can share the Causal DAG which contains all variables included in the analysis. If another modeller disagrees, they can refer to the assumptions in the Causal DAG to explain why. The Causal DAG can then be easily ammended and the tests repeated.
For consistency, the user should confirm that their treament and outcomes are well-defined (unambiguous and correspond to the values in the data). It might be worth adding a small check-list with items such as: "Are there different versions of the treatment X?" and "If so, do you expect these versions to yield the same outcome for all instances of the scenario?". An example of a treatment which has multiple versions is a vaccine. For the purpose of a high-level test, it might be suitable to conflate different vaccine versions (e.g. does the vaccine decrease deaths? -- all vaccines should so we don't care what type it is). But for more fine-grained tests, it would probably be necessary to distinguish between them (e.g. is Pfizer more effective at reducing deaths than Moderna?).
For each of the identifiability assumptions, we should append the necessary information to the contract (e.g. causal DAG, positivity violations, and consistency statement).
Modelling Assumptions
After identification, we need to estimate the value of an estimand using an appropriate statistical model. If the user opts for a parametric model, we have to additionally consider the restrictions this model places on the data and whether these are actually true. These are referred to as modelling assumptions. For example, we can only use a linear regression model of the form Y ~ mX + C if the X/Y relationship fits onto a straight line. Otherwise, we will extrapolate values from a misspecified model, yielding a biased estimate.
For each estimator that we support, we should add a list of modelling assumptions that must hold if the result is to be considered causal.
Overview
For a given causal test case, we cannot confirm whether the causal estimate is correct or not. This is because the correctness of the causal estimate depends on a number of identifiability and modelling assumptions. The best we can do is to make clear the assumptions on which the estimate is founded. For this reason, we will implement a causal inference contract.
A causal inference contract should be attached to every test result and it should state all of the assumptions on which a causal inference is made. That way, another modeller or tester has the ability to accept or dispute test results on the the basis of their identifiability or modelling assumptions. Therefore, if the Causal DAG is poorly specified, the modeller/tester can reject the results, modify the DAG and repeat the analysis to observe the difference.
Identifiability assumptions
The identifiability assumptions are assumptions that must hold for the identified estimand to be considered causal. The main identifiability assumptions we care about are:
If all of these assumptions hold, we know that our identified adjustment set is capable of isolating the causal effect of X on Y. However, we can only check if the positivity assumptions holds; we cannot check consistency and exchangeability (to my knowledge).
To check positivity, we can confirm that there is data within each stratum of the estimand. If not, the user can only continue if they wish to use a parametric estimator to extrapolate this missing values. However, this parametric estimator must be correctly specified for this to be considered valid. Alternatively, we can point the user towards the positivity violation, providing guidance as to the minimal set of executions which will enable causal inference. In some cases, the positivity violation cannot be fixed because it is not possible to observe a particular individual (known as a structural positivity violation). This requires the user to restrict the causal question to individuals outside of the positivity violation.
We cannot check consistency and exchangeability because these depend on domain expertise that cannot be learned from the data. For example, we cannot tell from the data whether all confounders have been considered and we cannot tell whether the data is consistent with the question. The best we can do here is to make the assumptions clear.
For exchangeability, the user can share the Causal DAG which contains all variables included in the analysis. If another modeller disagrees, they can refer to the assumptions in the Causal DAG to explain why. The Causal DAG can then be easily ammended and the tests repeated.
For consistency, the user should confirm that their treament and outcomes are well-defined (unambiguous and correspond to the values in the data). It might be worth adding a small check-list with items such as: "Are there different versions of the treatment X?" and "If so, do you expect these versions to yield the same outcome for all instances of the scenario?". An example of a treatment which has multiple versions is a vaccine. For the purpose of a high-level test, it might be suitable to conflate different vaccine versions (e.g. does the vaccine decrease deaths? -- all vaccines should so we don't care what type it is). But for more fine-grained tests, it would probably be necessary to distinguish between them (e.g. is Pfizer more effective at reducing deaths than Moderna?).
For each of the identifiability assumptions, we should append the necessary information to the contract (e.g. causal DAG, positivity violations, and consistency statement).
Modelling Assumptions
After identification, we need to estimate the value of an estimand using an appropriate statistical model. If the user opts for a parametric model, we have to additionally consider the restrictions this model places on the data and whether these are actually true. These are referred to as modelling assumptions. For example, we can only use a linear regression model of the form
Y ~ mX + Cif the X/Y relationship fits onto a straight line. Otherwise, we will extrapolate values from a misspecified model, yielding a biased estimate.For each estimator that we support, we should add a list of modelling assumptions that must hold if the result is to be considered causal.