docs

Author: jmafoster1

docs/source/background.rst

@@ -83,11 +83,12 @@ Background
 
    <div style="margin-top: 30px;"></div>
 
-The Causal Testing Framework primarily consists of the following 2 components: 1) Causal Specification and 2) Causal Test Case.
+The Causal Testing Framework primarily consists of the following 3 components: 1) Modelling Scenario, 2) Causal Graph, and 2) Causal Test Case.
 
 #.
-   :doc:`Causal Specification <../modules/causal_specification>`\ : To apply graphical causal inference techniques for testing, we need a *causal specification*.
-   This takes the form of a directed acyclic graph (DAG) that depicts causal relationships amongst inputs and outputs and a set of constraints over input variables that capture the modelling scenario of interest.
+   :doc:`Causal Graph <../modules/causal_specification>`\ : To apply graphical causal inference techniques for testing, we need a *specification*.
+   For this, we must specify the variables that are relevant to the modelling scenario of interest, and any constraints over them.
+   We must also specify the expected causal relationships between the variables as a directed acyclic graph (DAG).
 
 
 #.

docs/source/modules/causal_specification.rst

@@ -2,52 +2,46 @@
 Causal Specification
 =====================
 
-- In causal testing, our units of interest are specific usage **scenarios** of the system-under-test. For example,
-  when testing an epidemiological computational model, one scenario could focus on the simulation of the spread of a virus in a crowded indoors space.
-  For this scenario, our causal specification will describe how a number of interventions should **cause** some outputs to change e.g. opening a window should reduce the spread of the virus by some factor.
-
-- In order to isolate the causal effect of the defined interventions, the user needs to express the anticipated cause-effect relationships amongst the inputs and outputs involved in the scenario.
-  This is achieved using a causal DAG, a simple dot and arrow graph that does not contain any cycles where nodes are random variables that represent the
-  inputs and outputs in the scenario-under-test, and edges represent causality. For example, ``window --> infection_prob`` encodes the belief that opening or closing the
-  window should cause the probability of infection to change.
-
-- A causal specification is simply the combination of these components: a series of requirements for the scenario-under-test and a causal DAG representing causality
-  amongst the inputs and outputs.
-
-- Collectively, the components of the causal specification provide both contextual information in the form of constraints and requirements, as well as causal information in the form of a causal DAG.
-  It's these components that are used to design statistical experiments that can answer causal questions about the scenario-under-test.
+As in traditional software testing, the specification defines the expected behaviour of the system.
+In causal testing, this is made up of two components: the modelling scenario and the causal graph.
+These components are then used to design statistical experiments that can answer causal questions about the system-under-test.
 
 1. Modelling Scenario
-----------------------
+---------------------
 
-- Each scenario is defined as a series of constraints placed over a set of input variables. A constraint is simply a mapping
-  from an input variable to a specific value or distribution that characterises the scenario in question.
-  For example, a scenario simulating the spread of a virus in a crowded indoors space would likely place a constraint over the size of room,
-  the number of windows, and the number of people in the room.
+- In causal testing, our units of interest are specific usage **scenarios** of the system-under-test.
+  For example, when testing an epidemiological computational model, one scenario could focus on the simulation of the spread of a virus through a population.
+  For this scenario, we may then test how a number of interventions should **cause** some outputs to change e.g. vaccinations should reduce the total number of deaths.
+
+- Each scenario is defined as a series of constraints placed over a set of input variables.
+  A constraint is simply a mapping from an input variable to a specific value or distribution that characterises the scenario in question.
+  For example, a scenario simulating the spread of a virus would likely place constraints on the location, population demographics, and who is vaccinated.
 
 - Requirements for this scenario should describe how a particular intervention
-  (e.g. opening the window, changing the number of people, changing the size of the room etc.) is expected to cause a particular outcome (number of infections, deaths, R0 etc.) to change.
+  (e.g.changing the number of people, changing who is vaccinated, etc.) is expected to cause a particular outcome (number of infections, deaths, R0, etc.) to change.
   The way these requirements are expressed is up to the user, however, it is essential that they focus on the expected effect of an intervention.
 
-2. Causal DAG
---------------
+2. Causal Graph
+---------------
 
+To isolate the causal effect of the defined interventions, the user needs to express the anticipated cause-effect relationships amongst the inputs and outputs involved in the scenario.
+This is done using a directed acyclic graph (DAG) in which nodes represent variables in the system and edges represent causal effects.
 In order to apply CI techniques, we need to capture causality amongst the inputs and outputs in the scenario-under-test.
 Therefore, for each scenario, the user must define a causal DAG.
-As an example, consider the DAG shown below for the `Poisson Line Process example. <https://github.com/CITCOM-project/CausalTestingFramework/tree/main/examples/poisson-line-process>`_
-Here, the model has three inputs: ``width``, ``height``, and ``intensity``.
-These inputs control the number of lines (``num_lines_abs``) and polygons (``num_shapes_abs``) that are drawn, which then feed into the numbers of lines (``num_lines_unit``) and polygons (``num_shapes_unit``) per unit area.
-Note though that the ``num_lines_abs`` does not have a direct causal effect on ``num_shapes_unit``, since the number of polygons per unit area is defined entirely by the total number of polygons and the area of the sampling window.
+
+As an example, consider the DAG shown below for the `vaccinating the elderly example. <https://github.com/CITCOM-project/CausalTestingFramework/tree/main/examples/covasim_/vaccinating_elderly>`_
+This modelling scenario has two inputs `vaccine` and `max_doses` and three outputs `cum_vaccinations`, `cum_vaccinated`, and `cum_infections`.
+We do not expect `max_doses` to have a causal effect on any of the outputs since this remains constant throughout modelling scenario.
 
 .. container:: zoom-container
 
-   .. image:: ../../../examples/poisson-line-process/dag.png
+   .. image:: ../../../examples/covasim_/vaccinating_elderly/dag.png
       :class: zoomable-image
-      :alt: Schematic diagram of the Poisson Line Process DAG
+      :alt: Causal DAG of the vaccinating the elderly modelling scenario
 
-.. literalinclude:: ../../../examples/poisson-line-process/dag.dot
+.. literalinclude:: ../../../examples/covasim_/vaccinating_elderly/dag.dot
    :language: graphviz
-   :caption: **Figure:** Example Causal DAG for the Poisson line process example.
+   :caption: **Figure:** Example Causal DAG for the vaccinating the elderly example.
 
 .. raw:: html
 

docs/source/modules/causal_testing.rst

@@ -4,7 +4,7 @@ Causal Testing
 A ``causal test`` or ``causal test case`` is the expected change in an outcome that applying an intervention to the input should cause.
 In this context, an intervention is simply a function which manipulates the input configuration of the scenario-under-test in a way that is expected to cause a change to some outcome.
 Programmatically, the data structure of causal tests can either be a ``.json`` file or hard-coded (e.g. our :doc:`tutorials <../tutorials>` contain examples of how to
-encode your causal tests). Moreover, by ``causal testing`` we refer to the overall process and execution of using the ``causal specification`` and ``causal test case(s)``, including statistical estimators,
+encode your causal tests). Moreover, by ``causal testing`` we refer to the overall process and execution of using the ``modelling scenario``, ``causal graph``, and ``causal test case(s)``, including statistical estimators,
 to determine whether each test case passes or fails relative to the test oracle.
 
 Getting Started
@@ -33,17 +33,21 @@ Example: Testing Virus Spread in a Classroom
 In the following sections, we describe the end-to-end process of ``causal testing`` for a hypothetical epidemiological computational model in which we are testing the model within a classroom scenario.
 In particular, suppose we're interested in how various precautions, such as hand-washing and mask-wearing, can prevent the spread of a virus within a classroom.
 
-1. Specification
+1. Modelling Scenario
 ----------------
 
-For our causal specification, suppose we define the scenario with the following constraints:
+For our modelling scenario, suppose we define the scenario with the following constraints:
 
 * ``n_people ~ Uniform(20, 30)`` (There are between 20 and 30 people in the classroom).
 * ``environment = Grid(x,y ~ Uniform(20, 40))`` (The classroom is square grid of between 20x20 and 40x40 units).
 * ``n_infected_t0 = 1`` (One person is infected initially).
 * ``precaution = None`` (No precautions taken).
   We also specify the output we are interested in as ``n_infected_t5``\ , the number of people infected after five days of daily one hour lessons.
 
+
+2. Causal Graph
+----------------
+
 Then, we create a simple causal directed acyclic graph (DAG), which represents causality amongst these variables:
 
 .. raw:: html
@@ -135,7 +139,7 @@ Then, we create a simple causal directed acyclic graph (DAG), which represents c
 
 
 
-2. Causal Test Cases
+3. Causal Test Cases
 --------------------
 
 We then define a number of causal test cases to apply to the scenario-under-test. For example, supposing we expect mask wearing and hand washing to have a preventative effect:
@@ -147,14 +151,14 @@ We then define a number of causal test cases to apply to the scenario-under-test
 
 - To run these test cases observationally, we need *valid* observational data for the scenario-under-test. This means we can only use executions with between 20 and 30 people, a square environment of size betwen 20x20 and 40x40, and where a single person was initially infected. In addition, this data must contain executions both with and without the intervention. Next, we need to identify any sources of bias in this data and determine a procedure to counteract them. This is achieved automatically using graphical causal inference techniques that identify a set of variables that can be adjusted to obtain a causal estimate. Finally, for any categorical biasing variables, we need to make sure we have executions corresponding to each category otherwise we have a positivity violation (i.e. missing data). In the worst case, this at least guides the user to an area of the system-under-test that should be executed.
 
-3. Causal Testing
+4. Causal Testing
 -----------------
 
 - After obtaining suitable test data, we can now apply causal inference. First, as described above, we use our causal graph to identify a set of adjustment variables that mitigate all bias in the data. Next, we use statistical models to adjust for these variables (implementing the statistical procedure necessary to isolate the causal effect) and obtain the desired causal estimate. Depending on the statistical model used, we can also generate 95% confidence intervals (or confidence intervals at any confidence level for that matter).
 
 - In our example, the causal DAG tells us it is necessary to adjust for ``environment`` in order to obtain the causal effect of ``precaution`` on ``n_infected_t5``. Supposing the relationship is linear, we can employ a linear regression model of the form ``n_infected_t5 ~ p0*precaution + p1*environment`` to carry out this adjustment. If we use experimental data, only a single environment is used by design and therefore the adjustment has no impact. However, if we use observational data, the environment may vary and therefore this adjustment will look at the causal effect within different environments and then provide a weighted average, which turns out to be the partial coefficient ``p0``.
 
-4. Test Oracle Procedure
+5. Test Oracle Procedure
 -------------------------
 
 - After conducting causal inference, all that remains is to ascertain whether the causal effect is expected or not. In our example, this is simply a case of checking whether the causal effect on ``n_infected_t5`` falls within the specified range. However, in the future, we may wish to implement more complex oracles.