|
1 | 1 | { |
2 | 2 | "cells": [ |
3 | | - { |
4 | | - "cell_type": "markdown", |
5 | | - "id": "56965fba-b90b-4233-a819-bb747ecd9d81", |
6 | | - "metadata": {}, |
7 | | - "source": [ |
8 | | - "# Statistical Metamorphic Testing using the API" |
9 | | - ] |
10 | | - }, |
11 | 3 | { |
12 | 4 | "cell_type": "markdown", |
13 | 5 | "id": "5adf7cdc-fd96-47a4-a194-f1f060a4c0c5", |
|
34 | 26 | "Before diving into the details, a good first step is to define your file paths, including your input configurations:" |
35 | 27 | ] |
36 | 28 | }, |
| 29 | + { |
| 30 | + "cell_type": "markdown", |
| 31 | + "id": "56965fba-b90b-4233-a819-bb747ecd9d81", |
| 32 | + "metadata": {}, |
| 33 | + "source": [ |
| 34 | + "# Statistical Metamorphic Testing using the API" |
| 35 | + ] |
| 36 | + }, |
37 | 37 | { |
38 | 38 | "cell_type": "code", |
39 | 39 | "execution_count": 1, |
|
61 | 61 | "id": "4928e126-dc5e-4cb1-9136-23c3f8292924", |
62 | 62 | "metadata": {}, |
63 | 63 | "source": [ |
64 | | - "Under the hood, the CTF consists of 2 main components, namely, a Causal Specification and a Causal Test Case. Let's break down the Causal Specification first.\n", |
65 | | - "\n", |
66 | | - "A Causal Specification consists of two sub-components called a Modelling Scenario and a Causal Directed Acyclic Graph (DAG). \n", |
67 | | - "\n", |
68 | | - "Firstly, the modelling scenario specifies the (or part of the) system under test by defining the observable variables and any constraints that exist between those variables. The CTF currently supports three types of variables:\n", |
| 64 | + "Under the hood, the CTF consists of three main components, namely a Modelling Scenario, a Causal Directed Acyclic Graph (DAG), and a Causal Test Case.\n", |
| 65 | + "The modelling scenario specifies the (or part of the) system under test by defining the observable variables and any constraints that exist between those variables. The CTF currently supports three types of variables:\n", |
69 | 66 | "\n", |
70 | 67 | "- `Input` variables, which are inputs to the system.\n", |
71 | 68 | "\n", |
72 | 69 | "- `Output` variables, which are outputs from the system.\n", |
73 | 70 | "\n", |
74 | 71 | "- `Meta` variables, which are not directly observable but can be related to the system in terms of other `Input` or `Output` variables.\n", |
75 | 72 | "\n", |
76 | | - "Secondly, the causal DAG encodes information about the expected causal structure of the system through nodes representing variables and directed edges representing causal relationships, which is a model of how the data could have been generated. Together, the Causal DAG and modelling scenario form the `Causal Specification`.\n", |
| 73 | + "The causal DAG encodes information about the expected causal structure of the system through nodes representing variables and directed edges representing causal relationships, which is a model of how the data could have been generated.\n", |
| 74 | + "Together, the Causal DAG and modelling scenario form the causal specification.\n", |
77 | 75 | "\n", |
78 | 76 | "**Note**: The CTF currently doesn't support native visualisation tools, but it is possible to use existing frameworks such as NetworkX to visualise your DAG. Alternatively, browser-based environments such as [DAGitty](https://www.dagitty.net/) may also be useful." |
79 | 77 | ] |
|
83 | 81 | "id": "15354565-eeb5-4722-bf6b-0b987eabb8c2", |
84 | 82 | "metadata": {}, |
85 | 83 | "source": [ |
86 | | - "## Step 2: Create a Casual Specification" |
| 84 | + "## Step 2: Read in the Data" |
87 | 85 | ] |
88 | 86 | }, |
89 | 87 | { |
|
297 | 295 | "In this case, the PLT model has three positive floating-point input parameters: thee width and height of the sampling window, and the intensity of the Poisson process. The model then outputs the total number of lines intersecting the sampling window, and the number of polygons formed by the intersecting lines. Note: in this dataset, the output variables appended by the suffix `_unit` are normalised with respect to their respective areas (`width*height`)." |
298 | 296 | ] |
299 | 297 | }, |
| 298 | + { |
| 299 | + "cell_type": "markdown", |
| 300 | + "id": "b6526dd1-625b-48bb-9782-f80080b1b917", |
| 301 | + "metadata": {}, |
| 302 | + "source": [ |
| 303 | + "## Step 3: Create a Modelling Scenario" |
| 304 | + ] |
| 305 | + }, |
300 | 306 | { |
301 | 307 | "cell_type": "code", |
302 | 308 | "execution_count": 3, |
303 | 309 | "id": "ac297d2d-5a2f-4c33-bbdc-967d54e24e3f", |
304 | 310 | "metadata": {}, |
305 | 311 | "outputs": [], |
306 | 312 | "source": [ |
307 | | - "# Step 2: Create a Causal Specification using the Modelling Scenario and Causal DAG\n", |
308 | | - "\n", |
309 | 313 | "from causal_testing.specification.variable import Input, Output\n", |
310 | 314 | "from causal_testing.specification.causal_dag import CausalDAG\n", |
311 | 315 | "from causal_testing.specification.scenario import Scenario\n", |
|
347 | 351 | "id": "877d413d-ff96-4481-953f-891c19493531", |
348 | 352 | "metadata": {}, |
349 | 353 | "source": [ |
350 | | - "## Step 3: Create Causal Test Cases" |
| 354 | + "## Step 4: Create Causal Test Cases" |
351 | 355 | ] |
352 | 356 | }, |
353 | 357 | { |
354 | 358 | "cell_type": "markdown", |
355 | 359 | "id": "be854667-44de-4f40-a37d-fb35588f047a", |
356 | 360 | "metadata": {}, |
357 | 361 | "source": [ |
358 | | - "Now that we've created our Causal Specification, we're ready to create our Causal Tests. Causal tests are essentially metamorphic tests that are executed using statistical causal inference. A causal test expresses the change in a given output that we expect to see when we change a particular input in some way. \n", |
| 362 | + "Now that we've created our Modelling Scenario, we're ready to create our Causal Tests. Causal tests are essentially metamorphic tests that are executed using statistical causal inference. A causal test expresses the change in a given output that we expect to see when we change a particular input in some way. \n", |
359 | 363 | "\n", |
360 | 364 | "Firstly, a `base test case`, which specifies the relationship between the given output and input and the desired effect, is required to build a `causal test case`. Together, the causal test case forms the complete executable test, which is the minimum required to perform identification on the DAG.\n", |
361 | 365 | "\n", |
|
575 | 579 | "source": [ |
576 | 580 | "In a very similar way to the method above, we can test our second metamorphic relation that the number of polygons per unit area should be independent of sample width and height. Since we are only interested in whether there is some effect, we use the average treatment effect (ATE) instead of the risk ratio from above, which quantifies the additive change in outcome caused by the intervention.\n", |
577 | 581 | "\n", |
578 | | - "To investigate whether the width affects number of polygons per unit area, we need to execute a new set of test cases, but this time fixing the intensity and varying the width. Note: we don't need to redefine the causal specification, nor the perform identification again; but we have to redefine our base test case since we're now considering the Polygon's width as the treatment variable." |
| 582 | + "To investigate whether the width affects number of polygons per unit area, we need to execute a new set of test cases, but this time fixing the intensity and varying the width. Note: we don't need to redefine the modelling scenario, nor the perform identification again; but we have to redefine our base test case since we're now considering the Polygon's width as the treatment variable." |
579 | 583 | ] |
580 | 584 | }, |
581 | 585 | { |
|
823 | 827 | ], |
824 | 828 | "metadata": { |
825 | 829 | "kernelspec": { |
826 | | - "display_name": "Python 3 (ipykernel)", |
| 830 | + "display_name": "Python 3 (CI)", |
827 | 831 | "language": "python", |
828 | 832 | "name": "python3" |
829 | 833 | }, |
|
837 | 841 | "name": "python", |
838 | 842 | "nbconvert_exporter": "python", |
839 | 843 | "pygments_lexer": "ipython3", |
840 | | - "version": "3.10.12" |
| 844 | + "version": "3.11.14" |
841 | 845 | } |
842 | 846 | }, |
843 | 847 | "nbformat": 4, |
|
0 commit comments