Skip to content

Commit fa4cb8e

Browse files
jmafoster1jmafoster1
committed
Updated schematic
1 parent dc28758 commit fa4cb8e

9 files changed

Lines changed: 132 additions & 156 deletions

File tree

images/.gitignore renamed to docs/source/_static/images/.gitignore

File renamed without changes.

‎docs/source/_static/images/schematic-dark.png‎

16.1 KB
Loading

‎docs/source/_static/images/schematic.png‎

14.9 KB
Loading

‎docs/source/_static/images/schematic.tex‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
\documentclass{standalone}
2+
3+
\usepackage{tikz}
4+
\usetikzlibrary{arrows,positioning,shapes,calc,fit,overlay-beamer-styles, backgrounds}
5+
\usepackage{dsfont,pifont}
6+
\newcommand*{\expe}{\mathds{E}}
7+
\usepackage{amsmath}
8+
\usepackage{booktabs}
9+
\usepackage{fontawesome7}
10+
11+
\usepackage[default]{FiraSans}
12+
\usepackage[mathrm=sym]{unicode-math}
13+
\setmathfont{Fira Math}
14+
15+
\newcommand{\indep}{\perp \!\!\! \perp}
16+
17+
\begin{document}
18+
\tikzset{
19+
node/.style={circle, draw, minimum size=3ex, inner sep=0.2},
20+
edge/.style={->,> = latex'},
21+
}
22+
23+
\newcommand{\cmark}{\ding{51}}%
24+
\newcommand{\xmark}{\ding{55}}%
25+
26+
\begin{tikzpicture}[background rectangle/.style={fill=none}, show background rectangle, color=black]
27+
28+
% Test Case
29+
\begin{scope}[name prefix=test-, local bounding box=test-case]
30+
\node[draw=none, rectangle, anchor=north] (title) at (0, 0) {Causal Test Cases};
31+
\node[anchor=north,align=center] (tuple) at (title.south) {$I \to_{?} Y_3$\hspace{5mm}$X_2 \indep_? X_2$};
32+
\node[draw, rectangle] [fit=(title) (tuple)] {};
33+
\end{scope}
34+
35+
% ci
36+
\begin{scope}[name prefix=ci-, local bounding box=ci, shift={($(test-test-case.east) + (1, 0)$)}]
37+
\node[draw=none, rectangle, anchor=south west] (title) {Causal Inference};
38+
\node[draw=none, rectangle, anchor=north, align=center] (brain) at (title.south) {\faIcon{hexagon-nodes-bolt}};
39+
40+
\coordinate (top) at ({(0, 0)} |- test-title.north);
41+
\coordinate (bot) at ({(0, 0)} |- test-tuple.south);
42+
43+
\node[draw, rectangle] [fit=(title) (brain) (top) (bot)] {};
44+
\end{scope}
45+
46+
% Estimate
47+
\begin{scope}[name prefix=estimate-, local bounding box=estimate, shift={($(ci-ci.east)+(1, 0)$)}]
48+
\node[draw=none, rectangle, anchor=south west] (title) {Causal Estimate};
49+
\node[anchor=north] (table) at (title.south) {\faIcon{chart-line}};
50+
\coordinate (top) at ({(0, 0)} |- test-title.north);
51+
\coordinate (bot) at ({(0, 0)} |- test-tuple.south);
52+
\node[draw, rectangle] [fit=(title) (table) (top) (bot)] {};
53+
\end{scope}
54+
55+
% Oracle
56+
\begin{scope}[name prefix=oracle-, local bounding box=test-oracle, shift={($(estimate-estimate.east) + (1, 0)$)}]
57+
\node[draw=none, rectangle, anchor=south west] (title) {Test Oracle};
58+
\node[draw=none, rectangle, anchor=north] (scale) at (title.south) {\faIcon{scale-balanced}};
59+
60+
\coordinate (top) at ({(0, 0)} |- test-title.north);
61+
\coordinate (bot) at ({(0, 0)} |- test-tuple.south);
62+
\node[draw, rectangle] [fit=(title) (scale) (top) (bot)] {};
63+
\end{scope}
64+
65+
% Outcome
66+
\begin{scope}[name prefix=outcome-, local bounding box=test-outcome, shift={($(oracle-test-oracle.east) + (1, 0)$)}]
67+
\node[draw=none, rectangle, anchor=south west] (title) at (0,0) {Test Outcomes};
68+
\node[draw=none, anchor=north] (ok) at (title.south) {\cmark ~ \xmark};
69+
70+
\coordinate (top) at ({(0, 0)} |- test-title.north);
71+
\coordinate (bot) at ({(0, 0)} |- test-tuple.south);
72+
\node[draw, rectangle] (test-outcome) [fit=(outcome-title) (outcome-ok) (top) (bot)] {};
73+
\end{scope}
74+
75+
76+
% Causal DAG
77+
\begin{scope}[name prefix=dag-, shift={(0, 2)}]
78+
\node[node] (x1) at (-1, 0) {$X_1$};
79+
\node[node] (x2) at (-1, 1.4) {$X_2$};
80+
\node[node] (i) at (0, 0.7) {$I$};
81+
\node[node] (y1) at (1,0) {$Y_{1}$};
82+
\node[node] (y2) at (1,0.7) {$Y_2$};
83+
\node[node] (y3) at (1,1.4) {$Y_3$};
84+
85+
\draw[edge] (x1) to (i);
86+
\draw[edge] (x2) to (i);
87+
\draw[edge] (i) to (y1);
88+
\draw[edge] (i) to (y2);
89+
\draw[edge] (i) to (y3);
90+
\draw[edge] (x1) to (y1);
91+
\draw[edge] (x2) to (y3);
92+
\node[draw=none, rectangle] (nodes) [fit=(x1) (x2) (y1) (y2) (y3) (i)] {};
93+
\node[draw=none, rectangle, anchor=south] (title) at (nodes.north) {Causal DAG};
94+
\end{scope}
95+
% DAG outline
96+
\node[draw, rectangle] (dag) [fit=(dag-nodes) (dag-title) (dag-title)] {};
97+
98+
% Data
99+
\begin{scope}[name prefix=data-, local bounding box=test-data, shift={($(dag) + (5, 1.12)$)}]
100+
\node[draw=none, rectangle] (title) {Test Data};
101+
\node[anchor=north] (table) at (title.south) {
102+
\begin{tabular}{rrrrrr}
103+
\toprule
104+
$X_1$ & $X_2$ & $I$ & $Y_1$ & $Y_2$ & $Y_3$ \\
105+
\midrule
106+
1.2 & ``UK'' & 0.3 & 7.8 & 4 & True \\
107+
3.2 & ``UK'' & 0.1 & 7.6 & 8 & False \\
108+
\multicolumn{6}{c}{$\vdots$} \\
109+
\bottomrule
110+
\end{tabular}
111+
};
112+
\node[draw, rectangle] [fit=(title) (table)] {};
113+
\end{scope}
114+
115+
116+
%Information flow
117+
\draw[edge, dashed] (dag.290) -- (ci-ci.160);
118+
\draw[edge, dashed] (dag) -- (test-test-case.north);
119+
\draw[edge, dashed] (test-test-case) -- (ci-ci);
120+
121+
\draw[edge, dashed] (data-test-data.south) -- (data-test-data |- ci-ci.north);
122+
\draw[edge, dashed] (ci-ci) -- (estimate-estimate);
123+
124+
\draw[edge, dashed] (estimate-estimate) -- (oracle-test-oracle.west |- estimate-estimate);
125+
\draw[edge, dashed] (oracle-test-oracle.east |- outcome-test-outcome) -- (outcome-test-outcome);
126+
\end{tikzpicture}
127+
\end{document}

‎docs/source/index.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,10 @@ Welcome to the Causal Testing Framework
77
Motivation
88
----------
99

10-
A common problem in computer science is to develop robust and reliable software systems that can perform correctly under various input configurations and maintain consistency across complex, physical scenarios. However, software systems, and more specifically computational models, can be difficult to test: they may contain hundreds of parameters, making testing all possible inputs computationally infeasible; some models may be inherently non-deterministic, producing different outputs for the same inputs due to randomness; or there may exist hidden causal relationships between input-output pairs, causing errors that only appear under specific combinations of input configurations.
10+
From predicting the weather to simulating disease transmission, scientific software plays an increasingly pivotal role in developing scientific understanding that informs our everyday lives.
11+
However, they are also some of the most difficult software systems to properly test.
12+
They have large, complex input spaces, are computationally expensive to run, often rely on stochastic black-box components, and are applied in exploratory contexts where the expected outcomes are not known.
13+
From a practical standpoint, the time and effort that can be dedicated to testing is often limited, especially in an academic context, making it especially important to maximise the efficiency of the limited number of test runs we are able to perform.
1114

1215
The Framework
1316
-------------

‎images/schematic-dark.png‎

-23.3 KB
Binary file not shown.

‎images/schematic.png‎

-21.6 KB
Binary file not shown.

‎images/schematic.tex‎

Lines changed: 0 additions & 154 deletions
This file was deleted.

‎paper/paper.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ The user may also refine tests to validate the nature of a particular relationsh
6969
Next, the user supplies a set of runtime data in the form of a table with each column representing a variable and rows containing the value of each variable for a particular run of the software.
7070
Finally, the CTF automatically validates the causal properties by using the causal DAG to identify a statistical estimand [@pearl2009causality] (essentially a set of features in the data which must be controlled for), calculate a causal effect estimate from the supplied data, and validating this against the expected causal relationship.
7171

72-
![Causal Testing workflow.\label{fig:schematic}](../images/schematic.png)
72+
![Causal Testing workflow.\label{fig:schematic}](../docs/_static/images/schematic.png)
7373

7474
## Test Adequacy
7575
Because the properties being tested are completely separate from the data used to validate them, traditional coverage-based metrics are not appropriate here.

0 commit comments

Comments
 (0)