Skip to content

Commit c1e591d

Browse files
author
Jérôme FERET
committed
script fair comparison between classical and parametric analysis
1 parent c36a071 commit c1e591d

2 files changed

Lines changed: 76 additions & 36 deletions

File tree

‎examples/benchmarks/incremental-KaSa/output/experiments_output.tex‎

Lines changed: 13 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -7,38 +7,20 @@
77
\cmidrule(lr){4-6}
88
& & \bfseries\shortstack{analysis} & \bfseries\shortstack{initial\\analysis} & \bfseries\shortstack{disable\\rules} & \bfseries\shortstack{add\\a rule}\\
99
\midrule
10-
\texttt{fgf} & 41 & 0.0174 & 0.0387 & 0.00364 & 0.0152 \\
11-
\texttt{sfb} & 70 & 0.0271 & 0.0813 & 0.00642 & 0.049 \\
12-
\texttt{tgf\_v19} & 96 & 0.0803 & 0.111 & 0.011 & 0.0425 \\
13-
\texttt{tgf\_v20} & 98 & 0.0997 & 0.143 & 0.014 & 0.061 \\
14-
\texttt{tgf\_v21} & 210 & 0.224 & 0.303 & 0.0124 & 0.0964 \\
15-
\texttt{tgf\_20170401} & 214 & 0.2 & 0.257 & 0.0077 & 0.0936 \\
16-
\texttt{machine} & 219 & 0.237 & 0.26 & 0.012 & 0.121 \\
17-
\texttt{ensemble} & 231 & 0.126 & 0.154 & 0.00368 & 0.0406 \\
18-
\texttt{tgf\_20180419} & 291 & 0.208 & 0.249 & 0.00679 & 0.0883 \\
19-
\texttt{wnt\_20151228} & 355 & 0.955 & 0.635 & 0.0683 & 0.34 \\
20-
\texttt{wnt\_20160928} & 1418 & 3.23 & 3.34 & 0.0147 & 1.5 \\
21-
\texttt{wnt\_20170322} & 1469 & 3.58 & 3.69 & 0.0135 & 1.66 \\
22-
\texttt{korkut\_20170113} & 3915 & 3.3 & 3.24 & 0.0756 & 0.124 \\
23-
\texttt{korkut\_20170206} & 5749 & 14.2 & 14.2 & 0.207 & 1.24 \\
24-
\texttt{korkut\_20170117} & 12895 & 13.3 & 13.3 & 0.0213 & 0.143 \\
10+
\texttt{tgf\_v19} & 96 & 0.0827 & 0.132 & 0.0114 & 0.0436 \\
11+
\texttt{tgf\_20180419} & 291 & 0.212 & 0.244 & 0.00675 & 0.0849 \\
12+
\texttt{wnt\_20151228} & 355 & 0.94 & 1.09 & 0.0613 & 0.339 \\
13+
\texttt{wnt\_20170322} & 1469 & 3.65 & 3.9 & 0.0208 & 1.72 \\
14+
\texttt{korkut\_20170113} & 3915 & 3.33 & 3.36 & 0.0759 & 0.122 \\
15+
\texttt{korkut\_20170206} & 5749 & 13.9 & 13.8 & 0.206 & 1.2 \\
2516
\bottomrule
2617
\end{tabular}
2718
\AltTextCMSB{
28-
The table shows the number of rules and the runtimes of the evaluation for 15 models from the literature.
29-
The model fgf has 41 rules. The non-incremental analysis took 0.0174 seconds. The incremental initial analysis took 0.0387 seconds. Disabling 10 rules took 0.00364 seconds. Adding a rule took 0.0152 seconds.
30-
The model sfb has 70 rules. The non-incremental analysis took 0.0271 seconds. The incremental initial analysis took 0.0813 seconds. Disabling 10 rules took 0.00642 seconds. Adding a rule took 0.049 seconds.
31-
The model tgf\_v19 has 96 rules. The non-incremental analysis took 0.0803 seconds. The incremental initial analysis took 0.111 seconds. Disabling 10 rules took 0.011 seconds. Adding a rule took 0.0425 seconds.
32-
The model tgf\_v20 has 98 rules. The non-incremental analysis took 0.0997 seconds. The incremental initial analysis took 0.143 seconds. Disabling 10 rules took 0.014 seconds. Adding a rule took 0.061 seconds.
33-
The model tgf\_v21 has 210 rules. The non-incremental analysis took 0.224 seconds. The incremental initial analysis took 0.303 seconds. Disabling 10 rules took 0.0124 seconds. Adding a rule took 0.0964 seconds.
34-
The model tgf\_20170401 has 214 rules. The non-incremental analysis took 0.2 seconds. The incremental initial analysis took 0.257 seconds. Disabling 10 rules took 0.0077 seconds. Adding a rule took 0.0936 seconds.
35-
The model machine has 219 rules. The non-incremental analysis took 0.237 seconds. The incremental initial analysis took 0.26 seconds. Disabling 10 rules took 0.012 seconds. Adding a rule took 0.121 seconds.
36-
The model ensemble has 231 rules. The non-incremental analysis took 0.126 seconds. The incremental initial analysis took 0.154 seconds. Disabling 10 rules took 0.00368 seconds. Adding a rule took 0.0406 seconds.
37-
The model tgf\_20180419 has 291 rules. The non-incremental analysis took 0.208 seconds. The incremental initial analysis took 0.249 seconds. Disabling 10 rules took 0.00679 seconds. Adding a rule took 0.0883 seconds.
38-
The model wnt\_20151228 has 355 rules. The non-incremental analysis took 0.955 seconds. The incremental initial analysis took 0.635 seconds. Disabling 10 rules took 0.0683 seconds. Adding a rule took 0.34 seconds.
39-
The model wnt\_20160928 has 1418 rules. The non-incremental analysis took 3.23 seconds. The incremental initial analysis took 3.34 seconds. Disabling 10 rules took 0.0147 seconds. Adding a rule took 1.5 seconds.
40-
The model wnt\_20170322 has 1469 rules. The non-incremental analysis took 3.58 seconds. The incremental initial analysis took 3.69 seconds. Disabling 10 rules took 0.0135 seconds. Adding a rule took 1.66 seconds.
41-
The model korkut\_20170113 has 3915 rules. The non-incremental analysis took 3.3 seconds. The incremental initial analysis took 3.24 seconds. Disabling 10 rules took 0.0756 seconds. Adding a rule took 0.124 seconds.
42-
The model korkut\_20170206 has 5749 rules. The non-incremental analysis took 14.2 seconds. The incremental initial analysis took 14.2 seconds. Disabling 10 rules took 0.207 seconds. Adding a rule took 1.24 seconds.
43-
The model korkut\_20170117 has 12895 rules. The non-incremental analysis took 13.3 seconds. The incremental initial analysis took 13.3 seconds. Disabling 10 rules took 0.0213 seconds. Adding a rule took 0.143 seconds.
19+
The table shows the number of rules and the runtimes of the evaluation for 6 models from the literature.
20+
The model tgf\_v19 has 96 rules. The non-incremental analysis took 0.0827 seconds. The incremental initial analysis took 0.132 seconds. Disabling 10 rules took 0.0114 seconds. Adding a rule took 0.0436 seconds.
21+
The model tgf\_20180419 has 291 rules. The non-incremental analysis took 0.212 seconds. The incremental initial analysis took 0.244 seconds. Disabling 10 rules took 0.00675 seconds. Adding a rule took 0.0849 seconds.
22+
The model wnt\_20151228 has 355 rules. The non-incremental analysis took 0.94 seconds. The incremental initial analysis took 1.09 seconds. Disabling 10 rules took 0.0613 seconds. Adding a rule took 0.339 seconds.
23+
The model wnt\_20170322 has 1469 rules. The non-incremental analysis took 3.65 seconds. The incremental initial analysis took 3.9 seconds. Disabling 10 rules took 0.0208 seconds. Adding a rule took 1.72 seconds.
24+
The model korkut\_20170113 has 3915 rules. The non-incremental analysis took 3.33 seconds. The incremental initial analysis took 3.36 seconds. Disabling 10 rules took 0.0759 seconds. Adding a rule took 0.122 seconds.
25+
The model korkut\_20170206 has 5749 rules. The non-incremental analysis took 13.9 seconds. The incremental initial analysis took 13.8 seconds. Disabling 10 rules took 0.206 seconds. Adding a rule took 1.2 seconds.
4426
}

‎examples/benchmarks/incremental-KaSa/script_disable_and_add_rules‎

Lines changed: 63 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,67 @@ for MODEL in "${MODELS[@]}"; do
253253
break
254254
fi
255255

256+
257+
# PARAMETRIC MODEL ANALYSIS
258+
259+
if $VERBOSE; then
260+
echo "➖ Starting parametric models analysis:"
261+
fi
262+
263+
# choose 10 random rules to be added to the working set
264+
random_rules=$(paste -s -d ',' <(shuf -i 0-$NR_RULES_MODEL -n "$NR_RULES"))
265+
266+
if $VERBOSE; then
267+
echo "Random list of rules that are in the working set: $random_rules"
268+
fi
269+
270+
# choose a random valuation (true or false) for each of the rules in the working set
271+
false_indices=()
272+
for ((n=0; n<NR_RULES; n++)); do
273+
val=$((RANDOM % 2))
274+
if [ "$val" -eq 0 ]; then
275+
false_indices+=("$n")
276+
fi
277+
done
278+
279+
echo "q" > "$INPUT_INCREMENTAL_TMP"
256280

281+
if $VERBOSE; then
282+
echo "PARAMETRIC MODELS COMMANDS:"
283+
cat "$INPUT_INCREMENTAL_TMP"
284+
fi
285+
286+
cmd=(
287+
timeout "$TIMEOUT"
288+
"$KASA_INC_BINARY"
289+
"$EXAMPLE_MODEL"
290+
-syntax "$SYNTAX"
291+
$OPTIONS
292+
--rules-in-working-set "$random_rules"
293+
)
294+
if $VERBOSE; then
295+
printf '%q ' cat "$INPUT_INCREMENTAL_TMP" '|' "${cmd[@]}"
296+
printf '\n'
297+
fi
298+
{ cat "$INPUT_INCREMENTAL_TMP"| ${cmd[@]}; } >> "$STDOUT_OUTPUT_FILE_DECREMENTAL"
299+
EXIT_STATUS=$?
300+
301+
# Check if the program was killed by timeout
302+
if [ $EXIT_STATUS -eq 124 ]; then
303+
echo "⚠️ Timeout or memory overflow occurred for $MODEL in the decremental analysis."
304+
break
305+
fi
306+
307+
# copy runtimes from output/profiling.html to the csv output file
308+
# initial analysis
309+
store_runtime_from_profiling "Global initialization" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,1_initialization,$i,$NR_RULES"
310+
store_runtime_from_profiling "Initial states computation" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,2_initial_state,$i,$NR_RULES"
311+
store_runtime_from_profiling "Compute fixpoint of reachability analysis" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,3_fixpoint,$i,$NR_RULES"
312+
# projection to the chosen valuation (and printing)
313+
store_runtime_from_profiling "Print result of reachability analysis" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,5_print,$i,$NR_RULES"
314+
315+
316+
257317
# DECREMENTAL ANALYSIS
258318

259319
if $VERBOSE; then
@@ -312,13 +372,11 @@ for MODEL in "${MODELS[@]}"; do
312372
fi
313373

314374
# copy runtimes from output/profiling.html to the csv output file
315-
# initial analysis
316-
store_runtime_from_profiling "Global initialization" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,1_initialization,$i,$NR_RULES"
317-
store_runtime_from_profiling "Initial states computation" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,2_initial_state,$i,$NR_RULES"
318-
store_runtime_from_profiling "Compute fixpoint of reachability analysis" "$MODEL,$NR_RULES_MODEL,2_decremental,1_init,3_fixpoint,$i,$NR_RULES"
319375
# projection to the chosen valuation (and printing)
320376
store_runtime_from_profiling "Enable or disable a rule" "$MODEL,$NR_RULES_MODEL,2_decremental,4_disable,4_disable,$i,$NR_RULES"
321-
store_runtime_from_profiling "Print result of reachability analysis" "$MODEL,$NR_RULES_MODEL,2_decremental,4_disable,5_print,$i,$NR_RULES"
377+
store_runtime_from_profiling "Print result of reachability analysis" "$MODEL,$NR_RULES_MODEL,2_decremental,4_disable,5_print,$i,$NR_RULES"
378+
379+
322380

323381
# INCREMENTAL ADDING OFcode RULES
324382
# I redo the analysis instead of just adding the rules to he previous instance of the analysis, because the "update file" resets the profiling.html

0 commit comments

Comments
 (0)