Skip to content

Commit 86015ee

Browse files
[show][generate_dump] Add optional tcpdump packet capture to "show techsupport"
Why --- `show techsupport` collects a rich snapshot of device state, but it is purely a point-in-time view: it has no visibility into the traffic that was actually on the wire while the dump was taken. For a large class of network troubleshooting, packet-level evidence is exactly what is needed, and today it has to be gathered out of band with a separate tcpdump invocation and then manually correlated with the techsupport archive. This adds packet capture as a first-class, optional part of techsupport. When enabled, traffic is captured for the duration of the collection and the resulting pcap is bundled into the same techsupport tarball, so a single command produces both the device state and a matching capture taken over the same window, as one self-contained artifact. This is useful across the board -- protocol exchanges (BGP, BFD, LACP, LLDP, ARP/ND, DHCP), data-plane forwarding and drop analysis, and any investigation where seeing the packets matters -- and the optional BPF filter lets the operator scope the capture to just the traffic of interest. What ---- A new, opt-in tcpdump capture for `show techsupport`, disabled by default. show techsupport --with-tcpdump [--tcpdump-duration <seconds>] # 1-300, default 60 [--tcpdump-packet-limit <count>] # 1-100000, default 10000 [--tcpdump-filter <bpf-expression>] # e.g. 'udp port 3784' The tuning options are only meaningful together with --with-tcpdump; if they are supplied without it the command fails fast with a clear error rather than silently ignoring them. How --- show/main.py - Adds the four Click options above and forwards them to generate_dump as -T/-P/-L/-F. --tcpdump-duration and --tcpdump-packet-limit use click.IntRange so out-of-range values are rejected at the CLI; the documented defaults are applied when --with-tcpdump is given without explicit values. scripts/generate_dump - start_tcpdump_capture(): launches `tcpdump -i any` in the background at the start of collection so the capture overlaps data gathering. The capture is self-bounding via tcpdump's native options -G/-W (wall-clock duration) and -c (packet count), whichever is reached first, and -s 0 keeps full-length packets. An optional BPF filter is appended. tcpdump availability is checked first, and the capture is skipped gracefully (without failing techsupport) when it is absent. - wait_tcpdump_capture(): waits for the capture to finish. It is called *before* save_to_tar, because save_to_tar archives the dump directory by file content; awaiting the capture first guarantees the completed pcap is on disk and included in the archive instead of a truncated snapshot. In the common case (data collection outlasts the capture) the wait returns immediately. - handle_exit(): kills any still-running capture so an interrupt or timeout cannot leave an orphan tcpdump process behind. - getopts gains -T/-P/-L/-F, with validation that -P/-L/-F require -T. The pcap is written as tcpdump_<duration>s_<timestamp>.pcap inside the dump directory and therefore appears in the final techsupport archive. Backward compatibility ----------------------- The feature is entirely opt-in. Without --with-tcpdump, generate_dump and `show techsupport` behave exactly as before. Testing ------- tests/techsupport_test.py adds coverage for the CLI-to-generate_dump option mapping (defaults and explicit values), the requirement that the tuning options be used together with --with-tcpdump, IntRange bound enforcement, and a regression test asserting that wait_tcpdump_capture runs before save_to_tar in generate_dump. Signed-off-by: vaishnav-nexthop <vaishnav@nexthop.ai>
1 parent ec69c33 commit 86015ee

3 files changed

Lines changed: 252 additions & 3 deletions

File tree

scripts/generate_dump

Lines changed: 159 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,17 @@ IS_SUPERVISOR=false
5555

5656
ENABLE_FLOW_DUMP=false
5757

58+
# tcpdump capture options
59+
WITH_TCPDUMP=false
60+
TCPDUMP_DURATION=60
61+
TCPDUMP_PACKET_LIMIT=10000
62+
TCPDUMP_FILTER=""
63+
TCPDUMP_PID=""
64+
# Track if tcpdump options were explicitly set (for validation)
65+
TCPDUMP_DURATION_SET=false
66+
TCPDUMP_PACKET_LIMIT_SET=false
67+
TCPDUMP_FILTER_SET=false
68+
5869
# lock dirs/files
5970
LOCKDIR="/tmp/techsupport-lock"
6071
PIDFILE="${LOCKDIR}/PID"
@@ -69,6 +80,13 @@ rm_lock_and_exit()
6980
handle_exit()
7081
{
7182
ECODE=$?
83+
# Kill tcpdump if still running to prevent orphan processes
84+
if [ ! -z "$TCPDUMP_PID" ]; then
85+
echo "Killing tcpdump (PID: $TCPDUMP_PID)..." >&2
86+
kill $TCPDUMP_PID 2>/dev/null || true
87+
wait $TCPDUMP_PID 2>/dev/null || true
88+
TCPDUMP_PID=""
89+
fi
7290
echo "Cleaning up working directory $TARDIR"
7391
$RM -rf $TARDIR
7492
echo "Removing lock. Exit: $ECODE" >&2
@@ -102,6 +120,85 @@ escape_quotes() {
102120
echo $1 | sed 's/\"/\\\"/g'
103121
}
104122

123+
###############################################################################
124+
# Start tcpdump capture in background
125+
# Uses tcpdump's native -G (rotate) and -W (file count limit) options for
126+
# time-limited capture instead of external timeout command.
127+
#
128+
# Includes robustness checks:
129+
# - Verifies tcpdump is available
130+
# - Adds packet count limit to manage pcap file size
131+
#
132+
# Globals:
133+
# LOGDIR, TCPDUMP_PID
134+
# Arguments:
135+
# $1 - duration: capture duration in seconds
136+
# $2 - packet_limit: maximum number of packets to capture
137+
# $3 - filter: BPF filter expression (optional)
138+
# Returns:
139+
# 0 on success, 1 on failure (sets TCPDUMP_PID global variable on success)
140+
###############################################################################
141+
start_tcpdump_capture() {
142+
# Set ERR trap to invoke handle_error on any command failure within this function.
143+
# This ensures errors are logged with exit code and line number for debugging.
144+
trap 'handle_error $? $LINENO' ERR
145+
local duration=$1
146+
local packet_limit=$2
147+
local filter=$3
148+
local timestamp=$(date +%Y%m%d_%H%M%S)
149+
local pcap_file="${LOGDIR}/tcpdump_${duration}s_${timestamp}.pcap"
150+
151+
# Verify tcpdump is available
152+
if ! command -v tcpdump &> /dev/null; then
153+
echo "Error: tcpdump not found, skipping capture"
154+
return 1
155+
fi
156+
157+
# Ensure LOGDIR exists
158+
if [ ! -d "$LOGDIR" ]; then
159+
$MKDIR $V -p "$LOGDIR"
160+
fi
161+
162+
# Build tcpdump command
163+
# -i any: Capture on all interfaces
164+
# -G: Rotate dump file every <duration> seconds
165+
# -W 1: Exit after 1 file (i.e., after <duration> seconds)
166+
# -s 0: Capture full packets (no truncation)
167+
# -c: Packet count limit to manage file size
168+
local tcpdump_cmd=""
169+
if [ -n "$filter" ]; then
170+
tcpdump_cmd="tcpdump -i any -w ${pcap_file} -G ${duration} -W 1 -s 0 -c ${packet_limit} ${filter}"
171+
echo "tcpdump: BPF filter: $filter"
172+
else
173+
tcpdump_cmd="tcpdump -i any -w ${pcap_file} -G ${duration} -W 1 -s 0 -c ${packet_limit}"
174+
fi
175+
176+
echo "tcpdump: Running command: $tcpdump_cmd"
177+
$tcpdump_cmd &
178+
TCPDUMP_PID=$!
179+
echo "tcpdump: Started capture (PID: $TCPDUMP_PID) for $duration seconds (limit: $packet_limit packets)"
180+
echo "tcpdump: Output file: $pcap_file"
181+
return 0
182+
}
183+
184+
###############################################################################
185+
# Wait for tcpdump to complete
186+
# Globals:
187+
# TCPDUMP_PID
188+
# Arguments:
189+
# None
190+
# Returns:
191+
# None
192+
###############################################################################
193+
wait_tcpdump_capture() {
194+
if [ ! -z "$TCPDUMP_PID" ]; then
195+
echo "Waiting for tcpdump (PID: $TCPDUMP_PID) to complete..."
196+
wait $TCPDUMP_PID 2>/dev/null || true
197+
echo "tcpdump capture completed"
198+
TCPDUMP_PID=""
199+
fi
200+
}
201+
105202
save_bcmcmd() {
106203
trap 'handle_error $? $LINENO' ERR
107204
local start_t=$(date +%s%3N)
@@ -2633,6 +2730,11 @@ main() {
26332730
echo $BASE > $TECHSUPPORT_TIME_INFO
26342731
start_t=$(date +%s%3N)
26352732

2733+
# Start tcpdump capture if requested (runs in background during data collection)
2734+
if $WITH_TCPDUMP; then
2735+
start_tcpdump_capture "$TCPDUMP_DURATION" "$TCPDUMP_PACKET_LIMIT" "$TCPDUMP_FILTER"
2736+
fi
2737+
26362738
# Trigger BMC debug log dump task - Must be the first task to run
26372739
bmc_debug_log_dump_task_id=$(trigger_bmc_debug_log_dump)
26382740
if [ "$bmc_debug_log_dump_task_id" == "-1" ]; then
@@ -2865,9 +2967,17 @@ main() {
28652967

28662968
wait
28672969

2970+
# Wait for the tcpdump capture to finish BEFORE archiving. save_to_tar
2971+
# appends the dump tree by file content, so a capture still running at
2972+
# this point would be archived truncated (and the completed pcap, which
2973+
# is only ever added via this tree append, would never make it in).
2974+
if $WITH_TCPDUMP; then
2975+
wait_tcpdump_capture
2976+
fi
2977+
28682978
save_to_tar
28692979

2870-
save_sai_failure_dump
2980+
save_sai_failure_dump
28712981

28722982
if [[ "$asic" = "mellanox" ]] || [[ "$asic" = "nvidia-bluefield" ]]; then
28732983
collect_nvidia_sdk_dumps
@@ -3044,10 +3154,18 @@ OPTIONS
30443154
Collect the output of debug dump cli
30453155
-f
30463156
On DPU platforms, also collect a DPU flow dump
3157+
-T
3158+
Enable tcpdump capture during techsupport collection
3159+
-P PERIOD
3160+
Duration in seconds for tcpdump capture (default: 60). Requires -T
3161+
-L LIMIT
3162+
Maximum number of packets to capture (default: 10000). Requires -T
3163+
-F FILTER
3164+
BPF filter expression for tcpdump (e.g., 'port 179', 'udp port 3784'). Requires -T
30473165
EOF
30483166
}
30493167

3050-
while getopts ":xnvhzas:t:r:df" opt; do
3168+
while getopts ":xnvhzas:t:r:dfTP:L:F:" opt; do
30513169
case $opt in
30523170
x)
30533171
# enable bash debugging
@@ -3101,13 +3219,52 @@ while getopts ":xnvhzas:t:r:df" opt; do
31013219
f)
31023220
ENABLE_FLOW_DUMP=true
31033221
;;
3222+
T)
3223+
WITH_TCPDUMP=true
3224+
;;
3225+
P)
3226+
if ! [[ ${OPTARG} =~ ^[0-9]+$ ]]; then
3227+
echo "Invalid tcpdump duration value: ${OPTARG}, Please enter a numeric value."
3228+
exit $EXT_GENERAL
3229+
fi
3230+
TCPDUMP_DURATION="${OPTARG}"
3231+
TCPDUMP_DURATION_SET=true
3232+
;;
3233+
L)
3234+
if ! [[ ${OPTARG} =~ ^[0-9]+$ ]]; then
3235+
echo "Invalid tcpdump packet limit value: ${OPTARG}, Please enter a numeric value."
3236+
exit $EXT_GENERAL
3237+
fi
3238+
TCPDUMP_PACKET_LIMIT="${OPTARG}"
3239+
TCPDUMP_PACKET_LIMIT_SET=true
3240+
;;
3241+
F)
3242+
TCPDUMP_FILTER="${OPTARG}"
3243+
TCPDUMP_FILTER_SET=true
3244+
;;
31043245
/?)
31053246
echo "Invalid option: -$OPTARG" >&2
31063247
exit $EXT_GENERAL
31073248
;;
31083249
esac
31093250
done
31103251

3252+
# Validate that tcpdump options (-P, -L, -F) require -T to be specified
3253+
if ! $WITH_TCPDUMP; then
3254+
if $TCPDUMP_DURATION_SET; then
3255+
echo "Error: -P (tcpdump duration) requires -T (enable tcpdump) to be specified" >&2
3256+
exit $EXT_INVALID_ARGUMENT
3257+
fi
3258+
if $TCPDUMP_PACKET_LIMIT_SET; then
3259+
echo "Error: -L (tcpdump packet limit) requires -T (enable tcpdump) to be specified" >&2
3260+
exit $EXT_INVALID_ARGUMENT
3261+
fi
3262+
if $TCPDUMP_FILTER_SET; then
3263+
echo "Error: -F (tcpdump filter) requires -T (enable tcpdump) to be specified" >&2
3264+
exit $EXT_INVALID_ARGUMENT
3265+
fi
3266+
fi
3267+
31113268
# Check permissions before proceeding further
31123269
if [ `whoami` != root ] && ! $NOOP;
31133270
then

show/main.py

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1796,9 +1796,34 @@ def users(verbose):
17961796
@click.option('--debug-dump', is_flag=True, help="Collect Debug Dump Output")
17971797
@click.option('--redirect-stderr', '-r', is_flag=True, help="Redirect an intermediate errors to STDERR")
17981798
@click.option('--flow-dump', is_flag=True, help="Collect DPU flow dump (Only valid on DPU platforms)")
1799+
@click.option('--with-tcpdump', is_flag=True, help="Capture traffic with tcpdump during techsupport")
1800+
@click.option('--tcpdump-duration', 'tcpdump_duration', type=click.IntRange(1, 300), default=None,
1801+
help="Duration in seconds for tcpdump capture (default: 60, range: 1-300). Requires --with-tcpdump")
1802+
@click.option('--tcpdump-packet-limit', 'tcpdump_packet_limit', type=click.IntRange(1, 100000), default=None,
1803+
help="Maximum number of packets to capture (default: 10000, range: 1-100000). Requires --with-tcpdump")
1804+
@click.option('--tcpdump-filter', 'tcpdump_filter', type=str, default=None,
1805+
help="BPF filter expression for tcpdump (e.g., 'port 179', 'udp port 3784'). Requires --with-tcpdump")
17991806
def techsupport(since, global_timeout, cmd_timeout, verbose, allow_process_stop,
1800-
silent, debug_dump, redirect_stderr, flow_dump):
1807+
silent, debug_dump, redirect_stderr, flow_dump,
1808+
with_tcpdump, tcpdump_duration, tcpdump_packet_limit, tcpdump_filter):
18011809
"""Gather information for troubleshooting"""
1810+
1811+
# Validate: tcpdump options require --with-tcpdump
1812+
if not with_tcpdump:
1813+
if tcpdump_duration is not None:
1814+
raise click.UsageError("--tcpdump-duration requires --with-tcpdump to be specified")
1815+
if tcpdump_packet_limit is not None:
1816+
raise click.UsageError("--tcpdump-packet-limit requires --with-tcpdump to be specified")
1817+
if tcpdump_filter is not None:
1818+
raise click.UsageError("--tcpdump-filter requires --with-tcpdump to be specified")
1819+
1820+
# Apply defaults when --with-tcpdump is used
1821+
if with_tcpdump:
1822+
if tcpdump_duration is None:
1823+
tcpdump_duration = 60 # default 60 seconds
1824+
if tcpdump_packet_limit is None:
1825+
tcpdump_packet_limit = 10000 # default 10k packets
1826+
18021827
cmd = ["sudo"]
18031828

18041829
if global_timeout:
@@ -1825,6 +1850,15 @@ def techsupport(since, global_timeout, cmd_timeout, verbose, allow_process_stop,
18251850
cmd += ['-t', str(cmd_timeout)]
18261851
if redirect_stderr:
18271852
cmd += ["-r"]
1853+
1854+
# Add tcpdump options
1855+
if with_tcpdump:
1856+
cmd += ['-T']
1857+
cmd += ['-P', str(tcpdump_duration)]
1858+
cmd += ['-L', str(tcpdump_packet_limit)]
1859+
if tcpdump_filter:
1860+
cmd += ['-F', tcpdump_filter]
1861+
18281862
run_command(cmd, display_cmd=verbose)
18291863

18301864

tests/techsupport_test.py

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
import pytest
23
import show.main
34
from unittest.mock import patch, Mock
@@ -17,10 +18,67 @@
1718
(['--debug-dump', '--redirect-stderr'], ['generate_dump', '-v', '-d', '-t', '5', '-r']),
1819
(['--flow-dump'], ['generate_dump', '-v', '-f', '-t', '5']),
1920
(['--debug-dump', '--flow-dump'], ['generate_dump', '-v', '-d', '-f', '-t', '5']),
21+
# tcpdump: flag alone applies defaults (60s, 10k packets) -> -T -P -L
22+
(['--with-tcpdump'],
23+
['generate_dump', '-v', '-t', '5', '-T', '-P', '60', '-L', '10000']),
24+
# tcpdump: all options forwarded, filter as -F
25+
(['--with-tcpdump', '--tcpdump-duration', '30',
26+
'--tcpdump-packet-limit', '500', '--tcpdump-filter', 'udp port 3784'],
27+
['generate_dump', '-v', '-t', '5', '-T', '-P', '30', '-L', '500', '-F', 'udp port 3784']),
2028
]
2129
)
2230
def test_techsupport(run_command, cli_arguments, expected):
2331
runner = CliRunner()
2432
result = runner.invoke(show.main.cli.commands['techsupport'], cli_arguments)
2533
run_command.assert_called_with(EXPECTED_BASE_COMMAND + expected, display_cmd=False)
2634

35+
36+
@patch("show.main.run_command")
37+
@pytest.mark.parametrize(
38+
"cli_arguments,expected_msg",
39+
[
40+
(['--tcpdump-duration', '30'], "--tcpdump-duration requires --with-tcpdump"),
41+
(['--tcpdump-packet-limit', '500'], "--tcpdump-packet-limit requires --with-tcpdump"),
42+
(['--tcpdump-filter', 'port 179'], "--tcpdump-filter requires --with-tcpdump"),
43+
]
44+
)
45+
def test_techsupport_tcpdump_requires_flag(run_command, cli_arguments, expected_msg):
46+
runner = CliRunner()
47+
result = runner.invoke(show.main.cli.commands['techsupport'], cli_arguments)
48+
assert result.exit_code != 0
49+
assert expected_msg in result.output
50+
run_command.assert_not_called()
51+
52+
53+
@patch("show.main.run_command")
54+
@pytest.mark.parametrize(
55+
"cli_arguments",
56+
[
57+
['--with-tcpdump', '--tcpdump-duration', '0'], # below range (1-300)
58+
['--with-tcpdump', '--tcpdump-duration', '301'], # above range
59+
['--with-tcpdump', '--tcpdump-packet-limit', '0'], # below range (1-100000)
60+
['--with-tcpdump', '--tcpdump-packet-limit', '100001'], # above range
61+
]
62+
)
63+
def test_techsupport_tcpdump_out_of_range(run_command, cli_arguments):
64+
runner = CliRunner()
65+
result = runner.invoke(show.main.cli.commands['techsupport'], cli_arguments)
66+
assert result.exit_code != 0
67+
run_command.assert_not_called()
68+
69+
70+
def test_tcpdump_awaited_before_save_to_tar():
71+
"""The tcpdump pcap is written into the dump tree and is only added to the
72+
archive by save_to_tar (a `tar -rhf` that snapshots file content). So the
73+
capture must be awaited before save_to_tar, or a still-running capture is
74+
archived truncated and the completed pcap is lost.
75+
"""
76+
script = os.path.join(os.path.dirname(__file__), '..', 'scripts', 'generate_dump')
77+
with open(script) as f:
78+
lines = [ln.strip() for ln in f]
79+
wait_idx = next(i for i, ln in enumerate(lines) if ln == 'wait_tcpdump_capture')
80+
save_idx = next(i for i, ln in enumerate(lines) if ln == 'save_to_tar')
81+
assert wait_idx < save_idx, (
82+
"wait_tcpdump_capture (line {}) must precede save_to_tar (line {})".format(
83+
wait_idx + 1, save_idx + 1))
84+

0 commit comments

Comments
 (0)