@@ -10,21 +10,21 @@ Gizmos for working with CSVs
1010* [ csvplot] ( #csvplot ) -- line and scatter plots from CSV files
1111* [ csvstats] ( #csvstats ) -- histograms and summary statistics for CSV files
1212* [ csvcat] ( #csvcat ) -- concatenate CSV files
13- * [ can2k] ( #can2k ) -- parse NMEA 2000 GPS data from CAN logs
14- * [ qgsdir] ( #qgsdir ) -- generate QGIS projects from directories of CSV files
1513* [ csvdelta] ( #csvdelta ) -- calculate inter-row deltas for CSV files
16- * [ minpath] ( #minpath ) -- shorten file paths to minimal unique suffixes
17- * [ depconv] ( #depconv ) -- convert dependency graphs between formats
18- * [ depfilter] ( #depfilter ) -- filter or select subsets of dependency graphs
19- * [ deptransform] ( #deptransform ) -- transform dependency graphs
14+ * [ can2k] ( #can2k ) -- parse NMEA 2000 GPS data from CAN logs
2015* [ can2csv] ( #can2csv ) -- parse CAN logs into CSV files
2116* [ canspam] ( #canspam ) -- generate random CAN traffic
2217* [ canstruct] ( #canstruct ) -- reconstruct NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol
2318 sessions
19+ * [ qgsdir] ( #qgsdir ) -- generate QGIS projects from directories of CSV files
20+ * [ depconv] ( #depconv ) -- convert dependency graphs between formats
21+ * [ depfilter] ( #depfilter ) -- filter or select subsets of dependency graphs
22+ * [ deptransform] ( #deptransform ) -- transform dependency graphs
2423* [ depquery] ( #depquery ) -- query properties of dependency graphs
2524* [ depcluster] ( #depcluster ) -- cluster dependency graphs using community detection
2625* [ graphdiff] ( #graphdiff ) -- compare two dependency graphs
2726* [ bbclasses] ( #bbclasses ) -- generate BitBake recipe inheritance diagrams
27+ * [ minpath] ( #minpath ) -- shorten file paths to minimal unique suffixes
2828
2929# Philosophy
3030
@@ -125,6 +125,27 @@ foo,bar,baz
1251257,8,9
126126```
127127
128+ ## csvdelta
129+
130+ Calculate the inter-row deltas for a CSV column. Useful for understanding the time between events.
131+ Also supports mean-centering a column, or centering it around a specific value.
132+
133+ ``` sh
134+ $ csvdelta --column foo << EOF
135+ foo,bar
136+ 0,a
137+ 1,b
138+ 3,c
139+ 5,d
140+ EOF
141+
142+ foo,bar,foo-deltas
143+ 0,a,
144+ 1,b,1
145+ 3,c,2
146+ 5,d,2
147+ ```
148+
128149## can2k
129150
130151Parse NMEA 2000 GPS data out of a candump into a CSV file that QGIS can load with minimal effort.
@@ -142,59 +163,65 @@ src,seq_id,longitude_deg,latitude_deg,altitude_m,sog_mps,cog_deg_cwfn,cog_ref,me
142163>
143164> If you want to use ` can2k ` together with ` qgsdir ` , you need to use ` can2k --wkt ` .
144165
145- ## qgsdir
166+ ## can2csv
146167
147- Generate a QGIS project from a directory of CSV layer files. Each CSV file is assumed to have a
148- column of WKT geometries named ` geometry ` (QGIS's geometry heuristics don't appear to be exposed via
149- their Python API).
168+ Parse basic data from a CAN frame into a CSV record. Faster than ` sed ` , and also parses the canid.
169+ Useful in conjunction with ` csvdelta ` to understand message timing.
170+
171+ ` can2csv ` is not a real CAN parser, and does not understand any of the data transmitted via CAN.
150172
151173``` sh
152- $ can2k --wkt ./data/n2k-sample.log ./data/n2k.csv
153- $ qgsdir --open ./data/n2k.csv
174+ $ head -n 3 data/candump-random-data.log | can2csv
175+ timestamp,interface,canid,dlc,priority,src,dst,pgn,data
176+ 1739229594.465994,can0,0xE9790B5,8,3,0xB5,0x90,0x29700,CA3F871A5A6EE75F
177+ 1739229594.467052,can0,0xD15F192,8,3,0x92,0xF1,0x11500,500B3766CB2DED7C
154178```
155179
156- You may pass directories or files. If you pass a directory, the script is able to group layers by
157- subdirectory, leading to an easier-to-use layer tree.
180+ If you pass ` --reconstruct ` , then ` can2csv ` will reconstruct any transport layer sessions it can
181+ understand. Right now that's just NMEA 2000 Fast Packet, but ISO-11783 Transport Protocol is
182+ planned.
158183
159- ## csvdelta
184+ ## canspam
160185
161- Calculate the inter-row deltas for a CSV column. Useful for understanding the time between events.
162- Also supports mean-centering a column, or centering it around a specific value.
186+ The [ canspam] ( ./scripts/canspam ) script can generate random CAN traffic on a Linux CAN device. It's
187+ useful for inflating busload, or for generating random traffic to test ` can2csv ` against ;)
188+
189+ ## canstruct
190+
191+ The ` canstruct ` tool is a NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol transport session
192+ reconstruction tool. That is, you give it the individual 8-byte frames, and it gives you the
193+ reconstructed messages.
163194
164195``` sh
165- $ csvdelta --column foo << EOF
166- foo,bar
167- 0,a
168- 1,b
169- 3,c
170- 5,d
171- EOF
196+ $ cat data/abort-then-full.log
197+ (1750963033.251412) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
198+ (1750963033.270725) can0 18EC1C2A#FF01FFFFFF00EF00 // TP.Conn_Abort
199+ (1750963079.757877) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
200+ (1750963079.775206) can0 18EC1C2A#110401FFFF00EF00 // TP.CM_CTS
201+ (1750963079.778342) can0 14EB2A1C#0111111111111111 // TP.DT
202+ (1750963079.779468) can0 14EB2A1C#0222222222222222 // TP.DT
203+ (1750963079.780613) can0 14EB2A1C#0333333333333333 // TP.DT
204+ (1750963079.781778) can0 14EB2A1C#0444FFFFFFFFFFFF // TP.DT
205+ (1750963079.795905) can0 18EC1C2A#13160004FF00EF00 // TP.CM_EndofMsgACK
172206
173- foo,bar,foo-deltas
174- 0,a,
175- 1,b,1
176- 3,c,2
177- 5,d,2
207+ $ canstruct data/abort-then-full.log
208+ 2025-06-28T15:36:19.051620Z WARN csvizmo::can::tp: TP.CM_ABRT 0x1C < - 0x2A reason ExistingTransportSession pgn 0xEF00
209+ (1750963079.795905) can0 18EF2A1C#11111111111111222222222222223333333333333344
178210```
179211
180- ## minpath
212+ ## qgsdir
181213
182- Shorten file paths to the minimal unique suffix. Useful for displaying lists of files in a compact
183- way while keeping them distinguishable.
214+ Generate a QGIS project from a directory of CSV layer files. Each CSV file is assumed to have a
215+ column of WKT geometries named ` geometry ` (QGIS's geometry heuristics don't appear to be exposed via
216+ their Python API).
184217
185218``` sh
186- $ minpath << EOF
187- /home/user/project/src/main.rs
188- /home/user/project/src/lib.rs
189- /home/user/project/tests/main.rs
190- EOF
191-
192- src/main.rs
193- lib.rs
194- tests/main.rs
219+ $ can2k --wkt ./data/n2k-sample.log ./data/n2k.csv
220+ $ qgsdir --open ./data/n2k.csv
195221```
196222
197- Multiple options are available to customize and tune the output. See ` minpath --help ` for details.
223+ You may pass directories or files. If you pass a directory, the script is able to group layers by
224+ subdirectory, leading to an easier-to-use layer tree.
198225
199226## depconv
200227
@@ -387,52 +414,6 @@ flowchart LR
387414 a --> b
388415` ` `
389416
390- # # can2csv
391-
392- Parse basic data from a CAN frame into a CSV record. Faster than ` sed` , and also parses the canid.
393- Useful in conjunction with ` csvdelta` to understand message timing.
394-
395- ` can2csv` is not a real CAN parser, and does not understand any of the data transmitted via CAN.
396-
397- ` ` ` sh
398- $ head -n 3 data/candump-random-data.log | can2csv
399- timestamp,interface,canid,dlc,priority,src,dst,pgn,data
400- 1739229594.465994,can0,0xE9790B5,8,3,0xB5,0x90,0x29700,CA3F871A5A6EE75F
401- 1739229594.467052,can0,0xD15F192,8,3,0x92,0xF1,0x11500,500B3766CB2DED7C
402- ` ` `
403-
404- If you pass ` --reconstruct` , then ` can2csv` will reconstruct any transport layer sessions it can
405- understand. Right now that' s just NMEA 2000 Fast Packet, but ISO-11783 Transport Protocol is
406- planned.
407-
408- ## canspam
409-
410- The [canspam](./scripts/canspam) script can generate random CAN traffic on a Linux CAN device. It' s
411- useful for inflating busload, or for generating random traffic to test ` can2csv` against ; )
412-
413- # # canstruct
414-
415- The ` canstruct` tool is a NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol transport session
416- reconstruction tool. That is, you give it the individual 8-byte frames, and it gives you the
417- reconstructed messages.
418-
419- ` ` ` sh
420- $ cat data/abort-then-full.log
421- (1750963033.251412) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
422- (1750963033.270725) can0 18EC1C2A#FF01FFFFFF00EF00 // TP.Conn_Abort
423- (1750963079.757877) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
424- (1750963079.775206) can0 18EC1C2A#110401FFFF00EF00 // TP.CM_CTS
425- (1750963079.778342) can0 14EB2A1C#0111111111111111 // TP.DT
426- (1750963079.779468) can0 14EB2A1C#0222222222222222 // TP.DT
427- (1750963079.780613) can0 14EB2A1C#0333333333333333 // TP.DT
428- (1750963079.781778) can0 14EB2A1C#0444FFFFFFFFFFFF // TP.DT
429- (1750963079.795905) can0 18EC1C2A#13160004FF00EF00 // TP.CM_EndofMsgACK
430-
431- $ canstruct data/abort-then-full.log
432- 2025-06-28T15:36:19.051620Z WARN csvizmo::can::tp: TP.CM_ABRT 0x1C < - 0x2A reason ExistingTransportSession pgn 0xEF00
433- (1750963079.795905) can0 18EF2A1C#11111111111111222222222222223333333333333344
434- ` ` `
435-
436417# # bbclasses
437418
438419The [bbclasses](./scripts/bbclasses) script can parse BitBake recipes to generate an inheritance
@@ -502,3 +483,22 @@ flowchart LR
502483 poky/meta/conf/distro/include/ptest-packagelists.inc --> | " require" | poky/meta/classes-recipe/ptest.bbclass
503484 poky/meta/recipes-support/curl/curl_8.7.1.bb --> | " appends" | meta-work/recipes-support/curl/curl__.bbappend
504485` ` `
486+
487+ # # minpath
488+
489+ Shorten file paths to the minimal unique suffix. Useful for displaying lists of files in a compact
490+ way while keeping them distinguishable.
491+
492+ ` ` ` sh
493+ $ minpath << EOF
494+ /home/user/project/src/main.rs
495+ /home/user/project/src/lib.rs
496+ /home/user/project/tests/main.rs
497+ EOF
498+
499+ src/main.rs
500+ lib.rs
501+ tests/main.rs
502+ ` ` `
503+
504+ Multiple options are available to customize and tune the output. See ` minpath --help` for details.
0 commit comments