-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathChangeLog
More file actions
2797 lines (2695 loc) · 163 KB
/
ChangeLog
File metadata and controls
2797 lines (2695 loc) · 163 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
pmacct (Promiscuous mode IP Accounting package) v1.5.0rc3
pmacct is Copyright (C) 2003-2013 by Paolo Lucente
1.5.0rc2 -- 25-12-2013
+ nfacctd: introduced support for variable-length IPFIX fields for custom-
defined aggregation primitives: 'string' semantics is supported and
maximum expected length of the field should be specified as 'len'
primitive definition. Also PENs are now supported: field_type can be
<value> or <PEN>:<value>. Finally, 'raw' semantics to print raw data,
fixed or variable length in hex format was added.
+ pmacctd, uacctd: introducing custom-defined aggregation primitives in
libpcap and ULOG daemons. A new 'packet_ptr' keyword is supported in the
aggregate_primitives map for the task: it defines the base pointer in the
packet where to read the primitive value; intuitively, this is to be used
in conjunction with 'len'. The supported syntax is: <layer>:[<protocol
value>]+[<offset>]. 'layer' keys are: 'packet', 'mac', 'vlan', 'mpls',
'l3', 'l4', 'payload'. Examples are provided in 'examples/primitives.lst'.
+ nfacctd: introduced pro rating algorithm if sql_history is enabled and
nfacctd_time_new is disabled. Although ideal, the feature is disabled
by default for now and can be enabled by setting nfacctd_pro_rating to
true. Given a NetFlow/IPFIX flow duration greater than time-bins size as
configured by sql_history, bytes/packets counters are proportionally
distributed across all time-bins spanned by the flow. Many thanks to
Stefano Birmani for his support.
+ Introducing index_maps: enables indexing of maps to increase lookup speeds
on large maps and/or sustained lookup rates. Indexes are automatically
defined basing on structure and content of the map, up to a maximum of 8.
Indexing of pre_tag_map, bgp_peer_src_as_map, flows_to_rd_map is supported.
+ BGP daemon: introduced bgp_daemon_interval and bgp_daemon_batch config
directives: to prevent massive syncronization of BGP peers to contend
resources, BGP sessions are accepted in batches: these define the time
interval between any two batches and the amount of BGP peers in each batch
respectively.
+ Introducing historical accounting offset (ie. sql_history_offset) to set
an offset to timeslots basetime. If history is set to 30 mins (by default
creating 10:00, 10:30, 11:00, etc. time-bins), with an offset of, say,
900 seconds (so 15 mins) it will create 10:15, 10:45, 11:15, etc. time-
bins.
+ print, MongoDB, SQL plugins: improved placement of tuples in the correct
table when historical accounting (ie. sql_history) and dynamic table
names (ie. sql_table) features are both in use.
+ print, MongoDB, SQL plugins: dynamic file names (print plugin) and
tables (MongoDB and SQL plugins) can now include $peer_src_ip, $tag and
$tag2 variables: value is populated using the processed record value for
peer_src_ip, tag, tag2 primitives respectively.
+ print plugin: introduced print_latest_file to point latest filename for
print_output_file time-series. Until 1.5.0rc1 selection was automagic.
But having introduced variable spool directory structures and primitives-
related variables the existing basic scheme of producing pointers had to
be phased-out.
+ IMT plugin: added EOF in the client-server communication so to detect
uncompleted messages and print an error message. Thanks to Adam Jacob
Muller for his proposal.
+ Introduced [nf|sf|pm]acctd_pipe size and bgp_daemon_pipe_size config
directives to define the size of the kernel socket used read traffic data
and for BGP messaging respectively.
+ pmacctd, uacctd: mpls_top_label, mpls_bottom_label and mpls_stack_depth
primitives have been implemented.
+ pmacctd, uacctd: GTP tunnel handler now supports inspection of GTPv1.
+ pre_tag_map: results of evaluation of pre_tag_map, in case of a positive
match, overrides any tags passed by nfprobe/sfprobe plugins via NetFlow/
sFlow export.
+ pre_tag_map: stack keyword now supports logical or operator (A | B) in
addition to sum (A + B).
+ pre_tag_map: introduced 'mpls_pw_id' keyword to match the signalled MPLS
L2 VPNs Pseudowire ID. In NetFlow v9/IPFIX this is compared against IE
#249; in sFlow v5 this is compared against vll_vc_id field, extended MPLS
VC object.
+ Introduced log notifications facility: allows to note down specific log
notifications have been sent so to prevent excessive repetitive output.
! fix, plugin_hooks.c: plugin_buffer_size variables are bumped to u_int64_t
! fix, plugin_hooks.c: improved protection of internal pmacct buffering
(plugin_buffer_size, plugin_pipe_size) from inconsistencies: buffer is now
also invalidated by the core process upon first writing into it. Thanks to
Chris Wilson for his support.
! fix, plugin_hooks.c: a simple default value for plugin_pipe_size and
plugin_buffer_size is now picked if none is supplied. This is to get
around tricky estimates. 1.5.0rc1 release affected.
! fix, ll.c: ntohl() done against a char pointer instead of u_int32_t one
in MPLS handler was causing incorrect parsing of labels. Thanks to Marco
Marzetti for his support.
! fix, net_aggr.c: IPv6 networks debug messages now report correctly net
and mask information. Also IPv6 prefix to peer source/destination ASN was
crashing due to an incorrect pointer. Finally applying masks to IPv6
addresses was not done correctly. Thanks to Brent Van Dussen for
reporting the issue.
! fix, classifiers: slightly optimized search_class_id_status_table() and
added warning message if the amount of classifiers exceeds configured
number of classifier_table_num (by default 256).
! fix, pre_tag_map: if a JEQ can be resolved into multiple labels, stop to
the first occurrence.
! fix, nfacctd, sfacctd: IPv6 was not being correctly reported due to a
re-definition of NF9_FTYPE_IPV6. 1.5.0rc1 release affected. Thanks to
Andrew Boey for reporting the issue.
! fix, nfacctd: when historical accounting is enabled, ie. sql_history, not
assume anymore start and end timestamps to be of the same kind (ie. field
type #150/#151, #152/#153, etc.).
! fix, BGP daemon: default BGP RouterID used if supplied bgp_daemon_ip is
"0.0.0.0" or "::"
! fix, BGP daemon: the socket opened to accept BGP peerings is restricted
to che core process (ie. closed upon instantiating the plugins). Thanks
to Olivier Benghozi for reporting the issue.
! fix, BGP daemon: memory leak detected accepting vpnv4 and vpnv6 routes.
Thanks to Olivier Benghozi for his support solving the issue.
! fix, BGP daemon: compiling the package without IPv6 support and sending
ipv6 AF was resulting in a buffer overrun. Thanks to Joel Krauska for his
support resolving the issue.
! fix, IMT plugin: when gracefully exiting, ie. via a SIGINT signal, delete
the pipe file in place for communicating with the pmacct IMT client tool.
! fix, print, MongoDB, AMQP plugins: saved_basetime variable initialized
to basetime value. This prevents P_eval_historical_acct() to consume much
resources during the first time-bin, if historical accounting is enabled
(ie. print_history). 1.5.0rc1 release affected.
! fix, print, MongoDB and SQL plugins: purge function is escaped if there
are no elements on the queue to process.
! fix, AMQP plugin: removed amqp_set_socket() call so to be able to compile
against rabbitmq-c >= 0.4.1
! fix, MongoDB plugin: change of API between C driver version 0.8 and 0.7
affected mongo_create_index(). MongoDB C driver version test introduced.
Thanks to Maarten Bollen for reporting the issue.
! fix, print plugin: SEGV was received if no print_output_file is specified
ie. print to standard output.
! fix, MongoDB: optimized usage of BSON objects array structure.
! fix, MongoDB plugin: brought a few numerical fields, ie. VLAN IDs, CoS,
ToS, etc. to integer representation, ie. bson_append_int(), from string
one, ie. bson_append_string(). Thanks to Job Snijders for his support.
! fix, MySQL plugin: improved catching condition of sql_multi_value set too
little value. Thanks to Chris Wilson for reporting the issue.
! fix, nfprobe plugin: catch ENETUNREACH errors instead of bailing out.
Patch is courtesy by Mike Jager.
1.5.0rc1 -- 29-08-2013
+ Introducing custom-defined aggregation primitives: primitives are defined
via a file pointed by aggregate_primitives config directive. The feature
applies to NetFlow v9/IPFIX fields only, and with a pre-defined length.
Semantics supported are: 'u_int' (unsigned integer, presented as decimal
number), 'hex' (unsigned integer, presented as hexa- decimal number), 'ip'
(IP address), 'mac' (MAC address)and 'str' (string). Syntax along with
examples are available in the 'examples/primitives.lst' file.
+ Introducing JSON output in addition to tabular and CSV formats. Suitable
for injection in 3rd party tools, JSON has the advantage of being a self-
consisting format (ie. compared to CSV does not require a table title).
Library leveraged is Jansson, available at: http://www.digip.org/jansson/
+ Introducing RabbitMQ/AMQP pmacct plugin to publish network traffic data
to message exchanges. Unicast, broadcast, load-balancing scenarios being
supported. amqp_routing_key supports dynamic elements, like the value of
peer_src_ip and tag primitives or configured post_tag value, enabling
selective delivery of data to consumers. Messages are encoded in JSON
format.
+ pre_tag_map (and other maps): 'ip' key, which is compared against the IP
address originating NetFlow/IPFIX or the AgentId field in sFlow, can now
be an IP prefix, ie. XXX.XXX.XXX.XXX/NN, so to apply tag statements to
set of exporters or 0.0.0.0/0 to apply to any exporter. Many thanks to
Stefano Birmani for his support.
+ Re-introducing support for Cisco ASA NSEL export. Previously it was just
a hack. Now most of the proper work done for Cisco NEL is being reused:
post_nat_src_host (field type #40001), post_nat_dst_host (field type
#40002), post_nat_src_port (field type #40003), post_nat_dst_port (field
type #40004), fw_event (variant of nat_event, field type #40005) and
timestamp_start (observation time in msecs, field type #323).
+ Introducing MPLS-related aggregation primitives decoded from NetFlow v9/
IPFIX, mpls_label_top mpls_label_bottom and mpls_stack_depth, so to give
visibility in export scenarios on egress towards core, MPLS interfaces.
+ mpls_vpn_rd: primitive value can now be sourced from NetFlow v9/IPFIX
field types #234 (ingressVRFID) and #235 (egressVRFID). This is in
addition to existing method to source value from a flow_to_rd_map file.
+ networks_file: AS field can now be defined as "<peer_as>_<origin_as>",
Useful also to define (or override) elments of an internal port-to-port
traffic matrix.
+ print plugin: creation of intermediate directory levels is now supported;
directories can contain dynamic time-based elements hence the amount of
variables in a given pathname was also lifted to 32 from 8.
+ print plugin: introduced print_history configuration directive, which
supports same syntax as, for example, sql_history. When enabled, time-
related variables substitution of dynamic print_output_file names are
determined using this value instead of print_refresh_time one.
+ Introducing IP prefix labels, ie. for custom grouping of own IP address
space. The feature can be enabled by a --enable-plabel when configuring
the package for compiling. Labels can be defined via a networks_file.
+ mongo_user and mongo_passwd configuration directive have been added in
order to support authentication with MongoDB. If both are omitted, for
backward compatibility, authentication is disabled; if only one of the
two is specified instead, the other is set to its default value.
+ Introducing mongo_indexes_file config directive to define indexes in
collections with dynamic name. If the collection does not exist yet, it
is created. Index names are picked by MongoDB.
+ print plugin: introduced print_output_file_append config directive: if
set to true allows the plugin to append to an output file rather than
overwrite.
+ bgp_agent_map: added bgp_port key to lookup a NetFlow agent also against
a BGP session port (in addition to BGP session IP address/router ID): it
aims to support scenarios where BGP sessions do NAT traverals.
+ peer_dst_ip (BGP next-hop) can now be inferred by MPLS_TOP_LABEL_ADDR
(NetFlow v9/IPFIX field type #47). This field might replace BGP next-hop
when NetFlow is exported egress on MPLS-enabled core interfaces.
+ Introducing [nf|pm|sf|u]acctd_proc_name config directives to define the
name of the core process (by default always set to 'default'). This is
the equivalent to instantiate named plugins but for the core process.
Thanks to Brian Rak for bringing this up.
+ pre_tag_map: introduced key 'flowset_id' to tag NetFlow v9/IFPIX data
records basing on their flowset ID value, part of the flowset header.
+ pmacct client: introduced '-V' command-line option to verify version,
build info and compile options passed to the configure script; also a
new -a option now allows to retrieve supported aggregation primitives
and their description.
+ Check for mallopt() has been added at configure time. mallopt() calls
are introduced in order to disable glibc malloc() boundary checks.
! flow_to_rd_map replaces iface_to_rd_map, increasing its scope: it is
now possible to map <MPLS bottom label, BGP next-hop> couples to BGP/
MPLS VPN Route Distinguishers (RD). This is in addition to existing
mapping method basing on <ingress router, input interface>.
! fix, nfacctd, sfacctd: Setsocksize() call effectiveness is now verified
via a subsequent getsockopt(). If result is different than expected, an
informational log message is issued.
! fix, building system: removed stale check for FreeBSD4 and introduced
check for BSD systems. If on a BSD system, -DBSD is now passed over to
the compiler.
! fix, tee plugin: transparent mode now works on FreeBSD systems. Patch
is courtesy by Nikita V. Shirokov.
! fix, peer_dst_ip: uninitialized pointer variable was causing unexpected
behaviours. Thanks to Maarten Bollen for his support resolving this.
! fix, IMT plugin: selective queries with -M and -N switches verified not
working properly. Thanks to Acipia organization for providing a patch.
! fix, sql_common.c: src_port and dst_port primitives correctly spelled if
used in conjunction with BGP primitives. Thanks to Brent Van Dussen and
Elisa Jasinska for flagging the issue.
! fix, building system: added library checks in /usr/lib64 for OS's where
it is not linked to /lib where required.
! fix, print, MongoDB and AMQP plugins: P_test_zero_elem() obsoleted.
Instead, the cache structure 'valid' field is used to commit entries to
the backend.
! fix, nfacctd: in NetFlow v9/IPFIX, if no time reference is specified as
part of records, fall back to time reference in datagram header.
! fix, MongoDB plugin: mongo_insert_batch() now bails out with MONGO_FAIL
if something went wrong while processing elements in the batch and an
error message is issued. Typical reason for such condition is batch is
too big for the resources, mainly memory, available. Thanks very much to
Maarten Bollen for his support.
! fix, cfg_handlers.c: all functions parsing configuration directives, and
expecting string arguments, are now calling lower_string() so to act as
case insensitive.
! fix, IPv6 & NetFlow exporter IP address: upon enabling IPv6, NetFlow
exporter IP addresses were written as IPv4-mapped IPv6 address. This was
causing confusion when composing maps since the 'ip' field would change
depending on whether IPv6 was enabled or not. This is now fixed and IPv4-
mapped IPv6 addresses are now internally translated to plain IPv4 ones.
! fix, nfacctd: NetFlow v9/IPFIX source/destination peer ASN information
elements have been found mixed up and are now in proper order.
0.14.3 -- 03-05-2013
+ tee plugin: a new tee_receivers configuration directive allows multiple
receivers to be defined. Receivers can be optionally grouped, for example
for load-balancing (rr, hash) purposes, and attached a list of filters
(via tagging). The list is fully reloadable at runtime.
+ A new pkt_len_distrib aggregation primitive is introduced: it works by
defining length distribution bins, ie. "0-999,1000-1499,1500-9000" via
the new pkt_len_distrib_bins configuration directive. Maximum amount
of bins that can be defined is 255; lengths must be within the range
0-9000.
+ Introduced NAT primitives to support Cisco NetFlow Event Logging (NEL),
for Carrier Grade NAT (CGNAT) scenarios: nat_event, post_nat_src_host,
post_nat_dst_host, post_nat_src_port and post_nat_dst_port. Thanks to
Simon Lockhart for his input and support developing the feature.
+ Introduced timestamp primitives (to msec resolution) to support generic
logging functions: timestamp_start, timestamp_end (timestamp_end being
currently applicable only to traffic flows). These primitives must not
be confused with existing sql_history timestamps which are meant for the
opposite function instead, temporal aggregation.
+ networks_file: introduced support for (BGP) next-hop (peer_dst_ip) in
addition to existing fields. Improved debug output. Also introduced a
new networks_file_filter feature to make networks_file work as a filter
in addition to its resolver functionality: if set to true net and host
values not belonging to defined networks are zeroed out. See UPGRADE
document for backward compatibility.
+ BGP daemon: added support for IPv6 NLRI and IPv6 BGP next-hop elements
for rfc4364 BGP/MPLS Virtual Private Networks.
+ MongoDB plugin: introduced mongo_insert_batch directive to define the
amount of elements to be inserted per batch - allowing the plugin to
scale better. Thanks for the strong support to Michiel Muhlenbaumer and
Job Snijders.
+ pre_tag_map: 'set_qos' feature introduced: matching network traffic is
set 'tos' primitive to the specified value. This is useful if collecting
ingress NetFlow/IPFIX at both trusted and untrusted borders, allowing to
selectively override ToS values at untrusted ones. For consistency,
pre_tag_map keys id and id2 have been renamed to set_tag and set_tag2;
legacy jargon is still supported for backward compatibility.
+ sfacctd: improved support for L2 accounting, ethernet length is being
committed as packet length; this information gets replaced by any length
information will come from upper layers, if any is reported. Thanks to
Daniel Swarbrick for his support.
+ nfacctd: introduced nfacctd_peer_as directive to value peer_src_as and
peer_dst_as primitives from NetFlow/IPFIX export src_as and dst_as
values respectively (ie. as a result of a "ip flow-export .. peer-as"
config on the exporter). The directive can be plugin-specific.
+ print, memory plugins: print_output_separator allows to select separator
for CSV outputs. Default comma separator is generally fine except for
BGP AS-SET representation.
! Building sub-system: two popular configure switches, --enable-threads
and --enable-64bit, are now set to true by default.
! fix, print & mongodb plugins: added missing cases for src_net and dst_net
primitives. Thanks to John Hess for his support.
! fix, SQL plugins: improved handling of fork() calls when return value
is -1 (fork failed). Many thanks to Stefano Birmani for his valuable
support troubleshooting the issue.
! fix, ISIS daemon: linked list functions got isis_ prefix in order to
prevent namespace clashes with other libraries (ie. MySQL) we link
against. Thanks to Stefano Birmani for reporting the issue.
! fix, tee plugin: can't bridge AFs when in transparent mode is not fatal
error condition anymore to tackle transient interface conditions. Error
message is throttled to once per 60 secs. Thanks to Evgeniy Kozhuhovskiy
for his support troubleshooting the issue.
! fix, nfacctd: extra length checks introduced when parsing NetFlow v9/
IPFIX options and data template flowsets. Occasional daemon crashes were
verified upon receipt of malformed/incomplete template data.
! fix: plugins now bail out with an error message if core process is found
dead via a getppid() check.
- nfacctd_sql_log feature removed. The same can now be achieved with the
use of proper timestamp primitives (see above).
0.14.2 -- 14-01-2013
+ pmacct opens to MongoDB, a leading noSQL document-oriented database
via a new 'mongodb' plugin. Feature parity is maintained with all
existing plugins. The QUICKSTART doc includes a brief section on how
to getting started with it. Using MongoDB >= 2.2.0 is recommended;
MongoDB C driver is required.
+ GeoIP lookups support has been introduced: geoip_ipv4 and geoip_ipv6
config directives now allow to load Maxmind IPv4/IPv6 GeoIP database
files; two new traffic aggregation primitives are added to support the
feature: src_host_country and dst_host_country. Feature implemented
against all deamons and all plugins and supports both IPv4 and IPv6.
Thanks to Vincent Bernat for his patches and precious support.
+ networks_file: user-supplied files to define IP networks and their
associations to ASNs (optional) has been hooked up to the 'fallback'
(longest match wins) setting of [pm|u|sf|nf]acctd_net, [pm|u]acctd_as
and [sf|nf]acctd_as_new. Thanks to John Hess for his support.
+ A new sampling_rate traffic aggregation primitive has been introduced:
to report on the sampling rate to be applied to renormalize counters
(ie. useful to support troubleshooting of untrusted node exports and
hybrid scenarios where a partial sampling_map is supplied). If renorm
of counters is enabled (ie. [n|s]facctd_renormalize set to true) then
sampling_rate will show as 1 (ie. already renormalized).
+ sql_table, print_output_file, mongo_table: dynamic table names are
now enriched by a $ref variable, populated with the configured value
for refresh time, and a $hst variable, populated with the configured
value for sql_history (in secs).
+ Solved the limit of 64 traffic aggregation primitives: the original
64 bits bitmap is now split in a 16 bits index + 48 bits registry
with multiple entries (currently 2). cfg_set_aggregate() and, in
future, cfg_get_aggregate() functions are meant to safely manipulate
the new bitmap structure and detect mistakes in primitives definition.
! fix, print plugin: removed print_output_file limitation to 64 chars.
Now maximum filename length is imposed by underlying OS.
! fix, print plugin: primitives are selectively enabled for printing
based on 'aggregate' directive.
! fix, print plugin: pointer to latest file been generated is updated
at very last in the workflow.
! fix, ip_flow.c: incorrect initialization for IPv6 flow buffer. Thanks
to Mike Jager for reporting the issue and providing a patch.
! fix, pre_tag_map: improved matching of pre_tag_map primitives against
IPFIX fields. Thanks to Nikita V Shirokov for reporting the issue.
! fix, nfprobe plugin: improved handling of unsuccessful send() calls
in order to prevent file descriptors depletion and log failure cause.
Patch is courtesy by Mike Jager.
! fix, nfacctd: gracefully handling the case of NetFlow v9/IPFIX flowset
length of zero; unproper handling of the condition was causing nfacctd
to infinite loop over the packet; patch is courtesy by Mike Jager.
! fix, Setsocksize(): setsockopt() replaces Setsocksize() in certain
cases and Setsocksize() fix to len parameter. Patch is courtesy by
Vincent Bernat
0.14.1 -- 03-08-2012
+ nfacctd: introduced support for IPFIX variable-length IEs (RFC5101),
improved support for IPFIX PEN IEs.
+ nfacctd, sfacctd: positive/negative caching for bgp_agent_map and
sampling_map is being introduced. Cache entries are invalidated upon
reload of the maps.
+ bgp_agent_map: resolution of IPv4 NetFlow agents to BGP speakers
with IPv6 sessions is now possible. This is to support dual-stack
network deployments. Also the keyword 'filter' is introduced and
supported values are only 'ip' and 'ip6'.
+ nfacctd: etype primitive can be populated from IP_PROTOCOL_VERSION,
ie. Field Type #60, in addition to ETHERTYPE, ie. Field Type #256.
Should both be present the latter has priority over the former.
+ print plugin: introduced a pointer to the latest filename in the set,
ie. in cases when variable filenames are specified. The pointer comes
in the shape of a symlink called "<plugin name>-latest".
! fix, pretag_handlers.c: BGP next-hop handlers are now hooked to the
longest-match mechanism for destination IP prefix.
! fix, net_aggr.c: defining a networks_file configuration directive in
conjunction with --enable-ipv6 was causing a SEGVs. This is now solved.
! fix, uacctd: cache routine is now being called in order to resolve
in/out interface ifindexes. Patch is courtesy by Stig Thormodsrud.
! fix, BGP daemon: bgp_neighbors_file now lists also IPv6 BGP peerings.
! fix, sql_common.c: SQL writers due to safe action are now logged with
a warning message rather than debug.
! fix, PostgreSQL table schemas: under certain conditions, default
definition of stamp_inserted was generating a 'date/time field value
out of range: "0000-01-01 00:00:00"' error. Many thanks to Marcello
di Leonardo for reporting the issue and providing a fix.
! fix, IS-IS daemon: sockunion_print() function was found not portable
and has been removed.
! fix, BGP daemon: memcpy() replaced by ip6_addr_cpy() upon writing to
sockaddr_in6 structures.
! fix, EXAMPLES document has been renamed QUICKSTART for disambiguation
on filesystems where case-sensitive names are not supported.
! Several code cleanups. Patches are courtesy by Osama Abu Elsorour
and Ryan Steinmetz.
0.14.0 -- 11-04-2012
+ pmacct now integrates an IS-IS daemon within collectors; the daemon
is being run as a parallel thread within the collector core process;
a single L2 P2P neighborship, ie. over a GRE tunnel, is supported;
it implements P2P Hello, CSNP and PSNP - and does not send any LSP
information out. The daemon is currently used for route resolution.
It is well suited to several case-studies, popular one being: more
specific internal routes are carried within the IGP while they are
summarized in BGP crossing cluster boundaries.
+ A new aggregation primitive 'etype' has been introduced in order to
support accounting against the EtherType field of Ethernet frames.
The implementation is consistent across all data collection methods
and backends.
+ sfacctd: introduced support for samples generated on ACL matches in
Brocade (sFlow sample type: Enterprise: #1991, Format: #1). Thanks
to Elisa Jasinska and Brent Van Dussen for their support.
+ sfacctd, pre_tag_map: introduced sample_type key. In sFlow v2/v4/v5
this is compared against the sample type field. Value is expected
in <Enterprise>:<Format> notation.
! fix, signals.c: ignoring SIGINT and SIGTERM in my_sigint_handler()
to prevent multiple calls to fill_pipe_buffer(), condition that can
cause pipe buffer overruns. Patch is courtesy by Osama Abu Elsorour.
! fix, pmacctd: tunnel registry now correctly supports multiple tunnel
definitions for the same stack level.
! fix, print plugin: cos field now correctly shows up in the format
title while CSV format is selected and L2 primitives are enabled.
! fix, util.c: a feof() check has been added to the fread() call in
read_SQLquery_from_file(); thanks to Elisa Jasinska and Brent Van
Dussen for their support.
! fix, nfprobe: NetFlow output socket is now re-opened after failing
send() calls. Thanks to Maurizio Molina for reporting the problem.
! fix, sfacctd: length checks have been imporved while extracting
string tokens (ie. AS-PATH and BGP communities) from sFlow Extended
Gateway object. Thanks to Duncan Small for his support.
0.14.0rc3 -- 07-12-2011
+ BGP daemon: BGP/MPLS VPNs (rfc4364) implemented! This encompasses both
RIB storage (ie. virtualization layer) and lookup. bgp_iface_to_rd_map
map correlates <router IP, input/output interfaces (ifIndex)> couples
to Route Distinguishers (RDs). RD encapsulation types #0 (2-bytes ASN),
#1 (IP address) and #2 (4-bytes ASN) are supported. Examples provided:
examples/bgp_iface_to_rd.map and EXAMPLES files.
+ mpls_vpn_rd aggregation primitive has been added to the set. Also this
is being supported key in Pre-Tagging (pre_tag_map).
+ print plugin: introduced print_output_file feature to write statistics
to files. Output is text, formatted or CSV. Filenames can contain time-
based variables to make them dynamic. If filename is static instead,
content is overwritten over time.
+ print plugin: introduced print_time_roundoff feature to align time slots
nicely, same as per the sql_history_roundoff directive.
+ print plugin: introduced print_trigger_exec feature to execute custom
scripts at each print_refresh_time interval (ie. to process, expire,
gzip, etc. files). Feature is in sync with wrap-up of data commit to
screen or files.
+ pmacctd: introduced support for DLT_LOOP link-type (ie. OpenBSD tunnel
interfaces). Thanks to Neil Reilly for his support.
+ uacctd: a cache of ifIndex is introduced. Hash structure with conflict
chains and short expiration time (ie. to avoid getting tricked by cooked
interfaces devices a-la ppp0). The cache is an effort to gain speed-ups.
Implementation is courtesy by Stephen Hemminger, Vyatta.
+ Logging: introduced syslog-like timestamping when writing directly to
files. Also a separate FD per process is used and SIGHUP elicits files
reopening: all aimed at letting proper logs rotation by external tools.
+ Introduced plugin_pipe_backlog configuration directive: it induces a
backlog of buffers on the pipe before actually releasing them to the
plugin. The strategy helps optimizing inter-process communications, ie.
when plugins are quicker processing data than the Core process.
! fix, peer_src_ip primitive: has been disconnected from [ns]facctd_as_new
mechanism in order to ensure it's always representing a reference to the
NetFlow or sFlow emitter.
! fix, nfprobe: input and output VLAN ID field types have been aligned to
RFC3954, which appears to be also retroactively supported by IPFIX. The
new field types are #58 and #59 respectively. Thanks to Maurizio Molina
for pointing the issue out.
! fix, IMT plugin: fragmentation of the class table over multiple packets
to the pmacct IMT client was failing and has been resolved.
! fix, nfprobe: individual flows start and end timestamps are now filled
to the msec resolution. Thanks to Daniel Aschwanden for having reported
the issue.
! fix, uacctd: NETLINK_NO_ENOBUFS is set to prevent the daemon being
reported about ENOBUFS events by the underlying operating system. Works
on kernels 2.6.30+. Patch is courtesy by Stephen Hemminger, Vyatta.
! fix, uacctd: get_ifindex() can now return values greater than 2^15. Patch
is courtesy by Stephen Hemminger, Vyatta.
! fix, pmacctd, uacctd: case of zero IPv6 payload in conjunction with no
IPv6 next header is now supported. Thanks to Quirin Scheitle for having
reported the issue.
- Support for is_symmetric aggregation primitive is discontinued.
0.14.0rc2 -- 26-08-2011
+ sampling_map feature is introduced, allowing definition of static traffic
sampling mappings. Content of the map is reloadable at runtime. If a
specific router is not defined in the map, the sampling rate advertised
by the router itself, if any, is applied.
+ nfacctd: introduced support for 16 bits SAMPLER_IDs in NetFlow v9/IPFIX;
this appears to be the standard length with IOS-XR.
+ nfacctd: introduced support for (FLOW)_SAMPLING_INTERVAL fields as part
of the NetFlow v9/IPFIX data record. This case is not prevented by the
RFC although such information is typically exported as part of options.
It appears some probes, ie. FlowMon by Invea-Tech, are getting down this
way.
+ nfacctd, sfacctd: nfacctd_as_new and sfacctd_as_new got a new 'fallback'
option; when specified, lookup of BGP-related primitives is done against
BGP first and, if not successful, against the export protocol.
+ nfacctd, sfacctd: nfacctd_net and sfacctd_net got a new 'fallback' option
that when specified looks up network-related primitives (prefixes, masks)
against BGP first and, if not successful, against the export protocol. It
gets useful for resolving prefixes advertised only in the IGP.
+ sql_num_hosts feature is being introduced: defines, in MySQL and SQLite
plugins, whether IP addresses should be left numerical (in network bytes
ordering) or converted into strings. For backward compatibility, default
is to convert them into strings.
+ print_num_protos and sql_num_protos configuration directives have been
introduced to allow to handle IP protocols (ie. tcp, udp) in numerical
format. The default, backward compatible, is to look protocol names up.
The feature is built against all plugins and can also be activated via
the '-u' commandline switch.
! fix, nfacctd: NetFlow v9/IPFIX sampling option parsing now doesn't rely
anymore solely on finding a SamplerID field; as an alternative, presence
of a sampling interval field is also checked. Also a workaround is being
introduced for sampled NetFlow v9 & C7600: if samplerID within a data
record is defined and set to zero and no match was possible, then the
last samplerID defined is returned.
! nfacctd: (FLOW)_SAMPLING_INTERVAL fields as part of the NetFlow v9/IPFIX
data record are now supported also 16-bits long (in addition to 32-bits).
! fix, SQL plugins: sql_create_table() timestamp has been aligned with SQL
queries (insert, update, lock); furthermore sql_create_table() is invoked
every sql_refresh_time instead of every sql_history. Docs updated. Thanks
to Luis Galan for having reported the issue.
! fix, pmacct client: error code when connection is refused on UNIX socket
was 0; it has been changed to 1 to reflect the error condition. Thanks
to Mateusz Viste for reporting the issue.
! fix, building system: CFLAGS were not always honoured. Patch is courtesy
of Etienne Champetier
! fix, ll.c: empty return value was causing compiler with certain flags to
complain about the issue. Patch is courtesy of Ryan Steinmetz.
0.14.0rc1 -- 31-03-2011
+ IPFIX (IETF IP Flow Information Export protocol) replication and
collector capabilities have been introduced as part of nfacctd, the
NetFlow accounting daemon of the pmacct package.
+ nfprobe plugin: initial IPFIX export implementation. This is called
via a 'nfprobe_version: 10' configuration directive. pmacctd, the
promiscuous mode accounting daemon, and uacctd, the ULOG accounting
daemon, both part of the pmacct package are now supported.
+ Oracle's BrekeleyDB 11gR2 offers a perfect combination of technologies
by including an SQL API that is fully compatible with SQLite. As a
result pmacct now opens to BerkeleyDB 5.x via its SQLite3 plugin.
+ sfacctd: BGP-related traffic primitives (AS Path, local preference,
communities, etc.) are now read from sFlow Extended Gateway object if
sfacctd_as_new is set to false (default).
+ nfacctd, sfacctd: source and destination peer ASNs are now read from
NetFlow or sFlow data if [ns]facctd_as_new is set to false (default).
+ nfacctd: introduced support for NetFlow v9/IPFIX source and destination
peer ASN field types 128 and 129. The support is enabled at runtime by
setting to 'false' (default) the 'nfacctd_as_new' directive.
+ sfacctd: f_agent now points sFlow Agent ID instead of source IP address;
among the other things, this allows to compare BGP source IP address/BGP
Router-ID against the sFlow Agent ID.
+ PostgreSQL plugin: 'sql_delimiter' config directive being introduced:
if sql_use_copy is true, uses the supplied character as delimiter.Useful
in cases where the default delimiter is part of any of the supplied
strings.
+ pmacct client: introduced support for Comma-Separated Values (CSV) output
in addition to formatted-text. A -O commandline switch allows to enable
the feature.
! fix, MySQL/PostgreSQL/SQLite3 plugins: insert of data into the database
can get arbitrarily delayed under low traffic conditions. Many Thanks
to Elisa Jasinska and Brent Van Dussen for their great support in solving
the issue.
! fix, BGP daemon: multiple BGP capabilities per capability announcement
were not supported - breaking compliancy with RFC5492. The issue was
only verified against a OpenBGPd speaker. Patch is courtesy of Manuel
Guesdon.
! fix, initial effort made to document uacctd, the ULOG accounting daemon
0.12.5 -- 28-12-2010
+ nfacctd: introduced support for NAT L3/L4 field values via xlate_src
and xlate_dst configuration directives. Implementation follows IPFIX
standard for IPv4 and IPv6 (field types 225, 226, 227, 228, 281 and
282).
+ nfacctd: Cisco ASA NetFlow v9 NSEL field types 40001, 40002, 40003,
40004 and IPFIX/Cisco ASA NetFlow v9 NSEL msecs absolute timestamps
field types 152, 153 and 323 have been added.
+ nfacctd: introduced support for 'new' TCP/UDP source/destination ports
(field types 180, 181, 182, 183), as per IPFIX standard, basing on the
L4 protocol value (if any is specified as part of the export; otherwise
assume L4 is not TCP/UDP).
+ nfacctd, nfprobe: introduced support for application classification
via NetFlow v9 field type #95 (application ID) and application name
table option. This feature aligns with Cisco NBAR-NetFlow v9
integration feature.
+ nfacctd: introduced support for egress bytes and packet counters (field
types 23, 24) basing on the direction value (if any is specified as
part of the export; otherwise assume ingress as per RFC3954).
+ nfprobe: egress IPv4/IPv6 NetFlow v9 templates have been introduced;
compatibility with Cisco (no use of OUT_BYTES, OUT_OUT_PACKETS) taken
into account.
+ nfacctd: added support for egress datalink NetFlow v9 fields basing
on direction field.
+ nfacctd, sfacctd: aggregate_filter can now filter against TCP flags;
also, [ns]facctd_net directive can now be specified per-plugin.
+ BGP daemon: introduced support for IPv6 transport of BGP messaging.
+ BGP daemon: BGP peer information is now linked into the status table
for caching purposes. This optimization results in good CPU savings
in bigger deployments.
! fix, nfacctd, sfacctd: daemons were crashing on OpenBSD platform upon
setting an aggregate_filter configuration directive. Patch is courtesy
of Manuel Pata.
! fix, xflow_status.c: status entries were not properly linked to the
hash conflict chain resulting in a memory leak. However the maximum
number of table entries set by default was preventing the structure
to grow undefinitely.
! fix, sql_common.c: increased buffer size available for sql_table_schema
from 1KB to 8KB. Thanks to Michiel Muhlenbaumer his support.
! fix, bgp_agent_map has been improved to allow mapping of NetFlow/sFlow
agents making use of IPv6 transport to either a) IPv4 transport address
of BGP sessions or b) 32-bit BGP Router IDs. Mapping to IPv6 addresses
is however not (yet) possible.
! fix, nfprobe: encoding of NetFlow v9 option scope has been improved;
nfprobe source IPv4/IPv6 address, if specified via nfprobe_source_ip
directive, is now being written.
! fix, util.c: string copies in trim_spaces(), trim_all_spaces() and
strip_quotes() have been rewritten more safely. Patch is courtesy of
Dmitry Koplovich.
! fix, sfacctd: interface format is now merged back into interface value
fields so to ease keeping track of discards (and discard reasons) and
multicast fanout.
! fix, MySQL, SQLite3 plugins: sql table version 8 issued to provide
common naming convention when mapping primitives to database fields
among the supported RDBMS base. Thanks to Chris Wilson for his support.
! fix, pmacct client: numeric variables output converted to unsigned
from signed.
! fix, nfacctd_net, sfacctd_net: default value changed from null (and
related error message) to 'netflow' for nfacctd_net and 'sflow' for
sfacctd_net.
! fix, nfacctd, sfacctd: aggregate_filter was not catching L2 primitives
(VLAN, MAC addresses) when performing egress measurements.
0.12.4 -- 01-10-2010
+ BGP daemon: a new memory model is introduced by which IP prefixes
are being shared among the BGP peers RIBs - leading to consistent
memory savings whenever multiple BGP peers export full tables due
to the almost total overlap of information. Longest match nature
of IP lookups required to raise BGP peer awareness of the lookup
algorithm. Updated INTERNALS document to support estimation of the
memory footprint of the daemon.
+ BGP daemon: a new bgp_table_peer_buckets configuration directive
is introduced: per-peer routing information is attached to IP
prefixes and now hashed onto buckets with conflict chains. This
parameter sets the number of buckets of such hash structure; the
value is directly related to the number of expected BGP peers,
should never exceed such amount and is best set to 1/10 of the
expected number of peers.
+ nfprobe: support has been added to export direction field (NetFlow
v9 field type #61); its value, 0=ingress 1=egress, is determined
via nfprobe_direction configuration directive.
+ nfacctd: introduced support for Cisco ASA bytes counter, NetFlow v9
field type #85. Thanks to Ralf Reinartz for his support.
+ nfacctd: improved flow recognition heuristics for cases in which
IPv4/IPv6/input/output data are combined within the same NetFlow
v9 template. Thanks to Carsten Schoene for his support.
! fix, BGP daemon: bgp_nexthop_followup was not working correctly if
pointed to a non-existing next-hop.
! fix, nfv9_template.c: ignoring unsupported NetFlow v9 field types;
improved template logging. Thanks to Ralf Reinartz for his support.
! fix, print plugin: support for interfaces and network masks has
been added. Numeric variables output converted to unsigned from
signed.
0.12.3 -- 28-07-2010
+ 'cos' aggregation primitive has been implemented providing support
for 802.1p priority. Collection is supported via sFlow, libpcap and
ULOG; export is supported via sFlow.
+ BGP daemon: TCP MD5 signature implemented. New 'bgp_daemon_md5_file'
configuration directive is being added for the purpose of defining
peers and their respective MD5 keys, one per line, in CSV format.
The map is reloadable at runtime: existing MD5 keys are removed via
setsockopt(), new ones are installed as per the newly supplied map.
Sample map added in 'examples/bgp_md5.lst.example'.
+ BGP daemon: added support for RFC3107 (SAFI=4 label information) to
enable receipt of labeled IPv4/IPv6 unicast prefixes.
+ nfprobe, sfprobe: introduced the concept of traffic direction. As a
result, [ns]fprobe_direction and [ns]fprobe_ifindex configuration
directives have been implemented.
+ [ns]fprobe_direction defines traffic direction. It can be statically
defined via 'in' or 'out' keywords; values can also be dynamically
determined through a pre_tag_map (1=input, 2=output) by means of
'tag' and 'tag2' keywords.
+ [ns]fprobe_ifindex either statically associate an interface index
(ifIndex) to a given [ns]fprobe plugin or semi-dynamically via
lookups against a pre_tag_map by means of 'tag' and 'tag2' keywords.
+ sfprobe: sfprobe_ifspeed configuration directive is introduced and
aimed at statically associating an interface speed to an sfprobe
plugin.
+ sfprobe: Switch Extension Header support added. Enabler for this
development was support for 'cos' and in/out direction. Whereas
VLAN information was already supported as an aggregation primitive.
+ sfprobe: added support for Counter Samples for multiple interfaces.
Sampling function has been brought to the plugin so that Counter
Samples can be populated with real bytes/packets traffic levels.
! nfprobe, sfprobe: send buffer size is now aligned to plugin_pipe_size,
if specified, providing a way to tune buffers in case of sustained
exports.
! fix, addr.c: pm_ntohll() and pm_htonll() routines rewritten. These
are aimed at changing byte ordering of 64-bit variables.
! fix, BGP daemon: support for IPv6 global address/link-local address
next-hops as part of MP_REACH_NLRI parsing.
! fix, cfg_handlers.c: bgp_daemon and bgp_daemon_msglog parsing was
not correct, ie. enabled if specified as 'false'. Thanks to Brent
Van Dussen for reporting the issue.
! fix, bgp.c: found a CPU hog issue caused by missing cleanup of the
select() descriptors vector.
! fix, pmacct.c: in_iface/out_iface did erroneously fall inside a
section protected by the "--disable-l2" switch. Thanks to Brent
Van Dussen for reporting the issue.
0.12.2 -- 27-05-2010
+ A new 'tee' plugin is introduced bringing both NetFlow and sFlow
replication capabilities to pmacct. It supports transparent mode
(tee_transparent), coarse-grained filtering capabilities via the
Pre-Tagging infrastructure. Quickstart guide is included as part
of the EXAMPLES file (chapter XII).
+ nfprobe, sfprobe: introduced support for export of the BGP next-hop
information. Source data selection for BGP next-hop is being linked
to [pmacctd_as|uacctd_as] configuration directive. Hence it must be
set to 'bgp' in order for this feature to work.
+ nfprobe, sfprobe, BGP daemon: new set of features (nfprobe_ipprec,
sfprobe_ipprec, bgp_daemon_ipprec) allows to mark self-originated
sFlow, NetFlow and BGP datagrams with the supplied IP precedence
value.
+ peer_src_ip (IP address of the NetFlow emitter, agent ID of the
sFlow emitter) and peer_dst_ip (BGP next-hop) can now be filled
from NetFlow/sFlow protocols data other than BGP. To activate the
feature nfacctd_as_new/sfacctd_as_new have to be 'false' (default
value), 'true' or 'file'.
+ print plugin: introduced support for Comma-Separated Values (CSV)
output in addition to formatted-text. A new print_output feature
allows to switch between the two.
+ pmacctd: improved 802.1ad support. While recursing, outer VLAN is
always reported as value of the 'vlan' primitive.
! fix, pmacctd: 802.1p was kept integral part of the 'vlan' value.
Now a 0x0FFF mask is applied in order to return only the VLAN ID.
! fix, pkt_handlers.c: added trailing '\0' symbol when truncating
AS-PATH and BGP community strings due to length constraints.
! fix, sql_common.c: maximum SQL writers warning message was never
reached unless a recovery method is specifited. Thanks to Sergio
Charpinel Jr for reporting the issue.
! fix, MySQL and PostgreSQL plugins: PGRES_TUPLES_OK (PostgreSQL)
and errno 1050 (MySQL) are now considered valid return codes when
dynamic tables are involved (ie. sql_table_schema). Thanks to
Sergio Charpinel Jr for his support.
! fix, BGP daemon: pkt_bgp_primitives struct has been explicitely
64-bit aligned. Mis-alignment was causing crashes when buffering
was enabled (plugin_buffer_size). Verified on Solaris/sparc.
0.12.1 -- 07-04-2010
+ Input/output interfaces (SNMP indexes) have now been implemented
natively; it's therefore not required anymore to pass through the
(Pre-)tag infrastructure. As a result two aggregation primitives
are being introduced: 'in_iface' and 'out_iface'.
+ Support for source/destination IP prefix masks is introduced via
two new aggregation primitives: src_mask and dst_mask. These are
populated as defined by the [nf|sf|pm|u]acctd_net directive:
NetFlow/sFlow protocols, BGP, Network files (networks_file) or
static (networks_mask) being valid data sources.
+ A generic tunnel inspection infrastructure has been developed to
benefit both pmacctd and uacctd daemons. Handlers are defined via
configuration file. Once enabled daemons will account basing upon
tunnelled headers rather than the envelope. Currently the only
supported tunnel protocol is GTP, the GPRS tunnelling protocol
(which can be configured as: "tunnel_0: gtp, <UDP port>"). Up to
8 different tunnel stacks and up to 4 tunnel layers per stack are
supported. First matching stack, first matching layer wins.
+ uacctd: support for the MAC layer has been added for the Netlink/
ULOG Linux packet capturing framework.
+ 'nfprobe_source_ip' feature introduced: it allows to select the
IPv4/IPv6 address to be used to export NetFlow datagrams to the
collector.
+ nfprobe, sfprobe: network masks are now exported via NetFlow and
sFlow. 'pmacctd_net' and its equivalent directives define how to
populate src_mask and dst_mask values.
! cleanup, nfprobe/sfprobe: data source for 'src_as' and 'dst_as'
primitives is now expected to be always explicitely defined (in
line with how 'src_net' and 'dst_net' primitives work). See the
UPGRADE doc for the (limited) backward compatibility impact.
! Updated SQL documentation: sql/README.iface guides on 'in_iface'
and 'out_iface' primitives; sql/README.mask guides on 'src_mask'
and 'dst_mask' primitives; sql/README.is_symmetric guides on
'is_symmetric' primitive.
! fix, nfacctd.h: source and destination network masks were twisted
in the NetFlow v5 export structure definition. Affected releases
are: 0.12.0rc4 and 0.12.0.
! fix, nfprobe_plugin.c: l2_to_flowrec() was missing some variable
declaration when the package was configured for compilation with
--disable-l2. Thanks to Brent Van Dussen for reporting the issue.
! fix, bgp.c: bgp_attr_munge_as4path() return code was not defined
for some cases. This was causing some BGP messages to be marked
as malformed.
! fix, sfprobe: a dummy MAC layer was created whenever this was not
included as part of the captured packet. This behaviour has been
changed and header protocol is now set to 11 (IPv4) or 12 (IPv6)
accordingly. Thanks to Neil McKee for pointing the issue.
! workaround, building sub-system: PF_RING enabled libpcap was not
recognized due to missing of pcap_dispatch(). This is now fixed.
0.12.0 -- 16-02-2010
+ 'is_symmetric' aggregation primitive has been implemented: aimed
at easing detection of asymmetric traffic. It's based on rule
definitions supplied in a 'bgp_is_symmetric_map' map, reloadable
at runtime.
+ A new 'bgp_daemon_allow_file' configuration directive allows to
specify IP addresses that can establish a BGP session with the
collector's BGP thread. Many thanks to Erik van der Burg for
contributing the idea.
+ 'nfacctd_ext_sampling_rate' and 'sfacctd_ext_sampling_rate' are
introduced: they flag the daemon that captured traffic is being
sampled. Useful to tackle corner cases, ie. the sampling rate
reported by the NetFlow/sFlow agent is missing or incorrect.
+ The 'bgp_follow_nexthop' feature has been extended so that extra
IPv4/IPv6 prefixes can be supplied. Up to 32 IP prefixes are now
supported and a warning message is generated whenever a supplied
string fails parsing.
+ Pre-Tagging: implemented 'src_local_pref' and 'src_comms' keys.
These allow tagging based on source IP prefix local_pref (sourced
from either a map or BGP, ie. 'bgp_src_local_pref_type: map',
'bgp_src_local_pref_type: bgp') and standard BGP communities.
+ Pre-Tagging: 'src_peer_as' key was extended in order to match on
BGP-sourced data (bgp_peer_src_as_type: bgp).
+ Pre-Tagging: introduced 'comms' key to tag basing on up to 16
standard BGP communities attached to the destination IP prefix.
The lookup is done against the BGP RIB of the exporting router.
Comparisons can be done in either match-any or match-all fashion;
xidDocumentation and examples updated.
! fix, util.c: load_allow_file(), empty allow file was granting a
connection to everybody being confused with a 'no map' condition.
Now this case is properly recognized and correctly translates in
a reject all clause.
! fix, sql_common.c: log of NetFlow micro-flows to a SQL database
(nfacctd_sql_log directive) was not correctly getting committed
to the backend, when sql_history was disabled.
! fix, mysql|pgsql|sqlite_plugin.c: 'flows' aggregation primitive
was not suitable to mix-and-match with BGP related primitives
(ie. peer_dst_as, etc.) due to an incorrect check. Many thanks
to Zenon Mousmoulas for the bug report.
! fix, pretag_handlers.c: tagging against NetFlow v9 4-bytes in/out
interfaces was not working properly. Thanks to Zenon Mousmoulas
for reporting the issue.
0.12.0rc4 -- 21-12-2009
+ BGP-related source primitives are introduced, namely: src_as_path,
src_std_comm, src_ext_comm, src_local_pref and src_med. These add
to peer_src_as which was already implemented. All can be resolved
via reverse BGP lookups; peer_src_as, src_local_pref and src_med
can also be resolved via lookup maps which support checks like:
bgp_nexthop (RPF), peer_dst_as (RPF), input interface and source
MAC address. Many thanks to Zenon Mousmoulas and GRNET for their
fruitful cooperation.
+ Memory structures to store BGP-related primitives have been
optimized. Memory is now allocated only for primitives part of
the selected aggregation profile ('aggregate' config directive).
+ A new 'bgp_follow_nexthop' configuration directive is introduced
to follow the BGP next-hop up to the edge of the routing domain.
This is particularly aimed at networks not running MPLS, where
hop-by-hop routing is in place.
+ Lookup maps for BGP-related source primitives (bgp_src_med_map,
bgp_peer_src_as_map, bgp_src_local_pref_map): result of check(s)
can now be the keyword 'bgp', ie. 'id=bgp' which triggers a BGP
lookup. This is thought to handle exceptions to static mapping.
+ A new 'bgp_peer_as_skip_subas' configuration directive is being
introduced. When computing peer_src_as and peer_dst_as, returns
the first ASN which is not part of a BGP confederation; if only
confederated ASNs are on the AS-Path, the first one is returned
instead.
+ Pre-Tagging: support has been introduced for NetFlow v9 traffic
direction (ingress/egress).
+ Network masks part of NetFlow/sFlow export protocols can now be
used to compute src_net, dst_net and sum_net primitives. As a
result a set of directives [nfacctd|sfacctd|pmacctd|uacctd]_net
allows to globally select the method to resolve such primitives,
valid values being: netflow, sflow, file (networks_file), mask
(networks_mask) and bgp (bgp_daemon).
+ uacctd: introduced support for input/output interfaces, fetched
via NetLink/ULOG API; interfaces are available for Pre-Tagging,
and inclusion in NetFlow and sFlow exports. The implementation
is courtesy of Stig Thormodsrud.
+ nfprobe, sfprobe: new [nfprobe|sfprobe]_peer_as option to set
source/destination ASNs, part of the NetFlow and sFlow exports,
to the peer-AS rather than origin-AS. This feature depends on a
working BGP daemon thread setup.
! A few resource leaks were detected and fixed. Patch is courtesy
of Eric Sesterhenn.
! bgp/bgp.c: thread concurrency was detected upon daemon startup
under certain conditions. As a solution the BGP thread is being
granted a time advantage over the traffic collector thread.
! bgp/bgp.c: fixed a security issue which could have allowed a
malicious user to disrupt established working BGP sessions by
exploiting the implemented concept of BGP session replenishment;
this has been secured by a check against the session holdtime.
Many thanks to Erik van der Burg for spotting the issue.
! bgp/bgp.c: BGP listener socket now sets SO_REUSEADDR option for
quicker turn around times while stopping/starting the daemon.
! net_aggr.c: default route (0.0.0.0/0) was considered invalid;
this is now fixed.
0.12.0rc3 -- 28-10-2009
+ Support for NetFlow v9 sampling via Option templates and
data is introduced; this is twofold: a) 'nfacctd_renormalize'
configuration directive is now able to renormalize NetFlow v9
data on-the-fly by performing Option templates management; b)
'nfprobe', the NetFlow probe plugin, is able to flag sampling
rate (either internal or external) when exporting flows to the
collector.
+ '[pm|u]acctd_ext_sampling_rate' directives are introduced to
support external sampling rate scenarios: packet selection is
performed by the underlying packect capturing framework, ie.
ULOG, PF_RING. Making the daemon aware of the sampling rate,
allows to renormalize or export such information via NetFlow
or sFlow.
+ pmacctd: the IPv4/IPv6 fragment handler engine was reviewed
to make it sampling-friendly. The new code hooks get enabled
when external sampling (pmacctd_ext_sampling_rate) is defined.
+ A new 'uacctd' daemon is added to the set; it is based on the
Netlink ULOG packet capturing framework; this implies it works
only on Linux and can be optionally enabled when compling by
defining the '--enable-ulog' switch. The implementation is
fully orthogonal with the existing feature set. Thanks very
much to: A.O. Prokofiev for contributing the original idea
and code; Stig Thormodsrud for his support and review.
+ The 'tag2' primitive is introduced. Its aim is to support
traffic matrix scenarios by giving a second field dedicated
to tag traffic. In a pre_tag_map this can be employed via the
'id2' key. See examples in the 'examples/pretag.map.example'
document. SQL plugins write 'tag2' content in the 'agent_id2'
field. Read 'sql/README.agent_id2' document for reference.
+ Some new directives to control and re-define file attributes
written by the pmacct daemons, expecially when launched with
increased priviledges, are introduced: file_umask, files_uid,
files_gid. Files to which these apply include, ie. pidfile,
logfile and BGP neighbors file.
! fix, bgp/bgp.c: upon reaching bgp_daemon_max_peers threshold,
logs were flooded by warnings even when messages were coming
from a previously accepted BGP neighbor. Warnings are now sent
only when a new BGP connection is refused.
! fix, nfprobe/netflow9.c: tags (pre_tag_map, post_tag) were set
per pair of flows, not respecting their uni-directional nature.
It was generating hiding of some tags.
! fix, nfprobe/netflow9.c: templates were (wrongly) not being
included in the count of flows sent in NetFlow v9 datagrams.
While this was not generating any issues with parsing flows,
it was originating visualization issues in Wireshark.
! fix, SQL plugins: CPU hitting 100% has been determined when
sql_history is disabled but sql_history_roundoff is defined.
Thanks to Charlie Allom for reporting the issue.
! fix, sfacctd.c: input and output interfaces (non-expaneded
format) were not correcly decoded creating issues to Pre-
tagging. Thanks to Jussi Sjostrom for reporting the issue.
0.12.0rc2 -- 09-09-2009
+ BGP daemon thread has been tied up with both the NetFlow and
sFlow probe plugins, nfprobe and sfprobe, allowing to encode
dynamic ASN information (src_as, dst_as) instead of reading
it from text files. This finds special applicability within
open-source router solutions.
+ 'bgp_stdcomm_pattern_to_asn' feature is introduced: filters
BGP standard communities against the supplied pattern. The
first matching community is split using the ':' symbol. The
first part is mapped onto the peer AS field while the second
is mapped onto the origin AS field. The aim is to deal with
prefixes on the own address space. Ie. BGP standard community
XXXXX:YYYYY is mapped as: Peer-AS=XXXXX, Origin-AS=YYYYY.
+ 'bgp_neighbors_file' feature is introduced: writes a list of
the BGP neighbors in the established state to the specified
file. This gets particularly useful for automation purposes
(ie. auto-discovery of devices to poll via SNMP).
+ 'bgp_stdcomm_pattern' feature was improved by supporting the
regex '.' symbol which can be used to wildcard a pre-defined
number of characters, ie. '65534:64...' will match community
values in the range 64000-64999 only.
+ SQL preprocess layer: removed dependency between actions and
checks. Overral logics was reviewed to act more consistently
with recently introduced SQL cache entry status field.
+ SQL common layer: poll() timeout is now calculated adaptively
for increased deadline precision.
+ sql_startup_delay feature functionality was improved in order
to let it work as a sliding window to match NetFlow setups in
which a) mainain original flow timestamps and b) enable the
sql_dont_try_update feature is required.
! DST (Daylight Saving Time) support introduced to sql_history
and sql_refresh_time directives. Thanks to <alshu@tut.by> for
reporting the issue.
! fix, pmacctd.c: initial sfprobe plugin checks were disabling