Use file content heuristics to decide file reader. (#1962)

* Added heuristics file content detector that determines the content based on the magic number.

* Moved stream checkpoint outside format detector as it is not directly tied to it.

* Added a new factory function `createReader` that uses the new heuristics detection method.

* Add <algorithm> include.

* Added unit tests.

* Deprecated old factory function.

* Add byte-swapped zstd magic number.

* Lint

* Move enum closer to first usage.

* Added unit tests for file reader device factory.

* Revert indentation.

* Fixed StreamCheckpoint to restore original stream state.

* Moved isStreamSeekable helper to inside `CaptureFileFormatDetector`.

Move it out if it needs to be reused somewhere.

* Added pcap magic number for Alexey Kuznetzov's modified pcap format.

Libpcap supports reading this format since 0.9.1. The heuristics detection will identify such magic number as pcap and leave final support decision to the pcap backend infrastructure.

* Split the unit test into multiple smaller tests.

* Added helper to indicate if ZstSupport is enabled for PcapNg devices.

* Split pcap microsecond and nanosecond file heuristics tests.
Nanoseconds are skipped if they aren't supported.

* Skipping Zst test case if zst is not supported.

* Due to file heuristics returning PcapNG format on Zstd archive, if Zstd compression is unsupported, the format detector will return Unknown.

* Lint

* Added invalid device factory to pcap tag.

* Updated static zst archives to be actual archives.

* Centralized PTF test name width under a macro.

* Add Pcap++Test header files to test sources for IDE tooling.

* Fixed test output formatting.

* Lint

* Typo fix.

* Shortened test names.

* Simplified invalid file test.

* Simplified ZST tests.

* Added snoop test.

* Expanded granularity of file format detection.

Updated pcap file detection to return the precice format of Pcap instead of just `true` / `false`.
Updated detect format to always retuirn the detected format. Previous responsibility for unsupported zstd archive files has been passed up the call stack to the factory function `createReader`.

* Marked `checkSupport` functions as constexpr to enable compile time optimizations and branch pruning.

* Exclude json from pre-commit cppcheck as it is slow due to many define branches.

* Lint

* Fix runtime side effects inside constexpr function.

* Added a secondary factory function to separate mixed error handling methods.

* Revert deprecation message, as doxygen is unhappy.

* Update tests.

* Update deprecation warning to point to the function closer to the signature.

* Catch general exception instead of runtime error.

* Shortened deprecation message due to pre-commit warnings when its is on 1 line and doxygen errors when its in 2 lines.

* Fix braces.

* Simplfy test.

* Added tests for createReader failures.

* Simplified pcap detection to not require to read the entire pcap header.

* Added const qualifiers to detector methods.

* Added dedicated unit tests for CaptureFileFormatDetector.

* Added more tests for `createReader`.

* Add static assert for array indice checks.

* Updated detectPcap selection.

* Extracted capture format detector to remove it from publicly available headers.

* Fix includes.

* Removed duplicate files from tracking.

* Lint

* Trimmed pcapng sample.

* Change PcapNGZst to ZstArchive. Zst to PcapNG branch folding is done in `createReader` instead of having the Format detector assume that is what is intended.

* Added separate format value for "modified" pcap to separate the format detector from libpcap behaviour.

* Docs fix.

* Add automatic open functionality to `createReader` factory to mimic RAII initialization.

* Update docstring.

* Fix exception message assert.

* Refactored format tests to utilize the createReader factory.

* Fix nanoprecision test issues.

* Remove openDevice flag. Update create procedure to avoid exceptions on tryCreateDevice.

* Docs update + Lint

* Docs fix.

* Lint.

* Remove nano support checks as it should always be supported by the internal parser.

* Extend format detector to capture byte order from magic number.

* Add initial merge logic.

* Add methods to check magic numbers directly.

* Comment out unused variables.

* Add helper to convert format to pcap precision.

* Simplify code.

* Fixed errors.

* Disable pcap "modified" file format detection tests.

* Lint

* Fixed doxygen.

* Change Kuznetzov pcap magic number to constexpr.

* Change all temporary files to have `.tmp` extension. Remove `/temp/` subdirectory.

* Lint

* Revert `tryCreateReader` to use `createReader`

* Remove conditional branch that always evaluates to true.

* Lint

Author: Dimi1010

.pre-commit-config.yaml

@@ -1,4 +1,4 @@
-exclude: '.*\.(pcap|pcapng|dat)|(PacketExamples|PcapExamples|expected_output|pcap_examples).*\.txt'
+exclude: '.*\.(pcap|pcapng|dat)|(PacketExamples|PcapExamples|expected_output|pcap_examples).*\.(txt|zst|zstd)'
 fail_fast: false
 repos:
   - repo: local
@@ -37,6 +37,7 @@ repos:
         files: ^(Common\+\+|Packet\+\+|Pcap\+\+|Tests|Examples)/.*\.(cpp|h)$
       - id: cppcheck
         args: ["--std=c++14", "--language=c++", "--suppressions-list=cppcheckSuppressions.txt", "--inline-suppr", "--force"]
+        exclude: ^3rdParty/json
   - repo: https://github.com/BlankSpruce/gersemi
     rev: 0.26.1
     hooks:

Pcap++/CMakeLists.txt

@@ -108,6 +108,7 @@ target_link_libraries(
 )
 
 if(LIGHT_PCAPNG_ZSTD)
+  target_compile_definitions(Pcap++ PRIVATE -DPCPP_PCAPNG_ZSTD_SUPPORT)
   target_link_libraries(Pcap++ PRIVATE light_pcapng)
 endif()
 

Pcap++/header/PcapFileDevice.h

@@ -114,7 +114,32 @@ namespace pcpp
 		/// it returns an instance of PcapFileReaderDevice
 		/// @param[in] fileName The file name to open
 		/// @return An instance of the reader to read the file. Notice you should free this instance when done using it
+		/// @deprecated Prefer `createReader` or `tryCreateReader` due to selection of reader based on file content
+		/// instead of extension.
+		PCPP_DEPRECATED("Prefer `tryCreateReader` due to selection of reader based on file content.")
 		static IFileReaderDevice* getReader(const std::string& fileName);
+
+		/// @brief Creates an instance of the reader best fit to read the file.
+		///
+		/// The factory function uses heuristics based on the file content to decide the reader.
+		/// If the file type is known at compile time, it is better to construct a concrete reader instance directly.
+		///
+		/// @param[in] fileName The path to the file to open.
+		/// @return A unique pointer to a reader instance
+		/// @throws std::runtime_error If the file could not be opened or is of unsupported format.
+		/// @remarks The device is not opened automatically. Call `open()` on the returned device before using it.
+		static std::unique_ptr<IFileReaderDevice> createReader(const std::string& fileName);
+
+		/// @brief Tries to create an instance of the reader best fit to read the file.
+		///
+		/// The factory function uses heuristics based on the file content to decide the reader.
+		/// If the file type is known at compile time, it is better to construct a concrete reader instance directly.
+		///
+		/// @param fileName The path to the file to open.
+		/// @return A unique pointer to a reader instance, or nullptr if the file could not be opened or is of
+		/// unsupported format.
+		/// @remarks The device is not opened automatically. Call `open()` on the returned device before using it.
+		static std::unique_ptr<IFileReaderDevice> tryCreateReader(const std::string& fileName);
 	};
 
 	/// @class IFileWriterDevice