Merge pull request #2 from ncode/juliano/development

update readme and update development setup

Author: ncode

.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: Test and coverage
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 2
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '1.23'
+      - name: Run coverage
+        run:  go test -coverpkg=./... ./... -race -coverprofile=coverage.out -covermode=atomic
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v4
+        with:
+          verbose: true 
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/codeql.yml

@@ -0,0 +1,74 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ "main" ]
+  schedule:
+    - cron: '35 15 * * 3'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'go' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v3
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+
+        # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+        # queries: security-extended,security-and-quality
+
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v3
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+
+    #   If the Autobuild fails above, remove it and uncomment the following three lines.
+    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
+
+    # - run: |
+    #   echo "Run, Build Application using script"
+    #   ./location_of_script_within_repo/buildscript.sh
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v3
+      with:
+        category: "/language:${{matrix.language}}"

.github/workflows/go.yml

@@ -0,0 +1,28 @@
+# This workflow will build a golang project
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
+
+name: Go
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.23'
+
+    - name: Build
+      run: go build -v ./...
+
+    - name: Test
+      run: go test -v ./...

.gitignore

@@ -26,3 +26,7 @@ go.work.sum
 
 # idea
 .idea
+
+# ignore the log directory
+
+configs/development/logs/

README.md

@@ -1,2 +1,157 @@
-# vault-audit-filter
- A lightweight tool designed to receive audit events from the vault and apply filtering using a rule-based engine
+[![Go Report Card](https://goreportcard.com/badge/github.com/ncode/vault-audit-filter)](https://goreportcard.com/report/github.com/ncode/vault-audit-filter)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![codecov](https://codecov.io/gh/ncode/vault-audit-filter/graph/badge.svg?token=PTW9OYF19R)](https://codecov.io/gh/ncode/vault-audit-filter)
+
+# Vault Audit Filter
+
+`vault-audit-filter` is a Go-based tool designed to filter and log HashiCorp Vault audit logs based on configurable rules. It provides fine-grained control over how Vault audit events are processed and categorized, allowing you to capture critical events while reducing noise from routine operations.
+
+## Features
+
+- **Configurable Rule-Based Filtering**: Define rules to match specific audit events, such as read, write, delete, or specific paths in Vault.
+- **Multiple Rule Groups**: Organize rules into groups and log them to separate files.
+- **Dynamic Logging**: Log audit events to specified files with log rotation and size limits.
+- **Supports Multiple Operations**: Filters common Vault operations, including KV operations, metadata updates, and deletion events.
+- **Performance-Oriented**: Built with `gnet` to handle high concurrency.
+
+## Table of Contents
+
+- [Getting Started](#getting-started)
+- [Installation](#installation)
+- [Configuration](#configuration)
+- [Usage](#usage)
+- [Testing](#testing)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Getting Started
+
+These instructions will help you set up and run `vault-audit-filter` on your local machine.
+
+### Prerequisites
+
+- **Go**: Ensure you have Go 1.21 or later installed. You can download it here: <https://golang.org/dl/>
+- **Vault**: You should have HashiCorp Vault installed and configured. Instructions can be found here: <https://www.vaultproject.io/docs/install>
+
+### Installation
+
+Clone the repository:
+
+    git clone https://github.com/ncode/vault-audit-filter.git
+    cd vault-audit-filter
+
+### Build the Project
+
+To build the binary:
+
+    go build -o vault-audit-filter .
+
+### Running the Application
+
+Once you have built the project, you can run the `vault-audit-filter` executable:
+
+    ./vault-audit-filter --config config.yaml
+
+## Configuration
+
+`vault-audit-filter` uses a YAML-based configuration file that allows you to define rule groups, specify logging files, and configure Vault settings.
+
+### Sample Configuration (`config.yaml`)
+
+    vault:
+      address: "http://127.0.0.1:8200"
+      token: "your-vault-token"
+      audit_path: "/vault-audit-filter"
+      audit_address: "127.0.0.1:1269"
+      audit_description: "Vault Audit Filter Device"
+
+    rule_groups:
+      - name: "normal_operations"
+        rules:
+          - 'Request.Operation in ["read", "update"] && Request.Path startsWith "secret/data/" && Auth.PolicyResults.Allowed == true'
+        log_file:
+          file_path: "/var/log/vault_normal_operations.log"
+          max_size: 100      # Max size in MB
+          max_backups: 5     # Max number of backup files
+          max_age: 30        # Max age in days
+          compress: true     # Compress rotated files
+
+      - name: "critical_events"
+        rules:
+          - 'Request.Operation == "delete" && Auth.PolicyResults.Allowed == true'
+          - 'Request.Path startsWith "secret/metadata/" && Auth.PolicyResults.Allowed == true'
+        log_file:
+          file_path: "/var/log/vault_critical_events.log"
+          max_size: 100
+          max_backups: 5
+          max_age: 30
+          compress: true
+
+### Configuration Parameters
+
+- **Vault Settings**:
+  - `vault.address`: The address of your Vault instance.
+  - `vault.token`: Vault token for authentication.
+  - `vault.audit_path`: The path for Vault's audit device.
+  - `vault.audit_address`: The address for receiving audit logs.
+  - `vault.audit_description`: Description for the Vault audit device.
+
+- **Rule Groups**:
+  - `rule_groups.name`: The name of the rule group.
+  - `rule_groups.rules`: A list of expressions using `expr` to define rules for audit log filtering.
+  - `log_file.file_path`: The file path where matching logs will be written.
+  - `log_file.max_size`: The maximum size of the log file in MB before rotation.
+  - `log_file.max_backups`: The number of backup logs to keep.
+  - `log_file.max_age`: The maximum number of days to retain logs.
+  - `log_file.compress`: Whether to compress the old log files.
+
+### Rule Syntax
+
+Rules are written using the `expr` language, a simple and safe expression language for Go. Rules can be based on the following properties of audit logs:
+
+- `Request.Operation`: The type of operation (`read`, `update`, `delete`, etc.).
+- `Request.Path`: The Vault path being accessed.
+- `Auth.PolicyResults.Allowed`: Whether the operation was allowed.
+
+**Example Rule**:
+
+  'Request.Operation == "update" && Request.Path startsWith "secret/data/" && Auth.PolicyResults.Allowed == true'
+
+## Usage
+
+To run `vault-audit-filter` with your configuration file, use:
+
+```bash
+$ ./vault-audit-filter --config config.yaml
+```
+
+### Command-Line Options
+
+- `--config`: Specify the path to the configuration file (default is `config.yaml`).
+- `--log-level`: Set the logging level (`debug`, `info`, `warn`, `error`).
+
+### Environment Variables
+
+You can also define environment variables to override configuration file values. For example:
+
+```bash
+$ export VAULT_ADDRESS="http://127.0.0.1:8200"
+$ export VAULT_TOKEN="your-vault-token"
+```
+
+### Development
+
+For development purposes, you can use the provided Makefile located at `configs/development/Makefile` to build and run the project using Docker and Docker Compose. This is how I test my changes and have a playground of sorts.
+
+## Contributing
+
+We welcome contributions from the community! 
+Before submitting a pull request, ensure that:
+
+- The code compiles without errors.
+- All tests pass.
+- Your changes are well-documented.
+
+## License
+
+This project is licensed under the Apache License, Version 2.0. See the `LICENSE` file for details.