Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions .github/workflows/BSD-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@ name: Run tests on BSDs

on:
workflow_dispatch:
# push:
# branches: [ master ]
# paths-ignore:
# - 'docs/**'
push:
branches: [ master ]
paths-ignore:
- 'docs/**'

# pull_request:
# types: [opened, synchronize, reopened]
# paths-ignore:
# - 'docs/**'
pull_request:
types: [opened, synchronize, reopened]
paths-ignore:
- 'docs/**'

jobs:
test-freebsd:
Expand All @@ -36,7 +36,7 @@ jobs:
GOOS: freebsd

- name: Run tests in VM
uses: cross-platform-actions/action@v0.32.0
uses: cross-platform-actions/action@v1
with:
operating_system: freebsd
version: '${{ matrix.freebsd_version }}'
Expand Down Expand Up @@ -70,7 +70,7 @@ jobs:
GOOS: netbsd

- name: Run tests in VM
uses: cross-platform-actions/action@v0.32.0
uses: cross-platform-actions/action@v1
with:
operating_system: netbsd
version: '${{ matrix.netbsd_version }}'
Expand Down Expand Up @@ -104,7 +104,7 @@ jobs:
GOOS: openbsd

- name: Run tests in VM
uses: cross-platform-actions/action@v0.32.0
uses: cross-platform-actions/action@v1
with:
operating_system: openbsd
version: '${{ matrix.openbsd_version }}'
Expand Down
81 changes: 43 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,60 +1,65 @@
![Build](https://github.com/creativeprojects/resticprofile/workflows/Build/badge.svg)
![Run tests on FreeBSD, OpenBSD, NetBSD](https://github.com/creativeprojects/resticprofile/actions/workflows/BSD-tests.yml/badge.svg)
[![codecov](https://codecov.io/gh/creativeprojects/resticprofile/branch/master/graph/badge.svg?token=cUozgF9j4I)](https://codecov.io/gh/creativeprojects/resticprofile)

# resticprofile
Configuration profiles manager for [restic backup](https://restic.net/)

**resticprofile** is the missing link between a configuration file and restic backup. Creating a configuration file for restic has been [discussed before](https://github.com/restic/restic/issues/16), but seems to be a very low priority right now.

With resticprofile:

* You no longer need to remember command parameters and environment variables
* You can create multiple profiles inside one configuration file
* A profile can inherit all the options from another profile
* You can run the forget command before or after a backup (in a section called *retention*)
* You can check a repository before or after a backup
* You can create groups of profiles that will run sequentially
* You can run shell commands before or after running a profile: useful if you need to mount and unmount your backup disk for example
* You can run a shell command if an error occurred (at any time)
* You can send a backup stream via _stdin_
* You can start restic at a lower or higher priority (Priority Class in Windows, *nice* in all unixes) and/or _ionice_ (only available on Linux)
* It can check that you have enough memory before starting a backup. (I've had some backups that literally killed a server with swap disabled)
* You can generate cryptographically secure random keys to use as a restic key file
* You can easily schedule backups, retentions and checks (works for *systemd*, *crond*, *launchd* and *windows task scheduler*)
* You can generate a simple status file to send to some monitoring software and make sure your backups are running fine
* You can use a template syntax in your configuration file
* You can generate scheduled tasks using *crond*
* Get backup statistics in your status file
* Automatically clear up stale locks
* Export a **prometheus** file after a backup, or send the report to a push gateway automatically
* Run shell commands in the background when non fatal errors are detected from restic
* Send messages to HTTP hooks before, after a successful or failed job (backup, forget, check, prune, copy)
* Automatically initialize the secondary repository using `copy-chunker-params` flag
* Send resticprofile logs to a syslog server
* Preventing your system from idle sleeping
* See the help from both restic and resticprofile via the `help` command or `-h` flag
* Don't schedule a job when the system is running on battery
* **[new for v0.29.0]** Scheduling a group of profiles (configuration `v2` only)
**resticprofile** is the missing link between a configuration file and restic backup command line.

## Features

Here is a non-exhaustive list of what resticprofile offers:

* **Profiles**
* No need to remember command parameters and environment variables
* Create multiple profiles in one configuration file
* Profiles can inherit options from other profiles
* Create groups of profiles to run sequentially
* Easily schedule backups, retentions, and checks (supports *systemd*, *crond*, *launchd*, and *Windows Task Scheduler*)
* Use template syntax in your configuration file
* **[new for v0.29.0]** Schedule a group of profiles (configuration `v2` only)
* **Automation**
* Run the forget command before or after a backup (in a section called *retention*)
* Check a repository before or after a backup
* Run shell commands before or after running a profile, useful for mounting and unmounting backup disks
* Run a shell command if an error occurs
* Send a backup stream via _stdin_
* Start restic at different priorities (Priority Class in Windows, *nice* in Unix, and/or _ionice_ in Linux)
* Automatically clear stale locks
* **Monitoring**
* Generate a simple status file for monitoring software to ensure backups are running smoothly
* Export a prometheus file after a backup or send the report to a push gateway
* Run shell commands in the background when non-fatal errors are detected
* Send messages to HTTP hooks before, after a successful or failed job (backup, forget, check, prune, copy)
* Send resticprofile logs to a syslog server
* **Checks**
* Check for enough memory before starting a backup
* Avoid scheduling a job when the system is on battery
* **Misc**
* Generate cryptographically secure random keys for a restic key file
* Automatically initialize the secondary repository using the `copy-chunker-params` flag
* Prevent the system from idle sleeping
* View help for both restic and resticprofile via the `help` command or `-h` flag

## Configuration files

The configuration file accepts various formats:
* [TOML](https://github.com/toml-lang/toml) : configuration file with extension _.toml_ and _.conf_ to keep compatibility with versions before 0.6.0
* [TOML](https://github.com/toml-lang/toml) : configuration file with extension _.toml_ or _.conf_
* [JSON](https://en.wikipedia.org/wiki/JSON) : configuration file with extension _.json_
* [YAML](https://en.wikipedia.org/wiki/YAML) : configuration file with extension _.yaml_
* [HCL](https://github.com/hashicorp/hcl): configuration file with extension _.hcl_



<!--ts-->

<!--te-->
## Getting started

We recommend you start by reading the [getting started](https://creativeprojects.github.io/resticprofile/configuration/getting_started/index.html) section

# Using resticprofile
## Using resticprofile

The full documentation has been moved to [creativeprojects.github.io](https://creativeprojects.github.io/resticprofile/)

# Survey
## Survey

What are your most important features?
Please fill in the [survey](https://github.com/creativeprojects/resticprofile/issues/415)
Expand Down
71 changes: 71 additions & 0 deletions lock/lock_pid_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
//go:build !netbsd

package lock

import (
"bytes"
"os"
"os/signal"
"path/filepath"
"testing"

"github.com/creativeprojects/resticprofile/shell"
"github.com/shirou/gopsutil/v4/process"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestProcessPID(t *testing.T) {
t.Parallel()

var childPID int32
buffer := &bytes.Buffer{}

// use the lock helper binary (we only need to wait for some time, we don't need the locking part)
cmd := shell.NewCommand(lockBinary, []string{"lock", "-wait", "200", "-lock", filepath.Join(t.TempDir(), t.Name())})
cmd.Stdout = buffer
// SetPID method is called right after we forked and have a PID available
cmd.SetPID = func(pid int32) {
childPID = pid
running, err := process.PidExists(childPID)
assert.NoError(t, err)
assert.True(t, running)
}
_, _, err := cmd.Run()
require.NoError(t, err)

// at that point, the child process should be finished
running, err := process.PidExists(childPID)
assert.NoError(t, err)
assert.False(t, running)
}

func TestForceLockWithExpiredPID(t *testing.T) {
t.Parallel()

tempfile := getTempfile(t)
lock := NewLock(tempfile)
defer lock.Release()

assert.True(t, lock.TryAcquire())
assert.True(t, lock.HasLocked())

// run a child process
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
defer signal.Reset(os.Interrupt)

cmd := shell.NewSignalledCommand("echo", []string{"Hello World!"}, c)
cmd.SetPID = lock.SetPID
_, _, err := cmd.Run()
require.NoError(t, err)

// child process should be finished
// let's close the lockfile handle manually (unix doesn't actually care, but windows would complain)
lock.file.Close()

other := NewLock(tempfile)
defer other.Release()
assert.True(t, other.ForceAcquire())
assert.True(t, other.HasLocked())
}
57 changes: 0 additions & 57 deletions lock/lock_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,13 @@ import (
"fmt"
"os"
"os/exec"
"os/signal"
"path/filepath"
"regexp"
"testing"
"time"

"github.com/creativeprojects/resticprofile/platform"
"github.com/creativeprojects/resticprofile/shell"
"github.com/shirou/gopsutil/v4/process"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
Expand Down Expand Up @@ -136,31 +134,6 @@ func TestSetMorePID(t *testing.T) {
assert.Equal(t, int32(13), pid)
}

func TestProcessPID(t *testing.T) {
t.Parallel()

var childPID int32
buffer := &bytes.Buffer{}

// use the lock helper binary (we only need to wait for some time, we don't need the locking part)
cmd := shell.NewCommand(lockBinary, []string{"lock", "-wait", "200", "-lock", filepath.Join(t.TempDir(), t.Name())})
cmd.Stdout = buffer
// SetPID method is called right after we forked and have a PID available
cmd.SetPID = func(pid int32) {
childPID = pid
running, err := process.PidExists(childPID)
assert.NoError(t, err)
assert.True(t, running)
}
_, _, err := cmd.Run()
require.NoError(t, err)

// at that point, the child process should be finished
running, err := process.PidExists(childPID)
assert.NoError(t, err)
assert.False(t, running)
}

func TestForceLockIsAvailable(t *testing.T) {
t.Parallel()

Expand All @@ -187,36 +160,6 @@ func TestForceLockWithNoPID(t *testing.T) {
assert.False(t, other.HasLocked())
}

func TestForceLockWithExpiredPID(t *testing.T) {
t.Parallel()

tempfile := getTempfile(t)
lock := NewLock(tempfile)
defer lock.Release()

assert.True(t, lock.TryAcquire())
assert.True(t, lock.HasLocked())

// run a child process
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
defer signal.Reset(os.Interrupt)

cmd := shell.NewSignalledCommand("echo", []string{"Hello World!"}, c)
cmd.SetPID = lock.SetPID
_, _, err := cmd.Run()
require.NoError(t, err)

// child process should be finished
// let's close the lockfile handle manually (unix doesn't actually care, but windows would complain)
lock.file.Close()

other := NewLock(tempfile)
defer other.Release()
assert.True(t, other.ForceAcquire())
assert.True(t, other.HasLocked())
}

func TestForceLockWithRunningPID(t *testing.T) {
t.Parallel()

Expand Down
19 changes: 17 additions & 2 deletions shell/command_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -271,8 +271,23 @@ func TestVariablesRewrite(t *testing.T) {
func TestRunLocalCommand(t *testing.T) {
t.Parallel()

cmd := NewCommand("command.go", nil)
expected := "." + string(os.PathSeparator) + "command.go"
// find a file in the current directory
entries, err := os.ReadDir("./")
require.NoError(t, err)

localFilename := ""
for _, entry := range entries {
if !entry.IsDir() && entry.Name()[0] != '.' {
localFilename = entry.Name()
break
}
}
require.NotEmpty(t, localFilename, "no file found in current directory")

t.Logf("using local file %q", localFilename)

cmd := NewCommand(localFilename, nil)
expected := "." + string(os.PathSeparator) + localFilename

for _, shell := range []string{defaultShell, bashShell, powershell, powershell6, windowsShell} {
args := getArgumentsComposer(shell)(cmd)
Expand Down
12 changes: 0 additions & 12 deletions util/executable.go

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build linux
//go:build linux || freebsd || openbsd || netbsd

package util

Expand Down Expand Up @@ -36,7 +36,7 @@ func TestResolveExecutable(t *testing.T) {
})

t.Run("command in PATH", func(t *testing.T) {
// Testing with "ls" which should be available on most Linux systems
// Testing with "ls" which should be available on all unix systems
path, err := resolveExecutable("ls")
assert.NoError(t, err)
assert.NotEmpty(t, path)
Expand Down
12 changes: 12 additions & 0 deletions util/executable_os.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
//go:build darwin || windows

package util

import "os"

// Executable returns the path name for the executable that started the current process.
// On Darwin and Windows, it behaves like os.Executable.
// On Linux and BSDs, it returns the path to the executable as specified in the command line arguments.
func Executable() (string, error) {
return os.Executable()
}
6 changes: 3 additions & 3 deletions util/executable_linux.go → util/executable_symlink.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build linux
//go:build !darwin && !windows

package util

Expand All @@ -10,8 +10,8 @@ import (
)

// Executable returns the path name for the executable that started the current process.
// On non-Linux systems, it behaves like os.Executable.
// On Linux, it returns the path to the executable as specified in the command line arguments.
// On Darwin and Windows, it behaves like os.Executable.
// On Linux and BSDs, it returns the path to the executable as specified in the command line arguments.
func Executable() (string, error) {
return resolveExecutable(os.Args[0])
}
Expand Down
Loading
Loading