Distribute backup alert hooks

[dann1]

With this change a hook+exporter backup alert system was created. The hook is distributed via packaging in 7.2.1

root@PC04:~# ls -l /var/lib/one/remotes/hooks/metrics/
total 12
-rw-r----- 1 oneadmin oneadmin 8960 May 21 10:24 backup_restore_metrics.rb # looks like it was merged with no execution permission, pls consider this

And this is the reference we have for setting it up

7.5.2 Adding Hooks for Backup Subsystem Alarms
To enable alerts for backup and restore job statuses, you need to create hooks that will be triggered and monitor the status of the corresponding jobs.

Run the following commands to create the hook template files:

cat <<EOF > /tmp/backup.tmpl 
NAME="backup_state_hook" 
TYPE="api"
ARGUMENTS="$API backup" 
ARGUMENTS_STDIN="YES" 
CALL="one.vm.backup" COMMAND="/var/lib/one/remotes/hooks/update_backup_restore_metrics.rb" 
REMOTE="NO" 
EOF 

cat <<EOF > /tmp/restore.tmpl 
NAME="restore_state_hook" 
TYPE="api"
ARGUMENTS="$API restore" 
ARGUMENTS_STDIN="YES" 
CALL="one.vm.restore" COMMAND="/var/lib/one/remotes/hooks/update_backup_restore_metrics.rb" 
REMOTE="NO" 
EOF
Run the following commands to create the hooks from the template files and cleanup template files:

onehook create /tmp/backup.tmpl
onehook create /tmp/restore.tmpl
rm /tmp/{backup,restore}.tmpl

After the hooks are created, Alertmanager alerts for backup and restore job statuses will start working.

We need to automatically deploy opennebula with this, whenever features.prometheus is set to true in the inventory. Or if you think it shouldn't be tied to that, then have a way to signal it.

[sk4zuzu]

Hi @dann1 , why /var/lib/one/remotes/hooks/update_backup_restore_metrics.rb vs /var/lib/one/remotes/hooks/metrics/backup_restore_metrics.rb ? Is it because looks like it was merged with no execution permission, pls consider this or is there any other modification other than chmod needed ? 🤔

[sk4zuzu]

btw

cat <<EOF > /tmp/backup.tmpl 
NAME="backup_state_hook" 
TYPE="api"
ARGUMENTS="$API backup" 
ARGUMENTS_STDIN="YES" 
CALL="one.vm.backup" COMMAND="/var/lib/one/remotes/hooks/update_backup_restore_metrics.rb" 
REMOTE="NO" 
EOF

this can't be right 🤔 as bash will treat $API as a variable and (if undefined) put empty string there.. Here's clear example of here-doc https://docs.opennebula.io/7.0/product/integration_references/system_interfaces/hook_driver/#creating-hooks where $ is escaped.

[sk4zuzu]

Another thing, why those are called "state_hook" ? Those are of type "api", should be called "api_hook", no? 🤔

[sk4zuzu]

After researching it, it seems that we don't ship opennebula and openssl gems, so users have to diy it:

apt install gcc ruby-dev libssl-dev
gem install opennebula openssl

this is a significant requirement because of gcc and devel libraries, shouldn't all this be shipped within opennebula-rubygems ? 🤔

[sk4zuzu]

/bin/env GEM_HOME=/usr/share/one/gems/ ruby -I /usr/lib/one/ruby/ /var/lib/one/remotes/hooks/metrics/backup_restore_metrics.rb

[roamingengineer311]

@sk4zuzu

❯ onehook show 5
HOOK 5 INFORMATION                                                              
ID                : 5                   
NAME              : backup_api_test     
TYPE              : api                 
LOCK              : None                

HOOK TEMPLATE                                                                   
ARGUMENTS="$API backup"
ARGUMENTS_STDIN="YES"
CALL="one.vm.backup"
COMMAND="/var/lib/one/remotes/hooks/update_backup_restore_metrics.rb"
REMOTE="NO"

that is working hook example

[sk4zuzu]

Hi @roamingengineer311

No, it isn't. 🤗

I don't know what this file is /var/lib/one/remotes/hooks/update_backup_restore_metrics.rb, I don't have it. But that doesn't matter anymore I figured it out anyways, you can check the PR #224.

[roamingengineer311]

@sk4zuzu ah yes on my lab it is having an old name :)
lg!