Commit b923b85
Fetch useful info from clients and store it as client metadata (Velocidex#1203)
# Client metadata and asset management
Although I imagine everyone using Velociraptor mostly to fetch data en
masse using hunts, personally I usually look up individual clients for
investigations. Later on I will run targeted hunts based on
metadata/tags, and rarely hunt across the whole fleet.
One of the perhaps biggest upgrades for myself has been to be able to
look up clients based on their serial number or assigned owner (fetched
from an asset management system). I have since extended this system to
tie the client to external systems and agents like Defender/Intune/Entra
and Action1/NinjaOne based on its serial or agent ID.
I have used most of these artifacts for years now, but I have not had
the capacity to document how to use them. predictible has since written
a good [knowledge base
artifact](https://docs.velociraptor.app/knowledge_base/tips/automating_metadata/)
that covers most if not all the concepts, so releasing the following
artifacts may not require a lot of additional documentation. There
should be at least a complete example showing
- How to extend Custom.Generic.Client.Info with
Generic.Client.HW.Identification
- Configure Server.Monitor.StoreClientHWInfo (illustrate how to use
SerialColumn and HWInfoMetadata)
- Configure Server.Monitor.StoreClientInfo (illustrate how to use
InfoMetadata)
An excellent showcase would use Defender, ADStatus FIXME for more useful
fields.
Perhaps a knowledge base article or blog post (whichever is the most
suitable format) referring to predictible's existing article, then
simply setting up all artifacts and showing how to look up various
indexed metadata provided by these new artifacts is enough?
## Client artifacts
Gather information from the client system.
These are helper artifacts fetching information about the client. The
aim is to extract some sort of identifier, a computer serial or some
sort of ID, that will later be used to fetch a lot of telemetry from the
computer from another agent. – Rather than implementing all of that in
Veloriraptor. However, when the useful data is already there as part of
fetching this identifier, it will be included in the artifact.
The intended way to use of these artifacts is by calling them from
Custom.Generic.Client.Info, but they may be used on their own.
## Generic.Client.HW.Identification
This artifact fetches useful hardware identification values from all
OSes and has served me well for years. It would be nice to gather
experience from manufacturers that I have never run clients on. The only
changes it has seen recently is a very rudimentary attempt to detect
whether the client is running in a VM.
## Generic.Client.ADStatus
The primary use case for this artifact is to get a simple answer to
whether the computer is joined to an Active Directory domain or not.
Depending on the OS, a lot more details are provided.
## Generic.Client.Defender.Health
The primary use case for this artifact is to get the MDATP machine ID
for all three OSes, but an extensive list of health data from the agent
is fetched.
## TODO: A good fourth example
I have some ideas, but I am very open for suggestions that goes beyond
just my own particular needs.
## Server event artifacts
Store information from interrogation as client metadata.
Storing data from the Custom.Generic.Client.Info is easy and almost does
not require its own published artifact. However, the following two
artifacts saves the user from having to write this themselves, and lets
them configure their desired result from parameters.
### Server.Monitor.StoreClientHWInfo
This is a relatively simple artifact, but it is very useful. The
parameter SerialColumn lets you choose which metadata columns (defaults
assuming Generic.Client.HW.Identification is used) to use for serial
value, in a prioritised order. The artifact comes with common patterns
to ignore. It could be extended with some transforms for manufacturers
who store serials in a weird way.
### Server.Monitor.StoreClientInfo
Given a specified interrogation artifact, this artifacts stores
specified columns from specified sources as optionally renamed metadata
values. Sources should only contain one row, but if they contain more,
the first row is picked.
Example use of the InfoMetadata parameter:
| Source | Field | Alias |
|------------------|----------------------------------------|-------------|
| BasicInformation | OS | os |
| BasicInformation | Architecture | arch |
| ADStatus | DomainJoined | ad_joined |
| MDATPHealth | MDATPHealth.edrMachineId | defender_id |
| ComputerSetup | Details.computer setup.execution.timestamp |
cs_executed |
| ComputerSetup | Details.computer setup.git.hash | cs_version |
(The ComputerSetup source here is an artifact fetching metadata produced
by an ansible playbook run, telling us when it was last executed, and
what version.)
This artifact could probably be over-engineered to add some sort of
transformation logic to support multiple rows. However, I think it is
fine as it is.
I am not sure if it is fixed, but attempting to store empty metadata
created issues in some version, hence the conditional at the end.
# Asset management
Now that clients have various IDs set in metadata, it would be great to
pull useful data (not stored on the endpoint) from external sources
using APIs. Examples:
- Fetch assigned owner and assigned use from an asset management system
- Look up account name / account ID
- Fetch vulnerabilities and security issues/recommendations
- TODO: plenty of examples to fill
Use cases where this additional information is fetched and then used in
a notebook are topics for a chapter of its own. Right now we want to
fetch data to store as additional indexed metadata. We will use one
particular example: Fetching information about the assigned owner from
an asset management system.
Velociraptor will give us plenty of information about logged-in users,
but I want to know who actually owns the computer. This can then be used
to determine whether the last-logged in user shouldn't even be logged
into the system in the first place. Other information, like assigned
department, location, or other useful custom fields, could also be
valuable.
We will use Snipe-IT as our example asset management system. It is most
likely not used in large enterprises, but is open-source and although a
bit cumbersome, still powerful and useful. I have been planning to write
one for Atlassian Assets. The system in question matters little, since
they will have have some sort of REST API and very common concepts.
We want to do the following:
- Fetch assets (perhaps matching certain filters)
- Match assets by serial
- Transform some of the information
- Synchronise selected data from the asset and save as client metadata
We want this data updated, so the data will be synchronised either
- On demand (running the server artifact)
- Whenever the client is interrogated, giving us immediate updates for
new clients (done by a server event artifact)
- Periodically, so that changes in the assets are updated (e.g. new
owner) (done by a server event artifact)
The server event artifacts simply call the server artifacts. Since the
server artifact has a lot of important parameters, and since we cannot
expect the administrator to have to fill these in in three separate
places, the following is done:
- The server artifact saves its arguments to a server metadata value
- The event artifacts fetches all arguments from the same value
This may be elegant or ugly, depending on who you ask, but it works and
certainly beats the alternative.
## Server artifacts
### Server.Assets.SnipeIT.Sync
Querying Snipe-IT is trivial (at least with the QueryAPI server artifact
helper). However, in order to create a user-friendly way to select which
fields to synchronise, with support for custom fields and aliases,
additional logic is needed. The artifact may be used in three ways:
- Simply fetch all assets (matching filters)
- Fetch assets with a serial found in Velociraptor clients
- Synchronise client metadata
The first two are "read-only" actions and can be used with notebook
suggestions to
- Find clients missing serial numbers
- Find clients not in assets
- Find assets not running Velociraptor
## Server event artifacts
### Server.Monitor.SnipeIT.Sync.Interrogation
Update client metadata whenever it is interrogated. Shows what metadata
changed (from/to).
## Server.Monitor.SnipeIT.Sync.Periodic
Update client metadata at a specified interval. Shows what metadata
changed (from/to).
# Artifact table
Complete artifact list (with dependencies, if any):
| Name | Type | Deps. | Description |
|------|------|--------|-------------|
| Generic.Client.HW.Identification | Client | - | Pull hardware
identification information from the system, like serial number and
computer model |
| Generic.Client.ADStatus | Client | - | Fetch Active Directory status |
| Generic.Client.Defender.Health | Client | - | Get Microsoft Defender
health status |
| Server.Monitor.StoreClientHWInfo | Server | - | Store client HW info
as metadata |
| Server.Monitor.StoreClientInfo | Server | (HW.Identification) | Store
client info as metadata |
| Server.Utils.QueryAPI | Server | - | Query an HTTP API with login and
pagination support |
| Server.Assets.SnipeIT.Sync | Server | (QueryAPI) | Save key Snipe-IT
asset information as client metadata |
| Server.Monitor.SnipeIT.Sync.Interrogation | Server event |
SnipeIT.Sync | Update asset information when client is interrogated |
| Server.Monitor.SnipeIT.Sync.Periodic | Server event | SnipeIT.Sync |
Update asset information regularly at a specified interval |
---------
Co-authored-by: Mike Cohen <mike@velocidex.com>1 parent 168ba66 commit b923b85
7 files changed
Lines changed: 876 additions & 1 deletion
File tree
- .github/workflows
- content
- exchange/artifacts
- knowledge_base/tips
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
Lines changed: 170 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
0 commit comments