Skip to content

Commit b923b85

Browse files
misjescudette
andauthored
Fetch useful info from clients and store it as client metadata (Velocidex#1203)
# Client metadata and asset management Although I imagine everyone using Velociraptor mostly to fetch data en masse using hunts, personally I usually look up individual clients for investigations. Later on I will run targeted hunts based on metadata/tags, and rarely hunt across the whole fleet. One of the perhaps biggest upgrades for myself has been to be able to look up clients based on their serial number or assigned owner (fetched from an asset management system). I have since extended this system to tie the client to external systems and agents like Defender/Intune/Entra and Action1/NinjaOne based on its serial or agent ID. I have used most of these artifacts for years now, but I have not had the capacity to document how to use them. predictible has since written a good [knowledge base artifact](https://docs.velociraptor.app/knowledge_base/tips/automating_metadata/) that covers most if not all the concepts, so releasing the following artifacts may not require a lot of additional documentation. There should be at least a complete example showing - How to extend Custom.Generic.Client.Info with Generic.Client.HW.Identification - Configure Server.Monitor.StoreClientHWInfo (illustrate how to use SerialColumn and HWInfoMetadata) - Configure Server.Monitor.StoreClientInfo (illustrate how to use InfoMetadata) An excellent showcase would use Defender, ADStatus FIXME for more useful fields. Perhaps a knowledge base article or blog post (whichever is the most suitable format) referring to predictible's existing article, then simply setting up all artifacts and showing how to look up various indexed metadata provided by these new artifacts is enough? ## Client artifacts Gather information from the client system. These are helper artifacts fetching information about the client. The aim is to extract some sort of identifier, a computer serial or some sort of ID, that will later be used to fetch a lot of telemetry from the computer from another agent. – Rather than implementing all of that in Veloriraptor. However, when the useful data is already there as part of fetching this identifier, it will be included in the artifact. The intended way to use of these artifacts is by calling them from Custom.Generic.Client.Info, but they may be used on their own. ## Generic.Client.HW.Identification This artifact fetches useful hardware identification values from all OSes and has served me well for years. It would be nice to gather experience from manufacturers that I have never run clients on. The only changes it has seen recently is a very rudimentary attempt to detect whether the client is running in a VM. ## Generic.Client.ADStatus The primary use case for this artifact is to get a simple answer to whether the computer is joined to an Active Directory domain or not. Depending on the OS, a lot more details are provided. ## Generic.Client.Defender.Health The primary use case for this artifact is to get the MDATP machine ID for all three OSes, but an extensive list of health data from the agent is fetched. ## TODO: A good fourth example I have some ideas, but I am very open for suggestions that goes beyond just my own particular needs. ## Server event artifacts Store information from interrogation as client metadata. Storing data from the Custom.Generic.Client.Info is easy and almost does not require its own published artifact. However, the following two artifacts saves the user from having to write this themselves, and lets them configure their desired result from parameters. ### Server.Monitor.StoreClientHWInfo This is a relatively simple artifact, but it is very useful. The parameter SerialColumn lets you choose which metadata columns (defaults assuming Generic.Client.HW.Identification is used) to use for serial value, in a prioritised order. The artifact comes with common patterns to ignore. It could be extended with some transforms for manufacturers who store serials in a weird way. ### Server.Monitor.StoreClientInfo Given a specified interrogation artifact, this artifacts stores specified columns from specified sources as optionally renamed metadata values. Sources should only contain one row, but if they contain more, the first row is picked. Example use of the InfoMetadata parameter: | Source | Field | Alias | |------------------|----------------------------------------|-------------| | BasicInformation | OS | os | | BasicInformation | Architecture | arch | | ADStatus | DomainJoined | ad_joined | | MDATPHealth | MDATPHealth.edrMachineId | defender_id | | ComputerSetup | Details.computer setup.execution.timestamp | cs_executed | | ComputerSetup | Details.computer setup.git.hash | cs_version | (The ComputerSetup source here is an artifact fetching metadata produced by an ansible playbook run, telling us when it was last executed, and what version.) This artifact could probably be over-engineered to add some sort of transformation logic to support multiple rows. However, I think it is fine as it is. I am not sure if it is fixed, but attempting to store empty metadata created issues in some version, hence the conditional at the end. # Asset management Now that clients have various IDs set in metadata, it would be great to pull useful data (not stored on the endpoint) from external sources using APIs. Examples: - Fetch assigned owner and assigned use from an asset management system - Look up account name / account ID - Fetch vulnerabilities and security issues/recommendations - TODO: plenty of examples to fill Use cases where this additional information is fetched and then used in a notebook are topics for a chapter of its own. Right now we want to fetch data to store as additional indexed metadata. We will use one particular example: Fetching information about the assigned owner from an asset management system. Velociraptor will give us plenty of information about logged-in users, but I want to know who actually owns the computer. This can then be used to determine whether the last-logged in user shouldn't even be logged into the system in the first place. Other information, like assigned department, location, or other useful custom fields, could also be valuable. We will use Snipe-IT as our example asset management system. It is most likely not used in large enterprises, but is open-source and although a bit cumbersome, still powerful and useful. I have been planning to write one for Atlassian Assets. The system in question matters little, since they will have have some sort of REST API and very common concepts. We want to do the following: - Fetch assets (perhaps matching certain filters) - Match assets by serial - Transform some of the information - Synchronise selected data from the asset and save as client metadata We want this data updated, so the data will be synchronised either - On demand (running the server artifact) - Whenever the client is interrogated, giving us immediate updates for new clients (done by a server event artifact) - Periodically, so that changes in the assets are updated (e.g. new owner) (done by a server event artifact) The server event artifacts simply call the server artifacts. Since the server artifact has a lot of important parameters, and since we cannot expect the administrator to have to fill these in in three separate places, the following is done: - The server artifact saves its arguments to a server metadata value - The event artifacts fetches all arguments from the same value This may be elegant or ugly, depending on who you ask, but it works and certainly beats the alternative. ## Server artifacts ### Server.Assets.SnipeIT.Sync Querying Snipe-IT is trivial (at least with the QueryAPI server artifact helper). However, in order to create a user-friendly way to select which fields to synchronise, with support for custom fields and aliases, additional logic is needed. The artifact may be used in three ways: - Simply fetch all assets (matching filters) - Fetch assets with a serial found in Velociraptor clients - Synchronise client metadata The first two are "read-only" actions and can be used with notebook suggestions to - Find clients missing serial numbers - Find clients not in assets - Find assets not running Velociraptor ## Server event artifacts ### Server.Monitor.SnipeIT.Sync.Interrogation Update client metadata whenever it is interrogated. Shows what metadata changed (from/to). ## Server.Monitor.SnipeIT.Sync.Periodic Update client metadata at a specified interval. Shows what metadata changed (from/to). # Artifact table Complete artifact list (with dependencies, if any): | Name | Type | Deps. | Description | |------|------|--------|-------------| | Generic.Client.HW.Identification | Client | - | Pull hardware identification information from the system, like serial number and computer model | | Generic.Client.ADStatus | Client | - | Fetch Active Directory status | | Generic.Client.Defender.Health | Client | - | Get Microsoft Defender health status | | Server.Monitor.StoreClientHWInfo | Server | - | Store client HW info as metadata | | Server.Monitor.StoreClientInfo | Server | (HW.Identification) | Store client info as metadata | | Server.Utils.QueryAPI | Server | - | Query an HTTP API with login and pagination support | | Server.Assets.SnipeIT.Sync | Server | (QueryAPI) | Save key Snipe-IT asset information as client metadata | | Server.Monitor.SnipeIT.Sync.Interrogation | Server event | SnipeIT.Sync | Update asset information when client is interrogated | | Server.Monitor.SnipeIT.Sync.Periodic | Server event | SnipeIT.Sync | Update asset information regularly at a specified interval | --------- Co-authored-by: Mike Cohen <mike@velocidex.com>
1 parent 168ba66 commit b923b85

7 files changed

Lines changed: 876 additions & 1 deletion

File tree

.github/workflows/gh-pages.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ on:
99
jobs:
1010
deploy:
1111
runs-on: ubuntu-latest
12+
permissions:
13+
contents: write
14+
1215
steps:
1316
- uses: actions/checkout@v2
1417
with:
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: Generic.Client.ADStatus
2+
author: Andreas Misje – @misje
3+
description: |
4+
Get Active Directory join status from a computer.
5+
6+
`DomainJoined` (boolean) and `Domain` (uppercase) are returned on all
7+
operating systems. Additional columns depend on the OS:
8+
9+
- **Linux**: uses `realm list` (realm/sssd). Adds `_RealmType`,
10+
`_RealmName`, `AllowedUsers`, `AllowedGroups`.
11+
- **Windows**: uses `dsregcmd /status`. Adds `AzureAdJoined`,
12+
`EnterpriseJoined`, `NetBIOS`, `DeviceName`.
13+
- **macOS**: uses `dsconfigad -show`. Adds `DeviceName`, `AdminGroups`.
14+
15+
On Linux hosts without realm/sssd installed, `realm list` is absent and
16+
the artifact returns `DomainJoined: false`. Treat that as "unknown" if
17+
the deployment mixes hosts that may not have realm at all.
18+
19+
## Use as part of client interrogation
20+
21+
This artifact is designed to be called from a custom override of
22+
[`Generic.Client.Info`](/artifact_references/pages/generic.client.info/)
23+
so that AD status is collected on every interrogation. Combined with
24+
[`Server.Monitor.StoreClientInfo`](/exchange/artifacts/pages/server.monitor.storeclientinfo/),
25+
the result can be saved as searchable client metadata such as
26+
`ad_joined: true` or `ad_domain: AD.EXAMPLE.ORG`. See
27+
[How can I automatically add & update client metadata?](/knowledge_base/tips/automating_metadata/)
28+
for an in-depth walkthrough about how to save interrogation data as
29+
client metadata.
30+
31+
#metadata #activedirectory
32+
33+
type: CLIENT
34+
35+
implied_permissions:
36+
- EXECVE
37+
38+
sources:
39+
- query: |
40+
LET Info <= SELECT OS
41+
FROM info()
42+
43+
LET SplitString(String) = filter(regex='.+',
44+
list=split(string=String, sep=', ?'))
45+
46+
LET LinuxStatus = SELECT
47+
parse_string_with_regex(
48+
string=Stdout,
49+
regex=(''' +type: +(?P<_RealmType>.*)''', ''' +realm-name: +(?P<_RealmName>.*)''', ''' +domain-name: +(?P<Domain>.*)''', ''' +permitted-logins: +(?P<AllowedUsers>.*)''', ''' +permitted-groups: +(?P<AllowedGroups>.*)''', )) AS ADStatus
50+
FROM execve(
51+
argv=('realm', 'list'))
52+
53+
LET WindowsStatus = SELECT
54+
parse_string_with_regex(
55+
string=Stdout,
56+
regex=('''(?s)Device State.+AzureAdJoined *: *(?P<AzureAdJoined>\S+)''', '''(?s)Device State.+EnterpriseJoined *: *(?P<EnterpriseJoined>\S+)''', '''(?s)Device State.+DomainJoined *: *(?P<DomainJoined>\S+)''', '''(?s)Device State.+DomainName *: *(?P<NetBIOS>\S+)''', '''(?s)Device State.+Device Name *: *(?P<DeviceName>\S+)''', )) AS ADStatus
57+
FROM execve(
58+
argv=('dsregcmd', '/status'))
59+
60+
LET DarwinStatus = SELECT
61+
parse_string_with_regex(
62+
string=Stdout,
63+
regex=('''Active Directory Domain += +(?P<Domain>\S+)''', '''Computer Account += +(?P<DeviceName>\S+)\$''', ''' +Allowed admin groups += +(?P<AdminGroups>.+)''', )) AS ADStatus
64+
FROM execve(
65+
argv=('dsconfigad', '-show'))
66+
67+
LET ADDict = SELECT
68+
ADStatus + dict(Domain=upcase(string=ADStatus.Domain)) AS ADStatus
69+
FROM switch(
70+
linux={
71+
SELECT *
72+
FROM if(
73+
condition=Info[0].OS = 'linux',
74+
then={
75+
SELECT ADStatus + dict(
76+
DomainJoined=ADStatus != dict(),
77+
AllowedUsers=SplitString(String=ADStatus.AllowedUsers),
78+
AllowedGroups=SplitString(String=ADStatus.AllowedGroups)) AS ADStatus
79+
FROM LinuxStatus
80+
})
81+
},
82+
windows={
83+
SELECT
84+
*
85+
FROM if(
86+
condition=Info[0].OS = 'windows',
87+
then={
88+
SELECT
89+
ADStatus + dict(
90+
AzureAdJoined=ADStatus.AzureAdJoined = 'YES',
91+
EnterpriseJoined=ADStatus.EnterpriseJoined = 'YES',
92+
DomainJoined=ADStatus.DomainJoined = 'YES') +
93+
parse_string_with_regex(
94+
string=ADStatus.DeviceName,
95+
regex='''^(?P<DeviceName>[^.]+)\.(?P<Domain>\S+)$''') AS ADStatus
96+
FROM WindowsStatus
97+
})
98+
},
99+
darwin={
100+
SELECT
101+
*
102+
FROM if(
103+
condition=Info[0].OS = 'darwin',
104+
then={
105+
SELECT
106+
ADStatus + dict(
107+
DomainJoined=ADStatus != dict(),
108+
AdminGroups=SplitString(
109+
String=ADStatus.AdminGroups)) AS ADStatus
110+
FROM DarwinStatus
111+
})
112+
})
113+
114+
SELECT *
115+
FROM foreach(row=ADDict, column='ADStatus')
Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
name: Generic.Client.Defender.Health
2+
author: Andreas Misje – @misje
3+
description: |
4+
Get Microsoft Defender for Endpoint (MDATP) health and configuration.
5+
6+
MDATP is Microsoft's EDR product for Windows, macOS and Linux. This
7+
artifact retrieves all available agent status and configuration via the
8+
platform-native interface: `mdatp health --output json` on Linux and
9+
macOS, `Get-MpComputerStatus` through
10+
[`Windows.System.PowerShell`](/artifact_references/pages/windows.system.powershell/)
11+
on Windows.
12+
13+
A single row is returned with a dict called `MDATPHealth` whose
14+
shape differs across platforms. Linux and macOS are nearly
15+
identical; Windows differs significantly. Platform-specific extras:
16+
17+
Linux:
18+
19+
- `behaviorMonitoring`
20+
- `supplementaryEventsSubsystem`
21+
22+
macOS:
23+
24+
- `deviceControlEnforcementLevel`
25+
- `ecsConfigurationIds`
26+
- `fullDiskAccessEnabled`
27+
- `networkEventsSubsystem`
28+
- `tamperProtection`
29+
- `troubleshootingMode`
30+
31+
Windows values are parsed from text output and arrive as strings
32+
(e.g. the literal `'True'`, not the boolean `true`). Linux and macOS
33+
values are properly typed JSON. The notebook suggestion **Common
34+
MDATP health information** normalises a useful subset across all
35+
three platforms.
36+
37+
## Use as part of client interrogation
38+
39+
Although useful on its own to gather MDATP status from hosts, this
40+
artifact can also be called from a custom override of
41+
[`Generic.Client.Info`](/artifact_references/pages/generic.client.info/).
42+
Combined with
43+
[`Server.Monitor.StoreClientInfo`](/exchange/artifacts/pages/server.monitor.storeclientinfo/),
44+
selected fields can be saved as client metadata. In particular,
45+
`edrMachineId` can be used to identify the client in MDATP, and tie
46+
the client ID to machines in Microsoft Graph API queries. See
47+
[How can I automatically add & update client metadata?](/knowledge_base/tips/automating_metadata/)
48+
for an in-depth walkthrough about how to save interrogation data as
49+
client metadata.
50+
51+
#mdatp #metadata
52+
53+
type: CLIENT
54+
55+
implied_permissions:
56+
- EXECVE
57+
58+
sources:
59+
- query: |
60+
LET Info <= SELECT OS
61+
FROM info()
62+
63+
SELECT *
64+
FROM if(
65+
condition=Info[0].OS = 'linux',
66+
then={
67+
SELECT parse_json(data=Stdout) AS MDATPHealth
68+
FROM execve(argv=['/usr/bin/mdatp', 'health', '--output', 'json'])
69+
},
70+
else=if(
71+
condition=Info[0].OS = 'darwin',
72+
then={
73+
SELECT parse_json(data=Stdout) AS MDATPHealth
74+
FROM execve(argv=['/usr/local/bin/mdatp', 'health', '--output', 'json'])
75+
},
76+
else=if(
77+
condition=Info[0].OS = 'windows',
78+
then={
79+
SELECT
80+
to_dict(item={
81+
SELECT _key,
82+
_value
83+
FROM parse_records_with_regex(
84+
file=Stdout,
85+
accessor='data',
86+
regex='''^\s*(?P<_key>\S+)\s+:\s+(?P<_value>[^\r\n]+)''')
87+
}) + dict(
88+
edrMachineId=read_file(
89+
accessor='reg',
90+
filename='''HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Advanced Threat Protection\senseId''')) AS MDATPHealth
91+
FROM Artifact.Windows.System.PowerShell(
92+
Command='Get-MpComputerStatus')
93+
})))
94+
95+
notebook:
96+
- name: Common MDATP health information
97+
type: vql_suggestion
98+
template: |
99+
/*
100+
# Common MDATP health information
101+
*/
102+
LET ColumnTypes <= dict(`ClientId`='client')
103+
104+
LET S = scope()
105+
106+
LET Result <= SELECT
107+
ClientId,
108+
S.Fqdn || client_info(client_id=ClientId).os_info.fqdn AS Fqdn,
109+
S.MDATPHealth || dict() AS D
110+
FROM source()
111+
112+
SELECT *
113+
FROM foreach(
114+
row=Result,
115+
query={
116+
SELECT *
117+
FROM if(
118+
condition='AMEngineVersion' IN D,
119+
then={
120+
SELECT
121+
ClientId,
122+
Fqdn,
123+
D.edrMachineId AS edrMachineId,
124+
D.AMProductVersion AS appVersion,
125+
D.AMEngineVersion AS engineVersion,
126+
D.AntivirusSignatureVersion AS definitionsVersion,
127+
timestamp(string=D.AntivirusSignatureLastUpdated) AS definitionsUpdated,
128+
if(condition=D.DefenderSignaturesOutOfDate = NULL,
129+
then=NULL,
130+
else=D.DefenderSignaturesOutOfDate != 'False') AS definitionsUpToDate,
131+
if(condition=D.BehaviorMonitorEnabled = NULL,
132+
then=NULL,
133+
else=D.BehaviorMonitorEnabled = 'True') AS behaviorMonitoringEnabled,
134+
if(condition=D.RealTimeProtectionEnabled = NULL,
135+
then=NULL,
136+
else=D.RealTimeProtectionEnabled = 'True') AS realTimeProtectionEnabled,
137+
if(
138+
condition=D.IsTamperProtected = NULL,
139+
then=NULL,
140+
else=D.IsTamperProtected = 'True') AS tamperProtectionEnabled,
141+
if(
142+
condition=D.TroubleShootingMode = NULL,
143+
then=NULL,
144+
else=D.TroubleShootingMode != 'Disabled') AS troubleshootingModeEnabled
145+
FROM scope()
146+
},
147+
else={
148+
SELECT
149+
ClientId,
150+
Fqdn,
151+
D.edrMachineId AS edrMachineId,
152+
D.appVersion AS appVersion,
153+
D.engineVersion AS engineVersion,
154+
D.definitionsVersion AS definitionsVersion,
155+
timestamp(
156+
epoch=D.definitionsUpdated) AS definitionsUpdated,
157+
if(
158+
condition=D.definitionsStatus.`$type` = NULL,
159+
then=NULL,
160+
else=D.definitionsStatus.`$type` = 'upToDate') AS definitionsUpToDate,
161+
// Not available in Darwin:
162+
D.behaviorMonitoring.displayValue AS behaviorMonitoringEnabled,
163+
D.realTimeProtectionEnabled.value AS realTimeProtectionEnabled,
164+
// Not available in Linux:
165+
D.tamperProtection.displayValue AS tamperProtectionEnabled,
166+
// Not available in Linux:
167+
D.troubleshootingMode AS troubleshootingModeEnabled
168+
FROM scope()
169+
})
170+
})

0 commit comments

Comments
 (0)