Skip to content

Lots More Bots #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
donatj wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Lots More Bots #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d14755a
More modern mastodon
donatj May 14, 2025
cbe0665
Add support and tests for many more bots
donatj May 11, 2025
8c8b368
Simplify the bot dance
donatj May 14, 2025
11f241f
Simplify regex a little
donatj May 14, 2025
e482d47
Update Browser Constants
donatj May 15, 2025
e99f4fc
Merge commit 'e482d47' into bot-attack
donatj May 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ Predefined helper constants from `donatj\UserAgent\Browsers`
| `Browsers::ADSBOT_GOOGLE` | AdsBot-Google |
| `Browsers::ANDROID_BROWSER` | Android Browser |
| `Browsers::APPLEBOT` | Applebot |
| `Browsers::ARCHIVE_ORG_BOT` | archive.org_bot |
| `Browsers::BAIDUSPIDER` | Baiduspider |
| `Browsers::BINGBOT` | bingbot |
| `Browsers::BLACKBERRY_BROWSER` | BlackBerry Browser |
Expand All @@ -172,13 +173,16 @@ Predefined helper constants from `donatj\UserAgent\Browsers`
| `Browsers::CHATGPT_USER` | ChatGPT-User |
| `Browsers::CHROME` | Chrome |
| `Browsers::CURL` | curl |
| `Browsers::DISCORDBOT` | Discordbot |
| `Browsers::EDGE` | Edge |
| `Browsers::FACEBOOKEXTERNALHIT` | facebookexternalhit |
| `Browsers::FEEDVALIDATOR` | FeedValidator |
| `Browsers::FIREFOX` | Firefox |
| `Browsers::GOOGLEBOT` | Googlebot |
| `Browsers::GOOGLEBOT_IMAGE` | Googlebot-Image |
| `Browsers::GOOGLEBOT_VIDEO` | Googlebot-Video |
| `Browsers::GOOGLE_READ_ALOUD` | Google-Read-Aloud |
| `Browsers::GOOGLE_SAFETY` | Google-Safety |
| `Browsers::GPTBOT` | GPTBot |
| `Browsers::HEADLESSCHROME` | HeadlessChrome |
| `Browsers::IEMOBILE` | IEMobile |
Expand All @@ -195,6 +199,7 @@ Predefined helper constants from `donatj\UserAgent\Browsers`
| `Browsers::OAI_SEARCHBOT` | OAI-SearchBot |
| `Browsers::OCULUSBROWSER` | OculusBrowser |
| `Browsers::OPERA` | Opera |
| `Browsers::PINTERESTBOT` | Pinterestbot |
| `Browsers::PUFFIN` | Puffin |
| `Browsers::SAFARI` | Safari |
| `Browsers::SAILFISHBROWSER` | SailfishBrowser |
Expand All @@ -207,10 +212,16 @@ Predefined helper constants from `donatj\UserAgent\Browsers`
| `Browsers::UC_BROWSER` | UC Browser |
| `Browsers::VALVE_STEAM_TENFOOT` | Valve Steam Tenfoot |
| `Browsers::VIVALDI` | Vivaldi |
| `Browsers::WELLKNOWNBOT` | WellKnownBot |
| `Browsers::WGET` | Wget |
| `Browsers::WHALE` | Whale |
| `Browsers::WORDPRESS` | WordPress |
| `Browsers::WPBOT` | wpbot |
| `Browsers::YANDEX` | Yandex |
| `Browsers::YANDEXBOT` | YandexBot |
| `Browsers::YANDEXBOT` | YandexBot |
| `Browsers::YANDEXIMAGES` | YandexImages |
| `Browsers::YANDEXMOBILEBOT` | YandexMobileBot |
| `Browsers::YANDEXRCA` | YandexRCA |
| `Browsers::YANDEXUSERPROXY` | YandexUserproxy |

More information is available at [Donat Studios](https://donatstudios.com/PHP-Parser-HTTP_USER_AGENT).
42 changes: 41 additions & 1 deletion bin/constant_generator.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,49 @@
}
}

$browserExclusions = [
'ADKERNELTOPICCRAWLER',
'AHREFSBOT',
'AMAZONADBOT',
'AMAZONBOT',
'AWARIOBOT',
'BAIDUSPIDER_RENDER',
'BARKROWLER',
'BLEXBOT',
'BRAVEBOT',
'CENSYSINSPECT',
'COCCOCBOT_IMAGE',
'COCCOCBOT_WEB',
'DATAFORSEOBOT',
'DOTBOT',
'DVBOT',
'EV_CRAWLER',
'FLIPBOARDPROXY',
'HEEXYBOT',
'HUBSPOT_DOMAIN_CHECK',
'LEIKIBOT',
'MINIFLUX',
'MODATSCANNER',
'MOJEEKBOT',
'PERPLEXITY_USER',
'PERPLEXITYBOT',
'PETALBOT',
'PROXIMIC',
'SEMRUSHBOT',
'SEMRUSHBOT_SI',
'SEZNAMBOT',
'SMTBOT',
'SPIDERLING',
'SURDOTLYBOT',
'YETI'
];

$browserBody = "{$header}namespace donatj\UserAgent;\n\ninterface Browsers {\n\n";
$maxKey = max(array_map('strlen', array_keys($browsers)));
foreach( $browsers as $const => $val ) {
if( in_array($const, $browserExclusions, true) ) {
continue;
}
$browserBody .= sprintf("\tconst %-{$maxKey}s = %s;\n", $const, var_export(key($val), true));
}
$browserBody .= "\n}\n\n";
Expand All @@ -74,4 +114,4 @@
$platformBody .= "\n}\n\n";

file_put_contents(__DIR__ . '/../src/UserAgent/Browsers.php', $browserBody);
file_put_contents(__DIR__ . '/../src/UserAgent/Platforms.php', $platformBody);
file_put_contents(__DIR__ . '/../src/UserAgent/Platforms.php', $platformBody);
6 changes: 5 additions & 1 deletion bin/user_agent_sorter.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
$jsonfile = __DIR__ . '/../Tests/user_agents.dist.json';

$uas = json_decode(file_get_contents($jsonfile), true);
if( $uas === null ) {
echo "Failed to decode JSON\n";
die(1);
}

foreach( $uas as $key => &$val ) {
$val['key'] = $key;
Expand Down Expand Up @@ -97,4 +101,4 @@ function compare_version( $a, $b ) {
}

return $value;
}
}
115 changes: 63 additions & 52 deletions src/UserAgent/Browsers.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,58 +6,69 @@

interface Browsers {

const ADSBOT_GOOGLE = 'AdsBot-Google';
const ANDROID_BROWSER = 'Android Browser';
const APPLEBOT = 'Applebot';
const BAIDUSPIDER = 'Baiduspider';
const BINGBOT = 'bingbot';
const BLACKBERRY_BROWSER = 'BlackBerry Browser';
const BROWSER = 'Browser';
const BUNJALLOO = 'Bunjalloo';
const CAMINO = 'Camino';
const CHATGPT_USER = 'ChatGPT-User';
const CHROME = 'Chrome';
const CURL = 'curl';
const EDGE = 'Edge';
const FACEBOOKEXTERNALHIT = 'facebookexternalhit';
const FEEDVALIDATOR = 'FeedValidator';
const FIREFOX = 'Firefox';
const GOOGLEBOT = 'Googlebot';
const GOOGLEBOT_IMAGE = 'Googlebot-Image';
const GOOGLEBOT_VIDEO = 'Googlebot-Video';
const GPTBOT = 'GPTBot';
const HEADLESSCHROME = 'HeadlessChrome';
const IEMOBILE = 'IEMobile';
const IMESSAGEBOT = 'iMessageBot';
const KINDLE = 'Kindle';
const LYNX = 'Lynx';
const MASTODON = 'Mastodon';
const MIDORI = 'Midori';
const MIUIBROWSER = 'MiuiBrowser';
const MSIE = 'MSIE';
const MSNBOT_MEDIA = 'msnbot-media';
const NETFRONT = 'NetFront';
const NINTENDOBROWSER = 'NintendoBrowser';
const OAI_SEARCHBOT = 'OAI-SearchBot';
const OCULUSBROWSER = 'OculusBrowser';
const OPERA = 'Opera';
const PUFFIN = 'Puffin';
const SAFARI = 'Safari';
const SAILFISHBROWSER = 'SailfishBrowser';
const SAMSUNGBROWSER = 'SamsungBrowser';
const SILK = 'Silk';
const SLACKBOT = 'Slackbot';
const TELEGRAMBOT = 'TelegramBot';
const TIZENBROWSER = 'TizenBrowser';
const TWITTERBOT = 'Twitterbot';
const UC_BROWSER = 'UC Browser';
const VALVE_STEAM_TENFOOT = 'Valve Steam Tenfoot';
const VIVALDI = 'Vivaldi';
const WGET = 'Wget';
const WHALE = 'Whale';
const WORDPRESS = 'WordPress';
const YANDEX = 'Yandex';
const YANDEXBOT = 'YandexBot';
const ADSBOT_GOOGLE = 'AdsBot-Google';
const ANDROID_BROWSER = 'Android Browser';
const APPLEBOT = 'Applebot';
const ARCHIVE_ORG_BOT = 'archive.org_bot';
const BAIDUSPIDER = 'Baiduspider';
const BINGBOT = 'bingbot';
const BLACKBERRY_BROWSER = 'BlackBerry Browser';
const BROWSER = 'Browser';
const BUNJALLOO = 'Bunjalloo';
const CAMINO = 'Camino';
const CHATGPT_USER = 'ChatGPT-User';
const CHROME = 'Chrome';
const CURL = 'curl';
const DISCORDBOT = 'Discordbot';
const EDGE = 'Edge';
const FACEBOOKEXTERNALHIT = 'facebookexternalhit';
const FEEDVALIDATOR = 'FeedValidator';
const FIREFOX = 'Firefox';
const GOOGLEBOT = 'Googlebot';
const GOOGLEBOT_IMAGE = 'Googlebot-Image';
const GOOGLEBOT_VIDEO = 'Googlebot-Video';
const GOOGLE_READ_ALOUD = 'Google-Read-Aloud';
const GOOGLE_SAFETY = 'Google-Safety';
const GPTBOT = 'GPTBot';
const HEADLESSCHROME = 'HeadlessChrome';
const IEMOBILE = 'IEMobile';
const IMESSAGEBOT = 'iMessageBot';
const KINDLE = 'Kindle';
const LYNX = 'Lynx';
const MASTODON = 'Mastodon';
const MIDORI = 'Midori';
const MIUIBROWSER = 'MiuiBrowser';
const MSIE = 'MSIE';
const MSNBOT_MEDIA = 'msnbot-media';
const NETFRONT = 'NetFront';
const NINTENDOBROWSER = 'NintendoBrowser';
const OAI_SEARCHBOT = 'OAI-SearchBot';
const OCULUSBROWSER = 'OculusBrowser';
const OPERA = 'Opera';
const PINTERESTBOT = 'Pinterestbot';
const PUFFIN = 'Puffin';
const SAFARI = 'Safari';
const SAILFISHBROWSER = 'SailfishBrowser';
const SAMSUNGBROWSER = 'SamsungBrowser';
const SILK = 'Silk';
const SLACKBOT = 'Slackbot';
const TELEGRAMBOT = 'TelegramBot';
const TIZENBROWSER = 'TizenBrowser';
const TWITTERBOT = 'Twitterbot';
const UC_BROWSER = 'UC Browser';
const VALVE_STEAM_TENFOOT = 'Valve Steam Tenfoot';
const VIVALDI = 'Vivaldi';
const WELLKNOWNBOT = 'WellKnownBot';
const WGET = 'Wget';
const WHALE = 'Whale';
const WORDPRESS = 'WordPress';
const WPBOT = 'wpbot';
const YANDEX = 'Yandex';
const YANDEXBOT = 'YandexBot';
const YANDEXIMAGES = 'YandexImages';
const YANDEXMOBILEBOT = 'YandexMobileBot';
const YANDEXRCA = 'YandexRCA';
const YANDEXUSERPROXY = 'YandexUserproxy';

}

32 changes: 26 additions & 6 deletions src/UserAgentParser.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,10 @@ function parse_user_agent( $u_agent = null ) {
}

preg_match_all(<<<'REGEX'
%(?P<browser>Camino|Kindle(\ Fire)?|Firefox|Iceweasel|IceCat|Safari|MSIE|Trident|AppleWebKit|
%(?P<prev>.)?(?P<browser>Camino|Kindle(\ Fire)?|Firefox|Iceweasel|IceCat|Safari|MSIE|Trident|AppleWebKit|
TizenBrowser|(?:Headless)?Chrome|YaBrowser|Vivaldi|IEMobile|Opera|OPR|Silk|Midori|(?-i:Edge)|EdgA?|CriOS|UCBrowser|Puffin|
OculusBrowser|SamsungBrowser|SailfishBrowser|XiaoMi/MiuiBrowser|YaApp_Android|Whale|
Baiduspider|Applebot|Facebot|Googlebot|YandexBot|bingbot|Lynx|Version|Wget|curl|ChatGPT-User|GPTBot|OAI-SearchBot|
Applebot|Facebot|Googlebot|YandexBot|bingbot|Lynx|Version|
Valve\ Steam\ Tenfoot|Mastodon|
NintendoBrowser|PLAYSTATION\ (?:\d|Vita)+)
\)?;?
Expand All @@ -109,11 +109,31 @@ function parse_user_agent( $u_agent = null ) {
, $u_agent, $result);

// If nothing matched, return null (to avoid undefined index errors)
if( !isset($result[BROWSER][0], $result[BROWSER_VERSION][0]) ) {
if( preg_match('%^(?!Mozilla)(?P<browser>[A-Z0-9\-]+)([/ :](?P<version>[0-9A-Z.]+))?%ix', $u_agent, $result) ) {
return [ PLATFORM => $platform ?: null, BROWSER => $result[BROWSER], BROWSER_VERSION => empty($result[BROWSER_VERSION]) ? null : $result[BROWSER_VERSION] ];
}
if( !isset($result[BROWSER][0], $result[BROWSER_VERSION][0])
&& preg_match('%^(?!Mozilla)(?P<browser>[A-Z0-9\-]+)([/ :](?P<version>[0-9A-Z.]+))?%ix', $u_agent, $g_result) ) {
return [ PLATFORM => $platform, BROWSER => $g_result[BROWSER], BROWSER_VERSION => empty($g_result[BROWSER_VERSION]) ? null : $g_result[BROWSER_VERSION] ];
}

if(
(
empty($result[BROWSER][0])
|| ($result['prev'][0] !== '')
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition will throw an 'Undefined index' notice if $result['prev'][0] doesn't exist. The regex on line 99 uses (?P<prev>.)? with a ? quantifier, meaning the capture group may not be set. This condition should use isset() or !empty() to check for existence before accessing the array element. Suggested fix: || (isset($result['prev'][0]) && $result['prev'][0] !== '')

Suggested change
|| ($result['prev'][0] !== '')
|| (isset($result['prev'][0]) && $result['prev'][0] !== '')

Copilot uses AI. Check for mistakes.
)
&& preg_match(<<<'REGEX'
%[(;]\s*(?P<browser>[^(/;]+)
(?:[:/ ]v?(?P<version>[0-9A-Z.]+)[^;)\s]*)?
;?(?:\s*robot;)?\s*\+https?:%x
REGEX
, $u_agent, $bot_result)
) {
return [
PLATFORM => $platform,
BROWSER => trim($bot_result['browser']),
BROWSER_VERSION => empty($bot_result['version']) ? null : $bot_result['version'],
];
}

if( !isset($result[BROWSER][0], $result[BROWSER_VERSION][0]) ) {
return $return;
}

Expand Down
Loading