Website performance/reliability issues

[tresf]

Due to an influx of high-resource requests against the website, the website keeps going down. I've temporarily disabled the forums (again) to keep the main website alive... Some thoughts/tasks on this topic...

• Slow or prevent the bots constantly crawling our PHP pages, causing a lot of database requests
• Implement a more performant way to handle these requests
• Investigate some storms that have occurred by non-bots (possible DDoS)
• Review LSP code/database efficiency for long-running processes
• Review phpBB plugins for helping with long-running processes

HIstorically, we've allowed all websites to crawl our page as it helps people get answers they need quickly however the bots have gotten out of control... Here's a 12-hour snapshot of our nginx access logs....

Any additional help is greatly appreciated. This is a volunteer-lead website and resources are limited.

sudo cat /var/log/nginx/access.log |grep bingbot |wc -l

bot          hits
-----------------
bingbot     10717
PetalBot     7206
Amazonbot    5517
Googlebot    5085
SemrushBot   4670
MJ12bot      4114
AwarioBot    3899
UptimeRobot  3550
ChatGPT      1614
AhrefsBot     952
YandexBot     497
PerplexityBot 181
DuckDuckBot    24
DiscordBot     21
trendictionbot 18

[tresf]

At time of writing this, ChatGPTBot, YandexBot, PerplexityBot, DuckDuckBot, DiscordBot, trendictionbot all are relatively sane, clocking in at less than 1,000 requests in 12 hours (less than 2 requests per minute) whereas bing is 10x that and deserves to be blocked.

There's some argument to be made that we need to throttle this via our robots.txt file, but I don't want to kill all crawlers from accessing our /forum and /lsp, so this is a bandaid, as the PR clearly describes:

• Nuclear bot thwarting #418

... I have added a crawl delay to our robots.txt, but a quick search shows many bots don't honor this, so we'll have to see if re-enabling the more aggressive bots will honor this or not.

[tresf]

At time of writing this, ChatGPTBot

That didn't age well... once the forums were back online we got hammered. Killing both GPTBot as well as ChatGPTBot.

[headquarter8302]

Are the bot still hitting our servers? I recommend unblocking Googlebot and Duckduckbot since both also are responsible for webpage crawling. #418 has been committed for at least 4 days at this point, and I'm not sure how long these search engines keep unindexable websites on their results page, even with backlinks.

Except Bingbot, holy crap Microsoft needs to control their bot

[tresf]

Except Bingbot, holy crap Microsoft needs to control their bot

Google's the 4th highest offender.

Are the bot still hitting our servers?

I think so, but it's harder to detect if they don't identify themselves. I agree that we're punishing the bots to allow other nefarious traffic, but I'm not sure what else to do. Advice welcome here. Here's a snippet from just now...

47.239.155.134 - - [27/May/2025:13:52:40 +0000] "HEAD /forum/viewtopic.php?p=85931&sid=<redacted> HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2768.82 Safari/537.36"
47.239.156.153 - - [27/May/2025:13:52:40 +0000] "HEAD /forum/viewtopic.php?sid=<redacted>&t=37557 HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3816.80 Safari/537.36"
47.239.155.78  - - [27/May/2025:13:52:40 +0000] "HEAD /forum/index.php?sid=<redacted> HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3465.108 Safari/537.36"
47.239.156.85  - - [27/May/2025:13:52:40 +0000] "HEAD /forum/app.php/rules?sid=<redacted> HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3459.150 Safari/537.36"
47.239.155.175 - - [27/May/2025:13:52:40 +0000] "HEAD /forum/viewtopic.php?sid=<redacted>&t=36990 HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3183.100 Safari/537.36"
47.239.155.170 - - [27/May/2025:13:52:40 +0000] "HEAD /forum/viewforum.php?f=5&sid=<redacted> HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.2558.95 Safari/537.36"
47.239.156.85  - - [27/May/2025:13:52:40 +0000] "HEAD /forum/viewforum.php?f=19&sid=<redacted> HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.2413.150 Safari/537.36"
47.239.155.134 - - [27/May/2025:13:52:40 +0000] "GET  /forum/viewtopic.php?p=85931&sid=<redacted> HTTP/1.1" 200 11316 "-" "Mozilla/5.0 (Windows NT 6.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2768.82 Safari/537.36"
47.239.155.78  - - [27/May/2025:13:52:40 +0000] "GET  /forum/index.php?sid=<redacted> HTTP/1.1" 200 8220 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3465.108 Safari/537.36"
47.239.156.153 - - [27/May/2025:13:52:40 +0000] "GET  /forum/viewtopic.php?sid=<redacted>&t=37557 HTTP/1.1" 200 8845 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3816.80 Safari/537.36"
47.239.156.85  - - [27/May/2025:13:52:40 +0000] "GET  /forum/app.php/rules?sid=<redacted> HTTP/1.1" 200 5452 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3459.150 Safari/537.36"
47.239.155.175 - - [27/May/2025:13:52:40 +0000] "GET  /forum/viewtopic.php?sid=<redacted>&t=36990 HTTP/1.1" 200 12470 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3183.100 Safari/537.36"
47.239.155.170 - - [27/May/2025:13:52:40 +0000] "GET  /forum/viewforum.php?f=5&sid=<redacted> HTTP/1.1" 200 12599 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.2558.95 Safari/537.36"
47.239.156.85  - - [27/May/2025:13:52:40 +0000] "GET  /forum/viewforum.php?f=19&sid=<redacted> HTTP/1.1" 200 12383 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.2413.150 Safari/537.36"

[tresf]

After watching the behavior for a bit and reading some other websites' struggles, I've decided to take a more targeted approach.

Starting with 9c9928d, I've blocked all IP addresses registered to Alibaba cloud's ASN. This is likely to block a few legitimate users (e.g. VPN, etc) but that's just something we'll have to live with.

I noticed that most bots can't help but access pages that a user wouldn't normally visit, so by filtering on these pages I can pretty quickly identify a bot. Here's a quick command:

sudo cat /var/log/nginx/access.log |grep faq |grep " 200 " |awk '{print $1}'|sort|uniq |wc -l

What this does:

• Look for requests to our "frequently asked questions" page "faq"
• Look for successful page hits (filter out requests we've already filtered)
• Get the first field in the log, which is the IP address
• Sort the IP addresses and only return unique values

This returns 2,300 unique IPs since midnight of last night 963 of which are part of the 47.x.x.x range. This is why blocking ALL of Alibaba cloud IPs is essential to monitor this problem more closely. Fortunately, Alibaba must register these under its ASN, which can be found here: https://asn.ipinfo.app/list/AS45899. IPInfo kindly offers an nginx blacklist file, which is what was committed in 9c9928d.

Now, I can go through the rest of the IP ranges using https://ipinfo.io/<offending_ip>, and see how widespread the issue is by trying to figure out how often they're hammering us. At time of writing this, some other offenders:

• AS52974 - Henet Telecomunicacoes Ltda
• AS53118 - NETLINE TELECOM
• AS45899 - VNPT Corp

(just to name a few)

I'll continue monitoring for abusive IP ranges and add block files as-needed. Once this is done, we can likely re-enable many of the well-known crawlers so that we don't lose our Google/SEO indexing.

[tresf]

I'm not sure how long these search engines keep unindexable websites on their results page, even with backlinks.

Google's definitely ornery :D

Fortunately with the latest blacklists, we can probably turn some of these bots back on.

[tresf]

After the latest round of blacklists, we're down to ~600 unique IP addresses in a 12-hour period that are hitting typical bot pages like FAQ or delete_cookies. This is a huge improvement, so it seems that blocking the few major offenders has fixed the underlying issue.

Of the 600 or so IP addresses that are remaining, there's not an easily found pattern, most belong to ISPs and from what I can tell, none of these are so rampant that they should be blocked.

sudo cat /var/log/nginx/access.log |grep " 200 "|grep faq|awk '{print $1}'|sort|uniq|wc -l
630 # <--- returns this value

Now I'm going to re-enable some bot crawling that was blacklisted previously. I'll start with Google.