Stability Improvements (#489)

1. Issue: Broken QueueThread joining logic which relies on thread being
`joinable` which is unrelated to doing `join`, leading to the case where
the network thread crashes when it reaches the end of scope, due to a
`std::terminate` in the dtor of `std::thread` (throws on destruction
when unjoined). Fix: Using `std::jthread` which uses RAII to join on
destruction.
2. Issue: Slow hardware or overloaded server can cause a client to
disconnect and socket to close before `TNetwork::DisconnectClient` is
called, but `IsDisconnected()` will still be false. In this case,
`.remote_endpoint()` can fail, which throws an exception, which is not
caught and thus `std::terminate`s the server. I observed this, even
though it was only in extremely contrived scenarios, it still happened
and crashed the server. Fix: Wrapping the whole block in a try/catch.
Alternate fix would have been to pass an `error_code` but the result is
the same. This fix is not quite enough though, a later fix resolves the
remaining issue that this causes the `mClientMap` to ignore the
disconnection. Also, all paths that do `if (c.IsDisconnected())` will
fail to decrement the `mClientMap`, which isn't always the right
behavior afaik. Fixed in another fix though.
3. Issue: Connection limiting ("DDoS protection") is broken as
exceptions cause it to not decrement (and slowly fill up the
`mClientMap`) in special cases. Mutexes are locked and unlocked manually
which can (and will) lead to cases where the mutex is locked, an
exception is thrown in the subsequent line (`address().to_string()` can
throw, same with `.remote_endpoint()`, both of which are being called in
the locked context without RAII unlocking). Fix: Replace manual map and
mutex handling with a new class, `TConnectionLimiter`, and an associated
"Guard" object `TConnectionLimiter::TGuard` which uses RAII to correctly
keep track of IP-and-connection-count associations, the way the previous
code was trying to do. This works across exceptions and other weird
issues. Each connection's main thread now owns a guard, which, on
destruction, decrements the counter. This way both the per-IP limits as
well as the global limits are enforced. Also added some stats about this
to the `status` command to ensure that server owners can observe this in
action.
4. Issue: `.address().to_string()` can throw and is called even if
`accept` failed, in `TNetwork::TCPServerMain`'s accept loop. I didn't
observe this and it didn't cause crashes, but I was touching that part
anyway. Fix: Explicitly handle the error case first, then get the IP,
etc.
5. Issue: `ReadWithTimeout` spawned a new async context for each read,
and then ran that context's event loop in a new thread. This means that,
not only did every one of those reads SPAWN A THREAD(!!!), it also
started an io context, which gets an fd, so this made DDoS arguably more
effective, not less.
6. Issue: Client disconnect can race due to being done on multiple
threads (TOCTOU bug). For example `Looper` and a normal disconnect call
can happen at the same time, because they check for `is_open` which can
be true, and then change to false right after the check, causing a
segfault in asio internals. Fix: Added an atomic compare-and-swap (CAS)
mechanic that acts like a lock for the socket disconnect/close, and
adjusted other places that checked `is_open`.
8. Issue: Lua panic calls the panic handler, and if the error supplied
in the panic is not a string, or is otherwise invalid, it will trigger
another panic within the panic handler. This continues and eventually
crashes the program in one of many fun ways. Fix: Use raw lua functions
to check if the top of the stack is a string, and only then print it,
otherwise print that there was a panic and leave it at that.
9. Issue: `error()` crashes the server, due to `sol::error`'s
constructor expecting a `std::string` (`lua_tostring` or `__tostring`
meta method), which doesn't exist if the error is, for example, `nil`. I
reported this to sol2, but it might be an issue only in this older
version we're using. Fix: Fixed as part of the next issue:
10. Issue: `TLuaResult` was used/accessed from multiple threads,
including `sol::object` accessed from multiple threads. This lead to
each access of a `TLuaResult::Result` accessing the Lua stack of that
state (from outside that state's thread, which is unsafe). This
consistently lead to issues and sometimes crashes. Fix: `TLuaResult` now
always marshals results into a detached result variant. This allocates,
but this is unlikely to impact the hot paths, as most results will be
empty or have primitive types. **UPDATE 2026-04-29:** This fix caused
some other issues, so I added a result type that has no result value,
only a status. That seems to help.
11. Issue: HTTP retains a curl handle per thread and never cleans them
up. With one new thread per client, each doing an auth request at least,
this quickly exhausts all file descriptors. This manifests itself as
**dns resolutions failing**, as the server fails to open a socket to
send a DNS query. Fix: The HTTP code now retains a pool of reusable
handles, which clean up automatically via RAII. I tried to build this in
a way that doesn't modify the code too much, so I kept it global and
static.
12. Issue: Crash when accessing an expired `std::weak_ptr<TClient>`.
This can happen when we check for `.expired()`, and then `.lock()`,
which is, of course, another TOCTOU (time of check vs time of use) bug.
What youre SUPPOSED to do instead, is locking, which always returns a
`std::shared_ptr<>`, and then check `std::shared_ptr<>::operator bool`.
An expired `std::weak_ptr` will return a default-constructed
`std::shared_ptr`, which evaluates to `false` when converted to `bool`.
Fix: Replaced all uses of `.expired()` and other such checks with the
correct pattern. This was a lot of search and manual replace :D.

I used LLMs to help with writing unit-tests, but those do not compile
into the final executable anyway. If this is undesired, I'm happy to
remove that code.

---

By creating this pull request, I understand that code that is AI
generated or otherwise automatically generated may be rejected without
further discussion.
I declare that I fully understand all code I pushed into this PR, and
wrote all this code myself and own the rights to this code.

Author: WiserTixx

CMakeLists.txt

@@ -8,7 +8,7 @@ include(cmake/Vcpkg.cmake) # needs to happen before project()
 
 project(
     "BeamMP-Server" # replace this
-    VERSION 3.3.0
+    VERSION 3.9.2
 )
 
 include(cmake/StandardSettings.cmake)
@@ -39,6 +39,7 @@ set(PRJ_HEADERS
     include/TConfig.h
     include/TConsole.h
     include/THeartbeatThread.h
+    include/TIoPollThread.h
     include/TLuaEngine.h
     include/TLuaPlugin.h
     include/TNetwork.h
@@ -52,6 +53,8 @@ set(PRJ_HEADERS
     include/Settings.h
     include/Profiling.h
     include/ChronoWrapper.h
+    include/TConnectionLimiter.h
+    include/TLuaResult.h
 )
 # add all source files (.cpp) to this, except the one with main()
 set(PRJ_SOURCES
@@ -65,6 +68,7 @@ set(PRJ_SOURCES
     src/TConfig.cpp
     src/TConsole.cpp
     src/THeartbeatThread.cpp
+    src/TIoPollThread.cpp
     src/TLuaEngine.cpp
     src/TLuaPlugin.cpp
     src/TNetwork.cpp
@@ -78,6 +82,8 @@ set(PRJ_SOURCES
     src/Settings.cpp
     src/Profiling.cpp
     src/ChronoWrapper.cpp
+    src/TConnectionLimiter.cpp
+    src/TLuaResult.cpp
 )
 
 find_package(Lua REQUIRED)

cmake/Vcpkg.cmake

@@ -15,3 +15,5 @@ if(NOT DEFINED CMAKE_TOOLCHAIN_FILE)
     set(CMAKE_TOOLCHAIN_FILE ${CMAKE_SOURCE_DIR}/vcpkg/scripts/buildsystems/vcpkg.cmake)
 endif()
 
+# ensure SOL2 safeties are all ON
+add_compile_definitions(SOL_ALL_SAFETIES_ON=1)

include/Client.h

@@ -18,6 +18,7 @@
 
 #pragma once
 
+#include <atomic>
 #include <chrono>
 #include <memory>
 #include <optional>
@@ -68,8 +69,11 @@ class TClient final {
     std::string GetCarPositionRaw(int Ident);
     void SetUDPAddr(const ip::udp::endpoint& Addr) { mUDPAddress = Addr; }
     void SetTCPSock(ip::tcp::socket&& CSock) { mSocket = std::move(CSock); }
-    void Disconnect(std::string_view Reason);
-    bool IsDisconnected() const { return !mSocket.is_open(); }
+    // Returns true only for the thread that actually performs socket shutdown/close.
+    [[nodiscard]] bool Disconnect(std::string_view Reason);
+    bool IsDisconnected() const {
+        return mDisconnectState.load(std::memory_order_acquire) != EDisconnectState::Connected;
+    }
     // locks
     void DeleteCar(int Ident);
     [[nodiscard]] const std::unordered_map<std::string, std::string>& GetIdentifiers() const { return mIdentifiers; }
@@ -106,6 +110,12 @@ class TClient final {
     [[nodiscard]] const std::vector<uint8_t>& GetMagic() const { return mMagic; }
 
 private:
+    enum class EDisconnectState {
+        Connected,
+        Disconnecting,
+        Disconnected
+    };
+
     void InsertVehicle(int ID, const std::string& Data);
 
     TServer& mServer;
@@ -121,6 +131,8 @@ class TClient final {
     TSetOfVehicleData mVehicleData;
     SparseArray<std::string> mVehiclePosition;
     std::string mName = "Unknown Client";
+    // Once disconnect starts, this client is terminal and its socket must be treated as dead.
+    std::atomic<EDisconnectState> mDisconnectState { EDisconnectState::Connected };
     ip::tcp::socket mSocket;
     ip::udp::endpoint mUDPAddress {};
     int mUnicycleID = -1;

include/Common.h

@@ -132,6 +132,10 @@ class Application final {
     static inline Version mVersion { 3, 9, 2 };
 };
 
+/// Used to static_assert in std::visit
+template<class>
+inline constexpr bool AlwaysFalseV = false;
+
 void SplitString(std::string const& str, const char delim, std::vector<std::string>& out);
 std::string LowerString(std::string str);
 

include/TConnectionLimiter.h

@@ -0,0 +1,90 @@
+// BeamMP, the BeamNG.drive multiplayer mod.
+// Copyright (C) 2026 BeamMP Ltd., BeamMP team and contributors.
+//
+// BeamMP Ltd. can be contacted by electronic mail via contact@beammp.com.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published
+// by the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+#pragma once
+
+#include <mutex>
+#include <optional>
+#include <string>
+#include <unordered_map>
+#include <utility>
+
+class TConnectionLimiter {
+public:
+    struct TStats {
+        size_t CurrentGlobal = 0;
+        size_t MaxGlobal = 0;
+        size_t ActiveIpBuckets = 0;
+        size_t CurrentMaxPerIp = 0;
+        size_t MaxPerIp = 0;
+        size_t SaturatedIpBuckets = 0;
+    };
+
+    class TGuard {
+    public:
+        TGuard() = default;
+        TGuard(TConnectionLimiter* owner, std::string ip);
+
+        TGuard(const TGuard&) = delete;
+        TGuard& operator=(const TGuard&) = delete;
+
+        ~TGuard() { Release(); }
+
+        // not threadsafe
+        TGuard(TGuard&& other) noexcept { *this = std::move(other); }
+        // not threadsafe
+        TGuard& operator=(TGuard&& other) noexcept;
+
+    private:
+        friend class TConnectionLimiter;
+
+        // not threadsafe
+        void Release();
+
+        TConnectionLimiter* mOwner { nullptr };
+        std::string mIp;
+    };
+
+    TConnectionLimiter(size_t maxPerIp, size_t maxGlobal);
+
+    [[nodiscard]] std::optional<TGuard> TryAcquire(const std::string& ip);
+    [[nodiscard]] TStats GetStats();
+
+private:
+    void Release(const std::string& ip) {
+        std::unique_lock Lock { mMutex };
+        auto It = mPerIp.find(ip);
+        if (It != mPerIp.end()) {
+            // this guard exists to avoid underflow in case something goes wrong and this gets called too many times
+            if (It->second > 0)
+                --It->second;
+            if (It->second == 0)
+                mPerIp.erase(It);
+        }
+        // this guard exists to avoid underflow in case something goes wrong and this gets called too many times
+        if (mGlobal > 0)
+            --mGlobal;
+    }
+
+    const size_t mMaxPerIp;
+    const size_t mMaxGlobal;
+
+    std::mutex mMutex { };
+    std::unordered_map<std::string, size_t> mPerIp { };
+    size_t mGlobal = 0;
+};

include/TIoPollThread.h

@@ -0,0 +1,36 @@
+// BeamMP, the BeamNG.drive multiplayer mod.
+// Copyright (C) 2024 BeamMP Ltd., BeamMP team and contributors.
+//
+// BeamMP Ltd. can be contacted by electronic mail via contact@beammp.com.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published
+// by the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+#pragma once
+
+#include <boost/asio/io_context.hpp>
+#include <boost/asio/executor_work_guard.hpp>
+#include <thread>
+
+class TIoPollThread {
+public:
+    TIoPollThread();
+    ~TIoPollThread();
+
+    boost::asio::io_context& IoCtx() noexcept { return mIoCtx; }
+
+private:
+    boost::asio::io_context mIoCtx;
+    boost::asio::executor_work_guard<boost::asio::io_context::executor_type> mWorkGuard;
+    std::jthread mThread;
+};

include/TLuaEngine.h

@@ -19,13 +19,12 @@
 #pragma once
 
 #include "Profiling.h"
+#include "TLuaResult.h"
 #include "TNetwork.h"
 #include "TServer.h"
-#include <any>
 #include <chrono>
 #include <condition_variable>
 #include <filesystem>
-#include <initializer_list>
 #include <list>
 #include <lua.hpp>
 #include <memory>
@@ -34,6 +33,8 @@
 #include <queue>
 #include <random>
 #include <set>
+#include <sol/forward.hpp>
+#include <sol/protected_function_result.hpp>
 #include <toml.hpp>
 #include <unordered_map>
 #include <vector>
@@ -58,36 +59,10 @@ namespace fs = std::filesystem;
 /**
  * std::variant means, that TLuaArgTypes may be one of the Types listed as template args
  */
-using TLuaValue = std::variant<std::string, int, JsonString, bool, std::unordered_map<std::string, std::string>, float>;
-enum TLuaType {
-    String = 0,
-    Int = 1,
-    Json = 2,
-    Bool = 3,
-    StringStringMap = 4,
-    Float = 5,
-};
+using TLuaValue = std::variant<std::monostate, std::string, int, JsonString, bool, std::unordered_map<std::string, std::string>, float>;
 
 class TLuaPlugin;
 
-struct TLuaResult {
-    bool Ready;
-    bool Error;
-    std::string ErrorMessage;
-    sol::object Result { sol::lua_nil };
-    TLuaStateId StateId;
-    std::string Function;
-    std::shared_ptr<std::mutex> ReadyMutex {
-        std::make_shared<std::mutex>()
-    };
-    std::shared_ptr<std::condition_variable> ReadyCondition {
-        std::make_shared<std::condition_variable>()
-    };
-
-    void MarkAsReady();
-    void WaitUntilReady();
-};
-
 struct TLuaPluginConfig {
     static inline const std::string FileName = "PluginConfig.toml";
     TLuaStateId StateId;
@@ -166,7 +141,7 @@ class TLuaEngine : public std::enable_shared_from_this<TLuaEngine>, IThreaded {
         const std::optional<std::chrono::high_resolution_clock::duration>& Max = std::nullopt);
     void ReportErrors(const std::vector<std::shared_ptr<TLuaResult>>& Results);
     bool HasState(TLuaStateId StateId);
-    [[nodiscard]] std::shared_ptr<TLuaResult> EnqueueScript(TLuaStateId StateID, const TLuaChunk& Script);
+    [[nodiscard]] std::shared_ptr<TLuaVoidResult> EnqueueScript(TLuaStateId StateID, const TLuaChunk& Script);
     [[nodiscard]] std::shared_ptr<TLuaResult> EnqueueFunctionCall(TLuaStateId StateID, const std::string& FunctionName, const std::vector<TLuaValue>& Args, const std::string& EventName);
     void EnsureStateExists(TLuaStateId StateId, const std::string& Name, bool DontCallOnInit = false);
     void RegisterEvent(const std::string& EventName, TLuaStateId StateId, const std::string& FunctionName);
@@ -227,9 +202,9 @@ class TLuaEngine : public std::enable_shared_from_this<TLuaEngine>, IThreaded {
 
     // Debugging functions (slow)
     std::unordered_map<std::string /*event name */, std::vector<std::string> /* handlers */> Debug_GetEventsForState(TLuaStateId StateId);
-    std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaResult>>> Debug_GetStateExecuteQueueForState(TLuaStateId StateId);
+    std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaVoidResult>>> Debug_GetStateExecuteQueueForState(TLuaStateId StateId);
     std::vector<QueuedFunction> Debug_GetStateFunctionQueueForState(TLuaStateId StateId);
-    std::vector<TLuaResult> Debug_GetResultsToCheckForState(TLuaStateId StateId);
+    std::vector<TLuaResult::DetachedSnapshot> Debug_GetResultsToCheckForState(TLuaStateId StateId);
 
 private:
     void CollectAndInitPlugins();
@@ -242,7 +217,7 @@ class TLuaEngine : public std::enable_shared_from_this<TLuaEngine>, IThreaded {
         StateThreadData(const std::string& Name, TLuaStateId StateId, TLuaEngine& Engine);
         StateThreadData(const StateThreadData&) = delete;
         virtual ~StateThreadData() noexcept { beammp_debug("\"" + mStateId + "\" destroyed"); }
-        [[nodiscard]] std::shared_ptr<TLuaResult> EnqueueScript(const TLuaChunk& Script);
+        [[nodiscard]] std::shared_ptr<TLuaVoidResult> EnqueueScript(const TLuaChunk& Script);
         [[nodiscard]] std::shared_ptr<TLuaResult> EnqueueFunctionCall(const std::string& FunctionName, const std::vector<TLuaValue>& Args, const std::string& EventName);
         [[nodiscard]] std::shared_ptr<TLuaResult> EnqueueFunctionCallFromCustomEvent(const std::string& FunctionName, const std::vector<TLuaValue>& Args, const std::string& EventName, CallStrategy Strategy);
         void RegisterEvent(const std::string& EventName, const std::string& FunctionName);
@@ -254,7 +229,7 @@ class TLuaEngine : public std::enable_shared_from_this<TLuaEngine>, IThreaded {
         std::vector<std::string> GetStateTableKeys(const std::vector<std::string>& keys);
 
         // Debug functions, slow
-        std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaResult>>> Debug_GetStateExecuteQueue();
+        std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaVoidResult>>> Debug_GetStateExecuteQueue();
         std::vector<TLuaEngine::QueuedFunction> Debug_GetStateFunctionQueue();
 
     private:
@@ -279,7 +254,7 @@ class TLuaEngine : public std::enable_shared_from_this<TLuaEngine>, IThreaded {
         TLuaStateId mStateId;
         lua_State* mState;
         std::thread mThread;
-        std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaResult>>> mStateExecuteQueue;
+        std::queue<std::pair<TLuaChunk, std::shared_ptr<TLuaVoidResult>>> mStateExecuteQueue;
         std::recursive_mutex mStateExecuteQueueMutex;
         std::vector<QueuedFunction> mStateFunctionQueue;
         std::mutex mStateFunctionQueueMutex;