Skip to content

feat: add heartbeat feature to auto-terminate program on failure #425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wen2so wants to merge 3 commits into
base: development
Choose a base branch
from
Open

feat: add heartbeat feature to auto-terminate program on failure #425

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b10846b
feat: add heartbeat feature
wen2so Jul 17, 2025
75ed67f
feat: add heartbeat interval configuration option
wen2so Jul 17, 2025
4d0649c
Merge branch 'development' into dev/heartbeat
nam20485 Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions OdbDesignLib/App/OdbDesignArgs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ namespace Odb::Lib::App
return boolArg("disable-authentication", DEFAULT_DISABLE_AUTH);
}

int OdbDesignArgs::heartbeatInterval() const
{
return intArg("heartbeat-interval", DEFAULT_HEARTBEAT_INTERVAL);
}

std::string OdbDesignArgs::getUsageString() const
{
std::stringstream ss;
Expand All @@ -66,6 +71,7 @@ namespace Odb::Lib::App
ss << " --load-design <design> Design to load on startup (default: " << DEFAULT_LOAD_DESIGN << ")\n";
ss << " --load-all Load all designs on startup (default: " << (DEFAULT_LOAD_ALL ? "true" : "false") << ")\n";
ss << " --disable-authentication Disable authentication (default: " << (DEFAULT_DISABLE_AUTH ? "true" : "false") << ")\n";
ss << " --heartbeat-interval <interval> Heartbeat interval in seconds (default: " << DEFAULT_HEARTBEAT_INTERVAL << ")\n";
ss << " --help Print this help message\n";
return ss.str();
}
Expand Down
2 changes: 2 additions & 0 deletions OdbDesignLib/App/OdbDesignArgs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ namespace Odb::Lib::App
std::string loadDesign() const;
bool loadAll() const;
bool disableAuthentication() const;
int heartbeatInterval() const;

protected:
// Inherited via CommandLineArgs
Expand All @@ -34,6 +35,7 @@ namespace Odb::Lib::App
constexpr static const char* DEFAULT_LOAD_DESIGN = "";
constexpr static const bool DEFAULT_LOAD_ALL = false;
constexpr static const bool DEFAULT_DISABLE_AUTH = false;
constexpr static const int DEFAULT_HEARTBEAT_INTERVAL = 5;

};
}
8 changes: 8 additions & 0 deletions OdbDesignServer/Controllers/HealthCheckController.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ namespace Odb::App::Server
register_route_handler("/healthz/live", std::bind(&HealthCheckController::health_check_live, this, std::placeholders::_1));
register_route_handler("/healthz/ready", std::bind(&HealthCheckController::health_check_ready, this, std::placeholders::_1));
register_route_handler("/healthz/started", std::bind(&HealthCheckController::health_check_started, this, std::placeholders::_1));
register_route_handler("/healthz/heartbeat", std::bind(&HealthCheckController::health_check_heartbeat, this, std::placeholders::_1));
}

crow::response HealthCheckController::health_check_live(const crow::request& req)
Expand All @@ -38,4 +39,11 @@ namespace Odb::App::Server
{
return crow::response(crow::status::OK, "txt", "healthy: started");
}

crow::response HealthCheckController::health_check_heartbeat(const crow::request& req)
{
// update last heartbeat
OdbDesignServerApp::inst_->updateLastHeartbeat();
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This line uses the global singleton inst_ to access the application object. This creates a tight coupling and dependency on a global variable1. The RouteController base class already provides access to the server application via the m_serverApp member. You can use dynamic_cast to safely cast it to your specific application class and call the method. This approach uses dependency injection and avoids global state, which is a much cleaner design.

Style Guide References

Suggested change
OdbDesignServerApp::inst_->updateLastHeartbeat();
dynamic_cast<OdbDesignServerApp&>(m_serverApp).updateLastHeartbeat();

Footnotes

  1. Avoid global singletons. (link) ↩

Copy link
Copy Markdown
Owner

@nam20485 nam20485 Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Gemini states, you already have access to m_serverApp. You can use that and get rid of the singleton object.

return crow::response(crow::status::OK, "txt", "heartbeat received");
}
}
1 change: 1 addition & 0 deletions OdbDesignServer/Controllers/HealthCheckController.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ namespace Odb::App::Server
crow::response health_check_live(const crow::request& req);
crow::response health_check_ready(const crow::request& req);
crow::response health_check_started(const crow::request& req);
crow::response health_check_heartbeat(const crow::request& req);

};
}
33 changes: 32 additions & 1 deletion OdbDesignServer/OdbDesignServerApp.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
#include <thread>
#include <chrono>
#include <iostream>
#include "OdbDesignServerApp.h"
#include "Controllers/HelloWorldController.h"
#include "Controllers/FileUploadController.h"
Expand All @@ -11,9 +14,15 @@ using namespace Odb::Lib::App;

namespace Odb::App::Server
{
OdbDesignServerApp* OdbDesignServerApp::inst_ = nullptr;
int heartbeatInterval;
OdbDesignServerApp::OdbDesignServerApp(int argc, char* argv[])
: OdbServerAppBase(argc, argv)
{
{
inst_ = this;
heartbeatInterval = args().heartbeatInterval();
// set last heartbeat time to now
lastHeartbeat_.store(std::chrono::steady_clock::now(), std::memory_order_relaxed);
}

//OdbDesignServerApp::~OdbDesignServerApp()
Expand Down Expand Up @@ -44,6 +53,24 @@ namespace Odb::App::Server
m_vecControllers.push_back(std::make_shared<DesignsController>(*this));
}

void monitorHeartbeat()
{
auto lastTime = OdbDesignServerApp::inst_->lastHeartbeat_.load(std::memory_order_relaxed);
while (true)
{
std::this_thread::sleep_for(std::chrono::seconds(1));
auto now = std::chrono::steady_clock::now();
auto diff = now - lastTime;
// check heartbeat
if (diff > std::chrono::seconds(heartbeatInterval))
{
std::cerr << "Heartbeat timeout, exiting..." << std::endl;
exit(0);
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Calling exit(0) will cause an immediate, forceful termination of the process. This prevents proper cleanup. A more graceful shutdown can be achieved by stopping the Crow application1. This allows the run() method to return and the application to perform any necessary cleanup before exiting.

Style Guide References

Suggested change
exit(0);
OdbDesignServerApp::inst_->crow_app().stop();
return;

Footnotes

  1. Use crow_app().stop() for a graceful shutdown. (link) ↩

}
lastTime = OdbDesignServerApp::inst_->lastHeartbeat_.load(std::memory_order_relaxed);
}
}

bool OdbDesignServerApp::preServerRun()
{
// CORS
Expand All @@ -70,6 +97,10 @@ namespace Odb::App::Server
auto basicRequestAuth = std::make_unique<BasicRequestAuthentication>(BasicRequestAuthentication(disableAuth));
request_auth(std::move(basicRequestAuth));

// start heart beat monitor
std::thread heartbeatMonitor(monitorHeartbeat);
heartbeatMonitor.detach(); // run in background

return true;
}
}
10 changes: 9 additions & 1 deletion OdbDesignServer/OdbDesignServerApp.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,15 @@ namespace Odb::App::Server

//Utils::ExitCode Run() override;

protected:
static OdbDesignServerApp* inst_;
// store last heartbeat time
std::atomic<std::chrono::steady_clock::time_point> lastHeartbeat_;
Comment on lines +17 to +19
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

These members (inst_ and lastHeartbeat_) are public, which breaks encapsulation. The inst_ static pointer, in particular, implements a singleton pattern via a public member, which introduces global state and makes code difficult to test and reason about1. These should be made private.

Style Guide References

Footnotes

  1. Avoid public members for internal state. (link) ↩

void updateLastHeartbeat()
{
lastHeartbeat_.store(std::chrono::steady_clock::now(), std::memory_order_relaxed);
}

protected:
void add_controllers() override;


Expand Down
7 changes: 7 additions & 0 deletions swagger/odbdesign-server-0.9-swagger.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,13 @@ paths:
"200":
description: ""
/healthz/ready:
get:
tags: ["health check"]
parameters: []
responses:
"200":
description: ""
/healthz/heartbeat:
get:
tags: ["health check"]
parameters: []
Expand Down
Loading