LuaBridge crashes when passing metatable of userdata to luaL_tolstring (print)

[Mellnik]

Environment

• Windows 10 x64
• Visual Studio 2022 14.34.31937 toolchain
• Using latest LuaBridge commit from master branch [9092ace]
• Lua 5.4.4

Steps to reproduce

• Expose a C++ class to Lua.
• Get the metatable of an instance of such userdata class.
• Pass it to print.

Result

• A table and not a userdata is passed to lual_tolstring in print.
• lual_tolstring calls __tostring metamethod
• The function bound to __tostring is invoked in LuaBridge
• LuaBridge crashes because it received a table and not a valid userdata

I am not entirely sure if this is the fault of LuaBridge or Lua. Maybe @dmitry-t can share some insights.

Example C++ code with LuaBridge:

// Some example class with a tostring method.
class ExampleClass
{
public:
	ExampleClass() :
		a(0), b(0), c(0)
	{

	}

	string tostring() const
	{
		return "whatever";
	}

	int a, b, c;
};

lua_State* L = luaL_newstate();

luaL_openlibs(L);

// Expose example class and add a __tostring metamethod
luabridge::getGlobalNamespace(L)
	.beginClass<ExampleClass>("ExampleClass")
		.addConstructor<void(*) ()>()
		.addFunction("__tostring", &ExampleClass::tostring)
	.endClass()
	;

// Create an instance of the example class.
// Call getmetatable on the instance and pass it to print.
luaL_dostring(L, "local t = ExampleClass(); print(getmetatable(t));");

lua_close(L);

Call stack

main.exe!luabridge::detail::Userdata::getPointer() Line 46	C++
main.exe!luabridge::detail::Userdata::get<ExampleClass>(lua_State * L, int index, bool canBeConst) Line 252	C++
main.exe!luabridge::detail::CFunc::CallConstMember<std::string (__cdecl ExampleClass::*)(void)const>::f(lua_State * L) Line 290	C++
main.exe!precallC(lua_State * L, StackValue * func, int nresults, int(*)(lua_State *) f) Line 506	C
main.exe!luaD_precall(lua_State * L, StackValue * func, int nresults) Line 570	C
main.exe!ccall(lua_State * L, StackValue * func, int nResults, int inc) Line 607	C
main.exe!luaD_callnoyield(lua_State * L, StackValue * func, int nResults) Line 628	C
main.exe!lua_callk(lua_State * L, int nargs, int nresults, __int64 ctx, int(*)(lua_State *, int, __int64) k) Line 1024	C
main.exe!luaL_callmeta(lua_State * L, int obj, const char * event) Line 867	C
main.exe!luaL_tolstring(lua_State * L, int idx, unsigned __int64 * len) Line 885	C
main.exe!luaB_print(lua_State * L) Line 29	C
^print is called above.^


main.exe!precallC(lua_State * L, StackValue * func, int nresults, int(*)(lua_State *) f) Line 506	C
main.exe!luaD_precall(lua_State * L, StackValue * func, int nresults) Line 573	C
main.exe!luaV_execute(lua_State * L, CallInfo * ci) Line 1636	C
main.exe!ccall(lua_State * L, StackValue * func, int nResults, int inc) Line 611	C
main.exe!luaD_callnoyield(lua_State * L, StackValue * func, int nResults) Line 628	C
main.exe!f_call(lua_State * L, void * ud) Line 1042	C
main.exe!luaD_rawrunprotected(lua_State * L, void(*)(lua_State *, void *) f, void * ud) Line 147	C
main.exe!luaD_pcall(lua_State * L, void(*)(lua_State *, void *) func, void * u, __int64 old_top, __int64 ef) Line 926	C
main.exe!lua_pcallk(lua_State * L, int nargs, int nresults, int errfunc, __int64 ctx, int(*)(lua_State *, int, __int64) k) Line 1067	C
main.exe!main() Line 93	C++

[Mellnik]

Please see #309
This fixes the crash. But I still wonder if there is a deeper issue within LuaBridge...

[dmitry-t]

Thank you for your feedback. Would you mind to implement a unit test which reproduces the fail case?

[Mellnik]

I replied to the other issue. Think the commit from LuaBridge3 contains a test case for this.