Support thiscall and fastcall calling conventions (Windows x86-32)

[scc-tw]

Summary

Add thiscall and fastcall calling convention support to the NATIVE_CALL handler for Windows x86-32 targets.

Background

The current NATIVE_CALL implementation supports:

• cdecl (default): all args on stack, caller cleanup
• stdcall: all args on stack, callee cleanup (planned for next sprint)

Two additional Windows-only conventions are not yet implemented:

thiscall

• Usage: Default for C++ non-static member functions on MSVC
• Register args: this pointer in ECX, all other args on stack
• Stack cleanup: Callee (like stdcall)
• Complexity: Medium — need to extract this from arg list, put in ECX, push rest to stack
• Impact: Protected regions calling C++ methods through vtable dispatch. Rare in typical protection targets (crypto/license code is usually C).

fastcall

• Usage: __fastcall decorated functions, some Windows kernel APIs
• Register args: First 2 integer args in ECX, EDX; rest on stack
• Stack cleanup: Callee
• Complexity: Low — similar to a 2-register calling convention
• Impact: Very rare in protected regions. Some internal compiler helpers use fastcall.

Implementation Notes

TransitionEntry

Convention field already allocated: arg_count[17:16] = calling_convention.

• 00 = cdecl (default)
• 01 = stdcall
• 10 = thiscall ← this issue
• 11 = fastcall ← this issue

Trampoline changes (platform_call_x86_32.S / .asm)

thiscall:

; ECX = first arg (this pointer)
; Stack: remaining args, right-to-left
mov ecx, [desc + int_args[0]]   ; this → ECX
; Push int_args[argc-1] ... int_args[1] to stack (skip [0])
call target
; Callee cleans stack (like stdcall, no ADD ESP needed)

fastcall:

; ECX = first arg, EDX = second arg
; Stack: remaining args from arg[2] onward
mov ecx, [desc + int_args[0]]
mov edx, [desc + int_args[1]]
; Push int_args[argc-1] ... int_args[2] to stack
call target
; Callee cleans stack

classify_args changes

For thiscall: int_args[0] = this, rest go to stack buffer starting from int_args[1].
For fastcall: int_args[0] = ECX, int_args[1] = EDX, rest go to stack from int_args[2].

FP interaction

Both conventions: FP args go on stack as 8-byte doubles (same as cdecl). No xmm register passing on x86-32.

Testing

• thiscall: create a minimal C++ class with a method, call through VM
• fastcall: create a __fastcall function, verify ECX/EDX arg placement

Priority

Low — defer to after cdecl/stdcall are stable. Can be implemented incrementally without affecting existing ABI support.