[Request] Memory searching functionality for Lua scripts

[Davidebyzero]

As far as I can tell, there's no built-in memory search function in BizHawk Lua. So you either have to use the RAM Search UI and manually copy that into a script, or implement a search loop in Lua.

Here is an example, which I am using to display the current Worms Armageddon game engine time (using the DOSBox-X core). It works, but takes about 9 seconds even when the search is successful. On the bright side, it only has to be done once per started game round, but that's still somewhat inconvenient, and surely there are other TASes for which even more searching would be needed.

The built-in function should allow the memory domain and address range to be specified, along with the size of the needle in bytes (8 bytes in my example), and alignment (4 bytes in my example). Its implementation could then perhaps optimize by specializing for commonly used combinations of needle size and alignment (so that it doesn't have to be slowed down by using variables for those), e.g. by using templated code.

addr = 0

tastudio.addcolumn("time", "Time", 60)
local cache = {}

local id
id = event.onframeend(function()
    local needle0 = 0x5755D0
    local needle1 = 0x800

    if memory.read_u32_le(addr, "Physical RAM") ~= needle0 then -- or memory.read_u32_le(addr+4, "Physical RAM") ~= needle1 then
        local data = memory.read_bytes_as_array(0, 0x03000000, "Physical RAM")

        addr = 0
        for i = 0, #data - 8, 4 do
            local w0 = data[i+1] + (data[i+2] << 8) + (data[i+3] << 16) + (data[i+4] << 24)
            if w0 == needle0 then
                local w1 = data[i+5] + (data[i+6] << 8) + (data[i+7] << 16) + (data[i+8] << 24)
                if w1 == needle1 then
                    addr = i
                    console.log(string.format("Found at: 0x%08X", addr))
                    break
                end
            end
        end
    end
    if addr == 0 then
        event.unregisterbyid(id)
    else
        local value = memory.read_u32_le(addr + 8, "Physical RAM")
        msg = string.format("%d.%02d", value // 50, (value % 50) * 2)
        cache[emu.framecount()] = msg
        gui.text(1, 27, msg)
    end
end)

tastudio.onqueryitemtext(function(frame, column)
    if column == "time" then
        return cache[frame] or "?"
    end
end)

• Related to my other filed issue [Request] Adding custom TAStudio columns in Lua is very slow; a faster way to do this is needed #4756.

[vadosnaprimer]

On the bright side, it only has to be done once per started game round

You mean the addresses for the needed values change every level? Is there no pointer that would handle that in-game, that the script could just read and then infer the final address?

[YoshiRulz]

memory.read_bytes_as_binary_string + string.find is probably faster.

[Davidebyzero]

On the bright side, it only has to be done once per started game round

You mean the addresses for the needed values change every level? Is there no pointer that would handle that in-game, that the script could just read and then infer the final address?

Yes, it changes every round (level) because the game structures are allocated at the beginning of a round and deallocated at its end.

There is a pointer, but BizHawk DOSBox-X Lua doesn't provide access to the linear address space of the current process. It only provides access to Physical RAM. This makes it hard to find the pointer I want (and even if found, to then use that pointer); I suspect WA.exe's linear memory is fragmented across physical memory.

I pretty sure doing this properly would require reaching into Windows 98's page table, and I don't know how to do that. Especially since Lua doesn't provide access to the CR3 register (or any registers, with the DOSBox-X core).

---

Edit: Okay, I managed to find the page table with Claude Sonnet's help, and I can now follow the pointer chain. Finding the page table was slow, but it will be the same for an entire movie (and maybe even the same for every movie that uses the .hdd from the Windows 98 SE installation movie), so I can just hard-code that address.

I don't feel it would be proper for me to post the code that follows the pointer chain, as this used my inside information as a WA dev, but I can post the page table code (will do, as soon as I've finished the pointer chain code and made sure it works).

[Davidebyzero]

Doing Edit->Find in BizHawk's Hex Editor is also kind of slow; it takes 6 seconds. The 9 seconds taken by the Lua search is not actually much slower than that, and if YoshiRulz's suggestion is used, it'd probably be even faster.

[vadosnaprimer]

There is a pointer, but BizHawk DOSBox-X Lua doesn't provide access to the linear address space of the current process. It only provides access to Physical RAM.

What does this mean?

BTW there's a way to expose the entirety of the waterboxed (readable) memory, we did it for DSDA-Doom core.

[Davidebyzero]

What does this mean?

BTW there's a way to expose the entirety of the waterboxed (readable) memory, we did it for DSDA-Doom core.

I tried viewing "Waterbox PageData" and it just shows some very repetitive data that has nothing to do with the Win98 page table... anyway, here's the code that worked for me:

local RAM_SIZE = memory.getmemorydomainsize("Physical RAM")

local function read_u32(i)
    return memory.read_u32_le(i, "Physical RAM")
end

local physmem
local function cached_read_u32(base, offset)
    local i = base + offset
    return physmem[i+1] + (physmem[i+2] << 8) + (physmem[i+3] << 16) + (physmem[i+4] << 24)
end

if true then -- change to false for actual use
    physmem = memory.read_bytes_as_array(0, RAM_SIZE, "Physical RAM")
    PAGE_DIR_PHYS = nil

    local PAGE_SIZE = 0x1000
    local PDE_COUNT = 1024

    for addr = 0, RAM_SIZE - PAGE_SIZE, PAGE_SIZE do
        local selfref_index = -1
        local present_count = 0
        local valid_present = 0

        for i = 0, PDE_COUNT - 1 do
            local entry = cached_read_u32(addr, i * 4)
            if (entry & 1) == 1 then
                present_count = present_count + 1
                local entry_phys = entry & 0xFFFFF000
                if entry_phys == addr then
                    selfref_index = i
                end
                if entry_phys < RAM_SIZE then
                    valid_present = valid_present + 1
                end
            end
        end

        if selfref_index >= 0 and present_count >= 4 and present_count == valid_present then
            console.log(string.format("Page directory found at 0x%08X, self-ref index %d (vaddr 0x%08X), present=%d",
                addr, selfref_index, selfref_index * 0x400000, present_count))
            PAGE_DIR_PHYS = addr
            break
        end
    end
    if not PAGE_DIR_PHYS then
        console.log("Page directory not found")
    end
else
    PAGE_DIR_PHYS = 0x007F6000
end

local function virtualToPhysical(vaddr)
    local pdIndex = (vaddr >> 22) & 0x3FF
    local ptIndex = (vaddr >> 12) & 0x3FF
    local offset  = vaddr & 0xFFF

    -- Read Page Directory Entry
    local pde = read_u32(PAGE_DIR_PHYS + pdIndex * 4)
    if (pde & 1) == 0 then
        return nil  -- not present
    end

    -- Get Page Table physical address
    local ptPhys = pde & 0xFFFFF000
    if ptPhys >= RAM_SIZE then
        return nil  -- outside physical RAM
    end

    -- Read Page Table Entry
    local pte = read_u32(ptPhys + ptIndex * 4)
    if (pte & 1) == 0 then
        return nil  -- not present
    end

    -- Get physical page
    local pagePhys = pte & 0xFFFFF000
    if pagePhys >= RAM_SIZE then
        return nil  -- outside physical RAM
    end

    return pagePhys + offset
end

local function virtualToPhysical_offset(vaddr, offset)
    if vaddr == nil then
        return nil
    end
    return virtualToPhysical(vaddr + offset)
end

local function readVirtual_u32(vaddr, offset)
    if vaddr == nil then
        return nil
    end
    local paddr = virtualToPhysical(vaddr + offset)
    if paddr == nil then
        return nil
    end
    return read_u32(paddr)
end

local cache = {}
local frame_addr = nil

if PAGE_DIR_PHYS then
    tastudio.addcolumn("time", "Time", 60)

    event.onframeend(function()
        -- Only re-find the frame address if it becomes no longer valid, because
        -- otherwise it will be temporarily lost during context switches
        if not frame_addr or read_u32(frame_addr - 4) ~= 0x800 then
            frame_addr = virtualToPhysical_offset(readVirtual_u32(readVirtual_u32(readVirtual_u32(0x7A0884, 0), 0xA0), 0x488), 0x5CC)
        end

        if frame_addr then
            local value = read_u32(frame_addr)
            msg = string.format("%d.%02d", value // 50, (value % 50) * 2)
            cache[emu.framecount()] = msg
            gui.text(1, 27, msg)
        else
            cache[emu.framecount()] = ""
        end
    end)

    tastudio.onqueryitemtext(function(frame, column)
        if column == "time" then
            return cache[frame] or "?"
        end
    end)
end

[vadosnaprimer]

Page data is a map of which addresses are readable/writable/etc, but not their actual values. DSDA-Doom's "system bus" domain is actual values, including all waterboxed internal memory of the core itself.