TSR memory allocation via 0501 freed on Primary Client termination

[crazii]

Hi Japheth,

Thank you for the amazing HDPMI project that brings all kinds of possibilities to retro-computing.

Recently I encounter a problem that when a TSR allocates DPMI memory while the foreground DPMI program is active, the allocated memory lives until the foreground program terminated.

An exception of 0E (#PF) with error code 04 (ring3 fault with present bit=0) is generated on accessing that mem, right after the foreground program terminated, logs show the access was OK while the foreground program is alive.

I check the I31MEM.ASM: 1682 and 1696, it seems the gbl.cApps was 2 when the allocation happens, and probably also 2 when the foreground program terminated.

I'm aware this TSR behavior is not good, it causes memory fragmentations anyways. FYI the actual scenario is that the SBEMU TSR tries to init TSF with some MB mem allocation, during another instance of SBEMU with soundfont options enabled.

I was using a old fork of HDPMI so I don't know whether this problem still exist. If yes, please share you ideas on it, e.g. is it being fixed or not? As a DPMI TSR is less common for old programs, and the problem now can be avoided in a way, by some workarounds, e.g. pre-alloc some mem on TSR initialization.

Thanks again.

[Baron-von-Riedesel]

AFAICS, if a TSR needs dynamic memory allocation, it is supposed to alloc memory during init. If the size is unknown at that time, the application should alloc the minimum and realloc the block as needed.

However, there was a bug in HDPMI, fixed in v3.23:

þ fixed int 31h, ax=503h/505h: if base address of resized block had to
change, the block's owner was set to the current client.

It's just 2 lines, in I31MEM.ASM

;--- v3.23: _allocaddrspace has set bOwner to current client;
;---        the old block's owner must therefore be copied to new block.
	mov al, [esi].MEMITEM.bOwner
	mov [ebx].MEMITEM.bOwner, al

[crazii]

Thanks!

I don't remember how djgpp handle memory allocations, it probably uses multi blocks without resizing. if resizing is used, then will work with v3.23 I guess. I don't remember what version of the HDPMI fork. LOL.

There are some function limited to a TSR, e.g. function 000D not allowed for a PM (v1.0) TSR, but I didn't see restriction for TSRs on 0501 in the dpmi1.0 spec (don't know about 0.9 spec). Currently I used a hack to pre-alloc some memory working for some cases (e.g. files <= 4M). However, the real size is dynamic and depends on file size, determined by the soundfont file specified in cmd line option.

I'm leaving the code for now and maybe there are some other ways that come to.

[Baron-von-Riedesel]

I don't remember how djgpp handle memory allocations, it probably uses multi blocks without resizing.

I recall that fork vsbhdasf did the realloc "manually" in function tsfimpl_realloc(void *ptr, size_t size) - that is, it allocates a new block, copies the data and finally frees the old block. That's something a TSR should not do.

[crazii]

I integrated the 2 line changes from v3.23, and checked djgpp src, it by default do not call DPMI realloc 0503, but with alternative settings in _crt0_startup_flags (_CRT0_FLAG_UNIX_SBRK) to allow reallocation with based address changes. but when using that flag, SBEMU crashes after loading, I guess there is incompatibility between the flag and interrupt handling, or there are untested bugs with that flag in SBEMU.
so for now, the 2 line changes are not tested.

the problem is not just tsfimpl_realloc, which can be omitted by setting up TSF properly. because SBEMU cam initially install with no SF loaded, so all TSF memory are not allocated at all, but when a second instance of SBEMU runs with soundfont enabled, there'll be memory allocations in MBs.

there are 2 perpectives of realloc: c-runtime (djgpp) and DPMI, djgpp has heaps & pool and when that pool exhausted, it will request new memory from DPMI. currently I'm calling malloc() and free() to preserve memory, because free() won't return the memory to the DPMI server but still in the c-runtime heaps. there are other ways e.g. pre-alloc directly using int31h, but all are the same that to pre-allocate memory.

[crazii]

I think there might be some other ways to fix it in DPMI server. e.g. maintaining a client information for selectors so when any INT31h calls, DPMI server can get the right client from CS and set the correct client to requests. it also helps with cleaning up all the selectors and memories on client termination.

VDPMI has such codes, but it's almost totally in C so changes are a little bit easy, while whole assembly gives me headaches. I mean not to compare, it's just a non-assembly coder is feeling. :D

[awik32]

I think it is OK for a TSR to call DOS and do such things as file IO and memory allocation, depending on the InDOS flag and setting the current process (PSP) to itself (and restoring it when finished).

[crazii]

I think it is OK for a TSR to call DOS and do such things as file IO and memory allocation, depending on the InDOS flag and setting the current process (PSP) to itself (and restoring it when finished).

Yes, similar but with tiny differences. DOS is not reentrant and has a live shell (command.com) so it needs the InDOS flag, a DPMI service doesn't need a flag for its memory allocation, only needed to care about InDOS when using DOS related services, e.g. the protected mode INT21h.

[Baron-von-Riedesel]

think there might be some other ways to fix it in DPMI server. e.g. maintaining a client information for selectors so when any
INT31h calls, DPMI server can get the right client from CS and set the correct client to requests.

Things could surely be "improved", but HDPMI aims to be compatible with the Win9X DPMI host, and that host doesn't do such things. It's also doubtful, because any DPMI application can hook int 31h ( VSBHDA actually does that ), so the caller's CS does not really contain much information.

it also helps with cleaning up all the selectors and memory on client termination.

This is no problem for DPMI v1.0 hosts, since LDT, IDT and address space aren't global anymore - and HDPMI optionally supports this. Currently a DPMI TSR has to disable that option, but by implementing support for DPMI v1.0 TSRs ( int 31h 0xCxx) and shared memory ( int 31h 0xDxx ) this could be changed.

[crazii]

Things could surely be "improved", but HDPMI aims to be compatible with the Win9X DPMI host, and that host doesn't do such things. It's also doubtful, because any DPMI application can hook int 31h ( VSBHDA actually does that ), so the caller's CS does not really contain much information.

the caller CS can be useful as long as the int31h handler always goes to ring0 first, and then application handling result recorded (but I doubt the need to record an app-provided memory handle with CS/client-id because no old application hooks the memory allocation functions AFAIK), but that's another topic, and you have made a good point, as I posted in the beginning, there are no old PM programs found with such behavior, so actually it does not affect any old programs. and SBEMU as a new program does not necessarily rely on a corresponding changed behavior in DPMI server. I posted the issue with the intention to make HDPMI better by any chance, even only with some ideas or discussions that may not lead to any actual changes.

Thank you again.