Time independent skulltulas

[mracsys]

Implements #2356.

Two parts to this PR. There is the skulltula time of day feature, and a new framework to enable arbitrary changes to any of the scene or room files parsed to linked python objects. The framework automatically shifts all pointers within the files to handle new file locations and additions/deletions within the file. It will have multiple applications besides the skulltula feature:

• Detailed cutscene parsing instead of writing raw bytes (WIP). Also more easily allows writing completely new cutscenes.
• Adding any actor/object to any room, such as a sign outside Ganon’s castle for the bridge condition or anywhere else a hint might be helpful
• Entrance/exit additions, such as promoting grottos to the same framework as other entrances, or new loading planes for mixed pool door rando
• Space reclamation. Almost a MB is wasted in padding between scene and room files.
• More user friendly interface for MQ patching, and opens the door for other alternate scene options for anyone crazy enough to attempt it.
• Collision changes for problems like Reduce boss doors side range to prevent voiding out in Forest temple #2331 and for my vision of door rando without scene reloads between doors.

More details on the design of the parser is included in Notes/scene-and-room-patching.md.

GUI Impacts for Web

TL;DR, Creating Patch File is added as a progress bar milestone between Patching ROM and Starting compression.

Due to the increased time spent patching the rom caused by the significantly larger amount of changed bytes compared to the current randomizer, an additional status message milestone is added to the GUI progress bar: Creating Patch File. This was previously included in the Patching ROM milestone. Additionally, as these milestones alternate for each player in multiworld generation, the progress percentages for these two milestones were modified from a static number to one that scales based on the currently processing world. This requires resetting currentWorld after successful patching for subsequent generations to work when Generation Count is greater than 1.

Python Virtual Environment Addition

This PR makes use of the numpy package to more efficiently process changed bytes for patch generation compared to base python lists and dictionaries. While the randomizer has avoided external dependencies thus far, using numpy reduces the performance penalty substantially (times are from single world runs on an AMD 5900X with the S8 Tournament preset):

Task	Penalty with numpy	Penalty without numpy
ROM Generation	25% (1.6s)	66% (4.3s)
ZPF Generation	168% (8.4s)	278% (13.9s)

A virtual environment is automatically created and activated by the major entry points to the randomizer:

• GUI - GUI.py
• Console - OoTRandomizer.py
• CI - CI.py

Each of these entry points independently check if they are running in a virtual environment located in subfolder .venv at the local path. If the environment doesn't exist, it is created automatically. There is also check if all dependencies in requirements.txt are fulfilled via pip. If not fulfilled, they are installed automatically in the virtual environment. If running in a release build, any pip errors are captured and passed to the regular error popup. Otherwise, running via the scripts logs errors to the console before exiting. If venv and pip checks are successful, but either the current python environment is not the virtual environment, or new dependencies were added, the script relaunches itself via the subprocess module in the virtual environment with all arguments intact. This behavior can be added to additional entry points or custom scripts by importing the ensure_venv() function from Utils.py and adding it as the first line wherever it's needed. See gui_main(), start(), and the top level of CI.py for example usage.

numpy is pinned between the last version supported by python 3.8 (1.24.4) and the version in use on RealRob's custom_voice_packs branch (2.2.1). Both versions have been tested to work. For compatibility with RealRob's work, I strongly recommend increasing the minimum python version for the randomizer to 3.10, the earliest supported by numpy 2.2.1. 3.8 is EOL, and 3.9 will be EOL in October 2025. This PR does not change the minimum python version.

Unrelated Bugfixes

• Fix a typo in an exception for wad file generation (see

  OoT-Randomizer/Main.py

  Line 267 in a821705

  raise FileNotFoundError(f'Invalid path to Base WAD: "{input_file}"')

  )
• Fix Rom.copy() handling of the original property. On copy, the new object's original would refer to itself instead of the unmodified rom buffer. This was causing problems re-using the patched rom before cosmetics to output a rom and zpf at the same time. This isn't a problem with the current randomizer as it always fully repatches the rom for every output. The same issue also applied to the overlay_table, but that had no ill effects.

Todo

• Test compressed roms. DMA changes for compressor?
• Update pointers outside scene files to globals in them, mainly cutscenes.
• If 2 doesn’t fix problems moving file start addresses, figure that out
• Integrate framework with rando patching pipeline
• Move cutscene patches to framework
• Move MQ patching to framework
• Move any other scene changes to framework
• Add setting for skulltulas ignoring time of day. Add actors/objects where needed and patch all actor parameters.

Rando Patches to Test with New System

• Cutscene Patching
  • Cutscenes
    • Lost Woods Bridge Saria's Gift
    • Song cutscenes without songs as items
    • Song cutscenes with songs as items
    • Open Royal Family Tomb as both adult/child
    • Darunia's Dance for Saria's Song
    • Owl warps
    • Zelda escaping from Hyrule Castle for OoT check
    • Small cutscene after learning OoT song
    • Epona race start
    • Epona escapes to different Hyrule Field entrances
    • Burning Kak intro cutscene
    • Well draining cutscene (Kakariko)
    • Nabooru knuckle defeat
    • Rainbow bridge
    • Trial completion cutscenes
    • Ganon's Tower collapse
    • Phantom Ganon blue warp Deku Sprout cutscene skip
  • Cutscenes outside scene files
    • Jabu Jabu swallowing Link
    • Ruto pointing to dungeon reward in Big Octo room
    • Opening Door of Time
    • Master Sword pedestal cutscene
    • Well draining cutscene (Windmill)
  • wondertalk2 actor moves
    • Shadow Temple whispering maze (8x)
    • Shadow Temple Truthspinner (2x)
    • GTG Entrance (3x)
    • GTG Stalfos room (1x)
    • GTG Flame Wall Maze/Slopes Room (1x)
    • GTG Pushblock Room (2x)
    • GTG Rotating Statue Room (1x)
    • GTG Megaton Statue/Back Enemies Room (1x)
    • GTG Lava Room (3x)
    • GTG Dinolfos Room (1x)
    • GTG Inner Maze (1x)
    • GTG Shellblade Room/Toilet (1x)
    • Death Mountain Crater (1x)
    • Hideout Cells (1 per cell room)
• Other Patches
  • Duplicate Bazaar room for Kakariko
  • Move Sheik from pedastal in ToT
  • Ice Cavern alcove camera (duplicated in two spots in Patches.py)
  • Fire Temple boss loop unlocked door without keysanity
  • Non-MQ Water Temple door always unlocked
  • Graveyard ledge grabs
  • Owl removals
  • Jabu octorok position
  • Forest/Fire Temple switch heights
  • Kakariko carpenter starting position
  • Vanilla DC gossip stone fairy flag
  • Colossus Fairy entrance "...???" text
  • Forbid Sun's song in a bunch of cutscenes
  • Move Fado for adult trade shuffle
  • Spirit Shortcut actor tweaks
  • Gerudo Fortress gate guard reposition
  • Skip child stealth crawlspace exit
  • Silver rupee shuffle MQ DC/Spirit temp->permanent flags
  • Well ladder rupee reposition No longer relevant as of Fix for ladder cutscene softlock vanilla bug #2551
  • Shadow Temple redead shared flags for silver rupee shuffle
  • Song shuffle cutscene text boxes
  • Song shuffle location addresses
  • Shopsanity shop item objects in room headers
  • Cow shuffle repositions
    • Stable
    • Tower
    • Shuffled item actor params in set_cow_id_data
  • CSMC repositions
    • Vanilla Ganons Castle Light Trial
    • Vanilla Spirit Temple compass chest
    • Silver Gauntlet chest in glitched logic
  • Dead Hand spawns in vanilla Shadow/Well
  • Broken drops vanilla Spirit deku shield in anubis room
  • TCG shuffle temp flags to permanent/keysy
  • Remove "entrance blockers"
  • Scrub shuffle actor params in set_deku_salesman_data
  • Jabu stone actor
  • Keysy dungeon/boss doors
  • Ganons Tower first BK door unlock for pot shuffle
• Entrance Shuffle
  • Generate/write exit list for each scene
  • Jabu boss exit coordinates
  • Water Temple boss exit room number
  • Redirect LLR exits to main exit for OW shuffle
  • Redirect ZR<->Field exits to land from water
  • Spirit temp flag purge on entry through front door
  • Grotto actor data changes in set_grotto_shuffle_data
• MQ Patching
  • All rooms can be entered from all entrances
    • Deku Tree
    • Dodongo's Cavern
    • Jabu Jabu's Belly
    • Forest Temple
    • Fire Temple
    • Water Temple
    • Spirit Temple
    • Shadow Temple
    • Bottom of the Well
    • Ice Cavern
    • Gerudo Training Ground
    • Ganon's Castle
  • Ice Cavern scene header patch from old system
  • MQ Spirit Temple room 6 new alternate header
  • Shadow Temple MQ redead shared flags for silver rupee shuffle
  • DC MQ door flag move for silver rupee shuffle
  • Spirit Temple MQ front right chest temp -> permanent flag for silver rupee shuffle
  • Key doors are correct
  • Check overrides work
  • Scrub actor patching works
  • Cow actor patching works
  • Jabu MQ dungeon reward model override works
  • Keysy removes locks
• ASM/C patches
  • 0x26c10e3 - generic grotto ACTOR_EN_GS actor params 0x3818 -> 0x38FF (use grotto ID for hint text ID)

[r0bd0g]

I guess for logic you'd go around to each skulltula you affected that checks for night and change at_night to (at_night or [setting]). Something to look out for is the watchtower, which will no longer be climbable in child day without a means of killing the skulltula. In closed forest, the time of day access check is skipped, so there's special logic on the night skull there to confirm you can actually get it to night somehow, and you'd also have to add the new setting as an alternative for that.

[mzxrules]

man, rando sure is a hot mess when it comes to organization.

I find it kind of odd to just stuff these important type definitions after a bunch of function patching routines split out of patches.py. I think we need to consider moving this type access stuff into a new folder like "ztypes" or something

[mracsys]

You're right, I should have put more effort into structure. Some of the other utility classes like Vec3s and Vec3i that didn't fit well got put into Rom.py despite not fitting there either. I can refactor into the same file/folder structure as decomp for structs lifted from there, with a top level folder to keep it neat.

[r0bd0g]

I warned you about special logic that would be needed for KF GS Know It All House! As you have it currently, in closed forest, if you start at daytime, under this new setting the skulltula will be there but you won't have logic to collect it before beating deku or finding sun's song. That skull has an extra check on it to make sure it's actually present since closed forest breaks the assumption that you will always have access to moving time of day from spawn, so you need to add the new setting as an alternative way to have the time of day to access it.

(This isn't really related to your PR, you shouldn't change anything about this, but I wonder if all of those skulltulas that check for ToD in areas where time moves already, should just have the time of day check removed. It was nice to look at before but with the new setting it's become kind of bloated. I made a choice along those lines with Peahats when reviewing enemy drop shuffle b/c separate night and day logic for them was pretty convoluted for no real benefit.)

I don't know enough about glitched to say whether or not the logic is wrong absolutely for sure or not, but at a glance I don't trust it. For example, man on roof, you check for a weapon to kill the skulltula, but you always are checking it regardless of if the skulltula is potentially there at daytime. You should get a glitched logic expert to review it, or make no glitched logic changes and mark the settings as incompatible.

(Unrelated to your PR but I was glancing over the night skulls in glitched logic and noticed some logic for buying the Giant's Knife that is assuming that the Knife isn't shuffled. This shows the dangers of marking new settings as glitched logic compatible without getting a specialist's review!)

[fenhl]

Should this setting be enabled in the Easy Mode preset?

[mracsys]

It would make sense. Would that need to go through some sort of vote or poll?

[fenhl]

Current dev team call is in favor so I'd say you can just add it.

[shirosoluna]

@r0bd0g >

I don't know enough about glitched to say whether or not the logic is wrong absolutely for sure or not, but at a glance I don't trust it. For example, man on roof, you check for a weapon to kill the skulltula, but you always are checking it regardless of if the skulltula is potentially there at daytime. You should get a glitched logic expert to review it, or make no glitched logic changes and mark the settings as incompatible.

(Unrelated to your PR but I was glancing over the night skulls in glitched logic and noticed some logic for buying the Giant's Knife that is assuming that the Knife isn't shuffled. This shows the dangers of marking new settings as glitched logic compatible without getting a specialist's review!)

I wouldnt worry too much about glitched logic right now, because it's being updated entirely to have all settings included. There are way more important stuff that isnt compatible at all currently and in released versions. If this is going to be slated for 9.0 as well then we can make it incorporated. I dont know the timing for this PR.
Just offering support to not be concerned about glitched logic too too much atm

[fenhl]

There's not really any timing for this PR, it will be merged once it's been code reviewed and sufficiently tested. Since it's a rather large change, that might take a while, but so is glitched logic 3.0, so it's hard to say which is more likely to be merged first. I'd recommend not including the changes for this in glitched logic 3.0 for now or perhaps maintaining them separately.

[shirosoluna]

There's not really any timing for this PR, it will be merged once it's been code reviewed and sufficiently tested. Since it's a rather large change, that might take a while, but so is glitched logic 3.0, so it's hard to say which is more likely to be merged first. I'd recommend not including the changes for this in glitched logic 3.0 for now or perhaps maintaining them separately.

wasn't meaning that it would wait for it but that I'd incorporate it into logic when the time comes. Even if it's after being pulled.
They are maintained separately. But if new glitch logic comes later then still dont bother worry about current glitched logic. It's too incompatible with everything currently and one more setting wouldnt make a difference lol.

[fenhl]

I'll see if we can get a web generator for this branch set up to encourage more testing.

[mracsys]

FYI, I don't have much free time through at least fall, possibly into next year. I'm still around, but changes might be delayed.