Miyoo-matic

Play a Miyoo handheld console automatically with solenoids and a webcam.

Requirements

1. Python 3
2. SQLite Tools
3. Ardunio CLI
4. OBS Studio
5. Tesseract

FireRed ROM

PC First Time Setup

1. sqlite3 -init pc/setup.sql miyoomatic.db .quit
2. python -m venv myenv
3. ./venv/Scripts/Activate.ps1
4. pip install -r pc/src/requirements-api.txt -r pc/src/requirements-controller.txt
   • Or, if you're only running one of the two processes on this machine, install just that file.

Subsequent Setup

The app runs as two processes — start each in its own terminal.

1. ./venv/Scripts/Activate.ps1
2. ruff check
3. API (HTTP + WS + static): python pc/src/api.py — serves http://localhost:8001, WebSocket at /ws.
4. Controller — pick a backend:
   • Real console: python pc/src/main.py --backend arduino
   • mGBA on this desktop: python pc/src/main.py --backend emulator

Override the controller's API target with MIYOOMATIC_API=http://host:port.

Arduino Setup

1. arduino-cli config init
2. arduino-cli core update-index
3. arduino-cli core install arduino:avr
4. arduino-cli compile --fqbn arduino:avr:nano:cpu=atmega328old Serial
5. arduino-cli upload -p COM3 --fqbn arduino:avr:nano Serial

Extract Palette From ROM

• They are 16x 15-bit colors compressed with LZ10
• Pointers to all Pokemon palettes are stored in a single table. Its offset is located at 0x130
• Shiny palette pointer is at 0x134 - but it turns out we can use the "normal" address to locate those as well????? pointers are 24 bit?
• palette pointers are ordered according to internal id number, which is the same as the pokedex number for gen1 and gen2 atleast

Fun Palette SQL Queries

• Pokemon that uses a single color the most/least, by percentage
• Most/least common colors
  • By number of palettes they are in
  • By total number of pixels
• Pokemon with the least/most amount of colors
• Group pokemon by their background color
• Pokemon that represent a single color with multiple palette colors
  • Used by some shiny pokemon to add a hidden element?

Camera Settings

• Zoom: 10/10 - Zoom in as much as you can without cutting off any of the screen
• Brightness: 35/100 - I decreased this until the background looked completely dark
• Contrast: 50/100 - Unchanged
• Sharpness: 100/100 - Makes the edges of pixels much clearer
• Saturation: 50/100 - Unchanged
  • TODO: Increased to 75 to get closer to the look of the sprites

Troubleshooting

• Image Too Bright: Open the Windows Camera app, then launch the Python program
• Image Too Zoomed: Open "Manage cameras" and select your camera, ensure focus and close lid and program, then try running again.
• Start the program in the overworld
  • This lets the program best detect the screen's border
• Image Detection Issues
  • Clean the webcam lens and the screen

Testing

• Add unit tests

Resources

• Font from cufonfonts. converted TTF to WOFF online

• https://pseudopencv.site/utilities/hsvcolormask/ So helpful when I was trying to figure out useful HSV mask ranges for gender detection.
• https://voliol.neocities.org/articles/genIIIpalettes How colour palettes work in Fire Red
• https://pokemondb.net/tools/text-list helped me fill my database

Training

• Make training the image recognition model easy to do and update
• It only needs to classify items into 1 of 3 things or unknown, realistically

// TODO: Add setup commands for each script that only happen once to simplify prereqs or verify everything is prepared // TODO: this is giving the PC limited information about the screenshot. // if a script needs to take two different kinds then the PC will need to differentiate them somehow, maybe just have a few // like PrintScreen2, PrintScreen3, etc. and send a different byte over serial line. // Or better yet make the parameter 1,2,3, etc. instead of zero and send over another byte.

// TODO: Read button map from two 8 DIP switches? // TODO: make the delay in terms of ticks? like 60 ticks per second multiples? with a small constant. // start conditions: // floor 3 pokemon tower // menu closed but will hover pokemon if open // text speed is set to fast // have multiple ultra balls // PKMN #1 // - sweet scent as first ability // - must be able to always run (speed >= 67 (Lv20 Haunter max speed TODO: verify this) OR hold smoke ball (diff timing?)) // - else: have it fainted and next alive pokemon slot can always run // pokemon that is used in the encounter // - has single character nickname // - has a move that can sleep or freeze the gastly, near 100% catch rate with ultra ball.

// Note: doesn't work on ghost since it's a normal move. // - scyther with false swipe leaves opponent at 1HP, 100% catch rate with Great Ball

// TODO: maybe I can send a command to python saying I will take at least the sum of all the millseconds i // delay for, so you can go ahead and sleep your thread for that long and then wake up and start checking // for a message every 10ms or so. so the main thread isn't taking up too much CPU time. // TODO: NOOP command. would allow for easily commenting out commands from a script if not needed. // for example if im somewhere in the desert and it says the sun is beating or has a longer animation. I can comment something out // rather than changing the entire thing. id have to add to the original delay for example. and lose what the original delay value was. // or I could generate the script from a function that takes inputs like current weather event, etc.

// TODO: replace for loop with a variable for remembering the index and then just modulus that in my void loop. // that lets me interrupt the flow of the program more to compute other things in between script actions. // maybe I could use 7 seg display to show iterations // show

// TODO: Read a dip switch to select the script by index // int catch[12][2] = {};

// start conditions: // have >= 50 coins // at a slot machine with no coins yet inserted // Stop when i have >= 9900 coins (enough to buy 55 ABRAs) // Note: ~150 complete buys to probably receive a shiny. int slots[8][2] = { { Down, 100}, // coin { Down, 100}, // coin { Down, 100}, // coin { A, 100}, // pull { A, 100}, // pull { A, 100}, // pull // TODO: I think holding A fast forwards the coin payout... solenoids are not ideal for staying active though... {PrintScreen, -1}, // over serial: tell PC to take a photo { Wait, -1}, // wait for a message over serial // The PC will check the screenshot for number of coins won and use that to determine additional wait time before sending a resume command over serial // Or should the PC reply with a byte that can be used by the arduino to determine how long to wait? };

// TODO: Figure out how long an in-game step and a turn take and make those commands of their own eg. turnwalkup2, turnwalkup3, walkup1, walkup2, turnrunleft5, etc. // TODO: perma-run? can i always hold run with a 3d printed thing? no solenoid on it? // TODO: if i have an extra button outdoors maybe i can press select for my bike // TODO: cat the scripts: slot machine script + walk to prize corner + walk to casino + buy abra

// Down, Left, Up, A int walkToGameCorner[8][2] = { {Down, 1500}, // exit shop, turn + down 5 steps // Animation of door exit {Down, 200}, // down 1 step {Left, 1500}, // turn + left 5 steps { Up, 200}, // turn + up 1 step // Animation of door enter // Note: I could choose to go 3 up and right but that requires a new button. { Up, 1100}, // up 4 steps {Left, 1500}, // turn + left 2 steps { A, 600}, // select slot machine { A, 1300}, // yes to dialog };

int walkToPrizeCorner[2][2] = { {B, 600}, // exit slot machine {A, 1200}, // confirm dialog // turn + right 2 // turn + down 5 // turn + right 5 // turn + up 2 // up 4 };

// TODO: make a function that takes a string, A-Z a-z space 0-9 !?/-./ quotation mark and apostrophe open and close, ellipsis, male, female // int giveNickname[][2] = {

// };

// start conditions // have at least 55 spaces in your PC open // make it so the pokemon is inserted in the empty boxes. // TODO: make sure I dont run into any NPCs int walkToPokemonCenter[1][2] = { {Down, 200}, // down 1 step // turn + right 4 // turn + up 10 // turn + right 5 // turn + up 1 // up 3 // turn + right 4 // turn + up 3 // boot up PC // skip dialog // choose Bill's // skip dialog // skip dialog // down // down // select move pokemon };

// Whatever current box i am in // box is full of pokemon // cursor located at top right // if a shiny is in the box it will not be released and the arduino will be stuck waiting hovering that pokemon

// TODO: develop something to go to the next box, maybe part of the input can be the number of pokemon to check (55) // TODO: have a function fill the data of this array since it will be hundreds of commands long. int releaseNonShiny[][2] = {

};

// now i can either just check the color of the sprite or an easier way would likely be to open the summary and check for the star or a blue border instead of purple // a, select pokemon // down to summary // click summary // screenshot // press A to exit summary // wait // if not shiny: // press a on pokemon // down 4 or up 2 // press A on release // press up // press A // skip dialog (up arrow worked too) // skip dialog // end if // go right // if i != 0 and i % 5 == 0: // go down //};

// start conditions: // - have at least 180 coins // looking at a full box one to the left of the empty "RIP Abra" box // "caught pokemon will be sent directly to the PC, to whichever box you last accessed" // Note: buyClefairy, buyDratini, buyScyther and buyPorygon will all be very similar. int buyAbra[7][2] = { {A, 500}, // talk {A, 100}, // skip dialog {A, 100}, // select abra {A, 100}, // buy abra {B, 100}, // skip nickname OR press down then A (useful if I can't spare a B button, but will be slightly slower) // {Down, 100}, // {A, 100}, {A, 100}, // skip abra being put in PC // NOTE: Once you have filled the currently hovered box there will be an additional dialog! {A, 100}, // skip dialog // TODO: };

TODO: alternate loop style

while True:

try:

msg = incoming.get_nowait() # non-blocking

print(msg)

except queue.Empty:

pass

#TODO read https://docs.python.org/3/library/queue.html implementation and how it works with threads

State Machine

Overworld

Menu > Pokedex Menu > Pokemon Menu > Bag Menu > Michael Menu > Save Menu > Option Menu > Exit

Pokemon > #1 Pokemon > #2 Pokemon > #3 Pokemon > #4 Pokemon > #5 Pokemon > #6 Pokemon > Cancel

Pokemon > #1 > Menu > Summary Pokemon > #1 > Menu > SWEET SCENT Pokemon > #1 > Menu > Switch Pokemon > #1 > Menu > Item Pokemon > #1 > Menu > Cancel

Sweet Scent

Pokemon Appeared > Gastly Regular Pokemon Appeared > Gastly Shiny Pokemon Appeared > Cubone Regular Pokemon Appeared > Cubone Shiny Pokemon Appeared > Haunter Regular Pokemon Appeared > Haunter Shiny

Encounter > Fight Encounter > Bag Encounter > Pokemon Encounter > Run

Dialog

mGBA Emualtor

1. pip install pywin32 pydirectinput
2. Open mGBA with Fire red and your save file
3. Load game and get to overworld.
4. Open menu and have it hover Pokemon.
5. Set frame size to 1x
6. Emulation speed to 1x
7. Mute the emulator for your own sanity
8. python src/pc/emulator.py

Tests

TODO: Add test for is_menu_open, should return False for all templates except when the menu is open returns True

TODO: Add test for this, should return -1 for all templates except when the menu is open. Make the filename equal to the expected menu index returned by this function

NOTE: screenshots from mGBA are not compressed PNGs, PNGGauntlet reduces their size by a third to around 6kB.