Move documentation to new docs folder, remake Ghidra scripts with Ghidrathon (#30)

* Remove TODO that was completed

* Move building and contributing instructions into new `docs` folder

* Added WIP instructions for Ghidra

* Add link for decompiler setup to `BUILDING.md`, improve formatting

* Add Ghidra `export_functions_as_csv.py` port

* Rename scripts to remove `_in_ida` suffix

* Fix export functions script

* ignore automatically named functions

* fix formatting of address (prepend `0x`)

* `rename_functions` ported to ghidra

* Make docs say to change image base before opening

* Fix docs numbering

* Fix Ghidra function name prefixes

* Fix link markdown syntax

* fix rename functions to use correct memory

* port rename data function

* Add descriptions to Ghidra scripts

* Port export_data_as_csv script to ghidra

* Remove `export_names_as_syms.py` (not for LCE decomp)

* fix function addrs being lower case

* fix quality always being `U`

* Remove print

* Fix config function call

* Update instructions for Ghidra Switch Loader

* Fix rename function script to use block and not address space

* Add multichunk deletion back

* overwrite default functions

* add IDA instructions

Author: boreals-back-again

README.md

@@ -33,70 +33,12 @@ The Nintendo Switch Edition release only lasted for a year, before being replace
 While the latest version of Nintendo Switch Edition is v1.0.17, that version only matches up with Wii U Edition's Patch 35 (v560). Wii U Edition would get a further 8 updates (up until v688) before LCE was abandoned for good.
 
 ## Building
-The decomp toolchain was created for Linux & MacOS users. While it isn't a hard requirement, if you're running Windows its advised that you [setup WSL](https://learn.microsoft.com/en-us/windows/wsl/install) with an Ubuntu-like distro for the easiest setup.
-
-The instructions below assume you're running a form of Linux (WSL or native).
-
-### 1. Set up dependencies
-
-* Python 3.6 or newer with [pip](https://pip.pypa.io/en/stable/installation/)
-* Ninja
-* CMake 3.13+
-    * If you are on Ubuntu 18.04, you must first [update CMake by using the official CMake APT repository](https://apt.kitware.com/).
-* ccache (to speed up builds)
-* xdelta3
-* clang (for compiling Rust tools)
-
-Ubuntu users can install those dependencies by running:
-
-```shell
-sudo apt install python3 ninja-build cmake ccache xdelta3 clang libssl-dev libncurses5
-```
-
-Additionally, you'll also need:
-
-* A Rust toolchain ([follow the instructions here](https://www.rust-lang.org/tools/install))
-* The following Python modules: `capstone colorama cxxfilt pyelftools ansiwrap watchdog python-Levenshtein toml` (install them with `pip install ...`)
-
-### 2. Set up the project
-
-1. Clone this repository. If you are using WSL, please clone the repo *inside* WSL, *not* on the Windows side (for performance reasons).
-
-2. Run `git submodule update --init --recursive`
-
-    Next, you'll need to acquire the **original v1.0.17 `main` NSO executable**.
-
-3. Run `tools/setup.py [path to the NSO]`
-    * This will:
-        * install tools/check to check for differences in decompiled code
-        * convert the executable if necessary
-        * set up [Clang 4.0.1](https://releases.llvm.org/download.html#4.0.1) by downloading it from the official LLVM website
-        * create a build directory in `build/`
-    * If something goes wrong, follow the instructions given to you by the script.
-    * If you wish to use a CMake generator that isn't Ninja, use `--cmake_backend` to specify it.
-
-### 3. Build
-
-To start the build, just run
-
-```shell
-ninja -C build
-```
-
-By default, a multithreaded build is performed.
-
-To check whether everything built correctly, just run `tools/check` after the build completes.
+See [BUILDING.md](docs/BUILDING.md).
 
 ## Contributing
-As this project is in its very early stages, its hard to put guidelines on something that will evolve over time as contributors gain a better understanding of the game's internals.
-
-While almost all source paths aren't known, a few .cpp file names do appear in the Wii U Edition's global static constructors, along with a couple asserts giving file paths. Follow these file names wherever possible.
-
-Another point of reference is the Switch Edition, showing us the file path to `Minecraft.World/Calendar.cpp` and some other files in that folder. World generation should be stored in this folder based on `MinecraftWorld_RunStaticCtors` routing all worldgen static constructors inside of the folder.
-
-For another point of reference, you could also look at [the Minecraft: Pocket Edition decompilation](https://github.com/MCPE-RE/0.1.3j-core) for more inspiration. As a matter of fact, a lot of classes from that decompilation project share function names with this game's symbols. Both editions originated from Notch's messy Minecraft beta code, so it's no wonder that a lot of names were probably copied from Mojang's official mappings at the time.
+See [CONTRIBUTING.md](docs/CONTRIBUTING.md).
 
-##
+---
 
 Minecraft LCE Decompilation logo by [@break-core](https://github.com/break-core?tab=repositories)
 

docs/BUILDING.md

@@ -0,0 +1,58 @@
+# Building
+The decomp toolchain was created for Linux & MacOS users. While it isn't a hard requirement, if you're running Windows its advised that you [setup WSL](https://learn.microsoft.com/en-us/windows/wsl/install) with an Ubuntu-like distro for the easiest setup.
+
+The instructions below assume you're running a form of Linux (WSL or native).
+
+## 1. Set up dependencies
+
+* Python 3.6 or newer with [pip](https://pip.pypa.io/en/stable/installation/)
+* Ninja
+* CMake 3.13+
+    * If you are on Ubuntu 18.04, you must first [update CMake by using the official CMake APT repository](https://apt.kitware.com/).
+* ccache (to speed up builds)
+* xdelta3
+* clang (for compiling Rust tools)
+
+Ubuntu users can install those dependencies by running:
+
+```shell
+sudo apt install python3 ninja-build cmake ccache xdelta3 clang libssl-dev libncurses5
+```
+
+Additionally, you'll also need:
+
+* A Rust toolchain ([follow the instructions here](https://www.rust-lang.org/tools/install))
+* The following Python modules: `capstone colorama cxxfilt pyelftools ansiwrap watchdog python-Levenshtein toml` (install them with `pip install ...`)
+
+## 2. Set up the project
+
+1. Clone this repository. If you are using WSL, please clone the repo *inside* WSL, *not* on the Windows side (for performance reasons).
+
+2. Run `git submodule update --init --recursive`
+
+    Next, you'll need to acquire the **original v1.0.17 `main` NSO executable**.
+
+3. Run `tools/setup.py [path to the NSO]`
+    * This will:
+        * install tools/check to check for differences in decompiled code
+        * convert the executable if necessary
+        * set up [Clang 4.0.1](https://releases.llvm.org/download.html#4.0.1) by downloading it from the official LLVM website
+        * create a build directory in `build/`
+    * If something goes wrong, follow the instructions given to you by the script.
+    * If you wish to use a CMake generator that isn't Ninja, use `--cmake_backend` to specify it.
+
+## 3. Build
+
+To start the build, just run
+
+```shell
+ninja -C build
+```
+
+By default, a multithreaded build is performed.
+
+To check whether everything built correctly, just run `tools/check` after the build completes.
+
+## 4. Decompilers
+
+For instructions on setting up a decompiler, see [DECOMPILERS.md](DECOMPILERS.md).

docs/CONTRIBUTING.md

@@ -0,0 +1,8 @@
+# Contributing
+As this project is in its very early stages, its hard to put guidelines on something that will evolve over time as contributors gain a better understanding of the game's internals.
+
+While almost all source paths aren't known, a few .cpp file names do appear in the Wii U Edition's global static constructors, along with a couple asserts giving file paths. Follow these file names wherever possible.
+
+Another point of reference is the Switch Edition, showing us the file path to `Minecraft.World/Calendar.cpp` and some other files in that folder. World generation should be stored in this folder based on `MinecraftWorld_RunStaticCtors` routing all worldgen static constructors inside of the folder.
+
+For another point of reference, you could also look at [the Minecraft: Pocket Edition decompilation](https://github.com/MCPE-RE/0.1.3j-core) for more inspiration. As a matter of fact, a lot of classes from that decompilation project share function names with this game's symbols. Both editions originated from Notch's messy Minecraft beta code, so it's no wonder that a lot of names were probably copied from Mojang's official mappings at the time.

docs/DECOMPILERS.md

@@ -0,0 +1,15 @@
+# Decompilers
+
+## IDA Pro
+1. Install `nxo64.py` from `reswitched/loaders`. Instructions are available [here](https://github.com/reswitched/loaders?tab=readme-ov-file#installation)
+2. Open `ida64.exe` and disassemble `main.nso`.
+
+## Ghidra
+### Setup Instructions
+
+1. Setup `Ghidrathon`. Instructions are available [here](https://github.com/mandiant/Ghidrathon#installing-ghidrathon). Make sure `Ghidrathon` has access to the `toml` package.
+2. Setup `Ghidra Switch Loader`. Instructions are available [here](https://github.com/Adubbz/Ghidra-Switch-Loader).
+3. Create a new project. When importing `main.nso`, click `Options`. Then set `Base Address` to `0x7100000000`.
+4. Open the `Script Manager` (green play icon) from Ghidra's toolbar.
+5. Click on Manage Script Directories (bullet points icon) inside the Script Manager, and add `tools/ghidra`. Do not add `tools/common/ghidra_scripts`.
+6. Select the `NX-Switch` category, and use the scripts provided. Information and instructions for each individual script are shown inside the Script Manager.

tools/ghidra/export_data_as_csv.py

@@ -0,0 +1,46 @@
+# Exports the .bss data to a CSV
+#@author Boreal
+#@category NX-Switch
+
+import csv
+import os
+
+from ghidra.app.script import GhidraScript
+from ghidra.program.model.symbol import SymbolType
+from ghidra.program.model.address import AddressSet
+
+# this is java.io.file
+OutputFilePathObject = askFile("Save CSV file", "OK")
+output_file_path = OutputFilePathObject.getAbsolutePath()
+
+def is_valid_name(name: str) -> bool:
+    return not name.startswith(("DAT_"))
+
+listing = currentProgram().getListing()
+
+def get_data_heads(block):
+    start = block.getStart()
+    end = block.getEnd()
+    
+    addr = start
+    
+    print(start)
+    print(end)
+
+if output_file_path:
+    with open(output_file_path, 'w', newline='') as csvfile:
+        fieldnames = ['Address', 'Name']
+        csvwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
+
+        bss = currentProgram().getMemory().getBlock(".bss")
+        symbols = currentProgram().getSymbolTable().getSymbols(AddressSet(bss.getAddressRange()), SymbolType.LABEL, True)
+
+        for data in symbols:
+            csvwriter.writerow({
+                'Address': f'0x{data.getAddress().getOffset():016x}',
+                'Name': data.getName()
+            })
+        
+    print(f"CSV file '{output_file_path}' has been generated.")
+else:
+    print("Operation cancelled by the user.")

tools/ghidra/export_functions_as_csv.py

@@ -0,0 +1,50 @@
+# Export functions to a CSV (CURRENTLY NOT MATCHING IDA!!!)
+#@author Boreal
+#@category NX-Switch
+
+import csv
+import os
+
+from ghidra.app.script import GhidraScript
+
+# this is java.io.file
+OutputFilePathObject = askFile("Select the CSV file to update, checked functions will be preserved", "OK")
+output_file_path = OutputFilePathObject.getAbsolutePath()
+
+def is_valid_name(name: str) -> bool:
+    return not name.startswith(("sub_", "nullsub_", "j_", "FUN_", "thunk_FUN_"))
+
+# Load the existing csv if it exists
+existing_data = {}
+if output_file_path and os.path.exists(output_file_path):
+    with open(output_file_path, 'r', newline='') as csvfile:
+        reader = csv.DictReader(csvfile)
+        for row in reader:
+            address = row['Address']
+            existing_data[address] = row
+
+if output_file_path:
+    with open(output_file_path, 'w', newline='') as csvfile:
+        fieldnames = ['Address', 'Quality', 'Size', 'Name']
+        csvwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
+        csvwriter.writeheader()
+
+        for func in currentProgram().getListing().getFunctions(True):
+            func_name = func.getName() 
+            func_size = func.getBody().getNumAddresses()
+            address_raw = func.getEntryPoint().getOffset()
+            address = f"0x{address_raw:016x}"
+
+            quality = existing_data.get(address, {}).get('Quality', 'U')
+
+            csvwriter.writerow({
+                'Address': address,
+                'Quality': quality,
+                'Size': str(func_size).zfill(6),
+                'Name': func_name if is_valid_name(func_name) else ''
+            })
+
+    print(f"CSV file '{output_file_path}' has been updated.")
+else:
+    print("Operation cancelled by the user.")
+

tools/ghidra/rename_data.py

@@ -0,0 +1,39 @@
+# Renames data to those from the CSV
+#@author Boreal
+#@category NX-Switch
+
+import csv
+import os
+
+from ghidra.app.script import GhidraScript
+from ghidra.program.model.symbol import SourceType
+
+# this is java.io.file
+InputFilePathObject = askFile("Import from CSV file", "OK")
+input_file_path = InputFilePathObject.getAbsolutePath()
+
+if input_file_path:
+    with open(input_file_path, 'r', newline='') as csvfile:
+        csvreader = csv.reader(csvfile)
+
+        print(csvfile)
+
+        for row in csvreader:
+            if len(row) < 2:
+                continue
+
+            address_str, name = row[0], row[1]
+
+            if not address_str or not name:
+                continue
+
+            try:
+                address = toAddr(int(address_str, 16))
+            except ValueError:
+                print(f"Invalid address format: {address_str}")
+                continue
+            
+            data = getDataAt(address)
+
+            if data is not None:
+                data.getPrimarySymbol().setName(name, SourceType.USER_DEFINED)

tools/ghidra/rename_functions.py

@@ -0,0 +1,76 @@
+# Renames functions to those from the CSV
+#@author Boreal
+#@category NX-Switch
+
+import csv
+import os
+
+from ghidra.app.script import GhidraScript
+from ghidra.program.model.symbol import SourceType
+from ghidra.program.model.address import AddressSet
+
+common_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "common"))
+
+# util collides with java.util so this has to be done
+import importlib
+config_path = os.path.join(common_path, "util/config.py")
+spec = importlib.util.spec_from_file_location("config", config_path)
+config = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(config)
+
+adf = currentProgram().getAddressFactory()
+mem = currentProgram().getMemory()
+text_block = mem.getBlock(".text")
+
+csv_path = config.get_functions_csv_path()
+
+function_manager = currentProgram().getFunctionManager()
+
+
+def delete_multichunk_funcs():
+    for func in function_manager.getFunctions(True):
+        if func.getBody().getNumAddressRanges() > 1:
+            function_manager.removeFunction(func.getEntryPoint())
+            print(f"deleted multichunk function {func.getEntryPoint().getOffset():016x}")
+
+delete_multichunk_funcs()
+
+def can_overwrite_name(new_name: str):
+    if new_name == "":
+        return False
+
+    return True # we have to allow wii u symbols
+
+with open(csv_path, "r") as f:
+    reader = csv.reader(f)
+    next(reader)
+
+    prev_func = None
+
+    for fn in reader:
+        raw_addr = int(fn[0], 16)
+        size = int(fn[2])
+        name = fn[3]
+
+        addr = text_block.getStart().getNewAddress(raw_addr)
+        func = function_manager.getFunctionAt(addr)
+        
+        if func is None:
+            print(f"Creating function at {addr} {name}")
+            try:
+                func = function_manager.createFunction(None, addr, AddressSet(addr, addr.add(size - 1)), SourceType.USER_DEFINED)
+            except:
+                print(f"Creating function at {addr} failed")
+                continue
+
+        elif func.getEntryPoint() != addr:
+            print(f"Fixing function at {addr} with name {name}")
+            prev_func.setBody(AddressSet(prev_func.getEntryPoint(), func.getBody().getMaxAddress().add(1)))
+            func = function_manager.createFunction(None, addr, AddressSet(addr, addr.add(size - 1)), SourceType.USER_DEFINED)
+
+        if can_overwrite_name(name):
+            print(f"Renaming {addr} to {name}")
+            func.setName(name, SourceType.USER_DEFINED)
+            
+        prev_func = func
+    print("Renaming from functions completed.")