Complete rework of localization approach. New feature: Clickable transport markers

[mrmaffen]

This fixes #18 and lots more. I reworked the whole approach to how this addon is localized.

• Zone, dungeon and worldboss names are directly taken from pfDB whenever possible. Localized zone name matching should therefore no longer be a problem
• Whenever pfDB doesn't have a translated string available, it's still possible to add it manually in Locale_xxXX.lua. See "Windhorn Canyon" in Locale_deDE.lua (Edit: This particular line has been removed again, but it still shows how manual zone translations work)
• String parameters are used to construct phrases like "Boat to %Zone%", or "%Dungeon% - Armory", which enables translators to use the zone name from pfDB while still having full control over the wording.
• I have also made it possible to add custom localized strings for special cases. See "-- translations that need manual adjustment" in Locale_deDE.lua. The default way of constructing "Boat to %Zone%" didn't suffice here. This fine-grained control over each translation will hopefully also be helpful in other languages.
• All translations have an English fallback. String param mask and names.
• I have also added deDE, esES, ptBR and zhCN translations. I'm a native German speaker. Only know a little Spanish and Portuguese, but I'm quite confident the translations should be okay. Copied most of it from the existing files. Did my best with Chinese, but I wouldn't be surprised if something went horribly wrong. All of these, except German, could benefit from review.
• As a minor note: I have also adjusted some zeppelin and boat coordinates slightly, as well as splitting the combined zeppelin markers in two. This helps players to remember where the zeppelin they are looking for will dock.

Edit: Transport markers can now be clicked and it will take you to the destination's zone map. I have included this commit here, because I wanted to avoid merge conflicts.

[mrmaffen]

Just as a note that might help with gauging the stability of this PR: I have tested these changes with all relevant clients (enUS, deDE, esES, ptBR, zhCN) and confirmed proper functionality and working English fallbacks. Could have still missed something ofc. Let me know if you find something :)

[mrmaffen]

I have been using this new version for the last couple days and can confirm that everything seems to work as it should. One thing I noticed is that "Lower Karazhan Halls" seems to be the only zone-name that isn't available in pfDB (except for the newly added "Windhorn Canyon" and "Frostmane Hollow", but that's to be expected). Not sure why that is the case... "Karazhan Crypt" and "Tower of Karazhan" on the other hand are available. Maybe it makes sense to either manually translate "Lower Karazhan Halls" or to use the plain "Karazhan" instead.

[tilare]

Thanks for all your work, I still need some time to test it myself before merging

[mrmaffen]

I just noticed that the InstanceJournal compat stuff needs some tweaking. This isn't ready to push yet

[mrmaffen]

Done a056812

[Digitalvalues]

1. I don't know how limited 1.12.1 API is, but is it possible to depend on already installed pfQuest and refer to it's db instead of using a full local copy?
2. Why do we need all these blobs with all units and all zones names from the game? MMM uses only half of that data. Specifically, map zones and raid bosses.
3. Can only the current locale blob be initialized? Currently all locale blobs are initialized, even when we actually need one of them.

The main concern is ram usage, the 32-bit vanilla client is already starving for memory and constantly crashes even with LAA patch.

Thank you for your contribution.

[mrmaffen]

1. I don't know how limited 1.12.1 API is, but is it possible to depend on already installed pfQuest and refer to it's db instead of using a full local copy?

Done. When pfQuest and pfQuest-turtle are installed this saves 2-3mb

2. Why do we need all these blobs with all units and all zones names from the game? MMM uses only half of that data. Specifically, map zones and raid bosses.

If you really want to shave off every kilobyte, you could write a script that slims the whole pfDB to only the necessary zones and units, but we're talking about 2-3mb of memory here.

3. Can only the current locale blob be initialized? Currently all locale blobs are initialized, even when we actually need one of them.

Good catch. It was easy to implement and saved around 7-8mb of memory usage.

The main concern is ram usage, the 32-bit vanilla client is already starving for memory and constantly crashes even with LAA patch.

Thank you for your contribution.

Didn't know that users were struggling with OOM crashes. My client is actually running pretty stable.

[mrmaffen]

3. Can only the current locale blob be initialized? Currently all locale blobs are initialized, even when we actually need one of them.

Thinking about this some more, I was curious how pfQuest(-turtle) handles this. Turns out ... it doesn't. All locales are always being loaded into memory, which is about 90mb for all enUS clients and 80mb for all others. That definitely needs to change. I'll write a quick PR and mention you :)

[Digitalvalues]

Thank you for the commit.

If you really want to shave off every kilobyte, you could write a script that slims the whole pfDB to only the necessary zones and units, but we're talking about 2-3mb of memory here.

As now we depend on pfDB which already loads everything, it doesn't really make sense to reduce the data in MMM_pfDB fallback, so leave it as it is.

Could you also add ruRU locale? It's the only one missing from the PR, but pfDB supports it.

[mrmaffen]

Could you also add ruRU locale? It's the only one missing from the PR, but pfDB supports it.

No, unfortunately I cannot. After all the atrocities that Russian soldiers have committed in Ukraine and after all the drones that have been sent and are still being sent into apartment buildings just to kill civilians, I will not support Russia in any way and encourage other devs to do the same.

[Digitalvalues]

I don't share your point of view, and I see no correlation between Russian gamers, developers, content localization, and the government's current policy. Especially considering that Russian is one of the top ~5 most spoken languages in the world. Applying the same logic, you should also exclude the deDE locale due to the war crimes committed by Germany.

In any case, thanks for the contirbution, I will add it myself in a separate PR when this one is merged, as it makes sense to include it in this addon localization and we already have database sources.

[mrmaffen]

@Digitalvalues Yes absolutely. If Germany would be committing those war crimes today, I absolutely would want to boycott the whole country as well. I happen to be German myself and know how many decades it took until the stench of guilt lessened. Once Russia has lost this horrible war that they started, they can start apologizing. And maybe in a couple of decades I will be able to trust them again.

Especially considering that Russian is one of the top ~5 most spoken languages in the world.

It seems Russian propaganda has gotten the better of you. Russian is not even in the top 10. Delusions of grandeur ...

[mrmaffen]

Back to coding :D
I have added a simple feature that I always thought would feel very natural with this marker addon. Transport markers can now be clicked and it will take you to the destination's zone map. I have included this commit here, because I wanted to avoid merge conflicts. I will adjust the PR title

[mrmaffen]

The last pfDB commit (47f5bca) has introduced some issues. It seems pfQuest-turtle isn't actually loading its own table under the "xxXX-turtle" key but rather is patching the existing locale table. They don't exist side-by-side and the consequence is that certain zone names no longer are the same like the ones actually used in-game. Either pfQuest-turtle needs patching or we go back to manually loading our own pfDB.

Example to test here is "Dustwallow Marsh". In the German client it's translated as "Marschen von Dustwallow". This is what it's called in pfQuest as well. However pfQuest-turtle patches this for whatever reason and uses the wrong "Düstermarschen" translation.

I'll just write another commit that partially reverts my last pfDB changes so that our own MMM_pfDB is always used. Works perfectly fine there. I'll of course keep the code that skips all the unused locale data, so the memory usage should still be much lower than before.

[Digitalvalues]

I don’t think it is a good idea to always use the duplicated pfDB only because pfDB-turtle is overriding the translation of some zones. Incorrect translation should be fixed on pfDB-turtle side.

I assume you can find examples where pfDB-turtle version is more accurate than pfDB for other languages.

If you really want to address this issue in MMM addon, I think the best way to handle it is to cherry-pick the indexes of wrong translations and introduce a separate table with adjusted translations to be used by MMM when these specific indexes are accessed. Thereby, implement some kind of redirection mechanism.

[mrmaffen]

Yea you're right. I already wrote a short debug function that tells me which translations are missing. When I have the time, I'll add a temporary workaround for MMM_pfDB that provides those missing translations, if pfDB doesn't have them. That will ensure that once pfDB-turtle has been updated, it will automatically be prioritized over those temporary workaround translations. Not sure yet what I'll do about wrong translations (opposed to missing ones). I'll have to analyze how much of a problem they actually are. I'd assume they aren't, but we'll see ...

[mrmaffen]

Okay .. so I had some time to think about this problem and came to the conclusion that it would be best to just extract all needed strings from pfquest- and pfquest-turtle. I didn't like the dependency on other installed addons. This leads to a total mess in no time and is very confusing to non-dev users.
So I thought, why not have a system that takes advantage of the pfquest localization databse, while not being bloated at all and not being dependent on other addons? The result is a small lua helper tool: https://codeberg.org/hyperhumble/pfDB-Localization-Helper

Whenever pfquest or pfquest-turtle gets updated, or you modify MarkerData.lua, you can just run the tool again and all localized strings get extracted. Copy them over, replacing all existing units_filtered.lua and zones_filtered.lua files and you're done.

Temporary fixes of inaccurate pfDB strings can be found in zones_filtered_fixes.lua. Whenever you re-run the helper tool, you might wanna check if some of those fixes have become redundant by now being part of pfQuest-turtle.

Maybe this tool can also help other addon devs to localize their own projects. I'm just hoping we get a few steps closer to consistent translations across the board. In-game and in all addons.

[mrmaffen]

Oh and if you want to accept this PR, I'd suggest you squash the whole thing so that those big pfDB files don't pollute the history. Don't know what the default procedure here on github is...

[Digitalvalues]

Honestly, I don’t see any issues with depending on pfDB. You still depend on it, but now that dependency has become implicit. I think you should combine the best parts of both approaches from these commits: 47f5bca and ab25109.

The idea would be to check for an already initialized pfDB table by default. If it’s not found, then fall back to a filtered MMM_pfDB containing only a few zones and units. Clean and simple.

A helper tool would still be useful for updating this filtered data (btw, no access to it via the provided link).

[mrmaffen]

Honestly, I don’t see any issues with depending on pfDB. You still depend on it, but now that dependency has become implicit.

I strongly disagree. I see lots of issues. If players don't update their pfQuest and/or pfQuest-turtle, other addons that depend on it will break. And since we need to modify and even add to pfDB, you'd need a whole version check system, that somehow determines the pfDB version and then applies the correct patches. You'll get bug reports left and right by confused users that had no idea that their pfQuest version is the cause of some seemingly random zone not showing map markers. I could go on, but you get the point.

Dependency management is hard. Definitely not worth it here.

(btw, no access to it via the provided link).

fixed

[Digitalvalues]

If players don't update their pfQuest and/or pfQuest-turtle

I see your point, seems reasonable. Let’s wait for the approval and merge from @tilare