Bugfixes in preparation for switching to Weblate for translations by kattni · Pull Request #635 · beeware/beeware.github.io

kattni · 2025-05-27T02:58:12Z

This is the first step towards switching the website to Weblate for translations.

As discussed, I have updated the data models to indicate what fields should be translated. I also added a fix to the plugin itself to handle the duplicate message definition error, as the workaround we talked about addressed two out of what turned out to be a significant number of instances of this issue.

I need the following two things from @freakboy3742 at this point:

Verify that I have added translate = True to all applicable fields in the model files. These files are in the flowblocks and models directories. There may be one or two empty ones that I left out in my initial attempt to deal with the above-mentioned error, and I feel at this point that they should be included, so please be aware of that as you go through this.
Please tell me if this is a reasonable fix. I added if msg: to this block. My intention here is to skip generating any sections in the .pot file if there is no content in a given field, to avoid the above-mentioned error. It appears to be working, but I don't understand this enough to really grasp the implications of the change, so I would appreciate your input on it.
- https://github.com/kattni/beeware.github.io/blob/c8d09046ffb05bc40422e9545ecc34cda72aba0e/packages/lektor_i18n_plugin/lektor_i18n.py#L120-L127

Fixes #634

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

kattni · 2025-05-27T02:59:43Z

+        for msg, paths in self.translations.items():
+            if msg:
+                result += "#: %s\n" % " ".join(paths)
+                for token, repl in {'\\': '\\\\', '\n': '\\n', '\t': '\\t', '"': '\\"'}.items():
+                    msg = msg.replace(token, repl)
+                result += 'msgid "%s"\n' % msg
+                result += 'msgstr ""\n\n'
+        return result


@freakboy3742 I added the if msg: to this block, as described in the initial PR comment. Tagging it here for more convenient reference.

freakboy3742

Re Q1: I've flagged a couple of possible omissions in the translation markup.

I wonder if we might want to remove some translations as well. Is it reasonable for translators to keep up with blog posts? Event updates? I'm mostly wary that the PO file is 18k lines long, and because it's all one file, there's no good way to prioritise what is "important" content vs less important.

Re Q2: As far as I can make out, the if msg: change makes sense. The only potential risk I can see is on the "reverse" approach - if the code that tries to process the translation does a lookup of "", and doesn't find a match, does it raise an error? Based on my testing, it doesn't look like it does... so maybe that is success?

freakboy3742 · 2025-05-27T03:29:58Z

 label = Author
 type = string
 width = 1/2
+translate = True


Does the author name need to be translated?

No. I'm not sure why I flagged it. Will remove.

freakboy3742 · 2025-05-27T03:38:31Z

Presumably flowblocks/button-block.ini also has some translation opportunities in the button label?

Ah, right. Yes most likely. As I said, I'm not as familiar with the flowblocks. I wasn't thinking about it.

freakboy3742 · 2025-05-27T03:43:57Z

Not sure if this is the Tor version or the original (it looks like the original), but we should put a note at the top here describing the provenance.

It should be the Tor version. I will add this.

freakboy3742 · 2025-05-27T03:44:37Z

No need to include both of these; as long as we retain the license and describe provenance, we don't have to preserve all the content.

I'll keep this in mind for the next PR. This PR will ultimately not have the plugin in it yet.

kattni · 2025-05-27T20:56:30Z

I looked through the Tor updates. Only one of them has any details beyond the commit title. It appears to resolve the multiple-build issue. Here is what I found:

this moves the compilation of .po files and translation of contents.lr files earlier in the build process, in on_before_build_all instead of on_before_build, to ensure that the build queue accounts for the alternative content files (contents+*.lr) during the build process, instead of requiring multiple (expensive) build runs

the caveat is that translated alternative contents files are now only produced during a full build, which likely breaks "on-the-fly" creation of those files when authoring using the Lektor web admin interface

I am including this here because I don't quite know the implications of the caveat, and I wanted to make sure you were aware as it probably makes more sense to you than me.

Regarding your comments:

I wonder if we might want to remove some translations as well. Is it reasonable for translators to keep up with blog posts? Event updates? I'm mostly wary that the PO file is 18k lines long, and because it's all one file, there's no good way to prioritise what is "important" content vs less important.

I am inclined to agree with this. I initially assumed that there was a lot of consistency across the event updates, which if you base it on the last few tutorials, it's not an incorrect assumption, but looking through the .po file as a whole, this appears to not be the case. We can always add this back in if we get the whole site translated and people are looking for more.

Re Q2: As far as I can make out, the if msg: change makes sense. The only potential risk I can see is on the "reverse" approach - if the code that tries to process the translation does a lookup of "", and doesn't find a match, does it raise an error? Based on my testing, it doesn't look like it does... so maybe that is success?

I'll take maybe for an answer. Obviously if it eventually chokes, we can revisit it.

freakboy3742 · 2025-05-27T22:47:28Z

I am including this here because I don't quite know the implications of the caveat, and I wanted to make sure you were aware as it probably makes more sense to you than me.

I won't profess enough experience with Lektor to fully understand all the consequence of such a change - but it sounds to me like this is a fix for the 'needs 2 builds' problem you've been seeing. Definitely something worth keeping an eye on, though.

kattni · 2025-05-27T22:48:26Z

@freakboy3742 Ok, here are the updates:

I fixed the bugs in the couple of Resources links we found, specifically the lack of titles on the two latest ones, and the space in the directory name. I also updated the missed rST link to be Markdown.
I removed the translate tag from the blog posts and the individual events - I left them on the "main" page models. I believe that means it only applies to the "index" for each of them, but let me know if I'm missing something with this.
I updated the previously-missed /contributing/membership/ link to /membership/. I believe I verified this was a complete change, but please let me know if I missed anything - I was struggling within Git to deal with all the files generated and modified by the plugin, so I may have accidentally dropped one of the changes.
I removed the plugin and the related-config files for the purposes of this PR.

I think this is ready for another look.

kattni · 2025-05-27T22:49:44Z

I won't profess enough experience with Lektor to fully understand all the consequence of such a change - but it sounds to me like this is a fix for the 'needs 2 builds' problem you've been seeing. Definitely something worth keeping an eye on, though.

Understood. That is my take on it as well, however I wanted to ensure you were aware of it in case it ends up affecting our implementation.

freakboy3742

tl;dr - this looks like a completely uncontroversial set of changes. There's likely more work that could be done to improve the pagination even more (what even is the gutter for on those pages?...) - and the translation flags obviously won't do anything yet, other than indicate that things could be translated - but I'd be happy landing this set of changes as is.

kattni · 2025-05-27T23:35:32Z

@freakboy3742 Updated title, no longer draft status.

freakboy3742

Looks good - and one step closer to managed translations!

github-actions · 2025-05-27T23:48:09Z

Visit the preview URL for this PR (updated for commit 46967a9):

https://beeware-org--pr635-weblate-5eia3r6c.web.app

_{(expires Tue, 03 Jun 2025 23:47:54 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: b0da44bc067e7d9a4255c77cb2c5fce572218cec}

kattni · 2025-05-27T23:54:32Z

(what even is the gutter for on those pages?...)

I would also love to know the answer to this. I guess we'll have to ask the person who wrote it?

...the translation flags obviously won't do anything yet, other than indicate that things could be translated...

I verified this before submitting this as-is. They are ignored without the plugin.

freakboy3742 · 2025-05-27T23:56:46Z

(what even is the gutter for on those pages?...)

I would also love to know the answer to this. I guess we'll have to ask the person who wrote it?

Nah - that guy doesn't know anything... 🤣

...the translation flags obviously won't do anything yet, other than indicate that things could be translated...

I verified this before submitting this as-is. They are ignored without the plugin.

👍

johnzhou721 · 2025-06-13T23:10:14Z

I wonder if we might want to remove some translations as well. Is it reasonable for translators to keep up with blog posts? Event updates? I'm mostly wary that the PO file is 18k lines long, and because it's all one file, there's no good way to prioritise what is "important" content vs less important.

I'd definitely be interested to translate the blog posts, event updates etc, once I have more time.

Since we're maintaining the letkor-i18n-plugin, we could add code that splits those into another file (=component on Weblate) and sort them according to the date published that is listed in the lektor source file. When we're merging a little bit more work is needed because AFIAK there's no way for msgmerge to specify a custom sort key. So it'd involve loading the merged unsorted file and resorting it and saving it.

Do you think this would be a good way?

freakboy3742 · 2025-06-14T03:14:01Z

I wonder if we might want to remove some translations as well. Is it reasonable for translators to keep up with blog posts? Event updates? I'm mostly wary that the PO file is 18k lines long, and because it's all one file, there's no good way to prioritise what is "important" content vs less important.

I'd definitely be interested to translate the blog posts, event updates etc, once I have more time.

You're one person, translating one language. I'm very wary of setting up a bunch of complex infrastructure to support one contributor on one language, and thereby setting up expectations for other languages, unless we've got evidence that blog posts will be translated regularly.

Since we're maintaining the letkor-i18n-plugin, we could add code that splits those into another file (=component on Weblate) and sort them according to the date published that is listed in the lektor source file. When we're merging a little bit more work is needed because AFIAK there's no way for msgmerge to specify a custom sort key. So it'd involve loading the merged unsorted file and resorting it and saving it.

Do you think this would be a good way?

Is it possible? Maybe. Is it worth it? I doubt it, for the same reason that we omitted blog posts in the first place.

johnzhou721 · 2025-06-14T12:03:55Z

I agree with you on the first part and intended the second part to be a totally theoretical discussion… There’s some regular work in Spanish going on, but there’s nothing in the other langs so far… it’d make sense for a lot of people to be working on website translations for this to be worth it/

kattni added 2 commits May 26, 2025 21:02

Initial config for i18n plugin.

1b883e8

Update models, fix plugin.

c8d0904

kattni marked this pull request as draft May 27, 2025 02:58

kattni commented May 27, 2025

View reviewed changes

freakboy3742 reviewed May 27, 2025

View reviewed changes

kattni added 6 commits May 27, 2025 16:59

Suggested updates.

e3d2667

Remove space in directory, missed rST link.

91447d4

Various bugfixes, move pagination to page bottom.

6db7681

Update membership link.

94aeb1a

Remove plugin for future inclusion.

b0dae3a

Remove plugin configs for future inclusion.

46967a9

freakboy3742 reviewed May 27, 2025

View reviewed changes

kattni changed the title ~~WIP: Switch to Weblate for translations~~ Bugfixes in preparation for switching to Weblate for translations May 27, 2025

kattni marked this pull request as ready for review May 27, 2025 23:34

kattni requested a review from freakboy3742 May 27, 2025 23:38

freakboy3742 approved these changes May 27, 2025

View reviewed changes

freakboy3742 added the preview Approved for an automated preview label May 27, 2025

freakboy3742 merged commit 51c6a63 into beeware:lektor May 27, 2025
2 checks passed

freakboy3742 mentioned this pull request May 28, 2025

Switch to Weblate for translations #634

Closed

Uh oh!

Conversation

kattni commented May 27, 2025

PR Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kattni commented May 27, 2025

Uh oh!

freakboy3742 commented May 27, 2025

Uh oh!

kattni commented May 27, 2025

Uh oh!

kattni commented May 27, 2025

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

kattni commented May 27, 2025

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 27, 2025

Uh oh!

kattni commented May 27, 2025

Uh oh!

freakboy3742 commented May 27, 2025

Uh oh!

Uh oh!

johnzhou721 commented Jun 13, 2025

Uh oh!

freakboy3742 commented Jun 14, 2025

Uh oh!

johnzhou721 commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants