Change i18n-extract-messages build step to merge components

[pwithnall]

See #840 (comment)

The problem

Currently, the kolibri-tools i18n-extract-messages build utility is called multiple times by kolibri-explore-plugin, once for each component (template-ui, ek-components, the app). This was implemented in commit d4d678d because the utility only supports one --namespace/--searchPath argument pair.

This setup extracts the messages from each component into a CSV file named after the component, e.g. ek-components-messages.csv. This is then translated and committed as ek-components-messages.json.

At runtime, the webpack_asset tag (e.g. {% webpack_asset 'kolibri.core.default_frontend' %} in base.html) generates some inline JavaScript which calls kolibriCoreAppGlobal.registerLanguageAssets() on the translated messages, which are loaded from the JSON file and injected inline into the page in that call.

There is only a webpack asset for kolibri_explore_plugin.app, there is no webpack asset for ek-components or template-ui. This means that none of the runtime loading code actually loads the ek-components/template-ui messages in, and hence they are not translated at runtime.

There is prior art in Kolibri for how to handle translations from a sub-package, in the form of its kolibri-common package, which contains some common UI elements (and translations). It’s hooked in to the generated JSON file by kolibri-tools i18n-extract-messages: when passed an additional --namespace and --searchPath argument (for kolibri-common), it will extract messages from that namespace too, and bung them into the same output file.

This can be seen by doing ack -l searchPlaceholder in kolibri-explore-plugin vs ack -l resourcesUnavailable in Kolibri. The latter appears in kolibri/locale/*/LC_MESSAGES/kolibri.core.default_frontend-messages.json, whereas the former does not (and that means it’s not translated at runtime).

The fix

We need to revert d4d678d and instead change kolibri-tools i18n-extract-messages to support multiple --namespace/--searchPath pairs.

A short term workaround would be to manually move all the messages from {ek-components,template-ui}-messages.json into kolibri_explore_plugin.app-messages.json. But that would have to be repeated every time the strings are re-extracted or the translation is updated.

[dbnicholson]

I'm tempted to ditch the kolibri-tools translation flow completely and use formatjs directly everywhere like you did in loading-screen. In a sense this would be like how the backend translation process is just django's and not any custom think kolibri is doing. The benefits would be:

• We can control the process without having to wait for infrequent kolibri-tools releases if there are issues.
• We can tailor it to our own processes. I.e., Transifex vs Crowdin.
• We can just use formatjs as it was intended and not try to navigate Kolibri's runtime translation APIs.

The obvious downside is that we'd have to write all of that tooling.

[pwithnall]

I don’t see what the upside is. We’d essentially have to write and maintain a copy of kolibri-tools.

[manuq]

A short term workaround would be to manually move all the messages from {ek-components,template-ui}-messages.json into kolibri_explore_plugin.app-messages.json. But that would have to be repeated every time the strings are re-extracted or the translation is updated.

I did the short workaround in #848 . Here is the merge script if you want to put it into git. But is simple enough:

# Example: ./merge.py kolibri_explore_plugin.app-messages.json ek-components-messages.json template-ui-messages.json kolibri_explore_plugin.app-messages.json
import argparse
import os
import json

def _merge_translations(input_paths, output_path):

    all_json_data = {}
    for json_path in input_paths:
        with open(json_path) as json_file:
            json_data = json.load(json_file)
            all_json_data.update(json_data)

    with open(output_path, "w", encoding="utf8") as json_file:
        json.dump(
            all_json_data, json_file, ensure_ascii=False, sort_keys=True, indent=2
        )


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Merge JSON files."
    )
    parser.add_argument(
        "input_paths", help="Paths to JSON files to merge into output.", type=str, nargs="*"
    )
    parser.add_argument(
        "output_path", help="Path to JSON file to write.", type=str
    )

    args = vars(parser.parse_args())
    _merge_translations(**args)

[pwithnall]

Thanks @manuq!

I’m working on extending kolibri-tools to support multiple namespaces. We can go with the workaround for now. When kolibri-tools is ready and released, we’d revert d4d678d and use kolibri-tools i18n-extract-messages in a single pass.

[pwithnall]

I have a WIP branch here, but need to test it before proposing it upstream: https://github.com/endlessm/kolibri/tree/kolibri-tools-more-namespaces