Fix path encoding issues (#1153) by jkriegshauser · Pull Request #2654 · premake/premake-core

jkriegshauser · 2026-04-16T01:43:53Z

What does this PR do?

Solves #1153 by adding complete UTF-8 path support to both the embedded Lua 5.3 and premake5 functions

closes #1153

How does this PR change Premake's behavior?

Existing behavior should remain unchanged. There are a few other minor fixes that I will enumerate later

Anything else we should know?

Don't think so, but feel free to ask away.

Can squash if necessary before merging. Also probably need to take a look at the documentation.

Did you check all the boxes?

Focus on a single fix or feature; remove any unrelated formatting or code changes
Add unit tests showing fix or feature works; all tests pass
Mention any related issues (put closes #XXXX in comment to auto-close issue when PR is merged)
Follow our coding conventions
Minimize the number of commits
Align documentation to your changes

You can now support Premake on our OpenCollective. Your contributions help us spend more time responding to requests like these!

nickclark2016

I see a lot of what feels like duplicate code for handling MBCS conversions. Can we make a utility function somewhere that handles all of this in one spot? Definitely in favor of this change. It'll be nice having better UTF support within Premake.

jkriegshauser · 2026-04-19T16:24:10Z

I see a lot of what feels like duplicate code for handling MBCS conversions. Can we make a utility function somewhere that handles all of this in one spot? Definitely in favor of this change. It'll be nice having better UTF support within Premake.

I was doing a self-review and realized that as well. Going to make it an actual Lua auxiliary extension that hopefully I can work back into the Lua upstream. Working on that now.

nickclark2016 · 2026-04-19T16:41:26Z

I see a lot of what feels like duplicate code for handling MBCS conversions. Can we make a utility function somewhere that handles all of this in one spot? Definitely in favor of this change. It'll be nice having better UTF support within Premake.

I was doing a self-review and realized that as well. Going to make it an actual Lua auxiliary extension that hopefully I can work back into the Lua upstream. Working on that now.

Shall we defer this PR until it's in the Lua upstream, figure out what work needs to be done to bring in the latest Lua version, then readdress within Premake?

I'd like to get this change landed before 5.0 stabilizes, so we don't break any ABIs that may be relied upon by native Premake addons post 5.0.

jkriegshauser · 2026-04-19T18:11:02Z

Shall we defer this PR until it's in the Lua upstream, figure out what work needs to be done to bring in the latest Lua version, then readdress within Premake?

Personally I'd like to land it here first. Premake already is on an older version of Lua and has Lua modifications, plus I think it's a longer task to get any changes in Lua upstream.

I'd like to get this change landed before 5.0 stabilizes, so we don't break any ABIs that may be relied upon by native Premake addons post 5.0.

100% agree, there shouldn't be any API/ABI affect with this other than making UTF-8 work for filenames

jkriegshauser · 2026-04-20T06:08:25Z

@nickclark2016 alright, take a look now. All tests pass for me on linux and windows

nickclark2016 · 2026-04-20T15:45:04Z

Looks like there are still a fair amount of compilation failures. It's also worth noting that we attempt to support system installs of our vendored libraries, so we can't really change the interface without risking breaking that.

jkriegshauser · 2026-04-20T16:48:40Z

Looks like there are still a fair amount of compilation failures. It's also worth noting that we attempt to support system installs of our vendored libraries, so we can't really change the interface without risking breaking that.

Got it; I didn't previously realize that. I guess this support would be conditional based on whether one builds with the contrib. I'll work on getting that fixed up

jkriegshauser · 2026-04-20T22:20:52Z

@nickclark2016 can you kick off another CI run? I've rewritten history to only 3 commits, but also fixed several issues with mingw/system libs/clang on windows/etc.

jkriegshauser · 2026-04-21T04:21:10Z

Thanks, much closer. I'll look into the solaris compile issue and failing netbsd test tomorrow. Is there any way for me to be able to start CI jobs without adding a comment to request it manually?

nickclark2016 · 2026-04-21T04:24:35Z

I don't have permissions to allow that, but as soon as you land this, I believe any future PRs would not have this problem.

Also, those VM based actions are flakey. Looks like the issue was the action itself, not the build. Only thing left appears to be issues in the web actions.

Jarod42 · 2026-04-21T07:29:00Z

Is there any way for me to be able to start CI jobs without adding a comment to request it manually?

You might enable actions in your Clone repo.
Check your repo settings.

jkriegshauser · 2026-04-22T20:05:09Z

Alright, I think all the fixes are in now. Please kick off another build.

I believe that the website build issue is unrelated. Looks like the root cause is:

ValidationError: Invalid options object. Progress Plugin has been initialized
using an options object that does not match the API schema.

which doesn't seem related to my changes. Might need a fix like pinning the node version or a docusaurus update or a package-lock.json or something to lock the versions 🤷

nickclark2016 · 2026-04-22T20:07:21Z

Ah yeah, looks like we need to actually use a package-lock.json (pretty sure none of the core contributors are actually web devs)

jkriegshauser · 2026-04-22T20:30:05Z

Ah yeah, looks like we need to actually use a package-lock.json (pretty sure none of the core contributors are actually web devs)

Added another PR to fix the website here. I enabled actions on my fork and it passes there now (at least the check part; i obviously cannot publish)

nickclark2016 · 2026-04-22T20:57:21Z

Oh good. The auto jobs are working now.

jkriegshauser · 2026-04-22T20:58:28Z

Oh good. The auto jobs are working now.

yeah i just needed to land a change 😅

jkriegshauser · 2026-04-22T21:16:29Z

The dragonflybsd check that failed timed out while booting QEMU. Can you manually restart just that one job?

nickclark2016 · 2026-04-22T21:24:54Z

Giving this a deep dive now that everything is built and the tests are happy. Are the tests in a state where they should catch regressions? I'm looking at going in to update the dependencies in the very near future, so I don't want to break the improvements you've made here.

jkriegshauser · 2026-04-22T21:28:23Z

Are the tests in a state where they should catch regressions?

Yes, the tests should validate all of the changed functionality so regressions should be caught. Plus we define [_]UNICODE now so wmain() is used and win32 functions default to the W extension if not specified. It doesn't really affect the CRT functions though (remove(), stat(), etc.). It might be good to mention in the documentation somewhere that full UTF-8 is contingent on building with contrib Lua; otherwise we add the functions but we can't fix the actual internal Lua calls to ANSI functions. If you can point me at the desired place in the docs i can add a blurb.

If you have any issues updating dependencies just lmk. Next step is for me to try to get the Lua changes upstreamed there too.

nickclark2016 · 2026-04-22T21:33:05Z

https://premake.github.io/docs/Building-Premake

If we had better documentation on how to build with the contrib folder vs system, it would live here. To not dump that additional work on you after this massive lift, would you mind just opening an issue, dumping a blurb there about UTF8 support, and setting me as the assignee (if you have permission to do so)? Then I can get it massaged into an fuller section on the various build options.

jkriegshauser · 2026-04-22T21:37:33Z

@nickclark2016 can't assign, but mentioned you: #2658

If I get some spare time I can try to help out too.

nickclark2016 · 2026-04-22T21:38:42Z

@nickclark2016 can't assign, but mentioned you: #2658

On my list to address before 5.0 stable (hopefully soon), thanks!

nickclark2016

Overall, very happy with the state of this. General question to chew on:

Should we have the UTF8 changes better macro-ed off? This would let us turn on and off UTF8 support, which would be a nice to have for building using system dependencies instead of the contrib modules.

nickclark2016 · 2026-04-22T21:48:39Z

 		filter { "system:windows", "configurations:Release" }
 			incrementallink "Off"

+		filter { "system:windows" }


Does characterset "Unicode" not work?

Overall, very happy with the state of this.

☺️

Should we have the UTF8 changes better macro-ed off? This would let us turn on and off UTF8 support, which would be a nice to have for building using system dependencies instead of the contrib modules.

I don't think it's really necessary. If you build with system Lua on Windows, the _UTF8_ENABLED doesn't get set (will evaluate as nil) which is how you know if you have UTF-8 support. You still get some UTF-8 support but it cannot be relied upon (which is kind of how it was before this change). But ideally I manage to get the Lua changes upstream and we can update to that, so it might be a moot point.

Does characterset "Unicode" not work?

It does for vs (i tried with vs2022), but not for mingw. I'll push an update and I can make an issue if you want.

Let's go ahead an open an issue and have it reference this, I don't want to block this.

Issue created: #2659

nickclark2016 · 2026-04-23T02:20:19Z

I'm going to leave this open for a few days for others to review, but LGTM at this point. Thanks!!

dfagnou · 2026-04-23T20:03:24Z

approved

- Fix the regression introduced in premake#2654 - Update the document to match the long-standing existing behavior

nickclark2016 reviewed Apr 18, 2026

View reviewed changes

jkriegshauser changed the title ~~Fix path encoding issues (#1153)~~ Draft: Fix path encoding issues (#1153) Apr 19, 2026

jkriegshauser force-pushed the utf8-support-latest branch from 29547fe to 5ec46ca Compare April 20, 2026 06:07

jkriegshauser changed the title ~~Draft: Fix path encoding issues (#1153)~~ Fix path encoding issues (#1153) Apr 20, 2026

jkriegshauser force-pushed the utf8-support-latest branch from 5ec46ca to aba02b0 Compare April 20, 2026 22:17

jkriegshauser force-pushed the utf8-support-latest branch from aba02b0 to b87c647 Compare April 22, 2026 20:03

nickclark2016 mentioned this pull request Apr 22, 2026

Fix website build #2657

Merged

4 tasks

jkriegshauser added 2 commits April 22, 2026 13:55

Test Unicode functionality

6b1be35

Lua support for UTF-8 on Windows

b01da7d

jkriegshauser force-pushed the utf8-support-latest branch from b87c647 to 0ed0638 Compare April 22, 2026 20:55

jkriegshauser mentioned this pull request Apr 22, 2026

Documentation improvements around UTF-8 support #2658

Open

2 tasks

nickclark2016 reviewed Apr 22, 2026

View reviewed changes

Premake support for UTF-8

02866be

jkriegshauser force-pushed the utf8-support-latest branch from 0ed0638 to 02866be Compare April 23, 2026 02:08

nickclark2016 approved these changes Apr 23, 2026

View reviewed changes

jkriegshauser mentioned this pull request Apr 23, 2026

characterset "Unicode" doesn't work properly on MINGW #2659

Open

15 tasks

nickclark2016 mentioned this pull request Apr 23, 2026

5.0 Stable Milestone #2661

Open

7 tasks

nickclark2016 merged commit c8c60f8 into premake:master Apr 26, 2026
55 checks passed

mercury233 mentioned this pull request Apr 27, 2026

fix return value of os.writefile_ifnotequal() #2667

Open

6 tasks

mercury233 added a commit to mercury233/premake-core that referenced this pull request May 10, 2026

fix return value of os.writefile_ifnotequal()

489a72f

- Fix the regression introduced in premake#2654 - Update the document to match the long-standing existing behavior

Uh oh!

Conversation

jkriegshauser commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickclark2016 left a comment

Choose a reason for hiding this comment

Uh oh!

jkriegshauser commented Apr 19, 2026

Uh oh!

nickclark2016 commented Apr 19, 2026

Uh oh!

jkriegshauser commented Apr 19, 2026

Uh oh!

jkriegshauser commented Apr 20, 2026

Uh oh!

nickclark2016 commented Apr 20, 2026

Uh oh!

jkriegshauser commented Apr 20, 2026

Uh oh!

jkriegshauser commented Apr 20, 2026

Uh oh!

jkriegshauser commented Apr 21, 2026

Uh oh!

nickclark2016 commented Apr 21, 2026

Uh oh!

Jarod42 commented Apr 21, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026

Uh oh!

nickclark2016 commented Apr 22, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026

Uh oh!

nickclark2016 commented Apr 22, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026

Uh oh!

nickclark2016 commented Apr 22, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickclark2016 commented Apr 22, 2026

Uh oh!

jkriegshauser commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickclark2016 commented Apr 22, 2026

Uh oh!

nickclark2016 left a comment

Choose a reason for hiding this comment

Uh oh!

nickclark2016 Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

jkriegshauser Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

nickclark2016 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

jkriegshauser Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

nickclark2016 commented Apr 23, 2026

Uh oh!

dfagnou commented Apr 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jkriegshauser commented Apr 16, 2026 •

edited

Loading

jkriegshauser commented Apr 22, 2026 •

edited

Loading

jkriegshauser commented Apr 22, 2026 •

edited

Loading