Use prefixes and suffixes for manifest names to indicate platform.

[Hoikas]

This formalizes a system of prefixing and suffixing manifest names to indicate the platform. Given that the current examples of production manifests are:

External -> external client, windows, 32-bit
macExternal -> external client, macOS, 32-bit Intel

The baseline rule is that no prefix means Windows and no suffix means 32-bit intel. Therefore, all other operating systems and architectures must be suffixed. I included PowerPC suffixes for @dpogue whom I know is tinkering with PowerPC macs. Here is what I came up with:

Suffixes:

• : (no suffix) x86
• AMD64: AMD64/x86_64
• ARM: 32-bit arm (eg NVIDIA Tegra)
• ARM64: 64-bit arm (eg Apple Silicon, Snapdragon X)
• PPC: 32-bit PowerPC (eg iMac G4)
• PPC64: 64-bit PowerPC (eg PowerMac G5)

Prefixes:

• : (no prefix) Windows
• mac: macOS
• lin: Linux (presumably glibc based)

So, some sample manifests:

• External64: x64 Windows external client image manifest
• nixThinInternal: x86 Linux internal client manifest
• macInternalPatcherARM64: Apple Silicon macOS internal release patcher manifest

Currently, we do not build any universal/fat binaries for macOS, therefore nothing has been added for any kind of universal mac binary. I'm somewhat concerned about the UX if/when an architecture gets dropped and a user on that architecture downloads a new fat binary without that architecture. Aside from that, the best way to handle universal binaries, IMO, would be to add a Universal or Fat suffix (remember, no suffix is already established as 32-bit intel) and detect if the current binary is fat. If so, use the fat binary. If not, use the arch-specific one. This would match the Windows behavior where anyone running a 32-bit client on 64-bit Windows will continue to receive 32-bit clients, even though they can run a 64-bit client. However, as stated previously, we currently do not build universal binaries, so this is currently only a tangential point.

This proposal matches what I'm already doing in UruManifest - it assumes that amd64 windows binaries should go into a manifest suffixed with 64. The proposed mac changes to UruManifest will need adjustments because it wrongly reassigns the meaning of macExternal to "64-bit intel". There's also some other oddness in there about a mac bundle manifest that doesn't make any sense. All client files should go into the client manifest.

[dpogue]

Overall this looks good.

My one minor complaint (which isn't new) is that because this is hardcoded at build-time, there's no way to do something like ask plFilePatcher to download the files for another manifest to inspect them

[dpogue]

Strictly speaking, this is probably not going to work for the whole set of Linux/Solaris/BSD(s) that fall under "Unix-like", but I also imagine that beyond Linux nobody is going to be providing pre-built clients, so maybe this isn't something we should care about here.

I would maybe suggest using "lin" as a prefix for Linux builds, but we'd need to pipe that through as a define I guess...

[Hoikas]

I was wondering about that. I know that gaming on FreeBSD is a thing, but 🤷‍♂️

[dpogue]

There's no harm in having these, but again I'm not sure anyone is realistically going to be distributing prebuilt PPC clients (and even I haven't been adventurous enough to poke at PPC64 😛)

I wonder what the best way to handle a universal (i.e., x86_64 and arm64 in the same file) macOS build would be? Separate manifests pointing to the same files might be least hassle and avoid needing to teach the client too much?

[Hoikas]

As I mentioned in the PR, I think the universal manifest should be suffixed "Universal," and if the client detects that it's a fat binary, it can request the fat manifest. A bit of a special case, but so is all of macOS, IMO. Another hot take: just because a manifest is listed here doesn't mean it has to exist. If we only ever distribute a universal client, that's totally fine. It's just not what the build system spits out natively, so we need to handle the actual build output.

[dpogue]

I'd be interested in hearing if there's a compelling reason to not unify them on other OSes...

[Hoikas]

I've been evolving toward pretty strongly preferring that the patcher and client remain separate, especially on Windows. The patcher's job is (IMO) to ensure that the client is in a state where it can launch correctly. Sometimes, that means that the patcher has to install new redistributables or do other rather clever tasks.

Case in point, when OU upgraded the compiler from VS .NET to VS 2010, the patcher did not know how to install redistributables. So, I added this functionality to the patcher, which was recompiled with VS 2003, but the client was compiled with VS 2010. The MOULa server shipped down the VS .NET compiled patcher for quite some time so that players could quietly have the VC++2010 redistributable installed before the VS 2010 client launched. If the client and patcher are unified, these kinds of tricks become much more difficult. Although our patcher now has redist installing built in, I am wary of closing the door for these kinds of tricks.

[Hoikas]

Also, I still have the hope of plClient.exe not spawning any OS gui at all and all logging in related tasks occuring in the StartUp Age. Having a separate patcher makes that delineation of "basically no native GUI stuff" and "oh, look, dialogs" pretty clear.

[dpogue]

Ahh, that does sound like a pretty compelling reason to keep it that way (at least on Windows)

[colincornaby]

FWIW - I'm open to this discussion on macOS as well, but was driven by a few things:

• It's an unusual arrangement on macOS.
• I don't think we have a similar runtime problem - but there is the potential the client patches itself to a client with higher OS requirements, and then can no longer launch itself.

[dpogue]

macOS doesn't really have installable newer runtimes. You get what Apple shipped and you will never get a newer version without upgrading your OS.

[Hoikas]

My one minor complaint (which isn't new) is that because this is hardcoded at build-time, there's no way to do something like ask plFilePatcher to download the files for another manifest to inspect them

Yeah. I had another version of this written that allowed more runtime decision making. For example, allowing 32-bit Windows clients to get the 64-bit manifest if the OS x64. I eventually decided that's probably more over engineered than we need at this time.

[dgelessus]

Standardizing this naming is definitely a good idea, and the naming scheme looks good to me overall. Specific issues I have with the proposed prefixes/suffixes:

• nix is not a good name for Linux (as already mentioned by @dpogue), because compiled Linux binaries are Linux-specific and don't work on other Unix-likes (ignoring things like FreeBSD's Linux emulation).
  • Even linux may not be specific enough, because there are multiple incompatible libc implementations for Linux... glibc is by far the most common, but some distros use musl libc instead, such as Alpine Linux (though I don't think Alpine is commonly used for desktop GUI setups). Of course, we can just decide that "Linux" means "glibc-based Linux" and introduce another "OS" name for musl-based Linux later.
• The macExternal manifest has an established use on Cyan's shard, but doesn't contain a "real" 32-bit macOS client in the way that our Mac patcher code would expect it. Cyan's macExternal manifest was meant for the old Cider wrapper, so it actually contains 32-bit Windows binaries (plus some macOS binaries meant to update the Cider wrapper itself, but AFAIK that never worked, because the necessary symlinks were missing in the wrapper). This probably isn't an issue in practice, because roughly nobody uses the Cider wrapper anymore, but it may be safer to choose a different OS prefix or arch suffix to avoid any possible mixups.
• I don't think macOS needs separate Universal and architecture-specific manifests. To my knowledge, it's never necessary to use a single-arch binary instead of a universal binary containing the same architecture - all macOS versions handle universal binaries properly (and also have ways to force-select a specific architecture from a universal binary, if absolutely necessary).
  • @Hoikas makes a good point about future updates dropping an architecture from a universal binary. IMO that's an argument against a generic Universal manifest, because that doesn't communicate which architectures are in that universal binary. I think a better solution would be to structure the macOS manifests like for the other OSes - one manifest per architecture, and every patcher requests a manifest matching its current architecture - but let the shard operator decide whether each manifest points to a different single-arch client, or multiple manifests point to the same universal client. Then, if an architecture is dropped from a universal client, the manifest for that architecture can be changed to an appropriate alternative client (single-arch replacement or an old version) or removed entirely (which would at least give players a clear error).
• Minor thing: I would prefer an explicit X86_64/AMD64 suffix over simply 64, unless there's an established use for 64.

[Hoikas]

That's fair enough. Having arch specific manifests and leaving it up to the shard operator to populate them with whatever they want means less code we have to write, which is probably what @dpogue wanted me to understand. I think being more explicit about the suffix for amd64 is a good idea - AFAIK no one is actually using 64-bit manifests anywhere. Even though UruManifest makes AMD64 manifests suffixed 64, I don't think any client actually consumes those, so let's change that while we can.

[dgelessus]

I was thinking whether perhaps the manifest prefix/suffix should be determined in the CMake build and passed in as a predefined macro. As CMake provides a variable specifying the target architecture, I hoped it would be simpler to check that than the compiler-specific architecture macros. Unfortunately, the values of that CMake variable aren't standardized at all across OSes and compilers, so it's actually easier to check 2 compiler macros rather than CMake's 5 different possible names for x86 or arm32...

Also, if we ever want to support native universal builds, we can't set a single architecture name at configure time, because every compiler call will build for multiple architectures at once.

[dpogue]

I don't know if a universal app on macOS can even know whether it's universal or not

[dgelessus]

Only by parsing its own executable, I assume. More realistically, you'd inject that information at build time. (It would be okay to have CMake define a macro for "is universal" or "list of all targeted architectures" in a universal build, because the value would be the same for all architectures in the build.)

[dpogue]

(not a serious suggestion, but) you can do universal builds for Windows if you try hard enough: https://github.com/sjmulder/fatpack 😛

[Hoikas]

Also, if we ever want to support native universal builds, we can't set a single architecture name at configure time, because every compiler call will build for multiple architectures at once.

Can CMake even do that? I was under the impression that we would need to collect all of the binaries from CI and lipo them together.

I don't know if a universal app on macOS can even know whether it's universal or not

The first 32-bits of a universal binary should be ca fe ba be. (<3 Wikipedia)

[dpogue]

Can CMake even do that? I was under the impression that we would need to collect all of the binaries from CI and lipo them together.

CMake can absolutely do this! The problem is that other build systems (like autotools) cannot, and so vcpkg doesn't have a way to support universal builds of all the dependencies.

[Hoikas]

Ah, right, it's one of those multi-config generator problems. Wonderful.

[dgelessus]

The problem is that other build systems (like autotools) cannot

Even autoconf has some support for macOS universal builds - see "Compiling for Multiple Architectures" and AC_C_BIGENDIAN. The problem is that many builds in practice make some assumptions that conflict with how macOS compilers do universal builds, and it's often more work to refactor a complex build to remove those assumptions than to just run two single-arch builds and add a lipo step at the end.

Ah, right, it's one of those multi-config generator problems. Wonderful.

AFAIK, universal builds aren't a multi-config thing in CMake. What changes is that the target architecture variables become lists of architectures, and certain checks like "what's the target endianness" are impossible. The usual pattern of making everything a generator expression doesn't help with that either - you have to move all architecture-specific checks into preprocessor conditionals.

macOS universal builds work by passing multiple architectures to every compiler call - e. g. clang -arch arm64 -arch x86_64 -c -o hello.o hello.c will compile hello.c for arm64 and x86_64 into hello.o, and that object file will be a universal binary. Generally, this is pretty convenient to integrate into an existing build, because it's "just" adding some compiler flags. You only get problems when your build scripts (or the build system itself) make too many assumptions based on the target architecture, or perform unusual operations on the compiled files, or you have parts written entirely in assembly.

[Hoikas]

That's pretty cool - it's too bad that kind of build is pretty much incompatible with both us and vcpkg.

[colincornaby]

I'm going to follow up on the universal Mac app discussion (let me know I miss anything):

• I think we should ship a universal binary as a goal. If someone upgrades from an Intel Mac to a ARM Mac - we don't want them to keep using an Intel client. (I'm not actually sure if there is even a way for the server to kick the user over to an ARM client.)
• I don't have a good answer for what happens when an architecture is dropped. Obviously this will probably happen with Intel Macs someday. But I'm not sure how not doing universal builds would solve that.
• I think the idea of having multiple manifests pointing to the same binary isn't a bad one. But I'm still wondering about the client-changed-host-architectures scenario.
• We should also think about future Mac architectures and how we'd upgrade someone to a future architecture. Universal binary seems reasonable to me. In the past Apple has had to make breaking changes between different ARM architectures (ARMv7 vs ARMv8 vs ARMv9.) So we should account for that in the future we may need to deliver a new architecture. And again, rolling that up in a universal binary so it's transparent seems like the best path.
• I think to create a Mac universal binary we just do two separate compilations and then lipo them together. https://ss64.com/mac/lipo.html This was made easier by statically linking everything so there is only one binary to lipo. (Edit: I see this was discussed already.)

[dgelessus]

That's pretty cool - it's too bad that kind of build is pretty much incompatible with both us and vcpkg.

At least on our side, it would be very doable I think - only the endianness check should cause issues, and that can be substituted by a compile-time architecture check in universal builds. (The new CMAKE_<LANG>_BYTE_ORDER variables in CMake 3.20 allow handling this robustly and without complicating the common case where the byte order is static.)

We can't do anything about vcpkg of course, but most dependencies can be installed via MacPorts instead, which does universal binaries very well. I assume PhysX would be the biggest problem, with its custom scripts on top of CMake, but at worst we'd have to use lipo on the PhysX libraries.

[dpogue]

• I think we should ship a universal binary as a goal. If someone upgrades from an Intel Mac to a ARM Mac - we don't want them to keep using an Intel client. (I'm not actually sure if there is even a way for the server to kick the user over to an ARM client.)

There is no way for the server to tell someone to use a different manifest. This is the same issue we have with upgrading Win32 users to Win64 clients that Hoikas has mentioned.

• I don't have a good answer for what happens when an architecture is dropped. Obviously this will probably happen with Intel Macs someday. But I'm not sure how not doing universal builds would solve that.

In theory you'd be able to stop bundling older architectures into the universal build (to cut down the size) while still making them available as standalone builds.

• I think the idea of having multiple manifests pointing to the same binary isn't a bad one. But I'm still wondering about the client-changed-host-architectures scenario.

My thinking is that if that same binary is a universal build, that would effectively give us seamless upgrades with new architectures. The individual slices in that build would ask for their own arch-specific manifests.

• We should also think about future Mac architectures and how we'd upgrade someone to a future architecture. Universal binary seems reasonable to me. In the past Apple has had to make breaking changes between different ARM architectures (ARMv7 vs ARMv8 vs ARMv9.) So we should account for that in the future we may need to deliver a new architecture. And again, rolling that up in a universal binary so it's transparent seems like the best path.

Per my answer to the previous point, if someone on Apple Silicon is running an x86_64 client and it asks for the x64 manifest and gets delivered a universal build, the next launch should run the arm64 slice which would ask for the arm64 manifest going forward.

[colincornaby]

In theory you'd be able to stop bundling older architectures into the universal build (to cut down the size) while still making them available as standalone builds.

I think what I'm concerned about is that we'd no longer ship an Intel build because we'd lose our ability to build for Intel. We'll head that direction eventually as Xcode drops Intel support and our Github build agents move on. The executable size isn't that large. I don't think we'd do something like drop Intel out of the universal binary for size concerns.

[dpogue]

While it's possible technically to do a universal macOS build supporting ppc/ppc64/i386/x86_64/arm64 (and up to 19 other combinations), I wasn't seriously thinking anyone would include PowerPC or 32-bit Intel builds in the universal binary, so I'm less worried about needing to deal with endian-checking issues for a univerally-universal build. You have to use different compilers for PowerPC anyways, so that's always going to be a bit of a special case.

You can (for what it's worth) make macOS builds using clang on Linux and link with lld if you provide the SDK, and clang might keep Intel support around for a while. What's lacking there is the other Xcode tools like ibtool, plist-util, actool, codesign, and stuff like that.

[dpogue]

In any case, I think the goal of this PR is improvements on the current situation, not a perfect system that can handle every weird edge case we dream up

[colincornaby]

Yeah, I think what I'm replying to in the PR description is this specifically:

I'm somewhat concerned about the UX if/when an architecture gets dropped and a user on that architecture downloads a new fat binary without that architecture. Aside from that, the best way to handle universal binaries, IMO, would be to add a Universal or Fat suffix (remember, no suffix is already established as 32-bit intel) and detect if the current binary is fat.

On architecture dropping I'm concerned as well but unsure what to do about it as architecture dropping will likely happen when the tools drop support for an architecture. Which means universal or not we still have the same problem. If we drop Intel support an Intel manifest goes away and the user will be unable to patch. So while we don't have a migration path now - I'm concerned that we may need to consider one.

Edit: Or obviously if we have a universal binary this problem solves itself. (Except for the users being orphaned without any UX problem.)

[Hoikas]

If we drop Intel support an Intel manifest goes away and the user will be unable to patch.

That's true - but I think we can pretty gracefully handle the manifest being gone by presenting an error that says something like, "Sorry, but your Mac/PC is no longer supported." Whereas if the architecture is dropped from the universal binary, it's a bit of a quandary.

Anyway, as @dpogue says, my goal here is to improve upon the situation we find ourselves in. We have lots of Windows and macOS binaries for different configurations and architectures, but everything is all fuzzy and conflicted. I feel like what I have here is the most future proof and least restraining path forward - nothing here should actively prevent us from doing a universal binary in the future. But this will help us distribute working clients in a more predictable way now.

I would like to get a 👍 or 👎 so I can know if this is the direction I need to take UruManifest in.

[colincornaby]

I feel like what I have here is the most future proof and least restraining path forward - nothing here should actively prevent us from doing a universal binary in the future. But this will help us distribute working clients in a more predictable way now.

I would like to get a 👍 or 👎 so I can know if this is the direction I need to take UruManifest in.

Understood. Give me a few hours to consider - I'm not quite as bullish that there won't be issues in future. I'll get back to you with a thumbs up or down.

[colincornaby]

I'll go ahead and give this a thumbs up. But - IMO - the Mac client should never be shipped on a platform by platform basis. It should always be universal. (Within reason of course, we could lipo something like a PowerPC client but should not need to.)

Work was already done to prep the Mac client for a universal build via lipo, we just need to orchestrate that on the build system. So I'd be concerned if we were thinking about shipping per architecture.

[dgelessus]

One thing before I give my full final thumbs up: How do we proceed with the conflicting existing use of macExternal, etc. between the old Cider wrapper and our new native macOS client? I'm not that opposed to saying "we don't care about conflicts with the old Cider wrapper" - I just want to make sure that it's a conscious decision.

A possible workaround would be to use an explicit X86 suffix for non-Windows x86 manifests. Then the new macOS client would use macExternalX86 and won't conflict with the Cider wrapper's macExternal.

I agree that we should prefer universal builds for macOS if at all possible, because from the user's perspective, they're superior in all aspects except disk space (which shouldn't be a practical concern for an already multi-gigabyte game). But as explained, I think the macOS manifests should still be per-architecture, because that will "just do the right thing" for universal clients, especially when a new architecture is added or an old one is removed, while giving shard operators maximum flexibility with how they want to build their macOS clients.

I wouldn't worry that building for x86_64 macOS will become impossible any time soon. After all, @dpogue is still happily building for PowerPC, 20 years after Apple began dropping it 🙂 Building on GitHub Actions is a different question, once Apple inevitably drops support for targeting x86_64 from Xcode. But even that should still be possible, because besides the default Apple Clang from Xcode, the macOS runners apparently also have "regular" Clang installed from Homebrew, and I'd assume that upstream Clang will keep x86_64 macOS as a target for a long time.

[Hoikas]

One thing before I give my full final thumbs up: How do we proceed with the conflicting existing use of macExternal, etc. between the old Cider wrapper and our new native macOS client? I'm not that opposed to saying "we don't care about conflicts with the old Cider wrapper" - I just want to make sure that it's a conscious decision.

My thought process here was that while macExternal was a cider wrapper for the windows executables, the intention was for that client to run on a 32-bit intel mac. So, that's what that manifest indicates. AFAIK, the Cider wrapper hasn't worked since GameTap, so we shouldn't break anyone by shipping down something where the intent is the same but the implementation is quite different. If I've missed something, let me know!

[dgelessus]

The Cider wrapper still worked for quite a while with MOULa. IIRC, it finally broke with the update to PhysX 2.6.4, because the new PhysX installer didn't run properly under the wrapper. And ignoring that, no recent macOS supports running 32-bit binaries anymore. So it's definitely no longer in use now, but it was used for much longer than just the GameTap time.

More relevant is that (as far as I know) Cyan still hosts the macExternal, etc. manifests expected by the Cider wrapper, so if our patcher code tries to use those, it will mis-patch itself weirdly. Of course, that will only affect someone building a 32-bit macOS client themselves and then carelessly pointing that at Cyan's servers. So it's not really a practical issue, but juuuust possible enough that I worry that someone might do it 😄

[Hoikas]

Interesting. Do you remember when the PhysX 2.6.4 update happened on MOULa? I know when we did the VS 2010 update, we left a VS 2003 patcher up for an amount of time to ease the transition, and I'm wondering if the time elapsed is comparable. IIRC they left the VS 2003 patcher up for like a year.

[dgelessus]

Do you remember when the PhysX 2.6.4 update happened on MOULa?

Perhaps I'm misremembering - it might not have been the switch to PhysX 2.6.4, but rather when the PhysX_Setup.exe was changed from the old Ageia installer to the newer Nvidia "legacy" installer. But it definitely had something to do with the PhysX redistributable.

I have some old files from late May 2020 where I tried to retrofit the new PhysX files into the Cider wrapper (it actually worked IIRC, but I never uploaded the result anywhere). I wrote "OpenUru 159" in the file names, and Chogon's release notes for that build (from 2020-05-27) list "Physics update" as one of the changes, so that checks out.

In short: the old Cider wrapper was usable until roughly 5 years ago.