It's decision time. Please read.

[mszoek]

We're at a crossroads in this project and I need to decide the future direction. Gonna ramble a bit but it's important so please try to stick with me.

I started this project mostly as a joke and a challenge... "could I really do that?"... and I feel that I've shown how possible it is. I satisfied myself that my claws are sharp. While I was doing that, the project grew a life of its own and I don't want to let you all down.. Also, I'm still having fun with this and want to make it a reality! So here's the thing...

We have to get to Mach-O now. Let's not go into why or timing or whatever - I know we need to and why but I don't want to explain it all. We have to. Now, how?

If I stick with FreeBSD, the road ahead is mainly gonna be porting Apple tools like ld64, the cctools stuff, dyld, and so on, then getting alllll of FreeBSD to build as Mach-O instead of ELF, then fixing all the breakage. Vastly oversimplified but there... so that's a FUCKING SLOG and quite honestly I Don't Wanna. Plus, even when we get all that done, we still have to figure out how much of Mach and what other system calls I missed that real Mac binaries actually need. I know there's stuff missing. And stuff returns the wrong values. So this road leads to implementing lots of missing bits, and slowly getting towards compatibility. All of that takes time away from me working on the GUI and the interesting bits.

The other path is Darwin. I would essentially start with Apple's opensource 'distribution-macOS' for 10.15, set up a build system using BSD's make system, then add in the missing tools etc from FreeBSD, and layer our frameworks and GUI and such on top of the real Darwin/xnu. Now this... sounds fun. But also, it's my best shot to actually make Mac software run on it. The kernel, system libs, binary format, runtime linker, launchd, everything the same as Catalina. It's a big lift, but it frees me up afterwards to spend less time dicking around and more time making a slick desktop and getting apps to work.

So guess what? We're moving to Darwin!

And yeah, we're going to lose some things in that move. Hardware support - much better than xnu will give us. ZFS. Linux binary compatibility. Fantastic BSD performance, threading, stability... However. We also gain some things. A much better foundation for the goal. A platform that has a big Hackintosh developer community, as well as the PureDarwin folks and others. The toolchain will be identical. All the headers and versions are the right ones. Full Mach and all the syscalls. A good HFS implementation and likely APFS. There are a lot of good reasons.

I've been playing with all this (both paths) for a few days now. Oh... and I promise to not force-push a new main on you :)

[PlatinumLucario]

That's awesome! At least with Darwin, we'll be able to get the Mach-O stuff running. But despite the stuff that is lost in the move, they can be regained later with porting them. :)

[reactive-firewall]

And yeah, we're going to lose some things in that move. Hardware support - much better than xnu will give us. ZFS. Linux binary compatibility.

But despite the stuff that is lost in the move, they can be regained later with porting them. :)

Counter phrasing: "We lose more by not moving"; for linux there is darwin docker (via the containers framework we can run much of linux tools in a "typical" way) (also homebrew can help with a lot of porting); for ELF, much of freebsd 'ports' is ported by 'macports'.

and regarding hardware we might be able to leverage the bootloaders from Clover (by the hackntosh folks)

[RandomDSdevel]

     I like the thought of moving ahead with Darwin/XNU, but have you considered leaving anything in the code open to also coming back and making userland multi-platform later? Someobody else could take up FreeBSD backend support and making the shims and compatibility components for that at some point, if they wanted to. Some modularity might help. Maybe that's too difficult, though. Additionally, maybe the existing coupling assumptions needed in reproducing macOS behavior are still too tight, at least for now.

[RandomDSdevel]

     Regarding ZFS, at least this implementation exists for macOS. Darwin/XNU will still expect and require APFS for the filesystem of boot drives.

• (Distant stretch goal:

  Revive the glory fays of the earlier Mac OS X v10.5 developer betas and restore system- and kernel-level ZFS filesystem support.

• Even more distant further stretch goal:

  Add boot support for other filesystems.

  Not sure which, if any, could/would be suitable, however.)

(…Only if that could be done in some way that doesn't break macOS support for users running non-APFS boot filesystems, of course.)

[mszoek]

I like the thought of moving ahead with Darwin/XNU, but have you considered leaving anything in the code open to also coming back and making userland multi-platform later? Someobody else could take up FreeBSD backend support and making the shims and compatibility components for that at some point, if they wanted to. Some modularity might help. Maybe that's too difficult, though. Additionally, maybe the existing coupling assumptions needed in reproducing macOS behavior are still too tight, at least for now.

Everything will still be there in the branch if we do ever want to come back. I expect this will evolve as I work through getting everything to build and boot on xnu.

[mszoek]

Regarding ZFS, at least this implementation exists for macOS. Darwin/XNU will still expect and require APFS for the filesystem of boot drives.

* (Distant stretch goal:
  Revive the glory fays of the earlier Mac OS X v10.5 developer betas and restore system- and kernel-level ZFS filesystem support.

* Even more distant further stretch goal:
  Add boot support for other filesystems.
  Not sure which, if any, could/would be suitable, however.)

(…Only if that could be done in some way that doesn't break macOS support for users running non-APFS boot filesystems, of course.)

I really only used ZFS because it's good, FreeBSD supports it, and it is similar to APFS. Being able to keep it would be great but not critical. And of course, filesystem support can be added to a kernel fairly easily. There are projects that support ext2/3/4fs and NTFS on macOS, for example.

[reactive-firewall]

Also ZFS requires a patent-grant to implement and distribute, even for the open-source implementations (which is why one of those open-source grants' implementation is typically used).

APFS is a fair compromise as it provides most of the same capabilities (with minimal loss in performance on SSD (and backups on older drives can still have other more appropriate formats with Darwin)) But the transparent clustering of partitions in a container group is more user-friendly (albeit more tightly coupled) and provides key aspects of MacOS behaviors including hot-patch upgrades via the update volume and sandboxing virtual memory on a separate partition, and APFS like ZFS can handle multi-teribyte drives that are common today. Regarding extfs4 (2 and 3 are ill-suited) the use of it would have issues specifically with how MacOS likes to enforce encryption without bringing in a slew of Linux/GNU stuff that under-performs compared with APFS, so APFS (while not the ZFS) is best aligned with the other previously mentioned goals of MacOS compatibility/similarity. But I digress.

---

Add boot support for other filesystems.

Not sure which, if any, could/would be suitable, however.)

Interestingly the public headers from MacOS LibSystem (distantly based on FreeBSD btw) where the /usr/sys/*.h stuff on MacOS lives, contain a bit of insight on which could be suitable: the issue is XNU/Darwin are stripped of the implementation for non-Apple intended hardware. So some of the FreeBSD stuff could actually be brought back in, and that's why apple does just that from time-to-time. The difference is Apple is a for-profit company, and thus always driven in part by profit, where this project is not part of a for-profit company, and thus could explore less profitable, but more interesting experiments with hardware. Apple could support more hardware, but would need to take on the additional support overhead, and possibly go against their company mantra "Think Different℠" and think like open-source communities (e.g. Canonical comes to mind), but they don't see the profit in it (at this time? 🤷).

All that to say: So as an idea, we could fill the gaps with expanded HW support. I'm personally very interested in seeing if we could support some of the older Apple hardware in a sort-of 'lite' fashion, given MacOS often pushes the deprication of support for principles like security (intel CPUs and inherent speculative processing vulnerabilities, or bootloader that requires a hardware secure enclave), and performance (required multi-processors, and more memory for graphics, or actual audio DSP for sound, or SSD for APFS) but the older headers could be shimmed instead of depricated at the cost of performance and security (let's face it, the project isn't to compete with apple's MacOS, which as a for-profit OS will always strive to justify itself against open-source (especially free) alternatives)

Longer-term: Linux thrives in buisness because the industry figured out they can make a buisness of just supporting the free OS, (e.g. conviencnce-as-a-service/Support-aaS ) why not plan to have the same; the further you deviate with hardware the harder your setup might be. 🤷

@mszoek I leave these ideas to you.

Regarding drivers: apple, just lets venders define their drivers for stuff by giving them an API (very blackbox in many-ways), and ABI compatibility should give the project many of those. However, mac-ports is probably the best option besides the fact that we'll need to write/patch/shim many drivers if we want to expand beyond supporting Metal and USB-C/thunderbolt, Broadcom, etc. (e.g. CloverOS area of expertise)

[SergeySlice]

Well, and where we will get device drivers? FreeBSD has many drivers which absent in Darwin.

[PlatinumLucario]

How about we can do all three of them, depending on the situation with each driver?

Using the existing kexts would be top priority, then new native drivers for Darwin get made based on the Linux/FreeBSD driver source code and be maintained, then in the long term a compatibility shim can be made for various stuff from other OS' drivers that don't have any code for it available for rewriting.

So from top to lowest priority, from easiest to hardest: First option, second option, third option

[mszoek]

We're on the same page :)

[SergeySlice]

For example real MacOS has no graphic drivers for Tiger Lake while ravynOS does.
It is not a simple work to make a graphics driver. No successes in Hackintosh.
I think the better way is to port all FreeBSD sources to Mach-O.

[mszoek]

ravynOS is just using the EFI framebuffer to draw unaccelerated. We had accelerated drivers for AMD and Intel chip sets. Nvidia was a work in progress. If I can get similar framebuffer support out of xnu, we should be ok to start with.

Trust me, getting FreeBSD to build as Mach-O is not easy. The libc in particular has a LOT of assumptions about the binary format and capabilities of ELF objects that are no longer true with Mach-O. And even if we did, there's a lot left to do. I'd rather take a shot at porting some framebuffer code or video drivers that we have source for in other OSes. But we'll see.... this might be a dead end.

[reactive-firewall]

• Create a compatibility shim like FreeBSD did for Linux wifi drivers. This means implementing enough of the Linux or BSD kernel APIs that drivers use on top of xnu, then doing minor porting effort to bring in verbatim drivers from those other projects. This is the hardest approach but could unlock a wide range of drivers with mostly one-time effort.

IMHO

@mszoek take a look at my Shim for sys/cdefs.h for XNU/Darwin/FreeBSD/Musl libc at reactive-firewall/liberals, which I use in the same way for porting FreeBSD to musl libc (which is FAR more diverged from the BSDs than XNU is)

I think if the goal is maintainability with near-one-time efforts, the compatibility layer makes the most sense, and aligns with the goal of MacOS API compatibility best.

Hope this helps 🙇

[SergeySlice]

May be we will stay with FreeBSD as is but make some layer to launch flatpack as Mac application with all frameworks inside? Nothing outside. And translate system calls to a set of FreeBSD system calls (like wine).
I don't know how complex is it.

[reactive-firewall]

Why we can't use different methods for launch simple FreeBSD application or flat-pack?

In theory (because AFAIK the tools have not been publicly implemented ... yet? 🤷 ) there is no limit preventing us from using different methods, if we can figure out how to implement it (possibly from scratch) however, without a way to share common libraries like the libc or libc++ or libobjc via a common dynamic load object (e.g. ELF libc.so and Mach-O libSystem.dylib provide the same ABI logically in different compiled formats) I would assume we are talking static libs for the Mach-O (which absolutly breaks most of macOS ABI compatibility, as Mach-O defaults to assuming the existence of a global (e.g., MH_DYLINKER) dynamic lib loader) which will grow very, very, large. But I digress into my own speculation.

In the second case we will assume it is MachO application calling MachO libraries.

This is where I get lost; how do we even get to the second case? E.g., in C-lang how do we get the compiler to go from the C Language standard entry-point to the theoretical code that will identify either case? The format (e.g., ELF or Mach-O) of the binary needs to be identified before the code to load and run can even be parsed, all before it can be even checked for things like the MACH-O load header or ELF start code (e.g., all the stuff before the developer's main() function is called) which WILL be needed to launch either case. This is why I assumed an emulator (in traditional ELF format, and work something like wine but for Mach-O) or full fledged Virtual Machine would handle loading the flatpack, where we could then assume the second case as you mentioned, and do stuff like intercept calls to shared code with a custom MH_DYLINKER.

I've gone quite a-bit into explaining some of the why, not to say we can't do things, but rather to keep the focus on @mszoek's initial topic question:

We have to get to Mach-O now. Let's not go into why or timing or whatever - I know we need to and why but I don't want to explain it all. We have to. Now, how?

Important

Our question from the beginning remains: "Now, How?"

To that end I'm going to bow out of the conversation to leave space for others to contribute here

[mszoek]

Why we can't use different methods for launch simple FreeBSD application or flat-pack? In the second case we will assume it is MachO application calling MachO libraries.

Let's follow this line of thought. How does your app inside the flatpak communicate with things like Finder, launchd, and system frameworks? Where do those live? Your flatpak support code or host OS would have to implement all of those APIs and services. If you're going to do that, why not just make apps a native format?

[SergeySlice]

Because I want to launch closed-source applications. Else why all this?
May be you are right and we can make Darwin which is able to launch Mach-O application if we have all needed frameworks. Graphics drivers will be only flat framebuffer or may be even software OpenGL from private Apple's library... may be.

[CuriousTommy]

Because I want to launch closed-source applications

Flatpak is necessary for Linux distros because Linux distros don't care enough about offering a stable ABI platform for proprietary apps.

As long as RavynOS takes the proper responsibility to ensure backwards compatibility with proprietary apps, tools like Flatpaks would be unnecessary.

[kevincrans]

I wrote and wiped a big story, to conclude Linux and BSD bringing a macUlator:Darling? is a better idea than Linux/BSD apps on RavynOS/each-other. However I would still love to see the linuxulator ported.
And for true cross-platform, LLVM IR/.Net is the best.
But I would expect Flatpak macOS to just install mac ports and dependencies from mac ports too.

[ashifolfi]

yoooo this is so cool actually.

[reactive-firewall]

Plea to leverage git submodules when transitioning

@mszoek, this may be a bit technical, and off topic, but can we leverage git submodules (and I am willing to help get these all setup) for the projects that we are shifting to?

My Rationale:

maintaining the whole OS in the repo will:

• continue to slow down your workflow; git submodules can greatly simplify pulling from upstream
• keep the repo on the path of becoming too monolithic (git fsck + git autogc + git prune + git repack + git push --force --atomic could help too) ; git submodules are symbolic and will remain trivial (they are just a repo-config and git hash in size)
• remain cumbersome to update and even sync ; git submodules are modular and will improve the performance of syncs (e.g., O(mⁿ ) vs O((m*1)*n) for git graph-traversal)
• continue the unwieldy ad-hoc tracking of source providence right-now ; git submodules will simplify some of this by being a clear link between upstream and local components

and more:

• submodules are build-in to native git (unlike extensions like git-lfs which is also worth a look)
• submodules are supported as hyperlinks between GitHub projects on the GitHub UI
• submodules can even be recursive if needed (e.g., can be turtles all the way down)
• pivoting to Darwin is a breaking change already, so this gives the project the opportunity to leverage git submodules before re-stabalizing
• these submodules can be introduced incrementally on branches or all-together on the darwin branch.
• submodules allow for focused collaboration by modularizing the project without breaking the connections as whole

Suggestions for what to submodule

• Frameworks (recessively per .framework target) (each framework should be hermetic to git, eventually allowing us to pull-in other frameworks from others like cocoatron AND GNUStep AND perhaps swift, separately)
  • this also provides the ability to allow parallel work from the community (e.g., my interest is in re-factoring most of Foundation.framework and AppKit.framework so they are modernized and follow apple public API, but I don't care as much about the implementations of CoreData.framework or CoreGraphics.framework etc.)
• Core applications (per .app target)
  • even when required for RavynOS the ability to test building them on macOS individually will improve the ability to develop and "keep your focus on what you want to work on"
  • some apps can end-up being quite large, and from a maintainability perspective modularizing makes sense
  • fixing an app should not hold up the last-good-version of everything else
• any tools like cc-tool or libobjc (these can be almost anywhere they need to be)
• different maintainers can help maintain the portion they have expertise to contribute to
• future drivers (allow a path forward for the discussed ideas of future hardware support drivers)
• and finally the bootloaders and bootstrap tools (this is not my area of expertise, but I imaging the bootloaders are from somewhere that we can point at)

so what's the catch?

Mostly that these need to be configured initially and committed like any other change to the top-repository.

As far as configuration:

• configuration goes in .gitmodules (a section for EVERY submodule)
• automatic configuration will also go in the normal repo when activating/initializing a submodule locally in your clone ${GIT_DIR:-.git}/config (the workspace or local path as in git config --local --list or git config --worktree --list would list)
• each submodule requires a reference repo url (e.g. git url same as any git clone would - this can even be file://somepath for new empty submodules)
• each submodule needs a path in the repo to place the submodule worktree when checked-out (e.g. just like the path you pass to a git clone after the git-repo) -- I recommend these just be the name of the git-repo (e.g. https://github.com/ravynsoft/Frameworks.git could be named "Frameworks" and would then be placed by git at ./Framewoks inside the ravynOS clone) and any prefix needed (e.g., cc-tools could go in contrb/cc-tools)
• OPTIONALLY we can track any known branch in the sub-module's repo -- this is recommended and is typically master or main or for things like cPython might be 3.13, etc.)
• OPTIONALLY there are config fields for ignore rules and update modes -- I recommend these be set to ignore = dirty (ignore all local changes inside submodule) and update = checkout (always checkout an actual git-ref when updating the submodule)

As far as overhead: small network performance impact to fetch the multiple git-repos 🤷

As far as typical usage:

need to checkout/update submodules (e.g., after a git-pull)

$ #git fetch
$ #git pull
$ git submodule update --init --force --remote --checkout

• probably need to update areas of the build logic that use git-checkout to followup with the submodule update

---

everything to know about git-submodules (and more)

• git help submodules -- the official git-submodules manual
• https://github.blog/open-source/git/working-with-submodules/
• Example of actual .gitmodules submodule config: https://github.com/reactive-firewall/ash/blob/BSDLike-Ash/.gitmodules
  • and what they look like in github UI: https://github.com/reactive-firewall/ash/tree/BSDLike-Ash/musl