Darling logo

Monday, February 6, 2017

The Mach-O Transition: Darling in the Past 5 Years

Darling has been under development for almost five years now, which invites the questions — what has happened over the past years, are we getting anywhere and when will we get there.

Darling's History

Darling was initially based on the maloader project, which can still be found on GitHub. Looking back at these times, I regard this largely as a mistake, although I can argue that one learns from his/her mistakes. Back then, I had very little knowledge of macOS technologies and architecture, and maloader was the only existing proof of concept. It is a proof of concept that quickly allows you to get very primitive applications running, but it loses breath even faster as you move to more complex software.

Later on, realizing the limitations of maloader's approach, I embarked on a transition to building Apple's source code directly, instead of doing translation of high level APIs to their Linux counterparts (like maloader), e.g. by directly bridging libSystem to glibc. Although this requires a much larger time investment upfront to get even the very basic "Hello world" application running, it has proven to be the right thing to do. Huge compatibility issues that had plagued Darling since its inception were suddenly gone. And if you build one piece of original Apple source code, it quickly becomes possible to build more with very low effort.

At this point, I looked at Wine as a project of analogous goals and wanted to copy some of their ideas. This turned out to be a mistake as well, although this time it doesn't entail throwing away months' worth of work.

The Mach-O Transition

As you may know, macOS and other of Apple's operating systems use Mach-O as the format of choice for application and dynamic library binaries. This is in contrast with ELF used on other Unix-like platforms. As of now, the master branch of Darling still produces ELF files. This means ELF is used for all macOS libraries and frameworks, as well as various executables making up the macOS-like environment in Darling.

This is similar to what Wine does. If you peek inside Wine's directories (possibly /usr/lib/x86_64-linux-gnu/wine/wine on your system), you will find scores of .dll.so files. This take on building up a Windows-like runtime environment has (among other things) enabled the infamous "winelib" based applications to exist. It is very well possible that building ELFs instead of real PE DLLs bears no disadvantages for Wine; I cannot assess this. But it has brought numerous complications for Darling, up to a point where it was decided things had to be done differently.

The biggest hurdle is Apple's reliance on Mach-O specific features. Unlike ELF, which is a very lean format, Apple doesn't think twice before adding hosts of new features in Mach-O. They don't do so in many other areas, as their list of system calls would attest, and Mach-O is no exception. These features have no direct counterparts in ELF — two-level namespace, symbol reexports, specialized segments, extra initializer parameters or FAT files to name just a few. There are also other, more subtle differences, such as symbol prefixing with an underscore, which is a pain to deal with, as it required Darling to patch every assembly file taken from Apple.

This is why Darling now has a branch named using-machos-experiment where all of Darling's components, with the exception of a few small executables required to bootstrap Darling, are built as Mach-O files. Thanks to the Clang compiler, this isn't as hard or daunting as it may seem. Most Linux distributions provide Clang with multitargeting capabilities, meaning it can natively produce Mach-O object files, as if built on macOS itself. The only missing piece was the linker, which was very easy to add and build during Darling's build process.

What it Means

This change will bring Darling even closer to a real macOS-like environment. There will now be real .dylib files in /usr/lib when running macOS applications. Applications doing all kinds of exotic non-portable operations (such as loading a standard system .dylib file into memory and then asking the dynamic linker to use it) will find themselves at home. In this development branch, all components are built as fat Mach-O files containing both the x86_64 and i386 versions, meaning the build is done in one go.

This puts an end to various symbol name conflicts between macOS libraries and Linux libraries, which appear time from time despite efforts to counter them with ELF symbol versioning. It enables us to use Apple's original dynamic linker or Objective-C runtime, yet another step in reducing the compatibility gap between macOS and Darling. It also improves the chances that an original library coming from macOS would work when placed into Darling, leaving the possible EULA issues aside.

Technically, this has further impact on Darling's ability to interact with the outside world, e.g. with libasound or libpulse to play a sound, but these complications have very simple solutions.

Further Experiments

The Mach-O branch currently uses a neat trick to run 32-bit macOS applications. It actually loads them in a 64-bit Linux process and it transitions into 32-bit mode (and back) as needed. You may call it a technical exercise with few real-world benefits, and you could probably be right. We will see. But there are certain benefits, such as being able to map commpage at the right address (which would not be possible in a 32-bit Linux process).

Containerization

Darling still provides prefixes, but no longer manages them the same way as Wine. You probably know the annoying popup saying that Wine is updating the prefix. Well, this is gone in Darling.

Darling now uses overlayfs to overlay user prefixes on top of prefix default contents. Especially in the Mach-O branch, the files are no longer strewn across several places, everything is in <install-prefix>/libexec/darling.

After the overlay is set up, Darling switches to this overlay as its root file system (using pivot_root()) in its mount namespace. This and other uses of Linux namespaces bring Darling closer to how Docker or LXC work. In future, it could be optionally possible to isolate a Darling prefix into a separate network namespace, making it work more like a standalone system.

Future Development

Darling is no longer a one man show. Two skilled engineers — Sergey and Andrew — have joined in and have already greatly contributed.

Darling currently needs to improve in areas around Mach ports (implemented in a Linux kernel module) and their support in kqueue(). Going forward, the big question is whether to keep pursuing having our own code for Mach port support or try to port over original code from XNU. But after this is tackled, we should be able to run many of macOS daemons including launchd, notifyd, syslogd and others.

In the meantime, Sergey is experimenting with GUI applications, plugging all the needed layers together, and is figuring out how to make progress in this area. Andrew is helping bring Darling's environment closer to macOS, for example by building Perl inside Darling. This could sound stupid, but you would be surprised how often is Perl actually needed (and invoked) on macOS; and no, Darling absolutely cannot use Perl from the underlying Linux system.

While this may not be very interesting for end-users, every project must start with a rock-solid foundation to remain viable in the future. This is why GUI apps are a long-term goal, but you only see Darling being used with console applications. Console applications are not the primary goal, but are the means to an end.

Documentation and Debugging

Darling is currently severely underdocumented, which makes the initial learning process very difficult for new contributors. I have started a documentation project on Darling's wiki where I plan to explain both ideas and techniques specific to Darling, as well as general inner workings of macOS. The latter is also very important, because I would bet even most skilled macOS developers have vague knowledge at best of how, for instance, Mach ports work. Even less probably know about the commpage I mentioned earlier in the text.

Work is also underway for the using-machos-experiment branch to produce a gdb-darling debugger that would be able to read symbol and debug information not only from ELF files, but also from Mach-Os. This is an absolute must have for the Mach-O Transition, where 99% of Darling's code is compiled into Mach-O files, making debugging otherwise very complicated.

50 comments:

  1. Even if Darling can't run GUI application yet, I am still excited about the potential it has for the future!

    ReplyDelete
    Replies


    1. So, you're doing exactly what FreeBSD does with Linux apps all those years?علی یاسینی

      Delete
  2. Sounds like a fantastic base for the future.

    In another 10 years it will be complete enough to run OSX wine in Darling :)

    ReplyDelete
    Replies
    1. I know people like to joke about this, but Wine will never run under Darling. The short version is x86 has only 2 TLS registers available. See https://wiki.darlinghq.org/documentation:thread_local_storage#registers

      Therefore, without an extra level of indirection added into Darling or Wine, Wine would always end up conflicting with either Darling or glibc. And there is no reason to add special hacks for something without a real use case.

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Congrats! I'm very happy with this major advance!

    ReplyDelete
  5. So, you're doing exactly what FreeBSD does with Linux apps all those years?

    ReplyDelete
  6. Very nice read. Your success present & future are a success for us all. Keep up the amazing work - Im sure a lot of developers l ok ve geekt CLI Apps even though we also are hoping for GUI too someday in the future. +

    ReplyDelete
  7. Thanks, it's great somebody is working on this :)

    ReplyDelete
  8. Take a transport ride to Downtown New Brunswick. It's experienced an excellent renascence over the most recent couple of years. A lot of Museums. A lot of network occasions and theaters. A lot of spots to eat. Contact the New Brunswick community affiliation. Return Of Xander Cage Jackets They'll help. Have a great time.

    ReplyDelete
  9. Get the most recent neighborhood and national political news and updates in a single spot. Speak more loudly against the issues which influence the general population. Voice your interests against any political pioneer, a political issue or social issue.

    keyword : Latest Politics News, Trending News, Local and National News, Politics News in Hindi, Social and Political Issues, Live TV

    ReplyDelete


  10. نقل عفش من المدينة المنورة الى جدة نقل عفش من المدينة المنورة الى جدة
    نقل عفش من المدينة المنورة الى مكة نقل عفش من المدينة المنورة الى مكة
    نقل عفش من المدينة الى ينبع نقل عفش من المدينة الى ينبع

    ReplyDelete
  11. Thank you for the Post. You can connect us from remote areas. We have structured the advantageous strides by following which you can get accomplishment over your printer issues. These means are detailed by experienced geeks. Each progression is easy to pursue. We give learning to the two sorts of printers.
    How to Connect Dell Printer to Wireless Network

    ReplyDelete
  12. keep connected with dhankesari lottery draw result on-line. state lottery is one in all the foremost and wide spread lottery round the west bengal. Keep visiting our web site to west bengal state lottery draw results of the day.

    dhankesari lottery result 4pm

    dhankesari lottery result

    ReplyDelete
  13. after to refreshing my Windows 10 to its latest form, my associated HP printer is currently unfit to take any of my printing direction. I scanned Google for this, and found this is because of the driver issue. Additionally, I come to realize that, I can download HP printer's driver and related programming from www hp com 123 setup. Be that as it may, I don't have the foggiest idea how to download driver from this webpage. If you don't mind would someone be able to please help me viewing this as I needed to proceed with my printing fill in at the earliest opportunity.

    ReplyDelete
  14. awesome post. It was amazing and easy to understand. logo designs thanks for this

    ReplyDelete
  15. Here you get information related to Dhan Kesari , if you are Lottery Lover then this is best for You,

    ReplyDelete
  16. How to activate Paramount Network on Roku?

    You need to launch the paramount channel app on the Roku device. For the paramount channel download, you can get the channel app from the Roku channel hub. Once you get the channel app, you need to install the channel app and generate the code for the activation. Surf to the paramountnetwork.com/activate site and enter the code for the activation. After the channel activation, you will be asked to enter the tv provider user id and the password to link the channel app.
    For more details on the paramount channel app, you can get the guidance from our specialized team via the toll-free number +1-805-436-1200.

    ReplyDelete
  17. Unfortunately, DISH Anywhere app Roku is not available as a standalone app. Moreover, you can watch its content by the screen mirroring feature. First, enable the screen mirroring feature on your Smartphone and the Roku device. And make sure to connect them to the same wireless network. Further, download the app on the mobile device and install it by following the screen prompts. Finally, access the app and the content will appear on the TV screen. Play any video and start watching the Dish Anywhere programs on the Roku TV screen. Reach @ +1-844-221-7630

    ReplyDelete
  18. Unfortunately, DISH Anywhere app Roku is not available as a standalone app. Moreover, you can watch its content by the screen mirroring feature. First, enable the screen mirroring feature on your Smartphone and the Roku device. And make sure to connect them to the same wireless network. Further, download the app on the mobile device and install it by following the screen prompts. Finally, access the app and the content will appear on the TV screen. Play any video and start watching the Dish Anywhere programs on the Roku TV screen.

    ReplyDelete
  19. Cinemax on Roku is available as a stand-alone app. Download the app from the Roku channel store and activate it via its site for activation. Or else, you can watch it without cable using live TV streaming services like Hulu, PlayStation Vue, Amazon Prime Video. Make sure to create your account for these and choose the packages that consist of the Cinemax channel. Pay your bucks, and the plan will be active. Further, you can stream it via a big TV screen. Still having doubts, then visit our blogs and read it once and then start to add the channel. Reach @+1-866-218-6310

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Thanks for sharing this amazing post, keep sharing with us. Quickbooks technical support phone number

    ReplyDelete
  22. Having trouble with your Matlab assignments? Matlab Assignment Help team got you covered. We work 24/7 to ensure that all your assignments are well-taken care of. We ensure time and quality delivery at all times. To get the best Matlab Homework Help, visit our website now.

    ReplyDelete
  23. How to stream Disney now on Roku
    If you're not a Roku com link, this is an entertainment service that no one can afford to skip in 2020. When you are an active member of the Roku com link, you will be able to complete the activation of the Roku com link. We've got so many exciting things to say about activating the Roku device that you're sure to enjoy being part of this awesome Roku Com link User group.
    Roku Com Link
    Internet streaming is the current hottest sensation in the entertainment market, and Roku is a leader in the online streaming market. Roku is a powerful digital device that enables users to watch live movies, Television programs, and other channels from the online platform to their Television sets. Roku services can be accessed from the Roku com link. Roku Com Link Activation is a straightforward process, and you'll be able to use limitless content channels of excellent quality directly on your Television screen.
    You will download and install all popular video sharing apps such as YouTube, HULU, Amazing, and Netflix on your Roku unit. If you'd like to access the Roku TV Activate page, you can check out the official Roku Com Activate Link for more information.

    ReplyDelete
  24. When I take Matlab assignment help online, I expect the solution not only to fetch me good grades but also to help me with my exam revisions. I was not impressed at all by the Matlab assignment helper who tackled my signal processing homework. Even if it secured me a good grade, I am still having trouble understanding anything in the solution so technically I still have to deal with voluminous books to understand this topic. I hope you guys can try and simplify the solutions in the future to make them easier to understand.

    ReplyDelete
  25. hy
    thanks for sharing this blog

    plz visit 123movies


    ReplyDelete
  26. To fix HP printer driver Windows 10 issue , impair HP Smart Install, run ... of introducing HP printer is simple and won't take in excess of a couple of moments.

    ReplyDelete
  27. 123.hp.com/setup - Complete Hp Printer setup, Driver Software installation, 123 hp com wireless setup & Troubleshooting 123hpcom Printer.

    ReplyDelete
  28. Buy Adderall online at your door step from one & only trusted pharma in this ERA. Buying Adderall 30mg with Credit card.

    ReplyDelete
  29. Your blog provided us with valuable information to work with.every tips of your post are awesome. Thanks a lot for sharing. Keep blogging,

    Gastroenterolog

    ReplyDelete
  30. Excellent and very exciting site. Love to watch. Keep Rocking.
    KrogerStoresFeedback

    ReplyDelete
  31. Your website is really cool and this is a great inspiring article.

    tellthebell.com

    ReplyDelete
  32. 123.hp.com setup - Complete Hp Printer setup, Driver Software installation, 123 hp com wireless setup & Troubleshooting 123hpcom Printer.

    ReplyDelete