Darling logo

Tuesday, July 23, 2019

Darling Progress Report Q2 2019

Summary of Progress

We are very excited to say that in Q2 2019 (April 1 to June 30) we saw more community involvement than ever before. Many pull requests were submitted that spanned from bug fixes for our low level assembly to higher level modules such as the AppKit framework. Thanks to everyone for your contributions and we hope for this level of engagement to continue.

Darling Community Discord

While we have relied solely on GitHub issues for a long time, it has become apparent that a platform for people to ask general questions, share knowledge, and coordinate efforts is necessary. There is now an official Discord server that is open for everybody to join in order to discuss anything Darling related. Here is the invite link: https://discord.gg/XRD3mQA. Feel free to join and invite anyone who may be interested!

Darling Core

The community submitted more pull requests than we have ever received before in Q2.

James Urquhart's pull requests gave us more stubs for many frameworks including DrawSprocket, AGL, Carbon, CoreServices, and ApplicationServices. Wow, what a list, thank you James! James also fixed a very low level bug where system calls with large numbers of arguments wouldn't work properly for 32-bit applications. Finally, he fixed a problem where Darling was using clock_sleep instead of clock_nanosleep and a bug with the stat system call.

Commits sent in by kamillys gave us more stubs for the AddressBook and Carbon frameworks.

New Website

Sergey has remade our project website using Hugo. This new website looks very modern and adapts for mobile devices. This refresh looks very snazzy! The Frequently Asked Questions section of the site is perfectly placed to answer some of the most commonly sought after information. The whole page has the right balance of content without overwhelming the viewer with technical details. The new site also highlights the key features and goals of Darling, including "Fast, Free, Compatible, Easy to Use, and Native."



AppKit

Continued improvements were seen with our implementation of AppKit. Sergey squashed some bugs including crashing when no spelling engines were present and pointers having the wrong printing format. He got the X11 pasteboard to work with our AppKit. Ctrl+c/Ctrl+v away, even pretend it's Command if you want!

GitHub user kamillys also submitted a PR which gave AppKit more stubs for NSApplication and NSWorkspace.

AppKit also now also reexports ApplicationServices, solving some crashes due to undefined symbols when programs were being loaded.

Nested Frameworks

Some frameworks on macOS appear to be just one framework but under the hood are made up of sub-frameworks which are usually reexported so that when being linked to it appears to be one big framework. Until now, our build system didn't support nested frameworks. In June, Andrew added support for this and system frameworks such as Accelerate now have the same file structure as they do on macOS. This was accomplished using some CMake magic and carefully replicating the structure of some system frameworks.

Framework Nesting VS Reexporting

Reexporting is when a dynamic library presents itself as also containing the symbols of the library or framework it is reexporting. Symbols are typically functions, constants, or classes. This is why you can link to only AppKit and without linking to Foundation you can use Foundation symbols. Nested frameworks are when one framework is placed inside the other. In the case where a larger framework is broken into sub-frameworks (like CoreServices and ApplicationServices) you need to have the parent framework reexport those sub-frameworks in order to appear as one framework. Another use case for nesting frameworks is when your framework relies on a supporting library and it's only meant to be used by your framework. This is like keeping it private. In that case the parent framework wouldn't reexport it but it would instead link to its child framework.



Foundation

A good amount of attention was received by Foundation this time. Our NSKeyedArchiver/NSKeyedUnarchiver had many bug fixes and improvements submitted by  CuriousTommy. Stability in this area very important because NSKeyedArchiver is used to serialize objects extensively in almost every Cocoa application. In addition to those improvements, Tommy gave Foundation symbols for the NSScriptClassDefinition category of NSObject and fixed an issue with some some macros in NSObjCRuntime.h.

Stubs were also added by kamillys for NSScriptClassDescription and for some NSCallbacks functions that Foundation was missing.

These changes improve the general compatibility because as the name implies, Foundation is very fundamental and relied on by many other libraries and applications.

CoreFoundation

Error handling in NSSet was improved by Tommy by better checking parameters, particularly if someone tries to insert a nil object into a set, which isn't supposed to be possible but was possible before. Thanks Tommy!

Going Forward

Strong community involvement is always exciting and motivating. We received lots of feedback and pull requests! If you have any questions about ways you can contribute and learn, join our Discord linked at the top of this post. When first trying to help out an open source project it may seem unclear how you can help. If none of the open issues on GitHub jump out of you, the Discord is the perfect place to ask where you can start. The wiki also is very useful because it contains build instructions and workarounds for known build problems.

It was also nice to see the community step up and fix some 32-bit specific bugs because as anticipated in the last progress report, the release of macOS coming in a few months drops all support for 32-bit applications. Darling plans to continue supporting 32-bit applications for a long time and once GUI support improves it will be a strong option for applications that stop working due to the lack of 32-bit capabilities.

See you all in a few months,

- The Darling Project

Friday, May 3, 2019

Darling Progress Report Q1 2019

Summary of Progress

Darling made good progress in the first quarter of 2019. A total of 13 issues were closed from January 1 2019 to March 30 2019. The work continues for GUI app support. The thing holding us back the most was a lack of support for running a debugger (lldb) in Darling. Last summer Luboš took care of this complicated task which involved delving into Darling's multithreading and Mach APIs in #304 and was capped off with loading Mach-O binaries in the kernel with #364.

In the meantime, Sergey has been putting a lot of work into getting our new Cocoa stack to work and has produced exciting results. The last thing preventing us from announcing and shipping basic GUI application support is trouble around loading Linux's native OpenGL drivers and other files. Loading native Linux things doesn't reliably work with our current mechanism for filesystem virtualization, which is based on mount namespaces and chrooting.

Luboš is working on a replacement that we have dubbed "vchroot" (virtual chroot) that will be a one-time fix for this issue, but progress has been stalled due to time constraints. We would also like to welcome Jack Howell as the newest member of the project! Below are a few accomplishments which we would like to highlight.

Introduction from Jack

We would like to welcome Jack Howell to Darling! Here’s a bit about him:
“I'm excited to see all the recent progress around Darling because I think it might be time for the community to develop a software solution similar to Wine for macOS applications. With so many commercial applications and games being available for macOS,  Darling might have a special place in the ever coming but still not yet fully "mainstream compatible" desktop/workstation Linux experience. My current goal is to help the amazing people behind the Darling project get to a state where Darling is able to run commercial applications that utilize a minimal set of the macOS APIs. (Those are the ones that are theoretically more feasible to run in the short term)

A fair bit of the most popular games right now are in this category. Its no secret I would like to see Linux gaming advance even more and hopefully that future will include Darling at some point.

As for the technical side, I've been looking at the current issues with our Cocotron stack that are blocking a specific game's startup that I'm trying to bring up on Darling. Sergey has been an invaluable source of help during this process. Based on mostly his guidence I sent out an initial set of patches that resolved the issues I did see with the application. I continue testing and troubleshooting to see where the next area is that we will need to tackle.”


git now works in Darling

Before issue #365 was fixed by Sergey the popular version-control software git would crash while executing the index-pack routine. As the routine typically happens while cloning a repository, git was previously pretty much useless in Darling. With git now working, software development in Darling has become more of a reality and perhaps even more significant is that this is a step towards getting Homebrew working, as it relies on git extensively. The next step for getting Homebrew to run is HTTPS support for curl, which will require work in corecrypto. Some efforts towards getting curl to work have already been completed, including getting securityd to run.

Progress towards Xcode support

Some steps forward were made by Andrew for running Apple's Xcode IDE. While Xcode's GUI will not be working in Darling for the foreseeable future, some issues preventing programs the Xcode Command Line Tools were resolved. The now closed issue #445 documented this effort which resulted in hundreds of missing constants and classes being added to Darling's system frameworks.

While attempting to run xcodebuild a discrepancy with how Darling determined the bundle path for frameworks (issue #490) was exposed and promptly fixed.

While the Command Line Tools still don't work when being run through xcodebuild, like before workarounds exist for running them. We are now much closer than before to fully supporting the CLT.

Multithreading now works in Python 3

Previously Darling failed to support python programs that use the threading module. Thanks to Luboš, issue #462 has been fixed.

xtrace

Sergey has been working on improving xtrace, our Darwin syscall tracing tool. In addition to syscall names, xtrace can now display the arguments passed to syscalls, formatting them appropriately — as an integer, a pointer or as a string (featuring some basic syntax highlighting). In some cases — for many Mach traps and for some frequently-used BSD syscalls — xtrace can even display the arguments symbolically, such as displaying a value of 1537 passed to an open() call as O_WRONLY | O_CREAT | O_TRUNC. In general, xtrace is now mostly on par with, and in some aspects, better than the dtruss tool available on macOS.

The coolest new xtrace feature is support for parsing and displaying MIG routine calls. Whenever the traced process sends or receives a Mach message, xtrace prints a short description about the message, such as its destination and reply ports and the size of the message body. With MIG tracing support, xtrace can additionally decode what remote procedure call (RPC) this message actually encodes, and display the call similarly to how it displays syscalls, complete with arguments and return values. This makes it a lot easier to debug issues related to inter-process communication, because Mach IPC and MIG in particular are pervasively used on Darwin for everything from logging to inspecting and debugging other processes.

The xtrace work is not complete yet, but it's close to completion and will be merged in the near future.

Outlook

With the macOS 10.15 release that will be announced at WWDC this year, 32-bit applications will cease to function. Xcode 10 already lacks support for targeting 32-bit macOS using the SDK included with it. On the other hand, Darling has no plans to remove 32-bit support. The result of this may be an increase in demand for Darling as tech-savvy people try to find ways to run those apps. We wish to accelerate development as much as possible and spark interest in our project leading up to the macOS 10.15 release because this is a big opportunity for growth.

Apple is also rumored to transition towards the ARM architecture for their Macs. While it's hard to see ARM replacing Intel in Desktop Macs, it's possible for macOS to someday abandon the x64_64 architecture as well.

We aim to keep the community updated on the progress of Darling and given our level activity it seems fit to write four progress reports per year. We hope to have lots of good news about our progress when the time for the next progress report comes around!

Community

It is important that we know what people want to use Darling for in order to properly prioritize the project's development. The usual requests are applications such as Xcode, but some quite interesting uses of Darling in Linux based Continuous Integration (CI) servers which need macOS-only software have popped up recently. If there is something you want to use Darling for which may not be obvious please comment below!

Useful links

Our website
Darling Github
The wiki