From 5f1cb933ced23707900f74242ffd58fc2f9fc4fa Mon Sep 17 00:00:00 2001 From: Thomas A Date: Sat, 20 May 2023 20:59:36 -0700 Subject: [PATCH] Upload README.md and UPDATE_SOURCE.md --- README.apple.md | 416 ++++++++++++++++++++++++++++++++ README.md | 417 +-------------------------------- darling/notes/UPDATE_SOURCE.md | 18 ++ 3 files changed, 439 insertions(+), 412 deletions(-) create mode 100644 README.apple.md create mode 100644 darling/notes/UPDATE_SOURCE.md diff --git a/README.apple.md b/README.apple.md new file mode 100644 index 0000000..da6bb7b --- /dev/null +++ b/README.apple.md @@ -0,0 +1,416 @@ +What is XNU? +=========== + +XNU kernel is part of the Darwin operating system for use in macOS and iOS operating systems. XNU is an acronym for X is Not Unix. +XNU is a hybrid kernel combining the Mach kernel developed at Carnegie Mellon University with components from FreeBSD and a C++ API for writing drivers called IOKit. +XNU runs on x86_64 for both single processor and multi-processor configurations. + +XNU Source Tree +=============== + + * `config` - configurations for exported apis for supported architecture and platform + * `SETUP` - Basic set of tools used for configuring the kernel, versioning and kextsymbol management. + * `EXTERNAL_HEADERS` - Headers sourced from other projects to avoid dependency cycles when building. These headers should be regularly synced when source is updated. + * `libkern` - C++ IOKit library code for handling of drivers and kexts. + * `libsa` - kernel bootstrap code for startup + * `libsyscall` - syscall library interface for userspace programs + * `libkdd` - source for user library for parsing kernel data like kernel chunked data. + * `makedefs` - top level rules and defines for kernel build. + * `osfmk` - Mach kernel based subsystems + * `pexpert` - Platform specific code like interrupt handling, atomics etc. + * `security` - Mandatory Access Check policy interfaces and related implementation. + * `bsd` - BSD subsystems code + * `tools` - A set of utilities for testing, debugging and profiling kernel. + +How to build XNU +================ + +Building `DEVELOPMENT` kernel +----------------------------- + +The xnu make system can build kernel based on `KERNEL_CONFIGS` & `ARCH_CONFIGS` variables as arguments. +Here is the syntax: + + make SDKROOT= ARCH_CONFIGS= KERNEL_CONFIGS= + +Where: + + * \: path to macOS SDK on disk. (defaults to `/`) + * \: can be `debug`, `development`, `release`, `profile` and configures compilation flags and asserts throughout kernel code. + * \ : can be valid arch to build for. (E.g. `X86_64`) + +To build a kernel for the same architecture as running OS, just type + + $ make + $ make SDKROOT=macosx.internal + +Additionally, there is support for configuring architectures through `ARCH_CONFIGS` and kernel configurations with `KERNEL_CONFIGS`. + + $ make SDKROOT=macosx.internal ARCH_CONFIGS=X86_64 KERNEL_CONFIGS=DEVELOPMENT + $ make SDKROOT=macosx.internal ARCH_CONFIGS=X86_64 KERNEL_CONFIGS="RELEASE DEVELOPMENT DEBUG" + + +Note: + * By default, architecture is set to the build machine architecture, and the default kernel + config is set to build for DEVELOPMENT. + + +This will also create a bootable image, kernel.[config], and a kernel binary +with symbols, kernel.[config].unstripped. + +To intall the kernel into a DSTROOT, use the `install_kernels` target: + + $ make install_kernels DSTROOT=/tmp/xnu-dst + +Hint: +For a more satisfying kernel debugging experience, with access to all +local variables and arguments, but without all the extra check of the +DEBUG kernel, add something like: + CFLAGS_DEVELOPMENTARM64="-O0 -g -DKERNEL_STACK_MULTIPLIER=2" + CXXFLAGS_DEVELOPMENTARM64="-O0 -g -DKERNEL_STACK_MULTIPLIER=2" +to your make command. +Replace DEVELOPMENT and ARM64 with the appropriate build and platform. + + + * To build with RELEASE kernel configuration + + make KERNEL_CONFIGS=RELEASE SDKROOT=/path/to/SDK + + +Building FAT kernel binary +-------------------------- + +Define architectures in your environment or when running a make command. + + $ make ARCH_CONFIGS="X86_64" exporthdrs all + +Other makefile options +---------------------- + + * $ make MAKEJOBS=-j8 # this will use 8 processes during the build. The default is 2x the number of active CPUS. + * $ make -j8 # the standard command-line option is also accepted + * $ make -w # trace recursive make invocations. Useful in combination with VERBOSE=YES + * $ make BUILD_LTO=0 # build without LLVM Link Time Optimization + * $ make REMOTEBUILD=user@remotehost # perform build on remote host + * $ make BUILD_JSON_COMPILATION_DATABASE=1 # Build Clang JSON Compilation Database + +The XNU build system can optionally output color-formatted build output. To enable this, you can either +set the `XNU_LOGCOLORS` environment variable to `y`, or you can pass `LOGCOLORS=y` to the make command. + + +Debug information formats +========================= + +By default, a DWARF debug information repository is created during the install phase; this is a "bundle" named kernel.development.\.dSYM +To select the older STABS debug information format (where debug information is embedded in the kernel.development.unstripped image), set the BUILD_STABS environment variable. + + $ export BUILD_STABS=1 + $ make + + +Building KernelCaches +===================== + +To test the xnu kernel, you need to build a kernelcache that links the kexts and +kernel together into a single bootable image. +To build a kernelcache you can use the following mechanisms: + + * Using automatic kernelcache generation with `kextd`. + The kextd daemon keeps watching for changing in `/System/Library/Extensions` directory. + So you can setup new kernel as + + $ cp BUILD/obj/DEVELOPMENT/X86_64/kernel.development /System/Library/Kernels/ + $ touch /System/Library/Extensions + $ ps -e | grep kextd + + * Manually invoking `kextcache` to build new kernelcache. + + $ kextcache -q -z -a x86_64 -l -n -c /var/tmp/kernelcache.test -K /var/tmp/kernel.test /System/Library/Extensions + + + +Running KernelCache on Target machine +===================================== + +The development kernel and iBoot supports configuring boot arguments so that we can safely boot into test kernel and, if things go wrong, safely fall back to previously used kernelcache. +Following are the steps to get such a setup: + + 1. Create kernel cache using the kextcache command as `/kernelcache.test` + 2. Copy exiting boot configurations to alternate file + + $ cp /Library/Preferences/SystemConfiguration/com.apple.Boot.plist /next_boot.plist + + 3. Update the kernelcache and boot-args for your setup + + $ plutil -insert "Kernel Cache" -string "kernelcache.test" /next_boot.plist + $ plutil -replace "Kernel Flags" -string "debug=0x144 -v kernelsuffix=test " /next_boot.plist + + 4. Copy the new config to `/Library/Preferences/SystemConfiguration/` + + $ cp /next_boot.plist /Library/Preferences/SystemConfiguration/boot.plist + + 5. Bless the volume with new configs. + + $ sudo -n bless --mount / --setBoot --nextonly --options "config=boot" + + The `--nextonly` flag specifies that use the `boot.plist` configs only for one boot. + So if the kernel panic's you can easily power reboot and recover back to original kernel. + + + + +Creating tags and cscope +======================== + +Set up your build environment and from the top directory, run: + + $ make tags # this will build ctags and etags on a case-sensitive volume, only ctags on case-insensitive + $ make TAGS # this will build etags + $ make cscope # this will build cscope database + + +How to install a new header file from XNU +========================================= + +To install IOKit headers, see additional comments in [iokit/IOKit/Makefile](). + +XNU installs header files at the following locations - + + a. $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers + b. $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders + c. $(DSTROOT)/usr/include/ + d. $(DSTROOT)/System/DriverKit/usr/include/ + e. $(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders + +`Kernel.framework` is used by kernel extensions.\ +The `System.framework` and `/usr/include` are used by user level applications. \ +`/System/DriverKit/usr/include` is used by userspace drivers. \ +The header files in framework's `PrivateHeaders` are only available for ** Apple Internal Development **. + +The directory containing the header file should have a Makefile that +creates the list of files that should be installed at different locations. +If you are adding the first header file in a directory, you will need to +create Makefile similar to `xnu/bsd/sys/Makefile`. + +Add your header file to the correct file list depending on where you want +to install it. The default locations where the header files are installed +from each file list are - + + a. `DATAFILES` : To make header file available in user level - + `$(DSTROOT)/usr/include` + + b. `DRIVERKIT_DATAFILES` : To make header file available to DriverKit userspace drivers - + `$(DSTROOT)/System/DriverKit/usr/include` + + c. `PRIVATE_DATAFILES` : To make header file available to Apple internal in + user level - + `$(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders` + + d. `KERNELFILES` : To make header file available in kernel level - + `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers` + `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders` + + e. `PRIVATE_KERNELFILES` : To make header file available to Apple internal + for kernel extensions - + `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders` + +The Makefile combines the file lists mentioned above into different +install lists which are used by build system to install the header files. There +are two types of install lists: machine-dependent and machine-independent. +These lists are indicated by the presence of `MD` and `MI` in the build +setting, respectively. If your header is architecture-specific, then you should +use a machine-dependent install list (e.g. `INSTALL_MD_LIST`). If your header +should be installed for all architectures, then you should use a +machine-independent install list (e.g. `INSTALL_MI_LIST`). + +If the install list that you are interested does not exist, create it +by adding the appropriate file lists. The default install lists, its +member file lists and their default location are described below - + + a. `INSTALL_MI_LIST` : Installs header file to a location that is available to everyone in user level. + Locations - + $(DSTROOT)/usr/include + Definition - + INSTALL_MI_LIST = ${DATAFILES} + + b. `INSTALL_DRIVERKIT_MI_LIST` : Installs header file to a location that is + available to DriverKit userspace drivers. + Locations - + $(DSTROOT)/System/DriverKit/usr/include + Definition - + INSTALL_DRIVERKIT_MI_LIST = ${DRIVERKIT_DATAFILES} + + c. `INSTALL_MI_LCL_LIST` : Installs header file to a location that is available + for Apple internal in user level. + Locations - + $(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders + Definition - + INSTALL_MI_LCL_LIST = ${PRIVATE_DATAFILES} + + d. `INSTALL_KF_MI_LIST` : Installs header file to location that is available + to everyone for kernel extensions. + Locations - + $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers + Definition - + INSTALL_KF_MI_LIST = ${KERNELFILES} + + e. `INSTALL_KF_MI_LCL_LIST` : Installs header file to location that is + available for Apple internal for kernel extensions. + Locations - + $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders + Definition - + INSTALL_KF_MI_LCL_LIST = ${KERNELFILES} ${PRIVATE_KERNELFILES} + + f. `EXPORT_MI_LIST` : Exports header file to all of xnu (bsd/, osfmk/, etc.) + for compilation only. Does not install anything into the SDK. + Definition - + EXPORT_MI_LIST = ${KERNELFILES} ${PRIVATE_KERNELFILES} + + g. `INSTALL_MODULEMAP_INCDIR_MI_LIST` : Installs module map file to a + location that is available to everyone in user level, installing at the + root of INCDIR. + Locations - + $(DSTROOT)/usr/include + Definition - + INSTALL_MODULEMAP_INCDIR_MI_LIST = ${MODULEMAP_INCDIR_FILES} + +If you want to install the header file in a sub-directory of the paths +described in (1), specify the directory name using two variables +`INSTALL_MI_DIR` and `EXPORT_MI_DIR` as follows - + + INSTALL_MI_DIR = dirname + EXPORT_MI_DIR = dirname + +A single header file can exist at different locations using the steps +mentioned above. However it might not be desirable to make all the code +in the header file available at all the locations. For example, you +want to export a function only to kernel level but not user level. + + You can use C language's pre-processor directive (#ifdef, #endif, #ifndef) + to control the text generated before a header file is installed. The kernel + only includes the code if the conditional macro is TRUE and strips out + code for FALSE conditions from the header file. + + Some pre-defined macros and their descriptions are - + + a. `PRIVATE` : If defined, enclosed definitions are considered System + Private Interfaces. These are visible within xnu and + exposed in user/kernel headers installed within the AppleInternal + "PrivateHeaders" sections of the System and Kernel frameworks. + b. `KERNEL_PRIVATE` : If defined, enclosed code is available to all of xnu + kernel and Apple internal kernel extensions and omitted from user + headers. + c. `BSD_KERNEL_PRIVATE` : If defined, enclosed code is visible exclusively + within the xnu/bsd module. + d. `MACH_KERNEL_PRIVATE`: If defined, enclosed code is visible exclusively + within the xnu/osfmk module. + e. `XNU_KERNEL_PRIVATE`: If defined, enclosed code is visible exclusively + within xnu. + f. `KERNEL` : If defined, enclosed code is available within xnu and kernel + extensions and is not visible in user level header files. Only the + header files installed in following paths will have the code - + + $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers + $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders + g. `DRIVERKIT`: If defined, enclosed code is visible exclusively in the + DriverKit SDK headers used by userspace drivers. + +Conditional compilation +======================= + +`xnu` offers the following mechanisms for conditionally compiling code: + + a. *CPU Characteristics* If the code you are guarding has specific + characterstics that will vary only based on the CPU architecture being + targeted, use this option. Prefer checking for features of the + architecture (e.g. `__LP64__`, `__LITTLE_ENDIAN__`, etc.). + b. *New Features* If the code you are guarding, when taken together, + implements a feature, you should define a new feature in `config/MASTER` + and use the resulting `CONFIG` preprocessor token (e.g. for a feature + named `config_virtual_memory`, check for `#if CONFIG_VIRTUAL_MEMORY`). + This practice ensures that existing features may be brought to other + platforms by simply changing a feature switch. + c. *Existing Features* You can use existing features if your code is + strongly tied to them (e.g. use `SECURE_KERNEL` if your code implements + new functionality that is exclusively relevant to the trusted kernel and + updates the definition/understanding of what being a trusted kernel means). + +It is recommended that you avoid compiling based on the target platform. `xnu` +does not define the platform macros from `TargetConditionals.h` +(`TARGET_OS_OSX`, `TARGET_OS_IOS`, etc.). + + +There is a deprecated `TARGET_OS_EMBEDDED` macro, but this should be avoided +as it is in general too broad a definition for most functionality. +Please refer to TargetConditionals.h for a full picture. + +How to add a new syscall +======================== + + + + +Testing the kernel +================== + +XNU kernel has multiple mechanisms for testing. + + * Assertions - The DEVELOPMENT and DEBUG kernel configs are compiled with assertions enabled. This allows developers to easily + test invariants and conditions. + + * XNU Power On Self Tests (`XNUPOST`): The XNUPOST config allows for building the kernel with basic set of test functions + that are run before first user space process is launched. Since XNU is hybrid between MACH and BSD, we have two locations where + tests can be added. + + xnu/osfmk/tests/ # For testing mach based kernel structures and apis. + bsd/tests/ # For testing BSD interfaces. + Please follow the documentation at [osfmk/tests/README.md](osfmk/tests/README.md) + + * User level tests: The `tools/tests/` directory holds all the tests that verify syscalls and other features of the xnu kernel. + The make target `xnu_tests` can be used to build all the tests supported. + + $ make RC_ProjectName=xnu_tests SDKROOT=/path/to/SDK + + These tests are individual programs that can be run from Terminal and report tests status by means of std posix exit codes (0 -> success) and/or stdout. + Please read detailed documentation in [tools/tests/unit_tests/README.md](tools/tests/unit_tests/README.md) + + +Kernel data descriptors +======================= + +XNU uses different data formats for passing data in its api. The most standard way is using syscall arguments. But for complex data +it often relies of sending memory saved by C structs. This packaged data transport mechanism is fragile and leads to broken interfaces +between user space programs and kernel apis. `libkdd` directory holds user space library that can parse custom data provided by the +same version of kernel. The kernel chunked data format is described in detail at [libkdd/README.md](libkdd/README.md). + + +Debugging the kernel +==================== + +The xnu kernel supports debugging with a remote kernel debugging protocol (kdp). Please refer documentation at [technical note] [TN2063] +By default the kernel is setup to reboot on a panic. To debug a live kernel, the kdp server is setup to listen for UDP connections +over ethernet. For machines without ethernet port, this behavior can be altered with use of kernel boot-args. Following are some +common options. + + * `debug=0x144` - setups debug variables to start kdp debugserver on panic + * `-v` - print kernel logs on screen. By default XNU only shows grey screen with boot art. + * `kdp_match_name=en1` - Override default port selection for kdp. Supported for ethernet, thunderbolt and serial debugging. + +To debug a panic'ed kernel, use llvm debugger (lldb) along with unstripped symbol rich kernel binary. + + sh$ lldb kernel.development.unstripped + +And then you can connect to panic'ed machine with `kdp_remote [ip addr]` or `gdb_remote [hostip : port]` commands. + +Each kernel is packaged with kernel specific debug scripts as part of the build process. For security reasons these special commands +and scripts do not get loaded automatically when lldb is connected to machine. Please add the following setting to your `~/.lldbinit` +if you wish to always load these macros. + + settings set target.load-script-from-symbol-file true + +The `tools/lldbmacros` directory contains the source for each of these commands. Please follow the [README.md](tools/lldbmacros/README.md) +for detailed explanation of commands and their usage. + +[TN2118]: https://developer.apple.com/library/mac/technotes/tn2004/tn2118.html#//apple_ref/doc/uid/DTS10003352 "Kernel Core Dumps" +[TN2063]: https://developer.apple.com/library/mac/technotes/tn2063/_index.html "Understanding and Debugging Kernel Panics" +[Kernel Programming Guide]: https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF diff --git a/README.md b/README.md index da6bb7b..365e650 100644 --- a/README.md +++ b/README.md @@ -1,416 +1,9 @@ -What is XNU? -=========== +# darling-xnu -XNU kernel is part of the Darwin operating system for use in macOS and iOS operating systems. XNU is an acronym for X is Not Unix. -XNU is a hybrid kernel combining the Mach kernel developed at Carnegie Mellon University with components from FreeBSD and a C++ API for writing drivers called IOKit. -XNU runs on x86_64 for both single processor and multi-processor configurations. +## Upstream URL -XNU Source Tree -=============== +https://github.com/apple-oss-distributions/xnu/tree/xnu-7195.141.2 - * `config` - configurations for exported apis for supported architecture and platform - * `SETUP` - Basic set of tools used for configuring the kernel, versioning and kextsymbol management. - * `EXTERNAL_HEADERS` - Headers sourced from other projects to avoid dependency cycles when building. These headers should be regularly synced when source is updated. - * `libkern` - C++ IOKit library code for handling of drivers and kexts. - * `libsa` - kernel bootstrap code for startup - * `libsyscall` - syscall library interface for userspace programs - * `libkdd` - source for user library for parsing kernel data like kernel chunked data. - * `makedefs` - top level rules and defines for kernel build. - * `osfmk` - Mach kernel based subsystems - * `pexpert` - Platform specific code like interrupt handling, atomics etc. - * `security` - Mandatory Access Check policy interfaces and related implementation. - * `bsd` - BSD subsystems code - * `tools` - A set of utilities for testing, debugging and profiling kernel. +## Updating Source -How to build XNU -================ - -Building `DEVELOPMENT` kernel ------------------------------ - -The xnu make system can build kernel based on `KERNEL_CONFIGS` & `ARCH_CONFIGS` variables as arguments. -Here is the syntax: - - make SDKROOT= ARCH_CONFIGS= KERNEL_CONFIGS= - -Where: - - * \: path to macOS SDK on disk. (defaults to `/`) - * \: can be `debug`, `development`, `release`, `profile` and configures compilation flags and asserts throughout kernel code. - * \ : can be valid arch to build for. (E.g. `X86_64`) - -To build a kernel for the same architecture as running OS, just type - - $ make - $ make SDKROOT=macosx.internal - -Additionally, there is support for configuring architectures through `ARCH_CONFIGS` and kernel configurations with `KERNEL_CONFIGS`. - - $ make SDKROOT=macosx.internal ARCH_CONFIGS=X86_64 KERNEL_CONFIGS=DEVELOPMENT - $ make SDKROOT=macosx.internal ARCH_CONFIGS=X86_64 KERNEL_CONFIGS="RELEASE DEVELOPMENT DEBUG" - - -Note: - * By default, architecture is set to the build machine architecture, and the default kernel - config is set to build for DEVELOPMENT. - - -This will also create a bootable image, kernel.[config], and a kernel binary -with symbols, kernel.[config].unstripped. - -To intall the kernel into a DSTROOT, use the `install_kernels` target: - - $ make install_kernels DSTROOT=/tmp/xnu-dst - -Hint: -For a more satisfying kernel debugging experience, with access to all -local variables and arguments, but without all the extra check of the -DEBUG kernel, add something like: - CFLAGS_DEVELOPMENTARM64="-O0 -g -DKERNEL_STACK_MULTIPLIER=2" - CXXFLAGS_DEVELOPMENTARM64="-O0 -g -DKERNEL_STACK_MULTIPLIER=2" -to your make command. -Replace DEVELOPMENT and ARM64 with the appropriate build and platform. - - - * To build with RELEASE kernel configuration - - make KERNEL_CONFIGS=RELEASE SDKROOT=/path/to/SDK - - -Building FAT kernel binary --------------------------- - -Define architectures in your environment or when running a make command. - - $ make ARCH_CONFIGS="X86_64" exporthdrs all - -Other makefile options ----------------------- - - * $ make MAKEJOBS=-j8 # this will use 8 processes during the build. The default is 2x the number of active CPUS. - * $ make -j8 # the standard command-line option is also accepted - * $ make -w # trace recursive make invocations. Useful in combination with VERBOSE=YES - * $ make BUILD_LTO=0 # build without LLVM Link Time Optimization - * $ make REMOTEBUILD=user@remotehost # perform build on remote host - * $ make BUILD_JSON_COMPILATION_DATABASE=1 # Build Clang JSON Compilation Database - -The XNU build system can optionally output color-formatted build output. To enable this, you can either -set the `XNU_LOGCOLORS` environment variable to `y`, or you can pass `LOGCOLORS=y` to the make command. - - -Debug information formats -========================= - -By default, a DWARF debug information repository is created during the install phase; this is a "bundle" named kernel.development.\.dSYM -To select the older STABS debug information format (where debug information is embedded in the kernel.development.unstripped image), set the BUILD_STABS environment variable. - - $ export BUILD_STABS=1 - $ make - - -Building KernelCaches -===================== - -To test the xnu kernel, you need to build a kernelcache that links the kexts and -kernel together into a single bootable image. -To build a kernelcache you can use the following mechanisms: - - * Using automatic kernelcache generation with `kextd`. - The kextd daemon keeps watching for changing in `/System/Library/Extensions` directory. - So you can setup new kernel as - - $ cp BUILD/obj/DEVELOPMENT/X86_64/kernel.development /System/Library/Kernels/ - $ touch /System/Library/Extensions - $ ps -e | grep kextd - - * Manually invoking `kextcache` to build new kernelcache. - - $ kextcache -q -z -a x86_64 -l -n -c /var/tmp/kernelcache.test -K /var/tmp/kernel.test /System/Library/Extensions - - - -Running KernelCache on Target machine -===================================== - -The development kernel and iBoot supports configuring boot arguments so that we can safely boot into test kernel and, if things go wrong, safely fall back to previously used kernelcache. -Following are the steps to get such a setup: - - 1. Create kernel cache using the kextcache command as `/kernelcache.test` - 2. Copy exiting boot configurations to alternate file - - $ cp /Library/Preferences/SystemConfiguration/com.apple.Boot.plist /next_boot.plist - - 3. Update the kernelcache and boot-args for your setup - - $ plutil -insert "Kernel Cache" -string "kernelcache.test" /next_boot.plist - $ plutil -replace "Kernel Flags" -string "debug=0x144 -v kernelsuffix=test " /next_boot.plist - - 4. Copy the new config to `/Library/Preferences/SystemConfiguration/` - - $ cp /next_boot.plist /Library/Preferences/SystemConfiguration/boot.plist - - 5. Bless the volume with new configs. - - $ sudo -n bless --mount / --setBoot --nextonly --options "config=boot" - - The `--nextonly` flag specifies that use the `boot.plist` configs only for one boot. - So if the kernel panic's you can easily power reboot and recover back to original kernel. - - - - -Creating tags and cscope -======================== - -Set up your build environment and from the top directory, run: - - $ make tags # this will build ctags and etags on a case-sensitive volume, only ctags on case-insensitive - $ make TAGS # this will build etags - $ make cscope # this will build cscope database - - -How to install a new header file from XNU -========================================= - -To install IOKit headers, see additional comments in [iokit/IOKit/Makefile](). - -XNU installs header files at the following locations - - - a. $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers - b. $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders - c. $(DSTROOT)/usr/include/ - d. $(DSTROOT)/System/DriverKit/usr/include/ - e. $(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders - -`Kernel.framework` is used by kernel extensions.\ -The `System.framework` and `/usr/include` are used by user level applications. \ -`/System/DriverKit/usr/include` is used by userspace drivers. \ -The header files in framework's `PrivateHeaders` are only available for ** Apple Internal Development **. - -The directory containing the header file should have a Makefile that -creates the list of files that should be installed at different locations. -If you are adding the first header file in a directory, you will need to -create Makefile similar to `xnu/bsd/sys/Makefile`. - -Add your header file to the correct file list depending on where you want -to install it. The default locations where the header files are installed -from each file list are - - - a. `DATAFILES` : To make header file available in user level - - `$(DSTROOT)/usr/include` - - b. `DRIVERKIT_DATAFILES` : To make header file available to DriverKit userspace drivers - - `$(DSTROOT)/System/DriverKit/usr/include` - - c. `PRIVATE_DATAFILES` : To make header file available to Apple internal in - user level - - `$(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders` - - d. `KERNELFILES` : To make header file available in kernel level - - `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers` - `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders` - - e. `PRIVATE_KERNELFILES` : To make header file available to Apple internal - for kernel extensions - - `$(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders` - -The Makefile combines the file lists mentioned above into different -install lists which are used by build system to install the header files. There -are two types of install lists: machine-dependent and machine-independent. -These lists are indicated by the presence of `MD` and `MI` in the build -setting, respectively. If your header is architecture-specific, then you should -use a machine-dependent install list (e.g. `INSTALL_MD_LIST`). If your header -should be installed for all architectures, then you should use a -machine-independent install list (e.g. `INSTALL_MI_LIST`). - -If the install list that you are interested does not exist, create it -by adding the appropriate file lists. The default install lists, its -member file lists and their default location are described below - - - a. `INSTALL_MI_LIST` : Installs header file to a location that is available to everyone in user level. - Locations - - $(DSTROOT)/usr/include - Definition - - INSTALL_MI_LIST = ${DATAFILES} - - b. `INSTALL_DRIVERKIT_MI_LIST` : Installs header file to a location that is - available to DriverKit userspace drivers. - Locations - - $(DSTROOT)/System/DriverKit/usr/include - Definition - - INSTALL_DRIVERKIT_MI_LIST = ${DRIVERKIT_DATAFILES} - - c. `INSTALL_MI_LCL_LIST` : Installs header file to a location that is available - for Apple internal in user level. - Locations - - $(DSTROOT)/System/Library/Frameworks/System.framework/PrivateHeaders - Definition - - INSTALL_MI_LCL_LIST = ${PRIVATE_DATAFILES} - - d. `INSTALL_KF_MI_LIST` : Installs header file to location that is available - to everyone for kernel extensions. - Locations - - $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers - Definition - - INSTALL_KF_MI_LIST = ${KERNELFILES} - - e. `INSTALL_KF_MI_LCL_LIST` : Installs header file to location that is - available for Apple internal for kernel extensions. - Locations - - $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders - Definition - - INSTALL_KF_MI_LCL_LIST = ${KERNELFILES} ${PRIVATE_KERNELFILES} - - f. `EXPORT_MI_LIST` : Exports header file to all of xnu (bsd/, osfmk/, etc.) - for compilation only. Does not install anything into the SDK. - Definition - - EXPORT_MI_LIST = ${KERNELFILES} ${PRIVATE_KERNELFILES} - - g. `INSTALL_MODULEMAP_INCDIR_MI_LIST` : Installs module map file to a - location that is available to everyone in user level, installing at the - root of INCDIR. - Locations - - $(DSTROOT)/usr/include - Definition - - INSTALL_MODULEMAP_INCDIR_MI_LIST = ${MODULEMAP_INCDIR_FILES} - -If you want to install the header file in a sub-directory of the paths -described in (1), specify the directory name using two variables -`INSTALL_MI_DIR` and `EXPORT_MI_DIR` as follows - - - INSTALL_MI_DIR = dirname - EXPORT_MI_DIR = dirname - -A single header file can exist at different locations using the steps -mentioned above. However it might not be desirable to make all the code -in the header file available at all the locations. For example, you -want to export a function only to kernel level but not user level. - - You can use C language's pre-processor directive (#ifdef, #endif, #ifndef) - to control the text generated before a header file is installed. The kernel - only includes the code if the conditional macro is TRUE and strips out - code for FALSE conditions from the header file. - - Some pre-defined macros and their descriptions are - - - a. `PRIVATE` : If defined, enclosed definitions are considered System - Private Interfaces. These are visible within xnu and - exposed in user/kernel headers installed within the AppleInternal - "PrivateHeaders" sections of the System and Kernel frameworks. - b. `KERNEL_PRIVATE` : If defined, enclosed code is available to all of xnu - kernel and Apple internal kernel extensions and omitted from user - headers. - c. `BSD_KERNEL_PRIVATE` : If defined, enclosed code is visible exclusively - within the xnu/bsd module. - d. `MACH_KERNEL_PRIVATE`: If defined, enclosed code is visible exclusively - within the xnu/osfmk module. - e. `XNU_KERNEL_PRIVATE`: If defined, enclosed code is visible exclusively - within xnu. - f. `KERNEL` : If defined, enclosed code is available within xnu and kernel - extensions and is not visible in user level header files. Only the - header files installed in following paths will have the code - - - $(DSTROOT)/System/Library/Frameworks/Kernel.framework/Headers - $(DSTROOT)/System/Library/Frameworks/Kernel.framework/PrivateHeaders - g. `DRIVERKIT`: If defined, enclosed code is visible exclusively in the - DriverKit SDK headers used by userspace drivers. - -Conditional compilation -======================= - -`xnu` offers the following mechanisms for conditionally compiling code: - - a. *CPU Characteristics* If the code you are guarding has specific - characterstics that will vary only based on the CPU architecture being - targeted, use this option. Prefer checking for features of the - architecture (e.g. `__LP64__`, `__LITTLE_ENDIAN__`, etc.). - b. *New Features* If the code you are guarding, when taken together, - implements a feature, you should define a new feature in `config/MASTER` - and use the resulting `CONFIG` preprocessor token (e.g. for a feature - named `config_virtual_memory`, check for `#if CONFIG_VIRTUAL_MEMORY`). - This practice ensures that existing features may be brought to other - platforms by simply changing a feature switch. - c. *Existing Features* You can use existing features if your code is - strongly tied to them (e.g. use `SECURE_KERNEL` if your code implements - new functionality that is exclusively relevant to the trusted kernel and - updates the definition/understanding of what being a trusted kernel means). - -It is recommended that you avoid compiling based on the target platform. `xnu` -does not define the platform macros from `TargetConditionals.h` -(`TARGET_OS_OSX`, `TARGET_OS_IOS`, etc.). - - -There is a deprecated `TARGET_OS_EMBEDDED` macro, but this should be avoided -as it is in general too broad a definition for most functionality. -Please refer to TargetConditionals.h for a full picture. - -How to add a new syscall -======================== - - - - -Testing the kernel -================== - -XNU kernel has multiple mechanisms for testing. - - * Assertions - The DEVELOPMENT and DEBUG kernel configs are compiled with assertions enabled. This allows developers to easily - test invariants and conditions. - - * XNU Power On Self Tests (`XNUPOST`): The XNUPOST config allows for building the kernel with basic set of test functions - that are run before first user space process is launched. Since XNU is hybrid between MACH and BSD, we have two locations where - tests can be added. - - xnu/osfmk/tests/ # For testing mach based kernel structures and apis. - bsd/tests/ # For testing BSD interfaces. - Please follow the documentation at [osfmk/tests/README.md](osfmk/tests/README.md) - - * User level tests: The `tools/tests/` directory holds all the tests that verify syscalls and other features of the xnu kernel. - The make target `xnu_tests` can be used to build all the tests supported. - - $ make RC_ProjectName=xnu_tests SDKROOT=/path/to/SDK - - These tests are individual programs that can be run from Terminal and report tests status by means of std posix exit codes (0 -> success) and/or stdout. - Please read detailed documentation in [tools/tests/unit_tests/README.md](tools/tests/unit_tests/README.md) - - -Kernel data descriptors -======================= - -XNU uses different data formats for passing data in its api. The most standard way is using syscall arguments. But for complex data -it often relies of sending memory saved by C structs. This packaged data transport mechanism is fragile and leads to broken interfaces -between user space programs and kernel apis. `libkdd` directory holds user space library that can parse custom data provided by the -same version of kernel. The kernel chunked data format is described in detail at [libkdd/README.md](libkdd/README.md). - - -Debugging the kernel -==================== - -The xnu kernel supports debugging with a remote kernel debugging protocol (kdp). Please refer documentation at [technical note] [TN2063] -By default the kernel is setup to reboot on a panic. To debug a live kernel, the kdp server is setup to listen for UDP connections -over ethernet. For machines without ethernet port, this behavior can be altered with use of kernel boot-args. Following are some -common options. - - * `debug=0x144` - setups debug variables to start kdp debugserver on panic - * `-v` - print kernel logs on screen. By default XNU only shows grey screen with boot art. - * `kdp_match_name=en1` - Override default port selection for kdp. Supported for ethernet, thunderbolt and serial debugging. - -To debug a panic'ed kernel, use llvm debugger (lldb) along with unstripped symbol rich kernel binary. - - sh$ lldb kernel.development.unstripped - -And then you can connect to panic'ed machine with `kdp_remote [ip addr]` or `gdb_remote [hostip : port]` commands. - -Each kernel is packaged with kernel specific debug scripts as part of the build process. For security reasons these special commands -and scripts do not get loaded automatically when lldb is connected to machine. Please add the following setting to your `~/.lldbinit` -if you wish to always load these macros. - - settings set target.load-script-from-symbol-file true - -The `tools/lldbmacros` directory contains the source for each of these commands. Please follow the [README.md](tools/lldbmacros/README.md) -for detailed explanation of commands and their usage. - -[TN2118]: https://developer.apple.com/library/mac/technotes/tn2004/tn2118.html#//apple_ref/doc/uid/DTS10003352 "Kernel Core Dumps" -[TN2063]: https://developer.apple.com/library/mac/technotes/tn2063/_index.html "Understanding and Debugging Kernel Panics" -[Kernel Programming Guide]: https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF +When updating the source code, make sure to refer to the [`Updating Sources`](https://docs.darlinghq.org/contributing/updating-sources/index.html#updating-sources) section from the Darling Docs website. Additional details (if any) can be found in [`darling/notes/UPDATE_SOURCE.md`](darling/notes/UPDATE_SOURCE.md). \ No newline at end of file diff --git a/darling/notes/UPDATE_SOURCE.md b/darling/notes/UPDATE_SOURCE.md new file mode 100644 index 0000000..1a41956 --- /dev/null +++ b/darling/notes/UPDATE_SOURCE.md @@ -0,0 +1,18 @@ +# Differences Between Darlingserver's XNU Source And This Repo + +Darlingserver contains a striped down version of the XNU source that is only used for darlingserver. In comparison, this repo contains a full copy of the XNU source that can use for any purpose. + +This implies the following: +* All non-darlingserver XNU source changes or modifications should be made to this repo (not darlingserver). +* If the Darling SDK need any additional XNU header, the symlinks should point to this repo. + +# `libsyscall` Symlinks + +When updating XNU source, make sure to also restore the following symlinks: + +``` +# cd /path/to/darling-xnu +ln -s -t libsyscall/Platforms/MacOSX ../../../darling/src/libsystem_kernel/extra/Platforms/MacOSX/ppc +ln -s -t libsyscall ../gen/bsdsyscalls +ln -s -t libsyscall ../darling/src/libsystem_kernel/extra/nolegacy.h +``` \ No newline at end of file