Update Source To dyld-852.2

IMPCaches.md

@@ -0,0 +1,166 @@
+# IMP caches generation
+
+## Principle
+
+When you call an Objective-C method, Objective-C looks for the actual IMP (function pointer) given the class and selector. It then
+stores, in malloced memory, this [selector, IMP] pair in a hash table. 
+
+These hash tables are:
+
+* imperfect: the hash function is just a mask applied to the selector's
+address, so if two selectors have the same low bits, you will get a collision.
+Collisions are resolved through linear probing
+* expensive: the footprint of these hash tables is often around 70MB total on a carry device.
+
+We want to replace this with static hash tables, inside the shared cache, where you have:
+
+* a perfect hashing function
+* and the tables are in clean memory
+
+## Hash function
+
+Because the hash function is executed as part of `objc_msgSend`, it has to be blazingly fast. The algorithm below explains how we can
+make a hash function of the form `h(x) = (x >> shift) & mask` work, where `shift` and `mask` are two class-specific parameters.
+
+Note that we cannot use traditional perfect hash tables techniques as:
+
+* they require at least [1.44 bits of information per key](https://arxiv.org/pdf/1910.06416.pdf). 
+* they often require multiple lookups (two-level hashing).
+
+The idea is that because the input of the hash function is a selector's address, we have some control over the input... because the
+dyld shared cache builder is the one placing all the selectors.
+
+So now the problem becomes : given a set of classes and the selectors they implement, how do you place the selectors, and for each
+class, how do you find a shift and mask so that the hash table generated by (selector >> shift) & mask is perfect?
+
+(Note that the shift + mask idiom lets us use various "bit stripes" of the selector's address for various hash tables).
+
+## Algorithm
+
+There are basically two steps to the main algorithm:
+
+### Finding shifts and masks
+
+Assign some of the high bits of the selector's address, and find a shift and a mask for each class. This is a backtracking
+algorithm which goes through each class, one after another. As it goes through classes, it finds a shift and a mask compatible
+with the bit ranges that are already set on each selector's address, and assigns the corresponding bits of the selector's address.
+Note that because addresses are partial at this point (some of the bits are unset) it's very difficult to check for collisions,
+so selectors will end up at the same address.
+
+At each step of the algorithm, we go through a set of possible shifts and masks until we find one which works. If none work, we let
+the hash table grow to one more bit to make our job slightly easier. In practice few hash tables grow one more bit (82 out of 18k).
+If we cannot find a suitable shift and mask after backtracking a few times, we'll also allow ourselves to drop a class from the set
+and not generate an IMP cache for that one. This happens for a dozen classes or so with the current data set.
+
+Next we have to deal with collisions, and constraints on the addresses themselves because each selector is... a `char*`, which has
+a length. You cannot have an overlap between two selectors. So the idea here is to try to get rid of this constraint by assigning
+in step 1 just the address of a "bucket" which is 128 bytes long (7 bits). So step 1 will assign the high bits of the address
+(which will be the address of the bucket) and we can then place the selectors linearly in the buckets.
+
+### Shuffling selectors around
+
+Then you have to deal with address collisions. If the lengths of selectors in a given bucket add up to more than 128, you have
+to move some selectors out. So step 2 goes through all the selectors, checks if it fits within the bucket it's supposed to be in,
+and if it doesn't finds another suitable bucket.
+
+To do so, it iterates through all the classes the selector is in, and builds a "constraint set" applying to that selector's
+bucket's address (by basically looking at which slots in each hash table are free and combining all these constraints, which 
+impact a different "stripe" of the address due to the shifts and masks). Once we have the constraint set, we can find a different
+bucket for the selector without changing the shifts and masks. (If there is no other possible bucket, we allow ourselves to drop the
+classes involving this selector, too).
+
+Once each selector has a valid bucket, you can simply assign the low 7 bits of each address by looking at which selectors are in 
+each bucket and looking at their lengths.
+
+The problem is hard but not that hard given the number of classes we are targeting with this optimization:
+we are roughly targeting ~ 20k classes out of 120k and ~ 200k selectors out of 900k, so we have lots of "free space".
+
+Note that any holes we leave in the bucketization of the selectors can be filled later by placing all the selectors not targeted
+by the optimization and any selectors from OS executables inside them.
+
+## Shared cache builder setup
+
+The optimization is guided by a file dropped by the OrderFiles project (`/AppleInternal/OrderFiles/shared-cache-objc-optimizations.json`).
+The performance team is responsible for updating this file by looking at the caches on live or StressCycler devices with `objcdt dump-imp-caches`.
+
+This file specifies a list of classes and metaclasses the algorithm targets, and a list of flattening roots.
+
+We haven't explained flattening yet. When a class D(aughter) inherits from a class M(other), we should in theory add all of M's methods
+to D's IMP cache. However
+
+* This constrains the problem too much. The solver's job will be too difficult if the same selectors are in thousands of classes.
+* This makes the IMP cache sizes blow up. 
+
+So our scheme is to target only classes which are leaves in the inheritance tree with this optimization. For any selector that comes from
+parent classes, the cache lookup will fail and fall back to looking for the method in `super`. Because `super` is not a leaf class,
+it will have a dynamically allocated IMP cache and can cache there any selector that comes from the parents.
+
+However, this means that some very interesting classes from a memory standpoint (because we find their caches in many processes)
+get excluded because they have child classes. A solution to this is to turn on selector inheritance (add Mother's selectors
+to Daughter's cache) starting at some flattening root. Then the IMP cache will have a "fallback class" that is the superclass
+of the flattening root, and `objc_msgSend` will fallback to that class if it cannot find the selector in the child cache, 
+skipping over all the classes up to the flattening root (because we know all the selectors in that chain will be present
+in the child cache).
+
+Very early into the shared cache builder's life, the algorithm described above runs. To do so, it parses all of the source dylibs
+to find out which methods will end up in which class's cache (see section "what ends up in each cache" below).
+The output of the algorithm is:
+
+- for each class:
+  * a shift and a mask
+  * a list of methods that will end up in the cache with (source install name, source class name, source category name)
+- for each selector: an offset at which it needs to be placed relative to the beginning of the selectors section
+- a list of holes in the selector address space (or more accurately offset space) that can be used to add additional selectors
+    not targeted by the algorithm
+  
+Then, we do all the selector deduping work, and when we get to the ObjC optimizer:
+
+- we go through all the classes again to build a map from (source install name, source class name, source category name) to the 
+    actual IMP's address (which is not known before the shared cache is laid out)
+- we go through all the IMP caches returned by the algorithm, find the corresponding IMP, and emit the actual IMP cache in the 
+    objc optimizer's RO region.
+
+## Some code pointers
+
+Most of the logic lives in the single C++ file `IMPCaches.cpp` and the two headers `IMPCaches.hpp` and `IMPCachesBuilder.hpp`. A tour:
+
+### Types
+
+`IMPCaches::Selector` : A unique selector. Has a name, a list of classes it's used in, and an address (which is actually an offset from
+the beginning of the selectors section). Due to how the placement algorithm work, it also has a current
+partial address and the corresponding bitmask of fixed bits. The algorithm adds bits to the partial address
+as it makes progress and updates the mask accordingly
+
+`IMPCaches::AddressSpace` : a representation of the address space available in the selectors section, so that step 2 of the algorithm
+can check if selector buckets are overflowing.
+
+`IMPCaches::HoleMap` : represents the holes left by the selector placement algorithm, to be filled later with other selectors we did not target.
+
+`IMPCaches::Constraint` : represents a constraint on some of the bits of an address. It stores a set of allowed values for a given range of bits (shift and mask)
+
+`IMPCachesBuilder` : what actually computes the caches (part of the algorithm ; what actually lays down the caches lives in the
+ObjC optimizer)
+
+### Entry points
+
+* `IMPCachesBuilder::parseDylibs` : parses the source dylibs to figure out which method ends up in which cache.
+
+* `IMPCachesBuilder::findShiftsAndMasks` : finds the shift and mask for each class (see "Findings shifts and masks" above).
+
+* `IMPCachesBuilder::solveGivenShiftsAndMasks` : moves selectors around to make sure each bucket only contains 128 bytes worth of selectors.
+
+Then in `OptimizerObjC`:
+
+* `IMPMapBuilder` : Builds a map from (install name, class name, method name) to actual IMPs
+
+* `IMPCachesEmitter` : emits the actual IMP caches in the shared cache.
+
+### What ends up in each cache?
+
+`IMPCachesBuilder::parseDylibs` goes through all the classes and categories to decide, for each (class,selector) pair, which implementation we will actually call at runtime.
+The sequence is:
+
+* Get all the methods from the method lists
+* Attach any methods from non-cross-image categories (when we have cross-image categories, we prevent the class from getting an IMP cache). If there is any collision at this point we'll
+use the implementation from the main class (or from the first category we found).
+* Then we may inline some of the implementations from superclasses, see the explanation on flattening above.

bin/expand.rb

@@ -1,102 +0,0 @@
-#!/usr/bin/env ruby
-
-require 'yaml'
-
-if ENV["DRIVERKITROOT"]
-    $availCmd = ENV["SDKROOT"] + ENV["DRIVERKITROOT"] + "/usr/local/libexec/availability.pl";
-else
-    $availCmd = ENV["SDKROOT"] + "/usr/local/libexec/availability.pl";
-end
-
-def versionString(vers)
-    uvers = ""
-    if vers =~ /^(\d+)$/
-        uvers = "#{$1}_0"
-    elsif vers =~ /^(\d+).(\d+)$/
-        uvers = "#{$1}_#{$2}"
-    elsif vers =~ /^(\d+).(\d+).(\d+)$/
-        if $3 == 0
-            uvers = "#{$1}_#{$2}"
-        else
-            uvers = "#{$1}_#{$2}_#{$3}"
-        end
-    end
-    uvers
-end
-
-def versionHex(vers)
-    major = 0;
-    minor = 0;
-    revision = 0;
-
-    if vers =~ /^(\d+)$/
-        major = $1.to_i;
-    elsif vers =~ /^(\d+).(\d+)$/
-        major = $1.to_i;
-        minor = $2.to_i;
-    elsif vers =~ /^(\d+).(\d+).(\d+)$/
-        major = $1.to_i;
-        minor = $2.to_i;
-        revision = $3.to_i;
-    end
-    "0x00#{major.to_s(16).rjust(2, '0')}#{minor.to_s(16).rjust(2, '0')}#{revision.to_s(16).rjust(2, '0')}"
-end
-
-def expandVersions(prefix, arg)
-    versionList = `#{$availCmd} #{arg}`.gsub(/\s+/m, ' ').strip.split(" ")
-    versionList.each { |version|
-        puts "#define #{prefix}#{versionString(version)}".ljust(48, ' ') + versionHex(version)
-    }
-end
-
-def expandPlatformVersions(prefix, platform, arg)
-    versionList = `#{$availCmd} #{arg}`.gsub(/\s+/m, ' ').strip.split(" ")
-    versionList.each { |version|
-        puts "static dyld_build_version_t dyld_platform_version_#{prefix}_#{versionString(version)}".ljust(72, ' ') + "= { .platform = #{platform}, .version = #{versionHex(version)} };"
-    }
-end
-
-def versionSetsForOSes(versionSets, key, platform, target)
-    puts "#if #{target}"
-    versionSets.each { |k,v|
-        puts "#define dyld_#{k}_os_versions dyld_platform_version_#{platform}_#{versionString(v[key].to_s)}"
-    }
-    puts "#endif /* #{target} */"
-end
-
-def expandVersionSets()
-    versionSets = YAML.load(`#{$availCmd} --sets`)
-    versionSetsForOSes(versionSets, "macos", "macOS", "TARGET_OS_OSX")
-    versionSetsForOSes(versionSets, "ios", "iOS", "TARGET_OS_IOS")
-    versionSetsForOSes(versionSets, "tvos", "tvOS", "TARGET_OS_TV")
-    versionSetsForOSes(versionSets, "watchos", "watchOS", "TARGET_OS_WATCH")
-    versionSetsForOSes(versionSets, "bridgeos", "bridgeOS", "TARGET_OS_BRIDGE")
-end
-
-ARGF.each do |line|
-    if line =~ /^\/\/\@MAC_VERSION_DEFS\@$/
-        expandVersions("DYLD_MACOSX_VERSION_", "--macosx")
-    elsif line =~ /^\/\/\@IOS_VERSION_DEFS\@$/
-        expandVersions("DYLD_IOS_VERSION_", "--ios")
-    elsif line =~ /^\/\/\@WATCHOS_VERSION_DEFS\@$/
-        expandVersions("DYLD_WATCHOS_VERSION_", "--watchos")
-    elsif line =~ /^\/\/\@TVOS_VERSION_DEFS\@$/
-        expandVersions("DYLD_TVOS_VERSION_", "--appletvos")
-    elsif line =~ /^\/\/\@BRIDGEOS_VERSION_DEFS\@$/
-        expandVersions("DYLD_BRIDGEOS_VERSION_", "--bridgeos")
-    elsif line =~ /^\/\/\@MACOS_PLATFORM_VERSION_DEFS\@$/
-        expandPlatformVersions("macOS", "PLATFORM_MACOS", "--macosx")
-    elsif line =~ /^\/\/\@IOS_PLATFORM_VERSION_DEFS\@$/
-        expandPlatformVersions("iOS", "PLATFORM_IOS", "--ios")
-    elsif line =~ /^\/\/\@WATCHOS_PLATFORM_VERSION_DEFS\@$/
-        expandPlatformVersions("watchOS", "PLATFORM_WATCHOS", "--watchos")
-    elsif line =~ /^\/\/\@TVOS_PLATFORM_VERSION_DEFS\@$/
-        expandPlatformVersions("tvOS", "PLATFORM_TVOS", "--appletvos")
-    elsif line =~ /^\/\/\@BRIDGEOS_PLATFORM_VERSION_DEFS\@$/
-        expandPlatformVersions("bridgeOS", "PLATFORM_BRIDGEOS", "--bridgeos")
-    elsif line =~ /^\/\/\@VERSION_SET_DEFS\@$/
-        expandVersionSets()
-    else
-        puts line
-    end
-end

build-scripts/ContainerizedTestRunner-build-everything.sh

@@ -0,0 +1,9 @@
+#!/bin/sh
+
+source $SRCROOT/build-scripts/include.sh
+
+env -i PATH="${PATH}" xcodebuild install -sdk ${SDKROOT} -configuration ${CONFIGURATION} -target dyld -target libdyld -target dyld_tests -target dyld_usage TOOLCHAINS="${TOOLCHAINS}" DSTROOT=${DERIVED_FILES_DIR}/TestRoot OBJROOT=${DERIVED_FILES_DIR}/objroot DISABLE_SDK_METADATA_PARSING=YES XCTestGenPath=${DERIVED_FILES_DIR}/XCTestGenerated.h GCC_PREPROCESSOR_DEFINITIONS='$GCC_PREPROCESSOR_DEFINITIONS BUILD_FOR_TESTING=1' || exit_if_error $? "Build failed"
+${BUILT_PRODUCTS_DIR}/chroot_util -chroot ${DERIVED_FILES_DIR}/TestRoot -fallback / -add_file /bin/echo -add_file /bin/sh -add_file /bin/bash -add_file /bin/ls -add_file /usr/sbin/dtrace -add_file /sbin/mount -add_file /sbin/mount_devfs -add_file /usr/lib/libobjc-trampolines.dylib -add_file /usr/bin/leaks -add_file /System/iOSSupport/System/Library/Frameworks/UIKit.framework/UIKit || exit_if_error $? "Chroot build failed"
+/bin/mkdir -p ${DERIVED_FILES_DIR}/TestRoot/dev
+/bin/mkdir -m 777 -p ${DERIVED_FILES_DIR}/TestRoot/tmp
+

build-scripts/configure-dyld-archives.sh

@@ -0,0 +1,15 @@
+# link with all .a files in /usr/local/lib/dyld
+ls -1 ${SDKROOT}/usr/local/lib/dyld/*.a | grep -v libcompiler_rt > ${DERIVED_SOURCES_DIR}/archives.txt
+
+# link with crash report archive if it exists
+if [ -f ${SDKROOT}/usr/local/lib/libCrashReporterClient.a ]
+then
+  echo \"${SDKROOT}/usr/local/lib/libCrashReporterClient.a\" >> ${DERIVED_SOURCES_DIR}/archives.txt
+fi
+
+# link with crypto archive if it exists
+if [ -f ${SDKROOT}/usr/local/lib/libcorecrypto_static.a ]
+then
+  echo \"${SDKROOT}/usr/local/lib/libcorecrypto_static.a\" >> ${DERIVED_SOURCES_DIR}/archives.txt
+fi
+

build-scripts/dyld_tests-build.sh

@@ -0,0 +1,79 @@
+#!/bin/sh
+
+set -e
+
+source $SRCROOT/build-scripts/include.sh
+
+# Exit on failure
+
+OBJROOT_DYLD_APP_CACHE_UTIL="${TARGET_TEMP_DIR}/Objects_Dyld_App_Cache_Util"
+OBJROOT_RUN_STATIC="${TARGET_TEMP_DIR}/Objects_Run_Static"
+
+SYMROOT=${BUILD_DIR}/${CONFIGURATION}${EFFECTIVE_PLATFORM_NAME}/dyld_tests
+OBJROOT=${PROJECT_TEMP_DIR}/${CONFIGURATION}${EFFECTIVE_PLATFORM_NAME}
+SDKROOT=${SDKROOT:-$(xcrun -sdk macosx.internal --show-sdk-path)}
+DEPLOYMENT_TARGET_CLANG_FLAG_NAME=${DEPLOYMENT_TARGET_CLANG_FLAG_NAME:-"mmacosx-version-min"}
+DERIVED_FILES_DIR=${DERIVED_FILES_DIR}
+LDFLAGS="-L$BUILT_PRODUCTS_DIR"
+#LLBUILD=$(xcrun --sdk $SDKROOT --find llbuild 2> /dev/null)
+NINJA=${LLBUILD:-`xcrun  --sdk $SDKROOT --find ninja  2> /dev/null`}
+BUILD_TARGET=${ONLY_BUILD_TEST:-all}
+
+if [ ! -z "$LLBUILD" ]; then
+  NINJA="$LLBUILD ninja build"
+fi
+
+OSVERSION="10.14"
+if [ ! -z "$DEPLOYMENT_TARGET_CLANG_ENV_NAME" ]; then
+    OSVERSION=${!DEPLOYMENT_TARGET_CLANG_ENV_NAME}
+fi
+
+if [ -z "$SRCROOT" ]; then
+    echo "Error $$SRCROOT must be set"
+fi
+
+if [ -z "$ARCHS" ]; then
+    PLATFORM_NAME=${PLATFORM_NAME:macosx}
+    case "$PLATFORM_NAME" in
+       "watchos")   ARCHS="armv7k arm64_32"
+       ;;
+       "appletvos") ARCHS="arm64"
+       ;;
+       *)    ARCHS=${ARCHS_STANDARD}
+       ;;
+    esac
+fi
+
+if [ -z "$ARCHS" ]; then
+    ARCHS="x86_64"
+fi
+
+/bin/mkdir -p ${DERIVED_FILES_DIR}
+TMPFILE=$(mktemp ${DERIVED_FILES_DIR}/config.ninja.XXXXXX)
+
+echo "OBJROOT = $OBJROOT" >> $TMPFILE
+echo "OSFLAG = $DEPLOYMENT_TARGET_CLANG_FLAG_NAME" >> $TMPFILE
+echo "OSVERSION = $OSVERSION" >> $TMPFILE
+echo "SDKROOT = $SDKROOT" >> $TMPFILE
+echo "SRCROOT = $SRCROOT" >> $TMPFILE
+echo "SYMROOT = $SYMROOT" >> $TMPFILE
+echo "BUILT_PRODUCTS_DIR = $BUILT_PRODUCTS_DIR" >> $TMPFILE
+echo "INSTALL_GROUP = $INSTALL_GROUP" >> $TMPFILE
+echo "INSTALL_MODE_FLAG = $INSTALL_MODE_FLAG" >> $TMPFILE
+echo "INSTALL_OWNER = $INSTALL_OWNER" >> $TMPFILE
+echo "INSTALL_DIR = $INSTALL_DIR" >> $TMPFILE
+echo "USER_HEADER_SEARCH_PATHS = $USER_HEADER_SEARCH_PATHS" >> $TMPFILE
+echo "SYSTEM_HEADER_SEARCH_PATHS = $SYSTEM_HEADER_SEARCH_PATHS" >> $TMPFILE
+echo "ARCHS = $ARCHS" >> $TMPFILE
+echo "DERIVED_FILES_DIR = $DERIVED_FILES_DIR" >> $TMPFILE
+echo "LDFLAGS = $LDFLAGS" >> $TMPFILE
+
+/usr/bin/rsync -vc $TMPFILE ${DERIVED_FILES_DIR}/config.ninja
+/bin/rm -f $TMPFILE
+
+xcodebuild install -target run-static SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_RUN_STATIC}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+
+xcodebuild install -target dyld_app_cache_util -sdk macosx.internal -configuration ${CONFIGURATION} MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_DYLD_APP_CACHE_UTIL}" SRCROOT="${SRCROOT}" DSTROOT="${BUILT_PRODUCTS_DIR}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" INSTALL_PATH="/host_tools" RC_ARCHS="${NATIVE_ARCH_ACTUAL}" DISABLE_SDK_METADATA_PARSING=YES
+
+${SRCROOT}/testing/build_ninja.py ${DERIVED_FILES_DIR}/config.ninja || exit_if_error $? "Generating build.ninja failed"
+${NINJA} -C ${DERIVED_FILES_DIR} ${BUILD_TARGET} || exit_if_error $? "Ninja build failed"

build-scripts/dyld_tests-install.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+
+
+#LLBUILD=$(xcrun --sdk $SDKROOT --find llbuild 2> /dev/null)
+NINJA=${LLBUILD:-`xcrun  --sdk $SDKROOT --find ninja  2> /dev/null`}
+INSTALL_TARGET="install"
+
+if [ ! -z "$LLBUILD" ]; then
+  NINJA="$LLBUILD ninja build"
+fi
+
+if [ ! -z "$ONLY_BUILD_TEST" ]; then
+    INSTALL_TARGET="install-$BUILD_ONLY"
+fi
+
+${NINJA} -C ${DERIVED_FILES_DIR} ${INSTALL_TARGET} || exit_if_error $? "Ninja install failed"

build-scripts/generate-cache-config-header.sh

@@ -0,0 +1,53 @@
+#!/bin/sh
+
+/bin/echo "" > ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+if [ -z "${ARM_SDK}" ]; then
+    # if iOS SDK not available, use MacOSX SDK
+    ARM_SDK=`xcrun -sdk macosx.internal --show-sdk-path`
+fi
+
+SHARED_REGION_FILE="${ARM_SDK}/usr/include/mach/shared_region.h"
+
+
+if [ -r "${SHARED_REGION_FILE}" ]; then
+    /bin/echo -n "#define ARM_SHARED_REGION_START "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    awk '/define SHARED_REGION_BASE_ARM[ \t]/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+    /bin/echo -n "#define ARM_SHARED_REGION_SIZE "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    awk '/define SHARED_REGION_SIZE_ARM[ \t]/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+    /bin/echo -n "#define ARM64_SHARED_REGION_START "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    awk '/define SHARED_REGION_BASE_ARM64[ \t]/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+    /bin/echo -n "#define ARM64_SHARED_REGION_SIZE "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    awk '/define SHARED_REGION_SIZE_ARM64[ \t]/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+    grep SHARED_REGION_BASE_ARM64_32 "${SHARED_REGION_FILE}" > /dev/null 2>&1
+    if [ "$?" -eq "0" ]; then
+        /bin/echo -n "#define ARM64_32_SHARED_REGION_START "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+        awk '/define SHARED_REGION_BASE_ARM64_32/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+        /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+
+        /bin/echo -n "#define ARM64_32_SHARED_REGION_SIZE "  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+        awk '/define SHARED_REGION_SIZE_ARM64_32/ { print $3;}' "${SHARED_REGION_FILE}" >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+        /bin/echo ""  >> ${DERIVED_FILE_DIR}/dyld_cache_config.h
+    fi
+else
+    /bin/echo "ERROR: File needed to configure update_dyld_shared_cache does not exist '${SHARED_REGION_FILE}'"
+    exit 1
+fi
+
+if [ -r "${ARM_SDK}/AppleInternal/DirtyDataFiles/dirty-data-segments-order.txt" ]; then
+    mkdir -p "${DSTROOT}/${INSTALL_LOCATION}/usr/local/bin"
+    cp "${ARM_SDK}/AppleInternal/DirtyDataFiles/dirty-data-segments-order.txt"  "${DSTROOT}/${INSTALL_LOCATION}/usr/local/bin"
+fi
+if [ -r "${ARM_SDK}/AppleInternal/OrderFiles/dylib-order.txt" ]; then
+    mkdir -p "${DSTROOT}/${INSTALL_LOCATION}/usr/local/bin"
+    cp "${ARM_SDK}/AppleInternal/OrderFiles/dylib-order.txt"  "${DSTROOT}/${INSTALL_LOCATION}/usr/local/bin"
+fi
+

build-scripts/include.sh

@@ -0,0 +1,10 @@
+exit_if_error() {
+  local exit_code=$1
+  shift
+  [[ $exit_code ]] &&               # do nothing if no error code passed
+    ((exit_code != 0)) && {         # do nothing if error code is 0
+      printf 'ERROR: %s\n' "$@" >&2 # we can use better logging here
+      exit "$exit_code"             # we could also check to make sure
+                                    # error code is numeric when passed
+    }
+}

build-scripts/libdyld-generate-version-headers.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+
+if [ "${DRIVERKIT}" = 1 ]; then
+    RUNTIME_PREFIX="/System/DriverKit/Runtime"
+else
+    RUNTIME_PREFIX=""
+fi
+
+/bin/mkdir -p ${DERIVED_FILES_DIR}
+/bin/mkdir -p ${DSTROOT}${RUNTIME_PREFIX}/usr/local/include/mach-o/
+
+VERSIONS=${SDKROOT}${RUNTIME_PREFIX}/usr/local/include/dyld/for_dyld_priv.inc
+DYLD_PRIV_IN=${SRCROOT}/include/mach-o/dyld_priv.h
+DYLD_PRIV_OUT=${DSTROOT}${RUNTIME_PREFIX}/usr/local/include/mach-o/dyld_priv.h
+TMPFILE=$(mktemp ${DERIVED_FILES_DIR}/dyld_priv.h.XXXXXX)
+
+/bin/chmod 0644 $TMPFILE
+
+while IFS="" read -r p || [ -n "$p" ]
+do
+  case "$p" in
+    *@VERSION_DEFS* ) cat "$VERSIONS" >> $TMPFILE ;;
+    *               ) echo "$p" >> $TMPFILE ;;
+  esac
+done < $DYLD_PRIV_IN
+
+/usr/bin/rsync -vc $TMPFILE $DYLD_PRIV_OUT
+/bin/rm -f $TMPFILE

build-scripts/update_dyld_shared_cache-build.sh

@@ -0,0 +1,43 @@
+
+if [ "${ACTION}" = "install" ]
+then
+    OBJROOT_LOCAL="${TARGET_TEMP_DIR}/Objects_Local"
+    xcodebuild install -target dyld_shared_cache_builder SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_LOCAL}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+
+    # On macOS, also install dyld_shared_cache_builder to the platform so that root_util can find it.
+    if [ "${RC_PURPLE}" = "" ]
+    then
+      if [ "${PLATFORM_DIR}" != "" ]
+      then
+        # Note this is set to something like DEVELOPER_INSTALL_DIR=/Applications/Xcode.app/Contents/Developer
+        mkdir -p ${DSTROOT}/${DEVELOPER_INSTALL_DIR}/Platforms/MacOSX.platform/usr/local/bin/
+        cp ${DSTROOT}/usr/local/bin/dyld_shared_cache_builder ${DSTROOT}/${DEVELOPER_INSTALL_DIR}/Platforms/MacOSX.platform/usr/local/bin/dyld_shared_cache_builder
+      fi
+    fi
+
+    if [ "${RC_PURPLE}" = "YES" ]
+    then
+        OBJROOT_UTILS="${TARGET_TEMP_DIR}/Objects_Utils"
+        xcodebuild install -target dyld_closure_util -target dyld_shared_cache_util SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_UTILS}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+        if [ "${RC_BRIDGE}" != "YES" ]
+        then
+            OBJROOT_SIM="${TARGET_TEMP_DIR}/Objects_Sim"
+            xcodebuild install -target update_dyld_sim_shared_cache SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_SIM}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+        fi
+    else
+        OBJROOT_MAC="${TARGET_TEMP_DIR}/Objects_Mac"
+        xcodebuild install -target update_dyld_shared_cache_tool SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_MAC}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+        OBJROOT_MAC="${TARGET_TEMP_DIR}/Objects2_Mac"
+        xcodebuild install -target update_dyld_shared_cache_root_mode_tool SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_MAC}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" DISABLE_SDK_METADATA_PARSING=YES
+    fi
+fi
+
+# On macOS build the kernel linker in to /usr/lib too.  It defaults to the toolchain
+if [ "${ACTION}" != "installhdrs" ]
+then
+  if [ "${RC_PURPLE}" = "" ]
+  then
+    OBJROOT_MAC="${TARGET_TEMP_DIR}/Objects_Linker_Mac"
+    xcodebuild ${ACTION} -target libKernelCollectionBuilder SDKROOT="${SDKROOT}" MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET} OBJROOT="${OBJROOT_MAC}" SRCROOT="${SRCROOT}" DSTROOT="${DSTROOT}" SYMROOT="${SYMROOT}/usr/lib" RC_ProjectSourceVersion="${RC_ProjectSourceVersion}" LD_DYLIB_INSTALL_NAME="/usr/lib/libKernelCollectionBuilder.dylib" INSTALL_PATH="/usr/lib"
+  fi
+fi

chroot_util.cpp

@@ -0,0 +1,210 @@
+/*
+* Copyright (c) 2019 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* "Portions Copyright (c) 1999 Apple Computer, Inc.  All Rights
+* Reserved.  This file contains Original Code and/or Modifications of
+* Original Code as defined in and that are subject to the Apple Public
+* Source License Version 1.0 (the 'License').  You may not use this file
+* except in compliance with the License.  Please obtain a copy of the
+* License at http://www.apple.com/publicsource and read it before using
+* this file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT.  Please see the
+* License for the specific language governing rights and limitations
+* under the License."
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
+
+#include <cassert>
+#include <cstdio>
+#include <cstring>
+
+#include <fcntl.h>
+#include <dirent.h>
+#include <libgen.h>
+#include <unistd.h>
+#include <sys/attr.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/param.h>
+#include <mach-o/fat.h>
+#include <mach-o/loader.h>
+#include <copyfile.h>
+
+#include <set>
+#include <string>
+#include <vector>
+#include <functional>
+#include <filesystem>
+
+#include "StringUtils.h"
+#include "MachOFile.h"
+
+std::set<std::string> scanForDependencies(const std::string& path) {
+    __block std::set<std::string> result;
+    struct stat stat_buf;
+    int fd = open(path.c_str(), O_RDONLY, 0);
+    if (fd == -1) {
+        return result;
+    }
+
+    if (fstat(fd, &stat_buf) == -1) {
+        close(fd);
+        return result;
+    }
+
+    const void* buffer = mmap(NULL, (size_t)stat_buf.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+    if (buffer == MAP_FAILED) {
+        close(fd);
+        return result;
+    }
+
+    auto scanner = ^(const char *loadPath, bool isWeak, bool isReExport, bool isUpward, uint32_t compatVersion, uint32_t curVersion, bool &stop) {
+        if (isWeak) { return; } // We explicily avoid LC_LOAD_WEAK_DYLIB since we are trying to build a minimal chroot
+        if (loadPath[0] != '/') { return; } // Only include absolute dependencies
+        result.insert(loadPath);
+    };
+    Diagnostics diag;
+    if ( dyld3::FatFile::isFatFile(buffer) ) {
+        const dyld3::FatFile* ff = (dyld3::FatFile*)buffer;
+        ff->forEachSlice(diag, stat_buf.st_size, ^(uint32_t sliceCpuType, uint32_t sliceCpuSubType, const void* sliceStart, uint64_t sliceSize, bool& stop) {
+            const dyld3::MachOFile* mf = (dyld3::MachOFile*)sliceStart;
+            mf->forEachDependentDylib(scanner);
+         });
+    } else {
+        const dyld3::MachOFile* mf = (dyld3::MachOFile*)buffer;
+        if (mf->isMachO(diag, stat_buf.st_size)) {
+            mf->forEachDependentDylib(scanner);
+        }
+    }
+    close(fd);
+    return result;
+}
+
+std::string withoutPrefixPath(const std::string& path, const std::string& prefix ) {
+    std::string result = path;
+    size_t pos = result.find(prefix);
+    result.erase(pos, prefix.length());
+    return result;
+}
+
+void add_symlinks_to_dylib(const std::string path) {
+    static std::set<std::string> alreadyMatched;
+    size_t pos = path.rfind(".framework/Versions/");
+    auto prefixPath = path.substr(0, pos);
+    if (alreadyMatched.find(prefixPath) != alreadyMatched.end()) { return; }
+
+    if (pos == std::string::npos) { return; }
+//    fprintf(stderr, "PATH: %s\n", path.c_str());
+    size_t versionStart = pos+20;
+    size_t versionEnd = versionStart;
+    while (path[versionEnd] != '/') {
+        ++versionEnd;
+    }
+    size_t frameworkNameBegin = pos;
+    while (path[frameworkNameBegin-1] != '/') {
+        --frameworkNameBegin;
+    }
+    auto frameworkName = path.substr(frameworkNameBegin, pos-frameworkNameBegin);
+    auto version = path.substr(versionStart, versionEnd-versionStart);
+    std::string mainLinkPath = prefixPath + ".framework/" + frameworkName;
+    std::string mainLinkTarget = "Versions/Current/" + frameworkName;
+    std::string versionLinkPath =  prefixPath + ".framework/Versions/Current";
+    std::string versionLinkTarget =  version;;
+    alreadyMatched.insert(prefixPath);
+    if (!std::filesystem::exists(versionLinkPath)) {
+        std::filesystem::create_symlink(version, versionLinkPath);
+    }
+    if (!std::filesystem::exists(mainLinkPath)) {
+        std::filesystem::create_symlink(mainLinkTarget, mainLinkPath);
+    }
+}
+
+void add_symlink(const std::string& target, const std::string& path) {
+    if (!std::filesystem::exists(path)) {
+        std::filesystem::create_symlink(target, path);
+    }
+}
+
+void buildChroot(const std::string& chroot, const std::string& fallback, const std::vector<std::string>& binaries) {
+    auto chrootPath = std::filesystem::path(chroot);
+    auto fallbackPath = std::filesystem::path(fallback);
+
+    for (const auto& binary : binaries) {
+        if (std::filesystem::exists(chroot + binary)) { continue; }
+        std::filesystem::create_directories(std::filesystem::path(chroot + binary).parent_path());
+        std::filesystem::copy(fallback + binary, chroot + binary);
+    }
+    bool foundNewEntries = true;
+    std::set<std::string> scannedFiles;
+    std::string devfsPath = chroot + "/dev";
+    while (foundNewEntries) {
+        foundNewEntries = false;
+        for(auto file = std::filesystem::recursive_directory_iterator(chroot);
+            file != std::filesystem::recursive_directory_iterator();
+            ++file ) {
+            auto filePath = file->path().string();
+            if (filePath == devfsPath) {
+                file.disable_recursion_pending();
+                continue;
+            }
+            if (scannedFiles.find(filePath) != scannedFiles.end()) { continue; }
+            scannedFiles.insert(filePath);
+            auto candidates = scanForDependencies(filePath);
+            for (const auto& candidate : candidates) {
+                if (std::filesystem::exists(chroot + candidate)) { continue; }
+                if (!std::filesystem::exists(fallback + candidate)) { continue; }
+                std::filesystem::create_directories(std::filesystem::path(chroot + candidate).parent_path());
+                std::filesystem::copy(fallback + candidate, chroot + candidate);
+                add_symlinks_to_dylib(chroot + candidate);
+                foundNewEntries = true;
+            }
+        }
+    }
+    add_symlink("libSystem.B.dylib", chroot + "/usr/lib/libSystem.dylib");
+    add_symlink("libSystem.dylib", chroot + "/usr/lib/libc.dylib");
+}
+
+int main(int argc, const char * argv[]) {
+    std::vector<std::string> binaries;
+    std::vector<std::string> overlays;
+    std::string fallback;
+    std::string chroot;
+    for (int i = 1; i < argc; ++i) {
+        const char* arg = argv[i];
+        if (arg[0] == '-') {
+            if (strcmp(arg, "-chroot") == 0) {
+                chroot = argv[++i];
+            } else if (strcmp(arg, "-fallback") == 0) {
+                fallback = argv[++i];
+            } else if (strcmp(arg, "-add_file") == 0) {
+                binaries.push_back(argv[++i]);
+            } else {
+                fprintf(stderr, "unknown option: %s\n", arg);
+                exit(-1);
+            }
+        } else {
+            fprintf(stderr, "unknown option: %s\n", arg);
+            exit(-1);
+        }
+    }
+
+    if (chroot.length() == 0) {
+        fprintf(stderr, "No -chroot <dir>\n");
+        exit(-1);
+    }
+    if (fallback.length() == 0) {
+        fprintf(stderr, "No -fallback <dir>\n");
+        exit(-1);
+    }
+    buildChroot(chroot, fallback, binaries);
+    // insert code here...
+    return 0;
+}

configs/libdyld.xcconfig

@@ -4,5 +4,10 @@ LIBSYSTEM_LIBS[sdk=embedded*]       = -Wl,-upward-lsystem_platform     -Wl,-upwa
 LIBSYSTEM_LIBS[sdk=macosx*]         = -Wl,-upward-lsystem_platform     -Wl,-upward-lsystem_malloc -Wl,-upward-lsystem_c -Wl,-upward-lsystem_pthread     -Wl,-upward-lxpc -Wl,-upward-lsystem_blocks -Wl,-upward-lsystem_kernel     -Wl,-upward-lsystem_sandbox -Wl,-upward-ldispatch -Wl,-upward-lcorecrypto  -Wl,-upward-lcompiler_rt
 LIBSYSTEM_LIBS[sdk=driverkit*]      = -Wl,-upward-lsystem_platform     -Wl,-upward-lsystem_malloc -Wl,-upward-lsystem_c -Wl,-upward-lsystem_pthread  -Wl,-upward-lsystem_blocks -Wl,-upward-lsystem_kernel    -Wl,-upward-lcompiler_rt
 
+EXTRA_SECTIONS                      = 
+EXTRA_SECTIONS[arch=arm64e]         = -Wl,-sectcreate,__SHARED_CACHE,__cfstring,/dev/null
+
+USE_CHAINED_FIXUPS                  =
+
 INSTALL_PATH = /usr/lib/system
 

configs/update_dyld_sim_shared_cache.xcconfig

@@ -1,6 +1,6 @@
-
 #include "<DEVELOPER_DIR>/AppleInternal/XcodeConfig/PlatformSupportHost.xcconfig"
 
+INSTALL_PATH = $(SIMULATOR_RUNTIME_BUNDLE_INSTALL_DIR)/Contents/Resources
 
 //:configuration = Debug
 GCC_PREPROCESSOR_DEFINITIONS = BUILDING_CACHE_BUILDER=1 DEBUG=1

darling-scripts/add-version-macros.sh

@@ -1,14 +0,0 @@
-#!/bin/bash
-set -x
-
-pushd $(dirname "$0") > /dev/null
-SCRIPTDIR=$(pwd -P)
-popd > /dev/null
-
-export SRCROOT="$SCRIPTDIR/.."
-export SDKROOT="$SRCROOT/../../Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"
-
-# copied and modified from dyld.xcodeproj
-${SRCROOT}/bin/expand.rb < "${SRCROOT}/include/mach-o/dyld_priv.h" > "${SRCROOT}/include/mach-o/dyld_priv.h.tmp"
-
-mv "${SRCROOT}/include/mach-o/dyld_priv.h.tmp" "${SRCROOT}/include/mach-o/dyld_priv.h"

doc/man/man1/closured.1

@@ -1,10 +0,0 @@
-.Dd 3/1/17
-.Dt closured 1
-.Os Darwin
-.Sh NAME
-.Nm closured
-.Nd Daemon for building dyld closures.
-.Sh SYNOPSIS
-.Nm closured is a launchd managed daemon.
-.Sh DESCRIPTION
-.Nm closured is a launchd managed daemon.

doc/man/man1/dyld.1

@@ -1,4 +1,4 @@
-.TH DYLD 1 "June 1, 2017" "Apple Inc."
+.TH DYLD 1 "June 1, 2020" "Apple Inc."
 .SH NAME
 dyld \- the dynamic linker
 .SH SYNOPSIS
@@ -51,8 +51,6 @@ DYLD_PRINT_DOFS
 DYLD_PRINT_RPATHS
 .br
 DYLD_SHARED_CACHE_DIR
-.br
-DYLD_SHARED_CACHE_DONT_VALIDATE
 .SH DESCRIPTION
 The dynamic linker checks the following environment variables during the launch
 of each process.
@@ -229,13 +227,7 @@ that expansion was successful or not.
 .TP
 .B DYLD_SHARED_CACHE_DIR
 This is a directory containing dyld shared cache files.  This variable can be used in
-conjunction with DYLD_SHARED_REGION=private and DYLD_SHARED_CACHE_DONT_VALIDATE
-to run a process with an alternate shared cache.
-.TP
-.B DYLD_SHARED_CACHE_DONT_VALIDATE
-Causes dyld to not check that the inode and mod-time of files in the shared cache match
-the requested dylib on disk. Thus a program can be made to run with the dylib in the
-shared cache even though the real dylib has been updated on disk.
+conjunction with DYLD_SHARED_REGION=private to run a process with an alternate shared cache.
 .TP
 .SH DYNAMIC LIBRARY LOADING
 Unlike many other operating systems, Darwin does not locate dependent dynamic libraries

doc/man/man1/dyld_usage.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "DYLD_USAGE" "1" "2018-07-28" "" "dyld"
+.TH "DYLD_USAGE" "1" "2020-04-13" "" "dyld"
 .SH NAME
 dyld_usage \- report dynamic linker activity in real-time
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-\fBdyld_usage\fP \fB[\-e] [\-f mode] [\-t seconds] [\-R rawfile [\-S start_time]
+\fBdyld_usage\fP \fB[\-e] [\-f mode] [\-j] [\-h] [\-t seconds] [\-R rawfile [\-S start_time]
 [\-E end_time]] [pid | cmd [pid | cmd] ...]\fP
 .SH DESCRIPTION
 .sp
@@ -56,16 +56,27 @@ for maximum data display.
 .B \-e
 Exclude the specified list of pids and commands from the sample, and exclude
 \fBdyld_usage\fP by default.
+.INDENT 7.0
+.TP
+.B \-j
+.UNINDENT
+.sp
+Display output in JSON format.
+.UNINDENT
+.INDENT 0.0
+.TP
+.B \-h
+Display usage information and exit.
 .UNINDENT
 .INDENT 0.0
 .TP
 .B \-R
-specifies a raw trace file to process.
+Specify a raw trace file to process.
 .UNINDENT
 .INDENT 0.0
 .TP
 .B \-t
-specifies timeout in seconds (for use in automated tools).
+Specify timeout in seconds (for use in automated tools).
 .UNINDENT
 .SH DISPLAY
 .sp
@@ -107,6 +118,6 @@ processes named Mail.
 .SH AUTHOR
 Apple, Inc.
 .SH COPYRIGHT
-2000-2018, Apple, Inc.
+2000-2020, Apple, Inc.
 .\" Generated by docutils manpage writer.
 .

doc/man/man1/update_dyld_shared_cache.1

@@ -1,75 +0,0 @@
-.Dd June 1, 2017
-.Dt update_dyld_shared_cache 1
-.Os Darwin
-.Sh NAME
-.Nm update_dyld_shared_cache
-.Nd "Updates dyld's shared cache"
-.Sh SYNOPSIS
-.Nm
-.Op Fl root Ar directory 
-.Op Fl overlay Ar directory 
-.Op Fl arch Ar arch 
-.Op Fl force 
-.Op Fl debug
-.Op Fl universal_boot
-.Sh DESCRIPTION
-.Nm update_dyld_shared_cache
-ensures that dyld's shared cache is up-to-date.  This tool is normally
-only run by Apple's Installer and Software Update, as they are the only
-official ways for OS dylibs to be updated.  But if for some reason you
-used another mechanism to alter an OS dylib, you should manually run
-.Nm update_dyld_shared_cache . 
-.Pp
-Note that the new cache does not take effect until the OS is rebooted.  
-.Pp
-The dyld shared cache
-is mapped by dyld into a process at launch time. Later, when loading
-any mach-o image, dyld will first check if is in the share cache, and if
-it is will use that pre-bound version instead of opening, mapping, and binding
-the original file.  This results in significant performance improvements to
-launch time.
-.Pp
-.Nm update_dyld_shared_cache
-scans the directory /System/Library/Receipts/ for .bom files which list all files
-installed.  From that info it creates the set of OS dylibs to build into the dyld cache.
-.Pp
-.Nm update_dyld_shared_cache
-builds a separate cache file for each architecture.  The cache files and a readable text
-map of the cached are generated to /var/db/dyld.
-.Pp
-You must be root to run this tool.
-.Pp
-The options are as follows:
-.Bl -tag
-.It Fl root Ar directory
-This option specifies the root of an OS installation for which dyld's
-shared cache should be updated.  This is used by the Installer to update the
-dyld shared cache in a partition other than the one you into which you are currently 
-booted.  The cache files are created in the var/db/dyld directory of the specified directory.
-Note: if you are manually doing this, be sure to run the update_dyld_shared_cache tool
-that is in the partition being updated.  This assures the cache format created will
-match that expected when booting off that partition. 
-.It Fl overlay Ar directory
-This option specifies the root of a sparse directory tree.  When building
-the dyld shared cache, any corresponding mach-o files in the sparse directory 
-will override those in the boot partition.  This is used by Software
-Update to build a dyld shared cache for the update that is about to be
-installed.  The cache files
-are created in the var/db/dyld directory of the specified directory.
-.It Fl arch Ar arch
-By default 
-.Nm update_dyld_shared_cache 
-generates cache files for all architecture that the current machine
-can execute.  You can override this behavior by specifying one or more -arch options and list
-exactly which architectures should have their shared caches updated.
-.It Fl force
-This option will cause 
-.Nm update_dyld_shared_cache
-to regenerated the shared cache files even if they appear to be already up-to-date. 
-.It Fl debug
-This option prints out additional information about the work being done.
-.It Fl universal_boot
-This option builds caches for all machines.
-.El
-.Sh SEE ALSO
-.Xr dyld 1

doc/man/man3/dlopen_preflight.3

@@ -1,4 +1,4 @@
-.Dd April 17, 2006
+.Dd May 11, 2020
 .Os
 .Dt DLOPEN_PREFLIGHT 3
 .Sh NAME
@@ -16,6 +16,13 @@ It checks if the file and libraries it depends on are all compatible with the cu
 That is, they contain the correct architecture and are not otherwise ABI incompatible.
 .Pp
 .Fn dlopen_preflight
+was created for the PowerPC to Intel transition for use by apps with plugins that the user chooses to load.
+The app could use dlopen_preflight() to show only loadable plugins to the user (such as in a menu).
+.Pp
+This is potentially an expensive call because it may internally do the same as dlopen/dlclose.  Only
+use dlopen_preflight() if you need to show the user a list of potentially loadable plugins.
+.Pp
+.Fn dlopen_preflight
 was first available in Mac OS X 10.5.
 .Sh SEARCHING
 .Fn dlopen_preflight

doc/man/man3/dlsym.3

@@ -85,7 +85,6 @@ is the name used in C source code.  For example to find the address
 of function foo(), you would pass "foo" as the symbol name.  This
 is unlike the older dyld APIs which required a leading underscore.
 If you are looking up a C++ symbol, you need to use the mangled C++ symbol name.
-name.  
 .Sh SEE ALSO
 .Xr dlopen 3
 .Xr dlerror 3

doc/man/man3/dyld.3

@@ -76,7 +76,7 @@ is out of range zero is returned.
 .Fn _dyld_get_image_name
 returns the name of the image indexed by
 .Fa image_index.
-The C-string continues to be owned by dyld and should not deleted.
+The C-string continues to be owned by dyld and should not be deleted.
 If 
 .Fa image_index
 is out of range NULL is returned.

doc/rst/conf.py

@@ -232,18 +232,15 @@ for name in os.listdir(command_guide_path):
         header = f.readline().rstrip('\n')
 
         if len(header) != len(title):
-            print >>sys.stderr, (
-                "error: invalid header in %r (does not match title)" % (
-                    file_subpath,))
+            print("error: invalid header in {} (does not match title)".
+            format(file_subpath), file=sys.stderr)
         if ' - ' not in title:
-            print >>sys.stderr, (
-                ("error: invalid title in %r "
-                 "(expected '<name> - <description>')") % (
-                    file_subpath,))
+            print("error: invalid title in {} (expected '<name> - <description>')".
+            format(file_subpath), file=sys.stderr)
 
         # Split the name out of the title.
         name,description = title.split(' - ', 1)
-        man_pages.append((file_subpath.replace('.rst',''), name,
+        man_pages.append((name.replace('.rst',''), name,
                           description, man_page_authors, 1))
 
 # If true, show URL addresses after external links.

doc/rst/dyld_usage.rst

@@ -1,10 +1,10 @@
 dyld_usage - report dynamic linker activity in real-time
-==========================================================
+========================================================
 
 SYNOPSIS
 --------
 
-:program:`dyld_usage` **[-e] [-f mode] [-t seconds] [-R rawfile [-S start_time]
+:program:`dyld_usage` **[-e] [-f mode] [-j] [-h] [-t seconds] [-R rawfile [-S start_time]
 [-E end_time]] [pid | cmd [pid | cmd] ...]**
 
 DESCRIPTION
@@ -31,13 +31,21 @@ OPTIONS
   Exclude the specified list of pids and commands from the sample, and exclude
   :program:`dyld_usage` by default.
 
+  .. option:: -j
+
+  Display output in JSON format.
+
+.. option:: -h
+
+  Display usage information and exit.
+
 .. option:: -R
 
-  specifies a raw trace file to process.
+  Specify a raw trace file to process.
 
 .. option:: -t
 
-  specifies timeout in seconds (for use in automated tools).
+  Specify timeout in seconds (for use in automated tools).
 
 DISPLAY
 -------

dyld.xcodeproj/ContainerizedTestRunner.xctestplan

@@ -0,0 +1,25 @@
+{
+  "configurations" : [
+    {
+      "id" : "0ED518CA-34AA-4808-822E-3532C949AFBA",
+      "name" : "Configuration 1",
+      "options" : {
+
+      }
+    }
+  ],
+  "defaultOptions" : {
+    "codeCoverage" : false
+  },
+  "testTargets" : [
+    {
+      "parallelizable" : true,
+      "target" : {
+        "containerPath" : "container:dyld.xcodeproj",
+        "identifier" : "3715A2FD232320BC0059433D",
+        "name" : "ContainerizedTestRunner"
+      }
+    }
+  ],
+  "version" : 1
+}

dyld3/APIs.cpp

@@ -54,10 +54,13 @@
 #include "ClosureBuilder.h"
 #include "ClosureFileSystemPhysical.h"
 
+#include <dyld/VersionMap.h>
+
 #if __has_feature(ptrauth_calls)
 #include <ptrauth.h>
 #endif
 
+extern mach_header __dso_handle;
 
 namespace dyld {
     extern dyld_all_image_infos dyld_all_image_infos;
@@ -77,10 +80,6 @@ static const void *stripPointer(const void *ptr) {
 
 pthread_mutex_t RecursiveAutoLock::_sMutex = PTHREAD_RECURSIVE_MUTEX_INITIALIZER;
 
-// forward declaration
-static void dyld_get_image_versions_internal(const struct mach_header* mh, void (^callback)(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version));
-
-
 uint32_t _dyld_image_count(void)
 {
     log_apis("_dyld_image_count()\n");
@@ -121,6 +120,11 @@ const char* _dyld_get_image_name(uint32_t imageIndex)
     return gAllImages.imagePathByIndex(imageIndex);
 }
 
+const struct mach_header * _dyld_get_prog_image_header()
+{
+    log_apis("_dyld_get_prog_image_header()\n");
+    return gAllImages.mainExecutable();
+}
 
 static bool nameMatch(const char* installName, const char* libraryName)
 {
@@ -304,7 +308,18 @@ uint32_t dyld_get_sdk_version(const mach_header* mh)
 uint32_t dyld_get_program_sdk_version()
 {
 	log_apis("dyld_get_program_sdk_version()\n");
-    return dyld3::dyld_get_sdk_version(gAllImages.mainExecutable());
+    uint32_t result = dyld3::dyld_get_sdk_version(gAllImages.mainExecutable());
+#if TARGET_OS_OSX
+    // HACK: We didn't have time to fix all the zippered clients in the spring releases, so keep the mapping. We have resolved it for all new clients using the platform aware SPIs. Since we are doing to deprecate this SPI we will leave the hack in.
+    if (dyld_get_active_platform() == (dyld_platform_t)dyld3::Platform::iOSMac) {
+        if (result >= 0x000D0400) {
+            result = 0x000A0F04;
+        } else {
+            result = 0x000A0F00;
+        }
+    }
+#endif
+    return result;
 }
 
 uint32_t dyld_get_min_os_version(const mach_header* mh)
@@ -334,9 +349,7 @@ dyld_platform_t dyld_get_active_platform(void) {
 
 dyld_platform_t dyld_get_base_platform(dyld_platform_t platform) {
     switch (platform) {
-#ifdef PLATFORM_IOSMAC
-        case PLATFORM_IOSMAC:               return PLATFORM_IOS;
-#endif
+        case PLATFORM_MACCATALYST:               return PLATFORM_IOS;
         case PLATFORM_IOSSIMULATOR:         return PLATFORM_IOS;
         case PLATFORM_WATCHOSSIMULATOR:     return PLATFORM_WATCHOS;
         case PLATFORM_TVOSSIMULATOR:        return PLATFORM_TVOS;
@@ -355,8 +368,24 @@ bool dyld_is_simulator_platform(dyld_platform_t platform) {
     }
 }
 
+static
+dyld_build_version_t mapFromVersionSet(dyld_build_version_t version) {
+    if (version.platform != 0xffffffff) return version;
+    auto i = std::lower_bound(sVersionMap.begin(), sVersionMap.end(), version.version);
+    assert(i != sVersionMap.end());
+    switch(dyld3::dyld_get_base_platform(::dyld_get_active_platform())) {
+        case PLATFORM_MACOS: return { .platform = PLATFORM_MACOS, .version = i->macos };
+        case PLATFORM_IOS: return { .platform = PLATFORM_IOS, .version = i->ios };
+        case PLATFORM_WATCHOS: return { .platform = PLATFORM_WATCHOS, .version = i->watchos };
+        case PLATFORM_TVOS: return { .platform = PLATFORM_TVOS, .version = i->tvos };
+        case PLATFORM_BRIDGEOS: return { .platform = PLATFORM_BRIDGEOS, .version = i->bridgeos };
+        default: return { .platform = 0, .version = 0 };
+    }
+}
+
 bool dyld_sdk_at_least(const struct mach_header* mh, dyld_build_version_t version) {
     __block bool retval = false;
+    version = mapFromVersionSet(version);
     dyld3::dyld_get_image_versions(mh, ^(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version) {
         if (dyld3::dyld_get_base_platform(platform) == version.platform && sdk_version >= version.version) {
             retval = true;
@@ -367,6 +396,7 @@ bool dyld_sdk_at_least(const struct mach_header* mh, dyld_build_version_t versio
 
 bool dyld_minos_at_least(const struct mach_header* mh, dyld_build_version_t version) {
     __block bool retval = false;
+    version = mapFromVersionSet(version);
     dyld3::dyld_get_image_versions(mh, ^(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version) {
         if (dyld3::dyld_get_base_platform(platform) == version.platform && min_version >= version.version) {
             retval = true;
@@ -375,15 +405,7 @@ bool dyld_minos_at_least(const struct mach_header* mh, dyld_build_version_t vers
     return retval;
 }
 
-bool dyld_program_sdk_at_least(dyld_build_version_t version) {
-    return dyld3::dyld_sdk_at_least(gAllImages.mainExecutable(), version);
-}
-
-bool dyld_program_minos_at_least(dyld_build_version_t version) {
-    return dyld3::dyld_minos_at_least(gAllImages.mainExecutable(), version);
-}
-
-#if TARGET_OS_OSX || TARGET_OS_IOS
+#if TARGET_OS_OSX
 static
 uint32_t linkedDylibVersion(const mach_header* mh, const char *installname) {
     __block uint32_t retval = 0;
@@ -401,17 +423,18 @@ uint32_t linkedDylibVersion(const mach_header* mh, const char *installname) {
 #define PACKED_VERSION(major, minor, tiny) ((((major) & 0xffff) << 16) | (((minor) & 0xff) << 8) | ((tiny) & 0xff))
 
 static uint32_t deriveVersionFromDylibs(const struct mach_header* mh) {
+#if TARGET_OS_IOS
+    // 7.0 is the last version that was in iOSes mapping table, and it is the earliest version that support 64 bit binarie.
+    // Since we dropped 32 bit support, we know any binary with a version must be from 7.0
+    return 0x00070000;
+#elif TARGET_OS_OSX
     // This is a binary without a version load command, we need to infer things
     struct DylibToOSMapping {
         uint32_t dylibVersion;
         uint32_t osVersion;
     };
-    uint32_t linkedVersion = 0;
-#if TARGET_OS_OSX
-    linkedVersion = linkedDylibVersion(mh, "/usr/lib/libSystem.B.dylib");
+    uint32_t linkedVersion = linkedDylibVersion(mh, "/usr/lib/libSystem.B.dylib");
     static const DylibToOSMapping versionMapping[] = {
-        { PACKED_VERSION(88,1,3),   0x000A0400 },
-        { PACKED_VERSION(111,0,0),  0x000A0500 },
         { PACKED_VERSION(123,0,0),  0x000A0600 },
         { PACKED_VERSION(159,0,0),  0x000A0700 },
         { PACKED_VERSION(169,3,0),  0x000A0800 },
@@ -420,31 +443,6 @@ static uint32_t deriveVersionFromDylibs(const struct mach_header* mh) {
         // We don't need to expand this table because all recent
         // binaries have LC_VERSION_MIN_ load command.
     };
-#elif TARGET_OS_IOS
-        linkedVersion = linkedDylibVersion(mh, "/System/Library/Frameworks/Foundation.framework/Foundation");
-        static const DylibToOSMapping versionMapping[] = {
-            { PACKED_VERSION(678,24,0), 0x00020000 },
-            { PACKED_VERSION(678,26,0), 0x00020100 },
-            { PACKED_VERSION(678,29,0), 0x00020200 },
-            { PACKED_VERSION(678,47,0), 0x00030000 },
-            { PACKED_VERSION(678,51,0), 0x00030100 },
-            { PACKED_VERSION(678,60,0), 0x00030200 },
-            { PACKED_VERSION(751,32,0), 0x00040000 },
-            { PACKED_VERSION(751,37,0), 0x00040100 },
-            { PACKED_VERSION(751,49,0), 0x00040200 },
-            { PACKED_VERSION(751,58,0), 0x00040300 },
-            { PACKED_VERSION(881,0,0),  0x00050000 },
-            { PACKED_VERSION(890,1,0),  0x00050100 },
-            { PACKED_VERSION(992,0,0),  0x00060000 },
-            { PACKED_VERSION(993,0,0),  0x00060100 },
-            { PACKED_VERSION(1038,14,0),0x00070000 },
-            { PACKED_VERSION(0,0,0),    0x00070000 }
-            // We don't need to expand this table because all recent
-            // binaries have LC_VERSION_MIN_ load command.
-    };
-#else
-    static const DylibToOSMapping versionMapping[] = {};
-#endif
     if ( linkedVersion != 0 ) {
         uint32_t lastOsVersion = 0;
         for (const DylibToOSMapping* p=versionMapping; ; ++p) {
@@ -457,6 +455,7 @@ static uint32_t deriveVersionFromDylibs(const struct mach_header* mh) {
             lastOsVersion = p->osVersion;
         }
     }
+#endif
     return 0;
 }
 
@@ -471,9 +470,6 @@ static void dyld_get_image_versions_internal(const struct mach_header* mh, void
         if (sdk == 0) {
             sdk = deriveVersionFromDylibs(mh);
         }
-        if (platform == dyld3::Platform::iOSMac) {
-            sdk = 0x000A0F00;
-        }
         callback((const dyld_platform_t)platform, sdk, minOS);
     });
 
@@ -508,29 +504,24 @@ void dyld_get_image_versions(const struct mach_header* mh, void (^callback)(dyld
     // FIXME: Once dyld2 is gone gAllImages.mainExecutable() will be valid in all cases
     // and we can stop calling _NSGetMachExecuteHeader()
     if (mh == (const struct mach_header*)_NSGetMachExecuteHeader()) {
-        // Cache the main executable and short circuit parsing the
-        if (mainExecutablePlatform == 0) {
+        if (mainExecutablePlatform) {
+            return callback(mainExecutablePlatform, mainExecutableSDKVersion, mainExecutableMinOSVersion);
+        }
+        mainExecutablePlatform = ::dyld_get_active_platform();
         dyld_get_image_versions_internal(mh, ^(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version) {
-#if 0
-                //FIXME: Reenable this once Libc supports dynamic platforms.
-                if (platform == PLATFORM_MACOS && dyld_get_active_platform() == PLATFORM_IOSMAC) {
-                    //FIXME: This version should be generated at link time
-                    mainExecutablePlatform = PLATFORM_IOSMAC;
-                    mainExecutableSDKVersion = 0x000D0000;
-                    mainExecutableMinOSVersion = 0x000D0000;
-                } else {
-                    mainExecutablePlatform = platform;
-                    mainExecutableSDKVersion = sdk_version;
-                    mainExecutableMinOSVersion = min_version;
+            if (platform == PLATFORM_MACOS && dyld_get_base_platform(mainExecutablePlatform) == PLATFORM_IOS) {
+                // We are running with DYLD_FORCE_PLATFORM, use the current OSes values
+                dyld_get_image_versions_internal(&__dso_handle, ^(dyld_platform_t dyld_platform, uint32_t dyld_sdk_version, uint32_t dyld_min_version) {
+                    if (dyld_get_base_platform(dyld_platform) == PLATFORM_IOS) {
+                        mainExecutableSDKVersion = dyld_sdk_version;
+                        mainExecutableMinOSVersion = dyld_min_version;
                     }
-#else
-                mainExecutablePlatform = platform;
-                mainExecutableSDKVersion = sdk_version;
-                mainExecutableMinOSVersion = min_version;
-#endif
-                //FIXME: Assert if more than one command?
                 });
+            } else {
+                 mainExecutableSDKVersion = sdk_version;
+                 mainExecutableMinOSVersion = min_version;
              }
+        });
         return callback(mainExecutablePlatform, mainExecutableSDKVersion, mainExecutableMinOSVersion);
     }
 #if TARGET_OS_EMBEDDED
@@ -554,6 +545,120 @@ void dyld_get_image_versions(const struct mach_header* mh, void (^callback)(dyld
         dyld_get_image_versions_internal(mh, callback);
 }
 
+struct VIS_HIDDEN VersionSPIDispatcher {
+    static bool dyld_program_minos_at_least(dyld_build_version_t version) {
+        return dyld_program_minos_at_least_active(version);
+    }
+    static bool dyld_program_sdk_at_least(dyld_build_version_t version) {
+        return dyld_program_sdk_at_least_active(version);
+    }
+private:
+    // We put these into a struct to guarantee so we can control the placement to guarantee a version and the set equivalent
+    // Can be loaded via a single load pair instruction.
+    struct FastPathData {
+        uint32_t version;
+        uint32_t versionSetEquivalent;
+        dyld_platform_t platform;
+    };
+    static uint32_t findVersionSetEquuivalent(uint32_t version) {
+        uint32_t candidateVersion = 0;
+        uint32_t candidateVersionEquivalent = 0;
+        uint32_t newVersionSetVersion = 0;
+        for (const auto& i : sVersionMap) {
+            switch (dyld_get_base_platform(::dyld_get_active_platform())) {
+                case PLATFORM_MACOS:    newVersionSetVersion = i.macos; break;
+                case PLATFORM_IOS:      newVersionSetVersion = i.ios; break;
+                case PLATFORM_WATCHOS:  newVersionSetVersion = i.watchos; break;
+                case PLATFORM_TVOS:     newVersionSetVersion = i.tvos; break;
+                case PLATFORM_BRIDGEOS: newVersionSetVersion = i.bridgeos; break;
+                default: newVersionSetVersion = 0xffffffff; // If we do not know about the platform it is newer than everything
+            }
+            if (newVersionSetVersion > version) { break; }
+            candidateVersion = newVersionSetVersion;
+            candidateVersionEquivalent = i.set;
+        }
+        return candidateVersionEquivalent;
+    };
+
+    static void setVersionMappingFastPathData(const struct mach_header* mh) {
+        dyld3::dyld_get_image_versions(mh, ^(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version) {
+            minosFastPathData.platform = dyld_get_base_platform(::dyld_get_active_platform());
+            sdkFastPathData.platform = dyld_get_base_platform(::dyld_get_active_platform());
+            minosFastPathData.version = min_version;
+            minosFastPathData.versionSetEquivalent = findVersionSetEquuivalent(min_version);
+            sdkFastPathData.version = sdk_version;
+            sdkFastPathData.versionSetEquivalent = findVersionSetEquuivalent(sdk_version);
+        });
+    }
+
+    static void setupFastPath(void) {
+        setVersionMappingFastPathData((const struct mach_header*)_NSGetMachExecuteHeader());
+        dyld_program_minos_at_least_active = &dyld_program_minos_at_least_fast;
+        dyld_program_sdk_at_least_active = &dyld_program_sdk_at_least_fast;
+    }
+
+    static bool dyld_program_minos_at_least_slow (dyld_build_version_t version) {
+        setupFastPath();
+        return dyld_program_minos_at_least_fast(version);
+    }
+
+    static bool dyld_program_sdk_at_least_slow (dyld_build_version_t version) {
+        setupFastPath();
+        return dyld_program_sdk_at_least_fast(version);
+    }
+
+    // Fast path implementation of version checks for main executables
+    // This works by using the fact that are essentially 3 cases we care about:
+    // 1. Comparing the exctuable against any other platform (which should always return false)
+    // 2. Comparing the exctuable against a version set (platform 0xfffffff)
+    // 3. Comparing the exctuable againstt our base platform
+    //
+    // We achieve this by setting up a single compare (currentVersion >= version.version) and a couple of
+    // of simple tests that will all compile to conditional moves to setup that compare:
+    // 1. We setup the comapreVersion as 0. It will only keep that value if it is not a version set and it
+    //    it is not the platform we are testing against. 0 will be less than the value encoded in any well
+    //    formed binary, so the test will end up returning false
+    // 2. If the platform is 0xffffffff it is a version set. In the fast path setup we we calculated a value
+    //    that allows a direct comparison, so we set comapreVersion to that (versionSetEquivalent)
+    // 3. If it is a concrete platform and it matches the current platform running then we can set comapreVersion
+    //    to the actual version number that the was embedded in the binary, which is we stashed in the fast
+    //    path data
+
+    static bool dyld_program_minos_at_least_fast (dyld_build_version_t version) {
+        uint32_t currentVersion = 0;
+        if (version.platform == 0xffffffff) { currentVersion = minosFastPathData.versionSetEquivalent; }
+        if (version.platform == minosFastPathData.platform) { currentVersion = minosFastPathData.version; }
+        return (currentVersion >= version.version);
+    }
+
+    static bool dyld_program_sdk_at_least_fast (dyld_build_version_t version) {
+        uint32_t currentVersion = 0;
+        if (version.platform == 0xffffffff) { currentVersion = sdkFastPathData.versionSetEquivalent ; }
+        if (version.platform == sdkFastPathData.platform) { currentVersion = sdkFastPathData.version; }
+        return (currentVersion >= version.version);
+    }
+
+    static bool (*dyld_program_minos_at_least_active)(dyld_build_version_t version);
+    static bool (*dyld_program_sdk_at_least_active)(dyld_build_version_t version);
+    static FastPathData minosFastPathData;
+    static FastPathData sdkFastPathData;
+};
+
+bool (*VersionSPIDispatcher::dyld_program_minos_at_least_active)(dyld_build_version_t version) = &VersionSPIDispatcher::dyld_program_minos_at_least_slow;
+bool (*VersionSPIDispatcher::dyld_program_sdk_at_least_active)(dyld_build_version_t version) = &VersionSPIDispatcher::dyld_program_sdk_at_least_slow;
+VersionSPIDispatcher::FastPathData VersionSPIDispatcher::minosFastPathData = {0, 0, 0};
+VersionSPIDispatcher::FastPathData VersionSPIDispatcher::sdkFastPathData = {0, 0, 0};
+
+
+// We handle this directly instead of dispatching through dyld3::dyld_program_sdk_at_least because they are very perf sensitive
+bool dyld_program_minos_at_least(dyld_build_version_t version) {
+    return VersionSPIDispatcher::dyld_program_minos_at_least(version);
+}
+
+bool dyld_program_sdk_at_least(dyld_build_version_t version) {
+    return VersionSPIDispatcher::dyld_program_sdk_at_least(version);
+}
+
 uint32_t dyld_get_program_min_os_version()
 {
     log_apis("dyld_get_program_min_os_version()\n");
@@ -853,7 +958,8 @@ void* dlopen_internal(const char* path, int mode, void* callerAddress)
     else
         leafName = path;
 
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
+
+#if TARGET_OS_IPHONE
     // <rdar://problem/40235395> dyld3: dlopen() not working with non-canonical paths
     char canonicalPath[PATH_MAX];
     if ( leafName != path ) {
@@ -914,8 +1020,8 @@ bool dlopen_preflight_internal(const char* path)
     DYLD_LOAD_LOCK_THIS_BLOCK
     log_apis("dlopen_preflight(%s)\n", path);
 
-    // check if path is in dyld shared cache
-    if ( gAllImages.dyldCacheHasPath(path) )
+    // check if path is in dyld shared cache, or is a symlink to the cache
+    if ( _dyld_shared_cache_contains_path(path) )
         return true;
 
     // check if file is loadable
@@ -928,8 +1034,6 @@ bool dlopen_preflight_internal(const char* path)
         return true;
     }
 
-    // FIXME: may be symlink to something in dyld cache
-
     return false;
 }
 
@@ -1137,6 +1241,12 @@ bool _dyld_shared_cache_is_locally_built()
     return false;
 }
 
+uint32_t _dyld_launch_mode()
+{
+    return gAllImages.launchMode();
+}
+
+
 void _dyld_images_for_addresses(unsigned count, const void* addresses[], dyld_image_uuid_offset infos[])
 {
     log_apis("_dyld_images_for_addresses(%u, %p, %p)\n", count, addresses, infos);
@@ -1238,13 +1348,13 @@ void dyld_dynamic_interpose(const mach_header* mh, const dyld_interpose_tuple ar
 static void* mapStartOfCache(const char* path, size_t length)
 {
     struct stat statbuf;
-    if ( ::stat(path, &statbuf) == -1 )
+    if ( dyld3::stat(path, &statbuf) == -1 )
         return NULL;
 
     if ( statbuf.st_size < length )
         return NULL;
 
-    int cache_fd = ::open(path, O_RDONLY);
+    int cache_fd = dyld3::open(path, O_RDONLY, 0);
     if ( cache_fd < 0 )
         return NULL;
 
@@ -1278,7 +1388,7 @@ static const DyldSharedCache* findCacheInDirAndMap(const uuid_t cacheUuid, const
             if ( const DyldSharedCache* cache = (DyldSharedCache*)mapStartOfCache(cachePath, 0x00100000) ) {
                 uuid_t foundUuid;
                 cache->getUUID(foundUuid);
-                if ( ::memcmp(foundUuid, cacheUuid, 16) != 0 ) {
+                if ( (::memcmp(cache, "dyld_", 5) != 0) || (::memcmp(foundUuid, cacheUuid, 16) != 0) ) {
                     // wrong uuid, unmap and keep looking
                     ::munmap((void*)cache, 0x00100000);
                 }
@@ -1310,12 +1420,11 @@ int dyld_shared_cache_find_iterate_text(const uuid_t cacheUuid, const char* extr
     }
     if ( sharedCache == nullptr ) {
          // if not, look in default location for cache files
-    #if    __IPHONE_OS_VERSION_MIN_REQUIRED
-        const char* defaultSearchDir = IPHONE_DYLD_SHARED_CACHE_DIR;
+    #if TARGET_OS_IPHONE
+        sharedCache = findCacheInDirAndMap(cacheUuid, IPHONE_DYLD_SHARED_CACHE_DIR, sizeMapped);
     #else
-        const char* defaultSearchDir = MACOSX_DYLD_SHARED_CACHE_DIR;
+        sharedCache = findCacheInDirAndMap(cacheUuid, MACOSX_MRM_DYLD_SHARED_CACHE_DIR, sizeMapped);
    #endif
-        sharedCache = findCacheInDirAndMap(cacheUuid, defaultSearchDir, sizeMapped);
         // if not there, look in extra search locations
         if ( sharedCache == nullptr ) {
             for (const char** p = extraSearchDirs; *p != nullptr; ++p) {
@@ -1330,7 +1439,8 @@ int dyld_shared_cache_find_iterate_text(const uuid_t cacheUuid, const char* extr
 
     // get base address of cache
     __block uint64_t cacheUnslidBaseAddress = 0;
-    sharedCache->forEachRegion(^(const void *content, uint64_t vmAddr, uint64_t size, uint32_t permissions) {
+    sharedCache->forEachRegion(^(const void *content, uint64_t vmAddr, uint64_t size,
+                                 uint32_t initProt, uint32_t maxProt, uint64_t flags) {
         if ( cacheUnslidBaseAddress == 0 )
             cacheUnslidBaseAddress = vmAddr;
     });
@@ -1361,9 +1471,9 @@ int dyld_shared_cache_iterate_text(const uuid_t cacheUuid, void (^callback)(cons
     return dyld3::dyld_shared_cache_find_iterate_text(cacheUuid, extraSearchDirs, callback);
 }
     
-bool dyld_need_closure(const char* execPath, const char* tempDir)
+bool dyld_need_closure(const char* execPath, const char* dataContainerRootDir)
 {
-    log_apis("dyld_need_closure()\n");
+    log_apis("dyld_need_closure(%s)\n", execPath);
 
     // We don't need to build a closure if the shared cache has it already
     const DyldSharedCache* sharedCache = (DyldSharedCache*)gAllImages.cacheLoadAddress();
@@ -1372,11 +1482,29 @@ bool dyld_need_closure(const char* execPath, const char* tempDir)
             return false;
     }
 
+    // this SPI changed. Originally the second path was to $TMPDIR, now it is $HOME
+    // if called old way, adjust
+    size_t rootDirLen = strlen(dataContainerRootDir);
+    char homeFromTmp[PATH_MAX];
+    if ( (rootDirLen > 5) && (strcmp(&dataContainerRootDir[rootDirLen-4], "/tmp") == 0) && (rootDirLen < PATH_MAX) ) {
+        strlcpy(homeFromTmp, dataContainerRootDir, PATH_MAX);
+        homeFromTmp[rootDirLen-4] = '\0';
+        dataContainerRootDir = homeFromTmp;
+    }
+
+    // dummy up envp needed by buildClosureCachePath()
+    char strBuf[PATH_MAX+8]; // room for HOME= and max path
+    strcpy(strBuf, "HOME=");
+    strlcat(strBuf, dataContainerRootDir, sizeof(strBuf));
+    const char* envp[2];
+    envp[0] = strBuf;
+    envp[1] = nullptr;
 
     char closurePath[PATH_MAX];
-    if ( dyld3::closure::LaunchClosure::buildClosureCachePath(execPath, closurePath, tempDir, false) ) {
+    if ( dyld3::closure::LaunchClosure::buildClosureCachePath(execPath, envp, false, closurePath) ) {
         struct stat statbuf;
-        return (::stat(closurePath, &statbuf) != 0);
+        // if no file at location where closure would be stored, then need to build a closure
+        return (dyld3::stat(closurePath, &statbuf) != 0);
     }
 
     // Not containerized so no point in building a closure.
@@ -1389,7 +1517,14 @@ void _dyld_missing_symbol_abort()
     // dyld3 binds all such missing symbols to this one handler.
     // We need the crash log to contain the backtrace so someone can
     // figure out the symbol.
-    abort_report_np("missing lazy symbol called");
+
+    auto allImageInfos = gAllImages.oldAllImageInfo();
+    allImageInfos->errorKind           = DYLD_EXIT_REASON_SYMBOL_MISSING;
+    allImageInfos->errorClientOfDylibPath   = "<unknown>";
+    allImageInfos->errorTargetDylibPath     = "<unknown>";
+    allImageInfos->errorSymbol              = "<unknown>";
+
+    halt("missing lazy symbol called");
 }
 
 const char* _dyld_get_objc_selector(const char* selName)
@@ -1410,6 +1545,12 @@ void _dyld_for_each_objc_protocol(const char* protocolName,
     gAllImages.forEachObjCProtocol(protocolName, callback);
 }
 
+void _dyld_register_driverkit_main(void (*mainFunc)())
+{
+    log_apis("_dyld_register_driverkit_main()\n");
+    gAllImages.setDriverkitMain(mainFunc);
+}
+
 #if !TARGET_OS_DRIVERKIT
 struct dyld_func {
     const char*  name;
@@ -1424,6 +1565,10 @@ static const struct dyld_func dyld_funcs[] = {
     {"__dyld_get_image_name",           (void*)dyld3::_dyld_get_image_name },
     {"__dyld_get_image_header",         (void*)dyld3::_dyld_get_image_header },
     {"__dyld_get_image_vmaddr_slide",   (void*)dyld3::_dyld_get_image_vmaddr_slide },
+#if TARGET_OS_OSX
+    // <rdar://problem/59265987> support old licenseware plug ins on macOS
+    {"__dyld_lookup_and_bind",          (void*)dyld3::_dyld_lookup_and_bind },
+#endif
 };
 #endif
 

dyld3/APIs.h

@@ -82,6 +82,8 @@ intptr_t _dyld_get_image_vmaddr_slide(uint32_t imageIndex) TEMP_HIDDEN;
 
 const char* _dyld_get_image_name(uint32_t imageIndex) TEMP_HIDDEN;
 
+const struct mach_header * _dyld_get_prog_image_header() TEMP_HIDDEN;
+
 int32_t NSVersionOfLinkTimeLibrary(const char* libraryName) TEMP_HIDDEN;
 
 int32_t NSVersionOfRunTimeLibrary(const char* libraryName) TEMP_HIDDEN;
@@ -157,7 +159,9 @@ bool _dyld_shared_cache_optimized() TEMP_HIDDEN;
 
 bool _dyld_shared_cache_is_locally_built() TEMP_HIDDEN;
     
-bool dyld_need_closure(const char* execPath, const char* tempDir) TEMP_HIDDEN;
+uint32_t _dyld_launch_mode() TEMP_HIDDEN;
+
+bool dyld_need_closure(const char* execPath, const char* dataContainerRootDir) TEMP_HIDDEN;
 
 void _dyld_images_for_addresses(unsigned count, const void* addresses[], struct dyld_image_uuid_offset infos[]) TEMP_HIDDEN;
 
@@ -193,8 +197,10 @@ void _dyld_for_each_objc_class(const char* className,
 void _dyld_for_each_objc_protocol(const char* protocolName,
                                   void (^callback)(void* protocolPtr, bool isLoaded, bool* stop)) TEMP_HIDDEN;
 
+void _dyld_register_driverkit_main(void (*mainFunc)())TEMP_HIDDEN;
+
 // only in macOS and deprecated 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
 NSObjectFileImageReturnCode NSCreateObjectFileImageFromFile(const char* pathName, NSObjectFileImage *objectFileImage) TEMP_HIDDEN;
 NSObjectFileImageReturnCode NSCreateObjectFileImageFromMemory(const void *address, size_t size, NSObjectFileImage *objectFileImage) TEMP_HIDDEN;
 bool NSDestroyObjectFileImage(NSObjectFileImage objectFileImage) TEMP_HIDDEN;

dyld3/APIs_macOS.cpp

@@ -38,6 +38,10 @@
 
 #include <algorithm>
 
+#if __has_feature(ptrauth_calls)
+  #include <ptrauth.h>
+#endif
+
 #include "dlfcn.h"
 
 #include "AllImages.h"
@@ -55,7 +59,7 @@ void                    parseDlHandle(void* h, const MachOLoaded** mh, bool* don
 
 
 // only in macOS and deprecated 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
 
 // macOS needs to support an old API that only works with fileype==MH_BUNDLE.
 // In this deprecated API (unlike dlopen), loading and linking are separate steps.
@@ -70,7 +74,7 @@ NSObjectFileImageReturnCode NSCreateObjectFileImageFromFile(const char* path, NS
 
     // verify path exists
      struct stat statbuf;
-    if ( ::stat(path, &statbuf) == -1 )
+    if ( dyld3::stat(path, &statbuf) == -1 )
         return NSObjectFileImageFailure;
 
     // create ofi that just contains path. NSLinkModule does all the work
@@ -98,13 +102,13 @@ NSObjectFileImageReturnCode NSCreateObjectFileImageFromMemory(const void* memIma
     bool usable = false;
     const MachOFile* mf = (MachOFile*)memImage;
     if ( mf->hasMachOMagic() && mf->isMachO(diag, memImageSize) ) {
-        usable = (gAllImages.archs().grade(mf->cputype, mf->cpusubtype) != 0);
+        usable = (gAllImages.archs().grade(mf->cputype, mf->cpusubtype, false) != 0);
     }
     else if ( const FatFile* ff = FatFile::isFatFile(memImage) ) {
         uint64_t sliceOffset;
         uint64_t sliceLen;
         bool     missingSlice;
-        if ( ff->isFatFileWithSlice(diag, memImageSize, gAllImages.archs(), sliceOffset, sliceLen, missingSlice) ) {
+        if ( ff->isFatFileWithSlice(diag, memImageSize, gAllImages.archs(), false, sliceOffset, sliceLen, missingSlice) ) {
             mf = (MachOFile*)((long)memImage+sliceOffset);
             if ( mf->isMachO(diag, sliceLen) ) {
                 usable = true;
@@ -112,7 +116,7 @@ NSObjectFileImageReturnCode NSCreateObjectFileImageFromMemory(const void* memIma
         }
     }
     if ( usable ) {
-        if ( !mf->supportsPlatform(Platform::macOS) )
+        if ( !mf->builtForPlatform(Platform::macOS) )
             usable = false;
     }
     if ( !usable ) {
@@ -343,10 +347,15 @@ const char* NSLibraryNameForModule(NSModule m)
 
 static bool flatFindSymbol(const char* symbolName, void** symbolAddress, const mach_header** foundInImageAtLoadAddress)
 {
+    // <rdar://problem/59265987> allow flat lookup to find "_memcpy" even though it is not implemented as that name in any dylib
+    MachOLoaded::DependentToMachOLoaded finder = ^(const MachOLoaded* mh, uint32_t depIndex) {
+        return gAllImages.findDependent(mh, depIndex);
+    };
+
     __block bool result = false;
     gAllImages.forEachImage(^(const LoadedImage& loadedImage, bool& stop) {
         bool resultPointsToInstructions = false;
-        if ( loadedImage.loadedAddress()->hasExportedSymbol(symbolName, nullptr, symbolAddress, &resultPointsToInstructions) ) {
+        if ( loadedImage.loadedAddress()->hasExportedSymbol(symbolName, finder, symbolAddress, &resultPointsToInstructions) ) {
             *foundInImageAtLoadAddress = loadedImage.loadedAddress();
             stop = true;
             result = true;
@@ -446,8 +455,35 @@ void* NSAddressOfSymbol(NSSymbol symbol)
 {
     log_apis("NSAddressOfSymbol(%p)\n", symbol);
 
+	if ( symbol == nullptr )
+		return nullptr;
+
     // in dyld 1.0, NSSymbol was a pointer to the nlist entry in the symbol table
-    return (void*)symbol;
+    void *result = (void*)symbol;
+
+#if __has_feature(ptrauth_calls)
+    __block const MachOLoaded *module = nullptr;
+    gAllImages.infoForImageMappedAt(symbol, ^(const LoadedImage& foundImage, uint8_t permissions) {
+        module = foundImage.loadedAddress();
+    });
+
+    int64_t slide = module->getSlide();
+    __block bool resultPointsToInstructions = false;
+    module->forEachSection(^(const MachOAnalyzer::SectionInfo& sectInfo, bool malformedSectionRange, bool& stop) {
+        uint64_t sectStartAddr = sectInfo.sectAddr + slide;
+        uint64_t sectEndAddr = sectStartAddr + sectInfo.sectSize;
+        if ( ((uint64_t)result >= sectStartAddr) && ((uint64_t)result < sectEndAddr) ) {
+            resultPointsToInstructions = (sectInfo.sectFlags & S_ATTR_PURE_INSTRUCTIONS) || (sectInfo.sectFlags & S_ATTR_SOME_INSTRUCTIONS);
+            stop = true;
+        }
+    });
+
+    if (resultPointsToInstructions) {
+        result = __builtin_ptrauth_sign_unauthenticated(result, ptrauth_key_asia, 0);
+    }
+#endif
+
+    return result;
 }
 
 NSModule NSModuleForSymbol(NSSymbol symbol)

dyld3/AllImages.cpp

@@ -31,6 +31,7 @@
 #include <libkern/OSAtomic.h>
 #include <uuid/uuid.h>
 #include <mach-o/dyld_images.h>
+#include <libc_private.h>
 
 #include <vector>
 #include <algorithm>
@@ -45,6 +46,7 @@
 #include "Closure.h"
 #include "ClosureBuilder.h"
 #include "ClosureFileSystemPhysical.h"
+#include "RootsChecker.h"
 
 #include "objc-shared-cache.h"
 
@@ -59,12 +61,8 @@ extern "C" void __cxa_finalize_ranges(const __cxa_range_t ranges[], unsigned int
 
 extern "C" int  __cxa_atexit(void (*func)(void *), void* arg, void* dso);
 
-#ifdef DARLING
-#define kdebug_is_enabled(...) 0
-#endif
 
-
-VIS_HIDDEN bool gUseDyld3 = false;
+VIS_HIDDEN void* __ptrauth_dyld_address_auth gUseDyld3 = nullptr;
 
 
 namespace dyld3 {
@@ -87,8 +85,7 @@ void AllImages::init(const closure::LaunchClosure* closure, const DyldSharedCach
     _dyldCachePath      = dyldCachePath;
 
     if ( _dyldCacheAddress ) {
-        const dyld_cache_mapping_info* const fileMappings = (dyld_cache_mapping_info*)((uint64_t)_dyldCacheAddress + _dyldCacheAddress->header.mappingOffset);
-        _dyldCacheSlide = (uint64_t)dyldCacheLoadAddress - fileMappings[0].address;
+        _dyldCacheSlide = (uint64_t)dyldCacheLoadAddress - dyldCacheLoadAddress->unslidLoadAddress();
         _imagesArrays.push_back(dyldCacheLoadAddress->cachedDylibsImageArray());
         if ( auto others = dyldCacheLoadAddress->otherOSImageArray() )
             _imagesArrays.push_back(others);
@@ -116,11 +113,25 @@ void AllImages::init(const closure::LaunchClosure* closure, const DyldSharedCach
     _processDOFs = Loader::dtraceUserProbesEnabled();
 }
 
-void AllImages::setProgramVars(ProgramVars* vars)
+void AllImages::setProgramVars(ProgramVars* vars, bool keysOff, bool osBinariesOnly)
 {
     _programVars = vars;
-    const dyld3::MachOFile* mf = (dyld3::MachOFile*)_programVars->mh;
-    _archs = &GradedArchs::forCurrentOS(mf);
+    _archs = &GradedArchs::forCurrentOS(keysOff, osBinariesOnly);
+}
+
+void AllImages::setLaunchMode(uint32_t flags)
+{
+    _launchMode = flags;
+}
+
+AllImages::MainFunc AllImages::getDriverkitMain()
+{
+    return _driverkitMain;
+}
+
+void AllImages::setDriverkitMain(MainFunc mainFunc)
+{
+    _driverkitMain = mainFunc;
 }
 
 void AllImages::setRestrictions(bool allowAtPaths, bool allowEnvPaths)
@@ -221,7 +232,7 @@ void AllImages::mirrorToOldAllImageInfos()
         // <radr://problem/42668846> update UUID array if needed
         uint32_t nonCachedCount = 1; // always add dyld
         for (const LoadedImage& li : _loadedImages) {
-            if ( !li.loadedAddress()->inDyldCache() )
+            if ( _oldAllImageInfos->processDetachedFromSharedRegion || !li.loadedAddress()->inDyldCache())
                 ++nonCachedCount;
         }
         if ( nonCachedCount != _oldAllImageInfos->uuidArrayCount ) {
@@ -242,7 +253,7 @@ void AllImages::mirrorToOldAllImageInfos()
             dyldMF->getUuid(_oldUUIDArray[0].imageUUID);
             index = 1;
             for (const LoadedImage& li : _loadedImages) {
-                if ( !li.loadedAddress()->inDyldCache() ) {
+                if ( _oldAllImageInfos->processDetachedFromSharedRegion || !li.loadedAddress()->inDyldCache() ) {
                     _oldUUIDArray[index].imageLoadAddress = li.loadedAddress();
                     li.loadedAddress()->getUuid(_oldUUIDArray[index].imageUUID);
                     ++index;
@@ -260,13 +271,6 @@ void AllImages::addImages(const Array<LoadedImage>& newImages)
     // copy into _loadedImages
     withWriteLock(^(){
         _loadedImages.append(newImages);
-        // if any image not in the shared cache added, recompute bounds
-        for (const LoadedImage& li : newImages) {
-            if ( !((MachOAnalyzer*)li.loadedAddress())->inDyldCache() ) {
-                recomputeBounds();
-                break;
-            }
-        }
     });
 }
 
@@ -315,6 +319,13 @@ void AllImages::runImageNotifiers(const Array<LoadedImage>& newImages)
         _oldAllImageInfos->notification(dyld_image_adding, count, oldDyldInfo);
     }
 
+    // if any image not in the shared cache added, recompute bounds
+    for (const LoadedImage& li : newImages) {
+        if ( !((MachOAnalyzer*)li.loadedAddress())->inDyldCache() ) {
+            recomputeBounds();
+            break;
+        }
+    }
 
     // update immutable ranges
     for (const LoadedImage& li : newImages) {
@@ -322,7 +333,7 @@ void AllImages::runImageNotifiers(const Array<LoadedImage>& newImages)
             uintptr_t baseAddr = (uintptr_t)li.loadedAddress();
             li.image()->forEachDiskSegment(^(uint32_t segIndex, uint32_t fileOffset, uint32_t fileSize, int64_t vmOffset, uint64_t vmSize, uint8_t permissions, bool laterReadOnly, bool &stop) {
                 if ( (permissions & (VM_PROT_READ|VM_PROT_WRITE)) == VM_PROT_READ ) {
-                    addImmutableRange(baseAddr + vmOffset, baseAddr + vmOffset + vmSize);
+                    addImmutableRange(baseAddr + (uintptr_t)vmOffset, (uintptr_t)(baseAddr + vmOffset + vmSize));
                 }
             });
         }
@@ -352,7 +363,7 @@ void AllImages::runImageNotifiers(const Array<LoadedImage>& newImages)
             image->getUuid(uuid);
             fsid_t       fsid = {{ 0, 0 }};
             fsobj_id_t   fsobjid = { 0, 0 };
-            if ( !li.loadedAddress()->inDyldCache() && (stat(path, &stat_buf) == 0) ) {
+            if ( !li.loadedAddress()->inDyldCache() && (dyld3::stat(path, &stat_buf) == 0) ) {
                 fsobjid = *(fsobj_id_t*)&stat_buf.st_ino;
                 fsid    = {{ stat_buf.st_dev, 0 }};
             }
@@ -469,7 +480,7 @@ void AllImages::removeImages(const Array<LoadedImage>& unloadImages)
             image->getUuid(uuid);
             fsid_t       fsid = {{ 0, 0 }};
             fsobj_id_t   fsobjid = { 0, 0 };
-            if ( stat(path, &stat_buf) == 0 ) {
+            if ( dyld3::stat(path, &stat_buf) == 0 ) {
                 fsobjid = *(fsobj_id_t*)&stat_buf.st_ino;
                 fsid    = {{ stat_buf.st_dev, 0 }};
             }
@@ -922,6 +933,12 @@ void AllImages::breadthFirstRecurseDependents(Array<closure::ImageNum>& visited,
         if ( !findImageNum(depImageNum, depLi) )
             return;
         handler(depLi, depStop);
+        // <rdar://58466613> if there is an override of some dyld cache dylib, we need to store the override ImageNum in the visited set
+        if ( depImageNum != depLi.image()->imageNum() ) {
+            depImageNum = depLi.image()->imageNum();
+            if ( visited.contains(depImageNum) )
+                return;
+        }
         visited.push_back(depImageNum);
         if ( depStop ) {
             stopped = true;
@@ -959,7 +976,7 @@ const MachOLoaded* AllImages::mainExecutable() const
 const closure::Image* AllImages::mainExecutableImage() const
 {
     assert(_mainClosure != nullptr);
-    return _mainClosure->images()->imageForNum(_mainClosure->topImage());
+    return _mainClosure->images()->imageForNum(_mainClosure->topImageNum());
 }
 
 void AllImages::setMainPath(const char* path )
@@ -969,7 +986,7 @@ void AllImages::setMainPath(const char* path )
 
 const char* AllImages::imagePath(const closure::Image* image) const
 {
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
+#if TARGET_OS_IPHONE
     // on iOS and watchOS, apps may be moved on device after closure built
 	if ( _mainExeOverridePath != nullptr ) {
         if ( image == mainExecutableImage() )
@@ -980,14 +997,7 @@ const char* AllImages::imagePath(const closure::Image* image) const
 }
 
 dyld_platform_t AllImages::platform() const {
-    if (oldAllImageInfo()->version >= 16) { return (dyld_platform_t)oldAllImageInfo()->platform; }
-
-    __block dyld_platform_t result;
-    // FIXME: Remove this once we only care about version 16 or greater all image infos
-    dyld_get_image_versions(mainExecutable(), ^(dyld_platform_t platform, uint32_t sdk_version, uint32_t min_version) {
-        result = platform;
-    });
-    return result;
+    return (dyld_platform_t)oldAllImageInfo()->platform;
 }
 
 const GradedArchs& AllImages::archs() const
@@ -1029,7 +1039,7 @@ void AllImages::decRefCount(const mach_header* loadAddress)
 }
 
 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
 NSObjectFileImage AllImages::addNSObjectFileImage(const OFIInfo& image)
 {
     __block uint64_t imageNum = 0;
@@ -1202,13 +1212,15 @@ void Reaper::finalizeDeadImages()
 
 void Reaper::runTerminators(const LoadedImage& li)
 {
+    // <rdar://problem/71820555> Don't run static terminator for arm64e
+    const MachOAnalyzer* ma = (MachOAnalyzer*)li.loadedAddress();
+    if ( ma->isArch("arm64e") )
+        return;
+
     if ( li.image()->hasTerminators() ) {
         typedef void (*Terminator)();
         li.image()->forEachTerminator(li.loadedAddress(), ^(const void* terminator) {
             Terminator termFunc = (Terminator)terminator;
-#if __has_feature(ptrauth_calls)
-            termFunc = (Terminator)__builtin_ptrauth_sign_unauthenticated((void*)termFunc, 0, 0);
-#endif
             termFunc();
             log_initializers("dyld: called static terminator %p in %s\n", termFunc, li.image()->path());
         });
@@ -1470,13 +1482,20 @@ void AllImages::setObjCNotifiers(_dyld_objc_notify_mapped map, _dyld_objc_notify
     }
 }
 
-void AllImages::applyInterposingToDyldCache(const closure::Closure* closure)
+void AllImages::applyInterposingToDyldCache(const closure::Closure* closure, mach_port_t mach_task_self)
 {
     dyld3::ScopedTimer timer(DBG_DYLD_TIMING_APPLY_INTERPOSING, 0, 0, 0);
     const uintptr_t                 cacheStart              = (uintptr_t)_dyldCacheAddress;
     __block closure::ImageNum       lastCachedDylibImageNum = 0;
     __block const closure::Image*   lastCachedDylibImage    = nullptr;
     __block bool                    suspendedAccounting     = false;
+
+    if ( closure->findAttributePayload(closure::TypedBytes::Type::cacheOverrides) == nullptr )
+        return;
+
+    // make the cache writable for this block
+    DyldSharedCache::DataConstScopedWriter patcher(_dyldCacheAddress, mach_task_self, (DyldSharedCache::DataConstLogFunc)&log_segments);
+
     closure->forEachPatchEntry(^(const closure::Closure::PatchEntry& entry) {
         if ( entry.overriddenDylibInCache != lastCachedDylibImageNum ) {
             lastCachedDylibImage    = closure::ImageArray::findImage(imagesArrays(), entry.overriddenDylibInCache);
@@ -1491,7 +1510,9 @@ void AllImages::applyInterposingToDyldCache(const closure::Closure* closure)
         LoadedImage foundImage;
         switch ( entry.replacement.image.kind ) {
             case closure::Image::ResolvedSymbolTarget::kindImage:
-                assert(findImageNum(entry.replacement.image.imageNum, foundImage));
+                if ( !findImageNum(entry.replacement.image.imageNum, foundImage) ) {
+                    abort_report_np("cannot find replacement imageNum=0x%04X when patching cache to override imageNum=0x%04X\n", entry.replacement.image.imageNum, entry.overriddenDylibInCache);
+                }
                 newValue = (uintptr_t)(foundImage.loadedAddress()) + (uintptr_t)entry.replacement.image.offset;
                 break;
             case closure::Image::ResolvedSymbolTarget::kindSharedCache:
@@ -1664,9 +1685,14 @@ void AllImages::runAllInitializersInImage(const closure::Image* image, const Mac
     });
 }
 
-const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool rtldNoLoad, bool rtldLocal, bool rtldNoDelete, bool rtldNow, bool fromOFI, const void* callerAddress)
+// Note this is noinline to avoid having too much stack used if loadImage has to call due to an invalid closure
+__attribute__((noinline))
+const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool rtldNoLoad, bool rtldLocal,
+                                     bool rtldNoDelete, bool rtldNow, bool fromOFI, const void* callerAddress,
+                                     bool canUsePrebuiltSharedCacheClosure)
 {
     bool sharedCacheFormatCompatible = (_dyldCacheAddress != nullptr) && (_dyldCacheAddress->header.formatVersion == dyld3::closure::kFormatVersion);
+    canUsePrebuiltSharedCacheClosure &= sharedCacheFormatCompatible;
 
     // quick check if path is in shared cache and already loaded
     if ( _dyldCacheAddress != nullptr ) {
@@ -1690,7 +1716,7 @@ const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool r
                  sharedCacheFormatCompatible ) {
                 const dyld3::closure::ImageArray* images = _dyldCacheAddress->cachedDylibsImageArray();
                 const dyld3::closure::Image* image = images->imageForNum(dyldCacheImageIndex+1);
-                return loadImage(diag, image->imageNum(), nullptr, rtldLocal, rtldNoDelete, rtldNow, fromOFI);
+                return loadImage(diag, path, image->imageNum(), nullptr, rtldLocal, rtldNoDelete, rtldNow, fromOFI, callerAddress);
             }
         }
     }
@@ -1712,10 +1738,11 @@ const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool r
     // Then try again with forcing a new closure
     for (bool canUseSharedCacheClosure : { true, false }) {
         // We can only use a shared cache closure if the shared cache format is the same as libdyld.
-        canUseSharedCacheClosure &= sharedCacheFormatCompatible;
+        canUseSharedCacheClosure &= canUsePrebuiltSharedCacheClosure;
         closure::FileSystemPhysical fileSystem(nullptr, nullptr, _allowEnvPaths);
+        RootsChecker rootsChecker;
         closure::ClosureBuilder::AtPath atPathHanding = (_allowAtPaths ? closure::ClosureBuilder::AtPath::all : closure::ClosureBuilder::AtPath::onlyInRPaths);
-        closure::ClosureBuilder cb(_nextImageNum, fileSystem, _dyldCacheAddress, true, *_archs, closure::gPathOverrides, atPathHanding);
+        closure::ClosureBuilder cb(_nextImageNum, fileSystem, rootsChecker, _dyldCacheAddress, true, *_archs, closure::gPathOverrides, atPathHanding, true, nullptr, (dyld3::Platform)platform());
         newClosure = cb.makeDlopenClosure(path, _mainClosure, _loadedImages.array(), callerImageNum, rtldNoLoad, rtldNow, canUseSharedCacheClosure, &topImageNum);
         if ( newClosure == closure::ClosureBuilder::sRetryDlopenClosure ) {
             log_apis("   dlopen: closure builder needs to retry: %s\n", path);
@@ -1739,7 +1766,7 @@ const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool r
         if ( const closure::ImageArray* newArray = newClosure->images() ) {
             appendToImagesArray(newArray);
         }
-        log_apis("   dlopen: made closure: %p\n", newClosure);
+        log_apis("   dlopen: made %s closure: %p\n", newClosure->topImage()->variantString(), newClosure);
     }
 
     // if already loaded, just bump refCount and return
@@ -1775,14 +1802,16 @@ const MachOLoaded* AllImages::dlopen(Diagnostics& diag, const char* path, bool r
         }
     }
 
-    return loadImage(diag, topImageNum, newClosure, rtldLocal, rtldNoDelete, rtldNow, fromOFI);
+    return loadImage(diag, path, topImageNum, newClosure, rtldLocal, rtldNoDelete, rtldNow, fromOFI, callerAddress);
 }
 
 // Note this is noinline to avoid having too much stack used in the parent
 // dlopen method
 __attribute__((noinline))
-const MachOLoaded* AllImages::loadImage(Diagnostics& diag, closure::ImageNum topImageNum, const closure::DlopenClosure* newClosure,
-                                        bool rtldLocal, bool rtldNoDelete, bool rtldNow, bool fromOFI) {
+const MachOLoaded* AllImages::loadImage(Diagnostics& diag, const char* path,
+                                        closure::ImageNum topImageNum, const closure::DlopenClosure* newClosure,
+                                        bool rtldLocal, bool rtldNoDelete, bool rtldNow, bool fromOFI,
+                                        const void* callerAddress) {
     // Note this array is used as the storage to Loader so needs to be at least
     // large enough to handle whatever total number of images we need to do the dlopen
     STACK_ALLOC_OVERFLOW_SAFE_ARRAY(LoadedImage, newImages, 1024);
@@ -1793,9 +1822,10 @@ const MachOLoaded* AllImages::loadImage(Diagnostics& diag, closure::ImageNum top
     dyld3::Array<dyld3::closure::Image::ObjCSelectorImage>  selectorImages;
 
     // run loader to load all new images
+    RootsChecker rootsChecker;
     Loader loader(_loadedImages.array(), newImages, _dyldCacheAddress, imagesArrays(),
-                  selectorOpt, selectorImages,
-                  &dyld3::log_loads, &dyld3::log_segments, &dyld3::log_fixups, &dyld3::log_dofs);
+                  selectorOpt, selectorImages, rootsChecker, (dyld3::Platform)platform(),
+                  &dyld3::log_loads, &dyld3::log_segments, &dyld3::log_fixups, &dyld3::log_dofs, !rtldNow);
 
     // find Image* for top image, look in new closure first
     const closure::Image* topImage = nullptr;
@@ -1805,9 +1835,9 @@ const MachOLoaded* AllImages::loadImage(Diagnostics& diag, closure::ImageNum top
         topImage = closure::ImageArray::findImage(imagesArrays(), topImageNum);
     if ( newClosure == nullptr ) {
         if ( topImageNum < dyld3::closure::kLastDyldCacheImageNum )
-            log_apis("   dlopen: using image in dyld shared cache %p\n", topImage);
+            log_apis("   dlopen: using pre-built %s dlopen closure from dyld shared cache %p\n", topImage->variantString(), topImage);
         else
-            log_apis("   dlopen: using pre-built dlopen closure %p\n", topImage);
+            log_apis("   dlopen: using pre-built %s dlopen closure %p\n", topImage->variantString(), topImage);
     }
     LoadedImage topLoadedImage = LoadedImage::make(topImage);
     if ( rtldLocal && !topImage->inDyldCache() )
@@ -1822,9 +1852,19 @@ const MachOLoaded* AllImages::loadImage(Diagnostics& diag, closure::ImageNum top
     loader.completeAllDependents(diag, someCacheImageOverridden);
     if ( diag.hasError() )
         return nullptr;
-    loader.mapAndFixupAllImages(diag, _processDOFs, fromOFI);
-    if ( diag.hasError() )
+    bool closureOutOfDate;
+    bool recoverable;
+    loader.mapAndFixupAllImages(diag, _processDOFs, fromOFI, &closureOutOfDate, &recoverable);
+    if ( diag.hasError() ) {
+        // If we used a pre-built shared cache closure, and now found that it was out of date,
+        // try again and rebuild a new closure
+        // Note, newClosure is null in the case where we used a prebuilt closure
+        if ( closureOutOfDate && recoverable && (newClosure == nullptr) ) {
+            diag.clearError();
+            return dlopen(diag, path, false /* rtldNoLoad */, rtldLocal, rtldNoDelete, rtldNow, fromOFI, callerAddress, false);
+        }
         return nullptr;
+    }
 
     // Record if we had a root
     _someImageOverridden |= someCacheImageOverridden;
@@ -1845,7 +1885,7 @@ const MachOLoaded* AllImages::loadImage(Diagnostics& diag, closure::ImageNum top
 
     // if closure adds images that override dyld cache, patch cache
     if ( newClosure != nullptr )
-        applyInterposingToDyldCache(newClosure);
+        applyInterposingToDyldCache(newClosure, mach_task_self());
 
     runImageCallbacks(newImages);
 

dyld3/AllImages.h

@@ -30,13 +30,16 @@
 #include <os/lock_private.h>
 #include <mach-o/dyld_priv.h>
 
+#include <sys/types.h>
+
 #include "Closure.h"
 #include "Loading.h"
 #include "MachOLoaded.h"
 #include "DyldSharedCache.h"
+#include "PointerAuth.h"
 
 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
 // only in macOS and deprecated 
 struct VIS_HIDDEN OFIInfo
 {
@@ -56,6 +59,7 @@ public:
     typedef void                (*NotifyFunc)(const mach_header* mh, intptr_t slide);
     typedef void                (*LoadNotifyFunc)(const mach_header* mh, const char* path, bool unloadable);
     typedef void                (*BulkLoadNotifier)(unsigned count, const mach_header* mhs[], const char* paths[]);
+    typedef void                (*MainFunc)(void);
 
     void                        init(const closure::LaunchClosure* closure, const DyldSharedCache* dyldCacheLoadAddress, const char* dyldCachePath,
                                      const Array<LoadedImage>& initialImages);
@@ -63,13 +67,14 @@ public:
     void                        setHasCacheOverrides(bool someCacheImageOverriden);
     bool                        hasCacheOverrides() const;
     void                        setMainPath(const char* path);
+    void                        setLaunchMode(uint32_t flags);
     void                        applyInitialImages();
 
     void                        addImages(const Array<LoadedImage>& newImages);
     void                        removeImages(const Array<LoadedImage>& unloadImages);
     void                        runImageNotifiers(const Array<LoadedImage>& newImages);
     void                        runImageCallbacks(const Array<LoadedImage>& newImages);
-    void                        applyInterposingToDyldCache(const closure::Closure* closure);
+    void                        applyInterposingToDyldCache(const closure::Closure* closure, mach_port_t mach_task_self);
     void                        runStartupInitialzers();
     void                        runInitialzersBottomUp(const closure::Image* topImage);
     void                        runLibSystemInitializer(LoadedImage& libSystem);
@@ -99,6 +104,7 @@ public:
     const char*                 imagePath(const closure::Image*) const;
     dyld_platform_t             platform() const;
     const GradedArchs&          archs() const;
+    uint32_t                    launchMode() const { return _launchMode; }
     
     const Array<const closure::ImageArray*>& imagesArrays();
 
@@ -119,7 +125,7 @@ public:
     void                        notifyMonitorLoads(const Array<LoadedImage>& newImages);
     void                        notifyMonitorUnloads(const Array<LoadedImage>& unloadingImages);
 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
     NSObjectFileImage           addNSObjectFileImage(const OFIInfo&);
     void                        removeNSObjectFileImage(NSObjectFileImage);
     bool                        forNSObjectFileImage(NSObjectFileImage imageHandle,
@@ -132,7 +138,12 @@ public:
     void                        forEachObjCProtocol(const char* protocolName,
                                                     void (^callback)(void* protocolPtr, bool isLoaded, bool* stop)) const;
     
-    const MachOLoaded*          dlopen(Diagnostics& diag, const char* path, bool rtldNoLoad, bool rtldLocal, bool rtldNoDelete, bool forceBindLazies, bool fromOFI, const void* callerAddress);
+    MainFunc                    getDriverkitMain();
+    void                        setDriverkitMain(MainFunc mainFunc);
+
+    const MachOLoaded*          dlopen(Diagnostics& diag, const char* path, bool rtldNoLoad, bool rtldLocal, bool rtldNoDelete,
+                                       bool forceBindLazies, bool fromOFI, const void* callerAddress,
+                                       bool canUsePrebuiltSharedCacheClosure = true);
 
     struct ProgramVars
     {
@@ -142,7 +153,7 @@ public:
         const char***      environPtr;
         const char**       __prognamePtr;
     };
-    void                    setProgramVars(ProgramVars* vars);
+    void                    setProgramVars(ProgramVars* vars, bool keysOff, bool platformBinariesOnly);
 
     // Note these are to be used exclusively by forking
     void takeLockBeforeFork();
@@ -178,8 +189,10 @@ private:
         }                               array[2];  // programs with only main-exe and dyld cache fit in here
     };
 
-    const MachOLoaded*          loadImage(Diagnostics& diag, closure::ImageNum topImageNum, const closure::DlopenClosure* newClosure,
-                                          bool rtldLocal, bool rtldNoDelete, bool rtldNow, bool fromOFI);
+    const MachOLoaded*          loadImage(Diagnostics& diag, const char* path,
+                                          closure::ImageNum topImageNum, const closure::DlopenClosure* newClosure,
+                                          bool rtldLocal, bool rtldNoDelete, bool rtldNow, bool fromOFI,
+                                          const void* callerAddress);
 
 
     typedef void (*Initializer)(int argc, const char* argv[], char* envp[], const char* apple[], const ProgramVars* vars);
@@ -203,17 +216,13 @@ private:
     uintptr_t                   resolveTarget(closure::Image::ResolvedSymbolTarget target) const;
     void                        addImmutableRange(uintptr_t start, uintptr_t end);
 
-    void                        constructMachPorts(int slot);
-    void                        teardownMachPorts(int slot);
-    void                        forEachPortSlot(void (^callback)(int slot));
-    void                        sendMachMessage(int slot, mach_msg_id_t msg_id, mach_msg_header_t* msg_buffer, mach_msg_size_t msg_size);
-    void                        notifyMonitoringDyld(bool unloading, const Array<LoadedImage>& images);
-
     static void                 runAllStaticTerminatorsHelper(void*);
 
     typedef closure::ImageArray  ImageArray;
 
-    const closure::LaunchClosure*           _mainClosure         = nullptr;
+    typedef const closure::LaunchClosure* __ptrauth_dyld_address_auth MainClosurePtrType;
+
+    MainClosurePtrType                      _mainClosure         = nullptr;
     const DyldSharedCache*                  _dyldCacheAddress    = nullptr;
     const char*                             _dyldCachePath       = nullptr;
     uint64_t                                _dyldCacheSlide      = 0;
@@ -236,8 +245,11 @@ private:
     bool                                    _allowAtPaths        = false;
     bool                                    _allowEnvPaths       = false;
     bool                                    _someImageOverridden = false;
+    uint32_t                                _launchMode          = 0;
     uintptr_t                               _lowestNonCached     = 0;
     uintptr_t                               _highestNonCached    = UINTPTR_MAX;
+    
+    MainFunc                               _driverkitMain       = nullptr;
 #ifdef OS_UNFAIR_RECURSIVE_LOCK_INIT
     mutable os_unfair_recursive_lock        _globalLock      = OS_UNFAIR_RECURSIVE_LOCK_INIT;
 #else
@@ -250,7 +262,7 @@ private:
     GrowableArray<BulkLoadNotifier, 2, 2>   _loadBulkNotifiers;
     GrowableArray<DlopenCount, 4, 4>        _dlopenRefCounts;
     GrowableArray<LoadedImage, 16>          _loadedImages;
-#if __MAC_OS_X_VERSION_MIN_REQUIRED
+#if TARGET_OS_OSX
     uint64_t                                _nextObjectFileImageNum = 0;
     GrowableArray<OFIInfo, 4, 1>            _objectFileImages;
 #endif

dyld3/Array.h

@@ -26,9 +26,10 @@
 
 #include <algorithm>
 #include <stdint.h>
+#include <assert.h>
 #include <stddef.h>
 #include <mach/mach.h>
-#include <assert.h>
+#include <TargetConditionals.h>
 
 #if !TARGET_OS_DRIVERKIT && (BUILDING_LIBDYLD || BUILDING_DYLD)
   #include <CrashReporterClient.h>
@@ -160,13 +161,13 @@ inline void OverflowSafeArray<T,MAXCOUNT>::growTo(uintptr_t n)
 #if BUILDING_LIBDYLD
         //FIXME We should figure out a way to do this in dyld
         char crashString[256];
-        snprintf(crashString, 256, "OverflowSafeArray failed to allocate %lu bytes, vm_allocate returned: %d\n",
-                _overflowBufferSize, kr);
+        snprintf(crashString, 256, "OverflowSafeArray failed to allocate %llu bytes, vm_allocate returned: %d\n",
+                (uint64_t)_overflowBufferSize, kr);
         CRSetCrashLogMessage(crashString);
 #endif
         assert(0);
     }
-    ::memcpy((void*)_overflowBuffer, this->_elements, this->_usedCount*sizeof(T));
+    ::memcpy((void*)_overflowBuffer, (void*)this->_elements, this->_usedCount*sizeof(T));
     this->_elements = (T*)_overflowBuffer;
     this->_allocCount = _overflowBufferSize / sizeof(T);
 

dyld3/BootArgs.cpp

@@ -1,9 +1,25 @@
-//
-//  BootArgs.cpp
-//  dyld
-//
-//  Created by Louis Gerbarg on 11/14/18.
-//
+/*
+* Copyright (c) 2017 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* This file contains Original Code and/or Modifications of Original Code
+* as defined in and that are subject to the Apple Public Source License
+* Version 2.0 (the 'License'). You may not use this file except in
+* compliance with the License. Please obtain a copy of the License at
+* http://www.opensource.apple.com/apsl/ and read it before using this
+* file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+* Please see the License for the specific language governing rights and
+* limitations under the License.
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
 
 #include <sys/types.h>
 #include <sys/sysctl.h>
@@ -12,53 +28,36 @@
 #include "Loading.h" // For internalInstall()
 #include "BootArgs.h"
 
-namespace dyld3 {
-/*
-* Checks to see if there are any args that impact dyld. These args
-* can be set sevaral ways. These will only be honored on development
-* and Apple Internal builds.
-*/
-bool BootArgs::contains(const char* arg)
-{
-    //FIXME: Use strnstr(). Unfortunately we are missing an imp in libc.a
-#if TARGET_OS_SIMULATOR
-    return false;
-#else
-    // don't check for boot-args on customer installs
-    if ( !internalInstall() )
-        return false;
-
-    // get length of full boot-args string
-    size_t len;
-    if ( sysctlbyname("kern.bootargs", NULL, &len, NULL, 0) != 0 )
-        return false;
-
-    // get copy of boot-args string
-    char bootArgsBuffer[len];
-    if ( sysctlbyname("kern.bootargs", bootArgsBuffer, &len, NULL, 0) != 0 )
-        return false;
-
-    // return true if 'arg' is a sub-string of boot-args
-    return (strstr(bootArgsBuffer, arg) != nullptr);
+namespace dyld {
+#if defined(__x86_64__) && !TARGET_OS_SIMULATOR
+    bool isTranslated();
 #endif
 }
 
+namespace dyld3 {
+
 uint64_t BootArgs::_flags = 0;
 
 bool BootArgs::forceCustomerCache() {
     return (_flags & kForceCustomerCacheMask);
 }
 
+bool BootArgs::forceDevelopmentCache() {
+    return (_flags & kForceDevelopmentCacheMask);
+}
+
 bool BootArgs::forceDyld2() {
+#if defined(__x86_64__) && !TARGET_OS_SIMULATOR
+    if (dyld::isTranslated()) { return true; }
+#endif
     // If both force dyld2 and dyld3 are set then use dyld3
     if (_flags & kForceDyld3CacheMask) { return false; }
     if (_flags & kForceDyld2CacheMask) { return true; }
-    if (contains("force_dyld2=1")) { return true; }
     return false;
 }
 
 bool BootArgs::forceDyld3() {
-    return ((_flags & kForceDyld3CacheMask) || contains("force_dyld3=1"));
+    return (_flags & kForceDyld3CacheMask);
 }
 
 bool BootArgs::enableDyldTestMode() {
@@ -69,6 +68,14 @@ bool BootArgs::enableCompactImageInfo() {
     return (_flags & kEnableCompactImageInfoMask);
 }
 
+bool BootArgs::forceReadOnlyDataConst() {
+    return (_flags & kForceReadOnlyDataConstMask);
+}
+
+bool BootArgs::forceReadWriteDataConst() {
+    return (_flags & kForceReadWriteDataConstMask);
+}
+
 void BootArgs::setFlags(uint64_t flags) {
 #if TARGET_IPHONE_SIMULATOR
     return;

dyld3/BootArgs.h

@@ -1,9 +1,26 @@
-//
-//  BootArgs.hpp
-//  dyld
-//
-//  Created by Louis Gerbarg on 11/14/18.
-//
+/*
+* Copyright (c) 2017 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* This file contains Original Code and/or Modifications of Original Code
+* as defined in and that are subject to the Apple Public Source License
+* Version 2.0 (the 'License'). You may not use this file except in
+* compliance with the License. Please obtain a copy of the License at
+* http://www.opensource.apple.com/apsl/ and read it before using this
+* file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+* Please see the License for the specific language governing rights and
+* limitations under the License.
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
+
 
 #ifndef __DYLD_BOOTARGS_H__
 #define __DYLD_BOOTARGS_H__
@@ -15,19 +32,24 @@
 namespace dyld3 {
 #if BUILDING_DYLD
     struct VIS_HIDDEN BootArgs {
-        static bool contains(const char* arg);
         static bool forceCustomerCache();
+        static bool forceDevelopmentCache();
         static bool forceDyld2();
         static bool forceDyld3();
         static bool enableDyldTestMode();
         static bool enableCompactImageInfo();
+        static bool forceReadOnlyDataConst();
+        static bool forceReadWriteDataConst();
         static void setFlags(uint64_t flags);
     private:
         static const uint64_t kForceCustomerCacheMask = 1<<0;
         static const uint64_t kDyldTestModeMask = 1<<1;
+        static const uint64_t kForceDevelopmentCacheMask = 1<<2;
         static const uint64_t kForceDyld2CacheMask = 1<<15;
         static const uint64_t kForceDyld3CacheMask = 1<<16;
         static const uint64_t kEnableCompactImageInfoMask = 1<<17;
+        static const uint64_t kForceReadOnlyDataConstMask = 1<<18;
+        static const uint64_t kForceReadWriteDataConstMask = 1<<19;
         //FIXME: Move this into __DATA_CONST once it is enabled for dyld
         static uint64_t _flags;
     };

dyld3/Closure.cpp

@@ -31,6 +31,7 @@
 #include <sys/stat.h>
 #include <sys/sysctl.h>
 #include <System/machine/cpu_capabilities.h>
+#include <_simple.h>
 
 extern "C" {
   #include <corecrypto/ccdigest.h>
@@ -142,7 +143,7 @@ bool Image::representsImageNum(ImageNum num) const
         return true;
     if ( !flags.isDylib )
         return false;
-    if ( flags.inDyldCache )
+    if ( !flags.hasOverrideImageNum )
         return false;
     ImageNum cacheImageNum;
     if ( isOverrideOfDyldCacheImage(cacheImageNum) )
@@ -573,6 +574,16 @@ bool Image::hasPrecomputedObjC() const
     return getFlags().hasPrecomputedObjC;
 }
 
+bool Image::fixupsNotEncoded() const
+{
+    return getFlags().fixupsNotEncoded;
+}
+
+bool Image::rebasesNotEncoded() const
+{
+    return getFlags().rebasesNotEncoded;
+}
+
 void Image::forEachTerminator(const void* imageLoadAddress, void (^handler)(const void* terminator)) const
 {
     uint32_t size;
@@ -643,17 +654,10 @@ void Image::forEachFixup(void (^rebase)(uint64_t imageOffsetToRebase, bool& stop
     if ( stop )
         return;
 
-    for (const Image::BindPattern& bindPat : bindFixups()) {
-        uint64_t curBindOffset = bindPat.startVmOffset;
-        for (uint16_t i=0; i < bindPat.repeatCount; ++i) {
-            bind(curBindOffset, bindPat.target, stop);
-            curBindOffset += (pointerSize * (1 + bindPat.skipCount));
+    stop = this->forEachBind(bind);
     if ( stop )
-                break;
-        }
-        if ( stop )
-            break;
-    }
+        return;
+
 
     if (hasChainedFixups())
         chainedFixups(chainedStartsOffset(), chainedTargets(), stop);
@@ -729,6 +733,24 @@ void Image::forEachFixup(void (^rebase)(uint64_t imageOffsetToRebase, bool& stop
     }
 }
 
+bool Image::forEachBind(void (^bind)(uint64_t imageOffsetToBind, ResolvedSymbolTarget bindTarget, bool& stop)) const
+{
+    const uint32_t pointerSize = is64() ? 8 : 4;
+    bool stop = false;
+    for (const Image::BindPattern& bindPat : bindFixups()) {
+        uint64_t curBindOffset = bindPat.startVmOffset;
+        for (uint16_t i=0; i < bindPat.repeatCount; ++i) {
+            bind(curBindOffset, bindPat.target, stop);
+            curBindOffset += (pointerSize * (1 + bindPat.skipCount));
+            if ( stop )
+                break;
+        }
+        if ( stop )
+            break;
+    }
+    return stop;
+}
+
 void Image::forEachTextReloc(void (^rebase)(uint32_t imageOffsetToRebase, bool& stop),
                              void (^bind)(uint32_t imageOffsetToBind, ResolvedSymbolTarget bindTarget, bool& stop)) const
 {
@@ -888,6 +910,11 @@ void Image::forEachImageToInitBefore(void (^handler)(ImageNum imageToInit, bool&
     }
 }
 
+const char* Image::variantString() const
+{
+    return (this->fixupsNotEncoded() ? "minimal" : "full");
+}
+
 ////////////////////////////  ImageArray ////////////////////////////////////////
 
 size_t ImageArray::size() const
@@ -954,7 +981,11 @@ const Image* ImageArray::imageForNum(ImageNum num) const
 
 const Image* ImageArray::findImage(const Array<const ImageArray*> imagesArrays, ImageNum imageNum)
 {
-   for (const ImageArray* ia : imagesArrays) {
+    // Search image arrays backwards as the main closure, or dlopen closures, may rebuild closures
+    // for shared cache images which are not roots, but whose initialisers must rebuild as they depend
+    // on a root
+    for (uintptr_t index = imagesArrays.count(); index > 0; --index) {
+        const ImageArray* ia = imagesArrays[index - 1];
         if ( const Image* result = ia->imageForNum(imageNum) )
             return result;
     }
@@ -986,7 +1017,7 @@ const ImageArray* Closure::images() const
     return (ImageArray*)result;
 }
 
-ImageNum Closure::topImage() const
+ImageNum Closure::topImageNum() const
 {
     uint32_t size;
     const ImageNum* top = (ImageNum*)findAttributePayload(Type::topImage, &size);
@@ -995,6 +1026,13 @@ ImageNum Closure::topImage() const
     return *top;
 }
 
+const Image* Closure::topImage() const
+{
+    ImageNum                            imageNum      = this->topImageNum();
+    const dyld3::closure::ImageArray*   closureImages = this->images();
+    return closureImages->imageForNum(imageNum);
+}
+
 void Closure::forEachPatchEntry(void (^handler)(const PatchEntry& entry)) const
 {
 	forEachAttributePayload(Type::cacheOverrides, ^(const void* payload, uint32_t size, bool& stop) {
@@ -1021,6 +1059,7 @@ void Closure::deallocate() const
     ::vm_deallocate(mach_task_self(), (long)this, size());
 }
 
+
 ////////////////////////////  LaunchClosure ////////////////////////////////////////
 
 void LaunchClosure::forEachMustBeMissingFile(void (^handler)(const char* path, bool& stop)) const
@@ -1074,12 +1113,6 @@ bool LaunchClosure::builtAgainstDyldCache(uuid_t cacheUUID) const
     return true;
 }
 
-const char* LaunchClosure::bootUUID() const
-{
-    uint32_t size;
-    return (char*)findAttributePayload(Type::bootUUID, &size);
-}
-
 void LaunchClosure::forEachEnvVar(void (^handler)(const char* keyEqualValue, bool& stop)) const
 {
     forEachAttributePayload(Type::envVar, ^(const void* payload, uint32_t size, bool& stop) {
@@ -1155,6 +1188,23 @@ bool LaunchClosure::hasInsertedLibraries() const
     return getFlags().hasInsertedLibraries;
 }
 
+bool LaunchClosure::hasProgramVars(uint32_t& runtimeOffset) const
+{
+    if ( !getFlags().hasProgVars )
+        return false;
+    uint32_t payloadSize = 0;
+    const uint8_t* buffer = (const uint8_t*)findAttributePayload(Type::progVars, &payloadSize);
+    if (buffer == nullptr)
+        return false;
+    runtimeOffset = *((uint32_t*)buffer);
+    return true;
+}
+
+bool LaunchClosure::usedInterposing() const
+{
+    return getFlags().usedInterposing;
+}
+
 bool LaunchClosure::hasInterposings() const
 {
     __block bool result = false;
@@ -1252,86 +1302,69 @@ void LaunchClosure::duplicateClassesHashTable(const ObjCClassDuplicatesOpt*& dup
     duplicateClassesHashTable = (const ObjCClassDuplicatesOpt*)buffer;
 }
 
-static void putHexNibble(uint8_t value, char*& p)
-{
-    if ( value < 10 )
-        *p++ = '0' + value;
-    else
-        *p++ = 'A' + value - 10;
-}
 
-static void putHexByte(uint8_t value, char*& p)
+static bool getContainerLibraryCachesDir(const char* envp[], char libCacheDir[])
 {
-    value &= 0xFF;
-    putHexNibble(value >> 4,   p);
-    putHexNibble(value & 0x0F, p);
-}
-
-static bool hashBootAndFileInfo(const char* mainExecutablePath, char hashString[32])
-{
-    struct stat statbuf;
-    if ( ::stat(mainExecutablePath, &statbuf) != 0)
+    // $HOME is root of writable data container
+    const char* homeDir = _simple_getenv(envp, "HOME");
+    if ( homeDir == nullptr )
         return false;
-#if !DARLING && !TARGET_OS_DRIVERKIT // Temp until core crypto is available
-    const struct ccdigest_info* di = ccsha256_di();
-    ccdigest_di_decl(di, hashTemp); // defines hashTemp array in stack
-    ccdigest_init(di, hashTemp);
 
-    // put boot time into hash
-    const uint64_t* bootTime = ((uint64_t*)_COMM_PAGE_BOOTTIME_USEC);
-    ccdigest_update(di, hashTemp, sizeof(uint64_t), bootTime);
-
-    // put inode of executable into hash
-    ccdigest_update(di, hashTemp, sizeof(statbuf.st_ino), &statbuf.st_ino);
-
-    // put mod-time of executable into hash
-    ccdigest_update(di, hashTemp, sizeof(statbuf.st_mtime), &statbuf.st_mtime);
-
-    // complete hash computation and append as hex string
-    uint8_t hashBits[32];
-    ccdigest_final(di, hashTemp, hashBits);
-    char* s = hashString;
-    for (size_t i=0; i < sizeof(hashBits); ++i)
-        putHexByte(hashBits[i], s);
-    *s = '\0';
+    // Use realpath to block malicious values like HOME=/tmp/../usr/bin
+    char realHomePath[PATH_MAX];
+    if ( realpath(homeDir, realHomePath) != nullptr )
+        homeDir = realHomePath;
+#if TARGET_OS_OSX
+    // <rdar://problem/66593232> iOS apps on Apple Silicon macOS have a different data container location
+    if ( strstr(homeDir, "/Library/Containers/") == nullptr )
+        return false;
+#else
+    // <rdar://problem/47688842> dyld3 should only save closures to disk for containerized apps
+    if ( strncmp(homeDir, "/private/var/mobile/Containers/Data/", 36) != 0 )
+        return false;
 #endif
+
+    // return $HOME/Library/Caches/
+    strlcpy(libCacheDir, homeDir, PATH_MAX);
+    strlcat(libCacheDir, "/Library/Caches", PATH_MAX);
     return true;
 }
 
-bool LaunchClosure::buildClosureCachePath(const char* mainExecutablePath, char closurePath[], const char* tempDir,
-                                          bool makeDirsIfMissing)
+bool LaunchClosure::buildClosureCachePath(const char* mainExecutablePath, const char* envp[],
+                                          bool makeDirsIfMissing, char closurePath[])
 {
-    // <rdar://problem/47688842> dyld3 should only save closures to disk for containerized apps
-    if ( strstr(tempDir, "/Containers/Data/") == nullptr )
+    // get path to data container's Library/Caches/ dir
+    if ( !getContainerLibraryCachesDir(envp, closurePath) )
         return false;
 
-    strlcpy(closurePath, tempDir, PATH_MAX);
-    strlcat(closurePath, "/com.apple.dyld/", PATH_MAX);
-    
-    // make sure dyld sub-dir exists
-    if ( makeDirsIfMissing ) {
+    // make sure XXX/Library/Caches/ exists
     struct stat statbuf;
-        if ( ::stat(closurePath, &statbuf) != 0 ) {
+    if ( dyld3::stat(closurePath, &statbuf) != 0 )
+        return false;
+
+    // add dyld sub-dir
+    strlcat(closurePath, "/com.apple.dyld", PATH_MAX);
+    if ( makeDirsIfMissing ) {
+        if ( dyld3::stat(closurePath, &statbuf) != 0 ) {
             if ( ::mkdir(closurePath, S_IRWXU) != 0 )
                 return false;
         }
     }
 
+    // add <prog-name> + ".closure"
     const char* leafName = strrchr(mainExecutablePath, '/');
     if ( leafName == nullptr )
         leafName = mainExecutablePath;
     else
         ++leafName;
+    strlcat(closurePath, "/", PATH_MAX);
     strlcat(closurePath, leafName, PATH_MAX);
-    strlcat(closurePath, "-", PATH_MAX);
-
-    if ( !hashBootAndFileInfo(mainExecutablePath, &closurePath[strlen(closurePath)]) )
-        return false;
 
     strlcat(closurePath, ".closure", PATH_MAX);
     return true;
 }
 
+
 ////////////////////////////  ObjCStringTable ////////////////////////////////////////
     
 uint32_t ObjCStringTable::hash(const char *key, size_t keylen) const

dyld3/Closure.h

@@ -110,7 +110,6 @@ struct VIS_HIDDEN TypedBytes
         topImage                = 36,  // sizeof(ImageNum)
         libDyldEntry            = 37,  // sizeof(ResolvedSymbolTarget)
         libSystemNum            = 38,  // sizeof(ImageNum)
-        bootUUID                = 39,  // c-string 40
         mainEntry               = 40,  // sizeof(ResolvedSymbolTarget)
         startEntry              = 41,  // sizeof(ResolvedSymbolTarget)     // used by programs built with crt1.o
         cacheOverrides          = 42,  // sizeof(PatchEntry) * count       // used if process uses interposing or roots (cached dylib overrides)
@@ -120,6 +119,7 @@ struct VIS_HIDDEN TypedBytes
         classTable              = 46,  // (3 * uint32_t) + (sizeof(ObjCClassImage) * count) + classHashTable size + protocolHashTable size
         warning                 = 47,  // len = uint32_t + length path + 1, use one entry per warning
         duplicateClassesTable   = 48,  // duplicateClassesHashTable
+        progVars                = 49,  // sizeof(uint32_t)
     };
 
     Type         type          : 8;
@@ -165,6 +165,8 @@ struct VIS_HIDDEN Image : ContainerTypedBytes
     bool                hasObjC() const;
     bool                hasInitializers() const;
     bool                hasPrecomputedObjC() const;
+    bool                fixupsNotEncoded() const;   // minimal closure, dyld must parse and apply fixups
+    bool                rebasesNotEncoded() const;
     bool                hasTerminators() const;
     bool                hasReadOnlyData() const;
     bool                hasChainedFixups() const;
@@ -176,7 +178,6 @@ struct VIS_HIDDEN Image : ContainerTypedBytes
     bool                is64() const;
     bool                neverUnload() const;
     bool                cwdMustBeThisDir() const;
-    bool                isPlatformBinary() const;
     bool                overridableDylib() const;
     bool                hasFileModTimeAndInode(uint64_t& inode, uint64_t& mTime) const;
     void                forEachCDHash(void (^handler)(const uint8_t cdHash[20], bool& stop)) const;
@@ -192,6 +193,7 @@ struct VIS_HIDDEN Image : ContainerTypedBytes
     bool                hasPathWithHash(const char* path, uint32_t hash) const;
     bool                isOverrideOfDyldCacheImage(ImageNum& cacheImageNum) const;
     uint64_t            textSize() const;
+    const char*         variantString() const; // "minimal" or "full" closure
 
 	union ResolvedSymbolTarget
     {
@@ -227,7 +229,7 @@ struct VIS_HIDDEN Image : ContainerTypedBytes
             return (raw != rhs.raw);
         }
      };
-     static_assert(sizeof(ResolvedSymbolTarget) == 8);
+     static_assert(sizeof(ResolvedSymbolTarget) == 8, "Invalid size");
 
 
     // ObjC optimisations
@@ -359,6 +361,8 @@ struct VIS_HIDDEN Image : ContainerTypedBytes
     void                forEachTextReloc(void (^rebase)(uint32_t imageOffsetToRebase, bool& stop),
                                          void (^bind)(uint32_t imageOffsetToBind, ResolvedSymbolTarget bindTarget, bool& stop)) const;
 
+    bool                forEachBind(void (^bind)(uint64_t imageOffsetToBind, ResolvedSymbolTarget bindTarget, bool& stop)) const;
+
  	static_assert(sizeof(ResolvedSymbolTarget) == 8, "Overflow in size of SymbolTargetLocation");
 
     static uint32_t     hashFunction(const char*);
@@ -369,6 +373,8 @@ private:
     friend class ClosureBuilder;
     friend class ClosureWriter;
     friend class LaunchClosureWriter;
+    friend class RebasePatternBuilder;
+    friend class BindPatternBuilder;
     
     uint32_t             pageSize() const;
 
@@ -395,7 +401,10 @@ private:
                         hasReadOnlyData              : 1,
                         hasChainedFixups             : 1,
                         hasPrecomputedObjC           : 1,
-                        padding                      : 17;
+                        fixupsNotEncoded             : 1,
+                        rebasesNotEncoded            : 1,
+                        hasOverrideImageNum          : 1,
+                        padding                      : 14;
     };
 
     static_assert(sizeof(Flags) == sizeof(uint64_t), "Flags overflow");
@@ -656,7 +665,8 @@ struct VIS_HIDDEN Closure : public ContainerTypedBytes
 {
     size_t              size() const;
     const ImageArray*   images() const;
-    ImageNum            topImage() const;
+    ImageNum            topImageNum() const;
+    const Image*        topImage() const;
     void                deallocate() const;
 
     friend class ClosureWriter;
@@ -697,7 +707,6 @@ struct VIS_HIDDEN LaunchClosure : public Closure
     };
 
     bool                builtAgainstDyldCache(uuid_t cacheUUID) const;
-    const char*         bootUUID() const;
     void                forEachMustBeMissingFile(void (^handler)(const char* path, bool& stop)) const;
     void                forEachSkipIfExistsFile(void (^handler)(const SkippedFile& file, bool& stop)) const;
     void                forEachEnvVar(void (^handler)(const char* keyEqualValue, bool& stop)) const;
@@ -709,6 +718,7 @@ struct VIS_HIDDEN LaunchClosure : public Closure
     void                forEachInterposingTuple(void (^handler)(const InterposingTuple& tuple, bool& stop)) const;
     bool                usedAtPaths() const;
     bool                usedFallbackPaths() const;
+    bool                usedInterposing() const;
     bool                selectorHashTable(Array<Image::ObjCSelectorImage>& imageNums,
                                           const ObjCSelectorOpt*& hashTable) const;
     bool                classAndProtocolHashTables(Array<Image::ObjCClassImage>& imageNums,
@@ -717,9 +727,10 @@ struct VIS_HIDDEN LaunchClosure : public Closure
     void                duplicateClassesHashTable(const ObjCClassDuplicatesOpt*& duplicateClassesHashTable) const;
     bool                hasInsertedLibraries() const;
     bool                hasInterposings() const;
+    bool                hasProgramVars(uint32_t& runtimeOffset) const;
     
-    static bool         buildClosureCachePath(const char* mainExecutablePath, char closurePath[], const char* tempDir,
-                                              bool makeDirsIfMissing);
+    static bool         buildClosureCachePath(const char* mainExecutablePath, const char* envp[],
+                                              bool makeDirsIfMissing, char closurePath[]);
 
 private:
     friend class LaunchClosureWriter;
@@ -730,7 +741,9 @@ private:
                         usedFallbackPaths        :  1,
                         initImageCount           : 16,
                         hasInsertedLibraries     :  1,
-                        padding                  : 13;
+                        hasProgVars              :  1,
+                        usedInterposing          :  1,
+                        padding                  : 11;
     };
     const Flags&        getFlags() const;
 };

dyld3/ClosureBuilder.h

@@ -32,6 +32,7 @@
 #include "PathOverrides.h"
 #include "DyldSharedCache.h"
 #include "MachOAnalyzer.h"
+#include "MachOAnalyzerSet.h"
 #include "Map.h"
 #include "Loading.h"
 
@@ -43,25 +44,20 @@ struct objc_selopt_t;
 struct objc_protocolopt2_t;
 }
 
+typedef dyld3::MachOAnalyzerSet::WrappedMachO   WrappedMachO;
+
 namespace dyld3 {
 
+class RootsChecker;
 
 namespace closure {
 
+class ClosureAnalyzerSet;
 
-
-class VIS_HIDDEN ClosureBuilder
+class VIS_HIDDEN ClosureBuilder : public dyld3::MachOAnalyzerSet
 {
 public:
 
-    struct LaunchErrorInfo
-    {
-        uintptr_t       kind;
-        const char*     clientOfDylibPath;
-        const char*     targetDylibPath;
-        const char*     symbol;
-    };
-
     struct ResolvedTargetInfo
     {
         const MachOLoaded* foundInDylib;
@@ -75,26 +71,23 @@ public:
         int                libOrdinal;
     };
 
-    struct CacheDylibsBindingHandlers
-    {
-        void        (^rebase)(ImageNum, const MachOLoaded* imageToFix, uint32_t runtimeOffset);
-        void        (^bind)(ImageNum, const MachOLoaded* imageToFix, uint32_t runtimeOffset, Image::ResolvedSymbolTarget target, const ResolvedTargetInfo& targetInfo);
-        void        (^chainedBind)(ImageNum, const MachOLoaded*, const dyld_chained_starts_in_image* starts, const Array<Image::ResolvedSymbolTarget>& targets, const Array<ResolvedTargetInfo>& targetInfos);
-   };
+    typedef void (^DylibFixupHandler)(const MachOLoaded* fixupIn, uint64_t fixupLocRuntimeOffset, PointerMetaData pmd, const MachOAnalyzerSet::FixupTarget& target);
 
     enum class AtPath { none, all, onlyInRPaths };
 
-                                ClosureBuilder(uint32_t startImageNum, const FileSystem& fileSystem, const DyldSharedCache* dyldCache, bool dyldCacheIsLive,
+                                ClosureBuilder(uint32_t startImageNum, const FileSystem& fileSystem, const RootsChecker& rootsChecker,
+                                               const DyldSharedCache* dyldCache, bool dyldCacheIsLive,
                                                const GradedArchs& archs, const PathOverrides& pathOverrides,
                                                AtPath atPathHandling=AtPath::all, bool allowRelativePaths=true, LaunchErrorInfo* errorInfo=nullptr,
-                                               Platform platform=MachOFile::currentPlatform(), const CacheDylibsBindingHandlers* handlers=nullptr);
+                                               Platform platform=MachOFile::currentPlatform(), DylibFixupHandler dylibFixupHandler=nullptr);
                                 ~ClosureBuilder();
     Diagnostics&                diagnostics() { return _diag; }
 
     const LaunchClosure*        makeLaunchClosure(const LoadedFileInfo& fileInfo, bool allowInsertFailures);
 
     const LaunchClosure*        makeLaunchClosure(const char* mainPath,bool allowInsertFailures);
-
+    void                        makeMinimalClosures() { _makeMinimalClosure = true; }
+    void                        setCanSkipEncodingRebases() { _leaveRebasesAsOpcodes = true; }
 
     static const DlopenClosure* sRetryDlopenClosure;
     const DlopenClosure*        makeDlopenClosure(const char* dylibPath, const LaunchClosure* mainClosure, const Array<LoadedImage>& loadedList,
@@ -102,8 +95,10 @@ public:
                                                   closure::ImageNum* topImageNum);
 
     ImageNum                    nextFreeImageNum() const { return _startImageNum + _nextIndex; }
+    Platform                    platform() const { return _platform; }
     
     void                        setDyldCacheInvalidFormatVersion();
+    void                        disableInterposing() { _interposingDisabled = true; }
 
 
     struct PatchableExport
@@ -125,20 +120,20 @@ public:
         const char*             aliasPath;
     };
 
-    const ImageArray*           makeDyldCacheImageArray(bool customerCache, const Array<CachedDylibInfo>& dylibs, const Array<CachedDylibAlias>& aliases);
+    const ImageArray*           makeDyldCacheImageArray(const Array<CachedDylibInfo>& dylibs, const Array<CachedDylibAlias>& aliases);
 
     const ImageArray*           makeOtherDylibsImageArray(const Array<LoadedFileInfo>& otherDylibs, uint32_t cachedDylibsCount);
 
     static void                 buildLoadOrder(Array<LoadedImage>& loadedList, const Array<const ImageArray*>& imagesArrays, const Closure* toAdd);
 
 private:
-
+    friend ClosureAnalyzerSet;
 
     struct InitInfo
     {
-        uint32_t    initOrder;
-        bool        danglingUpward;
-        bool        visited;
+        uint32_t   initOrder       : 30,
+                   danglingUpward  :  1,
+                   visited         :  1;
     };
 
     struct BuilderLoadedImage
@@ -153,8 +148,11 @@ private:
                                     isBadImage              : 1,
                                     mustBuildClosure        : 1,
                                     hasMissingWeakImports   : 1,
-                                    padding                 : 12,
+                                    hasInterposingTuples    : 1,
+                                    padding                 : 11,
                                     overrideImageNum        : 12;
+        uint32_t                    exportsTrieOffset;
+        uint32_t                    exportsTrieSize;
         LoadedFileInfo              loadedFileInfo;
 
         // Convenience method to get the information from the loadedFileInfo
@@ -186,36 +184,27 @@ private:
                                       uint32_t compatVersion, bool canUseSharedCacheClosure);
     void                    buildImage(ImageWriter& writer, BuilderLoadedImage& forImage);
     void                    addSegments(ImageWriter& writer, const MachOAnalyzer* mh);
-    void                    addRebaseInfo(ImageWriter& writer, const MachOAnalyzer* mh);
-    void                    addSynthesizedRebaseInfo(ImageWriter& writer, const MachOAnalyzer* mh);
-    void                    addSynthesizedBindInfo(ImageWriter& writer, const MachOAnalyzer* mh);
-    void                    addBindInfo(ImageWriter& writer, BuilderLoadedImage& forImage);
-    void                    reportRebasesAndBinds(ImageWriter& writer, BuilderLoadedImage& forImage);
+    void                    addFixupInfo(ImageWriter& writer, BuilderLoadedImage& forImage);
     void                    addChainedFixupInfo(ImageWriter& writer, BuilderLoadedImage& forImage);
     void                    addInterposingTuples(LaunchClosureWriter& writer, const Image* image, const MachOAnalyzer* mh);
     void                    computeInitOrder(ImageWriter& writer, uint32_t loadIndex);
     void                    addClosureInfo(LaunchClosureWriter& closureWriter);
     void                    depthFirstRecurseSetInitInfo(uint32_t loadIndex, InitInfo initInfos[], uint32_t& initOrder, bool& hasError);
-    bool                    findSymbol(BuilderLoadedImage& fromImage, int libraryOrdinal, const char* symbolName, bool weakImport, bool lazyBind, uint64_t addend,
-                                       Image::ResolvedSymbolTarget& target, ResolvedTargetInfo& targetInfo);
-    bool                    findSymbolInImage(const MachOAnalyzer* macho, const char* symbolName, uint64_t addend, bool followReExports,
-                                              bool weakImport, Image::ResolvedSymbolTarget& target, ResolvedTargetInfo& targetInfo);
     const MachOAnalyzer*    machOForImageNum(ImageNum imageNum);
     ImageNum                imageNumForMachO(const MachOAnalyzer* mh);
     const MachOAnalyzer*    findDependent(const MachOLoaded* mh, uint32_t depIndex);
     BuilderLoadedImage&     findLoadedImage(ImageNum imageNum);
+    const BuilderLoadedImage& findLoadedImage(ImageNum imageNum) const;
     BuilderLoadedImage&     findLoadedImage(const MachOAnalyzer* mh);
     uint32_t                index(const BuilderLoadedImage&);
     bool                    expandAtLoaderPath(const char* loadPath, bool fromLCRPATH, const BuilderLoadedImage& loadedImage, char fixedPath[]);
-    bool                    expandAtExecutablePath(const char* loadPath, bool fromLCRPATH, char fixedPath[]);
+    bool                    expandAtExecutablePath(const char* loadPath, bool fromLCRPATH, bool fromLCRPATHinMain, char fixedPath[]);
     void                    addMustBeMissingPath(const char* path);
     void                    addSkippedFile(const char* path, uint64_t inode, uint64_t mtime);
-    const char*             strdup_temp(const char* path);
+    const char*             strdup_temp(const char* path) const;
     bool                    overridableDylib(const BuilderLoadedImage& forImage);
-    void                    forEachBind(BuilderLoadedImage& forImage, void (^handler)(uint64_t runtimeOffset, Image::ResolvedSymbolTarget target, const ResolvedTargetInfo& targetInfo, bool& stop),
-                                                                      void (^strongHandler)(const char* strongSymbolName),
-                                                                      void (^missingLazyBindHandler)());
-    bool                    findMissingSymbolHandler(Image::ResolvedSymbolTarget& target, ResolvedTargetInfo& targetInfo);
+    void                    addOperatorCachePatches(BuilderLoadedImage& forImage);
+    void                    addWeakDefCachePatch(uint32_t cachedDylibIndex, uint32_t exportCacheOffset, const FixupTarget& patchTarget);
 
     struct HashCString {
         static size_t hash(const char* v);
@@ -283,6 +272,7 @@ private:
         OverflowSafeArray<SelectorFixup>                                                    selectorFixups;
         Map<const char*, dyld3::closure::Image::ObjCImageOffset, HashCString, EqualCString> selectorMap;
         std::optional<uint64_t>                                                             methodNameVMOffset;
+        OverflowSafeArray<uint64_t>                                                         methodListFixups;
     };
 
     bool                    optimizeObjC(Array<ImageWriter>& writers);
@@ -292,6 +282,7 @@ private:
     void                    optimizeObjCClasses(const objc_opt::objc_clsopt_t* objcClassOpt,
                                                 const Map<const dyld3::MachOAnalyzer*, bool, HashPointer, EqualPointer>& sharedCacheImagesMap,
                                                 const Map<const char*, dyld3::closure::Image::ObjCDuplicateClass, HashCString, EqualCString>& duplicateSharedCacheClasses,
+                                                const Map<const char*, bool, HashCString, EqualCString>& duplicateClassesToIgnore,
                                                 ObjCOptimizerImage& image);
     void                    optimizeObjCProtocols(const objc_opt::objc_protocolopt2_t* objcProtocolOpt,
                                                   const Map<const dyld3::MachOAnalyzer*, bool, HashPointer, EqualPointer>& sharedCacheImagesMap,
@@ -302,25 +293,40 @@ private:
                                                          const char* duplicateDefinitionPath,
                                                          const char* canonicalDefinitionPath);
 
+    void                    parseObjCClassDuplicates(Map<const char*, bool, HashCString, EqualCString>& duplicateClassesToIgnore);
+
     static bool             inLoadedImageArray(const Array<LoadedImage>& loadedList, ImageNum imageNum);
     static void             buildLoadOrderRecurse(Array<LoadedImage>& loadedList, const Array<const ImageArray*>& imagesArrays, const Image* toAdd);
     void                    invalidateInitializerRoots();
+    Image::ResolvedSymbolTarget makeResolvedTarget(const FixupTarget& target) const;
+
+    // MachOAnalyzerSet implementations
+    void                mas_forEachImage(void (^handler)(const WrappedMachO& anImage, bool hidden, bool& stop)) const override;
+    bool                mas_fromImageWeakDefLookup(const WrappedMachO& fromWmo, const char* symbolName, uint64_t addend, CachePatchHandler patcher, FixupTarget& target) const override;
+    void                mas_mainExecutable(WrappedMachO& anImage) const override;
+    void*               mas_dyldCache() const override;
+    bool                wmo_dependent(const WrappedMachO* image, uint32_t depIndex, WrappedMachO& childObj, bool& missingWeakDylib) const override;
+    const char*         wmo_path(const WrappedMachO* image) const override;
+    ExportsTrie         wmo_getExportsTrie(const WrappedMachO* image) const override;
+    bool                wmo_missingSymbolResolver(const WrappedMachO* fromWmo, bool weakImport, bool lazyBind, const char* symbolName, const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const override;
+
 
     const FileSystem&                       _fileSystem;
+    const RootsChecker&                     _rootsChecker;
     const DyldSharedCache* const            _dyldCache;
     const PathOverrides&                    _pathOverrides;
     const GradedArchs&                      _archs;
     Platform const                          _platform;
     uint32_t const                          _startImageNum;
     const ImageArray*                       _dyldImageArray                 = nullptr;
-    const CacheDylibsBindingHandlers*       _handlers                       = nullptr;
+    DylibFixupHandler                       _dylibFixupHandler              = nullptr;
     const Array<CachedDylibAlias>*          _aliases                        = nullptr;
     const AtPath                            _atPathHandling                 = AtPath::none;
     uint32_t                                _mainProgLoadIndex              = 0;
     const char*                             _mainProgLoadPath               = nullptr;
     Diagnostics                             _diag;
     LaunchErrorInfo*                        _launchErrorInfo                = nullptr;
-    PathPool*                               _tempPaths                      = nullptr;
+    mutable PathPool*                       _tempPaths                      = nullptr;
     PathPool*                               _mustBeMissingPaths             = nullptr;
     OverflowSafeArray<SkippedFile>          _skippedFiles;
     uint32_t                                _nextIndex                      = 0;
@@ -328,7 +334,6 @@ private:
     OverflowSafeArray<Image::LinkedImage,65536> _dependencies;                  // all dylibs in cache need ~20,000 edges
     OverflowSafeArray<InterposingTuple>     _interposingTuples;
     OverflowSafeArray<Closure::PatchEntry>  _weakDefCacheOverrides;
-    OverflowSafeArray<const char*>          _weakDefsFromChainedBinds;
     OverflowSafeArray<uint8_t>              _objcSelectorsHashTable;
     OverflowSafeArray<Image::ObjCSelectorImage> _objcSelectorsHashTableImages;
     OverflowSafeArray<uint8_t>              _objcClassesHashTable;
@@ -341,21 +346,48 @@ private:
     bool                                    _makingDyldCacheImages          = false;
     bool                                    _dyldCacheIsLive                = false;    // means kernel is rebasing dyld cache content being viewed
     bool                                    _makingClosuresInCache          = false;
-    bool                                    _makingCustomerCache            = false;
     bool                                    _allowRelativePaths             = false;
     bool                                    _atPathUsed                     = false;
+    bool                                    _interposingTuplesUsed          = false;
     bool                                    _fallbackPathUsed               = false;
     bool                                    _allowMissingLazies             = false;
     bool                                    _dyldCacheInvalidFormatVersion  = false;
     bool                                    _foundNonCachedImage            = false;    // true means we have one or more images from disk we need to build closure(s) for
     bool                                    _foundDyldCacheRoots            = false;    // true if one or more images are roots of the shared cache
     bool                                    _foundMissingLazyBinds          = false;    // true if one or more images having missing lazy binds
+    bool                                    _makeMinimalClosure             = false;
+    bool                                    _interposingDisabled            = false;
+    bool                                    _leaveRebasesAsOpcodes          = false;
     ImageNum                                _libDyldImageNum                = 0;
     ImageNum                                _libSystemImageNum              = 0;
 };
 
+class VIS_HIDDEN RebasePatternBuilder
+{
+public:
+            RebasePatternBuilder(OverflowSafeArray<closure::Image::RebasePattern>& entriesStorage, uint64_t ptrSize);
+    void    add(uint64_t runtimeOffset);
+private:
+    OverflowSafeArray<closure::Image::RebasePattern>&   _rebaseEntries;
+    uint64_t                                            _lastLocation;
+    const uint64_t                                      _ptrSize;
+
+    static const Image::RebasePattern                   _s_maxLeapPattern;
+    static const uint64_t                               _s_maxLeapCount;
+};
 
 
+class VIS_HIDDEN BindPatternBuilder
+{
+public:
+            BindPatternBuilder(OverflowSafeArray<closure::Image::BindPattern>& entriesStorage, uint64_t ptrSize);
+    void    add(uint64_t runtimeOffset, Image::ResolvedSymbolTarget target, bool weakBindCoalese);
+private:
+    OverflowSafeArray<closure::Image::BindPattern>&   _bindEntries;
+    const uint64_t                                    _ptrSize;
+    uint64_t                                          _lastOffset;
+    Image::ResolvedSymbolTarget                       _lastTarget;
+};
 
 
 } //  namespace closure

dyld3/ClosureFileSystem.h

@@ -39,7 +39,7 @@ struct LoadedFileInfo {
     uint64_t     fileContentLen             = 0;
     uint64_t     sliceOffset                = 0;
     uint64_t     sliceLen            : 63,
-                 isSipProtected      : 1;
+                 isOSBinary          : 1;
     uint64_t     inode                      = 0;
     uint64_t     mtime                      = 0;
     void (*unload)(const LoadedFileInfo&)   = nullptr;

dyld3/ClosureFileSystemNull.cpp

@@ -34,9 +34,14 @@ bool FileSystemNull::loadFile(const char* path, LoadedFileInfo& info, char reale
 }
 
 void FileSystemNull::unloadFile(const LoadedFileInfo& info) const {
+    if (info.unload)
+        info.unload(info);
 }
 
 void FileSystemNull::unloadPartialFile(LoadedFileInfo& info, uint64_t keepStartOffset, uint64_t keepLength) const {
+    // Note we don't actually unload the data here, but we do want to update the offsets for other data structures to track where we are
+    info.fileContent = (const void*)((char*)info.fileContent + keepStartOffset);
+    info.fileContentLen = keepLength;
 }
 
 bool FileSystemNull::fileExists(const char* path, uint64_t* inode, uint64_t* mtime,

dyld3/ClosureFileSystemPhysical.cpp

@@ -23,13 +23,12 @@
 
 #include "ClosureFileSystemPhysical.h"
 
+#include <TargetConditionals.h>
+
 #include <string.h>
 #include <stdlib.h>
 #include <unistd.h>
 #include <fcntl.h>
-#if BUILDING_UPDATE_DYLD_CACHE_BUILDER
-  #include <rootless.h>
-#endif
 #include <sys/errno.h>
 #include <sys/mman.h>
 #include <sys/stat.h>
@@ -38,6 +37,9 @@
   #include <sandbox.h>
   #include <sandbox/private.h>
 #endif
+#include <TargetConditionals.h>
+#include "MachOFile.h"
+#include "MachOAnalyzer.h"
 
 using dyld3::closure::FileSystemPhysical;
 
@@ -45,7 +47,7 @@ bool FileSystemPhysical::getRealPath(const char possiblePath[MAXPATHLEN], char r
     __block bool success = false;
     // first pass: open file and ask kernel for canonical path
     forEachPath(possiblePath, ^(const char* aPath, unsigned prefixLen, bool& stop) {
-        int fd = ::open(aPath, O_RDONLY, 0);
+        int fd = dyld3::open(aPath, O_RDONLY, 0);
         if ( fd != -1 ) {
             char tempPath[MAXPATHLEN];
             success = (fcntl(fd, F_GETPATH, tempPath) == 0);
@@ -110,6 +112,8 @@ void FileSystemPhysical::forEachPath(const char* path, void (^handler)(const cha
     }
     if ( _rootPath != nullptr ) {
         strlcpy(altPath, _rootPath, PATH_MAX);
+        if ( path[0] != '/' )
+            strlcat(altPath, "/", PATH_MAX);
         strlcat(altPath, path, PATH_MAX);
         handler(altPath, (unsigned)strlen(_rootPath), stop);
         if ( stop )
@@ -144,9 +148,8 @@ bool FileSystemPhysical::loadFile(const char* path, LoadedFileInfo& info, char r
     // open file
     __block int fd;
     __block struct stat statBuf;
-    __block bool sipProtected = false;
     forEachPath(path, ^(const char* aPath, unsigned prefixLen, bool& stop) {
-        fd = ::open(aPath, O_RDONLY, 0);
+        fd = dyld3::open(aPath, O_RDONLY, 0);
         if ( fd == -1 ) {
             int openErrno = errno;
             if ( (openErrno == EPERM) && sandboxBlockedOpen(path) )
@@ -183,9 +186,6 @@ bool FileSystemPhysical::loadFile(const char* path, LoadedFileInfo& info, char r
                     }
                     else
                         strcpy(realerPath, realPathWithin);
-    #if BUILDING_UPDATE_DYLD_CACHE_BUILDER
-                    sipProtected = (rootless_check_trusted_fd(fd) == 0);
-    #endif
                     stop = true;
                 }
                 else {
@@ -217,7 +217,7 @@ bool FileSystemPhysical::loadFile(const char* path, LoadedFileInfo& info, char r
     info.fileContentLen = statBuf.st_size;
     info.sliceOffset = 0;
     info.sliceLen = statBuf.st_size;
-    info.isSipProtected = sipProtected;
+    info.isOSBinary = false;
     info.inode = statBuf.st_ino;
     info.mtime = statBuf.st_mtime;
     info.path  = path;
@@ -240,6 +240,27 @@ bool FileSystemPhysical::loadFile(const char* path, LoadedFileInfo& info, char r
     }
     info.fileContent = wholeFile;
 
+    // if this is an arm64e mach-o or a fat file with an arm64e slice we need to record if it is an OS binary
+#if TARGET_OS_OSX && __arm64e__
+    const MachOAnalyzer* ma = (MachOAnalyzer*)wholeFile;
+    if ( ma->hasMachOMagic() ) {
+        if ( (ma->cputype == CPU_TYPE_ARM64) && ((ma->cpusubtype & ~CPU_SUBTYPE_MASK) == CPU_SUBTYPE_ARM64E) ) {
+            if ( ma->isOSBinary(fd, 0, info.fileContentLen) )
+                info.isOSBinary = true;
+        }
+    }
+    else if ( const FatFile* fat = FatFile::isFatFile(wholeFile) ) {
+        Diagnostics diag;
+        fat->forEachSlice(diag, info.fileContentLen, ^(uint32_t sliceCpuType, uint32_t sliceCpuSubType, const void* sliceStart, uint64_t sliceSize, bool& stop) {
+            if ( (sliceCpuType == CPU_TYPE_ARM64) && ((sliceCpuSubType & ~CPU_SUBTYPE_MASK) == CPU_SUBTYPE_ARM64E) ) {
+                uint64_t sliceOffset = (uint8_t*)sliceStart-(uint8_t*)wholeFile;
+                const MachOAnalyzer* sliceMA = (MachOAnalyzer*)((uint8_t*)wholeFile + sliceOffset);
+                if ( sliceMA->isOSBinary(fd, sliceOffset, sliceSize) )
+                    info.isOSBinary = true;
+            }
+        });
+    }
+#endif
     // Set unmap as the unload method.
     info.unload = [](const LoadedFileInfo& info) {
         ::munmap((void*)info.fileContent, (size_t)info.fileContentLen);
@@ -257,11 +278,11 @@ void FileSystemPhysical::unloadFile(const LoadedFileInfo& info) const {
 void FileSystemPhysical::unloadPartialFile(LoadedFileInfo& info, uint64_t keepStartOffset, uint64_t keepLength) const {
     // Unmap from 0..keepStartOffset and (keepStartOffset+keepLength)..info.fileContentLen
     if (keepStartOffset)
-        ::munmap((void*)info.fileContent, (size_t)keepStartOffset);
+        ::munmap((void*)info.fileContent, (size_t)trunc_page(keepStartOffset));
     if ((keepStartOffset + keepLength) != info.fileContentLen) {
-        // Round up to page alignment
-        keepLength = (keepLength + PAGE_SIZE - 1) & (-PAGE_SIZE);
-        ::munmap((void*)((char*)info.fileContent + keepStartOffset + keepLength), (size_t)(info.fileContentLen - (keepStartOffset + keepLength)));
+        uintptr_t start = round_page((uintptr_t)info.fileContent + keepStartOffset + keepLength);
+        uintptr_t end = (uintptr_t)info.fileContent + info.fileContentLen;
+        ::munmap((void*)start, end - start);
     }
     info.fileContent = (const void*)((char*)info.fileContent + keepStartOffset);
     info.fileContentLen = keepLength;
@@ -272,7 +293,7 @@ bool FileSystemPhysical::fileExists(const char* path, uint64_t* inode, uint64_t*
     __block bool result = false;
     forEachPath(path, ^(const char* aPath, unsigned prefixLen, bool& stop) {
         struct stat statBuf;
-        if ( ::stat(aPath, &statBuf) == 0 ) {
+        if ( dyld3::stat(aPath, &statBuf) == 0 ) {
             if (inode)
                 *inode = statBuf.st_ino;
             if (mtime)

dyld3/ClosurePrinter.cpp

@@ -49,7 +49,7 @@ static std::string printTarget(const Array<const ImageArray*>& imagesArrays, Ima
                 return std::string("bind to ") + targetImage->leafName() + " - " + hex8(-signExtend);
             }
             else
-                return std::string("bind to ") + targetImage->leafName() + " + " + hex8(target.image.offset);
+                return std::string("bind to ") + hex4(target.image.imageNum) + "-" + targetImage->leafName() + " + " + hex8(target.image.offset);
             break;
         case Image::ResolvedSymbolTarget::kindSharedCache:
             return std::string("bind to dyld cache + ") + hex8(target.sharedCache.offset);
@@ -139,8 +139,6 @@ static const char* nameForType(TypedBytes::Type type) {
         return "libDyldEntry";
     case TypedBytes::Type::libSystemNum:
         return "libSystemNum";
-    case TypedBytes::Type::bootUUID:
-        return "bootUUID";
     case TypedBytes::Type::mainEntry:
         return "mainEntry";
     case TypedBytes::Type::startEntry:
@@ -159,6 +157,8 @@ static const char* nameForType(TypedBytes::Type type) {
         return "warning";
     case TypedBytes::Type::duplicateClassesTable:
         return "duplicateClassesTable";
+    case TypedBytes::Type::progVars:
+        return "programVars";
     }
 }
 
@@ -207,7 +207,6 @@ static Node buildImageNode(const Image* image, const Array<const ImageArray*>& i
     imageNode.map["has-plus-loads"].value = (image->mayHavePlusLoads() ? "true" : "false");
     imageNode.map["never-unload"].value = (image->neverUnload() ? "true" : "false");
     imageNode.map["has-precomputed-objc"].value = (image->hasPrecomputedObjC() ? "true" : "false");
-//    imageNode.map["platform-binary"].value = (image->isPlatformBinary() ? "true" : "false");
 //    if ( image->cwdMustBeThisDir() )
 //        imageNode.map["cwd-must-be-this-dir"].value = "true";
     if ( !image->inDyldCache() ) {
@@ -284,6 +283,12 @@ static Node buildImageNode(const Image* image, const Array<const ImageArray*>& i
             }
         });
 
+        uint64_t expectedInode;
+        uint64_t expectedMtime;
+        if ( image->hasFileModTimeAndInode(expectedInode, expectedMtime) ) {
+            imageNode.map["file-inode"].value = hex(expectedInode);
+            imageNode.map["file-mod-time"].value = hex(expectedMtime);
+        }
 
         if ( printFixups ) {
             image->forEachFixup(^(uint64_t imageOffsetToRebase, bool &stop) {
@@ -424,6 +429,10 @@ static Node buildImageNode(const Image* image, const Array<const ImageArray*>& i
         imageNode.map["override-of-dyld-cache-image"].value = ImageArray::findImage(imagesArrays, cacheImageNum)->path();
 	}
 
+    if ( image->inDyldCache() && image->overridableDylib() ) {
+        imageNode.map["overridable-dylib"].value = "true";
+    }
+
 
 #if 0
     // add things to init before this image
@@ -476,7 +485,7 @@ static Node buildClosureNode(const DlopenClosure* closure, const Array<const Ima
     closure->forEachPatchEntry(^(const Closure::PatchEntry& patchEntry) {
         Node patchNode;
         patchNode.map["func-dyld-cache-offset"].value = hex8(patchEntry.exportCacheOffset);
-        patchNode.map["func-image-num"].value         = hex8(patchEntry.overriddenDylibInCache);
+        patchNode.map["func-image-num"].value         = hex4(patchEntry.overriddenDylibInCache);
         patchNode.map["replacement"].value            = printTarget(imagesArrays, patchEntry.replacement);
         root.map["dyld-cache-fixups"].array.push_back(patchNode);
     });
@@ -514,7 +523,7 @@ static Node buildClosureNode(const LaunchClosure* closure, const Array<const Ima
     closure->forEachPatchEntry(^(const Closure::PatchEntry& patchEntry) {
         Node patchNode;
         patchNode.map["func-dyld-cache-offset"].value = hex8(patchEntry.exportCacheOffset);
-        patchNode.map["func-image-num"].value         = hex8(patchEntry.overriddenDylibInCache);
+        patchNode.map["func-image-num"].value         = hex4(patchEntry.overriddenDylibInCache);
         patchNode.map["replacement"].value            = printTarget(imagesArrays, patchEntry.replacement);
         root.map["dyld-cache-fixups"].array.push_back(patchNode);
     });
@@ -556,14 +565,6 @@ static Node buildClosureNode(const LaunchClosure* closure, const Array<const Ima
         root.map["interposing-tuples"].array.push_back(tupleNode);
     });
 
-    closure->forEachPatchEntry(^(const Closure::PatchEntry& patchEntry) {
-        Node patchNode;
-        patchNode.map["func-dyld-cache-offset"].value = hex8(patchEntry.exportCacheOffset);
-        patchNode.map["func-image-num"].value         = hex8(patchEntry.overriddenDylibInCache);
-        patchNode.map["replacement"].value            = printTarget(imagesArrays, patchEntry.replacement);
-        root.map["dyld-cache-fixups"].array.push_back(patchNode);
-    });
-
     root.map["initial-image-count"].value = decimal(closure->initialLoadCount());
 
     // add env-vars if they exist
@@ -699,6 +700,11 @@ static Node buildClosureNode(const LaunchClosure* closure, const Array<const Ima
         root.map["objc-duplicate-class-warnings"].array.push_back(warningNode);
     });
 
+    // add program vars info for old macOS binaries
+    uint32_t progVarsOffset;
+    if ( closure->hasProgramVars(progVarsOffset) )
+        root.map["program-vars-offset"].value = hex8(progVarsOffset);
+
 #if 0
 
 

dyld3/ClosureWriter.cpp

@@ -312,6 +312,16 @@ void ImageWriter::setBindInfo(const Array<Image::BindPattern>& fixups)
     append(TypedBytes::Type::bindFixups, fixups.begin(), (uint32_t)fixups.count()*sizeof(Image::BindPattern));
 }
 
+void ImageWriter::setFixupsNotEncoded()
+{
+    getFlags().fixupsNotEncoded = true;
+}
+
+void ImageWriter::setRebasesNotEncoded()
+{
+    getFlags().rebasesNotEncoded = true;
+}
+
 void ImageWriter::setChainedFixups(uint64_t runtimeStartsStructOffset, const Array<Image::ResolvedSymbolTarget>& targets)
 {
     getFlags().hasChainedFixups = true;
@@ -397,6 +407,7 @@ void ImageWriter::setAsOverrideOf(ImageNum imageNum)
 {
     uint32_t temp = imageNum;
     append(TypedBytes::Type::imageOverride, &temp, sizeof(temp));
+    getFlags().hasOverrideImageNum = true;
 }
 
 void ImageWriter::setInitsOrder(const ImageNum images[], uint32_t count)
@@ -521,6 +532,11 @@ void LaunchClosureWriter::setUsedAtPaths(bool value)
     getFlags().usedAtPaths = value;
 }
 
+void LaunchClosureWriter::setUsedInterposing(bool value)
+{
+    getFlags().usedInterposing = value;
+}
+
 void LaunchClosureWriter::setHasInsertedLibraries(bool value)
 {
     getFlags().hasInsertedLibraries = value;
@@ -616,15 +632,10 @@ void LaunchClosureWriter::setDyldCacheUUID(const uuid_t uuid)
     append(TypedBytes::Type::dyldCacheUUID, uuid, sizeof(uuid_t));
 }
 
-void LaunchClosureWriter::setBootUUID(const char* uuid)
+void LaunchClosureWriter::setHasProgramVars(uint32_t offset)
 {
-    unsigned len = (unsigned)strlen(uuid);
-    char temp[len+8];
-    strcpy(temp, uuid);
-    unsigned paddedSize = len+1;
-    while ( (paddedSize % 4) != 0 )
-        temp[paddedSize++] = '\0';
-    append(TypedBytes::Type::bootUUID, temp, paddedSize);
+    getFlags().hasProgVars = true;
+    append(TypedBytes::Type::progVars, &offset, sizeof(uint32_t));
 }
 
 void LaunchClosureWriter::setObjCSelectorInfo(const Array<uint8_t>& hashTable, const Array<Image::ObjCSelectorImage>& hashTableImages) {

dyld3/ClosureWriter.h

@@ -97,6 +97,7 @@ public:
     void        setMappingInfo(uint64_t sliceOffset, uint64_t vmSize);
     void        setFileInfo(uint64_t inode, uint64_t modTime);
     void        setRebaseInfo(const Array<Image::RebasePattern>&);
+    void        setRebasesNotEncoded();
     void        setTextRebaseInfo(const Array<Image::TextFixupPattern>&);
     void        setBindInfo(const Array<Image::BindPattern>&);
     void        setObjCFixupInfo(const Image::ResolvedSymbolTarget& objcProtocolClassTarget,
@@ -108,6 +109,7 @@ public:
     void        setAsOverrideOf(ImageNum);
     void        setInitsOrder(const ImageNum images[], uint32_t count);
     void        setChainedFixups(uint64_t runtimeStartsStructOffset, const Array<Image::ResolvedSymbolTarget>& targets);
+    void        setFixupsNotEncoded();
 
     const Image* currentImage();
 
@@ -154,17 +156,18 @@ public:
     void                    setStartEntry(Image::ResolvedSymbolTarget start);
     void                    setUsedFallbackPaths(bool);
     void                    setUsedAtPaths(bool);
+    void                    setUsedInterposing(bool);
     void                    setHasInsertedLibraries(bool);
     void                    setMustBeMissingFiles(const Array<const char*>& paths);
     void                    setMustExistFiles(const Array<LaunchClosure::SkippedFile>& files);
     void                    addInterposingTuples(const Array<InterposingTuple>& tuples);
     void                    setDyldCacheUUID(const uuid_t);
-    void                    setBootUUID(const char* uuid);
     void                    addEnvVar(const char* envVar);
     void                    setObjCSelectorInfo(const Array<uint8_t>& hashTable, const Array<Image::ObjCSelectorImage>& hashTableImages);
     void                    setObjCClassAndProtocolInfo(const Array<uint8_t>& classHashTable, const Array<uint8_t>& protocolHashTable,
                                                         const Array<Image::ObjCClassImage>& hashTableImages);
     void                    setObjCDuplicateClassesInfo(const Array<uint8_t>& hashTable);
+    void                    setHasProgramVars(uint32_t offset);
 
 private:
     LaunchClosure::Flags&   getFlags();

dyld3/Defines.h

@@ -0,0 +1,36 @@
+/*
+* Copyright (c) 2019 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* "Portions Copyright (c) 1999 Apple Computer, Inc.  All Rights
+* Reserved.  This file contains Original Code and/or Modifications of
+* Original Code as defined in and that are subject to the Apple Public
+* Source License Version 1.0 (the 'License').  You may not use this file
+* except in compliance with the License.  Please obtain a copy of the
+* License at http://www.apple.com/publicsource and read it before using
+* this file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT.  Please see the
+* License for the specific language governing rights and limitations
+* under the License."
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
+
+// For now just for compact info, but in the long run we can use this to centralize convenience macros and configuration options
+
+#ifndef DYLD_DEFINES_H
+#define DYLD_DEFINES_H
+
+#define VIS_HIDDEN __attribute__((visibility("hidden")))
+
+#ifndef BUILD_FOR_TESTING
+#define BUILD_FOR_TESTING 0
+#endif
+
+#endif /* DYLD_DEFINES_H */

dyld3/Diagnostics.cpp

@@ -21,6 +21,7 @@
  * @APPLE_LICENSE_HEADER_END@
  */
 
+#include <iostream>
 
 #include <stdint.h>
 #include <stdio.h>
@@ -41,6 +42,7 @@
 #include <dirent.h>
 #include <mach/mach.h>
 #include <mach/machine.h>
+#include <mach/mach_time.h>
 #include <mach-o/loader.h>
 #include <mach-o/nlist.h>
 #include <mach-o/fat.h>
@@ -95,9 +97,9 @@ void Diagnostics::error(const char* format, va_list list)
         return;
 
     if (_prefix.empty()) {
-        fprintf(stderr, "%s", _simple_string(_buffer));
+        fprintf(stderr, "%s\n", _simple_string(_buffer));
     } else {
-        fprintf(stderr, "[%s] %s", _prefix.c_str(), _simple_string(_buffer));
+        fprintf(stderr, "[%s] %s\n", _prefix.c_str(), _simple_string(_buffer));
     }
 #endif
 }
@@ -125,6 +127,14 @@ void Diagnostics::assertNoError() const
         abort_report_np("%s", _simple_string(_buffer));
 }
 
+bool Diagnostics::errorMessageContains(const char* subString) const
+{
+    if ( _buffer == nullptr )
+        return false;
+    return (strstr(_simple_string(_buffer), subString) != nullptr);
+}
+
+
 #if !BUILDING_CACHE_BUILDER
 const char* Diagnostics::errorMessage() const
 {
@@ -213,5 +223,53 @@ void Diagnostics::clearWarnings()
 #endif
 }
 
+#if BUILDING_CACHE_BUILDER
+void TimeRecorder::pushTimedSection() {
+    openTimings.push_back(mach_absolute_time());
+}
+
+void TimeRecorder::recordTime(const char* format, ...) {
+    uint64_t t = mach_absolute_time();
+    uint64_t previousTime = openTimings.back();
+    openTimings.pop_back();
+
+    char*   output_string = nullptr;
+    va_list list;
+    va_start(list, format);
+    vasprintf(&output_string, format, list);
+    va_end(list);
+
+    if (output_string != nullptr) {
+        timings.push_back(TimingEntry {
+            .time = t - previousTime,
+            .logMessage = std::string(output_string),
+            .depth = (int)openTimings.size()
+        });
+    }
+
+    openTimings.push_back(mach_absolute_time());
+}
+
+void TimeRecorder::popTimedSection() {
+    openTimings.pop_back();
+}
+
+static inline uint32_t absolutetime_to_milliseconds(uint64_t abstime)
+{
+    return (uint32_t)(abstime/1000/1000);
+}
+
+void TimeRecorder::logTimings() {
+    for (const TimingEntry& entry : timings) {
+        for (int i = 0 ; i < entry.depth ; i++) {
+            std::cerr << "  ";
+        }
+        std::cerr << "time to " << entry.logMessage << " " << absolutetime_to_milliseconds(entry.time) << "ms" << std::endl;
+    }
+
+    timings.clear();
+}
+#endif
+
 #endif
 

dyld3/Diagnostics.h

@@ -57,6 +57,7 @@ public:
     bool            noError() const;
     void            clearError();
     void            assertNoError() const;
+    bool            errorMessageContains(const char* subString) const;
 
 #if !BUILDING_CACHE_BUILDER
     const char*                     errorMessage() const;
@@ -68,6 +69,7 @@ public:
 #endif
 
 private:
+
     void*                    _buffer = nullptr;
 #if BUILDING_CACHE_BUILDER
     std::string              _prefix;
@@ -76,7 +78,37 @@ private:
 #endif
 };
 
+#if BUILDING_CACHE_BUILDER
 
+class VIS_HIDDEN TimeRecorder
+{
+public:
+    // Call pushTimedSection(), then mark events with recordTime. Call popTimedSection() to stop the current timing session.
+    // This is stack-based, so you can start a sub-timer with pushTimedSection() / recordTime / recordTime... / popTimedSection()
+    // inside a first timed section.
+    // Call logTimings() to print everything.
 
+    // Start a new timed section.
+    void pushTimedSection();
+
+    // Records the time taken since the last pushTimedSection() / recordTime() at the current level
+    void recordTime(const char* format, ...);
+
+    // Stop the current timed section and pop back one level.
+    void popTimedSection();
+
+    void logTimings();
+private:
+    struct TimingEntry {
+        uint64_t time;
+        std::string logMessage;
+        int depth;
+    };
+
+    std::vector<uint64_t> openTimings;
+    std::vector<TimingEntry> timings;
+};
+
+#endif /* BUILDING_CACHE_BUILDER */
 
 #endif // Diagnostics_h

dyld3/JSON.h

@@ -30,18 +30,59 @@
 
 #include <string>
 #include <map>
+#include <sstream>
 #include <vector>
 
 namespace dyld3 {
 namespace json {
 
+enum class NodeValueType {
+    Default,
+    String,
+    RawValue,
+};
+
 struct Node
 {
+    NodeValueType               type = NodeValueType::Default;
     std::string                 value;
     std::map<std::string, Node> map;
     std::vector<Node>           array;
+
+    inline Node()
+    : type(NodeValueType::Default), value(), map(), array() { }
+
+    inline Node(std::string string)
+    : type(NodeValueType::String), value(string), map(), array() { }
+
+    inline Node(const char *string) : Node(std::string{string}) { }
+
+    inline Node(bool b)
+    : type(NodeValueType::RawValue), value(b ? "true" : "false")
+    , map(), array() { }
+
+    inline Node(int64_t i64)
+    : type(NodeValueType::RawValue), value(), map(), array()
+    {
+        std::ostringstream os{};
+        os << i64;
+        value = os.str();
+    }
+
+    inline Node(uint64_t u64)
+    : type(NodeValueType::RawValue), value(), map(), array()
+    {
+        std::ostringstream os{};
+        os << u64;
+        value = os.str();
+    }
 };
 
+static inline Node makeNode(std::string value) {
+    Node node;
+    node.value = value;
+    return node;
+}
 
 } // namespace json
 } // namespace dyld3

dyld3/JSONReader.h

@@ -34,6 +34,7 @@ namespace dyld3 {
 namespace json {
 
 Node readJSON(Diagnostics& diags, const char* filePath);
+Node readJSON(Diagnostics& diags, const void* contents, size_t length);
 
 // Given a map node, returns the node representing the given value.
 // If it is missing, returns a sentinel node and sets an error on the diagnostic
@@ -46,6 +47,9 @@ const Node* getOptionalValue(Diagnostics& diags, const Node& node, const char* k
 // Parses an int from the given node, or throws an error if its not an integer payload
 uint64_t parseRequiredInt(Diagnostics& diags, const Node& node);
 
+// Parses a bool from the given node, or throws an error if its not a boolean payload
+bool parseRequiredBool(Diagnostics& diags, const Node& node);
+
 // Parses a string from the given node, or throws an error if its not a string payload
 const std::string& parseRequiredString(Diagnostics& diags, const Node& node);
 

dyld3/JSONReader.mm

@@ -94,6 +94,33 @@ uint64_t parseRequiredInt(Diagnostics& diags, const Node& node) {
     return atoi(node.value.c_str());
 }
 
+bool parseRequiredBool(Diagnostics& diags, const Node& node) {
+    if (diags.hasError())
+        return false;
+
+    if (!node.array.empty()) {
+        diags.error("Cannot get integer value from array node\n");
+        return false;
+    }
+    if (!node.map.empty()) {
+        diags.error("Cannot get integer value from value node\n");
+        return false;
+    }
+    if (node.value.empty()) {
+        diags.error("Cannot get integer value from empty node\n");
+        return false;
+    }
+
+    if ( (node.value == "true") || (node.value == "1") )
+        return true;
+
+    if ( (node.value == "false") || (node.value == "0") )
+        return false;
+
+    diags.error("Boolean value should be true/false/0/1\n");
+    return false;
+}
+
 const std::string& parseRequiredString(Diagnostics& diags, const Node& node) {
     static std::string sentinelString = "";
 
@@ -168,6 +195,12 @@ Node parseNode(Diagnostics& diags, id jsonObject) {
         return node;
     }
 
+    // NSBoolean -> string
+    if ([jsonObject isKindOfClass:[NSNumber class]]) {
+        node.value = [[(NSNumber*)jsonObject stringValue] UTF8String];
+        return node;
+    }
+
     diags.error("Unknown json deserialized type\n");
     return Node();
 }
@@ -193,5 +226,17 @@ Node readJSON(Diagnostics& diags, const char* filePath) {
     return node;
 }
 
+Node readJSON(Diagnostics& diags, const void * contents, size_t length) {
+    NSData* data = [NSData dataWithBytes:contents length:length];
+    NSError* error = nil;
+    id jsonObject = [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:&error];
+    if (!jsonObject) {
+        diags.error("Could not deserialize json because '%s'\n", [[error localizedFailureReason] UTF8String]);
+        return Node();
+    }
+
+    return parseNode(diags, jsonObject);
+}
+
 } //namespace json
 } //namespace dyld3

dyld3/JSONWriter.h

@@ -61,7 +61,7 @@ static inline void indentBy(uint32_t spaces, std::ostream& out) {
     }
 }
 
-static inline void printJSON(const Node& node, uint32_t indent, std::ostream& out)
+static inline void printJSON(const Node& node, uint32_t indent = 0, std::ostream& out = std::cout)
 {
     if ( !node.map.empty() ) {
         out << "{";
@@ -95,19 +95,54 @@ static inline void printJSON(const Node& node, uint32_t indent, std::ostream& ou
         out << "]";
     }
     else {
+        auto &value = node.value;
+        switch (node.type) {
+        case NodeValueType::Default:
+        case NodeValueType::String:
+            if (value.find('"') == std::string::npos) {
+                out << "\"" << value << "\"";
+            } else {
                 std::string escapedString;
-        escapedString.reserve(node.value.size());
-        for (char c : node.value) {
+                escapedString.reserve(value.size());
+                for (char c : value) {
                     if (c == '"')
                         escapedString += '\\';
                     escapedString += c;
                 }
                 out << "\"" << escapedString << "\"";
             }
+            break;
+        case NodeValueType::RawValue:
+            out << value;
+            break;
+        }
+    }
     if ( indent == 0 )
         out << "\n";
 }
 
+static inline void streamArrayBegin(bool &needsComma, std::ostream& out = std::cout)
+{
+    out << "[";
+    needsComma = false;
+}
+
+static inline void streamArrayNode(bool &needsComma, Node &node, std::ostream& out = std::cout)
+{
+    if (needsComma)
+        out << ",";
+    out << "\n";
+    indentBy(2, out);
+    printJSON(node, 2, out);
+    needsComma = true;
+}
+
+static inline void streamArrayEnd(bool &needsComma, std::ostream& out = std::cout)
+{
+    if (needsComma)
+        out << "\n";
+    out << "]\n";
+}
 
 } // namespace json
 } // namespace dyld3

dyld3/Loading.cpp

@@ -46,14 +46,17 @@
 //#include <dispatch/dispatch.h>
 #include <mach/vm_page_size.h>
 
+#include "ClosureFileSystemPhysical.h"
 #include "MachOFile.h"
 #include "MachOLoaded.h"
 #include "MachOAnalyzer.h"
 #include "Logging.h"
 #include "Loading.h"
+#include "RootsChecker.h"
 #include "Tracing.h"
 #include "dyld2.h"
 #include "dyld_cache_format.h"
+#include "libdyldEntryVector.h"
 
 #include "objc-shared-cache.h"
 
@@ -103,11 +106,15 @@ namespace dyld3 {
 Loader::Loader(const Array<LoadedImage>& existingImages, Array<LoadedImage>& newImagesStorage,
                const void* cacheAddress, const Array<const dyld3::closure::ImageArray*>& imagesArrays,
                const closure::ObjCSelectorOpt* selOpt, const Array<closure::Image::ObjCSelectorImage>& selImages,
-               LogFunc logLoads, LogFunc logSegments, LogFunc logFixups, LogFunc logDofs)
+               const RootsChecker& rootsChecker, dyld3::Platform platform,
+               LogFunc logLoads, LogFunc logSegments, LogFunc logFixups, LogFunc logDof,
+               bool allowMissingLazies, dyld3::LaunchErrorInfo* launchErrorInfo)
        : _existingImages(existingImages), _newImages(newImagesStorage),
          _imagesArrays(imagesArrays), _dyldCacheAddress(cacheAddress), _dyldCacheSelectorOpt(nullptr),
          _closureSelectorOpt(selOpt), _closureSelectorImages(selImages),
-         _logLoads(logLoads), _logSegments(logSegments), _logFixups(logFixups), _logDofs(logDofs)
+         _rootsChecker(rootsChecker), _allowMissingLazies(allowMissingLazies), _platform(platform),
+         _logLoads(logLoads), _logSegments(logSegments), _logFixups(logFixups), _logDofs(logDof), _launchErrorInfo(launchErrorInfo)
+
 {
 #if BUILDING_DYLD
     // This is only needed for dyld and the launch closure, not the dlopen closures
@@ -122,7 +129,7 @@ void Loader::addImage(const LoadedImage& li)
     _newImages.push_back(li);
 }
 
-LoadedImage* Loader::findImage(closure::ImageNum targetImageNum)
+LoadedImage* Loader::findImage(closure::ImageNum targetImageNum) const
 {
 #if BUILDING_DYLD
     // The launch images are different in dyld vs libdyld.  In dyld, the new images are
@@ -131,7 +138,7 @@ LoadedImage* Loader::findImage(closure::ImageNum targetImageNum)
         return info;
     }
 
-    for (uint64_t index = 0; index != _newImages.count(); ++index) {
+    for (uintptr_t index = 0; index != _newImages.count(); ++index) {
         LoadedImage& info = _newImages[index];
         if ( info.image()->representsImageNum(targetImageNum) ) {
             // Try cache this entry for next time
@@ -180,11 +187,16 @@ uintptr_t Loader::resolveTarget(closure::Image::ResolvedSymbolTarget target)
 
 void Loader::completeAllDependents(Diagnostics& diag, bool& someCacheImageOverridden)
 {
-    // accumulate all image overrides
-    STACK_ALLOC_ARRAY(ImageOverride, overrides, _existingImages.maxCount() + _newImages.maxCount());
+    bool iOSonMac = (_platform == Platform::iOSMac);
+#if (TARGET_OS_OSX && TARGET_CPU_ARM64)
+    if ( _platform == Platform::iOS )
+        iOSonMac = true;
+#endif
+    // accumulate all image overrides (512 is placeholder for max unzippered twins in dyld cache)
+    STACK_ALLOC_ARRAY(ImageOverride, overrides, _existingImages.maxCount() + _newImages.maxCount() + 512);
     for (const auto anArray : _imagesArrays) {
-        // ignore prebuilt Image* in dyld cache
-        if ( anArray->startImageNum() < dyld3::closure::kFirstLaunchClosureImageNum )
+        // ignore prebuilt Image* in dyld cache, except for MacCatalyst apps where unzipped twins can override each other
+        if ( (anArray->startImageNum() < dyld3::closure::kFirstLaunchClosureImageNum) && !iOSonMac )
             continue;
         anArray->forEachImage(^(const dyld3::closure::Image* image, bool& stop) {
             ImageOverride overrideEntry;
@@ -207,7 +219,7 @@ void Loader::completeAllDependents(Diagnostics& diag, bool& someCacheImageOverri
     uintptr_t index = 0;
     while ( (index < _newImages.count()) && diag.noError() ) {
         const closure::Image* image = _newImages[index].image();
-        //fprintf(stderr, "completeAllDependents(): looking at dependents of %s\n", image->path());
+        //dyld::log("completeAllDependents(): looking at dependents of %s\n", image->path());
         image->forEachDependentImage(^(uint32_t depIndex, closure::Image::LinkKind kind, closure::ImageNum depImageNum, bool& stop) {
             // check if imageNum needs to be changed to an override
             for (const ImageOverride& entry : overrides) {
@@ -241,8 +253,11 @@ void Loader::completeAllDependents(Diagnostics& diag, bool& someCacheImageOverri
     }
 }
 
-void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool fromOFI)
+void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool fromOFI, bool* closureOutOfDate, bool* recoverable)
 {
+    *closureOutOfDate = false;
+    *recoverable      = true;
+
     // scan array and map images not already loaded
     for (LoadedImage& info : _newImages) {
         if ( info.loadedAddress() != nullptr ) {
@@ -267,21 +282,21 @@ void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool from
         if ( info.image()->inDyldCache() ) {
             if ( info.image()->overridableDylib() ) {
                 struct stat statBuf;
-                if ( stat(info.image()->path(), &statBuf) == 0 ) {
-                    // verify file has not changed since closure was built
-                    uint64_t inode;
-                    uint64_t mtime;
-                    if ( info.image()->hasFileModTimeAndInode(inode, mtime) ) {
-                        if ( (statBuf.st_mtime != mtime) || (statBuf.st_ino != inode) ) {
+                if ( dyld3::stat(info.image()->path(), &statBuf) == 0 ) {
+                    dyld3::closure::FileSystemPhysical fileSystem;
+                    if ( _rootsChecker.onDiskFileIsRoot(info.image()->path(), (const DyldSharedCache*)_dyldCacheAddress, info.image(),
+                                                        &fileSystem, statBuf.st_ino, statBuf.st_mtime) ) {
+                        if ( ((const DyldSharedCache*)_dyldCacheAddress)->header.dylibsExpectedOnDisk ) {
                             diag.error("dylib file mtime/inode changed since closure was built for '%s'", info.image()->path());
-                        }
-                    }
-                    else {
+                        } else {
                             diag.error("dylib file not expected on disk, must be a root '%s'", info.image()->path());
                         }
+                        *closureOutOfDate = true;
+                    }
                 }
                 else if ( (_dyldCacheAddress != nullptr) && ((dyld_cache_header*)_dyldCacheAddress)->dylibsExpectedOnDisk ) {
                     diag.error("dylib file missing, was in dyld shared cache '%s'", info.image()->path());
+                    *closureOutOfDate = true;
                 }
             }
             if ( diag.noError() ) {
@@ -297,28 +312,28 @@ void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool from
             }
         }
         else {
-            mapImage(diag, info, fromOFI);
+            mapImage(diag, info, fromOFI, closureOutOfDate);
             if ( diag.hasError() )
                 break; // out of for loop
         }
 
     }
     if ( diag.hasError() ) {
-        // bummer, need to clean up by unmapping any images just mapped
-        for (LoadedImage& info : _newImages) {
-            if ( (info.state() == LoadedImage::State::mapped) && !info.image()->inDyldCache() && !info.leaveMapped() ) {
-                _logSegments("dyld: unmapping %s\n", info.image()->path());
-                unmapImage(info);
-            }
-        }
+        // need to clean up by unmapping any images just mapped
+        unmapAllImages();
         return;
     }
 
-    // apply fixups
+    // apply fixups to all but main executable
+    LoadedImage* mainInfo = nullptr;
     for (LoadedImage& info : _newImages) {
         // images in shared cache do not need fixups applied
         if ( info.image()->inDyldCache() )
             continue;
+        if ( info.loadedAddress()->filetype == MH_EXECUTE ) {
+            mainInfo = &info;
+            continue;
+        }
         // previously loaded images were previously fixed up
         if ( info.state() < LoadedImage::State::fixedUp ) {
             applyFixupsToImage(diag, info);
@@ -327,6 +342,26 @@ void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool from
             info.setState(LoadedImage::State::fixedUp);
         }
     }
+    if ( diag.hasError() ) {
+        // need to clean up by unmapping any images just mapped
+        unmapAllImages();
+        return;
+    }
+
+    if ( mainInfo != nullptr ) {
+        // now apply fixups to main executable
+        // we do it in this order so that if there is a problem with the dylibs in the closure
+        // the main executable is left untouched so the closure can be rebuilt
+        applyFixupsToImage(diag, *mainInfo);
+        if ( diag.hasError() ) {
+            // need to clean up by unmapping any images just mapped
+            unmapAllImages();
+            // we have already started fixing up the main executable, so we cannot retry the launch again
+            *recoverable = false;
+            return;
+        }
+        mainInfo->setState(LoadedImage::State::fixedUp);
+    }
 
     // find and register dtrace DOFs
     if ( processDOFs ) {
@@ -344,6 +379,18 @@ void Loader::mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool from
     }
 }
 
+void Loader::unmapAllImages()
+{
+    for (LoadedImage& info : _newImages) {
+        if ( !info.image()->inDyldCache() && !info.leaveMapped() ) {
+            if ( (info.state() == LoadedImage::State::mapped) || (info.state() == LoadedImage::State::fixedUp) ) {
+                _logSegments("dyld: unmapping %s\n", info.image()->path());
+                unmapImage(info);
+            }
+        }
+    }
+}
+
 bool Loader::sandboxBlocked(const char* path, const char* kind)
 {
 #if TARGET_OS_SIMULATOR || TARGET_OS_DRIVERKIT
@@ -370,7 +417,7 @@ bool Loader::sandboxBlockedStat(const char* path)
     return sandboxBlocked(path, "file-read-metadata");
 }
 
-void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
+void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI, bool* closureOutOfDate)
 {
     dyld3::ScopedTimer timer(DBG_DYLD_TIMING_MAP_IMAGE, info.image()->path(), 0, 0);
 
@@ -382,11 +429,7 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
     bool                    isCodeSigned  = image->hasCodeSignature(codeSignFileOffset, codeSignFileSize);
 
     // open file
-#if BUILDING_DYLD
-    int fd = dyld::my_open(info.image()->path(), O_RDONLY, 0);
-#else
-    int fd = ::open(info.image()->path(), O_RDONLY, 0);
-#endif
+    int fd = dyld3::open(info.image()->path(), O_RDONLY, 0);
     if ( fd == -1 ) {
         int openErr = errno;
         if ( (openErr == EPERM) && sandboxBlockedOpen(image->path()) )
@@ -399,7 +442,7 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
     // get file info
     struct stat statBuf;
 #if TARGET_OS_SIMULATOR
-    if ( stat(image->path(), &statBuf) != 0 ) {
+    if ( dyld3::stat(image->path(), &statBuf) != 0 ) {
 #else
     if ( fstat(fd, &statBuf) != 0 ) {
 #endif
@@ -418,6 +461,7 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
     if ( image->hasFileModTimeAndInode(inode, mtime) ) {
         if ( (statBuf.st_mtime != mtime) || (statBuf.st_ino != inode) ) {
             diag.error("file mtime/inode changed since closure was built for '%s'", image->path());
+            *closureOutOfDate = true;
             close(fd);
             return;
         }
@@ -433,6 +477,23 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
         }
     }
 
+    if ( isCodeSigned && (sliceOffset == 0) ) {
+        uint64_t expectedFileSize = round_page_kernel(codeSignFileOffset+codeSignFileSize);
+        uint64_t actualFileSize = round_page_kernel(statBuf.st_size);
+        if ( actualFileSize < expectedFileSize ) {
+            diag.error("File size too small for code signature");
+            *closureOutOfDate = true;
+            close(fd);
+            return;
+        }
+        if ( actualFileSize != expectedFileSize ) {
+            diag.error("File size doesn't match code signature");
+            *closureOutOfDate = true;
+            close(fd);
+            return;
+        }
+    }
+
     // register code signature
     uint64_t coveredCodeLength  = UINT64_MAX;
     if ( isCodeSigned ) {
@@ -447,6 +508,13 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
             if ( (errnoCopy == EPERM) || (errnoCopy == EBADEXEC) ) {
                 diag.error("code signature invalid (errno=%d) sliceOffset=0x%08llX, codeBlobOffset=0x%08X, codeBlobSize=0x%08X for '%s'",
                             errnoCopy, sliceOffset, codeSignFileOffset, codeSignFileSize, image->path());
+#if BUILDING_LIBDYLD
+                if ( errnoCopy == EBADEXEC ) {
+                    // dlopen closures many be prebuilt in to the shared cache with a code signature, but the dylib is replaced
+                    // with one without a code signature.  In that case, lets build a new closure
+                    *closureOutOfDate = true;
+                }
+#endif
             }
             else {
                 diag.error("fcntl(fd, F_ADDFILESIGS_RETURN) failed with errno=%d, sliceOffset=0x%08llX, codeBlobOffset=0x%08X, codeBlobSize=0x%08X for '%s'",
@@ -603,6 +671,7 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
             }
         }
         if ( diag.hasError() ) {
+            *closureOutOfDate = true;
             ::vm_deallocate(mach_task_self(), loadAddress, (vm_size_t)totalVMSize);
             return;
         }
@@ -610,7 +679,7 @@ void Loader::mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI)
 
 #endif
 
-#if __IPHONE_OS_VERSION_MIN_REQUIRED && !TARGET_OS_SIMULATOR
+#if (__arm__ || __arm64__) && !TARGET_OS_SIMULATOR
     // tell kernel about fairplay encrypted regions
     uint32_t fpTextOffset;
     uint32_t fpSize;
@@ -644,7 +713,7 @@ void Loader::registerDOFs(const Array<DOFInfo>& dofs)
     if ( dofs.empty() )
         return;
 
-    int fd = open("/dev/" DTRACEMNR_HELPER, O_RDWR);
+    int fd = ::open("/dev/" DTRACEMNR_HELPER, O_RDWR);
     if ( fd < 0 ) {
         _logDofs("can't open /dev/" DTRACEMNR_HELPER " to register dtrace DOF sections\n");
     }
@@ -682,18 +751,18 @@ void Loader::registerDOFs(const Array<DOFInfo>& dofs)
 
 bool Loader::dtraceUserProbesEnabled()
 {
-#ifdef DARLING
-	return false;
-#else
+#if !TARGET_OS_SIMULATOR
     uint8_t dofEnabled = *((uint8_t*)_COMM_PAGE_DTRACE_DOF_ENABLED);
     return ( (dofEnabled & 1) );
+#else
+    return false;
 #endif
 }
 
 
 void Loader::vmAccountingSetSuspended(bool suspend, LogFunc logger)
 {
-#if __arm__ || __arm64__
+#if TARGET_OS_IPHONE && !TARGET_OS_SIMULATOR
     // <rdar://problem/29099600> dyld should tell the kernel when it is doing fix-ups caused by roots
     logger("vm.footprint_suspend=%d\n", suspend);
     int newValue = suspend ? 1 : 0;
@@ -704,6 +773,21 @@ void Loader::vmAccountingSetSuspended(bool suspend, LogFunc logger)
 #endif
 }
 
+static const char* targetString(const MachOAnalyzerSet::FixupTarget& target)
+{
+    switch (target.kind ) {
+        case MachOAnalyzerSet::FixupTarget::Kind::rebase:
+            return "rebase";
+        case MachOAnalyzerSet::FixupTarget::Kind::bindAbsolute:
+            return "abolute";
+        case MachOAnalyzerSet::FixupTarget::Kind::bindToImage:
+            return target.foundSymbolName;
+        case MachOAnalyzerSet::FixupTarget::Kind::bindMissingSymbol:
+            return "missing";
+    }
+    return "";
+}
+
 void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
 {
     dyld3::ScopedTimer timer(DBG_DYLD_TIMING_APPLY_FIXUPS, (uint64_t)info.loadedAddress(), 0, 0);
@@ -716,6 +800,94 @@ void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
     
     if ( overrideOfCache )
         vmAccountingSetSuspended(true, _logFixups);
+    if ( image->fixupsNotEncoded() ) {
+        // make the cache writable for this block
+        // We do this lazily, only if we find a symbol which needs to be overridden
+        DyldSharedCache::DataConstLazyScopedWriter patcher((const DyldSharedCache*)_dyldCacheAddress, mach_task_self(), (DyldSharedCache::DataConstLogFunc)_logSegments);
+        auto* patcherPtr = &patcher;
+
+        WrappedMachO wmo((MachOAnalyzer*)info.loadedAddress(), this, (void*)info.image());
+        wmo.forEachFixup(diag,
+        ^(uint64_t fixupLocRuntimeOffset, PointerMetaData pmd, const FixupTarget& target, bool& stop) {
+            uintptr_t* fixUpLoc = (uintptr_t*)(imageLoadAddress + fixupLocRuntimeOffset);
+            uintptr_t value;
+            switch ( target.kind ) {
+                case MachOAnalyzerSet::FixupTarget::Kind::rebase:
+                case MachOAnalyzerSet::FixupTarget::Kind::bindToImage:
+                    value = (uintptr_t)(target.foundInImage._mh) + target.offsetInImage;
+                    break;
+                case MachOAnalyzerSet::FixupTarget::Kind::bindAbsolute:
+                    value = (uintptr_t)target.offsetInImage;
+                    break;
+                case MachOAnalyzerSet::FixupTarget::Kind::bindMissingSymbol:
+                    if ( _launchErrorInfo ) {
+                        _launchErrorInfo->kind              = DYLD_EXIT_REASON_SYMBOL_MISSING;
+                        _launchErrorInfo->clientOfDylibPath = info.image()->path();
+                        _launchErrorInfo->targetDylibPath   = target.foundInImage.path();
+                        _launchErrorInfo->symbol            = target.requestedSymbolName;
+                    }
+                    // we have no value to set, and forEachFixup() is about to finish
+                    return;
+            }
+    #if __has_feature(ptrauth_calls)
+            if ( pmd.authenticated )
+                value = MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signPointer(value, fixUpLoc, pmd.usesAddrDiversity, pmd.diversity, pmd.key);
+    #endif
+            if ( pmd.high8 )
+                value |= ((uint64_t)pmd.high8 << 56);
+            _logFixups("dyld: fixup: %s:%p = %p (%s)\n", leafName, fixUpLoc, (void*)value, targetString(target));
+            *fixUpLoc = value;
+        },
+        ^(uint32_t cachedDylibIndex, uint32_t exportCacheOffset, const FixupTarget& target) {
+#if BUILDING_LIBDYLD && __x86_64__
+            // Full dlopen closures don't patch weak defs.  Bail out early if we are libdyld to match this behaviour
+            return;
+#endif
+            patcherPtr->makeWriteable();
+            ((const DyldSharedCache*)_dyldCacheAddress)->forEachPatchableUseOfExport(cachedDylibIndex, exportCacheOffset, ^(dyld_cache_patchable_location patchLoc) {
+                uintptr_t* loc     = (uintptr_t*)(((uint8_t*)_dyldCacheAddress)+patchLoc.cacheOffset);
+                uintptr_t  newImpl = (uintptr_t)(target.foundInImage._mh) + target.offsetInImage + DyldSharedCache::getAddend(patchLoc);
+    #if __has_feature(ptrauth_calls)
+                if ( patchLoc.authenticated )
+                    newImpl = MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signPointer(newImpl, loc, patchLoc.usesAddressDiversity, patchLoc.discriminator, patchLoc.key);
+    #endif
+                // ignore duplicate patch entries
+                if ( *loc != newImpl ) {
+                    _logFixups("dyld: cache patch: %p = 0x%0lX\n", loc, newImpl);
+                    *loc = newImpl;
+                }
+            });
+        });
+#if BUILDING_LIBDYLD && TARGET_OS_OSX
+        // <rdar://problem/59265987> support old licenseware plugins on macOS using minimal closures
+        __block bool oldBinary = true;
+        info.loadedAddress()->forEachSupportedPlatform(^(Platform platform, uint32_t minOS, uint32_t sdk) {
+            if ( (platform == Platform::macOS) && (sdk >= 0x000A0F00) )
+                oldBinary = false;
+        });
+        if ( oldBinary ) {
+            // look for __DATA,__dyld section
+            info.loadedAddress()->forEachSection(^(const MachOAnalyzer::SectionInfo& sectInfo, bool malformedSectionRange, bool& stop) {
+                if ( (strcmp(sectInfo.sectName, "__dyld") == 0) && (strcmp(sectInfo.segInfo.segName, "__DATA") == 0) ) {
+                    // dyld_func_lookup is second pointer in __dyld section
+                    uintptr_t* dyldSection = (uintptr_t*)(sectInfo.sectAddr + (uintptr_t)info.loadedAddress());
+                    _logFixups("dyld: __dyld section: %p = %p\n", &dyldSection[1], &dyld3::compatFuncLookup);
+                    dyldSection[1] = (uintptr_t)&dyld3::compatFuncLookup;
+                 }
+            });
+        }
+#endif
+    }
+    else {
+        if ( image->rebasesNotEncoded() ) {
+            // <rdar://problem/56172089> some apps have so many rebases the closure file is too big, instead we go back to rebase opcodes
+            ((MachOAnalyzer*)imageLoadAddress)->forEachRebase(diag, true, ^(uint64_t imageOffsetToRebase, bool& stop) {
+                // this is a rebase, add slide
+                uintptr_t* fixUpLoc = (uintptr_t*)(imageLoadAddress + imageOffsetToRebase);
+                *fixUpLoc += slide;
+                _logFixups("dyld: fixup: %s:%p += %p\n", leafName, fixUpLoc, (void*)slide);
+            });
+        }
         image->forEachFixup(^(uint64_t imageOffsetToRebase, bool& stop) {
             // this is a rebase, add slide
             uintptr_t* fixUpLoc = (uintptr_t*)(imageLoadAddress + imageOffsetToRebase);
@@ -752,6 +924,12 @@ void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
             // this is a bind, set to target
             uintptr_t* fixUpLoc = (uintptr_t*)(imageLoadAddress + imageOffsetToBind);
             uintptr_t value = resolveTarget(bindTarget);
+#if __has_feature(ptrauth_calls)
+            // Sign the ISA on arm64e.
+            // Unfortunately a hard coded value here is not ideal, but this is ABI so we aren't going to change it
+            // This matches the value in libobjc __objc_opt_ptrs: .quad x@AUTH(da, 27361, addr)
+            value = MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signPointer(value, fixUpLoc, true, 27361, 2);
+#endif
             _logFixups("dyld: fixup objc protocol: %s:%p = %p\n", leafName, fixUpLoc, (void*)value);
             *fixUpLoc = value;
         },
@@ -771,7 +949,6 @@ void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
                 fixupTarget.image.imageNum = imageNum;
                 fixupTarget.image.offset = vmOffset;
             }
-
             uintptr_t* fixUpLoc = (uintptr_t*)(imageLoadAddress + imageOffsetToFixup);
             uintptr_t value = resolveTarget(fixupTarget);
             _logFixups("dyld: fixup objc selector: %s:%p(was '%s') = %p(now '%s')\n", leafName, fixUpLoc, (const char*)*fixUpLoc, (void*)value, (const char*)value);
@@ -785,7 +962,12 @@ void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
             _logFixups("dyld: fixup objc stable Swift: %s:%p = %p\n", leafName, fixUpLoc, (void*)value);
             *fixUpLoc = value;
         }, ^(uint64_t imageOffsetToFixup, bool &stop) {
-        // TODO: Implement this
+            // fixupObjCMethodList
+            // Set the method list to have the uniqued bit set
+            uint32_t* fixUpLoc = (uint32_t*)(imageLoadAddress + imageOffsetToFixup);
+            uint32_t value = (*fixUpLoc) | MachOAnalyzer::ObjCMethodList::methodListIsUniqued;
+            _logFixups("dyld: fixup objc method list: %s:%p = 0x%08x\n", leafName, fixUpLoc, value);
+            *fixUpLoc = value;
         });
 
 #if __i386__
@@ -803,6 +985,7 @@ void Loader::applyFixupsToImage(Diagnostics& diag, LoadedImage& info)
         if ( segmentsMadeWritable )
             setSegmentProtects(info, false);
 #endif
+    }
 
     // make any read-only data segments read-only
     if ( image->hasReadOnlyData() && !image->inDyldCache() ) {
@@ -832,13 +1015,112 @@ void Loader::setSegmentProtects(const LoadedImage& info, bool write)
 }
 #endif
 
+
+void Loader::forEachImage(void (^handler)(const LoadedImage& li, bool& stop)) const
+{
+    bool stop = false;
+    for (const LoadedImage& li : _existingImages) {
+        handler(li, stop);
+        if ( stop )
+            return;
+    }
+    for (const LoadedImage& li : _newImages) {
+        handler(li, stop);
+        if ( stop )
+            return;
+    }
+}
+
+void Loader::mas_forEachImage(void (^handler)(const WrappedMachO& wmo, bool hidden, bool& stop)) const
+{
+    forEachImage(^(const LoadedImage& li, bool& stop) {
+        WrappedMachO wmo((MachOAnalyzer*)li.loadedAddress(), this, (void*)li.image());
+        handler(wmo, li.hideFromFlatSearch(), stop);
+    });
+}
+
+
+bool Loader::wmo_missingSymbolResolver(const WrappedMachO* fromWmo, bool weakImport, bool lazyBind, const char* symbolName, const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const
+{
+    if ( weakImport ) {
+        target.offsetInImage = 0;
+        target.kind  = FixupTarget::Kind::bindAbsolute;
+        return true;
+    }
+
+    if ( lazyBind && _allowMissingLazies ) {
+        __block bool result = false;
+        forEachImage(^(const LoadedImage& li, bool& stop) {
+            if ( li.loadedAddress()->isDylib() && (strcmp(li.loadedAddress()->installName(), "/usr/lib/system/libdyld.dylib") == 0) ) {
+                WrappedMachO libdyldWmo((MachOAnalyzer*)li.loadedAddress(), this, (void*)li.image());
+                Diagnostics  diag;
+                if ( libdyldWmo.findSymbolIn(diag, "__dyld_missing_symbol_abort", 0, target) ) {
+                     // <rdar://problem/44315944> closures should bind missing lazy-bind symbols to a missing symbol handler in libdyld in flat namespace
+                     result = true;
+                }
+                stop = true;
+            }
+        });
+        return result;
+    }
+
+    // FIXME
+    return false;
+}
+
+
+void Loader::mas_mainExecutable(WrappedMachO& mainWmo) const
+{
+    forEachImage(^(const LoadedImage& li, bool& stop) {
+        if ( li.loadedAddress()->isMainExecutable() ) {
+            WrappedMachO wmo((MachOAnalyzer*)li.loadedAddress(), this, (void*)li.image());
+            mainWmo = wmo;
+            stop = true;
+        }
+    });
+}
+
+void* Loader::mas_dyldCache() const
+{
+    return (void*)_dyldCacheAddress;
+}
+
+
+bool Loader::wmo_dependent(const WrappedMachO* wmo, uint32_t depIndex, WrappedMachO& childWmo, bool& missingWeakDylib) const
+{
+    const closure::Image* image = (closure::Image*)(wmo->_other);
+    closure::ImageNum depImageNum = image->dependentImageNum(depIndex);
+    if ( depImageNum == closure::kMissingWeakLinkedImage ) {
+        missingWeakDylib = true;
+        return true;
+    }
+    else {
+        if ( LoadedImage* li = findImage(depImageNum) ) {
+            WrappedMachO foundWmo((MachOAnalyzer*)li->loadedAddress(), this, (void*)li->image());
+            missingWeakDylib = false;
+            childWmo = foundWmo;
+            return true;
+        }
+    }
+    return false;
+}
+
+
+const char* Loader::wmo_path(const WrappedMachO* wmo) const
+{
+    const closure::Image* image = (closure::Image*)(wmo->_other);
+    return image->path();
+}
+
+
+
 #if BUILDING_DYLD
 LoadedImage* Loader::LaunchImagesCache::findImage(closure::ImageNum imageNum,
                                                   Array<LoadedImage>& images) const {
     if ( (imageNum < _firstImageNum) || (imageNum >= _lastImageNum) )
         return nullptr;
 
-    uint64_t cacheIndex = imageNum - _firstImageNum;
+    unsigned int cacheIndex = imageNum - _firstImageNum;
     uint32_t imagesIndex = _imageIndices[cacheIndex];
     if ( imagesIndex == 0 )
         return nullptr;
@@ -847,16 +1129,17 @@ LoadedImage* Loader::LaunchImagesCache::findImage(closure::ImageNum imageNum,
     return &images[imagesIndex - 1];
 }
 
-void Loader::LaunchImagesCache::tryAddImage(closure::ImageNum imageNum,
-                                            uint64_t allImagesIndex) {
+void Loader::LaunchImagesCache::tryAddImage(closure::ImageNum imageNum, uint64_t allImagesIndex) const {
     if ( (imageNum < _firstImageNum) || (imageNum >= _lastImageNum) )
         return;
 
-    uint64_t cacheIndex = imageNum - _firstImageNum;
+    unsigned int cacheIndex = imageNum - _firstImageNum;
 
     // Note the index is offset by 1 so that 0's are not yet set
     _imageIndices[cacheIndex] = (uint32_t)allImagesIndex + 1;
 }
+ #endif
+
 
 void forEachLineInFile(const char* buffer, size_t bufferLen, void (^lineHandler)(const char* line, bool& stop))
 {
@@ -880,7 +1163,7 @@ void forEachLineInFile(const char* buffer, size_t bufferLen, void (^lineHandler)
 
 void forEachLineInFile(const char* path, void (^lineHandler)(const char* line, bool& stop))
 {
-    int fd = dyld::my_open(path, O_RDONLY, 0);
+    int fd = dyld3::open(path, O_RDONLY, 0);
     if ( fd != -1 ) {
         struct stat statBuf;
         if ( fstat(fd, &statBuf) == 0 ) {
@@ -894,14 +1177,13 @@ void forEachLineInFile(const char* path, void (^lineHandler)(const char* line, b
     }
 }
 
-#endif
 
 #if (BUILDING_LIBDYLD || BUILDING_DYLD)
     bool internalInstall()
     {
 #if TARGET_OS_SIMULATOR
         return false;
-#elif __IPHONE_OS_VERSION_MIN_REQUIRED
+#elif TARGET_OS_IPHONE
         uint32_t devFlags = *((uint32_t*)_COMM_PAGE_DEV_FIRM);
         return ( (devFlags & 1) == 1 );
 #else

dyld3/Loading.h

@@ -33,6 +33,7 @@
 
 #include "Closure.h"
 #include "MachOLoaded.h"
+#include "MachOAnalyzerSet.h"
 
 namespace objc_opt {
 struct objc_clsopt_t;
@@ -41,6 +42,15 @@ struct objc_selopt_t;
 
 namespace dyld3 {
 
+class RootsChecker;
+
+struct LaunchErrorInfo
+{
+    uintptr_t       kind;
+    const char*     clientOfDylibPath;
+    const char*     targetDylibPath;
+    const char*     symbol;
+};
 
 //
 // Tuple of info about a loaded image. Contains the loaded address, Image*, and state.
@@ -89,20 +99,23 @@ private:
 //
 // Utility class to recursively load dependents
 //
-class VIS_HIDDEN Loader {
+class VIS_HIDDEN Loader : public MachOAnalyzerSet {
 public:
         typedef bool (*LogFunc)(const char*, ...) __attribute__((format(printf, 1, 2)));
 
                         Loader(const Array<LoadedImage>& existingImages, Array<LoadedImage>& newImagesStorage,
                                const void* cacheAddress, const Array<const dyld3::closure::ImageArray*>& imagesArrays,
                                const closure::ObjCSelectorOpt* selOpt, const Array<closure::Image::ObjCSelectorImage>& selImages,
-                               LogFunc log_loads, LogFunc log_segments, LogFunc log_fixups, LogFunc log_dofs);
+                               const RootsChecker& rootsChecker, dyld3::Platform platform,
+                               LogFunc log_loads, LogFunc log_segments, LogFunc log_fixups, LogFunc log_dofs,
+                               bool allowMissingLazies=false, dyld3::LaunchErrorInfo* launchErrorInfo=nullptr);
 
     void                addImage(const LoadedImage&);
     void                completeAllDependents(Diagnostics& diag, bool& someCacheImageOverridden);
-    void                mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool fromOFI=false);
+    void                mapAndFixupAllImages(Diagnostics& diag, bool processDOFs, bool fromOFI, bool* closureOutOfDate, bool* recoverable);
     uintptr_t           resolveTarget(closure::Image::ResolvedSymbolTarget target);
-    LoadedImage*        findImage(closure::ImageNum targetImageNum);
+    LoadedImage*        findImage(closure::ImageNum targetImageNum) const;
+    void                forEachImage(void (^handler)(const LoadedImage& li, bool& stop)) const;
 
     static void         unmapImage(LoadedImage& info);
     static bool         dtraceUserProbesEnabled();
@@ -126,7 +139,7 @@ private:
     struct LaunchImagesCache {
         LoadedImage*                    findImage(closure::ImageNum targetImageNum,
                                                   Array<LoadedImage>& images) const;
-        void                            tryAddImage(closure::ImageNum targetImageNum, uint64_t allImagesIndex);
+        void                            tryAddImage(closure::ImageNum targetImageNum, uint64_t allImagesIndex) const;
 
         static const uint64_t           _cacheSize = 128;
         static const closure::ImageNum  _firstImageNum = closure::kFirstLaunchClosureImageNum;
@@ -135,11 +148,11 @@ private:
         // Note, the cache stores "indices + 1" into the _allImages array.
         // 0 means we haven't cached an entry yet
         uint32_t                        _cacheStorage[_cacheSize] = { 0 };
-        Array<uint32_t>                 _imageIndices = { &_cacheStorage[0], _cacheSize, _cacheSize };
+        mutable Array<uint32_t>         _imageIndices = { &_cacheStorage[0], _cacheSize, _cacheSize };
     };
 #endif
 
-    void                mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI);
+    void                mapImage(Diagnostics& diag, LoadedImage& info, bool fromOFI, bool* closureOutOfDate);
     void                applyFixupsToImage(Diagnostics& diag, LoadedImage& info);
     void                registerDOFs(const Array<DOFInfo>& dofs);
     void                setSegmentProtects(const LoadedImage& info, bool write);
@@ -147,6 +160,16 @@ private:
     bool                sandboxBlockedOpen(const char* path);
     bool                sandboxBlockedStat(const char* path);
     bool                sandboxBlocked(const char* path, const char* kind);
+    void                unmapAllImages();
+
+    // MachOAnalyzerSet support
+    void            mas_forEachImage(void (^handler)(const WrappedMachO& anImage, bool hidden, bool& stop)) const override;
+    void            mas_mainExecutable(WrappedMachO& anImage) const override;
+    void*           mas_dyldCache() const override;
+    bool            wmo_dependent(const WrappedMachO* image, uint32_t depIndex, WrappedMachO& childObj, bool& missingWeakDylib) const override;
+    const char*     wmo_path(const WrappedMachO* image) const override;
+    bool            wmo_missingSymbolResolver(const WrappedMachO* fromWmo, bool weakImport, bool lazyBind, const char* symbolName, const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const override;
+
 
     const Array<LoadedImage>&                       _existingImages;
     Array<LoadedImage>&                             _newImages;
@@ -155,13 +178,17 @@ private:
     const objc_opt::objc_selopt_t*                  _dyldCacheSelectorOpt;
     const closure::ObjCSelectorOpt*                 _closureSelectorOpt;
     const Array<closure::Image::ObjCSelectorImage>& _closureSelectorImages;
+    const RootsChecker&                             _rootsChecker;
 #if BUILDING_DYLD
     LaunchImagesCache                               _launchImagesCache;
 #endif
+    bool                                            _allowMissingLazies;
+    dyld3::Platform                                 _platform;
     LogFunc                                         _logLoads;
     LogFunc                                         _logSegments;
     LogFunc                                         _logFixups;
     LogFunc                                         _logDofs;
+    dyld3::LaunchErrorInfo*                         _launchErrorInfo;
 };
 
 

dyld3/MachOAnalyzer.h

@@ -44,7 +44,23 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
     using MachOLoaded::getDylibInstallName;
     using MachOLoaded::FoundSymbol;
     using MachOLoaded::findExportedSymbol;
+    using MachOLoaded::forEachGlobalSymbol;
+    using MachOLoaded::forEachLocalSymbol;
+    using MachOFile::canBePlacedInDyldCache;
+    using MachOFile::forEachLoadCommand;
+    using MachOFile::removeLoadCommand;
 
+    enum class Rebase {
+        unknown,
+        pointer32,
+        pointer64,
+        textPCrel32,
+        textAbsolute32,
+    };
+
+    static bool loadFromBuffer(Diagnostics& diag, const closure::FileSystem& fileSystem,
+                               const char* path, const GradedArchs& archs, Platform platform,
+                               closure::LoadedFileInfo& info);
     static closure::LoadedFileInfo load(Diagnostics& diag, const closure::FileSystem& fileSystem,
                                         const char* logicalPath, const GradedArchs& archs, Platform platform, char realerPath[MAXPATHLEN]);
     static const MachOAnalyzer*  validMainExecutable(Diagnostics& diag, const mach_header* mh, const char* path, uint64_t sliceLength,
@@ -52,26 +68,47 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
 
     typedef void (^ExportsCallback)(const char* symbolName, uint64_t imageOffset, uint64_t flags,
                                     uint64_t other, const char* importName, bool& stop);
-    bool                validMachOForArchAndPlatform(Diagnostics& diag, size_t mappedSize, const char* path, const GradedArchs& archs, Platform platform) const;
+    bool                validMachOForArchAndPlatform(Diagnostics& diag, size_t mappedSize, const char* path, const GradedArchs& archs, Platform platform, bool isOSBinary) const;
+
+    // Caches data useful for converting from raw data to VM addresses
+    struct VMAddrConverter {
+        uint64_t preferredLoadAddress                       = 0;
+        intptr_t slide                                      = 0;
+        uint16_t chainedPointerFormat                       = 0;
+        bool contentRebased                                 = false;
+#if !(BUILDING_LIBDYLD || BUILDING_DYLD)
+        enum class SharedCacheFormat : uint8_t {
+            none            = 0,
+            v2_x86_64_tbi   = 1,
+            v3              = 2
+        };
+        SharedCacheFormat sharedCacheChainedPointerFormat   = SharedCacheFormat::none;
+#endif
+
+        uint64_t convertToVMAddr(uint64_t v) const;
+    };
+
+    VMAddrConverter     makeVMAddrConverter(bool contentRebased) const;
+
     uint64_t            mappedSize() const;
     bool                hasObjC() const;
     bool                hasPlusLoadMethod(Diagnostics& diag) const;
+    bool                usesObjCGarbageCollection() const;
+    bool                isSwiftLibrary() const;
     uint64_t            preferredLoadAddress() const;
-    void                forEachLocalSymbol(Diagnostics& diag, void (^callback)(const char* symbolName, uint64_t n_value, uint8_t n_type, uint8_t n_sect, uint16_t n_desc, bool& stop)) const;
     void                forEachRPath(void (^callback)(const char* rPath, bool& stop)) const;
-
-    bool                isEncrypted() const;
+    bool                hasProgramVars(Diagnostics& diag, uint32_t& progVarsOffset) const;
     void                forEachCDHash(void (^handler)(const uint8_t cdHash[20])) const;
     bool                hasCodeSignature(uint32_t& fileOffset, uint32_t& size) const;
     bool                usesLibraryValidation() const;
     bool                isRestricted() const;
-    bool                getEntry(uint32_t& offset, bool& usesCRT) const;
+    bool                getEntry(uint64_t& offset, bool& usesCRT) const;
     bool                isSlideable() const;
-    bool                hasInitializer(Diagnostics& diag, bool contentRebased, const void* dyldCache=nullptr) const;
+    bool                hasInitializer(Diagnostics& diag, const VMAddrConverter& vmAddrConverter, const void* dyldCache=nullptr) const;
     void                forEachInitializerPointerSection(Diagnostics& diag, void (^callback)(uint32_t sectionOffset, uint32_t sectionSize, const uint8_t* content, bool& stop)) const;
-    void                forEachInitializer(Diagnostics& diag, bool contentRebased, void (^callback)(uint32_t offset), const void* dyldCache=nullptr) const;
-    bool                hasTerminators(Diagnostics& diag, bool contentRebased) const;
-    void                forEachTerminator(Diagnostics& diag, bool contentRebased, void (^callback)(uint32_t offset)) const;
+    void                forEachInitializer(Diagnostics& diag, const VMAddrConverter& vmAddrConverter, void (^callback)(uint32_t offset), const void* dyldCache=nullptr) const;
+    bool                hasTerminators(Diagnostics& diag, const VMAddrConverter& vmAddrConverter) const;
+    void                forEachTerminator(Diagnostics& diag, const VMAddrConverter& vmAddrConverter, void (^callback)(uint32_t offset)) const;
     void                forEachDOFSection(Diagnostics& diag, void (^callback)(uint32_t offset)) const;
     uint32_t            segmentCount() const;
     void                forEachExportedSymbol(Diagnostics& diag, ExportsCallback callback) const;
@@ -88,26 +125,36 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
     const void*         getBindOpcodes(uint32_t& size) const;
     const void*         getLazyBindOpcodes(uint32_t& size) const;
     const void*         getSplitSeg(uint32_t& size) const;
+    bool                hasSplitSeg() const;
+    bool                isSplitSegV1() const;
+    bool                isSplitSegV2() const;
     uint64_t            segAndOffsetToRuntimeOffset(uint8_t segIndex, uint64_t segOffset) const;
     bool                hasLazyPointers(uint32_t& runtimeOffset, uint32_t& size) const;
     void                forEachRebase(Diagnostics& diag, bool ignoreLazyPointer, void (^callback)(uint64_t runtimeOffset, bool& stop)) const;
+    void                forEachRebase(Diagnostics& diag, void (^callback)(uint64_t runtimeOffset, bool isLazyPointerRebase, bool& stop)) const;
     void                forEachTextRebase(Diagnostics& diag, void (^callback)(uint64_t runtimeOffset, bool& stop)) const;
     void                forEachBind(Diagnostics& diag, void (^callback)(uint64_t runtimeOffset, int libOrdinal, const char* symbolName,
                                                                         bool weakImport, bool lazyBind, uint64_t addend, bool& stop),
-                                                       void (^strongHandler)(const char* symbolName),
-                                                       void (^missingLazyBindHandler)()) const;
+                                    void (^strongHandler)(const char* symbolName)) const;
+    void                forEachBind(Diagnostics& diag, void (^callback)(uint64_t runtimeOffset, int libOrdinal, uint8_t type, const char* symbolName,
+                                                                        bool weakImport, bool lazyBind, uint64_t addend, bool& stop),
+                                    void (^strongHandler)(const char* symbolName)) const;
     void                forEachChainedFixupTarget(Diagnostics& diag, void (^callback)(int libOrdinal, const char* symbolName, uint64_t addend, bool weakImport, bool& stop)) const;
-    bool                canHavePrecomputedDlopenClosure(const char* path, void (^failureReason)(const char*)) const;
     void                forEachRebase(Diagnostics& diag, void (^handler)(const char* opcodeName, const LinkEditInfo& leInfo, const SegmentInfo segments[],
-                                                                             bool segIndexSet, uint32_t pointerSize, uint8_t segmentIndex, uint64_t segmentOffset, uint8_t type, bool& stop)) const;
+                                                                             bool segIndexSet, uint32_t pointerSize, uint8_t segmentIndex, uint64_t segmentOffset, Rebase kind, bool& stop)) const;
     void                forEachBind(Diagnostics& diag, void (^handler)(const char* opcodeName, const LinkEditInfo& leInfo, const SegmentInfo segments[],
                                                                        bool segIndexSet,  bool libraryOrdinalSet, uint32_t dylibCount, int libOrdinal,
                                                                        uint32_t pointerSize, uint8_t segmentIndex, uint64_t segmentOffset,
                                                                        uint8_t type, const char* symbolName, bool weakImport, bool lazyBind, uint64_t addend, bool& stop),
-                                                       void (^strongHandler)(const char* symbolName),
-                                                       void (^missingLazyBindHandler)()) const;
+                                                       void (^strongHandler)(const char* symbolName)) const;
     bool                canBePlacedInDyldCache(const char* path, void (^failureReason)(const char*)) const;
+    bool                canHavePrecomputedDlopenClosure(const char* path, void (^failureReason)(const char*)) const;
+#if BUILDING_APP_CACHE_UTIL
+    bool                canBePlacedInKernelCollection(const char* path, void (^failureReason)(const char*)) const;
+#endif
+    bool                usesClassicRelocationsInKernelCollection() const;
     uint32_t            loadCommandsFreeSpace() const;
+    bool                hasStompedLazyOpcodes() const;
  
 #if DEBUG
     void                validateDyldCacheDylib(Diagnostics& diag, const char* path) const;
@@ -115,10 +162,17 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
     void                withChainStarts(Diagnostics& diag, uint64_t startsStructOffsetHint, void (^callback)(const dyld_chained_starts_in_image*)) const;
     uint64_t            chainStartsOffset() const;
     uint16_t            chainedPointerFormat() const;
+    static uint16_t     chainedPointerFormat(const dyld_chained_fixups_header* chainHeader);
     bool                hasUnalignedPointerFixups() const;
+    const dyld_chained_fixups_header* chainedFixupsHeader() const;
+    bool                hasFirmwareChainStarts(uint16_t* pointerFormat, uint32_t* startsCount, const uint32_t** starts) const;
+    bool                isOSBinary(int fd, uint64_t sliceOffset, uint64_t sliceSize) const;  // checks if binary is codesigned to be part of the OS
+    static bool         sliceIsOSBinary(int fd, uint64_t sliceOffset, uint64_t sliceSize);
 
     const MachOAnalyzer*    remapIfZeroFill(Diagnostics& diag, const closure::FileSystem& fileSystem, closure::LoadedFileInfo& info) const;
     
+    bool                markNeverUnload(Diagnostics &diag) const;
+
     struct ObjCInfo {
         uint32_t    selRefCount;
         uint32_t    classDefCount;
@@ -140,20 +194,12 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
         dyld3::OverflowSafeArray<SectionInfo> sectionInfos = { buffer, sizeof(buffer) / sizeof(buffer[0]) };
     };
 
-    // Caches data useful for converting from raw data to VM addresses
-    struct VMAddrConverter {
-        uint64_t preferredLoadAddress   = 0;
-        intptr_t slide                  = 0;
-        uint16_t chainedPointerFormat   = 0;
-        bool contentRebased             = false;
-    };
-
     struct ObjCClassInfo {
         // These fields are all present on the objc_class_t struct
         uint64_t isaVMAddr                                  = 0;
         uint64_t superclassVMAddr                           = 0;
         //uint64_t methodCacheBuckets;
-        //uint64_t methodCacheProperties;
+        uint64_t methodCacheVMAddr                          = 0;
         uint64_t dataVMAddr                                 = 0;
 
         // This field is only present if this is a Swift object, ie, has the Swift
@@ -170,16 +216,28 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
         // These are from the class_ro_t which data points to
         enum class ReadOnlyDataField {
             name,
-            baseMethods
+            baseProtocols,
+            baseMethods,
+            baseProperties,
+            flags
         };
 
         uint64_t getReadOnlyDataField(ReadOnlyDataField field, uint32_t pointerSize) const;
         uint64_t nameVMAddr(uint32_t pointerSize) const {
             return getReadOnlyDataField(ReadOnlyDataField::name, pointerSize);
         }
+        uint64_t baseProtocolsVMAddr(uint32_t pointerSize) const {
+            return getReadOnlyDataField(ReadOnlyDataField::baseProtocols, pointerSize);
+        }
         uint64_t baseMethodsVMAddr(uint32_t pointerSize) const {
             return getReadOnlyDataField(ReadOnlyDataField::baseMethods, pointerSize);
         }
+        uint64_t basePropertiesVMAddr(uint32_t pointerSize) const {
+            return getReadOnlyDataField(ReadOnlyDataField::baseProperties, pointerSize);
+        }
+        uint64_t flags(uint32_t pointerSize) const {
+            return getReadOnlyDataField(ReadOnlyDataField::flags, pointerSize);
+        }
 
         // These are embedded in the Mach-O itself by the compiler
         enum FastDataBits {
@@ -205,6 +263,19 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
         }
     };
 
+    struct ObjCMethodList {
+        // This matches the bits in the objc runtime
+        enum : uint32_t {
+            methodListIsUniqued = 0x1,
+            methodListIsSorted  = 0x2,
+
+            // The size is bits 2 through 16 of the entsize field
+            // The low 2 bits are uniqued/sorted as above.  The upper 16-bits
+            // are reserved for other flags
+            methodListSizeMask  = 0x0000FFFC
+        };
+    };
+
     struct ObjCImageInfo {
         uint32_t version;
         uint32_t flags;
@@ -225,6 +296,11 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
         uint64_t nameLocationVMAddr;
     };
 
+    struct ObjCProperty {
+        uint64_t nameVMAddr;        // & const char *
+        uint64_t attributesVMAddr;  // & const char *
+    };
+
     struct ObjCCategory {
         uint64_t nameVMAddr;
         uint64_t clsVMAddr;
@@ -237,7 +313,7 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
     struct ObjCProtocol {
         uint64_t isaVMAddr;
         uint64_t nameVMAddr;
-        //uint64_t protocolsVMAddr;
+        uint64_t protocolsVMAddr;
         uint64_t instanceMethodsVMAddr;
         uint64_t classMethodsVMAddr;
         uint64_t optionalInstanceMethodsVMAddr;
@@ -249,10 +325,6 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
         //uint64_t extendedMethodTypesVMAddr;
         //uint64_t demangledNameVMAddr;
         //uint64_t classPropertiesVMAddr;
-
-        // Note this isn't in a protocol, but we use it in dyld to track if the protocol
-        // is large enough to avoid a reallocation in objc.
-        bool requiresObjCReallocation;
     };
 
     enum class PrintableStringResult {
@@ -266,25 +338,41 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
                                    SectionCache* sectionCache = nullptr,
                                    bool (^sectionHandler)(const SectionInfo& sectionInfo) = nullptr) const;
 
-    void forEachObjCClass(Diagnostics& diag, bool contentRebased,
+    void parseObjCClass(Diagnostics& diag, const VMAddrConverter& vmAddrConverter,
+                        uint64_t classVMAddr,
+                        void (^handler)(Diagnostics& diag,
+                                        uint64_t classSuperclassVMAddr,
+                                        uint64_t classDataVMAddr,
+                                        const ObjCClassInfo& objcClass)) const;
+
+    void forEachObjCClass(Diagnostics& diag, const VMAddrConverter& vmAddrConverter,
                           void (^handler)(Diagnostics& diag, uint64_t classVMAddr,
                                           uint64_t classSuperclassVMAddr, uint64_t classDataVMAddr,
                                           const ObjCClassInfo& objcClass, bool isMetaClass)) const;
 
-    void forEachObjCCategory(Diagnostics& diag, bool contentRebased,
+    void forEachObjCCategory(Diagnostics& diag, const VMAddrConverter& vmAddrConverter,
                              void (^handler)(Diagnostics& diag, uint64_t categoryVMAddr,
                                              const dyld3::MachOAnalyzer::ObjCCategory& objcCategory)) const;
 
-    void forEachObjCProtocol(Diagnostics& diag, bool contentRebased,
+    // lists all Protocols defined in the image
+    void forEachObjCProtocol(Diagnostics& diag, const VMAddrConverter& vmAddrConverter,
                              void (^handler)(Diagnostics& diag, uint64_t protocolVMAddr,
                                              const dyld3::MachOAnalyzer::ObjCProtocol& objCProtocol)) const;
 
     // Walk a method list starting from its vmAddr.
     // Note, classes, categories, protocols, etc, all share the same method list struture so can all use this.
-    void forEachObjCMethod(uint64_t methodListVMAddr, bool contentRebased,
-                           void (^handler)(uint64_t methodVMAddr, const ObjCMethod& method)) const;
+    void forEachObjCMethod(uint64_t methodListVMAddr, const VMAddrConverter& vmAddrConverter,
+                           void (^handler)(uint64_t methodVMAddr, const ObjCMethod& method),
+                           bool* isRelativeMethodList = nullptr) const;
 
-    void forEachObjCSelectorReference(Diagnostics& diag, bool contentRebased,
+    void forEachObjCProperty(uint64_t propertyListVMAddr, const VMAddrConverter& vmAddrConverter,
+                             void (^handler)(uint64_t propertyVMAddr, const ObjCProperty& property)) const;
+
+    // lists all Protocols on a protocol_list_t
+    void forEachObjCProtocol(uint64_t protocolListVMAddr, const VMAddrConverter& vmAddrConverter,
+                             void (^handler)(uint64_t protocolRefVMAddr, const ObjCProtocol& protocol)) const;
+
+    void forEachObjCSelectorReference(Diagnostics& diag, const VMAddrConverter& vmAddrConverter,
                                       void (^handler)(uint64_t selRefVMAddr, uint64_t selRefTargetVMAddr)) const;
 
     void forEachObjCMethodName(void (^handler)(const char* methodName)) const;
@@ -293,7 +381,7 @@ struct VIS_HIDDEN MachOAnalyzer : public MachOLoaded
 
     const ObjCImageInfo* objcImageInfo() const;
 
-    void forEachWeakDef(Diagnostics& diag, void (^handler)(const char* symbolName, uintptr_t imageOffset, bool isFromExportTrie)) const;
+    void forEachWeakDef(Diagnostics& diag, void (^handler)(const char* symbolName, uint64_t imageOffset, bool isFromExportTrie)) const;
     
 private:
 
@@ -307,7 +395,8 @@ private:
                     segSize           : 61;
 	};
 
-    enum class Malformed { linkeditOrder, linkeditAlignment, linkeditPermissions, dyldInfoAndlocalRelocs, segmentOrder, textPermissions, executableData, codeSigAlignment };
+    enum class Malformed { linkeditOrder, linkeditAlignment, linkeditPermissions, dyldInfoAndlocalRelocs, segmentOrder,
+                            textPermissions, executableData, writableData, codeSigAlignment, sectionsAddrRangeWithinSegment };
     bool                    enforceFormat(Malformed) const;
 
     const uint8_t*          getContentForVMAddr(const LayoutInfo& info, uint64_t vmAddr) const;
@@ -320,9 +409,10 @@ private:
     bool                    validBindInfo(Diagnostics& diag, const char* path) const;
     bool                    validMain(Diagnostics& diag, const char* path) const;
     bool                    validChainedFixupsInfo(Diagnostics& diag, const char* path) const;
+    bool                    validChainedFixupsInfoOldArm64e(Diagnostics& diag, const char* path) const;
 
     bool                    invalidRebaseState(Diagnostics& diag, const char* opcodeName, const char* path, const LinkEditInfo& leInfo, const SegmentInfo segments[],
-                                              bool segIndexSet, uint32_t pointerSize, uint8_t segmentIndex, uint64_t segmentOffset, uint8_t type) const;
+                                              bool segIndexSet, uint32_t pointerSize, uint8_t segmentIndex, uint64_t segmentOffset, Rebase kind) const;
     bool                    invalidBindState(Diagnostics& diag, const char* opcodeName, const char* path, const LinkEditInfo& leInfo, const SegmentInfo segments[],
                                               bool segIndexSet,  bool libraryOrdinalSet, uint32_t dylibCount, int libOrdinal, uint32_t pointerSize,
                                               uint8_t segmentIndex, uint64_t segmentOffset, uint8_t type, const char* symbolName) const;
@@ -335,19 +425,20 @@ private:
 
     void                    getAllSegmentsInfos(Diagnostics& diag, SegmentInfo segments[]) const;
     bool                    segmentHasTextRelocs(uint32_t segIndex) const;
-    uint64_t                relocBaseAddress(const SegmentInfo segmentsInfos[], uint32_t segCount) const;
+    uint64_t                localRelocBaseAddress(const SegmentInfo segmentsInfos[], uint32_t segCount) const;
+    uint64_t                externalRelocBaseAddress(const SegmentInfo segmentsInfos[], uint32_t segCount) const;
     bool                    segIndexAndOffsetForAddress(uint64_t addr, const SegmentInfo segmentsInfos[], uint32_t segCount, uint32_t& segIndex, uint64_t& segOffset) const;
     void                    parseOrgArm64eChainedFixups(Diagnostics& diag, void (^targetCount)(uint32_t totalTargets, bool& stop),
                                                                            void (^addTarget)(const LinkEditInfo& leInfo, const SegmentInfo segments[], bool libraryOrdinalSet, uint32_t dylibCount, int libOrdinal, uint8_t type, const char* symbolName, uint64_t addend, bool weakImport, bool& stop),
                                                                            void (^addChainStart)(const LinkEditInfo& leInfo, const SegmentInfo segments[], uint8_t segmentIndex, bool segIndexSet, uint64_t segmentOffset, uint16_t format, bool& stop)) const;
     bool                    contentIsRegularStub(const uint8_t* helperContent) const;
-    uint64_t                entryAddrFromThreadCmd(const thread_command* cmd) const;
     void                    recurseTrie(Diagnostics& diag, const uint8_t* const start, const uint8_t* p, const uint8_t* const end,
                                         OverflowSafeArray<char>& cummulativeString, int curStrOffset, bool& stop, MachOAnalyzer::ExportsCallback callback) const;
     void                    analyzeSegmentsLayout(uint64_t& vmSpace, bool& hasZeroFill) const;
 
 };
 
+
 } // namespace dyld3
 
 #endif /* MachOAnalyzer_h */

dyld3/MachOAnalyzerSet.cpp

@@ -0,0 +1,530 @@
+/*
+ * Copyright (c) 2019 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+#include <sys/types.h>
+#include <mach/mach.h>
+#include <assert.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <mach-o/reloc.h>
+#include <mach-o/nlist.h>
+#include <TargetConditionals.h>
+
+#include "MachOAnalyzerSet.h"
+#include "DyldSharedCache.h"
+
+#if BUILDING_DYLD
+  namespace dyld { void log(const char*, ...); }
+#endif
+
+namespace dyld3 {
+
+static bool hasHigh8(uint64_t addend)
+{
+    // distinguish negative addend from TBI
+    if ( (addend >> 56) == 0 )
+        return false;
+    return ( (addend >> 48) != 0xFFFF );
+}
+
+void MachOAnalyzerSet::WrappedMachO::forEachBind(Diagnostics& diag, FixUpHandler fixUpHandler, CachePatchHandler patchHandler) const
+{
+    const bool              is64            = _mh->is64();
+    __block int             lastLibOrdinal  = 256;
+    __block const char*     lastSymbolName  = nullptr;
+    __block uint64_t        lastAddend      = 0;
+    __block FixupTarget     target;
+    __block PointerMetaData pmd;
+    _mh->forEachBind(diag, ^(uint64_t runtimeOffset, int libOrdinal, const char* symbolName, bool weakImport, bool lazyBind, uint64_t addend, bool& stop) {
+        if ( (symbolName == lastSymbolName) && (libOrdinal == lastLibOrdinal) && (addend == lastAddend) ) {
+            // same symbol lookup as last location
+            fixUpHandler(runtimeOffset, pmd, target, stop);
+        }
+        else if ( this->findSymbolFrom(diag, libOrdinal, symbolName, weakImport, lazyBind, addend, patchHandler, target) ) {
+            pmd.high8 = 0;
+            if ( is64 && (target.addend != 0) ) {
+                if ( hasHigh8(target.addend) ) {
+                    pmd.high8  = (target.addend >> 56);
+                    target.offsetInImage &= 0x00FFFFFFFFFFFFFFULL;
+                    target.addend        &= 0x00FFFFFFFFFFFFFFULL;
+                }
+            }
+            if ( !target.skippableWeakDef ) {
+                fixUpHandler(runtimeOffset, pmd, target, stop);
+                lastSymbolName = symbolName;
+                lastLibOrdinal = libOrdinal;
+                lastAddend     = addend;
+            }
+        }
+        else {
+            // call handler with missing symbol before stopping
+            if ( target.kind == FixupTarget::Kind::bindMissingSymbol )
+                fixUpHandler(runtimeOffset, pmd, target, stop);
+            stop = true;
+        }
+    }, ^(const char* symbolName) {
+    });
+}
+
+MachOAnalyzerSet::PointerMetaData::PointerMetaData()
+{
+    this->diversity           = 0;
+    this->high8               = 0;
+    this->authenticated       = 0;
+    this->key                 = 0;
+    this->usesAddrDiversity   = 0;
+}
+
+MachOAnalyzerSet::PointerMetaData::PointerMetaData(const MachOLoaded::ChainedFixupPointerOnDisk* fixupLoc, uint16_t pointer_format)
+{
+    this->diversity           = 0;
+    this->high8               = 0;
+    this->authenticated       = 0;
+    this->key                 = 0;
+    this->usesAddrDiversity   = 0;
+    switch ( pointer_format ) {
+        case DYLD_CHAINED_PTR_ARM64E:
+        case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+        case DYLD_CHAINED_PTR_ARM64E_FIRMWARE:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
+            this->authenticated = fixupLoc->arm64e.authRebase.auth;
+            if ( this->authenticated ) {
+                this->key               = fixupLoc->arm64e.authRebase.key;
+                this->usesAddrDiversity = fixupLoc->arm64e.authRebase.addrDiv;
+                this->diversity         = fixupLoc->arm64e.authRebase.diversity;
+            }
+            else if ( fixupLoc->arm64e.bind.bind == 0 ) {
+                this->high8             = fixupLoc->arm64e.rebase.high8;
+            }
+            break;
+        case DYLD_CHAINED_PTR_64:
+        case DYLD_CHAINED_PTR_64_OFFSET:
+            if ( fixupLoc->generic64.bind.bind == 0 )
+                this->high8             = fixupLoc->generic64.rebase.high8;
+            break;
+    }
+}
+
+
+void MachOAnalyzerSet::WrappedMachO::forEachFixup(Diagnostics& diag, FixUpHandler fixup, CachePatchHandler patcher) const
+{
+    uint16_t        fmPointerFormat;
+    uint32_t        fmStartsCount;
+    const uint32_t* fmStarts;
+    const MachOAnalyzer* ma = _mh;
+    const uint64_t prefLoadAddr = ma->preferredLoadAddress();
+    if ( ma->hasChainedFixups() ) {
+        // build targets table
+        STACK_ALLOC_OVERFLOW_SAFE_ARRAY(FixupTarget, targets, 512);
+        ma->forEachChainedFixupTarget(diag, ^(int libOrdinal, const char* symbolName, uint64_t addend, bool weakImport, bool& stop) {
+            targets.default_constuct_back();
+            FixupTarget& foundTarget = targets.back();
+            if ( !this->findSymbolFrom(diag, libOrdinal, symbolName, weakImport, false, addend, patcher, foundTarget) ) {
+                // call handler with missing symbol before stopping
+                if ( foundTarget.kind == FixupTarget::Kind::bindMissingSymbol )
+                    fixup(0, PointerMetaData(), foundTarget, stop);
+                stop = true;
+            }
+        });
+        if ( diag.hasError() )
+            return;
+
+        // walk all chains
+        ma->withChainStarts(diag, ma->chainStartsOffset(), ^(const dyld_chained_starts_in_image* startsInfo) {
+            ma->forEachFixupInAllChains(diag, startsInfo, false, ^(MachOLoaded::ChainedFixupPointerOnDisk* fixupLoc,
+                                                                    const dyld_chained_starts_in_segment* segInfo, bool& fixupsStop) {
+                uint64_t fixupOffset = (uint8_t*)fixupLoc - (uint8_t*)ma;
+                uint64_t targetOffset;
+                uint32_t bindOrdinal;
+                int64_t  embeddedAddend;
+                PointerMetaData pmd(fixupLoc, segInfo->pointer_format);
+                if ( fixupLoc->isBind(segInfo->pointer_format, bindOrdinal, embeddedAddend) ) {
+                    if ( bindOrdinal < targets.count() ) {
+                        if ( embeddedAddend == 0 ) {
+                            if ( hasHigh8(targets[bindOrdinal].addend) ) {
+                                FixupTarget targetWithoutHigh8 = targets[bindOrdinal];
+                                pmd.high8 = (targetWithoutHigh8.addend >> 56);
+                                targetWithoutHigh8.offsetInImage &= 0x00FFFFFFFFFFFFFFULL;
+                                targetWithoutHigh8.addend        &= 0x00FFFFFFFFFFFFFFULL;
+                                fixup(fixupOffset, pmd, targetWithoutHigh8, fixupsStop);
+                            }
+                            else {
+                                fixup(fixupOffset, pmd, targets[bindOrdinal], fixupsStop);
+                            }
+                        }
+                        else {
+                            // pointer on disk encodes extra addend, make pseudo target for that
+                            FixupTarget targetWithAddend = targets[bindOrdinal];
+                            targetWithAddend.addend        += embeddedAddend;
+                            targetWithAddend.offsetInImage += embeddedAddend;
+                            fixup(fixupOffset, pmd, targetWithAddend, fixupsStop);
+                        }
+                    }
+                    else {
+                        diag.error("out of range bind ordinal %d (max %lu)", bindOrdinal, targets.count());
+                        fixupsStop = true;
+                    }
+                }
+                else if ( fixupLoc->isRebase(segInfo->pointer_format, prefLoadAddr, targetOffset) ) {
+                    FixupTarget rebaseTarget;
+                    rebaseTarget.kind               = FixupTarget::Kind::rebase;
+                    rebaseTarget.foundInImage       = *this;
+                    rebaseTarget.offsetInImage      = targetOffset & 0x00FFFFFFFFFFFFFFULL;
+                    rebaseTarget.isLazyBindRebase   = false; // FIXME
+                    fixup(fixupOffset, pmd, rebaseTarget, fixupsStop);
+                }
+            });
+        });
+    }
+    else if ( ma->hasFirmwareChainStarts(&fmPointerFormat, &fmStartsCount, &fmStarts) ) {
+        // This is firmware which only has rebases, the chain starts info is in a section (not LINKEDIT)
+        ma->forEachFixupInAllChains(diag, fmPointerFormat, fmStartsCount, fmStarts, ^(MachOLoaded::ChainedFixupPointerOnDisk* fixupLoc, bool& stop) {
+            uint64_t fixupOffset = (uint8_t*)fixupLoc - (uint8_t*)ma;
+            PointerMetaData pmd(fixupLoc, fmPointerFormat);
+            uint64_t targetOffset;
+            fixupLoc->isRebase(fmPointerFormat, prefLoadAddr, targetOffset);
+            FixupTarget rebaseTarget;
+            rebaseTarget.kind               = FixupTarget::Kind::rebase;
+            rebaseTarget.foundInImage       = *this;
+            rebaseTarget.offsetInImage      = targetOffset & 0x00FFFFFFFFFFFFFFULL;
+            rebaseTarget.isLazyBindRebase   = false;
+            fixup(fixupOffset, pmd, rebaseTarget, stop);
+        });
+    }
+    else {
+        // process all rebase opcodes
+        const bool is64 = ma->is64();
+        ma->forEachRebase(diag, ^(uint64_t runtimeOffset, bool isLazyPointerRebase, bool& stop) {
+            uint64_t* loc = (uint64_t*)((uint8_t*)ma + runtimeOffset);
+            uint64_t locValue = is64 ? *loc : *((uint32_t*)loc);
+            FixupTarget rebaseTarget;
+            PointerMetaData pmd;
+            if ( is64 )
+                pmd.high8  = (locValue >> 56);
+            rebaseTarget.kind               = FixupTarget::Kind::rebase;
+            rebaseTarget.foundInImage       = *this;
+            rebaseTarget.offsetInImage      = (locValue & 0x00FFFFFFFFFFFFFFULL) - prefLoadAddr;
+            rebaseTarget.isLazyBindRebase   = isLazyPointerRebase;
+            fixup(runtimeOffset, pmd, rebaseTarget, stop);
+        });
+        if ( diag.hasError() )
+            return;
+
+        // process all bind opcodes
+        this->forEachBind(diag, fixup, patcher);
+    }
+    if ( diag.hasError() )
+        return;
+
+    // main executable may define operator new/delete symbols that overrides weak-defs but have no fixups
+    if ( ma->isMainExecutable() && ma->hasWeakDefs() ) {
+        _set->wmo_findExtraSymbolFrom(this,  patcher);
+    }
+}
+
+
+bool MachOAnalyzerSet::wmo_findSymbolFrom(const WrappedMachO* fromWmo, Diagnostics& diag, int libOrdinal, const char* symbolName, bool weakImport,
+                                           bool lazyBind, uint64_t addend, CachePatchHandler patcher, FixupTarget& target) const
+{
+    target.libOrdinal = libOrdinal;
+    if ( libOrdinal == BIND_SPECIAL_DYLIB_FLAT_LOOKUP ) {
+        __block bool found = false;
+        this->mas_forEachImage(^(const WrappedMachO& anImage, bool hidden, bool& stop) {
+            // when an image is hidden (RTLD_LOCAL) it can still look up symbols in itself
+            if ( hidden && (fromWmo->_mh != anImage._mh) )
+                return;
+            if ( anImage.findSymbolIn(diag, symbolName, addend, target) ) {
+                stop = true;
+                found = true;
+            }
+        });
+        if ( found )
+            return true;
+        // see if missing symbol resolver can find something
+        if ( fromWmo->missingSymbolResolver(weakImport, lazyBind, symbolName, "flat namespace", fromWmo->path(), target) )
+            return true;
+        // fill out target info about missing symbol
+        target.kind                 = FixupTarget::Kind::bindMissingSymbol;
+        target.requestedSymbolName  = symbolName;
+        target.foundSymbolName      = nullptr;
+        target.foundInImage         = WrappedMachO();   // no image it should be in
+
+        diag.error("symbol '%s' not found, expected in flat namespace by '%s'", symbolName, fromWmo->path());
+        return false;
+    }
+    else if ( libOrdinal == BIND_SPECIAL_DYLIB_WEAK_LOOKUP ) {
+        if ( this->mas_fromImageWeakDefLookup(*fromWmo, symbolName, addend, patcher, target) ) {
+            target.weakCoalesced = true;
+            return true;
+        }
+
+        if ( !fromWmo->_mh->hasChainedFixups() ) {
+            // support old binaries where symbols have been stripped and have weak_bind to itself
+            target.skippableWeakDef = true;
+            return true;
+        }
+        // see if missing symbol resolver can find something
+        if ( fromWmo->missingSymbolResolver(weakImport, lazyBind, symbolName, "flat namespace", fromWmo->path(), target) )
+            return true;
+        // fill out target info about missing symbol
+        target.kind                 = FixupTarget::Kind::bindMissingSymbol;
+        target.requestedSymbolName  = symbolName;
+        target.foundSymbolName      = nullptr;
+        target.foundInImage         = WrappedMachO();   // no image it should be in
+
+        diag.error("symbol '%s' not found, expected to be weak-def coalesced in '%s'", symbolName, fromWmo->path());
+        return false;
+    }
+    else {
+        int depIndex = libOrdinal - 1;
+        bool missingWeakDylib = false;
+        WrappedMachO depHelper;
+        const WrappedMachO* targetImage = nullptr;
+        if ( libOrdinal == BIND_SPECIAL_DYLIB_SELF ) {
+            targetImage = fromWmo;
+        }
+        else if ( libOrdinal == BIND_SPECIAL_DYLIB_MAIN_EXECUTABLE ) {
+            this->mas_mainExecutable(depHelper);
+            targetImage = &depHelper;
+        }
+        else if ( fromWmo->dependent(depIndex, depHelper, missingWeakDylib) ) {
+            targetImage = &depHelper;
+        }
+        else {
+            diag.error("unknown library ordinal %d in %s", libOrdinal, fromWmo->path());
+            return false;
+        }
+        // use two-level namespace target image
+        if ( !missingWeakDylib && targetImage->findSymbolIn(diag, symbolName, addend, target) )
+            return true;
+
+        // see if missing symbol resolver can find something
+        const char* expectedInPath = missingWeakDylib ? "missing dylib" : targetImage->path();
+        if ( fromWmo->missingSymbolResolver(weakImport, lazyBind, symbolName, expectedInPath, fromWmo->path(), target) )
+            return true;
+
+        // fill out target info about missing symbol
+        target.kind                 = FixupTarget::Kind::bindMissingSymbol;
+        target.requestedSymbolName  = symbolName;
+        target.foundSymbolName      = nullptr;
+        target.foundInImage         = *targetImage;    // no image it is expected to be in
+
+        // symbol not found and not weak or lazy so error out
+        diag.error("symbol '%s' not found, expected in '%s', needed by '%s'", symbolName, expectedInPath, fromWmo->path());
+        return false;
+    }
+    return false;
+}
+
+// These are mangled symbols for all the variants of operator new and delete
+// which a main executable can define (non-weak) and override the
+// weak-def implementation in the OS.
+static const char* const sTreatAsWeak[] = {
+    "__Znwm", "__ZnwmRKSt9nothrow_t",
+    "__Znam", "__ZnamRKSt9nothrow_t",
+    "__ZdlPv", "__ZdlPvRKSt9nothrow_t", "__ZdlPvm",
+    "__ZdaPv", "__ZdaPvRKSt9nothrow_t", "__ZdaPvm",
+    "__ZnwmSt11align_val_t", "__ZnwmSt11align_val_tRKSt9nothrow_t",
+    "__ZnamSt11align_val_t", "__ZnamSt11align_val_tRKSt9nothrow_t",
+    "__ZdlPvSt11align_val_t", "__ZdlPvSt11align_val_tRKSt9nothrow_t", "__ZdlPvmSt11align_val_t",
+    "__ZdaPvSt11align_val_t", "__ZdaPvSt11align_val_tRKSt9nothrow_t", "__ZdaPvmSt11align_val_t"
+};
+
+void MachOAnalyzerSet::wmo_findExtraSymbolFrom(const WrappedMachO* fromWmo, CachePatchHandler patcher) const
+{
+    for (const char* weakSymbolName : sTreatAsWeak) {
+        Diagnostics exportDiag;
+        FixupTarget dummyTarget;
+        // pretend main executable does have a use of this operator new/delete and look up the impl
+        // this has the side effect of adding a cache patch if there is an impl outside the cache
+        wmo_findSymbolFrom(fromWmo, exportDiag, -3, weakSymbolName, true, false, 0, patcher, dummyTarget);
+    }
+
+}
+
+bool MachOAnalyzerSet::WrappedMachO::findSymbolIn(Diagnostics& diag, const char* symbolName, uint64_t addend, FixupTarget& target) const
+{
+    const MachOAnalyzer* ma  = _mh;
+
+    // if exports trie location not computed yet, do it now
+    ExportsTrie exportsTrie = this->getExportsTrie();
+
+    target.foundSymbolName = nullptr;
+    if ( exportsTrie.start ) {
+        if ( const uint8_t* node = this->_mh->trieWalk(diag, exportsTrie.start, exportsTrie.end, symbolName)) {
+            const uint8_t* p = node;
+            const uint64_t flags = this->_mh->read_uleb128(diag, p, exportsTrie.end);
+            if ( flags & EXPORT_SYMBOL_FLAGS_REEXPORT ) {
+                // re-export from another dylib, lookup there
+                const uint64_t libOrdinal = ma->read_uleb128(diag, p, exportsTrie.end);
+                const char* importedName = (char*)p;
+                if ( importedName[0] == '\0' )
+                    importedName = symbolName;
+                const int depIndex = (int)(libOrdinal - 1);
+                bool missingWeakDylib;
+                WrappedMachO depHelper;
+                if ( this->dependent(depIndex, depHelper, missingWeakDylib) && !missingWeakDylib ) {
+                    if ( depHelper.findSymbolIn(diag, importedName, addend, target) ) {
+                        target.requestedSymbolName = symbolName;
+                        return true;
+                    }
+                }
+                if ( !missingWeakDylib )
+                    diag.error("re-export ordinal %lld out of range for %s", libOrdinal, symbolName);
+                return false;
+            }
+            target.kind                 = FixupTarget::Kind::bindToImage;
+            target.requestedSymbolName  = symbolName;
+            target.foundSymbolName      = symbolName;
+            target.foundInImage         = *this;
+            target.isWeakDef            = false;
+            target.addend               = addend;
+            uint64_t trieValue = ma->read_uleb128(diag, p, exportsTrie.end);
+            switch ( flags & EXPORT_SYMBOL_FLAGS_KIND_MASK ) {
+                case EXPORT_SYMBOL_FLAGS_KIND_REGULAR:
+                    target.offsetInImage = trieValue + addend;
+                    if ( flags & EXPORT_SYMBOL_FLAGS_STUB_AND_RESOLVER ) {
+                        // for now, just return address of resolver helper stub
+                        // FIXME handle running resolver
+                        (void)this->_mh->read_uleb128(diag, p, exportsTrie.end);
+                    }
+                    if ( flags & EXPORT_SYMBOL_FLAGS_WEAK_DEFINITION )
+                        target.isWeakDef = true;
+                    break;
+                case EXPORT_SYMBOL_FLAGS_KIND_THREAD_LOCAL:
+                    // no type checking that client expected TLV yet
+                    target.offsetInImage = trieValue;
+                    break;
+                case EXPORT_SYMBOL_FLAGS_KIND_ABSOLUTE:
+                    target.kind          = FixupTarget::Kind::bindAbsolute;
+                    target.offsetInImage = trieValue + addend;
+                    break;
+                default:
+                    diag.error("unsupported exported symbol kind. flags=%llu at node offset=0x%0lX", flags, (long)(node-exportsTrie.start));
+                    return false;
+            }
+            return true;
+        }
+    }
+    else {
+        ma->forEachGlobalSymbol(diag, ^(const char* n_name, uint64_t n_value, uint8_t n_type, uint8_t n_sect, uint16_t n_desc, bool& stop) {
+            if ( strcmp(n_name, symbolName) == 0 ) {
+                target.kind                 = FixupTarget::Kind::bindToImage;
+                target.foundSymbolName      = symbolName;
+                target.requestedSymbolName  = symbolName;
+                target.foundInImage         = *this;
+                target.offsetInImage        = n_value - ma->preferredLoadAddress() + addend;
+                target.addend               = addend;
+                stop = true;
+            }
+        });
+        if ( target.foundSymbolName )
+            return true;
+    }
+
+    // symbol not exported from this image
+    // if this is a dylib and has re-exported dylibs, search those too
+    if ( (ma->filetype == MH_DYLIB) && ((ma->flags & MH_NO_REEXPORTED_DYLIBS) == 0) ) {
+        __block unsigned depIndex = 0;
+        ma->forEachDependentDylib(^(const char* loadPath, bool isWeak, bool isReExport, bool isUpward, uint32_t compatVersion, uint32_t curVersion, bool& stop) {
+            if ( isReExport ) {
+                bool missingWeakDylib;
+                WrappedMachO child;
+                 if ( this->dependent(depIndex, child, missingWeakDylib) && !missingWeakDylib ) {
+                    if ( child.findSymbolIn(diag, symbolName, addend, target) )
+                        stop = true;
+                }
+            }
+            ++depIndex;
+        });
+    }
+
+    return (target.foundSymbolName != nullptr);
+}
+
+
+MachOAnalyzerSet::ExportsTrie MachOAnalyzerSet::wmo_getExportsTrie(const WrappedMachO* wmo) const
+{
+    const uint8_t* start = nullptr;
+    const uint8_t* end   = nullptr;
+    uint32_t runtimeOffset;
+    uint32_t size;
+    if ( wmo->_mh->hasExportTrie(runtimeOffset, size) ) {
+        start = (uint8_t*)wmo->_mh + runtimeOffset;
+        end   = start + size;
+    }
+    return { start, end };
+}
+
+
+// scan all weak-def images in load order
+// return first non-weak defintion found
+// otherwise first weak definition found
+bool MachOAnalyzerSet::mas_fromImageWeakDefLookup(const WrappedMachO& fromWmo, const char* symbolName, uint64_t addend, CachePatchHandler patcher, FixupTarget& target) const
+{
+    // walk all images in load order, looking only at ones with weak-defs
+    const DyldSharedCache* dyldCache = (DyldSharedCache*)mas_dyldCache();
+    __block bool foundImpl = false;
+    this->mas_forEachImage(^(const WrappedMachO& anImage, bool hidden, bool& stop) {
+        if ( !anImage._mh->hasWeakDefs() )
+            return;
+        // when an image is hidden (RTLD_LOCAL) it can still look up symbols in itself
+        if ( hidden && (fromWmo._mh != anImage._mh) )
+            return;
+        FixupTarget tempTarget;
+        Diagnostics diag;
+        if ( anImage.findSymbolIn(diag, symbolName, addend, tempTarget) ) {
+            // ignore symbol re-exports, we will find the real definition later in forEachImage()
+            if ( anImage._mh != tempTarget.foundInImage._mh )
+                return;
+            if ( foundImpl && anImage._mh->inDyldCache() && (anImage._mh != target.foundInImage._mh) ) {
+                // we have already found the target, but now we see something in the dyld cache
+                // that also implements this symbol, so we need to change all caches uses of that
+                // to use the found one instead
+                uint32_t cachedDylibIndex = 0;
+                if ( dyldCache->findMachHeaderImageIndex(anImage._mh, cachedDylibIndex) ) {
+                    uintptr_t exportCacheOffset = (uint8_t*)tempTarget.foundInImage._mh + tempTarget.offsetInImage - (uint8_t*)dyldCache;
+                    patcher(cachedDylibIndex, (uint32_t)exportCacheOffset, target);
+                }
+            }
+            if ( !foundImpl ) {
+                // this is first found, so copy this to result
+                target        = tempTarget;
+                foundImpl     = true;
+            }
+            else if ( target.isWeakDef && !tempTarget.isWeakDef ) {
+                // we found a non-weak impl later on, switch to it
+                target        = tempTarget;
+            }
+        }
+    });
+    return foundImpl;
+}
+
+
+} // dyld3
+
+

dyld3/MachOAnalyzerSet.h

@@ -0,0 +1,199 @@
+/*
+ * Copyright (c) 2019 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+#ifndef MachOAnalyzerSet_h
+#define MachOAnalyzerSet_h
+
+
+#include "MachOAnalyzer.h"
+#include "Array.h"
+
+
+namespace dyld3 {
+
+//
+// MachOAnalyzerSet is an abstraction to deal with sets of mach-o files. For instance,
+// if a mach-o file A binds to a symbol in mach-o file B, the MachOAnalyzerSet lets you
+// evaulate the bind such that you know where in B, the bind pointer in A needs to point.
+//
+// The goal of MachOAnalyzerSet is to be the one place for code that handles mach-o
+// file interactions, such as two-level namespace binding, weak-def coalescing, and
+// dyld cache patching.
+//
+// Conceptually, MachOAnalyzerSet is an ordered list of mach-o files. Each file is modeled
+// as an WrappedMachO object.  This is a lightweight POD struct of three pointers that
+// can be copied around.  A WrappedMachO consists of the MachOAnalyzer* that is represents,
+// a pointer to the MachOAnalyzerSet it is in, and an abstract "other" pointer that the
+// concrete implementation of MachOAnalyzerSet defines.  All uses of mach-o files in
+// the MachOAnalyzerSet method uses WrappedMachO types.
+//
+//      // This is the key method on WrappedMachO. It is called during closure building to
+//      // compile down the fixups, as well as at runtime in dyld3 minimal closure mode
+//      // to parse LINKEDIT and rebase/bind pointers.
+//      void   forEachFixup(Diagnostics& diag, FixUpHandler, CachePatchHandler) const;
+//
+//
+// It would have been nice to have virtual methods on WrappedMachO, but C++ won't allow
+// objects with vtables to be copied.  So instead the methods on WrappedMachO simply
+// forward to virtual methods in the owning MachOAnalyzerSet object.  Therefore, there
+// are two kinds of methods on MachOAnalyzerSet: methods for a WrappedMachO start with wmo_,
+// whereas methods for the whole set start with mas_.
+//
+//      // Walk all images in the set in order.  "hidden" means the image was loaded with RTLD_LOCAL
+//      void    mas_forEachImage(void (^handler)(const WrappedMachO& wmo, bool hidden, bool& stop)) const = 0;
+//
+//      // fills in mainWmo with the WrappedMachO for the main executable in the set
+//      void    mas_mainExecutable(WrappedMachO& mainWmo) const = 0;
+//
+//      // returns a pointer to the start of the dyld cache used the mach-o files in the set
+//      void*   mas_dyldCache() const = 0;
+//
+//      // For weak-def coalescing. The file fromWmo needs to bind to symbolName.  All files with weak-defs should be searched.
+//       // As a side effect of doing this binding, it may find that the symbol is bound overrides something in the dyld cache.
+//      // In that case, the CachePatchHandler function is called with info about how to patch the dyld cache.
+//      // This function has a default implementation.  Only the dyld cache builder overrides this, because the set is all the
+//      // dylibs in the dyld cache, and coalescing should only look at files actually linked.
+//      bool    mas_fromImageWeakDefLookup(const WrappedMachO& fromWmo, const char* symbolName, uint64_t addend,
+//                                         CachePatchHandler patcher, FixupTarget& target) const;
+//
+//
+//      // For a given WrappedMachO (fromWmo), find the nth dependent dylib.  If depIndex is out of range, return false.
+//      // If child is weak-linked dylib that could not be loaded, set missingWeakDylib to true and return true.
+//      // Otherwise fill in childWmo and return true
+//      bool        wmo_dependent(const WrappedMachO* fromWmo, uint32_t depIndex, WrappedMachO& childWmo, bool& missingWeakDylib) const = 0;
+//
+//      // Returns the path to the specified WrappedMachO
+//      const char* wmo_path(const WrappedMachO* wmo) const = 0;
+//
+//      // Called if a symbol cannot be found.  If false is returned, then the symbol binding code will return an error.
+//      // If true is returned, then "target" must be set.  It may be set to "NULL" as absolute/0.
+//      bool        wmo_missingSymbolResolver(const WrappedMachO* fromWmo, bool weakImport, bool lazyBind, const char* symbolName,
+//                                            const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const = 0;
+//
+//      // Returns the exports trie for the given binary. There is a default implementation which walks the load commands.
+//      // This should only be overriden if the MachOAnalyzerSet caches the export trie location.
+//      ExportsTrie wmo_getExportsTrie(const WrappedMachO* wmo) const;
+//
+//      // This handles special symbols like C++ operator new which can exist in the main executable as non-weak but
+//      // coalesce with other weak implementations. It does not need to be overridden.
+//      void        wmo_findExtraSymbolFrom(const WrappedMachO* fromWmo, CachePatchHandler ph) const;
+//
+//      // This is core logic for two-level namespace symbol look ups.  It does not need to be overridden.
+//      bool        wmo_findSymbolFrom(const WrappedMachO* fromWmo, Diagnostics& diag, int libOrdinal, const char* symbolName,
+//                                     bool weakImport, bool lazyBind, uint64_t addend, CachePatchHandler ph, FixupTarget& target) const;
+//
+//
+
+struct VIS_HIDDEN MachOAnalyzerSet
+{
+public:
+    struct FixupTarget;
+
+    struct ExportsTrie { const uint8_t* start; const uint8_t* end; } ;
+
+    // Extra info needed when setting an actual pointer value at runtime
+    struct PointerMetaData
+    {
+                    PointerMetaData();
+                    PointerMetaData(const MachOLoaded::ChainedFixupPointerOnDisk* fixupLoc, uint16_t pointer_format);
+
+        uint32_t    diversity         : 16,
+                    high8             :  8,
+                    authenticated     :  1,
+                    key               :  2,
+                    usesAddrDiversity :  1;
+    };
+
+    typedef void (^FixUpHandler)(uint64_t fixupLocRuntimeOffset, PointerMetaData pmd, const FixupTarget& target, bool& stop);
+    typedef void (^CachePatchHandler)(uint32_t cachedDylibIndex, uint32_t exportCacheOffset, const FixupTarget& target);
+
+    struct WrappedMachO
+    {
+        const MachOAnalyzer*           _mh;
+        const MachOAnalyzerSet*        _set;
+        void*                          _other;
+
+                        WrappedMachO() : _mh(nullptr), _set((nullptr)), _other((nullptr)) { }
+                        WrappedMachO(const MachOAnalyzer* ma, const MachOAnalyzerSet* mas, void* o) : _mh(ma), _set(mas), _other(o) { }
+                        ~WrappedMachO() {} 
+
+        // Used by: dyld cache building, dyld3s fixup applying, app closure building traditional format, dyldinfo tool
+        void            forEachFixup(Diagnostics& diag, FixUpHandler, CachePatchHandler) const;
+
+        // convenience functions
+        bool            dependent(uint32_t depIndex, WrappedMachO& childObj, bool& missingWeakDylib) const { return _set->wmo_dependent(this, depIndex, childObj, missingWeakDylib); }
+        const char*     path() const { return (_set ? _set->wmo_path(this) : nullptr); }
+        ExportsTrie     getExportsTrie() const { return _set->wmo_getExportsTrie(this); }
+        bool            findSymbolFrom(Diagnostics& diag, int libOrdinal, const char* symbolName, bool weakImport, bool lazyBind, uint64_t addend, CachePatchHandler ph, FixupTarget& target) const
+                                        { return _set->wmo_findSymbolFrom(this, diag, libOrdinal, symbolName, weakImport, lazyBind, addend, ph, target); }
+        bool            missingSymbolResolver(bool weakImport, bool lazyBind, const char* symbolName, const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const
+                                        { return _set->wmo_missingSymbolResolver(this, weakImport, lazyBind, symbolName, expectedInDylibPath, clientPath, target); }
+
+        bool            findSymbolIn(Diagnostics& diag, const char* symbolName, uint64_t addend, FixupTarget& target) const;
+
+    private:
+        void            forEachBind(Diagnostics& diag, FixUpHandler, CachePatchHandler) const;
+    };
+
+
+    struct FixupTarget
+    {
+        enum class Kind { rebase, bindToImage, bindAbsolute, bindMissingSymbol };
+        WrappedMachO            foundInImage;
+        uint64_t                offsetInImage           = 0;     // includes addend
+        const char*             requestedSymbolName     = nullptr;
+        const char*             foundSymbolName         = nullptr;
+        uint64_t                addend                  = 0;     // already added into offsetInImage
+        int                     libOrdinal              = 0;
+        Kind                    kind                    = Kind::rebase;
+        bool                    isLazyBindRebase        = false; // target is stub helper in same image
+        bool                    isWeakDef               = false; // target symbol is a weak-def
+        bool                    weakCoalesced           = false; // target found searching all images
+        bool                    weakBoundSameImage      = false; // first weak-def was in same image as use
+        bool                    skippableWeakDef        = false; // old binary that stripped symbol, so def will never be found
+    };
+
+    virtual void    mas_forEachImage(void (^handler)(const WrappedMachO& wmo, bool hidden, bool& stop)) const = 0;
+    virtual bool    mas_fromImageWeakDefLookup(const WrappedMachO& fromWmo, const char* symbolName, uint64_t addend, CachePatchHandler patcher, FixupTarget& target) const;
+    virtual void    mas_mainExecutable(WrappedMachO& mainWmo) const = 0;
+    virtual void*   mas_dyldCache() const = 0;
+
+
+protected:
+    friend WrappedMachO;
+
+    virtual bool            wmo_dependent(const WrappedMachO* fromWmo, uint32_t depIndex, WrappedMachO& childWmo, bool& missingWeakDylib) const = 0;
+    virtual const char*     wmo_path(const WrappedMachO* wmo) const = 0;
+    virtual ExportsTrie     wmo_getExportsTrie(const WrappedMachO* wmo) const;
+    virtual bool            wmo_findSymbolFrom(const WrappedMachO* fromWmo, Diagnostics& diag, int libOrdinal, const char* symbolName, bool weakImport,
+                                               bool lazyBind, uint64_t addend, CachePatchHandler ph, FixupTarget& target) const;
+    virtual bool            wmo_missingSymbolResolver(const WrappedMachO* fromWmo, bool weakImport, bool lazyBind, const char* symbolName,
+                                                      const char* expectedInDylibPath, const char* clientPath, FixupTarget& target) const = 0;
+    virtual void            wmo_findExtraSymbolFrom(const WrappedMachO* fromWmo, CachePatchHandler ph) const;
+};
+
+
+
+} // namespace dyld3
+
+#endif /* MachOAnalyzerSet_h */

dyld3/MachOAppCache.cpp

@@ -0,0 +1,163 @@
+/*
+* Copyright (c) 2017 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* This file contains Original Code and/or Modifications of Original Code
+* as defined in and that are subject to the Apple Public Source License
+* Version 2.0 (the 'License'). You may not use this file except in
+* compliance with the License. Please obtain a copy of the License at
+* http://www.opensource.apple.com/apsl/ and read it before using this
+* file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+* Please see the License for the specific language governing rights and
+* limitations under the License.
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
+
+#include "MachOAppCache.h"
+
+#include <list>
+
+#include <CoreFoundation/CFArray.h>
+#include <CoreFoundation/CFPropertyList.h>
+#include <CoreFoundation/CFString.h>
+
+#ifndef LC_FILESET_ENTRY
+#define LC_FILESET_ENTRY      (0x35 | LC_REQ_DYLD) /* used with fileset_entry_command */
+struct fileset_entry_command {
+    uint32_t        cmd;        /* LC_FILESET_ENTRY */
+    uint32_t        cmdsize;    /* includes id string */
+    uint64_t        vmaddr;     /* memory address of the dylib */
+    uint64_t        fileoff;    /* file offset of the dylib */
+    union lc_str    entry_id;   /* contained entry id */
+    uint32_t        reserved;   /* entry_id is 32-bits long, so this is the reserved padding */
+};
+#endif
+
+namespace dyld3 {
+
+void MachOAppCache::forEachDylib(Diagnostics& diag, void (^callback)(const MachOAnalyzer* ma, const char* name, bool& stop)) const {
+    const intptr_t slide = getSlide();
+    forEachLoadCommand(diag, ^(const load_command *cmd, bool &stop) {
+        if (cmd->cmd == LC_FILESET_ENTRY) {
+            const fileset_entry_command* app_cache_cmd = (const fileset_entry_command*)cmd;
+            const char* name = (char*)app_cache_cmd + app_cache_cmd->entry_id.offset;
+            callback((const MachOAnalyzer*)(app_cache_cmd->vmaddr + slide), name, stop);
+            return;
+        }
+    });
+}
+
+void MachOAppCache::forEachPrelinkInfoLibrary(Diagnostics& diags,
+                                              void (^callback)(const char* bundleName, const char* relativePath,
+                                                               const std::vector<const char*>& deps)) const  {
+
+     __block std::list<std::string> nonASCIIStrings;
+     auto getString = ^(Diagnostics& diags, CFStringRef symbolNameRef) {
+         const char* symbolName = CFStringGetCStringPtr(symbolNameRef, kCFStringEncodingUTF8);
+         if ( symbolName != nullptr )
+             return symbolName;
+
+         CFIndex len = CFStringGetMaximumSizeForEncoding(CFStringGetLength(symbolNameRef), kCFStringEncodingUTF8);
+         char buffer[len + 1];
+         if ( !CFStringGetCString(symbolNameRef, buffer, len, kCFStringEncodingUTF8) ) {
+             diags.error("Could not convert string to ASCII");
+             return (const char*)nullptr;
+         }
+         buffer[len] = '\0';
+         nonASCIIStrings.push_back(buffer);
+         return nonASCIIStrings.back().c_str();
+     };
+
+    const uint8_t* prelinkInfoBuffer = nullptr;
+    uint64_t prelinkInfoBufferSize = 0;
+    prelinkInfoBuffer = (const uint8_t*)findSectionContent("__PRELINK_INFO", "__info", prelinkInfoBufferSize);
+    if ( prelinkInfoBuffer == nullptr )
+        return;
+
+    CFReadStreamRef readStreamRef = CFReadStreamCreateWithBytesNoCopy(kCFAllocatorDefault, prelinkInfoBuffer, prelinkInfoBufferSize, kCFAllocatorNull);
+    if ( !CFReadStreamOpen(readStreamRef) ) {
+        fprintf(stderr, "Could not open plist stream\n");
+        exit(1);
+    }
+    CFErrorRef errorRef = nullptr;
+    CFPropertyListRef plistRef = CFPropertyListCreateWithStream(kCFAllocatorDefault, readStreamRef, prelinkInfoBufferSize, kCFPropertyListImmutable, nullptr, &errorRef);
+    if ( errorRef != nullptr ) {
+        CFStringRef stringRef = CFErrorCopyFailureReason(errorRef);
+        fprintf(stderr, "Could not read plist because: %s\n", CFStringGetCStringPtr(stringRef, kCFStringEncodingASCII));
+        CFRelease(stringRef);
+        exit(1);
+    }
+    assert(CFGetTypeID(plistRef) == CFDictionaryGetTypeID());
+
+    // Get the "_PrelinkInfoDictionary" array
+    CFArrayRef prelinkInfoDictionaryArrayRef = (CFArrayRef)CFDictionaryGetValue((CFDictionaryRef)plistRef, CFSTR("_PrelinkInfoDictionary"));
+    assert(CFGetTypeID(prelinkInfoDictionaryArrayRef) == CFArrayGetTypeID());
+
+    for (CFIndex i = 0; i != CFArrayGetCount(prelinkInfoDictionaryArrayRef); ++i) {
+        CFDictionaryRef kextInfoDictionary = (CFDictionaryRef)CFArrayGetValueAtIndex(prelinkInfoDictionaryArrayRef, i);
+        assert(CFGetTypeID(kextInfoDictionary) == CFDictionaryGetTypeID());
+
+        CFStringRef bundleIdentifierStringRef = (CFStringRef)CFDictionaryGetValue((CFDictionaryRef)kextInfoDictionary, CFSTR("CFBundleIdentifier"));
+        assert(CFGetTypeID(bundleIdentifierStringRef) == CFStringGetTypeID());
+
+        const char* bundleID = getString(diags, bundleIdentifierStringRef);
+        if ( bundleID == nullptr )
+            return;
+
+        const char* relativePath = nullptr;
+        CFStringRef relativePathStringRef = (CFStringRef)CFDictionaryGetValue((CFDictionaryRef)kextInfoDictionary, CFSTR("_PrelinkExecutableRelativePath"));
+        if ( relativePathStringRef != nullptr ) {
+            assert(CFGetTypeID(relativePathStringRef) == CFStringGetTypeID());
+            relativePath = getString(diags, relativePathStringRef);
+            if ( relativePath == nullptr )
+                return;
+        }
+
+        std::vector<const char*> dependencies;
+
+        CFDictionaryRef bundleLibrariesDictionaryRef = (CFDictionaryRef)CFDictionaryGetValue((CFDictionaryRef)kextInfoDictionary, CFSTR("OSBundleLibraries"));
+        if (bundleLibrariesDictionaryRef != nullptr) {
+            // Add the libraries to the dependencies
+            // If we didn't have bundle libraries then a placeholder was added
+            assert(CFGetTypeID(bundleLibrariesDictionaryRef) == CFDictionaryGetTypeID());
+
+            struct ApplyContext {
+                Diagnostics*                diagnostics;
+                std::vector<const char*>*   dependencies                                    = nullptr;
+                const char* (^getString)(Diagnostics& diags, CFStringRef symbolNameRef)     = nullptr;
+            };
+
+            CFDictionaryApplierFunction callback = [](const void *key, const void *value, void *context) {
+                CFStringRef keyStringRef = (CFStringRef)key;
+                assert(CFGetTypeID(keyStringRef) == CFStringGetTypeID());
+
+                ApplyContext* applyContext = (ApplyContext*)context;
+                const char* depString = applyContext->getString(*applyContext->diagnostics, keyStringRef);
+                if ( !depString )
+                    return;
+
+                applyContext->dependencies->push_back(depString);
+            };
+
+            ApplyContext applyContext = { &diags, &dependencies, getString };
+            CFDictionaryApplyFunction(bundleLibrariesDictionaryRef, callback, &applyContext);
+
+            if ( diags.hasError() )
+                return;
+        }
+        callback(bundleID, relativePath, dependencies);
+    }
+
+    CFRelease(plistRef);
+    CFRelease(readStreamRef);
+}
+
+} // namespace dyld3

dyld3/MachOAppCache.h

@@ -0,0 +1,63 @@
+/*
+ * Copyright (c) 2017 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+#ifndef MachOAppCache_h
+#define MachOAppCache_h
+
+#include "MachOAnalyzer.h"
+
+namespace dyld3 {
+
+struct MachOAppCache : public MachOAnalyzer {
+    // Taken from kmod.h
+    enum {
+        kmodMaxName = 64
+    };
+    #pragma pack(push, 4)
+    struct KModInfo64_v1 {
+        uint64_t            next_addr;
+        int32_t             info_version;
+        uint32_t            id;
+        uint8_t             name[kmodMaxName];
+        uint8_t             version[kmodMaxName];
+        int32_t             reference_count;
+        uint64_t            reference_list_addr;
+        uint64_t            address;
+        uint64_t            size;
+        uint64_t            hdr_size;
+        uint64_t            start_addr;
+        uint64_t            stop_addr;
+    };
+    #pragma pack(pop)
+
+    void forEachDylib(Diagnostics& diag, void (^callback)(const MachOAnalyzer* ma, const char* name, bool& stop)) const;
+
+    // Walk the __PRELINK_INFO dictionary and return each bundle and its libraries
+    void forEachPrelinkInfoLibrary(Diagnostics& diags,
+                                   void (^callback)(const char* bundleName, const char* relativePath,
+                                                    const std::vector<const char*>& deps)) const;
+};
+
+} // namespace dyld3
+
+#endif /* MachOAppCache_h */

dyld3/MachOFile.cpp

@@ -25,6 +25,11 @@
 #include <assert.h>
 #include <string.h>
 #include <stdio.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/errno.h>
+#include <sys/fcntl.h>
+#include <unistd.h>
 #include <TargetConditionals.h>
 #include <mach/host_info.h>
 #include <mach/mach.h>
@@ -34,9 +39,50 @@
 #include "MachOFile.h"
 #include "SupportedArchs.h"
 
+#if BUILDING_DYLD || BUILDING_LIBDYLD
+    // define away restrict until rdar://60166935 is fixed
+    #define restrict
+    #include <subsystem.h>
+#endif
 
 namespace dyld3 {
 
+////////////////////////////  posix wrappers ////////////////////////////////////////
+
+// <rdar://problem/10111032> wrap calls to stat() with check for EAGAIN
+int stat(const char* path, struct stat* buf)
+{
+    int result;
+    do {
+#if BUILDING_DYLD || BUILDING_LIBDYLD
+        result = ::stat_with_subsystem(path, buf);
+#else
+        result = ::stat(path, buf);
+#endif
+    } while ((result == -1) && ((errno == EAGAIN) || (errno == EINTR)));
+
+    return result;
+}
+
+// <rdar://problem/13805025> dyld should retry open() if it gets an EGAIN
+int open(const char* path, int flag, int other)
+{
+    int result;
+    do {
+#if BUILDING_DYLD || BUILDING_LIBDYLD
+        if (flag & O_CREAT)
+            result = ::open(path, flag, other);
+        else
+            result = ::open_with_subsystem(path, flag);
+#else
+        result = ::open(path, flag, other);
+#endif
+    } while ((result == -1) && ((errno == EAGAIN) || (errno == EINTR)));
+
+    return result;
+}
+
+
 ////////////////////////////  FatFile ////////////////////////////////////////
 
 const FatFile* FatFile::isFatFile(const void* fileStart)
@@ -130,7 +176,8 @@ void FatFile::forEachSlice(Diagnostics& diag, uint64_t fileLen, void (^callback)
     }
 }
 
-bool FatFile::isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const GradedArchs& archs, uint64_t& sliceOffset, uint64_t& sliceLen, bool& missingSlice) const
+bool FatFile::isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const GradedArchs& archs, bool isOSBinary,
+                                 uint64_t& sliceOffset, uint64_t& sliceLen, bool& missingSlice) const
 {
     missingSlice = false;
     if ( (this->magic != OSSwapBigToHostInt32(FAT_MAGIC)) && (this->magic != OSSwapBigToHostInt32(FAT_MAGIC_64)) )
@@ -138,7 +185,7 @@ bool FatFile::isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const Grad
 
     __block int bestGrade = 0;
     forEachSlice(diag, fileLen, ^(uint32_t sliceCpuType, uint32_t sliceCpuSubType, const void* sliceStart, uint64_t sliceSize, bool& stop) {
-        if (int sliceGrade = archs.grade(sliceCpuType, sliceCpuSubType)) {
+        if (int sliceGrade = archs.grade(sliceCpuType, sliceCpuSubType, isOSBinary)) {
             if ( sliceGrade > bestGrade ) {
                 sliceOffset = (char*)sliceStart - (char*)this;
                 sliceLen    = sliceSize;
@@ -158,27 +205,48 @@ bool FatFile::isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const Grad
 
 ////////////////////////////  GradedArchs ////////////////////////////////////////
 
-const GradedArchs GradedArchs::i386          = { {{CPU_TYPE_I386,     CPU_SUBTYPE_I386_ALL,    1}} };
-const GradedArchs GradedArchs::x86_64        = { {{CPU_TYPE_X86_64,   CPU_SUBTYPE_X86_64_ALL,  1}} };
-const GradedArchs GradedArchs::x86_64h       = { {{CPU_TYPE_X86_64,   CPU_SUBTYPE_X86_64_H,    2}, {CPU_TYPE_X86_64, CPU_SUBTYPE_X86_64_ALL, 1}},     };
-const GradedArchs GradedArchs::arm64         = { {{CPU_TYPE_ARM64,    CPU_SUBTYPE_ARM64_ALL,   1}} };
+
+#define GRADE_i386        CPU_TYPE_I386,       CPU_SUBTYPE_I386_ALL,    false
+#define GRADE_x86_64      CPU_TYPE_X86_64,     CPU_SUBTYPE_X86_64_ALL,  false
+#define GRADE_x86_64h     CPU_TYPE_X86_64,     CPU_SUBTYPE_X86_64_H,    false
+#define GRADE_armv7       CPU_TYPE_ARM,        CPU_SUBTYPE_ARM64_ALL,   false
+#define GRADE_armv7s      CPU_TYPE_ARM,        CPU_SUBTYPE_ARM_V7,      false
+#define GRADE_armv7k      CPU_TYPE_ARM,        CPU_SUBTYPE_ARM_V7K,     false
+#define GRADE_arm64       CPU_TYPE_ARM64,      CPU_SUBTYPE_ARM64_ALL,   false
+#define GRADE_arm64e      CPU_TYPE_ARM64,      CPU_SUBTYPE_ARM64E,      false
+#define GRADE_arm64e_pb   CPU_TYPE_ARM64,      CPU_SUBTYPE_ARM64E,      true
+#define GRADE_arm64_32    CPU_TYPE_ARM64_32,   CPU_SUBTYPE_ARM64_32_V8, false
+
+const GradedArchs GradedArchs::i386              = { {{GRADE_i386,    1}} };
+const GradedArchs GradedArchs::x86_64            = { {{GRADE_x86_64,  1}} };
+const GradedArchs GradedArchs::x86_64h           = { {{GRADE_x86_64h, 2}, {GRADE_x86_64, 1}} };
+const GradedArchs GradedArchs::arm64             = { {{GRADE_arm64,   1}} };
 #if SUPPORT_ARCH_arm64e
-const GradedArchs GradedArchs::arm64e_compat = { {{CPU_TYPE_ARM64,    CPU_SUBTYPE_ARM64E,      2}, {CPU_TYPE_ARM64,  CPU_SUBTYPE_ARM64_ALL, 1}} };
-const GradedArchs GradedArchs::arm64e        = { {{CPU_TYPE_ARM64,    CPU_SUBTYPE_ARM64E,      1}} };
+const GradedArchs GradedArchs::arm64e_keysoff    = { {{GRADE_arm64e,    2}, {GRADE_arm64, 1}} };
+const GradedArchs GradedArchs::arm64e_keysoff_pb = { {{GRADE_arm64e_pb, 2}, {GRADE_arm64, 1}} };
+const GradedArchs GradedArchs::arm64e            = { {{GRADE_arm64e,    1}} };
+const GradedArchs GradedArchs::arm64e_pb         = { {{GRADE_arm64e_pb, 1}} };
 #endif
-const GradedArchs GradedArchs::armv7k        = { {{CPU_TYPE_ARM,      CPU_SUBTYPE_ARM_V7K,     1}} };
-const GradedArchs GradedArchs::armv7         = { {{CPU_TYPE_ARM,      CPU_SUBTYPE_ARM_V7,      1}} };
-const GradedArchs GradedArchs::armv7s        = { {{CPU_TYPE_ARM,      CPU_SUBTYPE_ARM_V7S,     2}, {CPU_TYPE_ARM,    CPU_SUBTYPE_ARM_V7, 1}} };
+const GradedArchs GradedArchs::armv7             = { {{GRADE_armv7,   1}} };
+const GradedArchs GradedArchs::armv7s            = { {{GRADE_armv7s,  2}, {GRADE_armv7, 1}} };
+const GradedArchs GradedArchs::armv7k            = { {{GRADE_armv7k,  1}} };
 #if SUPPORT_ARCH_arm64_32
-const GradedArchs GradedArchs::arm64_32      = { {{CPU_TYPE_ARM64_32, CPU_SUBTYPE_ARM64_32_V8, 1}} };
+const GradedArchs GradedArchs::arm64_32          = { {{GRADE_arm64_32, 1}} };
 #endif
 
-int GradedArchs::grade(uint32_t cputype, uint32_t cpusubtype) const
+int GradedArchs::grade(uint32_t cputype, uint32_t cpusubtype, bool isOSBinary) const
 {
     for (const CpuGrade* p = _orderedCpuTypes; p->type != 0; ++p) {
-        if ( (p->type == cputype) && (p->subtype == (cpusubtype & ~CPU_SUBTYPE_MASK)) )
+        if ( (p->type == cputype) && (p->subtype == (cpusubtype & ~CPU_SUBTYPE_MASK)) ) {
+            if ( p->osBinary ) {
+                if ( isOSBinary )
                     return p->grade;
             }
+            else {
+                return p->grade;
+            }
+        }
+    }
     return 0;
 }
 
@@ -206,13 +274,13 @@ static bool isHaswell()
 }
 #endif
 
-const GradedArchs& GradedArchs::forCurrentOS(const MachOFile* mainExecutable)
+const GradedArchs& GradedArchs::forCurrentOS(bool keysOff, bool osBinariesOnly)
 {
 #if __arm64e__
-    if ( mainExecutable->cpusubtype < CPU_SUBTYPE_ARM64E )
-        return arm64e_compat;
+    if ( osBinariesOnly )
+        return (keysOff ? arm64e_keysoff_pb : arm64e_pb);
     else
-        return arm64e;
+        return (keysOff ? arm64e_keysoff : arm64e);
 #elif __ARM64_ARCH_8_32__
     return arm64_32;
 #elif __arm64__
@@ -232,7 +300,7 @@ const GradedArchs& GradedArchs::forCurrentOS(const MachOFile* mainExecutable)
 #endif
 }
 
-const GradedArchs& GradedArchs::forName(const char* archName, bool forMainExecutable)
+const GradedArchs& GradedArchs::forName(const char* archName, bool keysOff)
 {
     if (strcmp(archName, "x86_64h") == 0 )
         return x86_64h;
@@ -240,7 +308,7 @@ const GradedArchs& GradedArchs::forName(const char* archName, bool forMainExecut
         return x86_64;
 #if SUPPORT_ARCH_arm64e
     else if (strcmp(archName, "arm64e") == 0 )
-        return forMainExecutable ? arm64e_compat : arm64e;
+        return keysOff ? arm64e_keysoff : arm64e;
 #endif
     else if (strcmp(archName, "arm64") == 0 )
         return arm64;
@@ -260,6 +328,7 @@ const GradedArchs& GradedArchs::forName(const char* archName, bool forMainExecut
 }
 
 
+
 ////////////////////////////  MachOFile ////////////////////////////////////////
 
 
@@ -285,7 +354,7 @@ const MachOFile::PlatformInfo MachOFile::_s_platformInfos[] = {
     { "tvOS",        Platform::tvOS,              LC_VERSION_MIN_TVOS     },
     { "watchOS",     Platform::watchOS,           LC_VERSION_MIN_WATCHOS  },
     { "bridgeOS",    Platform::bridgeOS,          LC_BUILD_VERSION        },
-    { "UIKitForMac", Platform::iOSMac,            LC_BUILD_VERSION        },
+    { "MacCatalyst", Platform::iOSMac,            LC_BUILD_VERSION        },
     { "iOS-sim",     Platform::iOS_simulator,     LC_BUILD_VERSION        },
     { "tvOS-sim",    Platform::tvOS_simulator,    LC_BUILD_VERSION        },
     { "watchOS-sim", Platform::watchOS_simulator, LC_BUILD_VERSION        },
@@ -303,6 +372,10 @@ size_t MachOFile::machHeaderSize() const
     return is64() ? sizeof(mach_header_64) : sizeof(mach_header);
 }
 
+uint32_t MachOFile::maskedCpuSubtype() const
+{
+    return (this->cpusubtype & ~CPU_SUBTYPE_MASK);
+}
 
 uint32_t MachOFile::pointerSize() const
 {
@@ -404,7 +477,7 @@ void MachOFile::packedVersionToString(uint32_t packedVersion, char versionString
     *s++ = '\0';
 }
 
-bool MachOFile::supportsPlatform(Platform reqPlatform) const
+bool MachOFile::builtForPlatform(Platform reqPlatform, bool onlyOnePlatform) const
 {
     __block bool foundRequestedPlatform = false;
     __block bool foundOtherPlatform     = false;
@@ -414,22 +487,73 @@ bool MachOFile::supportsPlatform(Platform reqPlatform) const
         else
             foundOtherPlatform = true;
     });
+    // if checking that this binary is built for exactly one platform, fail if more
+    if ( foundOtherPlatform && onlyOnePlatform )
+        return false;
     if ( foundRequestedPlatform )
         return true;
 
-    // we did find some platform info, but not requested, so return false
-    if ( foundOtherPlatform )
-        return false;
-
     // binary has no explict load command to mark platform
     // could be an old macOS binary, look at arch
-    if  ( reqPlatform == Platform::macOS ) {
+    if  ( !foundOtherPlatform && (reqPlatform == Platform::macOS) ) {
         if ( this->cputype == CPU_TYPE_X86_64 )
             return true;
         if ( this->cputype == CPU_TYPE_I386 )
             return true;
     }
 
+#if BUILDING_DYLDINFO
+    // Allow offline tools to analyze binaries dyld doesn't load, ie, those with platforms
+    if ( !foundOtherPlatform && (reqPlatform == Platform::unknown) )
+        return true;
+#endif
+
+    return false;
+}
+
+bool MachOFile::loadableIntoProcess(Platform processPlatform, const char* path) const
+{
+    if ( this->builtForPlatform(processPlatform) )
+        return true;
+
+    // Some host macOS dylibs can be loaded into simulator processes
+    if ( MachOFile::isSimulatorPlatform(processPlatform) && this->builtForPlatform(Platform::macOS)) {
+        static const char* macOSHost[] = {
+            "/usr/lib/system/libsystem_kernel.dylib",
+            "/usr/lib/system/libsystem_platform.dylib",
+            "/usr/lib/system/libsystem_pthread.dylib",
+            "/usr/lib/system/libsystem_platform_debug.dylib",
+            "/usr/lib/system/libsystem_pthread_debug.dylib",
+            "/usr/lib/system/host/liblaunch_sim.dylib",
+        };
+        for (const char* libPath : macOSHost) {
+            if (strcmp(libPath, path) == 0)
+                return true;
+        }
+    }
+
+    // If this is being called on main executable where we expect a macOS program, Catalyst programs are also runnable
+    if ( (this->filetype == MH_EXECUTE) && (processPlatform == Platform::macOS) && this->builtForPlatform(Platform::iOSMac, true) )
+        return true;
+#if (TARGET_OS_OSX && TARGET_CPU_ARM64)
+    if ( (this->filetype == MH_EXECUTE) && (processPlatform == Platform::macOS) && this->builtForPlatform(Platform::iOS, true) )
+        return true;
+#endif
+
+    bool iOSonMac = (processPlatform == Platform::iOSMac);
+#if (TARGET_OS_OSX && TARGET_CPU_ARM64)
+    // allow iOS binaries in iOSApp
+    if ( processPlatform == Platform::iOS ) {
+        // can load Catalyst binaries into iOS process
+        if ( this->builtForPlatform(Platform::iOSMac) )
+            return true;
+        iOSonMac = true;
+    }
+#endif
+    // macOS dylibs can be loaded into iOSMac processes
+    if ( (iOSonMac) && this->builtForPlatform(Platform::macOS, true) )
+        return true;
+
     return false;
 }
 
@@ -469,8 +593,10 @@ Platform MachOFile::currentPlatform()
     return Platform::tvOS;
 #elif TARGET_OS_IOS
     return Platform::iOS;
-#elif TARGET_OS_MAC
+#elif TARGET_OS_OSX
     return Platform::macOS;
+#elif TARGET_OS_DRIVERKIT
+    return Platform::driverKit;
 #else
     #error unknown platform
 #endif
@@ -542,6 +668,16 @@ bool MachOFile::isStaticExecutable() const
     return !hasLoadCommand(LC_LOAD_DYLINKER);
 }
 
+bool MachOFile::isKextBundle() const
+{
+    return (this->filetype == MH_KEXT_BUNDLE);
+}
+
+bool MachOFile::isFileSet() const
+{
+    return (this->filetype == MH_FILESET);
+}
+
 bool MachOFile::isPIE() const
 {
     return (this->flags & MH_PIE);
@@ -659,6 +795,52 @@ void MachOFile::forEachLoadCommand(Diagnostics& diag, void (^callback)(const loa
     }
 }
 
+void MachOFile::removeLoadCommand(Diagnostics& diag, void (^callback)(const load_command* cmd, bool& remove, bool& stop))
+{
+    bool stop = false;
+    const load_command* startCmds = nullptr;
+    if ( this->magic == MH_MAGIC_64 )
+        startCmds = (load_command*)((char *)this + sizeof(mach_header_64));
+    else if ( this->magic == MH_MAGIC )
+        startCmds = (load_command*)((char *)this + sizeof(mach_header));
+    else if ( hasMachOBigEndianMagic() )
+        return;  // can't process big endian mach-o
+    else {
+        const uint32_t* h = (uint32_t*)this;
+        diag.error("file does not start with MH_MAGIC[_64]: 0x%08X 0x%08X", h[0], h [1]);
+        return;  // not a mach-o file
+    }
+    const load_command* const cmdsEnd = (load_command*)((char*)startCmds + this->sizeofcmds);
+    auto cmd = (load_command*)startCmds;
+    const uint32_t origNcmds = this->ncmds;
+    unsigned bytesRemaining = this->sizeofcmds;
+    for (uint32_t i = 0; i < origNcmds; ++i) {
+        bool remove = false;
+        auto nextCmd = (load_command*)((char *)cmd + cmd->cmdsize);
+        if ( cmd->cmdsize < 8 ) {
+            diag.error("malformed load command #%d of %d at %p with mh=%p, size (0x%X) too small", i, this->ncmds, cmd, this, cmd->cmdsize);
+            return;
+        }
+        if ( (nextCmd > cmdsEnd) || (nextCmd < startCmds) ) {
+            diag.error("malformed load command #%d of %d at %p with mh=%p, size (0x%X) is too large, load commands end at %p", i, this->ncmds, cmd, this, cmd->cmdsize, cmdsEnd);
+            return;
+        }
+        callback(cmd, remove, stop);
+        if ( remove ) {
+            this->sizeofcmds -= cmd->cmdsize;
+            ::memmove((void*)cmd, (void*)nextCmd, bytesRemaining);
+            this->ncmds--;
+        } else {
+            bytesRemaining -= cmd->cmdsize;
+            cmd = nextCmd;
+        }
+        if ( stop )
+            break;
+    }
+    if ( cmd )
+     ::bzero(cmd, bytesRemaining);
+}
+
 const char* MachOFile::installName() const
 {
     const char*  name;
@@ -782,6 +964,48 @@ bool MachOFile::enforceCompatVersion() const
     return result;
 }
 
+const thread_command* MachOFile::unixThreadLoadCommand() const {
+    Diagnostics diag;
+    __block const thread_command* command = nullptr;
+    forEachLoadCommand(diag, ^(const load_command* cmd, bool& stop) {
+        if ( cmd->cmd == LC_UNIXTHREAD ) {
+            command = (const thread_command*)cmd;
+            stop = true;
+        }
+    });
+    return command;
+}
+
+
+uint32_t MachOFile::entryAddrRegisterIndexForThreadCmd() const
+{
+    switch ( this->cputype ) {
+        case CPU_TYPE_I386:
+            return 10; // i386_thread_state_t.eip
+        case CPU_TYPE_X86_64:
+            return 16; // x86_thread_state64_t.rip
+        case CPU_TYPE_ARM:
+            return 15; // arm_thread_state_t.pc
+        case CPU_TYPE_ARM64:
+            return 32; // arm_thread_state64_t.__pc
+    }
+    return ~0U;
+}
+
+
+uint64_t MachOFile::entryAddrFromThreadCmd(const thread_command* cmd) const
+{
+    assert(cmd->cmd == LC_UNIXTHREAD);
+    const uint32_t* regs32 = (uint32_t*)(((char*)cmd) + 16);
+    const uint64_t* regs64 = (uint64_t*)(((char*)cmd) + 16);
+
+    uint32_t index = entryAddrRegisterIndexForThreadCmd();
+    if (index == ~0U)
+        return 0;
+
+    return is64() ? regs64[index] : regs32[index];
+}
+
 
 void MachOFile::forEachSegment(void (^callback)(const SegmentInfo& info, bool& stop)) const
 {
@@ -981,8 +1205,94 @@ bool MachOFile::isSharedCacheEligiblePath(const char* dylibName) {
             || (strncmp(dylibName, "/Library/Apple/System/Library/", 30) == 0) );
 }
 
+static bool startsWith(const char* buffer, const char* valueToFind) {
+    return strncmp(buffer, valueToFind, strlen(valueToFind)) == 0;
+}
+
+static bool platformExcludesSharedCache_macOS(const char* installName) {
+    // Note: This function basically matches dontCache() from update dyld shared cache
+
+    if ( startsWith(installName, "/usr/lib/system/introspection/") )
+        return true;
+    if ( startsWith(installName, "/System/Library/QuickTime/") )
+        return true;
+    if ( startsWith(installName, "/System/Library/Tcl/") )
+        return true;
+    if ( startsWith(installName, "/System/Library/Perl/") )
+        return true;
+    if ( startsWith(installName, "/System/Library/MonitorPanels/") )
+        return true;
+    if ( startsWith(installName, "/System/Library/Accessibility/") )
+        return true;
+    if ( startsWith(installName, "/usr/local/") )
+        return true;
+    if ( startsWith(installName, "/usr/lib/pam/") )
+        return true;
+    // We no longer support ROSP, so skip all paths which start with the special prefix
+    if ( startsWith(installName, "/System/Library/Templates/Data/") )
+        return true;
+
+    // anything inside a .app bundle is specific to app, so should not be in shared cache
+    if ( strstr(installName, ".app/") != NULL )
+        return true;
+
+    return false;
+}
+
+static bool platformExcludesSharedCache_iOS(const char* installName) {
+    if ( strcmp(installName, "/System/Library/Caches/com.apple.xpc/sdk.dylib") == 0 )
+        return true;
+    if ( strcmp(installName, "/System/Library/Caches/com.apple.xpcd/xpcd_cache.dylib") == 0 )
+        return true;
+    return false;
+}
+
+static bool platformExcludesSharedCache_tvOS(const char* installName) {
+    return platformExcludesSharedCache_iOS(installName);
+}
+
+static bool platformExcludesSharedCache_watchOS(const char* installName) {
+    return platformExcludesSharedCache_iOS(installName);
+}
+
+static bool platformExcludesSharedCache_bridgeOS(const char* installName) {
+    return platformExcludesSharedCache_iOS(installName);
+}
+
+// Returns true if the current platform requires that this install name be excluded from the shared cache
+// Note that this overrides any exclusion from anywhere else.
+static bool platformExcludesSharedCache(Platform platform, const char* installName) {
+    switch (platform) {
+        case dyld3::Platform::unknown:
+            return false;
+        case dyld3::Platform::macOS:
+            return platformExcludesSharedCache_macOS(installName);
+        case dyld3::Platform::iOS:
+            return platformExcludesSharedCache_iOS(installName);
+        case dyld3::Platform::tvOS:
+            return platformExcludesSharedCache_tvOS(installName);
+        case dyld3::Platform::watchOS:
+            return platformExcludesSharedCache_watchOS(installName);
+        case dyld3::Platform::bridgeOS:
+            return platformExcludesSharedCache_bridgeOS(installName);
+        case dyld3::Platform::iOSMac:
+            return platformExcludesSharedCache_macOS(installName);
+        case dyld3::Platform::iOS_simulator:
+            return false;
+        case dyld3::Platform::tvOS_simulator:
+            return false;
+        case dyld3::Platform::watchOS_simulator:
+            return false;
+        case dyld3::Platform::driverKit:
+            return false;
+    }
+}
+
+
+
 bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(const char*)) const
 {
+
     if ( !isSharedCacheEligiblePath(path) ) {
         // Dont spam the user with an error about paths when we know these are never eligible.
         return false;
@@ -1006,6 +1316,28 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
         failureReason("install path does not match install name");
         return false;
     }
+    else if ( strstr(dylibName, "//") != 0 ) {
+        failureReason("install name should not include //");
+        return false;
+    }
+    else if ( strstr(dylibName, "./") != 0 ) {
+        failureReason("install name should not include ./");
+        return false;
+    }
+
+    __block bool platformExcludedFile = false;
+    forEachSupportedPlatform(^(Platform platform, uint32_t minOS, uint32_t sdk) {
+        if ( platformExcludedFile )
+            return;
+        if ( platformExcludesSharedCache(platform, dylibName) ) {
+            platformExcludedFile = true;
+            return;
+        }
+    });
+    if ( platformExcludedFile ) {
+        failureReason("install name is not shared cache eligible on platform");
+        return false;
+    }
 
     bool retval = true;
 
@@ -1025,6 +1357,7 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
     __block bool hasExtraInfo = false;
     __block bool hasDyldInfo = false;
     __block bool hasExportTrie = false;
+    __block bool hasLazyLoad = false;
     Diagnostics diag;
     forEachLoadCommand(diag, ^(const load_command* cmd, bool& stop) {
         if ( cmd->cmd == LC_SEGMENT_SPLIT_INFO )
@@ -1033,6 +1366,8 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
             hasDyldInfo = true;
         if ( cmd->cmd == LC_DYLD_EXPORTS_TRIE )
             hasExportTrie = true;
+        if ( cmd->cmd == LC_LAZY_LOAD_DYLIB )
+            hasLazyLoad = true;
     });
     if ( !hasExtraInfo ) {
         retval = false;
@@ -1042,6 +1377,10 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
         retval = false;
         failureReason("Old binary, missing dyld info or export trie");
     }
+    if ( hasLazyLoad ) {
+        retval = false;
+        failureReason("Has lazy load");
+    }
 
     // dylib can only depend on other dylibs in the shared cache
     __block bool allDepPathsAreGood = true;
@@ -1057,18 +1396,13 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
     }
 
     // dylibs with interposing info cannot be in cache
-    __block bool hasInterposing = false;
-    forEachSection(^(const SectionInfo& info, bool malformedSectionRange, bool &stop) {
-        if ( ((info.sectFlags & SECTION_TYPE) == S_INTERPOSING) || ((strcmp(info.sectName, "__interpose") == 0) && (strcmp(info.segInfo.segName, "__DATA") == 0)) )
-            hasInterposing = true;
-    });
-    if ( hasInterposing ) {
+    if ( hasInterposingTuples() ) {
         retval = false;
         failureReason("Has interposing tuples");
     }
 
-    // Temporarily kick out swift binaries on watchOS simulators as they have missing split seg
-    if ( supportsPlatform(Platform::watchOS_simulator) && isArch("i386") ) {
+    // Temporarily kick out swift binaries out of dyld cache on watchOS simulators as they have missing split seg
+    if ( (this->cputype == CPU_TYPE_I386) && builtForPlatform(Platform::watchOS_simulator) ) {
         if ( strncmp(dylibName, "/usr/lib/swift/", 15) == 0 ) {
             retval = false;
             failureReason("i386 swift binary");
@@ -1078,6 +1412,243 @@ bool MachOFile::canBePlacedInDyldCache(const char* path, void (^failureReason)(c
     return retval;
 }
 
+#if BUILDING_APP_CACHE_UTIL
+bool MachOFile::canBePlacedInKernelCollection(const char* path, void (^failureReason)(const char*)) const
+{
+    // only dylibs and the kernel itself can go in cache
+    if ( this->filetype == MH_EXECUTE ) {
+        // xnu
+    } else if ( this->isKextBundle() ) {
+        // kext's
+    } else {
+        failureReason("Not MH_KEXT_BUNDLE");
+        return false;
+    }
+
+    if ( this->filetype == MH_EXECUTE ) {
+        // xnu
+
+        // two-level namespace binaries cannot go in cache
+        if ( (this->flags & MH_TWOLEVEL) != 0 ) {
+            failureReason("Built with two level namespaces");
+            return false;
+        }
+
+        // xnu kernel cannot have a page zero
+        __block bool foundPageZero = false;
+        forEachSegment(^(const SegmentInfo &segmentInfo, bool &stop) {
+            if ( strcmp(segmentInfo.segName, "__PAGEZERO") == 0 ) {
+                foundPageZero = true;
+                stop = true;
+            }
+        });
+        if (foundPageZero) {
+            failureReason("Has __PAGEZERO");
+            return false;
+        }
+
+        // xnu must have an LC_UNIXTHREAD to point to the entry point
+        __block bool foundMainLC = false;
+        __block bool foundUnixThreadLC = false;
+        Diagnostics diag;
+        forEachLoadCommand(diag, ^(const load_command* cmd, bool& stop) {
+            if ( cmd->cmd == LC_MAIN ) {
+                foundMainLC = true;
+                stop = true;
+            }
+            else if ( cmd->cmd == LC_UNIXTHREAD ) {
+                foundUnixThreadLC = true;
+            }
+        });
+        if (foundMainLC) {
+            failureReason("Found LC_MAIN");
+            return false;
+        }
+        if (!foundUnixThreadLC) {
+            failureReason("Expected LC_UNIXTHREAD");
+            return false;
+        }
+
+        if (diag.hasError()) {
+            failureReason("Error parsing load commands");
+            return false;
+        }
+
+        // The kernel should be a static executable, not a dynamic one
+        if ( !isStaticExecutable() ) {
+            failureReason("Expected static executable");
+            return false;
+        }
+
+        // The kernel must be built with -pie
+        if ( !isPIE() ) {
+            failureReason("Expected pie");
+            return false;
+        }
+    }
+
+    if ( isArch("arm64e") && isKextBundle() && !hasChainedFixups() ) {
+        failureReason("Missing fixup information");
+        return false;
+    }
+
+    // dylibs with interposing info cannot be in cache
+    __block bool hasInterposing = false;
+    forEachSection(^(const SectionInfo& info, bool malformedSectionRange, bool &stop) {
+        if ( ((info.sectFlags & SECTION_TYPE) == S_INTERPOSING) || ((strcmp(info.sectName, "__interpose") == 0) && (strcmp(info.segInfo.segName, "__DATA") == 0)) )
+            hasInterposing = true;
+    });
+    if ( hasInterposing ) {
+        failureReason("Has interposing tuples");
+        return false;
+    }
+
+    // Only x86_64 is allowed to have RWX segments
+    if ( !isArch("x86_64") && !isArch("x86_64h") ) {
+        __block bool foundBadSegment = false;
+        forEachSegment(^(const SegmentInfo &info, bool &stop) {
+            if ( (info.protections & (VM_PROT_WRITE | VM_PROT_EXECUTE)) == (VM_PROT_WRITE | VM_PROT_EXECUTE) ) {
+                failureReason("Segments are not allowed to be both writable and executable");
+                foundBadSegment = true;
+                stop = true;
+            }
+        });
+        if ( foundBadSegment )
+            return false;
+    }
+
+    return true;
+}
+#endif
+
+static bool platformExcludesPrebuiltClosure_macOS(const char* path) {
+    // We no longer support ROSP, so skip all paths which start with the special prefix
+    if ( startsWith(path, "/System/Library/Templates/Data/") )
+        return true;
+
+    // anything inside a .app bundle is specific to app, so should not get a prebuilt closure
+    if ( strstr(path, ".app/") != NULL )
+        return true;
+
+    return false;
+}
+
+static bool platformExcludesPrebuiltClosure_iOS(const char* path) {
+    if ( strcmp(path, "/System/Library/Caches/com.apple.xpc/sdk.dylib") == 0 )
+        return true;
+    if ( strcmp(path, "/System/Library/Caches/com.apple.xpcd/xpcd_cache.dylib") == 0 )
+        return true;
+    return false;
+}
+
+static bool platformExcludesPrebuiltClosure_tvOS(const char* path) {
+    return platformExcludesPrebuiltClosure_iOS(path);
+}
+
+static bool platformExcludesPrebuiltClosure_watchOS(const char* path) {
+    return platformExcludesPrebuiltClosure_iOS(path);
+}
+
+static bool platformExcludesPrebuiltClosure_bridgeOS(const char* path) {
+    return platformExcludesPrebuiltClosure_iOS(path);
+}
+
+// Returns true if the current platform requires that this install name be excluded from the shared cache
+// Note that this overrides any exclusion from anywhere else.
+static bool platformExcludesPrebuiltClosure(Platform platform, const char* path) {
+    switch (platform) {
+        case dyld3::Platform::unknown:
+            return false;
+        case dyld3::Platform::macOS:
+            return platformExcludesPrebuiltClosure_macOS(path);
+        case dyld3::Platform::iOS:
+            return platformExcludesPrebuiltClosure_iOS(path);
+        case dyld3::Platform::tvOS:
+            return platformExcludesPrebuiltClosure_tvOS(path);
+        case dyld3::Platform::watchOS:
+            return platformExcludesPrebuiltClosure_watchOS(path);
+        case dyld3::Platform::bridgeOS:
+            return platformExcludesPrebuiltClosure_bridgeOS(path);
+        case dyld3::Platform::iOSMac:
+            return platformExcludesPrebuiltClosure_macOS(path);
+        case dyld3::Platform::iOS_simulator:
+            return false;
+        case dyld3::Platform::tvOS_simulator:
+            return false;
+        case dyld3::Platform::watchOS_simulator:
+            return false;
+        case dyld3::Platform::driverKit:
+            return false;
+    }
+}
+
+bool MachOFile::canHavePrecomputedDlopenClosure(const char* path, void (^failureReason)(const char*)) const
+{
+    __block bool retval = true;
+
+    // only dylibs can go in cache
+    if ( (this->filetype != MH_DYLIB) && (this->filetype != MH_BUNDLE) ) {
+        retval = false;
+        failureReason("not MH_DYLIB or MH_BUNDLE");
+    }
+
+    // flat namespace files cannot go in cache
+    if ( (this->flags & MH_TWOLEVEL) == 0 ) {
+        retval = false;
+        failureReason("not built with two level namespaces");
+    }
+
+    // can only depend on other dylibs with absolute paths
+    __block bool allDepPathsAreGood = true;
+    forEachDependentDylib(^(const char* loadPath, bool isWeak, bool isReExport, bool isUpward, uint32_t compatVersion, uint32_t curVersion, bool& stop) {
+        if ( loadPath[0] != '/' ) {
+            allDepPathsAreGood = false;
+            stop = true;
+        }
+    });
+    if ( !allDepPathsAreGood ) {
+        retval = false;
+        failureReason("depends on dylibs that are not absolute paths");
+    }
+
+    __block bool platformExcludedFile = false;
+    forEachSupportedPlatform(^(Platform platform, uint32_t minOS, uint32_t sdk) {
+        if ( platformExcludedFile )
+            return;
+        if ( platformExcludesPrebuiltClosure(platform, path) ) {
+            platformExcludedFile = true;
+            return;
+        }
+    });
+    if ( platformExcludedFile ) {
+        failureReason("file cannot get a prebuilt closure on this platform");
+        return false;
+    }
+
+    // dylibs with interposing info cannot have dlopen closure pre-computed
+    if ( hasInterposingTuples() ) {
+        retval = false;
+        failureReason("has interposing tuples");
+    }
+
+    // special system dylib overrides cannot have closure pre-computed
+    if ( strncmp(path, "/usr/lib/system/introspection/", 30) == 0 ) {
+        retval = false;
+        failureReason("override of OS dylib");
+    }
+
+    return retval;
+}
+
+bool MachOFile::hasInterposingTuples() const
+{
+    __block bool hasInterposing = false;
+    forEachSection(^(const SectionInfo& info, bool malformedSectionRange, bool &stop) {
+        if ( ((info.sectFlags & SECTION_TYPE) == S_INTERPOSING) || ((strcmp(info.sectName, "__interpose") == 0) && (strcmp(info.segInfo.segName, "__DATA") == 0)) )
+            hasInterposing = true;
+    });
+    return hasInterposing;
+}
 
 bool MachOFile::isFairPlayEncrypted(uint32_t& textOffset, uint32_t& size) const
 {
@@ -1145,12 +1716,19 @@ bool MachOFile::hasChainedFixups() const
 {
 #if SUPPORT_ARCH_arm64e
     // arm64e always uses chained fixups
-    if ( (this->cputype == CPU_TYPE_ARM64) && (this->cpusubtype == CPU_SUBTYPE_ARM64E) )
-        return true;
+    if ( (this->cputype == CPU_TYPE_ARM64) && (this->maskedCpuSubtype() == CPU_SUBTYPE_ARM64E) ) {
+        // Not all binaries have fixups at all so check for the load commands
+        return hasLoadCommand(LC_DYLD_INFO_ONLY) || hasLoadCommand(LC_DYLD_CHAINED_FIXUPS);
+    }
 #endif
     return hasLoadCommand(LC_DYLD_CHAINED_FIXUPS);
 }
 
+bool MachOFile::hasChainedFixupsLoadCommand() const
+{
+    return hasLoadCommand(LC_DYLD_CHAINED_FIXUPS);
+}
+
 uint64_t MachOFile::read_uleb128(Diagnostics& diag, const uint8_t*& p, const uint8_t* end)
 {
     uint64_t result = 0;
@@ -1191,7 +1769,7 @@ int64_t MachOFile::read_sleb128(Diagnostics& diag, const uint8_t*& p, const uint
         bit += 7;
     } while (byte & 0x80);
     // sign extend negative numbers
-    if ( (byte & 0x40) != 0 )
+    if ( ((byte & 0x40) != 0) && (bit < 64) )
         result |= (~0ULL) << bit;
     return result;
 }

dyld3/MachOFile.h

@@ -74,8 +74,15 @@
   #define S_INIT_FUNC_OFFSETS       0x16
 #endif
 
+#ifndef MH_FILESET
+    #define MH_FILESET              0xc     /* set of mach-o's */
+#endif
+
 namespace dyld3 {
 
+// replacements for posix that handle EINTR
+int	stat(const char* path, struct stat* buf) VIS_HIDDEN;
+int	open(const char* path, int flag, int other) VIS_HIDDEN;
 
 
 /// Returns true if (addLHS + addRHS) > b, or if the add overflowed
@@ -99,7 +106,7 @@ enum class Platform {
     tvOS                = 3,    // PLATFORM_TVOS
     watchOS             = 4,    // PLATFORM_WATCHOS
     bridgeOS            = 5,    // PLATFORM_BRIDGEOS
-    iOSMac              = 6,    // PLATFORM_IOSMAC
+    iOSMac              = 6,    // PLATFORM_MACCATALYST
     iOS_simulator       = 7,    // PLATFORM_IOSSIMULATOR
     tvOS_simulator      = 8,    // PLATFORM_TVOSSIMULATOR
     watchOS_simulator   = 9,    // PLATFORM_WATCHOSSIMULATOR
@@ -115,10 +122,11 @@ public:
     GradedArchs()                   = delete;
     GradedArchs(const GradedArchs&) = delete;
 
-    static const GradedArchs&  forCurrentOS(const MachOFile* mainExecutable);
-    static const GradedArchs&  forName(const char* archName, bool forMainExecutable = false);
+    static const GradedArchs&  forCurrentOS(bool keysOff, bool platformBinariesOnly);
+    static const GradedArchs&  forName(const char* archName, bool keysOff = false);
 
-    int                     grade(uint32_t cputype, uint32_t cpusubtype) const;
+
+    int                     grade(uint32_t cputype, uint32_t cpusubtype, bool platformBinariesOnly) const;
     const char*             name() const;
 
     // pre-built lists for existing hardware
@@ -128,7 +136,9 @@ public:
     static const GradedArchs arm64;           // A11 or earlier iPhone or iPad
 #if SUPPORT_ARCH_arm64e
     static const GradedArchs arm64e;            // A12 or later iPhone or iPad
-    static const GradedArchs arm64e_compat;   // A12 running arm64 main executable
+    static const GradedArchs arm64e_keysoff;    // A12 running with signing keys disabled
+    static const GradedArchs arm64e_pb;         // macOS Apple Silicon running platform binary
+    static const GradedArchs arm64e_keysoff_pb; // macOS Apple Silicon running with signing keys disabled
 #endif
     static const GradedArchs armv7k;          // watch thru series 3
     static const GradedArchs armv7s;          // deprecated
@@ -139,7 +149,7 @@ public:
 
 // private:
 // should be private, but compiler won't statically initialize static members above
-    struct CpuGrade { uint32_t type; uint32_t subtype; uint32_t grade; };
+    struct CpuGrade { uint32_t type; uint32_t subtype; bool osBinary; uint16_t grade; };
     const CpuGrade     _orderedCpuTypes[3];  // zero terminated
 };
 
@@ -149,7 +159,7 @@ struct VIS_HIDDEN FatFile : fat_header
 {
     static const FatFile*  isFatFile(const void* fileContent);
     void                   forEachSlice(Diagnostics& diag, uint64_t fileLen, void (^callback)(uint32_t sliceCpuType, uint32_t sliceCpuSubType, const void* sliceStart, uint64_t sliceSize, bool& stop)) const;
-    bool                   isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const GradedArchs& archs, uint64_t& sliceOffset, uint64_t& sliceLen, bool& missingSlice) const;
+    bool                   isFatFileWithSlice(Diagnostics& diag, uint64_t fileLen, const GradedArchs& archs, bool osBinary, uint64_t& sliceOffset, uint64_t& sliceLen, bool& missingSlice) const;
 
 private:
     bool                   isValidSlice(Diagnostics& diag, uint64_t fileLen, uint32_t sliceIndex,
@@ -180,18 +190,21 @@ struct VIS_HIDDEN MachOFile : mach_header
     bool            isMainExecutable() const;
     bool            isDynamicExecutable() const;
     bool            isStaticExecutable() const;
+    bool            isKextBundle() const;
+    bool            isFileSet() const;
     bool            isPreload() const;
     bool            isPIE() const;
     bool            isArch(const char* archName) const;
     const char*     archName() const;
     bool            is64() const;
+    uint32_t        maskedCpuSubtype() const;
     size_t          machHeaderSize() const;
     uint32_t        pointerSize() const;
     bool            uses16KPages() const;
-    bool            supportsPlatform(Platform) const;
+    bool            builtForPlatform(Platform, bool onlyOnePlatform=false) const;
+    bool            loadableIntoProcess(Platform processPlatform, const char* path) const;
     bool            isZippered() const;
     bool            inDyldCache() const;
-    bool            isSimulatorBinary() const;
     bool            getUuid(uuid_t uuid) const;
     bool            hasWeakDefs() const;
     bool            hasThreadLocalVariables() const;
@@ -200,12 +213,22 @@ struct VIS_HIDDEN MachOFile : mach_header
     const char*     installName() const;  // returns nullptr is no install name
     void            forEachDependentDylib(void (^callback)(const char* loadPath, bool isWeak, bool isReExport, bool isUpward, uint32_t compatVersion, uint32_t curVersion, bool& stop)) const;
     bool            canBePlacedInDyldCache(const char* path, void (^failureReason)(const char*)) const;
+#if BUILDING_APP_CACHE_UTIL
+    bool            canBePlacedInKernelCollection(const char* path, void (^failureReason)(const char*)) const;
+#endif
+    bool            canHavePrecomputedDlopenClosure(const char* path, void (^failureReason)(const char*)) const;
     bool            canBeFairPlayEncrypted() const;
     bool            isFairPlayEncrypted(uint32_t& textOffset, uint32_t& size) const;
     bool            allowsAlternatePlatform() const;
     bool            hasChainedFixups() const;
+    bool            hasChainedFixupsLoadCommand() const;
     void            forDyldEnv(void (^callback)(const char* envVar, bool& stop)) const;
     bool            enforceCompatVersion() const;
+    bool            hasInterposingTuples() const;
+
+    const thread_command*   unixThreadLoadCommand() const;
+    uint32_t                entryAddrRegisterIndexForThreadCmd() const;
+    uint64_t                entryAddrFromThreadCmd(const thread_command* cmd) const;
 
     struct SegmentInfo
     {
@@ -246,6 +269,7 @@ struct VIS_HIDDEN MachOFile : mach_header
 protected:
     bool            hasMachOBigEndianMagic() const;
     void            forEachLoadCommand(Diagnostics& diag, void (^callback)(const load_command* cmd, bool& stop)) const;
+    void            removeLoadCommand(Diagnostics& diag, void (^callback)(const load_command* cmd, bool& remove, bool& stop));
     bool            hasLoadCommand(uint32_t) const;
 
     const encryption_info_command* findFairPlayEncryptionLoadCommand() const;

dyld3/MachOLoaded.cpp

@@ -211,6 +211,7 @@ void MachOLoaded::getLayoutInfo(LayoutInfo& result) const
             result.linkeditFileSize     = (uint32_t)info.fileSize;
             result.linkeditSegIndex     = info.segIndex;
         }
+        result.lastSegIndex = info.segIndex;
     });
 }
 
@@ -583,6 +584,15 @@ bool MachOLoaded::findClosestSymbol(uint64_t address, const char** symbolName, u
         return false;
     uint64_t targetUnslidAddress = address - leInfo.layout.slide;
 
+    // find section index the address is in to validate n_sect
+    __block uint32_t sectionIndexForTargetAddress = 0;
+    forEachSection(^(const SectionInfo& sectInfo, bool malformedSectionRange, bool& stop) {
+        ++sectionIndexForTargetAddress;
+        if ( (sectInfo.sectAddr <= targetUnslidAddress) && (targetUnslidAddress < sectInfo.sectAddr+sectInfo.sectSize) ) {
+            stop = true;
+        }
+    });
+
     uint32_t               maxStringOffset  = leInfo.symTab->strsize;
     const char*            stringPool       =             (char*)getLinkEditContent(leInfo.layout, leInfo.symTab->stroff);
     const struct nlist*    symbols          = (struct nlist*)   (getLinkEditContent(leInfo.layout, leInfo.symTab->symoff));
@@ -595,10 +605,10 @@ bool MachOLoaded::findClosestSymbol(uint64_t address, const char** symbolName, u
         for (const struct nlist_64* s = globalsStart; s < globalsEnd; ++s) {
             if ( (s->n_type & N_TYPE) == N_SECT ) {
                 if ( bestSymbol == nullptr ) {
-                    if ( s->n_value <= targetUnslidAddress )
+                    if ( (s->n_value <= targetUnslidAddress) && (s->n_sect == sectionIndexForTargetAddress) )
                         bestSymbol = s;
                 }
-                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) ) {
+                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) && (s->n_sect == sectionIndexForTargetAddress) ) {
                     bestSymbol = s;
                 }
             }
@@ -609,10 +619,10 @@ bool MachOLoaded::findClosestSymbol(uint64_t address, const char** symbolName, u
         for (const struct nlist_64* s = localsStart; s < localsEnd; ++s) {
              if ( ((s->n_type & N_TYPE) == N_SECT) && ((s->n_type & N_STAB) == 0) ) {
                 if ( bestSymbol == nullptr ) {
-                    if ( s->n_value <= targetUnslidAddress )
+                    if ( (s->n_value <= targetUnslidAddress) && (s->n_sect == sectionIndexForTargetAddress) )
                         bestSymbol = s;
                 }
-                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) ) {
+                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) && (s->n_sect == sectionIndexForTargetAddress) ) {
                     bestSymbol = s;
                 }
             }
@@ -632,10 +642,10 @@ bool MachOLoaded::findClosestSymbol(uint64_t address, const char** symbolName, u
         for (const struct nlist* s = globalsStart; s < globalsEnd; ++s) {
             if ( (s->n_type & N_TYPE) == N_SECT ) {
                 if ( bestSymbol == nullptr ) {
-                    if ( s->n_value <= targetUnslidAddress )
+                    if ( (s->n_value <= targetUnslidAddress) && (s->n_sect == sectionIndexForTargetAddress) )
                         bestSymbol = s;
                 }
-                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) ) {
+                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) && (s->n_sect == sectionIndexForTargetAddress) ) {
                     bestSymbol = s;
                 }
             }
@@ -646,10 +656,10 @@ bool MachOLoaded::findClosestSymbol(uint64_t address, const char** symbolName, u
         for (const struct nlist* s = localsStart; s < localsEnd; ++s) {
              if ( ((s->n_type & N_TYPE) == N_SECT) && ((s->n_type & N_STAB) == 0) ) {
                 if ( bestSymbol == nullptr ) {
-                    if ( s->n_value <= targetUnslidAddress )
+                    if ( (s->n_value <= targetUnslidAddress) && (s->n_sect == sectionIndexForTargetAddress) )
                         bestSymbol = s;
                 }
-                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) ) {
+                else if ( (s->n_value <= targetUnslidAddress) && (bestSymbol->n_value < s->n_value) && (s->n_sect == sectionIndexForTargetAddress) ) {
                     bestSymbol = s;
                 }
             }
@@ -698,9 +708,8 @@ bool MachOLoaded::intersectsRange(uintptr_t start, uintptr_t length) const
 
 const uint8_t* MachOLoaded::trieWalk(Diagnostics& diag, const uint8_t* start, const uint8_t* end, const char* symbol)
 {
-    uint32_t visitedNodeOffsets[128];
-    int visitedNodeOffsetCount = 0;
-    visitedNodeOffsets[visitedNodeOffsetCount++] = 0;
+    STACK_ALLOC_OVERFLOW_SAFE_ARRAY(uint32_t, visitedNodeOffsets, 128);
+    visitedNodeOffsets.push_back(0);
     const uint8_t* p = start;
     while ( p < end ) {
         uint64_t terminalSize = *p++;
@@ -769,17 +778,14 @@ const uint8_t* MachOLoaded::trieWalk(Diagnostics& diag, const uint8_t* start, co
                 diag.error("malformed trie child, nodeOffset=0x%llX out of range\n", nodeOffset);
                return nullptr;
             }
-            for (int i=0; i < visitedNodeOffsetCount; ++i) {
-                if ( visitedNodeOffsets[i] == nodeOffset ) {
+            // check for cycles
+            for (uint32_t aVisitedNodeOffset : visitedNodeOffsets) {
+                if ( aVisitedNodeOffset == nodeOffset ) {
                     diag.error("malformed trie child, cycle to nodeOffset=0x%llX\n", nodeOffset);
                     return nullptr;
                 }
             }
-            visitedNodeOffsets[visitedNodeOffsetCount++] = (uint32_t)nodeOffset;
-            if ( visitedNodeOffsetCount >= 128 ) {
-                diag.error("malformed trie too deep\n");
-                return nullptr;
-            }
+            visitedNodeOffsets.push_back((uint32_t)nodeOffset);
             p = &start[nodeOffset];
         }
         else
@@ -791,7 +797,6 @@ const uint8_t* MachOLoaded::trieWalk(Diagnostics& diag, const uint8_t* start, co
 void MachOLoaded::forEachCDHashOfCodeSignature(const void* codeSigStart, size_t codeSignLen,
                                                void (^callback)(const uint8_t cdHash[20])) const
 {
-#ifndef DARLING
     forEachCodeDirectoryBlob(codeSigStart, codeSignLen, ^(const void *cdBuffer) {
         const CS_CodeDirectory* cd = (const CS_CodeDirectory*)cdBuffer;
         uint32_t cdLength = htonl(cd->length);
@@ -834,7 +839,6 @@ void MachOLoaded::forEachCDHashOfCodeSignature(const void* codeSigStart, size_t
             return;
         }
     });
-#endif
 }
 
 
@@ -907,7 +911,7 @@ void MachOLoaded::forEachCodeDirectoryBlob(const void* codeSigStart, size_t code
 
     // Note: The kernel sometimes chooses sha1 on watchOS, and sometimes sha256.
     // Embed all of them so that we just need to match any of them
-    const bool isWatchOS = this->supportsPlatform(Platform::watchOS);
+    const bool isWatchOS = this->builtForPlatform(Platform::watchOS);
     const bool isMainExecutable = this->isMainExecutable();
     auto hashRankFn = isWatchOS ? &hash_rank_watchOS_dylibs : &hash_rank;
 
@@ -967,36 +971,51 @@ uint64_t MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signExtendedAddend() co
         return addend19;
 }
 
-const char* MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::keyName() const
+const char* MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::keyName(uint8_t keyBits)
 {
     static const char* names[] = {
         "IA", "IB", "DA", "DB"
     };
-    assert(this->authBind.auth == 1);
-    uint8_t keyBits = this->authBind.key;
     assert(keyBits < 4);
     return names[keyBits];
 }
+const char* MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::keyName() const
+{
+    assert(this->authBind.auth == 1);
+    return keyName(this->authBind.key);
+}
+
+uint64_t MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signPointer(uint64_t unsignedAddr, void* loc, bool addrDiv, uint16_t diversity, uint8_t key)
+{
+    // don't sign NULL
+    if ( unsignedAddr == 0 )
+        return 0;
+        
+#if __has_feature(ptrauth_calls)
+    uint64_t extendedDiscriminator = diversity;
+    if ( addrDiv )
+        extendedDiscriminator = __builtin_ptrauth_blend_discriminator(loc, extendedDiscriminator);
+    switch ( key ) {
+        case 0: // IA
+            return (uintptr_t)__builtin_ptrauth_sign_unauthenticated((void*)unsignedAddr, 0, extendedDiscriminator);
+        case 1: // IB
+            return (uintptr_t)__builtin_ptrauth_sign_unauthenticated((void*)unsignedAddr, 1, extendedDiscriminator);
+        case 2: // DA
+            return (uintptr_t)__builtin_ptrauth_sign_unauthenticated((void*)unsignedAddr, 2, extendedDiscriminator);
+        case 3: // DB
+            return (uintptr_t)__builtin_ptrauth_sign_unauthenticated((void*)unsignedAddr, 3, extendedDiscriminator);
+    }
+    assert(0 && "invalid signing key");
+#else
+    assert(0 && "arm64e signing only arm64e");
+#endif
+}
+
 
 uint64_t MachOLoaded::ChainedFixupPointerOnDisk::Arm64e::signPointer(void* loc, uint64_t target) const
 {
     assert(this->authBind.auth == 1);
-#if __has_feature(ptrauth_calls)
-    uint64_t discriminator = authBind.diversity;
-    if ( authBind.addrDiv )
-        discriminator = __builtin_ptrauth_blend_discriminator(loc, discriminator);
-    switch ( authBind.key ) {
-        case 0: // IA
-            return (uint64_t)__builtin_ptrauth_sign_unauthenticated((void*)target, 0, discriminator);
-        case 1: // IB
-            return (uint64_t)__builtin_ptrauth_sign_unauthenticated((void*)target, 1, discriminator);
-        case 2: // DA
-            return (uint64_t)__builtin_ptrauth_sign_unauthenticated((void*)target, 2, discriminator);
-        case 3: // DB
-            return (uint64_t)__builtin_ptrauth_sign_unauthenticated((void*)target, 3, discriminator);
-    }
-#endif
-    return target;
+    return signPointer(target, loc, authBind.addrDiv, authBind.diversity, authBind.key);
 }
 
 uint64_t MachOLoaded::ChainedFixupPointerOnDisk::Generic64::unpackedTarget() const
@@ -1013,10 +1032,25 @@ uint64_t MachOLoaded::ChainedFixupPointerOnDisk::Generic64::signExtendedAddend()
     return newValue;
 }
 
+const char* MachOLoaded::ChainedFixupPointerOnDisk::Kernel64::keyName() const
+{
+    static const char* names[] = {
+        "IA", "IB", "DA", "DB"
+    };
+    assert(this->isAuth == 1);
+    uint8_t keyBits = this->key;
+    assert(keyBits < 4);
+    return names[keyBits];
+}
+
 bool MachOLoaded::ChainedFixupPointerOnDisk::isRebase(uint16_t pointerFormat, uint64_t preferedLoadAddress, uint64_t& targetRuntimeOffset) const
 {
     switch (pointerFormat) {
        case DYLD_CHAINED_PTR_ARM64E:
+       case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+       case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
+       case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+       case DYLD_CHAINED_PTR_ARM64E_FIRMWARE:
             if ( this->arm64e.bind.bind )
                 return false;
             if ( this->arm64e.authRebase.auth ) {
@@ -1024,14 +1058,25 @@ bool MachOLoaded::ChainedFixupPointerOnDisk::isRebase(uint16_t pointerFormat, ui
                 return true;
             }
             else {
-                targetRuntimeOffset = this->arm64e.unpackTarget() - preferedLoadAddress;
+                targetRuntimeOffset = this->arm64e.unpackTarget();
+                if ( (pointerFormat == DYLD_CHAINED_PTR_ARM64E) || (pointerFormat == DYLD_CHAINED_PTR_ARM64E_FIRMWARE) ) {
+                    targetRuntimeOffset -= preferedLoadAddress;
+                }
                 return true;
             }
             break;
         case DYLD_CHAINED_PTR_64:
+        case DYLD_CHAINED_PTR_64_OFFSET:
             if ( this->generic64.bind.bind )
                 return false;
-            targetRuntimeOffset = this->generic64.unpackedTarget() - preferedLoadAddress;
+            targetRuntimeOffset = this->generic64.unpackedTarget();
+            if ( pointerFormat == DYLD_CHAINED_PTR_64 )
+                targetRuntimeOffset -= preferedLoadAddress;
+            return true;
+            break;
+        case DYLD_CHAINED_PTR_64_KERNEL_CACHE:
+        case DYLD_CHAINED_PTR_X86_64_KERNEL_CACHE:
+            targetRuntimeOffset = this->kernel64.target;
             return true;
             break;
         case DYLD_CHAINED_PTR_32:
@@ -1040,45 +1085,89 @@ bool MachOLoaded::ChainedFixupPointerOnDisk::isRebase(uint16_t pointerFormat, ui
             targetRuntimeOffset = this->generic32.rebase.target - preferedLoadAddress;
             return true;
             break;
+        case DYLD_CHAINED_PTR_32_FIRMWARE:
+            targetRuntimeOffset = this->firmware32.target - preferedLoadAddress;
+            return true;
+            break;
         default:
             break;
     }
     assert(0 && "unsupported pointer chain format");
 }
 
-bool MachOLoaded::ChainedFixupPointerOnDisk::isBind(uint16_t pointerFormat, uint32_t& bindOrdinal) const
+bool MachOLoaded::ChainedFixupPointerOnDisk::isBind(uint16_t pointerFormat, uint32_t& bindOrdinal, int64_t& addend) const
 {
+    addend = 0;
     switch (pointerFormat) {
         case DYLD_CHAINED_PTR_ARM64E:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
+        case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+        case DYLD_CHAINED_PTR_ARM64E_FIRMWARE:
             if ( !this->arm64e.authBind.bind )
                 return false;
             if ( this->arm64e.authBind.auth ) {
+                if ( pointerFormat == DYLD_CHAINED_PTR_ARM64E_USERLAND24 )
+                    bindOrdinal = this->arm64e.authBind24.ordinal;
+                else
                     bindOrdinal = this->arm64e.authBind.ordinal;
                 return true;
             }
             else {
+                if ( pointerFormat == DYLD_CHAINED_PTR_ARM64E_USERLAND24 )
+                    bindOrdinal = this->arm64e.bind24.ordinal;
+                else
                     bindOrdinal = this->arm64e.bind.ordinal;
+                addend = this->arm64e.signExtendedAddend();
                 return true;
             }
             break;
         case DYLD_CHAINED_PTR_64:
+        case DYLD_CHAINED_PTR_64_OFFSET:
             if ( !this->generic64.bind.bind )
                 return false;
             bindOrdinal = this->generic64.bind.ordinal;
+            addend = this->generic64.bind.addend;
             return true;
             break;
         case DYLD_CHAINED_PTR_32:
             if ( !this->generic32.bind.bind )
                 return false;
             bindOrdinal = this->generic32.bind.ordinal;
+            addend = this->generic32.bind.addend;
             return true;
             break;
+        case DYLD_CHAINED_PTR_64_KERNEL_CACHE:
+        case DYLD_CHAINED_PTR_X86_64_KERNEL_CACHE:
+            return false;
         default:
             break;
     }
     assert(0 && "unsupported pointer chain format");
 }
 
+unsigned MachOLoaded::ChainedFixupPointerOnDisk::strideSize(uint16_t pointerFormat)
+{
+    switch (pointerFormat) {
+        case DYLD_CHAINED_PTR_ARM64E:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+        case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
+            return 8;
+        case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+        case DYLD_CHAINED_PTR_ARM64E_FIRMWARE:
+        case DYLD_CHAINED_PTR_32_FIRMWARE:
+        case DYLD_CHAINED_PTR_64:
+        case DYLD_CHAINED_PTR_64_OFFSET:
+        case DYLD_CHAINED_PTR_32:
+        case DYLD_CHAINED_PTR_32_CACHE:
+        case DYLD_CHAINED_PTR_64_KERNEL_CACHE:
+            return 4;
+        case DYLD_CHAINED_PTR_X86_64_KERNEL_CACHE:
+            return 1;
+    }
+    assert(0 && "unsupported pointer chain format");
+}
+
 #if BUILDING_DYLD || BUILDING_LIBDYLD
 void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_starts_in_image* starts, uintptr_t slide,
                                         Array<const void*> bindTargets, void (^logFixup)(void* loc, void* newValue)) const
@@ -1089,16 +1178,20 @@ void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_st
 #if __LP64__
   #if  __has_feature(ptrauth_calls)
            case DYLD_CHAINED_PTR_ARM64E:
+           case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+           case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+           case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
                if ( fixupLoc->arm64e.authRebase.auth ) {
                     if ( fixupLoc->arm64e.authBind.bind ) {
-                        if ( fixupLoc->arm64e.authBind.ordinal >= bindTargets.count() ) {
-                            diag.error("out of range bind ordinal %d (max %lu)", fixupLoc->arm64e.authBind.ordinal, bindTargets.count());
+                        uint32_t bindOrdinal = (segInfo->pointer_format == DYLD_CHAINED_PTR_ARM64E_USERLAND24) ? fixupLoc->arm64e.authBind24.ordinal : fixupLoc->arm64e.authBind.ordinal;
+                        if ( bindOrdinal >= bindTargets.count() ) {
+                            diag.error("out of range bind ordinal %d (max %lu)", bindOrdinal, bindTargets.count());
                             stop = true;
                             break;
                         }
                         else {
                             // authenticated bind
-                            newValue = (void*)(bindTargets[fixupLoc->arm64e.bind.ordinal]);
+                            newValue = (void*)(bindTargets[bindOrdinal]);
                             if (newValue != 0)  // Don't sign missing weak imports
                                 newValue = (void*)fixupLoc->arm64e.signPointer(fixupLoc, (uintptr_t)newValue);
                         }
@@ -1110,19 +1203,23 @@ void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_st
                 }
                 else {
                     if ( fixupLoc->arm64e.bind.bind ) {
-                        if ( fixupLoc->arm64e.bind.ordinal >= bindTargets.count() ) {
-                            diag.error("out of range bind ordinal %d (max %lu)", fixupLoc->arm64e.bind.ordinal, bindTargets.count());
+                        uint32_t bindOrdinal = (segInfo->pointer_format == DYLD_CHAINED_PTR_ARM64E_USERLAND24) ? fixupLoc->arm64e.bind24.ordinal : fixupLoc->arm64e.bind.ordinal;
+                        if ( bindOrdinal >= bindTargets.count() ) {
+                            diag.error("out of range bind ordinal %d (max %lu)", bindOrdinal, bindTargets.count());
                             stop = true;
                             break;
                         }
                         else {
                             // plain bind
-                            newValue = (void*)((long)bindTargets[fixupLoc->arm64e.bind.ordinal] + fixupLoc->arm64e.signExtendedAddend());
+                            newValue = (void*)((long)bindTargets[bindOrdinal] + fixupLoc->arm64e.signExtendedAddend());
                         }
                     }
                     else {
-                        // plain rebase
+                        // plain rebase (old format target is vmaddr, new format target is offset)
+                        if ( segInfo->pointer_format == DYLD_CHAINED_PTR_ARM64E )
                             newValue = (void*)(fixupLoc->arm64e.unpackTarget()+slide);
+                        else
+                            newValue = (void*)((uintptr_t)this + fixupLoc->arm64e.unpackTarget());
                    }
                 }
                 if ( logFixup )
@@ -1131,6 +1228,7 @@ void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_st
                 break;
   #endif
             case DYLD_CHAINED_PTR_64:
+            case DYLD_CHAINED_PTR_64_OFFSET:
                 if ( fixupLoc->generic64.bind.bind ) {
                     if ( fixupLoc->generic64.bind.ordinal >= bindTargets.count() ) {
                         diag.error("out of range bind ordinal %d (max %lu)", fixupLoc->generic64.bind.ordinal, bindTargets.count());
@@ -1142,7 +1240,11 @@ void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_st
                     }
                 }
                 else {
+                    // plain rebase (old format target is vmaddr, new format target is offset)
+                    if ( segInfo->pointer_format == DYLD_CHAINED_PTR_64 )
                         newValue = (void*)(fixupLoc->generic64.unpackedTarget()+slide);
+                    else
+                        newValue = (void*)((uintptr_t)this + fixupLoc->generic64.unpackedTarget());
                 }
                 if ( logFixup )
                     logFixup(fixupLoc, newValue);
@@ -1184,26 +1286,31 @@ void MachOLoaded::fixupAllChainedFixups(Diagnostics& diag, const dyld_chained_st
 }
 #endif
 
-bool MachOLoaded::walkChain(Diagnostics& diag, const dyld_chained_starts_in_segment* segInfo, uint32_t pageIndex, uint16_t offsetInPage,
-                            bool notifyNonPointers, void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const
+
+bool MachOLoaded::walkChain(Diagnostics& diag, ChainedFixupPointerOnDisk* chain, uint16_t pointer_format, bool notifyNonPointers, uint32_t max_valid_pointer,
+                            void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, bool& stop)) const
 {
+    const unsigned stride = ChainedFixupPointerOnDisk::strideSize(pointer_format);
     bool  stop = false;
-    uint8_t*                   pageContentStart = (uint8_t*)this + segInfo->segment_offset + (pageIndex * segInfo->page_size);
-    ChainedFixupPointerOnDisk* chain = (ChainedFixupPointerOnDisk*)(pageContentStart+offsetInPage);
     bool  chainEnd = false;
     while (!stop && !chainEnd) {
         // copy chain content, in case handler modifies location to final value
         ChainedFixupPointerOnDisk chainContent = *chain;
-        handler(chain, segInfo, stop);
+        handler(chain, stop);
         if ( !stop ) {
-            switch (segInfo->pointer_format) {
+            switch (pointer_format) {
                 case DYLD_CHAINED_PTR_ARM64E:
+                case DYLD_CHAINED_PTR_ARM64E_KERNEL:
+                case DYLD_CHAINED_PTR_ARM64E_USERLAND:
+                case DYLD_CHAINED_PTR_ARM64E_USERLAND24:
+                case DYLD_CHAINED_PTR_ARM64E_FIRMWARE:
                     if ( chainContent.arm64e.rebase.next == 0 )
                         chainEnd = true;
                     else
-                        chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chainContent.arm64e.rebase.next*8);
+                        chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chainContent.arm64e.rebase.next*stride);
                     break;
                 case DYLD_CHAINED_PTR_64:
+                case DYLD_CHAINED_PTR_64_OFFSET:
                     if ( chainContent.generic64.rebase.next == 0 )
                         chainEnd = true;
                     else
@@ -1215,15 +1322,28 @@ bool MachOLoaded::walkChain(Diagnostics& diag, const dyld_chained_starts_in_segm
                     else {
                         chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chainContent.generic32.rebase.next*4);
                         if ( !notifyNonPointers ) {
-                            while ( (chain->generic32.rebase.bind == 0) && (chain->generic32.rebase.target > segInfo->max_valid_pointer) ) {
+                            while ( (chain->generic32.rebase.bind == 0) && (chain->generic32.rebase.target > max_valid_pointer) ) {
                                 // not a real pointer, but a non-pointer co-opted into chain
                                 chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chain->generic32.rebase.next*4);
                             }
                         }
                     }
                     break;
+                case DYLD_CHAINED_PTR_64_KERNEL_CACHE:
+                case DYLD_CHAINED_PTR_X86_64_KERNEL_CACHE:
+                    if ( chainContent.kernel64.next == 0 )
+                        chainEnd = true;
+                    else
+                        chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chainContent.kernel64.next*stride);
+                    break;
+                case DYLD_CHAINED_PTR_32_FIRMWARE:
+                    if ( chainContent.firmware32.next == 0 )
+                        chainEnd = true;
+                    else
+                        chain = (ChainedFixupPointerOnDisk*)((uint8_t*)chain + chainContent.firmware32.next*4);
+                    break;
                 default:
-                    diag.error("unknown pointer format 0x%04X", segInfo->pointer_format);
+                    diag.error("unknown pointer format 0x%04X", pointer_format);
                     stop = true;
             }
         }
@@ -1231,14 +1351,25 @@ bool MachOLoaded::walkChain(Diagnostics& diag, const dyld_chained_starts_in_segm
     return stop;
 }
 
-void MachOLoaded::forEachFixupInAllChains(Diagnostics& diag, const dyld_chained_starts_in_image* starts, bool notifyNonPointers,
-                                          void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const
+void MachOLoaded::forEachFixupChainSegment(Diagnostics& diag, const dyld_chained_starts_in_image* starts,
+                                           void (^handler)(const dyld_chained_starts_in_segment* segInfo, uint32_t segIndex, bool& stop)) const
 {
     bool stopped = false;
     for (uint32_t segIndex=0; segIndex < starts->seg_count && !stopped; ++segIndex) {
         if ( starts->seg_info_offset[segIndex] == 0 )
             continue;
         const dyld_chained_starts_in_segment* segInfo = (dyld_chained_starts_in_segment*)((uint8_t*)starts + starts->seg_info_offset[segIndex]);
+        handler(segInfo, segIndex, stopped);
+    }
+}
+
+void MachOLoaded::forEachFixupInSegmentChains(Diagnostics& diag, const dyld_chained_starts_in_segment* segInfo, bool notifyNonPointers,
+                                              void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const
+{
+    auto adaptor = ^(ChainedFixupPointerOnDisk* fixupLocation, bool& stop) {
+         handler(fixupLocation, segInfo, stop);
+    };
+    bool stopped = false;
     for (uint32_t pageIndex=0; pageIndex < segInfo->page_count && !stopped; ++pageIndex) {
         uint16_t offsetInPage = segInfo->page_start[pageIndex];
         if ( offsetInPage == DYLD_CHAINED_PTR_START_NONE )
@@ -1250,16 +1381,40 @@ void MachOLoaded::forEachFixupInAllChains(Diagnostics& diag, const dyld_chained_
             while (!stopped && !chainEnd) {
                 chainEnd = (segInfo->page_start[overflowIndex] & DYLD_CHAINED_PTR_START_LAST);
                 offsetInPage = (segInfo->page_start[overflowIndex] & ~DYLD_CHAINED_PTR_START_LAST);
-                    if ( walkChain(diag, segInfo, pageIndex, offsetInPage, notifyNonPointers, handler) )
-                        stopped = true;
+                uint8_t* pageContentStart = (uint8_t*)this + segInfo->segment_offset + (pageIndex * segInfo->page_size);
+                ChainedFixupPointerOnDisk* chain = (ChainedFixupPointerOnDisk*)(pageContentStart+offsetInPage);
+                stopped = walkChain(diag, chain, segInfo->pointer_format, notifyNonPointers, segInfo->max_valid_pointer, adaptor);
                 ++overflowIndex;
             }
         }
         else {
             // one chain per page
-                walkChain(diag, segInfo, pageIndex, offsetInPage, notifyNonPointers, handler);
+            uint8_t* pageContentStart = (uint8_t*)this + segInfo->segment_offset + (pageIndex * segInfo->page_size);
+            ChainedFixupPointerOnDisk* chain = (ChainedFixupPointerOnDisk*)(pageContentStart+offsetInPage);
+            stopped = walkChain(diag, chain, segInfo->pointer_format, notifyNonPointers, segInfo->max_valid_pointer, adaptor);
         }
     }
+}
+
+void MachOLoaded::forEachFixupInAllChains(Diagnostics& diag, const dyld_chained_starts_in_image* starts, bool notifyNonPointers,
+                                          void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const
+{
+    bool stopped = false;
+    for (uint32_t segIndex=0; segIndex < starts->seg_count && !stopped; ++segIndex) {
+        if ( starts->seg_info_offset[segIndex] == 0 )
+            continue;
+        const dyld_chained_starts_in_segment* segInfo = (dyld_chained_starts_in_segment*)((uint8_t*)starts + starts->seg_info_offset[segIndex]);
+        forEachFixupInSegmentChains(diag, segInfo, notifyNonPointers, handler);
+    }
+}
+
+void MachOLoaded::forEachFixupInAllChains(Diagnostics& diag, uint16_t pointer_format, uint32_t starts_count, const uint32_t chain_starts[],
+                                          void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, bool& stop)) const
+{
+    for (uint32_t i=0; i < starts_count; ++i) {
+        ChainedFixupPointerOnDisk* chain = (ChainedFixupPointerOnDisk*)((uint8_t*)this + chain_starts[i]);
+        if ( walkChain(diag, chain, pointer_format, false, 0, handler) )
+            break;
     }
 }
 

dyld3/MachOLoaded.h

@@ -30,7 +30,7 @@
 #include "MachOFile.h"
 
 
-class CacheBuilder;
+class SharedCacheBuilder;
 
 namespace dyld3 {
 
@@ -79,6 +79,7 @@ struct VIS_HIDDEN MachOLoaded : public MachOFile
                                                   Array<const void*> bindTargets, void (^fixupLogger)(void* loc, void* newValue)) const;
 #endif
 
+    void                    forEachGlobalSymbol(Diagnostics& diag, void (^callback)(const char* symbolName, uint64_t n_value, uint8_t n_type, uint8_t n_sect, uint16_t n_desc, bool& stop)) const;
 
 
     // For use with new rebase/bind scheme were each fixup location on disk contains info on what
@@ -90,11 +91,15 @@ struct VIS_HIDDEN MachOLoaded : public MachOFile
             dyld_chained_ptr_arm64e_auth_bind   authBind;
             dyld_chained_ptr_arm64e_rebase      rebase;
             dyld_chained_ptr_arm64e_bind        bind;
+            dyld_chained_ptr_arm64e_bind24      bind24;
+            dyld_chained_ptr_arm64e_auth_bind24 authBind24;
 
             uint64_t        signExtendedAddend() const;
             uint64_t        unpackTarget() const;
             const char*     keyName() const;
             uint64_t        signPointer(void* loc, uint64_t target) const;
+            static uint64_t    signPointer(uint64_t unsignedPtr, void* loc, bool addrDiv, uint16_t diversity, uint8_t key);
+            static const char* keyName(uint8_t keyBits);
         };
 
         union Generic64 {
@@ -112,19 +117,27 @@ struct VIS_HIDDEN MachOLoaded : public MachOFile
             uint64_t        signExtendedAddend() const;
         };
 
+        struct Kernel64 : dyld_chained_ptr_64_kernel_cache_rebase {
+            const char* keyName() const;
+        };
+
+        struct Firm32 : dyld_chained_ptr_32_firmware_rebase { };
+
         typedef dyld_chained_ptr_32_cache_rebase Cache32;
 
         uint64_t            raw64;
         Arm64e              arm64e;
         Generic64           generic64;
+        Kernel64            kernel64;
 
         uint32_t            raw32;
         Generic32           generic32;
         Cache32             cache32;
-
+        Firm32              firmware32;
 
         bool                isRebase(uint16_t pointerFormat, uint64_t preferedLoadAddress, uint64_t& targetRuntimeOffset) const;
-        bool                isBind(uint16_t pointerFormat, uint32_t& bindOrdinal) const;
+        bool                isBind(uint16_t pointerFormat, uint32_t& bindOrdinal, int64_t& addend) const;
+        static unsigned     strideSize(uint16_t pointerFormat);
     };
 
 
@@ -135,6 +148,7 @@ struct VIS_HIDDEN MachOLoaded : public MachOFile
         uint32_t     linkeditFileOffset;
         uint32_t     linkeditFileSize;
         uint32_t     linkeditSegIndex;
+        uint32_t     lastSegIndex;
     };
 
     struct LinkEditInfo
@@ -152,11 +166,21 @@ struct VIS_HIDDEN MachOLoaded : public MachOFile
     };
     void                    getLinkEditPointers(Diagnostics& diag, LinkEditInfo&) const;
 
-    // use by dyldinfo
+    void forEachFixupChainSegment(Diagnostics& diag, const dyld_chained_starts_in_image* starts,
+                                  void (^handler)(const dyld_chained_starts_in_segment* segInfo, uint32_t segIndex, bool& stop)) const;
+
+    void forEachFixupInSegmentChains(Diagnostics& diag, const dyld_chained_starts_in_segment* segInfo, bool notifyNonPointers,
+                                     void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const;
+
+    // for dyld loaded images
     void                    forEachFixupInAllChains(Diagnostics& diag, const dyld_chained_starts_in_image* starts, bool notifyNonPointers,
                                                     void (^callback)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const;
+    // for preload images
+    void                    forEachFixupInAllChains(Diagnostics& diag, uint16_t pointer_format, uint32_t starts_count, const uint32_t chain_starts[],
+                                                    void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, bool& stop)) const;
+
 protected:
-    friend CacheBuilder;
+    friend SharedCacheBuilder;
 
     struct FoundSymbol {
         enum class Kind { headerOffset, absolute, resolverOffset };
@@ -169,18 +193,14 @@ protected:
         const char*         foundSymbolName;
     };
 
-
-protected:
-    friend CacheBuilder;
-
     bool                    findExportedSymbol(Diagnostics& diag, const char* symbolName, bool weakImport, FoundSymbol& foundInfo, DependentToMachOLoaded finder) const;
 
     void                    getLinkEditLoadCommands(Diagnostics& diag, LinkEditInfo& result) const;
     void                    getLayoutInfo(LayoutInfo&) const;
     const uint8_t*          getLinkEditContent(const LayoutInfo& info, uint32_t fileOffset) const;
     const uint8_t*          getExportsTrie(const LinkEditInfo& info, uint64_t& trieSize) const;
-    void                    forEachGlobalSymbol(Diagnostics& diag, void (^callback)(const char* symbolName, uint64_t n_value, uint8_t n_type, uint8_t n_sect, uint16_t n_desc, bool& stop)) const;
     void                    forEachLocalSymbol(Diagnostics& diag, void (^callback)(const char* symbolName, uint64_t n_value, uint8_t n_type, uint8_t n_sect, uint16_t n_desc, bool& stop)) const;
+
     uint32_t                dependentDylibCount() const;
     bool                    findClosestFunctionStart(uint64_t address, uint64_t* functionStartAddress) const;
 
@@ -190,9 +210,8 @@ protected:
     // On all other platforms this always returns a single best cd hash (ranked to match the kernel).
     // Note the callback parameter is really a CS_CodeDirectory.
     void                    forEachCodeDirectoryBlob(const void* codeSigStart, size_t codeSignLen, void (^callback)(const void* cd)) const;
-    bool                    walkChain(Diagnostics& diag, const dyld_chained_starts_in_segment* segInfo, uint32_t pageIndex, uint16_t offsetInPage,
-                                      bool notifyNonPointers, void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, const dyld_chained_starts_in_segment* segInfo, bool& stop)) const;
-
+    bool                    walkChain(Diagnostics& diag, ChainedFixupPointerOnDisk* start, uint16_t pointer_format, bool notifyNonPointers, uint32_t max_valid_pointer,
+                                        void (^handler)(ChainedFixupPointerOnDisk* fixupLocation, bool& stop)) const;
 
 };
 

dyld3/PathOverrides.cpp

@@ -33,13 +33,13 @@
 #include <limits.h>
 #include <sys/errno.h>
 #include <unistd.h>
+#include <TargetConditionals.h>
 
 #include <initializer_list>
 
 #include "PathOverrides.h"
 
 
-
 namespace dyld3 {
 namespace closure {
 
@@ -94,7 +94,7 @@ PathOverrides::~PathOverrides()
 
 void PathOverrides::forEachInsertedDylib(void (^handler)(const char* dylibPath, bool &stop)) const
 {
-    if ( _insertedDylibs != nullptr ) {
+    if ( _insertedDylibs != nullptr && _insertedDylibs[0] != '\0' ) {
         forEachInColonList(_insertedDylibs, nullptr, ^(const char* path, bool &stop) {
             handler(path, stop);
         });
@@ -401,6 +401,25 @@ void PathOverrides::forEachPathVariant(const char* initialPath, bool pathIsInDyl
             return;
     }
 
+    // try rootpath
+    bool searchiOSSupport = (platform == Platform::iOSMac);
+#if (TARGET_OS_OSX && TARGET_CPU_ARM64)
+    if ( platform == Platform::iOS ) {
+        searchiOSSupport = true;
+        // <rdar://problem/58959974> some old Almond apps reference old WebKit location
+        if ( strcmp(initialPath, "/System/Library/PrivateFrameworks/WebKit.framework/WebKit") == 0 )
+            initialPath = "/System/Library/Frameworks/WebKit.framework/WebKit";
+    }
+#endif
+    if ( searchiOSSupport ) {
+        char rtpath[strlen("/System/iOSSupport")+strlen(initialPath)+8];
+        strcpy(rtpath, "/System/iOSSupport");
+        strcat(rtpath, initialPath);
+        forEachImageSuffix(rtpath, false, pathIsInDyldCacheWhichCannotBeOverridden, stop, handler);
+        if ( stop )
+            return;
+    }
+
     // try original path
     forEachImageSuffix(initialPath, false, pathIsInDyldCacheWhichCannotBeOverridden, stop, handler);
     if ( stop )
@@ -533,7 +552,7 @@ const char* PathPool::add(const char* path)
     return _next->add(path);
 }
 
-void PathPool::forEachPath(void (^handler)(const char* path))
+void PathPool::forEachPath(void (^handler)(const char* path)) const
 {
     for (const char* s = _buffer; s < _current; ++s) {
         handler(s);
@@ -544,6 +563,20 @@ void PathPool::forEachPath(void (^handler)(const char* path))
         _next->forEachPath(handler);
 }
 
+bool PathPool::contains(const char* path) const
+{
+    for (const char* s = _buffer; s < _current; ++s) {
+        if ( strcmp(s, path) == 0 )
+            return true;
+        s += strlen(s);
+    }
+
+    if ( _next != nullptr )
+        return _next->contains(path);
+
+    return false;
+}
+
 
 
 } // namespace closure

dyld3/PathOverrides.h

@@ -42,7 +42,8 @@ public:
     static PathPool*    allocate();
     static void         deallocate(PathPool* pool);
     const char*         add(const char* path);
-    void                forEachPath(void (^handler)(const char* path));
+    void                forEachPath(void (^handler)(const char* path)) const;
+    bool                contains(const char* path) const;
 
 private:
     enum { kAllocationSize = 32*1024 };

dyld3/PointerAuth.h

@@ -0,0 +1,90 @@
+/*
+ * Copyright (c) 2017 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+
+
+#ifndef __DYLD_POINTER_AUTH_H__
+#define __DYLD_POINTER_AUTH_H__
+
+#include <ptrauth.h>
+
+#if __has_feature(ptrauth_calls)
+#define __ptrauth_dyld_address_auth __ptrauth(ptrauth_key_process_dependent_data, 1, 0)
+#define __ptrauth_dyld_function_ptr __ptrauth(ptrauth_key_process_dependent_code, 1, 0)
+#else
+#define __ptrauth_dyld_address_auth
+#define __ptrauth_dyld_function_ptr
+#endif
+
+namespace dyld3 {
+
+// On arm64e, signs the given pointer with the address of where it is stored.
+// Other archs just have a regular pointer
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wptrauth-null-pointers"
+template<typename T>
+struct AuthenticatedValue {
+    static_assert(sizeof(T) <= sizeof(uintptr_t));
+
+    AuthenticatedValue() {
+        this->value = ptrauth_sign_unauthenticated(nullptr, ptrauth_key_process_dependent_data, this);
+    }
+    ~AuthenticatedValue() = default;
+    AuthenticatedValue(const AuthenticatedValue& other) {
+        this->value = ptrauth_auth_and_resign(other.value,
+                                              ptrauth_key_process_dependent_data, &other,
+                                              ptrauth_key_process_dependent_data, this);
+    }
+    AuthenticatedValue(AuthenticatedValue&& other) {
+        this->value = ptrauth_auth_and_resign(other.value,
+                                              ptrauth_key_process_dependent_data, &other,
+                                              ptrauth_key_process_dependent_data, this);
+        other.value = ptrauth_sign_unauthenticated(nullptr, ptrauth_key_process_dependent_data, &other);
+    }
+
+    AuthenticatedValue& operator=(const AuthenticatedValue&) = delete;
+    AuthenticatedValue& operator=(AuthenticatedValue&&) = delete;
+
+    // Add a few convenience methods for interoperating with values of the given type
+    AuthenticatedValue& operator=(const T& other) {
+        this->value = ptrauth_sign_unauthenticated(other, ptrauth_key_process_dependent_data, this);
+        return *this;
+    }
+    bool operator==(const T& other) const {
+        return ptrauth_auth_data(this->value, ptrauth_key_process_dependent_data, this) == other;
+    }
+    bool operator!=(const T& other) const {
+        return ptrauth_auth_data(this->value, ptrauth_key_process_dependent_data, this) != other;
+    }
+
+private:
+    const void* value;
+};
+
+
+#pragma clang diagnostic pop
+
+} // namespace dyld3
+
+#endif // __DYLD_POINTER_AUTH_H__
+

dyld3/RootsChecker.cpp

@@ -0,0 +1,140 @@
+/*
+* Copyright (c) 2020 Apple Inc. All rights reserved.
+*
+* @APPLE_LICENSE_HEADER_START@
+*
+* This file contains Original Code and/or Modifications of Original Code
+* as defined in and that are subject to the Apple Public Source License
+* Version 2.0 (the 'License'). You may not use this file except in
+* compliance with the License. Please obtain a copy of the License at
+* http://www.opensource.apple.com/apsl/ and read it before using this
+* file.
+*
+* The Original Code and all software distributed under the License are
+* distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+* EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+* INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+* Please see the License for the specific language governing rights and
+* limitations under the License.
+*
+* @APPLE_LICENSE_HEADER_END@
+*/
+
+#include <TargetConditionals.h>
+
+#include "ClosureFileSystemPhysical.h"
+#include "DyldSharedCache.h"
+#include "MachOAnalyzer.h"
+#include "RootsChecker.h"
+
+#if DYLD_SIMULATOR_ROOTS_SUPPORT
+#include "SharedCacheRuntime.h"
+#endif
+
+namespace dyld3 {
+
+#if DYLD_SIMULATOR_ROOTS_SUPPORT
+static bool imageUUIDMatchesSharedCache(const char* path, const closure::FileSystem* fileSystem,
+                                       const DyldSharedCache* cache, const closure::Image* image) {
+    // We can only use the cache if the file UUID matches the shared cache
+    bool uuidMatchesCache = false;
+
+    // The simulator can't pass in a file system as its not hooked up to the vector.
+    // They can just pass in nullptr where needed and we'll assume its the physical file system
+    closure::FileSystemPhysical fileSystemPhysical;
+    if ( fileSystem == nullptr )
+        fileSystem = &fileSystemPhysical;
+
+    Diagnostics diag;
+    Platform platform = cache->platform();
+    const GradedArchs& archs = GradedArchs::forName(cache->archName(), true);
+    char realerPath[MAXPATHLEN];
+    dyld3::closure::LoadedFileInfo loadedFileInfo = dyld3::MachOAnalyzer::load(diag, *fileSystem, path, archs, platform, realerPath);
+    const dyld3::MachOAnalyzer* ma = (const dyld3::MachOAnalyzer*)loadedFileInfo.fileContent;
+    if ( ma != nullptr ) {
+        uuid_t uuid;
+        uuid_t image_uuid;
+        if ( !ma->getUuid(uuid) )
+            memset(&uuid, 0, sizeof(uuid_t));
+        if ( !image->getUuid(image_uuid) )
+            memset(&image_uuid, 0, sizeof(uuid_t));
+        uuidMatchesCache = ( memcmp(uuid, image_uuid, sizeof(uuid_t)) == 0 );
+        fileSystem->unloadFile(loadedFileInfo);
+    }
+    return uuidMatchesCache;
+}
+
+bool RootsChecker::uuidMatchesSharedCache(const char* path, const closure::FileSystem* fileSystem,
+                                          const DyldSharedCache* cache) {
+    const dyld3::closure::ImageArray* images = cache->cachedDylibsImageArray();
+    const closure::Image* image = nullptr;
+    uint32_t imageIndex;
+    if ( cache->hasImagePath(path, imageIndex) ) {
+        image = images->imageForNum(imageIndex+1);
+    }
+    return imageUUIDMatchesSharedCache(path, fileSystem, cache, image);
+}
+#endif
+
+bool RootsChecker::onDiskFileIsRoot(const char* path, const DyldSharedCache* cache,
+                                    const closure::Image* image,
+                                    const closure::FileSystem* fileSystem,
+                                    uint64_t inode, uint64_t modtime) const {
+
+    // Upon entry, we know the dylib exists and has a mod time and inode.  Now
+    // we need to see if its a root or not
+
+    // Note we don't check if dylibs are expected on disk here.  We assume that an
+    // image only has a mod time and inode if its expected on disk
+    uint64_t expectedINode;
+    uint64_t expectedMtime;
+    if ( image->hasFileModTimeAndInode(expectedINode, expectedMtime) ) {
+        if ( (expectedMtime == modtime) && (expectedINode == inode) )
+            return false;
+        // mod time or inode don't match, so this is a root
+        return true;
+    }
+
+#if DYLD_SIMULATOR_ROOTS_SUPPORT
+    if ( strcmp(path, "/usr/lib/system/libsystem_kernel.dylib") == 0 ) {
+        // If this was a root when launchd checked, then assume we are a root now
+        if ( libsystemKernelIsRoot )
+            return true;
+
+        // If the file system is read-only, then this cannot be a root now
+        if ( !fileSystemCanBeModified )
+            return false;
+
+        // Possibly a root.  Open the file and check
+        return !imageUUIDMatchesSharedCache(path, fileSystem, cache, image);
+    } else if ( strcmp(path, "/usr/lib/system/libsystem_platform.dylib") == 0 ) {
+        // If this was a root when launchd checked, then assume we are a root now
+        if ( libsystemPlatformIsRoot )
+            return true;
+
+        // If the file system is read-only, then this cannot be a root now
+        if ( !fileSystemCanBeModified )
+            return false;
+
+        // Possibly a root.  Open the file and check
+        return !imageUUIDMatchesSharedCache(path, fileSystem, cache, image);
+    } else if ( strcmp(path, "/usr/lib/system/libsystem_pthread.dylib") == 0 ) {
+        // If this was a root when launchd checked, then assume we are a root now
+        if ( libsystemPThreadIsRoot )
+            return true;
+
+        // If the file system is read-only, then this cannot be a root now
+        if ( !fileSystemCanBeModified )
+            return false;
+
+        // Possibly a root.  Open the file and check
+        return !imageUUIDMatchesSharedCache(path, fileSystem, cache, image);
+    }
+#endif
+
+    // If we aren't a special simulator dylib, then we must be a root
+    return true;
+}
+
+} // namespace dyld3

dyld3/RootsChecker.h

@@ -0,0 +1,94 @@
+/*
+ * Copyright (c) 2020 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+
+
+#ifndef __DYLD_ROOTS_CHECKER_H__
+#define __DYLD_ROOTS_CHECKER_H__
+
+#include <TargetConditionals.h>
+
+#if TARGET_OS_OSX && (BUILDING_DYLD || BUILDING_CLOSURE_UTIL) && !TARGET_OS_SIMULATOR
+#define DYLD_SIMULATOR_ROOTS_SUPPORT 1
+#else
+#define DYLD_SIMULATOR_ROOTS_SUPPORT 0
+#endif
+
+class DyldSharedCache;
+
+namespace dyld3 {
+
+namespace closure {
+class FileSystem;
+struct Image;
+}
+
+#define VIS_HIDDEN __attribute__((visibility("hidden")))
+
+class VIS_HIDDEN RootsChecker
+{
+public:
+    RootsChecker() = default;
+
+    // Does a on-disk binary represent a root.
+    // If the cache expects dylibs on disk, then so long as the dylib matches
+    // the mod time and inode in the cache, it is not a root.
+    // For the macOS simulator support dylibs, which are on disk, whether they
+    // are roots depends on the state we are tracking in the comm page.
+    // For all other dylibs and all other platforms, on-disk dylibs are always roots
+    bool onDiskFileIsRoot(const char* path, const DyldSharedCache* cache, const closure::Image* image,
+                          const closure::FileSystem* fileSystem,
+                          uint64_t inode, uint64_t modtime) const;
+
+#if DYLD_SIMULATOR_ROOTS_SUPPORT
+    static bool uuidMatchesSharedCache(const char* path, const closure::FileSystem* fileSystem,
+                                       const DyldSharedCache* cache);
+
+    void setLibsystemKernelIsRoot(bool v) {
+        libsystemKernelIsRoot = v;
+    }
+    void setLibsystemPlatformIsRoot(bool v) {
+        libsystemPlatformIsRoot = v;
+    }
+    void setLibsystemPThreadIsRoot(bool v) {
+        libsystemPThreadIsRoot = v;
+    }
+    void setFileSystemCanBeModified(bool v) {
+        fileSystemCanBeModified = v;
+    }
+#endif
+
+private:
+    // By default, assume nothing is a root, but that the file system is writable.
+    // This should be conservatively correct.
+#if DYLD_SIMULATOR_ROOTS_SUPPORT
+    bool libsystemKernelIsRoot      = false;
+    bool libsystemPlatformIsRoot    = false;
+    bool libsystemPThreadIsRoot     = false;
+    bool fileSystemCanBeModified    = true;
+#endif
+};
+
+} // namespace dyld3
+
+#endif // __DYLD_ROOTS_CHECKER_H__

dyld3/SharedCacheRuntime.cpp

@@ -26,6 +26,7 @@
 #include <stdint.h>
 #include <string.h>
 #include <unistd.h>
+#include <errno.h>
 #include <fcntl.h>
 #include <sys/param.h>
 #include <sys/types.h>
@@ -53,12 +54,20 @@
 // should be in mach/shared_region.h
 extern "C" int __shared_region_check_np(uint64_t* startaddress);
 extern "C" int __shared_region_map_and_slide_np(int fd, uint32_t count, const shared_file_mapping_np mappings[], long slide, const dyld_cache_slide_info2* slideInfo, size_t slideInfoSize);
+extern "C" int __shared_region_map_and_slide_2_np(uint32_t files_count, const shared_file_np files[], uint32_t mappings_count, const shared_file_mapping_slide_np mappings[]);
 
+#ifndef VM_PROT_NOAUTH
+#define VM_PROT_NOAUTH  0x40  /* must not interfere with normal prot assignments */
+#endif
+
+extern bool gEnableSharedCacheDataConst;
 
 namespace dyld {
-    extern int  my_stat(const char* path, struct stat* buf);
-    extern int  my_open(const char* path, int flag, int other);
     extern void log(const char*, ...);
+    extern void logToConsole(const char* format, ...);
+#if defined(__x86_64__) && !TARGET_OS_SIMULATOR
+    bool isTranslated();
+#endif
 }
 
 
@@ -67,10 +76,10 @@ namespace dyld3 {
 
 struct CacheInfo
 {
-    int                     fd;
-    shared_file_mapping_np  mappings[3];
-    uint64_t                slideInfoAddressUnslid;
-    size_t                  slideInfoSize;
+    shared_file_mapping_slide_np            mappings[DyldSharedCache::MaxMappings];
+    uint32_t                                mappingsCount;
+    // All mappings come from the same file
+    int                                     fd               = 0;
     uint64_t                                sharedRegionStart;
     uint64_t                                sharedRegionSize;
     uint64_t                                maxSlide;
@@ -169,57 +178,123 @@ static void rebaseChainV4(uint8_t* pageContent, uint16_t startOffset, uintptr_t
 }
 #endif
 
-static void getCachePath(const SharedCacheOptions& options, size_t pathBufferSize, char pathBuffer[])
-{
+#if TARGET_OS_OSX
+bool getMacOSCachePath(char pathBuffer[], size_t pathBufferSize,
+                       const char* cacheDir, bool useHaswell) {
+    // Clear old attempts at finding a cache, if any
+    pathBuffer[0] = '\0';
+
     // set cache dir
-    if ( options.cacheDirOverride != nullptr ) {
-        strlcpy(pathBuffer, options.cacheDirOverride, pathBufferSize);
-    }
-    else {
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
-        strlcpy(pathBuffer, IPHONE_DYLD_SHARED_CACHE_DIR, sizeof(IPHONE_DYLD_SHARED_CACHE_DIR));
-#else
-        strlcpy(pathBuffer, MACOSX_DYLD_SHARED_CACHE_DIR, sizeof(MACOSX_DYLD_SHARED_CACHE_DIR));
-#endif
-    }
+    strlcpy(pathBuffer, cacheDir, pathBufferSize);
 
     // append file component of cache file
     if ( pathBuffer[strlen(pathBuffer)-1] != '/' )
         strlcat(pathBuffer, "/", pathBufferSize);
-#if __x86_64__ && !__IPHONE_OS_VERSION_MIN_REQUIRED
-    if ( options.useHaswell ) {
+
+#if __x86_64__
+    if ( useHaswell ) {
         size_t len = strlen(pathBuffer);
         struct stat haswellStatBuf;
         strlcat(pathBuffer, DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME_H, pathBufferSize);
-        if ( dyld::my_stat(pathBuffer, &haswellStatBuf) == 0 )
-            return;
+        if ( dyld3::stat(pathBuffer, &haswellStatBuf) == 0 )
+            return true;
         // no haswell cache file, use regular x86_64 cache
         pathBuffer[len] = '\0';
     }
 #endif
 
+    struct stat statBuf;
+    strlcat(pathBuffer, DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME, pathBufferSize);
+    if ( dyld3::stat(pathBuffer, &statBuf) == 0 )
+        return true;
+
+    return false;
+}
+#endif // TARGET_OS_OSX
+
+static void getCachePath(const SharedCacheOptions& options, size_t pathBufferSize, char pathBuffer[])
+{
+#if TARGET_OS_OSX
+
+    if ( options.cacheDirOverride != nullptr ) {
+        getMacOSCachePath(pathBuffer, pathBufferSize, options.cacheDirOverride, options.useHaswell);
+    } else {
+        getMacOSCachePath(pathBuffer, pathBufferSize, MACOSX_MRM_DYLD_SHARED_CACHE_DIR, options.useHaswell);
+    }
+
+#else // TARGET_OS_OSX
+
+    // Non-macOS path
+    if ( options.cacheDirOverride != nullptr ) {
+        strlcpy(pathBuffer, options.cacheDirOverride, pathBufferSize);
+    } else {
+        strlcpy(pathBuffer, IPHONE_DYLD_SHARED_CACHE_DIR, sizeof(IPHONE_DYLD_SHARED_CACHE_DIR));
+    }
+
+    // append file component of cache file
+    if ( pathBuffer[strlen(pathBuffer)-1] != '/' )
+        strlcat(pathBuffer, "/", pathBufferSize);
+
     strlcat(pathBuffer, DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME, pathBufferSize);
 
-#if __IPHONE_OS_VERSION_MIN_REQUIRED && !TARGET_OS_SIMULATOR
+#if TARGET_OS_IPHONE && !TARGET_OS_SIMULATOR
     // use .development cache if it exists
-    struct stat enableStatBuf;
+    if ( BootArgs::forceCustomerCache() ) {
+        // The boot-arg always wins.  Use the customer cache if we are told to
+        return;
+    }
+    if ( !dyld3::internalInstall() ) {
+        // We can't use the development cache on customer installs
+        return;
+    }
+    if ( BootArgs::forceDevelopmentCache() ) {
+        // The boot-arg always wins.  Use the development cache if we are told to
+        strlcat(pathBuffer, DYLD_SHARED_CACHE_DEVELOPMENT_EXT, pathBufferSize);
+        return;
+    }
+
+    // If only one or the other caches exists, then use the one we have
     struct stat devCacheStatBuf;
     struct stat optCacheStatBuf;
-    bool developmentDevice = dyld3::internalInstall();
-    bool enableFileExists = (dyld::my_stat(IPHONE_DYLD_SHARED_CACHE_DIR "enable-dylibs-to-override-cache", &enableStatBuf) == 0);
-    bool devCacheExists = (dyld::my_stat(IPHONE_DYLD_SHARED_CACHE_DIR DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME DYLD_SHARED_CACHE_DEVELOPMENT_EXT, &devCacheStatBuf) == 0);
-    bool optCacheExists = (dyld::my_stat(IPHONE_DYLD_SHARED_CACHE_DIR DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME, &optCacheStatBuf) == 0);
-    if ( !BootArgs::forceCustomerCache() && developmentDevice && ((enableFileExists && (enableStatBuf.st_size < ENABLE_DYLIBS_TO_OVERRIDE_CACHE_SIZE) && devCacheExists) || !optCacheExists) )
+    bool devCacheExists = (dyld3::stat(IPHONE_DYLD_SHARED_CACHE_DIR DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME DYLD_SHARED_CACHE_DEVELOPMENT_EXT, &devCacheStatBuf) == 0);
+    bool optCacheExists = (dyld3::stat(IPHONE_DYLD_SHARED_CACHE_DIR DYLD_SHARED_CACHE_BASE_NAME ARCH_NAME, &optCacheStatBuf) == 0);
+    if ( !devCacheExists ) {
+        // If the dev cache doesn't exist, then use the customer cache
+        return;
+    }
+    if ( !optCacheExists ) {
+        // If the customer cache doesn't exist, then use the development cache
         strlcat(pathBuffer, DYLD_SHARED_CACHE_DEVELOPMENT_EXT, pathBufferSize);
+        return;
+    }
+
+    // Finally, check for the sentinels
+    struct stat enableStatBuf;
+    //struct stat sentinelStatBuf;
+    bool enableFileExists = (dyld3::stat(IPHONE_DYLD_SHARED_CACHE_DIR "enable-dylibs-to-override-cache", &enableStatBuf) == 0);
+    // FIXME: rdar://problem/59813537 Re-enable once automation is updated to use boot-arg
+    bool sentinelFileExists = false;
+    //bool sentinelFileExists = (dyld3::stat(MACOSX_MRM_DYLD_SHARED_CACHE_DIR "enable_development_mode", &sentinelStatBuf) == 0);
+    if ( enableFileExists && (enableStatBuf.st_size < ENABLE_DYLIBS_TO_OVERRIDE_CACHE_SIZE) ) {
+        // if the old enable file exists, use the development cache
+        strlcat(pathBuffer, DYLD_SHARED_CACHE_DEVELOPMENT_EXT, pathBufferSize);
+        return;
+    }
+    if ( sentinelFileExists ) {
+        // If the new sentinel exists, then use the development cache
+        strlcat(pathBuffer, DYLD_SHARED_CACHE_DEVELOPMENT_EXT, pathBufferSize);
+        return;
+    }
 #endif
 
+#endif //!TARGET_OS_OSX
 }
 
 
 int openSharedCacheFile(const SharedCacheOptions& options, SharedCacheLoadInfo* results)
 {
     getCachePath(options, sizeof(results->path), results->path);
-    return dyld::my_open(results->path, O_RDONLY, 0);
+    return dyld3::open(results->path, O_RDONLY, 0);
 }
 
 static bool validMagic(const SharedCacheOptions& options, const DyldSharedCache* cache)
@@ -258,15 +333,56 @@ static bool validPlatform(const SharedCacheOptions& options, const DyldSharedCac
 }
 
 #if !TARGET_OS_SIMULATOR
-static void verboseSharedCacheMappings(const shared_file_mapping_np mappings[3])
+static void verboseSharedCacheMappings(const shared_file_mapping_slide_np mappings[DyldSharedCache::MaxMappings],
+                                       uint32_t mappingsCount)
 {
-    for (int i=0; i < 3; ++i) {
-        dyld::log("        0x%08llX->0x%08llX init=%x, max=%x %s%s%s\n",
-            mappings[i].sfm_address, mappings[i].sfm_address+mappings[i].sfm_size-1,
-            mappings[i].sfm_init_prot, mappings[i].sfm_init_prot,
-            ((mappings[i].sfm_init_prot & VM_PROT_READ) ? "read " : ""),
-            ((mappings[i].sfm_init_prot & VM_PROT_WRITE) ? "write " : ""),
-            ((mappings[i].sfm_init_prot & VM_PROT_EXECUTE) ? "execute " : ""));
+    for (int i=0; i < mappingsCount; ++i) {
+        const char* mappingName = "";
+        if ( mappings[i].sms_max_prot & VM_PROT_WRITE ) {
+            if ( mappings[i].sms_max_prot & VM_PROT_NOAUTH ) {
+                // __DATA*
+                mappingName = "data";
+            } else {
+                // __AUTH*
+                mappingName = "auth";
+            }
+        }
+        uint32_t init_prot = mappings[i].sms_init_prot & (VM_PROT_READ | VM_PROT_WRITE | VM_PROT_EXECUTE);
+        uint32_t max_prot = mappings[i].sms_max_prot & (VM_PROT_READ | VM_PROT_WRITE | VM_PROT_EXECUTE);
+        dyld::log("        0x%08llX->0x%08llX init=%x, max=%x %s%s%s%s\n",
+            mappings[i].sms_address, mappings[i].sms_address+mappings[i].sms_size-1,
+            init_prot, max_prot,
+            ((mappings[i].sms_init_prot & VM_PROT_READ) ? "read " : ""),
+            ((mappings[i].sms_init_prot & VM_PROT_WRITE) ? "write " : ""),
+            ((mappings[i].sms_init_prot & VM_PROT_EXECUTE) ? "execute " : ""),
+            mappingName);
+    }
+}
+
+
+static void verboseSharedCacheMappingsToConsole(const shared_file_mapping_slide_np mappings[DyldSharedCache::MaxMappings],
+                                                uint32_t mappingsCount)
+{
+    for (int i=0; i < mappingsCount; ++i) {
+        const char* mappingName = "";
+        if ( mappings[i].sms_max_prot & VM_PROT_WRITE ) {
+            if ( mappings[i].sms_max_prot & VM_PROT_NOAUTH ) {
+                // __DATA*
+                mappingName = "data";
+            } else {
+                // __AUTH*
+                mappingName = "auth";
+            }
+        }
+        uint32_t init_prot = mappings[i].sms_init_prot & (VM_PROT_READ | VM_PROT_WRITE | VM_PROT_EXECUTE);
+        uint32_t max_prot = mappings[i].sms_max_prot & (VM_PROT_READ | VM_PROT_WRITE | VM_PROT_EXECUTE);
+        dyld::logToConsole("dyld: mapping 0x%08llX->0x%08llX init=%x, max=%x %s%s%s%s\n",
+                           mappings[i].sms_address, mappings[i].sms_address+mappings[i].sms_size-1,
+                           init_prot, max_prot,
+                           ((mappings[i].sms_init_prot & VM_PROT_READ) ? "read " : ""),
+                           ((mappings[i].sms_init_prot & VM_PROT_WRITE) ? "write " : ""),
+                           ((mappings[i].sms_init_prot & VM_PROT_EXECUTE) ? "execute " : ""),
+                           mappingName);
     }
 }
 #endif
@@ -282,7 +398,7 @@ static bool preflightCacheFile(const SharedCacheOptions& options, SharedCacheLoa
     }
 
     struct stat cacheStatBuf;
-    if ( dyld::my_stat(results->path, &cacheStatBuf) != 0 ) {
+    if ( dyld3::stat(results->path, &cacheStatBuf) != 0 ) {
         results->errorMessage = "shared cache file stat() failed";
         ::close(fd);
         return false;
@@ -307,40 +423,42 @@ static bool preflightCacheFile(const SharedCacheOptions& options, SharedCacheLoa
         ::close(fd);
         return false;
     }
-    if ( (cache->header.mappingCount != 3) || (cache->header.mappingOffset > 0x138) ) {
+    if ( (cache->header.mappingCount < 3) || (cache->header.mappingCount > DyldSharedCache::MaxMappings) || (cache->header.mappingOffset > 0x168) ) {
         results->errorMessage = "shared cache file mappings are invalid";
         ::close(fd);
         return false;
     }
     const dyld_cache_mapping_info* const fileMappings = (dyld_cache_mapping_info*)&firstPage[cache->header.mappingOffset];
-    if (  (fileMappings[0].fileOffset != 0)
-      || ((fileMappings[0].address + fileMappings[0].size) > fileMappings[1].address)
-      || ((fileMappings[1].address + fileMappings[1].size) > fileMappings[2].address)
-      || ((fileMappings[0].fileOffset + fileMappings[0].size) != fileMappings[1].fileOffset)
-      || ((fileMappings[1].fileOffset + fileMappings[1].size) != fileMappings[2].fileOffset)
+    const dyld_cache_mapping_info* textMapping = &fileMappings[0];
+    const dyld_cache_mapping_info* firstDataMapping = &fileMappings[1];
+    const dyld_cache_mapping_info* linkeditMapping = &fileMappings[cache->header.mappingCount - 1];
+    if (  (textMapping->fileOffset != 0)
+      || ((fileMappings[0].address + fileMappings[0].size) > firstDataMapping->address)
+      || ((fileMappings[0].fileOffset + fileMappings[0].size) != firstDataMapping->fileOffset)
       || ((cache->header.codeSignatureOffset + cache->header.codeSignatureSize) != cacheFileLength)
-      || (fileMappings[0].maxProt != (VM_PROT_READ|VM_PROT_EXECUTE))
-      || (fileMappings[1].maxProt != (VM_PROT_READ|VM_PROT_WRITE))
-      || (fileMappings[2].maxProt != VM_PROT_READ) ) {
-        results->errorMessage = "shared cache file mappings are invalid";
+      || (textMapping->maxProt != (VM_PROT_READ|VM_PROT_EXECUTE))
+      || (linkeditMapping->maxProt != VM_PROT_READ) ) {
+        results->errorMessage = "shared cache text/linkedit mappings are invalid";
         ::close(fd);
         return false;
     }
 
-    if ( cache->header.mappingOffset >= 0xF8 ) {
-        if ( (fileMappings[0].address != cache->header.sharedRegionStart) || ((fileMappings[2].address + fileMappings[2].size) > (cache->header.sharedRegionStart+cache->header.sharedRegionSize)) ) {
-            results->errorMessage = "shared cache file mapping addressses invalid";
+    // Check the __DATA mappings
+    for (unsigned i = 1; i != (cache->header.mappingCount - 1); ++i) {
+        if ( ((fileMappings[i].address + fileMappings[i].size) > fileMappings[i + 1].address)
+          || ((fileMappings[i].fileOffset + fileMappings[i].size) != fileMappings[i + 1].fileOffset)
+          || (fileMappings[i].maxProt != (VM_PROT_READ|VM_PROT_WRITE)) ) {
+            results->errorMessage = "shared cache data mappings are invalid";
             ::close(fd);
             return false;
         }
     }
-    else {
-        if ( (fileMappings[0].address != SHARED_REGION_BASE) || ((fileMappings[2].address + fileMappings[2].size) > (SHARED_REGION_BASE+SHARED_REGION_SIZE)) ) {
+
+    if ( (textMapping->address != cache->header.sharedRegionStart) || ((linkeditMapping->address + linkeditMapping->size) > (cache->header.sharedRegionStart+cache->header.sharedRegionSize)) ) {
         results->errorMessage = "shared cache file mapping addressses invalid";
         ::close(fd);
         return false;
     }
-    }
 
     // register code signature of cache file
     fsignatures_t siginfo;
@@ -375,28 +493,59 @@ static bool preflightCacheFile(const SharedCacheOptions& options, SharedCacheLoa
     ::munmap(mappedData, sizeof(firstPage));
 
     // fill out results
-    info->fd = fd;
-    for (int i=0; i < 3; ++i) {
-        info->mappings[i].sfm_address       = fileMappings[i].address;
-        info->mappings[i].sfm_size          = fileMappings[i].size;
-        info->mappings[i].sfm_file_offset   = fileMappings[i].fileOffset;
-        info->mappings[i].sfm_max_prot      = fileMappings[i].maxProt;
-        info->mappings[i].sfm_init_prot     = fileMappings[i].initProt;
+    info->mappingsCount = cache->header.mappingCount;
+    // We have to emit the mapping for the __LINKEDIT before the slid mappings
+    // This is so that the kernel has already mapped __LINKEDIT in to its address space
+    // for when it copies the slid info for each __DATA mapping
+    for (int i=0; i < cache->header.mappingCount; ++i) {
+        uint64_t    slideInfoFileOffset = 0;
+        uint64_t    slideInfoFileSize   = 0;
+        vm_prot_t   authProt            = 0;
+        vm_prot_t   initProt            = fileMappings[i].initProt;
+        if ( cache->header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset) ) {
+            // Old cache without the new slid mappings
+            if ( i == 1 ) {
+                // Add slide info to the __DATA mapping
+                slideInfoFileOffset = cache->header.slideInfoOffsetUnused;
+                slideInfoFileSize   = cache->header.slideInfoSizeUnused;
+                // Don't set auth prot to anything interseting on the old mapppings
+                authProt = 0;
+            }
+        } else {
+            // New cache where each mapping has a corresponding slid mapping
+            const dyld_cache_mapping_and_slide_info* slidableMappings = (const dyld_cache_mapping_and_slide_info*)&firstPage[cache->header.mappingWithSlideOffset];
+            slideInfoFileOffset = slidableMappings[i].slideInfoFileOffset;
+            slideInfoFileSize   = slidableMappings[i].slideInfoFileSize;
+            if ( (slidableMappings[i].flags & DYLD_CACHE_MAPPING_AUTH_DATA) == 0 )
+                authProt = VM_PROT_NOAUTH;
+            if ( (slidableMappings[i].flags & DYLD_CACHE_MAPPING_CONST_DATA) != 0 ) {
+                // The cache was built with __DATA_CONST being read-only.  We can override that
+                // with a boot-arg
+                if ( !gEnableSharedCacheDataConst )
+                    initProt |= VM_PROT_WRITE;
+            }
+        }
+
+        // Add a file for each mapping
+        info->fd                        = fd;
+        info->mappings[i].sms_address               = fileMappings[i].address;
+        info->mappings[i].sms_size                  = fileMappings[i].size;
+        info->mappings[i].sms_file_offset           = fileMappings[i].fileOffset;
+        info->mappings[i].sms_slide_size            = 0;
+        info->mappings[i].sms_slide_start           = 0;
+        info->mappings[i].sms_max_prot              = fileMappings[i].maxProt;
+        info->mappings[i].sms_init_prot             = initProt;
+        if ( slideInfoFileSize != 0 ) {
+            uint64_t offsetInLinkEditRegion = (slideInfoFileOffset - linkeditMapping->fileOffset);
+            info->mappings[i].sms_slide_start   = (user_addr_t)(linkeditMapping->address + offsetInLinkEditRegion);
+            info->mappings[i].sms_slide_size    = (user_addr_t)slideInfoFileSize;
+            info->mappings[i].sms_init_prot    |= (VM_PROT_SLIDE | authProt);
+            info->mappings[i].sms_max_prot     |= (VM_PROT_SLIDE | authProt);
+        }
     }
-    info->mappings[1].sfm_max_prot  |= VM_PROT_SLIDE;
-    info->mappings[1].sfm_init_prot |= VM_PROT_SLIDE;
-    info->slideInfoAddressUnslid  = fileMappings[2].address + cache->header.slideInfoOffset - fileMappings[2].fileOffset;
-    info->slideInfoSize           = (long)cache->header.slideInfoSize;
-    if ( cache->header.mappingOffset >= 0xf8 ) {
     info->sharedRegionStart = cache->header.sharedRegionStart;
     info->sharedRegionSize  = cache->header.sharedRegionSize;
     info->maxSlide          = cache->header.maxSlide;
-    }
-    else {
-        info->sharedRegionStart = SHARED_REGION_BASE;
-        info->sharedRegionSize  = SHARED_REGION_SIZE;
-        info->maxSlide          = SHARED_REGION_SIZE - (fileMappings[2].address + fileMappings[2].size - fileMappings[0].address);
-    }
     return true;
 }
 
@@ -404,14 +553,10 @@ static bool preflightCacheFile(const SharedCacheOptions& options, SharedCacheLoa
 #if !TARGET_OS_SIMULATOR
 
 // update all __DATA pages with slide info
-static bool rebaseDataPages(bool isVerbose, CacheInfo& info, SharedCacheLoadInfo* results)
+static bool rebaseDataPages(bool isVerbose, const dyld_cache_slide_info* slideInfo, const uint8_t *dataPagesStart,
+                            uint64_t sharedRegionStart, SharedCacheLoadInfo* results)
 {
-    uint64_t dataPagesStart = info.mappings[1].sfm_address;
-    const dyld_cache_slide_info* slideInfo = nullptr;
-    if ( info.slideInfoSize != 0 ) {
-        slideInfo = (dyld_cache_slide_info*)(info.slideInfoAddressUnslid + results->slide);
-    }
-    const dyld_cache_slide_info* slideInfoHeader = (dyld_cache_slide_info*)slideInfo;
+    const dyld_cache_slide_info* slideInfoHeader = slideInfo;
     if ( slideInfoHeader != nullptr ) {
         if ( slideInfoHeader->version == 2 ) {
             const dyld_cache_slide_info2* slideHeader = (dyld_cache_slide_info2*)slideInfo;
@@ -450,6 +595,7 @@ static bool rebaseDataPages(bool isVerbose, CacheInfo& info, SharedCacheLoadInfo
              for (int i=0; i < slideHeader->page_starts_count; ++i) {
                  uint8_t* page = (uint8_t*)(dataPagesStart + (pageSize*i));
                  uint64_t delta = slideHeader->page_starts[i];
+                 //dyld::log("page[%d]: page_starts[i]=0x%04X\n", i, delta);
                  if ( delta == DYLD_CACHE_SLIDE_V3_PAGE_ATTR_NO_REBASE )
                      continue;
                  delta = delta/sizeof(uint64_t); // initial offset is byte based
@@ -459,7 +605,7 @@ static bool rebaseDataPages(bool isVerbose, CacheInfo& info, SharedCacheLoadInfo
                      delta = loc->plain.offsetToNextPointer;
                      if ( loc->auth.authenticated ) {
 #if __has_feature(ptrauth_calls)
-                        uint64_t target = info.sharedRegionStart + loc->auth.offsetFromSharedCacheBase + results->slide;
+                        uint64_t target = sharedRegionStart + loc->auth.offsetFromSharedCacheBase + results->slide;
                         MachOLoaded::ChainedFixupPointerOnDisk ptr;
                         ptr.raw64 = *((uint64_t*)loc);
                         loc->raw = ptr.arm64e.signPointer(loc, target);
@@ -532,42 +678,66 @@ static bool reuseExistingCache(const SharedCacheOptions& options, SharedCacheLoa
             // we don't know the path this cache was previously loaded from, assume default
             getCachePath(options, sizeof(results->path), results->path);
             if ( options.verbose ) {
-                const shared_file_mapping_np* const mappings = (shared_file_mapping_np*)(cacheBaseAddress + existingCache->header.mappingOffset);
+                const dyld_cache_mapping_and_slide_info* const mappings = (const dyld_cache_mapping_and_slide_info*)(cacheBaseAddress + existingCache->header.mappingWithSlideOffset);
                 dyld::log("re-using existing shared cache (%s):\n", results->path);
-                shared_file_mapping_np slidMappings[3];
-                for (int i=0; i < 3; ++i) {
-                    slidMappings[i] = mappings[i];
-                    slidMappings[i].sfm_address += results->slide;
+                shared_file_mapping_slide_np slidMappings[DyldSharedCache::MaxMappings];
+                for (int i=0; i < DyldSharedCache::MaxMappings; ++i) {
+                    slidMappings[i].sms_address = mappings[i].address;
+                    slidMappings[i].sms_size = mappings[i].size;
+                    slidMappings[i].sms_file_offset = mappings[i].fileOffset;
+                    slidMappings[i].sms_max_prot = mappings[i].maxProt;
+                    slidMappings[i].sms_init_prot = mappings[i].initProt;
+                    slidMappings[i].sms_address += results->slide;
+                    if ( existingCache->header.mappingOffset > __offsetof(dyld_cache_header, mappingWithSlideOffset) ) {
+                        // New caches have slide info on each new mapping
+                        const dyld_cache_mapping_and_slide_info* const slidableMappings = (dyld_cache_mapping_and_slide_info*)(cacheBaseAddress + existingCache->header.mappingWithSlideOffset);
+                        assert(existingCache->header.mappingWithSlideCount <= DyldSharedCache::MaxMappings);
+                        if ( !(slidableMappings[i].flags & DYLD_CACHE_MAPPING_AUTH_DATA) ) {
+                            slidMappings[i].sms_max_prot  |= VM_PROT_NOAUTH;
+                            slidMappings[i].sms_init_prot |= VM_PROT_NOAUTH;
                         }
-                verboseSharedCacheMappings(slidMappings);
+                        if ( (slidableMappings[i].flags & DYLD_CACHE_MAPPING_CONST_DATA) != 0 ) {
+                            // The cache was built with __DATA_CONST being read-only.  We can override that
+                            // with a boot-arg
+                            if ( !gEnableSharedCacheDataConst )
+                                slidMappings[i].sms_init_prot |= VM_PROT_WRITE;
+                        }
+                    }
+                }
+                verboseSharedCacheMappings(slidMappings, existingCache->header.mappingCount);
             }
         }
         else {
             results->errorMessage = "existing shared cache in memory is not compatible";
         }
+
         return true;
     }
     return false;
 }
 
-static long pickCacheASLR(CacheInfo& info)
+static long pickCacheASLRSlide(CacheInfo& info)
 {
     // choose new random slide
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
+#if TARGET_OS_IPHONE || (TARGET_OS_OSX && TARGET_CPU_ARM64)
     // <rdar://problem/20848977> change shared cache slide for 32-bit arm to always be 16k aligned
-    long slide = ((arc4random() % info.maxSlide) & (-16384));
-#else
-    long slide = ((arc4random() % info.maxSlide) & (-4096));
-#endif
-
-    // <rdar://problem/32031197> respect -disable_aslr boot-arg
-    if ( BootArgs::contains("-disable_aslr") )
+    long slide;
+    if (info.maxSlide == 0)
         slide = 0;
-
-    // update mappings
-    for (uint32_t i=0; i < 3; ++i) {
-        info.mappings[i].sfm_address += slide;
+    else
+        slide = ((arc4random() % info.maxSlide) & (-16384));
+#else
+    long slide;
+    if (info.maxSlide == 0)
+        slide = 0;
+    else
+        slide = ((arc4random() % info.maxSlide) & (-4096));
+#if defined(__x86_64__) && !TARGET_OS_SIMULATOR
+    if (dyld::isTranslated()) {
+        slide &= (-16384);
     }
+#endif
+#endif
     
     return slide;
 }
@@ -578,16 +748,68 @@ static bool mapCacheSystemWide(const SharedCacheOptions& options, SharedCacheLoa
     if ( !preflightCacheFile(options, results, &info) )
         return false;
 
-    const dyld_cache_slide_info2* slideInfo = nullptr;
-    if ( info.slideInfoSize != 0 ) {
-        results->slide = pickCacheASLR(info);
-        slideInfo = (dyld_cache_slide_info2*)(info.slideInfoAddressUnslid + results->slide);
+    int result = 0;
+    if ( info.mappingsCount != 3 ) {
+        uint32_t maxSlide = options.disableASLR ? 0 : (uint32_t)info.maxSlide;
+
+        shared_file_np file;
+        file.sf_fd = info.fd;
+        file.sf_mappings_count = info.mappingsCount;
+        // For the new syscall, this is actually the max slide.  The kernel now owns the actual slide
+        file.sf_slide = maxSlide;
+        result = __shared_region_map_and_slide_2_np(1, &file, info.mappingsCount, info.mappings);
+    } else {
+        // With the old syscall, dyld has to choose the slide
+        results->slide = options.disableASLR ? 0 : pickCacheASLRSlide(info);
+
+        // update mappings based on the slide we choose
+        for (uint32_t i=0; i < info.mappingsCount; ++i) {
+            info.mappings[i].sms_address += results->slide;
+            if ( info.mappings[i].sms_slide_size != 0 )
+                info.mappings[i].sms_slide_start += (uint32_t)results->slide;
+        }
+
+        // If we get here then we don't have the new kernel function, so use the old one
+        const dyld_cache_slide_info2*   slideInfo       = nullptr;
+        size_t                          slideInfoSize   = 0;
+        shared_file_mapping_np mappings[3];
+        for (unsigned i = 0; i != 3; ++i) {
+            mappings[i].sfm_address         = info.mappings[i].sms_address;
+            mappings[i].sfm_size            = info.mappings[i].sms_size;
+            mappings[i].sfm_file_offset     = info.mappings[i].sms_file_offset;
+            mappings[i].sfm_max_prot        = info.mappings[i].sms_max_prot;
+            mappings[i].sfm_init_prot       = info.mappings[i].sms_init_prot;
+            if ( info.mappings[i].sms_slide_size != 0 ) {
+                slideInfo       = (dyld_cache_slide_info2*)info.mappings[i].sms_slide_start;
+                slideInfoSize   = (size_t)info.mappings[i].sms_slide_size;
+            }
+        }
+        result = __shared_region_map_and_slide_np(info.fd, 3, mappings, results->slide, slideInfo, slideInfoSize);
     }
 
-    int result = __shared_region_map_and_slide_np(info.fd, 3, info.mappings, results->slide, slideInfo, info.slideInfoSize);
     ::close(info.fd);
     if ( result == 0 ) {
-        results->loadAddress = (const DyldSharedCache*)(info.mappings[0].sfm_address);
+        results->loadAddress = (const DyldSharedCache*)(info.mappings[0].sms_address);
+        if ( info.mappingsCount != 3 ) {
+            // We don't know our own slide any more as the kernel owns it, so ask for it again now
+            if ( reuseExistingCache(options, results) ) {
+
+                // update mappings based on the slide the kernel chose
+                for (uint32_t i=0; i < info.mappingsCount; ++i) {
+                    info.mappings[i].sms_address += results->slide;
+                    if ( info.mappings[i].sms_slide_size != 0 )
+                        info.mappings[i].sms_slide_start += (uint32_t)results->slide;
+                }
+
+                if ( options.verbose )
+                    verboseSharedCacheMappingsToConsole(info.mappings, info.mappingsCount);
+                return true;
+            }
+            // Uh oh, we mapped the kernel, but we didn't find the slide
+            if ( options.verbose )
+                dyld::logToConsole("dyld: error finding shared cache slide for system wide mapping\n");
+            return false;
+        }
     }
     else {
         // could be another process beat us to it
@@ -601,7 +823,7 @@ static bool mapCacheSystemWide(const SharedCacheOptions& options, SharedCacheLoa
 
     if ( options.verbose ) {
         dyld::log("mapped dyld cache file system wide: %s\n", results->path);
-        verboseSharedCacheMappings(info.mappings);
+        verboseSharedCacheMappings(info.mappings, info.mappingsCount);
     }
     return true;
 }
@@ -616,12 +838,18 @@ static bool mapCachePrivate(const SharedCacheOptions& options, SharedCacheLoadIn
 
     // compute ALSR slide
     results->slide = 0;
-#if !TARGET_OS_SIMULATOR // simulator caches do not support sliding
-    if ( info.slideInfoSize != 0 ) {
-        results->slide = pickCacheASLR(info);
-    }
+#if !TARGET_OS_SIMULATOR
+    results->slide = options.disableASLR ? 0 : pickCacheASLRSlide(info);
 #endif
-    results->loadAddress = (const DyldSharedCache*)(info.mappings[0].sfm_address);
+
+    // update mappings
+    for (uint32_t i=0; i < info.mappingsCount; ++i) {
+        info.mappings[i].sms_address += (uint32_t)results->slide;
+        if ( info.mappings[i].sms_slide_size != 0 )
+            info.mappings[i].sms_slide_start += (uint32_t)results->slide;
+    }
+
+    results->loadAddress = (const DyldSharedCache*)(info.mappings[0].sms_address);
 
     // deallocate any existing system wide shared cache
     deallocateExistingSharedCache();
@@ -634,18 +862,18 @@ static bool mapCachePrivate(const SharedCacheOptions& options, SharedCacheLoadIn
 #endif
 
     // map cache just for this process with mmap()
-    for (int i=0; i < 3; ++i) {
-        void* mmapAddress = (void*)(uintptr_t)(info.mappings[i].sfm_address);
-        size_t size = (size_t)(info.mappings[i].sfm_size);
+    for (int i=0; i < info.mappingsCount; ++i) {
+        void* mmapAddress = (void*)(uintptr_t)(info.mappings[i].sms_address);
+        size_t size = (size_t)(info.mappings[i].sms_size);
         //dyld::log("dyld: mapping address %p with size 0x%08lX\n", mmapAddress, size);
         int protection = 0;
-        if ( info.mappings[i].sfm_init_prot & VM_PROT_EXECUTE )
+        if ( info.mappings[i].sms_init_prot & VM_PROT_EXECUTE )
             protection   |= PROT_EXEC;
-        if ( info.mappings[i].sfm_init_prot & VM_PROT_READ )
+        if ( info.mappings[i].sms_init_prot & VM_PROT_READ )
             protection   |= PROT_READ;
-        if ( info.mappings[i].sfm_init_prot & VM_PROT_WRITE )
+        if ( info.mappings[i].sms_init_prot & VM_PROT_WRITE )
             protection   |= PROT_WRITE;
-        off_t offset = info.mappings[i].sfm_file_offset;
+        off_t offset = info.mappings[i].sms_file_offset;
         if ( ::mmap(mmapAddress, size, protection, MAP_FIXED | MAP_PRIVATE, info.fd, offset) != mmapAddress ) {
             // failed to map some chunk of this shared cache file
             // clear shared region
@@ -662,11 +890,22 @@ static bool mapCachePrivate(const SharedCacheOptions& options, SharedCacheLoadIn
 #if TARGET_OS_SIMULATOR // simulator caches do not support sliding
     return true;
 #else
-    bool success = rebaseDataPages(options.verbose, info, results);
+
+    // Change __DATA_CONST to read-write for this block
+    DyldSharedCache::DataConstScopedWriter patcher(results->loadAddress, mach_task_self(), options.verbose ? &dyld::log : nullptr);
+
+    __block bool success = true;
+    for (int i=0; i < info.mappingsCount; ++i) {
+        if ( info.mappings[i].sms_slide_size == 0 )
+            continue;
+        const dyld_cache_slide_info* slideInfoHeader = (const dyld_cache_slide_info*)info.mappings[i].sms_slide_start;
+        const uint8_t* mappingPagesStart = (const uint8_t*)info.mappings[i].sms_address;
+        success &= rebaseDataPages(options.verbose, slideInfoHeader, mappingPagesStart, info.sharedRegionStart, results);
+    }
 
     if ( options.verbose ) {
         dyld::log("mapped dyld cache file private to process (%s):\n", results->path);
-        verboseSharedCacheMappings(info.mappings);
+        verboseSharedCacheMappings(info.mappings, info.mappingsCount);
     }
     return success;
 #endif
@@ -710,7 +949,7 @@ bool findInSharedCacheImage(const SharedCacheLoadInfo& loadInfo, const char* dyl
 
     if ( loadInfo.loadAddress->header.formatVersion != dyld3::closure::kFormatVersion ) {
         // support for older cache with a different Image* format
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
+#if TARGET_OS_IPHONE
         uint64_t hash = 0;
         for (const char* s=dylibPathToFind; *s != '\0'; ++s)
                 hash += hash*4 + *s;
@@ -718,7 +957,7 @@ bool findInSharedCacheImage(const SharedCacheLoadInfo& loadInfo, const char* dyl
         const dyld_cache_image_info* const start = (dyld_cache_image_info*)((uint8_t*)loadInfo.loadAddress + loadInfo.loadAddress->header.imagesOffset);
         const dyld_cache_image_info* const end = &start[loadInfo.loadAddress->header.imagesCount];
         for (const dyld_cache_image_info* p = start; p != end; ++p) {
-#if __IPHONE_OS_VERSION_MIN_REQUIRED
+#if TARGET_OS_IPHONE
             // on iOS, inode is used to hold hash of path
             if ( (p->modTime == 0) && (p->inode != hash) )
                 continue;
@@ -741,34 +980,7 @@ bool findInSharedCacheImage(const SharedCacheLoadInfo& loadInfo, const char* dyl
     if ( loadInfo.loadAddress->hasImagePath(dylibPathToFind, imageIndex) ) {
         results->image = images->imageForNum(imageIndex+1);
     }
- #if __MAC_OS_X_VERSION_MIN_REQUIRED
-    else {
-        // <rdar://problem/32740215> handle symlink to cached dylib
-        if ( loadInfo.loadAddress->header.dylibsExpectedOnDisk ) {
-            struct stat statBuf;
-            if ( dyld::my_stat(dylibPathToFind, &statBuf) == 0 ) {
-                // on macOS we store the inode and mtime of each dylib in the cache in the dyld_cache_image_info array
-                const dyld_cache_image_info* const start = (dyld_cache_image_info*)((uint8_t*)loadInfo.loadAddress + loadInfo.loadAddress->header.imagesOffset);
-                const dyld_cache_image_info* const end = &start[loadInfo.loadAddress->header.imagesCount];
-                for (const dyld_cache_image_info* p = start; p != end; ++p) {
-                    if ( (p->inode == statBuf.st_ino) && (p->modTime == statBuf.st_mtime) ) {
-                        imageIndex = (uint32_t)(p - start);
-                        results->image = images->imageForNum(imageIndex+1);
-                        break;
-                    }
-                }
-            }
-        }
-        else {
-            char resolvedPath[PATH_MAX];
-            if ( realpath(dylibPathToFind, resolvedPath) != nullptr ) {
-                if ( loadInfo.loadAddress->hasImagePath(resolvedPath, imageIndex) ) {
-                    results->image = images->imageForNum(imageIndex+1);
-                }
-            }
-        }
-    }
-#endif
+
     if ( results->image == nullptr )
         return false;
 
@@ -793,7 +1005,7 @@ void deallocateExistingSharedCache()
 #if TARGET_OS_SIMULATOR
     // dyld deallocated macOS shared cache before jumping into dyld_sim
 #else
-    // <rdar://problem/5077374> remove the shared region sub-map
+    // <rdar://problem/50773474> remove the shared region sub-map
     uint64_t existingCacheAddress = 0;
     if ( __shared_region_check_np(&existingCacheAddress) == 0 ) {
         ::mmap((void*)((long)SHARED_REGION_BASE), SHARED_REGION_SIZE, PROT_NONE, MAP_FIXED | MAP_PRIVATE| MAP_ANON, 0, 0);

dyld3/SharedCacheRuntime.h

@@ -30,6 +30,7 @@
 #include <stdint.h>
 
 #include "DyldSharedCache.h"
+#include "PointerAuth.h"
 
 namespace dyld3 {
 
@@ -38,10 +39,12 @@ struct SharedCacheOptions {
     bool            forcePrivate;
     bool            useHaswell;
     bool            verbose;
+    bool            disableASLR;
 };
 
 struct SharedCacheLoadInfo {
-    const DyldSharedCache*       loadAddress;
+    typedef const DyldSharedCache* __ptrauth_dyld_address_auth DyldCachePtrType;
+    DyldCachePtrType             loadAddress;
     long                         slide;
     const char*                  errorMessage;
     char                         path[256];

dyld3/SupportedArchs.h

@@ -25,6 +25,8 @@
 #ifndef _DYLD_SUPPORTED_ARCHS_H_
 #define _DYLD_SUPPORTED_ARCHS_H_
 
+#include <TargetConditionals.h>
+
 
 
 #endif // _DYLD_SUPPORTED_ARCHS_H_

dyld3/Tracing.cpp

@@ -28,12 +28,9 @@
 #include <mach/mach.h>
 #include <kern/kcdata.h>
 #include <mach-o/dyld_priv.h>
+
 #include "Loading.h"
 #include "Tracing.h"
-#ifdef DARLING
-#define kdebug_trace(...)
-#define kdebug_is_enabled(...) false
-#endif
 
 // Workaround for header issues in rdar://49073930
 // #include <System/os/reason_private.h>
@@ -52,7 +49,7 @@ void kdebug_trace_dyld_image(const uint32_t code,
                        const fsid_t fsid,
                        const mach_header* load_addr)
 {
-    uint64_t id = kdebug_trace_string(code, 0, imagePath);
+    uint64_t id = kdebug_trace_string(KDBG_CODE(DBG_DYLD, DBG_DYLD_UUID, code), 0, imagePath);
 #if __ARM_ARCH_7K__
     uint32_t *uuid = (uint32_t *)uuid_bytes;
     kdebug_trace(KDBG_CODE(DBG_DYLD, DBG_DYLD_UUID, code + 2), uuid[0],
@@ -72,7 +69,7 @@ void kdebug_trace_dyld_image(const uint32_t code,
                  ((uint64_t)fsobjid.fid_generation << 32),
                  id, 0, 0);
 #endif /* !__ARM_ARCH_7K__ */
-    kdebug_trace_string(code, id, nullptr);
+    kdebug_trace_string(KDBG_CODE(DBG_DYLD, DBG_DYLD_UUID, code), id, nullptr);
 }
 
 // FIXME

dyld3/Tracing.h

@@ -35,10 +35,6 @@
 #include <System/sys/kdebug.h>
 #include <System/sys/reason.h>
 
-#ifdef DARLING
-#define kdebug_trace_string(...) ((uint64_t)-1)
-#endif
-
 
 #define DBG_DYLD_INTERNAL_SUBCLASS              (7)
 #define DBG_DYLD_API_SUBCLASS                   (8)
@@ -69,7 +65,8 @@
 #define DBG_DYLD_DEBUGGING_VM_UNMAP             (KDBG_CODE(DBG_DYLD, DBG_DYLD_DEBUGGING_SUBCLASS, 1))
 #define DBG_DYLD_DEBUGGING_MAP_LOOP             (KDBG_CODE(DBG_DYLD, DBG_DYLD_DEBUGGING_SUBCLASS, 2))
 #define DBG_DYLD_DEBUGGING_MARK                 (KDBG_CODE(DBG_DYLD, DBG_DYLD_DEBUGGING_SUBCLASS, 3))
-
+#define DBG_DYLD_TASK_NOTIFY_REGISTER           (KDBG_CODE(DBG_DYLD, DBG_DYLD_DEBUGGING_SUBCLASS, 4))
+#define DBG_DYLD_TASK_NOTIFY_DEREGISTER         (KDBG_CODE(DBG_DYLD, DBG_DYLD_DEBUGGING_SUBCLASS, 5))
 
 #define VIS_HIDDEN __attribute__((visibility("hidden")))
 

dyld3/libdyldEntryVector.cpp

@@ -38,7 +38,8 @@ extern "C" char start;
 
 VIS_HIDDEN const char** appleParams;
 
-extern bool gUseDyld3;
+extern void* __ptrauth_dyld_address_auth gUseDyld3;
+extern bool gEnableSharedCacheDataConst;
 
 namespace dyld3 {
 
@@ -57,7 +58,8 @@ static const char* leafName(const char* argv0)
         return argv0;
 }
 
-static void entry_setVars(const mach_header* mainMH, int argc, const char* argv[], const char* envp[], const char* apple[])
+static void entry_setVars(const mach_header* mainMH, int argc, const char* argv[], const char* envp[], const char* apple[],
+                          bool keysOff, bool platformBinariesOnly, bool enableSharedCacheDataConst)
 {
     NXArgc       = argc;
     NXArgv       = argv;
@@ -70,11 +72,13 @@ static void entry_setVars(const mach_header* mainMH, int argc, const char* argv[
     sVars.NXArgvPtr     = &NXArgv;
     sVars.environPtr    = (const char***)&environ;
     sVars.__prognamePtr = &__progname;
-    gAllImages.setProgramVars(&sVars);
+    gAllImages.setProgramVars(&sVars, keysOff, platformBinariesOnly);
 
-    gUseDyld3 = true;
+    gUseDyld3 = (void*)1;
 
     setLoggingFromEnvs(envp);
+
+    gEnableSharedCacheDataConst = enableSharedCacheDataConst;
 }
 
 static void entry_setHaltFunction(void (*func)(const char* message) __attribute__((noreturn)) )
@@ -108,10 +112,11 @@ static void entry_setNotifyMonitoringDyld(void (*notifyMonitoringDyld)(bool unlo
 
 static void entry_setInitialImageList(const closure::LaunchClosure* closure,
                                       const DyldSharedCache* dyldCacheLoadAddress, const char* dyldCachePath,
-                                const Array<LoadedImage>& initialImages, LoadedImage& libSystem)
+                                      const Array<LoadedImage>& initialImages, LoadedImage& libSystem,
+                                      mach_port_t mach_task_self)
 {
     gAllImages.init(closure, dyldCacheLoadAddress, dyldCachePath, initialImages);
-    gAllImages.applyInterposingToDyldCache(closure);
+    gAllImages.applyInterposingToDyldCache(closure, mach_task_self);
 
     // run initializer for libSytem.B.dylib
     // this calls back into _dyld_initializer which calls gAllIimages.addImages()
@@ -147,6 +152,24 @@ static void entry_setHasCacheOverrides(bool someCacheImageOverriden)
     gAllImages.setHasCacheOverrides(someCacheImageOverriden);
 }
 
+
+static void entry_setProgramVars(ProgramVars* progVars)
+{
+    // this entry only called when running crt1.o based old macOS programs
+    gAllImages.setProgramVars((AllImages::ProgramVars*)progVars, false, false);
+}
+
+static void entry_setLaunchMode(uint32_t flags)
+{
+    gAllImages.setLaunchMode(flags);
+}
+
+static MainFunc entry_getDriverkitMain(void)
+{
+    return gAllImages.getDriverkitMain();
+}
+
+
 static_assert((closure::kFormatVersion & LibDyldEntryVector::kBinaryFormatVersionMask) == closure::kFormatVersion, "binary format version overflow");
 
 const LibDyldEntryVector entryVectorForDyld = {
@@ -163,7 +186,10 @@ const LibDyldEntryVector entryVectorForDyld = {
     &entry_setRestrictions,
     &entry_setNotifyMonitoringDyldMain,
     &entry_setNotifyMonitoringDyld,
-    &entry_setHasCacheOverrides
+    &entry_setHasCacheOverrides,
+    &entry_setProgramVars,
+    &entry_setLaunchMode,
+    &entry_getDriverkitMain,
 };
 
 VIS_HIDDEN void _dyld_atfork_prepare()

dyld3/libdyldEntryVector.h

@@ -27,30 +27,35 @@
 #define __DYLD_ENTRY_VECTOR_H__
 
 #include <mach-o/loader.h>
+#include <sys/types.h>
 #include <Availability.h>
 
 #include "Loading.h"
 
 struct dyld_all_image_infos;
 class DyldSharedCache;
+struct ProgramVars;
 
 namespace dyld3 {
 
+typedef void  (*MainFunc)(void);
 
 struct LibDyldEntryVector
 {
-    enum { kCurrentVectorVersion = 7 };
+    enum { kCurrentVectorVersion = 10 };
     // The 32-bit caches steal bits to make rebase chains, so use 32-bits for the binary format version storage, but mask only some to actually use
     enum { kBinaryFormatVersionMask = 0x00FFFFFF };
 
     uint32_t    vectorVersion;              // should be kCurrentVectorVersion
     uint32_t    binaryFormatVersion;        // should be dyld3::closure::kFormatVersion
-    void        (*setVars)(const mach_header* mainMH, int argc, const char* argv[], const char* envp[], const char* apple[]);
+    void        (*setVars)(const mach_header* mainMH, int argc, const char* argv[], const char* envp[], const char* apple[],
+                           bool keysOff, bool platformBinariesOnly, bool enableSharedCacheDataConst);
     void        (*setHaltFunction)(void (*func)(const char* message) __attribute__((noreturn)) );
     void        (*setOldAllImageInfo)(dyld_all_image_infos*);
     void        (*setInitialImageList)(const closure::LaunchClosure* closure,
                                        const DyldSharedCache* dyldCacheLoadAddress, const char* dyldCachePath,
-                                        const Array<LoadedImage>& initialImages, LoadedImage& libSystem);
+                                       const Array<LoadedImage>& initialImages, LoadedImage& libSystem,
+                                       mach_port_t mach_task_self);
     void        (*runInitialzersBottomUp)(const mach_header* topImageLoadAddress);
     void        (*startFunc)();
     // added in version 3
@@ -66,6 +71,15 @@ struct LibDyldEntryVector
                                                                             const char* imagePaths[]));
     // added in version 7
     void        (*setHasCacheOverrides)(bool someCacheImageOverriden);
+
+    // added in version 8
+    void        (*setProgramVars)(struct ProgramVars* progVars);
+    
+    // added in version 9
+    void        (*setLaunchMode)(uint32_t flags);
+
+    // added in version 10
+    MainFunc    (*getDriverkitMain)(void);
 };
 
 extern const LibDyldEntryVector entryVectorForDyld;

dyld3/shared-cache/AppCacheBuilder.h

@@ -0,0 +1,204 @@
+/*
+ * Copyright (c) 2017 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+#ifndef AppCacheBuilder_h
+#define AppCacheBuilder_h
+
+#include "CacheBuilder.h"
+#include "MachOFileAbstraction.hpp"
+#include "MachOAppCache.h"
+
+#include <list>
+
+#include <CoreFoundation/CFDictionary.h>
+
+class AppCacheBuilder final : public CacheBuilder {
+public:
+    struct Options {
+        enum class AppCacheKind {
+            none,
+            kernel,                 // The base first party kernel collection with xnu and boot time kexts
+            pageableKC,              // Other first party kexts which are usually only for some HW, eg, different GPUs
+            kernelCollectionLevel2, // Placeholder until we find a use for this
+            auxKC,                  // Third party kexts, or Apple kexts updated out of band with the OS, if any
+        };
+
+        enum class StripMode {
+            none,           // Don't strip anything
+            all,            // Strip everything
+            allExceptKernel // Strip everything other than the static kernel
+        };
+
+        AppCacheKind    cacheKind = AppCacheKind::none;
+        StripMode       stripMode = StripMode::none;
+    };
+    AppCacheBuilder(const DyldSharedCache::CreateOptions& dyldCacheOptions, const Options& appCacheOptions,
+                    const dyld3::closure::FileSystem& fileSystem);
+    ~AppCacheBuilder() override final;
+
+    // Wraps up a loaded macho with the other information required by kext's, eg, the list of dependencies
+    struct InputDylib {
+        LoadedMachO                     dylib;
+        std::string                     dylibID;
+        std::vector<std::string>        dylibDeps;
+        CFDictionaryRef                 infoPlist   = nullptr;
+        std::string                     bundlePath;
+        Diagnostics*                    errors          = nullptr;
+        CacheBuilder::DylibStripMode    stripMode = CacheBuilder::DylibStripMode::stripNone;
+    };
+
+    // The payload for -sectcreate
+    struct CustomSegment {
+        struct CustomSection {
+            std::string                     sectionName;
+            std::vector<uint8_t>            data;
+            uint64_t                        offsetInRegion = 0;
+        };
+
+        std::string                 segmentName;
+        std::vector<CustomSection>  sections;
+        Region*                     parentRegion = nullptr;
+    };
+
+    bool                    addCustomSection(const std::string& segmentName,
+                                             CustomSegment::CustomSection section);
+    void                    setExistingKernelCollection(const dyld3::MachOAppCache* appCacheMA);
+    void                    setExistingPageableKernelCollection(const dyld3::MachOAppCache* appCacheMA);
+    void                    setExtraPrelinkInfo(CFDictionaryRef dictionary);
+    void                    buildAppCache(const std::vector<InputDylib>& dylibs);
+    void                    writeFile(const std::string& path);
+    void                    writeBuffer(uint8_t*& buffer, uint64_t& bufferSize) const;
+
+private:
+
+    enum {
+        // The fixup format can't support more than 3 levels right now
+        numFixupLevels = 4
+    };
+
+    struct AppCacheDylibInfo : CacheBuilder::DylibInfo
+    {
+        // From CacheBuilder::DylibInfo
+        // const LoadedMachO*              input;
+        // std::string                     dylibID;
+        // std::vector<SegmentMappingInfo> cacheLocation;
+
+        DylibStripMode                  stripMode       = DylibStripMode::stripNone;
+        std::vector<std::string>        dependencies;
+        CFDictionaryRef                 infoPlist       = nullptr;
+        Diagnostics*                    errors          = nullptr;
+        std::string                     bundlePath;
+    };
+
+    static_assert(std::is_move_constructible<AppCacheDylibInfo>::value);
+
+    void                    forEachDylibInfo(void (^callback)(const DylibInfo& dylib, Diagnostics& dylibDiag)) override final;
+
+    void                    forEachCacheDylib(void (^callback)(const dyld3::MachOAnalyzer* ma,
+                                                               const std::string& dylibID,
+                                                               DylibStripMode stripMode,
+                                                               const std::vector<std::string>& dependencies,
+                                                               Diagnostics& dylibDiag,
+                                                               bool& stop)) const;
+
+    const DylibInfo*                    getKernelStaticExecutableInputFile() const;
+    const dyld3::MachOAnalyzer*         getKernelStaticExecutableFromCache() const;
+
+    void                                makeSortedDylibs(const std::vector<InputDylib>& dylibs);
+    void                                allocateBuffer();
+    void                                assignSegmentRegionsAndOffsets();
+    void                                copyRawSegments();
+    void                                assignSegmentAddresses();
+    void                                generateCacheHeader();
+    void                                generatePrelinkInfo();
+    uint32_t                            getCurrentFixupLevel() const;
+    void                                processFixups();
+    void                                writeFixups();
+    void                                fipsSign();
+    void                                generateUUID();
+    uint64_t                            numRegions() const;
+    uint64_t                            numBranchRelocationTargets();
+    uint64_t                            fixupsPageSize() const;
+    uint64_t                            numWritablePagesToFixup(uint64_t numBytesToFixup) const;
+    bool                                fixupsArePerKext() const;
+    void                                forEachRegion(void (^callback)(const Region& region)) const;
+
+    Options                             appCacheOptions;
+    const dyld3::MachOAppCache*         existingKernelCollection = nullptr;
+    const dyld3::MachOAppCache*         pageableKernelCollection = nullptr;
+    CFDictionaryRef                     extraPrelinkInfo         = nullptr;
+
+    std::vector<AppCacheDylibInfo>      sortedDylibs;
+    std::vector<InputDylib>             codelessKexts;
+    std::vector<CustomSegment>          customSegments;
+    Region                              cacheHeaderRegion;
+    Region                              readExecuteRegion;
+    Region                              branchStubsRegion;
+    Region                              dataConstRegion;
+    Region                              branchGOTsRegion;
+    Region                              readWriteRegion;
+    Region                              hibernateRegion;
+    Region                              readOnlyTextRegion;
+    std::list<Region>                   customDataRegions;       // -sectcreate
+    Region                              prelinkInfoRegion;
+    std::list<Region>                   nonSplitSegRegions;
+    // Region                           _readOnlyRegion;        // This comes from the base class
+    Region                              fixupsSubRegion;        // This will be in the __LINKEDIT when we write the file
+
+    // This is the base address from the statically linked kernel, or 0 for other caches
+    uint64_t                            cacheBaseAddress = 0;
+    // For x86_64 only, we want to keep the address of the hibernate segment from xnu
+    // We'll ensure this is mapped lower than the cache base address
+    uint64_t                            hibernateAddress = 0;
+
+    uint16_t                            chainedPointerFormat = 0;
+
+    // The dictionary we ultimately store in the __PRELINK_INFO region
+    CFMutableDictionaryRef              prelinkInfoDict = nullptr;
+
+    // Cache header fields
+    // FIXME: 32-bit
+
+    struct CacheHeader64 {
+        typedef std::pair<segment_command_64*, Region*> SegmentCommandAndRegion;
+        typedef std::pair<fileset_entry_command*, const DylibInfo*> DylibCommandAndInfo;
+
+        mach_header_64*                         header              = nullptr;
+
+        uint64_t                                numLoadCommands     = 0;
+        uint64_t                                loadCommandsSize    = 0;
+
+        uuid_command*                           uuid                = nullptr;
+        build_version_command*                  buildVersion        = nullptr;
+        thread_command*                         unixThread          = nullptr;
+        symtab_command*                         symbolTable         = nullptr;
+        dysymtab_command*                       dynSymbolTable      = nullptr;
+        linkedit_data_command*                  chainedFixups       = nullptr;
+        std::vector<SegmentCommandAndRegion>    segments;
+        std::vector<DylibCommandAndInfo>        dylibs;
+    };
+
+    CacheHeader64 cacheHeader;
+};
+
+#endif /* AppCacheBuilder_h */

dyld3/shared-cache/BuilderUtils.h

@@ -1,33 +0,0 @@
-/*
- * Copyright (c) 2017 Apple Inc. All rights reserved.
- *
- * @APPLE_LICENSE_HEADER_START@
- *
- * This file contains Original Code and/or Modifications of Original Code
- * as defined in and that are subject to the Apple Public Source License
- * Version 2.0 (the 'License'). You may not use this file except in
- * compliance with the License. Please obtain a copy of the License at
- * http://www.opensource.apple.com/apsl/ and read it before using this
- * file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
- * Please see the License for the specific language governing rights and
- * limitations under the License.
- *
- * @APPLE_LICENSE_HEADER_END@
- */
-
-
-#ifndef BuilderUtils_h
-#define BuilderUtils_h
-
-dispatch_group_t buildGroup();
-void makeBoms(dyld3::Manifest& manifest, const std::string& masterDstRoot);
-bool build(Diagnostics& diags, dyld3::Manifest& manifest, const std::string& masterDstRoot, bool dedupe, bool verbose, bool skipWrites, bool agileChooseSHA256CdHash,
-           bool emitDevCaches, bool isLocallyBuiltCache);
-
-#endif /* BuilderUtils_h */

dyld3/shared-cache/BuilderUtils.mm

@@ -1,317 +0,0 @@
-/*
- * Copyright (c) 2017 Apple Inc. All rights reserved.
- *
- * @APPLE_LICENSE_HEADER_START@
- *
- * This file contains Original Code and/or Modifications of Original Code
- * as defined in and that are subject to the Apple Public Source License
- * Version 2.0 (the 'License'). You may not use this file except in
- * compliance with the License. Please obtain a copy of the License at
- * http://www.opensource.apple.com/apsl/ and read it before using this
- * file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
- * Please see the License for the specific language governing rights and
- * limitations under the License.
- *
- * @APPLE_LICENSE_HEADER_END@
- */
-
-
-#include <set>
-#include <array>
-#include <string>
-#include <sstream>
-#include <iomanip> // std::setfill, std::setw
-#include <pthread.h>
-#include <mach/mach.h>
-#include <sys/types.h>
-#include <sys/sysctl.h>
-#include <dispatch/dispatch.h>
-
-#include <Bom/Bom.h>
-#include <Security/Security.h>
-#include <Security/SecCodeSigner.h>
-#include <CommonCrypto/CommonCrypto.h>
-
-#include "Manifest.h"
-#include "Diagnostics.h"
-#include "FileUtils.h"
-
-#include "BuilderUtils.h"
-
-static dispatch_queue_t write_queue = dispatch_queue_create("com.apple.dyld.cache-builder.write", DISPATCH_QUEUE_CONCURRENT);
-static dispatch_group_t build_group = dispatch_group_create();
-
-dispatch_group_t buildGroup() {
-    return build_group;
-}
-
-void insertFileInBom(const std::string& path, BOMBom bom)
-{
-    std::vector<std::string> components;
-    std::vector<std::string> processed_components;
-    std::stringstream ss(path);
-    std::string item;
-
-    while (std::getline(ss, item, '/')) {
-        if (!item.empty()) {
-            components.push_back(item);
-        }
-    }
-
-    std::string partialPath = ".";
-    std::string lastComponent = components.back();
-    components.pop_back();
-    BOMFSObject fso = BOMFSObjectNew(BOMDirectoryType);
-    BOMFSObjectSetFlags(fso, B_PATHONLY);
-    BOMFSObjectSetPathName(fso, ".", true);
-    BOMFSObjectSetShortName(fso, ".", true);
-    (void)BOMBomInsertFSObject(bom, fso, false);
-    BOMFSObjectFree(fso);
-
-    for (const auto& component : components) {
-        partialPath = partialPath + "/" + component;
-        fso = BOMFSObjectNew(BOMDirectoryType);
-        BOMFSObjectSetFlags(fso, B_PATHONLY);
-        BOMFSObjectSetPathName(fso, partialPath.c_str(), true);
-        BOMFSObjectSetShortName(fso, component.c_str(), true);
-        (void)BOMBomInsertFSObject(bom, fso, false);
-        BOMFSObjectFree(fso);
-    }
-
-    partialPath = partialPath + "/" + lastComponent;
-    fso = BOMFSObjectNew(BOMFileType);
-    BOMFSObjectSetFlags(fso, B_PATHONLY);
-    BOMFSObjectSetPathName(fso, partialPath.c_str(), true);
-    BOMFSObjectSetShortName(fso, lastComponent.c_str(), true);
-    (void)BOMBomInsertFSObject(bom, fso, false);
-    BOMFSObjectFree(fso);
-}
-
-void makeBoms(dyld3::Manifest& manifest, const std::string& masterDstRoot)
-{
-    mkpath_np((masterDstRoot + "/Boms/").c_str(), 0755);
-
-    manifest.forEachConfiguration([&manifest, &masterDstRoot](const std::string& configName) {
-        auto                     config = manifest.configuration(configName);
-        std::vector<std::string> prodBomPaths;
-        std::vector<std::string> devBomPaths;
-
-        std::string runtimePath =  "/System/Library/Caches/com.apple.dyld/";
-        if (manifest.platform() == dyld3::Platform::macOS) {
-            runtimePath =  "/private/var/db/dyld/";
-        }
-
-        for (auto& arch : config.architectures) {
-            std::string cachePath = "dyld_shared_cache_" + arch.first;
-            prodBomPaths.push_back(cachePath);
-            if (manifest.platform() != dyld3::Platform::macOS) {
-                cachePath += ".development";
-            }
-            devBomPaths.push_back(cachePath);
-            char buffer[MAXPATHLEN];
-            sprintf(buffer, "%s/Boms/%s.prod.bom", masterDstRoot.c_str(), configName.c_str());
-            BOMBom bom = BOMBomNew(buffer);
-            for (auto& path : prodBomPaths) {
-                insertFileInBom(runtimePath + path, bom);
-            }
-            BOMBomFree(bom);
-
-            sprintf(buffer, "%s/Boms/%s.dev.bom", masterDstRoot.c_str(), configName.c_str());
-            bom = BOMBomNew(buffer);
-            for (auto& path : devBomPaths) {
-                insertFileInBom(runtimePath + path, bom);
-            }
-            BOMBomFree(bom);
-            
-            sprintf(buffer, "%s/Boms/%s.full.bom", masterDstRoot.c_str(), configName.c_str());
-            bom = BOMBomNew(buffer);
-            for (auto& path : prodBomPaths) {
-                insertFileInBom(runtimePath + path, bom);
-            }
-            for (auto& path : devBomPaths) {
-                insertFileInBom(runtimePath + path, bom);
-            }
-            BOMBomFree(bom);
-        }
-    });
-}
-
-bool build(Diagnostics& diags, dyld3::Manifest& manifest, const std::string& masterDstRoot, bool dedupe, bool verbose,
-           bool skipWrites, bool agileChooseSHA256CdHash, bool emitDevCaches, bool isLocallyBuiltCache)
-{
-    dispatch_queue_t                   queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
-    dispatch_queue_t                   warningQueue = dispatch_queue_create("com.apple.dyld.cache-builder.warnings", DISPATCH_QUEUE_SERIAL);
-    std::vector<std::set<std::string>> dedupedCacheSets;
-    if (dedupe) {
-        manifest.forEachConfiguration([&manifest, &dedupedCacheSets](const std::string& configName) {
-            auto config = manifest.configuration(configName);
-            bool dupeFound = false;
-            
-            for (auto& cacheSet : dedupedCacheSets) {
-                if (config == manifest.configuration(*cacheSet.begin())) {
-                    cacheSet.insert(configName);
-                    dupeFound = true;
-                    break;
-                }
-            }
-            
-            if (!dupeFound) {
-                std::set<std::string> temp;
-                temp.insert(configName);
-                dedupedCacheSets.push_back(temp);
-            }
-        });
-    } else {
-        manifest.forEachConfiguration([&manifest, &dedupedCacheSets](const std::string& configName) {
-            std::set<std::string> temp;
-            temp.insert(configName);
-            dedupedCacheSets.push_back(temp);
-        });
-    }
-    
-    std::vector<dyld3::BuildQueueEntry> buildQueue;
-    
-    for (auto& cacheSet : dedupedCacheSets) {
-        //FIXME we may want to consider moving to hashes of UUID sets
-        std::string setName;
-        
-        for (auto& archName : cacheSet) {
-            if (!setName.empty()) {
-                setName += "|";
-            }
-            setName += archName;
-        }
-        
-        std::stringstream fileNameStream;
-#pragma clang diagnostic push
-#pragma clang diagnostic ignored "-Wdeprecated-declarations"
-        std::array<uint8_t, CC_SHA1_DIGEST_LENGTH> digest = { 0 };
-        CC_SHA1(setName.c_str(), (unsigned int)setName.length(), &digest[0]);
-#pragma clang diagnostic pop
-
-        fileNameStream << std::hex << std::uppercase << std::setfill('0');
-        for (int c : digest) {
-            fileNameStream << std::setw(2) << c;
-        }
-        
-        std::string fileName(fileNameStream.str());
-        
-        if (dedupe) {
-            for (auto& config : cacheSet) {
-                if (!skipWrites) {
-                    int err = symlink(("DedupedConfigs/" + fileName).c_str(), (masterDstRoot + "/" + config).c_str());
-                    if (err) {
-                        diags.warning("Could not create symlink '%s' -> 'DedupedConfigs/%s' (%d)", config.c_str(), fileName.c_str(), err);
-                    }
-                }
-            }
-        }
-        
-        manifest.configuration(*cacheSet.begin()).forEachArchitecture([&masterDstRoot, &dedupe, &fileName, &setName, &manifest,
-                                                                       &buildQueue, &cacheSet, skipWrites, verbose, emitDevCaches, isLocallyBuiltCache](const std::string& arch) {
-            std::string configPath;
-            std::string runtimePath =  "/System/Library/Caches/com.apple.dyld/";
-            if (manifest.platform() == dyld3::Platform::macOS) {
-                runtimePath =  "/private/var/db/dyld/";
-            }
-            if (dedupe) {
-                configPath = masterDstRoot + "/DedupedConfigs/" + fileName + runtimePath;
-            } else {
-                configPath = masterDstRoot + runtimePath;
-            }
-
-            mkpath_np(configPath.c_str(), 0755);
-            if (manifest.platform() == dyld3::Platform::macOS) {
-                buildQueue.push_back(manifest.makeQueueEntry(configPath + "dyld_shared_cache_" + arch, cacheSet, arch, false, setName + "/" + arch,
-                                                             isLocallyBuiltCache, skipWrites, verbose));
-            } else {
-                if (emitDevCaches)
-                    buildQueue.push_back(manifest.makeQueueEntry(configPath + "dyld_shared_cache_" + arch + ".development", cacheSet, arch, false, setName + "/" + arch,
-                                                                 isLocallyBuiltCache, skipWrites, verbose));
-                buildQueue.push_back(manifest.makeQueueEntry(configPath + "dyld_shared_cache_" + arch, cacheSet, arch, true, setName + "/" + arch,
-                                                             isLocallyBuiltCache, skipWrites, verbose));
-            }
-        });
-    }
-
-    __block bool cacheBuildFailure = false;
-    __block std::set<std::string> warnings;
-    __block std::set<std::string> errors;
-
-    dispatch_sync(warningQueue, ^{
-        auto manifestWarnings = diags.warnings();
-        //warnings.insert(manifestWarnings.begin(), manifestWarnings.end());
-    });
-
-    bool requuiresConcurrencyLimit = false;
-    dispatch_semaphore_t concurrencyLimit = NULL;
-    // Limit cuncurrency to 8 threads for machines with 32GB of RAM and to 1 thread if we have 4GB or less of memory
-    uint64_t memSize = 0;
-    size_t sz = sizeof(memSize);;
-    if ( sysctlbyname("hw.memsize", &memSize, &sz, NULL, 0) == 0 ) {
-        if ( memSize <= 0x100000000ULL ) {
-            fprintf(stderr, "Detected 4Gb or less of memory, limiting concurrency to 1 thread\n");
-            requuiresConcurrencyLimit = true;
-            concurrencyLimit = dispatch_semaphore_create(1);
-        } else if ( memSize <= 0x800000000ULL ) {
-            fprintf(stderr, "Detected 32Gb or less of memory, limiting concurrency to 8 threads\n");
-            requuiresConcurrencyLimit = true;
-            concurrencyLimit = dispatch_semaphore_create(8);
-        }
-    }
-
-    dispatch_apply(buildQueue.size(), queue, ^(size_t index) {
-        auto queueEntry = buildQueue[index];
-        pthread_setname_np(queueEntry.options.loggingPrefix.substr(0, MAXTHREADNAMESIZE - 1).c_str());
-        
-        // Horrible hack to limit concurrency in low spec build machines.
-        if (requuiresConcurrencyLimit) { dispatch_semaphore_wait(concurrencyLimit, DISPATCH_TIME_FOREVER); }
-        DyldSharedCache::CreateResults results = DyldSharedCache::create(queueEntry.options, queueEntry.fileSystem, queueEntry.dylibsForCache, queueEntry.otherDylibsAndBundles, queueEntry.mainExecutables);
-        if (requuiresConcurrencyLimit) { dispatch_semaphore_signal(concurrencyLimit); }
-
-        dispatch_sync(warningQueue, ^{
-            warnings.insert(results.warnings.begin(), results.warnings.end());
-            bool chooseSecondCdHash = agileChooseSHA256CdHash;
-            if (agileChooseSHA256CdHash && !results.agileSignature) {
-                // Ignore this option for caches that are not signed agile (which is the majority).
-                chooseSecondCdHash = false;
-            }
-            for (const auto& configName : queueEntry.configNames) {
-                auto& configResults = manifest.configuration(configName).architecture(queueEntry.options.archs->name()).results;
-                for (const auto& mh : results.evictions) {
-                    configResults.exclude(mh, "VM overflow, evicting");
-                }
-                configResults.warnings = results.warnings;
-                if (queueEntry.options.optimizeStubs) {
-                    configResults.productionCache.cdHash = chooseSecondCdHash ? results.cdHashSecond : results.cdHashFirst;
-                } else {
-                    configResults.developmentCache.cdHash =  chooseSecondCdHash ? results.cdHashSecond : results.cdHashFirst;
-                }
-            }
-        });
-        if (!results.errorMessage.empty()) {
-            fprintf(stderr, "[%s] ERROR: %s\n", queueEntry.options.loggingPrefix.c_str(), results.errorMessage.c_str());
-            cacheBuildFailure = true;
-        } else if (skipWrites) {
-            fprintf(stderr, "[%s] Skipped writing cache to: %s\n", queueEntry.options.loggingPrefix.c_str(), queueEntry.outputPath.c_str());
-        }
-    });
-    
-    // print any warnings
-    for (const std::string& warn : warnings) {
-        fprintf(stderr, "[WARNING] %s\n", warn.c_str());
-    }
-    
-    int err = sync_volume_np(masterDstRoot.c_str(), SYNC_VOLUME_FULLSYNC | SYNC_VOLUME_WAIT);
-    if (err) {
-        fprintf(stderr, "Volume sync failed errnor=%d (%s)\n", err, strerror(err));
-    }
-    
-    return !cacheBuildFailure;
-}

dyld3/shared-cache/CacheBuilder.h

@@ -35,8 +35,7 @@
 #include "DyldSharedCache.h"
 #include "Diagnostics.h"
 #include "MachOAnalyzer.h"
-
-
+#include "IMPCaches.hpp"
 
 template <typename P> class LinkeditOptimizer;
 
@@ -44,6 +43,7 @@ template <typename P> class LinkeditOptimizer;
 class CacheBuilder {
 public:
     CacheBuilder(const DyldSharedCache::CreateOptions& options, const dyld3::closure::FileSystem& fileSystem);
+    virtual ~CacheBuilder();
 
     struct InputFile {
         enum State {
@@ -69,30 +69,26 @@ public:
         InputFile*                      inputFile;
     };
 
-    void                                        build(std::vector<InputFile>& inputFiles,
-                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
-    void                                        build(const std::vector<LoadedMachO>& dylibs,
-                                                      const std::vector<LoadedMachO>& otherOsDylibsInput,
-                                                      const std::vector<LoadedMachO>& osExecutables,
-                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
-    void                                        build(const std::vector<DyldSharedCache::MappedMachO>&  dylibsToCache,
-                                                      const std::vector<DyldSharedCache::MappedMachO>&  otherOsDylibs,
-                                                      const std::vector<DyldSharedCache::MappedMachO>&  osExecutables,
-                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
-    void                                        writeFile(const std::string& path);
-    void                                        writeBuffer(uint8_t*& buffer, uint64_t& size);
-    void                                        writeMapFile(const std::string& path);
-    std::string                                 getMapFileBuffer(const std::string& cacheDisposition) const;
-    void                                        deleteBuffer();
     std::string                                 errorMessage();
-    const std::set<std::string>                 warnings();
-    const std::set<const dyld3::MachOAnalyzer*> evictions();
-    const bool                                  agileSignature();
-    const std::string                           cdHashFirst();
-    const std::string                           cdHashSecond();
-    const std::string                           uuid() const;
 
-    void forEachCacheDylib(void (^callback)(const std::string& path));
+    struct Region
+    {
+        uint8_t*    buffer                          = nullptr;
+        uint64_t    bufferSize                      = 0;
+        uint64_t    sizeInUse                       = 0;
+        uint64_t    unslidLoadAddress               = 0;
+        uint64_t    cacheFileOffset                 = 0;
+        uint8_t     initProt                        = 0;
+        uint8_t     maxProt                         = 0;
+        std::string name;
+        uint64_t    index                  = ~0ULL; // The index of this region in the final binary
+
+        // Each region can optionally have its own slide info
+        uint8_t*    slideInfoBuffer                 = nullptr;
+        uint64_t    slideInfoBufferSizeAllocated    = 0;
+        uint64_t    slideInfoFileOffset             = 0;
+        uint64_t    slideInfoFileSize               = 0;
+    };
 
     struct SegmentMappingInfo {
         const void*     srcSegment;
@@ -104,6 +100,8 @@ public:
         uint32_t        dstCacheFileSize;
         uint32_t        copySegmentSize;
         uint32_t        srcSegmentIndex;
+        // Used by the AppCacheBuilder to work out which one of the regions this segment is in
+        const Region*   parentRegion            = nullptr;
     };
 
     struct DylibTextCoalescer {
@@ -114,9 +112,12 @@ public:
         DylibSectionOffsetToCacheSectionOffset objcMethNames;
         DylibSectionOffsetToCacheSectionOffset objcMethTypes;
 
-        bool sectionWasCoalesced(std::string_view sectionName) const;
-        DylibSectionOffsetToCacheSectionOffset& getSectionCoalescer(std::string_view sectionName);
-        const DylibSectionOffsetToCacheSectionOffset& getSectionCoalescer(std::string_view sectionName) const;
+        DylibSectionOffsetToCacheSectionOffset cfStrings;
+
+        bool segmentWasCoalesced(std::string_view segmentName) const;
+        bool sectionWasCoalesced(std::string_view segmentName, std::string_view sectionName) const;
+        DylibSectionOffsetToCacheSectionOffset& getSectionCoalescer(std::string_view segmentName, std::string_view sectionName);
+        const DylibSectionOffsetToCacheSectionOffset& getSectionCoalescer(std::string_view segmentName, std::string_view sectionName) const;
     };
 
     struct CacheCoalescedText {
@@ -132,76 +133,132 @@ public:
             uint64_t                             savedSpace       = 0;
         };
 
+        struct CFSection {
+            uint8_t*                             bufferAddr         = nullptr;
+            uint32_t                             bufferSize         = 0;
+            uint64_t                             bufferVMAddr       = 0;
+            uint64_t                             cacheFileOffset    = 0;
+
+            // The install name of the dylib for the ISA
+            const char*                          isaInstallName     = nullptr;
+            const char*                          isaClassName       = "___CFConstantStringClassReference";
+            uint64_t                             isaVMOffset        = 0;
+        };
+
         StringSection objcClassNames;
         StringSection objcMethNames;
         StringSection objcMethTypes;
 
+        CFSection     cfStrings;
+
         void parseCoalescableText(const dyld3::MachOAnalyzer* ma,
+                                  DylibTextCoalescer& textCoalescer,
+                                  const IMPCaches::SelectorMap& selectors,
+                                  IMPCaches::HoleMap& selectorHoleMap);
+        void parseCFConstants(const dyld3::MachOAnalyzer* ma,
                               DylibTextCoalescer& textCoalescer);
         void clear();
 
         StringSection& getSectionData(std::string_view sectionName);
         const StringSection& getSectionData(std::string_view sectionName) const;
+        uint64_t getSectionVMAddr(std::string_view segmentName, std::string_view sectionName) const;
+        uint8_t* getSectionBufferAddr(std::string_view segmentName, std::string_view sectionName) const;
+        uint64_t getSectionObjcTag(std::string_view segmentName, std::string_view sectionName) const;
     };
 
     class ASLR_Tracker
     {
     public:
+        ASLR_Tracker() = default;
         ~ASLR_Tracker();
 
+        ASLR_Tracker(ASLR_Tracker&&) = delete;
+        ASLR_Tracker(const ASLR_Tracker&) = delete;
+        ASLR_Tracker& operator=(ASLR_Tracker&& other) = delete;
+        ASLR_Tracker& operator=(const ASLR_Tracker& other) = delete;
+
         void        setDataRegion(const void* rwRegionStart, size_t rwRegionSize);
-        void        add(void* p);
+        void        add(void* loc, uint8_t level = (uint8_t)~0);
+        void        setHigh8(void* p, uint8_t high8);
+        void        setAuthData(void* p, uint16_t diversity, bool hasAddrDiv, uint8_t key);
+        void        setRebaseTarget32(void*p, uint32_t targetVMAddr);
+        void        setRebaseTarget64(void*p, uint64_t targetVMAddr);
         void        remove(void* p);
-        bool        has(void* p);
+        bool        has(void* loc, uint8_t* level = nullptr) const;
         const bool* bitmap()        { return _bitmap; }
         unsigned    dataPageCount() { return _pageCount; }
+        unsigned    pageSize() const { return _pageSize; }
         void        disable()       { _enabled = false; };
+        bool        hasHigh8(void* p, uint8_t* highByte) const;
+        bool        hasAuthData(void* p, uint16_t* diversity, bool* hasAddrDiv, uint8_t* key) const;
+        bool        hasRebaseTarget32(void* p, uint32_t* vmAddr) const;
+        bool        hasRebaseTarget64(void* p, uint64_t* vmAddr) const;
+
+        // Get all the out of band rebase targets.  Used for the kernel collection builder
+        // to emit the classic relocations
+        std::vector<void*> getRebaseTargets() const;
 
     private:
 
+        enum {
+#if BUILDING_APP_CACHE_UTIL
+            // The x86_64 kernel collection needs 1-byte aligned fixups
+            kMinimumFixupAlignment = 1
+#else
+            // Shared cache fixups must be at least 4-byte aligned
+            kMinimumFixupAlignment = 4
+#endif
+        };
+
         uint8_t*     _regionStart    = nullptr;
-        uint8_t*     _endStart       = nullptr;
+        uint8_t*     _regionEnd      = nullptr;
         bool*        _bitmap         = nullptr;
         unsigned     _pageCount      = 0;
         unsigned     _pageSize       = 4096;
         bool         _enabled        = true;
+
+        struct AuthData {
+            uint16_t    diversity;
+            bool        addrDiv;
+            uint8_t     key;
+        };
+        std::unordered_map<void*, uint8_t>  _high8Map;
+        std::unordered_map<void*, AuthData> _authDataMap;
+        std::unordered_map<void*, uint32_t> _rebaseTarget32;
+        std::unordered_map<void*, uint64_t> _rebaseTarget64;
+
+        // For kernel collections to work out which other collection a given
+        // fixup is relative to
+#if BUILDING_APP_CACHE_UTIL
+        uint8_t*                            _cacheLevels = nullptr;
+#endif
     };
 
     typedef std::map<uint64_t, std::set<void*>> LOH_Tracker;
 
-    struct Region
-    {
-        uint8_t*    buffer                 = nullptr;
-        uint64_t    bufferSize             = 0;
-        uint64_t    sizeInUse              = 0;
-        uint64_t    unslidLoadAddress      = 0;
-        uint64_t    cacheFileOffset        = 0;
+    // For use by the LinkeditOptimizer to work out which symbols to strip on each binary
+    enum class DylibStripMode {
+        stripNone,
+        stripLocals,
+        stripExports,
+        stripAll
     };
 
-private:
+    struct DylibInfo
+    {
+        const LoadedMachO*              input;
+        std::string                     dylibID;
+        std::vector<SegmentMappingInfo> cacheLocation;
+        DylibTextCoalescer              textCoalescer;
+
+        // <class name, metaclass> -> pointer
+        std::unordered_map<IMPCaches::ClassKey, std::unique_ptr<IMPCaches::ClassData>, IMPCaches::ClassKeyHasher> impCachesClassData;
+    };
+
+protected:
     template <typename P>
     friend class LinkeditOptimizer;
 
-    struct ArchLayout
-    {
-        uint64_t    sharedMemoryStart;
-        uint64_t    sharedMemorySize;
-        uint64_t    textAndDataMaxSize;
-        uint64_t    sharedRegionPadding;
-        uint64_t    pointerDeltaMask;
-        const char* archName;
-        uint16_t    csPageSize;
-        uint8_t     sharedRegionAlignP2;
-        uint8_t     slideInfoBytesPerPage;
-        bool        sharedRegionsAreDiscontiguous;
-        bool        is64;
-        bool        useValueAdd;
-    };
-
-    static const ArchLayout  _s_archLayout[];
-    static const char* const _s_neverStubEliminateDylibs[];
-    static const char* const _s_neverStubEliminateSymbols[];
-
     struct UnmappedRegion
     {
         uint8_t*    buffer                 = nullptr;
@@ -209,117 +266,58 @@ private:
         uint64_t    sizeInUse              = 0;
     };
 
-    struct DylibInfo
-    {
-        const LoadedMachO*              input;
-        std::string                     runtimePath;
-        std::vector<SegmentMappingInfo> cacheLocation;
-        DylibTextCoalescer              textCoalescer;
-    };
+    // Virtual methods overridden by the shared cache builder and app cache builder
+    virtual void forEachDylibInfo(void (^callback)(const DylibInfo& dylib, Diagnostics& dylibDiag)) = 0;
 
-    void        makeSortedDylibs(const std::vector<LoadedMachO>& dylibs, const std::unordered_map<std::string, unsigned> sortOrder);
-    void        processSelectorStrings(const std::vector<LoadedMachO>& executables);
-    void        parseCoalescableSegments();
-    void        assignSegmentAddresses();
-
-    uint64_t    cacheOverflowAmount();
-    size_t      evictLeafDylibs(uint64_t reductionTarget, std::vector<const LoadedMachO*>& overflowDylibs);
-
-    void        fipsSign();
-    void        codeSign();
-    uint64_t    pathHash(const char* path);
-    void        writeCacheHeader();
     void        copyRawSegments();
-    void        adjustAllImagesForNewSegmentLocations();
-    void        writeSlideInfoV1();
-    void        writeSlideInfoV3(const bool bitmap[], unsigned dataPageCoun);
-    uint16_t    pageStartV3(uint8_t* pageContent, uint32_t pageSize, const bool bitmap[]);
-    void        findDylibAndSegment(const void* contentPtr, std::string& dylibName, std::string& segName);
-    void        addImageArray();
-    void        buildImageArray(std::vector<DyldSharedCache::FileAlias>& aliases);
-    void        addOtherImageArray(const std::vector<LoadedMachO>&, std::vector<const LoadedMachO*>& overflowDylibs);
-    void        addClosures(const std::vector<LoadedMachO>&);
-    void        markPaddingInaccessible();
-
-    bool        writeCache(void (^cacheSizeCallback)(uint64_t size), bool (^copyCallback)(const uint8_t* src, uint64_t size, uint64_t dstOffset));
-
-    template <typename P> void writeSlideInfoV2(const bool bitmap[], unsigned dataPageCount);
-    template <typename P> bool makeRebaseChainV2(uint8_t* pageContent, uint16_t lastLocationOffset, uint16_t newOffset, const struct dyld_cache_slide_info2* info);
-    template <typename P> void addPageStartsV2(uint8_t* pageContent, const bool bitmap[], const struct dyld_cache_slide_info2* info,
-                                             std::vector<uint16_t>& pageStarts, std::vector<uint16_t>& pageExtras);
-
-    template <typename P> void writeSlideInfoV4(const bool bitmap[], unsigned dataPageCount);
-    template <typename P> bool makeRebaseChainV4(uint8_t* pageContent, uint16_t lastLocationOffset, uint16_t newOffset, const struct dyld_cache_slide_info4* info);
-    template <typename P> void addPageStartsV4(uint8_t* pageContent, const bool bitmap[], const struct dyld_cache_slide_info4* info,
-                                             std::vector<uint16_t>& pageStarts, std::vector<uint16_t>& pageExtras);
+    void        adjustAllImagesForNewSegmentLocations(uint64_t cacheBaseAddress,
+                                                      ASLR_Tracker& aslrTracker, LOH_Tracker* lohTracker,
+                                                      const CacheBuilder::CacheCoalescedText* coalescedText);
 
     // implemented in AdjustDylibSegemnts.cpp
-    void        adjustDylibSegments(const DylibInfo& dylib, Diagnostics& diag) const;
+    void        adjustDylibSegments(const DylibInfo& dylib, Diagnostics& diag,
+                                    uint64_t cacheBaseAddress,
+                                    CacheBuilder::ASLR_Tracker& aslrTracker,
+                                    CacheBuilder::LOH_Tracker* lohTracker,
+                                    const CacheBuilder::CacheCoalescedText* coalescedText) const;
 
     // implemented in OptimizerLinkedit.cpp
-    void        optimizeLinkedit();
-
-    // implemented in OptimizerObjC.cpp
-    void        optimizeObjC();
-    uint32_t    computeReadOnlyObjC(uint32_t selRefCount, uint32_t classDefCount, uint32_t protocolDefCount);
-    uint32_t    computeReadWriteObjC(uint32_t imageCount, uint32_t protocolDefCount);
+    void        optimizeLinkedit(UnmappedRegion* localSymbolsRegion,
+                                 const std::vector<std::tuple<const mach_header*, const char*, DylibStripMode>>& images);
 
     // implemented in OptimizerBranches.cpp
-    void        optimizeAwayStubs();
-
-    typedef std::unordered_map<std::string, const dyld3::MachOAnalyzer*> InstallNameToMA;
-
-    typedef uint64_t                                                CacheOffset;
+    void        optimizeAwayStubs(const std::vector<std::pair<const mach_header*, const char*>>& images,
+                                  int64_t cacheSlide, uint64_t cacheUnslidAddr,
+                                  const DyldSharedCache* dyldCache,
+                                  const char* const neverStubEliminateSymbols[]);
 
     const DyldSharedCache::CreateOptions&       _options;
     const dyld3::closure::FileSystem&           _fileSystem;
     Region                                      _readExecuteRegion;
-    Region                                      _readWriteRegion;
     Region                                      _readOnlyRegion;
     UnmappedRegion                              _localSymbolsRegion;
-    UnmappedRegion                              _codeSignatureRegion;
     vm_address_t                                _fullAllocatedBuffer;
     uint64_t                                    _nonLinkEditReadOnlySize;
     Diagnostics                                 _diagnostics;
-    std::set<const dyld3::MachOAnalyzer*>       _evictions;
-    const ArchLayout*                           _archLayout;
-    uint32_t                                    _aliasCount;
-    uint64_t                                    _slideInfoFileOffset;
-    uint64_t                                    _slideInfoBufferSizeAllocated;
-    uint8_t*                                    _objcReadOnlyBuffer;
-    uint64_t                                    _objcReadOnlyBufferSizeUsed;
-    uint64_t                                    _objcReadOnlyBufferSizeAllocated;
-    uint8_t*                                    _objcReadWriteBuffer;
-    uint64_t                                    _objcReadWriteBufferSizeAllocated;
+    TimeRecorder                                _timeRecorder;
     uint64_t                                    _allocatedBufferSize;
     CacheCoalescedText                          _coalescedText;
-    uint64_t                                    _selectorStringsFromExecutables;
-    std::vector<DylibInfo>                      _sortedDylibs;
-    InstallNameToMA                             _installNameToCacheDylib;
-    std::unordered_map<std::string, uint32_t>   _dataDirtySegsOrder;
+    bool                                        _is64                       = false;
     // Note this is mutable as the only parallel writes to it are done atomically to the bitmap
     mutable ASLR_Tracker                        _aslrTracker;
-    std::map<void*, std::string>                _missingWeakImports;
     mutable LOH_Tracker                         _lohTracker;
-    const dyld3::closure::ImageArray*           _imageArray;
-    uint32_t                                    _sharedStringsPoolVmOffset;
-    uint8_t                                     _cdHashFirst[20];
-    uint8_t                                     _cdHashSecond[20];
-    std::unordered_map<const dyld3::MachOLoaded*, std::set<CacheOffset>>        _dylibToItsExports;
-    std::set<std::pair<const dyld3::MachOLoaded*, CacheOffset>>                 _dylibWeakExports;
-    std::unordered_map<CacheOffset, std::vector<dyld_cache_patchable_location>> _exportsToUses;
-    std::unordered_map<CacheOffset, std::string>                                _exportsToName;
 };
 
-
-
-
 inline uint64_t align(uint64_t addr, uint8_t p2)
 {
     uint64_t mask = (1 << p2);
     return (addr + mask - 1) & (-mask);
 }
 
+inline uint8_t* align_buffer(uint8_t* addr, uint8_t p2)
+{
+    return (uint8_t *)align((uintptr_t)addr, p2);
+}
 
 
 #endif /* CacheBuilder_h */

dyld3/shared-cache/DyldSharedCache.cpp

@@ -41,7 +41,7 @@
 #include <vector>
 #include <unordered_map>
 #include <unordered_set>
-#include "CacheBuilder.h"
+#include "SharedCacheBuilder.h"
 #include "FileUtils.h"
 #endif
 
@@ -59,6 +59,10 @@
 #include <sstream>
 #endif
 
+#if (BUILDING_LIBDYLD || BUILDING_DYLD)
+VIS_HIDDEN bool gEnableSharedCacheDataConst = false;
+#endif
+
 
 #if BUILDING_CACHE_BUILDER
 DyldSharedCache::CreateResults DyldSharedCache::create(const CreateOptions&               options,
@@ -68,7 +72,7 @@ DyldSharedCache::CreateResults DyldSharedCache::create(const CreateOptions&
                                                        const std::vector<MappedMachO>&    osExecutables)
 {
     CreateResults       results;
-    CacheBuilder   cache(options, fileSystem);
+    SharedCacheBuilder  cache(options, fileSystem);
     if (!cache.errorMessage().empty()) {
         results.errorMessage = cache.errorMessage();
         return results;
@@ -115,7 +119,7 @@ DyldSharedCache::CreateResults DyldSharedCache::create(const CreateOptions&
 
 bool DyldSharedCache::verifySelfContained(std::vector<MappedMachO>& dylibsToCache,
                                           std::unordered_set<std::string>& badZippered,
-                                          MappedMachO (^loader)(const std::string& runtimePath),
+                                          MappedMachO (^loader)(const std::string& runtimePath, Diagnostics& diag),
                                           std::vector<std::pair<DyldSharedCache::MappedMachO, std::set<std::string>>>& rejected)
 {
     // build map of dylibs
@@ -132,6 +136,7 @@ bool DyldSharedCache::verifySelfContained(std::vector<MappedMachO>& dylibsToCach
     }
 
     // check all dependencies to assure every dylib in cache only depends on other dylibs in cache
+    __block std::set<std::string> missingWeakDylibs;
     __block bool doAgain = true;
     while ( doAgain ) {
         __block std::vector<DyldSharedCache::MappedMachO> foundMappings;
@@ -141,6 +146,8 @@ bool DyldSharedCache::verifySelfContained(std::vector<MappedMachO>& dylibsToCach
             if ( badDylibs.count(dylib.runtimePath) != 0 )
                 continue;
             dylib.mh->forEachDependentDylib(^(const char* loadPath, bool isWeak, bool isReExport, bool isUpward, uint32_t compatVersion, uint32_t curVersion, bool& stop) {
+                if ( isWeak && (missingWeakDylibs.count(loadPath) != 0) )
+                    return;
                 if ( knownDylibs.count(loadPath) == 0 ) {
                     doAgain = true;
                     if ( badZippered.count(loadPath) != 0 ) {
@@ -151,10 +158,23 @@ bool DyldSharedCache::verifySelfContained(std::vector<MappedMachO>& dylibsToCach
                         badZippered.insert(dylib.mh->installName());
                         return;
                     }
+                    Diagnostics diag;
                     MappedMachO foundMapping;
                     if ( badDylibs.count(loadPath) == 0 )
-                        foundMapping = loader(loadPath);
+                        foundMapping = loader(loadPath, diag);
                     if ( foundMapping.length == 0 ) {
+                        // We allow weakly linked dylibs to be missing only if they are not present on disk
+                        // The shared cache doesn't contain enough information to patch them in later if they are
+                        // found on disk, so we don't want to pull something in to cache and cut it off from a dylib it
+                        // could have used.
+                        if ( isWeak ) {
+                            missingWeakDylibs.insert(loadPath);
+                            return;
+                        }
+
+                        if (diag.hasError())
+                            badDylibs[dylib.runtimePath].insert(diag.errorMessage());
+                        else
                             badDylibs[dylib.runtimePath].insert(std::string("Could not find dependency '") + loadPath +"'");
                         knownDylibs.erase(dylib.runtimePath);
                         knownDylibs.erase(dylib.mh->installName());
@@ -209,7 +229,14 @@ const T DyldSharedCache::getAddrField(uint64_t addr) const {
     return (const T)(addr + slide);
 }
 
-void DyldSharedCache::forEachRegion(void (^handler)(const void* content, uint64_t vmAddr, uint64_t size, uint32_t permissions)) const
+uint64_t DyldSharedCache::getCodeSignAddress() const
+{
+    auto mappings = (const dyld_cache_mapping_info*)((uint8_t*)this + header.mappingOffset);
+    return mappings[header.mappingCount-1].address + mappings[header.mappingCount-1].size;
+}
+
+void DyldSharedCache::forEachRegion(void (^handler)(const void* content, uint64_t vmAddr, uint64_t size,
+                                                    uint32_t initProt, uint32_t maxProt, uint64_t flags)) const
 {
     // <rdar://problem/49875993> sanity check cache header
     if ( strncmp(header.magic, "dyld_v1", 7) != 0 )
@@ -218,10 +245,18 @@ void DyldSharedCache::forEachRegion(void (^handler)(const void* content, uint64_
         return;
     if ( header.mappingCount > 20 )
         return;
+    if ( header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset) ) {
         const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
         const dyld_cache_mapping_info* mappingsEnd = &mappings[header.mappingCount];
         for (const dyld_cache_mapping_info* m=mappings; m < mappingsEnd; ++m) {
-        handler((char*)this + m->fileOffset, m->address, m->size, m->initProt);
+            handler((char*)this + m->fileOffset, m->address, m->size, m->initProt, m->maxProt, 0);
+        }
+    } else {
+        const dyld_cache_mapping_and_slide_info* mappings = (const dyld_cache_mapping_and_slide_info*)((char*)this + header.mappingWithSlideOffset);
+        const dyld_cache_mapping_and_slide_info* mappingsEnd = &mappings[header.mappingCount];
+        for (const dyld_cache_mapping_and_slide_info* m=mappings; m < mappingsEnd; ++m) {
+            handler((char*)this + m->fileOffset, m->address, m->size, m->initProt, m->maxProt, m->flags);
+        }
     }
 }
 
@@ -236,7 +271,8 @@ bool DyldSharedCache::inCache(const void* addr, size_t length, bool& readOnly) c
     uintptr_t unslidStart = (uintptr_t)addr - slide;
 
     // quick out if after end of cache
-    if ( unslidStart > (mappings[2].address + mappings[2].size) )
+    const dyld_cache_mapping_info* lastMapping = &mappings[header.mappingCount - 1];
+    if ( unslidStart > (lastMapping->address + lastMapping->size) )
         return false;
 
     // walk cache regions
@@ -252,6 +288,13 @@ bool DyldSharedCache::inCache(const void* addr, size_t length, bool& readOnly) c
     return false;
 }
 
+bool DyldSharedCache::isAlias(const char* path) const {
+    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
+    uintptr_t slide = (uintptr_t)this - (uintptr_t)(mappings[0].address);
+    // paths for aliases are store between cache header and first segment
+    return path < ((char*)mappings[0].address + slide);
+}
+
 void DyldSharedCache::forEachImage(void (^handler)(const mach_header* mh, const char* installName)) const
 {
     const dyld_cache_image_info*   dylibs   = (dyld_cache_image_info*)((char*)this + header.imagesOffset);
@@ -293,6 +336,62 @@ void DyldSharedCache::forEachImageEntry(void (^handler)(const char* path, uint64
     }
 }
 
+const bool DyldSharedCache::hasLocalSymbolsInfo() const
+{
+    return (header.localSymbolsOffset != 0 && header.mappingOffset > offsetof(dyld_cache_header,localSymbolsSize));
+}
+
+const void* DyldSharedCache::getLocalNlistEntries() const
+{
+    // check for cache without local symbols info
+    if (!this->hasLocalSymbolsInfo())
+        return nullptr;
+    const auto localInfo = (dyld_cache_local_symbols_info*)((uint8_t*)this + header.localSymbolsOffset);
+    return (uint8_t*)localInfo + localInfo->nlistOffset;
+}
+
+const uint32_t DyldSharedCache::getLocalNlistCount() const
+{
+    // check for cache without local symbols info
+     if (!this->hasLocalSymbolsInfo())
+        return 0;
+    const auto localInfo = (dyld_cache_local_symbols_info*)((uint8_t*)this + header.localSymbolsOffset);
+    return localInfo->nlistCount;
+}
+
+const char* DyldSharedCache::getLocalStrings() const
+{
+    // check for cache without local symbols info
+     if (!this->hasLocalSymbolsInfo())
+        return nullptr;
+    const auto localInfo = (dyld_cache_local_symbols_info*)((uint8_t*)this + header.localSymbolsOffset);
+    return (char*)localInfo + localInfo->stringsOffset;
+}
+
+const uint32_t DyldSharedCache::getLocalStringsSize() const
+{
+    // check for cache without local symbols info
+     if (!this->hasLocalSymbolsInfo())
+        return 0;
+    const auto localInfo = (dyld_cache_local_symbols_info*)((uint8_t*)this + header.localSymbolsOffset);
+    return localInfo->stringsSize;
+}
+
+ void DyldSharedCache::forEachLocalSymbolEntry(void (^handler)(uint32_t dylibOffset, uint32_t nlistStartIndex, uint32_t nlistCount, bool& stop)) const
+{
+    // check for cache without local symbols info
+     if (!this->hasLocalSymbolsInfo())
+        return;
+    const auto localInfo = (dyld_cache_local_symbols_info*)((uint8_t*)this + header.localSymbolsOffset);
+    const auto localEntries = (dyld_cache_local_symbols_entry*)((uint8_t*)localInfo + localInfo->entriesOffset);
+    bool stop = false;
+    for (uint32_t i = 0; i < localInfo->entriesCount; i++) {
+        dyld_cache_local_symbols_entry localEntry = localEntries[i];
+        handler(localEntry.dylibOffset, localEntry.nlistStartIndex, localEntry.nlistCount, stop);
+    }
+}
+
+
 const mach_header* DyldSharedCache::getIndexedImageEntry(uint32_t index, uint64_t& mTime, uint64_t& inode) const
 {
     const dyld_cache_image_info*   dylibs   = (dyld_cache_image_info*)((char*)this + header.imagesOffset);
@@ -302,10 +401,17 @@ const mach_header* DyldSharedCache::getIndexedImageEntry(uint32_t index, uint64_
     return (mach_header*)((uint8_t*)this + dylibs[index].address - mappings[0].address);
 }
 
+
+ const char* DyldSharedCache::getIndexedImagePath(uint32_t index) const
+{
+    auto dylibs = (const dyld_cache_image_info*)((char*)this + header.imagesOffset);
+    return (char*)this + dylibs[index].pathFileOffset;
+}
+
 void DyldSharedCache::forEachImageTextSegment(void (^handler)(uint64_t loadAddressUnslid, uint64_t textSegmentSize, const uuid_t dylibUUID, const char* installName, bool& stop)) const
 {
     // check for old cache without imagesText array
-    if ( header.mappingOffset < 123 )
+    if ( (header.mappingOffset <= __offsetof(dyld_cache_header, imagesTextOffset)) || (header.imagesTextCount == 0) )
         return;
 
     // walk imageText table and call callback for each entry
@@ -358,15 +464,16 @@ std::string DyldSharedCache::mapFile() const
     __block std::vector<uint64_t>   regionFileOffsets;
 
     result.reserve(256*1024);
-    forEachRegion(^(const void* content, uint64_t vmAddr, uint64_t size, uint32_t permissions) {
+    forEachRegion(^(const void* content, uint64_t vmAddr, uint64_t size,
+                    uint32_t initProt, uint32_t maxProt, uint64_t flags) {
         regionStartAddresses.push_back(vmAddr);
         regionSizes.push_back(size);
         regionFileOffsets.push_back((uint8_t*)content - (uint8_t*)this);
         char lineBuffer[256];
         const char* prot = "RW";
-        if ( permissions == (VM_PROT_EXECUTE|VM_PROT_READ) )
+        if ( maxProt == (VM_PROT_EXECUTE|VM_PROT_READ) )
             prot = "EX";
-        else if ( permissions == VM_PROT_READ )
+        else if ( maxProt == VM_PROT_READ )
             prot = "RO";
         if ( size > 1024*1024 )
             sprintf(lineBuffer, "mapping  %s %4lluMB 0x%0llX -> 0x%0llX\n", prot, size/(1024*1024), vmAddr, vmAddr+size);
@@ -409,7 +516,8 @@ uint64_t DyldSharedCache::mappedSize() const
 {
     __block uint64_t startAddr = 0;
     __block uint64_t endAddr = 0;
-    forEachRegion(^(const void* content, uint64_t vmAddr, uint64_t size, uint32_t permissions) {
+    forEachRegion(^(const void* content, uint64_t vmAddr, uint64_t size,
+                    uint32_t initProt, uint32_t maxProt, uint64_t flags) {
         if ( startAddr == 0 )
             startAddr = vmAddr;
         uint64_t end = vmAddr+size;
@@ -473,6 +581,16 @@ bool DyldSharedCache::hasImagePath(const char* dylibPath, uint32_t& imageIndex)
     return false;
 }
 
+bool DyldSharedCache::isOverridablePath(const char* dylibPath) const
+{
+    // all dylibs in customer dyld cache cannot be overridden except libdispatch.dylib
+    if ( header.cacheType == kDyldSharedCacheTypeProduction ) {
+        return (strcmp(dylibPath, "/usr/lib/system/libdispatch.dylib") == 0);
+    }
+    // in dev caches we can override all paths
+    return true;
+}
+
 bool DyldSharedCache::hasNonOverridablePath(const char* dylibPath) const
 {
     // all dylibs in customer dyld cache cannot be overridden except libdispatch.dylib
@@ -480,18 +598,19 @@ bool DyldSharedCache::hasNonOverridablePath(const char* dylibPath) const
     if ( header.cacheType == kDyldSharedCacheTypeProduction ) {
         uint32_t imageIndex;
         pathIsInDyldCacheWhichCannotBeOverridden = this->hasImagePath(dylibPath, imageIndex);
-        if ( pathIsInDyldCacheWhichCannotBeOverridden && (strcmp(dylibPath, "/usr/lib/system/libdispatch.dylib") == 0) )
+        if ( pathIsInDyldCacheWhichCannotBeOverridden && isOverridablePath(dylibPath) )
             pathIsInDyldCacheWhichCannotBeOverridden = false;
     }
     return pathIsInDyldCacheWhichCannotBeOverridden;
 }
 
+#if !BUILDING_LIBDSC
 const dyld3::closure::Image* DyldSharedCache::findDlopenOtherImage(const char* path) const
 {
     const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
     if ( mappings[0].fileOffset != 0 )
         return nullptr;
-    if ( header.mappingOffset < sizeof(dyld_cache_header) )
+    if ( header.mappingOffset < __offsetof(dyld_cache_header, otherImageArrayAddr) )
         return nullptr;
     if ( header.otherImageArrayAddr == 0 )
         return nullptr;
@@ -511,9 +630,6 @@ const dyld3::closure::Image* DyldSharedCache::findDlopenOtherImage(const char* p
     return nullptr;
 }
 
-
-
-
 const dyld3::closure::LaunchClosure* DyldSharedCache::findClosure(const char* executablePath) const
 {
     const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
@@ -575,7 +691,23 @@ void DyldSharedCache::forEachDlopenImage(void (^handler)(const char* runtimePath
         }
     }
 }
-#endif
+
+void DyldSharedCache::forEachDylibPath(void (^handler)(const char* dylibPath, uint32_t index)) const
+{
+    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
+    uintptr_t      slide                = (uintptr_t)this - (uintptr_t)(mappings[0].address);
+    const uint8_t* dylibTrieStart       = (uint8_t*)(this->header.dylibsTrieAddr + slide);
+    const uint8_t* dylibTrieEnd         = dylibTrieStart + this->header.dylibsTrieSize;
+
+   std::vector<DylibIndexTrie::Entry> dylibEntries;
+    if ( Trie<DylibIndex>::parseTrie(dylibTrieStart, dylibTrieEnd, dylibEntries) ) {
+        for (DylibIndexTrie::Entry& entry : dylibEntries ) {
+            handler(entry.name.c_str(), entry.info.index);
+        }
+    }
+}
+#endif // !BUILDING_LIBDYLD && !BUILDING_DYLD
+#endif // !BUILDING_LIBDSC
 
 const dyld3::closure::ImageArray* DyldSharedCache::cachedDylibsImageArray() const
 {
@@ -594,7 +726,7 @@ const dyld3::closure::ImageArray* DyldSharedCache::cachedDylibsImageArray() cons
 const dyld3::closure::ImageArray* DyldSharedCache::otherOSImageArray() const
 {
     // check for old cache without imagesArray
-    if ( header.mappingOffset < sizeof(dyld_cache_header) )
+    if ( header.mappingOffset < __offsetof(dyld_cache_header, otherImageArrayAddr) )
         return nullptr;
 
     if ( header.otherImageArrayAddr == 0 )
@@ -665,6 +797,59 @@ void DyldSharedCache::forEachPatchableUseOfExport(uint32_t imageIndex, uint32_t
     }
 }
 
+#if (BUILDING_LIBDYLD || BUILDING_DYLD)
+void DyldSharedCache::changeDataConstPermissions(mach_port_t machTask, uint32_t permissions,
+                                                 DataConstLogFunc logFunc) const {
+
+    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
+    uintptr_t slide = (uintptr_t)this - (uintptr_t)(mappings[0].address);
+
+    if ( (permissions & VM_PROT_WRITE) != 0 )
+        permissions |= VM_PROT_COPY;
+
+    forEachRegion(^(const void *, uint64_t vmAddr, uint64_t size,
+                    uint32_t initProt, uint32_t maxProt, uint64_t flags) {
+        void* content = (void*)(vmAddr + slide);
+        if ( ( flags & DYLD_CACHE_MAPPING_CONST_DATA) == 0 )
+            return;
+        if ( logFunc != nullptr ) {
+            logFunc("dyld: marking shared cache range 0x%x permissions: 0x%09lX -> 0x%09lX\n",
+                    permissions, (long)content, (long)content + size);
+        }
+        kern_return_t result = vm_protect(machTask, (vm_address_t)content, (vm_size_t)size, false, permissions);
+        if ( result != KERN_SUCCESS ) {
+            if ( logFunc != nullptr )
+                logFunc("dyld: failed to mprotect shared cache due to: %d\n", result);
+        }
+    });
+}
+
+DyldSharedCache::DataConstLazyScopedWriter::DataConstLazyScopedWriter(const DyldSharedCache* cache, mach_port_t machTask, DataConstLogFunc logFunc)
+    : cache(cache), machTask(machTask), logFunc(logFunc) {
+}
+
+DyldSharedCache::DataConstLazyScopedWriter::~DataConstLazyScopedWriter() {
+    if ( wasMadeWritable )
+        cache->changeDataConstPermissions(machTask, VM_PROT_READ, logFunc);
+}
+
+void DyldSharedCache::DataConstLazyScopedWriter::makeWriteable() {
+    if ( wasMadeWritable )
+        return;
+    if ( !gEnableSharedCacheDataConst )
+        return;
+    if ( cache == nullptr )
+        return;
+    wasMadeWritable = true;
+    cache->changeDataConstPermissions(machTask, VM_PROT_READ | VM_PROT_WRITE, logFunc);
+}
+
+DyldSharedCache::DataConstScopedWriter::DataConstScopedWriter(const DyldSharedCache* cache, mach_port_t machTask, DataConstLogFunc logFunc)
+    : writer(cache, machTask, logFunc) {
+    writer.makeWriteable();
+}
+#endif
+
 #if !(BUILDING_LIBDYLD || BUILDING_DYLD)
 // MRM map file generator
 std::string DyldSharedCache::generateJSONMap(const char* disposition) const {
@@ -738,25 +923,68 @@ std::string DyldSharedCache::generateJSONDependents() const {
 
 #endif
 
+#if !(BUILDING_LIBDYLD || BUILDING_DYLD)
+dyld3::MachOAnalyzer::VMAddrConverter DyldSharedCache::makeVMAddrConverter(bool contentRebased) const {
+    typedef dyld3::MachOAnalyzer::VMAddrConverter VMAddrConverter;
 
+    __block VMAddrConverter::SharedCacheFormat pointerFormat = VMAddrConverter::SharedCacheFormat::none;
+    __block uint64_t pointerValueAdd = 0;;
+    forEachSlideInfo(^(uint64_t mappingStartAddress, uint64_t mappingSize, const uint8_t *mappingPagesStart, uint64_t slideInfoOffset, uint64_t slideInfoSize, const dyld_cache_slide_info *slideInfoHeader) {
+        assert(slideInfoHeader->version >= 2);
+        if ( slideInfoHeader->version == 2 ) {
+            const dyld_cache_slide_info2* slideInfo = (dyld_cache_slide_info2*)(slideInfoHeader);
+            assert(slideInfo->delta_mask == 0x00FFFF0000000000);
+            pointerFormat   = VMAddrConverter::SharedCacheFormat::v2_x86_64_tbi;
+            pointerValueAdd = slideInfo->value_add;
+        } else if ( slideInfoHeader->version == 3 ) {
+            pointerFormat   = VMAddrConverter::SharedCacheFormat::v3;
+            pointerValueAdd = unslidLoadAddress();
+        } else {
+            assert(false);
+        }
+    });
 
-const dyld_cache_slide_info* DyldSharedCache::slideInfo() const
-{
     const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
     uintptr_t slide = (uintptr_t)this - (uintptr_t)(mappings[0].address);
 
-    uint64_t offsetInLinkEditRegion = (header.slideInfoOffset - mappings[2].fileOffset);
+    VMAddrConverter vmAddrConverter;
+    vmAddrConverter.preferredLoadAddress            = pointerValueAdd;
+    vmAddrConverter.slide                           = slide;
+    vmAddrConverter.chainedPointerFormat            = 0;
+    vmAddrConverter.sharedCacheChainedPointerFormat = pointerFormat;
+    vmAddrConverter.contentRebased                  = contentRebased;
+
+    return vmAddrConverter;
+}
+#endif
+
+const dyld_cache_slide_info* DyldSharedCache::legacyCacheSlideInfo() const
+{
+    assert(header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset));
+    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
+    uintptr_t slide = (uintptr_t)this - (uintptr_t)(mappings[0].address);
+
+    uint64_t offsetInLinkEditRegion = (header.slideInfoOffsetUnused - mappings[2].fileOffset);
     return (dyld_cache_slide_info*)((uint8_t*)(mappings[2].address) + slide + offsetInLinkEditRegion);
 }
 
-const uint8_t* DyldSharedCache::dataRegionStart() const
+const dyld_cache_mapping_info* DyldSharedCache::legacyCacheDataRegionMapping() const
 {
+    assert(header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset));
+    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
+    return &mappings[1];
+}
+
+const uint8_t* DyldSharedCache::legacyCacheDataRegionBuffer() const
+{
+    assert(header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset));
     const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((char*)this + header.mappingOffset);
     uintptr_t slide = (uintptr_t)this - (uintptr_t)(mappings[0].address);
     
-    return (uint8_t*)(mappings[1].address) + slide;
+    return (uint8_t*)(legacyCacheDataRegionMapping()->address) + slide;
 }
 
+#if !BUILDING_LIBDSC
 const objc_opt::objc_opt_t* DyldSharedCache::objcOpt() const {
     // Find the objc image
     const dyld3::MachOAnalyzer* objcMA = nullptr;
@@ -810,7 +1038,7 @@ const void* DyldSharedCache::objcOptPtrs() const {
     int64_t slide = objcMA->getSlide();
     uint32_t pointerSize = objcMA->pointerSize();
     objcMA->forEachSection(^(const dyld3::MachOAnalyzer::SectionInfo& info, bool malformedSectionRange, bool& stop) {
-        if ( strncmp(info.segInfo.segName, "__DATA", 6) != 0 )
+        if ( (strncmp(info.segInfo.segName, "__DATA", 6) != 0) && (strncmp(info.segInfo.segName, "__AUTH", 6) != 0) )
             return;
         if (strcmp(info.sectName, "__objc_opt_ptrs") != 0)
             return;
@@ -827,6 +1055,97 @@ const void* DyldSharedCache::objcOptPtrs() const {
 
     return objcPointersContent;
 }
+#endif
+
+std::pair<const void*, uint64_t> DyldSharedCache::getObjCConstantRange() const {
+    const dyld3::MachOAnalyzer* libDyldMA = nullptr;
+    uint32_t imageIndex;
+    if ( hasImagePath("/usr/lib/system/libdyld.dylib", imageIndex) ) {
+        const dyld3::closure::ImageArray* images = cachedDylibsImageArray();
+        const dyld3::closure::Image* image = images->imageForNum(imageIndex+1);
+        libDyldMA = (const dyld3::MachOAnalyzer*)((uintptr_t)this + image->cacheOffset());
+
+        std::pair<const void*, uint64_t> ranges = { nullptr, 0 };
+#if TARGET_OS_OSX
+        ranges.first = libDyldMA->findSectionContent("__DATA", "__objc_ranges", ranges.second);
+#else
+        ranges.first = libDyldMA->findSectionContent("__DATA_CONST", "__objc_ranges", ranges.second);
+#endif
+        return ranges;
+    }
+
+    return { nullptr, 0 };
+}
+
+bool DyldSharedCache::hasSlideInfo() const {
+    if ( header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset) ) {
+        return header.slideInfoSizeUnused != 0;
+    } else {
+        const dyld_cache_mapping_and_slide_info* slidableMappings = (const dyld_cache_mapping_and_slide_info*)((char*)this + header.mappingWithSlideOffset);
+        for (uint32_t i = 0; i != header.mappingWithSlideCount; ++i) {
+            if ( slidableMappings[i].slideInfoFileSize != 0 ) {
+                return true;
+            }
+        }
+    }
+    return false;
+}
+
+void DyldSharedCache::forEachSlideInfo(void (^handler)(uint64_t mappingStartAddress, uint64_t mappingSize,
+                                                       const uint8_t* mappingPagesStart,
+                                                       uint64_t slideInfoOffset, uint64_t slideInfoSize,
+                                                       const dyld_cache_slide_info* slideInfoHeader)) const {
+    if ( header.mappingOffset <= __offsetof(dyld_cache_header, mappingWithSlideOffset) ) {
+        // Old caches should get the slide info from the cache header and assume a single data region.
+        const dyld_cache_mapping_info* dataMapping = legacyCacheDataRegionMapping();
+        uint64_t dataStartAddress = dataMapping->address;
+        uint64_t dataSize = dataMapping->size;
+        const uint8_t* dataPagesStart = legacyCacheDataRegionBuffer();
+        const dyld_cache_slide_info* slideInfoHeader = legacyCacheSlideInfo();
+
+        handler(dataStartAddress, dataSize, dataPagesStart,
+                header.slideInfoOffsetUnused, header.slideInfoSizeUnused, slideInfoHeader);
+    } else {
+        const dyld_cache_mapping_and_slide_info* slidableMappings = (const dyld_cache_mapping_and_slide_info*)((char*)this + header.mappingWithSlideOffset);
+        const dyld_cache_mapping_and_slide_info* linkeditMapping = &slidableMappings[header.mappingWithSlideCount - 1];
+        uint64_t sharedCacheSlide = (uint64_t)this - unslidLoadAddress();
+
+        for (uint32_t i = 0; i != header.mappingWithSlideCount; ++i) {
+            if ( slidableMappings[i].slideInfoFileOffset != 0 ) {
+                // Get the data pages
+                uint64_t dataStartAddress = slidableMappings[i].address;
+                uint64_t dataSize = slidableMappings[i].size;
+                const uint8_t* dataPagesStart = (uint8_t*)dataStartAddress + sharedCacheSlide;
+
+                // Get the slide info
+                uint64_t offsetInLinkEditRegion = (slidableMappings[i].slideInfoFileOffset - linkeditMapping->fileOffset);
+                const dyld_cache_slide_info* slideInfoHeader = (dyld_cache_slide_info*)((uint8_t*)(linkeditMapping->address) + sharedCacheSlide + offsetInLinkEditRegion);
+                handler(dataStartAddress, dataSize, dataPagesStart,
+                        slidableMappings[i].slideInfoFileOffset, slidableMappings[i].slideInfoFileSize, slideInfoHeader);
+            }
+        }
+    }
+}
+
+#if BUILDING_LIBDYLD
+const char* DyldSharedCache::getCanonicalPath(const char *path) const {
+    uint32_t dyldCacheImageIndex;
+    if ( hasImagePath(path, dyldCacheImageIndex) )
+        return getIndexedImagePath(dyldCacheImageIndex);
+#if TARGET_OS_OSX
+    // on macOS support "Foo.framework/Foo" symlink
+    char resolvedPath[PATH_MAX];
+    realpath(path, resolvedPath);
+    int realpathErrno = errno;
+    // If realpath() resolves to a path which does not exist on disk, errno is set to ENOENT
+    if ( (realpathErrno == ENOENT) || (realpathErrno == 0) ) {
+        if ( hasImagePath(resolvedPath, dyldCacheImageIndex) )
+            return getIndexedImagePath(dyldCacheImageIndex);
+    }
+#endif
+    return nullptr;
+}
+#endif
 
 #if !(BUILDING_LIBDYLD || BUILDING_DYLD)
 void DyldSharedCache::fillMachOAnalyzersMap(std::unordered_map<std::string,dyld3::MachOAnalyzer*> & dylibAnalyzers) const {

dyld3/shared-cache/DyldSharedCache.h

@@ -28,6 +28,10 @@
 #include <TargetConditionals.h>
 #include <uuid/uuid.h>
 
+#if (BUILDING_LIBDYLD || BUILDING_DYLD)
+#include <sys/types.h>
+#endif
+
 #if !(BUILDING_LIBDYLD || BUILDING_DYLD)
 #include <set>
 #include <string>
@@ -40,6 +44,7 @@
 #include "Diagnostics.h"
 #include "MachOAnalyzer.h"
 #include "Closure.h"
+#include "JSON.h"
 
 namespace objc_opt {
 struct objc_opt_t;
@@ -57,15 +62,22 @@ public:
         Agile      = 2
     };
 
+    enum class LocalSymbolsMode {
+        keep,
+        unmap,
+        strip
+    };
+
     struct CreateOptions
     {
         std::string                                 outputFilePath;
         std::string                                 outputMapFilePath;
         const dyld3::GradedArchs*                   archs;
         dyld3::Platform                             platform;
-        bool                                        excludeLocalSymbols;
+        LocalSymbolsMode                            localSymbolMode;
         bool                                        optimizeStubs;
-        bool                                        optimizeObjC;
+        bool                                        optimizeDyldDlopens;
+        bool                                        optimizeDyldLaunches;
         CodeSigningDigestMode                       codeSigningDigestMode;
         bool                                        dylibsRemovedDuringMastering;
         bool                                        inodesAreSameAsRuntime;
@@ -76,6 +88,7 @@ public:
         bool                                        evictLeafDylibsOnOverflow;
         std::unordered_map<std::string, unsigned>   dylibOrdering;
         std::unordered_map<std::string, unsigned>   dirtyDataSegmentOrdering;
+        dyld3::json::Node                           objcOptimizations;
         std::string                                 loggingPrefix;
     };
 
@@ -118,7 +131,7 @@ public:
     // outset the set.  It will call back the loader function to try to find any mising dylibs.
     static bool verifySelfContained(std::vector<MappedMachO>& dylibsToCache,
                                     std::unordered_set<std::string>& badZippered,
-                                    MappedMachO (^loader)(const std::string& runtimePath), std::vector<std::pair<DyldSharedCache::MappedMachO, std::set<std::string>>>& excluded);
+                                    MappedMachO (^loader)(const std::string& runtimePath, Diagnostics& diag), std::vector<std::pair<DyldSharedCache::MappedMachO, std::set<std::string>>>& excluded);
 
 
     //
@@ -154,7 +167,7 @@ public:
     //
     std::string         mapFile() const;
 
-#endif // TARGET_OS_OSX
+#endif // BUILDING_CACHE_BUILDER
 
 
     //
@@ -181,12 +194,30 @@ public:
     bool                hasImagePath(const char* dylibPath, uint32_t& imageIndex) const;
 
 
+    //
+    // Is this path (which we know is in the shared cache), overridable
+    //
+    bool                isOverridablePath(const char* dylibPath) const;
+
+
     //
     // Path is to a dylib in the cache and this is an optimized cache so that path cannot be overridden
     //
     bool                hasNonOverridablePath(const char* dylibPath) const;
 
 
+    //
+    // Check if shared cache contains local symbols info
+    //
+    const bool          hasLocalSymbolsInfo() const;
+
+
+    //
+    // Get code signature mapped address
+    //
+    uint64_t             getCodeSignAddress() const;
+
+
     //
     // Searches cache for dylib with specified mach_header
     //
@@ -199,13 +230,29 @@ public:
 
 
     //
-    // Iterates over each dylib in the cache
+    // Get image entry from index
     //
     const mach_header*  getIndexedImageEntry(uint32_t index, uint64_t& mTime, uint64_t& node) const;
 
 
+    // iterates over all dylibs and aliases
+    void forEachDylibPath(void (^handler)(const char* dylibPath, uint32_t index)) const;
+
+
     //
-    // Iterates over each dylib in the cache
+    // Get image path from index
+    //
+    const char*         getIndexedImagePath(uint32_t index) const;
+
+#if BUILDING_LIBDYLD
+    //
+    // Get the canonical (dylib) path for a given path, which may be a symlink to something in the cache
+    //
+    const char*         getCanonicalPath(const char* path) const;
+#endif
+
+    //
+    // Iterates over each text segment in the cache
     //
     void                forEachImageTextSegment(void (^handler)(uint64_t loadAddressUnslid, uint64_t textSegmentSize, const uuid_t dylibUUID, const char* installName, bool& stop)) const;
 
@@ -213,14 +260,48 @@ public:
     //
     // Iterates over each of the three regions in the cache
     //
-    void                forEachRegion(void (^handler)(const void* content, uint64_t vmAddr, uint64_t size, uint32_t permissions)) const;
+    void                forEachRegion(void (^handler)(const void* content, uint64_t vmAddr, uint64_t size,
+                                                      uint32_t initProt, uint32_t maxProt, uint64_t flags)) const;
 
 
+    //
+    // Get local symbols nlist entries
+    //
+    const void*         getLocalNlistEntries() const;
+
+
+    //
+    // Get local symbols nlist count
+    //
+    const uint32_t      getLocalNlistCount() const;
+
+
+    //
+    // Get local symbols strings
+    //
+    const char*  getLocalStrings() const;
+
+
+    //
+    // Get local symbols strings size
+    //
+    const uint32_t       getLocalStringsSize() const;
+
+
+     //
+     // Iterates over each local symbol entry in the cache
+     //
+     void                forEachLocalSymbolEntry(void (^handler)(uint32_t dylibOffset, uint32_t nlistStartIndex, uint32_t nlistCount, bool& stop)) const;
+
     //
     // Returns if an address range is in this cache, and if so if in a read-only area
     //
     bool                inCache(const void* addr, size_t length, bool& readOnly) const;
 
+    //
+    // Returns true if a path is an alternate path (symlink)
+    //
+    bool isAlias(const char* path) const;
 
     //
     // returns address the cache would load at if unslid
@@ -278,12 +359,17 @@ public:
     //
     // Returns the pointer to the slide info for this cache
     //
-    const dyld_cache_slide_info* slideInfo() const;
+    const dyld_cache_slide_info* legacyCacheSlideInfo() const;
+
+    //
+    // Returns a pointer to the __DATA region mapping in the cache
+    //
+    const dyld_cache_mapping_info* legacyCacheDataRegionMapping() const;
 
     //
     // Returns a pointer to the start of the __DATA region in the cache
     //
-    const uint8_t* dataRegionStart() const;
+    const uint8_t* legacyCacheDataRegionBuffer() const;
 
     //
     // Returns a pointer to the shared cache optimized Objective-C data structures
@@ -295,6 +381,15 @@ public:
     //
     const void* objcOptPtrs() const;
 
+    // Returns true if the cache has any slide info, either old style on a single data region
+    // or on each individual data mapping
+    bool                hasSlideInfo() const;
+
+    void                forEachSlideInfo(void (^handler)(uint64_t mappingStartAddress, uint64_t mappingSize,
+                                                         const uint8_t* mappingPagesStart,
+                                                         uint64_t slideInfoOffset, uint64_t slideInfoSize,
+                                                         const dyld_cache_slide_info* slideInfoHeader)) const;
+
 
     //
     // returns true if the offset is in the TEXT of some cached dylib and sets *index to the dylib index
@@ -322,6 +417,45 @@ public:
         return dummy.arm64e.keyName();
     }
 
+#if (BUILDING_LIBDYLD || BUILDING_DYLD)
+
+    typedef void (*DataConstLogFunc)(const char*, ...) __attribute__((format(printf, 1, 2)));
+    void changeDataConstPermissions(mach_port_t machTask, uint32_t permissions, DataConstLogFunc logFunc) const;
+
+    struct DataConstLazyScopedWriter {
+        DataConstLazyScopedWriter(const DyldSharedCache* cache, mach_port_t machTask, DataConstLogFunc logFunc);
+        ~DataConstLazyScopedWriter();
+
+        // Delete all other kinds of constructors to make sure we don't accidentally copy these around
+        DataConstLazyScopedWriter() = delete;
+        DataConstLazyScopedWriter(const DataConstLazyScopedWriter&) = delete;
+        DataConstLazyScopedWriter(DataConstLazyScopedWriter&&) = delete;
+        DataConstLazyScopedWriter& operator=(const DataConstLazyScopedWriter&) = delete;
+        DataConstLazyScopedWriter& operator=(DataConstLazyScopedWriter&&) = delete;
+
+        void makeWriteable();
+
+        const DyldSharedCache*  cache           = nullptr;
+        mach_port_t             machTask        = MACH_PORT_NULL;
+        DataConstLogFunc        logFunc         = nullptr;
+        bool                    wasMadeWritable = false;
+    };
+
+    struct DataConstScopedWriter {
+        DataConstScopedWriter(const DyldSharedCache* cache, mach_port_t machTask, DataConstLogFunc logFunc);
+        ~DataConstScopedWriter() = default;
+
+        // Delete all other kinds of constructors to make sure we don't accidentally copy these around
+        DataConstScopedWriter() = delete;
+        DataConstScopedWriter(const DataConstScopedWriter&) = delete;
+        DataConstScopedWriter(DataConstScopedWriter&&) = delete;
+        DataConstScopedWriter& operator=(const DataConstScopedWriter&) = delete;
+        DataConstScopedWriter& operator=(DataConstScopedWriter&&) = delete;
+
+        DataConstLazyScopedWriter writer;
+    };
+#endif
+
 #if !(BUILDING_LIBDYLD || BUILDING_DYLD)
     // MRM map file generator
     std::string generateJSONMap(const char* disposition) const;
@@ -336,8 +470,26 @@ public:
     std::string generateJSONDependents() const;
 #endif
 
+    // Note these enum entries are only valid for 64-bit archs.
+    enum class ConstantClasses {
+        cfStringAtomSize = 32
+    };
+
+    // Returns the start and size of the range in the shared cache of the ObjC constants, such as
+    // all of the CFString's which have been moved in to a contiguous range
+    std::pair<const void*, uint64_t> getObjCConstantRange() const;
+
+#if !(BUILDING_LIBDYLD || BUILDING_DYLD)
+    dyld3::MachOAnalyzer::VMAddrConverter makeVMAddrConverter(bool contentRebased) const;
+#endif
+
     dyld_cache_header header;
 
+    // The most mappings we could generate.
+    // For now its __TEXT, __DATA_CONST, __DATA_DIRTY, __DATA, __LINKEDIT,
+    // and optionally also __AUTH, __AUTH_CONST, __AUTH_DIRTY
+    static const uint32_t MaxMappings = 8;
+
 private:
     // Returns a variable of type "const T" which corresponds to the header field with the given unslid address
     template<typename T>

dyld3/shared-cache/FileUtils.cpp

@@ -38,9 +38,7 @@
 #include <dispatch/dispatch.h>
 #include <mach-o/dyld.h>
 #include <System/sys/csr.h>
-#ifndef DARLING
 #include <rootless.h>
-#endif
 
 #include <string>
 #include <fstream>
@@ -49,15 +47,8 @@
 #include "FileUtils.h"
 #include "StringUtils.h"
 #include "Diagnostics.h"
+#include "JSONReader.h"
 
-#if __MAC_OS_X_VERSION_MIN_REQUIRED < 101200
-extern "C" int rootless_check_trusted_fd(int fd) __attribute__((weak_import));
-#endif
-
-#ifdef DARLING
-static int rootless_check_trusted(const char* path) { return -1; }
-static int rootless_check_trusted_class(const char* path, const char* cls) { return -1; }
-#endif
 
 void iterateDirectoryTree(const std::string& pathPrefix, const std::string& path, bool (^dirFilter)(const std::string& path), void (^fileCallback)(const std::string& path, const struct stat&), bool processFiles, bool recurse)
 {
@@ -69,7 +60,7 @@ void iterateDirectoryTree(const std::string& pathPrefix, const std::string& path
     }
     while (dirent* entry = readdir(dir)) {
         struct stat statBuf;
-        std::string dirAndFile = path + "/" + entry->d_name;
+        std::string dirAndFile = path + (path.back() != '/' ? "/" : "") + entry->d_name;
         std::string fullDirAndFile = pathPrefix + dirAndFile;
          switch ( entry->d_type ) {
             case DT_REG:
@@ -142,49 +133,6 @@ const void* mapFileReadOnly(const char* path, size_t& mappedSize)
     return nullptr;
 }
 
-static bool sipIsEnabled()
-{
-    static bool             rootlessEnabled;
-    static dispatch_once_t  onceToken;
-    // Check to make sure file system protections are on at all
-    dispatch_once(&onceToken, ^{
-        rootlessEnabled = (csr_check(CSR_ALLOW_UNRESTRICTED_FS) != 0);
-    });
-    return rootlessEnabled;
-}
-
-bool isProtectedBySIP(const std::string& path)
-{
-    if ( !sipIsEnabled() )
-        return false;
-
-    return (rootless_check_trusted(path.c_str()) == 0);
-}
-
-bool isProtectedBySIPExceptDyld(const std::string& path)
-{
-    if ( !sipIsEnabled() )
-        return false;
-
-    return (rootless_check_trusted_class(path.c_str(), "dyld") == 0);
-}
-
-bool isProtectedBySIP(int fd)
-{
-    if ( !sipIsEnabled() )
-        return false;
-
-#if __MAC_OS_X_VERSION_MIN_REQUIRED >= 101200
-    return (rootless_check_trusted_fd(fd) == 0);
-#else
-    // fallback to using rootless_check_trusted
-    char realPath[MAXPATHLEN];
-    if ( fcntl(fd, F_GETPATH, realPath) == 0 )
-        return (rootless_check_trusted(realPath) == 0);
-    return false;
-#endif
-}
-
 bool fileExists(const std::string& path)
 {
     struct stat statBuf;

dyld3/shared-cache/FileUtils.h

@@ -36,6 +36,7 @@
 #include <dispatch/dispatch.h>
 
 #include "DyldSharedCache.h"
+#include "JSON.h"
 
 class Diagnostics;
 
@@ -73,10 +74,6 @@ bool safeSave(const void* buffer, size_t bufferLen, const std::string& path);
 
 const void* mapFileReadOnly(const char* path, size_t& mappedSize);
 
-bool isProtectedBySIP(const std::string& path);
-bool isProtectedBySIPExceptDyld(const std::string& path);
-bool isProtectedBySIP(int fd);
-
 bool fileExists(const std::string& path);
 
 std::unordered_map<std::string, uint32_t> parseOrderFile(const std::string& orderFileData);

dyld3/shared-cache/IMPCaches.hpp

@@ -0,0 +1,380 @@
+//
+//  IMPCaches.hpp
+//  dyld_shared_cache_builder
+//
+//  Created by Thomas Deniau on 18/12/2019.
+//
+
+#ifndef IMPCaches_hpp
+#define IMPCaches_hpp
+
+#include <vector>
+#include <unordered_map>
+#include <unordered_set>
+#include <set>
+#include <map>
+#include <string>
+#include <random>
+
+#define DEBUG_SLOTS 1
+
+template <typename P> class objc_method_t;
+
+namespace IMPCaches {
+class Selector;
+class Constraint;
+
+class ClassData {
+private:
+#if DEBUG_SLOTS
+    std::vector<const Selector*> slots;
+#else
+    std::vector<bool> slots;
+#endif
+
+    // Scratchpad for constraintForMethod
+    std::vector<int> allowedValues;
+
+    void resetSlots();
+
+public:
+    bool isMetaclass;
+
+    // True most of the time, but can also be false in 2 cases:
+    // * We have entries for classes for which we don't want to generate IMP caches
+    //   when they are superclasses of "interesting" (= for which we want an IMP cache)
+    //   classes, so that we can properly attach non-cross-image categories to them and
+    //   reference the right IMP for child classes which are actually interesting
+    // * We can also have failed to generate a cache for this class, or for a class
+    //   in its flattening superclass hierarchy
+    bool shouldGenerateImpCache;
+
+    // Class has duplicates or is a child of a class with duplicates.
+    bool isPartOfDuplicateSet = false;
+
+    // This is set when we drop a class because a class in its flattening superclasss
+    // hierarchy was dropped. In that case, we won't try to flip its shouldGenerateImpCache
+    // value back to true when restoring a snapshot. (We could keep track of all the
+    // dependencies but it would be very messy and the reward is only a few classes here and there).
+    bool droppedBecauseFlatteningSuperclassWasDropped = false;
+
+    uint8_t backtracks = 0;
+
+    // For debug purposes
+    const char* name;
+
+    struct Method {
+        const char* installName = nullptr;
+        const char* className = nullptr;
+        const char* categoryName = nullptr;
+        Selector* selector = nullptr;
+        bool wasInlined = false;
+        bool fromFlattening = false;
+    };
+
+    std::vector<Method> methods;
+
+    std::string description() const;
+
+    int neededBits;
+
+    void didFinishAddingMethods();
+
+    // Not const because this uses the slots as a scratchpad
+    Constraint constraintForMethod(const Selector* m);
+
+    bool operator<(const ClassData& r) const {
+        if (isMetaclass != r.isMetaclass) return isMetaclass;
+        return strcmp(name, r.name) < 0;
+    }
+
+    struct PlacementAttempt {
+        int numberOfBitsToSet;
+        int shift;
+        int neededBits;
+
+        PlacementAttempt(int _numberOfBitsToSet, int _shift, int _neededBits) : numberOfBitsToSet(_numberOfBitsToSet), shift(_shift), neededBits(_neededBits) {
+
+        }
+
+        std::string description() const;
+
+        friend bool operator<(const PlacementAttempt& l, const PlacementAttempt& r)
+        {
+            return  std::tie(l.neededBits, l.numberOfBitsToSet, l.shift)
+            < std::tie(r.neededBits, r.numberOfBitsToSet, r.shift);
+        }
+
+        int mask() const {
+            return (1 << neededBits) - 1;
+        }
+
+        struct PreviousMethodAddress {
+            int address;
+            int fixedBitsMask;
+        };
+
+        struct PreviousState {
+            int neededBits;
+            int shift;
+            int mask;
+            std::unordered_map<Selector*, PreviousMethodAddress> methods;
+        };
+
+        struct Result {
+            bool success;
+            PreviousState previousState;
+        };
+    };
+
+    // Possibilities we go through in the greedy backtracking algorithm findShiftsAndMask()
+    // Each class has a set (attempts()) of shift-mask possibilities, ordered, and we go through them
+    // sequentially.
+    PlacementAttempt attemptForShift(int shift, int neededBits) const;
+    std::vector<PlacementAttempt> attempts() const;
+
+    int shift;
+
+    inline int modulo() const {
+        return 1 << neededBits;
+    }
+
+    inline int mask() const {
+        return modulo() - 1;
+    }
+
+    // Attempt to place the class with the shift/mask in the attempt argument.
+    typename PlacementAttempt::Result applyAttempt(const PlacementAttempt& attempt, std::minstd_rand & randomNumberGenerator);
+
+    // Reassign the addresses as they were before we produced resultToBacktrackFrom
+    void backtrack(typename PlacementAttempt::Result& resultToBacktrackFrom);
+
+    // Did we have to grow the size of the hash table to one more bit when attempting to place it?
+    bool hadToIncreaseSize() const;
+
+    // Not const because this stomps over the slots array for checking
+    bool checkConsistency();
+
+    // Size in bytes needed for this hash table in the shared cache.
+    size_t sizeInSharedCache() const;
+
+    // Used to store the location of a flattening root's superclass, so that
+    // we can compute its final vmAddr once we are in the ObjC optimizer.
+    struct ClassLocator {
+        const char *installName;
+        unsigned segmentIndex;
+        unsigned segmentOffset;
+        bool operator==(const ClassLocator & other);
+    };
+
+    std::string_view flatteningRootName;
+    std::set<std::string_view> flattenedSuperclasses;
+    std::optional<ClassLocator> flatteningRootSuperclass;
+
+    // For debug purposes
+    bool operator==(const ClassData &other);
+};
+
+/// A unique selector. Has a name, a list of classes it's used in, and an address (which is actually an offset from
+/// the beginning of the selectors section). Due to how the placement algorithm work, it also has a current
+/// partial address and the corresponding bitmask of fixed bits. The algorithm adds bits to the partial address
+/// as it makes progress and updates the mask accordingly
+class Selector {
+public:
+    // For debug purposes
+    const char* name;
+    
+    /// Classes the selector is used in
+    std::vector<IMPCaches::ClassData*> classes;
+    
+    /// Which bits of address are already set
+    int fixedBitsMask;
+    
+    /// Current 128-byte bucket index for this selector. Only the bits in fixedBitsMask are actually frozen.
+    int inProgressBucketIndex;
+    
+    /// Full offset of the selector, including its low 7 bits (so, not the bucket's index ; inProgressBucketIndex (assuming all bits are setr by now) << 7 + some low bits)
+    int offset; // including low bits
+
+    /// Number of bits that you would need to freeze if you were to use this selector with this shift and mask.
+    int numberOfBitsToSet(int shift, int mask) const {
+        int fixedBits = (fixedBitsMask >> shift) & mask;
+        return __builtin_popcount(mask) - __builtin_popcount(fixedBits);
+    }
+
+    int numberOfSetBits() const {
+        return __builtin_popcount(fixedBitsMask);
+    }
+
+    unsigned int size() const {
+        return (unsigned int)strlen(name) + 1;
+    }
+
+    // For debug purposes
+    bool operator==(const Selector &other) const {
+        return (strcmp(name, other.name) == 0)
+            && (inProgressBucketIndex == other.inProgressBucketIndex)
+            && (fixedBitsMask == other.fixedBitsMask);
+    }
+
+    bool operator!=(const Selector &other) const {
+        return !(*this == other);
+    }
+
+};
+
+class AddressSpace;
+std::ostream& operator<<(std::ostream& o, const AddressSpace& c);
+
+struct Hole {
+    // [startAddress, endAddress[
+    int startAddress;
+    int endAddress;
+    int size() const {
+        return endAddress - startAddress;
+    }
+
+    // All our intervals are non-overlapping
+    bool operator<(const Hole& other) const {
+        auto a = std::make_tuple(size(), startAddress);
+        auto b = std::make_tuple(other.size(), other.startAddress);
+        return a < b;
+    }
+};
+
+/// Represents the holes left by the selector placement algorithm, to be filled later with other selectors we did not target.
+class HoleMap {
+public:
+    
+    /// Returns the position at which we should place a string of size `size`.
+    int addStringOfSize(unsigned size);
+    
+    /// Total size of all the holes
+    unsigned long totalHoleSize() const;
+    
+    // Empty the hole map.
+    void clear();
+
+    HoleMap();
+
+private:
+    friend class AddressSpace;
+    friend std::ostream& operator<<  (std::ostream& o, const HoleMap& m);
+
+    int endAddress = 0;
+    std::set<IMPCaches::Hole> holes;
+};
+
+// A selector that is known to be at offset 0, to let objc easily compute
+// the offset of a selector given the SEL.
+constexpr std::string_view magicSelector = "\xf0\x9f\xa4\xaf";
+
+/// This is used to place the selectors in 128-byte buckets.
+/// The "indices" below are the indices of the 128-byte bucket. To get an actual selector offset from this,
+/// we shift the index to the left by 7 bits, and assign low bits depending on the length of each selector (this happens
+/// in @see computeLowBits()).
+/// The goal of this class is to validate that selectors can actually be placed in the buckets without overflowing
+/// the 128-byte total length limit (based on the length of each individual selector)
+class AddressSpace {
+public:
+    int sizeAtIndex(int idx) const;
+    int sizeAvailableAfterIndex(int idx) const;
+    bool canPlaceMethodAtIndex(const Selector* method, int idx) const;
+    void placeMethodAtIndex(Selector* method, int idx);
+    
+    // If we decided to drop any classes, remove the selectors that were only present in them
+    void removeUninterestingSelectors();
+    
+    // Once all selectors are placed in their 128-byte buckets,
+    // actually assign the low 7 bits for each, and make a map of the
+    // holes so that we can fill them with other selectors later.
+    void computeLowBits(HoleMap& selectorsHoleMap) const;
+
+    std::string description() const;
+
+    static const int maximumIndex = (1 << 17) - 1;
+    static constexpr int bagSizeShift = 7;
+
+    friend std::ostream& operator<<  (std::ostream& o, const AddressSpace& c);
+private:
+    inline int bagSizeAtIndex(int idx) const {
+        static constexpr int bagSize = 1 << bagSizeShift;
+        static constexpr int bag0Size = bagSize - (magicSelector.length() + 1);
+        return idx ? bagSize : bag0Size;
+    }
+    bool canPlaceWithoutFillingOverflowCellAtIndex(int idx) const;
+    std::unordered_map<int, std::vector<Selector*>> methodsByIndex;
+    std::unordered_map<int, int> sizes;
+};
+
+/// Represents a constraint on some of the bits of an address
+/// It stores a set of allowed values for a given range of bits (shift and mask)
+class Constraint {
+public:
+    int mask;
+    int shift;
+    std::unordered_set<int> allowedValues;
+
+    Constraint intersecting(const Constraint& other) const;
+    friend std::ostream& operator << (std::ostream& o, const Constraint& c);
+
+    bool operator==(const Constraint& other) const {
+        return mask == other.mask &&
+                shift == other.shift &&
+                allowedValues == other.allowedValues;
+    }
+
+    struct Hasher {
+        size_t operator()(const IMPCaches::Constraint& c) const {
+            return c.shift << 24 | c.mask << 16 | c.allowedValues.size() << 8 | *c.allowedValues.begin();
+        }
+    };
+};
+
+/// Merges several Constraints together to generate a simplified constraint equivalent to the original set of constraints
+class ConstraintSet {
+    std::unordered_set<Constraint, Constraint::Hasher> constraints;
+
+public:
+    std::optional<Constraint> mergedConstraint;
+
+    bool add(const Constraint& c);
+    void clear();
+};
+
+class SelectorMap {
+public:
+    using UnderlyingMap = std::map<std::string_view, std::unique_ptr<IMPCaches::Selector>>;
+    UnderlyingMap map;
+    SelectorMap();
+};
+
+// Implemented in OptimizerObjC
+size_t sizeForImpCacheWithCount(int entries);
+
+struct ClassKey {
+    std::string_view name;
+    bool metaclass;
+
+    size_t hash() const {
+        std::size_t seed = 0;
+        seed ^= std::hash<std::string_view>()(name) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+        seed ^= std::hash<bool>()(metaclass) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+        return seed;
+    }
+
+    bool operator==(const ClassKey &other) const {
+        return (name == other.name) && (metaclass == other.metaclass);
+    }
+};
+
+struct ClassKeyHasher {
+    size_t operator()(const ClassKey& k) const {
+        return k.hash();
+    }
+};
+
+}
+
+
+#endif /* IMPCaches_hpp */

dyld3/shared-cache/IMPCachesBuilder.hpp

@@ -0,0 +1,186 @@
+//
+//  IMPCachesBuilder.hpp
+//  dyld
+//
+//  Created by Thomas Deniau on 20/01/2020.
+//
+
+#pragma once
+
+#include "CacheBuilder.h"
+#include "IMPCaches.hpp"
+#include <random>
+
+namespace IMPCaches {
+
+class IMPCachesBuilder {
+public:
+    SelectorMap selectors;
+
+    /// Parses source dylibs to figure out which methods end up in which caches
+    bool parseDylibs(Diagnostics& diag);
+
+    /// Builds a map of the class hierarchy across all dylibs. This is especially used to resolve
+    /// cross-dylib dependencies for superclasses and categories.
+    void buildClassesMap(Diagnostics& diag);
+
+    /// The entry point of the algorithm
+    void buildPerfectHashes(HoleMap& holeMap, Diagnostics& diag);
+
+    /// Regenerate the hole map if we needed to evict dylibs.
+    void computeLowBits(HoleMap& holeMap);
+    
+    // Total size the hash tables will need.
+    size_t totalIMPCachesSize() const;
+
+    void clear() {
+        // FIXME: Implement this
+    }
+
+    /** Classes for which we want to generate IMP caches according to the input JSON config laid down by OrderFiles
+     * The value is the index of the class in the json file (they are ordered by decreasing order of importance)
+     * This isn't just a vector because we also need to test for membership quickly */
+    std::unordered_map<std::string_view, int> neededClasses;
+    std::unordered_map<std::string_view, int> neededMetaclasses;
+
+    // Classes for which we don't generate IMP caches, but which we need to track
+    // to attach categories to them and find the right implementation for
+    // inlined selectors
+    std::unordered_set<std::string_view> trackedClasses;
+    std::unordered_set<std::string_view> trackedMetaclasses;
+
+    // List of classes with the same name that appear in different images.
+    // We should not try to play with fire and try to support duplicated
+    // classes in IMP caches.
+
+    using ClassSet = std::unordered_set<ClassKey, ClassKeyHasher>;
+    ClassSet duplicateClasses;
+
+    /// Selectors which we want to inline into child classes' caches.
+    std::unordered_set<std::string_view> selectorsToInline;
+    std::vector<const Selector*> inlinedSelectors;
+
+    // Class hierarchies to flatten:
+    // In every class, include every selector including
+    // the ones from superclasses up to the flattening root.
+    // This lets us enable constant caches for some of the classes which are not leaves.
+    // We avoid the pyramid of doom by making sure selectors from superclasses are
+    // included in child caches, up until some flattening root, and msgSend will
+    // fallback to the superclass of the flattening root if it can't find the selector
+    // it expects.
+    std::unordered_set<std::string_view> metaclassHierarchiesToFlatten;
+    std::unordered_set<std::string_view> classHierarchiesToFlatten;
+
+    /// All the dylibs the algorith, works on.
+    std::vector<CacheBuilder::DylibInfo> & dylibs;
+
+    IMPCachesBuilder(std::vector<CacheBuilder::DylibInfo>& allDylibs, const dyld3::json::Node& optimizerConfiguration, Diagnostics& diag, TimeRecorder& timeRecorder, const dyld3::closure::FileSystem& fileSystem);
+
+    struct ObjCClass {
+
+        const dyld3::MachOAnalyzer* superclassMA = nullptr;
+        const uint8_t*      metaClass           = nullptr;
+        const uint8_t*      superclass          = nullptr;
+        uint64_t            methodListVMaddr    = 0;
+        const char*         className           = nullptr;
+        bool                isRootClass         = false;
+        bool                isMetaClass         = false;
+
+        const IMPCaches::ClassData::ClassLocator superclassLocator() const {
+            uint64_t loadAddress = superclassMA->preferredLoadAddress();
+            __block std::optional<unsigned> segmentIndex;
+            __block std::optional<unsigned> segmentOffset;
+            uint64_t superclassVMAddr = (superclass - (const uint8_t*)superclassMA) + loadAddress;
+            superclassMA->forEachSegment(^(const dyld3::MachOAnalyzer::SegmentInfo &info, bool &stop) {
+                if (info.vmAddr <= superclassVMAddr && (info.vmAddr + info.vmSize) > superclassVMAddr) {
+                    segmentIndex = info.segIndex;
+                    segmentOffset = (unsigned)(superclassVMAddr - info.vmAddr);
+                    stop = true;
+                    return;
+                }
+            });
+            assert(segmentIndex.has_value() && segmentOffset.has_value());
+            return IMPCaches::ClassData::ClassLocator {
+                .installName = superclassMA->installName(),
+                .segmentIndex = *segmentIndex,
+                .segmentOffset = *segmentOffset,
+            };
+        }
+    };
+    struct ObjCCategory {
+        const dyld3::MachOAnalyzer* classMA = nullptr;
+        const uint8_t* cls = nullptr;
+    };
+
+private:
+
+    /// Is this a class for which we want to generate an IMP cache?
+    bool isClassInteresting(const ObjCClass& theClass) const;
+
+    /// Is this a class for which we want to generate an IMP cache, or on which we want to track method attachment by categories?
+    bool isClassInterestingOrTracked(const ObjCClass& theClass) const;
+
+    /// Adds a method to a given class's cache.
+    void addMethod(IMPCaches::ClassData* classDataPtr, const char* methodName, const char* installName, const char* className, const char* catName, bool inlined, bool fromFlattening);
+
+    /// Inline a method from a parent's cache to a child's cache.
+    void inlineMethodIfNeeded(IMPCaches::ClassData* classToInlineIn, const char* classToInlineFrom, const char* catToInlineFrom, const char* installNameToInlineFrom, const char* name, std::set<Selector*> & seenSelectors, bool isFlattening);
+
+    /// Map from location (address in the mmapped source dylib) to class info built by buildClassesMap()
+    std::unordered_map<const uint8_t*, ObjCClass> objcClasses;
+
+    /// Map from location (address in the mmapped source dylib) to category info built by buildClassesMap()
+    std::unordered_map<const uint8_t*, ObjCCategory> objcCategories;
+
+    /// Address space where the selectors are going to live. Used to determine at which address we'll actually layout each selector.
+    IMPCaches::AddressSpace addressSpace;
+
+    /// Find a shift and a mask for each class. Returns the number of classes that we could not place.
+    int findShiftsAndMasks();
+    
+    /// Shuffles selectors around to satisfy size constraints.  Returns the number of classes that we could not place.
+    int solveGivenShiftsAndMasks();
+
+    /// Determine classes we need to track category attachment on (for later inlining)
+    void buildTrackedClasses(CacheBuilder::DylibInfo& dylib, const dyld3::MachOAnalyzer* ma);
+
+    /// Parses the method lists in the source dylibs to determine what will end up in which IMP cache.
+    void populateMethodLists(CacheBuilder::DylibInfo& dylib, const dyld3::MachOAnalyzer* ma, int* duplicateClassCount);
+
+    /// Go through categories and add the methods from the category to the corresponding class's IMP cache
+    void attachCategories(CacheBuilder::DylibInfo& dylib, const dyld3::MachOAnalyzer* ma);
+
+    // Inline some selectors (driven by the OrderFiles) from parent caches into child caches.
+    void inlineSelectors(CacheBuilder::DylibInfo& dylib, std::unordered_map<std::string_view, CacheBuilder::DylibInfo*> & dylibsByInstallName, const dyld3::MachOAnalyzer* ma);
+
+    void fillAllClasses(std::vector<IMPCaches::ClassData*> & allClasses);
+    void fillAllMethods(std::vector<IMPCaches::Selector*> & allMethods);
+    void removeUninterestingClasses();
+
+    struct TargetClassFindingResult {
+        bool success;
+        const dyld3::MachOLoaded* foundInDylib;
+        const uint8_t* location;
+    };
+
+    struct BindTarget {
+        std::string                 symbolName      = "";
+        const dyld3::MachOAnalyzer* targetDylib     = nullptr;
+        bool                        isWeakImport    = false;
+    };
+
+    struct DylibAndDeps {
+        const dyld3::MachOAnalyzer*         ma = nullptr;
+        __block std::vector<std::string>    dependentLibraries;
+    };
+
+    TargetClassFindingResult findTargetClass(const dyld3::MachOAnalyzer* ma, uint64_t targetClassVMAddr, uint64_t targetClassPointerVMAddr, const char* logContext, const std::unordered_map<uint64_t, uint64_t> & bindLocations, std::vector<BindTarget> & bindTargets, std::unordered_map<std::string, DylibAndDeps> &dylibMap);
+
+    const std::string * nameAndIsMetaclassPairFromNode(const dyld3::json::Node & node, bool* metaclass);
+
+    Diagnostics& _diagnostics;
+    TimeRecorder& _timeRecorder;
+    const dyld3::closure::FileSystem& _fileSystem;
+};
+
+}

dyld3/shared-cache/MachOFileAbstraction.hpp

@@ -49,6 +49,19 @@
 #define DYLD_CACHE_ADJ_V2_IMAGE_OFF_32			0x0C
 #define DYLD_CACHE_ADJ_V2_THREADED_POINTER_64   0x0D
 
+#ifndef LC_FILESET_ENTRY
+#define LC_FILESET_ENTRY      (0x35 | LC_REQ_DYLD) /* used with fileset_entry_command */
+struct fileset_entry_command {
+    uint32_t        cmd;        /* LC_FILESET_ENTRY */
+    uint32_t        cmdsize;    /* includes id string */
+    uint64_t        vmaddr;     /* memory address of the dylib */
+    uint64_t        fileoff;    /* file offset of the dylib */
+    union lc_str    entry_id;   /* contained entry id */
+    uint32_t        reserved;   /* entry_id is 32-bits long, so this is the reserved padding */
+};
+#endif
+
+
 #include "FileAbstraction.hpp"
 //#include "Architectures.hpp"
 
@@ -775,8 +788,12 @@ public:
             }
         }
 
-		if (strcmp(segname, "__DATA") == 0)
-			return getSection("__DATA_CONST", sectname);
+        if (strcmp(segname, "__DATA") == 0) {
+            if (const macho_section<P>* dataConst = getSection("__DATA_CONST", sectname))
+                return dataConst;
+            if (const macho_section<P>* dataDirty = getSection("__DATA_DIRTY", sectname))
+                return dataDirty;
+        }
         return NULL;
     }
 
@@ -849,6 +866,77 @@ private:
 	dyld_info_command	fields;
 };
 
+
+//
+// mach-o build version load command
+//
+template <typename P>
+class macho_build_version_command {
+public:
+	uint32_t		cmd() const                             INLINE { return E::get32(fields.cmd); }
+	void			set_cmd(uint32_t value)                 INLINE { E::set32(fields.cmd, value); }
+
+	uint32_t		cmdsize() const							INLINE { return E::get32(fields.cmdsize); }
+	void			set_cmdsize(uint32_t value)				INLINE { E::set32(fields.cmdsize, value); }
+
+    uint32_t        platform() const                        INLINE { return E::get32(fields.platform); }
+    void            set_platform(uint32_t value)            INLINE { E::set32(fields.platform, value); }
+
+    uint32_t        minos() const                           INLINE { return E::get32(fields.minos); }
+    void            set_minos(uint32_t value)               INLINE { E::set32(fields.minos, value); }
+
+    uint32_t        sdk() const                             INLINE { return E::get32(fields.sdk); }
+    void            set_sdk(uint32_t value)                 INLINE { E::set32(fields.sdk, value); }
+
+    uint32_t        ntools() const                          INLINE { return E::get32(fields.ntools); }
+    void            set_ntools(uint32_t value)              INLINE { E::set32(fields.ntools, value); }
+
+	typedef typename P::E		E;
+private:
+	build_version_command	fields;
+};
+
+//
+// mach-o routines load command
+//
+template <typename P> struct macho_fileset_entry_command_content {};
+template <> struct macho_fileset_entry_command_content<Pointer32<BigEndian> >      { fileset_entry_command		fields; enum { CMD = LC_FILESET_ENTRY	}; };
+template <> struct macho_fileset_entry_command_content<Pointer64<BigEndian> >	    { fileset_entry_command	    fields; enum { CMD = LC_FILESET_ENTRY	}; };
+template <> struct macho_fileset_entry_command_content<Pointer32<LittleEndian> >   { fileset_entry_command		fields; enum { CMD = LC_FILESET_ENTRY	}; };
+template <> struct macho_fileset_entry_command_content<Pointer64<LittleEndian> >   { fileset_entry_command	    fields; enum { CMD = LC_FILESET_ENTRY	}; };
+
+template <typename P>
+class macho_fileset_entry_command {
+public:
+	uint32_t		cmd() const							INLINE { return E::get32(cache_entry_id.fields.cmd); }
+	void			set_cmd(uint32_t value)				INLINE { E::set32(cache_entry_id.fields.cmd, value); }
+
+	uint32_t		cmdsize() const						INLINE { return E::get32(cache_entry_id.fields.cmdsize); }
+	void			set_cmdsize(uint32_t value)			INLINE { E::set32(cache_entry_id.fields.cmdsize, value); }
+
+	uint64_t		vmaddr() const				        INLINE { return P::getP(cache_entry_id.fields.vmaddr); }
+	void			set_vmaddr(uint64_t value)	        INLINE { P::setP(cache_entry_id.fields.vmaddr, value); }
+
+	uint64_t		fileoff() const				        INLINE { return P::getP(cache_entry_id.fields.fileoff); }
+	void			set_fileoff(uint64_t value)	        INLINE { P::setP(cache_entry_id.fields.fileoff, value); }
+
+    uint32_t        entry_id_offset() const             INLINE { return E::get32(cache_entry_id.fields.entry_id.offset); }
+    void            set_entry_id_offset(uint32_t value) INLINE { E::set32(cache_entry_id.fields.entry_id.offset, value);  }
+
+    const char*     entry_id() const                    INLINE { return (const char*)&cache_entry_id.fields + entry_id_offset(); }
+    void            set_entry_id(const char* value)     INLINE {
+        set_entry_id_offset(sizeof(cache_entry_id.fields));
+        strcpy(((char*)&cache_entry_id.fields) + sizeof(cache_entry_id.fields), value);
+    }
+
+	typedef typename P::E		E;
+	enum {
+		CMD = macho_fileset_entry_command_content<P>::CMD
+	};
+private:
+	macho_fileset_entry_command_content<P>	cache_entry_id;
+};
+
 #ifndef NO_ULEB 
 inline uint64_t read_uleb128(const uint8_t*& p, const uint8_t* end) {
 	uint64_t result = 0;
@@ -884,7 +972,7 @@ inline int64_t read_sleb128(const uint8_t*& p, const uint8_t* end)
 		bit += 7;
 	} while (byte & 0x80);
 	// sign extend negative numbers
-	if ( (byte & 0x40) != 0 )
+    if ( ((byte & 0x40) != 0) && (bit < 64) )
 		result |= (~0ULL) << bit;
 	return result;
 }

dyld3/shared-cache/Manifest.h

@@ -1,264 +0,0 @@
-/*
- * Copyright (c) 2017 Apple Inc. All rights reserved.
- *
- * @APPLE_LICENSE_HEADER_START@
- *
- * This file contains Original Code and/or Modifications of Original Code
- * as defined in and that are subject to the Apple Public Source License
- * Version 2.0 (the 'License'). You may not use this file except in
- * compliance with the License. Please obtain a copy of the License at
- * http://www.opensource.apple.com/apsl/ and read it before using this
- * file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
- * Please see the License for the specific language governing rights and
- * limitations under the License.
- *
- * @APPLE_LICENSE_HEADER_END@
- */
-
-
-#ifndef Manifest_h
-#define Manifest_h
-
-#include <map>
-#include <set>
-#include <memory>
-#include <string>
-#include <vector>
-
-#include <unordered_map>
-#include <unordered_set>
-
-#include <assert.h>
-#include <uuid/uuid.h>
-
-#import <Foundation/Foundation.h>
-
-#include "DyldSharedCache.h"
-#include "Diagnostics.h"
-#include "MachOAnalyzer.h"
-#include "ClosureFileSystemPhysical.h"
-
-extern std::string toolDir();
-
-namespace dyld3 {
-
-struct VIS_HIDDEN UUID {
-    UUID() {}
-    UUID(const UUID& other) { uuid_copy(_bytes, other._bytes); }
-    UUID(const uuid_t other) { uuid_copy(&_bytes[0], other); }
-    UUID(const dyld3::MachOAnalyzer* ml) { ml->getUuid(_bytes); }
-    bool operator<(const UUID& other) const { return uuid_compare(_bytes, other._bytes) < 0; }
-    bool operator==(const UUID& other) const { return uuid_compare(_bytes, other._bytes) == 0; }
-    bool operator!=(const UUID& other) const { return !(*this == other); }
-
-    size_t hash() const
-    {
-        size_t retval = 0;
-        for (size_t i = 0; i < (16 / sizeof(size_t)); ++i) {
-            retval ^= ((size_t*)(&_bytes[0]))[i];
-        }
-        return retval;
-    }
-    const unsigned char* get() const { return _bytes; };
-
-private:
-    uuid_t _bytes;
-};
-
-struct BuildQueueEntry {
-    DyldSharedCache::CreateOptions            options;
-    dyld3::closure::FileSystemPhysical        fileSystem;
-    std::vector<DyldSharedCache::MappedMachO> dylibsForCache;
-    std::vector<DyldSharedCache::MappedMachO> otherDylibsAndBundles;
-    std::vector<DyldSharedCache::MappedMachO> mainExecutables;
-    std::string                               outputPath;
-    std::set<std::string>                     configNames;
-};
-
-struct Manifest {
-    struct UUIDInfo {
-        const MachOAnalyzer* mh;
-        uint64_t sliceFileOffset;
-        std::size_t size;
-        std::string runtimePath;
-        std::string buildPath;
-        std::string installName;
-        std::string arch;
-        UUID uuid;
-        UUIDInfo(const MachOAnalyzer* M, std::size_t S, uint64_t SO, UUID U, std::string A, std::string RP, std::string BP, std::string IN)
-            : mh(M), size(S), arch(A), uuid(U), runtimePath(RP), buildPath(BP), installName(IN), sliceFileOffset(SO) {}
-        UUIDInfo() : UUIDInfo(nullptr, 0, 0, UUID(), "", "", "", "") {}
-    };
-
-    struct Project {
-        std::vector<std::string> sources;
-    };
-
-
-    struct SegmentInfo {
-        std::string name;
-        uint64_t    startAddr;
-        uint64_t    endAddr;
-    };
-
-    struct CacheInfo {
-        std::vector<SegmentInfo> regions;
-        std::string              cdHash;
-    };
-
-    struct CacheImageInfo {
-        bool                     included;
-        std::string              exclusionInfo;
-        UUID                     uuid;
-        std::string              installname;
-        std::vector<SegmentInfo> segments;
-        CacheImageInfo(void)
-            : included(true)
-        {
-        }
-    };
-
-    struct Results {
-        std::string failure;
-        std::map<UUID, CacheImageInfo> dylibs;
-        std::map<UUID, CacheImageInfo> bundles;
-        std::map<UUID, CacheImageInfo> executables;
-
-        std::set<std::string> warnings;
-        CacheInfo             developmentCache;
-        CacheInfo             productionCache;
-        CacheImageInfo& dylibForInstallname(const std::string& installname);
-        void exclude(const dyld3::MachOAnalyzer* ml, const std::string& reason);
-        void exclude(Manifest& manifest, const UUID& uuid, const std::string& reason);
-    };
-
-    struct Architecture {
-        mutable Results     results;
-
-        bool operator==(const Architecture& O) const;
-        bool operator!=(const Architecture& other) const;
-    };
-
-    struct Configuration {
-        std::string                         platformName;
-        std::string                         device;
-        std::string                         disposition;
-        std::string                         metabomTag;
-        std::set<std::string>               metabomTags;
-        std::set<std::string>               metabomExcludeTags;
-        std::set<std::string>               metabomRestrictTags;
-        std::set<std::string>               restrictedInstallnames;
-        std::map<std::string, Architecture> architectures;
-
-        bool operator==(const Configuration& O) const;
-        bool operator!=(const Configuration& other) const;
-        const Architecture& architecture(const std::string& architecture) const;
-        void forEachArchitecture(std::function<void(const std::string& archName)> lambda) const;
-    };
-
-    const std::map<std::string, Project>& projects();
-    const Configuration& configuration(const std::string& configuration) const;
-    void forEachConfiguration(std::function<void(const std::string& configName)> lambda) const;
-
-    void addProjectSource(const std::string& project, const std::string& source, bool first = false);
-
-    const std::string projectPath(const std::string& projectName);
-    const bool        empty(void);
-    const std::string dylibOrderFile() const;
-    void setDylibOrderFile(const std::string& dylibOrderFile);
-
-    const std::string dirtyDataOrderFile() const;
-    void setDirtyDataOrderFile(const std::string& dirtyDataOrderFile);
-
-    const std::string metabomFile() const;
-    void setMetabomFile(const std::string& metabomFile);
-
-    const Platform platform() const;
-    void setPlatform(const Platform platform);
-
-    const std::string& build() const;
-    void setBuild(const std::string& build);
-    const uint32_t version() const;
-    void setVersion(const uint32_t manifestVersion);
-    bool normalized;
-
-    Manifest(Diagnostics& D, const std::string& path, bool populateIt = true);
-    void populate(const std::set<std::string>& overlays);
-
-    BuildQueueEntry makeQueueEntry(const std::string& outputPath, const std::set<std::string>& configs, const std::string& arch, bool optimizeStubs, const std::string& prefix,
-                                   bool isLocallyBuiltCache, bool skipWrites, bool verbose);
-
-    void write(const std::string& path);
-    void writeJSON(const std::string& path);
-    void        canonicalize(void);
-    void        calculateClosure();
-    const MachOAnalyzer* machOForUUID(const UUID& uuid) const;
-    const std::string buildPathForUUID(const UUID& uuid);
-    const std::string runtimePathForUUID(const UUID& uuid);
-    const std::string& installNameForUUID(const UUID& uuid);
-    DyldSharedCache::MappedMachO machoForPathAndArch(const std::string& path, const std::string& arch) const;
-    void remove(const std::string& config, const std::string& arch);
-    void runConcurrently(dispatch_queue_t queue, dispatch_semaphore_t concurrencyLimitingSemaphore, std::function<void(const std::string configuration, const std::string architecture)> lambda);
-    bool filterForConfig(const std::string& configName);
-    std::set<std::string> resultsForConfiguration(const std::string& configName);
-
-    // These are used by MRM to support having the Manifest give us a list of files/symlinks from the BOM but we use MRM for the actual cache generation
-    void forEachMachO(std::string configuration, std::function<void(const std::string &buildPath, const std::string &runtimePath, const std::string &arch, bool shouldBeExcludedIfLeaf)> lambda);
-
-    void forEachSymlink(std::string configuration, std::function<void(const std::string &fromPath, const std::string &toPath)> lambda);
-
-private:
-    NSDictionary*    _manifestDict;
-    Diagnostics&      _diags;
-    std::map<UUID, UUIDInfo> _uuidMap;
-    std::map<std::pair<std::string, std::string>, UUID> _installNameMap;
-    std::vector<std::pair<std::string, std::string>>    _symlinks;
-    static dispatch_queue_t _identifierQueue;
-    uint32_t                _manifestVersion;
-    std::string             _build;
-    std::string             _dylibOrderFile;
-    std::string             _dirtyDataOrderFile;
-    std::string             _metabomFile;
-    Platform                _platform;
-    std::map<std::string, Project>               _projects;
-    std::map<std::string, Configuration>         _configurations;
-    std::map<std::string, std::set<std::string>> _metabomTagMap;
-    std::map<std::string, std::set<std::string>> _metabomSymlinkTagMap;
-    std::map<std::string, std::set<std::string>> _metabomExcludeTagMap;
-    std::map<std::string, std::set<std::string>> _metabomRestrictedTagMap;
-
-    std::vector<DyldSharedCache::MappedMachO> dylibsForCache(const std::string& configuration, const std::string& architecture);
-    std::vector<DyldSharedCache::MappedMachO> otherDylibsAndBundles(const std::string& configuration, const std::string& architecture);
-    std::vector<DyldSharedCache::MappedMachO> mainExecutables(const std::string& configuration, const std::string& architecture);
-
-    const UUIDInfo& infoForUUID(const UUID& uuid) const;
-    const UUIDInfo infoForInstallNameAndarch(const std::string& installName, const std::string arch) const;
-    void insert(std::vector<DyldSharedCache::MappedMachO>& mappedMachOs, const CacheImageInfo& imageInfo);
-    bool loadParser(const void* p, size_t sliceLength, uint64_t sliceOffset, const std::string& runtimePath, const std::string& buildPath, const std::set<std::string>& architectures);
-    bool loadParsers(const std::string& pathToMachO, const std::string& runtimePath, const std::set<std::string>& architectures);
-    void dedupeDispositions();
-    void calculateClosure(const std::string& configuration, const std::string& architecture);
-    void canonicalizeDylib(const std::string& installname);
-    template <typename P>
-    void canonicalizeDylib(const std::string& installname, const uint8_t* p);
-    void addImplicitAliases(void);
-};
-}
-
-namespace std {
-template <>
-struct hash<dyld3::UUID> {
-    size_t operator()(const dyld3::UUID& x) const
-    {
-        return x.hash();
-    }
-};
-}
-
-#endif /* Manifest_h */

dyld3/shared-cache/ObjC2Abstraction.hpp

@@ -44,7 +44,7 @@ struct entsize_iterator {
     entsize_iterator() { } 
     
     entsize_iterator(const Tlist& list, uint32_t start = 0)
-        : entsize(list.getEntsize()), index(start), current(&list.get(start)) 
+        : entsize(list.getEntsize()), index(start), current((T*)list.get(start))
     { }
     
     const entsize_iterator<P,T,Tlist>& operator += (ptrdiff_t count) {
@@ -152,101 +152,403 @@ public:
     }
 };
 
-
 template <typename P>
-class objc_method_list_t;  // forward reference
+class objc_method_list_t {
 
-
-template <typename P>
-class objc_method_t {
     typedef typename P::uint_t pint_t;
+
+    template <typename PtrTy>
+    class objc_method_small_t {
+        typedef typename PtrTy::uint_t pint_t;
+        int32_t name;   // SEL
+        int32_t types;  // const char *
+        int32_t imp;    // IMP
+        friend class objc_method_list_t<PtrTy>;
+
+        objc_method_small_t() = delete;
+        ~objc_method_small_t() = delete;
+        objc_method_small_t(const objc_method_small_t& other) = delete;
+        objc_method_small_t(objc_method_small_t&& other) = delete;
+        objc_method_small_t& operator=(const objc_method_small_t& other) = delete;
+        objc_method_small_t& operator=(objc_method_small_t&& other) = delete;
+
+    public:
+
+        pint_t getName(ContentAccessor* cache, bool isOffsetToSel) const {
+            // We want to return the VM address of the "const char*" our selector
+            // reference is pointing at.
+            pint_t* nameRef = (pint_t*)((uint8_t*)&name + name);
+            if ( isOffsetToSel ) {
+                // Offset is directly to the SEL, not a selRef
+                return (pint_t)cache->vmAddrForContent(nameRef);
+            } else {
+                return (pint_t)PtrTy::getP(*nameRef);
+            }
+        }
+        // We want to update the selRef we are pointing at with the new content
+        // We may share the same selRef with other method lists or @SEL expressions, but as
+        // all of them want the same uniqued selector anyway, its safe to overwrite it here for
+        // everyone.
+        void setName(ContentAccessor* cache, pint_t newNameVMAddr, bool isOffsetToSel) {
+            if ( isOffsetToSel ) {
+                // Offset is directly to the SEL, not a selRef
+                void* namePtr = cache->contentForVMAddr(newNameVMAddr);
+                this->name = (int32_t)(intptr_t)((uint8_t*)namePtr - (uint8_t*)&this->name);
+            } else {
+                pint_t* selRef = (pint_t*)((uint8_t*)&name + name);
+                PtrTy::setP(*selRef, newNameVMAddr);
+            }
+        }
+        // Returns the vmAddr of the types
+        pint_t getTypes(ContentAccessor* cache) const {
+            pint_t* typesRef = (pint_t*)((uint8_t*)&types + types);
+            return (pint_t)cache->vmAddrForContent(typesRef);
+        }
+        void setTypes(ContentAccessor* cache, pint_t newTypesVMAddr) {
+            void* typesPtr = cache->contentForVMAddr(newTypesVMAddr);
+            this->types = (int32_t)(intptr_t)((uint8_t*)typesPtr - (uint8_t*)&this->types);
+        }
+        // Returns the vmAddr of the IMP
+        pint_t getIMP(ContentAccessor* cache) const {
+            pint_t* impRef = (pint_t*)((uint8_t*)&imp + imp);
+            return (pint_t)cache->vmAddrForContent(impRef);
+        }
+        void setIMP(ContentAccessor* cache, pint_t newIMPVMAddr) {
+            void* impPtr = cache->contentForVMAddr(newIMPVMAddr);
+            this->imp = (int32_t)(intptr_t)((uint8_t*)impPtr - (uint8_t*)&this->imp);
+        }
+
+        // Swap the contents of this value and other
+        // This has to recompute all of the relative offsets
+        void swap(objc_method_small_t<PtrTy>* other) {
+            // Get our targets
+            uint8_t* ourNameTarget  = (uint8_t*)&this->name + this->name;
+            uint8_t* ourTypesTarget = (uint8_t*)&this->types + this->types;
+            uint8_t* ourIMPTarget   = (uint8_t*)&this->imp + this->imp;
+            // Get their targets
+            uint8_t* theirNameTarget  = (uint8_t*)&other->name + other->name;
+            uint8_t* theirTypesTarget = (uint8_t*)&other->types + other->types;
+            uint8_t* theirIMPTarget   = (uint8_t*)&other->imp + other->imp;
+            // Set our targets
+            this->name = (int32_t)(intptr_t)(theirNameTarget - (uint8_t*)&this->name);
+            this->types = (int32_t)(intptr_t)(theirTypesTarget - (uint8_t*)&this->types);
+            this->imp = (int32_t)(intptr_t)(theirIMPTarget - (uint8_t*)&this->imp);
+            // Set their targets
+            other->name = (int32_t)(intptr_t)(ourNameTarget - (uint8_t*)&other->name);
+            other->types = (int32_t)(intptr_t)(ourTypesTarget - (uint8_t*)&other->types);
+            other->imp = (int32_t)(intptr_t)(ourIMPTarget - (uint8_t*)&other->imp);
+        }
+
+        struct SortBySELAddress :
+            public std::binary_function<const objc_method_small_t<PtrTy>&,
+                                        const objc_method_small_t<PtrTy>&, bool>
+        {
+            SortBySELAddress(ContentAccessor* cache, bool isOffsetToSel)
+                : cache(cache), isOffsetToSel(isOffsetToSel) { }
+
+            bool operator() (const objc_method_small_t<PtrTy>& lhs,
+                             const objc_method_small_t<PtrTy>& rhs)
+            {
+                return lhs.getName(cache, isOffsetToSel) < rhs.getName(cache, isOffsetToSel);
+            }
+
+            ContentAccessor* cache  = nullptr;
+            bool isOffsetToSel      = false;
+        };
+    };
+
+    template <typename PtrTy>
+    class objc_method_large_t {
+        typedef typename PtrTy::uint_t pint_t;
         pint_t name;   // SEL
         pint_t types;  // const char *
         pint_t imp;    // IMP
-    friend class objc_method_list_t<P>;
-public:
-    pint_t getName() const { return (pint_t)P::getP(name); }
-    void setName(pint_t newName) { P::setP(name, newName); }
+        friend class objc_method_list_t<PtrTy>;
+    public:
+        pint_t getName() const {
+            return (pint_t)PtrTy::getP(name);
+        }
+        void setName(pint_t newName) {
+            PtrTy::setP(name, newName);
+        }
+        pint_t getTypes() const {
+            return (pint_t)PtrTy::getP(types);
+        }
+        void setTypes(pint_t newTypes) {
+            PtrTy::setP(types, newTypes);
+        }
+        pint_t getIMP() const {
+            return (pint_t)PtrTy::getP(imp);
+        }
+        void setIMP(pint_t newIMP) {
+            PtrTy::setP(imp, newIMP);
+        }
 
         struct SortBySELAddress :
-        public std::binary_function<const objc_method_t<P>&, 
-                                    const objc_method_t<P>&, bool>
+            public std::binary_function<const objc_method_large_t<PtrTy>&,
+                                        const objc_method_large_t<PtrTy>&, bool>
         {
-        bool operator() (const objc_method_t<P>& lhs, 
-                         const objc_method_t<P>& rhs)
+            bool operator() (const objc_method_large_t<PtrTy>& lhs,
+                             const objc_method_large_t<PtrTy>& rhs)
             {
                 return lhs.getName() < rhs.getName();
             }
         };
-};
+    };
+
+    // Temporary struct to use when sorting small methods as their int32_t offsets can't reach
+    // from the stack where temporary values are placed, in to the shared cache buffer where the data lives
+    struct TempMethod {
+        // Relative methods in the shared cache always use direct offsets to the SEL
+        // at the point where this is running.  That means we don't need to indirect through
+        // a SEL reference.
+        pint_t selVMAddr;
+        pint_t typesVMAddr;
+        pint_t impVMAddr;
+    };
+
+    template <typename PtrTy>
+    struct SortBySELAddress :
+        public std::binary_function<const TempMethod&,
+                                    const TempMethod&, bool>
+    {
+        SortBySELAddress(ContentAccessor* cache) : cache(cache) { }
+
+        bool operator() (const TempMethod& lhs,
+                         const TempMethod& rhs)
+        {
+            return lhs.selVMAddr < rhs.selVMAddr;
+        }
+
+        ContentAccessor* cache = nullptr;
+    };
 
-template <typename P>
-class objc_method_list_t {
     uint32_t entsize;
     uint32_t count;
-    objc_method_t<P> first;
+    union {
+        objc_method_small_t<P> small;
+        objc_method_large_t<P> large;
+    } first;
 
     void* operator new (size_t, void* buf) { return buf; }
 
-public:
+    enum : uint32_t {
+        // If this is set, the relative method lists name_offset field is an
+        // offset directly to the SEL, not a SEL ref.
+        relativeMethodSelectorsAreDirectFlag    = 0x40000000,
 
-    typedef entsize_iterator<P, objc_method_t<P>, objc_method_list_t<P> > method_iterator;
+        // If this is set, then method lists are the new relative format, not
+        // the old pointer based format
+        relativeMethodFlag                      = 0x80000000,
+
+        // The upper 16-bits are all defined to be flags
+        methodListFlagsMask                     = 0xFFFF0000
+    };
+
+    uint32_t getFlags() const {
+        return (P::E::get32(entsize) & methodListFlagsMask);
+    }
+
+    typedef entsize_iterator<P, objc_method_small_t<P>, objc_method_list_t<P> > small_method_iterator;
+    typedef entsize_iterator<P, objc_method_large_t<P>, objc_method_list_t<P> > large_method_iterator;
+
+    small_method_iterator beginSmall() {
+        assert(usesRelativeMethods());
+        return small_method_iterator(*this, 0);
+    }
+    small_method_iterator endSmall() {
+        assert(usesRelativeMethods());
+        return small_method_iterator(*this, getCount());
+    }
+
+    large_method_iterator beginLarge() {
+        assert(!usesRelativeMethods());
+        return large_method_iterator(*this, 0);
+    }
+    large_method_iterator endLarge() {
+        assert(!usesRelativeMethods());
+        return large_method_iterator(*this, getCount());
+    }
+
+public:
 
     uint32_t getCount() const { return P::E::get32(count); }
 
-    uint32_t getEntsize() const {return P::E::get32(entsize)&~(uint32_t)3;}
-
-    objc_method_t<P>& get(uint32_t i) const { return *(objc_method_t<P> *)((uint8_t *)&first + i * getEntsize()); }
+    uint32_t getEntsize() const {
+        return P::E::get32(entsize) & ~(uint32_t)3 & ~methodListFlagsMask;
+    }
 
     uint32_t byteSize() const { 
         return byteSizeForCount(getCount(), getEntsize()); 
     }
 
-    static uint32_t byteSizeForCount(uint32_t c, uint32_t e = sizeof(objc_method_t<P>)) { 
-        return sizeof(objc_method_list_t<P>) - sizeof(objc_method_t<P>) + c*e;
+    static uint32_t byteSizeForCount(uint32_t c, uint32_t e) {
+        return sizeof(entsize) + sizeof(count) + c*e;
     }
 
-    method_iterator begin() { return method_iterator(*this, 0); }
-    method_iterator end() { return method_iterator(*this, getCount()); }
-    const method_iterator begin() const { return method_iterator(*this, 0); }
-    const method_iterator end() const { return method_iterator(*this, getCount()); }
+    bool usesRelativeMethods() const {
+        return (P::E::get32(entsize) & relativeMethodFlag) != 0;
+    }
 
-    void setFixedUp() { P::E::set32(entsize, getEntsize() | 3); }
+    void setFixedUp() {
+        P::E::set32(entsize, getEntsize() | 3 | getFlags());
+    }
 
-    void getPointers(std::set<void*>& pointersToRemove) {
-        for(method_iterator it = begin(); it != end(); ++it) {
-            objc_method_t<P>& entry = *it;
-            pointersToRemove.insert(&(entry.name));
-            pointersToRemove.insert(&(entry.types));
-            pointersToRemove.insert(&(entry.imp));
+    void setMethodListSelectorsAreDirect() {
+        P::E::set32(entsize, getEntsize() | getFlags() | relativeMethodSelectorsAreDirectFlag);
+    }
+
+    void sortMethods(ContentAccessor* cache, pint_t *typelist, bool isOffsetToSel) {
+        if ( usesRelativeMethods() ) {
+            // At this point we assume we are using offsets directly to selectors.  This
+            // is so that the TempMethod struct can also use direct offsets and not track the
+            // SEL reference VMAddrs
+            assert(isOffsetToSel);
+
+            if ( typelist == nullptr ) {
+                // This is the case when we are sorting the methods on a class.
+                // Only protocols have a type list which causes the other sort to be used
+                // We can't sort the small methods in place as their 32-bit offsets can't reach
+                // the VM space where the shared cache is being created.  Instead create a list
+                // of large methods and sort those.
+
+                std::vector<TempMethod> largeMethods;
+                for (unsigned i = 0 ; i != count; ++i) {
+                    const objc_method_small_t<P>* smallMethod = (const objc_method_small_t<P>*)get(i);
+                    TempMethod largeMethod;
+                    largeMethod.selVMAddr = smallMethod->getName(cache, isOffsetToSel);
+                    largeMethod.typesVMAddr = smallMethod->getTypes(cache);
+                    largeMethod.impVMAddr = smallMethod->getIMP(cache);
+                    largeMethods.push_back(largeMethod);
+                }
+
+                SortBySELAddress<P> sorter(cache);
+                std::stable_sort(largeMethods.begin(), largeMethods.end(), sorter);
+
+                for (unsigned i = 0 ; i != count; ++i) {
+                    const TempMethod& largeMethod = largeMethods[i];
+                    objc_method_small_t<P>* smallMethod = (objc_method_small_t<P>*)get(i);
+                    smallMethod->setName(cache, largeMethod.selVMAddr, isOffsetToSel);
+                    smallMethod->setTypes(cache, largeMethod.typesVMAddr);
+                    smallMethod->setIMP(cache, largeMethod.impVMAddr);
+                }
+
+#if 0
+                // Check the method lists are sorted
+                {
+                    typename objc_method_small_t<P>::SortBySELAddress sorter(cache);
+                    for (uint32_t i = 0; i < getCount(); i++) {
+                        for (uint32_t j = i+1; j < getCount(); j++) {
+                            objc_method_small_t<P>* mi = (objc_method_small_t<P>*)get(i);
+                            objc_method_small_t<P>* mj = (objc_method_small_t<P>*)get(j);
+                            if ( mi->getName(cache) == mj->getName(cache) )
+                                continue;
+                            if (! sorter(*mi, *mj)) {
+                                assert(false);
+                            }
+                        }
+                    }
+                }
+#endif
+            }
+            else {
+                typename objc_method_small_t<P>::SortBySELAddress sorter(cache, isOffsetToSel);
+                // can't easily use std::stable_sort here
+                for (uint32_t i = 0; i < getCount(); i++) {
+                    for (uint32_t j = i+1; j < getCount(); j++) {
+                        objc_method_small_t<P>* mi = (objc_method_small_t<P>*)get(i);
+                        objc_method_small_t<P>* mj = (objc_method_small_t<P>*)get(j);
+                        if (! sorter(*mi, *mj)) {
+                            mi->swap(mj);
+                            if (typelist) std::swap(typelist[i], typelist[j]);
+                        }
+                    }
+                }
+            }
+        } else {
+            typename objc_method_large_t<P>::SortBySELAddress sorter;
+
+            if ( typelist == nullptr ) {
+                // This is the case when we are sorting the methods on a class.
+                // Only protocols have a type list which causes the other sort to be used
+                std::stable_sort(beginLarge(), endLarge(), sorter);
+            }
+            else {
+                // can't easily use std::stable_sort here
+                for (uint32_t i = 0; i < getCount(); i++) {
+                    for (uint32_t j = i+1; j < getCount(); j++) {
+                        objc_method_large_t<P>* mi = (objc_method_large_t<P>*)get(i);
+                        objc_method_large_t<P>* mj = (objc_method_large_t<P>*)get(j);
+                        if (! sorter(*mi, *mj)) {
+                            std::swap(*mi, *mj);
+                            if (typelist) std::swap(typelist[i], typelist[j]);
+                        }
+                    }
+                }
+            }
+        }
+        // mark method list as sorted
+        this->setFixedUp();
+    }
+
+    pint_t getName(ContentAccessor* cache, uint32_t i, bool isOffsetToSel) {
+        pint_t name = 0;
+        if ( usesRelativeMethods() ) {
+            small_method_iterator it = beginSmall() + i;
+            objc_method_small_t<P>& method = *it;
+            name = method.getName(cache, isOffsetToSel);
+        } else {
+            large_method_iterator it = beginLarge() + i;
+            objc_method_large_t<P>& method = *it;
+            name = method.getName();
+        }
+        return name;
+    }
+
+    void setName(ContentAccessor* cache, uint32_t i, pint_t name, bool isOffsetToSel) {
+        if ( usesRelativeMethods() ) {
+            small_method_iterator it = beginSmall() + i;
+            objc_method_small_t<P>& method = *it;
+            method.setName(cache, name, isOffsetToSel);
+        } else {
+            large_method_iterator it = beginLarge() + i;
+            objc_method_large_t<P>& method = *it;
+            method.setName(name);
         }
     }
 
-    static void addPointers(uint8_t* methodList, CacheBuilder::ASLR_Tracker& aslrTracker) {
-        objc_method_list_t<P>* mlist = (objc_method_list_t<P>*)methodList;
-        for(method_iterator it = mlist->begin(); it != mlist->end(); ++it) {
-            objc_method_t<P>& entry = *it;
-            aslrTracker.add(&(entry.name));
-            aslrTracker.add(&(entry.types));
-            aslrTracker.add(&(entry.imp));
-        }
+    const char* getStringName(ContentAccessor* cache, uint32_t i, bool isOffsetToSel) {
+        return (const char*)cache->contentForVMAddr(getName(cache, i, isOffsetToSel));
     }
 
-    static objc_method_list_t<P>* newMethodList(size_t newCount, uint32_t newEntsize) {
-        void *buf = ::calloc(byteSizeForCount(newCount, newEntsize), 1);
-        return new (buf) objc_method_list_t<P>(newCount, newEntsize);
+    pint_t getImp(uint32_t i, ContentAccessor* cache) {
+        pint_t name = 0;
+        if ( usesRelativeMethods() ) {
+            small_method_iterator it = beginSmall() + i;
+            objc_method_small_t<P>& method = *it;
+            name = method.getIMP(cache);
+        } else {
+            large_method_iterator it = beginLarge() + i;
+            objc_method_large_t<P>& method = *it;
+            name = method.getIMP();
+        }
+        return name;
+    }
+
+    void* get(uint32_t i) const {
+        if ( usesRelativeMethods() ) {
+            return (void*)(objc_method_small_t<P> *)((uint8_t *)&first + i * getEntsize());
+        } else {
+            return (void*)(objc_method_large_t<P> *)((uint8_t *)&first + i * getEntsize());
+        }
     }
 
     void operator delete(void * p) { 
         ::free(p); 
     }
 
-    objc_method_list_t(uint32_t newCount, 
-                       uint32_t newEntsize = sizeof(objc_method_t<P>))
-        : entsize(newEntsize), count(newCount) 
-    { }
-
 private:
+
     // use newMethodList instead
     void* operator new (size_t);
 };
@@ -293,7 +595,7 @@ public:
 
     uint32_t getEntsize() const { return P::E::get32(entsize); }
 
-    objc_ivar_t<P>& get(pint_t i) const { return *(objc_ivar_t<P> *)((uint8_t *)&first + i * P::E::get32(entsize)); }
+    void* get(pint_t i) const { return (void*)(objc_ivar_t<P> *)((uint8_t *)&first + i * P::E::get32(entsize)); }
 
     uint32_t byteSize() const { 
         return byteSizeForCount(getCount(), getEntsize()); 
@@ -358,7 +660,7 @@ public:
 
     uint32_t getEntsize() const { return P::E::get32(entsize); }
 
-    objc_property_t<P>& get(uint32_t i) const { return *(objc_property_t<P> *)((uint8_t *)&first + i * getEntsize()); }
+    void* get(uint32_t i) const { return (objc_property_t<P> *)((uint8_t *)&first + i * getEntsize()); }
 
     uint32_t byteSize() const { 
         return byteSizeForCount(getCount(), getEntsize()); 
@@ -432,6 +734,7 @@ class objc_protocol_t {
 public:
     pint_t getIsaVMAddr() const { return (pint_t)P::getP(isa); }
     void setIsaVMAddr(pint_t newIsa) { P::setP(isa, newIsa); }
+    void* getISALocation() const { return (void*)&isa; }
 
     const char *getName(ContentAccessor* cache) const { return (const char *)cache->contentForVMAddr(P::getP(name)); }
 
@@ -491,6 +794,7 @@ public:
         if (instanceProperties)      aslrTracker.add(&instanceProperties);
         if (extendedMethodTypes)     aslrTracker.add(&extendedMethodTypes);
         if (demangledName)           aslrTracker.add(&demangledName);
+        if (classProperties)         aslrTracker.add(&classProperties);
     }
 };
 
@@ -641,12 +945,15 @@ public:
     objc_class_t<P> *getIsa(ContentAccessor* cache) const { return (objc_class_t<P> *)cache->contentForVMAddr(P::getP(isa)); }
 
     objc_class_t<P> *getSuperclass(ContentAccessor* cache) const { return (objc_class_t<P> *)cache->contentForVMAddr(P::getP(superclass)); }
-    
     const pint_t* getSuperClassAddress() const { return &superclass; }
 
     // Low bit marks Swift classes.
     objc_class_data_t<P> *getData(ContentAccessor* cache) const { return (objc_class_data_t<P> *)cache->contentForVMAddr(P::getP(data & ~0x3LL)); }
 
+    objc_class_t<P> *getVTable(ContentAccessor* cache) const { return (objc_class_t<P> *)cache->contentForVMAddr(P::getP(vtable)); }
+
+    pint_t* getVTableAddress() { return &vtable; }
+
     objc_method_list_t<P> *getMethodList(ContentAccessor* cache) const {
         objc_class_data_t<P>* d = getData(cache);
         return d->getMethodList(cache);
@@ -753,9 +1060,9 @@ public:
         objc_ivar_list_t<P> *ivars = data->getIvarList(cache);
         if (ivars) {
             for (pint_t i = 0; i < ivars->getCount(); i++) {
-                objc_ivar_t<P>& ivar = ivars->get(i);
+                objc_ivar_t<P>* ivar = (objc_ivar_t<P>*)ivars->get(i);
                 //fprintf(stderr, "visiting ivar: %s\n", ivar.getName(cache));
-                ivarVisitor.visitIvar(cache, header, cls, &ivar);
+                ivarVisitor.visitIvar(cache, header, cls, ivar);
             }
         } else {
             //fprintf(stderr, "no ivars\n");
@@ -768,14 +1075,20 @@ public:
     }
 };
 
+enum class ClassWalkerMode {
+    ClassesOnly,
+    ClassAndMetaclasses,
+};
+
 // Call visitor.visitClass() on every class.
 template <typename P, typename V>
 class ClassWalker {
     typedef typename P::uint_t pint_t;
     V& _visitor;
+    ClassWalkerMode _mode;
 public:
     
-    ClassWalker(V& visitor) : _visitor(visitor) { }
+    ClassWalker(V& visitor, ClassWalkerMode mode = ClassWalkerMode::ClassesOnly) : _visitor(visitor), _mode(mode) { }
     
     void walk(ContentAccessor* cache, const macho_header<P>* header)
     {   
@@ -783,8 +1096,14 @@ public:
         
         for (pint_t i = 0; i < classList.count(); i++) {
             objc_class_t<P>* cls = classList.get(i);
+            if (cls) {
                 //fprintf(stderr, "visiting class: %s\n", cls->getName(cache));
-            if (cls) _visitor.visitClass(cache, header, cls);
+                _visitor.visitClass(cache, header, cls);
+                if (_mode == ClassWalkerMode::ClassAndMetaclasses) {
+                    //fprintf(stderr, "visiting metaclass: %s\n", cls->getIsa(cache)->getName(cache));
+                    _visitor.visitClass(cache, header, cls->getIsa(cache));
+                }
+            }
         }
     }
 };
@@ -900,10 +1219,10 @@ public:
             objc_class_t<P> *cls = classes.get(i);
             objc_method_list_t<P> *mlist;
             if ((mlist = cls->getMethodList(cache))) {
-                mVisitor.visitMethodList(mlist);
+                mVisitor.visitMethodList(cache, mlist);
             }
             if ((mlist = cls->getIsa(cache)->getMethodList(cache))) {
-                mVisitor.visitMethodList(mlist);
+                mVisitor.visitMethodList(cache, mlist);
             }
         }
         
@@ -914,10 +1233,10 @@ public:
             objc_category_t<P> *cat = cats.get(i);
             objc_method_list_t<P> *mlist;
             if ((mlist = cat->getInstanceMethods(cache))) {
-                mVisitor.visitMethodList(mlist);
+                mVisitor.visitMethodList(cache, mlist);
             }
             if ((mlist = cat->getClassMethods(cache))) {
-                mVisitor.visitMethodList(mlist);
+                mVisitor.visitMethodList(cache, mlist);
             }
         }
 
@@ -930,19 +1249,19 @@ public:
             pint_t *typelist = proto->getExtendedMethodTypes(cache);
 
             if ((mlist = proto->getInstanceMethods(cache))) {
-                mVisitor.visitProtocolMethodList(mlist, typelist);
+                mVisitor.visitProtocolMethodList(cache, mlist, typelist);
                 if (typelist) typelist += mlist->getCount();
             }
             if ((mlist = proto->getClassMethods(cache))) {
-                mVisitor.visitProtocolMethodList(mlist, typelist);
+                mVisitor.visitProtocolMethodList(cache, mlist, typelist);
                 if (typelist) typelist += mlist->getCount();
             }
             if ((mlist = proto->getOptionalInstanceMethods(cache))) {
-                mVisitor.visitProtocolMethodList(mlist, typelist);
+                mVisitor.visitProtocolMethodList(cache, mlist, typelist);
                 if (typelist) typelist += mlist->getCount();
             }
             if ((mlist = proto->getOptionalClassMethods(cache))) {
-                mVisitor.visitProtocolMethodList(mlist, typelist);
+                mVisitor.visitProtocolMethodList(cache, mlist, typelist);
                 if (typelist) typelist += mlist->getCount();
             }
         }
@@ -960,25 +1279,36 @@ class SelectorOptimizer {
     std::set<pint_t> selectorRefVMAddrs;
 
     friend class MethodListWalker<P, SelectorOptimizer<P,V> >;
-    void visitMethodList(objc_method_list_t<P> *mlist)
+    void visitMethodList(ContentAccessor* cache, objc_method_list_t<P> *mlist)
     {
         // Gather selectors. Update method names.
         for (uint32_t m = 0; m < mlist->getCount(); m++) {
-            pint_t oldValue = mlist->get(m).getName();
+            // Read names as relative offsets to selRefs
+            pint_t oldValue = mlist->getName(cache, m, false);
             pint_t newValue = mVisitor.visit(oldValue);
-            mlist->get(m).setName(newValue);
+            // And write names as relative offsets to SELs themselves.
+            mlist->setName(cache, m, newValue, true);
         }
+        // Set this method list as now being relative offsets directly to the selector string
+        if ( mlist->usesRelativeMethods() )
+            mlist->setMethodListSelectorsAreDirect();
+
         // Do not setFixedUp: the methods are not yet sorted.
     }
 
-    void visitProtocolMethodList(objc_method_list_t<P> *mlist, pint_t *types)
+    void visitProtocolMethodList(ContentAccessor* cache, objc_method_list_t<P> *mlist, pint_t *types)
     {
-        visitMethodList(mlist);
+        visitMethodList(cache, mlist);
     }
 
 public:
 
-    SelectorOptimizer(V& visitor) : mVisitor(visitor) { }
+    SelectorOptimizer(V& visitor, bool& relativeMethodListSelectorsAreDirect) : mVisitor(visitor) {
+        // This pass requires that relative method lists are initially indirected via the selector
+        // ref.  After this pass runs we'll use relative offsets to the selectors themselves
+        assert(!relativeMethodListSelectorsAreDirect);
+        relativeMethodListSelectorsAreDirect = true;
+    }
 
     void visitCoalescedStrings(const CacheBuilder::CacheCoalescedText& coalescedText) {
         mVisitor.visitCoalescedStrings(coalescedText);
@@ -1160,37 +1490,27 @@ class MethodListSorter {
     typedef typename P::uint_t pint_t;
 
     uint32_t _optimized;
+    bool     _isOffsetToSel;
 
     friend class MethodListWalker<P, MethodListSorter<P> >;
-    void visitMethodList(objc_method_list_t<P> *mlist)
-    {
-        typename objc_method_t<P>::SortBySELAddress sorter;
-        std::stable_sort(mlist->begin(), mlist->end(), sorter);
-        mlist->setFixedUp();
+
+    void sortMethodList(ContentAccessor* cache, objc_method_list_t<P> *mlist, pint_t *typelist) {
+        mlist->sortMethods(cache, typelist, _isOffsetToSel);
         _optimized++;
     }
 
-    void visitProtocolMethodList(objc_method_list_t<P> *mlist, pint_t *typelist)
+    void visitMethodList(ContentAccessor* cache, objc_method_list_t<P> *mlist)
     {
-        typename objc_method_t<P>::SortBySELAddress sorter;
-        // can't easily use std::stable_sort here
-        for (uint32_t i = 0; i < mlist->getCount(); i++) {
-            for (uint32_t j = i+1; j < mlist->getCount(); j++) {
-                objc_method_t<P>& mi = mlist->get(i);
-                objc_method_t<P>& mj = mlist->get(j);
-                if (! sorter(mi, mj)) {
-                    std::swap(mi, mj);
-                    if (typelist) std::swap(typelist[i], typelist[j]);
-                }
-            }
+        sortMethodList(cache, mlist, nullptr);
     }
 
-        mlist->setFixedUp();
-        _optimized++;
+    void visitProtocolMethodList(ContentAccessor* cache, objc_method_list_t<P> *mlist, pint_t *typelist)
+    {
+        sortMethodList(cache, mlist, typelist);
     }
 
 public:
-    MethodListSorter() : _optimized(0) { }
+    MethodListSorter(bool isOffsetToSel) : _optimized(0), _isOffsetToSel(isOffsetToSel) { }
 
     size_t optimized() const { return _optimized; }
 

dyld3/shared-cache/OptimizerBranches.cpp

@@ -52,11 +52,13 @@ static const bool verbose = false;
 template <typename P>
 class StubOptimizer {
 public:
-                            StubOptimizer(const DyldSharedCache* cache, macho_header<P>* mh, Diagnostics& diags);
+                            StubOptimizer(int64_t cacheSlide, uint64_t cacheUnslidAddr,
+                                          const std::string& archName, macho_header<P>* mh,
+                                          const char* dylibID, Diagnostics& diags);
     void                    buildStubMap(const std::unordered_set<std::string>& neverStubEliminate);
     void                    optimizeStubs();
     void                    optimizeCallSites(std::unordered_map<uint64_t, uint64_t>& targetAddrToOptStubAddr);
-    const char*             installName() { return _installName; }
+    const char*             dylibID() { return _dylibID; }
     const uint8_t*          exportsTrie() {
         if ( _dyldInfo != nullptr )
             return &_linkeditBias[_dyldInfo->export_off()];
@@ -79,7 +81,7 @@ public:
     uint32_t                _branchToReUsedOptimizedStubCount = 0;
 
 private:
-    Diagnostics             _diagnostics;
+    Diagnostics&            _diagnostics;
 
     typedef std::function<bool(uint8_t callSiteKind, uint64_t callSiteAddr, uint64_t stubAddr, uint32_t& instruction)> CallSiteHandler;
     typedef typename P::uint_t pint_t;
@@ -107,6 +109,7 @@ private:
     int32_t                 getDisplacementFromThumbBranch(uint32_t instruction, uint32_t instrAddr);
     uint32_t                setDisplacementInThumbBranch(uint32_t instruction,  uint32_t instrAddr,
                                                          int32_t displacement, bool targetIsThumb);
+    uint32_t                cpuSubtype() { return ((dyld3::MachOFile*)_mh)->maskedCpuSubtype(); }
 
 
     struct AddressAndName { pint_t targetVMAddr; const char* targetName; };
@@ -120,11 +123,10 @@ private:
     macho_header<P>*                        _mh;
     int64_t                                 _cacheSlide          = 0;
     uint64_t                                _cacheUnslideAddr    = 0;
-    bool                                    _chainedFixups       = false;
     uint32_t                                _linkeditSize        = 0;
     uint64_t                                _linkeditAddr        = 0;
     const uint8_t*                          _linkeditBias        = nullptr;
-    const char*                             _installName         = nullptr;
+    const char*                             _dylibID             = nullptr;
     const macho_symtab_command<P>*          _symTabCmd           = nullptr;
     const macho_dysymtab_command<P>*        _dynSymTabCmd        = nullptr;
     const macho_dyld_info_command<P>*       _dyldInfo            = nullptr;
@@ -144,16 +146,14 @@ private:
 
 
 template <typename P>
-StubOptimizer<P>::StubOptimizer(const DyldSharedCache* cache, macho_header<P>* mh, Diagnostics& diags)
-: _mh(mh), _diagnostics(diags)
+StubOptimizer<P>::StubOptimizer(int64_t cacheSlide, uint64_t cacheUnslidAddr,
+                                const std::string& archName,
+                                macho_header<P>* mh, const char* dylibID,
+                                Diagnostics& diags)
+    : _mh(mh), _dylibID(dylibID),
+      _cacheSlide(cacheSlide), _cacheUnslideAddr(cacheUnslidAddr),
+      _diagnostics(diags)
 {
-    _cacheSlide = (long)cache - cache->unslidLoadAddress();
-    _cacheUnslideAddr = cache->unslidLoadAddress();
-#if SUPPORT_ARCH_arm64e
-    _chainedFixups = (strcmp(cache->archName(), "arm64e") == 0);
-#else
-    _chainedFixups = false;
-#endif
     const macho_load_command<P>* const cmds = (macho_load_command<P>*)((uint8_t*)mh + sizeof(macho_header<P>));
     const uint32_t cmd_count = mh->ncmds();
     macho_segment_command<P>* segCmd;
@@ -161,9 +161,6 @@ StubOptimizer<P>::StubOptimizer(const DyldSharedCache* cache, macho_header<P>* m
     const macho_load_command<P>* cmd = cmds;
     for (uint32_t i = 0; i < cmd_count; ++i) {
         switch (cmd->cmd()) {
-            case LC_ID_DYLIB:
-                _installName = ((macho_dylib_command<P>*)cmd)->name();
-                break;
             case LC_SYMTAB:
                 _symTabCmd = (macho_symtab_command<P>*)cmd;
                 break;
@@ -221,17 +218,17 @@ uint32_t StubOptimizer<P>::lazyPointerAddrFromArmStub(const uint8_t* stubInstruc
     int32_t  stubData   = E::get32(*(uint32_t*)(stubInstructions+12));
     if ( stubInstr1 != 0xe59fc004 ) {
         _diagnostics.warning("first instruction of stub (0x%08X) is not 'ldr ip, pc + 12' for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     if ( stubInstr2 != 0xe08fc00c ) {
         _diagnostics.warning("second instruction of stub (0x%08X) is not 'add ip, pc, ip' for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     if ( stubInstr3 != 0xe59cf000 ) {
         _diagnostics.warning("third instruction of stub (0x%08X) is not 'ldr pc, [ip]' for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     return stubVMAddr + 12 + stubData;
@@ -244,7 +241,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64Stub(const uint8_t* stubInstr
     uint32_t stubInstr1 = E::get32(*(uint32_t*)stubInstructions);
     if ( (stubInstr1 & 0x9F00001F) != 0x90000010 ) {
         _diagnostics.warning("first instruction of stub (0x%08X) is not ADRP for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     int32_t adrpValue = ((stubInstr1 & 0x00FFFFE0) >> 3) | ((stubInstr1 & 0x60000000) >> 29);
@@ -253,7 +250,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64Stub(const uint8_t* stubInstr
     uint32_t stubInstr2 = E::get32(*(uint32_t*)(stubInstructions + 4));
     if ( (stubInstr2 & 0xFFC003FF) != 0xF9400210 ) {
         _diagnostics.warning("second instruction of stub (0x%08X) is not LDR for stub at addr 0x%0llX in %s",
-                stubInstr2, (uint64_t)stubVMAddr, _installName);
+                stubInstr2, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     uint32_t ldrValue = ((stubInstr2 >> 10) & 0x00000FFF);
@@ -267,7 +264,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64_32Stub(const uint8_t* stubIn
     uint32_t stubInstr1 = E::get32(*(uint32_t*)stubInstructions);
     if ( (stubInstr1 & 0x9F00001F) != 0x90000010 ) {
         _diagnostics.warning("first instruction of stub (0x%08X) is not ADRP for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     int32_t adrpValue = ((stubInstr1 & 0x00FFFFE0) >> 3) | ((stubInstr1 & 0x60000000) >> 29);
@@ -276,7 +273,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64_32Stub(const uint8_t* stubIn
     uint32_t stubInstr2 = E::get32(*(uint32_t*)(stubInstructions + 4));
     if ( (stubInstr2 & 0xFFC003FF) != 0xB9400210 ) {
         _diagnostics.warning("second instruction of stub (0x%08X) is not LDR for stub at addr 0x%0llX in %s",
-                stubInstr2, (uint64_t)stubVMAddr, _installName);
+                stubInstr2, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     uint32_t ldrValue = ((stubInstr2 >> 10) & 0x00000FFF);
@@ -294,7 +291,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64eStub(const uint8_t* stubInst
     // ADRP  X17, dyld_mageLoaderCache@page
     if ( (stubInstr1 & 0x9F00001F) != 0x90000011 ) {
         _diagnostics.warning("first instruction of stub (0x%08X) is not ADRP for stub at addr 0x%0llX in %s",
-                stubInstr1, (uint64_t)stubVMAddr, _installName);
+                stubInstr1, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     int32_t adrpValue = ((stubInstr1 & 0x00FFFFE0) >> 3) | ((stubInstr1 & 0x60000000) >> 29);
@@ -305,7 +302,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64eStub(const uint8_t* stubInst
     uint32_t stubInstr2 = E::get32(*(uint32_t*)(stubInstructions + 4));
     if ( (stubInstr2 & 0xFFC003FF) != 0x91000231 ) {
         _diagnostics.warning("second instruction of stub (0x%08X) is not ADD for stub at addr 0x%0llX in %s",
-                             stubInstr2, (uint64_t)stubVMAddr, _installName);
+                             stubInstr2, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     uint32_t addValue = ((stubInstr2 & 0x003FFC00) >> 10);
@@ -314,7 +311,7 @@ uint64_t StubOptimizer<P>::lazyPointerAddrFromArm64eStub(const uint8_t* stubInst
     uint32_t stubInstr3 = E::get32(*(uint32_t*)(stubInstructions + 8));
     if ( stubInstr3 != 0xF9400230 ) {
         _diagnostics.warning("second instruction of stub (0x%08X) is not LDR for stub at addr 0x%0llX in %s",
-                stubInstr2, (uint64_t)stubVMAddr, _installName);
+                stubInstr2, (uint64_t)stubVMAddr, _dylibID);
         return 0;
     }
     return (stubVMAddr & (-4096)) + adrpValue*4096 + addValue;
@@ -356,19 +353,19 @@ void StubOptimizer<P>::buildStubMap(const std::unordered_set<std::string>& never
                             default:
                                 if ( symbolIndex >= _symTabCmd->nsyms() ) {
                                     _diagnostics.warning("symbol index out of range (%d of %d) for stub at addr 0x%0llX in %s",
-                                        symbolIndex, _symTabCmd->nsyms(), (uint64_t)stubVMAddr, _installName);
+                                        symbolIndex, _symTabCmd->nsyms(), (uint64_t)stubVMAddr, _dylibID);
                                     continue;
                                 }
                                 const macho_nlist<P>* sym = &symbolTable[symbolIndex];
                                 uint32_t stringOffset = sym->n_strx();
                                 if ( stringOffset > _symTabCmd->strsize() ) {
                                     _diagnostics.warning("symbol string offset out of range (%u of %u) for stub at addr 0x%0llX in %s",
-                                        stringOffset, sym->n_strx(), (uint64_t)stubVMAddr, _installName);
+                                        stringOffset, sym->n_strx(), (uint64_t)stubVMAddr, _dylibID);
                                     continue;
                                 }
                                 const char* symName = &symbolStrings[stringOffset];
                                 if ( neverStubEliminate.count(symName) ) {
-                                    //fprintf(stderr, "stubVMAddr=0x%llX, not bypassing stub to %s in %s because target is interposable\n", (uint64_t)stubVMAddr, symName, _installName);
+                                    //fprintf(stderr, "stubVMAddr=0x%llX, not bypassing stub to %s in %s because target is interposable\n", (uint64_t)stubVMAddr, symName, _dylibID);
                                     _stubsLeftInterposable++;
                                     continue;
                                 }
@@ -376,19 +373,19 @@ void StubOptimizer<P>::buildStubMap(const std::unordered_set<std::string>& never
                                 pint_t targetLPAddr = 0;
                                 switch ( _mh->cputype() ) {
                                     case CPU_TYPE_ARM64:
-                                    case CPU_TYPE_ARM64_32:
 #if SUPPORT_ARCH_arm64e
-                                        if (_mh->cpusubtype() == CPU_SUBTYPE_ARM64E)
+                                        if (cpuSubtype() == CPU_SUBTYPE_ARM64E)
                                             targetLPAddr = (pint_t)lazyPointerAddrFromArm64eStub(stubInstrs, stubVMAddr);
                                         else
-#endif
-#if SUPPORT_ARCH_arm64_32
-                                        if (_mh->cputype() == CPU_TYPE_ARM64_32)
-                                            targetLPAddr = (pint_t)lazyPointerAddrFromArm64_32Stub(stubInstrs, stubVMAddr);
-                                        else
 #endif
                                             targetLPAddr = (pint_t)lazyPointerAddrFromArm64Stub(stubInstrs, stubVMAddr);
                                         break;
+#if SUPPORT_ARCH_arm64_32
+                                    case CPU_TYPE_ARM64_32:
+                                        if (cpuSubtype() == CPU_TYPE_ARM64_32)
+                                            targetLPAddr = (pint_t)lazyPointerAddrFromArm64_32Stub(stubInstrs, stubVMAddr);
+                                        break;
+#endif
                                     case CPU_TYPE_ARM:
                                         targetLPAddr = (pint_t)lazyPointerAddrFromArmStub(stubInstrs, (uint32_t)stubVMAddr);
                                         break;
@@ -415,40 +412,29 @@ void StubOptimizer<P>::buildStubMap(const std::unordered_set<std::string>& never
                                 break;
                             default:
                                 lpValue = (pint_t)P::getP(lpContent[j]);
-
-                                // Fixup threaded rebase/bind
-                                if ( _chainedFixups ) {
-                                    dyld3::MachOLoaded::ChainedFixupPointerOnDisk ptr;
-                                    ptr.raw64 = lpValue;
-                                    assert(ptr.arm64e.authRebase.bind == 0);
-                                    if ( ptr.arm64e.authRebase.auth ) {
-                                        lpValue = (pint_t)(_cacheUnslideAddr + ptr.arm64e.authRebase.target);
-                                    }
-                                    else {
-                                        lpValue = (pint_t)ptr.arm64e.unpackTarget();
-                                    }
-                                }
-
                                 lpVMAddr = (pint_t)sect->addr() + j * sizeof(pint_t);
                                 if ( symbolIndex >= _symTabCmd->nsyms() ) {
                                     _diagnostics.warning("symbol index out of range (%d of %d) for lazy pointer at addr 0x%0llX in %s",
-                                            symbolIndex, _symTabCmd->nsyms(), (uint64_t)lpVMAddr, _installName);
+                                            symbolIndex, _symTabCmd->nsyms(), (uint64_t)lpVMAddr, _dylibID);
                                     continue;
                                 }
                                 const macho_nlist<P>* sym = &symbolTable[symbolIndex];
                                 uint32_t stringOffset = sym->n_strx();
                                 if ( stringOffset > _symTabCmd->strsize() ) {
                                     _diagnostics.warning("symbol string offset out of range (%u of %u) for lazy pointer at addr 0x%0llX in %s",
-                                            stringOffset, sym->n_strx(), (uint64_t)lpVMAddr, _installName);
+                                            stringOffset, sym->n_strx(), (uint64_t)lpVMAddr, _dylibID);
                                     continue;
                                 }
                                 const char* symName = &symbolStrings[stringOffset];
                                 if ( (lpValue > textSegStartAddr) && (lpValue< textSegEndAddr) ) {
-                                    //fprintf(stderr, "skipping lazy pointer at 0x%0lX to %s in %s because target is within dylib\n", (long)lpVMAddr, symName, _installName);
+                                    //fprintf(stderr, "skipping lazy pointer at 0x%0lX to %s in %s because target is within dylib\n", (long)lpVMAddr, symName, _dylibID);
                                 }
                                 else if ( (sizeof(pint_t) == 8) && ((lpValue % 4) != 0) ) {
-                                    _diagnostics.warning("lazy pointer at 0x%0llX does not point to 4-byte aligned address(0x%0llX) in %s",
-                                                (uint64_t)lpVMAddr, (uint64_t)lpValue, _installName);
+                                    // Only warn on lazy pointers which correspond to call targets
+                                    if ( sectionType == S_LAZY_SYMBOL_POINTERS ) {
+                                        _diagnostics.warning("lazy pointer at 0x%0llX does not point to 4-byte aligned address(0x%0llX) for symbol '%s' in %s",
+                                                             (uint64_t)lpVMAddr, (uint64_t)lpValue, symName, _dylibID);
+                                    }
                                 }
                                 else {
                                    _lpAddrToTargetAddr[lpVMAddr] = lpValue;
@@ -473,7 +459,7 @@ void StubOptimizer<P>::forEachCallSiteToAStub(CallSiteHandler handler)
     const uint8_t* infoStart = &_linkeditBias[_splitSegInfoCmd->dataoff()];
     const uint8_t* infoEnd = &infoStart[_splitSegInfoCmd->datasize()];
     if ( *infoStart++ != DYLD_CACHE_ADJ_V2_FORMAT ) {
-        _diagnostics.error("malformed split seg info in %s", _installName);
+        _diagnostics.error("malformed split seg info in %s", _dylibID);
         return;
     }
 
@@ -497,7 +483,7 @@ void StubOptimizer<P>::forEachCallSiteToAStub(CallSiteHandler handler)
             for (uint64_t k=0; k < fromOffsetCount; ++k) {
                 uint64_t kind = read_uleb128(p, infoEnd);
                 if ( kind > 13 ) {
-                    _diagnostics.error("bad kind (%llu) value in %s\n", kind, _installName);
+                    _diagnostics.error("bad kind (%llu) value in %s\n", kind, _dylibID);
                 }
                 uint64_t fromSectDeltaCount = read_uleb128(p, infoEnd);
                 uint64_t fromSectionOffset = 0;
@@ -551,7 +537,7 @@ template <typename P>
 uint32_t StubOptimizer<P>::setDisplacementInThumbBranch(uint32_t instruction,  uint32_t instrAddr,
                                                         int32_t displacement, bool targetIsThumb) {
     if ( (displacement > 16777214) || (displacement < (-16777216)) ) {
-        _diagnostics.error("thumb branch out of range at 0x%0X in %s", instrAddr, _installName);
+        _diagnostics.error("thumb branch out of range at 0x%0X in %s", instrAddr, _dylibID);
         return 0;
     }
     bool is_bl = ((instruction & 0xD000F800) == 0xD000F000);
@@ -571,12 +557,12 @@ uint32_t StubOptimizer<P>::setDisplacementInThumbBranch(uint32_t instruction,  u
     }
     else if (is_b) {
         if ( !targetIsThumb ) {
-            _diagnostics.error("no pc-rel thumb branch instruction that switches to arm mode at 0x%0X in %s", instrAddr, _installName);
+            _diagnostics.error("no pc-rel thumb branch instruction that switches to arm mode at 0x%0X in %s", instrAddr, _dylibID);
             return 0;
         }
     }
     else {
-        _diagnostics.error("not b/bl/blx at 0x%0X in %s", instrAddr, _installName);
+        _diagnostics.error("not b/bl/blx at 0x%0X in %s", instrAddr, _dylibID);
         return 0;
     }
     uint32_t s = (uint32_t)(displacement >> 24) & 0x1;
@@ -602,13 +588,13 @@ void StubOptimizer<P>::optimizeArmCallSites(std::unordered_map<uint64_t, uint64_
             bool is_blx = ((instruction & 0xD000F800) == 0xC000F000);
             bool is_b = ((instruction & 0xD000F800) == 0x9000F000);
             if ( !is_bl && !is_blx && !is_b ){
-                _diagnostics.warning("non-branch instruction at 0x%0llX in %s", callSiteAddr, _installName);
+                _diagnostics.warning("non-branch instruction at 0x%0llX in %s", callSiteAddr, _dylibID);
                 return false;
             }
             int32_t brDelta = getDisplacementFromThumbBranch(instruction, (uint32_t)callSiteAddr);
             pint_t targetAddr = (pint_t)callSiteAddr + 4 + brDelta;
             if ( targetAddr != stubAddr ) {
-                _diagnostics.warning("stub target mismatch at callsite 0x%0llX in %s", callSiteAddr, _installName);
+                _diagnostics.warning("stub target mismatch at callsite 0x%0llX in %s", callSiteAddr, _dylibID);
                 return false;
             }
             // ignore branch if not to a known stub
@@ -870,7 +856,7 @@ void StubOptimizer<P>::optimizeCallSites(std::unordered_map<uint64_t, uint64_t>&
         case CPU_TYPE_ARM64:
             optimizeArm64CallSites(targetAddrToOptStubAddr);
 #if SUPPORT_ARCH_arm64e
-            if (_mh->cpusubtype() == CPU_SUBTYPE_ARM64E)
+            if (cpuSubtype() == CPU_SUBTYPE_ARM64E)
                 optimizeArm64eStubs();
             else
 #endif
@@ -891,38 +877,47 @@ void StubOptimizer<P>::optimizeCallSites(std::unordered_map<uint64_t, uint64_t>&
         _diagnostics.verbose("dylib has %6u BLs to %4u stubs. Changed %5u, %5u, %5u BLs to use direct branch, optimized stub, neighbor's optimized stub. "
                              "%5u stubs left interposable, %4u stubs optimized. path=%s\n",
                             _branchToStubCount, _stubCount, _branchOptimizedToDirectCount, _branchToOptimizedStubCount, _branchToReUsedOptimizedStubCount,
-                            _stubsLeftInterposable, _stubOptimizedCount, _installName);
+                            _stubsLeftInterposable, _stubOptimizedCount, _dylibID);
     }
 
 }
 
 template <typename P>
-void bypassStubs(DyldSharedCache* cache, const std::string& archName, std::unordered_map<uint64_t, uint64_t>& targetAddrToOptStubAddr,
-                 const char* const neverStubEliminateDylibs[], const char* const neverStubEliminateSymbols[],
+void bypassStubs(std::vector<std::pair<const mach_header*, const char*>> images,
+                 const std::string& archName,
+                 int64_t cacheSlide, uint64_t cacheUnslidAddr,
+                 const DyldSharedCache* dyldCache,
+                 const char* const neverStubEliminateSymbols[],
                  Diagnostics& diags)
 {
+    std::unordered_map<uint64_t, uint64_t> targetAddrToOptStubAddr;
     diags.verbose("Stub elimination optimization:\n");
 
     // construct a StubOptimizer for each image
     __block std::vector<StubOptimizer<P>*> optimizers;
-    cache->forEachImage(^(const mach_header* mh, const char* installName) {
-        optimizers.push_back(new StubOptimizer<P>(cache, (macho_header<P>*)mh, diags));
-    });
+    for (std::pair<const mach_header*, const char*> image : images) {
+        optimizers.push_back(new StubOptimizer<P>(cacheSlide, cacheUnslidAddr, archName,
+                                                  (macho_header<P>*)image.first, image.second,
+                                                  diags));
+    }
 
     // build set of functions to never stub-eliminate because tools may need to override them
     std::unordered_set<std::string> neverStubEliminate;
     for (const char* const* p=neverStubEliminateSymbols; *p != nullptr; ++p) {
         neverStubEliminate.insert(*p);
     }
-    for (const char* const* d=neverStubEliminateDylibs; *d != nullptr; ++d) {
+
+#if !BUILDING_APP_CACHE_UTIL
+    // Customer shared caches support overriding libdispatch
+    if ( dyldCache != nullptr ) {
         for (StubOptimizer<P>* op : optimizers) {
-            if ( strcmp(op->installName(), *d) == 0 ) {
+            if ( dyldCache->isOverridablePath(op->dylibID()) ) {
                 // add all exports
                 const uint8_t* exportsStart = op->exportsTrie();
                 const uint8_t* exportsEnd = exportsStart + op->exportsTrieSize();
                 std::vector<ExportInfoTrie::Entry> exports;
                 if ( !ExportInfoTrie::parseTrie(exportsStart, exportsEnd, exports) ) {
-                    diags.error("malformed exports trie in %s", *d);
+                    diags.error("malformed exports trie in %s", op->dylibID());
                     return;
                 }
                 for(const ExportInfoTrie::Entry& entry : exports) {
@@ -931,6 +926,7 @@ void bypassStubs(DyldSharedCache* cache, const std::string& archName, std::unord
             }
         }
     }
+#endif
 
     // build maps of stubs-to-lp and lp-to-target
     for (StubOptimizer<P>* op : optimizers)
@@ -954,20 +950,32 @@ void bypassStubs(DyldSharedCache* cache, const std::string& archName, std::unord
         delete op;
 }
 
-void CacheBuilder::optimizeAwayStubs()
+void CacheBuilder::optimizeAwayStubs(const std::vector<std::pair<const mach_header*, const char*>>& images,
+                                     int64_t cacheSlide, uint64_t cacheUnslidAddr,
+                                     const DyldSharedCache* dyldCache,
+                                     const char* const neverStubEliminateSymbols[])
 {
     std::unordered_map<uint64_t, uint64_t> targetAddrToOptStubAddr;
-
-    DyldSharedCache* dyldCache = (DyldSharedCache*)_readExecuteRegion.buffer;
-    std::string archName = dyldCache->archName();
+    std::string archName = _options.archs->name();
 #if SUPPORT_ARCH_arm64_32
-    if ( startsWith(archName, "arm64_32") )
-        bypassStubs<Pointer32<LittleEndian> >(dyldCache, archName, targetAddrToOptStubAddr, _s_neverStubEliminateDylibs, _s_neverStubEliminateSymbols, _diagnostics);
-    else
+    if ( startsWith(archName, "arm64_32") ) {
+        bypassStubs<Pointer32<LittleEndian> >(images, archName, cacheSlide, cacheUnslidAddr,
+                                              dyldCache, neverStubEliminateSymbols,
+                                              _diagnostics);
+        return;
+    }
 #endif
-    if ( startsWith(archName, "arm64") )
-        bypassStubs<Pointer64<LittleEndian> >(dyldCache, archName, targetAddrToOptStubAddr, _s_neverStubEliminateDylibs, _s_neverStubEliminateSymbols, _diagnostics);
-    else if ( archName == "armv7k" )
-        bypassStubs<Pointer32<LittleEndian>>(dyldCache, archName, targetAddrToOptStubAddr, _s_neverStubEliminateDylibs, _s_neverStubEliminateSymbols, _diagnostics);
+    if ( startsWith(archName, "arm64") ) {
+        bypassStubs<Pointer64<LittleEndian> >(images, archName, cacheSlide, cacheUnslidAddr,
+                                              dyldCache, neverStubEliminateSymbols,
+                                              _diagnostics);
+        return;
+    }
+    if ( archName == "armv7k" ) {
+        bypassStubs<Pointer32<LittleEndian> >(images, archName, cacheSlide, cacheUnslidAddr,
+                                              dyldCache, neverStubEliminateSymbols,
+                                              _diagnostics);
+        return;
+    }
     // no stub optimization done for other arches
 }

dyld3/shared-cache/OptimizerLinkedit.cpp

@@ -53,7 +53,12 @@ class SortedStringPool
 public:
     // add a string and symbol table entry index to be updated later
     void add(uint32_t symbolIndex, const char* symbolName) {
-        _map[symbolName].push_back(symbolIndex);
+        _map[symbolName].push_back({ symbolIndex, false });
+    }
+
+    // add a string and symbol table entry index to be updated later
+    void addIndirect(uint32_t symbolIndex, const char* symbolName) {
+        _map[symbolName].push_back({ symbolIndex, true });
     }
 
     // copy sorted strings to buffer and update all symbol's string offsets
@@ -66,8 +71,13 @@ public:
             // append string to pool
             strcpy(&dstStringPool[poolOffset], symName.c_str());
             //  set each string offset of each symbol using it
-            for (uint32_t symbolIndex : entry.second) {
-                symbolTable[symbolIndex].set_n_strx(poolOffset);
+            for (std::pair<uint32_t, bool> symbolIndexAndIndirect : entry.second) {
+                if ( symbolIndexAndIndirect.second ) {
+                    // Indirect
+                    symbolTable[symbolIndexAndIndirect.first].set_n_value(poolOffset);
+                } else {
+                    symbolTable[symbolIndexAndIndirect.first].set_n_strx(poolOffset);
+                }
             }
             poolOffset += symName.size() + 1;
         }
@@ -85,7 +95,7 @@ public:
 
 
 private:
-    std::map<std::string, std::vector<uint32_t>> _map;
+    std::map<std::string, std::vector<std::pair<uint32_t, bool>>> _map;
 };
 
 
@@ -103,11 +113,12 @@ struct LocalSymbolInfo
 template <typename P>
 class LinkeditOptimizer {
 public:
-                    LinkeditOptimizer(void* cacheBuffer, macho_header<P>* mh, Diagnostics& diag);
+                    LinkeditOptimizer(const void* containerBuffer, macho_header<P>* mh, const char* dylibID,
+                                      Diagnostics& diag);
 
     uint32_t        linkeditSize() { return _linkeditSize; }
     uint64_t        linkeditAddr() { return _linkeditAddr; }
-    const char*     installName() { return _installName; }
+    const char*     dylibID() { return _dylibID; }
     void            copyWeakBindingInfo(uint8_t* newLinkEditContent, uint32_t& offset);
     void            copyLazyBindingInfo(uint8_t* newLinkEditContent, uint32_t& offset);
     void            copyBindingInfo(uint8_t* newLinkEditContent, uint32_t& offset);
@@ -124,6 +135,9 @@ public:
                                        uint32_t sharedSymbolTableStartOffset, uint32_t sharedSymbolTableCount,
                                        uint32_t sharedSymbolStringsOffset, uint32_t sharedSymbolStringsSize);
 
+    typedef CacheBuilder::DylibStripMode DylibStripMode;
+    void            setStripMode(DylibStripMode stripMode);
+
     macho_header<P>*                        machHeader() { return _mh; }
     const std::vector<const char*>          getDownwardDependents() { return _downDependentPaths; }
     const std::vector<const char*>          getAllDependents() { return _allDependentPaths; }
@@ -138,8 +152,11 @@ public:
     const std::vector<macho_segment_command<P>*>&  segCmds() { return _segCmds; }
 
 
-    static void optimizeLinkedit(CacheBuilder& builder);
-    static void mergeLinkedits(CacheBuilder& builder, std::vector<LinkeditOptimizer<P>*>& optimizers);
+    static void optimizeLinkedit(CacheBuilder& builder, const void* containerBuffer,
+                                 CacheBuilder::UnmappedRegion* localSymbolsRegion,
+                                 const std::vector<std::tuple<const mach_header*, const char*, DylibStripMode>>& images);
+    static void mergeLinkedits(CacheBuilder& builder, CacheBuilder::UnmappedRegion* localSymbolsRegion,
+                               std::vector<LinkeditOptimizer<P>*>& optimizers);
 
 private:
 
@@ -147,12 +164,12 @@ private:
     typedef typename P::E E;
 
     macho_header<P>*                        _mh;
-    void*                                   _cacheBuffer;
+    const void*                             _containerBuffer;
     Diagnostics&                            _diagnostics;
     uint32_t                                _linkeditSize        = 0;
     uint64_t                                _linkeditAddr        = 0;
     const uint8_t*                          _linkeditBias       = nullptr;
-    const char*                             _installName        = nullptr;
+    const char*                             _dylibID            = nullptr;
     macho_symtab_command<P>*                _symTabCmd          = nullptr;
     macho_dysymtab_command<P>*              _dynSymTabCmd       = nullptr;
     macho_dyld_info_command<P>*             _dyldInfo           = nullptr;
@@ -182,348 +199,14 @@ private:
     uint32_t                                _newDataInCodeOffset            = 0;
     uint32_t                                _newIndirectSymbolTableOffset   = 0;
     uint64_t                                _dyldSectionAddr                = 0;
-};
-
-
-
-template <typename P>
-class AcceleratorTables {
-public:
-                AcceleratorTables(DyldSharedCache* cache, uint64_t linkeditStartAddr, Diagnostics& diag, const std::vector<LinkeditOptimizer<P>*>& optimizers);
-
-    uint32_t    totalSize() const;
-    void        copyTo(uint8_t* buffer);
-
-private:
-    typedef typename P::E  E;
-
-    struct NodeChain;
-
-    struct DepNode {
-        std::vector<DepNode*>   _dependents;
-        unsigned                _depth;
-        const char*             _installName;
-
-                                DepNode() : _depth(0), _installName(nullptr) { }
-        void                    computeDepth();
-        static void             verifyUnreachable(DepNode* target, NodeChain& chain, Diagnostics& diag, std::unordered_set<DepNode*>& visitedNodes, const std::vector<DepNode*>& from);
-    };
-
-    struct NodeChain {
-        NodeChain*   prev;
-        DepNode*     node;
-    };
-
-    std::unordered_map<macho_header<P>*, DepNode>                   _depDAG;
-    std::vector<dyld_cache_image_info_extra>                        _extraInfo;
-    std::vector<uint8_t>                                            _trieBytes;
-    std::vector<uint16_t>                                           _reExportArray;
-    std::vector<uint16_t>                                           _dependencyArray;
-    std::vector<uint16_t>                                           _bottomUpArray;
-    std::vector<dyld_cache_accelerator_initializer>                 _initializers;
-    std::vector<dyld_cache_accelerator_dof>                         _dofSections;
-    std::vector<dyld_cache_range_entry>                             _rangeTable;
-    std::unordered_map<macho_header<P>*, uint32_t>                  _machHeaderToImageIndex;
-    std::unordered_map<std::string, macho_header<P>*>               _dylibPathToMachHeader;
-    std::unordered_map<macho_header<P>*, LinkeditOptimizer<P>*>     _machHeaderToOptimizer;
-    dyld_cache_accelerator_info                                     _acceleratorInfoHeader;
+    DylibStripMode                          _stripMode                 = DylibStripMode::stripAll;
 };
 
 
 template <typename P>
-void AcceleratorTables<P>::AcceleratorTables::DepNode::verifyUnreachable(AcceleratorTables<P>::DepNode* target, struct AcceleratorTables<P>::NodeChain& chain, Diagnostics& diag,
-                                                                        std::unordered_set<DepNode*>& visitedNodes, const std::vector<AcceleratorTables<P>::DepNode*>& from) {
-    for (DepNode* node : from) {
-        bool foundCycle = (node == target);
-        for (NodeChain* c = &chain; c->prev != nullptr; c = c->prev) {
-            if ( c->node == target ) {
-                foundCycle = true;
-                break;
-            }
-        }
-        if ( foundCycle ) {
-            NodeChain* chp = &chain;
-            std::string msg = std::string("found cycle for ") + target->_installName;
-            while (chp != nullptr) {
-                msg = msg + "\n  " + chp->node->_installName;
-                chp = chp->prev;
-            }
-            diag.warning("%s", msg.c_str());
-            return;
-        }
-
-        if ( visitedNodes.count(node) )
-            continue;
-        visitedNodes.insert(node);
-        NodeChain nextChain;
-        nextChain.prev = &chain;
-        nextChain.node = node;
-        verifyUnreachable(target, nextChain, diag, visitedNodes, node->_dependents);
-    }
-}
-
-const uint16_t kBranchIslandDylibIndex = 0x7FFF;
-
-template <typename P>
-AcceleratorTables<P>::AcceleratorTables(DyldSharedCache* cache, uint64_t linkeditStartAddr, Diagnostics& diag, const std::vector<LinkeditOptimizer<P>*>& optimizers)
-{
-    // build table mapping tables to map between mach_header, index, and optimizer
-    for ( LinkeditOptimizer<P>* op : optimizers ) {
-        _machHeaderToOptimizer[op->machHeader()] = op;
-    }
-    const dyld_cache_mapping_info* mappings = (dyld_cache_mapping_info*)((uint8_t*)cache + cache->header.mappingOffset);
-    uint64_t cacheStartAddress = mappings[0].address;
-    const dyld_cache_image_info* images = (dyld_cache_image_info*)((uint8_t*)cache + cache->header.imagesOffset);
-    for (unsigned i=0; i < cache->header.imagesCount; ++i) {
-        uint64_t segCacheFileOffset = images[i].address - cacheStartAddress;
-        macho_header<P>* mhMapped = (macho_header<P>*)((uint8_t*)cache+segCacheFileOffset);
-        const char* path = (char*)cache + images[i].pathFileOffset;
-        _dylibPathToMachHeader[path] = mhMapped;
-        // don't add alias entries (path offset in pool near start of cache) to header->index map
-        if ( images[i].pathFileOffset > segCacheFileOffset )
-            _machHeaderToImageIndex[mhMapped] = i;
-    }
-
-
-    // build DAG of image dependencies
-    for (LinkeditOptimizer<P>* op : optimizers) {
-        _depDAG[op->machHeader()]._installName = op->installName();
-    }
-    for (LinkeditOptimizer<P>* op : optimizers) {
-        DepNode& node = _depDAG[op->machHeader()];
-        for (const char* depPath : op->getDownwardDependents()) {
-            macho_header<P>* depMH = _dylibPathToMachHeader[depPath];
-            if ( depMH != nullptr ) {
-                DepNode* depNode = &_depDAG[depMH];
-                node._dependents.push_back(depNode);
-            }
-        }
-    }
-
-    // check for cycles in DAG
-    for (auto& entry : _depDAG) {
-        DepNode* node = &entry.second;
-        NodeChain chain;
-        chain.prev = nullptr;
-        chain.node = node;
-        std::unordered_set<DepNode*> visitedNodes;
-        DepNode::verifyUnreachable(node, chain, diag, visitedNodes, node->_dependents);
-    }
-
-    // compute depth for each DAG node
-    for (auto& entry : _depDAG) {
-        entry.second.computeDepth();
-    }
-
-    // build sorted (bottom up) list of images
-    std::vector<macho_header<P>*> sortedMachHeaders;
-    sortedMachHeaders.reserve(optimizers.size());
-    for (LinkeditOptimizer<P>* op : optimizers) {
-        if ( strcmp(op->installName(), "dyld_shared_cache_branch_islands") != 0 )
-            sortedMachHeaders.push_back(op->machHeader());
-        else
-            _machHeaderToImageIndex[op->machHeader()] = kBranchIslandDylibIndex;
-    }
-    std::sort(sortedMachHeaders.begin(), sortedMachHeaders.end(),
-              [&](macho_header<P>* lmh, macho_header<P>* rmh) -> bool {
-                if ( _depDAG[lmh]._depth != _depDAG[rmh]._depth )
-                    return (_depDAG[lmh]._depth < _depDAG[rmh]._depth);
-                else
-                    return (lmh < rmh);
-              });
-
-    // build zeroed array of extra infos
-    dyld_cache_image_info_extra emptyExtra;
-    emptyExtra.exportsTrieAddr              = 0;
-    emptyExtra.weakBindingsAddr             = 0;
-    emptyExtra.exportsTrieSize              = 0;
-    emptyExtra.weakBindingsSize             = 0;
-    emptyExtra.dependentsStartArrayIndex    = 0;
-    emptyExtra.reExportsStartArrayIndex     = 0;
-    _extraInfo.insert(_extraInfo.begin(), sortedMachHeaders.size(), emptyExtra);
-
-    //for ( macho_header<P>* mh : sortedMachHeaders ) {
-    //    fprintf(stderr, "depth: %3d mh: %p  path: %s\n", _depDAG[mh]._depth, mh, _machHeaderToOptimizer[mh]->installName());
-    //}
-
-    // build dependency table
-    _dependencyArray.push_back(0xFFFF); // reserve 0 slot to be "no-dependencies"
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        unsigned index = _machHeaderToImageIndex[mh];
-        auto depPaths = op->getAllDependents();
-        if ( depPaths.empty() ) {
-            _extraInfo[index].dependentsStartArrayIndex = 0;
-        }
-        else {
-            _extraInfo[index].dependentsStartArrayIndex = (uint32_t)_dependencyArray.size();
-            auto downPaths = op->getDownwardDependents();
-            for (const char* depPath : depPaths) {
-                macho_header<P>* depMH = _dylibPathToMachHeader[depPath];
-                uint16_t depIndex = _machHeaderToImageIndex[depMH];
-                if ( std::find(downPaths.begin(), downPaths.end(), depPath) == downPaths.end())
-                    depIndex |= 0x8000;
-                _dependencyArray.push_back(depIndex);
-            }
-            _dependencyArray.push_back(0xFFFF); // mark end of list
-       }
-    }
-
-    // build re-exports table
-    _reExportArray.push_back(0xFFFF); // reserve 0 slot to be "no-re-exports"
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        unsigned index = _machHeaderToImageIndex[mh];
-        auto reExPaths = op->getReExportPaths();
-        if ( reExPaths.empty() ) {
-            _extraInfo[index].reExportsStartArrayIndex = 0;
-        }
-        else {
-            _extraInfo[index].reExportsStartArrayIndex = (uint32_t)_reExportArray.size();
-            for (const char* reExPath : reExPaths) {
-                macho_header<P>* reExMH = _dylibPathToMachHeader[reExPath];
-                uint32_t reExIndex = _machHeaderToImageIndex[reExMH];
-                _reExportArray.push_back(reExIndex);
-            }
-            _reExportArray.push_back(0xFFFF); // mark end of list
-       }
-    }
-
-    // build ordered list of initializers
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        unsigned index = _machHeaderToImageIndex[mh];
-        _bottomUpArray.push_back(index);
-        for (uint64_t initializer : op->initializerAddresses()) {
-            //fprintf(stderr, "0x%08llX %s\n", initializer, op->installName());
-            dyld_cache_accelerator_initializer entry;
-            entry.functionOffset = (uint32_t)(initializer-cacheStartAddress);
-            entry.imageIndex = _machHeaderToImageIndex[mh];
-            _initializers.push_back(entry);
-        }
-    }
-
-    // build ordered list of DOF sections
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        assert(op != NULL);
-        unsigned imageIndex = _machHeaderToImageIndex[mh];
-        for (auto& sect : op->dofSections()) {
-            //fprintf(stderr, "0x%08llX %s\n", initializer, op->installName());
-            dyld_cache_accelerator_dof entry;
-            entry.sectionAddress = sect->addr();
-            entry.sectionSize    = (uint32_t)sect->size();
-            entry.imageIndex     = imageIndex;
-            _dofSections.push_back(entry);
-        }
-    }
-
-
-    // register exports trie and weak binding info in each dylib with image extra info
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        unsigned index = _machHeaderToImageIndex[mh];
-        _extraInfo[index].exportsTrieAddr  = op->exportsTrieLinkEditOffset() + linkeditStartAddr;
-        _extraInfo[index].exportsTrieSize  = op->exportsTrieLinkEditSize();
-        _extraInfo[index].weakBindingsAddr = op->weakBindingLinkEditOffset() + linkeditStartAddr;
-        _extraInfo[index].weakBindingsSize = op->weakBindingLinkEditSize();
-    }
-
-    // record location of __DATA/__dyld section in libdyld.dylib
-    macho_header<P>* libdyldMH = _dylibPathToMachHeader["/usr/lib/system/libdyld.dylib"];
-    LinkeditOptimizer<P>* libdyldOp = _machHeaderToOptimizer[libdyldMH];
-    uint64_t dyldSectionAddr = libdyldOp->dyldSectionAddress();
-
-    // build range table for fast address->image lookups
-    for (macho_header<P>* mh : sortedMachHeaders) {
-        LinkeditOptimizer<P>* op = _machHeaderToOptimizer[mh];
-        unsigned imageIndex = _machHeaderToImageIndex[mh];
-        for (const macho_segment_command<P>* segCmd : op->segCmds()) {
-            if ( strcmp(segCmd->segname(), "__LINKEDIT") == 0 )
-                continue;
-            dyld_cache_range_entry entry;
-            entry.startAddress = segCmd->vmaddr();
-            entry.size         = (uint32_t)segCmd->vmsize();
-            entry.imageIndex   = imageIndex;
-            _rangeTable.push_back(entry);
-        }
-    }
-    std::sort(_rangeTable.begin(), _rangeTable.end(),
-              [&](const dyld_cache_range_entry& lRange, const dyld_cache_range_entry& rRange) -> bool {
-                return (lRange.startAddress < rRange.startAddress);
-              });
-
-    // build trie that maps install names to image index
-    std::vector<DylibIndexTrie::Entry> dylibEntrys;
-    for (auto &x : _dylibPathToMachHeader) {
-        const std::string& path = x.first;
-        unsigned index = _machHeaderToImageIndex[x.second];
-        dylibEntrys.push_back(DylibIndexTrie::Entry(path, DylibIndex(index)));
-    }
-    DylibIndexTrie dylibsTrie(dylibEntrys);
-    dylibsTrie.emit(_trieBytes);
-    while ( (_trieBytes.size() % 4) != 0 )
-        _trieBytes.push_back(0);
-
-    // fill out header
-    _acceleratorInfoHeader.version              = 1;
-    _acceleratorInfoHeader.imageExtrasCount     = (uint32_t)_extraInfo.size();
-    _acceleratorInfoHeader.imagesExtrasOffset   = ALIGN_AS_TYPE(sizeof(dyld_cache_accelerator_info), dyld_cache_image_info_extra);
-    _acceleratorInfoHeader.bottomUpListOffset   = _acceleratorInfoHeader.imagesExtrasOffset + _acceleratorInfoHeader.imageExtrasCount*sizeof(dyld_cache_image_info_extra);
-    _acceleratorInfoHeader.dylibTrieOffset      = _acceleratorInfoHeader.bottomUpListOffset + _acceleratorInfoHeader.imageExtrasCount*sizeof(uint16_t);
-    _acceleratorInfoHeader.dylibTrieSize        = (uint32_t)_trieBytes.size();
-    _acceleratorInfoHeader.initializersOffset   = ALIGN_AS_TYPE(_acceleratorInfoHeader.dylibTrieOffset + _acceleratorInfoHeader.dylibTrieSize, dyld_cache_accelerator_initializer);
-    _acceleratorInfoHeader.initializersCount    = (uint32_t)_initializers.size();
-    _acceleratorInfoHeader.dofSectionsOffset    = ALIGN_AS_TYPE(_acceleratorInfoHeader.initializersOffset + _acceleratorInfoHeader.initializersCount*sizeof(dyld_cache_accelerator_initializer), dyld_cache_accelerator_initializer);
-    _acceleratorInfoHeader.dofSectionsCount     = (uint32_t)_dofSections.size();
-    _acceleratorInfoHeader.reExportListOffset   = ALIGN_AS_TYPE(_acceleratorInfoHeader.dofSectionsOffset + _acceleratorInfoHeader.dofSectionsCount*sizeof(dyld_cache_accelerator_dof), dyld_cache_accelerator_dof);
-    _acceleratorInfoHeader.reExportCount        = (uint32_t)_reExportArray.size();
-    _acceleratorInfoHeader.depListOffset        = ALIGN_AS_TYPE(_acceleratorInfoHeader.reExportListOffset + _acceleratorInfoHeader.reExportCount*sizeof(uint16_t), uint16_t);
-    _acceleratorInfoHeader.depListCount         = (uint32_t)_dependencyArray.size();
-    _acceleratorInfoHeader.rangeTableOffset     = ALIGN_AS_TYPE(_acceleratorInfoHeader.depListOffset + _acceleratorInfoHeader.depListCount*sizeof(uint16_t), dyld_cache_range_entry);
-    _acceleratorInfoHeader.rangeTableCount      = (uint32_t)_rangeTable.size();
-    _acceleratorInfoHeader.dyldSectionAddr      = dyldSectionAddr;
-}
-
-
-template <typename P>
-void AcceleratorTables<P>::DepNode::computeDepth()
-{
-    if ( _depth != 0 )
-        return;
-    _depth = 1;
-    for (DepNode* node : _dependents) {
-        node->computeDepth();
-        if ( node->_depth >= _depth )
-            _depth = node->_depth + 1;
-    }
-}
-
-template <typename P>
-uint32_t AcceleratorTables<P>::totalSize() const
-{
-    return (uint32_t)align(_acceleratorInfoHeader.rangeTableOffset + _acceleratorInfoHeader.rangeTableCount*sizeof(dyld_cache_range_entry), 14);
-}
-
-template <typename P>
-void AcceleratorTables<P>::copyTo(uint8_t* buffer)
-{
-    memcpy(buffer, &_acceleratorInfoHeader, sizeof(dyld_cache_accelerator_info));
-    memcpy(&buffer[_acceleratorInfoHeader.imagesExtrasOffset], &_extraInfo[0],       _extraInfo.size()*sizeof(dyld_cache_image_info_extra));
-    memcpy(&buffer[_acceleratorInfoHeader.bottomUpListOffset], &_bottomUpArray[0],   _bottomUpArray.size()*sizeof(uint16_t));
-    memcpy(&buffer[_acceleratorInfoHeader.initializersOffset], &_initializers[0],    _initializers.size()*sizeof(dyld_cache_accelerator_initializer));
-    memcpy(&buffer[_acceleratorInfoHeader.reExportListOffset], &_reExportArray[0],   _reExportArray.size()*sizeof(uint16_t));
-    memcpy(&buffer[_acceleratorInfoHeader.dofSectionsOffset],  &_dofSections[0],     _dofSections.size()*sizeof(dyld_cache_accelerator_dof));
-    memcpy(&buffer[_acceleratorInfoHeader.depListOffset],      &_dependencyArray[0], _dependencyArray.size()*sizeof(uint16_t));
-    memcpy(&buffer[_acceleratorInfoHeader.rangeTableOffset],   &_rangeTable[0],      _rangeTable.size()*sizeof(dyld_cache_range_entry));
-    memcpy(&buffer[_acceleratorInfoHeader.dylibTrieOffset],    &_trieBytes[0],       _trieBytes.size());
-}
-
-
-
-template <typename P>
-LinkeditOptimizer<P>::LinkeditOptimizer(void* cacheBuffer, macho_header<P>* mh, Diagnostics& diag)
-: _mh(mh), _cacheBuffer(cacheBuffer), _diagnostics(diag)
+LinkeditOptimizer<P>::LinkeditOptimizer(const void* containerBuffer, macho_header<P>* mh,
+                                        const char* dylibID, Diagnostics& diag)
+: _mh(mh), _dylibID(dylibID), _containerBuffer(containerBuffer), _diagnostics(diag)
 {
     const unsigned origLoadCommandsSize = mh->sizeofcmds();
     unsigned bytesRemaining = origLoadCommandsSize;
@@ -539,9 +222,6 @@ LinkeditOptimizer<P>::LinkeditOptimizer(void* cacheBuffer, macho_header<P>* mh,
     for (uint32_t i = 0; i < cmdCount; ++i) {
         bool remove = false;
         switch (cmd->cmd()) {
-            case LC_ID_DYLIB:
-                _installName = ((macho_dylib_command<P>*)cmd)->name();
-                break;
             case LC_SYMTAB:
                 _symTabCmd = (macho_symtab_command<P>*)cmd;
                 break;
@@ -621,6 +301,7 @@ LinkeditOptimizer<P>::LinkeditOptimizer(void* cacheBuffer, macho_header<P>* mh,
                     }
                 }
                 break;
+            case LC_DYLD_CHAINED_FIXUPS:
             case LC_SEGMENT_SPLIT_INFO:
                 remove = true;
                 break;
@@ -643,6 +324,11 @@ LinkeditOptimizer<P>::LinkeditOptimizer(void* cacheBuffer, macho_header<P>* mh,
     mh->set_sizeofcmds(origLoadCommandsSize - bytesRemaining);
 }
 
+template <typename P>
+void LinkeditOptimizer<P>::setStripMode(DylibStripMode stripMode) {
+    _stripMode = stripMode;
+}
+
 /*
 static void dumpLoadCommands(const uint8_t* mheader)
 {
@@ -746,6 +432,7 @@ void LinkeditOptimizer<P>::updateLoadCommands(uint32_t mergedLinkeditStartOffset
     _symTabCmd->set_strsize(sharedSymbolStringsSize);
 
     // update dynamic symbol table to have proper offsets into shared symbol table
+    if ( _dynSymTabCmd != nullptr ) {
         _dynSymTabCmd->set_ilocalsym(0);
         _dynSymTabCmd->set_nlocalsym(_newLocalSymbolCount);
         _dynSymTabCmd->set_iextdefsym(_newExportedSymbolsStartIndex-_newLocalSymbolsStartIndex);
@@ -760,6 +447,7 @@ void LinkeditOptimizer<P>::updateLoadCommands(uint32_t mergedLinkeditStartOffset
         _dynSymTabCmd->set_extreloff(0);
         _dynSymTabCmd->set_locreloff(0);
         _dynSymTabCmd->set_nlocrel(0);
+    }
 
     // update dyld info
     if ( _dyldInfo != nullptr ) {
@@ -867,11 +555,27 @@ void LinkeditOptimizer<P>::copyLocalSymbols(uint8_t* newLinkEditContent, SortedS
                                             bool redact, std::vector<LocalSymbolInfo>& localSymbolInfos,
                                             std::vector<macho_nlist<P>>& unmappedLocalSymbols, SortedStringPool<P>& localSymbolsStringPool)
 {
-    LocalSymbolInfo localInfo;
-    localInfo.dylibOffset = (uint32_t)(((uint8_t*)_mh) - (uint8_t*)_cacheBuffer);
+    localSymbolInfos.push_back(LocalSymbolInfo());
+
+    LocalSymbolInfo& localInfo = localSymbolInfos.back();
+    localInfo.dylibOffset = (uint32_t)(((uint8_t*)_mh) - (uint8_t*)_containerBuffer);
     localInfo.nlistStartIndex = (uint32_t)unmappedLocalSymbols.size();
     localInfo.nlistCount = 0;
     _newLocalSymbolsStartIndex = symbolIndex;
+    _newLocalSymbolCount = 0;
+
+    switch (_stripMode) {
+        case CacheBuilder::DylibStripMode::stripNone:
+        case CacheBuilder::DylibStripMode::stripExports:
+            break;
+        case CacheBuilder::DylibStripMode::stripLocals:
+        case CacheBuilder::DylibStripMode::stripAll:
+            return;
+    }
+
+    if ( _dynSymTabCmd == nullptr )
+        return;
+
     const char* strings = (char*)&_linkeditBias[_symTabCmd->stroff()];
     const macho_nlist<P>* const symbolTable = (macho_nlist<P>*)(&_linkeditBias[_symTabCmd->symoff()]);
     const macho_nlist<P>* const firstExport = &symbolTable[_dynSymTabCmd->ilocalsym()];
@@ -904,7 +608,6 @@ void LinkeditOptimizer<P>::copyLocalSymbols(uint8_t* newLinkEditContent, SortedS
     }
     _newLocalSymbolCount = symbolIndex - _newLocalSymbolsStartIndex;
     localInfo.nlistCount = (uint32_t)unmappedLocalSymbols.size() - localInfo.nlistStartIndex;
-    localSymbolInfos.push_back(localInfo);
 }
 
 
@@ -912,6 +615,20 @@ template <typename P>
 void LinkeditOptimizer<P>::copyExportedSymbols(uint8_t* newLinkEditContent, SortedStringPool<P>& stringPool, uint32_t& offset, uint32_t& symbolIndex)
 {
     _newExportedSymbolsStartIndex = symbolIndex;
+    _newExportedSymbolCount = 0;
+
+    switch (_stripMode) {
+        case CacheBuilder::DylibStripMode::stripNone:
+        case CacheBuilder::DylibStripMode::stripLocals:
+            break;
+        case CacheBuilder::DylibStripMode::stripExports:
+        case CacheBuilder::DylibStripMode::stripAll:
+            return;
+    }
+
+    if ( _dynSymTabCmd == nullptr )
+        return;
+
     const char* strings = (char*)&_linkeditBias[_symTabCmd->stroff()];
     const macho_nlist<P>* const symbolTable = (macho_nlist<P>*)(&_linkeditBias[_symTabCmd->symoff()]);
     const macho_nlist<P>* const firstExport = &symbolTable[_dynSymTabCmd->iextdefsym()];
@@ -940,6 +657,20 @@ template <typename P>
 void LinkeditOptimizer<P>::copyImportedSymbols(uint8_t* newLinkEditContent, SortedStringPool<P>& stringPool, uint32_t& offset, uint32_t& symbolIndex)
 {
     _newImportedSymbolsStartIndex = symbolIndex;
+    _newImportedSymbolCount = 0;
+
+    if ( _dynSymTabCmd == nullptr )
+        return;
+
+    switch (_stripMode) {
+        case CacheBuilder::DylibStripMode::stripNone:
+            break;
+        case CacheBuilder::DylibStripMode::stripLocals:
+        case CacheBuilder::DylibStripMode::stripExports:
+        case CacheBuilder::DylibStripMode::stripAll:
+            return;
+    }
+
     const char* strings = (char*)&_linkeditBias[_symTabCmd->stroff()];
     const macho_nlist<P>* const symbolTable = (macho_nlist<P>*)(&_linkeditBias[_symTabCmd->symoff()]);
     const macho_nlist<P>* const firstImport = &symbolTable[_dynSymTabCmd->iundefsym()];
@@ -964,6 +695,10 @@ template <typename P>
 void LinkeditOptimizer<P>::copyIndirectSymbolTable(uint8_t* newLinkEditContent, uint32_t& offset)
 {
     _newIndirectSymbolTableOffset = offset;
+
+    if ( _dynSymTabCmd == nullptr )
+        return;
+
      const uint32_t* const indirectTable = (uint32_t*)&_linkeditBias[_dynSymTabCmd->indirectsymoff()];
     uint32_t* newIndirectTable = (uint32_t*)&newLinkEditContent[offset];
     for (uint32_t i=0; i < _dynSymTabCmd->nindirectsyms(); ++i) {
@@ -977,7 +712,9 @@ void LinkeditOptimizer<P>::copyIndirectSymbolTable(uint8_t* newLinkEditContent,
 }
 
 template <typename P>
-void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<LinkeditOptimizer<P>*>& optimizers)
+void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder,
+                                          CacheBuilder::UnmappedRegion* localSymbolsRegion,
+                                          std::vector<LinkeditOptimizer<P>*>& optimizers)
 {
     // allocate space for new linkedit data
     uint64_t totalUnoptLinkeditsSize = builder._readOnlyRegion.sizeInUse - builder._nonLinkEditReadOnlySize;
@@ -1027,9 +764,11 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
         builder._diagnostics.verbose("  lazy bindings size:      %5uKB\n", (offset-startLazyBindingsInfosOffset)/1024);
     }
 
+    bool unmapLocals = ( builder._options.localSymbolMode == DyldSharedCache::LocalSymbolsMode::unmap );
+
     // copy symbol table entries
     std::vector<macho_nlist<P>> unmappedLocalSymbols;
-    if ( builder._options.excludeLocalSymbols )
+    if ( unmapLocals )
         unmappedLocalSymbols.reserve(0x01000000);
     std::vector<LocalSymbolInfo> localSymbolInfos;
         localSymbolInfos.reserve(optimizers.size());
@@ -1039,7 +778,7 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
     uint32_t sharedSymbolTableExportsCount = 0;
     uint32_t sharedSymbolTableImportsCount = 0;
     for (LinkeditOptimizer<P>* op : optimizers) {
-         op->copyLocalSymbols(newLinkEdit, stringPool, offset, symbolIndex, builder._options.excludeLocalSymbols,
+         op->copyLocalSymbols(newLinkEdit, stringPool, offset, symbolIndex, unmapLocals,
                              localSymbolInfos, unmappedLocalSymbols, localSymbolsStringPool);
         uint32_t x = symbolIndex;
         op->copyExportedSymbols(newLinkEdit, stringPool, offset, symbolIndex);
@@ -1081,7 +820,6 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
     uint64_t newLinkeditAlignedSize = align(offset, 14);
     builder._diagnostics.verbose("  symbol table size:       %5uKB (%d exports, %d imports)\n", (sharedSymbolTableEndOffset-sharedSymbolTableStartOffset)/1024, sharedSymbolTableExportsCount, sharedSymbolTableImportsCount);
     builder._diagnostics.verbose("  symbol string pool size: %5uKB\n", sharedSymbolStringsSize/1024);
-    builder._sharedStringsPoolVmOffset = (uint32_t)((builder._readOnlyRegion.unslidLoadAddress - builder._readExecuteRegion.unslidLoadAddress) + builder._nonLinkEditReadOnlySize + sharedSymbolStringsOffset);
 
     // overwrite mapped LINKEDIT area in cache with new merged LINKEDIT content
     builder._diagnostics.verbose("LINKEDITS optimized from %uMB to %uMB\n", (uint32_t)totalUnoptLinkeditsSize/(1024*1024), (uint32_t)newLinkeditUnalignedSize/(1024*1024));
@@ -1089,27 +827,8 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
     ::free(newLinkEdit);
     builder._readOnlyRegion.sizeInUse = builder._nonLinkEditReadOnlySize + newLinkeditAlignedSize;
 
-    // If making cache for customers, add extra accelerator tables for dyld
-    DyldSharedCache* cacheHeader = (DyldSharedCache*)builder._readExecuteRegion.buffer;
-    if ( builder._options.optimizeStubs ) {
-        uint64_t addrWhereAccTablesWillBe     = builder._readOnlyRegion.unslidLoadAddress+builder._readOnlyRegion.sizeInUse;
-        uint64_t addrWhereMergedLinkWillStart = builder._readOnlyRegion.unslidLoadAddress+builder._nonLinkEditReadOnlySize;
-        AcceleratorTables<P> tables(cacheHeader, addrWhereMergedLinkWillStart, builder._diagnostics, optimizers);
-        uint32_t tablesSize = tables.totalSize();
-        if ( tablesSize < (builder._readOnlyRegion.bufferSize - builder._readOnlyRegion.sizeInUse) ) {
-            tables.copyTo(builder._readOnlyRegion.buffer+builder._readOnlyRegion.sizeInUse);
-            cacheHeader->header.accelerateInfoAddr = addrWhereAccTablesWillBe;
-            cacheHeader->header.accelerateInfoSize = tablesSize;
-            builder._readOnlyRegion.sizeInUse += align(tablesSize, 14);
-            builder._diagnostics.verbose("Accelerator tables %uMB\n", (uint32_t)tablesSize/(1024*1024));
-       }
-        else {
-            builder._diagnostics.warning("not enough room to add dyld accelerator tables");
-        }
-    }
-
     // overwrite end of un-opt linkedits to create a new unmapped region for local symbols
-    if ( builder._options.excludeLocalSymbols ) {
+    if ( unmapLocals ) {
         const uint32_t entriesOffset = sizeof(dyld_cache_local_symbols_info);
         const uint32_t entriesCount  = (uint32_t)localSymbolInfos.size();
         const uint32_t nlistOffset   = (uint32_t)align(entriesOffset + entriesCount * sizeof(dyld_cache_local_symbols_info), 4); // 16-byte align start
@@ -1140,12 +859,10 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
             ::memcpy(newLocalsSymbolTable, &unmappedLocalSymbols[0], nlistCount*sizeof(macho_nlist<P>));
             // copy string pool
             localSymbolsStringPool.copyPoolAndUpdateOffsets(((char*)infoHeader)+stringsOffset, newLocalsSymbolTable);
-            // update cache header
-            cacheHeader->header.localSymbolsSize    = localsBufferSize;
             // return buffer of local symbols, caller to free() it
-            builder._localSymbolsRegion.buffer      = (uint8_t*)localsBuffer;
-            builder._localSymbolsRegion.bufferSize  = localsBufferSize;
-            builder._localSymbolsRegion.sizeInUse   = localsBufferSize;
+            localSymbolsRegion->buffer      = (uint8_t*)localsBuffer;
+            localSymbolsRegion->bufferSize  = localsBufferSize;
+            localSymbolsRegion->sizeInUse   = localsBufferSize;
         }
         else {
             builder._diagnostics.warning("could not allocate local symbols");
@@ -1164,14 +881,16 @@ void LinkeditOptimizer<P>::mergeLinkedits(CacheBuilder& builder, std::vector<Lin
 
 
 template <typename P>
-void LinkeditOptimizer<P>::optimizeLinkedit(CacheBuilder& builder)
+void LinkeditOptimizer<P>::optimizeLinkedit(CacheBuilder& builder, const void* containerBuffer,
+                                            CacheBuilder::UnmappedRegion* localSymbolsRegion,
+                                            const std::vector<std::tuple<const mach_header*, const char*, DylibStripMode>>& images)
 {
-    DyldSharedCache* cache = (DyldSharedCache*)builder._readExecuteRegion.buffer;
     // construct a LinkeditOptimizer for each image
     __block std::vector<LinkeditOptimizer<P>*> optimizers;
-    cache->forEachImage(^(const mach_header* mh, const char*) {
-        optimizers.push_back(new LinkeditOptimizer<P>(cache, (macho_header<P>*)mh, builder._diagnostics));
-    });
+    for (std::tuple<const mach_header*, const char*, DylibStripMode> image : images) {
+        optimizers.push_back(new LinkeditOptimizer<P>(containerBuffer, (macho_header<P>*)std::get<0>(image), std::get<1>(image), builder._diagnostics));
+        optimizers.back()->setStripMode(std::get<2>(image));
+    }
 #if 0
     // add optimizer for each branch pool
     for (uint64_t poolOffset : branchPoolOffsets) {
@@ -1180,20 +899,24 @@ void LinkeditOptimizer<P>::optimizeLinkedit(CacheBuilder& builder)
     }
 #endif
     // merge linkedit info
-    mergeLinkedits(builder, optimizers);
+    mergeLinkedits(builder, localSymbolsRegion, optimizers);
 
     // delete optimizers
     for (LinkeditOptimizer<P>* op : optimizers)
         delete op;
 }
 
-void CacheBuilder::optimizeLinkedit()
+void CacheBuilder::optimizeLinkedit(UnmappedRegion* localSymbolsRegion,
+                                    const std::vector<std::tuple<const mach_header*, const char*, DylibStripMode>>& images)
 {
-    if ( _archLayout->is64 ) {
-        return LinkeditOptimizer<Pointer64<LittleEndian>>::optimizeLinkedit(*this);
+    const void* buffer = (const void*)_fullAllocatedBuffer;
+    if ( _is64 ) {
+        return LinkeditOptimizer<Pointer64<LittleEndian>>::optimizeLinkedit(*this, buffer,
+                                                                            localSymbolsRegion, images);
     }
     else {
-         return LinkeditOptimizer<Pointer32<LittleEndian>>::optimizeLinkedit(*this);
+         return LinkeditOptimizer<Pointer32<LittleEndian>>::optimizeLinkedit(*this, buffer,
+                                                                             localSymbolsRegion, images);
     }
 }
 

dyld3/shared-cache/OptimizerObjC.cpp

@@ -32,11 +32,12 @@
 
 #include "DyldSharedCache.h"
 #include "Diagnostics.h"
-#include "CacheBuilder.h"
+#include "SharedCacheBuilder.h"
 #include "FileAbstraction.hpp"
 #include "MachOFileAbstraction.hpp"
 #include "MachOLoaded.h"
 #include "MachOAnalyzer.h"
+#include "MachOAnalyzerSet.h"
 
 #ifndef MH_HAS_OBJC
   #define MH_HAS_OBJC            0x40000000
@@ -117,27 +118,11 @@ public:
         _cacheStart         = (uint8_t*)cache;
         _cacheUnslideAddr   = cache->unslidLoadAddress();
         _slide              = (uint64_t)cache - _cacheUnslideAddr;
-#if SUPPORT_ARCH_arm64e
-        _chainedFixups      = (strcmp(cache->archName(), "arm64e") == 0);
-#else
-        _chainedFixups      = false;
-#endif
      }
 
     // Converts from an on disk vmAddr to the real vmAddr
     // That is, for a chained fixup, decodes the chain, for a non-chained fixup, does nothing.
     uint64_t vmAddrForOnDiskVMAddr(uint64_t vmaddr) {
-        if ( _chainedFixups ) {
-            dyld3::MachOLoaded::ChainedFixupPointerOnDisk ptr;
-            ptr.raw64 = vmaddr;
-            assert(ptr.arm64e.authRebase.bind == 0);
-            if ( ptr.arm64e.authRebase.auth ) {
-                vmaddr = _cacheUnslideAddr + ptr.arm64e.authRebase.target;
-            }
-            else {
-                vmaddr = ptr.arm64e.unpackTarget();
-            }
-        }
         return vmaddr;
     }
 
@@ -164,7 +149,6 @@ private:
     uint64_t                        _slide;
     uint64_t                        _cacheUnslideAddr;
     uint8_t*                        _cacheStart;
-    bool                            _chainedFixups;
 };
 
 
@@ -328,7 +312,7 @@ public:
     {
         if (cls->isMetaClass(cache)) return;
 
-        const char *name = cls->getName(cache);
+        const char* name = cls->getName(cache);
         uint64_t name_vmaddr = cache->vmAddrForContent((void*)name);
         uint64_t cls_vmaddr = cache->vmAddrForContent(cls);
         uint64_t hinfo_vmaddr = cache->vmAddrForContent(_hInfos.hinfoForHeader(cache, header));
@@ -348,6 +332,404 @@ public:
     size_t count() const { return _count; }
 };
 
+
+/// Builds a map from (install name, class name, method name) to actual IMPs
+template <typename P>
+class IMPMapBuilder
+{
+private:
+    typedef typename P::uint_t pint_t;
+
+public:
+
+    struct MapKey {
+        std::string_view installName;
+        std::string_view className;
+        std::string_view methodName;
+        bool isInstanceMethod;
+
+        bool operator==(const MapKey& other) const {
+            return isInstanceMethod == other.isInstanceMethod &&
+                    installName == other.installName &&
+                    className == other.className &&
+                    methodName == other.methodName;
+        }
+
+        size_t hash() const {
+            std::size_t seed = 0;
+            seed ^= std::hash<std::string_view>()(installName) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+            seed ^= std::hash<std::string_view>()(className) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+            seed ^= std::hash<std::string_view>()(methodName) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+            seed ^= std::hash<bool>()(isInstanceMethod) + 0x9e3779b9 + (seed<<6) + (seed>>2);
+            return seed;
+        }
+    };
+
+    struct MapKeyHasher {
+        size_t operator()(const MapKey& k) const {
+            return k.hash();
+        }
+    };
+
+    std::unordered_map<MapKey, pint_t, MapKeyHasher> impMap;
+    bool relativeMethodListSelectorsAreDirect;
+
+    IMPMapBuilder(bool relativeMethodListSelectorsAreDirect)
+        : relativeMethodListSelectorsAreDirect(relativeMethodListSelectorsAreDirect) { }
+
+    void visitClass(ContentAccessor* cache,
+                    const macho_header<P>* header,
+                    objc_class_t<P>* cls)
+    {
+        objc_method_list_t<P> *methodList = cls->getMethodList(cache);
+        if (methodList == nullptr) return;
+
+        const dyld3::MachOAnalyzer* ma = (const dyld3::MachOAnalyzer*)header;
+        bool isInstanceMethod = !cls->isMetaClass(cache);
+        const char* className = cls->getName(cache);
+        const char* installName = ma->installName();
+
+        for (uint32_t n = 0; n < methodList->getCount(); n++) {
+            // do not clobber an existing entry if any, because categories win
+            impMap.try_emplace(MapKey{
+                .installName = installName,
+                .className = className,
+                .methodName = methodList->getStringName(cache, n, relativeMethodListSelectorsAreDirect),
+                .isInstanceMethod = isInstanceMethod
+            }, methodList->getImp(n, cache));
+        }
+    }
+
+    void visit(ContentAccessor* cache, const macho_header<P>* header) {
+        const dyld3::MachOAnalyzer* ma = (const dyld3::MachOAnalyzer*)header;
+
+        // Method lists from categories
+        PointerSection<P, objc_category_t<P> *>
+            cats(cache, header, "__DATA", "__objc_catlist");
+        for (pint_t i = 0; i < cats.count(); i++) {
+            objc_category_t<P> *cat = cats.get(i);
+            objc_class_t<P>* cls = cat->getClass(cache);
+            if (cls == nullptr)
+                continue;
+
+            objc_method_list_t<P> *instanceMethods = cat->getInstanceMethods(cache);
+            if (instanceMethods != nullptr) {
+                for (uint32_t n = 0; n < instanceMethods->getCount(); n++) {
+                    MapKey k {
+                        .installName = ma->installName(),
+                        .className = cls->getName(cache),
+                        .methodName = instanceMethods->getStringName(cache, n, relativeMethodListSelectorsAreDirect),
+                        .isInstanceMethod = true
+                    };
+                    //printf("Adding %s %s %s %d cat %s\n", k.installName.data(), k.className.data(), k.methodName.data(), k.isInstanceMethod, k.catName->data());
+                    impMap[k] = instanceMethods->getImp(n, cache);
+                }
+            }
+            objc_method_list_t<P> *classMethods = cat->getClassMethods(cache);
+            if (classMethods != nullptr) {
+                for (uint32_t n = 0; n < classMethods->getCount(); n++) {
+                    MapKey k {
+                        .installName = ma->installName(),
+                        .className = cls->getName(cache),
+                        .methodName = classMethods->getStringName(cache, n, relativeMethodListSelectorsAreDirect),
+                        .isInstanceMethod = false
+                    };
+                    //printf("Adding %s %s %s %d cat %s\n", k.installName.data(), k.className.data(), k.methodName.data(), k.isInstanceMethod, k.catName->data());
+                    impMap[k] = classMethods->getImp(n, cache);
+                }
+            }
+        }
+    }
+};
+
+// List of offsets in libobjc that the shared cache optimization needs to use.
+template <typename T>
+struct objc_opt_imp_caches_pointerlist_tt {
+    T selectorStringVMAddrStart;
+    T selectorStringVMAddrEnd;
+    T inlinedSelectorsVMAddrStart;
+    T inlinedSelectorsVMAddrEnd;
+};
+
+template <typename P>
+class IMPCachesEmitter
+{
+    typedef typename P::uint_t pint_t;
+
+private:
+    Diagnostics&                 diag;
+    const IMPMapBuilder<P>&      impMapBuilder;
+    uint64_t                     selectorStringVMAddr;
+    uint8_t*&                    readOnlyBuffer;
+    size_t&                      readOnlyBufferSize;
+    uint8_t*&                    readWriteBuffer;
+    size_t&                      readWriteBufferSize;
+    CacheBuilder::ASLR_Tracker& aslrTracker;
+
+    std::map<std::string_view, const CacheBuilder::DylibInfo*> _dylibInfos;
+    std::map<std::string_view, const macho_header<P>*> _dylibs;
+    const std::vector<const IMPCaches::Selector*> inlinedSelectors;
+
+    struct ImpCacheHeader {
+        int32_t  fallback_class_offset;
+        uint32_t cache_shift :  5;
+        uint32_t cache_mask  : 11;
+        uint32_t occupied    : 14;
+        uint32_t has_inlines :  1;
+        uint32_t bit_one     :  1;
+    };
+
+    struct ImpCacheEntry {
+        uint32_t selOffset;
+        uint32_t impOffset;
+    };
+
+public:
+
+    static size_t sizeForImpCacheWithCount(int entries) {
+        return sizeof(ImpCacheHeader) + entries * sizeof(ImpCacheEntry);
+    }
+
+    struct ImpCacheContents {
+        struct bucket_t {
+            uint32_t sel_offset = 0;
+            uint64_t imp = 0;
+        };
+        std::vector<bucket_t>   buckets;
+        uint64_t                occupiedBuckets = 0;
+        bool hasInlines = false;
+
+        uint64_t capacity() const
+        {
+            return buckets.size();
+        }
+
+        uint64_t occupied() const {
+            return occupiedBuckets;
+        }
+
+        void incrementOccupied() {
+            ++occupiedBuckets;
+        }
+
+        void insert(uint64_t slot, uint64_t selOffset, uint64_t imp) {
+            bucket_t& b = buckets[slot];
+            assert(b.imp == 0);
+
+            if (!b.imp) incrementOccupied();
+            assert((uint32_t)selOffset == selOffset);
+            b.sel_offset = (uint32_t)selOffset;
+            b.imp = imp;
+        }
+
+        void fillBuckets(const IMPCaches::ClassData* classData, bool metaclass, const IMPMapBuilder<P> & classRecorder) {
+            const std::vector<IMPCaches::ClassData::Method> & methods = classData->methods;
+            buckets.resize(classData->modulo());
+            for (const IMPCaches::ClassData::Method& method : methods) {
+                typename IMPMapBuilder<P>::MapKey k {
+                    .installName = method.installName,
+                    .className = method.className,
+                    .methodName = method.selector->name,
+                    .isInstanceMethod = !metaclass
+                };
+
+                pint_t imp = classRecorder.impMap.at(k);
+                int slot = (method.selector->inProgressBucketIndex >> classData->shift) & classData->mask();
+                insert(slot, method.selector->offset, imp);
+                hasInlines |= (method.wasInlined && !method.fromFlattening);
+            }
+        }
+
+        std::pair<uint64_t, uint64_t>
+        write(ContentAccessor* cache,
+              uint64_t cacheSelectorStringVMAddr, uint64_t clsVMAddr,
+              uint8_t*& buf, size_t& bufSize, Diagnostics& diags) {
+            constexpr bool log = false;
+            uint64_t spaceRequired = sizeof(ImpCacheEntry) * capacity();
+
+            if (spaceRequired > bufSize) {
+                diags.error("Not enough space for imp cache");
+                return { 0, 0 };
+            }
+
+            // Convert from addresses to offsets and write out
+            ImpCacheEntry* offsetBuckets = (ImpCacheEntry*)buf;
+            // printf("Buckets: 0x%08llx\n", cache->vmAddrForContent(offsetBuckets));
+            for (uint64_t index = 0; index != buckets.size(); ++index) {
+                bucket_t bucket = buckets[index];
+                if (bucket.sel_offset == 0 && bucket.imp == 0) {
+                    // Empty bucket
+                    offsetBuckets[index].selOffset = 0xFFFFFFFF;
+                    offsetBuckets[index].impOffset = 0;
+                } else {
+                    int64_t selOffset = (int64_t)bucket.sel_offset;
+                    int64_t impOffset = clsVMAddr - bucket.imp;
+                    assert((int32_t)impOffset == impOffset);
+                    assert((int32_t)selOffset == selOffset);
+                    offsetBuckets[index].selOffset = (int32_t)selOffset;
+                    offsetBuckets[index].impOffset = (int32_t)impOffset;
+                    if (log) {
+                        diags.verbose("[IMP Caches] Coder[%lld]: %#08llx (sel: %#08x, imp %#08x) %s\n", index,
+                           cache->vmAddrForOnDiskVMAddr(bucket.imp),
+                           (int32_t)selOffset, (int32_t)impOffset,
+                           (const char*)cache->contentForVMAddr(cacheSelectorStringVMAddr + bucket.sel_offset));
+                    }
+                }
+            }
+
+            buf += spaceRequired;
+            bufSize -= spaceRequired;
+
+            return { cache->vmAddrForContent(offsetBuckets), (uint64_t)buckets.size() };
+        }
+    };
+
+    IMPCachesEmitter(Diagnostics& diags, const IMPMapBuilder<P>& builder, uint64_t selectorStringVMAddr, uint8_t*& roBuf, size_t& roBufSize, uint8_t* &rwBuf, size_t& rwBufSize, const std::vector<CacheBuilder::DylibInfo> & dylibInfos, const std::vector<const macho_header<P>*> & dylibs, CacheBuilder::ASLR_Tracker& tracker)
+        : diag(diags), impMapBuilder(builder), selectorStringVMAddr(selectorStringVMAddr), readOnlyBuffer(roBuf), readOnlyBufferSize(roBufSize), readWriteBuffer(rwBuf), readWriteBufferSize(rwBufSize), aslrTracker(tracker) {
+            for (const CacheBuilder::DylibInfo& d : dylibInfos) {
+                _dylibInfos[d.dylibID] = &d;
+            }
+            for (const macho_header<P>* d : dylibs) {
+                const dyld3::MachOAnalyzer* ma = (const dyld3::MachOAnalyzer*) d;
+                _dylibs[ma->installName()] = d;
+            }
+        }
+
+    // Returns true if we should filter this class out from getting an imp cache
+    bool filter(ContentAccessor* cache, const dyld3::MachOAnalyzer* ma, const objc_class_t<P>* cls) {
+        const CacheBuilder::DylibInfo* d = _dylibInfos[ma->installName()];
+        IMPCaches::ClassKey key {
+            .name = cls->getName(cache),
+            .metaclass = cls->isMetaClass(cache)
+        };
+        return (d->impCachesClassData.find(key) == d->impCachesClassData.end());
+    }
+
+    void visitClass(ContentAccessor* cache,
+                    const macho_header<P>* header,
+                    objc_class_t<P>* cls)
+    {
+        // If we ran out of space then don't try to optimize more
+        if (diag.hasError())
+            return;
+
+        const dyld3::MachOAnalyzer* ma = (const dyld3::MachOAnalyzer*) header;
+        if (filter(cache, ma, cls)) {
+            *cls->getVTableAddress() = 0;
+            return;
+        }
+
+        const char* className = cls->getName(cache);
+
+        if (cls->getVTable(cache) != 0) {
+            diag.error("Class '%s' has non-zero vtable\n", className);
+            return;
+        }
+
+        const CacheBuilder::DylibInfo* d = _dylibInfos[ma->installName()];
+        IMPCaches::ClassKey key {
+            .name = cls->getName(cache),
+            .metaclass = cls->isMetaClass(cache)
+        };
+        IMPCaches::ClassData* data = (d->impCachesClassData.at(key)).get();
+#if 0
+        for (const objc_method_t<P>& method : methods) {
+            printf("  0x%llx: 0x%llx (%s)\n", method.getImp(), method.getName(),
+                   (const char*)cache->contentForVMAddr(method.getName()));
+        }
+#endif
+
+        uint64_t clsVMAddr = cache->vmAddrForContent(cls);
+
+        if (data->mask() > 0x7ff) {
+            diag.verbose("Cache for class %s (%#08llx) is too large (mask: %#x)\n",
+                         className, clsVMAddr, data->mask());
+            return;
+        }
+
+        ImpCacheContents impCache;
+        impCache.fillBuckets(data, cls->isMetaClass(cache), impMapBuilder);
+
+        constexpr bool log = false;
+        if (log) {
+            printf("Writing cache for %sclass %s (%#08llx)\n", cls->isMetaClass(cache) ? "meta" : "", className, clsVMAddr);
+        }
+
+        struct ImpCacheHeader {
+            int32_t  fallback_class_offset;
+            uint32_t cache_shift :  5;
+            uint32_t cache_mask  : 11;
+            uint32_t occupied    : 14;
+            uint32_t has_inlines :  1;
+            uint32_t bit_one     :  1;
+        };
+        pint_t* vtableAddr = cls->getVTableAddress();
+
+        // the alignment of ImpCaches to 16 bytes is only needed for arm64_32.
+        ImpCacheHeader* cachePtr = (ImpCacheHeader*)align_buffer(readOnlyBuffer, sizeof(pint_t) == 4 ? 4 : 3);
+
+        assert(readOnlyBufferSize > sizeof(ImpCacheHeader));
+
+        uint64_t occupied = impCache.occupied();
+        int64_t fallback_class_offset = *(cls->getSuperClassAddress()) - clsVMAddr;
+
+        if (data->flatteningRootSuperclass) {
+            // If we are a class being flattened (inheriting all the selectors of
+            // its superclasses up to and including the flattening root), the fallback class
+            // should be the first superclass which is not flattened.
+            
+            // Find the VMAddr of that superclass, given its segment index and offset
+            // in the source dylib.
+            const auto & superclass = *(data->flatteningRootSuperclass);
+            const macho_header<P> * d = _dylibs[superclass.installName];
+            __block uint64_t superclassVMAddr = 0;
+            const dyld3::MachOAnalyzer *ma = (const dyld3::MachOAnalyzer *)d;
+            ma->forEachSegment(^(const dyld3::MachOAnalyzer::SegmentInfo &info, bool &stop) {
+                if (info.segIndex == superclass.segmentIndex) {
+                    superclassVMAddr = info.vmAddr + superclass.segmentOffset;
+                    stop = true;
+                }
+            });
+
+            assert(superclassVMAddr > 0);
+            fallback_class_offset = superclassVMAddr - clsVMAddr;
+        }
+
+        assert((int32_t)fallback_class_offset == fallback_class_offset);
+        assert((uint32_t)occupied == occupied);
+
+        *cachePtr = (ImpCacheHeader){
+            .fallback_class_offset = (int32_t)fallback_class_offset,
+            .cache_shift = (uint32_t)(data->shift + 7),
+            .cache_mask = (uint32_t)data->mask(),
+            .occupied = (uint32_t)occupied,
+            .has_inlines = impCache.hasInlines,
+            .bit_one = 1, // obj-c plays HORRENDOUS games here
+        };
+
+        // is this right?
+        int64_t vmaddr = cache->vmAddrForContent(readOnlyBuffer);
+        assert((pint_t)vmaddr == (uint64_t)vmaddr);
+        *vtableAddr =  (pint_t)cache->vmAddrForContent(readOnlyBuffer);
+        aslrTracker.add(vtableAddr);
+        readOnlyBuffer += sizeof(ImpCacheHeader);
+        readOnlyBufferSize -= sizeof(ImpCacheHeader);
+
+        impCache.write(cache, selectorStringVMAddr, clsVMAddr, readOnlyBuffer, readOnlyBufferSize, diag);
+    }
+
+    void emitInlinedSelectors(const std::vector<const IMPCaches::Selector*> selectors) {
+        // FIXME: this should be in constant memory
+        for (const IMPCaches::Selector* s : selectors) {
+            assert(readWriteBufferSize >= sizeof(pint_t));
+            *(pint_t*)readWriteBuffer = (pint_t)(selectorStringVMAddr + s->offset);
+            aslrTracker.add(readWriteBuffer);
+            readWriteBuffer += sizeof(pint_t);
+            readWriteBufferSize -= sizeof(pint_t);
+        }
+    }
+};
+
 template <typename P>
 class ProtocolOptimizer
 {
@@ -387,7 +769,7 @@ public:
         for (pint_t i = 0; i < protocols.count(); i++) {
             objc_protocol_t<P> *proto = protocols.get(i);
 
-            const char *name = proto->getName(cache);
+            const char* name = proto->getName(cache);
             if (_protocolNames.count(name) == 0) {
                 if (proto->getSize() > sizeof(objc_protocol_t<P>)) {
                     _diagnostics.error("objc protocol is too big");
@@ -411,7 +793,8 @@ public:
                                uint8_t *& rwdest, size_t& rwremaining,
                                uint8_t *& rodest, size_t& roremaining, 
                                CacheBuilder::ASLR_Tracker& aslrTracker,
-                               pint_t protocolClassVMAddr)
+                               pint_t protocolClassVMAddr,
+                               const dyld3::MachOAnalyzerSet::PointerMetaData& PMD)
     {
         if (_protocolCount == 0) return NULL;
 
@@ -440,6 +823,12 @@ public:
             if (!proto->getIsaVMAddr()) {
                 proto->setIsaVMAddr(protocolClassVMAddr);
             }
+
+            // If the objc runtime signed the Protocol ISA, then we need to too
+            if ( PMD.authenticated ) {
+                aslrTracker.setAuthData(proto->getISALocation(), PMD.diversity, PMD.usesAddrDiversity, PMD.key);
+            }
+
             if (oldSize < sizeof(*proto)) {
                 // Protocol object is old. Populate new fields.
                 proto->setSize(sizeof(objc_protocol_t<P>));
@@ -519,7 +908,7 @@ template <typename P>
 void addObjcSegments(Diagnostics& diag, DyldSharedCache* cache, const mach_header* libobjcMH,
                      uint8_t* objcReadOnlyBuffer, uint64_t objcReadOnlyBufferSizeAllocated,
                      uint8_t* objcReadWriteBuffer, uint64_t objcReadWriteBufferSizeAllocated,
-                     uint32_t objcRwFileOffset)
+                     uint64_t objcRwFileOffset)
 {
     // validate there is enough free space to add the load commands
     const dyld3::MachOAnalyzer* libobjcMA = ((dyld3::MachOAnalyzer*)libobjcMH);
@@ -588,6 +977,43 @@ void addObjcSegments(Diagnostics& diag, DyldSharedCache* cache, const mach_heade
     }
 }
 
+template <typename P> static inline void emitIMPCaches(ContentAccessor& cacheAccessor,
+                                         std::vector<CacheBuilder::DylibInfo> & allDylibs,
+                                         std::vector<const macho_header<P>*> & sizeSortedDylibs,
+                                         bool relativeMethodListSelectorsAreDirect,
+                                         uint64_t selectorStringVMAddr,
+                                         uint8_t* optROData, size_t& optRORemaining,
+                                         uint8_t* optRWData, size_t& optRWRemaining,
+                                         CacheBuilder::ASLR_Tracker& aslrTracker,
+                                         const std::vector<const IMPCaches::Selector*> & inlinedSelectors,
+                                         uint8_t* &inlinedSelectorsStart,
+                                         uint8_t* &inlinedSelectorsEnd,
+                                         Diagnostics& diag,
+                                         TimeRecorder& timeRecorder) {
+    diag.verbose("[IMP caches] computing IMP map\n");
+
+    IMPMapBuilder<P> classRecorder(relativeMethodListSelectorsAreDirect);
+    for (const macho_header<P>* mh : sizeSortedDylibs) {
+        ClassWalker<P, IMPMapBuilder<P>> classWalker(classRecorder, ClassWalkerMode::ClassAndMetaclasses);
+        classWalker.walk(&cacheAccessor, mh);
+        classRecorder.visit(&cacheAccessor, mh);
+    }
+
+    timeRecorder.recordTime("compute IMP map");
+    diag.verbose("[IMP caches] emitting IMP caches\n");
+
+    IMPCachesEmitter<P> impCachesEmitter(diag, classRecorder, selectorStringVMAddr, optROData, optRORemaining, optRWData, optRWRemaining, allDylibs, sizeSortedDylibs, aslrTracker);
+    ClassWalker<P, IMPCachesEmitter<P>> impEmitterClassWalker(impCachesEmitter, ClassWalkerMode::ClassAndMetaclasses);
+    for (const macho_header<P>* mh : sizeSortedDylibs) {
+        impEmitterClassWalker.walk(&cacheAccessor, mh);
+        if (diag.hasError())
+            return;
+    }
+
+    inlinedSelectorsStart = optRWData;
+    impCachesEmitter.emitInlinedSelectors(inlinedSelectors);
+    inlinedSelectorsEnd = optRWData;
+}
 
 template <typename P>
 void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::ASLR_Tracker& aslrTracker,
@@ -595,7 +1021,11 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
                     const std::map<void*, std::string>& missingWeakImports, Diagnostics& diag,
                     uint8_t* objcReadOnlyBuffer, uint64_t objcReadOnlyBufferSizeUsed, uint64_t objcReadOnlyBufferSizeAllocated,
                     uint8_t* objcReadWriteBuffer, uint64_t objcReadWriteBufferSizeAllocated,
-                    uint32_t objcRwFileOffset)
+                    uint64_t objcRwFileOffset,
+                    std::vector<CacheBuilder::DylibInfo> & allDylibs,
+                    const std::vector<const IMPCaches::Selector*> & inlinedSelectors,
+                    bool impCachesSuccess,
+                    TimeRecorder& timeRecorder)
 {
     typedef typename P::E           E;
     typedef typename P::uint_t      pint_t;
@@ -616,6 +1046,7 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
     __block const mach_header*      libobjcMH = nullptr;
     __block const macho_section<P> *optROSection = nullptr;
     __block const macho_section<P> *optPointerListSection = nullptr;
+    __block const macho_section<P> *optImpCachesPointerSection = nullptr;
     __block std::vector<const macho_header<P>*> objcDylibs;
     cache->forEachImage(^(const mach_header* machHeader, const char* installName) {
         const macho_header<P>* mh = (const macho_header<P>*)machHeader;
@@ -623,6 +1054,9 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
             libobjcMH = (mach_header*)mh;
             optROSection = mh->getSection("__TEXT", "__objc_opt_ro");
             optPointerListSection = mh->getSection("__DATA", "__objc_opt_ptrs");
+            if ( optPointerListSection == nullptr )
+                optPointerListSection = mh->getSection("__AUTH", "__objc_opt_ptrs");
+            optImpCachesPointerSection = mh->getSection("__DATA_CONST", "__objc_scoffs");
         }
         if ( mh->getSection("__DATA", "__objc_imageinfo") || mh->getSection("__OBJC", "__image_info") ) {
             objcDylibs.push_back(mh);
@@ -637,6 +1071,9 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
         diag.warning("libobjc's pointer list section missing (metadata not optimized)");
         return;
     }
+    if ( optImpCachesPointerSection == nullptr ) {
+        diag.warning("libobjc's magical shared cache offsets list section missing (metadata not optimized)");
+    }
     // point optROData into space allocated in dyld cache
     uint8_t* optROData = objcReadOnlyBuffer + objcReadOnlyBufferSizeUsed;
     size_t optRORemaining = objcReadOnlyBufferSizeAllocated - objcReadOnlyBufferSizeUsed;
@@ -767,7 +1204,11 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
         return (uint8_t*)(((uintptr_t)ptr + 0x7) & ~0x7);
     };
 
-    SelectorOptimizer<P, ObjCSelectorUniquer<P> > selOptimizer(uniq);
+    // Relative method lists names are initially an offset to a selector reference.
+    // Eventually we'll update them to offsets directly to the selector string.
+    bool relativeMethodListSelectorsAreDirect = false;
+
+    SelectorOptimizer<P, ObjCSelectorUniquer<P> > selOptimizer(uniq, relativeMethodListSelectorsAreDirect);
     selOptimizer.visitCoalescedStrings(coalescedText);
     for (const macho_header<P>* mh : sizeSortedDylibs) {
         LegacySelectorUpdater<P, ObjCSelectorUniquer<P>>::update(&cacheAccessor, mh, uniq);
@@ -865,7 +1306,7 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
     //
     // This is SAFE: modified binaries are still usable as unsorted lists.
     // This must be done AFTER uniquing selectors.
-    MethodListSorter<P> methodSorter;
+    MethodListSorter<P> methodSorter(relativeMethodListSelectorsAreDirect);
     for (const macho_header<P>* mh : sizeSortedDylibs) {
         methodSorter.optimize(&cacheAccessor, mh);
     }
@@ -887,10 +1328,26 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
                protocolOptimizer.protocolCount());
 
     pint_t protocolClassVMAddr = (pint_t)P::getP(optPointerList->protocolClass);
+
+    // Get the pointer metadata from the magic protocolClassVMAddr symbol
+    // We'll transfer it over to the ISA on all the objc protocols when we set their ISAs
+    dyld3::MachOAnalyzerSet::PointerMetaData protocolClassPMD;
+    uint16_t    protocolClassAuthDiversity  = 0;
+    bool        protocolClassAuthIsAddr     = false;
+    uint8_t     protocolClassAuthKey        = 0;
+    if ( aslrTracker.hasAuthData((void*)&optPointerList->protocolClass, &protocolClassAuthDiversity, &protocolClassAuthIsAddr, &protocolClassAuthKey) ) {
+        protocolClassPMD.diversity           = protocolClassAuthDiversity;
+        protocolClassPMD.high8               = 0;
+        protocolClassPMD.authenticated       = 1;
+        protocolClassPMD.key                 = protocolClassAuthKey;
+        protocolClassPMD.usesAddrDiversity   = protocolClassAuthIsAddr;
+    }
+
     err = protocolOptimizer.writeProtocols(&cacheAccessor,
                                            optRWData, optRWRemaining,
                                            optROData, optRORemaining,
-                                           aslrTracker, protocolClassVMAddr);
+                                           aslrTracker, protocolClassVMAddr,
+                                           protocolClassPMD);
     if (err) {
         diag.warning("%s", err);
         return;
@@ -942,6 +1399,56 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
     
     diag.verbose("  updated  % 6ld ivar offsets\n", ivarOffsetOptimizer.optimized());
 
+    //
+    // Build imp caches
+    //
+    // Objc has a magic section of imp cache base pointers.  We need these to
+    // offset everything else from
+    const CacheBuilder::CacheCoalescedText::StringSection& methodNames = coalescedText.getSectionData("__objc_methname");
+    uint64_t selectorStringVMAddr = methodNames.bufferVMAddr;
+    uint64_t selectorStringVMSize = methodNames.bufferSize;
+    uint64_t impCachesVMSize = 0; // We'll calculate this later
+
+    uint64_t optRODataRemainingBeforeImpCaches = optRORemaining;
+
+    timeRecorder.pushTimedSection();
+    
+    uint8_t* inlinedSelectorsStart = optRWData;
+    uint8_t* inlinedSelectorsEnd = optRWData;
+    
+    if (impCachesSuccess) {
+        emitIMPCaches<P>(cacheAccessor, allDylibs, sizeSortedDylibs, relativeMethodListSelectorsAreDirect,
+                         selectorStringVMAddr, optROData, optRORemaining, optRWData, optRWRemaining,
+                         aslrTracker, inlinedSelectors, inlinedSelectorsStart, inlinedSelectorsEnd, diag, timeRecorder);
+    }
+
+    uint8_t* alignedROData = alignPointer(optROData);
+    optRORemaining -= (alignedROData - optROData);
+    optROData = alignedROData;
+
+    impCachesVMSize = optRODataRemainingBeforeImpCaches - optRORemaining;
+    timeRecorder.recordTime("emit IMP caches");
+    timeRecorder.popTimedSection();
+
+    diag.verbose("[IMP Caches] Imp caches size: %'lld bytes\n\n", impCachesVMSize);
+
+    // Update the pointers in the pointer list section
+    if (optImpCachesPointerSection) {
+        if (optImpCachesPointerSection->size() < sizeof(objc_opt::objc_opt_pointerlist_tt<pint_t>)) {
+            diag.warning("libobjc's pointer list section is too small (metadata not optimized)");
+            return;
+        }
+        auto *impCachePointers = (objc_opt_imp_caches_pointerlist_tt<pint_t> *)cacheAccessor.contentForVMAddr(optImpCachesPointerSection->addr());
+        impCachePointers->selectorStringVMAddrStart = (pint_t)selectorStringVMAddr;
+        impCachePointers->selectorStringVMAddrEnd   = (pint_t)(selectorStringVMAddr + selectorStringVMSize);
+        impCachePointers->inlinedSelectorsVMAddrStart = (pint_t)cacheAccessor.vmAddrForContent(inlinedSelectorsStart);
+        impCachePointers->inlinedSelectorsVMAddrEnd = (pint_t)cacheAccessor.vmAddrForContent(inlinedSelectorsEnd);
+
+        aslrTracker.add(&impCachePointers->selectorStringVMAddrStart);
+        aslrTracker.add(&impCachePointers->selectorStringVMAddrEnd);
+        aslrTracker.add(&impCachePointers->inlinedSelectorsVMAddrStart);
+        aslrTracker.add(&impCachePointers->inlinedSelectorsVMAddrEnd);        
+    }
 
     // Collect flags.
     uint32_t headerFlags = 0;
@@ -990,7 +1497,7 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
 
 
     // Now that objc has uniqued the selector references, we can apply the LOHs so that ADRP/LDR -> ADRP/ADD
-    if (forProduction) {
+    {
         const bool logSelectors = false;
         uint64_t lohADRPCount = 0;
         uint64_t lohLDRCount = 0;
@@ -1102,19 +1609,41 @@ void doOptimizeObjC(DyldSharedCache* cache, bool forProduction, CacheBuilder::AS
 
 } // anon namespace
 
-void CacheBuilder::optimizeObjC()
+size_t IMPCaches::sizeForImpCacheWithCount(int count) {
+    // The architecture should not be relevant here as it's all offsets and fixed int sizes.
+    // It was just the most logical place to host this function in.
+
+    size_t size64 = IMPCachesEmitter<Pointer64<LittleEndian>>::sizeForImpCacheWithCount(count);
+    size_t size32 = IMPCachesEmitter<Pointer32<LittleEndian>>::sizeForImpCacheWithCount(count);
+    assert(size64 == size32);
+
+    return size64;
+}
+
+void SharedCacheBuilder::optimizeObjC(bool impCachesSuccess, const std::vector<const IMPCaches::Selector*> & inlinedSelectors)
 {
-    uint32_t objcRwFileOffset = (uint32_t)((_objcReadWriteBuffer - _readWriteRegion.buffer) + _readWriteRegion.cacheFileOffset);
     if ( _archLayout->is64 )
-        doOptimizeObjC<Pointer64<LittleEndian>>((DyldSharedCache*)_readExecuteRegion.buffer, _options.optimizeStubs, _aslrTracker, _lohTracker,
-                                                _coalescedText, _missingWeakImports,
-                                                _diagnostics, _objcReadOnlyBuffer, _objcReadOnlyBufferSizeUsed, _objcReadOnlyBufferSizeAllocated,
-                                                _objcReadWriteBuffer, _objcReadWriteBufferSizeAllocated, objcRwFileOffset);
+        doOptimizeObjC<Pointer64<LittleEndian>>((DyldSharedCache*)_readExecuteRegion.buffer,
+            _options.optimizeStubs,
+            _aslrTracker, _lohTracker,
+            _coalescedText,
+            _missingWeakImports, _diagnostics,
+            _objcReadOnlyBuffer,
+            _objcReadOnlyBufferSizeUsed,
+            _objcReadOnlyBufferSizeAllocated,
+            _objcReadWriteBuffer, _objcReadWriteBufferSizeAllocated,
+            _objcReadWriteFileOffset, _sortedDylibs, inlinedSelectors, impCachesSuccess, _timeRecorder);
     else
-        doOptimizeObjC<Pointer32<LittleEndian>>((DyldSharedCache*)_readExecuteRegion.buffer, _options.optimizeStubs, _aslrTracker, _lohTracker,
-                                                _coalescedText, _missingWeakImports,
-                                                _diagnostics, _objcReadOnlyBuffer, _objcReadOnlyBufferSizeUsed, _objcReadOnlyBufferSizeAllocated,
-                                                _objcReadWriteBuffer, _objcReadWriteBufferSizeAllocated, objcRwFileOffset);
+        doOptimizeObjC<Pointer32<LittleEndian>>((DyldSharedCache*)_readExecuteRegion.buffer,
+            _options.optimizeStubs,
+            _aslrTracker, _lohTracker,
+            _coalescedText,
+            _missingWeakImports, _diagnostics,
+            _objcReadOnlyBuffer,
+            _objcReadOnlyBufferSizeUsed,
+            _objcReadOnlyBufferSizeAllocated,
+            _objcReadWriteBuffer, _objcReadWriteBufferSizeAllocated,
+            _objcReadWriteFileOffset, _sortedDylibs, inlinedSelectors, impCachesSuccess, _timeRecorder);
 }
 
 static uint32_t hashTableSize(uint32_t maxElements, uint32_t perElementData)
@@ -1127,13 +1656,16 @@ static uint32_t hashTableSize(uint32_t maxElements, uint32_t perElementData)
 
 // The goal here is to allocate space in the dyld shared cache (while it is being laid out) that will contain
 // the objc structures that previously were in the __objc_opt_ro section.
-uint32_t CacheBuilder::computeReadOnlyObjC(uint32_t selRefCount, uint32_t classDefCount, uint32_t protocolDefCount)
+uint32_t SharedCacheBuilder::computeReadOnlyObjC(uint32_t selRefCount, uint32_t classDefCount, uint32_t protocolDefCount)
 {
     return 0xA000 + hashTableSize(selRefCount, 5) + hashTableSize(classDefCount, 12) + hashTableSize(protocolDefCount, 8);
 }
 
 // Space to replace the __objc_opt_rw section.
-uint32_t CacheBuilder::computeReadWriteObjC(uint32_t imageCount, uint32_t protocolDefCount)
+uint32_t SharedCacheBuilder::computeReadWriteObjC(uint32_t imageCount, uint32_t protocolDefCount)
 {
-    return 8*imageCount + protocolDefCount*12*(_archLayout->is64 ? 8 : 4);
+    uint8_t pointerSize = _archLayout->is64 ? 8 : 4;
+    return 8*imageCount
+         + protocolDefCount*12*pointerSize
+         + (int)_impCachesBuilder->inlinedSelectors.size() * pointerSize;
 }

dyld3/shared-cache/SharedCacheBuilder.h

@@ -0,0 +1,173 @@
+/*
+ * Copyright (c) 2017 Apple Inc. All rights reserved.
+ *
+ * @APPLE_LICENSE_HEADER_START@
+ *
+ * This file contains Original Code and/or Modifications of Original Code
+ * as defined in and that are subject to the Apple Public Source License
+ * Version 2.0 (the 'License'). You may not use this file except in
+ * compliance with the License. Please obtain a copy of the License at
+ * http://www.opensource.apple.com/apsl/ and read it before using this
+ * file.
+ *
+ * The Original Code and all software distributed under the License are
+ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
+ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
+ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
+ * Please see the License for the specific language governing rights and
+ * limitations under the License.
+ *
+ * @APPLE_LICENSE_HEADER_END@
+ */
+
+
+#ifndef SharedCacheBuilder_h
+#define SharedCacheBuilder_h
+
+#include "CacheBuilder.h"
+#include "DyldSharedCache.h"
+#include "ClosureFileSystem.h"
+#include "IMPCachesBuilder.hpp"
+
+class SharedCacheBuilder : public CacheBuilder {
+public:
+    SharedCacheBuilder(const DyldSharedCache::CreateOptions& options, const dyld3::closure::FileSystem& fileSystem);
+
+    void                                        build(std::vector<InputFile>& inputFiles,
+                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
+    void                                        build(const std::vector<LoadedMachO>& dylibs,
+                                                      const std::vector<LoadedMachO>& otherOsDylibsInput,
+                                                      const std::vector<LoadedMachO>& osExecutables,
+                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
+    void                                        build(const std::vector<DyldSharedCache::MappedMachO>&  dylibsToCache,
+                                                      const std::vector<DyldSharedCache::MappedMachO>&  otherOsDylibs,
+                                                      const std::vector<DyldSharedCache::MappedMachO>&  osExecutables,
+                                                      std::vector<DyldSharedCache::FileAlias>& aliases);
+
+    void                                        writeFile(const std::string& path);
+    void                                        writeBuffer(uint8_t*& buffer, uint64_t& size);
+    void                                        writeMapFile(const std::string& path);
+    std::string                                 getMapFileBuffer() const;
+    std::string                                 getMapFileJSONBuffer(const std::string& cacheDisposition) const;
+    void                                        deleteBuffer();
+    const std::set<std::string>                 warnings();
+    const std::set<const dyld3::MachOAnalyzer*> evictions();
+    const bool                                  agileSignature();
+    const std::string                           cdHashFirst();
+    const std::string                           cdHashSecond();
+    const std::string                           uuid() const;
+
+    void                                        forEachCacheDylib(void (^callback)(const std::string& path));
+    void                                        forEachCacheSymlink(void (^callback)(const std::string& path));
+
+    void                                        forEachDylibInfo(void (^callback)(const DylibInfo& dylib, Diagnostics& dylibDiag)) override final;
+
+private:
+
+    void        writeSlideInfoV1();
+
+    template <typename P> void writeSlideInfoV2(const bool bitmap[], unsigned dataPageCount);
+    template <typename P> bool makeRebaseChainV2(uint8_t* pageContent, uint16_t lastLocationOffset, uint16_t newOffset, const struct dyld_cache_slide_info2* info);
+    template <typename P> void addPageStartsV2(uint8_t* pageContent, const bool bitmap[], const struct dyld_cache_slide_info2* info,
+                                             std::vector<uint16_t>& pageStarts, std::vector<uint16_t>& pageExtras);
+
+    void        writeSlideInfoV3(const bool bitmap[], unsigned dataPageCoun);
+    uint16_t    pageStartV3(uint8_t* pageContent, uint32_t pageSize, const bool bitmap[]);
+    void        setPointerContentV3(dyld3::MachOLoaded::ChainedFixupPointerOnDisk* loc, uint64_t targetVMAddr, size_t next);
+
+    template <typename P> void writeSlideInfoV4(const bool bitmap[], unsigned dataPageCount);
+    template <typename P> bool makeRebaseChainV4(uint8_t* pageContent, uint16_t lastLocationOffset, uint16_t newOffset, const struct dyld_cache_slide_info4* info);
+    template <typename P> void addPageStartsV4(uint8_t* pageContent, const bool bitmap[], const struct dyld_cache_slide_info4* info,
+                                             std::vector<uint16_t>& pageStarts, std::vector<uint16_t>& pageExtras);
+
+    struct ArchLayout
+    {
+        uint64_t    sharedMemoryStart;
+        uint64_t    sharedMemorySize;
+        uint64_t    sharedRegionPadding;
+        uint64_t    pointerDeltaMask;
+        const char* archName;
+        uint16_t    csPageSize;
+        uint8_t     sharedRegionAlignP2;
+        uint8_t     slideInfoBytesPerPage;
+        bool        sharedRegionsAreDiscontiguous;
+        bool        is64;
+        bool        useValueAdd;
+    };
+
+    static const ArchLayout  _s_archLayout[];
+    static const char* const _s_neverStubEliminateSymbols[];
+
+    void        makeSortedDylibs(const std::vector<LoadedMachO>& dylibs, const std::unordered_map<std::string, unsigned> sortOrder);
+    void        processSelectorStrings(const std::vector<LoadedMachO>& executables, IMPCaches::HoleMap& selectorsHoleMap);
+    void        parseCoalescableSegments(IMPCaches::SelectorMap& selectorMap, IMPCaches::HoleMap& selectorsHoleMap);
+    void        assignSegmentAddresses();
+    void        assignMultipleDataSegmentAddresses(uint64_t& addr, uint32_t totalProtocolDefCount);
+
+    uint64_t    dataRegionsTotalSize() const;
+    uint64_t    dataRegionsSizeInUse() const;
+
+    // Return the earliest data region by address
+    const Region* firstDataRegion() const;
+
+    // Return the lateset data region by address
+    const Region* lastDataRegion() const;
+
+    uint64_t    cacheOverflowAmount();
+    size_t      evictLeafDylibs(uint64_t reductionTarget, std::vector<const LoadedMachO*>& overflowDylibs);
+
+    void        fipsSign();
+    void        codeSign();
+    uint64_t    pathHash(const char* path);
+    void        writeCacheHeader();
+    void        findDylibAndSegment(const void* contentPtr, std::string& dylibName, std::string& segName);
+    void        addImageArray();
+    void        buildImageArray(std::vector<DyldSharedCache::FileAlias>& aliases);
+    void        addOtherImageArray(const std::vector<LoadedMachO>&, std::vector<const LoadedMachO*>& overflowDylibs);
+    void        addClosures(const std::vector<LoadedMachO>&);
+    void        markPaddingInaccessible();
+
+    bool        writeCache(void (^cacheSizeCallback)(uint64_t size), bool (^copyCallback)(const uint8_t* src, uint64_t size, uint64_t dstOffset));
+
+    // implemented in OptimizerObjC.cpp
+    void        optimizeObjC(bool impCachesSuccess, const std::vector<const IMPCaches::Selector*> & inlinedSelectors);
+    uint32_t    computeReadOnlyObjC(uint32_t selRefCount, uint32_t classDefCount, uint32_t protocolDefCount);
+    uint32_t    computeReadWriteObjC(uint32_t imageCount, uint32_t protocolDefCount);
+
+    void        emitContantObjects();
+
+    typedef std::unordered_map<std::string, const dyld3::MachOAnalyzer*> InstallNameToMA;
+
+    typedef uint64_t                                                CacheOffset;
+
+    std::vector<DylibInfo>                      _sortedDylibs;
+    std::vector<Region>                         _dataRegions; // 1 or more __DATA regions.
+    UnmappedRegion                              _codeSignatureRegion;
+    std::set<const dyld3::MachOAnalyzer*>       _evictions;
+    const ArchLayout*                           _archLayout                             = nullptr;
+    uint32_t                                    _aliasCount                             = 0;
+    uint8_t*                                    _objcReadOnlyBuffer                     = nullptr;
+    uint64_t                                    _objcReadOnlyBufferSizeUsed             = 0;
+    uint64_t                                    _objcReadOnlyBufferSizeAllocated        = 0;
+    uint8_t*                                    _objcReadWriteBuffer                    = nullptr;
+    uint64_t                                    _objcReadWriteBufferSizeAllocated       = 0;
+    uint64_t                                    _objcReadWriteFileOffset                = 0;
+    uint64_t                                    _selectorStringsFromExecutables         = 0;
+    InstallNameToMA                             _installNameToCacheDylib;
+    std::unordered_map<std::string, uint32_t>   _dataDirtySegsOrder;
+    std::map<void*, std::string>                _missingWeakImports;
+    const dyld3::closure::ImageArray*           _imageArray                             = nullptr;
+    uint8_t                                     _cdHashFirst[20];
+    uint8_t                                     _cdHashSecond[20];
+    bool                                        _someDylibsUsedChainedFixups            = false;
+    std::unordered_map<const dyld3::MachOLoaded*, std::set<CacheOffset>>        _dylibToItsExports;
+    std::set<std::pair<const dyld3::MachOLoaded*, CacheOffset>>                 _dylibWeakExports;
+    std::unordered_map<CacheOffset, std::vector<dyld_cache_patchable_location>> _exportsToUses;
+    std::unordered_map<CacheOffset, std::string>                                _exportsToName;
+    IMPCaches::IMPCachesBuilder* _impCachesBuilder;
+};
+
+
+
+#endif /* SharedCacheBuilder_h */

dyld3/shared-cache/StringUtils.h

@@ -82,7 +82,7 @@ inline void putHexByte(uint8_t value, char*& p)
     putHexNibble(value & 0x0F, p);
 }
 
-inline uint8_t hexCharToUInt(const char hexByte, uint8_t& value) {
+inline bool hexCharToUInt(const char hexByte, uint8_t& value) {
     if (hexByte >= '0' && hexByte <= '9') {
         value = hexByte - '0';
         return true;
@@ -122,20 +122,21 @@ inline uint64_t hexToUInt64(const char* startHexByte, const char** endHexByte) {
     return retval;
 }
 
-inline bool hexToBytes(const char* startHexByte, uint32_t length, uint8_t buffer[]) {
-    if (startHexByte == nullptr)
+inline bool hexStringToBytes(const char* hexString, uint8_t buffer[], unsigned bufferMaxSize, unsigned& bufferLenUsed)
+{
+    bufferLenUsed = 0;
+    bool high = true;
+    for (const char* s=hexString; *s != '\0'; ++s) {
+        if ( bufferLenUsed > bufferMaxSize )
             return false;
-    const char *currentHexByte = startHexByte;
-    for (uint32_t i = 0; i < length; ++i) {
         uint8_t value;
-        if (!hexCharToUInt(currentHexByte[i], value)) {
+        if ( !hexCharToUInt(*s, value) )
             return false;
-        }
-        if (i%2 == 0) {
-            buffer[i/2] = value << 4;
-        } else {
-            buffer[(i-1)/2] |= value;
-        }
+        if ( high )
+            buffer[bufferLenUsed] = value << 4;
+        else
+            buffer[bufferLenUsed++] |= value;
+        high = !high;
     }
     return true;
 }