[libc++][Android] Support libc++ testing on Android (#69274)

I could probably break this commit into more pieces.

---

This patch adds libc++ support for Android L (Android 5.0+) and up,
tested using the Android team's current compiler, a recent version of
the AOSP sysroot, and the x86[-64] Android Emulator.

CMake and Lit Configuration:

Add runtimes/cmake/android/Arch-${ARCH}.cmake files that configure CMake
to cross-compile to Android without using CMake's built-in NDK support
(which only works with an actual packaged NDK).

Add libcxx/cmake/caches/AndroidNDK.cmake that builds and tests libc++
(and libc++abi) for Android. This file configures libc++ to match what
the NDK distributes, e.g.:
- libc++_shared.so (includes libc++abi objects, there is no
libc++abi.so). libunwind is linked statically but not exported.
 - libc++_static.a (does not include libc++abi) and libc++abi.a
 - `std::__ndk1` namespace
- All the libraries are built with `__ANDROID_API__=21`, even when they
are linked to something targeting a higher API level.

(However, when the Android LLVM team builds these components, they do
not use these CMake cache files. Instead they use Python scripts to
configure the builds. See
https://android.googlesource.com/toolchain/llvm_android/.)

Add llvm-libc++[abi].android-ndk.cfg.in files that test the Android
NDK's libc++_shared.so. These files can target old or new Android
devices. The Android LLVM team uses these test files to test libc++ for
both arm/arm64 and x86/x86_64 architectures.

The Android testing mode works by setting %{executor} to adb_run.py,
which uses `adb push` and `adb shell` to run tests remotely. adb_run.py
always runs tests as the "shell" user even on an old emulator where "adb
unroot" doesn't work. The script has workarounds for old Android
devices. The script uses a Unix domain socket on the host
(--job-limit-socket) to restrict concurrent adb invocations. Compiling
the tests is a major part of libc++ testing run-time, so it's desirable
to exploit all the host cores without overburdening the test devices,
which can have far fewer cores.

BuildKite CI:

Add a builder to run-buildbot, `android-ndk-*`, that uses Android Clang
and an Android sysroot to build libc++, then starts an Android emulator
container to run tests.

Run the emulator and an adb server in a separate Docker container
(libcxx-ci-android-emulator), and create a separate Docker image for
each emulator OS system image. Set ADB_SERVER_SOCKET to connect to the
container's adb server. Running the only adb server inside the container
makes cleanup more reliable between test runs, e.g. the adb client
doesn't create a `~/.android` directory and the adb server can be
restarted along with the emulator using docker stop/run. (N.B. The
emulator insists on connecting to an adb server and will start one
itself if it can't connect to one.)

The suffix to the android-ndk-* job is a label that concisely specifies
an Android SDK emulator image. e.g.:
 - "system-images;android-21;default;x86" ==> 21-def-x86
 - "system-images;android-33;google_apis;x86_64" ==> 33-goog-x86_64

Fixes: https://github.com/llvm/llvm-project/issues/69270
Differential Revision: https://reviews.llvm.org/D139147

libcxx/cmake/caches/AndroidNDK.cmake

@@ -0,0 +1,38 @@
+# Build libc++abi and libc++ closely resembling what is shipped in the Android
+# NDK.
+
+# The NDK names the libraries libc++_shared.so and libc++_static.a. Using the
+# libc++_shared.so soname ensures that the library doesn't interact with the
+# libc++.so in /system/lib[64].
+set(LIBCXX_SHARED_OUTPUT_NAME c++_shared CACHE STRING "")
+set(LIBCXX_STATIC_OUTPUT_NAME c++_static CACHE STRING "")
+
+# The NDK libc++ uses a special namespace to help isolate its symbols from those
+# in the platform's STL (e.g. /system/lib[64]/libc++.so, but possibly stlport on
+# older versions of Android).
+set(LIBCXX_ABI_VERSION 1 CACHE STRING "")
+set(LIBCXX_ABI_NAMESPACE __ndk1 CACHE STRING "")
+
+# CMake doesn't add a version suffix to an Android shared object filename,
+# (because CMAKE_PLATFORM_NO_VERSIONED_SONAME is set), so it writes both a
+# libc++_shared.so ELF file and a libc++_shared.so linker script to the same
+# output path (the script clobbers the binary). Turn off the linker script.
+set(LIBCXX_ENABLE_ABI_LINKER_SCRIPT OFF CACHE BOOL "")
+
+set(LIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY ON CACHE BOOL "")
+set(LIBCXXABI_ENABLE_SHARED OFF CACHE BOOL "")
+
+# Clang links libc++ by default, but it doesn't exist yet. The libc++ CMake
+# files specify -nostdlib++ to avoid this problem, but CMake's default "compiler
+# works" testing doesn't pass that flag, so force those tests to pass.
+set(CMAKE_C_COMPILER_WORKS ON CACHE BOOL "")
+set(CMAKE_CXX_COMPILER_WORKS ON CACHE BOOL "")
+
+# Use adb to push tests to a locally-connected device (e.g. emulator) and run
+# them.
+set(LIBCXX_TEST_CONFIG "llvm-libc++-android-ndk.cfg.in" CACHE STRING "")
+set(LIBCXXABI_TEST_CONFIG "llvm-libc++abi-android-ndk.cfg.in" CACHE STRING "")
+
+# CMAKE_SOURCE_DIR refers to the "<monorepo>/runtimes" directory.
+set(LIBCXX_EXECUTOR "${CMAKE_SOURCE_DIR}/../libcxx/utils/adb_run.py" CACHE STRING "")
+set(LIBCXXABI_EXECUTOR "${LIBCXX_EXECUTOR}" CACHE STRING "")

libcxx/docs/index.rst

@@ -130,6 +130,7 @@ Target platform Target architecture       Notes
 macOS 10.9+     i386, x86_64, arm64       Building the shared library itself requires targetting macOS 10.13+
 FreeBSD 12+     i386, x86_64, arm
 Linux           i386, x86_64, arm, arm64  Only glibc-2.24 and later and no other libc is officially supported
+Android 5.0+    i386, x86_64, arm, arm64
 Windows         i386, x86_64              Both MSVC and MinGW style environments, ABI in MSVC environments is :doc:`unstable <DesignDocs/ABIVersioning>`
 AIX 7.2TL5+     powerpc, powerpc64
 =============== ========================= ============================

libcxx/test/configs/llvm-libc++-android-ndk.cfg.in

@@ -0,0 +1,47 @@
+# This testing configuration handles running the test suite against LLVM's
+# libc++ using adb and a libc++_shared.so library on Android.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+import re
+import site
+
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+
+import libcxx.test.android
+import libcxx.test.config
+import libcxx.test.params
+
+config.substitutions.append(('%{flags}',
+    '--sysroot @CMAKE_SYSROOT@' if '@CMAKE_SYSROOT@' else ''
+))
+
+compile_flags = '-nostdinc++ -I %{include} -I %{target-include} -I %{libcxx}/test/support'
+if re.match(r'i686-linux-android(21|22|23)$', config.target_triple):
+    # 32-bit x86 Android has a bug where the stack is sometimes misaligned.
+    # The problem appears limited to versions before Android N (API 24) and only
+    # __attribute__((constructor)) functions. Compile with -mstackrealign to
+    # work around the bug.
+    # TODO: Consider automatically doing something like this in Clang itself (LIBCXX-ANDROID-FIXME)
+    # See https://github.com/android/ndk/issues/693.
+    compile_flags += ' -mstackrealign'
+config.substitutions.append(('%{compile_flags}', compile_flags))
+
+# The NDK library is called "libc++_shared.so". Use LD_LIBRARY_PATH to find
+# libc++_shared.so because older Bionic dynamic loaders don't support rpath
+# lookup.
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib++ -L %{lib} -lc++_shared'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor}' +
+    ' --job-limit-socket ' + libcxx.test.android.adb_job_limit_socket() +
+    ' --prepend-path-env LD_LIBRARY_PATH /data/local/tmp/libc++ --execdir %T -- '
+))
+
+libcxx.test.config.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/utils/adb_run.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+"""adb_run.py is a utility for running a libc++ test program via adb.
+"""
+
+import argparse
+import hashlib
+import os
+import re
+import shlex
+import socket
+import subprocess
+import sys
+from typing import List, Optional, Tuple
+
+
+# Sync a host file /path/to/dir/file to ${REMOTE_BASE_DIR}/run-${HASH}/dir/file.
+REMOTE_BASE_DIR = "/data/local/tmp/adb_run"
+
+g_job_limit_socket = None
+g_verbose = False
+
+
+def run_adb_sync_command(command: List[str]) -> None:
+    """Run an adb command and discard the output, unless the command fails. If
+    the command fails, dump the output instead, and exit the script with
+    failure.
+    """
+    if g_verbose:
+        sys.stderr.write(f"running: {shlex.join(command)}\n")
+    proc = subprocess.run(command, universal_newlines=True,
+                          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+                          stderr=subprocess.STDOUT, encoding="utf-8")
+    if proc.returncode != 0:
+        # adb's stdout (e.g. for adb push) should normally be discarded, but
+        # on failure, it should be shown. Print it to stderr because it's
+        # unrelated to the test program's stdout output. A common error caught
+        # here is "No space left on device".
+        sys.stderr.write(f"{proc.stdout}\n"
+                         f"error: adb command exited with {proc.returncode}: "
+                         f"{shlex.join(command)}\n")
+        sys.exit(proc.returncode)
+
+
+def sync_test_dir(local_dir: str, remote_dir: str) -> None:
+    """Sync the libc++ test directory on the host to the remote device."""
+
+    # Optimization: The typical libc++ test directory has only a single
+    # *.tmp.exe file in it. In that case, skip the `mkdir` command, which is
+    # normally necessary because we don't know if the target directory already
+    # exists on the device.
+    local_files = os.listdir(local_dir)
+    if len(local_files) == 1:
+        local_file = os.path.join(local_dir, local_files[0])
+        remote_file = os.path.join(remote_dir, local_files[0])
+        if not os.path.islink(local_file) and os.path.isfile(local_file):
+            run_adb_sync_command(["adb", "push", "--sync", local_file,
+                                  remote_file])
+            return
+
+    assert os.path.basename(local_dir) == os.path.basename(remote_dir)
+    run_adb_sync_command(["adb", "shell", "mkdir", "-p", remote_dir])
+    run_adb_sync_command(["adb", "push", "--sync", local_dir,
+                          os.path.dirname(remote_dir)])
+
+
+def build_env_arg(env_args: List[str], prepend_path_args: List[Tuple[str, str]]) -> str:
+    components = []
+    for arg in env_args:
+        k, v = arg.split("=", 1)
+        components.append(f"export {k}={shlex.quote(v)}; ")
+    for k, v in prepend_path_args:
+        components.append(f"export {k}={shlex.quote(v)}${{{k}:+:${k}}}; ")
+    return "".join(components)
+
+
+def run_command(args: argparse.Namespace) -> int:
+    local_dir = args.execdir
+    assert local_dir.startswith("/")
+    assert not local_dir.endswith("/")
+
+    # Copy each execdir to a subdir of REMOTE_BASE_DIR. Name the directory using
+    # a hash of local_dir so that concurrent adb_run invocations don't create
+    # the same intermediate parent directory. At least `adb push` has trouble
+    # with concurrent mkdir syscalls on common parent directories. (Somehow
+    # mkdir fails with EAGAIN/EWOULDBLOCK, see internal Google bug,
+    # b/289311228.)
+    local_dir_hash = hashlib.sha1(local_dir.encode()).hexdigest()
+    remote_dir = f"{REMOTE_BASE_DIR}/run-{local_dir_hash}/{os.path.basename(local_dir)}"
+    sync_test_dir(local_dir, remote_dir)
+
+    adb_shell_command = (
+        # Set the environment early so that PATH can be overridden. Overriding
+        # PATH is useful for:
+        #  - Replacing older shell utilities with toybox (e.g. on old devices).
+        #  - Adding a `bash` command that delegates to `sh` (mksh).
+        f"{build_env_arg(args.env, args.prepend_path_env)}"
+
+        # Set a high oom_score_adj so that, if the test program uses too much
+        # memory, it is killed before anything else on the device. The default
+        # oom_score_adj is -1000, so a test using too much memory typically
+        # crashes the device.
+        "echo 1000 >/proc/self/oom_score_adj; "
+
+        # If we're running as root, switch to the shell user. The libc++
+        # filesystem tests require running without root permissions. Some x86
+        # emulator devices (before Android N) do not have a working `adb unroot`
+        # and always run as root. Non-debug builds typically lack `su` and only
+        # run as the shell user.
+        #
+        # Some libc++ tests create temporary files in the working directory,
+        # which might be owned by root. Before switching to shell, make the
+        # cwd writable (and readable+executable) to every user.
+        #
+        # N.B.:
+        #  - Avoid "id -u" because it wasn't supported until Android M.
+        #  - The `env` and `which` commands were also added in Android M.
+        #  - Starting in Android M, su from root->shell resets PATH, so we need
+        #    to modify it again in the new environment.
+        #  - Avoid chmod's "a+rwx" syntax because it's not supported until
+        #    Android N.
+        #  - Defining this function allows specifying the arguments to the test
+        #    program (i.e. "$@") only once.
+        "run_without_root() {"
+        "  chmod 777 .;"
+        "  case \"$(id)\" in"
+        "    *\"uid=0(root)\"*)"
+        "    if command -v env >/dev/null; then"
+        "      su shell \"$(command -v env)\" PATH=\"$PATH\" \"$@\";"
+        "    else"
+        "      su shell \"$@\";"
+        "    fi;;"
+        "    *) \"$@\";;"
+        "  esac;"
+        "}; "
+    )
+
+    # Older versions of Bionic limit the length of argv[0] to 127 bytes
+    # (SOINFO_NAME_LEN-1), and the path to libc++ tests tend to exceed this
+    # limit. Changing the working directory works around this limit. The limit
+    # is increased to 4095 (PATH_MAX-1) in Android M (API 23).
+    command_line = [arg.replace(local_dir + "/", "./") for arg in args.command]
+
+    # Prior to the adb feature "shell_v2" (added in Android N), `adb shell`
+    # always created a pty:
+    #  - This merged stdout and stderr together.
+    #  - The pty converts LF to CRLF.
+    #  - The exit code of the shell command wasn't propagated.
+    # Work around all three limitations, unless "shell_v2" is present.
+    proc = subprocess.run(["adb", "features"], check=True,
+                          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+                          encoding="utf-8")
+    adb_features = set(proc.stdout.strip().split())
+    has_shell_v2 = "shell_v2" in adb_features
+    if has_shell_v2:
+        adb_shell_command += (
+            f"cd {remote_dir} && run_without_root {shlex.join(command_line)}"
+        )
+    else:
+        adb_shell_command += (
+            f"{{"
+            f"  stdout=$("
+            f"    cd {remote_dir} && run_without_root {shlex.join(command_line)};"
+            f"    echo -n __libcxx_adb_exit__=$?"
+            f"  ); "
+            f"}} 2>&1; "
+            f"echo -n __libcxx_adb_stdout__\"$stdout\""
+        )
+
+    adb_command_line = ["adb", "shell", adb_shell_command]
+    if g_verbose:
+        sys.stderr.write(f"running: {shlex.join(adb_command_line)}\n")
+
+    if has_shell_v2:
+        proc = subprocess.run(adb_command_line, shell=False, check=False,
+                              encoding="utf-8")
+        return proc.returncode
+    else:
+        proc = subprocess.run(adb_command_line, shell=False, check=False,
+                              stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                              encoding="utf-8")
+        # The old `adb shell` mode used a pty, which converted LF to CRLF.
+        # Convert it back.
+        output = proc.stdout.replace("\r\n", "\n")
+
+        if proc.returncode:
+            sys.stderr.write(f"error: adb failed:\n"
+                             f"  command: {shlex.join(adb_command_line)}\n"
+                             f"  output: {output}\n")
+            return proc.returncode
+
+        match = re.match(r"(.*)__libcxx_adb_stdout__(.*)__libcxx_adb_exit__=(\d+)$",
+                     output, re.DOTALL)
+        if not match:
+            sys.stderr.write(f"error: could not parse adb output:\n"
+                             f"  command: {shlex.join(adb_command_line)}\n"
+                             f"  output: {output}\n")
+            return 1
+
+        sys.stderr.write(match.group(1))
+        sys.stdout.write(match.group(2))
+        return int(match.group(3))
+
+
+def connect_to_job_limiter_server(sock_addr: str) -> None:
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+
+    try:
+        sock.connect(sock_addr)
+    except (FileNotFoundError, ConnectionRefusedError) as e:
+        # Copying-and-pasting an adb_run.py command-line from a lit test failure
+        # is likely to fail because the socket no longer exists (or is
+        # inactive), so just give a warning.
+        sys.stderr.write(f"warning: could not connect to {sock_addr}: {e}\n")
+        return
+
+    # The connect call can succeed before the server has called accept, because
+    # of the listen backlog, so wait for the server to send a byte.
+    sock.recv(1)
+
+    # Keep the socket open until this process ends, then let the OS close the
+    # connection automatically.
+    global g_job_limit_socket
+    g_job_limit_socket = sock
+
+
+def main() -> int:
+    """Main function (pylint wants this docstring)."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--execdir", type=str, required=True)
+    parser.add_argument("--env", type=str, required=False, action="append",
+                        default=[], metavar="NAME=VALUE")
+    parser.add_argument("--prepend-path-env", type=str, nargs=2, required=False,
+                        action="append", default=[],
+                        metavar=("NAME", "PATH"))
+    parser.add_argument("--job-limit-socket")
+    parser.add_argument("--verbose", "-v", default=False, action="store_true")
+    parser.add_argument("command", nargs=argparse.ONE_OR_MORE)
+    args = parser.parse_args()
+
+    global g_verbose
+    g_verbose = args.verbose
+    if args.job_limit_socket is not None:
+        connect_to_job_limiter_server(args.job_limit_socket)
+    return run_command(args)
+
+
+if __name__ == '__main__':
+    sys.exit(main())

libcxx/utils/ci/BOT_OWNERS.txt

@@ -15,3 +15,8 @@ D: Armv7, Armv8, AArch64
 N: LLVM on Power
 E: powerllvm@ca.ibm.com
 D: AIX, ppc64le
+
+N: Android libc++
+E: rprichard@google.com
+H: rprichard
+D: Emulator-based x86[-64] libc++ CI testing

libcxx/utils/ci/run-buildbot

@@ -156,6 +156,13 @@ function generate-cmake-libcxx-win() {
           "${@}"
 }
 
+function generate-cmake-android() {
+    generate-cmake-base \
+          -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
+          -DLIBCXX_CXX_ABI=libcxxabi \
+          "${@}"
+}
+
 function check-runtimes() {
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
@@ -706,6 +713,49 @@ aix)
     check-abi-list
     check-runtimes
 ;;
+android-ndk-*)
+    clean
+
+    ANDROID_EMU_IMG="${BUILDER#android-ndk-}"
+    . "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android/emulator-functions.sh"
+    if ! validate_emu_img "${ANDROID_EMU_IMG}"; then
+        echo "error: android-ndk suffix must be a valid emulator image (${ANDROID_EMU_IMG})" >&2
+        exit 1
+    fi
+    ARCH=$(arch_of_emu_img ${ANDROID_EMU_IMG})
+
+    # Use the Android compiler by default.
+    export CC=${CC:-/opt/android/clang/clang-current/bin/clang}
+    export CXX=${CXX:-/opt/android/clang/clang-current/bin/clang++}
+
+    # The NDK libc++_shared.so is always built against the oldest supported API
+    # level. When tests are run against a device with a newer API level, test
+    # programs can be built for any supported API level, but building for the
+    # newest API (i.e. the system image's API) is probably the most interesting.
+    PARAMS="target_triple=$(triple_of_arch ${ARCH})$(api_of_emu_img ${ANDROID_EMU_IMG})"
+    generate-cmake-android -C "${MONOREPO_ROOT}/runtimes/cmake/android/Arch-${ARCH}.cmake" \
+                           -C "${MONOREPO_ROOT}/libcxx/cmake/caches/AndroidNDK.cmake" \
+                           -DCMAKE_SYSROOT=/opt/android/ndk/sysroot \
+                           -DLIBCXX_TEST_PARAMS="${PARAMS}" \
+                           -DLIBCXXABI_TEST_PARAMS="${PARAMS}"
+    check-abi-list
+    ${NINJA} -vC "${BUILD_DIR}" install-cxx install-cxxabi
+
+    # Start the emulator and make sure we can connect to the adb server running
+    # inside of it.
+    "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android/start-emulator.sh" ${ANDROID_EMU_IMG}
+    trap "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android/stop-emulator.sh" EXIT
+    . "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android/setup-env-for-emulator.sh"
+
+    # Create adb_run early to avoid concurrent `mkdir -p` of common parent
+    # directories.
+    adb shell mkdir -p /data/local/tmp/adb_run
+    adb push "${BUILD_DIR}/lib/libc++_shared.so" /data/local/tmp/libc++/libc++_shared.so
+    echo "+++ Running the libc++ tests"
+    ${NINJA} -vC "${BUILD_DIR}" check-cxx
+    echo "+++ Running the libc++abi tests"
+    ${NINJA} -vC "${BUILD_DIR}" check-cxxabi
+;;
 #################################################################
 # Insert vendor-specific internal configurations below.
 #

libcxx/utils/ci/vendor/android/Dockerfile.emulator

@@ -0,0 +1,59 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+FROM ubuntu:jammy
+
+RUN apt-get update && apt-get install -y \
+    curl \
+    netcat-openbsd \
+    openjdk-11-jdk \
+    sudo \
+    unzip \
+    && rm -rf /var/lib/apt/lists/*
+
+ENV ANDROID_HOME /opt/android/sdk
+
+RUN curl -sL https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip -o cmdline-tools.zip && \
+    mkdir -p ${ANDROID_HOME} && \
+    unzip cmdline-tools.zip -d ${ANDROID_HOME}/cmdline-tools && \
+    mv ${ANDROID_HOME}/cmdline-tools/cmdline-tools ${ANDROID_HOME}/cmdline-tools/latest && \
+    rm cmdline-tools.zip
+ENV PATH="${ANDROID_HOME}/cmdline-tools/latest/bin:${PATH}"
+
+RUN yes | sdkmanager --licenses
+RUN sdkmanager --install emulator
+ENV PATH="${ANDROID_HOME}/emulator:${PATH}"
+
+ARG API  # e.g. 21
+RUN sdkmanager --install "platforms;android-${API}"
+
+ARG TYPE  # one of: default, google_apis, or google_apis_playstore
+ARG ABI   # e.g. armeabi-v7a, x86
+ENV EMU_PACKAGE_NAME="system-images;android-${API};${TYPE};${ABI}"
+RUN sdkmanager --install "${EMU_PACKAGE_NAME}"
+
+COPY ./emulator-entrypoint.sh /opt/emulator/bin/emulator-entrypoint.sh
+COPY ./emulator-wait-for-ready.sh /opt/emulator/bin/emulator-wait-for-ready.sh
+ENV PATH="/opt/emulator/bin:${PATH}"
+ENV PATH="${ANDROID_HOME}/platform-tools:${PATH}"
+
+# Setup password-less sudo so that /dev/kvm permissions can be changed. Run the
+# emulator in an unprivileged user for reliability (and it might require it?)
+RUN echo "ALL ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers
+RUN useradd --create-home emulator
+USER emulator
+WORKDIR /home/emulator
+
+# Size of emulator /data partition in megabytes.
+ENV EMU_PARTITION_SIZE=8192
+
+EXPOSE 5037
+
+HEALTHCHECK CMD emulator-wait-for-ready.sh 5
+
+ENTRYPOINT ["emulator-entrypoint.sh"]

libcxx/utils/ci/vendor/android/build-emulator-images.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+THIS_DIR="$(cd "$(dirname "$0")" && pwd)"
+. "${THIS_DIR}/emulator-functions.sh"
+
+build_image() {
+    local EMU_IMG="$1"
+    validate_emu_img_syntax "${EMU_IMG}"
+    docker build -t $(docker_image_of_emu_img ${EMU_IMG}) \
+        -f Dockerfile.emulator . \
+        --build-arg API=$(api_of_emu_img ${EMU_IMG}) \
+        --build-arg TYPE=$(type_of_emu_img ${EMU_IMG}) \
+        --build-arg ABI=$(abi_of_arch $(arch_of_emu_img ${EMU_IMG}))
+}
+
+cd "${THIS_DIR}"
+
+build_image 21-def-x86
+build_image 33-goog-x86_64

libcxx/utils/ci/vendor/android/emulator-entrypoint.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+# This script is the entrypoint of an Android Emulator Docker container.
+
+set -e
+
+# The container's /dev/kvm has the same UID+GID as the host device. Changing the
+# ownership inside the container doesn't affect the UID+GID on the host.
+sudo chown emulator:emulator /dev/kvm
+
+# Always use a copy of platform-tools provided by the host to ensure that the
+# versions of adb match between the host and the emulator.
+if [ ! -x /mnt/android-platform-tools/platform-tools/adb ]; then
+    echo "error: This image requires platform-tools mounted at" \
+         "/mnt/android-platform-tools containing platform-tools/adb" >&2
+    exit 1
+fi
+sudo cp -r /mnt/android-platform-tools/platform-tools /opt/android/sdk
+
+# Start an adb host server. `adb start-server` blocks until the port is ready.
+# Use ADB_REJECT_KILL_SERVER=1 to ensure that an adb protocol version mismatch
+# doesn't kill the adb server.
+ADB_REJECT_KILL_SERVER=1 adb -a start-server
+
+# This syntax (using an IP address of 127.0.0.1 rather than localhost) seems to
+# prevent the adb client from ever spawning an adb host server.
+export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
+
+# The AVD could already exist if the Docker container were stopped and then
+# restarted.
+if [ ! -d ~/.android/avd/emulator.avd ]; then
+    # N.B. AVD creation takes a few seconds and creates a mostly-empty
+    # multi-gigabyte userdata disk image. (It's not useful to create the AVDs in
+    # advance.)
+    avdmanager --verbose create avd --name emulator \
+        --package "${EMU_PACKAGE_NAME}" --device pixel_5
+fi
+
+# Use exec so that the emulator is PID 1, so that `docker stop` kills the
+# emulator.
+exec emulator @emulator -no-audio -no-window \
+    -partition-size "${EMU_PARTITION_SIZE}"

libcxx/utils/ci/vendor/android/emulator-functions.sh

@@ -0,0 +1,110 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+# Bash functions for managing the names of emulator system images.
+
+# Parse the image name and set variables: API, TYPE, and ARCH.
+__parse_emu_img() {
+    if [[ "${1}" =~ ([0-9]+)-(def|goog|play)-(arm|arm64|x86|x86_64)$ ]]; then
+        API=${BASH_REMATCH[1]}
+        case ${BASH_REMATCH[2]} in
+            def) TYPE=default ;;
+            goog) TYPE=google_apis ;;
+            play) TYPE=google_apis_playstore ;;
+        esac
+        ARCH=${BASH_REMATCH[3]}
+        return 0
+    else
+        return 1
+    fi
+}
+
+# Check that the emulator image name has valid syntax.
+validate_emu_img_syntax() {
+    local EMU_IMG="${1}"
+    local API TYPE ARCH
+    if ! __parse_emu_img "${EMU_IMG}"; then
+        echo "\
+error: invalid emulator image name: ${EMU_IMG}
+  expected \"\${API}-\${TYPE}-\${ARCH}\" where API is a number, TYPE is one of
+  (def|goog|play), and ARCH is one of arm, arm64, x86, or x86_64." >&2
+        return 1
+    fi
+}
+
+docker_image_of_emu_img() {
+    echo "android-emulator-${1}"
+}
+
+# Check that the emulator image name has valid syntax and that the Docker image
+# is present. On failure, writes an error to stderr and exits the script.
+validate_emu_img() {
+    local EMU_IMG="${1}"
+    if ! validate_emu_img_syntax "${EMU_IMG}"; then
+        return 1
+    fi
+    # Make sure Docker is working before trusting other Docker commands.
+    # Temporarily suppress command echoing so we only show 'docker info' output
+    # on failure, and only once.
+    if (set +x; !(docker info &>/dev/null || docker info)); then
+        echo "error: Docker is required for emulator usage but 'docker info' failed" >&2
+        return 1
+    fi
+    local DOCKER_IMAGE=$(docker_image_of_emu_img ${EMU_IMG})
+    if ! docker image inspect ${DOCKER_IMAGE} &>/dev/null; then
+        echo "error: emulator Docker image (${DOCKER_IMAGE}) is not installed" >&2
+        return 1
+    fi
+}
+
+api_of_emu_img() {
+    local API TYPE ARCH
+    __parse_emu_img "${1}"
+    echo ${API}
+}
+
+type_of_emu_img() {
+    local API TYPE ARCH
+    __parse_emu_img "${1}"
+    echo ${TYPE}
+}
+
+arch_of_emu_img() {
+    local API TYPE ARCH
+    __parse_emu_img "${1}"
+    echo ${ARCH}
+}
+
+# Expand the short emu_img string into the full SDK package string identifying
+# the system image.
+sdk_package_of_emu_img() {
+    local API TYPE ARCH
+    __parse_emu_img "${1}"
+    echo "system-images;android-${API};${TYPE};$(abi_of_arch ${ARCH})"
+}
+
+# Return the Android ABI string for an architecture.
+abi_of_arch() {
+    case "${1}" in
+        arm) echo armeabi-v7a ;;
+        arm64) echo aarch64-v8a ;;
+        x86) echo x86 ;;
+        x86_64) echo x86_64 ;;
+        *) echo "error: unhandled arch ${1}" >&2; exit 1 ;;
+    esac
+}
+
+triple_of_arch() {
+    case "${1}" in
+        arm) echo armv7a-linux-androideabi ;;
+        arm64) echo aarch64-linux-android ;;
+        x86) echo i686-linux-android ;;
+        x86_64) echo x86_64-linux-android ;;
+        *) echo "error: unhandled arch ${1}" >&2; exit 1 ;;
+    esac
+}

libcxx/utils/ci/vendor/android/emulator-wait-for-ready.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -ex
+
+# Time to wait in seconds. The emulator ought to start in 5-15 seconds or so,
+# so add a safety factor in case something takes longer in CI.
+TIMEOUT=${1-300}
+
+# This syntax (using an IP address of 127.0.0.1 rather than localhost) seems to
+# prevent the adb client from ever spawning an adb host server.
+export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
+
+# Invoke nc first to ensure that something is listening to port 5037. Otherwise,
+# invoking adb might fork an adb server.
+#
+# TODO: Consider waiting for `adb shell getprop dev.bootcomplete 2>/dev/null
+# | grep 1 >/dev/null` as well. It adds ~4 seconds to 21-def-x86 and ~15 seconds
+# to 33-goog-x86_64 and doesn't seem to be necessary for running libc++ tests.
+timeout ${TIMEOUT} bash -c '
+until (nc -z localhost 5037 && adb wait-for-device); do
+    sleep 0.5
+done
+'

libcxx/utils/ci/vendor/android/setup-env-for-emulator.sh

@@ -0,0 +1,13 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+export ADB_SERVER_SOCKET="tcp:$(docker inspect \
+    -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' \
+    libcxx-ci-android-emulator):5037"
+
+echo "setup-env-for-emulator.sh: setting ADB_SERVER_SOCKET to ${ADB_SERVER_SOCKET}"

libcxx/utils/ci/vendor/android/start-emulator.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+# Starts a new Docker container using a Docker image containing the Android
+# Emulator and an OS image. Stops and removes the old container if it exists
+# already.
+
+set -e
+
+THIS_DIR="$(cd "$(dirname "$0")" && pwd)"
+. "${THIS_DIR}/emulator-functions.sh"
+
+EMU_IMG="${1}"
+if ! validate_emu_img "${EMU_IMG}"; then
+    echo "error: The first argument must be a valid emulator image." >&2
+    exit 1
+fi
+
+"${THIS_DIR}/stop-emulator.sh"
+
+# Start the container.
+docker run --name libcxx-ci-android-emulator --detach --device /dev/kvm \
+    -eEMU_PARTITION_SIZE=8192 \
+    --volume android-platform-tools:/mnt/android-platform-tools \
+    $(docker_image_of_emu_img ${EMU_IMG})
+ERR=0
+docker exec libcxx-ci-android-emulator emulator-wait-for-ready.sh || ERR=${?}
+echo "Emulator container initial logs:"
+docker logs libcxx-ci-android-emulator
+if [ ${ERR} != 0 ]; then
+    exit ${ERR}
+fi
+
+# Make sure the device is accessible from outside the emulator container and
+# advertise to the user that this script exists.
+. "${THIS_DIR}/setup-env-for-emulator.sh"
+adb wait-for-device

libcxx/utils/ci/vendor/android/stop-emulator.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+THIS_DIR="$(cd "$(dirname "$0")" && pwd)"
+. "${THIS_DIR}/emulator-functions.sh"
+
+# Cleanup the emulator if it's already running.
+if docker container inspect libcxx-ci-android-emulator &>/dev/null; then
+    echo "Stopping existing emulator container..."
+    docker stop libcxx-ci-android-emulator
+
+    echo "Emulator container final logs:"
+    docker logs libcxx-ci-android-emulator
+
+    echo "Removing existing emulator container..."
+    docker rm libcxx-ci-android-emulator
+fi

libcxx/utils/libcxx/test/android.py

@@ -0,0 +1,97 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+import atexit
+import os
+import re
+import select
+import socket
+import subprocess
+import tempfile
+import threading
+from typing import List
+
+
+def _get_cpu_count() -> int:
+    # Determine the number of cores by listing a /sys directory. Older devices
+    # lack `nproc`. Even if a static toybox binary is pushed to the device, it may
+    # return an incorrect value. (e.g. On a Nexus 7 running Android 5.0, toybox
+    # nproc returns 1 even though the device has 4 CPUs.)
+    job = subprocess.run(["adb", "shell", "ls /sys/devices/system/cpu"],
+                         encoding="utf8", check=False,
+                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    if job.returncode == 1:
+        # Maybe adb is missing, maybe ANDROID_SERIAL needs to be defined, maybe the
+        # /sys subdir isn't there. Most errors will be handled later, just use one
+        # job. (N.B. The adb command still succeeds even if ls fails on older
+        # devices that lack the shell_v2 adb feature.)
+        return 1
+    # Make sure there are no CR characters in the output. Pre-shell_v2, the adb
+    # stdout comes from a master pty so newlines are CRLF-delimited. On Windows,
+    # LF might also get expanded to CRLF.
+    cpu_listing = job.stdout.replace('\r', '\n')
+
+    # Count lines that match "cpu${DIGITS}".
+    result = len([line for line in cpu_listing.splitlines()
+                  if re.match(r'cpu(\d)+$', line)])
+
+    # Restrict the result to something reasonable.
+    if result < 1:
+        result = 1
+    if result > 1024:
+        result = 1024
+
+    return result
+
+
+def _job_limit_socket_thread(temp_dir: tempfile.TemporaryDirectory,
+                             server: socket.socket, job_count: int) -> None:
+    """Service the job limit server socket, accepting only as many connections
+    as there should be concurrent jobs.
+    """
+    clients: List[socket.socket] = []
+    while True:
+        rlist = list(clients)
+        if len(clients) < job_count:
+            rlist.append(server)
+        rlist, _, _ = select.select(rlist, [], [])
+        for sock in rlist:
+            if sock == server:
+                new_client, _ = server.accept()
+                new_client.send(b"x")
+                clients.append(new_client)
+            else:
+                sock.close()
+                clients.remove(sock)
+
+
+def adb_job_limit_socket() -> str:
+    """An Android device can frequently have many fewer cores than the host
+    (e.g. 4 versus 128). We want to exploit all the device cores without
+    overburdening it.
+
+    Create a Unix domain socket that only allows as many connections as CPUs on
+    the Android device.
+    """
+
+    # Create the job limit server socket.
+    temp_dir = tempfile.TemporaryDirectory(prefix="libcxx_")
+    sock_addr = temp_dir.name + "/adb_job.sock"
+    server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    server.bind(sock_addr)
+    server.listen(1)
+
+    # Spawn a thread to service the socket. As a daemon thread, its existence
+    # won't prevent interpreter shutdown. The temp dir will still be removed on
+    # shutdown.
+    cpu_count = _get_cpu_count()
+    threading.Thread(target=_job_limit_socket_thread,
+                     args=(temp_dir, server, cpu_count),
+                     daemon=True).start()
+
+    return sock_addr

libcxxabi/test/configs/llvm-libc++abi-android-ndk.cfg.in

@@ -0,0 +1,40 @@
+# This testing configuration handles running the test suite against LLVM's
+# libc++abi using adb and a libc++_shared.so library on Android.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+import re
+import site
+
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+
+import libcxx.test.android
+import libcxx.test.config
+import libcxx.test.params
+
+config.substitutions.append(('%{flags}',
+    '--sysroot @CMAKE_SYSROOT@' if '@CMAKE_SYSROOT@' else ''
+))
+config.substitutions.append(('%{compile_flags}',
+    '-nostdinc++ -I %{include} -I %{cxx-include} -I %{cxx-target-include} %{maybe-include-libunwind} -I %{libcxx}/test/support -I %{libcxx}/src -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS'
+))
+
+# The NDK library is called "libc++_shared.so". Use LD_LIBRARY_PATH to find
+# libc++_shared.so because older Bionic dynamic loaders don't support rpath
+# lookup. The Android libc++ shared library exports libc++abi, so we don't need
+# to link with -lc++abi.
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib++ -L %{lib} -lc++_shared'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor}' +
+    ' --job-limit-socket ' + libcxx.test.android.adb_job_limit_socket() +
+    ' --prepend-path-env LD_LIBRARY_PATH /data/local/tmp/libc++ --execdir %T -- '
+))
+
+libcxx.test.config.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

llvm/utils/lit/lit/TestingConfig.py

@@ -43,6 +43,7 @@ class TestingConfig(object):
             "TSAN_OPTIONS",
             "UBSAN_OPTIONS",
             "ADB",
+            "ADB_SERVER_SOCKET",
             "ANDROID_SERIAL",
             "SSH_AUTH_SOCK",
             "SANITIZER_IGNORE_CVE_2016_2143",

runtimes/cmake/android/Arch-arm.cmake

@@ -0,0 +1,7 @@
+include(${CMAKE_CURRENT_LIST_DIR}/Common.cmake)
+
+set(CMAKE_SYSTEM_PROCESSOR "armv7-a" CACHE STRING "")
+set(CMAKE_ASM_COMPILER_TARGET "armv7a-linux-androideabi21" CACHE STRING "")
+set(CMAKE_C_COMPILER_TARGET   "armv7a-linux-androideabi21" CACHE STRING "")
+set(CMAKE_CXX_COMPILER_TARGET "armv7a-linux-androideabi21" CACHE STRING "")
+set(ANDROID_NATIVE_API_LEVEL 21 CACHE STRING "")

runtimes/cmake/android/Arch-arm64.cmake

@@ -0,0 +1,7 @@
+include(${CMAKE_CURRENT_LIST_DIR}/Common.cmake)
+
+set(CMAKE_SYSTEM_PROCESSOR "aarch64" CACHE STRING "")
+set(CMAKE_ASM_COMPILER_TARGET "aarch64-linux-android21" CACHE STRING "")
+set(CMAKE_C_COMPILER_TARGET   "aarch64-linux-android21" CACHE STRING "")
+set(CMAKE_CXX_COMPILER_TARGET "aarch64-linux-android21" CACHE STRING "")
+set(ANDROID_NATIVE_API_LEVEL 21 CACHE STRING "")

runtimes/cmake/android/Arch-x86.cmake

@@ -0,0 +1,7 @@
+include(${CMAKE_CURRENT_LIST_DIR}/Common.cmake)
+
+set(CMAKE_SYSTEM_PROCESSOR "i686" CACHE STRING "")
+set(CMAKE_ASM_COMPILER_TARGET "i686-linux-android21" CACHE STRING "")
+set(CMAKE_C_COMPILER_TARGET   "i686-linux-android21" CACHE STRING "")
+set(CMAKE_CXX_COMPILER_TARGET "i686-linux-android21" CACHE STRING "")
+set(ANDROID_NATIVE_API_LEVEL 21 CACHE STRING "")

runtimes/cmake/android/Arch-x86_64.cmake

@@ -0,0 +1,7 @@
+include(${CMAKE_CURRENT_LIST_DIR}/Common.cmake)
+
+set(CMAKE_SYSTEM_PROCESSOR "x86_64" CACHE STRING "")
+set(CMAKE_ASM_COMPILER_TARGET "x86_64-linux-android21" CACHE STRING "")
+set(CMAKE_C_COMPILER_TARGET   "x86_64-linux-android21" CACHE STRING "")
+set(CMAKE_CXX_COMPILER_TARGET "x86_64-linux-android21" CACHE STRING "")
+set(ANDROID_NATIVE_API_LEVEL 21 CACHE STRING "")

runtimes/cmake/android/Common.cmake

@@ -0,0 +1,6 @@
+set(CMAKE_SYSTEM_NAME "Android" CACHE STRING "")
+
+# Set the CMake system version to "1" to inhibit CMake's built-in support for
+# compiling using the Android NDK, which gets in the way when we're not using an
+# NDK.
+set(CMAKE_SYSTEM_VERSION "1" CACHE STRING "")