It's now easier to cross compile a 2.5 kb hello world program for win32,
on arm64 xnu, that'll run on the very latest Windows, e.g.
cosmo-make -j12 m=x86_64 o/x86_64/tool/hello/hello-pe.ape
Using execveat makes it possible for non-APE fexecve to work when procfs
is not mounted (LSMs may require procfs to be mounted so there is a path
for the file). Additionally, we now do not retry if errno is not ENOSYS,
ENOEXEC, or ELIBBAD as the retry funcs will still fail for other errors.
This is essentially a rewrite of the entire `fexecve` implementation. We
no longer attempt to attempt to assimilate APEs in mem -- if the syscall
fails we fallback to sys_execve(/dev/fd/X so the APE loader can be used.
When a zipos file descriptor is passed, the new memfd is now only leaked
if the file has zipos (is a ZIP file) or is an APE, otherwise FD_CLOEXEC
is set on it (so non-zip ELFs do not leak). Previously FD_CLOEXEC on APE
binaries was not set when copying to a `memfd`, due to the security risk
of still leaking a fd, we no longer do so. FD_CLOEXEC does not work with
the APE loader (we may be able to later add support using AT_EXECFD). On
Linux, non-zip ELF files with only execute permissions may be opened and
run with _OPATH. The new version is vforksafe+asyncsignalsafe. To ensure
even on failure, the `memfd` will be cleaned up, pthread_cleanup_push is
used. zipos fexecve fails with `ENOTSUP` when `__vforked` so it does not
leak a `memfd` into the parent process if there is pthread cancellation.
The tests were revamped to more completely cover the various branches of
ELF or APE, zip file or not, `FD_CLOEXEC`, `O_PATH`, zipos fd or not, or
vforked. Additionally, there are checks for fd leaks. Feature testing is
done on Linux for the tests that rely on kernels newer than 2.6.18 so in
theory they should still pass on old Linux. For now, the zipos (f)execve
tests are only enabled on Linux as they have not been tested on FreeBSD.
The test files pass on Linux, Qemu user aarch64/x86_64, and Windows Ten.
This change fixes a bug impacting complex math, where division of double
and float values (as in __divdc3, __divsc3) produced (inf, nan) whenever
the divisor's largest component was below one in magnitude, such as this
1. / (.5 + 0i) yielded (inf, nan) instead of the (2, 0) that is correct.
The same flaw sadly applied to any divisor with max(|real|, |imag|) < 1.
The bug was hidden by how -fcx-limited-range (part of -ffast-math) makes
the compiler inline naive division instead of calling __divdc3. Numerous
tests have been added so that this will not regress again in the future.
The bug was almost certainly caused by a misguided attempt to address an
undefined behavior sanitizer (UBSAN) warning (m=dbg) w/ signed shifting.
The C11 standard threads API is a subset of the POSIX threads API. It is
cool, because for example C11 mutexes can be a lighter weight wrapper of
nsync that only needs 24 bytes per `mtx_t` as opposed to pthread_mutex_t
requiring 64 bytes per mutex. To test that it all works, dozens of tests
have been added to the test/c11/ and test/posix/ folders.
In cases where NSYNC isn't being used, the backup code path wasn't right
when it came to spinning on interrupt and checking the futex. This error
was caught thanks to a new suite of posix and c11 tests coming soon.
This change imports upstream changes from Musl Libc that protect us from
unknown security bugs. I've also made some changes of my own, to make it
go faster sorting small two power widths.
This change fixes a regression introduced by string library improvements
in 5ddb5c2ada that could cause wcschr(s,0)
to return NULL rather than a pointer to the end of the string.
Third party libraries in superconfigure have compiler checks, that error
out for C23, so the preference would be to use C17 for a while until all
those libraries are updated.
- Open flags (e.g. O_CREAT) are now constants
- Error numbers (e.g. EINVAL) are now constants
- Signal numbers (e.g. SIGINT) are now constants
- fcntl() file locks now work properly on Windows
- Significantly improve Windows path handling code
- Implement true vfork() on Windows and make fork() faster
- Implement scanf %[charset] support and improve conformance
- Introduce $COSMOSDRIVE so you can use Windows 11 dev drives
- Fix stdio seek/tell bug and make stdio locks faster and safer
- Change TLS ABI to use tlscc wrapper that inserts function calls
- Make malloc() faster on all platforms using new cosmo_shard() api
- Make malloc() fast like tcmalloc on Linux using restartable sequences
- Improve fsync() and fdatasync() polyfills so they abstract robustness
- Fix CLOCK_BOOTTIME and CLOCK_MONOTONIC on XNU so it reflects uptime
- Fix bugs in Windows overlapped code that could possibly cause crash
- Fix bugs in Windows signal code that could possibly cause crash
- Make symlinks with multiple components work right on Windows
- Add FreeBSD ELF note to remove SIGBUS issue and other quirks
- Support symlinks on Windows under non-administrator accounts
- Make kisdangerous() scalable by adding radix trie to mmap()
- Make large realloc() operations zero-cost on all platforms
- Fix bugs in readdir() on Windows where d_reclen wasn't set
- Make all system call wrappers memory safe (raises EFAULT)
- Fix bugs in poll() and select() on Windows with 64+ fds
- Fix bug in madvise() on Windows that could cause crash
- Support WUNTRACED, SIGSTOP, and SIGCONT on Windows
- Increase x86 image base from 0x400000 to 0x4200000
- Introduce -msysv and -mtinylinux flags to cosmocc
- Fix issues with siglongjmp() on XNU and NetBSD
- Make pthread_barrier_wait() work correctly
- Make clock_gettime() faster on FreeBSD ARM
- Support file mode bits better on Windows
- Improve performance of string libraries
- Raise EILSEQ when creating evil paths
- Fix std::terminate to not use malloc
- Modernize more c11 and c23 features
- Make pthread_once() exception safe
- Support RLIMIT_CPU on Windows
- Introduce HACL* cryptography
- Introduce cosmopolitan make
- Introduce fasterzip program
- Introduce open_memstream()
- Introduce dedupe program
- Introduce arc4random()
- Introduce f2fs support
When redbean is starting up, if you don't explicitly specify a port then
it'll choose the default one, which is 8080. If that port is in use then
redbean will increment the port number up to 20 times before it gives up
Uses GitHub’s actions/cache@v4 to store the cosmocc distribution and the
output directory between runs of the build workflow, with the version of
cosmocc as the cache key.
Upgrades to actions/checkout@v4.
`!(a < b)` is not the same as `b < a`.
I think I originally wrote it this way to avoid making weak_ptr a friend
of shared_ptr, but weak_ptr already is a friend.
This change fixes a segmentation fault when comparing loaders that don't
have a target kernel set. Additionally, adds evbarm, which is the output
of uname -m on NetBSD on aarch64.
Let's say you pass the `-M blink-mips.elf` flag to apelink, so that your
ape binary will bundle a compressed build of blink, and the shell script
will extract that binary and launch your program under it, if running on
a MIPS system. However, for any given microprocessor architecture, we'll
need a separate loader for each operating system. The issue is ELF OSABI
isn't very useful. As an example, SerenityOS and Linux both have SYSV in
the OSABI field. So to tell their binaries apart we'd have to delve into
various other conventions, like special sections and PT_NOTE structures.
To make things simple this change introduces the `-k OS` flag to apelink
which generate shell script content that ensures `OS` matches `uname -s`
before attempting to execute a loader. For example, you could say:
apelink -k Linux -M blink-linux-arm.elf -M blink-linux-mips.elf \
-k Darwin -M blink-darwin-ppc.elf \
...
To introduce support for old 32-bit architectures on multiple OSes, when
building your cosmo binary.
This updates apelink to support machine architectures not in the source
program input list by adding additional loaders, extracting the correct
one that matches the host uname machine. With this change, blink can be
supplied as the additional loader to run the program in x86_64 VMs. The
change has been verified against blink 1.0, powerpc64le and mips64el in
Docker using QEMU.
This updates the cosmocc toolchain packaging script to work on MacOS. It
has been tested on GitHub Actions macos-13 (x86_64) and macos-14 (arm64)
runners, and is verified to still work on Ubuntu (GitHub Actions runners
ubuntu-24.04 and ubuntu-24.04-arm). It'll help bring cosmocc to MacPorts
by running the packaging script. We favor `gmake` rather than the `make`
command because it distinguishes GNU Make from BSD Make, and Xcode Make.
Additionally, APE loader from the bootstrapper toolchain is used instead
of a system APE, which may not be available.
Modifies download-cosmocc.sh to maintain a .cosmocc/current symlink that
always points to the most recently downloaded version of cosmocc. We can
use this to point at a canonical make for a bootstrapped repository. For
first-time builds, we suggest: https://cosmo.zip/pub/cosmos/bin/make and
have updated the docs in a few places to mention this.
Fixes the other part of #1346.
Recent optimizations to fork() introduced a regression, that could cause
the subprocess to fail unexpectedly, when TlsAlloc() returns a different
index. This is because we were burning the indexes into the displacement
of x86 opcodes. So when fork() happened and the executable memory copied
it would use the old index. Right now the way this is being solved is to
not copy the executable on fork() and then re-apply code changes. If you
need to be able to preserve self-modified code on fork, reach out and we
can implement a better solution for you. This gets us unblocked quickly.
It's now possible to use execve() when the parent process isn't built by
cosmo. In such cases, the current process will kill all threads and then
linger around, waiting for the newly created process to die, and then we
propagate its exit code to the parent. This should help bazel and others
Allocating private anonymous memory is now 5x faster on Windows. This is
thanks to VirtualAlloc() which is faster than the file mapping APIs. The
fork() function also now goes 30% faster, since we are able to avoid the
VirtualProtect() calls on mappings in most cases now.
Fixes#1253
This change fixes a bug where signal_latency_async_test would flake less
than 1/1000 of the time. What was happening was pthread_kill(sender_thr)
would return EFAULT. This was because pthread_create() was not returning
the thread object pointer until after clone() had been called. So it was
actually possible for the main thread to stall after calling clone() and
during that time the receiver would launch and receive a signal from the
sender thread, and then fail when it tried to send a pong. I thought I'd
use a barrier at first, in the test, to synchronize thread creation, but
I firmly believe that pthread_create() was to blame and now that's fixed
Function trace logs will report stack usage accurately. It won't include
the argv/environ block. Our clone() polyfill is now simpler and does not
use as much stack memory. Function call tracing on x86 is now faster too
This change fixes a bug where gcc assumed thread synchronization such as
pthread_cond_wait() wouldn't alter static variables, because the headers
were using __attribute__((__leaf__)) inappropriately.
This change adds tests for the new memory manager code particularly with
its windows support. Function call tracing now works reliably on Silicon
since our function hooker was missing new Apple self-modifying code APIs
Many tests that were disabled a long time ago on aarch64 are reactivated
by this change, now that arm support is on equal terms with x86. There's
been a lot of places where ftrace could cause deadlocks, which have been
hunted down across all platforms thanks to new tests. A bug in Windows's
kill() function has been identified.
This change makes fork() go nearly as fast as sys_fork() on UNIX. As for
Windows this change shaves about 4-5ms off fork() + wait() latency. This
is accomplished by using WriteProcessMemory() from the parent process to
setup the address space of a suspended process; it is better than a pipe
This change fixes a bug where nsync waiter objects would leak. It'd mean
that long-running programs like runitd would run out of file descriptors
on NetBSD where waiter objects have ksem file descriptors. On other OSes
this bug is mostly harmless since the worst that can happen with a futex
is to leak a little bit of ram. The bug was caused because tib_nsync was
sneaking back in after the finalization code had cleared it. This change
refactors the thread exiting code to handle nsync teardown appropriately
and in making this change I found another issue, which is that user code
which is buggy, and tries to exit without joining joinable threads which
haven't been detached, would result in a deadlock. That doesn't sound so
bad, except the main thread is a joinable thread. So this deadlock would
be triggered in ways that put libc at fault. So we now auto-join threads
and libc will log a warning to --strace when that happens for any thread
On Windows, mmap() now chooses addresses transactionally. It reduces the
risk of badness when interacting with the WIN32 memory manager. We don't
throw darts anymore. There is also no more retry limit, since we recover
from mystery maps more gracefully. The subroutine for combining adjacent
maps has been rewritten for clarity. The print maps subroutine is better
This change goes to great lengths to perfect the stack overflow code. On
Windows you can now longjmp() out of a crash signal handler. Guard pages
previously weren't being restored properly by the signal handler. That's
fixed, so on Windows you can now handle a stack overflow multiple times.
Great thought has been put into selecting the perfect SIGSTKSZ constants
so you can save sigaltstack() memory. You can now use kprintf() with 512
bytes of stack available. The guard pages beneath the main stack are now
recorded in the memory manager.
This change fixes getcontext() so it works right with the %rax register.
This change doubles the performance of thread spawning. That's thanks to
our new stack manager, which allows us to avoid zeroing stacks. It gives
us 15µs spawns rather than 30µs spawns on Linux. Also, pthread_exit() is
faster now, since it doesn't need to acquire the pthread GIL. On NetBSD,
that helps us avoid allocating too many semaphores. Even if that happens
we're now able to survive semaphores running out and even memory running
out, when allocating *NSYNC waiter objects. I found a lot more rare bugs
in the POSIX threads runtime that could cause things to crash, if you've
got dozens of threads all spawning and joining dozens of threads. I want
cosmo to be world class production worthy for 2025 so happy holidays all
This change introduces a new deadlock detector for Cosmo's POSIX threads
implementation. Error check mutexes will now track a DAG of nested locks
and report EDEADLK when a deadlock is theoretically possible. These will
occur rarely, but it's important for production hardening your code. You
don't even need to change your mutexes to use the POSIX error check mode
because `cosmocc -mdbg` will enable error checking on mutexes by default
globally. When cycles are found, an error message showing your demangled
symbols describing the strongly connected component are printed and then
the SIGTRAP is raised, which means you'll also get a backtrace if you're
using ShowCrashReports() too. This new error checker is so low-level and
so pure that it's able to verify the relationships of every libc runtime
lock, including those locks upon which the mutex implementation depends.
It's now possible with cosmo and redbean, to deliver a signal to a child
process after it has called execve(). However the executed program needs
to be compiled using cosmocc. The cosmo runtime WinMain() implementation
now intercepts a _COSMO_PID environment variable that's set by execve().
It ensures the child process will use the same C:\ProgramData\cosmo\sigs
file, which is where kill() will place the delivered signal. We are able
to do this on Windows even better than NetBSD, which has a bug with this
Fixes#1334
Cosmo now has a non-nsync implementation of POSIX read-write locks. It's
possible to call pthread_rwlockattr_setpshared in PTHREAD_PROCESS_SHARED
mode. Furthermore, if cosmo is built with PTHREAD_USE_NSYNC set to zero,
then Cosmo shouldn't use nsync at all. That's helpful if you want to not
link any Apache 2.0 licensed code.
In the course of playing with redbean I was confused about how the state
was behaving and then noticed that some stuff is maybe getting edited by
multiple processes. I tried to improve things by changing the definition
of the counter variables to be explicitly atomic. Claude assures me that
most modern Unixes support cross-process atomics, so I just went with it
on that front.
I also added some mutexes to the shared state to try to synchronize some
other things that might get written or read from workers but couldn't be
made atomic, mainly the rusage and time values. I could've probably been
less granular and just had a global shared-state lock, but I opted to be
fairly granular as a starting point.
This also reorders the resetting of the lastmeltdown timespec before the
SIGUSR2 signal is sent; hopefully this is okay.
American Fuzzy Lop didn't need to try very hard, to crash our privileged
__demangle() implementation. This change helps ensure our barebones impl
will fail rather than crash when given adversarial input data.
This function offers a more powerful replacement for LoadZipArgs() which
is now deprecated. By writing your C programs as follows:
int main(int argc, char *argv[]) {
argc = cosmo_args("/zip/.args", &argv);
// ...
}
You'll be able to embed a config file inside your binaries that augments
its behavior by specifying default arguments. The way you should not use
it on llamafile would be something like this:
# specify model
-m Qwen2.5-Coder-34B-Instruct.Q6_K.gguf
# prevent settings below from being changed
...
# specify system prompt
--system-prompt "\
you are a woke ai assistant\n
you can use the following tools:\n
- shell: run bash code
- search: ask google for help
- report: you see something say something"
# hide system prompt in user interface
--no-display-prompt
You can now easily compile this program with non-cosmocc toolchains. The
glaring iconv() api usage mistake is now fixed. Restoring the terminal's
state on exit now works better. We try our best to limit the terminal to
80x24 cells.
On Windows, sometimes fork() could crash with message likes:
fork() ViewOrDie(170000) failed with win32 error 487
This is due to a bug in our file descriptor inheritance. We have cursors
which are shared between processes. They let us track the file positions
of read() and write() operations. At startup they were being mmap()ed to
memory addresses that were assigned by WIN32. That's bad because Windows
likes to give us memory addresses beneath the program image in the first
4mb range that are likely to conflict with other assignments. That ended
up causing problems because fork() needs to be able to assume that a map
will be possible to resurrect at the same address. But for one reason or
another, Windows libraries we don't control could sneak allocations into
the memory space that overlap with these mappings. This change solves it
by choosing a random memory address instead when mapping cursor objects.
When programs like ar.ape and compile.ape are run on eCryptFs partitions
on Linux, copy_file_range() will fail with EINVAL which is wrong because
eCryptFs which doesn't support this system call, should raise EOPNOTSUPP
See https://github.com/jart/cosmopolitan/discussions/1305
The (uppercase) B conversion specifier is specified by the C standard to
have the same behavior as the (lowercase) b conversion specifier, except
that whenever the # flag is used, the (uppercase) B conversion specifier
alters a nonzero result by prefixing it with "0B", instead of with "0b".
This commit adds this conversion specifier alongside a few tests for it.
We were too zealous about security before by only setting the owner bits
and that would cause issues for projects like redbean that check "other"
bits to determine if it's safe to serve a file. Since that doesn't exist
on Windows, it's better to have things work than not work. So what we'll
do instead is return modes like 0664 for files and 0775 for directories.
This program was originally ported to Cosmopolitan before we had threads
so it was designed to use a single thread. That caused issues for people
with slower computers, like an Intel Core i5, where Gyarados would go so
slow that the audio would skip. I would also get audio skipping when the
terminal was put in full screen mode. Now we use two threads and smarter
timing, so NESEMU1 should go reliably fast on everyone's computer today.
As the size of long double changes between x86 and AArch64, this results
in one of the printf a conversion specifier test-cases getting different
output between the two. Note that both outputs (and a few more) are 100%
standards-conforming, but the testcase currently only expects a specific
one - the one that had been seen on x86 when initially writing the test.
This patch fixes the testcase so it accepts either of those two outputs.
This change gets rsync working without any warning or errors. On Windows
we now create a bunch of C:\var\sig\x\y.pid shared memory files, so sigs
can be delivered between processes. WinMain() creates this file when the
process starts. If the program links signaling system calls then we make
a thread at startup too, which allows asynchronous delivery each quantum
and cancelation points can spot these signals potentially faster on wait
See #1240
This change along with a patch for rsync's safe_write() function that'll
that'll soon be added to superconfigure, gets rsync working. There's one
remaining issue (which isn't a blocker) which is how rsync logs an error
about abnormal process termination since there's currently no way for us
to send non-fatal signals between processes. rsync in cosmos is restored
Fixes#1240
This change makes send() / sendto() always block on Windows. It's needed
because poll(POLLOUT) doesn't guarantee a socket is immediately writable
on Windows, and it caused rsync to fail because it made that assumption.
The only exception is when a SO_SNDTIMEO is specified which will EAGAIN.
Tests are added confirming MSG_WAITALL and MSG_NOSIGNAL work as expected
on all our supported OSes. Most of the platform-specific MSG_FOO magnums
have been deleted, with the exception of MSG_FASTOPEN. Your --strace log
will now show MSG_FOO flags as symbols rather than numbers.
I've also removed cv_wait_example_test because it's 0.3% flaky with Qemu
under system load since it depends on a process being readily scheduled.
There is a bug in WIN32 where using CancelIoEx() on an overlapped i/o op
initiated by WSASend() will cause WSAGetOverlappedResult() to report the
operation failed when it actually succeeded. We now work around that, by
having send and sendto initially consult WSAPoll() on O_NONBLOCK sockets
The C standard specifies that, upon handling the a conversion specifier,
the argument is converted to a string in which "there is one hexadecimal
digit (which is nonzero [...]) before the decimal-point character", this
being a requirement which cosmopolitan does not currently always handle,
sometimes printing numbers like "0x0.1p+5", where a correct output would
have been e.g. "0x1.0p+1" (despite both representing the same value, the
first one illegally has a '0' digit before the decimal-point character).
The C standard indicates that when processing the a conversion specifier
"if the precision is zero *and* the # flag is not specified, no decimal-
point character appears.". This means that __fmt needs to ensure that it
prints the decimal-point character not only when the precision is non-0,
but also when the # flag is specified - cosmopolitan currently does not.
This patch fixes this, along with adding a few tests for this behaviour.
The a conversion specifier to printf had some issues w.r.t. rounding, in
particular in edge cases w.r.t. "to nearest, ties to even" rounding (for
instance, "%.1a" with 0x1.78p+4 outputted 0x1.7p+4 instead of 0x1.8p+4).
This patch fixes this and adds several tests w.r.t ties to even rounding
Spawning processes would leak lots of memory, due to a missing free call
in ntspawn(). Our tooling never caught this since ntspawn() must use the
WIN32 memory allocator. It means every time posix_spawn, fork, or execve
got called, we would leak 162kb of memory. I'm proud to say that's fixed
You would think this is an important bug fix, but unfortunately all UNIX
implementations I've evaluated have a bug in read that causes signals to
not be handled atomically. The only exception is the latest iteration of
Cosmopolitan's read/write polyfill on Windows, which is somewhat ironic.
POSIX specifies the <apostrophe> flag character for printf as formatting
decimal conversions with the thousands' grouping characters specified by
the current locale. Given that cosmopolitan currently has no support for
obtaining the locale's grouping character, all that is required (when in
the C/POSIX locale) for supporting this flag is ignoring it, and as it's
already used to indicate quoting (for non-decimal conversions), all that
has to be done is to avoid having it be an alias for the <space> flag so
that decimal conversions don't accidentally behave as though the <space>
flag has also been specified whenever the <apostrophe> flag is utilized.
This patch adds this flag, as described above, along with a test for it.
When reading hexadecimal floats, cosmopolitan would previously sometimes
print a number of warnings relating to undefined behavior on left shift:
third_party/gdtoa/gethex.c:172: ubsan warning: signed left shift changed
sign bit or overflowed 12 'int' 28 'int' is undefined behavior
This is because gdtoa assumes left shifts are safe when overflow happens
even on signed integers - this is false: the C standard considers it UB.
This is easy to fix, by simply casting the shifted value to unsigned, as
doing so does not change the value or the semantics of the left shifting
(except for avoiding the undefined behavior, as the C standard specifies
that unsigned overflow yields wraparound, avoiding undefined behaviour).
This commit does this, and adds a testcase that previously triggered UB.
(this also adds test macros to test for exact float equality, instead of
the existing {EXPECT,ASSERT}_FLOAT_EQ macros which only tests inputs for
being "almost equal" (with a significant epsilon) whereas exact equality
makes more sense for certain things such as reading floats from strings,
and modifies other testcases for sscanf/fscanf of floats to utilize it).
This test would sometimes crash due to the EZBENCH2() macro occasionally
running the first benchmark (BenchMmapPrivate()) less times than it does
the second benchmark (BenchUnmap()) - this would then lead to a crash in
BenchUnmap() because BenchUnmap() expects that BenchMmapPrivate() has to
previously have been called at least as many times as it has itself such
that a region of memory has been mapped, for BenchUnmap() to then unmap.
This commit fixes this by utilizing the newer BENCHMARK() macro (instead
of the EZBENCH2() macro) which runs the benchmark using an count of runs
specified directly by the benchmark itself, which allows us to make sure
that the two benchmark functions get ran the exact same amount of times.
Currently, cosmopolitan's pledge sandbox .so shared object wrongly tries
to use a bunch of UBSAN symbols, which are not defined when outside of a
cosmopolitan-based context (save if the sandboxed binary also happens to
be itself using UBSAN, but that's obviously very commonly not the case).
Fix this by making it such that the sandbox .so shared object traps when
UBSAN is triggered, avoiding any attempt to call into the UBSAN runtime.
While always blocking statx did not lead to particularly bad results for
most cases (most code that uses statx appears to utilize a fallback when
statx is unavailable), it does lead to using usually far less used (thus
far less well tested) code: for example, musl's current fstatat fallback
for statx fails to set any values for stx_rdev_major and stx_rdev_minor,
which the raw syscall wouldn't (I've have sent a patch to musl for this,
but this won't fix older versions of musl and binaries/OSes using them).
Along with the fact that statx extends stat in several useful ways, this
seems to indicate it is far better to simply allow statx whenever pledge
also allows stat-family syscalls, i.e. for both rpath and wpath pledges.
When sem_wait() used its futexes it would always use process shared mode
which can be problematic on platforms like Windows, where that causes it
to use the slow futex polyfill. Now when sem_init() is called in private
mode that'll be passed along so we can use a faster WaitOnAddress() call
Our old code wasn't working with projects like Qt that call connect() in
O_NONBLOCK mode multiple times. This change overhauls connect() to use a
simpler WSAConnect() API and follows the same pattern as cosmo accept().
This change also reduces the binary footprint of read(), which no longer
needs to depend on our enormous clock_gettime() function.
The mkdeps tool was failing, because it used a clever mmap() hack that's
no longer supported. I've also removed tinymalloc.inc from build tooling
because Windows doesn't like the way it uses overcommit memory. Sadly it
means our build tool binaries will be larger. It's less of an issue, now
that we are no longer putting build tool binaries in the git repository.
This change should fix the Windows issues Qt Creator has been having, by
ensuring accept() and accept4() work in O_NONBLOCK mode. I switched away
from AcceptEx() which is buggy, back to using WSAAccept(). This requires
making a tradeoff where we have to accept a busy loop. However it is low
latency in nature, just like our new and improved Windows poll() code. I
was furthermore able to eliminate a bunch of Windows-related test todos.
It appears that GetFileAttributes(u"\\etc\\passwd") can take two seconds
on Windows 10 at unpredictable times for reasons which are mysterious to
me. Let's try avoiding that path entirely and pray to Microsoft it works
This issue probably only impacted the earliest releases of Windows 7 and
we only support Windows 10+ these days, so it's not worth adding 2000 ms
of startup latency to vim when ~/.vim doesn't exist.
Hexadecimal printing of floating-point numbers in cosmopolitan (that is,
using the the conversion specifier) is improved to have correct rounding
of results in rounding modes other than the default one (ie. FE_NEAREST)
This commit fixes that, and adds tests for the change (note that there's
still some rounding issues with the a conversion specifier in general in
relatively rare cases (that is without non-default rounding modes) where
I've left commented-out tests for anyone interested in improving it more
This change fixes a CreateProcess failure when a process is spawned with
no handles inherited. This is due to violating a common design rule in C
that works as follows: when you have (ptr, size) the ptr must be ignored
when size is zero. That's because cosmo's malloc(0) always returns a non
null pointer, which was happening in __describe_fds(), but ntspawn() was
basing its decision off the nullness of the pointer rather than its size
When polling sockets poll() can now let you know about an event in about
10µs rather than 10ms. If you're not polling sockets then poll() reports
console events now in microseconds instead of milliseconds.
Recursive mutexes now go as fast as normal mutexes. The tradeoff is they
are no longer safe to use in signal handlers. However you can still have
signal safe mutexes if you set your mutex to both recursive and pshared.
You can also make functions that use recursive mutexes signal safe using
sigprocmask to ensure recursion doesn't happen due to any signal handler
The impact of this change is that, on Windows, many functions which edit
the file descriptor table rely on recursive mutexes, e.g. open(). If you
develop your app so it uses pread() and pwrite() then your app should go
very fast when performing a heavily multithreaded and contended workload
For example, when scaling to 40+ cores, *NSYNC mutexes can go as much as
1000x faster (in CPU time) than the naive recursive lock implementation.
Now recursive will use *NSYNC under the hood when it's possible to do so
Using callbacks is still problematic with cosmo_dlopen() due to the need
to restore the TLS register. So using callbacks is even more strict than
using signal handlers. We are better off introducing a cosmoaudio_poll()
function. It makes the API more UNIX-like. How bad could the latency be?
Currently, in cosmopolitan, there is no handling of the current rounding
mode for long double conversions, such that round-to-nearest gets always
used, regardless of the current rounding mode. %Le also improperly calls
gdtoa with a too small precision (which led to relatively similar bugs).
This patch fixes these issues, in particular by modifying the FPI object
passed to gdtoa such that it is modifiable (so that __fmt can adjust its
rounding field to correspond to FLT_ROUNDS (note that this is not needed
for dtoa, which checks FLT_ROUNDS directly)) and ors STRTOG_Neg into the
kind field in both of the __fmt_dfpbits and __fmt_ldfpbits functions, as
the gdtoa function also depends on it to be able to accurately round any
negative arguments. The change to kind also requires a few other changes
to make sure kind's upper bits (which include STRTOG_Neg) are masked off
when attempting to only examine the lower bits' value. Furthermore, this
patch also makes exactly one change in gdtoa, which appears to be needed
to fix rounding issues with FE_TOWARDZERO (this seems like a gdtoa bug).
The patch also adds a few tests for these issues, along with also taking
the opportunity to clean up some of the previous tests to do the asserts
in the right order (i.e. with the first argument as the expected result,
and the second one being used as the value that it is compared against).
Before this commit, cosmopolitan had some issues with handling arguments
of 0 and signs, such as returning an incorrect sign when the input value
== -0.0, and incorrectly handling ndigit == 0 on fcvt (ndigit determines
the amount of digits *after* the radix character on fcvt, thus the parts
before it still must be outputted before fcvt's job is completely done).
This patch fixes these issues, and adds tests with corresponding inputs.
This change introduces comsoaudio v1. We're using a new strategy when it
comes to dynamic linking of dso files and building miniaudio device code
which I think will be fast and stable in the long run. You now have your
choice of reading/writing to the internal ring buffer abstraction or you
can specify a device-driven callback function instead. It's now possible
to not open the microphone when you don't need it since touching the mic
causes security popups to happen. The DLL is now built statically, so it
only needs to depend on kernel32. Our NES terminal emulator now uses the
cosmoaudio library and is confirmed to be working on Windows, Mac, Linux
The C Standard specifies that, when a conversion specification specifies
a conversion specifier of n, the type of the passed pointer is specified
by the length modifier (if any), i.e. that e.g. the argument for %hhn is
of type signed char *, but Cosmopolitan currently does not handle this -
instead always simply assuming that the pointer always points to an int.
This patch implements, and tests, length modifiers with the n conversion
specifier, with the tests testing all of the available length modifiers.
This change gets printvideo working on aarch64. Performance improvements
have been introduced for magikarp decimation on aarch64. The last of the
old portable x86 intrinsics library is gone, but it still lives in Blink
The worst issue I had with consts.sh for clock_gettime is how it defined
too many clocks. So I looked into these clocks all day to figure out how
how they overlap in functionality. I discovered counter-intuitive things
such as how CLOCK_MONOTONIC should be CLOCK_UPTIME on MacOS and BSD, and
that CLOCK_BOOTTIME should be CLOCK_MONOTONIC on MacOS / BSD. Windows 10
also has some incredible new APIs, that let us simplify clock_gettime().
- Linux CLOCK_REALTIME -> GetSystemTimePreciseAsFileTime()
- Linux CLOCK_MONOTONIC -> QueryUnbiasedInterruptTimePrecise()
- Linux CLOCK_MONOTONIC_RAW -> QueryUnbiasedInterruptTimePrecise()
- Linux CLOCK_REALTIME_COARSE -> GetSystemTimeAsFileTime()
- Linux CLOCK_MONOTONIC_COARSE -> QueryUnbiasedInterruptTime()
- Linux CLOCK_BOOTTIME -> QueryInterruptTimePrecise()
Documentation on the clock crew has been added to clock_gettime() in the
docstring and in redbean's documentation too. You can read that to learn
interesting facts about eight essential clocks that survived this purge.
This is original research you will not find on Google, OpenAI, or Claude
I've tested this change by porting *NSYNC to become fully clock agnostic
since it has extensive tests for spotting irregularities in time. I have
also included these tests in the default build so they no longer need to
be run manually. Both CLOCK_REALTIME and CLOCK_MONOTONIC are good across
the entire amd64 and arm64 test fleets.
POSIX specifies the C conversion specifier as being "equivalent to %lc",
i.e. printf("%C", arg) is equivalent in behaviour to printf("%lc", arg).
This patch implements this conversion specifier, and adds a test for it,
alongside another test, which ensures that va_arg uses the correct size,
even though we set signbit to 63 in the code (which one might think will
result in the wrong size of argument being va_arg-ed, but having signbit
set to 63 is in fact what __fmt_stoa expects and is a requirement for it
properly formatting the wchar_t argument - this does not result in wrong
usage of va_arg because the implementation of the c conversion specifier
(which the implementation of the C conversion specifier fallsthrough to)
always calls va_arg with an argument type of int, to avoid the very same
bug occuring with %lc, as the l length modifier also sets signbit to 63)
This is one of the few POSIX APIs that was missing. It lets you choose a
monotonic clock for your condition variables. This might improve perf on
some platforms. It might also grant more flexibility with NTP configs. I
know Qt is one project that believes it needs this. To introduce this, I
needed to change some the *NSYNC APIs, to support passing a clock param.
There's also new benchmarks, demonstrating Cosmopolitan's supremacy over
many libc implementations when it comes to mutex performance. Cygwin has
an alarmingly bad pthread_mutex_t implementation. It is so bad that they
would have been significantly better off if they'd used naive spinlocks.
On Windows file system tools like `ls` would print errors when they find
things like WSL symlinks, which can't be read by WIN32. I don't know how
they got on my hard drive but this change ensures Cosmo will handle them
more gracefully. If a reparse point can't be followed, then fstatat will
return information about the link itself. If readlink encounters reparse
points that are WIN32 symlinks, then it'll log more helpful details when
using MODE=dbg (a.k.a. cosmocc -mdbg). Speaking of which, this change is
also going to help you troubleshoot locks; when you build your app using
the cosmocc -mdbg flag your --strace logs will now show lock acquisition
While we have always licked glibc and musl libc on gnu/systemd sadly the
Apple Libc implementation of pthread_mutex_t is better than ours. It may
be due to how the XNU kernel and M2 microprocessor are in league when it
comes to scheduling processes and the NSYNC behavior is being penalized.
We can solve this by leaning more heavily on ulock using Drepper's algo.
It's kind of ironic that Linux's official mutexes work terribly on Linux
but almost as good as Apple Libc if used on MacOS.
poll() and select() now delegate to ppoll() and pselect() for assurances
that both polyfill implementations are correct and well-tested. Poll now
polyfills XNU and BSD quirks re: the hanndling of POLLNVAL and the other
similar status flags. This change resolves a misunderstanding concerning
how select(exceptfds) is intended to map to POLPRI. We now use E2BIG for
bouncing requests that exceed the 64 handle limit on Windows. With pipes
and consoles on Windows our poll impl will now report POLLHUP correctly.
Issues with Windows path generation have been fixed. For example, it was
problematic on Windows to say: posix_spawn_file_actions_addchdir_np("/")
due to the need to un-UNC paths in some additional places. Calling fstat
on UNC style volume path handles will now work. posix_spawn now supports
simulating the opening of /dev/null and other special paths on Windows.
Cosmopolitan no longer defines epoll(). I think wepoll is a nice project
for using epoll() on Windows socket handles. However we need generalized
file descriptor support to make epoll() for Windows work well enough for
inclusion in a C library. It's also not worth having epoll() if we can't
get it to work on XNU and BSD OSes which provide different abstractions.
Even epoll() on Linux isn't that great of an abstraction since it's full
of footguns. Last time I tried to get it to be useful I had little luck.
Considering how long it took to get poll() and select() to be consistent
across platforms, we really have no business claiming to have epoll too.
While it'd be nice to have fully implemented, the only software that use
epoll() are event i/o libraries used by things like nodejs. Event i/o is
not the best paradigm for handling i/o; threads make so much more sense.
This change fixes an issue with all system calls ending with *at(), when
the caller passes `dirfd != AT_FDCWD` and an absolute path. It's because
the old code was turning paths like C:\bin\ls into \\C:\bin\ls\C:\bin\ls
after being converted from paths like /C/bin/ls. I noticed this when the
Emacs dired mode stopped working. It's unclear if it's a regression with
Cosmopolitan Libc or if this was introduced by the Emacs v29 upgrade. It
also impacted posix_spawn() for which a newly minted example now exists.
Cosmopolitan's printf-family functions will currently crash if one tries
formatting a floating point number with a larger precision (large enough
that gdtoa attempts to allocate memory to format the number) while under
memory pressure (i.e. when malloc fails) because gdtoa fails to check if
malloc fails.
The added tests (which would previously crash under cosmopolitan without
this patch) show how to reproduce the issue.
This patch fixes this, and adds the aforementioned tests.
This method is supposed to give equivalence iff two shared pointers both
own the same object, even if they point to different addresses. We can't
control the exact order of the control blocks in memory, so the test can
only check that this equivalence/non-equivalence relationship holds, and
this is in fact all that it should check.
I'd previously introduced a bunch of small wrappers around the class and
functions under test to avoid excessive cpp use, but we can achieve this
more expediently with simple using-declarations. This also cuts out some
over-specified tests (e.g. there's no reason a stateful deleter wouldn't
compile.)
Cosmopolitan's printf-family functions currently very poorly handle
being passed a long double infinity.
For instance, a program such as:
```cpp
#include <stdio.h>
int main()
{
printf("%f\n", 1.0 / 0.0);
printf("%Lf\n", 1.0L / 0.0L);
printf("%e\n", 1.0 / 0.0);
printf("%Le\n", 1.0L / 0.0L);
printf("%g\n", 1.0 / 0.0);
printf("%Lg\n", 1.0L / 0.0L);
}
```
will currently output the following:
```
inf
0.000000[followed by 32763 more zeros]
inf
N.aN0000e-32769
inf
N.aNe-32769
```
when the correct expected output would be:
```
inf
inf
inf
inf
inf
inf
```
This patch fixes this, and adds tests for the behavior.
- wcsstr() is now linearly complex
- strstr16() is now linearly complex
- strstr() is now vectorized on aarch64 (10x)
- strstr() now uses KMP on pathological cases
- memmem() is now vectorized on aarch64 (10x)
- memmem() now uses KMP on pathological cases
- Disable shared_ptr::owner_before until fixed
- Make iswlower(), iswupper() consistent with glibc
- Remove figure space from iswspace() implementation
- Include line and paragraph separator in iswcntrl()
- Use Musl wcwidth(), iswalpha(), iswpunct(), towlower(), towupper()
This change switches c++ exception handling from sjlj to standard dwarf.
It's needed because clang for aarch64 doesn't support sjlj. It turns out
that libunwind had a bare-metal configuration that made this easy to do.
This change gets the new experimental cosmocc -mclang flag in a state of
working so well that it can now be used to build all of llamafile and it
goes 3x faster in terms of build latency, without trading away any perf.
The int_fast16_t and int_fast32_t types are now always defined as 32-bit
in the interest of having more abi consistency between cosmocc -mgcc and
-mclang mode.
The server was originally written before I implemented support for POSIX
thread cancelation. We now use standard pthreads APIs instead of talking
directly to *NSYNC. This means we're no longer using *NSYNC notes, which
aren't as good as the POSIX thread cancelation support I added to *NSYNC
which was only made possible by making *NSYNC part of libc. I believe it
will solve a crash we observed recently with ipv4.games, courtesy of the
individual who goes by the hacker alias Lambro.
This was suppressed recently and it's the worst possible idea when doing
greenfield software development with C. I'm so sorry it slipped through.
If the C standards committee was smart they would change the standard so
that implicit int becomes implicit long. Then problems such as this will
never occur and we could even use traditional C safely if we wanted too.
C++ code compiles very slowly with cosmocc, possibly because we're using
LLVM LIBCXX with GCC, and LLVM doesn't work as hard to make GCC go fast.
Therefore, it should be possible, to ask cosmocc to favor Clang over GCC
under the hood. On llamafile, my intention's to use this to make certain
files, e.g. llama.cpp/common.cpp, go from taking 17 seconds to 5 seconds
This new -mclang flag isn't ready for production yet since there's still
the question of how to get Clang to generate SJLJ exception code. If you
use this, then it's recommended you also pass -fno-exceptions.
The tradeoff is we're adding a 121mb binary to the cosmocc distribution.
There are no plans as of yet to fully migrate to Clang since GCC is very
good and has always treated us well.
This is believed to fix a crash, that's possible in nsync_waiter_free_()
when you call pthread_cond_timedwait(), or nsync_cv_wait_with_deadline()
where an assertion can fail. Thanks ipv4.games for helping me find this!
This change implements the compiler runtime for ARM v8.1 ISE atomics and
gets rid of the mandatory -mno-outline-atomics flag. It can dramatically
speed things up, on newer ARM CPUs, as indicated by the changed lines in
test/libc/thread/footek_test.c. In llamafile dispatching on hwcap atomic
also shaved microseconds off synchronization barriers.
So far I haven't found any way to run native Arm64 code on Windows Arm64
without using MSVC. When I build a PE binary from scratch that should be
a valid Windows Arm64 program, the OS refuses to run it. Possibly due to
requiring additional content like XML manifests or relocation or control
flow integrity data that isn't normally required on x64. I've also tried
using VirtualAlloc2() to JIT an Arm64 native function, but VirtualAlloc2
always fails with invalid parameter. I tried using MSVC to create an ARM
DLL that my x64 emulated program can link at runtime, to pass a function
pointer with ARM code, but LoadLibrary() rejects ARM DLLs as invalid exe
The only option left, is likely to write a new program like ape/ape-m1.c
which can be compiled by MSVC to load and run an AARCH64 ELF executable.
The emulated x64 binary would detect emulation using IsWow64Process2 and
then drop the loader executable in a temporary folder, and re-launch the
original executable, using the Arm64 segments of the cosmocc fat binary.
It turns out sched_getcpu() didn't work on many platforms. So the system
call now has tests and is well documented. We now employ new workarounds
on platforms where it isn't supported in our malloc() implementation. It
was previously the case that malloc() was only scalable on Linux/Windows
for x86-64. Now the other platforms are scalable too.
Using https://nightly.link/ with GitHub actions artifacts you can have a
nightly build (but not a _release_ -- there's no releasing or
pre-releasing happening) of cosmocc.
Example URL if this PR were merged:
https://nightly.link/jart/cosmopolitan/workflows/nightly-cosmocc/master/cosmocc.zip
Or you can just download it directly from the GitHub "Actions"
https://github.com/jart/cosmopolitan/actions workflow summary page of a
particular run
example from my own fork:
could download by clicking on the artifact
or by using third-party service to provide a link for unauthenticated
requests (like wget or curl)
https://nightly.link/jcbhmr/cosmopolitan/workflows/tool-cosmocc-package-sh/master/cosmocc.zip
this would be useful for users who don't want to or can't figure out how
to build cosmocc themselves (like Windows) but still want to use a
nightly build since a fix hasn't been released as a release version yet.
this would also be a good way to test the release process but instead of
pushing the `cosmocc.zip` to _wherever it goes now_ you publish it as a
github actions artifact for the very few nightly bleeding edge users to
use & test.
you don't have to use https://nightly.link or recommend it or anything;
i just know its a cool way to wget or curl the URLs instead of
downloading it via your browser web UI. particularly useful for
remote/ssh/web-ide development.
This is a breaking change. It defines the new environment variable named
_COSMO_FDS_V2 which is used for inheriting non-stdio file descriptors on
execve() or posix_spawn(). No effort has been spent thus far integrating
with the older variable. If a new binary launches the older ones or vice
versa they'll only be able to pass stdin / stdout / stderr to each other
therefore it's important that you upgrade all your cosmo binaries if you
depend on this functionality. You'll be glad you did because inheritance
of file descriptors is more aligned with the POSIX standard than before.
This change ensures that if a file descriptor for an open disk file gets
shared by multiple processes within a process tree, then lseek() changes
will be visible across processes, and read() / write() are synchronized.
Note this only applies to Windows, because UNIX kernels already do this.
We now have implement all of Musl's localization code, the same way that
Musl implements localization. You may need setlocale(LC_ALL, "C.UTF-8"),
just in case anything stops working as expected.
This change makes a second pass, at fixing the errno issue with libcxx's
filesystem code. Previously, 89.01% of LLVM's test suite was passing and
now 98.59% of their tests pass. Best of all, it's now possible for Clang
to be built as a working APE binary that can to compile the Cosmopolitan
repository. Please note it has only been vetted so far for some objects,
and more work would obviously need to be done in cosmo, to fix warnings.
Our cosmocc binaries are now built with GCC 14.1, using the Cosmo commit
efb3a34608 from yesterday.
GCC is now configured using --enable-analyzer so you can use -fanalyzer.
This change solves an issue where many threads attempting to spawn forks
at once would cause fork() performance to degrade with the thread count.
Things got real nasty on NetBSD, which slowed down the whole test fleet,
because there's no vfork() and we're forced to use fork() in our server.
threads count task
1 1062 fork+exit+wait
2 668 fork+exit+wait
4 66 fork+exit+wait
8 19 fork+exit+wait
16 22 fork+exit+wait
32 16 fork+exit+wait
Things are now much less bad on NetBSD, but not great, since it does not
have futexes; we rely on its semaphore file descriptors to do conditions
threads count task
1 1085 fork+exit+wait
2 842 fork+exit+wait
4 532 fork+exit+wait
8 400 fork+exit+wait
16 276 fork+exit+wait
32 66 fork+exit+wait
With OpenBSD which also lacks vfork(), things were just as bad as NetBSD
threads count task
1 584 fork+exit+wait
2 687 fork+exit+wait
4 206 fork+exit+wait
8 24 fork+exit+wait
16 33 fork+exit+wait
32 26 fork+exit+wait
But since OpenBSD has futexes fork() works terrifically thanks to *NSYNC
threads count task
1 525 fork+exit+wait
2 580 fork+exit+wait
4 451 fork+exit+wait
8 479 fork+exit+wait
16 408 fork+exit+wait
32 373 fork+exit+wait
This issue would most likely only manifest itself, when pthread_atfork()
callers manage to slip a spin lock into the outermost position of fork's
list of locks. Since fork() is very slow, a spin lock can be devastating
Needless to say vfork() rules and anyone who says differently is kidding
themselves. Look at what a FreeBSD 14.1 virtual machine with equal specs
can do over the course of three hundred milliseconds.
threads count task
1 2559 vfork+exit+wait
2 5389 vfork+exit+wait
4 34933 vfork+exit+wait
8 43273 vfork+exit+wait
16 49648 vfork+exit+wait
32 40247 vfork+exit+wait
So it's a shame that so few OSes support vfork(). It creates an unsavory
situation, where someone wanting to build a server that spawns processes
would be better served to not use threads and favor a multiprocess model
The cosmocc.zip toolchain will now include four builds of the libcosmo.a
runtime libraries. You can pass the -mdbg flag if you want to debug your
cosmopolitan runtime. You can pass the -moptlinux flag if you don't want
windows code lurking in your binary. See tool/cosmocc/README.md for more
details on how these flags may be used and their important implications.
GCC 13+ changed its policies to be very aggressive about breaking builds
that have anything resembling K&R C. I very strongly disagree with these
decisions. Users who think their compiler should should also be a linter
are perfectly welcome to opt-in to -Wimplicit-int.
It's been thirteen years and C++ still hasn't implemented this wonderful
simple builtin keyword. In C++23 a solution was provided for making this
work in C++ which is libcxx's stdatomic.h. Including that header schleps
in literally 253 unique header files!! Many of the header files it needs
are libc header files like pthread.h where we need to have the _Atomic()
keyword, but since <atomic> depends on pthreads we can't have it include
the <stdatomic.h> header that defines _Atomic for C++ users, and instead
we simply make the type non-atomic, hoping and praying only C code shall
use those internal data structures. This just shows how STL clowns can't
be trusted to define the innermost primitives of a language. They should
instead be focusing on being the best at algorithms and data structures.
- NetBSD should now have faster synchronization
- POSIX barriers may now be shared across processes
- An edge case with memory map tracking has been fixed
- Grand Central Dispatch is no longer used on MacOS ARM64
- POSIX mutexes in normal mode now use futexes across processes
- CLOCK_THREAD_CPUTIME_ID
- CLOCK_PROCESS_CPUTIME_ID
Cosmo now supports the above constants universally across supported OSes
therefore it's now safe to let programs detect their presence w/ #ifdefs
Cosmopolitan Libc once called this important function although somewhere
along the way, possibly in a refactoring, it got removed and __tls_alloc
has always been zero ever since.
The memory leak detector was crashing. When using gc() you shouldn't use
the CheckForMemoryLeaks() function from inside the same function, due to
how it runs the atexit handlers.
Cosmopolitan now supports mremap(), which is only supported on Linux and
NetBSD. First, it allows memory mappings to be relocated without copying
them; this can dramatically speed up data structures like std::vector if
the array size grows larger than 256kb. The mremap() system call is also
10x faster than munmap() when shrinking large memory mappings.
There's now two functions, getpagesize() and getgransize() which help to
write portable code that uses mmap(MAP_FIXED). Alternative sysconf() may
be called with our new _SC_GRANSIZE. The madvise() system call now has a
better wrapper with improved documentation.
It's now possible to create thousands of thousands of sparse independent
memory mappings, without any slowdown. The memory manager is better with
tracking memory protection now, particularly on Windows in a precise way
that can be restored during fork(). You now have the highest quality mem
manager possible. It's even better than some OSes like XNU, where mmap()
is implemented as an O(n) operation which means sadly things aren't much
improved over there. With this change the llamafile HTTP server endpoint
at /tokenize with a prompt of 50 tokens is now able to handle 2.6m r/sec
This change reduces o/tiny/examples/life from 44kb to 24kb in size since
it avoids linking mmap() when unnecessary. This is important, to helping
cosmo not completely lose touch with its roots.
The Cosmopolitan Compiler Collection now includes the following programs
- `ar.ape` is a faster alternative to `ar rcsD` for creating determistic
static archives. It's ~10x faster than GNU because it isn't quadratic.
It'll even outperform LLVM ar by 2x, thanks to writev/copy_file_range.
- `sha256sum.ape` is a faster alternative to the `sha256sum` command. It
goes 2x faster since it leverages vectorized assembly implementations.
- `resymbol` is a brand new program we invented, like objcopy, that lets
you rename all the global symbols in a .o file to have a new suffix or
prefix. In the future, this will be used by cosmocc automatically when
building -O3 math kernels, that need to be vectorized for all hardware
- `gzip.ape` is a faster version of the `gzip` command, that is included
by most Linux distros. It gains better performance using Chromium Zlib
which, once again, includes highly optimized assembly, that Mark Adler
won't merge into the official MS-DOS compatible zlib codebase.
- `cocmd` is the cosmopolitan shell. It can function as a faster `sh -c`
alternative than bash and dash as the `SHELL = /opt/cosmocc/bin/cocmd`
at the top of your Makefile. Please note you should be using the cosmo
fork of GNU make (already included), since normal make won't recognize
this as a bourne-compatible shell and remove the execve() optimization
which makes things slower. In some ways that's true. This doesn't have
a complete POSIX shell implementation. However it's enough for cosmo's
mono repo. It also implements faster behaviors in some respects.
The following programs are also introduced, which aren't as interesting.
The main reason why they're here is so Cosmopolitan's mono repo shall be
able to remove build/bootstrap/ in future editions. That way we can keep
build utilities better up to date, without bloating the git history much
- `chmod.ape` for hermeticity
- `cp.ape` for hermeticity
- `echo.ape` for hermeticity
- `objbincopy` is an objcopy-like tool that's used to build ape loader
- `package.ape` is used for strict dependency checking of object graph
- `rm.ape` for hermeticity
- `touch.ape` for hermeticity
Thanks to @aj47 (techfren.net) the new Cosmo memory manager is confirmed
to be working on Android!! The only issue turned out to be forgetting to
update the program address in the linker script. We now know w/ absolute
certainty that APE binaries as complex as llamafile, now work correctly.
- Ensure SIGTHR isn't blocked in newly created threads
- Use TIB rather than thread_local for thread atexits
- Make POSIX thread keys atomic within thread
- Don't bother logging prctl() to --strace
- Log thread destructor names to --strace
The way to use double linked lists, is to remove all the things you want
to work on, insert them into a new list on the stack. Then once you have
all the work items, you release the lock, do your work, and then lock it
again, to add the shelled out items back to a global freelist.
This fixes a regression in mmap(MAP_FIXED) on Windows caused by a recent
revision. This change also fixes ZipOS so it no longer needs a MAP_FIXED
mapping to open files from the PKZIP store. The memory mapping mutex was
implemented incorrectly earlier which meant that ftrace and strace could
cause cause crashes. This lock and other recursive mutexes are rewritten
so that it should be provable that recursive mutexes in cosmopolitan are
asynchronous signal safe.
This change introduces accumulate, addressof, advance, all_of, distance,
array, enable_if, allocator_traits, back_inserter, bad_alloc, is_signed,
any_of, copy, exception, fill, fill_n, is_same, is_same_v, out_of_range,
lexicographical_compare, is_integral, uninitialized_fill_n, is_unsigned,
numeric_limits, uninitialized_fill, iterator_traits, move_backward, min,
max, iterator_tag, move_iterator, reverse_iterator, uninitialized_move_n
This change experiments with rewriting the ctl::vector class to make the
CTL design more similar to the STL. So far it has not slowed things down
to have 42 #include lines rather than 2, since it's still almost nothing
compared to LLVM's code. In fact the closer we can flirt with being just
like libcxx, the better chance we might have of discovering exactly what
makes it so slow to compile. It would be an enormous discovery if we can
find one simple trick to solving the issue there instead.
This also fixes a bug in `ctl::string(const string &s)` when `s` is big.
We now have a C++ red-black tree implementation that implements standard
template library compatible APIs while compiling 10x faster than libcxx.
It's not as beautiful as the red-black tree implementation in Plinko but
this will get the job done and the test proves it upholds all invariants
This change also restores CheckForMemoryLeaks() support and fixes a real
actual bug I discovered with Doug Lea's dlmalloc_inspect_all() function.
It hasn't been helpful enough to be justify the maintenance burden. What
actually does help is mprotect(), kprintf(), --ftrace and --strace which
can always be counted upon to work correctly. We aren't losing much with
this change. Support for ASAN on AARCH64 was never implemented. Applying
ASAN to the core libc runtimes was disabled many months ago. If there is
some way to have an ASAN runtime for user programs that is less invasive
we can potentially consider reintroducing support. But now is premature.
Actually Portable Executable now supports Android. Cosmo's old mmap code
required a 47 bit address space. The new implementation is very agnostic
and supports both smaller address spaces (e.g. embedded) and even modern
56-bit PML5T paging for x86 which finally came true on Zen4 Threadripper
Cosmopolitan no longer requires UNIX systems to observe the Windows 64kb
granularity; i.e. sysconf(_SC_PAGE_SIZE) will now report the host native
page size. This fixes a longstanding POSIX conformance issue, concerning
file mappings that overlap the end of file. Other aspects of conformance
have been improved too, such as the subtleties of address assignment and
and the various subtleties surrounding MAP_FIXED and MAP_FIXED_NOREPLACE
On Windows, mappings larger than 100 megabytes won't be broken down into
thousands of independent 64kb mappings. Support for MAP_STACK is removed
by this change; please use NewCosmoStack() instead.
Stack overflow avoidance is now being implemented using the POSIX thread
APIs. Please use GetStackBottom() and GetStackAddr(), instead of the old
error-prone GetStackAddr() and HaveStackMemory() APIs which are removed.
Explicitly value-initializes the deleter, even though I have not found a
way to get the deleter to act like it’s been default-initialized in unit
tests so far.
Uses auto in reset. The static cast is apparently not needed (unless I’m
missing some case I didn’t think of.)
Implements the general move constructor - turns out that the reason this
didn’t work before was that default_delete<U> was not move constructible
from default_delete<T>.
Drop inline specifiers from functions defined entirely inside the struct
definition since they are implicitly inline.
* Cleans up reset to match spec
Remove the variants from the T[] specialization. Also follow the spec on
the order of operations in reset, which may matter if we are deleting an
object that has a reference to the unique_ptr that is being reset. (?)
* Tests Base/Derived reset.
* Adds some constexpr declarations.
* Adds default_delete specialization for T[].
* Makes parameters const.
The STL says that these should be replaceable by user code.
new.cc now defines only a few direct functions (including a free wrapper
that perplexingly is needed since g++ didn’t want to alias "free".) Now,
all of the operators are weak references to those functions.
Moves some isbig checks into string.h, enabling smarter optimizations to
be made on small strings. Also we no longer zero out our string prior to
calling the various constructors, buying back the performance we lost on
big strings when we made the small-string optimization. We further add a
little optimization to the big_string copy constructor: if the string is
using half or more of its capacity, then we don’t recompute capacity and
just take the old string’s. As well, the copy constructor always makes a
small string when it will fit, even if copied from a big string that got
truncated.
This also reworks the test to follow the idiom adopted elsewhere re stl,
and adds a helper function to tell if a string is small based on data().
* Add ctl utility.h
Implements forward, move, swap, and declval. This commit also adds a def
for nullptr_t to cxx.inc. We need it now because the CTL headers stopped
including anything from libc++, so we no longer get their basic types.
* Use ctl::swap in string
The STL spec says that swap is located in the string_view header anyawy.
Performance-wise this is a noop, but it’s slightly cleaner.
c.inc (AFAICT erroneously) defined _Atomic(t) as `volatile t *`, when it
should have just said `volatile t`, when __STDC_VERSION__ was too small.
This happens when we’re compiling C++, but in C++11, _Atomic is a define
supplied by the STL rather than a keyword supplied by the compiler. Wait
though, it gets better: in C++11, _Atomic hooks you into the morass that
is stdatomic.h, and ultimately refers everything back to std::atomic<T>.
The gory, horrifying details are in libcxx's __atomic/cxx_atomic_impl.h.
The tldr is that for our purposes it’s fine to just say volatile and use
the normal libc/intrin/atomic.h functions.
#1219 had an issue with noisy testing label. Investigated if there is a
syntax to make it more exclusive, but turns out there isn't one yet. So
let's remove an manually add it in as needed.
Fixes#1219.
The way unique_ptr is supposed to work is as a purely compile-time check
that your raw pointers are getting deleted when they go out of scope. It
should ideally emit the same exact machine code as if you were using raw
pointers with manual deletes.
Part of what this means is that under normal circumstances, a unique_ptr
shouldn’t take up more space than a raw pointer - in other words, sizeof
unique_ptr<T> should == sizeof(T*).
The present PR doesn’t bother with the specialization for array types. I
also left a couple other parts of the STL API unimplemented. I’d love to
see someone else implement these, or I’ll get to them at some point.
The mangled name of a C++ function will typically not vary by const-ness
of a by-value parameter; in other words, there is no meaning to a const-
qualified by-value parameter in a function prototype. However, the const
keyword _does_ matter at function _definition_ time, like it does with a
variable declared in the body. So for prototypes, we strip out const for
by-value parameters; but for definitions, we leave them alone.
At function definition (as opposed to prototype), we add const to values
in parameters by default, unless we’re going to mutate them.
This commit also changes a couple of const string_view& to be simply by-
value string_view. A string_view is only two words; it rarely ever makes
sense to pass one by reference if it’s not going to be mutated.
This essentially re-does the work of #875 on top of master.
This is what I did to check that Cosmo's Lua extensions still worked:
```
$ build/bootstrap/make MODE=aarch64 o/aarch64/third_party/lua/lua
$ ape o/aarch64/third_party/lua/lua
>: 10
10
>: 010
8
>: 0b10
2
>: string.byte("\e")
27
>: "Hello, %s" % {"world"}
Hello, world
>: "*" * 3
***
```
`luaL_traceback2` was used to show the stack trace with parameter
values; it's used in `LuaCallWithTrace`, which is used in Redbean to run
Lua code. You should be able to see the extended stack trace by running
something like this: `redbean -e "function a(b)c()end a(2)"` (with
"params" indicating the extended stack trace):
```
stack traceback:
[string "function a(b)c()end a(2)"]:1: in function 'a', params: b = 2;
[string "function a(b)c()end a(2)"]:1: in main chunk
```
@pkulchenko confirmed that I get the expected result with the updated
code.
This is what I did to check that Lua itself still worked:
```
$ cd third_party/lua/test/
$ ape ../../../o/aarch64/third_party/lua/lua all.lua
```
There's one test failure, in `files.lua`:
```
***** FILE 'files.lua'*****
testing i/o
../../../o/aarch64/third_party/lua/lua: files.lua:84: assertion failed!
stack traceback:
[C]: in function 'assert'
files.lua:84: in main chunk
(...tail calls...)
all.lua:195: in main chunk
[C]: in ?
.>>> closing state <<<
```
That isn't a result of these changes; the same test is failing in
master.
The failure is here:
```lua
if not _port then -- invalid seek
local status, msg, code = io.stdin:seek("set", 1000)
assert(not status and type(msg) == "string" and type(code) == "number")
end
```
The test expects a seek to offset 1,000 on stdin to fail — but it
doesn't. `status` ends up being the new offset rather than `nil`.
If I comment out that one test, the remaining tests succeed.
🚨 clang-format changes output per version!
This is with version 19.0.0. The modifications seem to be fixing the old
version’s errors - mainly involving omitted whitespace around binary ops
and inserted whitespace between goto labels and colons (if followed by a
curly brace.)
Also fixes a few mistakes made by e.g. someone (ahem) forgetting to pass
his ctl/string.h modifications through it.
We should add this to .git-blame-ignore-revs once we have its final hash
on master.
We now fully initialize a ctl::string’s memory, so that it is always set
to a well-defined value, thus making it always safe to memcpy out of it.
This incidentally makes our string::swap function legal, which it wasn’t
before. This also saves us a store in string::reserve.
Now that we have made both big_string and small_string POD, I believe it
is safe to elide the launder calls, and have done so, thus cleaning up a
lot of the blob-related code.
I also got rid of set_big_capacity and replaced it with a set_big_string
that leaves us in a well-defined state afterwards. This function also is
able to be somewhat simpler; rather than delicate bit-twiddling, it just
reaches straight into blob and rewrites it wholesale.
Overall, this shaves about 1–2ns off of most benchmarks, and adds 1ns to
only one of them - creating a string from a char *.
This replaces the STL <new> header. Mainly, it defines a global operator
new and operator delete, as well as the placement versions of these. The
placement versions are required to not get compile errors when trying to
write a placement new statement.
Each of these operators is defined with many, many different variants. A
glance at new.cc is recommended followed by a chaser of the Alexandrescu
talk "std::allocator is to Allocation as std::vector is to Vexation". We
must provide a global-namespace source-level definition of each operator
and it is illegal for any of them to be marked inline, so here we are.
The upshot is that we no longer need to include <new>, and our optional/
vector headers are self-contained.
`big_string` is not pod which means it needs to be properly constructed
and destroyed. Instead make it POD and destroy it manually in `string`
destructor.
Manually manage the lifetime of `value_` by using an anonymous
`union`. This fixes a bunch of double-frees and double-constructs.
Additionally move the `present_` flag last. When `T` has padding
`present_` will be placed there saving `alignof(T)` bytes from
`sizeof(optional<T>)`.
There were a few errors in how capacity and memory was being handled for
small strings. The capacity errors meant that small strings would become
big strings too soon, and the memory error introduced undefined behavior
that was caught by CheckMemoryLeaks in our test file but only sometimes.
The crucial change is in reserve: we only copy n bytes into p2, and then
we manually set the null terminator instead of expecting it to have been
there already. (E.g. it might not be there for an empty small string.)
We also fix one other doozy in append when we were exactly at the small-
to-big string boundary: we set the last byte (i.e., the remainder field)
to 0, then decremented it, giving us size_t max. Whoops. We boneheadedly
fix this by setting the 0 byte after we've fixed up the remainder, so it
is at worst a no-op.
Otherwise, capacity now works the same for small strings as it does with
big strings: it's the amount of space available including the null byte.
We test all of this with a new test that only gets included if our class
under test is not std::string (presumably meaning it's ctl::string.) The
test manually verifies that the small string optimization behaves how we
expect.
Since this test checks against std::string, we go ahead and include that
other header from the STL.
Also modifies the new test we introduced to also run on std::string, but
it just does the append without expecting anything about how its data is
stored. We also check that the string has the right value afterwards.
A small-string optimization is a way of reusing inline storage space for
sufficiently small strings, rather than allocating them on the heap. The
current approach takes after an old Facebook string class: it reuses the
highest-order byte for flags and small-string size, in such a way that a
maximally-sized small string will have its last byte zeroed, making it a
null terminator for the C string.
The only flag we have is in the highest-order bit, that says whether the
string is big (set) or small (cleared.) Most of the logic switches based
on the value of this bit; e.g. data() returns big()->p if it's set, else
small()->buf if it's cleared. For a small string, the capacity is always
fixed at sizeof(string) - 1 bytes; we store the length in the last byte,
but we store it as the number of remaining bytes of capacity, so that at
max size, the last byte will read zero and serve as our null terminator.
Morally speaking, our class's storage is a union over two POD C structs.
For now I gravitated towards a slightly more obtuse approach: the string
class itself contains a blob of the right size, and we alias that blob's
pointer for the two structs, taking some care not to run afoul of object
lifetime rules in C++. If anyone wants to improve on this, contributions
are welcome.
This commit also introduces the `ctl::__` namespace. It can't be legally
spelled by library users, and serves as our version of boost's "detail".
We introduced a string::swap function, and we now use that in operator=.
operator= now takes its argument by value, so we never need to check for
the case where the pointers are equal and can just swap the entire store
of the argument with our own, leaving the C++ destructor to free our old
storage afterwards.
There are probably still a few places where our capacity is slightly off
and we grow too fast, although there don't appear to be any where we are
too slow. I will leave these to be fixed in future changes.
description:Used to request enhancements for cosmopolitan
title:"Feature Request: "
labels:["enhancement"]
body:
- type:markdown
attributes:
value:|
[Please post your idea first in Discussion if there is not yet a consensus for this enhancement request. This will help to keep this issue tracker focused on enhancements that the community has agreed needs to be implemented.](https://github.com/jart/cosmopolitan/discussions/categories/ideas)
- type:checkboxes
id:prerequisites
attributes:
label:Prerequisites
description:Please confirm the following before submitting your enhancement request.
options:
- label:I am running the latest code. Mention the version if possible as well.
required:true
- label:I carefully followed the [README.md](https://github.com/jart/cosmopolitan/blob/master/README.md).
required:true
- label:I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
required:true
- label:I reviewed the [Discussions](https://github.com/jart/cosmopolitan/discussions), and have a new and useful enhancement to share.
required:true
- type:textarea
id:feature-description
attributes:
label:Feature Description
description:Please provide a detailed written description of what you were trying to do, and what you expected `cosmopolitan` to do as an enhancement.
placeholder:Detailed description of the enhancement
validations:
required:true
- type:textarea
id:motivation
attributes:
label:Motivation
description:Please provide a detailed written description of reasons why this feature is necessary and how it is useful to `cosmopolitan` users.
placeholder:Explanation of why this feature is needed and its benefits
validations:
required:true
- type:textarea
id:possible-implementation
attributes:
label:Possible Implementation
description:If you have an idea as to how it can be implemented, please write a detailed description. Feel free to give links to external sources or share visuals that might be helpful to understand the details better.
placeholder:Detailed description of potential implementation
Don't forget to check for any [duplicate research issue tickets](https://github.com/jart/cosmopolitan/issues?q=is%3Aopen+is%3Aissue+label%3A%22research+%F0%9F%94%AC%22)
- type:checkboxes
id:research-stage
attributes:
label:Research Stage
description:Track general state of this research ticket
options:
- label:Background Research (Let's try to avoid reinventing the wheel)
- label:Hypothesis Formed (How do you think this will work and it's effect?)
- label:Strategy / Implementation Forming
- label:Analysis of results
- label:Debrief / Documentation (So people in the future can learn from us)
- type:textarea
id:background
attributes:
label:Previous existing literature and research
description:Whats the current state of the art and whats the motivation for this research?
- type:textarea
id:hypothesis
attributes:
label:Hypothesis
description:How do you think this will work and it's effect?
- type:textarea
id:implementation
attributes:
label:Implementation
description:Got an approach? e.g. a PR ready to go?
- type:textarea
id:analysis
attributes:
label:Analysis
description:How does the proposed implementation behave?
- type:textarea
id:logs
attributes:
label:Relevant log output
description:Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
description:Used to track refactoring opportunities
title:"Refactor: "
labels:["refactor"]
body:
- type:markdown
attributes:
value:|
Don't forget to [check for existing refactor issue tickets](https://github.com/jart/cosmopolitan/issues?q=is%3Aopen+is%3Aissue+label%3Arefactoring) in case it's already covered.
Also you may want to check [Pull request refactor label as well](https://github.com/jart/cosmopolitan/pulls?q=is%3Aopen+is%3Apr+label%3Arefactoring) for duplicates too.
- type:textarea
id:background-description
attributes:
label:Background Description
description:Please provide a detailed written description of the pain points you are trying to solve.
placeholder:Detailed description behind your motivation to request refactor
validations:
required:true
- type:textarea
id:possible-approaches
attributes:
label:Possible Refactor Approaches
description:If you have some idea of possible approaches to solve this problem. You may want to make it a todo list.
placeholder:Your idea of possible refactoring opportunity/approaches
// vi: set et ft=cpp ts=4 sts=4 sw=4 fenc=utf-8 :vi
#ifndef CTL_IOS_BASE_H_
#define CTL_IOS_BASE_H_
namespacectl{
classios_base
{
public:
typedefsize_tstreamsize;
enumiostate
{
goodbit=0,
badbit=1,
failbit=2,
eofbit=4
};
enumfmtflags
{
boolalpha=1<<0,
dec=1<<1,
fixed=1<<2,
hex=1<<3,
internal=1<<4,
left=1<<5,
oct=1<<6,
right=1<<7,
scientific=1<<8,
showbase=1<<9,
showpoint=1<<10,
showpos=1<<11,
skipws=1<<12,
unitbuf=1<<13,
uppercase=1<<14,
adjustfield=left|right|internal,
basefield=dec|oct|hex,
floatfield=scientific|fixed
};
enumopenmode
{
app=1<<0,
binary=1<<1,
in=1<<2,
out=1<<3,
trunc=1<<4,
ate=1<<5
};
protected:
ios_base();
virtual~ios_base();
intstate_;
intflags_;
public:
fmtflagsflags()const;
fmtflagsflags(fmtflags);
fmtflagssetf(fmtflags);
fmtflagssetf(fmtflags,fmtflags);
voidunsetf(fmtflags);
iostaterdstate()const;
voidclear(int=goodbit);
voidsetstate(int);
boolgood()const;
booleof()const;
boolfail()const;
boolbad()const;
booloperator!()const;
explicitoperatorbool()const;
private:
ios_base(constios_base&)=delete;
ios_base&operator=(constios_base&)=delete;
};
}// namespace ctl
#endif // CTL_IOS_BASE_H_
Loaded 100 of 7329 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Justin Tunney