mirror of
https://github.com/mozilla/gecko-dev.git
synced 2024-10-19 08:15:31 +00:00
2f5add2222
As previous measurements have shown, creating/appending files on Windows/NTFS is slow because the CloseHandle() Win32 API takes 1-3ms to complete. This is apparently due to a fundamental issue with NTFS extents. A way to work around this slowness is to use multiple threads for I/O so file closing doesn't block execution as much. This commit updates the file copier to use a thread pool of 4 threads when processing file copies. Additional threads appear to have diminishing returns. On my i7-6700K, this reduces the time for processing the tests install manifest (24,572 files) on Windows from ~22.0s to ~12.5s in the best case. Using the thread pool globally resulted in a performance regression on Linux. Given the performance sensitivity of manifest copying, I thought it best to implement a slightly redundant non-Windows branch to preserve performance. For the record, that same machine running Linux is capable of processing nearly the same install manifest (24,616 files) in ~2.2s in the best case. MozReview-Commit-ID: B9LbKaOoO1u --HG-- extra : rebase_source : e9fee3861a70e1da9d18448f8435f4bd3e28c647 |
||
|---|---|---|
| .. | ||
| bitstring | ||
| blessings | ||
| compare-locales | ||
| configobj | ||
| eme | ||
| futures | ||
| jsmin | ||
| lldbutils | ||
| mach | ||
| mock-1.0.0 | ||
| mozboot | ||
| mozbuild | ||
| mozversioncontrol/mozversioncontrol | ||
| psutil | ||
| pyasn1 | ||
| pyasn1-modules | ||
| PyECC | ||
| pystache | ||
| pyyaml | ||
| redo | ||
| requests | ||
| rsa | ||
| slugid | ||
| virtualenv | ||
| which | ||
| mach_commands.py | ||
| moz.build | ||
| README | ||
This directory contains common Python code. The basic rule is that if Python code is cross-module (that's "module" in the Mozilla meaning - as in "module ownership") and is MPL-compatible, it should go here. What should not go here: * Python that is not MPL-compatible (see other-licenses/) * Python that has good reason to remain close to its "owning" (Mozilla) module (e.g. it is only being consumed from there). Historical information can be found at https://bugzilla.mozilla.org/show_bug.cgi?id=775243 ## pyyaml | pystache Used in taskcluster related mach commands to update download from github and remove .git and tests. Then run tests in taskcluster/tests/