Prior to this patch setting sandbox_vars['CFLAGS'] would cause those to
override any variables set by gyp via cflags_mozilla. In general
sanbox_vars are used to override values, and there's room for
a clearer interaction between moz.build and gyp, but discarding cflags
set in cflags_mozilla is almost certainly not the intended behavior here,
and this solves the immediate issue.
MozReview-Commit-ID: 1uCgXr5lagA
--HG--
extra : rebase_source : a330108f64d0e155d2e08206f2fe9a6c5eec8e59
It's not enough to only export the include folder for openssl, but for
linking the lib folder is also necessary. Also DEP_OPENSSL_INCLUDE
shouldn't be set directly, because it's set by Cargo itself based
on the output of openssl-sys's build script.
MozReview-Commit-ID: Ijbx7STgCXq
--HG--
extra : rebase_source : 9f93d20f974eb85e0e96fdfb017a402a263330ef
bz reported that mach output was slow under xterm on Mac. He tracked
this down to the terminal footer clearing until end of screen. It's
likely this is triggering a vsync or repaint or something, as he measured
each line to take 16ms to display.
Changing the clearing operation from "end of screen" to "end of line"
should be a cheaper operation and should speed up terminal writing.
This code has been historically sensitive to change. There's a
non-insignificant risk this will break a terminal configuration. I've
tested this with PuTTY, terminal.app on OS X (including under screen),
and iTerm2.app and it seems to "just work."
MozReview-Commit-ID: C2sYYboksj1
--HG--
extra : rebase_source : 0586084c6db8f6f9a126c44e1a44b54274f7cb9f
The former removed the /opt from the toolchain job names, and the latter
made optimize_task() return a single value.
--HG--
extra : rebase_source : 57beab6bf646c8202e3d83d2de3e547be62f39c3
The change to l10n packager from bug 780562 worked in practice because
no chrome category had exclusively manifest entries with flags, which
we're changing in this bug.
It turns out this was only due to a missing change in the patch for bug
780562.
--HG--
extra : rebase_source : 3c8c31c37d8fb48bb99b1758bcd8ef5f32c71fe0
The change to l10n packager from bug 780562 worked in practice because
no chrome category had exclusively manifest entries with flags, which
we're changing in this bug.
It turns out this was only due to a missing change in the patch for bug
780562.
--HG--
extra : rebase_source : 9f782e115f97063c97f165ed95eb4beeb72f86d0
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 100092c2f2ff609362a724fff60f46dd6e49c94e
extra : intermediate-source : d10f5ccd882b965fcad39914f7c3c930d1301a41
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : fea376f642e20e8c7723506fd4a73e3f8ac5d0e5
extra : intermediate-source : a872303fd08497bbde0e3b4cea09a88a4182810e
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
The JARManifests are relatively straightforward since they can be
converted into FinalTargetFiles by _consume_jar_manifest(). Since these
can contain RenamedSourcePaths, we no longer assert an error for those.
The main difference is that some JARManifests can result in
preprocessing .css files, which use a different marker than the default.
The two odd cases are bookmarks.html.in and buildconfig.html. The first
relies on an l10n define in a Makefile, while the latter needs CXXFLAGS.
The faster make backend works around these by defining
BOOKMARKS_INCLUDE_DIR to the en-US version, and skips the CXXFLAGS case
by defining BUILD_FASTER=1, so that is what the tup backend does for now
as well.
MozReview-Commit-ID: LlmNjQ2g40j
--HG--
extra : rebase_source : 2665a8115fac5329f2ddb49887b08607bcd4aa84
Some jar.mn files create symlinks with a different name than the file
they point to. The tup backend should support this by using the object's
target_basename as the destination filename.
MozReview-Commit-ID: IB4moXA7KWH
--HG--
extra : rebase_source : dfa85f369932514bb9aa029184f7b7292a9ae01c
Passing a file object intead of a string of the file's contents defers reading
the file to a place Python can free the resulting string earlier, reducing
peak memory consumption when packaging symbols and avoiding a MemoryError on
Windows 32 builders in automation.
MozReview-Commit-ID: H0R6BbjwhOu
--HG--
extra : rebase_source : 181b592ab503d0545f34d167d7c05428455b6671
We install the tooltool packages to ${HOME}/.mozbuild by default, and
${HOME} may have spaces on Windows. The arguments to the command-line
options need to be quoted appropriately.
DONTBUILD because NPOTB
We'll need this information to be able to locate mach later, since we
can't assume that mach is just several directories above the script
we're currently running.
If we're running bootstrap from inside a mozilla-central checkout, we'll
have all the necessary files that we need to install tooltool packages
via `mach artifact toolchain`. If we're running bootstrap from a
downloaded bootstrap.py, however, it's possible that we failed to clone
the repository. We need to account for that situation and provide
instructions on how to deal with it.
A consequence of these two mechanisms for bootstrap is that we'll have
to make sure we're invoking `mach artifact toolchain` correctly in both
cases, which we'll handle in subsequent commits.
Now that we're installing Stylo packages through `mach`, we need to have
`mach` available, which is only true after we've checked for a clone of
the tree. We move the block performing the installation in this commit
and then fixup resulting problems in future commits.
Using this command is more robust than our current method, and brings
several benefits, such as smart caching of the downloaded toolchain. We
change the clang package downloaded for Windows with this change, but
bindgen has been updated to work well with LLVM 5.0, so there should be
no problems.
Most complaints about compiler warnings being printed at the end of the
build (4abe611696ab - bug 1360764) center around the volume. And most
of the volume is due to 3rd party projects. Since work to suppress the
warnings in 3rd party projects has been slow, let's temporarily filter
out those warnings.
This commit adds a list of suppressed directories corresponding to
3rd party projects responsible for most compiler warnings. For
non-automation builds, we don't print the warnings. Instead, we
print a summary of how many warnings were suppressed.
Not all 3rd party projects were listed. The intent is to only list the
high volume ones so the warnings list is reasonable by default. I
believe the current implementation achieves that. But I wouldn't
be surprised if it requires refinement. Perfect is the enemy of good.
MozReview-Commit-ID: ExDRidsHROC
--HG--
extra : rebase_source : 49123193ea872a92208dc54a71b5f31208c10cd1
geckodriver is the Mozilla implementation of the WebDriver remote
control interface for Gecko, and provides an HTTPD proxy that
translates the WebDriver protocol to Marionette.
Building this as part of the Firefox build will allow us to run
WPT WebDriver tests to verify our implementation of Marionette and
geckodriver. It also makes it less painful to make changes across
projects.
This change will cause the geckodriver program to be built as part
of regular Firefox builds, except on macOS and Android, and when artifact
builds are enabled.
RUST_PROGRAMS in cross-compile environments cause the wrong linker to
be used. When this bug is fixed, we should be able to enable building
of geckodriver on macOS. This work is tracked in:
https://bugzilla.mozilla.org/show_bug.cgi?id=1329737
On Android, we may one to build a binary for the host system to use
(x86_64), instead of an ARM binary for the emulator.
MozReview-Commit-ID: FG5tmPv4iut
--HG--
extra : rebase_source : 091728fd2582458325689fc6e3d8b317428802d8
If we had an unsuccessful build, then duplicating the warnings at the
end of the compilation job would just be burying the actual error in a
blizzard of noise.
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : cd029cdbbcccc1d16f03d63a5f1fdf60be5db4fd
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 24db84974b54a714ba82dfad7ff68fd1a5bf656a
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.
--HG--
extra : rebase_source : ceb08da909121a2ac0a2cdaba7970e4594dde09f
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 19ec875917546abc147b234a815e1a64c204b92a
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 10d494fc910982c3e34f4744592edca906d3a85d
We're getting an intermittent failure running `hg manifest` in CI. I
have no clue why.
What I do know is that we now have the mozversioncontrol Python package
for containing utility code for interacting with version control. It is
slightly more robust and I'm willing to support it more than I am
check_utils.py.
This commit adds a new API to our abstract repository class to obtain the
files in the working directory by querying version control.
Since I suspect cwd was coming into play in automation, I've also
added a utility function to mozversioncontrol to attempt to find
a version control checkout from the current working directory. It
simply traces ancestor paths looking for a .hg or .git directory.
Finally, I've ported all callers of the now-deleted API to the new
one. The old code had some "../.." paths in it, meaning it only
worked when cwd was just right. Since we resolve the absolute path
to the checkout and store it on the repo object, I've updated the
code so it should work no matter what cwd is as long as a repo can
be found. I'm not 100% confident I found all consumers assuming cwd.
But it's a start.
I'm not 100% confident this will fix the intermittent issues in CI. But
at least we should get a better error message and at least we'll be
running less hacky code.
MozReview-Commit-ID: AmCraHXcTEX
--HG--
extra : rebase_source : 815ae369776577ad374333920fd645d412a55148
Windows repackaging for complete mar files will also need to pull a
different value out of the application.ini file, so this code should be
shareable.
MozReview-Commit-ID: CzCoNRYcBPX
--HG--
extra : rebase_source : a5e4b31ba876d811436a7d8d15462fa85f0762f8
The chmod permissions need to be in octal format to get the expected
permissions settings. This only seems to affect the output if we run the
repackaging command on Linux, but it should still be fixed.
MozReview-Commit-ID: to4v7dkSBl
--HG--
extra : rebase_source : 9d2289511ebd05aea8a4c6d37136f258e2463dee
Python can run these files with 'python -m 7z_exe_foo', but using
'import 7z_exe_foo' is not allowed because the module begins with a
number.
See also: https://stackoverflow.com/a/17487228
MozReview-Commit-ID: 97iDdXlZJ1a
--HG--
rename : python/mozbuild/mozbuild/action/7z_exe_archive.py => python/mozbuild/mozbuild/action/exe_7z_archive.py
rename : python/mozbuild/mozbuild/action/7z_exe_extract.py => python/mozbuild/mozbuild/action/exe_7z_extract.py
extra : rebase_source : 1941986a7e6e56305b742710c73b90a3f7b5226b
It makes more sense to have the repackage commands in a separate
directory, since Windows repackaging will add several new types.
MozReview-Commit-ID: 1wA7F7k4NXf
--HG--
rename : python/mozbuild/mozbuild/repackage.py => python/mozbuild/mozbuild/repackaging/dmg.py
extra : rebase_source : b4b4f2fd5c45900aaf125928c144c15cfa1e115e
This patch stops GENERATED_FILES from being run during export for artifact
builds and prevents EXPORTS from being processed because some of these will
depend on generated headers.
MozReview-Commit-ID: BHZBFbVHwPT
--HG--
extra : rebase_source : fe21e07c5bb8ef957a0006ec6a06eb833692ebaf
