Commit Graph

705 Commits

Author SHA1 Message Date
Chris AtLee
4a7ce514be Bug 1540276: Migrate authenticode signing to autograph r=Callek,mshal
Differential Revision: https://phabricator.services.mozilla.com/D47114

--HG--
extra : moz-landing-system : lando
2019-09-30 13:57:32 +00:00
Nick Alexander
6790ecb3a2 Bug 1550146 - Part 2: Make mach "build" run "stage-package && android assemble-app" for mobile/android. r=glandium
Android artifacts (GeckoView AARs, GeckoViewExample (and Fennec) APKs)
require native libraries (`libxul.so`) and an omnijar (`omni.ja`).
These are produced by `mach package` (really, the `stage-package`
target).  Engineers essentially never want a build without a package
for mobile/android.  This adds mobile/android-only tiers that run
`mach package` and then `mach android assemble-app`.  The latter
consumes `libxul.so` and `omni.ja` to produce _all the things_
relevant to GeckoView engineers.

Differential Revision: https://phabricator.services.mozilla.com/D41450

--HG--
extra : moz-landing-system : lando
2019-08-21 03:07:16 +00:00
Mike Hommey
d427c37271 Bug 1570798 - Restore relative path for MOZ_FETCHES_DIR. r=nalexander
Bug 1563711 changed MOZ_FETCHES_DIR to make the MOZ_ANDROID_FAT_AAR_*
environment variables absolute paths. Unfortunately, the replacement
relies on non-osx/windows tasks to run in docker-worker, which is not
necessarily true, and that makes the MOZ_FETCHES_DIR wrong in
non-osx/windows generic-worker tasks.  Apparently, that currently works,
but that's not guaranteed to stay this way.

The MOZ_ANDROID_FAT_AAR_* environment variables don't need to be
absolute paths, though. MOZ_FETCHES_DIR is normalized by run-task, and
MOZ_ANDROID_FAT_AAR_* can be set relative to that, which we do here.

Differential Revision: https://phabricator.services.mozilla.com/D40334
2019-08-03 07:08:39 +09:00
Johan Lorenzo
13648fa0df Bug 1563711 - Remove target.maven.zip r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D38175

--HG--
extra : moz-landing-system : lando
2019-07-30 15:35:12 +00:00
Mike Hommey
6098f22649 Bug 1565706 - Allow to run mach build rust without setting MOZ_RUST_TIER. r=froydnj
At the same time, make things a little more tangible in the tiers setup
in baseconfig.mk.

Differential Revision: https://phabricator.services.mozilla.com/D37942

--HG--
extra : moz-landing-system : lando
2019-07-15 23:50:31 +00:00
Mike Shal
26b533deae Bug 1557785 - Process install manifests in 3-tier Windows builds; r=firefox-build-system-reviewers,chmanchester
Windows 1-tier PGO builds only partially clobber between the
profile-generate and profile-use stages, so that exports/installed files
don't have to be reprocessed. Unfortunately we can't skip the install
manifests in 3-tier PGO because the profile-generate build happens on a
different machine, so we have to differentiate between 1-tier and 3-tier
PGO builds. A new variable, MOZ_1TIER_PGO, is used for this purpose.

Eventually this logic can be cleaned up in bug 1557788 once all PGO
builds use the 3-tier model.

Differential Revision: https://phabricator.services.mozilla.com/D34802

--HG--
extra : moz-landing-system : lando
2019-06-19 22:06:14 +00:00
Mike Hommey
6f6e8cbd72 Bug 1551639 - Always update buildid but avoid rebuilding libxul. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D33772

--HG--
extra : moz-landing-system : lando
2019-06-06 04:35:02 +00:00
Nick Alexander
5893476bdb Bug 1508976 - Produce a multi-architecture GeckoView "fat AAR". r=glandium
This follows the model set down for EME artifacts:

- a new tier is added that uses a new Python build action to fetch and
  artifacts
- the action unpacks the fetched artifacts and moves specific inputs
  into places expected by the build and packager
- in automation, MOZ_ARTIFACT_TASK* is used to ensure the artifacts
  come from the correct tasks

In this case, the artifact fetching is done entirely in a new Python
build action that internally uses `mach artifact install --job ...`.
The action also verifies that the fetched artifacts are compatible and
that we're not assembling a fat AAR that is nonsensical.  The specific
inputs are not used in the Fennec APK that is produced; they're only
used in the GeckoView AAR that is produced.

The artifact fetching itself required tweaking to fetch only
`target.maven.zip` artifacts and to not unpack them.

The specific inputs used are the native libraries (libs/$ARCH/*.so)
and the architecture-specific preference files ($ARCH/greprefs.js and
defaults/pref/$ARCH/geckoview-prefs.js).  None of these inputs are
impacted by l10n.

Differential Revision: https://phabricator.services.mozilla.com/D31572

--HG--
extra : moz-landing-system : lando
2019-05-30 15:22:09 +00:00
Nathan Froyd
2b3a15afe9 Bug 1542746 - use a static MOZ_PROFILE_ORDER_FILE; r=nalexander,chmanchester
We're planning on switching to IR-based profiling, so we can't use the
frontend-based instrumentation to collect the order in which functions
are executed...at least not during the build itself.  Performance tests
indicate that not having the order information decreases performance
significantly.  So we're going to check in static files for Win32 and
Win64 and use those to perform the ordering.  It's OK if these files are
slightly out of date; as of this writing, builds that generate and then
use these files complain that ~1/3 of the functions can't be found (!).
We're just trying to do something slightly smarter than whatever the
linker default is.

Differential Revision: https://phabricator.services.mozilla.com/D31132

--HG--
extra : moz-landing-system : lando
2019-05-24 01:54:59 +00:00
Mike Shal
e28f6db31f Bug 1537574 - Use mozbuild's backend-out-of-date logic for RecursiveMake; r=firefox-build-system-reviewers,chmanchester
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).

The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.

Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.

Differential Revision: https://phabricator.services.mozilla.com/D26262

--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
2019-04-10 22:14:27 +00:00
Nathan Froyd
a780e0f5a1 Bug 1541068 - add Rust compilation directories to GARBAGE_DIRS; r=nalexander
We add to `GARBAGE_DIRS` in the toplevel `Makefile.in` because all of
our Rust libraries share a single `CARGO_TARGET_DIR`, located in
topobjdir.

We add to `GARBAGE_DIRS` for Rust programs because Rust programs
currently do not share compilation artifacts with Rust libraries (as our
libraries are built with `panic=abort` and our programs are not, sharing
compilation artifacts between the two is a non-starter).

Differential Revision: https://phabricator.services.mozilla.com/D26762

--HG--
extra : moz-landing-system : lando
2019-04-09 18:59:19 +00:00
Andreea Pavel
a7f868ac25 Backed out changeset 95b3298fd2d4 (bug 1537574) for windows task timeouts a=backout
--HG--
rename : python/mozbuild/mozbuild/gen_test_backend.py => build/gen_test_backend.py
2019-04-08 22:23:03 +03:00
Mike Shal
96425af79a Bug 1537574 - Use mozbuild's backend-out-of-date logic for RecursiveMake; r=firefox-build-system-reviewers,chmanchester
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).

The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.

Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.

Differential Revision: https://phabricator.services.mozilla.com/D26262

--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
2019-04-08 16:37:56 +00:00
Mike Hommey
c7e738d69d Bug 1535501 - Allow to get full crashreporter symbols on artifact builds. r=mshal
Differential Revision: https://phabricator.services.mozilla.com/D23605

--HG--
extra : moz-landing-system : lando
2019-03-15 21:56:14 +00:00
Mike Hommey
e8b6f15845 Bug 1529194 - Allow to opt-in to widevine on aarch64 Windows. r=firefox-build-system-reviewers,chmanchester
This uses artifacts from a win32 build to get the necessary DLLs.

Differential Revision: https://phabricator.services.mozilla.com/D20597

--HG--
extra : moz-landing-system : lando
2019-03-05 08:41:04 +00:00
Mike Hommey
fa7300a70b Bug 1531680 - Don't disable LTO on standalone profile-generate builds. r=chmanchester
On one-go MOZ_PGO builds, it's generally not wanted to do LTO during the
profile-generate phase. And the build system doesn't really support
different build options between both phases in this case, so we relied
on MOZ_PROFILE_GENERATE to disable the LTO flags.

However, in standalone profile-generate builds, if --enable-lto is
passed explicitly, the build should respect that choice.

So instead of checking MOZ_PROFILE_GENERATE to disable the LTO flags,
we disable them when MOZ_LTO is not set, and we force it to be disabled
during the profile-generate phase of one-go MOZ_PGO builds.

Differential Revision: https://phabricator.services.mozilla.com/D21659

--HG--
extra : moz-landing-system : lando
2019-03-05 22:09:46 +00:00
Mike Hommey
d3af9d50b9 Bug 1529195 - Allow to opt-in to symbols/host-bin artifacts from the CLI. r=chmanchester
and to opt-out of test artifacts.

Depends on D20445

Differential Revision: https://phabricator.services.mozilla.com/D20446

--HG--
extra : moz-landing-system : lando
2019-02-21 22:53:43 +00:00
Mike Hommey
ef3ad686ee Bug 1512504 - Remove support for MSVC. r=froydnj
Consequently, this removes:
- MOZ_LIBPRIO, which is now always enabled.
- non_msvc_compiler, which is now always true.
- The cl.py wrapper, since it's not used anymore.
- CL_INCLUDES_PREFIX, which was only used for the cl.py wrapper.
- NONASCII, which was only there to ensure CL_INCLUDES_PREFIX still
  worked in non-ASCII cases.

This however keeps a large part of detecting and configuring for MSVC,
because we still do need it for at least headers, libraries, and midl.

Depends on D19614

Differential Revision: https://phabricator.services.mozilla.com/D19615

--HG--
extra : moz-landing-system : lando
2019-02-14 21:45:27 +00:00
Mike Shal
49fc6484f8 Bug 1507334 - Remove '10' argument to profileserver.py; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers
Support for running profileserver.py multiple times was originally added
in bug 663251 and subsequently removed in bug 672563. Since then we've
been copying around the argument to profileserver.py from mozconfigs in
buildbot-configs, to mozconfigs in mozilla-central, to client.mk, and
ultimately to Makefile.in. It no longer does anything and can be
removed.

Differential Revision: https://phabricator.services.mozilla.com/D15748

--HG--
extra : moz-landing-system : lando
2019-01-10 01:34:39 +00:00
Mike Hommey
e489736f60 Bug 1518703 - Remove R_386_PC32 check. r=froydnj,ted
The test is actually redundant with the TEXTREL check in
check_binary.py (R_386_PC32 relocations will only happen as text
relocations)

Differential Revision: https://phabricator.services.mozilla.com/D16002

--HG--
extra : moz-landing-system : lando
2019-01-10 00:07:53 +00:00
Chris Manchester
b101c5a8f3 Bug 1517532 - Extend artifact builds to find and download artifacts from pgo builds when MOZ_PGO is set. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D15988

--HG--
extra : moz-landing-system : lando
2019-01-08 23:03:27 +00:00
Chris Manchester
25e14c4f13 Bug 1507338 - Add a configure option for MOZ_PROFILE_GENERATE. r=firefox-build-system-reviewers,mshal
Differential Revision: https://phabricator.services.mozilla.com/D12999

--HG--
extra : moz-landing-system : lando
2019-01-03 16:16:38 +00:00
Chris Manchester
d69003537d Bug 1498031 - Merge code paths for running configure between Tup and Make based backends. r=firefox-build-system-reviewers,mshal
This addresses a related issue along the way: a build that results in running
configure would not update the value of self.config_environment (and therefore
self.substs) as seen from the build driver, so out of date values would have
been used.

The changes to Makefile.in and client.mk made exploit the assumption that by
he time anything in the Make build is running, config.status is up to date.
Users running "make" without the benefit of "mach" will need to manually run
configure when necessary in order to take this into account.

Differential Revision: https://phabricator.services.mozilla.com/D8473

--HG--
extra : moz-landing-system : lando
2018-10-16 22:21:36 +00:00
Ted Mielczarek
e11506c714 bug 1490325 - write sccache log directly to artifact directory, make logs slightly more verbose. r=mshal
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.

The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.

Differential Revision: https://phabricator.services.mozilla.com/D6187

--HG--
extra : moz-landing-system : lando
2018-09-18 20:02:17 +00:00
Gabriele Svelto
b48b3e4db0 Bug 1473821 - The MOZ_PGO_INSTRUMENTED environment variable is not used anymore so remove all remaining mentions of it r=gps
Differential Revision: https://phabricator.services.mozilla.com/D3219

--HG--
extra : moz-landing-system : lando
2018-08-17 16:40:10 +00:00
Mike Hommey
f50e1c662a Bug 1483779 - Enable PGO on clang-cl 32-bits builds. r=dmajor
--HG--
extra : source : 5fcbe08fb321a5076f56c380b554c63068efbb00
2018-08-16 16:44:36 +09:00
arthur.iakab
07aa46fd13 Backed out 2 changesets (bug 1479800, bug 1483779)for frequent cgx and arm64 failures
Backed out changeset 5fcbe08fb321 (bug 1483779)
Backed out changeset dccd5299c5ad (bug 1479800)
2018-08-17 09:50:30 +03:00
Mike Hommey
152366b300 Bug 1483779 - Enable PGO on clang-cl 32-bits builds. r=dmajor 2018-08-17 10:19:52 +09:00
Mike Hommey
6d4eb0a92e Bug 1464170 - Support PGO in clang builds on Linux. r=dmajor
This uses the same code path as for clang-cl builds.
2018-08-10 05:54:11 +09:00
David Major
45083c4934 Bug 1341525: Enable PGO in 64-bit clang-cl builds. r=froydnj 2018-08-01 17:59:23 -04:00
Andreea Pavel
11f1c09ffe Backed out changeset 7a3882f27aa3 (bug 1341525) for failing win Sa builds on a CLOSED TREE
--HG--
extra : amend_source : dbf7c208749c67b6bb8674eb5ba55c6514f89a2d
2018-08-02 02:26:13 +03:00
David Major
ebe3264f11 Bug 1341525: Enable PGO in 64-bit clang-cl builds. r=froydnj 2018-08-01 17:59:23 -04:00
Andreea Pavel
250fd9a027 Backed out changeset 2011dae40293 (bug 1341525) for failing win Sa builds 2018-08-02 01:33:30 +03:00
David Major
622dccd28f Bug 1341525: Enable PGO in 64-bit clang-cl builds. r=froydnj
--HG--
extra : rebase_source : 06038cb081f7b2b2789a8b867b03340bf103b05e
2018-08-01 17:59:23 -04:00
Chris AtLee
38cced543a Bug 1466537: Disable full crashsymbols by branch r=ted
Differential Revision: https://phabricator.services.mozilla.com/D1558

--HG--
extra : moz-landing-system : lando
2018-07-30 20:23:14 +00:00
Aaron Klotz
9e27dd8849 Bug 1473423: Allow test symbols to be included in full symbol archive in try builds; r=ted
This patch moves the logic for include/exclude/compress from the makefile into
symbols_archive.py, and adds an additional check to the full-archive case to
allow the inclusion of test symbols in try builds.

--HG--
extra : amend_source : 306419aae90a1935ced49fec89112df05142501c
2018-07-11 17:38:37 -06:00
Nathan Froyd
5c5a455e97 Bug 1444171 - use the order file during re-link with clang-cl; r=glandium
Now that we've generated an order file of the first N functions invoked
during startup, let's tell the linker about said functions so it can
cluster them appropriately.
2018-07-09 18:36:14 -04:00
Nathan Froyd
e8a82fdba9 Bug 1444171: Perform an actual clobber for profiledbuild with clang-cl; r=gps
clang-cl-style PGO is more akin to what we have for GCC rather than
MSVC, so we should do a full clobber for maybe_clobber_profiledbuild.
2018-07-09 18:26:09 -04:00
Chris Manchester
843faaf73a Bug 1470552 - Update Makefile.in to refer to stub file target for buildid.h and source-repo.h generated in backend.mk r=froydnj
MozReview-Commit-ID: CxLkQjX8veW

--HG--
extra : rebase_source : 6db24d50fbb73a7a7ac2e0e771bc0fb9c73a443d
2018-06-22 15:10:39 -07:00
Ciure Andrei
3b1168a508 Backed out 1 changesets (bug 1470552) for build bustages mozbuild/test/backend/test_build.py CLOSED TREE
Backed out changeset 567df8598cdb (bug 1470552)
2018-06-23 00:16:35 +03:00
Chris Manchester
cc743e536a Bug 1470552 - Update Makefile.in to refer to stub file target for buildid.h and source-repo.h generated in backend.mk r=froydnj
MozReview-Commit-ID: JLgCA23U8zg

--HG--
extra : rebase_source : e0365b3b03d40de785838d8ca44148979b39f255
2018-06-22 13:29:00 -07:00
Chris Manchester
636b4fc4c0 Bug 1461836 - Write out complete configure dependencies from configure for consumption by make and non-make backends. r=mshal
MozReview-Commit-ID: 792seCZ2rs1

--HG--
extra : rebase_source : e6450cd2947e2fd55085a2af469535421bc07bfb
2018-05-21 14:01:50 -07:00
Nathan Froyd
5873523189 Bug 1461383 - fix test_build.py for local builds; r=chmanchester,f=mshal
test_build.py fails on local builds with messages like:

make[4]: Entering directory '/tmp/tmp65BjCH'
make[4]: *** No rule to make target 'buildid.h'.  Stop.
make[4]: Leaving directory '/tmp/tmp65BjCH'
/home/froydnj/src/gecko-dev.git/config/faster/rules.mk:76: recipe for target '/tmp/tmp65BjCH/buildid.h' failed
make[3]: *** [/tmp/tmp65BjCH/buildid.h] Error 2
make[3]: *** Waiting for unfinished jobs....
make[4]: Entering directory '/tmp/tmp65BjCH'
make[4]: *** No rule to make target 'source-repo.h'.  Stop.
make[4]: Leaving directory '/tmp/tmp65BjCH'
/home/froydnj/src/gecko-dev.git/config/faster/rules.mk:76: recipe for target '/tmp/tmp65BjCH/source-repo.h' failed
make[3]: *** [/tmp/tmp65BjCH/source-repo.h] Error 2
make[3]: Leaving directory '/tmp/tmp65BjCH/faster'
Makefile:155: recipe for target 'faster' failed

The tests pass in automation, however, because automation always defines
the rules for buildid.h and source-repo.h in the toplevel Makefile.in.
For local builds, however, those rules are not defined to avoid build
churn.  Let's ensure that the necessary rules are defined during testing
as well.
2018-05-15 10:05:23 -04:00
Nathan Froyd
33635829ac Bug 1459721 - part 7 - remove dist_idl install manifest; r=chmanchester
We no longer need to install the IDL files to the objdir for
processing. \o/
2018-05-15 10:05:23 -04:00
Chris Manchester
efa96c69f0 Bug 1454771 - Move tup invocation from Makefile.in to mach. r=mshal
MozReview-Commit-ID: HkhK4oe93Vm

--HG--
extra : rebase_source : cbfc0b05deae9ca5b871952bfb771320f7d51d9e
2018-04-25 14:50:18 -07:00
Chris Manchester
d5e62a783b Bug 1451530 - Pass the current topobjdir to our Tup invocation. r=mshal
MozReview-Commit-ID: D6UNvxwfGTa

--HG--
extra : rebase_source : b3bb9b559862a18ea058c15e2b668afea458752f
2018-04-04 15:13:13 -07:00
Mike Shal
e43c6624d2 Bug 1387098 - enable tup quiet mode; r=froydnj
Tup can have a lot of unnecessary log lines, particularly on a fresh
build when it is scanning directories for the first time. Turning on
--quiet when MOZ_AUTOMATION is set will help trim down the log size
significantly.

MozReview-Commit-ID: AaRppurkXsA

--HG--
extra : rebase_source : dca18a0e3657853eacfc6cdbb935998cadb339ce
2018-02-23 14:27:58 -05:00
Chris Manchester
fcf16fe1e7 Bug 1445398 - Do not re-generate buildid.h for every Tup build. r=mshal
MozReview-Commit-ID: ErkTDOU8lYH

--HG--
extra : rebase_source : 085b0cb43b625fbb2691aa6bc788b62549927a02
2018-03-22 16:03:12 -07:00
Chris Manchester
7f9cffd5ea Bug 1298328 - Fix dependency between buildid.h and libxul, do not re-generate buildid.h for every developer build. r=ted
For automation builds, force buildid.h to be regenerated, and always
#include buildid.h in nsAppRunner.cpp. This will eliminate the mismatch at the
cost of re-linking libxul on every build.

For developer builds, always #include buildid.h in nsAppRunner.cpp, but do not
force buildid.h to be re-generated for incremental builds. This means we will
never have a mismatch between platform and application buildids, although
subsequent builds are no longer guaranteed to have a new buildid.

MozReview-Commit-ID: 19hvu8AQXgN

--HG--
extra : rebase_source : 9f4fd063f89b9b2165d5889fdaa8f83104c4411c
2018-03-09 13:49:55 -08:00
Gurzau Raul
70976c79b0 Backed out changeset 2d981ccf4c2e (bug 1298328) for failing X tests on /experiments/test/xpcshell/test_conditions.js on a CLOSED TREE 2018-03-09 01:53:26 +02:00