This patch integrates the majority of the mach-perftest-notebook project from the github project to the in-tree mozperftest package/tool. Certain portions of it are disabled in this integration (posting to iodide, and transform searching).
Differential Revision: https://phabricator.services.mozilla.com/D70134
--HG--
extra : moz-landing-system : lando
This commit removes `test_fix_stack_using_bpsyms.py`. That test can't easily be
modified to work with `fix_stacks.py` because it relies on internal
implementation details of `fix_stack_using_bpsym.py`. The unit testing done in
the `fix-stacks` repo provides test coverage that is as good or better.
Differential Revision: https://phabricator.services.mozilla.com/D66924
--HG--
extra : moz-landing-system : lando
This solves an edge case where tab completing a Python 2 command with global
arguments was using the wrong Python version.
Differential Revision: https://phabricator.services.mozilla.com/D56445
--HG--
extra : moz-landing-system : lando
This works for me and should allow trees to reopen.
Differential Revision: https://phabricator.services.mozilla.com/D70415
--HG--
extra : histedit_source : 529e1cb5cdb3030f145f44fc1ebf0ea9ae00aa8f
As noted by Jamie, accessible labels for richlistitem elements come from the
label elements that have value attributes within them.
In bug 1608202, we accidentally reused the same fluent message for the buttons
in about:downloads and the library download view (DownloadsViewUI.jsm) and
the context menuitems that do the same thing. This meant that
those menuitems gained a tooltip they shouldn't have, and the buttons gained
a label and accesskey they shouldn't have. The latter caused the
accessibility regression described in the bug.
This patch separates out the two usecases for the same string. I also checked
the other l10nIds used in DownloadsViewUI.jsm, and as far as I can tell this
is the only one that is reused in this way.
Differential Revision: https://phabricator.services.mozilla.com/D70219
--HG--
extra : moz-landing-system : lando
Since the test goes through all moz.build files disregarding DIRS and
the conditions that may disable directories, in some cases, moz.builds
can fail to be evaluated properly because of missing variables in
config.status. This time (because it's not the first), it's
LLVM_DLLTOOL.
After fixing that, it turns out many of the files/directories pointed to
by Files() directives were removed or moved.
While here, make the test script python3-ready.
Differential Revision: https://phabricator.services.mozilla.com/D70157
--HG--
extra : moz-landing-system : lando
Most (all?) mach xpcshell-test calls result in calling install_tests(None) while
most mach test calls for the same test/directory result in calling install_tests()
with a collection of test objects. Providing test objects allows install_tests()
to optimize which tests are installed, but there have been several recent bugs
that appear to be related to that optimization. Let's rely less on that optimization
and make things consistent between test/xpcshell-test. (There's a parallel
consideration for test vs mochitest.)
Differential Revision: https://phabricator.services.mozilla.com/D70052
--HG--
extra : moz-landing-system : lando
Update code in a few places to work with Python 3, especially with regard to how we invoke subprocesses with text output. Note that the Marionette test harness code needs to be updated so that text is written out in the correct encoding (see bug 1615748).
Differential Revision: https://phabricator.services.mozilla.com/D69012
--HG--
extra : moz-landing-system : lando
This also does a few remaining python 2 incompatible changes to
.configure files.
Differential Revision: https://phabricator.services.mozilla.com/D69538
--HG--
extra : moz-landing-system : lando
ConfigureSandbox._implied_options is a list of ReadOnlyNamespaces.
In python 3.5, ReadOnlyNamespaces end up with no guarantee in the order
of their __dict__. So when comparing
ReadOnlyNamespace(a=1, b=2)
and
ReadOnlyNamespace(a=3, b=4)
It's not guaranteed that the a's are compared before the b's.
In ConfigureSandbox._implied_options, some of those ReadOnlyNamespace
fields are SandboxDependsFunctions, which actually raise an error when
they're being compared, because we don't want that to happen in the
sandbox.
So when using python 3, configure would randomly fail when trying to
remove items from the ConfigureSandbox._implied_options list because
removing an item from a list scans the list to find the first element
that matches.
And ConfigureSandbox._implied_options needs to be ordered, which is why
it's currently a list.
So instead of removing by value, we create a new list with the remaining
values. But because the loop recurses, and needs the updated list, we
filter first.
Differential Revision: https://phabricator.services.mozilla.com/D69535
--HG--
extra : moz-landing-system : lando
When the build is stalled for some random reason, and mach executed a
python subcommand (this may or may not be limited to python 3, I'm not
sure, and it doesn't really matter since it's a problem on python 3,
which matters most), the subcommand may not have actually sent its last
bits of output before the stall because its output is a pipe and in that
case python uses buffered outputs.
Now, when your build is completely stalled and you ctrl+C, you end up
without these bits of output, and in some cases, those bits of output
can contain actual information, like... tracebacks.
A real life example of this is bug 1624670 when running mach build or
mach configure with the patches from bug 1627163 applied, and configure
stalls without printing out the ValueError message at all.
Differential Revision: https://phabricator.services.mozilla.com/D69925
--HG--
extra : moz-landing-system : lando
The effect of the deleted code has been to not install required head files
when those head files are not in the same directory as the test's manifest;
that seems wrong. I am slightly concerned about the comment justifying the
short-cut, but I cannot find any ill effect from removing the code, and doing
so allows test_attachments_downloader.js to run with either xpcshell-test or test.
Differential Revision: https://phabricator.services.mozilla.com/D70051
--HG--
extra : moz-landing-system : lando
Support for python 2 is completely dropped. It wouldn't be too much code
to support both python 2 and 3 but since configure is going to switch to
python 3 shortly after, it's not worth the effort.
Differential Revision: https://phabricator.services.mozilla.com/D69525
--HG--
extra : moz-landing-system : lando
The definition for reftest/crashtest variants of web-platform-tests in web-platform.yml has a trailing `s` which is technically incorrect.
This causes problems when querying ActiveData for the runtimes for these subsuites since the suite name recorded in ActiveData has the trailing s.
Changes:
- remove the trailing `s` from the definitions
Differential Revision: https://phabricator.services.mozilla.com/D68069
--HG--
extra : moz-landing-system : lando
This makes the --config-path option relative to topsrcdir rather then the current
directory.
Using a relative path is difficult for a developer running mach lint locally as
the command has to change as the working directory changes. A stable base path also
facilitates creating an alias in machrc.
Differential Revision: https://phabricator.services.mozilla.com/D68909
--HG--
extra : moz-landing-system : lando
This was designed to be used outside of the `unified build system` in order to keep
individual files syntax sane and to use the `compile_commands.json` outside of the
`unified build` environment.
Differential Revision: https://phabricator.services.mozilla.com/D68968
--HG--
extra : moz-landing-system : lando
`ProcessPoolExecutor` will naturally default to the number of CPUs on
the machine and will also handle edge cases on Windows.
Differential Revision: https://phabricator.services.mozilla.com/D68185
--HG--
extra : moz-landing-system : lando
time.clock() is deprecated since Python 3.3 and gone in Python 3.8.
Depends on D67773
Differential Revision: https://phabricator.services.mozilla.com/D67774
--HG--
extra : moz-landing-system : lando
Use finer granularity for some reftest/mochitest SCHEDULES.exclusive entries,
so that reftest-plain does not run when only crashtests are modified, and
vice versa; similarly, break up mochitest into mochitest/browser-chrome/chrome/
a11y. Use schedules-component instead of category.
Differential Revision: https://phabricator.services.mozilla.com/D60085
--HG--
extra : moz-landing-system : lando
In the process, fixed a few bugs:
- the template eg `{file_adds % "\n{file}"}` produced a leading blank line,
which led to everything being linted.
- 'd' was replaced with 'r' in diff_filters, but the replacement was discarded.
- as a result of the above, `hg status -d` was being used ("show only deleted (but tracked) files") and now it is `hg status -r` ("show only removed files"). I *think* this is what was intended?
Differential Revision: https://phabricator.services.mozilla.com/D66714
--HG--
extra : moz-landing-system : lando
Adds --config-paths parameter to mach mozlint where test configuration
files (.yml) can be located. The default path "tools/lint" is set automatically
and additional paths have preference over the default.
This allows Thunderbird to run mozlint-based tests in Taskcluster with a
different configuration.
Differential Revision: https://phabricator.services.mozilla.com/D67077
--HG--
extra : moz-landing-system : lando
While we validate user specified paths early and only consider the 'AM' (added
+ modified) diff filter from versioncontrol, it's still possible to get
non-existent files when using --outgoing. This situation can happen when you
have a commit series that first modifies a file in an earlier commit, and then
moves/deletes it in a later commit.
To get around this, just silently filter out any file paths that don't exist.
Differential Revision: https://phabricator.services.mozilla.com/D67411
--HG--
extra : moz-landing-system : lando
Dictionary iteration under Python 3 is in an inherently unpredictable order, and while we try to keep DEFINES ordered through the use of OrderedDicts, if at any point we populate DEFINES directly or indirectly while iterating through the contents of a non-ordered dictionary, the order of the DEFINES (and therefore the contents of the output Makefile) will be nondeterministic as well. This patch makes a number of changes to ensure that we only ever populate DEFINES in a deterministic fashion. (Note that in Python 3.7 and later, the built-in dict class actually has deterministic ordering, so these changes are technically only necessary until our minimum Python version becomes 3.7.)
Differential Revision: https://phabricator.services.mozilla.com/D66089
--HG--
extra : moz-landing-system : lando
LLVM's new pass manager has been in the works for several years and has better optimization (sometimes much better) than the legacy pass manager. I think it's in good enough shape for us to try at this point.
While we only really need the new pass manager for shippable builds, as a general principle I'd like to use it as much as possible, to help catch bugs for upstream. Therefore this patch enables the new pass manager by default for all clang builds, with the only exceptions being compilers older than version 9, and xcode clang where we can't trust the version number. There isn't a specific problem with older versions; I just don't want to sign up for the support cost of debugging people's local builds that may be fixed already.
I don't expect it to be necessary, but just in case, an opt-out is available via `ac_add_options --disable-new-pass-manager`.
Differential Revision: https://phabricator.services.mozilla.com/D66109
--HG--
extra : rebase_source : 91df800146700e4958b8e645ebbd3cf7b11a2f1e
extra : source : 2f5aba2e2c099a1df26e3444ccec2be0b4ff4613
LLVM's new pass manager has been in the works for several years and has better optimization (sometimes much better) than the legacy pass manager. I think it's in good enough shape for us to try at this point.
While we only really need the new pass manager for shippable builds, as a general principle I'd like to use it as much as possible, to help catch bugs for upstream. Therefore this patch enables the new pass manager by default for all clang builds, with the only exceptions being compilers older than version 9, and xcode clang where we can't trust the version number. There isn't a specific problem with older versions; I just don't want to sign up for the support cost of debugging people's local builds that may be fixed already.
I don't expect it to be necessary, but just in case, an opt-out is available via `ac_add_options --disable-new-pass-manager`.
Differential Revision: https://phabricator.services.mozilla.com/D66109
--HG--
extra : moz-landing-system : lando
Dictionary iteration under Python 3 is in an inherently unpredictable order, and while we try to keep DEFINES ordered through the use of OrderedDicts, if at any point we populate DEFINES directly or indirectly while iterating through the contents of a non-ordered dictionary, the order of the DEFINES (and therefore the contents of the output Makefile) will be nondeterministic as well. This patch makes a number of changes to ensure that we only ever populate DEFINES in a deterministic fashion. (Note that in Python 3.7 and later, the built-in dict class actually has deterministic ordering, so these changes are technically only necessary until our minimum Python version becomes 3.7.)
Differential Revision: https://phabricator.services.mozilla.com/D66089
--HG--
extra : moz-landing-system : lando
Note: while we can use time.monotonic in fetch-content, we can't in
mach artifact toolchain yet because it's still python2.
Differential Revision: https://phabricator.services.mozilla.com/D65690
--HG--
extra : moz-landing-system : lando
This was cargo culted from the autoconf equivalent, and while it makes a
command that does "$(PROG) foo" work because it becomes ": foo", that
may or may not actually be a desirable outcome.
OTOH, we do have some places where there are some "ifdef PROG" that are
just plain wrong when PROG is always actually set.
One place I do know that does check if the value is not ":" is for
OBJCOPY, which is still set from autoconf.
All in all, looking at all the check_prog(allow_missing=True) we have in
python configure, it doesn't seem anything is checking for ":", and that
doesn't seem like the right status quo.
Differential Revision: https://phabricator.services.mozilla.com/D65276
--HG--
extra : moz-landing-system : lando
On Linux and Mac, this makes `dmd.py` *much* faster when it is first run on a
DMD data file.
On Windows, this makes DMD actually usable locally. Previously the stacks
weren't fixed and so were rubbish.
Differential Revision: https://phabricator.services.mozilla.com/D57271
--HG--
extra : moz-landing-system : lando
Major version number bump for both mozlog and mozcrash + requirements update.
This is very much like the previous mozlog major bump - see bug 1591384.
This patch does not update testing/web-platform/tests/tools/wptrunner/requirements.txt;
I expect that will be automatically updated by the pyup.io bot once mozlog is re-released.
Differential Revision: https://phabricator.services.mozilla.com/D65135
--HG--
extra : moz-landing-system : lando
The preprocessor adds line markers in preprocessed files with line
numbers and file they came from. Bug 1528892 changed those markers
to be independent of the topobjdir and topsrcdir, by replacing them
with $OBJDIR and $SRCDIR, respectively.
This goes further, making these paths always use forward-slash, and
never backwards-slash, making the preprocessed files identical whether
the build occurred on Windows or Unix. (well, except when building
for different targets for target-specific sections)
Differential Revision: https://phabricator.services.mozilla.com/D64714
--HG--
extra : moz-landing-system : lando
Essentially:
FilterPath("docs").contains("docshell")
was returning True because "docshell" startswith "docs".
Differential Revision: https://phabricator.services.mozilla.com/D64661
--HG--
extra : moz-landing-system : lando
Windows programs run via Wine don't like Unix absolute paths (they look
like command line arguments), so we need to use relative paths.
Mingw already run fxc2 via wine, but for some reason it doesn't care
about the Unix absolute paths. genshaders does need some adjustements to
run properly with the real fxc.
Now, on actual Windows, because the temporary directory where
tempfile.NamedTemporaryFile creates files by default is not necessarily
on the same drive as where the command runs from, a relative path can't
be constructed. So we also force the temporary file to be created in the
current (obj) directory.
There is no similar concern for other files because we only go from
objdir to srcdir, and the build system already doesn't support both
being on a separate drive.
While here, flush stdout when the genshared script writes to it, so that
the messages are printed out immediately rather than randomly, later,
after output from subprocesses.
Differential Revision: https://phabricator.services.mozilla.com/D64294
--HG--
extra : moz-landing-system : lando
Makepkg has a commented config in /etc/makepkg.conf to set the path for packages to be built:
```
#PKGDEST=/home/packages
```
When you uncommit this line, packages will be built in the specified path, instead of cwd. This causes the bootstrap script to fail.
This commit will override the PKGDEST config so that package is built in cwd and could be located by the following lines of script.
Differential Revision: https://phabricator.services.mozilla.com/D64195
--HG--
extra : moz-landing-system : lando
The motivating change here was wpt-sync try pushes failing because
`libgraphitewasm.dylib` wasn't getting packaged in Mac artifact builds.
We add `libosclientcerts.dylib` because packaging was complaining about
that library as well.
Differential Revision: https://phabricator.services.mozilla.com/D63705
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
Install moz-phab using the correct command for the current operating system as
per moz-phab's documentation.
Differential Revision: https://phabricator.services.mozilla.com/D59139
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
This fixes str-vs-bytes in code paths that are only executed
with unapplied migrations, where we validate some parts of the
generated commits.
Differential Revision: https://phabricator.services.mozilla.com/D63049
--HG--
extra : moz-landing-system : lando
NodeJS 8.x is End-of-Lifed and is no longer receiving security fixes. 10.19.0 is now the oldest Long Term Support version of NodeJS, and it has just been released with several HTTP security fixes.
Differential Revision: https://phabricator.services.mozilla.com/D62781
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
This patch works around the removal of the platform.linux_distribution() function in Python 3.8 and later. It uses the in-tree third party distro package to provide linux distribution information to build system bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D60359
--HG--
extra : moz-landing-system : lando
This change adds `-ferror-limit=0` to the "command" lines in the
compile_commands.json file that's built by mach.
Differential Revision: https://phabricator.services.mozilla.com/D61392
--HG--
extra : moz-landing-system : lando
Changes:
Build string using the `format()` instead of `%` and relocate the `expandtabs` call to not trigger a AttributeError exception.
Differential Revision: https://phabricator.services.mozilla.com/D60939
--HG--
extra : moz-landing-system : lando
Changes:
Use compatibility layer provided by six for `iteritems` and `itervalues`.
Make `urlparse` import compatible with both 2/3.
Differential Revision: https://phabricator.services.mozilla.com/D60944
--HG--
extra : moz-landing-system : lando
