These imports make an out-of-tree build of SpiderMonkey depend on
the reftest module, which means SpiderMonkey implicitly depends on
layout/tools/reftest.
Firefox automation now uploads resource usage JSON files as
job artifacts (see bug 1272202). Now that the build system
writes the same data format and `mach resource-usage` can
read this data format, let's teach `mach resource-usage`
to load arbitrary URLs. This allows people to view system
resource usage for arbitrary jobs in automation.
Currently, you have to look at Treeherder to find the URL to
the build-resources.json artifact. Perhaps in the future
we can make finding the URL easier. Or we could integrate
source resource viewing into Treeherder itself (this is
probably preferred).
This commit continues the tradition of `mach resource-usage`
being a hacked up mess. Instead of loading the URL in
the browser, we download the URL from Python then serve it
from the HTTP server running as part of `mach resource-usage`.
This is somewhat horrible. But it was easiest to implement.
It also conveniently bypasses any cross origin request
restrictions the browser may impose. So it is useful.
MozReview-Commit-ID: IR1Cfs7SrRN
--HG--
extra : rebase_source : b33231d482ca891c1b3331f472009518bf57f8c3
We currently have our own system monitor serialization in
building.py. It predates as_dict() from mozsystemmonitor. Let's
use the "upstream" data format so we only have a single format
to consume.
This change required updating the in-tree resource viewer to
be compatible with the new data format.
This commit stops short of getting rid of the existing
data massaging code in building.py. Another day perhaps.
MozReview-Commit-ID: 1OJrSiyJjMX
--HG--
extra : rebase_source : 9782b2164d1735ed0872fe8c1637204d5b3b1313
For now, only the following two directories will be linted:
python/mozlint
tools/lint
New directories can be added by adding them to the 'include'
directive in tools/lint/flake8.lint. They all default to the
configuration specified in topsrcdir/.flake8. Subdirectories
can override this configuration by creating their own .flake8
file.
MozReview-Commit-ID: Eag48Lnkp3l
--HG--
extra : rebase_source : 6d98c9fef80055a48cc2622848aa04c3045b747e
The current algorithm for filtering down tests is too naive. For example, given the following
directory structured:
parent
- foo
- bar
- baz
And the following include/exclude directives:
include = ['foo']
exclude = ['foo/bar']
Then running ./mach lint parent and ./mach lint foo/baz should both lint all files in baz but
no files in bar. This provides a nice way to include/exclude directories, while allowing the
underlying linters to find appropriate files to lint *within* those directories.
tl;dr - Straight file paths (no globs) will be passed straight to the underlying linter as is.
While paths with globs will first be resolved to a list of matching file paths.
MozReview-Commit-ID: Eag48Lnkp3l
--HG--
extra : rebase_source : 18ce2231bc4198b1e811d39f9803f0d5e03d982e
So a few changes here:
- node_modules is downloaded using tooltool so that we dont need to rely on external infrastructure.
- We have a npm-shrinkwrap.json file that version locks all of our node packages.
- eslint, eslint-plugin-mozilla etc. are now all installed locally.
In reality this means that we don't hit the network and we don't force users into installing global packages.
./mach eslint --setup has also been improved. We install packages locally and display the path of the user's eslint binary (useful for configuring editors).
eslint-plugin-mozilla has been moved from testing/eslint-plugin-mozilla to /testing/eslint/eslint-plugin-mozilla.
The node_modules directory for eslint and other plugins is located in testing/eslint/.
MozReview-Commit-ID: 4SFSxzka6BS
--HG--
rename : testing/eslint-plugin-mozilla/LICENSE => testing/eslint/eslint-plugin-mozilla/LICENSE
rename : testing/eslint-plugin-mozilla/docs/balanced-listeners.rst => testing/eslint/eslint-plugin-mozilla/docs/balanced-listeners.rst
rename : testing/eslint-plugin-mozilla/docs/import-browserjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-browserjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-headjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-headjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/index.rst => testing/eslint/eslint-plugin-mozilla/docs/index.rst
rename : testing/eslint-plugin-mozilla/docs/mark-test-function-used.rst => testing/eslint/eslint-plugin-mozilla/docs/mark-test-function-used.rst
rename : testing/eslint-plugin-mozilla/docs/no-aArgs.rst => testing/eslint/eslint-plugin-mozilla/docs/no-aArgs.rst
rename : testing/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst => testing/eslint/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst
rename : testing/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst => testing/eslint/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst
rename : testing/eslint-plugin-mozilla/docs/var-only-at-top-level.rst => testing/eslint/eslint-plugin-mozilla/docs/var-only-at-top-level.rst
rename : testing/eslint-plugin-mozilla/lib/globals.js => testing/eslint/eslint-plugin-mozilla/lib/globals.js
rename : testing/eslint-plugin-mozilla/lib/helpers.js => testing/eslint/eslint-plugin-mozilla/lib/helpers.js
rename : testing/eslint-plugin-mozilla/lib/index.js => testing/eslint/eslint-plugin-mozilla/lib/index.js
rename : testing/eslint-plugin-mozilla/lib/processors/xbl-bindings.js => testing/eslint/eslint-plugin-mozilla/lib/processors/xbl-bindings.js
rename : testing/eslint-plugin-mozilla/lib/rules/.eslintrc => testing/eslint/eslint-plugin-mozilla/lib/rules/.eslintrc
rename : testing/eslint-plugin-mozilla/lib/rules/balanced-listeners.js => testing/eslint/eslint-plugin-mozilla/lib/rules/balanced-listeners.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js => testing/eslint/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-aArgs.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-aArgs.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js
rename : testing/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js => testing/eslint/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js
rename : testing/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js => testing/eslint/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js
rename : testing/eslint-plugin-mozilla/moz.build => testing/eslint/eslint-plugin-mozilla/moz.build
rename : testing/eslint-plugin-mozilla/package.json => testing/eslint/eslint-plugin-mozilla/package.json
extra : rebase_source : cf689f6cc170b9a22018c981a768f545f952e019
Currently a bug in python (https://bugs.python.org/issue8296) is preventing a KeyboardInterrupt from
reaching the parent process, meaning we can't kill the process with SIGINT. There is a workaround to
this bug, but instead I decided to ignore SIGINT in the parent process completely. Now, each child
process is responsible for handling SIGINT on its own. Since child processes should all shutdown
relatively quickly anyway, this effectively also ends the parent process.
The benefit of doing it this way is that each child process can return the results they have collected
to date. So when a developer hits Ctrl-C, they'll still see some (but not all) formatted lint output.
The downside is that a poorly implemented external linter could block the parent process from exiting
quickly, but if this happens we should just fix the linter.
MozReview-Commit-ID: 2tRJgtmoPYP
--HG--
extra : rebase_source : e72282bcc29c9b22690168dc973a817236cdf8ca
Firefox automation now uploads resource usage JSON files as
job artifacts (see bug 1272202). Now that the build system
writes the same data format and `mach resource-usage` can
read this data format, let's teach `mach resource-usage`
to load arbitrary URLs. This allows people to view system
resource usage for arbitrary jobs in automation.
Currently, you have to look at Treeherder to find the URL to
the build-resources.json artifact. Perhaps in the future
we can make finding the URL easier. Or we could integrate
source resource viewing into Treeherder itself (this is
probably preferred).
This commit continues the tradition of `mach resource-usage`
being a hacked up mess. Instead of loading the URL in
the browser, we download the URL from Python then serve it
from the HTTP server running as part of `mach resource-usage`.
This is somewhat horrible. But it was easiest to implement.
It also conveniently bypasses any cross origin request
restrictions the browser may impose. So it is useful.
MozReview-Commit-ID: IR1Cfs7SrRN
--HG--
extra : rebase_source : 91f5f807c19643ac4d1edb8f6652110813f7e53f
We currently have our own system monitor serialization in
building.py. It predates as_dict() from mozsystemmonitor. Let's
use the "upstream" data format so we only have a single format
to consume.
This change required updating the in-tree resource viewer to
be compatible with the new data format.
This commit stops short of getting rid of the existing
data massaging code in building.py. Another day perhaps.
MozReview-Commit-ID: 1OJrSiyJjMX
--HG--
extra : rebase_source : b7c7824b84110f118223dc483b03398855fe9965
Origins will be set for any caller of CommandLineHelper.add, but will only
be propagated if args are added to extra_args. This results in an incorrect
origin recorded for mozconfig injected arguments.
MozReview-Commit-ID: 9mJCaNHyd5C
Originally, the changes to FakeCompiler allowing overlays was meant to
be used for compiler target platform, but it turns out the
simplifications this allows on the compiler definitions themselves are
nice.
We use _pretty_path when specifying the targets of generated files, so
we need to use _pretty_path for the inputs as well. Otherwise make won't
know that they refer to the same file, and result in "No rule to make
target" errors.
MozReview-Commit-ID: JTdLFbkX1J0
Some generated files will depend on other generated files, but still
need to be in the export tier because they are C++ headers.
MozReview-Commit-ID: AFvp92lF0xy
Mozlint provides two main benefits:
1. A common system for defining lints across multiple languages
2. A common interface and result format for running them
This commit only adds the core library, it does not add any consumers of mozlint just yet.
MozReview-Commit-ID: CSQzq5del5k
--HG--
extra : rebase_source : b520b96177281a1b1770edf53a01cbc2196f494f
- enable debug artifact from a mozconfig file based on MOZ_DEBUG environment variable
- OSX debug artifact builds have 'mac64' instead of 'mac' into their file name
(fix debug artifact build download on OSX)
MozReview-Commit-ID: 7kAvsTfwaCb
--HG--
extra : transplant_source : %E6v%25%B79M%02%7E%A9%8B%FF%24%03%D1%BDo%AB%0F%B49
Because some older python 2.7 versions throw bogus errors when using the
exec statement as a function, use the function we added in bug 1264831
everywhere we use exec in the various configure tests. It doesn't take
much to trigger them, and the following changes ends up doing exactly
that.
Our current build system support for Rust compiles any Rust crate into a
so-called staticlib, which is a static library (.a file) that includes
the Rust runtime. That staticlib is then linked into libxul. For
supporting multiple crates, this approach breaks down, as linking
multiple copies of the Rust runtime is going to fail.
For supporting multiple crates, the approach taken here is to compile
each crate into a so-called rlib, which is essentially a staticlib
without the Rust runtime linked in. The build system takes note of
every crate destined for linking with libxul (treating them like static
libraries generated from C/C++ files), and generates a super-crate,
whimsically named "rul", that is compiled as a staticlib (so as to
include the Rust runtime) and then linked into libxul. Thus only one
copy of the Rust runtime is included, and the Rust compiler can take
care of any inter-crate dependencies.
This patch currently only supports Rust code in shared libraries, not in
binaries.
This accounts for default unittest and pytest output formatting,
in addition to mozunit.
MozReview-Commit-ID: 749CD0xQezX
--HG--
extra : rebase_source : 7a451c61d1ec41303b859b8fff4ec3dd2f84064c
The format provided to the build system by the manifest parser is highly redundant:
every test lists all variables for that test, and many tests use a large
support-files entry in DEFAULT that ends up in individual test objects. This
patch stores these DEFAULTS per-manifest rather than per-test to save disk
space, resulting in about a ~22mb smaller all-tests.json file. The
in-memory representation of tests is not changed by this patch, as the defaults
are again propagated to individual tests as all-tests.json is read by the test
resolver.
MozReview-Commit-ID: CEJaevfS5s7
imply_option has no effect when the resolved value is None, so the same
logic can be applied when checking for unknown implied options.
This allows to imply options that may not always exist (because they are
in a configure file that is optionally included).
Ideally, it would be better not to do this, but until we have something
better than optionally included configure files for
--disable-compile-environment, this is a necessary evil.
So far, everything was essentially executed at "declaration". This
made the sandbox code simpler, but to improve on the tooling around
python configure (for tests and introspection), we need to have more
flexibility, which executing everything at declaration doesn't give.
With this change, only @depends functions depending on --help, as
well as templates, are executed at the moment the moz.configure
files are included in the sandbox. The remainder is executed at the
end.
MozReview-Commit-ID: AQ3w2oCPQeN
--HG--
extra : topic : bar
extra : rebase_source : 2e0de1688fca11bd62c88bd66ef9cf9ad97206f8
extra : amend_source : 45c025594d645a9252989c8f13387de05e49d4dd
Whether it uses locale._parse_localename or nl_langinfo makes it have completely
different results in weird and/or widespread locale settings (LC_ALL=UTF-8 or
LC_ALL=C).
These config options can be defined in ~/.mozbuild/machrc or topsrcdir/machrc.
Aliases work similar to the identically named option in an hgrc.
For example:
[alias]
browser-test = mochitest -f browser
mochitest = mochitest -f plain
MozReview-Commit-ID: CnOocEslRUI
--HG--
extra : rebase_source : 2a6fa154aca7fea8f159ed840728951a37bc52ec
Runtime configs have been implemented for awhile, but disabled. This patch
enables configuration. Config files will be loaded in the following order
(later files override earlier ones):
1a. $MACHRC
1b. $MOZBUILD_STATE_PATH/machrc (if $MACHRC is unset)
2. topsrcdir/machrc
3. CLI via --settings
Note: .machrc may be used instead of machrc if desired.
MozReview-Commit-ID: IntONAZLGML
--HG--
extra : rebase_source : ff79b129eaea7cca5064d30fa6ddc76fceb9669b
This adds a |mach settings locale-gen| subcommand to automatically generate locale
specific documentation for settings. It also refactors |mach settings-create| to
|mach settings| and moves |mach settings| to |mach settings -l|. Finally it performs
some misc cleanup mostly related to locales.
MozReview-Commit-ID: 1VWLcb9ehAH
--HG--
extra : rebase_source : 8f580217123d79e66323ca4be948a3297ae4ced3
Some sections should support user-defined options. For example, in an [alias] section, the option names
are not well-defined, rather specified by the user. This patch allows user-defined option names for any
section that has a 'section.*' option defined. Even with 'section.*', option types are still well-defined.
MozReview-Commit-ID: L34W9v9Fy28
--HG--
extra : rebase_source : 9333f552edead9bf1cf464e28ef8fbbb9bed5597
Defining settings was a little complicated. First it required overriding a '_register_settings'
method, and then it required making N calls to a second 'register_setting' method within that.
This patch simplifies the process of defining settings by only requiring a
'config_settings' attribute. This attribute should be a list of tuples:
[
('<section>.<option>', '<type>', <default>, set(<choices)),
]
`default` and `choices` are optional. Alternatively, 'config_settings' can be a callable
that returns a list of tuples in the same format. This still allows for greater flexibility
for more advanced cases.
MozReview-Commit-ID: F4DTTNJdJsa
--HG--
extra : rebase_source : e3dd455ba559cd3992c9c1b3eaf021c9e0707cc1
Check if the INSIDE_EMACS environment variable is set and change the
log level to WARNING to not confuse the emacs/mi with logging messages.
MozReview-Commit-ID: 5AWZ6swGJsE
--HG--
extra : transplant_source : b%24%8Ff6%968%8A%02%E2%07%DD%C6Y9E%CB%7C.%E4
This requires a change to how we process test manifests in the build system:
now, whenever we see a support file mentioned in a manifest, we require that
file isn't already in that test's support files, but if we see a support file
that was already seen in some other test, the entry is ignored, but it is not
an error. As a result of this change, several duplicate support-files entries
needed to be removed.
MozReview-Commit-ID: G0juyxzcaB8
--HG--
rename : testing/mozbase/manifestparser/tests/test_default_skipif.py => testing/mozbase/manifestparser/tests/test_default_overrides.py
For example, say there is a command 'foo' that has a subcommand 'bar'. Prior to this, it was not
possible to run:
./mach foo
as its own independent command. The above would instead print the subcommand help for 'bar'.
MozReview-Commit-ID: JU4dXoxnCyu
--HG--
extra : rebase_source : bb15532ad39456b270071bc60d7b15e15af04e48
Our current build system support for Rust compiles any Rust crate into a
so-called staticlib, which is a static library (.a file) that includes
the Rust runtime. That staticlib is then linked into libxul. For
supporting multiple crates, this approach breaks down, as linking
multiple copies of the Rust runtime is going to fail.
For supporting multiple crates, the approach taken here is to compile
each crate into a so-called rlib, which is essentially a staticlib
without the Rust runtime linked in. The build system takes note of
every crate destined for linking with libxul (treating them like static
libraries generated from C/C++ files), and generates a super-crate,
whimsically named "rul", that is compiled as a staticlib (so as to
include the Rust runtime) and then linked into libxul. Thus only one
copy of the Rust runtime is included, and the Rust compiler can take
care of any inter-crate dependencies.
This patch currently only supports Rust code in shared libraries, not in
binaries. The handling for the rul crate is placed in the common
backend, with a special hook for derived backends to handle shared
library objects.
When reading config.log, with old-configure output, we may get non-ascii
strings, but that currently fails because we're using plain open() to
read it. So use encoded_open() instead (which does the same job for
other files in the same script).
Because the build system can be encapsulated in mach, python configure
can have a pipe as stdout/stderr, and in that case, sys.stdout/stderr
have an ascii encoding, failing to print out anything that doesn't
fit in ascii, consequently failing to print the things we've read from
config.log. So reopen stdout and stderr with the right encoding in
the configure output handler.
This moves test installation for test files out of the monolithic install
manifest for $objdir/_tests, and determines the test and support files
to install based on the object derived from all-tests.json. Additionally,
the files resulting from TEST_HARNESS_FILES are installed, as some tests
will depend on them.
As a result, the time to install tests when invoking the test runner will
scale with the number of tests requested to run rather than the entire set
of tests in the tree, resulting in significantly less overhead.
MozReview-Commit-ID: LeIrUVh1yD4
This extracts the logic from the emitter that handles support files in ini
manifests to a seperate function in testing.py, so that this logic can be
re-used to determine how to install all the files necessary to run a particular
test fon the corresponding object in all-tests.json.
MozReview-Commit-ID: GSEhEGm09IL
DONTBUILD NPOTB
If the target triple is noisy, let's just accept -arm and -i386
instead of being strict. We can be more strict when things have
settled.
MozReview-Commit-ID: FDNJ3TuY51d
--HG--
extra : rebase_source : 3db70abe6818fbe0360406f20de899867232df02
extra : amend_source : d324c229b1713adb844407078d6cc4d00d84b56d
A few notes:
* This doesn't accommodate general OMNIJAR_NAME definitions. The
current name (assets/omni.ja) is baked into the product in a few
places, and is very unlikely to change, so we just error out if this
isn't true.
* This makes the package-manifest.in file authoritative for what goes
into assets/, libs/, and the APK root. Previously,
package-manifest.in wrote into assets/ and libs/ but
upload-files-APK.mk also had a convoluted DIST_FILES filtering
process to work through before a file actually made it into the APK.
* This is intentional about repackaging. It simplifies the repackage
step rather than trying to make unpackage-then-repackage the same as
just package. I pretty much never get repackaging correct the first
time; this should help. (I've manually tested it.)
* The ALREADY_SZIPPED during repackaging is subsumed by the previous
check if UNPACKAGE is set. The custom linker expects stored, not
deflated, libraries, so there's some small legwork to accommodate
that in mozjar.
MozReview-Commit-ID: JvVtIUSX685
--HG--
extra : rebase_source : fd8a9cfe3dc364d23b1065995db599f99e676e38
The initial goal of templates was to provide a way to write shorter
constructs for some generic tasks during configure. But the limitations
of the sandbox and the properties of templates made them used for more
general functions.
Consequently, this led to templates having to be available from
anywhere, which, in turn, led to difficult to introspect constructs.
With bug 1257823, we've made almost everything use set_config and
similar functions from the global scope, but we don't enforce that
those primitives are only used at the global scope.
This change does that: it enforces that primitives are only used at
the global scope. Or in templates.
Now, since templates were used for other purposes than generic uses
of other primitives, we now allow non-template functions to be declared.
Those can be used everywhere, but don't have access to the sandbox
primitives.
Currently, we have @advanced, that gives the decorated functions access
to all the builtins and consequently, to the import statement.
That is a quite broad approach and doesn't allow to easily introspect
what functions are importing which modules.
This change introduces a new decorator that allows to import modules one
by one into the decorated functions.
Note: by the end of the change series, @advanced will be gone.
So far, we've been using the lowercase of the variable name, but it's
not enough for some special cases. Those special cases could do their
own business, but then, they'd have to duplicate 90% of check_prog,
which is less desirable.
While DummyFunction is descriptive of what the instances are (and they
can't even be called), the various uses of isintance(obj, DummyFunction)
are kind of confusing, especially when they are in moz.configure land
(and this bug is about to add another one).
