"Automation Error" is not sufficient to turn a mozharness job orange.
mochitest-plain failures of this type normally cause job failure by not
printing out the test suite summary. This patch uses the same technique
for geckoview tests: If a crash is detected, do not print a test summary,
so that mozharness will subsequently fail the job.
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 64aa804965d24bb90b103c00c692a2ac6859e408
This also adds a utility function for synthesizing native touch
events to Eventutils.js.
I did not add a test for searchbar because of intermittent issues
with showing the contextmenu (that are not reproducible manually).
I believe this is rather related to searchbar functionality than
my patches.
MozReview-Commit-ID: Dqm92Saosxz
--HG--
extra : rebase_source : e59df4f487f60cea137fbf8aea71a854a5706de9
This also adds a utility function for synthesizing native touch
events to Eventutils.js.
I did not add a test for searchbar because of intermittent issues
with showing the contextmenu (that are not reproducible manually).
I believe this is rather related to searchbar functionality than
my patches.
MozReview-Commit-ID: Dqm92Saosxz
--HG--
extra : rebase_source : d5c4333609b68773e62447bd3158cadfa89b803b
SpecialPowers.loadChromeScript() sends a script to the child process,
then creates a sandbox, and runs the script in that sandbox. There are
various sandboxOptions that can be passed when creating a sandbox, and
it would be nice to have that functionality for loadChromeScript.
I just need this for wantGlobalProperties, but I might as well make it
as general as possible. I'm not sure all of the types it can take can
actually be serialized across processes, but I guess that's okay.
MozReview-Commit-ID: GoJjXdjizFk
--HG--
extra : rebase_source : 9c2bc190dbf5a080978953cffd64205e8b816367
For a while now it has been making the content process sandbox less strict.
MozReview-Commit-ID: Am6fGzViaLk
--HG--
extra : rebase_source : 0bc037f205896c866559a7ab1f7e2c042c3142db
The browser-chrome test suite now detects and reports unhandled rejections of native Promises, in addition to those created by Promise.jsm. The whitelisting mechanism is updated to use primarily the PromiseTestUtils.expectUncaughtRejection function. Tests will fail if a rejection that is not whitelisted occurs, or if a whitelisted rejection does not occur anymore.
MozReview-Commit-ID: 1beGB5GG8Ty
--HG--
extra : rebase_source : b6573f8e2001f91d0e5a50f6376b191459549e94
extra : intermediate-source : 0411e687044ecc7b56684196238e6e6e68a9d685
extra : source : 8d53be05afc59519c5ce8cfae96d284a972fda71
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
This will create a mochitest selftest harness based on |mach python-test|. There
is also a basic test that checks whether TEST-PASS and TEST-UNEXPECTED-FAIL work.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : d73b37305590a415e350ee45785a85635e7d4209
This is case that got hit with the new mochitest selftest harness. In this scenario, several MochitestDesktop instances
(which call commandline.setup_logging in their constructor) are instantiated in the same interpreter. Because mozlog
implicitly saves the logger state, this meant that setup_logging kept appending duplicate handlers to the existing ones.
I believe that the intent of 'setup_logging' is to get a brand new logger, so it should ensure logger state is reset
on subsequent calls.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : f267489bef99f3ac3d657357002a0001610a038f
This means they will be copied to $PROFILE/extensions, which the sandbox allows
access to; if they are installed as temporary addons, loading frame scripts in
the content process tries to read from wherever they happen to be on disk. This
breaks running tests with a packaged build once we have full read-restrictions
for the content process sandbox.
MozReview-Commit-ID: 7ZiiM9FMXfG
--HG--
extra : rebase_source : d2cf3a2d06df9099dc6056fae351200eaa1d0ca9
The whitelisting function thisTestLeaksUncaughtRejectionsAndShouldBeFixed was replaced by expectUncaughtRejection, and existing calls did not take effect anymore.
MozReview-Commit-ID: 3uOxkgWYWEz
--HG--
extra : rebase_source : 5a10a3ebbfe0ce2a801330041f95447c313a9a70
extra : source : 6f0394b523a66dab444b8551deb8f3c6c81d8f31
The browser-chrome test suite now detects and reports unhandled rejections of native Promises, in addition to those created by Promise.jsm. The whitelisting mechanism is updated to use primarily the PromiseTestUtils.expectUncaughtRejection function. Tests will fail if a rejection that is not whitelisted occurs, or if a whitelisted rejection does not occur anymore.
MozReview-Commit-ID: 1beGB5GG8Ty
--HG--
extra : rebase_source : 64395c5fdf25deebd60dfbf2cf5df3cbf7ca8abb
extra : amend_source : 0a3f13419c050662680f2bd110d724b3bf991732
extra : source : 8d53be05afc59519c5ce8cfae96d284a972fda71
The browser-chrome self-test files now use the setExpectedFailuresForSelfTest function to specify the exact number of assertion failures that will be triggered. Also, most failures are now intercepted when specifying the "fail-if" property in a "browser.ini" manifest, while previously only those triggered using the "ok" function were intercepted. This allows re-enabling several browser-chome self-tests.
MozReview-Commit-ID: DlDjWaJPfvH
--HG--
extra : rebase_source : d9977dac6ecbd2b28f5697d22ce6edf4e1d4f899
extra : source : 6da58c7bb247d3e879012bea8d848eb68f16e36e
The whitelisting function thisTestLeaksUncaughtRejectionsAndShouldBeFixed was replaced by expectUncaughtRejection, and existing calls did not take effect anymore.
MozReview-Commit-ID: 3uOxkgWYWEz
--HG--
extra : rebase_source : 3a7720091180a770b32b595f8094c0d20170166d
The browser-chrome test suite now detects and reports unhandled rejections of native Promises, in addition to those created by Promise.jsm. The whitelisting mechanism is updated to use primarily the PromiseTestUtils.expectUncaughtRejection function. Tests will fail if a rejection that is not whitelisted occurs, or if a whitelisted rejection does not occur anymore.
MozReview-Commit-ID: 1beGB5GG8Ty
--HG--
extra : rebase_source : 59e5b84cb431f3ca28287d30a3da8fbea1363ec5
The browser-chrome self-test files now use the setExpectedFailuresForSelfTest function to specify the exact number of assertion failures that will be triggered. Also, most failures are now intercepted when specifying the "fail-if" property in a "browser.ini" manifest, while previously only those triggered using the "ok" function were intercepted. This allows re-enabling several browser-chome self-tests.
MozReview-Commit-ID: DlDjWaJPfvH
--HG--
extra : rebase_source : 498914c84ab69dd484fb5487ad9967073c331fd3
Several mochitests call `assertSnapshots`, which prints a reftest-like output
for use with the reftest analyzer. Update these lines to check failure patterns
before printing, like the regular assertion methods do.
MozReview-Commit-ID: CfChoar7bp8
Several mochitests call `assertSnapshots`, which prints a reftest-like output
for use with the reftest analyzer. Update these lines to check failure patterns
before printing, like the regular assertion methods do.
MozReview-Commit-ID: CfChoar7bp8
The mochitest-chrome harness sends a custom "contentEvent" event from redirect.html
to a listener in browser-test.js. It is possible for redirect.html to be loaded
and send contentEvent before the listener is set up in browser-test.js. In the
absence of a better synchronization strategy, redirect.html now retries the send
after a few seconds. If the first contentEvent was received, the second will be
ignored; if the first contentEvent was not received, the second should avoid an
intermittent harness hang and timeout.
Our linux32-debug build is very slow to startup when running mochitests on aws.
Sometimes we see similar behavior on other linux platforms. Intermittently,
in this environment, startup takes longer than the 120 seconds that marionette
waits, resulting in test failures in bug 1261598. Increasing the marionette
startup timeout to 180 seconds appears to effectively avoid these failures.
This uses XPCOM to replace the default process selector with one that always
asks for a new process and then put the old one back again. This comes with a
test to prove that it works.
MozReview-Commit-ID: Bq6KP4VzP7W
--HG--
extra : rebase_source : a42662d67f7eeca8bc5870d3c70c086abf582164
This uses XPCOM to replace the default process selector with one that always
asks for a new process and then put the old one back again. This comes with a
test to prove that it works.
MozReview-Commit-ID: Bq6KP4VzP7W
--HG--
extra : rebase_source : 4d22d07c6ccfb42718c65fd80cd8a0e20d02f72c
This is pretty straight-forward.
Sadly, this will require local developers to add a "skin" product
flavor to their invocations, like:
./mach gradle app:assembleLocalAustralisDebug
In addition, this shows how many different variants of the Gradle
product flavor are embedded into our automation configurations. I
can't solve that at this time.
Since I was here, I took the time to rename "automation" to
"official", which makes "localAustralis" the default in Android
Studio, avoiding a common issue with new builders producing an APK
that doesn't include omni.ja in the IDE.
MozReview-Commit-ID: CtU7zFpNCob
This will make it easier to correctly star these failures, which
otherwise simply show up as a false positive leak.
MozReview-Commit-ID: 5P6AMRyYmtI
--HG--
extra : rebase_source : 6e63bffb6cdcb389d6533acbc44c2a206009a99e
IMEContentObserver notifies IME of 3 notifications at most when editor is changed.
The order is:
1. text change (with merged range if 2 or more change occurred during an edit transaction)
2. selection change (only the latest selection change. other changes occurred before that during an editor transaction are ignored)
3. position change (scrolled, resized, window moved, etc)
This does not check the behavior in designMode because some operation in testWithHTMLEditor() causes unexpected behavior, e.g., moving focus. It *might* be bug of design mode. However, it doesn't matter for this bug. The important thing of this bug is, there should be automated tests for IMEContentObserver. And fortunately, IMEContentObserver does not check the type of editor. So, it's enough to test only contenteditable element for HTMLEditor at least for now. Therefore, I gave up to test it in designMode for now.
MozReview-Commit-ID: 7L6ZlbVMU2P
--HG--
extra : rebase_source : 8282fe7aa2f4d405f2576f05d46b60b044223855
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
This eliminates a 2 minute timeout seen at the end of Android mochitests
and reftests. Attempts to shutdown the web server were failing because
they were directed at IP 10.0.2.2 -- the loopback address for the
Android emulator.
We don't currently support H.264 on Android in automation, but we can improve
our test coverage by running the VP8 portion of this test in the meantime.
MozReview-Commit-ID: 3SPCTaqlfMk
--HG--
extra : rebase_source : cae3251f489e45f56b04378074083d6b4fd24666
The devicemanager killProcess() is updated to use force-stop first, then
use kill if force-stop does not work.
Browser test harnesses are updated to check if killProcess() worked, and
warn if it failed.
The gecko messages are now in the "process_output" action, rather than
in the "log" action (except for a few legacy cases), so examine both
when looking for LSAN messages.
MozReview-Commit-ID: 82r1p8WLwFa
--HG--
extra : rebase_source : 5af1c529e58f5ba90a3fd222e3cbbc67a850a08c
There were options already being passed to BrowserTestUtils.removeTab, with only
a single property being observed, "dontRemove". This caused BrowserTestUtils.removeTab
to return a Promise once a tab is removed, but didn't actually remove the tab (as the
calling test would be responsible for that themselves). This patch removes that option,
and adds a method to BrowserTestUtils called tabRemoved to use for that case instead.
The options being passed to removeTab are now forwarded along directly to tabbrowser's
removeTab method.
MozReview-Commit-ID: JzfZuoZmlJ0
--HG--
extra : rebase_source : 71afc1f82ecd979b101a9f1a1ef1766185eefd75
There were options already being passed to BrowserTestUtils.removeTab, with only
a single property being observed, "dontRemove". This caused BrowserTestUtils.removeTab
to return a Promise once a tab is removed, but didn't actually remove the tab (as the
calling test would be responsible for that themselves). This patch removes that option,
and adds a method to BrowserTestUtils called tabRemoved to use for that case instead.
The options being passed to removeTab are now forwarded along directly to tabbrowser's
removeTab method.
MozReview-Commit-ID: JzfZuoZmlJ0
--HG--
extra : rebase_source : cd9e7834f2f507b91cac8e9bb8e1dd58e2ba33d5
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
With bug 1256472 fixed, we have greater probability of out-of-process browsers being in the midst
of teardown and destruction when tests complete. There's already TabDestroyObserver in the parent
process making sure that the TabParent's are properly cleaned up, but we need to be more aggressive
about clearing out remaining nsIDOMWindow's and DocShells in the content processes.
This change more or less mirrors what's already going on in browser-test.js.
MozReview-Commit-ID: FZnNLpbfTEY
--HG--
extra : rebase_source : b31ce374f22d851d742b20086ff6676d82663c02
./mach robocop calls through to mochitest_options.py to validate the passed command line options, so it's not necessary to define a default Robocop APK path from two different places.
MozReview-Commit-ID: 8CryDqOKDBF
--HG--
extra : rebase_source : 843c1395fe384bc2a03bacf9a61b4abbdc4fca4c
By using --wait-for-jsdebugger, we can allow the test suite to start
automatically and also ensure the JS debugger has a chance to connect.
There's still an extra click to get the tests running though (at least on macOS)
because the test harness needs to be focused and the extra process used for the
Browser Toolbox removes the focus from it.
MozReview-Commit-ID: 1Eg7lqG3KST
--HG--
extra : rebase_source : 5df1ac74a3e083ac6c2293fb1cf8a27509b81274
Mochitest currently converts unstructured logs (e.g output from gecko) to 'info' messages. But
this means those messages won't be validated against mozharness' error logs. This change first
gets unstructured messages logged as process_output, and also ensures the StructuredOutputParser
in mozharness checks process_output messages against the error list.
MozReview-Commit-ID: KPTQnulwzyK
--HG--
extra : rebase_source : bdcdbf5567355f28ab88d17b27f44d5dfa0467c2
* Remove eslint rules for PSM which are redundant with toolkit/.eslintrc.js
* Fix missing plugins block in mochitest.eslintrc.js
* Disable brace-style checking in mixed-content mochitests which use boilerplate where calls to runTest and afterNavigationTest all use opening brace on newline. I've left this for a follow-up.
* Fix lint errors resulting from new rules defined by toolkit's eslintrc.js
MozReview-Commit-ID: EepCLrzAsdM
--HG--
extra : rebase_source : e74e008403d9cd70703d60cf829af01dbede0353
Mochitest currently converts unstructured logs (e.g output from gecko) to 'info' messages. But
this means those messages won't be validated against mozharness' error logs. This change first
gets unstructured messages logged as process_output, and also ensures the StructuredOutputParser
in mozharness checks process_output messages against the error list.
MozReview-Commit-ID: KPTQnulwzyK
--HG--
extra : rebase_source : c9038ff87843b1cb46de506e245715f24a60ed63
Mochitest currently converts unstructured logs (e.g output from gecko) to 'info' messages. But
this means those messages won't be validated against mozharness' error logs. This change first
gets unstructured messages logged as process_output, and also ensures the StructuredOutputParser
in mozharness checks process_output messages against the error list.
MozReview-Commit-ID: KPTQnulwzyK
--HG--
extra : rebase_source : 52f2f048aee5bd40cde29030e7668b321366e9ec
This patch is generated by the following sed script:
find . ! -wholename '*/.hg*' -type f \( -iname '*.html' -o -iname '*.xhtml' -o -iname '*.xul' -o -iname '*.js' \) -exec sed -i -e 's/\(\(text\|application\)\/javascript\);version=1.[0-9]/\1/g' {} \;
MozReview-Commit-ID: AzhtdwJwVNg
--HG--
extra : rebase_source : e8f90249454c0779d926f87777f457352961748d
This patch allows the use of the flag '--jscov-dir-prefix' for mochitest plain tests to enable code coverage collection with the JS Debugger. It also enables the mochitest-plain tests for the linux64-jsdcov build platform.
MozReview-Commit-ID: 6RqMEZ1I0D7
--HG--
extra : rebase_source : 351754541801f69f7c54807f6bdd3a3d1baf9222
Include the current flavor in the "No tests found" error message. Also suggest
what may have went wrong.
MozReview-Commit-ID: 5LEQFVDoJrT
--HG--
extra : rebase_source : 6abe8c6f467a11ff8abc33a37eefab80505e3cfb
Previously the run-wizard script would add a command to source the virtualenv in ~/.bashrc after
mozharness finished setting things up. This is fragile, assumes people are using bash, etc. Plus
it appeared to intermittently fail for some users.
Instead, this activates the virtualenv directly from individual mach commands that need it. This
guarantees we will always be using the virtualenv if required (and won't be using it if not). The
'activate_this.py' script is invoked the same way that we do it for in-tree mach commands:
https://dxr.mozilla.org/mozilla-central/rev/9c06e744b1befb3a2e2fdac7414ce18220774a1d/python/mozbuild/mozbuild/virtualenv.py#456
MozReview-Commit-ID: CfcoiVJXQTl
--HG--
extra : rebase_source : da01d1ce1bd9b41c89922e989f857c4de8c09341
This fixes a regression in bug 1332573 where if no tests are found by the getActiveTests
function, the status variable never gets set. This ensures that we always set status, and
that it will be set to a non-zero return code if any of the calls to runApp fail.
MozReview-Commit-ID: 20M7FcBs0DF
--HG--
extra : rebase_source : d33a1a052986df628a90146e6de29aa415d8a1ad
This was tricky because synthesizeMouse would compute the incorrect
coordinates if the requested event target was in a sub-frame. With this patch,
we deal correctly with sub-frames.
MozReview-Commit-ID: KpUKxFXKMrl
--HG--
extra : rebase_source : 08dbbd1abe04886ac4fe14fad11312ba0f63873b
On architectures like alpha and ia64, the glibc does not use the
canonical ABI version number 6 but 6.1 and therefore the filename
of the C library is not libc.so.6 but libc.so.6.1. We're therefore
making the Python code more flexible and use find_library from
ctypes.util to determine the filename from the environment instead
of hard-coding it.
--HG--
extra : rebase_source : 66e1caf57f9743dc10698b93c7e1f1a1b3b67e85
