The suite name is no part of the suite_start message and can be printed via a
mozlog formatter. We shouldn't be printing this directly to stdout from |mach
mochitest|.
MozReview-Commit-ID: TOuqfEF14
--HG--
extra : rebase_source : 23bb214e6c1eedccf95abe08bb06b187728599d5
The suite name is no part of the suite_start message and can be printed via a
mozlog formatter. We shouldn't be printing this directly to stdout from |mach
mochitest|.
MozReview-Commit-ID: TOuqfEF14
--HG--
extra : rebase_source : ebcf6e37b77a0ca1e6da242df9066d4b41286d5c
SynthesizeDrop synthesizes a dnd operation by starting a drag session and firing mouse events. Tweak the API to follow the order of real use cases.
MozReview-Commit-ID: 1SdJPUtSVKq
The suite name is no part of the suite_start message and can be printed via a
mozlog formatter. We shouldn't be printing this directly to stdout from |mach
mochitest|.
MozReview-Commit-ID: TOuqfEF14
--HG--
extra : rebase_source : 15db5dc64476c52c91e59868c3653d5962870a5d
When 'summary_on_shutdown' is True (which is the case for |mach test| and |mach
mochitest|), BaseSummaryFormatters will save the summary information until the
'shutdown' action is received at the end of the logger's lifetime.
Summary information will no longer be dumped on 'suite_end'.
MozReview-Commit-ID: HKtVr5PxfOy
--HG--
extra : rebase_source : f350f09111deb510b27a4e55797243dda3160869
We should follow the real use cases to synthesize mouse or touch events to generate pointer events so that the related logic is covered by these test cases.
MozReview-Commit-ID: 9xSgjSL0Azt
We should follow the real use cases to synthesize mouse or touch events to generate pointer events so that the related logic is covered by these test cases.
MozReview-Commit-ID: 9xSgjSL0Azt
In order for |mach test| and |mach mochitest| to log an overall summary,
every test harness invocation they make needs to use the same structured
logger (otherwise the affected suite will be missing from the summary).
MozReview-Commit-ID: 8LJw7r8SItk
--HG--
extra : rebase_source : 1417dce3817bae94ad61a5250065c6cbc35857e4
Suite names are currently only used by formatters to print out
an overall summary from |mach test| and |mach mochitest|. So
this doesn't need to be exact and can be tweaked further at a
later date.
If multiple test invocations have the same suite name, their
results will be merged in the overall summary. If a suite name
is missing, the summary will contain a placeholder name.
MozReview-Commit-ID: K1xpb9hUQRX
--HG--
extra : rebase_source : cc8cc8b36255d939dd5dffd3c5444c34951ac8e2
This is no longer necessary because the formatters now have the ability
to dump failure summaries which include failing tests and subtest results.
This also needs to be removed because it is throwing off the summary counts.
MozReview-Commit-ID: GbQkk0xQRds
--HG--
extra : rebase_source : 7bfbd81e1e7019ab639a98069fd7f0da994bfff7
This function makes it possible to listen for multiple events from the
content process, even when there are frameloader swaps.
This commit also adds a checkFn param to firstBrowserLoaded, which is
useful.
MozReview-Commit-ID: 93ItHIPSGVU
These magic locations evolve over time. Baking them into
moz.configure is the easiest way to share them across the build
system, and pushing them into a new |mach android *| command continues
a pattern that has been very successful.
MozReview-Commit-ID: CyxVQ0LHHgl
--HG--
extra : rebase_source : 8350d71665f0126aa4ee2c8fec32c4b8e34dc772
These magic locations evolve over time. Baking them into
moz.configure is the easiest way to share them across the build
system, and pushing them into a new |mach android *| command continues
a pattern that has been very successful.
MozReview-Commit-ID: CyxVQ0LHHgl
--HG--
extra : rebase_source : 8350d71665f0126aa4ee2c8fec32c4b8e34dc772
This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
--HG--
extra : rebase_source : 08dc3c05318b572ae1026227d0369fa8bf21b20f
This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
--HG--
extra : rebase_source : 08dc3c05318b572ae1026227d0369fa8bf21b20f
When psutil is available, use it, but also add any processes found in the
process log to the list of children. Hopefully this will ensure that all
browser children are successfully killed.
When running mochitests in automation, we expect no more than 1 instance of
Firefox to be running at any time: the browser under test. In local testing
scenarios, additional browser instances might be running.
Warn when mochitests find additional 'firefox' instances running before launching
a browser for testing, to make it easier to detect anomalies in automation.
This patch also checks for I/O errors when deleting leftover crashdump files
in the test harness and reports them if the files couldn't be deleted. This
won't prevent tests from leaking crash dumps if they're not written correctly,
but it will make their presence visible and won't cause the harness to hang if
it cannot delete them.
MozReview-Commit-ID: FLvBJxEKkH5
--HG--
extra : rebase_source : 6a0242b670ceabc2307c5345b0ea7108857a10d2
extra : source : 95b1010395826d06cdc77cbf43faf7d8681504eb
mochitests needs access to TESTING_JS_MODULES which are installed in $(OBJDIR)\_tests\modules\
MozReview-Commit-ID: CMgDlj4uKeP
--HG--
extra : rebase_source : 0a32b71db56bd68fc369d58117075dabf0465727
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
--HG--
rename : python/mozbuild/mozbuild/test/test_testing.py => testing/mozbase/moztest/tests/test_resolve.py
extra : rebase_source : 7ff11f9ec455547533082d20cb5371845f7a4f21
This adds a global instance that can be used by invoking assertion methods directly on the imported Assert object. The test suites set the global reporter function to the one for the currently running test.
MozReview-Commit-ID: 8dksVc9o7r
--HG--
extra : rebase_source : 3e382c6d24c6019d29963811c37469cfc23b928f
This patch includes a bunch of somewhat related fixes, these are:
- Ensuring that when a mochitest calls SimpleTest.expectChildProcessCrash()
the harness will wait for the crashes to be recorded before deleting the
dump files. This involves a message round-trip between the content and
parent process so to minimize its performance impact on all the non-crashing
tests it is done only when required.
- As an additional optimization, the SimpleTest harness will not send a
message to the content process anymore whenever it receives an
ipc:content-shutdown event, instead it does it only for abnormal shutdowns.
- Manually fixing remaining mochitests causing crashes to wait for crashes to
be recorded before finishing and deleting the dump files.
- Modifying BrowserTestUtils.crashBrowser() so that it optionally does not
delete the dump files, this is useful for tests that submit their dumps and
thus delete them on their own.
MozReview-Commit-ID: 4SLJ8BjJ18n
--HG--
extra : source : b5452a41bb962c6929292c5c538e19ac28d84fe7
We are seeing failures starting websocketProcessBridge due to the port being
in use. It is most likely a previously started websocketProcessBridge process that
is using the port.
On Windows, mozprocess.kill() calls TerminateJobObject/TerminateProcess and
GetExitCodeProcess, but these are asynchronous and don't wait for the process to
actually exit. Adding a wait call should guarantee the process has exited before
continuing which will hopefully ensure the port is free by the time we start an
additional websocketProcessBridge.
MozReview-Commit-ID: HGyjEsy1Ons
--HG--
extra : rebase_source : f56034e1fc0febae07d9b2728eded0a48975baca
The goal is to use a newer Android-Gradle build plugin version (2.3.3
is latest stable). That requires a modern Gradle (anything 3.3+, but
3.4.1 is the default from my Android Studio), and also a newer
build-tools (25.0.3 is latest stable).
The locations of lint output changed, and we want to use the standard
output location because it's difficult to accommodate variant details
in custom names. We change the location of findbugs output to follow
suit.
This requires either:
- fixing lint errors
- adding to the lint whitelist
- using the new lint baseline
It's best to use the new lint baseline, which will happen in the next commit.
MozReview-Commit-ID: D19FzIDCJrE
--HG--
extra : rebase_source : 12d132c0c3e0dbe2b8873b31360ea96d612de44c
Set the idle preference to be the current time, so that idle-daily won't kick in for 24 hours.
MozReview-Commit-ID: 6OJCSm8RaeZ
--HG--
extra : rebase_source : 71217263ddd5b9299e8463254f48ad2d9918b8a2
The main bug this fixes is that on reftest, the objdir needs to be added to the
whitelist on Windows. However, this only happens when running on Linux for some
reason.
Changing the --work-path and --obj-path arguments to --sandbox-read-whitelist
was more of a drive-by cleanup than anything necessary.
MozReview-Commit-ID: Dq8ZLETMzeM
--HG--
extra : rebase_source : 3d2cdda125205e76f86235eb373074899fe0789a
If Firefox crashes while mochitests or reftests are running, Marionette
will trigger an IOError exception which currently gets logged immediately,
and causes no post-test checks to be performed. This results in missing
crash and leak checks, and an unclear failure message on Treeherder.
With this change only the IOError from Marionette gets deferred until
all post-test checks are done. This fixes the failure messages, and will
put PROCESS_CRASH or leak log output first.
MozReview-Commit-ID: JCYP5LlPE1m
--HG--
extra : rebase_source : a4a9455402b01db8ef1dbafccc7a726d2927ec03
This adds new test verification steps for mochitest/reftest/xpcshell tests
with MOZ_CHAOSMODE=3 (thread scheduling and network thread scheduling).
Enabling all chaos mode features seems to destabilize test verification
so I am only enabling these features for now.
This patch makes it possible to disable mochitest-devtools tests on linux64-jsdcov through the 'coverage' flag.
MozReview-Commit-ID: 1gkgbFo19MZ
--HG--
extra : rebase_source : 9c6d311cb467e4424d42848e1caf024f9702b8cd
Currently the mochitest and reftest runners are forcing a timeout of 60s
(or 900s for valgrind or debug builds) when calling `start_session` of
Marionette. While this method still offers a timeout parameter, the
timeout should be set via the `startup_timeout` argument for Marionette.
Reason is that Marionette doesn't control the browser instance and
is getting called right after the application gets launched. As such
slow running builds can cause timeouts once it takes longer than 60s
for the Marionette server to get started.
By using `startup_timeout` the timeout will even be configurable via
the command line and mozharness config files.
MozReview-Commit-ID: EV7GklBcJjU
--HG--
extra : rebase_source : 9d3c623c49deb92d68c40ba4410c812c864e06bd
This patch re-organizes control flow in the browser-chrome harness so that
extra memory cleanup only happens just before the browser is closed, even
when running with --repeat.
Bug 1394910 modified the runtests.py return code when one or more tests
failed, but that causes mozharness to emit a starrable error. This patch
returns to the old behavior of normally returning 0 from runtests.py when
tests fail. To allow for test verification, verifyTests() is updated to
explicitly check for failed tests, instead of relying on the runTests()
return code.
This includes code for downloading a Firefox binary, downloading + setting up a tests.zip and
running output through mozharness' output parsers. This is all stuff that will also be required
for the reftest selftests.
I couldn't think of a better location to put this stuff, suggestions welcome.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : a328f6bc90e73fe23f9054933cd01a30065419f6
This patch gently removes support for __exposedProps__ by changing
ExposedPropertiesOnly::check() to always return false, while still
failing silently in deny for some kinds of access.
The tests that I changed all involve testing the behavior with
__exposedProps__. I adjusted them to expect it to fail, or to adjust
the error message they get when they fail. That seemed better than
deleting them entirely.
Note that test_bug1065185.html had a bug, so that it never executed
the first case. I fixed that, and then fixed up the test to work when
__exposedProps__ is not supported.
This also removes various bits of the test framework that use
__exposedProps__, but don't actually need to.
MozReview-Commit-ID: 8fvkAmITmXY
--HG--
extra : rebase_source : ef7e2c55adc12511f17f3865ebb46c343875f0b3
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
This renames killNamedOrphans to killNamedProc and removes the check for
parent proc id, so that any xpcshell or ssltunnel process is killed before
starting a mochitest or reftest run. For reftests, this is moved out of
the desktop harness and into the remote/android harness, since desktop
reftests do not use xpcshell or ssltunnel.
Overriding Services.scriptloader has a lot of consequences in terms of path
name magling and choice of target global when no target object is passed.
Since this test code doesn't care about callers outside of XPCOMUtils, it
should only override the instance used by XPCOMUtils, and not the instance
used by other callers.
MozReview-Commit-ID: 2LEchjzn0U5
--HG--
extra : rebase_source : 739fe2b03adb0b4637056b25d6b747a4e1f5a4a9
We should now only maintain Photon flavor. Remove all Australis related configuration in build scripts.
MozReview-Commit-ID: H4LE8LAso42
--HG--
extra : rebase_source : 2d5a05e43b261d573677834210a7b3fb18aebcac
This will only work if runByManifest is enabled, otherwise the harness will error out. It's also
illegal to set this on an individual test, it must be on the entire manifest.
MozReview-Commit-ID: LWYa3Sk1uyW
--HG--
extra : rebase_source : ea4add087c795324745a0a5fc18b3ef50b2292b0
This mechanically replaces nsILocalFile with nsIFile in
*.js, *.jsm, *.sjs, *.html, *.xul, *.xml, and *.py.
MozReview-Commit-ID: 4ecl3RZhOwC
--HG--
extra : rebase_source : 412880ea27766118c38498d021331a3df6bccc70
Two small changes: 1. Test verification uses a copy of the runtests.py
options, so that any changes made to the options object in one step do
not affect the next step. 2. Use mochitest url format
.../tests?testname=xxx instead of .../xxx?... when a single test is
being run, regardless of --repeat; this is required so that other
options, notably --keep-open, are respected.
--verify runs tests many times, in various configurations, in hopes of
finding intermittent failures. This option is added mostly for the -
coming soon! - test-verify task in automation, but can be used from
mach too, as a convenient way to stress tests of interest.
Changes were made to the SpawnTask.js file, which is where the add_task
implementation lives for this API.
MozReview-Commit-ID: 7bPlcrrJkCi
--HG--
extra : rebase_source : ce14fda2e71508d3e9dea39ad62e65a57b432779
.skip() allows you to skip a specific task from running and .only() allows you
to focus on only one specific task.
MozReview-Commit-ID: 36qQOhICN7s
--HG--
extra : rebase_source : 4b8e15a65dd9370b87dfdba8c85c64aae76dd4a0
Test harnesses may use STYLO_FORCE_ENABLED, so we need to fold this into mozinfo
for mochitest annotations.
MozReview-Commit-ID: JyadTE7apX6
--HG--
extra : rebase_source : 4eda1f150c012a8f5c0a58fdc921d8a6e03c4e68
This also starts running the selftests on linux debug builds, since that's the only place that we
can test assertions and leaks.
MozReview-Commit-ID: JTdTLOLWn5r
--HG--
extra : rebase_source : 340aca0c4e5f9697b1d652fd192332e47a1acab9
extra : histedit_source : 2d4b542d2122b4c6d2d48fc9c49848d5453e4533
This patch renames the mozinfo flag 'coverage' to 'ccov' to avoid ambiguity in whether a test is being skipped for linux64-ccov or for linux64-jsdcov. It also removes the 'runtests.py' mozinfo hack and renames all occurrences of 'coverage' that are used for skipping tests in linux64-ccov.
MozReview-Commit-ID: IF2640bDQP7
--HG--
extra : rebase_source : 614020325e30d1ca9e01aaf08479b8a4ffaec888
This also starts running the selftests on linux debug builds, since that's the only place that we
can test assertions and leaks.
MozReview-Commit-ID: JTdTLOLWn5r
--HG--
extra : rebase_source : 643f8d15a550fa39f4d3daba75c60560deefc96e
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 952043a441b41b2f58ec4abc51ac15fa71fc142f
The whitelisting function thisTestLeaksUncaughtRejectionsAndShouldBeFixed was replaced by expectUncaughtRejection, and existing calls did not take effect anymore.
MozReview-Commit-ID: 3uOxkgWYWEz
--HG--
extra : rebase_source : 6b27b77da78743880529adae7adf4ec2aa64dfea
extra : intermediate-source : 5809a30a0c4817509beb1078cf336d8367b8107a
extra : source : 6f0394b523a66dab444b8551deb8f3c6c81d8f31
The mochitest harness uses testEnd multiple times to log various failures. This can result
in several testEnd messages, which will soon cause mozlog to spit out an error. Instead,
these should be testStatus.
This also starts using mozlog's assertion_count log action to log test assertions (again,
instead of testEnd).
MozReview-Commit-ID: FFsyicSso5Y
--HG--
extra : rebase_source : 5e8d02714fcf24f2b86a9867b0403bbda0d00f91
"Automation Error" is not sufficient to turn a mozharness job orange.
mochitest-plain failures of this type normally cause job failure by not
printing out the test suite summary. This patch uses the same technique
for geckoview tests: If a crash is detected, do not print a test summary,
so that mozharness will subsequently fail the job.
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 64aa804965d24bb90b103c00c692a2ac6859e408
This also adds a utility function for synthesizing native touch
events to Eventutils.js.
I did not add a test for searchbar because of intermittent issues
with showing the contextmenu (that are not reproducible manually).
I believe this is rather related to searchbar functionality than
my patches.
MozReview-Commit-ID: Dqm92Saosxz
--HG--
extra : rebase_source : e59df4f487f60cea137fbf8aea71a854a5706de9
This also adds a utility function for synthesizing native touch
events to Eventutils.js.
I did not add a test for searchbar because of intermittent issues
with showing the contextmenu (that are not reproducible manually).
I believe this is rather related to searchbar functionality than
my patches.
MozReview-Commit-ID: Dqm92Saosxz
--HG--
extra : rebase_source : d5c4333609b68773e62447bd3158cadfa89b803b
SpecialPowers.loadChromeScript() sends a script to the child process,
then creates a sandbox, and runs the script in that sandbox. There are
various sandboxOptions that can be passed when creating a sandbox, and
it would be nice to have that functionality for loadChromeScript.
I just need this for wantGlobalProperties, but I might as well make it
as general as possible. I'm not sure all of the types it can take can
actually be serialized across processes, but I guess that's okay.
MozReview-Commit-ID: GoJjXdjizFk
--HG--
extra : rebase_source : 9c2bc190dbf5a080978953cffd64205e8b816367
For a while now it has been making the content process sandbox less strict.
MozReview-Commit-ID: Am6fGzViaLk
--HG--
extra : rebase_source : 0bc037f205896c866559a7ab1f7e2c042c3142db
The browser-chrome test suite now detects and reports unhandled rejections of native Promises, in addition to those created by Promise.jsm. The whitelisting mechanism is updated to use primarily the PromiseTestUtils.expectUncaughtRejection function. Tests will fail if a rejection that is not whitelisted occurs, or if a whitelisted rejection does not occur anymore.
MozReview-Commit-ID: 1beGB5GG8Ty
--HG--
extra : rebase_source : b6573f8e2001f91d0e5a50f6376b191459549e94
extra : intermediate-source : 0411e687044ecc7b56684196238e6e6e68a9d685
extra : source : 8d53be05afc59519c5ce8cfae96d284a972fda71
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
This will create a mochitest selftest harness based on |mach python-test|. There
is also a basic test that checks whether TEST-PASS and TEST-UNEXPECTED-FAIL work.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : d73b37305590a415e350ee45785a85635e7d4209
