Magic-Mirror/gecko-dev
1
0
Fork 0
mirror of https://github.com/mozilla/gecko-dev.git synced 2025-02-24 11:27:49 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Bug 1763980 - Port most of the mochitest-plain documentation to the in-tree docs. r=jgraham,Gijs

Browse Source 
These come from:

 * https://github.com/mdn/archived-content/blob/main/files/en-us/mozilla/projects/mochitest/index.html

With some manual clean-up and review.

Differential Revision: https://phabricator.services.mozilla.com/D143326
This commit is contained in:
Emilio Cobos Álvarez 2022-04-11 12:00:01 +00:00
parent c7e2f47014
commit 82da829168
4 changed files with 609 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/config.yml
View File

@ -50,6 +50,7 @@ categories:
- testing/geckodriver
- testing/test-verification
- testing/webrender
- testing/mochitest-plain
- testing/xpcshell
- web-platform
- gtest

314
testing/docs/mochitest-plain/faq.md Normal file
View File

@ -0,0 +1,314 @@
# Mochitest FAQ
## SSL and https-enabled tests
Mochitests must be run from http://mochi.test/ to succeed. However, some tests
may require use of additional protocols, hosts, or ports to test cross-origin
functionality.
The Mochitest harness addresses this need by mirroring all content of the
original server onto a variety of other servers through the magic of proxy
autoconfig and SSL tunneling. The full list of schemes, hosts, and ports on
which tests are served, is specified in `build/pgo/server-locations.txt`.
The origins described there are not the same, as some of them specify
particular SSL certificates for testing purposes, while some allow pages on
that server to request elevated privileges; read the file for full details.
It works as follows: The Mochitest harness includes preference values which
cause the browser to use proxy autoconfig to match requested URLs with servers.
The `network.proxy.autoconfig_url` preference is set to a data: URL that
encodes the JavaScript function, `FindProxyForURL`, which determines the host
of the given URL. In the case of SSL sites to be mirrored, the function maps
them to an SSL tunnel, which transparently forwards the traffic to the actual
server, as per the description of the CONNECT method given in RFC 2817. In this
manner a single HTTP server at http://127.0.0.1:8888 can successfully emulate
dozens of servers at distinct locations.
## What if my tests aren't done when onload fires?
Use `add_task()`, or call `SimpleTest.waitForExplicitFinish()` before onload
fires (and `SimpleTest.finish()` when you're done).
## How can I get the full log output for my test in automation for debugging?
Add the following to your test:
```
SimpleTest.requestCompleteLog();
```
## What if I need to change a preference to run my test?
The `SpecialPowers` object provides APIs to get and set preferences:
```js
await SpecialPowers.pushPrefEnv({ set: [["your-preference", "your-value" ]] });
// ...
await SpecialPowers.popPrefEnv(); // Implicit at the end of the test too.
```
You can also set prefs directly in the manifest:
```ini
[DEFAULT]
prefs =
browser.chrome.guess_favicon=true
```
If you need to change a pref when running a test locally, you can use the
`--setpref` flag:
```
./mach mochitest --setpref="javascript.options.jit.chrome=false" somePath/someTestFile.html
```
Equally, if you need to change a string pref:
```
./mach mochitest --setpref="webgl.osmesa=string with whitespace" somePath/someTestFile.html
```
## Can tests be run under a chrome URL?
Yes, use [mochitest-chrome](../chrome-tests/index.rst).
## How do I change the HTTP headers or status sent with a file used in a Mochitest?
Create a text file next to the file whose headers you want to modify. The name
of the text file should be the name of the file whose headers you're modifying
followed by `^headers^`. For example, if you have a file `foo.jpg`, the
text file should be named `foo.jpg^headers^`. (Don't try to actually use the
headers file in any other way in the test, because the HTTP server's
hidden-file functionality prevents any file ending in exactly one ^ from being
served.)
Edit the file to contain the headers and/or status you want to set, like so:
```
HTTP 404 Not Found
Content-Type: text/html
Random-Header-of-Doom: 17
```
The first line sets the HTTP status and a description (optional) associated
with the file. This line is optional; you don't need it if you're fine with the
normal response status and description.
Any other lines in the file describe additional headers which you want to add
or overwrite (most typically the Content-Type header, for the latter case) on
the response. The format follows the conventions of HTTP, except that you don't
need to have HTTP line endings and you can't use a header more than once (the
last line for a particular header wins). The file may end with at most one
blank line to match Unix text file conventions, but the trailing newline isn't
strictly necessary.
## How do I write tests that check header values, method types, etc. of HTTP requests?
To write such a test, you simply need to write an SJS (server-side JavaScript)
for it. See the [testing HTTP server](/networking/http_server_for_testing.rst)
docs for less mochitest-specific documentation of what you can do in SJS
scripts.
An SJS is simply a JavaScript file with the extension .sjs which is loaded in a
sandbox. Don't forget to reference it from your `mochitest.ini` file too!
```ini
[DEFAULT]
support-files =
test_file.sjs
```
The global property `handleRequest` defined by the script is then executed with
request and response objects, and the script populates the response based on the
information in the request.
Here's an example of a simple SJS:
```js
function handleRequest(request, response) {
// Allow cross-origin, so you can XHR to it!
response.setHeader("Access-Control-Allow-Origin", "*", false);
// Avoid confusing cache behaviors
response.setHeader("Cache-Control", "no-cache", false);
response.setHeader("Content-Type", "text/plain", false);
response.write("Hello world!");
}
```
The file is run, for example, at either
http://mochi.test:8888/tests/PATH/TO/YOUR/test_file.sjs,
http://{server-location}/tests/PATH/TO/YOUR/test_file.sjs - see
`build/pgo/server-locations.txt` for server locations!
If you want to actually execute the file, you need to reference it somehow. For
instance, you can XHR to it OR you could use a HTML element:
```js
var xhr = new XMLHttpRequest();
xhr.open("GET", "http://test/tests/dom/manifest/test/test_file.sjs");
xhr.onload = function(e){ console.log("loaded!", this.responseText)}
xhr.send();
```
The exact properties of the request and response parameters are defined in the
`nsIHttpRequestMetadata` and `nsIHttpResponse` interfaces in
`nsIHttpServer.idl`. However, here are a few useful ones:
* `.scheme` (string). The scheme of the request.
* `.host` (string). The scheme of the request.
* `.port` (string). The port of the request.
* `.method` (string). The HTTP method.
* `.httpVersion` (string). The protocol version, typically "1.1".
* `.path` (string). Path of the request,
* `.headers` (object). Name and values representing the headers.
* `.queryString` (string). The query string of the requested URL.
* `.bodyInputStream` ??
* `.getHeader(name)`. Gets a request header by name.
* `.hasHeader(name)` (boolean). Gets a request header by name.
**Note**: The browser is free to cache responses generated by your script. If
you ever want an SJS to return different data for multiple requests to the same
URL, you should add a `Cache-Control: no-cache` header to the response to
prevent the test from accidentally failing, especially if it's manually run
multiple times in the same Mochitest session.
## How do I keep state across loads of different server-side scripts?
Server-side scripts in Mochitest are run inside sandboxes, with a new sandbox
created for each new load. Consequently, any variables set in a handler don't
persist across loads. To support state storage, use the `getState(k)` and
`setState(k, v)` methods defined on the global object. These methods expose a
key-value storage mechanism for the server, with keys and values as strings.
(Use JSON to store objects and other structured data.) The myriad servers in
Mochitest are in reality a single server with some proxying and tunnelling
magic, so a stored state is the same in all servers at all times.
The `getState` and `setState` methods are scoped to the path being loaded. For
example, the absolute URLs `/foo/bar/baz, /foo/bar/baz?quux, and
/foo/bar/baz#fnord` all share the same state; the state for /foo/bar is entirely
separate.
You should use per-path state whenever possible to avoid inter-test dependencies
and bugs.
However, in rare cases it may be necessary for two scripts to collaborate in
some manner, and it may not be possible to use a custom query string to request
divergent behaviors from the script.
For this use case only you should use the `getSharedState(k, v)` and
`setSharedState(k, v)` methods defined on the global object. No restrictions
are placed on access to this whole-server shared state, and any script may add
new state that any other script may delete. To avoid conflicts, you should use
a key within a faux namespace so as to avoid accidental conflicts. For example,
if you needed shared state for an HTML5 video test, you might use a key like
`dom.media.video:sharedState`.
A further form of state storage is provided by the `getObjectState(k)` and
`setObjectState(k, v)` methods, which will store any `nsISupports` object.
These methods reside on the `nsIHttpServer` interface, but a limitation of
the sandbox object used by the server to process SJS responses means that the
former is present in the SJS request handler's global environment with the
signature `getObjectState(k, callback)`, where callback is a function to be
invoked by `getObjectState` with the object corresponding to the provided key
as the sole argument.
Note that this value mapping requires the value to be an XPCOM object; an
arbitrary JavaScript object with no `QueryInterface` method is insufficient.
If you wish to store a JavaScript object, you may find it useful
to provide the object with a `QueryInterface` implementation and then make
use of `wrappedJSObject` to reveal the actual JavaScript object through the
wrapping performed by XPConnect.
For further details on state-saving mechanisms provided by `httpd.js`, see
`netwerk/test/httpserver/nsIHttpServer.idl` and the
`nsIHttpServer.get(Shared|Object)?State` methods.
## How do I write a SJS script that responds asynchronously?
Sometimes you need to respond to a request asynchronously, for example after
waiting for a short period of time. You can do this by using the
`processAsync()` and `finish()` functions on the response object passed to the
`handleRequest()` function.
`processAsync()` must be called before returning from `handleRequest()`. Once
called, you can at any point call methods on the request object to send
more of the response. Once you are done, call the `finish()` function. For
example you can use the `setState()` / `getState()` functions described above to
store a request and later retrieve and finish it. However be aware that the
browser often reorders requests and so your code must be resilient to that to
avoid intermittent failures.
```js
let { setTimeout } = Cu.import("resource://gre/modules/Timer.jsm")
function handleRequest(request, response) {
response.processAsync();
response.setHeader("Content-Type", "text/plain", false);
response.write("hello...");
setTimeout(function() {
response.write("world!");
response.finish();
}, 5 * 1000);
}
```
For more details, see the `processAsync()` function documentation in
`netwerk/test/httpserver/nsIHttpServer.idl`.
## How do I get access to the files on the server as XPCOM objects from an SJS script?
If you need access to a file, because it's easier to store image data in a file
than directly in an SJS script, use the presupplied `SERVER_ROOT` object
state available to SJS scripts running in Mochitest:
```js
function handleRequest(req, res) {
var file;
getObjectState("SERVER_ROOT", function(serverRoot) {
file = serverRoot.getFile("tests/content/media/test/320x240.ogv");
});
// file is now an XPCOM object referring to the given file
res.write("file: " + file);
}
```
The path you specify is used as a path relative to the root directory served by
`httpd.js`, and an `nsIFile` corresponding to the file at that location is
returned.
Beware of typos: the file you specify doesn't actually have to exist
because file objects are mere encapsulations of string paths.
## Diagnosing and fixing leakcheck failures
Mochitests output a log of the windows and docshells that are created during the
test during debug builds. At the end of the test, the test runner runs a
leakcheck analysis to determine if any of them did not get cleaned up before the
test was ended.
Leaks can happen for a variety of reasons. One common one is that a JavaScript
event listener is retaining a reference that keeps the window alive.
```js
// Add an observer.
Services.obs.addObserver(myObserver, "event-name");
// Make sure and clean it up, or it may leak!
Services.obs.removeObserver(myObserver, "event-name");
```
Other sources of issues include accidentally leaving a window, or iframe
attached to the DOM, or setting an iframe's src to a blank string (creating an
about:blank page), rather than removing the iframe.
Finding the leak can be difficult, but the first step is to reproduce it
locally. Ensure you are on a debug build and the `MOZ_QUIET` environment flag
is not enabled. The leakcheck test analyzes the test output. After reproducing
the leak in the test, start commenting out code until the leak goes away. Then
once the leak stop reproducing, find the exact location where it is happening.
See [this post](https://crisal.io/words/2019/11/13/shutdown-leak-hunting.html)
for more advanced debugging techniques involving CC and GC logs.

292
testing/docs/mochitest-plain/index.md Normal file
View File

@ -0,0 +1,292 @@
# Mochitest
## DISCLAIMER
If you are testing web platform code, prefer using use a [wpt
test](/web-platform/index.rst) (preferably upstreamable ones).
## Introduction
Mochitest is an automated testing framework built on top of the
[MochiKit](https://mochi.github.io/mochikit/) JavaScript libraries.
Only things that can be tested using JavaScript (with chrome privileges!) can be
tested with this framework. Given some creativity, that's actually much more
than you might first think, but it's not possible to write Mochitest tests to
directly test a non-scripted C++ component, for example. (Use a compiled-code
test like [GTest](/gtest/index.rst) to do that.)
## Running tests
To run a single test (perhaps a new test you just added) or a subset of the
entire Mochitest suite, pass a path parameter to the `mach` command.
For example, to run only the test `test_CrossSiteXHR.html` in the Mozilla source
tree, you would run this command:
```
./mach test dom/security/test/cors/test_CrossSiteXHR.html
```
To run all the tests in `dom/svg/`, this command would work:
```
./mach test dom/svg/
```
You can also pass a manifest path to run all tests on that manifest:
```
./mach test dom/base/test/mochitest.ini
```
## Running flavors and subsuites
Flavors are variations of the default configuration used to run Mochitest. For
example, a flavor might have a slightly different set of prefs set for it, a
custom extension installed or even run in a completely different scope.
The Mochitest flavors are:
* **plain** - The most basic and common Mochitest. They run in content scope,
but can access certain privileged APIs with SpecialPowers.
* **browser** - These often test the browser UI itself and run in browser
window scope.
* **chrome** - These run in chrome scope and are typically used for testing
privileged JavaScript APIs. More information can be found
[here](../chrome-tests/index.rst).
* **a11y** - These test the accessibility interfaces. They can be found under
the top `accessible` directory and run in chrome scope. Note that these run
without e10s / fission.
A subsuite is similar to a flavor, except that it has an identical
configuration. It is just logically separated from the "default" subsuite for
display purposes. For example, devtools is a subsuite of the browser flavor.
There is no difference in how these two jobs are run. It exists so that the
devtools team can easily see and run their tests.
**Note**: There are also tags, which are similar to subsuites. Although they
both are used to logically group related sets of tests, they behave
differently. For example, applying a subsuite to a test removes that test from
the default set, whereas, a tag does not remove it.
By default, mach finds and runs every test in the given subdirectory no matter
which flavor or subsuite it belongs to. But sometimes, you might only want to
run a specific flavor or subsuite. This can be accomplished using the `--flavor`
(or `-f`) and `--subsuite` options respectively. For example:
```
./mach mochitest -f plain # runs all plain tests
./mach mochitest -f browser --subsuite devtools # runs all browser tests in the devtools subsuite
./mach mochitest -f chrome dom/indexedDB # runs all chrome tests in the dom/indexedDB subdirectory
```
In many cases, it won't be necessary to filter by flavor or subsuite as running
specific directories will do it implicitly. For example running:
```
./mach mochitest devtools/
```
Is a rough equivalent to running the `devtools` subsuite. There might be
situations where you might want to run tests that don't belong to any subsuite.
To do this, use:
```
./mach mochitest --subsuite default
```
## Debugging individual tests
If you need to debug an individual test, you could reload the page containing
the test with the debugger attached. If attaching a debugger before the problem
shows up is hard (for example, if the browser crashes as the test is loading),
you can specify a debugger when you run mochitest:
```
./mach mochitest --debugger=gdb ...
```
See also the `--debugger-args` and `--debugger-interactive` arguments. You can
also use the `--jsdebugger` argument to debug JavaScript.
## Finding errors
Search for the string `TEST-UNEXPECTED-FAIL` to find unexpected failures. You
can also search for `SimpleTest FINISHED` to see the final test summary.
## Logging results
The output from a test run can be sent to the console and/or a file (by default
the results are only displayed in the browser). There are several levels of
detail to choose from. The levels are `DEBUG`, `INFO`, `WARNING`, `ERROR` and
`CRITICAL`, where `DEBUG` produces the highest detail (everything), and
`CRITICAL` produces the least.
Mochitest uses structured logging. This means that you can use a set of command
line arguments to configure the log output. To log to stdout using the mach
formatter and log to a file in JSON format, you can use `--log-mach=-`
`--log-raw=mochitest.log`. By default the file logging level for all your
formatters is `INFO` but you can change this using `--log-mach-level=<level>`.
To turn on logging to the console use `--console-level=<level>`.
For example, to log test run output with the default (tbpl) formatter to the
file `~/mochitest.log` at `DEBUG` level detail you would use:
```
./mach mochitest --log-tbpl=~/mochitest.log --log-tbpl-level=DEBUG
```
## Headless mode
The tests must run in a focused window, which effectively prevents any other
user activity on the engaged computer. You can avoid this by using the
`--headless` argument or `MOZ_HEADLESS=1` environment variable.
```
./mach mochitest --headless ...
```
## Writing tests
A Mochitest plain test is simply an HTML or XHTML file that contains some
JavaScript to test for some condition.
### Asynchronous Tests
Sometimes tests involve asynchronous patterns, such as waiting for events or
observers. In these cases, you need to use `add_task`:
```js
add_task(async function my_test() {
let keypress = new Promise(...);
// .. simulate keypress
await keypress;
// .. run test
});
```
Or alternatively, manually call `waitForExplicitFinish` and `finish`:
```js
SimpleTest.waitForExplicitFinish();
addEventListener("keypress", function() {
// ... run test ...
SimpleTest.finish();
}, false);
// ... simulate key press ...
```
If you need more time, `requestLongerTimeout(number)` can be quite useful.
`requestLongerTimeout()` takes an integer factor that is a multiplier for the
default 45 seconds timeout. So a factor of 2 means: "Wait for at last 90s
(2*45s)". This is really useful if you want to pause execution to do a little
debugging.
### Test functions
Each test must contain some JavaScript that will run and tell Mochitest whether
the test has passed or failed. `SimpleTest.js` provides a number of functions
for the test to use, to communicate the results back to Mochitest. These
include:
* `ok(expressionThatShouldBeTrue, "Description of the check")` -- tests a value for its truthfulness
* `is(actualValue, expectedValue, "Description of the check")` -- compares two values (using Object.is)
* `isnot(actualValue, unexpectedValue, "Description of the check")` -- opposite of is()
If you want to include a test for something that currently fails, don't just
comment it out! Instead, use one of the "todo" equivalents so we notice if it
suddenly starts passing (at which point the test can be re-enabled):
* `todo(falseButShouldBeTrue, "Description of the check")`
* `todo_is(actualValue, expectedValue, "Description of the check")`
* `todo_isnot(actualValue, unexpectedValue, "Description of the check")`
Tests can call a function `info("Message string")` to write a message to the
test log.
In addition to mochitest assertions, mochitest supports the
[CommonJS standard assertions](http://wiki.commonjs.org/wiki/Unit_Testing/1.1),
like [nodejs' assert module](https://nodejs.org/api/assert.html#assert) but
implemented in `Assert.jsm`. These are auto-imported in the browser flavor, but
need to be imported manually in other flavors.
### Helper functions
Right now, useful helpers derived from MochiKit are available in
[`testing/mochitest/tests/SimpleTest/SimpleTest.js`](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js).
Although all of Mochikit is available at `testing/mochitest/MochiKit`, only
include files that you require to minimize test load times. Bug 367569 added
`sendChar`, `sendKey`, and `sendString` helpers. These are available in
`testing/mochitest/tests/SimpleTest/EventUtils.js`.
If you need to access some data files from your Mochitest, you can get an URI
for them by using `SimpleTest.getTestFileURL("relative/path/to/data.file")`.
Then you can eventually fetch their content by using `XMLHttpRequest` or so.
### Adding tests to the tree
`mach addtest` is the preferred way to add a test to the tree:
```
./mach addtest --suite mochitest-{plain,chrome,browser-chrome} path/to/new/test
```
That will add the manifest entry to the relevant manifest (`mochitest.ini`,
`chrome.ini`, etc. depending on the flavor) to tell the build system about your
new test, as well as creating the file based on a template.
```ini
[test_new_feature.html]
```
Optionally, you can specify metadata for your test, like whether to skip the
test on certain platforms:
```ini
[test_new_feature.html]
skip-if = os == 'win'
```
The [mochitest.ini format](/build/buildsystem/test_manifests.rst), which is
recognized by the parser, defines a long list of metadata.
### Adding a new mochitest.ini or chrome.ini file
If a `mochitest.ini` or `chrome.ini` file does not exist in the test directory
where you want to add a test, add them and update the moz.build file in the
directory for your test. For example, in `gfx/layers/moz.build`, we add
these two manifest files:
```python
MOCHITEST_MANIFESTS += ['apz/test/mochitest.ini']
MOCHITEST_CHROME_MANIFESTS += ['apz/test/chrome.ini']
```
<!-- TODO: This might be outdated.*
## Getting Stack Traces
To get stack when Mochitest crashes:
* Get a minidump_stackwalk binary for your platform from http://hg.mozilla.org/build/tools/file/tip/breakpad/
* Set the MINIDUMP_STACKWALK environment variable to point to the absolute path of the binary.
If the resulting stack trace doesn't have line numbers, run `mach buildsymbols`
to generate the requisite symbol files.
-->
## FAQ
See the [Mochitest FAQ page](faq.md) for other features and such that you may
want to use, such as SSL-enabled tests, custom http headers, async tests, leak
debugging, prefs...

3
testing/web-platform/docs/index.rst
View File

@ -112,7 +112,8 @@ Under ``testing/web-platform`` are the following directories:
``mozilla/tests``
Tests that will not be upstreamed and may
make use of Mozilla-specific features.
make use of Mozilla-specific features. They can access
the ``SpecialPowers`` APIs.
``mozilla/meta``
Metadata for the Mozilla-specific tests.