mirror of
https://github.com/mozilla/gecko-dev.git
synced 2025-02-17 14:25:49 +00:00
125 lines
5.3 KiB
Plaintext
125 lines
5.3 KiB
Plaintext
Design Specification for ./mach manifest skip-fails
|
|
===================================================
|
|
|
|
The first of the mach manifest subcommands is skip-fails. This command
|
|
can be used to automatically edit manifests to skip tests that are
|
|
failing as well as file the corresponding bugs for the failures. This
|
|
is particularly useful when “greening up” a new platform.
|
|
|
|
The user documentation for skip-fails is here:
|
|
https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html#using-mach-manifest-skip-fails
|
|
|
|
The skip-fails command knows how to manage failures for TOML, WPT
|
|
and REFTEST manifests. Additional developer design notes are included here.
|
|
|
|
|
|
Running skip-fails for TOML
|
|
---------------------------
|
|
|
|
The user documentation for TOML manifests is:
|
|
https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html
|
|
|
|
1. Classify the failure based on the following:
|
|
a. If less than 3 runs, classify "unknown"
|
|
b. If zero failures, classify "success"
|
|
c. If the ratio of failed runs / total runs is less than 40%
|
|
classify "intermittent"
|
|
d. Else edit the manifest to skip this failure
|
|
|
|
As design changes for skip-fails / TOML, notes will be added here.
|
|
|
|
Running skip-fails for WPT
|
|
---------------------------
|
|
|
|
The user documentation for WPT manifests is:
|
|
https://firefox-source-docs.mozilla.org/web-platform/index.html
|
|
|
|
1. Classify the failure based on the following:
|
|
a. If less than 3 runs, classify "unknown"
|
|
b. If zero failures, classify "success"
|
|
c. If the ratio of failed runs / total runs is less than 40%
|
|
classify "intermittent"
|
|
d. Else edit the manifest to skip this failure
|
|
|
|
As design changes for skip-fails / WPT, notes will be added here.
|
|
|
|
|
|
Running skip-fails for REFTEST
|
|
------------------------------
|
|
|
|
The user documentation for REFTEST manifests is:
|
|
https://firefox-source-docs.mozilla.org/layout/Reftest.html
|
|
|
|
Current Design notes for skip-fails / REFTEST:
|
|
|
|
As skip-fails for REFTEST finds a failure it will take the following actions:
|
|
|
|
1. Classify the failure based on the following:
|
|
a. If less than 3 runs, classify "unknown"
|
|
b. If zero failures, classify "success"
|
|
c. If the ratio of failed runs / total runs is less than 40%
|
|
classify "low frequency intermittent" (INTERMITTENT)
|
|
c. If the ratio of failed runs / total runs is less than 80%
|
|
classify "high frequency intermittent" (DISABLE_INTERMITTENT)
|
|
d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE)
|
|
|
|
2. Propose a new range (differences, pixels)
|
|
a. For "low frequency intermittent" errors do nothing
|
|
b. For "high frequency intermittent" set the range 0-Y (i.e. always
|
|
start at zero to avoid UNEXPECTED PASS)
|
|
c. For "failure" set the range X-Y (i.e. the range as reported in
|
|
reftest_errorsummary.log). If the failure is UNEXPECTED PASS
|
|
then set the range to 0-Y.
|
|
d. NOTE: The maximum for differences and pixels will be increased
|
|
by 5% over the reported values in reftest_errorsummary.log
|
|
e. NOTE: If for whatever reason the range is calculated to be 0-0 then
|
|
remove the fuzzy-if entirely
|
|
|
|
3. Merge with an existing fuzzy-if for this OS platform if the other
|
|
criteria differ in only one dimension. Current dimensions include:
|
|
|
|
OS: gtkWidget, winWidget, cocoaWidget, Android
|
|
build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer
|
|
is64Bit
|
|
fission
|
|
useDrawSnapshot
|
|
swgl
|
|
<OTHER>
|
|
|
|
4. Based on a command line switch choose do use implicit variables (or not).
|
|
a. The following variables can implicit values:
|
|
is64Bit: true
|
|
fission: true
|
|
useDrawSnapshot: false
|
|
swgl: false
|
|
If the value of an implicit variable matches it's default then it may be
|
|
elided from the conditional expression.
|
|
If the value of an implicit variable is the oppposite of it's default
|
|
value then it may be specified in the condition, e.g. !is64Bit
|
|
When merging an existing condition with a new failure condition where
|
|
the one dimension of difference is an implicit variable then the
|
|
condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit)
|
|
That is essential so that the reftest sandbox does not add the implicit
|
|
variable and cover only one of the two possible values.
|
|
b. When implicit variables are not used then the following heuristics apply:
|
|
The variable is64Bit will be elided for all platforms except for winWidget.
|
|
The variable useDrawSnapshot will be elided for all platforms except gtkWidget.
|
|
|
|
5. Position of fuzzy-if conditions
|
|
In reftest manifest the rightmost applicable condition is used.
|
|
|
|
a. First sorting criterion is based on OS in this order:
|
|
OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"]
|
|
This is because appleSilicon is a narrower specification than
|
|
cocoaWidget and would be ignored if not to the right of cocoaWidget.
|
|
|
|
b. The second sorting criterion is the number of dimensions that participate
|
|
in the expression (including implicits) from least dimensions to
|
|
greatest dimensions.
|
|
|
|
The last fuzzy-if will be placed on the line immediately to the left of the
|
|
"type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present).
|
|
Non fuzzy-if statements will appear before the fuzzy-if statements.
|
|
In this way the more generic conditions will only be superceded by very specific
|
|
conditions.
|