Commit Graph
55 Commits
Author SHA1 Message Date
abhishekmadan30 11291923e8 Bug 2016843 - Updated taskcluster version for test_manifests and test_testing. r=releng-reviewers,mach-reviewers,bhearsum,ahochheiden
Differential Revision: https://phabricator.services.mozilla.com/D283617
2026-02-18 04:31:12 +00:00
Tom Marble eb8e7f929b Bug 2000007 - Move mach manifest tools to testing/manifest (bis) r=ahal
Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D273654
2025-11-21 21:46:21 +00:00
Sandor Molnar 8ac7e8c9ad Revert "Bug 2000007 - Move mach manifest tools to testing/manifest r=ahal,mach-reviewers" for causing spider-monkey failures
This reverts commit c48f4b18bd.
2025-11-20 20:55:49 +00:00
Tom Marble c48f4b18bd Bug 2000007 - Move mach manifest tools to testing/manifest r=ahal,mach-reviewers
Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D272540
2025-11-20 20:05:35 +00:00
Tom Marble 5d5d1047f8 Bug 1991977 - Implement skip-fails --replace-tbd r=ahal
Updates skipfails.py to support --replace-tbd
- Improves on --known-intermittents and --new-failures modes by
  recognizing previously handled failures
- Caches error log context (to support self tests)
- Consistently reports when querying treeherder ("Retrieving...")
  in verbose mode
- Updates write_tasks to support writing Mock'ed tasks
- Updates write_json to ensure a final newline (to match the JSON linter)

Adds test_modes.py to test all the greening up modes

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D271085
2025-11-04 21:52:32 +00:00
Tom Marble 1da2f2c00f Bug 1997188 - resolve python-test dependencies r=ahal,mach-reviewers
Updated python/mach/mach/site.py to better report dependency failures
Updated dependencies for `./mach python-test --subsuite skip-fails`
  in testing/test/test_skipfails.in which includes bumping
  the version of typing-extensions of due to pydantic
Bumped the vendored version of typing-extensions to 4.15

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D270715
2025-10-30 21:44:41 +00:00
Tom Marble 2f890b64ba Bug 1991973 - Implement skip-fails --new-failures r=jmaher
Added new flags for mach manifest skip-fails:
  --new-failures
  --replace-tbd  (full functionality still WIP)
Improvements to skipfails.py
  Added more type hinting
  Added SkipfailsMode helper class for different modes
  Added Action class for deferred actions
  More performance optimizations (caching treeherder queries)
  Bug comments now include +/- 10 lines of the error log
Improved add_skip_if in toml.py by returning status when
  not adding redundant conditions
Updated design document SKIP-FAILS.txt

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D269302
2025-10-21 22:17:33 +00:00
Tom Marble 4424263193 Bug 1991971 - Implement skip-fails --known-intermittents r=jmaher
Enhanced mach manifest skip-fails to support --known-intermittents
  where bugzilla won't be updated and skip-if conditions will
  only be added to manifests for failures that are known intermittents

Documented --known-intermittents mode in SKIP-FAILS.txt

Fixed handling of included manifests from mozci task info
  by making sure the _included_ manifest is the one listed
  in the failure and the test path is recorded without the
  include manifest prefix

Fixed handling of a failure to ensure that the task_id
  used to create the skip-if condition is from one of
  the failing tasks

Removed use of FailedPlatform in TOML (non high-freq) manifests
  as it unnecessarily included some unused, negated variants
  in skip-if conidtions

Fixed handling of multiple runtimes in mozinfo.test_variant

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D268463
2025-10-15 21:21:30 +00:00
Tom Marble 594fad536a Bug 1991969 - Implement mach manifest skip-fails cache r=jmaher
Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D267340
2025-10-06 18:28:01 +00:00
Tom Marble dc2fe06115 Bug 1971610 - skipfails carryover comments from previous platform r=jmaher
Adds arguments to ./mach skip-fails
  -c, --carryover       Set carryover mode (only skip failures for platform
                        matches)
  -r, --failure-ratio FAILURE_RATIO
                        Ratio of test failures/total to skip [0.4]

Ensures that only carryover conditions are considered in --carryover mode

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D265321
2025-09-25 18:32:57 +00:00
Alexandru Marc 052367bd68 Revert "Bug 1971610 - skipfails carryover comments from previous platform r=jmaher" for causing dt failures
This reverts commit e1cc362579.
2025-09-25 03:37:54 +00:00
Tom Marble e1cc362579 Bug 1971610 - skipfails carryover comments from previous platform r=jmaher
Adds arguments to ./mach skip-fails
  -c, --carryover       Set carryover mode (only skip failures for platform
                        matches)
  -r, --failure-ratio FAILURE_RATIO
                        Ratio of test failures/total to skip [0.4]

Ensures that only carryover conditions are considered in --carryover mode

Signed-off-by: Tom Marble <[email protected]>

Differential Revision: https://phabricator.services.mozilla.com/D265321
2025-09-24 23:25:24 +00:00
Joel Maher c9669a84dc Bug 1982728 - Fix skipfails edge case with known intermittent tests. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D260903
2025-08-13 18:08:36 +00:00
Julien Cristau 582c6b4f19 Bug 1976273 - update test_skipfails dependencies for compatibility with python 3.13. r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D256486
2025-07-10 07:45:00 +00:00
Brian Grinstead 93d970ac5b Bug 1976132 - Update mach intermittents to handle wpt paths;r=jmaher
Cut/paste some code from testing/skipfails.py so it can be used without side effects
and tested more easily.

Differential Revision: https://phabricator.services.mozilla.com/D256477
2025-07-09 19:11:55 +00:00
Joel Maher a2bc75a972 Bug 1975610 - fixes to skip fails. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D256195
2025-07-07 18:51:09 +00:00
Joel Maher fb4d9970de Bug 1973753 - cleanup skipfails and wpt/mochitest harnesses for supporting android emulator. r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D254846
2025-06-26 01:16:55 +00:00
Arnaud Vergnet 031aeb3c1b Bug 1953958: Add mach command to disable high frequency test failures r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D252328
2025-06-03 13:19:38 +00:00
Alexandru Marc 61352170e6 Revert "Bug 1953958: Add mach command to disable high frequency test failures r=jmaher" for causing py3 failures
This reverts commit b0cecf8bfb.
2025-06-02 14:28:46 +00:00
Arnaud Vergnet b0cecf8bfb Bug 1953958: Add mach command to disable high frequency test failures r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D252036
2025-06-02 11:55:28 +00:00
Alexandru Marc d928219c12 Revert "Bug 1953958: Add mach command to disable high frequency test failures r=jmaher" for causing js-bench-sm failures complaining about markdown2 module
This reverts commit b104c617d4.
2025-05-30 17:35:13 +00:00
Arnaud Vergnet b104c617d4 Bug 1953958: Add mach command to disable high frequency test failures r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D251131
2025-05-30 12:49:56 +00:00
Joel Maher e994ac9cb8 Bug 1968729 - green up crashtest and reftest on ubuntu 24.04 x11 docker image. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D251360
2025-05-28 15:11:12 +00:00
Cristian Tuns 35de665d8f Revert "Bug 1968729 - green up crashtest and reftest on ubuntu 24.04 x11 docker image. r=aryx" for causing multiple reftest failures
This reverts commit 85cbc71c7d.
2025-05-27 23:36:01 +00:00
Joel Maher 85cbc71c7d Bug 1968729 - green up crashtest and reftest on ubuntu 24.04 x11 docker image. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D251360
2025-05-27 21:48:40 +00:00
Alex Hochheiden 9e91a1f021 Bug 1715287 - [lint] Enable remaining pyupgrade rules and lint tree r=linter-reviewers,webdriver-reviewers,translations-reviewers,whimboo,sylvestre
Differential Revision: https://phabricator.services.mozilla.com/D245320
2025-04-13 17:48:23 +00:00
Alex Hochheiden c1e140fe43 Bug 1715287 - [lint] Fix pyupgrade lint errors from new configuration r=linter-reviewers,webdriver-reviewers,taskgraph-reviewers,ahal,whimboo
Differential Revision: https://phabricator.services.mozilla.com/D245269
2025-04-13 17:48:23 +00:00
Joel Maher c972cfbd78 Bug 1957771 - fix skip-fails edge cases. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D244259
2025-04-03 12:50:59 +00:00
Arnaud Vergnet 5e518728f0 Bug 1949711: Add macosx 11.20 rosetta hack r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D239511
2025-03-06 14:12:47 +00:00
Arnaud Vergnet fb5a92e3a7 Bug 1949711: Convert macosx version 11.00 to 11.20 for skipfails r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D239510
2025-03-06 14:12:47 +00:00
Arnaud Vergnet 92bb5adbbb Bug 1949711: Do not optimize skip-if conditions if no data available in test-run-matrix r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D239509
2025-03-06 14:12:46 +00:00
Arnaud Vergnet 52ac95adee Bug 1949711: Fix test_skipfails input r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D239507
2025-03-06 14:12:45 +00:00
Arnaud Vergnet 539daaf130 Bug 1938934 - failedplatform: fix tests r=jmaher,aryx
Patch https://phabricator.services.mozilla.com/D234740 changed the input
data type but tests were not updated, giving typing errors

Differential Revision: https://phabricator.services.mozilla.com/D235578
2025-01-27 16:45:12 +00:00
Arnaud Vergnet 4689c3547c Bug 1938934 - skipfails: share platform handling logic r=jmaher,taskgraph-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D233474
2025-01-27 16:45:11 +00:00
Joel Maher 699bf5c87f Bug 1941654 - Make skipfails adjustments to support .list files more reliably. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D234775
2025-01-20 01:43:57 +00:00
Alexandru Marc d1875bbf93 Backed out 2 changesets (bug 1938934) for causing py3 fails
Backed out changeset 6709e35f6d82 (bug 1938934)
Backed out changeset 2abd783a7515 (bug 1938934)
2025-01-15 22:49:13 +02:00
Arnaud Vergnet 38a32d4548 Bug 1938934 - skipfails: share platform handling logic r=jmaher,taskgraph-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D233474
2025-01-15 16:34:03 +00:00
Arnaud Vergnet f78b6aaf4f Bug 1935017 - skipfails: fix crash on wpt and list manifests r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D232611
2024-12-19 18:18:57 +00:00
Arnaud Vergnet 348cc2caaa Bug 1930106 - skipfails: handle combined test variants r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D231934
2024-12-13 21:12:50 +00:00
Joel Maher d000ab2575 Bug 1930280 - use proper test path (for .list manifests) when querying bugzilla component information. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D231872
2024-12-12 13:10:03 +00:00
Arnaud Vergnet 9a4cc22493 Bug 1930106 - skipfails: use test file path to differentiate tests r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D231272
2024-12-05 16:07:43 +00:00
Arnaud Vergnet 21ea3b043d Bug 1930106 - skipfails: include build types and test variants r=jmaher
Tries to simplify the conditions as much as possible by checking the
possible permutations.

Differential Revision: https://phabricator.services.mozilla.com/D229584
2024-12-04 16:33:15 +00:00
Joel Maher 2b39d06b6c Bug 1935017 - fix skipfails.py to track platforms/configs/paths better. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D231013
2024-12-04 05:21:30 +00:00
Joel Maher 26d8b53bfb Bug 1920961 - remove line number tests from skip-fails. r=aryx
Differential Revision: https://phabricator.services.mozilla.com/D224166
2024-10-01 11:23:27 +00:00
Timothy Nikkel e0933122d8 Bug 1919368. When constructing surface pipes for images downscaling must come after premultiplying alpha. r=gfx-reviewers,nical
The downscaling filter depends on operating on premultiplied alpha data.

Differential Revision: https://phabricator.services.mozilla.com/D222597
2024-09-25 11:56:04 +00:00
Tom Marble 0a1353ab66 Bug 1892075 - mach manifest skip-fails support for reftest r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D215390
2024-07-11 17:35:45 +00:00
Tom Marble 4c1edeefb9 Bug 1880571 - mach manifest skip-fails: initial WPT support r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D203038
2024-03-18 18:13:55 +00:00
Tom Marble 4d12d5031c Bug 1879290 - fixes test_skipfails.py::test_get_failures_5 r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D201099
2024-02-08 20:22:04 +00:00
Tom Marble 9862fcd8e7 Bug 1877537 - Update tomlkit to 0.12.3 and mozci to 2.4.0 r=jmaher,mach-reviewers,ahal
Differential Revision: https://phabricator.services.mozilla.com/D200114
2024-01-31 18:57:03 +00:00
Tom Marble 923cca501a Bug 1869639 - mach manifest skip-fails: disable long running manifests r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D199777
2024-01-26 22:10:45 +00:00