Commit Graph
104 Commits
Author SHA1 Message Date
Elinor Fung 78083e7dd0 Add test for comhost with managed host (#66360) 2022-03-09 20:52:18 -08:00
Elinor Fung 627851323f Add tests for ijwhost (#66308) 2022-03-08 09:14:43 -08:00
Aaron Robinson 6fe1a80959 Use CMakeProjectReference instead of ProjectReference (#66202)
* Use CMakeProjectReference instead of ProjectReference

Changing this permits usage of "\runtime\dotnet.cmd build" as opposed
  to forcing users to use "\runtime\dotnet.cmd msbuild".
2022-03-04 19:40:27 -08:00
Marek Fišera 1ac45f5161 [wasm] Fix WebSocket tests on NodeJS (#64330)
- Move all SetScriptCommands to a target, so they are placed after scenario in the generated script.
- Add WasmXHarnessArgsCli as an alternative variable that can be used from cli.
- Fix NodeJS symbolic links on Helix.
- Copy TestEchoMiddleware and RemoteLoopMiddleware even for Scenario=NodeJS.
- Enable System.Net.WebSockets.Client.Tests on NodeJS.
2022-02-09 10:27:24 +01:00
Maxim Lipnin 0b17270554 Update the apple testing doc (#63119) 2021-12-24 15:33:24 +03:00
Tomáš Rylek c3683b06b0 Rename 'excludemonofailures' to 'mono' (#62749) 2021-12-14 07:38:55 +01:00
Nathan Ricci 1d4b5f684c Updated testing document; patching and various mono-specific msbuild targets are no longer needed. (#62588) 2021-12-09 13:40:03 -05:00
Simon RozsivalandAlexander Köplinger 6cf785d9fb Support Process.Start() on MacCatalyst (#61507)
* Modify existing tests

* Enable process start and kill on MacCatalyst

* Typo

* Temporarily enable ProcessTests for MacCatalyst

* Temporarily enable ProcessTests for MacCatalyst - attempt 2

* Allow running Mac Catalyst builds in App Sandbox

* Allow enabling app sandbox for the other Mac Catalyst sample

* Add missing parameter to XCode project generator

* Remove unnecessary MacCatalyst detection

* Enable App Sandbox for Mac Catalyst tests

* Create a separate test branch for App Sandbox

* Remove the restriction to enable app sandbox just for Mac Catalyst apps

* Do not throw PNSE for Mac Catalyst

* Clean-up platform-specific conditions

* Build correct Process implementation for MacCatalyst

* Try to get more information from CI for further investigation of the failing build

* Revert "Try to get more information from CI for further investigation of the failing build"

This reverts commit fc63a370d688aa0d1dc890e6928a8d031d148e56.

* Add MacCatalyst target framework for System.Diagnostics.Process

* Add supported platform annotations for Mac Catalyst

* Fix annotations

* Remove incorrectly placed attributes

* Update attributes including the reference file

* Change platform attributes

* Generate app.entitlements in the BuildAppleAppBundles test build target

* Revert "Generate app.entitlements in the BuildAppleAppBundles test build target"

This reverts commit abbe2241586e3469c3d3e9d4ecd2cea2e09d3641.

* Enable AppSandbox when generating CMakeLists.txt for libraries tests

* Try implementing a workaround for app sandbox mode

* Fix app sandbox detection bug

* Add explanation comment

* Enable more tests for MacCatalyst

* Add apple app builder input validation

* Add Mac Catalyst w/ App Sandbox enabled to runtime-manual test pipeline

* Enable networking in App Sandbox mode

* Skip test which won't work on Mac Catalyst

* Skip some tests in app sandbox mode

* Update src/tasks/AppleAppBuilder/AppleAppBuilder.cs

Co-authored-by: Alexander Köplinger <[email protected]>

* Add a clarification comment for a networking entitlement

* Simplify supported platform condition

* Remove temporarily enabled test

* Remove unnecessary attributes

* Replace checking env variable with checking errno in libproc

* Update docs

* Remove unnecessary changes

* Temporarily enable running System.Diagnostics.Process.Tests for this PR

This reverts commit 02d370c2cf730bf67a3cf6fca8fcb8c5e4ff1ccd.

* Revert "Temporarily enable running System.Diagnostics.Process.Tests for this PR"

This reverts commit dc72f0f1d068314317d088d33ec87f9aa6447854.

* Fix job suffix in runtime-manual

* Remove attributes

* Revert "Remove attributes"

This reverts commit 704e9fa73cd40854d1becc2889569bab3c47d62d.

* Try changing the order of attributes to please CI

* Update ApiCompat baseline

* ApiCompat step 1: remove all attributes for MaxWorkingSet setter

* Revert "Update ApiCompat baseline"

This reverts commit a2ad03207c5f9f40b9dd2c335a1b0f31c37a0ac5.

* Revert "ApiCompat step 1: remove all attributes for MaxWorkingSet setter"

This reverts commit be72a3d2564afb89a83198ef010b3d4622799601.

* Update ApiCompat baseline

* Update ApiCompat baseline after pulling upstream main

* Remove trailing whitespace

Co-authored-by: Alexander Köplinger <[email protected]>
2021-12-07 20:33:38 +01:00
Jeremy KoritzinskyandAaron Robinson 7099f4b12a Use reflection or a compile-time only shim assembly to reference unexposed corelib types. (#61802)
Co-authored-by: Aaron Robinson <[email protected]>
2021-11-19 19:28:42 -08:00
Maxim Lipnin b9a2c48aa8 Clean up the documentation for the library testing (#59401) 2021-11-08 12:11:54 +03:00
Aleksey Kliger (λgeek) 72bcf2f77a [tests] In test wrapper scripts if -debug is gdb or lldb, add separator before corerun and args (#60416)
* [tests] If -debug is LLDB, add separator before corerun and args

   For runtime test shell scripts, if a debugger is specified with `-debug=/usr/bin/lddb` or the like, add a separator so that `lldb` doesn't try to interpret `-p` as a PID argument.

   Also update mono workflow doc

* Also support gdb as the debugger
2021-10-20 10:17:54 -04:00
Aleksey Kliger (λgeek) 10d8964fb0 [docs] Add LLDB debugging instructions for Mono running a runtime test (#60410)
* [docs] Add LLDB debugging instructions for Mono running a runtime test

* Also add a note about building a single test
2021-10-14 12:54:31 -04:00
Aleksey Kliger (λgeek) 89cea2e659 [docs] Add instructions for debugging native runtime code on Android (#59183) 2021-09-17 09:01:13 -04:00
Viktor Hofer 67354a2fb4 Upgrade to net7.0 and remove target frameworks (#58011)
* Remove EOL tfms and upgrade to net7.0

* Make linker tests net7 aware

Make installer tests run on net7.0

Don't attempt to use live apphost

Fix payloads

Regenerate test files

Fix package testing
2021-09-08 22:04:23 +02:00
Jeremy KoritzinskyandAaron Robinson f3c705ef29 Implement support for passing a .env file to corerun to easily set environment variables for stress testing. (#58222)
* Implement support for passing a .env file to corerun to easily set environment variables for stress testing.

* Load the parsed dotenv into the process.

* Implement support for # comments

* Clean up code

* Avoid copy

* PR feedback

* Update docs and bash/batch scripts to support the dotenv files.

* Use what clang recommends to not copy

* Apply suggestions from code review

Co-authored-by: Aaron Robinson <[email protected]>

Co-authored-by: Aaron Robinson <[email protected]>
2021-09-03 02:18:59 +00:00
Aleksey Kliger (λgeek) d033e1a902 Don't build ReferenceSystemPrivateCoreLib tests with Mono (#58232)
New tests that extend the System.Private.CoreLib API should add to the ref assemblies in src/libraries and the tests themselves should be library tests.

Updated the workflow doc: the Clr.Native subset is all that is need if we're interested in testing Mono.

Contributes to https://github.com/dotnet/runtime/issues/58266

* [tests] Use RuntimeFlavor to build ReferenceSystemPrivateCoreLib tests

   When building tests that reference System.Private.CoreLib, use the one from the specified RuntimeFlavor (or fall back to coreclr if it's unset - although src/tests/build.sh sets it always)

* Update docs: We only need to build the Clr.Native subset if we're interested in testing Mono.

* flavor the pipelines to build the runtime-specific System.Private.CoreLib for the runtime tests.

* Don't build ReferenceSystemPrivateCoreLib tests with Mono

   We should avoid adding new ReferenceSystemPrivateCoreLib tests
2021-08-30 12:11:42 -04:00
Vitek Karas 0042240dd8 Add zip tool to the install list for Android tests (#57569)
On default WSL Ubuntu `zip` is not installed, and trying to build Android tests fails with rather cryptic errors.
2021-08-17 08:00:48 -07:00
Tomáš Rylek 3b40c7cfc4 Add support for building just a subset of runtime tests (#57142)
This change adds four new options to the runtime test build command:

1) test <test project path relative to src\tests> - build just that
one test.

2) dir <directory relative to src\tests> - build all tests in the
directory.

3) tree <directory relative to src\tests> - build all tests in the
given subtree.

4) all - use clean rebuild (i.e. don't apply incrementalism)
when building the tests.

Thanks

Tomas
2021-08-12 15:54:17 +02:00
Fan YangandJuan Hoyos e0ba450d7f [Mono] Update instructions for running runtime tests on Android (#57166)
* Update instructions for running runtime tests on Android

* Update docs/workflow/testing/mono/testing.md

Co-authored-by: Juan Hoyos <[email protected]>

Co-authored-by: Juan Hoyos <[email protected]>
2021-08-11 12:13:29 -04:00
Jeremy Koritzinsky 3dc99e900b Build C++/CLI tests as .NET Core C++/CLI (#56502)
* Build IJW assemblies against LKG SDK ref pack.

* Allow building against the live ref assemblies by passing `-cmakeargs -DCPP_CLI_LIVE_REF_ASSEMBLIES=1` to the test build script.

* Add test instructions

* Error handling

* Add support for the stdcall-mangled _CorDllMain entrypoint on x86 since MSVC uses it if it exists.

* Write out ref-pack path.
2021-08-02 16:53:48 +00:00
Ankit JainandLarry Ewing d953229d54 [wasm] Run Wasm.Build.Tests against workloads (#54451)
Co-authored-by: Larry Ewing <[email protected]>
2021-07-14 10:47:03 -04:00
Jeff Handley 69d67390e5 Fix the daily build links in docs (#55443)
* Fix the daily build links in docs

* Update the NuGet feed referenced
2021-07-09 22:32:23 -04:00
Noah FalkandViktor Hofer d783a8c812 Update library testing docs page to reduce confusion (#54324)
* Add warning on unmaintained testing doc page

* Update testing.md

Some example text that seems more clear to me, but only offered as a suggestion. 
Feel free to adjust it or if you want to use it as-is please double check what I wrote is accurate : )

I think the useful elements are:
1. Being explicit about what workflow steps need to happen in total
2. Being explicit about which commands are covering the entire workflow and which ones are only covering a part of it
3. Show the simple "do-it-all" options first before showing more complex partial options. Glancing at the first example and blindly copying it should land in the pit of success.

Co-authored-by: Viktor Hofer <[email protected]>
2021-06-23 09:00:47 +02:00
Pavel Savara fb61be479e [wasm][testing] hosting echo server in xharness process (#52923)
- move tests to inner loop
- include echo middleware in xharness server
- improve doc
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53592 for lack of TRACE
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53591 for content on GET/HEAD
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53874 for HttpRequestMessage.Headers.Host
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53872 for NPE on System.Net.Http.BrowserHttpHandler
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53876
- include middleware in Helix correlation payload
2021-06-09 17:32:58 +02:00
b443b8ccce Update testing docs with x86 instructions (#48715)
* Update testing docs with x86 instructions

* Apply suggestions from code review

Co-authored-by: Santiago Fernandez Madero <[email protected]>

* add more examples

* Apply suggestions from code review

Co-authored-by: Dan Moseley <[email protected]>

* Update docs/workflow/testing/libraries/testing.md

Co-authored-by: Santiago Fernandez Madero <[email protected]>

Co-authored-by: Santiago Fernandez Madero <[email protected]>
Co-authored-by: Dan Moseley <[email protected]>
Co-authored-by: Eric StJohn <[email protected]>
2021-05-24 11:45:21 -07:00
Aaron RobinsonandElinor Fung 66abf41f1e corerun supports user defined properties from command line. (#51774)
* corerun supports user defined properties from command line.

* Update workflow help doc for corerun.

Co-authored-by: Elinor Fung <[email protected]>
2021-04-26 20:46:55 -07:00
6ca3d9105d Shift Most of Wasm AOT test build to helix (#48226)
Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

## This is done by:
1. building the test assemblies on the build machine
   - the wasm part of the build is not executed on the build machine,
      because it has the AOT build part
2. Zip up the test assembly+friends, *and* any bits required to run the wasm
    app build for that on helix (eg. emsdk, wasm app targets, cross compiler etc)
3. Send all this to helix, and use a custom `aot-build.proj`
     - which recreates all the build inputs for the `WasmBuildApp` target
        using the paths for the assets on helix
     - then we can run `WasmBuildApp` for the build, resulting in a wasm app
       bundle.
4. Run the tests!

- We already have the bits required for building wasm apps on helix, supported
for `Wasm.Build.Tests`, which we can use here too.

## Trimming:

- Since, AOT can be so expensive, we use `EnableAggressiveTrimming=true`(EAT), but
   that means that we could have issues due to trimming.
- And it can sometimes be unclear whether the build/test failures are due to trimming
     or AOT.

- Because these builds+test runs are different from other builds, owing to the
   "build partially on helix" step, a normal EAT build would not be the same as 
- to help with testing this, we add two lanes to `runtime-staging`:
   - `*_Mono_AOT`: builds AOT+EAT on helix
   - `*_Mono_EAT`: builds EAT, on helix
      - this is required because we want to run almost the same kinda
         build: 1. build test assembly; *2. send to helix; 3. build wasm app;* 4. run tests

- This should effectively mean that we can see which errors might be due to EAT, and
   which are clearly because of EAT+AOT.

Co-authored-by: Steve Pfister <[email protected]>
Co-authored-by: Marek Safar <[email protected]>
Co-authored-by: Mitchell Hwang <[email protected]>
Co-authored-by: Ankit Jain <[email protected]>
2021-04-21 20:58:25 -04:00
Elinor Fung 189829cbf6 Use simpler command in example for building individual coreclr test (#51364)
* Use simpler command in example for building individual coreclr test

* Remove comment about using 'dotnet msbuild'
2021-04-16 10:56:28 -07:00
Ankit Jain 8d102e3fc2 [wasm][tests] Enable line numbers in stack traces (#50893) 2021-04-12 13:01:25 -04:00
Alexander Köplinger bc9f00c226 Cleanup PlatformSpecific/SkipOnMono attributes that skip a platform (#50907)
We have a bunch of test assemblies that don't make sense on some platforms, e.g. Browser.
Right now we're skipping them via `[SkipOnMono("reason", TestPlatforms.Browser)]` but there's nothing that inherently ties this to Mono other than the current implementation.

The more generic `SkipOnPlatform` attribute can be used instead.
We can also use it in places where we do `[PlatformSpecific(~TestPlatforms....)]` to avoid the double inversion.
2021-04-09 13:20:18 +02:00
erhanatesoglu aa64d2e712 Fixed broken link (#50964)
The link to xunit on github has changed.
2021-04-08 17:56:01 -04:00
Elinor Fung 5ca4992150 Clarify instructions for building individual test with native dependency (#50869) 2021-04-07 13:59:03 -07:00
Ankit Jain 8a1101d2b5 [wasm][tests] Add properties to allow passing args to xharness (#50678)
* [wasm][tests] Add properties to allow passing args to xharness

New properties:
- `$(WasmXHarnessArgs)`
- `$(WasmXHarnessMonoArgs)`
- `$(WasmTestAppArgs)`

Xharness command line is built with these:

`dotnet xharness wasm test .. $(WasmXHarnessArgs) .. -- $(WasmXHarnessMonoArgs) .. --run .. $(WasmTestAppArgs)`

* [wasm][tests] extract wasm bits into tests.wasm.targets

* [wasm] set EMSDK_PATH for makefile run-tests- targets

* [wasm] Update target dependencies for wasm

* Don't overwrite RunScriptCommand, thus fixing wasm build tests

* Track property name changes
2021-04-07 13:06:55 -04:00
Ryan Lucia 240fcec9d8 Update iOS workflow docs to clarify simulator/device os split (#50366) 2021-04-05 07:55:20 -04:00
SingleAccretion af5eae3c87 Delete missed mono/netcore from the documentation (#50557) 2021-04-01 09:27:57 -04:00
Fan Yang 1ce7460578 Update testing-android.md (#50484) 2021-03-31 10:03:27 -07:00
Steve Pfister bc5b227c9f Remove openssl dependency from android (#49282) 2021-03-23 00:17:12 -04:00
Elinor Fung 5738f60553 Update coreclr test doc to call out LibrariesConfiguration (#50002)
Include build configuration in error message about missing artifacts
2021-03-22 18:02:26 -07:00
Elinor Fung a4c89d6b52 Clarify host testing instructions (#49797) 2021-03-18 09:30:37 -07:00
Steve MacLean e422345323 Apple Silicon platform detection (#49155)
* Add Apple Silicon PlatformDetection

* Disable OSVersion_ValidVersion_OSX on Apple Silicon

* Update docs
2021-03-04 17:28:07 -05:00
Viktor Hofer 3553284bea Rename mention to master branch to main (#48824) 2021-03-01 10:09:31 +01:00
Elinor Fung 9ce547d084 Add example for installing/launching emulator to Android test doc (#47889) 2021-02-05 10:03:38 -08:00
Zoltan Varga 382de3d308 Remove the src/mono/netcore directory, move its contents to src/mono. (#47847)
* Remove the src/mono/netcore directory, move its contents to src/mono.

* Fix docs and make targets.

* Remove the netcore dir itself.

* Fix samples.
2021-02-04 18:59:14 -05:00
Elinor Fung f31473d2ff Add doc about running host tests (#47744) 2021-02-02 12:27:24 -08:00
Kenneth Pouncey a78f7d2622 [browser][docs] Add docs for running tests in different browser (#47109)
* Add documentation on running tests using different Browsers other than chrome.

- XHARNESS_COMMAND="test-browser --browser=safari"
- Add ability to run against different browsers from the make file target `run-browser-tests` by adding `XHARNESS_BROWSER= ` to command line.

At the moment supported values are:
- `chrome`
- `safari`
- `firefox`

* Add gecko firefox requirements to testing docs.
2021-01-19 05:38:45 +01:00
Kenneth Pouncey 6299cefa03 [browser][docs] Add docs for running outerloop and browser instance (#46737)
* [browser][docs] Add testing docs for running outerloop and browser instance

* Add information for GitHub interface.
2021-01-08 10:49:38 +01:00
Dan MoseleyandJan Kotas d4d50db676 Improve generated dump debugging instructions (#46493)
* Improve generated dump debugging instructions

* Update eng/testing/debug-dump-template.md

Co-authored-by: Jan Kotas <[email protected]>

* Update eng/testing/debug-dump-template.md

Co-authored-by: Jan Kotas <[email protected]>

* Update eng/testing/debug-dump-template.md

Co-authored-by: Jan Kotas <[email protected]>

* Update debug-dump-template.md

* Fix ```

* fix some ```cmd

* Reverse .cmd to .bat

* Open only for reading

* Improve foldername

* Install only one SOS

Co-authored-by: Jan Kotas <[email protected]>
2021-01-05 07:57:42 -08:00
Fan Yang 60c6086aa7 Update testing.md (#46295)
The current framework relays on CoreCLR to create the Core_Root and then patch it with mono runtime. So building CoreCLR is required before building runtime tests for desktop Mono
2020-12-21 14:04:30 -05:00
Fan Yang 1aac928655 Update testing.md (#46043)
skipstressdependencies switch was removed from the runtime tests build script.
2020-12-14 15:39:06 -05:00
Jan Kotas 10381a2cdc Find&Replace coreclr/src -> coreclr (#45761) 2020-12-08 10:40:17 -08:00