46 Commits
Author SHA1 Message Date
Marek Fišera 1ac45f5161 [wasm] Fix WebSocket tests on NodeJS (#64330)
- Move all SetScriptCommands to a target, so they are placed after scenario in the generated script.
- Add WasmXHarnessArgsCli as an alternative variable that can be used from cli.
- Fix NodeJS symbolic links on Helix.
- Copy TestEchoMiddleware and RemoteLoopMiddleware even for Scenario=NodeJS.
- Enable System.Net.WebSockets.Client.Tests on NodeJS.
2022-02-09 10:27:24 +01:00
Maxim Lipnin 0b17270554 Update the apple testing doc (#63119) 2021-12-24 15:33:24 +03:00
Simon RozsivalandAlexander Köplinger 6cf785d9fb Support Process.Start() on MacCatalyst (#61507)
* Modify existing tests

* Enable process start and kill on MacCatalyst

* Typo

* Temporarily enable ProcessTests for MacCatalyst

* Temporarily enable ProcessTests for MacCatalyst - attempt 2

* Allow running Mac Catalyst builds in App Sandbox

* Allow enabling app sandbox for the other Mac Catalyst sample

* Add missing parameter to XCode project generator

* Remove unnecessary MacCatalyst detection

* Enable App Sandbox for Mac Catalyst tests

* Create a separate test branch for App Sandbox

* Remove the restriction to enable app sandbox just for Mac Catalyst apps

* Do not throw PNSE for Mac Catalyst

* Clean-up platform-specific conditions

* Build correct Process implementation for MacCatalyst

* Try to get more information from CI for further investigation of the failing build

* Revert "Try to get more information from CI for further investigation of the failing build"

This reverts commit fc63a370d688aa0d1dc890e6928a8d031d148e56.

* Add MacCatalyst target framework for System.Diagnostics.Process

* Add supported platform annotations for Mac Catalyst

* Fix annotations

* Remove incorrectly placed attributes

* Update attributes including the reference file

* Change platform attributes

* Generate app.entitlements in the BuildAppleAppBundles test build target

* Revert "Generate app.entitlements in the BuildAppleAppBundles test build target"

This reverts commit abbe2241586e3469c3d3e9d4ecd2cea2e09d3641.

* Enable AppSandbox when generating CMakeLists.txt for libraries tests

* Try implementing a workaround for app sandbox mode

* Fix app sandbox detection bug

* Add explanation comment

* Enable more tests for MacCatalyst

* Add apple app builder input validation

* Add Mac Catalyst w/ App Sandbox enabled to runtime-manual test pipeline

* Enable networking in App Sandbox mode

* Skip test which won't work on Mac Catalyst

* Skip some tests in app sandbox mode

* Update src/tasks/AppleAppBuilder/AppleAppBuilder.cs

Co-authored-by: Alexander Köplinger <[email protected]>

* Add a clarification comment for a networking entitlement

* Simplify supported platform condition

* Remove temporarily enabled test

* Remove unnecessary attributes

* Replace checking env variable with checking errno in libproc

* Update docs

* Remove unnecessary changes

* Temporarily enable running System.Diagnostics.Process.Tests for this PR

This reverts commit 02d370c2cf730bf67a3cf6fca8fcb8c5e4ff1ccd.

* Revert "Temporarily enable running System.Diagnostics.Process.Tests for this PR"

This reverts commit dc72f0f1d068314317d088d33ec87f9aa6447854.

* Fix job suffix in runtime-manual

* Remove attributes

* Revert "Remove attributes"

This reverts commit 704e9fa73cd40854d1becc2889569bab3c47d62d.

* Try changing the order of attributes to please CI

* Update ApiCompat baseline

* ApiCompat step 1: remove all attributes for MaxWorkingSet setter

* Revert "Update ApiCompat baseline"

This reverts commit a2ad03207c5f9f40b9dd2c335a1b0f31c37a0ac5.

* Revert "ApiCompat step 1: remove all attributes for MaxWorkingSet setter"

This reverts commit be72a3d2564afb89a83198ef010b3d4622799601.

* Update ApiCompat baseline

* Update ApiCompat baseline after pulling upstream main

* Remove trailing whitespace

Co-authored-by: Alexander Köplinger <[email protected]>
2021-12-07 20:33:38 +01:00
Maxim Lipnin b9a2c48aa8 Clean up the documentation for the library testing (#59401) 2021-11-08 12:11:54 +03:00
Aleksey Kliger (λgeek) 89cea2e659 [docs] Add instructions for debugging native runtime code on Android (#59183) 2021-09-17 09:01:13 -04:00
Viktor Hofer 67354a2fb4 Upgrade to net7.0 and remove target frameworks (#58011)
* Remove EOL tfms and upgrade to net7.0

* Make linker tests net7 aware

Make installer tests run on net7.0

Don't attempt to use live apphost

Fix payloads

Regenerate test files

Fix package testing
2021-09-08 22:04:23 +02:00
Vitek Karas 0042240dd8 Add zip tool to the install list for Android tests (#57569)
On default WSL Ubuntu `zip` is not installed, and trying to build Android tests fails with rather cryptic errors.
2021-08-17 08:00:48 -07:00
Noah FalkandViktor Hofer d783a8c812 Update library testing docs page to reduce confusion (#54324)
* Add warning on unmaintained testing doc page

* Update testing.md

Some example text that seems more clear to me, but only offered as a suggestion. 
Feel free to adjust it or if you want to use it as-is please double check what I wrote is accurate : )

I think the useful elements are:
1. Being explicit about what workflow steps need to happen in total
2. Being explicit about which commands are covering the entire workflow and which ones are only covering a part of it
3. Show the simple "do-it-all" options first before showing more complex partial options. Glancing at the first example and blindly copying it should land in the pit of success.

Co-authored-by: Viktor Hofer <[email protected]>
2021-06-23 09:00:47 +02:00
Pavel Savara fb61be479e [wasm][testing] hosting echo server in xharness process (#52923)
- move tests to inner loop
- include echo middleware in xharness server
- improve doc
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53592 for lack of TRACE
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53591 for content on GET/HEAD
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53874 for HttpRequestMessage.Headers.Host
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53872 for NPE on System.Net.Http.BrowserHttpHandler
- more granular ActiveIssue https://github.com/dotnet/runtime/issues/53876
- include middleware in Helix correlation payload
2021-06-09 17:32:58 +02:00
b443b8ccce Update testing docs with x86 instructions (#48715)
* Update testing docs with x86 instructions

* Apply suggestions from code review

Co-authored-by: Santiago Fernandez Madero <[email protected]>

* add more examples

* Apply suggestions from code review

Co-authored-by: Dan Moseley <[email protected]>

* Update docs/workflow/testing/libraries/testing.md

Co-authored-by: Santiago Fernandez Madero <[email protected]>

Co-authored-by: Santiago Fernandez Madero <[email protected]>
Co-authored-by: Dan Moseley <[email protected]>
Co-authored-by: Eric StJohn <[email protected]>
2021-05-24 11:45:21 -07:00
6ca3d9105d Shift Most of Wasm AOT test build to helix (#48226)
Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

## This is done by:
1. building the test assemblies on the build machine
   - the wasm part of the build is not executed on the build machine,
      because it has the AOT build part
2. Zip up the test assembly+friends, *and* any bits required to run the wasm
    app build for that on helix (eg. emsdk, wasm app targets, cross compiler etc)
3. Send all this to helix, and use a custom `aot-build.proj`
     - which recreates all the build inputs for the `WasmBuildApp` target
        using the paths for the assets on helix
     - then we can run `WasmBuildApp` for the build, resulting in a wasm app
       bundle.
4. Run the tests!

- We already have the bits required for building wasm apps on helix, supported
for `Wasm.Build.Tests`, which we can use here too.

## Trimming:

- Since, AOT can be so expensive, we use `EnableAggressiveTrimming=true`(EAT), but
   that means that we could have issues due to trimming.
- And it can sometimes be unclear whether the build/test failures are due to trimming
     or AOT.

- Because these builds+test runs are different from other builds, owing to the
   "build partially on helix" step, a normal EAT build would not be the same as 
- to help with testing this, we add two lanes to `runtime-staging`:
   - `*_Mono_AOT`: builds AOT+EAT on helix
   - `*_Mono_EAT`: builds EAT, on helix
      - this is required because we want to run almost the same kinda
         build: 1. build test assembly; *2. send to helix; 3. build wasm app;* 4. run tests

- This should effectively mean that we can see which errors might be due to EAT, and
   which are clearly because of EAT+AOT.

Co-authored-by: Steve Pfister <[email protected]>
Co-authored-by: Marek Safar <[email protected]>
Co-authored-by: Mitchell Hwang <[email protected]>
Co-authored-by: Ankit Jain <[email protected]>
2021-04-21 20:58:25 -04:00
Ankit Jain 8d102e3fc2 [wasm][tests] Enable line numbers in stack traces (#50893) 2021-04-12 13:01:25 -04:00
Alexander Köplinger bc9f00c226 Cleanup PlatformSpecific/SkipOnMono attributes that skip a platform (#50907)
We have a bunch of test assemblies that don't make sense on some platforms, e.g. Browser.
Right now we're skipping them via `[SkipOnMono("reason", TestPlatforms.Browser)]` but there's nothing that inherently ties this to Mono other than the current implementation.

The more generic `SkipOnPlatform` attribute can be used instead.
We can also use it in places where we do `[PlatformSpecific(~TestPlatforms....)]` to avoid the double inversion.
2021-04-09 13:20:18 +02:00
erhanatesoglu aa64d2e712 Fixed broken link (#50964)
The link to xunit on github has changed.
2021-04-08 17:56:01 -04:00
Ankit Jain 8a1101d2b5 [wasm][tests] Add properties to allow passing args to xharness (#50678)
* [wasm][tests] Add properties to allow passing args to xharness

New properties:
- `$(WasmXHarnessArgs)`
- `$(WasmXHarnessMonoArgs)`
- `$(WasmTestAppArgs)`

Xharness command line is built with these:

`dotnet xharness wasm test .. $(WasmXHarnessArgs) .. -- $(WasmXHarnessMonoArgs) .. --run .. $(WasmTestAppArgs)`

* [wasm][tests] extract wasm bits into tests.wasm.targets

* [wasm] set EMSDK_PATH for makefile run-tests- targets

* [wasm] Update target dependencies for wasm

* Don't overwrite RunScriptCommand, thus fixing wasm build tests

* Track property name changes
2021-04-07 13:06:55 -04:00
Ryan Lucia 240fcec9d8 Update iOS workflow docs to clarify simulator/device os split (#50366) 2021-04-05 07:55:20 -04:00
Fan Yang 1ce7460578 Update testing-android.md (#50484) 2021-03-31 10:03:27 -07:00
Steve Pfister bc5b227c9f Remove openssl dependency from android (#49282) 2021-03-23 00:17:12 -04:00
Steve MacLean e422345323 Apple Silicon platform detection (#49155)
* Add Apple Silicon PlatformDetection

* Disable OSVersion_ValidVersion_OSX on Apple Silicon

* Update docs
2021-03-04 17:28:07 -05:00
Viktor Hofer 3553284bea Rename mention to master branch to main (#48824) 2021-03-01 10:09:31 +01:00
Elinor Fung 9ce547d084 Add example for installing/launching emulator to Android test doc (#47889) 2021-02-05 10:03:38 -08:00
Kenneth Pouncey a78f7d2622 [browser][docs] Add docs for running tests in different browser (#47109)
* Add documentation on running tests using different Browsers other than chrome.

- XHARNESS_COMMAND="test-browser --browser=safari"
- Add ability to run against different browsers from the make file target `run-browser-tests` by adding `XHARNESS_BROWSER= ` to command line.

At the moment supported values are:
- `chrome`
- `safari`
- `firefox`

* Add gecko firefox requirements to testing docs.
2021-01-19 05:38:45 +01:00
Kenneth Pouncey 6299cefa03 [browser][docs] Add docs for running outerloop and browser instance (#46737)
* [browser][docs] Add testing docs for running outerloop and browser instance

* Add information for GitHub interface.
2021-01-08 10:49:38 +01:00
Anirudh Agnihotry b37f10a504 Changing Windows_NT -> Windows for enabling TargetPlatformMoniker feature of the sdk for platform specific tfms. (#43651)
* eng change

* coreclr change

* libraries change

* tests change

* installer & mono change

* py files changed

* .md change

* .yml change

* remove redundant targetos property

* Windows_NT -> windows after ninja change

* fixing runtime build failure
2020-11-02 14:25:58 -08:00
Alexander Köplinger e658ddbc68 Simplify webassembly targets (#41869)
Add a subset for compiling only the wasm runtime: `mono.wasmruntime`
This allows us to rebuild the wasm runtime without building the full libraries subset.

Updated documentation with new guidance.

Updated how we build the dotnet.timezones.blat.
2020-09-05 00:41:32 +02:00
Youssef VictorandJan Kotas d14b50ae21 Fix trailing whitespaces (#40891)
* Trim trailing whitespaces

* Match raw with rendered

* Delete extra asterisks and |

* Update ELT Hooks - tail calls.md

Co-authored-by: Jan Kotas <[email protected]>
2020-08-19 17:29:01 -07:00
David Cantú 54fb5973a0 /p:Configuration must use "Release" instead of "release" (#40816)
I am not very familiar with Unix so I was not aware why I was having below error:
```
/home/jozky/runtime/.dotnet/sdk/5.0.100-preview.8.20362.3/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(379,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.browser-wasm was not downloaded. Try running a NuGet restore with the RuntimeIdentifier 'browser-wasm'. [/home/jozky/runtime/src/libraries/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj]
```

I tried running `dotnet restore --runtime browser-wasm` as suggested by the error but that didn't help.
The correct thing to do was to pass `Release` (with upper case R) to `Configuration` parameter.

cc @safern
2020-08-13 22:19:23 -07:00
Aleksey Kliger (λgeek) 2bfc063880 [docs] Fix wasm testing example directories (#38624)
And use a testsuite that passes for the example
2020-07-01 12:04:16 -04:00
Maxim Lipnin 84c64c5801 Add initial doc about testing libraries on WebAssembly (#37478)
* Add initial doc about testing libraries on WebAssembly

* Address feedback

* Address feedback
2020-06-08 18:01:51 +02:00
Egor Bogatov 00acdc1514 [mono] Use "dotnet publish" for Android sample with ILLink (#36593) 2020-05-16 16:15:56 +03:00
Egor Bogatov ec2209e736 [docs] How to run tests on iOS and Android (#36297) 2020-05-15 15:02:05 +03:00
Santiago Fernandez Madero df8e2d751e Update testing.md (#36286) 2020-05-12 12:49:05 -04:00
Viktor HoferandAlexander Köplinger 2cd041bbee Document F5 and Test Explorer prerequisites (#36117)
* Document F5 and Test Explorer prerequisites

* Update docs/workflow/testing/visualstudio.md

Co-authored-by: Alexander Köplinger <[email protected]>
2020-05-08 21:03:17 +02:00
Viktor Hofer 000046f6c8 Change netcoreapp5.0 to net5.0 (#35176)
* Change netcoreapp5.0 to net5.0

* Remove tfm downgrades

* Rename S.R.CS.Unsfae include folder
2020-05-06 13:53:21 +02:00
Viktor Hofer 8e0147ecdf Redesign the subset feature (#34663)
* Redesign subset feature and remove subsetcategory

Implement proposal from https://github.com/dotnet/runtime/issues/34403.
Remove subsetcategory and allow all subsets to specified via the -subset
switch.
2020-04-08 03:09:31 +02:00
Viktor Hofer 42183b1b86 Enable restore for ref and src projects in libs (#33553)
- Use RestoreUseStaticGraphEvaluation which improves no-op restore by 10-15x down to 10-20 seconds.
- .builds msbuild files renamed to .proj as RestoreUseStaticGraphEvaluation throws for non .proj files without an env var set.
- Introducing subsets for libraries and mono and replacing -buildtests switch which was only working for libraries in favor of the subset switch -subset tests which works consistently.
- Fixing the Microsoft.DotNet.CodeAnalysis analyzer which wasn't running and adding missing exclusions.
- Separating restore and build phases in different parts in the repo (ie for installer.tasks) as generated props and targets need to be imported which requires a reevaluation in the build phase.
- Fix eng/docker/build-docker-sdk.ps1 by using the official build entrypoints (cc @alnikola)
- Remove a few depprojs in favor of project restore (faster restore :))
- Fix root code coverage measurement not working correctly
- Traversal support instead of dir.traversal.targets or manual build target defines.
- Introduce a root Build.proj entrypoint which is responsible for building and restoring the repository. This is necessary to enable the new NuGet fast restore which works best and fastest with a single entrypoint.
- Avoid binclashes in libraries and between libraries and installer (netstandard.depproj vs netstandard.csproj)
- Upgrading the SDK to 5.0 latest
- Code cleanup
2020-04-06 11:54:10 +02:00
Jan Kotas 479067315a .NET Core to .NET branding changes (#33694)
Contributes to #33680
2020-03-28 07:00:19 -07:00
Karel Zikmund cee4eca178 Add how to run outer loop libraries tests (#32940)
* Add how to run outer loop libraries tests
2020-03-27 09:55:14 -07:00
Anirudh AgnihotryandViktor Hofer fef9ca70f6 Replacing OSGroup with TargetOS for coreclr, libraries and installer (#32833)
* Remove OSGroup from coreclr and corefx

* remove from installer

* fixing coreclr tests

* Some documentation changes

Co-Authored-By: Viktor Hofer <[email protected]>

* moving buildos calculation to config.props

* BuildOS -> TargetOS

* remove osgroup suffix

* removing comment and correcting typo

* dotnet msbuild -> dotnet build at relevant places in docs

* moving more properties to configuration.props

* refactoring properties

* moving TargetArchitecture property to loveBuildd.Targets

* adding configuration.props to signing

* Fix installer build

* BuildOS -> Target os and removing slash

Co-authored-by: Viktor Hofer <[email protected]>
2020-03-05 07:51:06 -08:00
Viktor Hofer 33f2807fc6 Remove BuildAndTest and RebuildAndTest target in favor of Test (#33151)
* Implicitly build when invoking the test target

To get closer to the VSTest behavior this changes the Test target to
implicitly invoke the Build target unless /p:TestNoBuild is passed in.
(VSTest uses VSTestNoBuild which is controlled by the
dotnet test --no-build flag)

* Fix error logging for test outputs

* Update dotnet msbuild to build docs

* Update ReportGenerator global tool version

Updating ReportGenerator's version to 4.5.0 which removes unnecessary
logging (args passed in).
2020-03-04 19:40:19 +01:00
Marie PíchováandViktor Hofer 12f1f58ec0 Removed misleading doc and added -vs testing (#32618)
Co-authored-by: Viktor Hofer <[email protected]>
2020-02-21 22:18:37 +01:00
Anirudh Agnihotry 86a086377b updating docs and removing BuildConfiguration and ConfigurationGroup (#32062)
* build config and update docs

* remove configurationGroup and update docs

* addressing feedback

* viktor comments
2020-02-20 14:39:38 -08:00
Viktor Hofer 0896dd8204 Update testing with configuration system changes 2020-02-20 17:39:21 +01:00
Alexander Köplinger aac8f91cc5 Ensure ActiveIssue attribute is only used with issue links (#32535)
* Ensure ActiveIssue attribute is only used with issue links

The constructor that takes an `int` was removed in https://github.com/dotnet/arcade/pull/4831.

Update docs to recommend using issue links instead of numbers.

Fix two tests that were using ActiveIssue incorrectly.

* PR feedback
2020-02-19 08:58:17 -05:00
Santiago Fernandez Madero 6b101e01c3 Remove workarounds in build scripts for netcoreapp and netfx versionless 2020-01-16 15:21:43 -08:00
9900dfb4b2 Organize some markdowns and fix links (#1159)
* Move files into hierarchy

* fix more links

* Fix botr and features links

* Remove bad apostrophe

* spelling

* Apply suggestions from code review

Co-Authored-By: Youssef Victor <[email protected]>

* Update docs/coding-guidelines/package-projects.md

Co-Authored-By: Jan Kotas <[email protected]>

Co-authored-by: Youssef Victor <[email protected]>
Co-authored-by: Jan Kotas <[email protected]>
2019-12-26 15:22:57 -08:00