Compare commits

..
167 Commits
Author SHA1 Message Date
Alexander Köplinger d9001b5ae7 Bump bockbuild to get GTK NSPasteboardType change (#18815)
Backport of https://github.com/mono/mono/pull/18080.

Approved for 2019-08 by @marek-safar in https://github.com/mono/mono/pull/18080#issuecomment-585451815
2020-02-12 23:46:45 +01:00
Alexander Köplinger e69d4b57d7 Bump bockbuild for Pango patch
Fixes bug #1048838, correct SF font not loading in VSMac

Backport of https://github.com/mono/mono/pull/18566
2020-01-28 13:13:48 +01:00
Alexander Köplinger fd9f379d29 [sdks] Use Xcode 11.3 stable version for iOS/Mac SDKs
As requested by the xamarin-macios team.

(cherry picked from commit 7e65a73d4d)
2019-12-18 22:08:19 +01:00
monojenkins e3084ecece [debugger] skip suspend for unattached threads (#18138)
[2019-08] [debugger] skip suspend for unattached threads

When the debugger tries to suspend all the VM threads, it can happen with cooperative-suspend that it tries to suspend a thread that has previously been attached, but then did a "light" detach (that only unsets the domain). With the domain set to `NULL`, looking up a JitInfo for a given `ip` will result into a crash, but it isn't even necessarily needed for the purpose of suspending a thread.

More details: Consider the following:
```
(lldb) c
error: Process is running.  Use 'process interrupt' to pause execution.
Process 12832 stopped
* thread #9, name = 'tid_a31f', queue = 'NSOperationQueue 0x8069b670 (QOS: UTILITY)', stop reason = breakpoint 1.1
    frame #0: 0x00222870 WatchWCSessionAppWatchOSExtension`debugger_interrupt_critical(info=0x81365400, user_data=0xb04dc718) at debugger-agent.c:2716:6
   2713         MonoDomain *domain = (MonoDomain *) mono_thread_info_get_suspend_state (info)->unwind_data [MONO_UNWIND_DATA_DOMAIN];
   2714         if (!domain) {
   2715                 /* not attached */
-> 2716                 ji = NULL;
   2717         } else {
   2718                 ji = mono_jit_info_table_find_internal ( domain, MONO_CONTEXT_GET_IP (&mono_thread_info_get_suspend_state (info)->ctx), TRUE, TRUE);
   2719         }
Target 0: (WatchWCSessionAppWatchOSExtension) stopped.
(lldb) bt
* thread #9, name = 'tid_a31f', queue = 'NSOperationQueue 0x8069b670 (QOS: UTILITY)', stop reason = breakpoint 1.1
  * frame #0: 0x00222870 WatchWCSessionAppWatchOSExtension`debugger_interrupt_critical(info=0x81365400, user_data=0xb04dc718) at debugger-agent.c:2716:6
    frame #1: 0x004e177b WatchWCSessionAppWatchOSExtension`mono_thread_info_safe_suspend_and_run(id=0xb0767000, interrupt_kernel=0, callback=(WatchWCSessionAppWatchOSExtension`debugger_interrupt_critical at debugger-agent.c:2708), user_data=0xb04dc718) at mono-threads.c:1358:19
    frame #2: 0x00222799 WatchWCSessionAppWatchOSExtension`notify_thread(key=0x03994508, value=0x81378c00, user_data=0x00000000) at debugger-agent.c:2747:2
    frame #3: 0x00355bd8 WatchWCSessionAppWatchOSExtension`mono_g_hash_table_foreach(hash=0x8007b4e0, func=(WatchWCSessionAppWatchOSExtension`notify_thread at debugger-agent.c:2733), user_data=0x00000000) at mono-hash.c:310:4
    frame #4: 0x0021e955 WatchWCSessionAppWatchOSExtension`suspend_vm at debugger-agent.c:2844:3
    frame #5: 0x00225ff0 WatchWCSessionAppWatchOSExtension`process_event(event=EVENT_KIND_THREAD_START, arg=0x039945d0, il_offset=0, ctx=0x00000000, events=0x805b28e0, suspend_policy=2) at debugger-agent.c:4012:3
    frame #6: 0x00227c7c WatchWCSessionAppWatchOSExtension`process_profiler_event(event=EVENT_KIND_THREAD_START, arg=0x039945d0) at debugger-agent.c:4072:2
    frame #7: 0x0021b174 WatchWCSessionAppWatchOSExtension`thread_startup(prof=0x00000000, tid=2957889536) at debugger-agent.c:4149:2
    frame #8: 0x0037912d WatchWCSessionAppWatchOSExtension`mono_profiler_raise_thread_started(tid=2957889536) at profiler-events.h:103:1
    frame #9: 0x003d53da WatchWCSessionAppWatchOSExtension`fire_attach_profiler_events(tid=0xb04dd000) at threads.c:1120:2
    frame #10: 0x003d4d83 WatchWCSessionAppWatchOSExtension`mono_thread_attach(domain=0x801750a0) at threads.c:1547:2
    frame #11: 0x003df1a1 WatchWCSessionAppWatchOSExtension`mono_threads_attach_coop_internal(domain=0x801750a0, cookie=0xb04dcc0c, stackdata=0xb04dcba8) at threads.c:6008:3
    frame #12: 0x003df287 WatchWCSessionAppWatchOSExtension`mono_threads_attach_coop(domain=0x00000000, dummy=0xb04dcc0c) at threads.c:6045:9
    frame #13: 0x005034b8 WatchWCSessionAppWatchOSExtension`::xamarin_switch_gchandle(self=0x80762c20, to_weak=false) at runtime.m:1805:2
    frame #14: 0x005065c1 WatchWCSessionAppWatchOSExtension`::xamarin_retain_trampoline(self=0x80762c20, sel="retain") at trampolines.m:693:2
    frame #15: 0x657ea520 libobjc.A.dylib`objc_retain + 64
    frame #16: 0x4b4d9caa WatchConnectivity`__66-[WCSession onqueue_handleDictionaryMessageRequest:withPairingID:]_block_invoke + 279
    frame #17: 0x453c7df7 Foundation`__NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 12
    frame #18: 0x453c7cf4 Foundation`-[NSBlockOperation main] + 88
    frame #19: 0x453cacee Foundation`__NSOPERATION_IS_INVOKING_MAIN__ + 27
    frame #20: 0x453c6ebd Foundation`-[NSOperation start] + 835
    frame #21: 0x453cb606 Foundation`__NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 27
    frame #22: 0x453cb12e Foundation`__NSOQSchedule_f + 194
    frame #23: 0x453cb26e Foundation`____addOperations_block_invoke_4 + 20
    frame #24: 0x65de007b libdispatch.dylib`_dispatch_call_block_and_release + 15
    frame #25: 0x65de126f libdispatch.dylib`_dispatch_client_callout + 14
    frame #26: 0x65de3788 libdispatch.dylib`_dispatch_continuation_pop + 421
    frame #27: 0x65de2ee3 libdispatch.dylib`_dispatch_async_redirect_invoke + 818
    frame #28: 0x65df087d libdispatch.dylib`_dispatch_root_queue_drain + 354
    frame #29: 0x65df0ff3 libdispatch.dylib`_dispatch_worker_thread2 + 109
    frame #30: 0x66024fa0 libsystem_pthread.dylib`_pthread_wqthread + 208
    frame #31: 0x66024e44 libsystem_pthread.dylib`start_wqthread + 36
```

Going further, `info` is about this thread:
```
(lldb) p/x *(int *)(((char *) info->node.key) + 0xa0)
(int) $2 = 0x01243f93
(lldb) thread list
Process 12832 stopped
  thread #1: tid = 0x1243ee1, 0x65f7e396 libsystem_kernel.dylib`mach_msg_trap + 10, name = 'tid_303', queue = 'com.apple.main-thread'
  thread #2: tid = 0x1243ee6, 0x65f816e2 libsystem_kernel.dylib`__recvfrom + 10
  thread #3: tid = 0x1243ee7, 0x65f81aea libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker'
  thread #4: tid = 0x1243ee9, 0x65f7e3d2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'Finalizer'
  thread #5: tid = 0x1243eea, 0x65f816e2 libsystem_kernel.dylib`__recvfrom + 10, name = 'Debugger agent'
  thread #6: tid = 0x1243f1d, 0x65f7e396 libsystem_kernel.dylib`mach_msg_trap + 10, name = 'com.apple.uikit.eventfetch-thread'
  thread #8: tid = 0x1243f93, 0x65f7e396 libsystem_kernel.dylib`mach_msg_trap + 10, name = 'tid_6d0f', queue = 'NSOperationQueue 0x8069b300 (QOS: UTILITY)'
* thread #9: tid = 0x12443a9, 0x00222870 WatchWCSessionAppWatchOSExtension`debugger_interrupt_critical(info=0x81365400, user_data=0xb04dc718) at debugger-agent.c:2716:6, name = 'tid_a31f', queue = 'NSOperationQueue 0x8069b670 (QOS: UTILITY)', stop reason = breakpoint 1.1
  thread #10: tid = 0x1244581, 0x65f7fd32 libsystem_kernel.dylib`__workq_kernreturn + 10
(lldb) thread select 8
* thread #8, name = 'tid_6d0f', queue = 'NSOperationQueue 0x8069b300 (QOS: UTILITY)'
    frame #0: 0x65f7e396 libsystem_kernel.dylib`mach_msg_trap + 10
libsystem_kernel.dylib`mach_msg_trap:
->  0x65f7e396 <+10>: retl
    0x65f7e397 <+11>: nop

libsystem_kernel.dylib`mach_msg_overwrite_trap:
    0x65f7e398 <+0>:  movl   $0xffffffe0, %eax         ; imm = 0xFFFFFFE0
    0x65f7e39d <+5>:  calll  0x65f85f44                ; _sysenter_trap
(lldb) bt
* thread #8, name = 'tid_6d0f', queue = 'NSOperationQueue 0x8069b300 (QOS: UTILITY)'
  * frame #0: 0x65f7e396 libsystem_kernel.dylib`mach_msg_trap + 10
    frame #1: 0x65f7e8ff libsystem_kernel.dylib`mach_msg + 47
    frame #2: 0x66079679 libxpc.dylib`_xpc_send_serializer + 104
    frame #3: 0x660794da libxpc.dylib`_xpc_pipe_simpleroutine + 80
    frame #4: 0x66079852 libxpc.dylib`xpc_pipe_simpleroutine + 43
    frame #5: 0x66043a8f libsystem_trace.dylib`___os_activity_stream_reflect_block_invoke_2 + 30
    frame #6: 0x65de126f libdispatch.dylib`_dispatch_client_callout + 14
    frame #7: 0x65de3d71 libdispatch.dylib`_dispatch_block_invoke_direct + 257
    frame #8: 0x65de3c62 libdispatch.dylib`dispatch_block_perform + 112
    frame #9: 0x6604349a libsystem_trace.dylib`_os_activity_stream_reflect + 725
    frame #10: 0x6604ef19 libsystem_trace.dylib`_os_log_impl_stream + 468
    frame #11: 0x6604e44d libsystem_trace.dylib`_os_log_impl_flatten_and_send + 6410
    frame #12: 0x6604cb3b libsystem_trace.dylib`_os_log + 137
    frame #13: 0x6604f4aa libsystem_trace.dylib`_os_log_impl + 31
    frame #14: 0x4b4eb4e9 WatchConnectivity`WCSerializePayloadDictionary + 393
    frame #15: 0x4b4d7c4d WatchConnectivity`-[WCSession onqueue_sendResponseDictionary:identifier:] + 195
    frame #16: 0x4b4da435 WatchConnectivity`__66-[WCSession onqueue_handleDictionaryMessageRequest:withPairingID:]_block_invoke.411 + 35
    frame #17: 0x453c7df7 Foundation`__NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 12
    frame #18: 0x453c7cf4 Foundation`-[NSBlockOperation main] + 88
    frame #19: 0x453cacee Foundation`__NSOPERATION_IS_INVOKING_MAIN__ + 27
    frame #20: 0x453c6ebd Foundation`-[NSOperation start] + 835
    frame #21: 0x453cb606 Foundation`__NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 27
    frame #22: 0x453cb12e Foundation`__NSOQSchedule_f + 194
    frame #23: 0x453cb067 Foundation`____addOperations_block_invoke_2 + 20
    frame #24: 0x65dedf49 libdispatch.dylib`_dispatch_block_async_invoke2 + 77
    frame #25: 0x65de4461 libdispatch.dylib`_dispatch_block_async_invoke_and_release + 17
    frame #26: 0x65de126f libdispatch.dylib`_dispatch_client_callout + 14
    frame #27: 0x65de3788 libdispatch.dylib`_dispatch_continuation_pop + 421
    frame #28: 0x65de2ee3 libdispatch.dylib`_dispatch_async_redirect_invoke + 818
    frame #29: 0x65df087d libdispatch.dylib`_dispatch_root_queue_drain + 354
    frame #30: 0x65df0ff3 libdispatch.dylib`_dispatch_worker_thread2 + 109
    frame #31: 0x66024fa0 libsystem_pthread.dylib`_pthread_wqthread + 208
    frame #32: 0x66024e44 libsystem_pthread.dylib`start_wqthread + 36
```
which is a thread in a "light" detach state (aka. coop detach), where we only unset the domain:
https://github.com/mono/mono/blob/4cefdcb7ce2d939ee78fb45d1b4913eb3bc064fd/mono/metadata/threads.c#L6084-L6111

Fixes https://github.com/mono/mono/issues/17926



<!--
Thank you for your Pull Request!

If you are new to contributing to Mono, please try to do your best at conforming to our coding guidelines http://www.mono-project.com/community/contributing/coding-guidelines/ but don't worry if you get something wrong. One of the project members will help you to get things landed.

Does your pull request fix any of the existing issues? Please use the following format: Fixes #issue-number
-->


Backport of #18105.

/cc @lewurm
2019-12-12 19:06:03 +01:00
monojenkins da5b1c675d [2019-08] Enable GSS on android (#18116)
Backport of #17838.
2019-12-11 14:04:12 +01:00
monojenkins bef1e63358 [interp] track valuetype stack for CEE_RET too (#17998)
Backport of #17968

Fixes https://github.com/mono/mono/issues/17924
2019-12-03 10:47:01 -05:00
Bernhard Urban-Forster 56325f4097 [arm] if mtriple is provided, do not set -march
E.g. when passing `-march=arm -mtriple=armv7s-ios`, `-march` will win and thus ignore `armv7s`.

Fixes https://github.com/mono/mono/issues/17931

* Caused by 5e318afd2e
* Related with 4cd5068230
* Almost fixed by e431093f73

Hopefully we are done with it.
2019-12-02 17:24:04 +01:00
monojenkins c4a1b40fbd [2019-08] [runtime] Treat calling a non-virtual method through… (#17961)
* [runtime] Treat calling a non-virtual method through an open delegate the same as the normal case instead of the OPEN_VIRTUAL case.

* [amd64] Fix the return value mapping in gsharedvt calls, the return address might be in different registers in the caller and callee.

* [runtime] Always initialize del->method_ptr even if interp_method is set, JITted code makes calls through it in mixed mode.

* Add a test for open delegates and gsharedvt.

* Increase nrgctx trampoline count, its needed by dynamic-method-churn.exe in full-aot-interp mode.
2019-12-02 17:20:35 +01:00
Ankit Jain 0c9250280d [2019-08] bump msbuild+roslyn to track mono-2019-08 (#17954)
* Bump msbuild to track mono-2019-08

* Bump roslyn to 3.4.0-beta4-19569-03

.. to match version from `dotnet/toolset` `releaes/3.1.1xx`: https://github.com/dotnet/toolset/blob/597688dfa01a77b8f502bfea9d79feecfc63e33d/eng/Versions.props#L36

* Bump nuget.exe to 5.4.0
2019-11-29 11:20:11 +01:00
Ankit Jain 14aac0c541 Bump msbuild to track mono-2019-08 (#17854) 2019-11-25 10:29:09 -06:00
monojenkins a77128ca79 [merp] Remove extraneous waitpid invocation (#17762)
The extraneous call basically guarantees that the following waitpid invocation will fail. Fixes merp stating that its process has failed when it hasn't.
2019-11-14 12:17:44 +01:00
monojenkins 296a9afdb2 [MacSDK] Bump xamarin-gtk-theme.py to latest revision from private bockbuild (#17786)
This adds some accessibility fixes from https://github.com/mono/xamarin-gtk-theme/pull/35

Backport of #17785.
2019-11-14 09:26:22 +01:00
Larry Ewing e59c3fbb68 Bump msbuild packages to 1e1d0f2caad3731357cca18b298536e514f5519b (#17724)
Pull in dotnet/toolset versions

```
Microsoft.NET.Sdk: 3.1.100-preview3.19553.1 (from 3.1.100-preview2.19528.2)
Microsoft.NET.Sdk.Razor: 3.1.0-preview3.19553.1 (from 3.1.0-preview2.19528.1)
Microsoft.NET.Sdk.Web: 3.1.100-preview3.19554.3 (from 3.1.100-preview2.19529.1)
Microsoft.DotNet.Cli.Runtime: 3.1.100-preview3.19554.3 (from 3.1.100-preview2.19529.1)
```
2019-11-06 11:06:29 +01:00
Alexander Köplinger 062f0ab8ca [sdks] Use Xcode 11.2 stable version for iOS/Mac SDKs
As requested by the xamarin-macios team.
2019-11-04 15:25:39 +01:00
monojenkins e1ef774391 [2019-08] Bump CoreFX to pickup corefx PR #367 to fix #17133. (#17622)
Bump CoreFX to pickup mono/corefx#367 to fix #17133.

Backport of #17590.
2019-11-01 14:20:37 -04:00
Ankit Jain 6d1f88e0ad Bump msbuild to get SDK updates from https://github.com/mono/msbuild/pull/150 2019-11-01 11:22:36 -05:00
Larry Ewing a3f3bfc4c3 Bump nuget to the latest suggested version 2019-11-01 11:22:36 -05:00
Larry Ewing 9bd3079f1c [2019-08] bump msbuild with more p2 packages 2019-11-01 11:22:36 -05:00
Thays Grazia 6ac1ff75a2 [dim][regression] Explicit interface override (#17583) (#17627)
* Precedence order:
-> Override interface method in class
-> Method in class
-> DIM
2019-10-30 21:41:45 -03:00
Alexander Köplinger a119807a01 [acceptance-tests] Bump mono/roslyn to get build fix
See https://github.com/mono/roslyn/commit/6bfcd4e770609a38b8523e83ce1ec943162f30bb

(cherry picked from commit 1340095581)
2019-10-29 21:44:45 +01:00
Aleksey Kliger d234d34b70 [2019-08][merp] Install sigterm handler in EnableMicrosoftTelemetry
Partly revert https://github.com/mono/mono/pull/17296 backports.

Follow-up work for backports of https://github.com/mono/mono/pull/17537 backports.

Addresses https://github.com/mono/mono/issues/17419
2019-10-28 14:31:34 -04:00
monojenkins 444a9a3fc4 [2019-08] [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping (#17568)
* [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping

The primary reason for this is gaining the ability to interpret SIGTERM properly as a crashing signal when outside of 'dump mode', and as a utility signal during the 'thread summarizer' dumping process.
2019-10-26 18:49:25 -04:00
Steve Pfister 8946e49a97 Added SR.missing from original backport 2019-10-24 21:31:39 -05:00
Steve Pfister 5c784c9747 Bump to mono/corefx@fb41040
Includes upstream corefx changes that makes NTLM auth work for more scenarios.

Backport of https://github.com/mono/mono/pull/17530
2019-10-24 21:31:39 -05:00
Larry Ewing fdb704901e Revert "[2019-08] rollback msbuild/roslyn updates for a preview" (#17543)
This reverts commit 1b2e536b22.
now that packages have been built.
2019-10-24 17:40:31 -04:00
monojenkins 92a177923c [2019-08] [debugger] Changing how debugger handles exception w… (#17524)
* Removing all fixes to fix #15203 and #17083, and trying a new approach to fix this #17084, together.

The thing is when we are executing on hybrid suspend the caller is executed inside try catch in the runtime_invoke_ method that is generated. And this try catch should be ignored for the debugger exception, otherwise it's the last try catch in main thread, but should always be considered to generate the stack trace.

Fixes #17084

* This test check the case that we were missing. On preemptive this test always runs perfectly, on Hybrid, never run well until this PR.

* Changing how call unit test. Changing variable name.
2019-10-24 11:54:14 -04:00
Larry Ewing 1b2e536b22 [2019-08] rollback msbuild/roslyn updates for a preview 2019-10-23 16:22:43 -04:00
Ankit Jain ec4ef9bc34 Bump roslyn-binaries to get roslyn 3.4.0-beta3-19521-01
.. from https://github.com/mono/roslyn-binaries/pull/20 .
2019-10-23 11:46:23 -05:00
Ankit Jain cc8deca901 Bump msbuild to pick up sdks+roslyn updates
.. from https://github.com/mono/msbuild/pull/147 .
2019-10-23 11:46:23 -05:00
Ankit Jain 335f0109c5 Bump msbuild to track mono-2019-08 2019-10-23 11:46:23 -05:00
Zoltan Varga 90d6e496e4 Fix checks so stack overflows work again. 2019-10-23 11:38:40 -05:00
Zoltan Varga 4b60e7f930 [jit] Avoid running mono_handle_native_crash () on the altstack, it can't produce a backtrace. AMD64 only for now. 2019-10-23 11:38:40 -05:00
Marius Ungureanu 989333d6ae Bump Bockbuild to pick-up gtk# binding change (#17486)
Fixes https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448
Which causes quite a few broken clipboard bugs and possible memory corruption
2019-10-23 14:32:30 +02:00
monojenkins baa12e5d24 [profiler] Fix coverage profiler on macos (#17422)
If we have coverage profiling enabled for a method (mono_profiler_coverage_instrumentation_enabled, with coverage_filter callback), then we should be able to allocate a coverage info structure (mono_profiler_coverage_alloc). After https://github.com/mono/mono/commit/52429673dc617fd437518757c04c7448211075e9, coverage_filter was reporting wrapper methods as ok to profile, while mono_profiler_coverage_alloc was refusing to allocate the necessary structure, leading to crashes.
2019-10-22 13:24:39 +03:00
monojenkins ef75d4bef7 [2019-08] BeginConnect complete to early when not using callback. (#17416)
* BeginConnect complete to early when not using callback.

The following commit, https://github.com/mono/mono/commit/068e8fe0424806753d7ab72a7b9bf0e54b58408b#diff-33b28a9dc0b8032650ed77213481a99f
changed the way we do DNS resolve in BeginConnect using host name. Instead
of doing a blocking DNS resolve, it switched to an async DNS resolve, followed
by an async connect. The returned IAsyncResult was just representing the
completion of the DNS resolve task and not the underlying connect. It
also turns out that part of doing EndConnect, we did some validation
of the used IAsyncResult making sure it was an instance of SocketAsyncResult,
meaning that if EndConnect was called on the returned IAsyncResult from
BeginConnect, EndConnect would throw and exception.

NOTE, this only happens if you use pattern like this:

var result = sock.BeginConnect (hostName, port, null, null);
result.AsyncWaitHandle.WaitOne (2000);
sock.EndConnect (result);

It did not happened if you used callbacks, since then EndConnect would
be called as part of callback and that IAsyncResult was of correct type,
or if you did a BeginConnect with an EndPoint or IPAddress[] since that
won't trigger the DNS resolve.

Fix is to make sure we hand out the SocketAsyncResult instance that will
be used in our internal connect calls from BeginConnect. We also make
sure, we chain the DNS resolve task together with a task doing the async
connect (using the same SocketAsyncResult instance). This will make sure
the IAsyncResult returned from BeginConnect is the same one as used in
internal connect call, so it will only complete when the connect call
completes.

* Adding test for BeginConnect using host name.

* Bump API snapshot submodule

Backport of #17035.
2019-10-21 17:48:30 +02:00
monojenkins 01dbbb746f [2019-08] Enable GSS on Linux (#17410)
* Fix GSS on Linux

* fix build

* use gssapi_krb5

* enable on linux

* [csproj] Update project files

* enabled GSS for monodroid

* [csproj] Update project files
2019-10-18 11:34:18 +02:00
Filip Navara 51a3dc37da Fix SafeHandle marshalling in ref/in/out parameters (#17330)
* Fix SafeHandle marshalling in ref/in/out parameters

* Reorder code to make it exception safe, update comments

* Revert the code reordering because it breaks the invariant that new handle returned from native code should always create new SafeHandle on the managed side

* Fix test_0_safehandle_ref test. The test was expecting incorrect behavior, verified on .NET Core 3 and .NET Framework 4.8.

* Address PR feedback

* Fix cut & paste error

* Fix build

* Really fix the test_0_safehandle_ref tests

* Add more tests for SafeHandle marshalling

* Add exclusion for broken CoreFX test
2019-10-17 14:12:31 +02:00
Cody Russell 3eb5f34f54 [GTK] Bump bockbuild for GtkViewport autoscrolling patch. (#17321)
(cherry picked from commit 37dc33a257)
2019-10-16 17:03:54 +02:00
Kyle White b601371d5f Update MERP event type to MonoAppCrash
* Update parameters in MERP WerReportMetadata xml

To enable backend analysis, we need to update our parameters to match our back end event type, which is MonoAppCrash.

It doesn't make sense to me that the previous implementation constructed the XML here in Mono, but depended on the user (VSMac in this case) to add the "correct" event type.  The parameters in the XML, must match exactly what is expected for the given EventType, and so we should ensure they are always in-sync by defining them here.  Thus, I removed the usage of `eventType` in `mono_merp_enable` and hard coded the expected eventType.

* [merp] Don't pass eventType from managed

The eventType must me MONO_MERP_EVENT_TYPE_STR ("MonoAppCrash") - the backend
determines the format of the other fields based on the event type, so it
doesn't make sense to pass other values from managed since the runtime controls
the other fields.

For compatability Mono.Runtime.EnableMicrosoftTelemetry will still take a
string parameter (now called "unused" instead of "eventType_str") but it will
not pass it down to the internal call.

* Bump corlib version

* [merp] Revert paramerters in WerReportMetadata.xml

Still need event type to be `MonoAppCrash` but the parameters will be the same as before.
2019-10-15 19:10:14 +02:00
Alexander Köplinger 6184ff007b [2019-08][ci] Use Xcode11.1 and 11.2beta2 for XI/XM Mono SDK builds (#17324)
As requested by the xamarin-macios team.
2019-10-15 11:44:41 +02:00
monojenkins 8969f2cc99 [2019-08] [merp] Include any managed methods in the 'unmanaged_frames' portion … (#17316)
* [merp] Include any managed methods in the 'unmanaged_frames' portion of the output also

This can help correlate the two lists when we have mixed managed & unmanaged stacks in our crash output.

* Wehn symbol/function address is not found, still output the IaP

* Don't set native_offset to ip, that's not what it means

* [merp] Bump protocol version

* Minor formatting fix


* Remove unused var

* Refactor filling in managed info for stack frame
2019-10-14 17:12:39 -04:00
monojenkins 3009440108 [2019-08][merp] Don't install SIGTERM handler in EnableMicrosoftTelemetry (#17308)
* [merp] Don't call install_state_summarizer in EnableMicrosoftTelemetry

install_state_summarizer installs a SIGTERM handler. Unfortunately
sigterm_signal_handler assumes that it will only be called when a crash is
already in progress (it assumes it won't be the supervisor, and so it doesn't
setup the memory for capturing a crash report among other things).

* [merp] Register SIGTERM handler in dump_native_stacktrace

Previously it was installed by EnableMicrosoftTelemetry, which is too early.
Now we get here without a handler installed even when mono_merp_enabled returns true.
2019-10-14 20:54:05 +02:00
Alexander Köplinger df5e13f95d [tests] Bump corefx to get Azure testhost change (#17275)
* Bump corefx to get Azure testhost change

See https://github.com/mono/corefx/pull/360

* Update ClientWebSocketTest as well

(cherry picked from commit d457346e06)
2019-10-11 09:39:00 +02:00
monojenkins 11e1499c22 [2019-08] [merp] Print missing status marker file for stage 1 (setup) (#17220)
* [merp] Print missing status marker file for stage 1 (setup)

* Zeroing is not necessary, snprintf() always null-terminates in this case

* Remove unused variable
2019-10-10 23:34:08 -04:00
Vlad Brezae 7dbad3c361 [arm] Fix fetching of method addresses (#17253)
[2019-08][arm] Fix fetching of method addresses

After https://github.com/mono/mono/commit/9ff3b0d65ee4, in an aot image we can emit addresses either as bl offset (which uses 4 bytes) or as ldr pc, =<label>; <addr> (which uses 8 bytes). Before this commit we were dereferencing an instruction pointer, in get_call_table_entry, assuming 8 bytes per entry. This could overflow and crash if we used in fact only 4 bytes per entry. Fix this by including the entry size in the aot image.

Fixes random crashes on arm CI on full-aotmixed suite.

Backport of https://github.com/mono/mono/pull/17099
2019-10-09 21:45:56 +02:00
monojenkins 9a88a36789 [sgen] Fix invalid value passed to write barrier (#17247)
[2019-08] [sgen] Fix invalid value passed to write barrier

When doing memory copy dest is the destination address and src is the source address. If we are copying a single reference, we need to emit a write barrier and pass to the write barrier the value, not the pointer where the value is stored. Otherwise nursery checks will fail and we won't mark the card table.

Backport of #17236.

/cc @marek-safar @BrzVlad
2019-10-09 15:46:05 +02:00
Alexander Köplinger 0f241c975c [2019-08] Add drawing type converters to mobile profiles (#17240)
* [2019-08] Add drawing type converters to mobile profiles

Needed for https://github.com/xamarin/xamarin-android/issues/3726

* Bump API snapshot submodule

* [csproj] Update project files
2019-10-09 12:16:40 +02:00
Ankit Jain 7ebe1a1763 Update Roslyn to 3.4.0-beta2-19477-01
.. to match roslyn-binaries PR: https://github.com/mono/roslyn-binaries/pull/19
2019-10-08 12:06:41 -05:00
Ankit Jain b759449ba8 Bump msbuild to track mono-2019-08
.. which updates SDKs and roslyn (match what mono has) in
https://github.com/mono/msbuild/pull/144 :

```
Microsoft.NET.Sdk:                      3.1.100-preview1.19506.1
Microsoft.NET.Sdk.Web:                  3.1.100-preview1.19506.3
Microsoft.NET.Sdk.Razor:                3.1.0-preview1.19506.2

Microsoft.NETCore.App:                  3.1.0-preview1.19506.1
Microsoft.DotNet.Cli.Runtime:           3.1.100-preview1.19506.9

NuGet.Build.Tasks:                      5.3.0-rtm.6251

Roslyn:                                 3.4.0-beta2-19477-01
Based on dotnet/toolset release/3.1.1xx branch HEAD
(e9c6794f1f124e273571237eb72a949ee5c950c6):

https://github.com/dotnet/toolset/blob/e9c6794f1f124e273571237eb72a949ee5c950c6/eng/Versions.props#L36
```
2019-10-08 12:06:41 -05:00
monojenkins 617f399efc [IO] Remove read-only logic in mono_w32_get_disk_free_space (#17211)
[2019-08] [IO] Remove read-only logic in mono_w32_get_disk_free_space

Fixes #17151

This brings us in line with GetDiskFreeSpaceExW on Windows as well as .NET Core.

I checked with the Android folks and it seems like this won't fail their test - all they check is that the returned value isn't `long.MaxValue`.

Backport of #17177.

/cc @akoeplinger @CoffeeFlux
2019-10-08 02:15:58 +02:00
monojenkins 77258ea112 [2019-08] [debugger][exception] Debugger breaks on handled exceptions (#17202)
* If there is a perform_wait_callback in the stack there will be another catch generated by the owner thread, so we don't need to throw, we can continue and find the next catch. Fixes #17083

* Reverting unit test changed on commit 405d521.

* Fixing assert when calling mono_jit_info_get_method if it was a trampoline.
2019-10-07 17:10:58 -04:00
Ankit Jain f83c321f88 Bump msbuild to track mono-2019-08 (#17193)
.. to track https://github.com/mono/msbuild/pull/143 .
	`Update SDKs to track vs16.4 and `.NET Core 3.1 Dev``
2019-10-07 16:10:44 +02:00
monojenkins 1ecd094b44 [2019-08] [Mono.Debugger.Soft] Fix VirtualMachine detaching (#17077)
* Revert "[Mono.Debugger.Soft] Properly close connections"

This reverts commit 2da7707373.

This was previously reverted because it was causing issues and it still does. It is a hack fix. We signal the connection for closing and next we send the dispose command over the network. This is obviously racy.

* [Mono.Debugger.Soft] Make sure receiving thread always closes

When we want to close the connection, setting the close flag is not enough, because a thread can be already stuck in a blocking receive, without getting to check the flag. We fix this by additionaly breaking the connection, by using socket.Shutdown.

Fixes https://github.com/mono/mono/issues/7377

* Bump API snapshot submodule
2019-10-06 09:57:37 +03:00
monojenkins 54a33be9de [merp] Put thread into async context before running summarizer (#17197)
[2019-08] [merp] Put thread into async context before running summarizer

Followup work for https://github.com/mono/mono/pull/17090

In order to get managed method info for unmanaged stacktraces, we call
`mini_jit_info_table_find_ext` which calls `decode_exception_debug_info` which may
allocate if it's not in async context.

Do the switch in `mono_threads_summarize_execute` so that non-supervising threads
all switch when they run the `sigterm_signal_handler`.  `mono_threads_summarize`
already does it for the supervisor thread.

Fixes https://github.com/mono/mono/issues/17180



Backport of #17194.

/cc @lambdageek
2019-10-05 21:05:59 +02:00
Alexander Köplinger 72128bb00d Bump libgdiplus to 6.0.4
(cherry picked from commit 0600ea1256)
2019-10-05 17:46:04 +02:00
Johan Lorensson 65a972c033 Always do copy_stack_data on entering GC safe/unsafe mode. (#17184) 2019-10-05 10:01:40 -04:00
monojenkins 9e6def1553 [merp] exit_status is 0 if we ran the uploader successfully (#17187)
Not 1 (TRUE)
2019-10-04 17:22:59 -04:00
monojenkins 8a707cc012 [2019-08] [reflection] Only duplicate MonoMarshalSpec strings for custom types (#17189)
[2019-08] [reflection] Only duplicate MonoMarshalSpec strings for custom types

Fixes #16570

We were treating the union as though it was a different type than it really is, and so we got junk data for the pointers and attempting to access them caused a crash.

Also fixed a typo seen along the way.

Backport of #17175.

/cc @lambdageek @CoffeeFlux
2019-10-04 22:46:03 +02:00
monojenkins bd72952cf8 [2019-08] [merp] Don't overrun buffer in copy_summary_string_safe … (#17178)
* [merp] Don't overrun buffer in copy_summary_string_safe

MonoFrameSummary:str_destr is an array of MONO_MAX_SUMMARY_NAME_LEN bytes, not
MONO_MAX_SUMMARY_NAME_LEN + 1 bytes.

Fixes Coverity CID 1454563

* [merp] Use g_strlcpy for copy_summary_string_safe

Fixes Coverity CID 1454563

We would sometimes write to MonoSummaryFrame:str_descr which is
MONO_MAX_SUMMARY_NAME_LEN bytes long at index MONO_MAX_SUMMARY_NAME_LEN which
is one past the end of the array.

* nit: rename confusing parameter names

old names were confusing - we were copying from 'out' to 'in'.  Now we copy to
'dest' from 'src'
2019-10-04 01:20:33 -04:00
Ankit Jain b6efc0cc90 Bump msbuild to track xplat-master (#17132)
.. which is tracking dotnet branch `release/3.1.1xx`.
2019-10-03 23:47:01 +02:00
monojenkins 2869cd5f67 Bump ikvm to get https://github.com/mono/ikvm-fork/pull/13 (#17170) 2019-10-03 20:46:55 +02:00
monojenkins a64a25695d [2019-08] [merp] Use macOS version not Darwin version in MERP reports (#17147)
* Use macOS version not Darwin version in MERP

Fixes #17004

* Update mono-merp.c

Ensure this is only for Mac

* Don't allocate in macos_version_string

Use a static buffer.

Also some formatting fixes.

* Simplify error path code when getting the product version


* Add G_STRING_CONSTANT_AND_LENGTH macro

Cherrypicked from
https://github.com/mono/mono/commit/8c288f06664c65050b153da0a8952209adc958a0
2019-10-03 10:44:50 -04:00
Alexis Christoforides 57f068438d [2019-08] [merp] Add API method that whitelists all native libraries (#17128)
Backport of #16899 and #15622

* [merp] Add API method that whitelists all native libraries

This is so that complex consumers such as Visual Studio for Mac can forego attempting to enumerate all loaded libraries, register them one at a time and potentially keep the list updated

* Set out_module, to avoid getting no useful information in the printout

It seems that without this, most/all entries in the unmanaged frames list are have no useful information.

* Add stub of new function when crash reporting is disabled

* Remove problematic assert from SIGILL handler

mono_handle_native_crash() function can actually return if the native crash has triggered or has been trigerred by  more than one signal. So it is possible to reach the point of the assert

* Guard writing to out_module

* Fix pointing to stack on string && some add tracing

* [merp] Fix logic of 'is_managed' attribute

* [merp] Output native library module name when 'whitelist all' mode is enabled (#16899)

Contributes to  https://github.com/mono/mono/issues/16689

* Bump MONO_CORLIB_VERSION
2019-10-01 22:08:28 -04:00
Alexander Köplinger 528103728f [2019-08] Update NETStandard2.1 APIs (#17123)
* Update NETStandard2.1 APIs (#17080)

Contributes to #17064

* Add System.Data.DataSetExtensions to mobile profiles

* Bump API snapshot submodule
2019-09-30 22:59:41 +02:00
monojenkins d8441deead [crashing] Remove Mono signal handlers when starting to handle a crash (#17078)
[2019-08] [crashing] Remove Mono signal handlers when starting to handle a crash

Fixes https://github.com/mono/mono/issues/16803

Backport of #17002.

/cc @marek-safar @alexischr
2019-09-27 01:36:00 +02:00
monojenkins 7ae0f38fd7 [runtime] Respect runtime_version in mono_init_internal (#17086)
[2019-08] [runtime] Respect runtime_version in mono_init_internal

In 6d2c77fb33 we changed the representation of
the runtimes that we will probe from a NULL-terminated array to a linked list.
Unfortunately we made a typo, and in the case where the caller of
`mono_init_internal` provided a runtime_version, we still picked
`DEFAULT_RUNTIME_VERSION` instead of the given `runtime_version`.

This is an issue for embedders like XA that use the "mobile" runtime - all
their BCL assemblies are compiled with the "mobile" version number (2.0.5.0) but
Mono running with the default runtime configuration will remap to 4.0.0.0.  As
a result, when there are references to 2.0.5.0 assemblies, the load requests
will be remapped to 4.0.0.0 which will never see the already loaded 2.0.5.0
assemblies, and we will fall back on the embedder's preload hook or filesystem
probing every single time.  This is a performance fail.




Backport of #17085.

/cc @lambdageek
2019-09-26 21:36:12 +02:00
Rolf Bjarne Kvinge 61ca7fe3a4 [sdks] Bump min iOS version to 7.0. (#17070)
Because that's the min iOS version in all current versions of Xamarin.iOS.
2019-09-26 14:14:22 +02:00
Steve Pfister c99adb97ed [2019-08] Fix time zone issue when jumping into DST (#17027)
There is an issue with jumping into DST for some time zones when the incorrect date-time offset is returned for date-time in UTC (which comes from DateTime.Now). The fix is to just check if the incoming date-time is in UTC
Also I added set of tests for some time zones but they verify jumping into DST in general.

Fixes #16395

Backport of https://github.com/mono/mono/pull/16430
2019-09-25 17:03:13 +02:00
Alexander Köplinger 6a8aced231 [2019-08] Bump bockbuild to get recent fixes (#17058)
- https://github.com/mono/bockbuild/pull/121
- https://github.com/mono/bockbuild/pull/122

Backport of https://github.com/mono/mono/pull/17054
2019-09-25 16:56:28 +02:00
monojenkins af008b207d [2019-08] [threads] clear small_id_key TLS when unregistering a thread (#17015)
[2019-08] [threads] clear small_id_key TLS when unregistering a thread

Fixes
```
* thread #12, name = 'tid_a507', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1be66144)
  * frame #0: 0x1be66144 libsystem_c.dylib`__abort + 184
    frame #1: 0x1be6608c libsystem_c.dylib`abort + 152
    frame #2: 0x003e1fa0 monotouchtest`log_callback(log_domain=0x00000000, log_level="error", message="* Assertion at ../../../../../mono/utils/hazard-pointer.c:158, condition `mono_bitset_test_fast (small_id_table, id)' not met\n", fatal=4, user_data=0x00000000) at runtime.m:1251:3
    frame #3: 0x003abf44 monotouchtest`monoeg_g_logv_nofree(log_domain=0x00000000, log_level=G_LOG_LEVEL_ERROR, format=<unavailable>, args=<unavailable>) at goutput.c:149:2 [opt]
    frame #4: 0x003abfb4 monotouchtest`monoeg_assertion_message(format=<unavailable>) at goutput.c:184:22 [opt]
    frame #5: 0x003904dc monotouchtest`mono_thread_small_id_free(id=<unavailable>) at hazard-pointer.c:0:2 [opt]
    frame #6: 0x003a0a74 monotouchtest`unregister_thread(arg=0x15c88400) at mono-threads.c:588:2 [opt]
    frame #7: 0x00336110 monotouchtest`mono_thread_detach_if_exiting at threads.c:1571:4 [opt]
    frame #8: 0x003e7a14 monotouchtest`::xamarin_release_trampoline(self=0x166452f0, sel="release") at trampolines.m:644:3
    frame #9: 0x001cdc40 monotouchtest`::-[__Xamarin_NSTimerActionDispatcher release](self=0x166452f0, _cmd="release") at registrar.m:83445:3
    frame #10: 0x1ce2ae68 Foundation`_timerRelease + 80
    frame #11: 0x1c31b56c CoreFoundation`CFRunLoopTimerInvalidate + 612
    frame #12: 0x1c31a554 CoreFoundation`__CFRunLoopTimerDeallocate + 32
    frame #13: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame #14: 0x1c2be6e8 CoreFoundation`__CFArrayReleaseValues + 500
    frame #15: 0x1c2be4c4 CoreFoundation`CFArrayRemoveAllValues + 104
    frame #16: 0x1c31ff64 CoreFoundation`__CFSetApplyFunction_block_invoke + 24
    frame #17: 0x1c3b2e18 CoreFoundation`CFBasicHashApply + 116
    frame #18: 0x1c31ff10 CoreFoundation`CFSetApplyFunction + 160
    frame #19: 0x1c3152cc CoreFoundation`__CFRunLoopDeallocate + 204
    frame #20: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame #21: 0x1c304a80 CoreFoundation`__CFTSDFinalize + 144
    frame #22: 0x1bfa324c libsystem_pthread.dylib`_pthread_tsd_cleanup + 644
    frame #23: 0x1bf9cc08 libsystem_pthread.dylib`_pthread_exit + 80
    frame #24: 0x1bf9af24 libsystem_pthread.dylib`pthread_exit + 36
    frame #25: 0x0039df84 monotouchtest`mono_threads_platform_exit(exit_code=<unavailable>) at mono-threads-posix.c:145:2 [opt]
    frame #26: 0x0033bb84 monotouchtest`start_wrapper(data=0x1609e1c0) at threads.c:1296:2 [opt]
    frame #27: 0x1bf9b914 libsystem_pthread.dylib`_pthread_body + 128
    frame #28: 0x1bf9b874 libsystem_pthread.dylib`_pthread_start + 44
    frame #29: 0x1bfa3b94 libsystem_pthread.dylib`thread_start + 4
```


Debugged by @lambdageek. Contributes to https://github.com/mono/mono/issues/10641

Edit: C/p into PR description so it's in the commit message:

We think what happens is:

1. `start_wrapper` runs `unregister_thread` when the thread is exiting.  At that point the `small_id` for the thread is cleared from the hazard pointer bitset by `mono_thread_small_id_free (small_id)`

2. Some other TLS destructor runs and attaches the thread again (which runs `mono_thread_info_register_small_id` which first calls `mono_thread_info_get_small_id` which tries to get the small id from the `small_id_key` TLS key and so the new `MonoThreadInfo` has the same `small_id` as the previously destroyed `MonoThreadInfo` - but the hazard pointer bitset is **not** updated).

3. This other TLS destructor runs and calls `mono_thread_detach_if_exiting` which calls `unregister_thread` again.

4. `unregister_thread` calls `	mono_thread_small_id_free (small_id)` a second time which asserts because we already cleared that id from the hazard pointer bitset.



Backport of #16973.

/cc @lewurm
2019-09-24 23:15:54 +02:00
monojenkins 6d40f13121 [2019-08] [debugger] Crash when debugging iOS application that throws (#16989)
* Doing the same fix from Android to IOS, I removed the ifdef, I don't think that this will happen when not using IOS nor Android, but I prefer to let the if there to avoid the exception. On mini-exceptions.c check if there is no ji we can continue and generate the exception without checking wrapper_type.

Fixes #16824

* Removing unnecessary check.
2019-09-24 20:50:09 +02:00
Jo Shields 4546440471 Add mono-hang-watchdog.in to tarball (#17018)
Remove custom distdir target and use standard EXTRA_DIST to do so.

(cherry picked from commit 75c21692a0)
2019-09-24 16:09:53 +02:00
monojenkins 4c31bdabf3 [2019-08] [debugger] New way to filter exceptions to support VSWin features (#17000)
* Creating a new version of MOD_KIND_EXCEPTION_ONLY to support VsWin features, discussed with Joaquin Jares. Because we don't have this functionality of get every other exception that is not specified, VsWin was getting every exception and filtering later. But this causes a slowness during the debugger because we stop all thread on each exception that we get, even if they are caught and we don't want to stop.

* Fixing other unit test because now we have 2 nested classes. Fixing indentation.

* Creating a flag in the MOD_KIND_EXCEPTION_ONLY protocol event and not creating a new protocol event.

* Changing what was suggested by Zoltan.

* Coding style

* Bump API snapshot submodule
2019-09-24 11:08:20 -03:00
Alexander Köplinger d73a286a2c [ci] Make ios/mac sdks archive URL more predictable
Requested by xamarin-macios team.

(cherry picked from commit 0a3c9c451b)
2019-09-24 15:27:12 +02:00
monojenkins dacef517fa [merp] Use function names even in 'private crashes' mode (#17003)
Partially fixes https://github.com/mono/mono/issues/16689

Backport of #16897.
2019-09-24 11:41:06 +02:00
monojenkins d1ca1a5d2e [2019-08] [System.Xml] Fix deserialization issue (#17014)
[This ](https://github.com/mono/mono/pull/11194/files#diff-ae54d9565e735005686191de0f19ed50R431-R435)change in `XmlTypeMapElementInfo.GetElement`  introduced the logic for handling the case of derived types deserialization. 
For some cases (like https://github.com/mono/mono/issues/16918) it breaks existing functionality because it returns wrong element for given element name after searching withing derived types collection.

![image](https://user-images.githubusercontent.com/3258267/65416085-bedc0500-ddff-11e9-941c-f24045c64121.png)

Unfortunately, what I can do here is just reverting the related changes.

Fixes https://github.com/mono/mono/issues/16918

Backport of #16965.

/cc @marek-safar @MaximLipnin
2019-09-24 11:35:39 +02:00
monojenkins 1d55b355f3 Correctly check for HAVE_STAT_BIRTHTIME in configure.ac checks for System.Native (#17001)
The check was done incorrectly, the include files need to be separated by newlines otherwise we would get a false negative result.

config.log contained this:

```
conftest.c:279:23: warning: extra tokens at end of #include directive [-Wextra-tokens]
#include <sys/types.h>, #include <sys/stat.h>
                      ^
                      //
```

This means that when doing File.GetCreationTime() we'd fall back to returning the last modified time.

It regressed when we switched to System.IO.File to the CoreFX implementation i.e. System.Native in https://github.com/mono/mono/commit/8f5cef936491e8b20888bbf18d426482c890637c.

Fixes https://github.com/mono/mono/issues/16974
2019-09-24 10:12:30 +02:00
monojenkins bb522edb0d [2019-08] Restore Previous ExecutionContext Capture Behavior (#16997)
Restores previous ExecutionContext Capture behavior where we were copying a new SynchronizationContext as opposed to a reference to the existing one.

The old behavior caused SynchronizationContextAwaitTaskContinuations to not post to a SynchronizationContext.

Fixes https://github.com/mono/mono/issues/16759

Reverts a fix for https://github.com/mono/mono/issues/12421

Backport of https://github.com/mono/mono/pull/16945
2019-09-23 19:06:16 -04:00
Bernhard Urban 7f7b5c9f3e [interp] avoid exception checkpoint when in thread is in GC Safe state (#16967)
[2019-08] [interp] avoid exception checkpoint when in thread is in GC Safe state

Backport of https://github.com/mono/mono/pull/16955
2019-09-23 21:46:06 +02:00
Alexander Köplinger c87f99dd81 [ci] Fix typo in sdks-archive.groovy 2019-09-23 15:53:44 +02:00
Alexander Köplinger 065499c2ef [ci] Fix typo in sdks-archive.groovy 2019-09-23 15:48:59 +02:00
Alexander Köplinger 9f7b2c8d0a [ci] Fix typo in sdks-archive.groovy 2019-09-23 15:44:44 +02:00
Alexander Köplinger 2c003ec71d [ci] Build iOS/Mac Mono sdks archive using Xcode 11
This was requested by the xamarin-macios team.

(cherry picked from commit 6ad929c317)
2019-09-23 15:43:33 +02:00
Bernhard Urban e02e823ef7 [arm64_32] fix pointer size in CallInfo
Note that in the output below `a`, `b`, etc. are NSObjects, so they are really passed as pointer.

Before:
```
* thread #1, name = 'tid_303', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x04f301d4 aWatchOSExtension`::xamarin_localized_string_format_9(format="hello%@%@%@%@%@%@%@%@%@", a=0, b=1, c=2, d=3, e=4, f=5, g=6, h=7, i=0x00000000) at nsstring-localization.m:81:9
   78   void *
   79   xamarin_localized_string_format_9 (NSString *format, id a, id b, id c, id d, id e, id f, id g, id h, id i)
   80   {
-> 81           return [NSString localizedStringWithFormat: format, a, b, c, d, e, f, g, h, i];
   82   }
   83
   84   }
(lldb) mbt 4
* thread #1
  * frame #0: 0x04f301d4 aWatchOSExtension`::xamarin_localized_string_format_9(format="hello%@%@%@%@%@%@%@%@%@", a=0, b=1, c=2, d=3, e=4, f=5, g=6, h=7, i=0x00000000) at nsstring-localization.m:81:9
    frame #1: 0x04d9984c aWatchOSExtension`interp_to_native_trampoline + 156
    frame #2: 0x04f41d5c aWatchOSExtension`ves_pinvoke_method(frame=0x05167af8, sig=0x1518afd0, addr=(aWatchOSExtension`::xamarin_localized_string_format_9(NSString *, id, id, id, id, id, id, id, id, id) at nsstring-localization.m:80), string_ctor=0, context=0x14550960, save_last_error=0) at interp.c:1411:2 [opt]
    NSString::xamarin_localized_string_format_9 @ 68 "calli.nat" || frame #3: 0x04f3bfa4 aWatchOSExtension`interp_exec_method_full(frame=0x05167d38, context=<unavailable>, clause_args=0x00000000, error=0x05168540) at interp.c:3290:5 [opt]
```

After:
```
* thread #1, name = 'tid_303', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x04c2c1e0 aWatchOSExtension`::xamarin_localized_string_format_9(format="hello%@%@%@%@%@%@%@%@%@", a=0, b=1, c=2, d=3, e=4, f=5, g=6, h=7, i=8) at nsstring-localization.m:81:9
   78   void *
   79   xamarin_localized_string_format_9 (NSString *format, id a, id b, id c, id d, id e, id f, id g, id h, id i)
   80   {
-> 81           return [NSString localizedStringWithFormat: format, a, b, c, d, e, f, g, h, i];
   82   }
   83
   84   }
```
2019-09-23 11:53:10 +02:00
monojenkins 70d6903053 [2019-08] [merp] Use a separate program as the hang supervisor. (#16900)
* [merp] Use a separate program as the hang supervisor.

Fixes https://github.com/mono/mono/issues/15646

macOS does not like signals being sent from the forked supervisor process, possibly towards anywhere but definitely when sent to the parent process. The following message is currently spewed after the supervisor process attempting to send a SIGSEGV to a hanged Mono process:
"The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec()."

We follow that direction and introduce a new binary that, when available in the mono executable's binary directory, is used to abort the parent process for us.

* Experiment: remove line possibly causing bad compilation under C++
2019-09-20 16:21:34 +02:00
Rolf Bjarne Kvinge 4bff2b6370 [offsets-tool] Install clang into the user-specific python directory.
Because the system location isn't world writeable on Catalina.
2019-09-20 16:19:44 +02:00
EgorBo 81894ec8ca Implement WriteCore and ReadCore in DeflateStream 2019-09-20 16:18:28 +02:00
Alexander Köplinger bfbf823ca1 [ci] Remove more XCODE32_DIR usages (#16964)
Follow-up to https://github.com/mono/mono/pull/16936

(cherry picked from commit 74b9ca426a)
2019-09-20 14:01:06 +02:00
Alexander Köplinger ce01b20a4d Add net_4.8.xml to EXTRA_DIST and bump binary-reference-assemblies again
The file was missed in https://github.com/mono/mono/pull/16912.
Also pull in another reference assemblies update.

(cherry picked from commit 857284b412)
2019-09-20 12:00:41 +02:00
Alexander Köplinger 7a587d7fa6 Add .NET 4.8 reference assemblies (#16912)
Brings in https://github.com/mono/reference-assemblies/pull/20

(cherry picked from commit f6a080bb79)
2019-09-20 12:00:35 +02:00
Rolf Bjarne Kvinge 35e454a8f6 [sdks] Remove the mac32 build. (#16936)
Xamarin.Mac doesn't need nor use it anymore.

This also removes the last part of the mono archive that required Xcode 9.4.
2019-09-19 14:34:30 -05:00
monojenkins 75eb342f53 [2019-08] [System] Make FileSystemWatcher backend non-static (#16926)
[2019-08] [System] Make FileSystemWatcher backend non-static

The FileSystemWatcher `watcher` field is the backend IFileWatcher that's
supposed to be used for the actual operations.

Each backend has a `bool GetInstance (out watcher)` method that returns a
singleton object, so each new instance of FileSystemWatcher gets the same
IFileWatcher instance.  (They will get different `watcher_handle` instances
which are used by some backends to uniquely identify this FileSystemWatcher).

However when the first FileSystemWatcher instance is `Dispose`d, it will set
`watcher = null` which will prevent the remaining instances from calling

    watcher?.StopDispatching (watcher_handle);
    watcher?.Dispose (watcher_handle);

which means that the backend won't properly cleanup resources for any other FSW
instances that have other `watcher_handle` values.

Addresses https://github.com/mono/mono/issues/16709



<!--
Thank you for your Pull Request!

If you are new to contributing to Mono, please try to do your best at conforming to our coding guidelines http://www.mono-project.com/community/contributing/coding-guidelines/ but don't worry if you get something wrong. One of the project members will help you to get things landed.

Does your pull request fix any of the existing issues? Please use the following format: Fixes #issue-number
-->


Backport of #16922.

/cc @steveisok @lambdageek
2019-09-19 14:45:54 -04:00
monojenkins 5881981f79 [2019-08] [mini] Add missing membars when initializing rgctx entries (#16909)
[2019-08] [mini] Add missing membars when initializing rgctx entries

Whenever we are publishing changes to the rgctx arrays, we are racing with rgctx_lazy_fetch_trampoline, so we need to make sure any pointers we set there must have their contents initialized.


Backport of #16904.

/cc @marek-safar @BrzVlad
2019-09-18 11:36:20 -04:00
monojenkins 6290b6cd6e Temporarily disable embedded ppdb data decompression (#16911)
Backport of #16868.
2019-09-18 14:57:42 +02:00
monojenkins a0e7f9eaf2 [2019-08] [arm64_32] make "Debug Mode" work on Watch series 4 with --interpreter (#16886)
[2019-08] [arm64_32] make "Debug Mode" work on Watch series 4 with --interpreter

Main objective for reviewers: This PR shouldn't change anything for `arm64`.

Context: "Debug Mode" on the Apple Watch allows execution of regular machine code and thus is pretty similar to FullAOT on iPhone, i.e. it doesn't require bitcode, but _still_ doesn't allow `signal(3)` so cooperative suspend is required.

Apple Watch Series 4 and onward use an ARM64 chip, but with a different ABI, also known as `arm64_32`. This PR changes the AOT compiler and related trampolines so that the AOT compiler can generate all necessary code to run the interpreter on this target.


Changes:

--------------------------------------

[arm64_32] set register size to 8 in cross compiler too 

-----------------------------------------

 [machine] update `target_mgreg_t` definition

`SIZEOF_REGISTER` is usually the same as `TARGET_SIZEOF_VOID_P`, except when `MONO_ARCH_ILP32 is defined.

`target_mgreg_t` should represent the width of the hardware register, that is, `SIZEOF_REGISTER`.

---------------------------------------------

[arm64_32] fix AOT image layout 

---------------------------------------------

 [arm64_32] make trampolines bitwidth aware

* replace pointer loads/stores with bitwidth-dependent macro
* reflect `specific` trampolines so that only two instructions are used for ILP32, since the related pair in the data page only needs 2 * 32bits too
* replace some `sizeof (target_mgreg_t)` with `TARGET_SIZEOF_VOID_P` (former is hardware register size, latter is pointer size)

---------------------------------------------
These regression tests are _not_ passing: https://github.com/mono/mono/issues/16864 Also some of monotouch-tests are crashing, see https://github.com/mono/mono/issues/16819

Here is a screenshot of a debug session in VSMac:
<img width="1321" alt="Screenshot 2019-09-16 at 19 13 22" src="https://user-images.githubusercontent.com/75403/64981326-1dfbc000-d8bc-11e9-9d16-48036f36b638.png">


Contributes to https://github.com/mono/mono/issues/10641


Backport of #16865.

/cc @lewurm
2019-09-17 17:06:03 -04:00
monojenkins 6275840a7f Rename bundle identifier for the various Mono.frameworks we create for Xamarin.iOS. Fixes xamarin/xamarin-macios#7005. (#16901)
This is the equivalent of
https://github.com/xamarin/xamarin-macios/commit/a9abc07cb357f9f012e3da66c70fa09d96bf02f7,
just for the Mono.frameworks in the mono archive.

Fixes https://github.com/xamarin/xamarin-macios/issues/7005.
2019-09-17 21:18:06 +02:00
monojenkins 25f6093283 [corlib] Fix building nunit-lite twice (#16895)
We sometimes saw the following build error in Mono SDKs builds on Jenkins:

```
[2019-09-16T01:03:30.709Z] CSC     [xammac] nunit-lite-console.exe
[2019-09-16T01:03:30.709Z] error CS0009: Metadata file '/Users/builder/jenkins/workspace/archive-mono/2019-08/mac/release/mcs/class/lib/xammac/nunitlite.dll' could not be opened -- PE image doesn't contain managed metadata.
```

Taking a closer look we found out that nunit-lite was trying to be built twice for the same profile which caused a race where the file was corrupted.

The reason for this is that the corlib Makefile defines two test assemblies: the normal PROFILE_corlib_test.dll and BinarySerializationOverVersionsTest.dll and
integrating the BinarySerializationOverVersionsTest.dll via the `test-vts` target incorrectly overwrote the `test` target instead of using `test-local`.

Backport of #16894.
2019-09-17 20:26:09 +02:00
monojenkins 7ec17ba1be [2019-08] [android sdk] Add aprofutil tool (#16884)
Backport of #16866.
2019-09-17 15:58:17 +02:00
monojenkins f755f3b539 [metadata] Fix leaks when handling a few attributes (#16850)
[2019-08] [metadata] Fix leaks when handling a few attributes

Callers of mono_reflection_create_custom_attr_data_args_noalloc were leaking some of the returned information. Accessed attributes are FixedBufferAttribute and UnmanagedFunctionPointerAttribute.

Backport of #16675.

/cc @BrzVlad
2019-09-16 20:16:02 -04:00
monojenkins 5f9a2db39b [2019-08] Fix infrequent hangs in test-runner. (#16854)
* Fix infrequent hangs in test-runner.

test-runner has been plagued with very infrequent hangs causing long
timeouts on CI. Turns out that there is a race in AsyncStreamReader
used by Process class for handling stdout/stderr when closing down
process. Since AsyncStreamReader::Dispose didn't make sure no pending
async read requests were still in flight before closing the stream
and underlying handle, it led to a race condition deadlocking the
complete process, hanging test-runner. A similar race has also been
observed but instead of causing a deadlock, it could also manifest as an
System.ObjectDisposedException when doing EndRead while another thread
is disposing the underlying stream. Hitting that race didn't deadlock
the process but failed to complete invoke of test-runner.

Fix is to better protect the async callback and the thread doing the
dispose call. In order to make sure we don't close underlying handle
while still having async reads in flight, fix also checks async result
for completion and if not completed when trying to dispose AsyncStreamReader
it will try to cancel outstanding pending IO and wait for completion
before closing the stream.

Since the issue has only been seen on Windows, the fix will try
to cancel pending IO and wait for completion before closing stream
on Windows. On other platforms, old behavior will be preserved, meaning
that the implementation will still be vunarable to race between pending
reads and close of stream, but since problem has not been observed on
other platforms and since there is no good way of cancel outstanding
blocking reads, old behavior is kept.

At some point in future we should look into replacing the implementation
of AsyncStreamReader and use cancelation tokens, but since we don't have
all corefx support around cancelation in Mono BCL, that currently won't
solve the issue, meaning that we need to bring in more classes. Since the
problem also occur in test-runner using the build BCL profile, even more
needs to be added to that profile in order to get similar support. But due
to the infrequency of the problem and that it has only been observed on
Windows, the isolated fix seems more reasonable.

Backport of #16793.
2019-09-16 16:32:33 +02:00
Bernhard Urban f31f5ea1f1 [2019-08] [threads] do not convert NULL thread name (#16828)
Avoids this warning printed in the VSMac application output when debugging:
```
2019-09-13 17:47:27.315540+0200 aWatchOSExtension[31768:24665480] ../../../../../mono/eglib/giconv.c:1029: assertion 'str != NULL' failed
```

With this trace:
```
(lldb) mbt
* thread #2
  * frame #0: 0x00583453 aWatchOSExtension`break_on_me at giconv.c:1023:2
    frame #1: 0x005834a6 aWatchOSExtension`monoeg_g_utf16_to_utf8(str=0x00000000, len=0, items_read=0x00000000, items_written=0x00000000, err=0x00000000) at giconv.c:1036:3
    frame #2: 0x0046d871 aWatchOSExtension`mono_thread_get_name_utf8(thread=0x24d08120) at threads.c:1812:16
    frame #3: 0x002cb7ef aWatchOSExtension`thread_commands(command=2, p="�>\x02\x18\x10", end="�>\x02\x18\x10", buf=0xb0ae0d20) at debugger-agent.c:8818:13
    frame #4: 0x002c7170 aWatchOSExtension`debugger_thread(arg=0x00000000) at debugger-agent.c:9932:10
    frame #5: 0x0047652b aWatchOSExtension`start_wrapper_internal(start_info=0x00000000, stack_ptr=0xb0ae1000) at threads.c:1221:3
    frame #6: 0x0047616c aWatchOSExtension`start_wrapper(data=0x7b63a050) at threads.c:1294:8
    frame #7: 0x077365f8 libsystem_pthread.dylib`_pthread_body + 137
    frame #8: 0x077397f7 libsystem_pthread.dylib`_pthread_start + 78
    frame #9: 0x077357ce libsystem_pthread.dylib`thread_start + 34
```

Introduced by b5c0c831da and seemingly fixed by accident in https://github.com/mono/mono/commit/7b64f1cd005ef4f8e7f0aa5655debca2754b1755#diff-f1a9287559d3aa620b058619f47f00c9R1832

Instead of backporting 7b64f1cd00 to 2019-08, I'm proposing this PR for the release branch.
2019-09-15 19:52:52 -05:00
monojenkins 20308e6f87 [aot] Do not wrap tool_prefix path when calling strip (#16820)
[2019-08] [aot] Do not put quotes around wrapped path

Should fix: https://github.com/xamarin/xamarin-android/issues/3619

As it results in double quoting, which vanishes the effect and breaks
AOT for XA on Windows.

The actual error:

    [aot-compiler stdout] Stripping the binary: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Stable\MSBuild\Xamarin\Android\ndk\arm-linux-androideabi-"strip" --strip-symbol=\$a --strip-symbol=\$d obj\Release\90\aot\armeabi-v7a\libaot-Xamarin.Android.Arch.Core.Common.dll.so.tmp
    [aot-compiler stderr] '""C:\Program' is not recognized as an internal or external command,
    [aot-compiler stderr] operable program or batch file.
    [aot-compiler stderr] AOT of image C:\Users\peter\source\repos\App9\App9\App9.Android\obj\Release\90\android\assets\Xamarin.Android.Support.CoordinaterLayout.dll failed.


Backport of #16792.

/cc @lewurm @radekdoulik
2019-09-13 13:56:04 -04:00
Radek Doulik cecda47c48 [aprofutil] Add -p and -f options
-p PORT allows easier retrieval of AOT profile from the application
 running with aot profiler, using the socket connection on local PORT

-f instructs the tool to try setup adb port forwarding to/from Android
 device or emulator

Example usage, retrieve the AOT profile from Android:

    > mono aprofutil.exe -s -v -f -p 9998 -o my.aprof
    Calling 'adb forward tcp:9998 tcp:9998'...
    Reading from '127.0.0.1:9998'...
    Read total 94903 bytes...
    Summary:
    	Modules:          6
    	Types:          262
    	Methods:      1,162
    Going to write the profile to 'my.aprof'
2019-09-12 13:05:43 +02:00
Steve Pfister 824cc12ac3 Bump to mono/corefx@e79cf5b
Backport of https://github.com/mono/mono/pull/16774
2019-09-12 08:57:24 +02:00
Alexander Köplinger b77dc06a7e [aprofutil] Install the tool correctly (#16112)
It needed a few tweaks to be usable when Mono is installed to the prefix.
2019-09-11 12:08:38 -05:00
Radek Doulik 1848d78d60 [aotprof-tool] Initial import of AOT profiler tool (#15384)
* [aotprof-tool] Initial import

Imported https://github.com/radekdoulik/aotprofile-tool and updated it
to use Mono.Profiler.Log

* [aotprof-tool] Added man page

* [aotprof-tool] Use the shortened name in README.md
2019-09-11 12:08:38 -05:00
monojenkins da0086e304 [2019-08] Add RenamedEvent* to FSW sources from CoreFX (#16756)
Add RenamedEvent* to FSW sources from CoreFX

Fixes #16486 - This was only partially fixed until now

* [csproj] Update project files

* Add test files which test these code paths on corefx
2019-09-11 12:11:57 -04:00
monojenkins 0297b21b03 [msbuild][roslyn] Bump msbuild and roslyn to pull in new versions (#16768) 2019-09-11 07:42:21 -05:00
monojenkins 40631e3b9e [2019-08] [aot] move method_addresses to data.rel.so section to avoid text relocations (#16751)
[2019-08] [aot] move method_addresses to data.rel.so section to avoid text relocations

Another revert of a revert! 🎉 

After some discussions with @grendello and @radekdoulik, we came up with this solution. It's a better attempt to fix https://github.com/mono/mono/issues/16369 as it (1) works within the restrictions of Android 10 and (2) still works for armv7 AOT. 


`.text` relocations aren't allowed in Android 10 (and this change shouldn't hurt on Linux).

AOT compilation:
```console
$ MONO_PATH=/Users/lewurm/Downloads/xamarin.android-oss-v10.0.99.136_Darwin-x86_64_pr_766265b9-Debug/bin/Debug/lib/xamarin.android/xbuild-frameworks/MonoAndroid/v1.0/ \
MONO_ENV_OPTIONS="" \
./sdks/out/android-cross-arm-release/bin/armv7-linux-android-mono-sgen \
--aot=keep-temps,outfile=here.dll.so,asmwriter,mtriple=armv7-linux-gnueabi,tool-prefix=/Users/lewurm/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-,ld-flags= \
~/Downloads/xamarin.android-oss-v10.0.99.136_Darwin-x86_64_pr_766265b9-Debug/bin/Debug/lib/xamarin.android/xbuild-frameworks/MonoAndroid/v10.0/Mono.Android.dll

Mono Ahead of Time compiler - compiling assembly /Users/lewurm/Downloads/xamarin.android-oss-v10.0.99.136_Darwin-x86_64_pr_766265b9-Debug/bin/Debug/lib/xamarin.android/xbuild-frameworks/MonoAndroid/v10.0/Mono.Android.dll
AOTID 7DDD05DA-37E0-813A-8C27-3E1634391BB3
Compiled: 144382/144382
Executing the native assembler: "/Users/lewurm/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-as"   -mfpu=vfp3 -o /var/folders/p3/5279mmgn1p575bz28j0ngfqw0000gn/T/mono_aot_550yHJ.o /var/folders/p3/5279mmgn1p575bz28j0ngfqw0000gn/T/mono_aot_550yHJ
Executing the native linker: "/Users/lewurm/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ld"  -shared -o here.dll.so.tmp  /var/folders/p3/5279mmgn1p575bz28j0ngfqw0000gn/T/mono_aot_550yHJ.o
Stripping the binary: "/Users/lewurm/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-strip" --strip-symbol=\$a --strip-symbol=\$d here.dll.so.tmp
Retained input file.
JIT time: 15031 ms, Generation time: 7374 ms, Assembly+Link time: 75130 ms.
```

Before:
```console
$ grep -C 6 'method_addresses:' /var/folders/p3/5279mmgn1p575bz28j0ngfqw0000gn/T/mono_aot_550yHJ
        .align 3
jit_code_end:

        .byte 0,0,0,0
.text 1
        .align 3
method_addresses:
        .local method_addresses
        .type method_addresses,#function
ldr pc,=.Lm_0
.ltorg
ldr pc,=.Lm_1
.ltorg
$ arm-linux-androideabi-readelf -a here.dll.so | grep -i textrel
 0x00000016 (TEXTREL)                    0x0
 0x0000001e (FLAGS)                      TEXTREL
```

After switching to `.data.rel.so`:
```console
$ grep -C 6 'method_addresses:' /var/folders/p3/5279mmgn1p575bz28j0ngfqw0000gn/T/mono_aot_1YhAgn  # assembly file produced by AOT compiler
jit_code_end:

        .byte 0,0,0,0
.section ".data.rel.ro"
.subsection 0
        .align 3
method_addresses:
        .local method_addresses
        .type method_addresses,#object
ldr pc,=.Lm_0
.ltorg
ldr pc,=.Lm_1
.ltorg
$ arm-linux-androideabi-readelf -a here.dll.so | grep -i textrel
$ echo $?
1
```

Backport of #16729.

/cc @lewurm
2019-09-10 12:35:47 -04:00
Thays Grazia 68b77674e2 Vtable [i] can be null so this should be check before use it. Fixes #16712 2019-09-09 19:17:14 +02:00
Bernhard Urban 4a0b4f41ed [mini] publish global patches after JitInfo has been added
Fixes https://github.com/mono/mono/issues/14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in https://github.com/mono/mono/issues/14080

As far as I can tell this only affects ARM64, ARM and PPC.
2019-09-09 18:33:23 +02:00
Thays Grazia 7a1f63fde6 [debugger][android] It was not initialising seq_points on MonoCompile on Android, so when was compiling dynamic methods, seq_points wasn't created and we got the assert when try to single step. 2019-09-06 23:28:08 +02:00
Martin Baulig 29b1ac19c9 Fix the tests. 2019-09-03 19:24:31 -05:00
Martin Baulig 71e34edce2 X509CertificateImplBtls.PrivateKey should allow to actually set the key. 2019-09-03 19:24:31 -05:00
monojenkins 1ecb6e59de [2019-08] [corlib] Add hack for Hebrew calendar data (#16641)
* [corlib] Add hack for Hebrew calendar data

* Fix month names for the Hebrew calendar

* [csproj] Update project files
2019-09-03 14:14:33 -04:00
monojenkins 17ba402d80 [2019-08] Remove MAX_PATH usage in w32process (#16595)
* Remove MAX_PATH usage in w32process

* Fixes

* Shrink buffer temporarily

* More fixes

* Reset buffer size to 260

* Fix length check and add notes

* Actually free things

* Add tracing and zero len at start

* Typo

* Fix another leak

* Simplify logic
2019-09-02 14:10:13 -05:00
monojenkins ded02bc115 [2019-08] [debugger][android] Fixing single step inside Exception on Android (#16597)
* Fixing single step inside Exception on Android... The method is null because on ss_create_init_args the IL is -1 when calling method Mono_UnhandledException_internal what I think it's expected, because of that I ignore the single step on android if there is no method. (debugger-engine.c) After that the sequence point cannot be found because it's calling a method of class DynamicMethodNameCounter what I think it's expected, because of that I remove the assert on android. (debugger-agent.c) Fixes #14772

* Changing comment to make it clear...
2019-08-31 08:23:52 -04:00
monojenkins 585f244aed [2019-08] Bump ikvm to pick up caa8e7f54279a47422626005d228447f5d3670d5. (#16574)
* Bump ikvm to pick up caa8e7f54279a47422626005d228447f5d3670d5.

* [monodis] Add a missing cattr parent case.
2019-08-30 12:28:45 +02:00
Alexander Köplinger 23cc371009 Build pkg-config as 64bit only, libtiff as universal and remove gtk-quartz-engine.py (#16548)
See https://github.com/mono/bockbuild/pull/116

(cherry picked from commit 309cf1bcee)

# Conflicts:
#	external/bockbuild
2019-08-29 20:31:53 +02:00
monojenkins ce6fa810dd [2019-08] Use the right FileSystemWatcher sources (#16554)
* Use the right FileSystemWatcher sources

The corefx code for FileSystemEventArgs has some changes to support
constructing one with a null filename.

Fixes #16486

* Remove the mono sources for the newly replaced files

* [csproj] Update project files
2019-08-29 14:35:26 +02:00
monojenkins bff42f3540 [2019-08] Implement Read and WriteAsyncMemory in Mono DeflateStream (#16549)
Resolves #16122

Since we put in the corefx version of GZipStream, it is expected that
our version of DeflateStream actually implements Write+ReadAsyncMemory.

This change does that by just calling the plain WriteAsync & ReadAsync
methods until we decide (or not) to pull in the corefx DeflateStream.

Backport of #16523.
2019-08-29 14:34:58 +02:00
monojenkins 076bab44f6 [2019-08] [arm/arm64] fix fconv/rconv issues with LLVM (#16532)
* [ci] add make -C mono/mini llvmaotcheck to FullAOT LLVM lanes

* [arm64] use opcode emulation for fconv/rconv

LLVM eagerly optimizes constants such as NaN in a way that does not align with .NET behaviour. In the future we can change it so that we only do opcode emulation when LLVM is used.

Fixes https://github.com/mono/mono/issues/16411

* [arm] account only 4 bytes on stack for single precision arguments

Fixes `test_0_arm64_small_stack_args` in gshared.cs when
* `test_0_arm64_small_stack_args` is compiled with LLVM
* `Foo3.Floats` is compiled with Mini

* [arm] use opcode emulation for fconv/rconv

* [llvmaotcheck] require sse3 being available for regression tests

* [amd64] use opcode emulation for fconv/rconv

* fix cast and signature for mono_rconv_u4
2019-08-28 13:28:52 +02:00
monojenkins 75f2ac15bd [threadpool-io] Throw if backend won't be able to register a FD (#16500)
The poll+select i/o selector backend can't handle file descriptor ids greater
than FD_SETSIZE.  This can happen if too many files are open and we want to
wait on it.

Previously, mono would fail in the i/o selector thread by which point it was
too late to do anything.

With this change we will fail eagerly on the thread that calls IOSelector.Add
by throwing a NotSupportedException.

Addresses https://github.com/mono/mono/issues/15931
2019-08-27 15:17:41 -04:00
Steve Pfister 44e1854943 Allow file:// based Uri's to be used in HttpClient when the BaseAddress is set
Fixes https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/967582

Bumps the HttpClient part in mono/corefx at: https://github.com/mono/corefx/commit/ee3acec16b37bec5016da129e1e3840097ca629b

The test that would have caught this failure was disabled previously via:
https://github.com/mono/mono/commit/2f411750acfd2132f9b4970ef8cf11d9a0f62df9

(cherry picked from commit 580defd967)
2019-08-27 20:02:46 +02:00
Alexander Köplinger 53cc8a61dc [acceptance-tests] Bump roslyn test repo to remove a problematic nuget feed
(cherry picked from commit 8d422436f5)
2019-08-27 19:48:29 +02:00
monojenkins 6717bcb8b6 [2019-08] [w32socket] Translate some errno codes to WSA_ errors imprecisely. (#16488)
* [w32socket] Translate some errno codes to WSA_ errors imprecisely.

These aren't a perfect match, but better than falling into the `default` case
where Mono asserts and crashes.

Addresses part of https://github.com/mono/mono/issues/16024

* [w32error] Add WSA_INVALID_PARAMETER and WSA_INVALID_HANDLE
2019-08-27 01:28:14 +02:00
monojenkins d82d780e42 [2019-08] [android] remove text relocations in AOT/arm32 (#16472)
* Revert "[aot] partial revert of #14043 (#14135)"

This reverts commit 223e0bd78d.

* Revert "[aot] change method table generation so that it works for larger assemblies on ARM (#14043)"

This reverts commit 9ff3b0d65e.
2019-08-27 01:25:25 +02:00
Alexander Köplinger 13bc35face [utils] Don't use MAP_32BIT on Apple platforms, fixes crash with XCode 11 beta6 (#16480)
As of XCode 11 beta6 the MacOSX SDK defines the MAP_32BIT symbol.

This causes the `mono_valloc()` function to try to use the `MAP_32BIT` flag for `mmap()`.

However as mentioned in the comment for the symbol in mman.h it seems to require a special entitlement which isn't available/documented anywhere yet.
This in turn causes the mmap call to fail presumably because we're missing that entitlement.

Instead we now skip setting this flag on Apple platforms to make `mmap()` behave like it did before.

Backport of https://github.com/mono/mono/pull/16441
2019-08-26 19:39:50 +02:00
monojenkins 8df98444c1 Pass --clr-memory-model in csc/vbc wrapper script (#16444)
[2019-08] Pass --clr-memory-model in csc/vbc wrapper script

Should help with random CSC crashes on ARM (we've been using it in the Mono build for a while).


Backport of #16439.

/cc @akoeplinger
2019-08-26 08:15:49 -04:00
monojenkins d63841057e [coop] Add a handle frame in construct_culture (#16442)
[2019-08] [coop] Add a handle frame in construct_culture

Otherwise we see warnings like these:

```
ves_icall_System_Globalization_CultureInfo_internal_get_cultures_raw USED 680 handles
```


Backport of #16299.

/cc @akoeplinger @lambdageek
2019-08-26 07:30:48 -04:00
monojenkins dc98767858 Fix issue in SafeHandle.DangerousReleaseInternal() that caused #16034. (#16423)
Using the CoreCLR version of this line fixed the problem and my investigations
also revealed why.

The `SafePipeHandle` contains a reference to both the Socket as well as the
`SafeSocketHandle`, taking ownership of them.  If there are no additional
references to either of them, so that both are collected in the same GC pass,
their finalizer may be invoked in any order.

If the Socket gets finalized first, it's `SafeSocketHandle` will have a `_mstate`
of 8 (two references) at the beginning of `DangerousReleaseInternal()` - which
will set it to 6 (one reference plus disposed flag).

Then when the `SafePipeHandle` gets collected, it will call `DangerousRelease ()`
on it's `SafeSocketHandle`.  As mentioned above, it's `_mstate` will be 6.
Now we own the handle, so we'll decrease it's reference count by one and set the
`Closed` flag - the new `_mstate` should thus be 3.

However, using

    new_state = (old_state & RefCount_Mask) - RefCount_One;

would set it to 1 - thus effectively clearing the `Disposed' flag.  And since it's
reference count is now also zero, the next finalize pass will then throw.

The CoreCLR version of this class uses

    new_state = old_state - RefCount_One;

which fixes this.

Backport of #16244.
2019-08-23 15:33:56 +02:00
Steve Pfister 472f0820d3 [2019-08] Add Compiler Server Workaround for 32-bit Linux (#16421)
Currently, if the process is 32-bit, roslyn tries to determine how much
memory is available. To do that, it tries to pinvoke into GlobalMemoryStatusEx,
which is not supported on mono.

Without it, the compiler server bombs on https://github.com/dotnet/roslyn/blob/0e63260c5afb3fb5b74c357dd250e500172bcd63/src/Compilers/Server/VBCSCompiler/NamedPipeClientConnection.cs#L55-L59

Backport of https://github.com/mono/mono/pull/16405
2019-08-23 14:48:26 +02:00
Alexander Köplinger 2ad05ee764 Bump corefx to get https://github.com/mono/corefx/pull/336 2019-08-23 14:44:13 +02:00
monojenkins 32ded2e340 [android] mono_dl_open_file: use g_file_test only on absolute paths (#16393)
The intention of calling `g_file_test (file, G_FILE_TEST_EXISTS)` where file is
the name of a shared library we want to open is to speed up probing for
non-existent libraries.

See https://github.com/mono/mono/pull/12074

The problem is that if file is just a simple "libdl.so" then `dlopen (file)`
doesn't just look for it in the current working directory, it will probe some
other paths too.  (For example on desktop linux you'd also look in all the
directories in LD_LIBRARY_PATH).  So the g_file_test() call is not a robust way
to avoid calling dlopen if the filename is relative.

But it actually broke more things: https://github.com/xamarin/xamarin-android/issues/3388

When probing for "libdl.so" on Android mono_lookup_pinvoke_call will first try
prepending some paths that it knows about and we end up calling
`dlopen ("/system/lib/libdl.so")` which will fail because Bionic has security
restrictions on what code can dlopen something from /system/lib with an
absolute path.  Eventually mono_lookup_pinvoke_call will go back to trying the
bare "libdl.so" which hits `g_file_test` and returns NULL.

The new code only does the file test if we pass it an absolute path, which
gives Bionic's dlopen a chance to deal with relative paths however it needs to.
2019-08-22 13:43:27 -04:00
monojenkins d80d33b053 [2019-08] [profiler] Rename helper functions to have common prefix (#16372)
* [profiler] Rename helper functions to have common prefix

Unfortunately https://github.com/mono/mono/pull/16335 didn't fix the new symbols showing up in the log profiler module.
The reason is that we actually pass `--disable-visibility-hidden` for iOS builds in https://github.com/mono/mono/blob/0b6d95e8ce7c882e2db9e89fa0dec850c687f65e/sdks/builds/ios.mk#L98.

Rename the helper functions instead to have a common prefix so they can be ignored in xamarin-macios tests.

* Fix Windows build
2019-08-20 22:57:39 +02:00
Alexander Köplinger 04c9f2f442 Bump corefx to get https://github.com/mono/corefx/pull/332 2019-08-20 20:53:25 +02:00
Jo Shields c0f0ff02fc Pass -fvisibility=hidden to profiler modules. (#16335)
* Pass -fvisibility=hidden to profiler modules.

This appears to not break the profilers:

```
directhex@breakfast:/tmp$ nm -g /tmp/ponyponyponypony/lib/libmono-profiler-log.so | grep close_socket_fd
directhex@breakfast:/tmp$
directhex@breakfast:/tmp$ /tmp/ponyponyponypony/bin/mono --profile=log:report bottle.exe
99 bottles of beer on the wall, 99 bottles of beer.
Take one down and pass it around, 99 bottles of beer on the wall.

[..]

No more bottles of beer on the wall, no more bottles of beer.
Go to the store and buy some more, 99 bottles of beer on the wall.

Mono log profiler data
	Profiler version: 3.0
	Data version: 17
	Arguments: log:report
	Architecture: x86-64
	Operating system: linux
	Mean timer overhead: 86 nanoseconds
	Program startup: Mon Aug 19 14:18:30 2019
	Program ID: 17715
	Server listening on: 41539

JIT summary
	Compiled methods: 753
	Generated code size: 228046
	JIT helpers: 0
	JIT helpers code size: 0

GC summary
	GC resizes: 0
	Max heap size: 0
	Object moves: 0

Metadata summary
	Loaded images: 3
	Loaded assemblies: 3

Exception summary
	Throws: 0
	Executed finally clauses: 3

Thread summary
	Thread: 0x7fcd968c9700, name: "Finalizer"
	Thread: 0x7fcd966c8700, name: "Profiler Sampler"
	Thread: 0x7fcd9b036780, name: "Main"
	Thread: 0x7fcd964c7700, name: "Profiler Helper"
	Thread: 0x7fcd954c5700, name: "Profiler Dumper"
	Thread: 0x7fcd95cc6700, name: "Profiler Writer"

Domain summary
	Domain: (nil), friendly name: "bottle.exe"

Context summary
	Context: (nil), domain: (nil)

Counters:
	Mono System:
		User Time                      : 70.000ms
		System Time                    : 0.000ms
		Total Time                     : 140.000ms
		Working Set                    : 22032384
		Private Bytes                  : 63455232
		Virtual Bytes                  : 449630208
		Page Faults                    : 3004
		CPU Load Average - 1min        : 1.870000
		CPU Load Average - 5min        : 2.670000
		CPU Load Average - 15min       : 2.900000
	Mono JIT:
		Methods from AOT               : 536
		Methods JITted using mono JIT  : 154
		Methods JITted using LLVM      : 0

```

* Add missing annotations and includes

(cherry picked from commit 3070ab6a96)
2019-08-20 16:18:12 +02:00
monojenkins a8c9ef3b66 [2019-08] Add missing Dispose() call on error in MonoTlsStream.CreateStream(). (#16328)
* Add missing `Dispose()` call on error in `MonoTlsStream.CreateStream()`.

* Remove the `finally` block to make it more easy to read.
2019-08-20 08:34:15 -04:00
Alexander Köplinger 31f480c436 Bump bockbuild to get gtksharp 2019-08-20 11:52:04 +02:00
monojenkins 6df5f69b26 [2019-08] [corlib] Fix sharing mode validation rules (#16281)
Fixes https://github.com/mono/mono/issues/16032

When we switched to corefx File.Copy on unix, the behavior changed where it opened a source and destination stream in order to pass the safe handles to Interop.Sys.CopyFile. This exposed a flaw in our sharing validation rules in MonoIO.Open, which threw an exception in this scenario:

new FileStream("source.txt", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite, 2048, FileOptions.DeleteOnClose);
new FileStream("source.txt", FileMode.Open, FileAccess.Read, FileShare.Read, 2048, FileOptions.None);

Simplified the FileShare check down to:

1. If the caller specifies FileShare.None, then it's an exclusive lock and no
   sharing is allowed.

2. If the caller specifies some other FileShare option, then FileShare.None
    cannot be specified.
2019-08-16 07:55:46 -04:00
monojenkins 94ec04bfdb Exclude Authenticode tests on Watch (#16283)
1002a4de2b added Authenticode cert support on non-WatchOS. The tests,
however, were not non-WatchOS-only. Fix this, so Watch doesn't try to
run Authenticode tests.
2019-08-16 00:31:59 -04:00
monojenkins a5184ed8ed Add missing FCONV_TO_U8 on ILP32 (WatchOS 5) (#16277) 2019-08-15 13:57:36 -04:00
monojenkins 4087eba140 Fix failing OSX Android lane (#16275)
Linker order matters when using .a files, so zlib.a MUST come after
sgen.a if sgen.a was compiled with support for compressed ppdb
2019-08-15 10:40:36 -04:00
monojenkins 2ef68dc530 [cominterop] Fix copy-paste error in ComVisibleAttribute testing helper. (#16269)
This type was introduced with 29a95bafd8.

Backport of #16254.
2019-08-15 14:36:55 +02:00
monojenkins 64f9c2de6f [2019-08] Fix linking of OS-provided zlib (#16245)
* Fix linking of OS-provided zlib

This is a reduced version of https://github.com/mono/mono/pull/16091
without the Z_PREFIX stuff

* Get rid of HAVE_ZLIB, be explicit about HAVE_SYS_ZLIB vs HAVE_STATIC_ZLIB

Backport of #16177.
2019-08-14 21:20:12 +02:00
monojenkins 2f373a9ad4 (mono.1) Update the MONO_THREADS_SUSPEND description (#16237) 2019-08-14 19:23:47 +02:00
monojenkins c51c60988b [sre] Outline typebuilder_setup_fields loop body (#16204)
The loop body uses coop handles, so wrap it in HANDLE_FUNCTION_ENTER/HANDLE_FUNCTION_RETURN

Addresses part of #16179

Backport of #16184.
2019-08-14 12:29:43 +02:00
Marek Safar f333a19c3a Bump linker 2019-08-14 10:18:03 +02:00
Alexander Köplinger abf46116ef [pkg] Bump libgdiplus to 6.0.2
(cherry picked from commit b60aaea791)
2019-08-13 12:13:33 +02:00
Sebastien Pouliot ec590ad3eb [2019-08][sdk] Enable the SRE runtime code inside iOS builds
This will allow the interpreter to use System.Reflection.Emit.

Note: There is a small size increase (about 50kb) for an application
built in release mode with the LLVM backend (even if the interpreter
is now used).

Backport of https://github.com/mono/mono/pull/16151
2019-08-12 21:43:36 +02:00
UnknownShadow200 dc534a8d3f [WinForms] Fix NotifyIcon not showing in systray on 64bit linux
https://bugzilla.xamarin.com/show_bug.cgi?id=14976

Credit goes to datnhanlz for noting that removing XChangeProperty(DisplayHandle, hwnd.whole_window, _XEMBED_INFO, _XEMBED_INFO, 32, PropertyMode.Replace, atoms, 2); would also fix the bug.

Note that the type of Atom is actually unsigned long (see usr/include/X11/X.h), however the code used a C# int for the atoms. As such the code worked without a problem on 32 bit linux, but broke on 64 bit linux.

I switched to using IntPtr instead to make the code also work on 64-bit.
2019-08-12 21:22:30 +02:00
Ankit Jain 835c26154c [msbuild] Bump to track xplat-master and update roslyn (#16154)
Upstream msbuild doesn't have a branch for vs16.4, so tracking master
for now.

Prompted by mono/msbuild#126:

This includes:

- merge mono-2019-06
- SDK update to track dotnet release/3.0.100-preview8
- Roslyn updated to 3.3.0-beta2-19381-14
2019-08-12 17:03:28 +02:00
Nikita Voronchev 03be2643f5 [System.Drawing.Design] Allow context be a null in the ColorEditor.EditValue (#16174)
Backport of #16141
2019-08-12 12:26:54 +02:00
Alexander Köplinger f69128c70a Bump corefx to fix build break 2019-08-10 23:47:59 +02:00
Alexander Köplinger ac12ebc375 Bump corefx to get https://github.com/mono/corefx/pull/327 2019-08-10 16:22:01 +02:00
Alexander Köplinger 595b7821e3 Bump corefx to get https://github.com/mono/corefx/pull/323 2019-08-09 16:20:31 +02:00
Alexander Köplinger 409c0a7c2d [ci] Trigger PR builds for netcore only on master branch
The work is happening in master only for now.

(cherry picked from commit fde0a5b628)
2019-08-09 01:00:47 +02:00
Alexis Christoforides a9658665bf [2019-08] [System] Guard against attempting to creating out-of-range spans in m… (#16124)
* [System] Guard against attempting to creating out-of-range spans in macOS FileSystemWatcher

Backport of #16049
2019-08-09 00:16:37 +02:00
Alexander Köplinger 4fd89f9b11 [tests] Fix accessing field of MarshalByRefObject in remoting1.cs (#16119)
We need to copy it to a local variable first.
This was exposed by a new Roslyn optimization for String.Concat (see https://github.com/dotnet/roslyn/issues/37830).

(cherry picked from commit ec35e14636)
2019-08-09 00:13:22 +02:00
monojenkins 0707345b89 [2019-08] Fixed RuntimeType.GetConstructorCandidates("") to return 0 elements. (#16109)
* Fixed RuntimeType.GetConstructorCandidates("") to return 0 elements.

https://github.com/mono/mono/issues/16010

The current behavior for GetConstructorCandidates when the name is null, "",
or "*" is to return the set of constructors for the type. For "", .NET Core and
.NET Framework return 0 elements.  This PR adds a check for "" to match that
behavior.

* Removed unncessary changes

* Unnecessary spaces

* Made the if check a bit more readable

Backport of #16054.
2019-08-08 16:49:04 +02:00
monojenkins 0bb83d064d [2019-08] Bump linker (#16088)
* Bump linker

* Bump api-doc-tools and add NEW_CECIL define to mdoc

* Bump linker

* Bump API snapshot submodule

* [csproj] Update project files
2019-08-08 12:12:33 +02:00
monojenkins 3115d3a0c0 [2019-08] [arm] respect I1 and I2 on gsharedvt out transition (#16081)
* [arm] respect I1 and I2 on gsharedvt out transition

In order to trigger the problem, we must call into gsharedvt code ("in" transition) and then escape the gsharedvt world ("out" transition). The bug happens in the latter. Also some other stars need to align (I haven't fully understood it), thus the repro is quite long.

The provided repro only triggers with LLVM, which we don't have enabled on CI yet: https://github.com/mono/mono/issues/15999

Fixes https://github.com/mono/mono/issues/15261
Fixes https://github.com/mono/mono/issues/15262
Fixes https://github.com/mono/mono/issues/15263
Fixes https://github.com/mono/mono/issues/15307

* fix it on amd64 too

* more amd64
2019-08-07 12:43:11 +02:00
Thays Grazia 8c27e2adf9 allow emitting DIM 2019-08-07 09:26:13 +02:00
Aleksey Kliger f8e9a15ed7 [w32socket] Translate ELOOP and ENAMETOOLONG
Translate to WSAELOOP and WSAENAMETOOLONG, respectively.

See https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2
2019-08-07 09:23:30 +02:00
monojenkins d1bd30f22d [2019-08] Bump api-doc-tools to latest master (#16050)
* Bump api-doc-tools to latest master

Disable building mdoc.exe in mcs mode since it no longer builds because of new C# features.

* Fix mcs build by disabling more mdoc invocations

* Bump monodroid_tools to compile against v4.7 reference assemblies

mdoc needs ValueTuple which is not in v4.6.

Backport of #16035.
2019-08-06 20:19:27 +02:00
monojenkins 1c57af4dc0 [2019-08] DeflateStream: check IsInvalid before disposing (#16053)
Backport of #16048.
2019-08-06 14:54:56 +02:00
Jo Shields 85cecac278 Bump version to 6.6 (#16029)
* Bump version to 6.6
2019-08-05 15:31:59 -04:00
2972 changed files with 156097 additions and 154706 deletions

No files matched your search

+1 -7
View File
@@ -1,6 +1,3 @@
# diff settings
*.cs diff=csharp
# ensure LF endings on all checkouts
configure.ac crlf=input
config.rpath crlf=input
@@ -10,7 +7,6 @@ mkinstalldirs crlf=input
*.sources crlf=input
.gitattributes crlf=input
*akefile* crlf=input
m4/mono-output.m4 crlf=input
# ensure native line endings on checkout
*.c crlf
@@ -20,10 +16,8 @@ m4/mono-output.m4 crlf=input
*.sln crlf
*.*proj* crlf
*.bat text eol=crlf
*.cmd text eol=crlf
# don't do anything to line-endings. Let CRLFs/LFs go into the repo, and CRLF/LFs on checkout
*.bat -crlf
*.xml -crlf
# CRLF Handling
+31 -39
View File
@@ -15,15 +15,15 @@
# Order is important. The last matching pattern has the most precedence.
/acceptance-tests @akoeplinger @kg
/acceptance-tests/profiler-stress* @BrzVlad @akoeplinger
/acceptance-tests/profiler-stress* @alexrp @akoeplinger
/data @directhex @akoeplinger
/data/gdb @vargaz
/data/lldb @vargaz
/data/gdb @lewurm @vargaz
/data/lldb @lewurm @vargaz
/docs @marek-safar @akoeplinger
/docs/riscv.md @alexrp
/docs/sources/mono-api-profiler.html @BrzVlad
/docs/riscv.md @alexrp @lewurm
/docs/sources/mono-api-profiler.html @alexrp
/external @akoeplinger @marek-safar
/external/api-snapshot @marek-safar @akoeplinger
@@ -34,16 +34,16 @@
/external/corefx @akoeplinger @marek-safar @egorbo @steveisok
/external/corert @akoeplinger @marek-safar @egorbo @steveisok
/external/linker @marek-safar
/external/llvm-project @SamMonoRT @vargaz @imhameed
/external/llvm @SamMonoRT @vargaz
/external/nuget-buildtasks @radical @akoeplinger
/external/roslyn-binaries @kg @marek-safar @akoeplinger @lewing
/libgc @BrzVlad @vargaz @naricc
/llvm @vargaz @SamMonoRT @imhameed
/llvm @vargaz @SamMonoRT
/man @marek-safar @miguel
/man/*prof* @BrzVlad
/man/*prof* @alexrp
/mcs @marek-safar @akoeplinger @egorbo
@@ -65,24 +65,24 @@
/mcs/class/System/System.Security.Cryptography.X509Certificates @baulig @steveisok
/mcs/class/System.Data* @egorbo @steveisok
/mcs/errors @marek-safar
/mcs/ilasm @marek-safar
/mcs/jay @marek-safar
/mcs/mcs @marek-safar
/mcs/tests @marek-safar
/mcs/errors @marek-safar @lewurm
/mcs/ilasm @marek-safar @lewurm
/mcs/jay @marek-safar @lewurm
/mcs/mcs @marek-safar @lewurm
/mcs/tests @marek-safar @lewurm
/mcs/tools @marek-safar @akoeplinger @steveisok
/mcs/tools/gensources @kg @akoeplinger
/mcs/tools/resx2sr @baulig
/mcs/tools/upload-to-sentry @directhex @akoeplinger
/mono/arch @vargaz
/mono/arch @vargaz @lewurm
/mono/btls @baulig @marek-safar
/mono/dis @lambdageek @vargaz
/mono/eglib @vargaz @lambdageek
/mono/benchmark @SamMonoRT
/mono/benchmark @SamMonoRT @lewurm
/mono/metadata @vargaz @lambdageek @thaystg
/mono/metadata/*profiler* @lambdageek
@@ -92,36 +92,27 @@
/mono/metadata/thread* @lateralusX @lambdageek
/mono/metadata/w32* @lateralusX @lambdageek
/mono/metadata/*-win* @lateralusX @lambdageek
/mono/metadata/appdomain* @vargaz @lambdageek @CoffeeFlux
/mono/metadata/assembly* @vargaz @lambdageek @CoffeeFlux
/mono/metadata/domain* @vargaz @lambdageek @CoffeeFlux
/mono/metadata/loader* @vargaz @lambdageek @CoffeeFlux
/mono/metadata/native-library* @vargaz @lambdageek @CoffeeFlux
/mono/metadata/*-unix* @lambdageek @CoffeeFlux
/mono/mini @vargaz @lambdageek @SamMonoRT
/mono/mini/*cfgdump* @vargaz
/mono/mini/*exceptions* @vargaz @BrzVlad
/mono/mini @vargaz @lewurm @lambdageek @SamMonoRT
/mono/mini/*cfgdump* @lewurm @vargaz
/mono/mini/*exceptions* @lewurm @vargaz
/mono/mini/*ppc* @lewurm @vargaz
/mono/mini/*riscv* @alexrp @lewurm
/mono/mini/*type-check* @lambdageek
/mono/mini/*llvm* @vargaz @imhameed
/mono/mini/interp/* @BrzVlad
/mono/mini/*profiler* @BrzVlad @lambdageek
/mono/mini/*type-check* @lewurm @lambdageek
/mono/mini/interp/* @lewurm @BrzVlad
/mono/mini/*profiler* @alexrp @lambdageek
/mono/mini/debugger-agent.c @vargaz @thaystg @DavidKarlas @lambdageek
/mono/native @baulig @egorbo @alexischr @marek-safar
/mono/profiler @BrzVlad @lambdageek
/mono/profiler @alexrp @lambdageek
/mono/sgen @BrzVlad @lambdageek
/mono/tools/offsets-tool @rolfbjarne
/mono/utils @vargaz @lambdageek
/mono/utils/atomic* @vargaz
/mono/utils/mono-hwcap* @vargaz
/mono/utils/mono-mem* @vargaz
/mono/utils @vargaz @lewurm @lambdageek
/mono/utils/atomic* @lewurm @vargaz
/mono/utils/mono-hwcap* @lewurm @vargaz
/mono/utils/mono-mem* @lewurm @vargaz
/mono/utils/mono-merp* @alexischr @lambdageek
/mono/utils/mono-state* @alexischr @lambdageek
/mono/utils/mono-threads* @lambdageek @vargaz
@@ -137,14 +128,14 @@
/runtime @akoeplinger @marek-safar
/samples @lambdageek
/samples/profiler @BrzVlad @lambdageek
/samples @lambdageek @lewurm
/samples/profiler @alexrp @lambdageek
/samples/size @alexrp @lambdageek
/scripts @akoeplinger @directhex
/scripts/ci @akoeplinger @directhex
/sdks @marek-safar @vargaz @lambdageek
/sdks @marek-safar @lewurm @vargaz @lambdageek
/sdks/builds/android* @alexischr @jonpryor
/sdks/builds/mxe* @alexischr @jonpryor
/sdks/builds/ios* @vargaz @rolfbjarne
@@ -158,5 +149,6 @@
/tools @lambdageek @vargaz
/tools/locale-builder @marek-safar @egorbo
/tools/nuget-hash-extractor @marek-safar @radical @akoeplinger
/tools/pedump @lambdageek
/tools/offsets-tool @lewurm @rolfbjarne
/tools/pedump @lambdageek @lewurm
/tools/sgen @brzvlad @vargaz @naricc
+1 -1
View File
@@ -43,5 +43,5 @@ Please paste the stack trace here if available.
```
<!--
You can join us on Discord (https://aka.ms/dotnet-discord) in the #monovm channel to discuss your reported issue
You can join us at https://gitter.im/mono/mono to discuss your reported issue
-->
+1 -1
View File
@@ -18,7 +18,6 @@
*.vssscc
*_i.c
*_p.c
*.mdb
*.ncb
*.suo
*.tlb
@@ -141,6 +140,7 @@ sdks/builds/llvm-*/
sdks/builds/android-*/
mcs/build/compiler-server.log
##############################################################################
# Arcade output directories
##############################################################################
+4 -7
View File
@@ -58,16 +58,13 @@
[submodule "external/api-snapshot"]
path = external/api-snapshot
url = git://github.com/mono/api-snapshot.git
[submodule "external/llvm"]
path = external/llvm
url = git://github.com/mono/llvm.git
branch = release_60
[submodule "external/helix-binaries"]
path = external/helix-binaries
url = git://github.com/mono/helix-binaries.git
[submodule "external/illinker-test-assets"]
path = external/illinker-test-assets
url = git://github.com/mono/illinker-test-assets.git
[submodule "external/llvm-project"]
path = external/llvm-project
url = git://github.com/dotnet/llvm-project.git
branch = release/6.x
[submodule "external/bdwgc"]
path = external/bdwgc
url = git://github.com/Unity-Technologies/bdwgc.git
+1 -1
View File
@@ -76,7 +76,7 @@ The Licenses
These are the licenses used in Mono, the files are located:
### MIT License
### MIT X11 License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
+51 -15
View File
@@ -2,11 +2,33 @@ ACLOCAL_AMFLAGS = -I m4
AM_CFLAGS = $(WERROR_CFLAGS)
SUBDIRS = @MONO_SUBDIRS@
noinst_SUBDIRS = @MONO_NOINST_SUBDIRS@
DIST_SUBDIRS = $(SUBDIRS) m4 netcore
MONOTOUCH_SUBDIRS = $(libgc_dir) mono
if !ENABLE_NETCORE
# Some tools might not build when cross-compiling
if CROSS_COMPILING
tools_dir =
else
tools_dir = tools
endif
if ENABLE_MSVC
build_with_msvc = msvc
build_without_msvc =
else
build_with_msvc =
build_without_msvc = msvc
endif
if ENABLE_NETCORE
update_submodules =
SUBDIRS = mk llvm mono runtime netcore
else
update_submodules = update_submodules
SUBDIRS = $(build_with_msvc) mk po $(libgc_dir) llvm mono $(ikvm_native_dir) support data runtime scripts man samples $(tools_dir) $(build_without_msvc) $(docs_dir) acceptance-tests
endif
# Keep in sync with SUBDIRS
DIST_SUBDIRS = $(build_with_msvc) m4 mk po $(libgc_dir) llvm mono ikvm-native support data runtime scripts man samples tools $(build_without_msvc) docs acceptance-tests netcore
all: $(update_submodules)
@@ -24,10 +46,9 @@ EXTRA_DIST= \
winconfig.h \
code_of_conduct.md \
external \
netcore/Makefile \
mcs/class/referencesource
DISTCHECK_CONFIGURE_FLAGS = EXTERNAL_RUNTIME=false
DISTCHECK_CONFIGURE_FLAGS = EXTERNAL_MCS=false EXTERNAL_RUNTIME=false
# Distribute the 'mcs' tree too
GIT_DIR ?= $(srcdir)/.git
@@ -45,9 +66,6 @@ dist-hook:
rm -f `find $(top_distdir)/external -path '*\.exe' -not -path '*/roslyn-binaries/*'`
rm -f `find $(top_distdir)/external -path '*\.dll' -not -path '*/binary-reference-assemblies/*' -not -path '*/roslyn-binaries/*' -not -path '*/helix-binaries/*'`
rm -rf "$(top_distdir)/external/linker/test"
rm -rf "$(top_distdir)/external/llvm-project/lldb/test"
rm -rf "$(top_distdir)/external/llvm-project/libcxx/test"
rm -rf "$(top_distdir)/external/llvm-project/clang/test"
pkgconfigdir = $(libdir)/pkgconfig
noinst_DATA = mono-uninstalled.pc
@@ -94,10 +112,30 @@ mcs-do-compiler-tests:
bootstrap-world: compiler-tests
$(MAKE) install
install:
for mydir in $(filter-out $(noinst_SUBDIRS),$(SUBDIRS)); do \
(cd $${mydir} && ${MAKE} install) \
done
if INSTALL_MONOTOUCH
monotouch-do-build: config.h
@list='$(MONOTOUCH_SUBDIRS)'; for subdir in $$list; do \
case "x$$subdir" in \
xmono ) target="monotouch-do-build";; \
* ) target="all";; \
esac; \
echo "Making $$target in $$subdir"; \
(cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$target); \
done;
(cd runtime && $(MAKE) $(AM_MAKEFLAGS) monotouch-do-build)
monotouch-do-clean:
@list='$(MONOTOUCH_SUBDIRS)'; for subdir in $$list; do \
case "x$$subdir" in \
xmono ) target="monotouch-do-clean";; \
* ) target="clean";; \
esac; \
echo "Making $$target in $$subdir"; \
(cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$target); \
done;
(cd runtime && $(MAKE) $(AM_MAKEFLAGS) monotouch-do-clean)
endif
update-csproj:
-rm msvc/scripts/order
@@ -141,5 +179,3 @@ update-solution-files:
update-solution-files-with-tests:
$(MAKE) "GENPROJ_ARGS=2012 true true" update-solution-files
endif
+1 -1
View File
@@ -2,7 +2,7 @@
<configuration>
<packageSources>
<clear />
<add key="dotnet-eng" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json" />
<add key="dotnet-core" value="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json" />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
</packageSources>
</configuration>
+6 -7
View File
@@ -5,13 +5,7 @@ standards for C# and the Common Language Runtime.
The Mono project is part of the [.NET Foundation](https://www.dotnetfoundation.org/)
Join us on [Discord](https://aka.ms/dotnet-discord) in the `#monovm` channel:
<a href="https://aka.ms/dotnet-discord">
<img src="https://img.shields.io/discord/732297728826277939?style=flat-square&label=Discord&logo=discord&logoColor=white&color=7289DA">
</a>
### Contents
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/mono/mono?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
1. [Compilation and Installation](#compilation-and-installation)
2. [Using Mono](#using-mono)
@@ -440,6 +434,7 @@ Disables compilation for the SSA optimization
framework, and the various SSA-based optimizations.
* `--enable-llvm`
* `--enable-loadedllvm`
* This enables the use of LLVM as a code generation engine
for Mono. The LLVM code generator and optimizer will be
@@ -452,6 +447,10 @@ full details and up-to-date information on this feature.
* You will need to have an LLVM built that Mono can link
against.
* The `--enable-loadedllvm` variant will make the LLVM backend
into a runtime-loadable module instead of linking it directly
into the main mono binary.
* `--enable-big-arrays` - Enable use of arrays with indexes larger
than Int32.MaxValue.
+9 -30
View File
@@ -104,10 +104,12 @@ if grep "^AM_PROG_LIBTOOL" configure.ac >/dev/null; then
fi
fi
# Parse parameters
#
# Plug in the extension module
#
has_ext_mod=false
ext_mod_args=''
has_disable_boehm=false
for PARAM; do
if [[ $PARAM =~ "--enable-extension-module" ]] ; then
has_ext_mod=true
@@ -115,14 +117,8 @@ for PARAM; do
ext_mod_args=`echo $PARAM | cut -d= -f2`
fi
fi
if [[ $PARAM =~ "--disable-boehm" ]] ; then
has_disable_boehm=true
fi
done
#
# Plug in the extension module
#
if test x$has_ext_mod = xtrue; then
pushd $top_srcdir../mono-extensions/scripts
sh ./prepare-repo.sh $ext_mod_args || exit 1
@@ -154,34 +150,17 @@ automake $am_opt ||
echo "Running autoconf ..."
autoconf || { echo "**Error**: autoconf failed."; exit 1; }
# Update all submodules recursively to ensure everything is checked out
if test -e $srcdir/scripts/update_submodules.sh; then
(cd $srcdir && scripts/update_submodules.sh)
fi
if test x$has_disable_boehm = xfalse -a -d $srcdir/external/bdwgc; then
echo Running external/bdwgc/autogen.sh ...
(cd $srcdir/external/bdwgc ; NOCONFIGURE=1 ./autogen.sh "$@")
echo Done running external/bdwgc/autogen.sh ...
if test -d $srcdir/libgc; then
echo Running libgc/autogen.sh ...
(cd $srcdir/libgc ; NOCONFIGURE=1 ./autogen.sh "$@")
echo Done running libgc/autogen.sh ...
fi
if test x$MONO_EXTRA_CONFIGURE_FLAGS != x; then
echo "MONO_EXTRA_CONFIGURE_FLAGS is $MONO_EXTRA_CONFIGURE_FLAGS"
fi
host_conf_flag=
build_uname_all=`(uname -a) 2>/dev/null`
case "$build_uname_all" in
CYGWIN*)
if [[ "$@" != *"--host="* ]]; then
echo "Missing --host parameter, configure using ./configure --host=i686-w64-mingw32 or --host=x86_64-w64-mingw32"
echo "Falling back using --host=x86_64-w64-mingw32 as default."
host_conf_flag="--host=x86_64-w64-mingw32"
fi
;;
esac
conf_flags="$MONO_EXTRA_CONFIGURE_FLAGS --enable-maintainer-mode --enable-compile-warnings $host_conf_flag" #--enable-iso-c
conf_flags="$MONO_EXTRA_CONFIGURE_FLAGS --enable-maintainer-mode --enable-compile-warnings" #--enable-iso-c
if test x$NOCONFIGURE = x; then
echo Running $srcdir/configure $conf_flags "$@" ...
+48 -4
View File
@@ -1,6 +1,50 @@
# Code of Conduct
# Contributor Code of Conduct
This project has adopted the code of conduct defined by the Contributor Covenant
to clarify expected behavior in our community.
As contributors and maintainers of this project, and in the interest of
fostering an open and welcoming community, we pledge to respect all people who
contribute through reporting issues, posting feature requests, updating
documentation, submitting pull requests or patches, and other activities.
For more information, see the [.NET Foundation Code of Conduct](https://dotnetfoundation.org/code-of-conduct).
We are committed to making participation in this project a harassment-free
experience for everyone, regardless of level of experience, gender, gender
identity and expression, sexual orientation, disability, personal appearance,
body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery
* Personal attacks
* Trolling or insulting/derogatory comments
* Public or private harassment
* Publishing other's private information, such as physical or electronic
addresses, without explicit permission
* Other unethical or unprofessional conduct
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
By adopting this Code of Conduct, project maintainers commit themselves to
fairly and consistently applying these principles to every aspect of managing
this project. Project maintainers who do not follow or enforce the Code of
Conduct may be permanently removed from the project team.
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting a project maintainer at miguel@xamarin.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. Maintainers are
obligated to maintain confidentiality with regard to the reporter of an
incident.
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 1.3.0, available at
[http://contributor-covenant.org/version/1/3/0/][version]
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/3/0/
+402 -627
View File
File diff suppressed because it is too large. Load diff
+13 -13
View File
@@ -1,9 +1,8 @@
import lldb
import traceback
def print_frames(thread, num_frames, current_thread):
# TODO: Make output header similar to bt.
print('%c thread #%i' % ('*' if current_thread else ' ', thread.idx))
print '%c thread #%i' % ('*' if current_thread else ' ', thread.idx)
if current_thread:
selected_frame = thread.GetSelectedFrame()
@@ -15,16 +14,17 @@ def print_frames(thread, num_frames, current_thread):
if function_name == "interp_exec_method_full":
try:
s = 'frame->imethod->method'
methoddesc = frame.EvaluateExpression('(char*) mono_method_full_name (' + s + ', 1)').summary[1:-1]
klassname = frame.EvaluateExpression('(char*) ' + s + '->klass->name').summary[1:-1]
methodname = frame.EvaluateExpression('(char*) ' + s + '->name').summary[1:-1]
ipoffset = frame.EvaluateExpression('ip').GetValueAsUnsigned()
insn = ''
if ipoffset != 0:
ipoffset -= frame.EvaluateExpression('frame->imethod->code').GetValueAsUnsigned()
insn = "\"" + frame.EvaluateExpression('mono_interp_opname (*ip)').summary[1:-1] + "\""
var = '%s @ %d %s || %s' % (methoddesc, ipoffset, insn, frame)
ipoffset -= frame.EvaluateExpression('imethod->code').GetValueAsUnsigned()
insn = "\"" + frame.EvaluateExpression('mono_interp_opname [*ip]').summary[1:-1] + "\""
var = '%s::%s @ %d %s || %s' % (klassname, methodname, ipoffset, insn, frame)
except Exception as e:
traceback.print_exc()
print "DBG: execfail:" + str(e)
elif function_name == "mono_interp_transform_method":
try:
s = 'runtime_method->method'
@@ -32,7 +32,7 @@ def print_frames(thread, num_frames, current_thread):
methodname = frame.EvaluateExpression('(char*) ' + s + '->name').summary[1:-1]
var = 'transforming %s::%s || %s' % (klassname, methodname, frame)
except Exception as e:
print("DBG: transformfail:" + str(e))
print "DBG: transformfail:" + str(e)
elif pc[0] == '0':
try:
framestr = frame.EvaluateExpression('(char*)mono_pmip((void*)%s)' % pc).summary[1:-1]
@@ -40,7 +40,7 @@ def print_frames(thread, num_frames, current_thread):
except:
pass
print(' %c %s' % ('*' if current_thread and frame.idx == selected_frame.idx else ' ', var))
print ' %c %s' % ('*' if current_thread and frame.idx == selected_frame.idx else ' ', var)
def monobt(debugger, command, result, dict):
opts = {'all_bt': False, 'num_frames': None}
@@ -50,20 +50,20 @@ def monobt(debugger, command, result, dict):
elif command.isdigit():
opts['num_frames'] = int(command)
elif command != '':
print('error: monobt [<number>|all]')
print 'error: monobt [<number>|all]'
return
target = debugger.GetSelectedTarget()
process = target.process
if not process.IsValid():
print('error: invalid process')
print 'error: invalid process'
return
if opts['all_bt']:
for thread in process.threads:
print_frames(thread, len(thread), process.selected_thread == thread)
print('')
print ''
else:
thread = process.selected_thread
num_frames = len(thread) if opts['num_frames'] is None else opts['num_frames']
@@ -76,4 +76,4 @@ def __lldb_init_module (debugger, dict):
# Add any commands contained in this module to LLDB
debugger.HandleCommand('command script add -f monobt.monobt monobt')
debugger.HandleCommand('command alias mbt monobt')
print('"monobt" command installed')
print '"monobt" command installed'
@@ -242,7 +242,7 @@ namespace HtmlAgilityPack
}
/// <summary>
/// Checks for existence of attribute with given name
/// Checks for existance of attribute with given name
/// </summary>
/// <param name="name"></param>
/// <returns></returns>
+1 -1
View File
@@ -22,7 +22,7 @@ Layout
as well as the upcoming Monodoc tools and index creation
tools).
The files can contain any amount of extra information.
The files can contain any ammount of extra information.
The documentation from the source code is then merged with
these files and the header/footer information to produce
+39 -61
View File
@@ -2,58 +2,36 @@
These are some useful links and notes pertaining to the RISC-V port of Mono.
## Useful RISC-V documentation
## Useful RISC-V documents
* [RISC-V User-Level ISA Specification](https://riscv.org/specifications)
* [RISC-V Privileged ISA Specification](https://riscv.org/specifications/privileged-isa)
* [RISC-V Debug Specification](https://riscv.org/specifications/debug-specification)
* [RISC-V ELF psABI Specification](https://github.com/riscv/riscv-elf-psabi-doc)
* [RISC-V C API Specification](https://github.com/riscv/riscv-c-api-doc)
* [RISC-V Toolchain Conventions](https://github.com/riscv/riscv-toolchain-conventions)
* [RISC-V Assembly Programmer's Manual](https://github.com/riscv/riscv-asm-manual)
* [RISC-V User-Level ISA Specification](https://riscv.org/specifications/)
* [RISC-V Privileged ISA Specification](https://riscv.org/specifications/privileged-isa/)
* [RISC-V Debug Specification Standard](https://github.com/riscv/riscv-debug-spec/blob/master/riscv-debug-spec.pdf)
* [RISC-V Assembly Programmer's Manual](https://github.com/riscv/riscv-asm-manual/blob/master/riscv-asm.md)
* [RISC-V ELF psABI Specification](https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md)
## Useful RISC-V repositories
* [RISC-V Organization](https://github.com/riscv)
* [RISC-V GNU Toolchain](https://github.com/riscv/riscv-gnu-toolchain)
* [RISC-V Binutils](https://github.com/riscv/riscv-binutils-gdb)
* [RISC-V GCC](https://github.com/riscv/riscv-gcc)
* [RISC-V Glibc](https://github.com/riscv/riscv-glibc)
* [RISC-V Newlib](https://github.com/riscv/riscv-newlib)
* [RISC-V Linux](https://github.com/riscv/riscv-linux)
* [RISC-V LLD](https://github.com/riscv/riscv-lld)
* [RISC-V Tools](https://github.com/riscv/riscv-tools)
* [RISC-V GNU Toolchain](https://github.com/riscv/riscv-gnu-toolchain)
* [RISC-V Binutils](https://github.com/riscv/riscv-binutils-gdb)
* [RISC-V GCC](https://github.com/riscv/riscv-gcc)
* [RISC-V Glibc](https://github.com/riscv/riscv-glibc)
* [RISC-V Newlib](https://github.com/riscv/riscv-newlib)
* [RISC-V QEMU](https://github.com/riscv/riscv-qemu)
* [RISC-V Opcodes](https://github.com/riscv/riscv-opcodes)
* [RISC-V Tests](https://github.com/riscv/riscv-tests)
* [lowRISC Organization](https://github.com/lowrisc)
* [RISC-V LLVM](https://github.com/lowrisc/riscv-llvm)
## Setting up a cross environment
### Debian/Ubuntu packages
Setting up a cross environment with a Linux toolchain and QEMU is quite easy.
This is the most painless way of getting a functional toolchain installed. Note
that these packages may not be available on older distributions; this is tested
on Ubuntu 19.04.
First, add this to your `$HOME/.bashrc` (or some other script that you run):
```bash
export RISCV=$HOME/riscv
export PATH=$RISCV/bin:$PATH
export QEMU_LD_PREFIX=/usr/riscv64-linux-gnu
```
Next, install the toolchain packages like so:
```bash
# apt install autoconf automake binutils-riscv64-linux-gnu build-essential gcc-riscv64-linux-gnu gdb-multiarch g++-riscv64-linux-gnu libtool qemu qemu-system-misc qemu-user qemu-user-static
```
You will now have all the toolchain binaries in `/usr/bin`.
### Building manually
This approach may be somewhat unstable since you will be using the latest
versions of the various parts of the toolchain.
First, add this to your `$HOME/.bashrc` (or some other script that you run):
First, add these to your `.bashrc` (or some other script that you run):
```bash
export RISCV=$HOME/riscv
@@ -61,42 +39,47 @@ export PATH=$RISCV/bin:$PATH
export QEMU_LD_PREFIX=$RISCV/sysroot
```
Next, install some dependencies needed to build the toolchain:
Install some dependencies needed to build the toolchain:
```console
# apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev
# apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev libusb-1.0-0-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev device-tree-compiler pkg-config libexpat-dev libglib2.0-dev libpixman-1-dev
```
Finally, build the toolchain:
Now you can build the toolchain:
```console
$ git clone --recursive [email protected]:riscv/riscv-gnu-toolchain.git
$ git clone --recursive [email protected]:riscv/riscv-tools.git
$ cd riscv-tools
$ ./build.sh
$ cd riscv-gnu-toolchain
$ ./configure --prefix=$RISCV --enable-linux --enable-multilib --with-arch=rv64imafdc --with-abi=lp64d
$ ./configure --prefix=$RISCV --enable-multilib
$ make linux
$ cd ../..
$ git clone --recursive [email protected]:riscv/riscv-qemu.git
$ cd riscv-qemu
$ mkdir build
$ cd build
$ ../configure --prefix=$RISCV --disable-werror
$ make
$ make install
```
This will install the built toolchain binaries in `$RISCV/bin`.
## Building Mono with a cross toolchain
Building Mono is quite straightforward:
```console
$ ./autogen.sh --prefix=$RISCV/sysroot --host=riscv64-linux-gnu
$ ./autogen.sh --prefix=$RISCV/sysroot --host=riscv64-unknown-linux-gnu
$ make
$ make install
```
(Note: You may need to use `--host=riscv64-unknown-linux-gnu` instead if you
built the toolchain manually.)
You can set `CFLAGS` as appropriate to change the RISC-V options, such as which
standard extensions and ABI to use. For example, to use the 64-bit soft float
ABI:
```console
$ CFLAGS="-mabi=lp64" ./autogen.sh --prefix=$RISCV/sysroot --host=riscv64-linux-gnu
$ CFLAGS="-mabi=lp64" ./autogen.sh --prefix=$RISCV/sysroot --host=riscv64-unknown-linux-gnu
```
Note that, since this is a cross build, the `mcs` directory won't be built. You
@@ -106,24 +89,19 @@ Mono and copy them into `$RISCV/sysroot`.
You can run Mono with QEMU like this:
```console
$ qemu-riscv64 ./mono/mini/mono hello.exe
$ qemu-riscv64 $RISCV/sysroot/bin/mono hello.exe
```
## Debugging
Debugging with GDB currently requires a manually built toolchain.
It can be done like so:
```console
$ qemu-riscv64 -g 12345 ./mono/mini/mono --interp basic.exe &
$ riscv64-unknown-elf-gdb -ex 'target remote localhost:12345' -ex 'b main' -ex 'c' ./mono/mini/mono
$ qemu-riscv64 -g 12345 ./mono/mini/mono-sgen --interp basic.exe &
$ riscv64-unknown-elf-gdb -ex 'target remote localhost:12345' -ex 'b main' -ex 'c' ./mono/mini/mono-sgen
```
## Things to be done
Things that need to be done beyond the basic 64-bit port, in no particular
order:
In no particular order:
* Complete the soft float port.
* Complete the 32-bit port.
-6
View File
@@ -109,12 +109,6 @@ typedef struct {
<h3>Write Barriers</h3>
<p>SGen is a concurrent and generational GC, features which require
tracking changes to the state of the heap. This is achieved through
write barriers. Whenever native code is changing the state of the
heap by storing references into another managed object, it needs to
do it using this write barrier API.
<h4><a name="api:mono_gc_wbarrier_arrayref_copy">mono_gc_wbarrier_arrayref_copy</a></h4>
<h4><a name="api:mono_gc_wbarrier_generic_nostore">mono_gc_wbarrier_generic_nostore</a></h4>
<h4><a name="api:mono_gc_wbarrier_generic_store">mono_gc_wbarrier_generic_store</a></h4>
+10 -10
View File
@@ -1,27 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
<Dependencies>
<ProductDependencies>
<Dependency Name="Microsoft.Private.CoreFx.NETCoreApp" Version="5.0.0-alpha.1.19563.3">
<Dependency Name="Microsoft.Private.CoreFx.NETCoreApp" Version="5.0.0-alpha1.19375.11">
<Uri>https://github.com/dotnet/corefx</Uri>
<Sha>5cee7c97d602f294e27c582d4dab81ec388f1d7b</Sha>
<Sha>97d5957fc71853961082e22873d926269b291b92</Sha>
</Dependency>
<Dependency Name="Microsoft.NETCore.App" Version="5.0.0-alpha.1.19564.1">
<Dependency Name="Microsoft.NETCore.App" Version="5.0.0-alpha1.19375.9">
<Uri>https://github.com/dotnet/core-setup</Uri>
<Sha>c77948d92a2f950140f09384f057cb893ec3955a</Sha>
<Sha>2977763fd35b8dc120f4eac94314aba7b05ae9ee</Sha>
</Dependency>
<Dependency Name="Microsoft.Net.Compilers" Version="3.3.1-beta4-19476-06">
<Dependency Name="Microsoft.Net.Compilers" Version="3.3.0-beta2-19381-14">
<Uri>https://github.com/dotnet/roslyn</Uri>
<Sha>39d545dc14bc4d5a546e5e2f58d1ebaca72fcd61</Sha>
<Sha>ef3a7a3863ae53b610a4b0c3682a35cad0829583</Sha>
</Dependency>
</ProductDependencies>
<ToolsetDependencies>
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="5.0.0-beta.20201.2">
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="1.0.0-beta.19379.1">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>bce0a98620c1c5a110b2bba9912f3d5929069c6b</Sha>
<Sha>193793488c727c2d12a04b4abe5e6ea63ad20526</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="5.0.0-beta.20201.2">
<Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="2.0.0-beta.19379.1">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>bce0a98620c1c5a110b2bba9912f3d5929069c6b</Sha>
<Sha>193793488c727c2d12a04b4abe5e6ea63ad20526</Sha>
</Dependency>
</ToolsetDependencies>
</Dependencies>
+4 -4
View File
@@ -1,14 +1,14 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<VersionPrefix>5.0.0</VersionPrefix>
<VersionPrefix>1.0.0</VersionPrefix>
<PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
<!-- Package versions -->
<!-- corefx -->
<MicrosoftPrivateCoreFxNETCoreAppVersion>5.0.0-alpha.1.19563.3</MicrosoftPrivateCoreFxNETCoreAppVersion>
<MicrosoftPrivateCoreFxNETCoreAppVersion>5.0.0-alpha1.19375.11</MicrosoftPrivateCoreFxNETCoreAppVersion>
<!-- core-setup -->
<MicrosoftNETCoreAppVersion>5.0.0-alpha.1.19564.1</MicrosoftNETCoreAppVersion>
<MicrosoftNETCoreAppVersion>5.0.0-alpha1.19375.9</MicrosoftNETCoreAppVersion>
<!-- roslyn -->
<MicrosoftNetCompilersVersion>3.3.1-beta4-19476-06</MicrosoftNetCompilersVersion>
<MicrosoftNetCompilersVersion>3.3.0-beta2-19374-02</MicrosoftNetCompilersVersion>
</PropertyGroup>
</Project>
+158
View File
@@ -0,0 +1,158 @@
param(
[Parameter(Mandatory=$true)][string] $InputPath, # Full path to directory where NuGet packages to be checked are stored
[Parameter(Mandatory=$true)][string] $ExtractPath, # Full path to directory where the packages will be extracted during validation
[Parameter(Mandatory=$true)][string] $SymbolToolPath # Full path to directory where dotnet symbol-tool was installed
)
Add-Type -AssemblyName System.IO.Compression.FileSystem
function FirstMatchingSymbolDescriptionOrDefault {
param(
[string] $FullPath, # Full path to the module that has to be checked
[string] $TargetServerParam, # Parameter to pass to `Symbol Tool` indicating the server to lookup for symbols
[string] $SymbolsPath
)
$FileName = [System.IO.Path]::GetFileName($FullPath)
$Extension = [System.IO.Path]::GetExtension($FullPath)
# Those below are potential symbol files that the `dotnet symbol` might
# return. Which one will be returned depend on the type of file we are
# checking and which type of file was uploaded.
# The file itself is returned
$SymbolPath = $SymbolsPath + "\" + $FileName
# PDB file for the module
$PdbPath = $SymbolPath.Replace($Extension, ".pdb")
# PDB file for R2R module (created by crossgen)
$NGenPdb = $SymbolPath.Replace($Extension, ".ni.pdb")
# DBG file for a .so library
$SODbg = $SymbolPath.Replace($Extension, ".so.dbg")
# DWARF file for a .dylib
$DylibDwarf = $SymbolPath.Replace($Extension, ".dylib.dwarf")
.\dotnet-symbol.exe --symbols --modules --windows-pdbs $TargetServerParam $FullPath -o $SymbolsPath | Out-Null
if (Test-Path $PdbPath) {
return "PDB"
}
elseif (Test-Path $NGenPdb) {
return "NGen PDB"
}
elseif (Test-Path $SODbg) {
return "DBG for SO"
}
elseif (Test-Path $DylibDwarf) {
return "Dwarf for Dylib"
}
elseif (Test-Path $SymbolPath) {
return "Module"
}
else {
return $null
}
}
function CountMissingSymbols {
param(
[string] $PackagePath # Path to a NuGet package
)
# Ensure input file exist
if (!(Test-Path $PackagePath)) {
throw "Input file does not exist: $PackagePath"
}
# Extensions for which we'll look for symbols
$RelevantExtensions = @(".dll", ".exe", ".so", ".dylib")
# How many files are missing symbol information
$MissingSymbols = 0
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$PackageGuid = New-Guid
$ExtractPath = Join-Path -Path $ExtractPath -ChildPath $PackageGuid
$SymbolsPath = Join-Path -Path $ExtractPath -ChildPath "Symbols"
[System.IO.Compression.ZipFile]::ExtractToDirectory($PackagePath, $ExtractPath)
# Makes easier to reference `symbol tool`
Push-Location $SymbolToolPath
Get-ChildItem -Recurse $ExtractPath |
Where-Object {$RelevantExtensions -contains $_.Extension} |
ForEach-Object {
if ($_.FullName -Match "\\ref\\") {
Write-Host "`t Ignoring reference assembly file" $_.FullName
return
}
$SymbolsOnMSDL = FirstMatchingSymbolDescriptionOrDefault $_.FullName "--microsoft-symbol-server" $SymbolsPath
$SymbolsOnSymWeb = FirstMatchingSymbolDescriptionOrDefault $_.FullName "--internal-server" $SymbolsPath
Write-Host -NoNewLine "`t Checking file" $_.FullName "... "
if ($SymbolsOnMSDL -ne $null -and $SymbolsOnSymWeb -ne $null) {
Write-Host "Symbols found on MSDL (" $SymbolsOnMSDL ") and SymWeb (" $SymbolsOnSymWeb ")"
}
else {
$MissingSymbols++
if ($SymbolsOnMSDL -eq $null -and $SymbolsOnSymWeb -eq $null) {
Write-Host "No symbols found on MSDL or SymWeb!"
}
else {
if ($SymbolsOnMSDL -eq $null) {
Write-Host "No symbols found on MSDL!"
}
else {
Write-Host "No symbols found on SymWeb!"
}
}
}
}
Pop-Location
return $MissingSymbols
}
function CheckSymbolsAvailable {
if (Test-Path $ExtractPath) {
Remove-Item $ExtractPath -Force -Recurse -ErrorAction SilentlyContinue
}
Get-ChildItem "$InputPath\*.nupkg" |
ForEach-Object {
$FileName = $_.Name
# These packages from Arcade-Services include some native libraries that
# our current symbol uploader can't handle. Below is a workaround until
# we get issue: https://github.com/dotnet/arcade/issues/2457 sorted.
if ($FileName -Match "Microsoft\.DotNet\.Darc\.") {
Write-Host "Ignoring Arcade-services file: $FileName"
Write-Host
return
}
elseif ($FileName -Match "Microsoft\.DotNet\.Maestro\.Tasks\.") {
Write-Host "Ignoring Arcade-services file: $FileName"
Write-Host
return
}
Write-Host "Validating $FileName "
$Status = CountMissingSymbols "$InputPath\$FileName"
if ($Status -ne 0) {
Write-Error "Missing symbols for $Status modules in the package $FileName"
}
Write-Host
}
}
CheckSymbolsAvailable
+83
View File
@@ -0,0 +1,83 @@
<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
<Project Sdk="Microsoft.NET.Sdk">
<!--
This MSBuild file is intended to be used as the body of the default
publishing release pipeline. The release pipeline will use this file
to invoke the PushToStaticFeed task that will read the build asset
manifest and publish the assets described in the manifest to
informed target feeds.
-->
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
</PropertyGroup>
<Import Project="$(MSBuildThisFileDirectory)DefaultVersions.props" Condition="Exists('$(MSBuildThisFileDirectory)DefaultVersions.props')" />
<!--
This won't be necessary once we solve this issue:
https://github.com/dotnet/arcade/issues/2266
-->
<Import Project="$(MSBuildThisFileDirectory)ArtifactsCategory.props" Condition="Exists('$(MSBuildThisFileDirectory)ArtifactsCategory.props')" />
<Import Project="$(NuGetPackageRoot)microsoft.dotnet.build.tasks.feed\$(MicrosoftDotNetBuildTasksFeedVersion)\build\Microsoft.DotNet.Build.Tasks.Feed.targets" />
<Target Name="PublishToFeed">
<Error Condition="'$(ArtifactsCategory)' == ''" Text="ArtifactsCategory: The artifacts' category produced by the build wasn't provided." />
<Error Condition="'$(AccountKeyToStaticFeed)' == ''" Text="AccountKeyToStaticFeed: Account key for target feed wasn't provided." />
<Error Condition="'$(ManifestsBasePath)' == ''" Text="Full path to asset manifests directory wasn't provided." />
<Error Condition="'$(BlobBasePath)' == '' AND '$(PackageBasePath)' == ''" Text="A valid full path to BlobBasePath of PackageBasePath is required." />
<ItemGroup>
<!-- Include all manifests found in the manifest folder. -->
<ManifestFiles Include="$(ManifestsBasePath)*.xml" />
</ItemGroup>
<Error Condition="'@(ManifestFiles)' == ''" Text="No manifest file was found in the provided path: $(ManifestsBasePath)" />
<!--
For now the type of packages being published will be informed for the whole build.
Eventually this will be specified on a per package basis:
TODO: https://github.com/dotnet/arcade/issues/2266
-->
<PropertyGroup>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == '.NETCORE'">https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == '.NETCOREVALIDATION'">https://dotnetfeed.blob.core.windows.net/arcade-validation/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETCORE'">https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETCORETOOLING'">https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ENTITYFRAMEWORKCORE'">https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETEXTENSIONS'">https://dotnetfeed.blob.core.windows.net/aspnet-extensions/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'CORECLR'">https://dotnetfeed.blob.core.windows.net/dotnet-coreclr/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'CORESDK'">https://dotnetfeed.blob.core.windows.net/dotnet-sdk/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'TOOLSINTERNAL'">https://dotnetfeed.blob.core.windows.net/dotnet-tools-internal/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'TOOLSET'">https://dotnetfeed.blob.core.windows.net/dotnet-toolset/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'WINDOWSDESKTOP'">https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'NUGETCLIENT'">https://dotnetfeed.blob.core.windows.net/nuget-nugetclient/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETENTITYFRAMEWORK6'">https://dotnetfeed.blob.core.windows.net/aspnet-entityframework6/index.json</TargetStaticFeed>
<TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETBLAZOR'">https://dotnetfeed.blob.core.windows.net/aspnet-blazor/index.json</TargetStaticFeed>
</PropertyGroup>
<Error
Condition="'$(TargetStaticFeed)' == ''"
Text="'$(ArtifactsCategory)' wasn't recognized as a valid artifact category. Valid categories are: '.NetCore' and '.NetCoreValidation'" />
<!-- Iterate publishing assets from each manifest file. -->
<PushArtifactsInManifestToFeed
ExpectedFeedUrl="$(TargetStaticFeed)"
AccountKey="$(AccountKeyToStaticFeed)"
BARBuildId="$(BARBuildId)"
MaestroApiEndpoint="$(MaestroApiEndpoint)"
BuildAssetRegistryToken="$(BuildAssetRegistryToken)"
Overwrite="$(OverrideAssetsWithSameName)"
PassIfExistingItemIdentical="$(PassIfExistingItemIdentical)"
MaxClients="$(MaxParallelUploads)"
UploadTimeoutInMinutes="$(MaxUploadTimeoutInMinutes)"
AssetManifestPath="%(ManifestFiles.Identity)"
BlobAssetsBasePath="$(BlobBasePath)"
PackageAssetsBasePath="$(PackageBasePath)"/>
</Target>
<ItemGroup>
<PackageReference Include="Microsoft.DotNet.Build.Tasks.Feed" Version="$(MicrosoftDotNetBuildTasksFeedVersion)" />
</ItemGroup>
</Project>
+82
View File
@@ -0,0 +1,82 @@
<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
<Project Sdk="Microsoft.NET.Sdk">
<!--
This MSBuild file is intended to be used as the body of the default
publishing release pipeline. The release pipeline will use this file
to invoke the PublishSymbols tasks to publish symbols to MSDL and SymWeb.
Parameters:
- PDBArtifactsDirectory : Full path to directory containing PDB files to be published.
- BlobBasePath : Full path containing *.symbols.nupkg packages to be published.
- DotNetSymbolServerTokenMsdl : PAT to access MSDL.
- DotNetSymbolServerTokenSymWeb : PAT to access SymWeb.
- DotNetSymbolExpirationInDays : Expiration days for published packages. Default is 3650.
-->
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
</PropertyGroup>
<Import Project="$(NuGetPackageRoot)microsoft.symboluploader.build.task\$(SymbolUploaderVersion)\build\PublishSymbols.targets" />
<Target Name="PublishSymbols">
<ItemGroup>
<FilesToPublishToSymbolServer Include="$(PDBArtifactsDirectory)\*.pdb"/>
<PackagesToPublishToSymbolServer Include="$(BlobBasePath)\*.symbols.nupkg"/>
<!--
These packages from Arcade-Services include some native libraries that
our current symbol uploader can't handle. Below is a workaround until
we get issue: https://github.com/dotnet/arcade/issues/2457 sorted.
-->
<PackagesToPublishToSymbolServer Remove="$(BlobBasePath)\Microsoft.DotNet.Darc.*" />
<PackagesToPublishToSymbolServer Remove="$(BlobBasePath)\Microsoft.DotNet.Maestro.Tasks.*" />
</ItemGroup>
<PropertyGroup>
<DotNetSymbolExpirationInDays Condition="'$(DotNetSymbolExpirationInDays)' == ''">3650</DotNetSymbolExpirationInDays>
<PublishToSymbolServer>true</PublishToSymbolServer>
<PublishToSymbolServer Condition="'@(FilesToPublishToSymbolServer)' == '' and '@(PackagesToPublishToSymbolServer)' == ''">false</PublishToSymbolServer>
</PropertyGroup>
<Message
Importance="High"
Text="No symbol package(s) were found to publish."
Condition="$(PublishToSymbolServer) == false" />
<!-- Symbol Uploader: MSDL -->
<Message Importance="High" Text="Publishing symbol packages to MSDL ..." Condition="$(PublishToSymbolServer)" />
<PublishSymbols PackagesToPublish="@(PackagesToPublishToSymbolServer)"
FilesToPublish="@(FilesToPublishToSymbolServer)"
PersonalAccessToken="$(DotNetSymbolServerTokenMsdl)"
SymbolServerPath="https://microsoftpublicsymbols.artifacts.visualstudio.com/DefaultCollection"
ExpirationInDays="$(DotNetSymbolExpirationInDays)"
VerboseLogging="true"
DryRun="false"
ConvertPortablePdbsToWindowsPdbs="false"
PdbConversionTreatAsWarning=""
Condition="$(PublishToSymbolServer)"/>
<!--
Symbol Uploader: SymWeb
Watson, VS insertion testings and the typical internal dev usage require SymWeb.
Currently we need to call the task twice (https://github.com/dotnet/core-eng/issues/3489).
-->
<Message Importance="High" Text="Publishing symbol packages to SymWeb ..." Condition="$(PublishToSymbolServer)" />
<PublishSymbols PackagesToPublish="@(PackagesToPublishToSymbolServer)"
FilesToPublish="@(FilesToPublishToSymbolServer)"
PersonalAccessToken="$(DotNetSymbolServerTokenSymWeb)"
SymbolServerPath="https://microsoft.artifacts.visualstudio.com/DefaultCollection"
ExpirationInDays="$(DotNetSymbolExpirationInDays)"
VerboseLogging="true"
DryRun="false"
ConvertPortablePdbsToWindowsPdbs="false"
PdbConversionTreatAsWarning=""
Condition="$(PublishToSymbolServer)"/>
</Target>
<ItemGroup>
<PackageReference Include="Microsoft.SymbolUploader.Build.Task" Version="$(SymbolUploaderVersion)" />
</ItemGroup>
</Project>
-141
View File
@@ -1,141 +0,0 @@
# This file is a temporary workaround for internal builds to be able to restore from private AzDO feeds.
# This file should be removed as part of this issue: https://github.com/dotnet/arcade/issues/4080
#
# What the script does is iterate over all package sources in the pointed NuGet.config and add a credential entry
# under <packageSourceCredentials> for each Maestro managed private feed. Two additional credential
# entries are also added for the two private static internal feeds: dotnet3-internal and dotnet3-internal-transport.
#
# This script needs to be called in every job that will restore packages and which the base repo has
# private AzDO feeds in the NuGet.config.
#
# See example YAML call for this script below. Note the use of the variable `$(dn-bot-dnceng-artifact-feeds-rw)`
# from the AzureDevOps-Artifact-Feeds-Pats variable group.
#
# - task: PowerShell@2
# displayName: Setup Private Feeds Credentials
# condition: eq(variables['Agent.OS'], 'Windows_NT')
# inputs:
# filePath: $(Build.SourcesDirectory)/eng/common/SetupNugetSources.ps1
# arguments: -ConfigFile $(Build.SourcesDirectory)/NuGet.config -Password $Env:Token
# env:
# Token: $(dn-bot-dnceng-artifact-feeds-rw)
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)][string]$ConfigFile,
[Parameter(Mandatory = $true)][string]$Password
)
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
. $PSScriptRoot\tools.ps1
# Add source entry to PackageSources
function AddPackageSource($sources, $SourceName, $SourceEndPoint, $creds, $Username, $Password) {
$packageSource = $sources.SelectSingleNode("add[@key='$SourceName']")
if ($packageSource -eq $null)
{
$packageSource = $doc.CreateElement("add")
$packageSource.SetAttribute("key", $SourceName)
$packageSource.SetAttribute("value", $SourceEndPoint)
$sources.AppendChild($packageSource) | Out-Null
}
else {
Write-Host "Package source $SourceName already present."
}
AddCredential -Creds $creds -Source $SourceName -Username $Username -Password $Password
}
# Add a credential node for the specified source
function AddCredential($creds, $source, $username, $password) {
# Looks for credential configuration for the given SourceName. Create it if none is found.
$sourceElement = $creds.SelectSingleNode($Source)
if ($sourceElement -eq $null)
{
$sourceElement = $doc.CreateElement($Source)
$creds.AppendChild($sourceElement) | Out-Null
}
# Add the <Username> node to the credential if none is found.
$usernameElement = $sourceElement.SelectSingleNode("add[@key='Username']")
if ($usernameElement -eq $null)
{
$usernameElement = $doc.CreateElement("add")
$usernameElement.SetAttribute("key", "Username")
$sourceElement.AppendChild($usernameElement) | Out-Null
}
$usernameElement.SetAttribute("value", $Username)
# Add the <ClearTextPassword> to the credential if none is found.
# Add it as a clear text because there is no support for encrypted ones in non-windows .Net SDKs.
# -> https://github.com/NuGet/Home/issues/5526
$passwordElement = $sourceElement.SelectSingleNode("add[@key='ClearTextPassword']")
if ($passwordElement -eq $null)
{
$passwordElement = $doc.CreateElement("add")
$passwordElement.SetAttribute("key", "ClearTextPassword")
$sourceElement.AppendChild($passwordElement) | Out-Null
}
$passwordElement.SetAttribute("value", $Password)
}
function InsertMaestroPrivateFeedCredentials($Sources, $Creds, $Password) {
$maestroPrivateSources = $Sources.SelectNodes("add[contains(@key,'darc-int')]")
Write-Host "Inserting credentials for $($maestroPrivateSources.Count) Maestro's private feeds."
ForEach ($PackageSource in $maestroPrivateSources) {
Write-Host "`tInserting credential for Maestro's feed:" $PackageSource.Key
AddCredential -Creds $creds -Source $PackageSource.Key -Username $Username -Password $Password
}
}
if (!(Test-Path $ConfigFile -PathType Leaf)) {
Write-PipelineTelemetryError -Category 'Build' -Message "Eng/common/SetupNugetSources.ps1 returned a non-zero exit code. Couldn't find the NuGet config file: $ConfigFile"
ExitWithExitCode 1
}
if (!$Password) {
Write-PipelineTelemetryError -Category 'Build' -Message 'Eng/common/SetupNugetSources.ps1 returned a non-zero exit code. Please supply a valid PAT'
ExitWithExitCode 1
}
# Load NuGet.config
$doc = New-Object System.Xml.XmlDocument
$filename = (Get-Item $ConfigFile).FullName
$doc.Load($filename)
# Get reference to <PackageSources> or create one if none exist already
$sources = $doc.DocumentElement.SelectSingleNode("packageSources")
if ($sources -eq $null) {
$sources = $doc.CreateElement("packageSources")
$doc.DocumentElement.AppendChild($sources) | Out-Null
}
# Looks for a <PackageSourceCredentials> node. Create it if none is found.
$creds = $doc.DocumentElement.SelectSingleNode("packageSourceCredentials")
if ($creds -eq $null) {
$creds = $doc.CreateElement("packageSourceCredentials")
$doc.DocumentElement.AppendChild($creds) | Out-Null
}
# Insert credential nodes for Maestro's private feeds
InsertMaestroPrivateFeedCredentials -Sources $sources -Creds $creds -Password $Password
$dotnet3Source = $sources.SelectSingleNode("add[@key='dotnet3']")
if ($dotnet3Source -ne $null) {
AddPackageSource -Sources $sources -SourceName "dotnet3-internal" -SourceEndPoint "https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3-internal/nuget/v2" -Creds $creds -Username "dn-bot" -Password $Password
AddPackageSource -Sources $sources -SourceName "dotnet3-internal-transport" -SourceEndPoint "https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3-internal-transport/nuget/v2" -Creds $creds -Username "dn-bot" -Password $Password
}
$dotnet31Source = $sources.SelectSingleNode("add[@key='dotnet3.1']")
if ($dotnet31Source -ne $null) {
AddPackageSource -Sources $sources -SourceName "dotnet3.1-internal" -SourceEndPoint "https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal/nuget/v2" -Creds $creds -Username "dn-bot" -Password $Password
AddPackageSource -Sources $sources -SourceName "dotnet3.1-internal-transport" -SourceEndPoint "https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal-transport/nuget/v2" -Creds $creds -Username "dn-bot" -Password $Password
}
$doc.Save($filename)
-149
View File
@@ -1,149 +0,0 @@
#!/usr/bin/env bash
# This file is a temporary workaround for internal builds to be able to restore from private AzDO feeds.
# This file should be removed as part of this issue: https://github.com/dotnet/arcade/issues/4080
#
# What the script does is iterate over all package sources in the pointed NuGet.config and add a credential entry
# under <packageSourceCredentials> for each Maestro's managed private feed. Two additional credential
# entries are also added for the two private static internal feeds: dotnet3-internal and dotnet3-internal-transport.
#
# This script needs to be called in every job that will restore packages and which the base repo has
# private AzDO feeds in the NuGet.config.
#
# See example YAML call for this script below. Note the use of the variable `$(dn-bot-dnceng-artifact-feeds-rw)`
# from the AzureDevOps-Artifact-Feeds-Pats variable group.
#
# - task: Bash@3
# displayName: Setup Private Feeds Credentials
# inputs:
# filePath: $(Build.SourcesDirectory)/eng/common/SetupNugetSources.sh
# arguments: $(Build.SourcesDirectory)/NuGet.config $Token
# condition: ne(variables['Agent.OS'], 'Windows_NT')
# env:
# Token: $(dn-bot-dnceng-artifact-feeds-rw)
ConfigFile=$1
CredToken=$2
NL='\n'
TB=' '
source="${BASH_SOURCE[0]}"
# resolve $source until the file is no longer a symlink
while [[ -h "$source" ]]; do
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
source="$(readlink "$source")"
# if $source was a relative symlink, we need to resolve it relative to the path where the
# symlink file was located
[[ $source != /* ]] && source="$scriptroot/$source"
done
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
. "$scriptroot/tools.sh"
if [ ! -f "$ConfigFile" ]; then
Write-PipelineTelemetryError -Category 'Build' "Error: Eng/common/SetupNugetSources.sh returned a non-zero exit code. Couldn't find the NuGet config file: $ConfigFile"
ExitWithExitCode 1
fi
if [ -z "$CredToken" ]; then
Write-PipelineTelemetryError -category 'Build' "Error: Eng/common/SetupNugetSources.sh returned a non-zero exit code. Please supply a valid PAT"
ExitWithExitCode 1
fi
if [[ `uname -s` == "Darwin" ]]; then
NL=$'\\\n'
TB=''
fi
# Ensure there is a <packageSources>...</packageSources> section.
grep -i "<packageSources>" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding <packageSources>...</packageSources> section."
ConfigNodeHeader="<configuration>"
PackageSourcesTemplate="${TB}<packageSources>${NL}${TB}</packageSources>"
sed -i.bak "s|$ConfigNodeHeader|$ConfigNodeHeader${NL}$PackageSourcesTemplate|" NuGet.config
fi
# Ensure there is a <packageSourceCredentials>...</packageSourceCredentials> section.
grep -i "<packageSourceCredentials>" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding <packageSourceCredentials>...</packageSourceCredentials> section."
PackageSourcesNodeFooter="</packageSources>"
PackageSourceCredentialsTemplate="${TB}<packageSourceCredentials>${NL}${TB}</packageSourceCredentials>"
sed -i.bak "s|$PackageSourcesNodeFooter|$PackageSourcesNodeFooter${NL}$PackageSourceCredentialsTemplate|" NuGet.config
fi
PackageSources=()
# Ensure dotnet3-internal and dotnet3-internal-transport are in the packageSources if the public dotnet3 feeds are present
grep -i "<add key=\"dotnet3\"" $ConfigFile
if [ "$?" == "0" ]; then
grep -i "<add key=\"dotnet3-internal\">" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding dotnet3-internal to the packageSources."
PackageSourcesNodeFooter="</packageSources>"
PackageSourceTemplate="${TB}<add key=\"dotnet3-internal\" value=\"https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3-internal/nuget/v2\" />"
sed -i.bak "s|$PackageSourcesNodeFooter|$PackageSourceTemplate${NL}$PackageSourcesNodeFooter|" $ConfigFile
fi
PackageSources+=('dotnet3-internal')
grep -i "<add key=\"dotnet3-internal-transport\"" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding dotnet3-internal-transport to the packageSources."
PackageSourcesNodeFooter="</packageSources>"
PackageSourceTemplate="${TB}<add key=\"dotnet3-internal-transport\" value=\"https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3-internal-transport/nuget/v2\" />"
sed -i.bak "s|$PackageSourcesNodeFooter|$PackageSourceTemplate${NL}$PackageSourcesNodeFooter|" $ConfigFile
fi
PackageSources+=('dotnet3-internal-transport')
fi
# Ensure dotnet3.1-internal and dotnet3.1-internal-transport are in the packageSources if the public dotnet3.1 feeds are present
grep -i "<add key=\"dotnet3.1\"" $ConfigFile
if [ "$?" == "0" ]; then
grep -i "<add key=\"dotnet3.1-internal\"" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding dotnet3.1-internal to the packageSources."
PackageSourcesNodeFooter="</packageSources>"
PackageSourceTemplate="${TB}<add key=\"dotnet3.1-internal\" value=\"https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal/nuget/v2\" />"
sed -i.bak "s|$PackageSourcesNodeFooter|$PackageSourceTemplate${NL}$PackageSourcesNodeFooter|" $ConfigFile
fi
PackageSources+=('dotnet3.1-internal')
grep -i "<add key=\"dotnet3.1-internal-transport\">" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding dotnet3.1-internal-transport to the packageSources."
PackageSourcesNodeFooter="</packageSources>"
PackageSourceTemplate="${TB}<add key=\"dotnet3.1-internal-transport\" value=\"https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal-transport/nuget/v2\" />"
sed -i.bak "s|$PackageSourcesNodeFooter|$PackageSourceTemplate${NL}$PackageSourcesNodeFooter|" $ConfigFile
fi
PackageSources+=('dotnet3.1-internal-transport')
fi
# I want things split line by line
PrevIFS=$IFS
IFS=$'\n'
PackageSources+="$IFS"
PackageSources+=$(grep -oh '"darc-int-[^"]*"' $ConfigFile | tr -d '"')
IFS=$PrevIFS
for FeedName in ${PackageSources[@]} ; do
# Check if there is no existing credential for this FeedName
grep -i "<$FeedName>" $ConfigFile
if [ "$?" != "0" ]; then
echo "Adding credentials for $FeedName."
PackageSourceCredentialsNodeFooter="</packageSourceCredentials>"
NewCredential="${TB}${TB}<$FeedName>${NL}<add key=\"Username\" value=\"dn-bot\" />${NL}<add key=\"ClearTextPassword\" value=\"$CredToken\" />${NL}</$FeedName>"
sed -i.bak "s|$PackageSourceCredentialsNodeFooter|$NewCredential${NL}$PackageSourceCredentialsNodeFooter|" $ConfigFile
fi
done
+83
View File
@@ -0,0 +1,83 @@
<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
<Project Sdk="Microsoft.NET.Sdk">
<!--
This MSBuild file is intended to be used as the body of the default
publishing release pipeline. The release pipeline will use this file
to invoke the SignCheck tool to validate that packages about to
be published are correctly signed.
Parameters:
- PackageBasePath : Directory containing all files that need to be validated.
- SignCheckVersion : Version of SignCheck package to be used.
- SignValidationExclusionList : ItemGroup containing exclusion list to be forwarded to SignCheck.
- EnableJarSigningCheck : Whether .jar files should be validated.
- EnableStrongNameCheck : Whether strong name check should be performed.
-->
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
</PropertyGroup>
<!--
From 'Signing.props' we import $(SignValidationExclusionList)
-->
<Import Project="$(MSBuildThisFileDirectory)Signing.props" Condition="Exists('$(MSBuildThisFileDirectory)Signing.props')" />
<Target Name="ValidateSigning">
<PropertyGroup>
<SignCheckToolPath>$(NuGetPackageRoot)Microsoft.DotNet.SignCheck\$(SignCheckVersion)\tools\Microsoft.DotNet.SignCheck.exe</SignCheckToolPath>
<SignCheckInputDir>$(PackageBasePath)</SignCheckInputDir>
<SignCheckLog>signcheck.log</SignCheckLog>
<SignCheckErrorLog>signcheck.errors.log</SignCheckErrorLog>
<SignCheckExclusionsFile>signcheck.exclusions.txt</SignCheckExclusionsFile>
</PropertyGroup>
<ItemGroup>
<!--
Documentation for these arguments is available here:
https://github.com/dotnet/arcade/tree/master/src/SignCheck
-->
<SignCheckArgs Include="--recursive" />
<SignCheckArgs Include="--traverse-subfolders" />
<SignCheckArgs Include="--file-status AllFiles" />
<SignCheckArgs Include="--log-file $(SignCheckLog)" />
<SignCheckArgs Include="--error-log-file $(SignCheckErrorLog)" />
<SignCheckArgs Include="--input-files $(SignCheckInputDir)" />
<SignCheckArgs Include="--exclusions-file $(SignCheckExclusionsFile)" Condition="'@(SignValidationExclusionList)' != ''" />
<SignCheckArgs Include="--verify-jar" Condition="'$(EnableJarSigningCheck)' == 'true'" />
<SignCheckArgs Include="--verify-strongname" Condition="'$(EnableStrongNameCheck)' == 'true'" />
</ItemGroup>
<WriteLinesToFile
File="$(SignCheckExclusionsFile)"
Lines="@(SignValidationExclusionList)"
Condition="'@(SignValidationExclusionList)' != ''"
Overwrite="true"
Encoding="Unicode"/>
<!--
IgnoreExitCode='true' because the tool doesn't return '0' on success.
-->
<Exec
Command="&quot;$(SignCheckToolPath)&quot; @(SignCheckArgs, ' ')"
IgnoreExitCode='true'
ConsoleToMsBuild="false"
StandardErrorImportance="high" />
<Error
Text="Signing validation failed. Check $(SignCheckErrorLog) for more information."
Condition="Exists($(SignCheckErrorLog)) and '$([System.IO.File]::ReadAllText($(SignCheckErrorLog)))' != ''" />
<Message
Text="##vso[artifact.upload containerfolder=LogFiles;artifactname=LogFiles]{SignCheckErrorLog}"
Condition="Exists($(SignCheckErrorLog)) and '$([System.IO.File]::ReadAllText($(SignCheckErrorLog)))' != ''" />
</Target>
<ItemGroup>
<PackageReference Include="Microsoft.DotNet.SignCheck" Version="$(SignCheckVersion)" />
</ItemGroup>
</Project>
+184
View File
@@ -0,0 +1,184 @@
param(
[Parameter(Mandatory=$true)][string] $InputPath, # Full path to directory where Symbols.NuGet packages to be checked are stored
[Parameter(Mandatory=$true)][string] $ExtractPath, # Full path to directory where the packages will be extracted during validation
[Parameter(Mandatory=$true)][string] $SourceLinkToolPath, # Full path to directory where dotnet SourceLink CLI was installed
[Parameter(Mandatory=$true)][string] $GHRepoName, # GitHub name of the repo including the Org. E.g., dotnet/arcade
[Parameter(Mandatory=$true)][string] $GHCommit # GitHub commit SHA used to build the packages
)
# Cache/HashMap (File -> Exist flag) used to consult whether a file exist
# in the repository at a specific commit point. This is populated by inserting
# all files present in the repo at a specific commit point.
$global:RepoFiles = @{}
$ValidatePackage = {
param(
[string] $PackagePath # Full path to a Symbols.NuGet package
)
# Ensure input file exist
if (!(Test-Path $PackagePath)) {
throw "Input file does not exist: $PackagePath"
}
# Extensions for which we'll look for SourceLink information
# For now we'll only care about Portable & Embedded PDBs
$RelevantExtensions = @(".dll", ".exe", ".pdb")
Write-Host -NoNewLine "Validating" ([System.IO.Path]::GetFileName($PackagePath)) "... "
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$ExtractPath = Join-Path -Path $using:ExtractPath -ChildPath $PackageId
$FailedFiles = 0
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Directory]::CreateDirectory($ExtractPath);
$zip = [System.IO.Compression.ZipFile]::OpenRead($PackagePath)
$zip.Entries |
Where-Object {$RelevantExtensions -contains [System.IO.Path]::GetExtension($_.Name)} |
ForEach-Object {
$FileName = $_.FullName
$Extension = [System.IO.Path]::GetExtension($_.Name)
$FakeName = -Join((New-Guid), $Extension)
$TargetFile = Join-Path -Path $ExtractPath -ChildPath $FakeName
# We ignore resource DLLs
if ($FileName.EndsWith(".resources.dll")) {
return
}
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, $TargetFile, $true)
$ValidateFile = {
param(
[string] $FullPath, # Full path to the module that has to be checked
[string] $RealPath,
[ref] $FailedFiles
)
# Makes easier to reference `sourcelink cli`
Push-Location $using:SourceLinkToolPath
$SourceLinkInfos = .\sourcelink.exe print-urls $FullPath | Out-String
if ($LASTEXITCODE -eq 0 -and -not ([string]::IsNullOrEmpty($SourceLinkInfos))) {
$NumFailedLinks = 0
# We only care about Http addresses
$Matches = (Select-String '(http[s]?)(:\/\/)([^\s,]+)' -Input $SourceLinkInfos -AllMatches).Matches
if ($Matches.Count -ne 0) {
$Matches.Value |
ForEach-Object {
$Link = $_
$CommitUrl = -Join("https://raw.githubusercontent.com/", $using:GHRepoName, "/", $using:GHCommit, "/")
$FilePath = $Link.Replace($CommitUrl, "")
$Status = 200
$Cache = $using:RepoFiles
if ( !($Cache.ContainsKey($FilePath)) ) {
try {
$Uri = $Link -as [System.URI]
# Only GitHub links are valid
if ($Uri.AbsoluteURI -ne $null -and $Uri.Host -match "github") {
$Status = (Invoke-WebRequest -Uri $Link -UseBasicParsing -Method HEAD -TimeoutSec 5).StatusCode
}
else {
$Status = 0
}
}
catch {
$Status = 0
}
}
if ($Status -ne 200) {
if ($NumFailedLinks -eq 0) {
if ($FailedFiles.Value -eq 0) {
Write-Host
}
Write-Host "`tFile $RealPath has broken links:"
}
Write-Host "`t`tFailed to retrieve $Link"
$NumFailedLinks++
}
}
}
if ($NumFailedLinks -ne 0) {
$FailedFiles.value++
$global:LASTEXITCODE = 1
}
}
Pop-Location
}
&$ValidateFile $TargetFile $FileName ([ref]$FailedFiles)
}
$zip.Dispose()
if ($FailedFiles -eq 0) {
Write-Host "Passed."
}
}
function ValidateSourceLinkLinks {
if (!($GHRepoName -Match "^[^\s\/]+/[^\s\/]+$")) {
Write-Host "GHRepoName should be in the format <org>/<repo>"
$global:LASTEXITCODE = 1
return
}
if (!($GHCommit -Match "^[0-9a-fA-F]{40}$")) {
Write-Host "GHCommit should be a 40 chars hexadecimal string"
$global:LASTEXITCODE = 1
return
}
$RepoTreeURL = -Join("https://api.github.com/repos/", $GHRepoName, "/git/trees/", $GHCommit, "?recursive=1")
$CodeExtensions = @(".cs", ".vb", ".fs", ".fsi", ".fsx", ".fsscript")
try {
# Retrieve the list of files in the repo at that particular commit point and store them in the RepoFiles hash
$Data = Invoke-WebRequest $RepoTreeURL | ConvertFrom-Json | Select-Object -ExpandProperty tree
foreach ($file in $Data) {
$Extension = [System.IO.Path]::GetExtension($file.path)
if ($CodeExtensions.Contains($Extension)) {
$RepoFiles[$file.path] = 1
}
}
}
catch {
Write-Host "Problems downloading the list of files from the repo. Url used: $RepoTreeURL"
$global:LASTEXITCODE = 1
return
}
if (Test-Path $ExtractPath) {
Remove-Item $ExtractPath -Force -Recurse -ErrorAction SilentlyContinue
}
# Process each NuGet package in parallel
$Jobs = @()
Get-ChildItem "$InputPath\*.symbols.nupkg" |
ForEach-Object {
$Jobs += Start-Job -ScriptBlock $ValidatePackage -ArgumentList $_.FullName
}
foreach ($Job in $Jobs) {
Wait-Job -Id $Job.Id | Receive-Job
}
}
Measure-Command { ValidateSourceLinkLinks }
+48 -60
View File
@@ -18,7 +18,6 @@ Param(
[switch] $sign,
[switch] $pack,
[switch] $publish,
[switch] $clean,
[switch][Alias('bl')]$binaryLog,
[switch] $ci,
[switch] $prepareMachine,
@@ -26,55 +25,49 @@ Param(
[Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
)
# Unset 'Platform' environment variable to avoid unwanted collision in InstallDotNetCore.targets file
# some computer has this env var defined (e.g. Some HP)
if($env:Platform) {
$env:Platform=""
}
function Print-Usage() {
Write-Host "Common settings:"
Write-Host " -configuration <value> Build configuration: 'Debug' or 'Release' (short: -c)"
Write-Host " -platform <value> Platform configuration: 'x86', 'x64' or any valid Platform value to pass to msbuild"
Write-Host " -verbosity <value> Msbuild verbosity: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic] (short: -v)"
Write-Host " -binaryLog Output binary log (short: -bl)"
Write-Host " -help Print help and exit"
Write-Host ""
Write-Host "Actions:"
Write-Host " -restore Restore dependencies (short: -r)"
Write-Host " -build Build solution (short: -b)"
Write-Host " -rebuild Rebuild solution"
Write-Host " -deploy Deploy built VSIXes"
Write-Host " -deployDeps Deploy dependencies (e.g. VSIXes for integration tests)"
Write-Host " -test Run all unit tests in the solution (short: -t)"
Write-Host " -integrationTest Run all integration tests in the solution"
Write-Host " -performanceTest Run all performance tests in the solution"
Write-Host " -pack Package build outputs into NuGet packages and Willow components"
Write-Host " -sign Sign build outputs"
Write-Host " -publish Publish artifacts (e.g. symbols)"
Write-Host " -clean Clean the solution"
Write-Host ""
Write-Host "Advanced settings:"
Write-Host " -projects <value> Semi-colon delimited list of sln/proj's to build. Globbing is supported (*.sln)"
Write-Host " -ci Set when running on CI server"
Write-Host " -prepareMachine Prepare machine for CI run, clean up processes after build"
Write-Host " -warnAsError <value> Sets warnaserror msbuild parameter ('true' or 'false')"
Write-Host " -msbuildEngine <value> Msbuild engine to use to run build ('dotnet', 'vs', or unspecified)."
Write-Host ""
Write-Host "Command line arguments not listed above are passed thru to msbuild."
Write-Host "The above arguments can be shortened as much as to be unambiguous (e.g. -co for configuration, -t for test, etc.)."
}
. $PSScriptRoot\tools.ps1
function Print-Usage() {
Write-Host "Common settings:"
Write-Host " -configuration <value> Build configuration: 'Debug' or 'Release' (short: -c)"
Write-Host " -platform <value> Platform configuration: 'x86', 'x64' or any valid Platform value to pass to msbuild"
Write-Host " -verbosity <value> Msbuild verbosity: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic] (short: -v)"
Write-Host " -binaryLog Output binary log (short: -bl)"
Write-Host " -help Print help and exit"
Write-Host ""
Write-Host "Actions:"
Write-Host " -restore Restore dependencies (short: -r)"
Write-Host " -build Build solution (short: -b)"
Write-Host " -rebuild Rebuild solution"
Write-Host " -deploy Deploy built VSIXes"
Write-Host " -deployDeps Deploy dependencies (e.g. VSIXes for integration tests)"
Write-Host " -test Run all unit tests in the solution (short: -t)"
Write-Host " -integrationTest Run all integration tests in the solution"
Write-Host " -performanceTest Run all performance tests in the solution"
Write-Host " -pack Package build outputs into NuGet packages and Willow components"
Write-Host " -sign Sign build outputs"
Write-Host " -publish Publish artifacts (e.g. symbols)"
Write-Host ""
Write-Host "Advanced settings:"
Write-Host " -projects <value> Semi-colon delimited list of sln/proj's to build. Globbing is supported (*.sln)"
Write-Host " -ci Set when running on CI server"
Write-Host " -prepareMachine Prepare machine for CI run, clean up processes after build"
Write-Host " -warnAsError <value> Sets warnaserror msbuild parameter ('true' or 'false')"
Write-Host " -msbuildEngine <value> Msbuild engine to use to run build ('dotnet', 'vs', or unspecified)."
Write-Host ""
Write-Host "Command line arguments not listed above are passed thru to msbuild."
Write-Host "The above arguments can be shortened as much as to be unambiguous (e.g. -co for configuration, -t for test, etc.)."
}
function InitializeCustomToolset {
if (-not $restore) {
return
}
$script = Join-Path $EngRoot 'restore-toolset.ps1'
$script = Join-Path $EngRoot "restore-toolset.ps1"
if (Test-Path $script) {
. $script
@@ -85,17 +78,13 @@ function Build {
$toolsetBuildProj = InitializeToolset
InitializeCustomToolset
$bl = if ($binaryLog) { '/bl:' + (Join-Path $LogDir 'Build.binlog') } else { '' }
$platformArg = if ($platform) { "/p:Platform=$platform" } else { '' }
$bl = if ($binaryLog) { "/bl:" + (Join-Path $LogDir "Build.binlog") } else { "" }
$platformArg = if ($platform) { "/p:Platform=$platform" } else { "" }
if ($projects) {
# Re-assign properties to a new variable because PowerShell doesn't let us append properties directly for unclear reasons.
# Explicitly set the type as string[] because otherwise PowerShell would make this char[] if $properties is empty.
[string[]] $msbuildArgs = $properties
# Resolve relative project paths into full paths
$projects = ($projects.Split(';').ForEach({Resolve-Path $_}) -join ';')
$msbuildArgs += "/p:Projects=$projects"
$properties = $msbuildArgs
}
@@ -120,15 +109,7 @@ function Build {
}
try {
if ($clean) {
if (Test-Path $ArtifactsDir) {
Remove-Item -Recurse -Force $ArtifactsDir
Write-Host 'Artifacts directory deleted.'
}
exit 0
}
if ($help -or (($null -ne $properties) -and ($properties.Contains('/help') -or $properties.Contains('/?')))) {
if ($help -or (($null -ne $properties) -and ($properties.Contains("/help") -or $properties.Contains("/?")))) {
Print-Usage
exit 0
}
@@ -138,7 +119,14 @@ try {
$nodeReuse = $false
}
if ($restore) {
# Import custom tools configuration, if present in the repo.
# Note: Import in global scope so that the script set top-level variables without qualification.
$configureToolsetScript = Join-Path $EngRoot "configure-toolset.ps1"
if (Test-Path $configureToolsetScript) {
. $configureToolsetScript
}
if (($restore) -and ($null -eq $env:DisableNativeToolsetInstalls)) {
InitializeNativeTools
}
@@ -146,7 +134,7 @@ try {
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message $_
Write-PipelineTelemetryError -Category "InitializeToolset" -Message $_
ExitWithExitCode 1
}
+12 -12
View File
@@ -26,7 +26,6 @@ usage()
echo " --pack Package build outputs into NuGet packages and Willow components"
echo " --sign Sign build outputs"
echo " --publish Publish artifacts (e.g. symbols)"
echo " --clean Clean the solution"
echo ""
echo "Advanced settings:"
@@ -63,7 +62,6 @@ publish=false
sign=false
public=false
ci=false
clean=false
warn_as_error=true
node_reuse=true
@@ -84,9 +82,6 @@ while [[ $# > 0 ]]; do
usage
exit 0
;;
-clean)
clean=true
;;
-configuration|-c)
configuration=$2
shift
@@ -201,15 +196,20 @@ function Build {
ExitWithExitCode 0
}
if [[ "$clean" == true ]]; then
if [ -d "$artifacts_dir" ]; then
rm -rf $artifacts_dir
echo "Artifacts directory deleted."
fi
exit 0
# Import custom tools configuration, if present in the repo.
configure_toolset_script="$eng_root/configure-toolset.sh"
if [[ -a "$configure_toolset_script" ]]; then
. "$configure_toolset_script"
fi
if [[ "$restore" == true ]]; then
# TODO: https://github.com/dotnet/arcade/issues/1468
# Temporary workaround to avoid breaking change.
# Remove once repos are updated.
if [[ -n "${useInstalledDotNetCli:-}" ]]; then
use_installed_dotnet_cli="$useInstalledDotNetCli"
fi
if [[ "$restore" == true && -z ${DisableNativeToolsetInstalls:-} ]]; then
InitializeNativeTools
fi
@@ -0,0 +1,41 @@
set(CROSS_NDK_TOOLCHAIN $ENV{ROOTFS_DIR}/../)
set(CROSS_ROOTFS ${CROSS_NDK_TOOLCHAIN}/sysroot)
set(CLR_CMAKE_PLATFORM_ANDROID "Android")
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_VERSION 1)
set(CMAKE_SYSTEM_PROCESSOR arm)
## Specify the toolchain
set(TOOLCHAIN "arm-linux-androideabi")
set(CMAKE_PREFIX_PATH ${CROSS_NDK_TOOLCHAIN})
set(TOOLCHAIN_PREFIX ${TOOLCHAIN}-)
find_program(CMAKE_C_COMPILER ${TOOLCHAIN_PREFIX}clang)
find_program(CMAKE_CXX_COMPILER ${TOOLCHAIN_PREFIX}clang++)
find_program(CMAKE_ASM_COMPILER ${TOOLCHAIN_PREFIX}clang)
find_program(CMAKE_AR ${TOOLCHAIN_PREFIX}ar)
find_program(CMAKE_LD ${TOOLCHAIN_PREFIX}ar)
find_program(CMAKE_OBJCOPY ${TOOLCHAIN_PREFIX}objcopy)
find_program(CMAKE_OBJDUMP ${TOOLCHAIN_PREFIX}objdump)
add_compile_options(--sysroot=${CROSS_ROOTFS})
add_compile_options(-fPIE)
add_compile_options(-mfloat-abi=soft)
include_directories(SYSTEM ${CROSS_NDK_TOOLCHAIN}/include/c++/4.9.x/)
include_directories(SYSTEM ${CROSS_NDK_TOOLCHAIN}/include/c++/4.9.x/arm-linux-androideabi/)
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -B ${CROSS_ROOTFS}/usr/lib/gcc/${TOOLCHAIN}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -L${CROSS_ROOTFS}/lib/${TOOLCHAIN}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} --sysroot=${CROSS_ROOTFS}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -fPIE -pie")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_FIND_ROOT_PATH "${CROSS_ROOTFS}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
@@ -0,0 +1,42 @@
set(CROSS_NDK_TOOLCHAIN $ENV{ROOTFS_DIR}/../)
set(CROSS_ROOTFS ${CROSS_NDK_TOOLCHAIN}/sysroot)
set(CLR_CMAKE_PLATFORM_ANDROID "Android")
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_VERSION 1)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
## Specify the toolchain
set(TOOLCHAIN "aarch64-linux-android")
set(CMAKE_PREFIX_PATH ${CROSS_NDK_TOOLCHAIN})
set(TOOLCHAIN_PREFIX ${TOOLCHAIN}-)
find_program(CMAKE_C_COMPILER ${TOOLCHAIN_PREFIX}clang)
find_program(CMAKE_CXX_COMPILER ${TOOLCHAIN_PREFIX}clang++)
find_program(CMAKE_ASM_COMPILER ${TOOLCHAIN_PREFIX}clang)
find_program(CMAKE_AR ${TOOLCHAIN_PREFIX}ar)
find_program(CMAKE_LD ${TOOLCHAIN_PREFIX}ar)
find_program(CMAKE_OBJCOPY ${TOOLCHAIN_PREFIX}objcopy)
find_program(CMAKE_OBJDUMP ${TOOLCHAIN_PREFIX}objdump)
add_compile_options(--sysroot=${CROSS_ROOTFS})
add_compile_options(-fPIE)
## Needed for Android or bionic specific conditionals
add_compile_options(-D__ANDROID__)
add_compile_options(-D__BIONIC__)
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -B ${CROSS_ROOTFS}/usr/lib/gcc/${TOOLCHAIN}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -L${CROSS_ROOTFS}/lib/${TOOLCHAIN}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} --sysroot=${CROSS_ROOTFS}")
set(CROSS_LINK_FLAGS "${CROSS_LINK_FLAGS} -fPIE -pie")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} ${CROSS_LINK_FLAGS}" CACHE STRING "" FORCE)
set(CMAKE_FIND_ROOT_PATH "${CROSS_ROOTFS}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
+51 -45
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
set -e
__NDK_Version=r21
__NDK_Version=r14
usage()
{
@@ -16,11 +16,11 @@ usage()
echo.
echo "By default, the NDK will be downloaded into the cross/android-rootfs/android-ndk-$__NDK_Version directory. If you already have an NDK installation,"
echo "you can set the NDK_DIR environment variable to have this script use that installation of the NDK."
echo "By default, this script will generate a file, android_platform, in the root of the ROOTFS_DIR directory that contains the RID for the supported and tested Android build: android.28-arm64. This file is to replace '/etc/os-release', which is not available for Android."
echo "By default, this script will generate a file, android_platform, in the root of the ROOTFS_DIR directory that contains the RID for the supported and tested Android build: android.21-arm64. This file is to replace '/etc/os-release', which is not available for Android."
exit 1
}
__ApiLevel=28 # The minimum platform for arm64 is API level 21 but the minimum version that support glob(3) is 28. See $ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/glob.h
__ApiLevel=21 # The minimum platform for arm64 is API level 21
__BuildArch=arm64
__AndroidArch=aarch64
__AndroidToolchain=aarch64-linux-android
@@ -53,20 +53,13 @@ for i in "$@"
done
# Obtain the location of the bash script to figure out where the root of the repo is.
__ScriptBaseDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
__CrossDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
__CrossDir="$__ScriptBaseDir/../../../.tools/android-rootfs"
if [[ ! -f "$__CrossDir" ]]; then
mkdir -p "$__CrossDir"
fi
# Resolve absolute path to avoid `../` in build logs
__CrossDir="$( cd "$__CrossDir" && pwd )"
__NDK_Dir="$__CrossDir/android-ndk-$__NDK_Version"
__lldb_Dir="$__CrossDir/lldb"
__ToolchainDir="$__CrossDir/android-ndk-$__NDK_Version"
__Android_Cross_Dir="$__CrossDir/android-rootfs"
__NDK_Dir="$__Android_Cross_Dir/android-ndk-$__NDK_Version"
__libunwind_Dir="$__Android_Cross_Dir/libunwind"
__lldb_Dir="$__Android_Cross_Dir/lldb"
__ToolchainDir="$__Android_Cross_Dir/toolchain/$__BuildArch"
if [[ -n "$TOOLCHAIN_DIR" ]]; then
__ToolchainDir=$TOOLCHAIN_DIR
@@ -85,47 +78,60 @@ echo "Target Toolchain location: $__ToolchainDir"
if [ ! -d $__NDK_Dir ]; then
echo Downloading the NDK into $__NDK_Dir
mkdir -p $__NDK_Dir
wget -q --progress=bar:force:noscroll --show-progress https://dl.google.com/android/repository/android-ndk-$__NDK_Version-linux-x86_64.zip -O $__CrossDir/android-ndk-$__NDK_Version-linux-x86_64.zip
unzip -q $__CrossDir/android-ndk-$__NDK_Version-linux-x86_64.zip -d $__CrossDir
wget -nv -nc --show-progress https://dl.google.com/android/repository/android-ndk-$__NDK_Version-linux-x86_64.zip -O $__Android_Cross_Dir/android-ndk-$__NDK_Version-linux-x86_64.zip
unzip -q $__Android_Cross_Dir/android-ndk-$__NDK_Version-linux-x86_64.zip -d $__Android_Cross_Dir
fi
if [ ! -d $__lldb_Dir ]; then
mkdir -p $__lldb_Dir
echo Downloading LLDB into $__lldb_Dir
wget -q --progress=bar:force:noscroll --show-progress https://dl.google.com/android/repository/lldb-2.3.3614996-linux-x86_64.zip -O $__CrossDir/lldb-2.3.3614996-linux-x86_64.zip
unzip -q $__CrossDir/lldb-2.3.3614996-linux-x86_64.zip -d $__lldb_Dir
wget -nv -nc --show-progress https://dl.google.com/android/repository/lldb-2.3.3614996-linux-x86_64.zip -O $__Android_Cross_Dir/lldb-2.3.3614996-linux-x86_64.zip
unzip -q $__Android_Cross_Dir/lldb-2.3.3614996-linux-x86_64.zip -d $__lldb_Dir
fi
echo "Download dependencies..."
__TmpDir=$__CrossDir/tmp/$__BuildArch/
mkdir -p "$__TmpDir"
# Create the RootFS for both arm64 as well as aarch
rm -rf $__Android_Cross_Dir/toolchain
# combined dependencies for coreclr, installer and libraries
__AndroidPackages="libicu"
__AndroidPackages+=" libandroid-glob"
__AndroidPackages+=" liblzma"
__AndroidPackages+=" krb5"
__AndroidPackages+=" openssl"
echo Generating the $__BuildArch toolchain
$__NDK_Dir/build/tools/make_standalone_toolchain.py --arch $__BuildArch --api $__ApiLevel --install-dir $__ToolchainDir
for path in $(wget -qO- http://termux.net/dists/stable/main/binary-$__AndroidArch/Packages |\
grep -A15 "Package: \(${__AndroidPackages// /\\|}\)" | grep -v "static\|tool" | grep Filename); do
# Install the required packages into the toolchain
# TODO: Add logic to get latest pkg version instead of specific version number
rm -rf $__Android_Cross_Dir/deb/
rm -rf $__Android_Cross_Dir/tmp
if [[ "$path" != "Filename:" ]]; then
echo "Working on: $path"
wget -qO- http://termux.net/$path | dpkg -x - "$__TmpDir"
fi
done
mkdir -p $__Android_Cross_Dir/deb/
mkdir -p $__Android_Cross_Dir/tmp/$arch/
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libicu_60.2_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libicu_60.2_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libicu-dev_60.2_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libicu-dev_60.2_$__AndroidArch.deb
cp -R "$__TmpDir/data/data/com.termux/files/usr/"* "$__ToolchainDir/sysroot/usr/"
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libandroid-glob-dev_0.4_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libandroid-glob-dev_0.4_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libandroid-glob_0.4_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libandroid-glob_0.4_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libandroid-support-dev_22_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libandroid-support-dev_22_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libandroid-support_22_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libandroid-support_22_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/liblzma-dev_5.2.3_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/liblzma-dev_5.2.3_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/liblzma_5.2.3_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/liblzma_5.2.3_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libunwind-dev_1.2.20170304_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libunwind-dev_1.2.20170304_$__AndroidArch.deb
wget -nv -nc http://termux.net/dists/stable/main/binary-$__AndroidArch/libunwind_1.2.20170304_$__AndroidArch.deb -O $__Android_Cross_Dir/deb/libunwind_1.2.20170304_$__AndroidArch.deb
echo Unpacking Termux packages
dpkg -x $__Android_Cross_Dir/deb/libicu_60.2_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libicu-dev_60.2_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libandroid-glob-dev_0.4_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libandroid-glob_0.4_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libandroid-support-dev_22_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libandroid-support_22_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/liblzma-dev_5.2.3_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/liblzma_5.2.3_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libunwind-dev_1.2.20170304_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
dpkg -x $__Android_Cross_Dir/deb/libunwind_1.2.20170304_$__AndroidArch.deb $__Android_Cross_Dir/tmp/$__AndroidArch/
cp -R $__Android_Cross_Dir/tmp/$__AndroidArch/data/data/com.termux/files/usr/* $__ToolchainDir/sysroot/usr/
# Generate platform file for build.sh script to assign to __DistroRid
echo "Generating platform file..."
echo "RID=android.${__ApiLevel}-${__BuildArch}" > $__ToolchainDir/sysroot/android_platform
echo "Now to build coreclr, libraries and installers; run:"
echo ROOTFS_DIR=\$\(realpath $__ToolchainDir/sysroot\) ./build.sh --cross --arch $__BuildArch \
--subsetCategory coreclr
echo ROOTFS_DIR=\$\(realpath $__ToolchainDir/sysroot\) ./build.sh --cross --arch $__BuildArch \
--subsetCategory libraries
echo ROOTFS_DIR=\$\(realpath $__ToolchainDir/sysroot\) ./build.sh --cross --arch $__BuildArch \
--subsetCategory installer
echo "RID=android.21-arm64" > $__ToolchainDir/sysroot/android_platform
echo Now run:
echo CONFIG_DIR=\`realpath cross/android/$__BuildArch\` ROOTFS_DIR=\`realpath $__ToolchainDir/sysroot\` ./build.sh cross $__BuildArch skipgenerateversion skipnuget cmakeargs -DENABLE_LLDBPLUGIN=0
+28 -72
View File
@@ -2,16 +2,15 @@
usage()
{
echo "Usage: $0 [BuildArch] [CodeName] [lldbx.y] [--skipunmount] --rootfsdir <directory>]"
echo "Usage: $0 [BuildArch] [LinuxCodeName] [lldbx.y] [--skipunmount] --rootfsdir <directory>]"
echo "BuildArch can be: arm(default), armel, arm64, x86"
echo "CodeName - optional, Code name for Linux, can be: trusty, xenial(default), zesty, bionic, alpine. If BuildArch is armel, LinuxCodeName is jessie(default) or tizen."
echo " for FreeBSD can be: freebsd11 or freebsd12."
echo "lldbx.y - optional, LLDB version, can be: lldb3.9(default), lldb4.0, lldb5.0, lldb6.0 no-lldb. Ignored for alpine and FReeBSD"
echo "LinuxCodeName - optional, Code name for Linux, can be: trusty, xenial(default), zesty, bionic, alpine. If BuildArch is armel, LinuxCodeName is jessie(default) or tizen."
echo "lldbx.y - optional, LLDB version, can be: lldb3.9(default), lldb4.0, lldb5.0, lldb6.0 no-lldb. Ignored for alpine"
echo "--skipunmount - optional, will skip the unmount of rootfs folder."
exit 1
}
__CodeName=xenial
__LinuxCodeName=xenial
__CrossDir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
__InitialDir=$PWD
__BuildArch=arm
@@ -26,9 +25,8 @@ __UbuntuPackages="build-essential"
__AlpinePackages="alpine-base"
__AlpinePackages+=" build-base"
__AlpinePackages+=" linux-headers"
__AlpinePackagesEdgeTesting=" lldb-dev"
__AlpinePackagesEdgeMain=" llvm9-libs"
__AlpinePackagesEdgeMain+=" python3"
__AlpinePackages+=" lldb-dev"
__AlpinePackages+=" llvm-dev"
# symlinks fixer
__UbuntuPackages+=" symlinks"
@@ -54,15 +52,6 @@ __AlpinePackages+=" krb5-dev"
__AlpinePackages+=" openssl-dev"
__AlpinePackages+=" zlib-dev"
__FreeBSDBase="12.1-RELEASE"
__FreeBSDPkg="1.10.5"
__FreeBSDPackages="libunwind"
__FreeBSDPackages+=" icu"
__FreeBSDPackages+=" libinotify"
__FreeBSDPackages+=" lttng-ust"
__FreeBSDPackages+=" llvm-90"
__FreeBSDPackages+=" krb5"
__UnprocessedBuildArgs=
while :; do
if [ $# -le 0 ]; then
@@ -91,7 +80,7 @@ while :; do
__BuildArch=armel
__UbuntuArch=armel
__UbuntuRepo="http://ftp.debian.org/debian/"
__CodeName=jessie
__LinuxCodeName=jessie
;;
x86)
__BuildArch=x86
@@ -120,36 +109,36 @@ while :; do
unset __LLDB_Package
;;
trusty) # Ubuntu 14.04
if [ "$__CodeName" != "jessie" ]; then
__CodeName=trusty
if [ "$__LinuxCodeName" != "jessie" ]; then
__LinuxCodeName=trusty
fi
;;
xenial) # Ubuntu 16.04
if [ "$__CodeName" != "jessie" ]; then
__CodeName=xenial
if [ "$__LinuxCodeName" != "jessie" ]; then
__LinuxCodeName=xenial
fi
;;
zesty) # Ubuntu 17.04
if [ "$__CodeName" != "jessie" ]; then
__CodeName=zesty
if [ "$__LinuxCodeName" != "jessie" ]; then
__LinuxCodeName=zesty
fi
;;
bionic) # Ubuntu 18.04
if [ "$__CodeName" != "jessie" ]; then
__CodeName=bionic
if [ "$__LinuxCodeName" != "jessie" ]; then
__LinuxCodeName=bionic
fi
;;
jessie) # Debian 8
__CodeName=jessie
__LinuxCodeName=jessie
__UbuntuRepo="http://ftp.debian.org/debian/"
;;
stretch) # Debian 9
__CodeName=stretch
__LinuxCodeName=stretch
__UbuntuRepo="http://ftp.debian.org/debian/"
__LLDB_Package="liblldb-6.0-dev"
;;
buster) # Debian 10
__CodeName=buster
__LinuxCodeName=buster
__UbuntuRepo="http://ftp.debian.org/debian/"
__LLDB_Package="liblldb-6.0-dev"
;;
@@ -159,22 +148,14 @@ while :; do
usage;
exit 1;
fi
__CodeName=
__LinuxCodeName=
__UbuntuRepo=
__Tizen=tizen
;;
alpine)
__CodeName=alpine
__LinuxCodeName=alpine
__UbuntuRepo=
;;
freebsd11)
__FreeBSDBase="11.3-RELEASE"
;&
freebsd12)
__CodeName=freebsd
__BuildArch=x64
__SkipUnmount=1
;;
--skipunmount)
__SkipUnmount=1
;;
@@ -210,50 +191,25 @@ if [ -d "$__RootfsDir" ]; then
rm -rf $__RootfsDir
fi
if [[ "$__CodeName" == "alpine" ]]; then
if [[ "$__LinuxCodeName" == "alpine" ]]; then
__ApkToolsVersion=2.9.1
__AlpineVersion=3.9
__AlpineVersion=3.7
__ApkToolsDir=$(mktemp -d)
wget https://github.com/alpinelinux/apk-tools/releases/download/v$__ApkToolsVersion/apk-tools-$__ApkToolsVersion-x86_64-linux.tar.gz -P $__ApkToolsDir
tar -xf $__ApkToolsDir/apk-tools-$__ApkToolsVersion-x86_64-linux.tar.gz -C $__ApkToolsDir
mkdir -p $__RootfsDir/usr/bin
cp -v /usr/bin/qemu-$__QEMUArch-static $__RootfsDir/usr/bin
$__ApkToolsDir/apk-tools-$__ApkToolsVersion/apk \
-X http://dl-cdn.alpinelinux.org/alpine/v$__AlpineVersion/main \
-X http://dl-cdn.alpinelinux.org/alpine/v$__AlpineVersion/community \
-U --allow-untrusted --root $__RootfsDir --arch $__AlpineArch --initdb \
add $__AlpinePackages
$__ApkToolsDir/apk-tools-$__ApkToolsVersion/apk \
-X http://dl-cdn.alpinelinux.org/alpine/edge/testing \
-X http://dl-cdn.alpinelinux.org/alpine/edge/main \
-U --allow-untrusted --root $__RootfsDir --arch $__AlpineArch --initdb \
add $__AlpinePackagesEdgeMain
$__ApkToolsDir/apk-tools-$__ApkToolsVersion/apk \
-X http://dl-cdn.alpinelinux.org/alpine/edge/testing \
-U --allow-untrusted --root $__RootfsDir --arch $__AlpineArch --initdb \
add $__AlpinePackagesEdgeTesting
add $__AlpinePackages
rm -r $__ApkToolsDir
elif [[ "$__CodeName" == "freebsd" ]]; then
mkdir -p $__RootfsDir/usr/local/etc
wget -O - https://download.freebsd.org/ftp/releases/amd64/${__FreeBSDBase}/base.txz | tar -C $__RootfsDir -Jxf - ./lib ./usr/lib ./usr/libdata ./usr/include ./usr/share/keys ./etc ./bin/freebsd-version
# For now, ask for 11 ABI even on 12. This can be revisited later.
echo "ABI = \"FreeBSD:11:amd64\"; FINGERPRINTS = \"${__RootfsDir}/usr/share/keys\"; REPOS_DIR = [\"${__RootfsDir}/etc/pkg\"]; REPO_AUTOUPDATE = NO; RUN_SCRIPTS = NO;" > ${__RootfsDir}/usr/local/etc/pkg.conf
echo "FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/\${ABI}/quarterly", mirror_type: \"srv\", signature_type: \"fingerprints\", fingerprints: \"${__RootfsDir}/usr/share/keys/pkg\", enabled: yes }" > ${__RootfsDir}/etc/pkg/FreeBSD.conf
mkdir -p $__RootfsDir/tmp
# get and build package manager
wget -O - https://github.com/freebsd/pkg/archive/${__FreeBSDPkg}.tar.gz | tar -C $__RootfsDir/tmp -zxf -
cd $__RootfsDir/tmp/pkg-${__FreeBSDPkg}
./autogen.sh && ./configure --prefix=$__RootfsDir/host && make install
rm -rf $__RootfsDir/tmp/pkg-${__FreeBSDPkg}
# install packages we need.
$__RootfsDir/host/sbin/pkg -r $__RootfsDir -C $__RootfsDir/usr/local/etc/pkg.conf update
$__RootfsDir/host/sbin/pkg -r $__RootfsDir -C $__RootfsDir/usr/local/etc/pkg.conf install --yes $__FreeBSDPackages
elif [[ -n $__CodeName ]]; then
qemu-debootstrap --arch $__UbuntuArch $__CodeName $__RootfsDir $__UbuntuRepo
cp $__CrossDir/$__BuildArch/sources.list.$__CodeName $__RootfsDir/etc/apt/sources.list
elif [[ -n $__LinuxCodeName ]]; then
qemu-debootstrap --arch $__UbuntuArch $__LinuxCodeName $__RootfsDir $__UbuntuRepo
cp $__CrossDir/$__BuildArch/sources.list.$__LinuxCodeName $__RootfsDir/etc/apt/sources.list
chroot $__RootfsDir apt-get update
chroot $__RootfsDir apt-get -f -y install
chroot $__RootfsDir apt-get -y install $__UbuntuPackages
@@ -263,7 +219,7 @@ elif [[ -n $__CodeName ]]; then
umount $__RootfsDir/*
fi
if [[ "$__BuildArch" == "arm" && "$__CodeName" == "trusty" ]]; then
if [[ "$__BuildArch" == "arm" && "$__LinuxCodeName" == "trusty" ]]; then
pushd $__RootfsDir
patch -p1 < $__CrossDir/$__BuildArch/trusty.patch
patch -p1 < $__CrossDir/$__BuildArch/trusty-lttng-2.4.patch
+37 -61
View File
@@ -1,11 +1,7 @@
set(CROSS_ROOTFS $ENV{ROOTFS_DIR})
set(TARGET_ARCH_NAME $ENV{TARGET_BUILD_ARCH})
if(EXISTS ${CROSS_ROOTFS}/bin/freebsd-version)
set(CMAKE_SYSTEM_NAME FreeBSD)
else()
set(CMAKE_SYSTEM_NAME Linux)
endif()
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_VERSION 1)
if(TARGET_ARCH_NAME STREQUAL "armel")
@@ -31,17 +27,10 @@ elseif(TARGET_ARCH_NAME STREQUAL "arm64")
elseif(TARGET_ARCH_NAME STREQUAL "x86")
set(CMAKE_SYSTEM_PROCESSOR i686)
set(TOOLCHAIN "i686-linux-gnu")
elseif (CMAKE_SYSTEM_NAME STREQUAL "FreeBSD")
set(CMAKE_SYSTEM_PROCESSOR "x86_64")
set(triple "x86_64-unknown-freebsd11")
else()
message(FATAL_ERROR "Arch is ${TARGET_ARCH_NAME}. Only armel, arm, arm64 and x86 are supported!")
endif()
if(DEFINED ENV{TOOLCHAIN})
set(TOOLCHAIN $ENV{TOOLCHAIN})
endif()
# Specify include paths
if(TARGET_ARCH_NAME STREQUAL "armel")
if(DEFINED TIZEN_TOOLCHAIN)
@@ -50,53 +39,50 @@ if(TARGET_ARCH_NAME STREQUAL "armel")
endif()
endif()
if("$ENV{__DistroRid}" MATCHES "android.*")
if(TARGET_ARCH_NAME STREQUAL "arm")
set(ANDROID_ABI armeabi-v7a)
elseif(TARGET_ARCH_NAME STREQUAL "arm64")
set(ANDROID_ABI arm64-v8a)
# add_compile_param - adds only new options without duplicates.
# arg0 - list with result options, arg1 - list with new options.
# arg2 - optional argument, quick summary string for optional using CACHE FORCE mode.
macro(add_compile_param)
if(NOT ${ARGC} MATCHES "^(2|3)$")
message(FATAL_ERROR "Wrong using add_compile_param! Two or three parameters must be given! See add_compile_param description.")
endif()
foreach(OPTION ${ARGV1})
if(NOT ${ARGV0} MATCHES "${OPTION}($| )")
set(${ARGV0} "${${ARGV0}} ${OPTION}")
if(${ARGC} EQUAL "3") # CACHE FORCE mode
set(${ARGV0} "${${ARGV0}}" CACHE STRING "${ARGV2}" FORCE)
endif()
endif()
# extract platform number required by the NDK's toolchain
string(REGEX REPLACE ".*\\.([0-9]+)-.*" "\\1" ANDROID_PLATFORM "$ENV{__DistroRid}")
set(ANDROID_TOOLCHAIN clang)
set(FEATURE_EVENT_TRACE 0) # disable event trace as there is no lttng-ust package in termux repository
set(CMAKE_SYSTEM_LIBRARY_PATH "${CROSS_ROOTFS}/usr/lib")
set(CMAKE_SYSTEM_INCLUDE_PATH "${CROSS_ROOTFS}/usr/include")
# include official NDK toolchain script
include(${CROSS_ROOTFS}/../build/cmake/android.toolchain.cmake)
elseif (CMAKE_SYSTEM_NAME STREQUAL "FreeBSD")
# we cross-compile by instructing clang
set(CMAKE_C_COMPILER_TARGET ${triple})
set(CMAKE_CXX_COMPILER_TARGET ${triple})
set(CMAKE_ASM_COMPILER_TARGET ${triple})
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
else()
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
set(CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN "${CROSS_ROOTFS}/usr")
set(CMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN "${CROSS_ROOTFS}/usr")
set(CMAKE_ASM_COMPILER_EXTERNAL_TOOLCHAIN "${CROSS_ROOTFS}/usr")
endif()
endforeach()
endmacro()
# Specify link flags
add_compile_param(CROSS_LINK_FLAGS "--sysroot=${CROSS_ROOTFS}")
add_compile_param(CROSS_LINK_FLAGS "--gcc-toolchain=${CROSS_ROOTFS}/usr")
add_compile_param(CROSS_LINK_FLAGS "--target=${TOOLCHAIN}")
add_compile_param(CROSS_LINK_FLAGS "-fuse-ld=gold")
if(TARGET_ARCH_NAME STREQUAL "armel")
if(DEFINED TIZEN_TOOLCHAIN) # For Tizen only
add_link_options("-B${CROSS_ROOTFS}/usr/lib/gcc/${TIZEN_TOOLCHAIN}")
add_link_options("-L${CROSS_ROOTFS}/lib")
add_link_options("-L${CROSS_ROOTFS}/usr/lib")
add_link_options("-L${CROSS_ROOTFS}/usr/lib/gcc/${TIZEN_TOOLCHAIN}")
add_compile_param(CROSS_LINK_FLAGS "-B${CROSS_ROOTFS}/usr/lib/gcc/${TIZEN_TOOLCHAIN}")
add_compile_param(CROSS_LINK_FLAGS "-L${CROSS_ROOTFS}/lib")
add_compile_param(CROSS_LINK_FLAGS "-L${CROSS_ROOTFS}/usr/lib")
add_compile_param(CROSS_LINK_FLAGS "-L${CROSS_ROOTFS}/usr/lib/gcc/${TIZEN_TOOLCHAIN}")
endif()
elseif(TARGET_ARCH_NAME STREQUAL "x86")
add_link_options(-m32)
add_compile_param(CROSS_LINK_FLAGS "-m32")
endif()
# Specify compile options
add_compile_param(CMAKE_EXE_LINKER_FLAGS "${CROSS_LINK_FLAGS}" "TOOLCHAIN_EXE_LINKER_FLAGS")
add_compile_param(CMAKE_SHARED_LINKER_FLAGS "${CROSS_LINK_FLAGS}" "TOOLCHAIN_EXE_LINKER_FLAGS")
add_compile_param(CMAKE_MODULE_LINKER_FLAGS "${CROSS_LINK_FLAGS}" "TOOLCHAIN_EXE_LINKER_FLAGS")
if(TARGET_ARCH_NAME MATCHES "^(arm|armel|arm64)$" AND NOT "$ENV{__DistroRid}" MATCHES "android.*")
# Specify compile options
add_compile_options("--sysroot=${CROSS_ROOTFS}")
add_compile_options("--target=${TOOLCHAIN}")
add_compile_options("--gcc-toolchain=${CROSS_ROOTFS}/usr")
if(TARGET_ARCH_NAME MATCHES "^(arm|armel|arm64)$")
set(CMAKE_C_COMPILER_TARGET ${TOOLCHAIN})
set(CMAKE_CXX_COMPILER_TARGET ${TOOLCHAIN})
set(CMAKE_ASM_COMPILER_TARGET ${TOOLCHAIN})
@@ -104,17 +90,7 @@ endif()
if(TARGET_ARCH_NAME MATCHES "^(arm|armel)$")
add_compile_options(-mthumb)
if (NOT DEFINED CLR_ARM_FPU_TYPE)
set (CLR_ARM_FPU_TYPE vfpv3)
endif (NOT DEFINED CLR_ARM_FPU_TYPE)
add_compile_options (-mfpu=${CLR_ARM_FPU_TYPE})
if (NOT DEFINED CLR_ARM_FPU_CAPABILITY)
set (CLR_ARM_FPU_CAPABILITY 0x7)
endif (NOT DEFINED CLR_ARM_FPU_CAPABILITY)
add_definitions (-DCLR_ARM_FPU_CAPABILITY=${CLR_ARM_FPU_CAPABILITY})
add_compile_options(-mfpu=vfpv3)
if(TARGET_ARCH_NAME STREQUAL "armel")
add_compile_options(-mfloat-abi=softfp)
if(DEFINED TIZEN_TOOLCHAIN)
@@ -127,7 +103,7 @@ elseif(TARGET_ARCH_NAME STREQUAL "x86")
add_compile_options(-Wno-error=unused-command-line-argument)
endif()
# Set LLDB include and library paths for builds that need lldb.
# Set LLDB include and library paths
if(TARGET_ARCH_NAME MATCHES "^(arm|armel|x86)$")
if(TARGET_ARCH_NAME STREQUAL "x86")
set(LLVM_CROSS_DIR "$ENV{LLVM_CROSS_HOME}")
@@ -155,7 +131,7 @@ if(TARGET_ARCH_NAME MATCHES "^(arm|armel|x86)$")
endif()
endif()
set(CMAKE_FIND_ROOT_PATH "${CROSS_ROOTFS}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
+8 -22
View File
@@ -1,14 +1,13 @@
param (
$darcVersion = $null,
$versionEndpoint = 'https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16',
$verbosity = 'minimal',
$toolpath = $null
$versionEndpoint = "https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16"
)
$verbosity = "m"
. $PSScriptRoot\tools.ps1
function InstallDarcCli ($darcVersion, $toolpath) {
$darcCliPackageName = 'microsoft.dotnet.darc'
function InstallDarcCli ($darcVersion) {
$darcCliPackageName = "microsoft.dotnet.darc"
$dotnetRoot = InitializeDotNetCli -install:$true
$dotnet = "$dotnetRoot\dotnet.exe"
@@ -24,24 +23,11 @@ function InstallDarcCli ($darcVersion, $toolpath) {
$darcVersion = $(Invoke-WebRequest -Uri $versionEndpoint -UseBasicParsing).Content
}
$arcadeServicesSource = 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
$arcadeServicesSource = 'https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json'
Write-Host "Installing Darc CLI version $darcVersion..."
Write-Host 'You may need to restart your command window if this is the first dotnet tool you have installed.'
if (-not $toolpath) {
Write-Host "'$dotnet' tool install $darcCliPackageName --version $darcVersion --add-source '$arcadeServicesSource' -v $verbosity -g"
& "$dotnet" tool install $darcCliPackageName --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity -g
}else {
Write-Host "'$dotnet' tool install $darcCliPackageName --version $darcVersion --add-source '$arcadeServicesSource' -v $verbosity --tool-path '$toolpath'"
& "$dotnet" tool install $darcCliPackageName --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity --tool-path "$toolpath"
}
Write-Host "You may need to restart your command window if this is the first dotnet tool you have installed."
& "$dotnet" tool install $darcCliPackageName --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity -g
}
try {
InstallDarcCli $darcVersion $toolpath
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Darc' -Message $_
ExitWithExitCode 1
}
InstallDarcCli $darcVersion
+8 -26
View File
@@ -2,8 +2,7 @@
source="${BASH_SOURCE[0]}"
darcVersion=''
versionEndpoint='https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16'
verbosity='minimal'
versionEndpoint="https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16"
while [[ $# > 0 ]]; do
opt="$(echo "$1" | awk '{print tolower($0)}')"
@@ -16,14 +15,6 @@ while [[ $# > 0 ]]; do
versionEndpoint=$2
shift
;;
--verbosity)
verbosity=$2
shift
;;
--toolpath)
toolpath=$2
shift
;;
*)
echo "Invalid argument: $1"
usage
@@ -43,6 +34,7 @@ while [[ -h "$source" ]]; do
[[ $source != /* ]] && source="$scriptroot/$source"
done
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
verbosity=m
. "$scriptroot/tools.sh"
@@ -56,27 +48,17 @@ function InstallDarcCli {
InitializeDotNetCli
local dotnet_root=$_InitializeDotNetCli
if [ -z "$toolpath" ]; then
local tool_list=$($dotnet_root/dotnet tool list -g)
if [[ $tool_list = *$darc_cli_package_name* ]]; then
echo $($dotnet_root/dotnet tool uninstall $darc_cli_package_name -g)
fi
else
local tool_list=$($dotnet_root/dotnet tool list --tool-path "$toolpath")
if [[ $tool_list = *$darc_cli_package_name* ]]; then
echo $($dotnet_root/dotnet tool uninstall $darc_cli_package_name --tool-path "$toolpath")
fi
local uninstall_command=`$dotnet_root/dotnet tool uninstall $darc_cli_package_name -g`
local tool_list=$($dotnet_root/dotnet tool list -g)
if [[ $tool_list = *$darc_cli_package_name* ]]; then
echo $($dotnet_root/dotnet tool uninstall $darc_cli_package_name -g)
fi
local arcadeServicesSource="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json"
local arcadeServicesSource="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json"
echo "Installing Darc CLI version $darcVersion..."
echo "You may need to restart your command shell if this is the first dotnet tool you have installed."
if [ -z "$toolpath" ]; then
echo $($dotnet_root/dotnet tool install $darc_cli_package_name --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity -g)
else
echo $($dotnet_root/dotnet tool install $darc_cli_package_name --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity --tool-path "$toolpath")
fi
echo $($dotnet_root/dotnet tool install $darc_cli_package_name --version $darcVersion --add-source "$arcadeServicesSource" -v $verbosity -g)
}
InstallDarcCli
+11 -12
View File
@@ -1,27 +1,26 @@
[CmdletBinding(PositionalBinding=$false)]
Param(
[string] $verbosity = 'minimal',
[string] $architecture = '',
[string] $version = 'Latest',
[string] $runtime = 'dotnet',
[string] $RuntimeSourceFeed = '',
[string] $RuntimeSourceFeedKey = ''
[string] $verbosity = "minimal",
[string] $architecture = "",
[string] $version = "Latest",
[string] $runtime = "dotnet"
)
. $PSScriptRoot\tools.ps1
$dotnetRoot = Join-Path $RepoRoot '.dotnet'
$dotnetRoot = Join-Path $RepoRoot ".dotnet"
$installdir = $dotnetRoot
try {
if ($architecture -and $architecture.Trim() -eq 'x86') {
$installdir = Join-Path $installdir 'x86'
if ($architecture -and $architecture.Trim() -eq "x86") {
$installdir = Join-Path $installdir "x86"
}
InstallDotNet $installdir $version $architecture $runtime $true -RuntimeSourceFeed $RuntimeSourceFeed -RuntimeSourceFeedKey $RuntimeSourceFeedKey
}
InstallDotNet $installdir $version $architecture $runtime $true
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message $_
ExitWithExitCode 1
}
+5 -45
View File
@@ -11,13 +11,9 @@ while [[ -h "$source" ]]; do
done
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
. "$scriptroot/tools.sh"
version='Latest'
architecture=''
runtime='dotnet'
runtimeSourceFeed=''
runtimeSourceFeedKey=''
while [[ $# > 0 ]]; do
opt="$(echo "$1" | awk '{print tolower($0)}')"
case "$opt" in
@@ -33,56 +29,20 @@ while [[ $# > 0 ]]; do
shift
runtime="$1"
;;
-runtimesourcefeed)
shift
runtimeSourceFeed="$1"
;;
-runtimesourcefeedkey)
shift
runtimeSourceFeedKey="$1"
;;
*)
Write-PipelineTelemetryError -Category 'Build' -Message "Invalid argument: $1"
echo "Invalid argument: $1"
usage
exit 1
;;
esac
shift
done
# Use uname to determine what the CPU is.
cpuname=$(uname -p)
# Some Linux platforms report unknown for platform, but the arch for machine.
if [[ "$cpuname" == "unknown" ]]; then
cpuname=$(uname -m)
fi
case $cpuname in
aarch64)
buildarch=arm64
;;
amd64|x86_64)
buildarch=x64
;;
armv*l)
buildarch=arm
;;
i686)
buildarch=x86
;;
*)
echo "Unknown CPU $cpuname detected, treating it as x64"
buildarch=x64
;;
esac
. "$scriptroot/tools.sh"
dotnetRoot="$repo_root/.dotnet"
if [[ $architecture != "" ]] && [[ $architecture != $buildarch ]]; then
dotnetRoot="$dotnetRoot/$architecture"
fi
InstallDotNet $dotnetRoot $version "$architecture" $runtime true $runtimeSourceFeed $runtimeSourceFeedKey || {
InstallDotNet $dotnetRoot $version "$architecture" $runtime true || {
local exit_code=$?
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "dotnet-install.sh failed (exit code '$exit_code')." >&2
echo "dotnet-install.sh failed (exit code '$exit_code')." >&2
ExitWithExitCode $exit_code
}
@@ -1,13 +0,0 @@
param(
[string] $token
)
. $PSScriptRoot\pipeline-logging-functions.ps1
# Write-PipelineSetVariable will no-op if a variable named $ci is not defined
# Since this script is only ever called in AzDO builds, just universally set it
$ci = $true
Write-PipelineSetVariable -Name 'VSS_NUGET_ACCESSTOKEN' -Value $token -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'VSS_NUGET_URI_PREFIXES' -Value 'https://dnceng.pkgs.visualstudio.com/;https://pkgs.dev.azure.com/dnceng/;https://devdiv.pkgs.visualstudio.com/;https://pkgs.dev.azure.com/devdiv/' -IsMultiJobVariable $false
+26 -25
View File
@@ -3,39 +3,39 @@ Param(
[Parameter(Mandatory=$true)][string] $gitHubPat, # GitHub personal access token from https://github.com/settings/tokens (no auth scopes needed)
[Parameter(Mandatory=$true)][string] $azdoPat, # Azure Dev Ops tokens from https://dev.azure.com/dnceng/_details/security/tokens (code read scope needed)
[Parameter(Mandatory=$true)][string] $outputFolder, # Where the graphviz.txt file will be created
[string] $darcVersion, # darc's version
[string] $darcVersion = '1.1.0-beta.19175.6', # darc's version
[string] $graphvizVersion = '2.38', # GraphViz version
[switch] $includeToolset # Whether the graph should include toolset dependencies or not. i.e. arcade, optimization. For more about
# toolset dependencies see https://github.com/dotnet/arcade/blob/master/Documentation/Darc.md#toolset-vs-product-dependencies
)
$ErrorActionPreference = "Stop"
. $PSScriptRoot\tools.ps1
Import-Module -Name (Join-Path $PSScriptRoot "native\CommonLibrary.psm1")
function CheckExitCode ([string]$stage)
{
$exitCode = $LASTEXITCODE
if ($exitCode -ne 0) {
Write-PipelineTelemetryError -Category 'Arcade' -Message "Something failed in stage: '$stage'. Check for errors above. Exiting now..."
Write-Host "Something failed in stage: '$stage'. Check for errors above. Exiting now..."
ExitWithExitCode $exitCode
}
}
try {
$ErrorActionPreference = 'Stop'
. $PSScriptRoot\tools.ps1
Import-Module -Name (Join-Path $PSScriptRoot 'native\CommonLibrary.psm1')
Push-Location $PSScriptRoot
Write-Host 'Installing darc...'
Write-Host "Installing darc..."
. .\darc-init.ps1 -darcVersion $darcVersion
CheckExitCode 'Running darc-init'
CheckExitCode "Running darc-init"
$engCommonBaseDir = Join-Path $PSScriptRoot 'native\'
$engCommonBaseDir = Join-Path $PSScriptRoot "native\"
$graphvizInstallDir = CommonLibrary\Get-NativeInstallDirectory
$nativeToolBaseUri = 'https://netcorenativeassets.blob.core.windows.net/resource-packages/external'
$installBin = Join-Path $graphvizInstallDir 'bin'
$nativeToolBaseUri = "https://netcorenativeassets.blob.core.windows.net/resource-packages/external"
$installBin = Join-Path $graphvizInstallDir "bin"
Write-Host 'Installing dot...'
Write-Host "Installing dot..."
.\native\install-tool.ps1 -ToolName graphviz -InstallPath $installBin -BaseUri $nativeToolBaseUri -CommonLibraryDirectory $engCommonBaseDir -Version $graphvizVersion -Verbose
$darcExe = "$env:USERPROFILE\.dotnet\tools"
@@ -51,36 +51,37 @@ try {
$graphVizImageFilePath = "$outputFolder\graph.png"
$normalGraphFilePath = "$outputFolder\graph-full.txt"
$flatGraphFilePath = "$outputFolder\graph-flat.txt"
$baseOptions = @( '--github-pat', "$gitHubPat", '--azdev-pat', "$azdoPat", '--password', "$barToken" )
$baseOptions = @( "--github-pat", "$gitHubPat", "--azdev-pat", "$azdoPat", "--password", "$barToken" )
if ($includeToolset) {
Write-Host 'Toolsets will be included in the graph...'
$baseOptions += @( '--include-toolset' )
Write-Host "Toolsets will be included in the graph..."
$baseOptions += @( "--include-toolset" )
}
Write-Host 'Generating standard dependency graph...'
Write-Host "Generating standard dependency graph..."
& "$darcExe" get-dependency-graph @baseOptions --output-file $normalGraphFilePath
CheckExitCode 'Generating normal dependency graph'
CheckExitCode "Generating normal dependency graph"
Write-Host 'Generating flat dependency graph and graphviz file...'
Write-Host "Generating flat dependency graph and graphviz file..."
& "$darcExe" get-dependency-graph @baseOptions --flat --coherency --graphviz $graphVizFilePath --output-file $flatGraphFilePath
CheckExitCode 'Generating flat and graphviz dependency graph'
CheckExitCode "Generating flat and graphviz dependency graph"
Write-Host "Generating graph image $graphVizFilePath"
$dotFilePath = Join-Path $installBin "graphviz\$graphvizVersion\release\bin\dot.exe"
& "$dotFilePath" -Tpng -o"$graphVizImageFilePath" "$graphVizFilePath"
CheckExitCode 'Generating graphviz image'
CheckExitCode "Generating graphviz image"
Write-Host "'$graphVizFilePath', '$flatGraphFilePath', '$normalGraphFilePath' and '$graphVizImageFilePath' created!"
}
catch {
if (!$includeToolset) {
Write-Host 'This might be a toolset repo which includes only toolset dependencies. ' -NoNewline -ForegroundColor Yellow
Write-Host 'Since -includeToolset is not set there is no graph to create. Include -includeToolset and try again...' -ForegroundColor Yellow
Write-Host "This might be a toolset repo which includes only toolset dependencies. " -NoNewline -ForegroundColor Yellow
Write-Host "Since -includeToolset is not set there is no graph to create. Include -includeToolset and try again..." -ForegroundColor Yellow
}
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Arcade' -Message $_
ExitWithExitCode 1
} finally {
Pop-Location
Pop-Location
}
+16 -21
View File
@@ -35,7 +35,7 @@ File path to global.json file
#>
[CmdletBinding(PositionalBinding=$false)]
Param (
[string] $BaseUri = 'https://netcorenativeassets.blob.core.windows.net/resource-packages/external',
[string] $BaseUri = "https://netcorenativeassets.blob.core.windows.net/resource-packages/external",
[string] $InstallDirectory,
[switch] $Clean = $False,
[switch] $Force = $False,
@@ -45,27 +45,26 @@ Param (
)
if (!$GlobalJsonFile) {
$GlobalJsonFile = Join-Path (Get-Item $PSScriptRoot).Parent.Parent.FullName 'global.json'
$GlobalJsonFile = Join-Path (Get-Item $PSScriptRoot).Parent.Parent.FullName "global.json"
}
Set-StrictMode -version 2.0
$ErrorActionPreference='Stop'
$ErrorActionPreference="Stop"
. $PSScriptRoot\pipeline-logging-functions.ps1
Import-Module -Name (Join-Path $PSScriptRoot 'native\CommonLibrary.psm1')
Import-Module -Name (Join-Path $PSScriptRoot "native\CommonLibrary.psm1")
try {
# Define verbose switch if undefined
$Verbose = $VerbosePreference -Eq 'Continue'
$Verbose = $VerbosePreference -Eq "Continue"
$EngCommonBaseDir = Join-Path $PSScriptRoot 'native\'
$EngCommonBaseDir = Join-Path $PSScriptRoot "native\"
$NativeBaseDir = $InstallDirectory
if (!$NativeBaseDir) {
$NativeBaseDir = CommonLibrary\Get-NativeInstallDirectory
}
$Env:CommonLibrary_NativeInstallDir = $NativeBaseDir
$InstallBin = Join-Path $NativeBaseDir 'bin'
$InstallerPath = Join-Path $EngCommonBaseDir 'install-tool.ps1'
$InstallBin = Join-Path $NativeBaseDir "bin"
$InstallerPath = Join-Path $EngCommonBaseDir "install-tool.ps1"
# Process tools list
Write-Host "Processing $GlobalJsonFile"
@@ -75,7 +74,7 @@ try {
}
$NativeTools = Get-Content($GlobalJsonFile) -Raw |
ConvertFrom-Json |
Select-Object -Expand 'native-tools' -ErrorAction SilentlyContinue
Select-Object -Expand "native-tools" -ErrorAction SilentlyContinue
if ($NativeTools) {
$NativeTools.PSObject.Properties | ForEach-Object {
$ToolName = $_.Name
@@ -113,21 +112,18 @@ try {
}
$toolInstallationFailure = $true
} else {
# We cannot change this to Write-PipelineTelemetryError because of https://github.com/dotnet/arcade/issues/4482
Write-Host $errMsg
Write-Error $errMsg
exit 1
}
}
}
if ((Get-Variable 'toolInstallationFailure' -ErrorAction 'SilentlyContinue') -and $toolInstallationFailure) {
# We cannot change this to Write-PipelineTelemetryError because of https://github.com/dotnet/arcade/issues/4482
Write-Host 'Native tools bootstrap failed'
exit 1
}
}
else {
Write-Host 'No native tools defined in global.json'
Write-Host "No native tools defined in global.json"
exit 0
}
@@ -135,18 +131,17 @@ try {
exit 0
}
if (Test-Path $InstallBin) {
Write-Host 'Native tools are available from ' (Convert-Path -Path $InstallBin)
Write-Host "Native tools are available from" (Convert-Path -Path $InstallBin)
Write-Host "##vso[task.prependpath]$(Convert-Path -Path $InstallBin)"
return $InstallBin
}
else {
Write-PipelineTelemetryError -Category 'NativeToolsBootstrap' -Message 'Native tools install directory does not exist, installation failed'
Write-Error "Native tools install directory does not exist, installation failed"
exit 1
}
exit 0
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'NativeToolsBootstrap' -Message $_
ExitWithExitCode 1
Write-Host $_
Write-Host $_.Exception
exit 1
}
+11 -43
View File
@@ -12,7 +12,6 @@ retry_wait_time_seconds=30
global_json_file="$(dirname "$(dirname "${scriptroot}")")/global.json"
declare -A native_assets
. $scriptroot/pipeline-logging-functions.sh
. $scriptroot/native/common-library.sh
while (($# > 0)); do
@@ -34,14 +33,6 @@ while (($# > 0)); do
force=true
shift 1
;;
--donotabortonfailure)
donotabortonfailure=true
shift 1
;;
--donotdisplaywarnings)
donotdisplaywarnings=true
shift 1
;;
--downloadretries)
download_retries=$2
shift 2
@@ -60,8 +51,6 @@ while (($# > 0)); do
echo " - (default) %USERPROFILE%/.netcoreeng/native"
echo ""
echo " --clean Switch specifying not to install anything, but cleanup native asset folders"
echo " --donotabortonfailure Switch specifiying whether to abort native tools installation on failure"
echo " --donotdisplaywarnings Switch specifiying whether to display warnings during native tools installation on failure"
echo " --force Clean and then install tools"
echo " --help Print help and exit"
echo ""
@@ -81,7 +70,8 @@ function ReadGlobalJsonNativeTools {
# Only extract the contents of the object.
local native_tools_list=$(echo $native_tools_section | awk -F"[{}]" '{print $2}')
native_tools_list=${native_tools_list//[\" ]/}
native_tools_list=$( echo "$native_tools_list" | sed 's/\s//g' | sed 's/,/\n/g' )
native_tools_list=${native_tools_list//,/$'\n'}
native_tools_list="$(echo -e "${native_tools_list}" | tr -d '[[:space:]]')"
local old_IFS=$IFS
while read -r line; do
@@ -102,7 +92,6 @@ if [[ -z $install_directory ]]; then
fi
install_bin="${native_base_dir}/bin"
installed_any=false
ReadGlobalJsonNativeTools
@@ -114,12 +103,11 @@ else
for tool in "${!native_assets[@]}"
do
tool_version=${native_assets[$tool]}
installer_path="$native_installer_dir/install-$tool.sh"
installer_command="$installer_path"
installer_name="install-$tool.sh"
installer_command="$native_installer_dir/$installer_name"
installer_command+=" --baseuri $base_uri"
installer_command+=" --installpath $install_bin"
installer_command+=" --version $tool_version"
echo $installer_command
if [[ $force = true ]]; then
installer_command+=" --force"
@@ -129,29 +117,11 @@ else
installer_command+=" --clean"
fi
if [[ -a $installer_path ]]; then
$installer_command
if [[ $? != 0 ]]; then
if [[ $donotabortonfailure = true ]]; then
if [[ $donotdisplaywarnings != true ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Execution Failed"
fi
else
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Execution Failed"
exit 1
fi
else
$installed_any = true
fi
else
if [[ $donotabortonfailure == true ]]; then
if [[ $donotdisplaywarnings != true ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Execution Failed: no install script"
fi
else
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Execution Failed: no install script"
exit 1
fi
$installer_command
if [[ $? != 0 ]]; then
echo "Execution Failed" >&2
exit 1
fi
done
fi
@@ -164,10 +134,8 @@ if [[ -d $install_bin ]]; then
echo "Native tools are available from $install_bin"
echo "##vso[task.prependpath]$install_bin"
else
if [[ $installed_any = true ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Native tools install directory does not exist, installation failed"
exit 1
fi
echo "Native tools install directory does not exist, installation failed" >&2
exit 1
fi
exit 0
+15 -13
View File
@@ -6,8 +6,9 @@ param(
[switch] $IsFeedPrivate
)
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
. $PSScriptRoot\tools.ps1
# Sets VSS_NUGET_EXTERNAL_FEED_ENDPOINTS based on the "darc-int-*" feeds defined in NuGet.config. This is needed
@@ -20,7 +21,7 @@ function SetupCredProvider {
)
# Install the Cred Provider NuGet plugin
Write-Host 'Setting up Cred Provider NuGet plugin in the agent...'
Write-Host "Setting up Cred Provider NuGet plugin in the agent..."
Write-Host "Getting 'installcredprovider.ps1' from 'https://github.com/microsoft/artifacts-credprovider'..."
$url = 'https://raw.githubusercontent.com/microsoft/artifacts-credprovider/master/helpers/installcredprovider.ps1'
@@ -28,18 +29,18 @@ function SetupCredProvider {
Write-Host "Writing the contents of 'installcredprovider.ps1' locally..."
Invoke-WebRequest $url -OutFile installcredprovider.ps1
Write-Host 'Installing plugin...'
Write-Host "Installing plugin..."
.\installcredprovider.ps1 -Force
Write-Host "Deleting local copy of 'installcredprovider.ps1'..."
Remove-Item .\installcredprovider.ps1
if (-Not("$env:USERPROFILE\.nuget\plugins\netcore")) {
Write-PipelineTelemetryError -Category 'Arcade' -Message 'CredProvider plugin was not installed correctly!'
Write-Host "CredProvider plugin was not installed correctly!"
ExitWithExitCode 1
}
else {
Write-Host 'CredProvider plugin was installed correctly!'
Write-Host "CredProvider plugin was installed correctly!"
}
# Then, we set the 'VSS_NUGET_EXTERNAL_FEED_ENDPOINTS' environment variable to restore from the stable
@@ -48,7 +49,7 @@ function SetupCredProvider {
$nugetConfigPath = "$RepoRoot\NuGet.config"
if (-Not (Test-Path -Path $nugetConfigPath)) {
Write-PipelineTelemetryError -Category 'Build' -Message 'NuGet.config file not found in repo root!'
Write-Host "NuGet.config file not found in repo's root!"
ExitWithExitCode 1
}
@@ -80,7 +81,7 @@ function SetupCredProvider {
}
else
{
Write-Host 'No internal endpoints found in NuGet.config'
Write-Host "No internal endpoints found in NuGet.config"
}
}
@@ -98,7 +99,7 @@ function InstallDotNetSdkAndRestoreArcade {
& $dotnet restore $restoreProjPath
Write-Host 'Arcade SDK restored!'
Write-Host "Arcade SDK restored!"
if (Test-Path -Path $restoreProjPath) {
Remove-Item $restoreProjPath
@@ -112,22 +113,23 @@ function InstallDotNetSdkAndRestoreArcade {
try {
Push-Location $PSScriptRoot
if ($Operation -like 'setup') {
if ($Operation -like "setup") {
SetupCredProvider $AuthToken
}
elseif ($Operation -like 'install-restore') {
elseif ($Operation -like "install-restore") {
InstallDotNetSdkAndRestoreArcade
}
else {
Write-PipelineTelemetryError -Category 'Arcade' -Message "Unknown operation '$Operation'!"
Write-Host "Unknown operation '$Operation'!"
ExitWithExitCode 1
}
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Arcade' -Message $_
ExitWithExitCode 1
}
finally {
Pop-Location
Pop-Location
}
+2 -2
View File
@@ -30,7 +30,7 @@ function SetupCredProvider {
rm installcredprovider.sh
if [ ! -d "$HOME/.nuget/plugins" ]; then
Write-PipelineTelemetryError -category 'Build' 'CredProvider plugin was not installed correctly!'
echo "CredProvider plugin was not installed correctly!"
ExitWithExitCode 1
else
echo "CredProvider plugin was installed correctly!"
@@ -42,7 +42,7 @@ function SetupCredProvider {
local nugetConfigPath="$repo_root/NuGet.config"
if [ ! "$nugetConfigPath" ]; then
Write-PipelineTelemetryError -category 'Build' "NuGet.config file not found in repo's root!"
echo "NuGet.config file not found in repo's root!"
ExitWithExitCode 1
fi
+3 -2
View File
@@ -1,6 +1,6 @@
[CmdletBinding(PositionalBinding=$false)]
Param(
[string] $verbosity = 'minimal',
[string] $verbosity = "minimal",
[bool] $warnAsError = $true,
[bool] $nodeReuse = $true,
[switch] $ci,
@@ -18,8 +18,9 @@ try {
MSBuild @extraArgs
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Build' -Message $_
ExitWithExitCode 1
}
-2
View File
@@ -152,8 +152,6 @@ function Get-File {
}
else {
Write-Verbose "Downloading $Uri"
# Don't display the console progress UI - it's a huge perf hit
$ProgressPreference = 'SilentlyContinue'
while($Attempt -Lt $DownloadRetries)
{
try {
+5 -5
View File
@@ -34,7 +34,7 @@ function ExpandZip {
echo "'Force flag enabled, but '$output_directory' exists. Removing directory"
rm -rf $output_directory
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Unable to remove '$output_directory'"
echo Unable to remove '$output_directory'>&2
return 1
fi
fi
@@ -45,7 +45,7 @@ function ExpandZip {
echo "Extracting archive"
tar -xf $zip_path -C $output_directory
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Unable to extract '$zip_path'"
echo "Unable to extract '$zip_path'" >&2
return 1
fi
@@ -117,7 +117,7 @@ function DownloadAndExtract {
# Download file
GetFile "$uri" "$temp_tool_path" $force $download_retries $retry_wait_time_seconds
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Failed to download '$uri' to '$temp_tool_path'."
echo "Failed to download '$uri' to '$temp_tool_path'." >&2
return 1
fi
@@ -125,7 +125,7 @@ function DownloadAndExtract {
echo "extracting from $temp_tool_path to $installDir"
ExpandZip "$temp_tool_path" "$installDir" $force $download_retries $retry_wait_time_seconds
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Failed to extract '$temp_tool_path' to '$installDir'."
echo "Failed to extract '$temp_tool_path' to '$installDir'." >&2
return 1
fi
@@ -148,7 +148,7 @@ function NewScriptShim {
fi
if [[ ! -f $tool_file_path ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' "Specified tool file path:'$tool_file_path' does not exist"
echo "Specified tool file path:'$tool_file_path' does not exist" >&2
return 1
fi
-117
View File
@@ -1,117 +0,0 @@
#!/usr/bin/env bash
source="${BASH_SOURCE[0]}"
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
. $scriptroot/common-library.sh
base_uri=
install_path=
version=
clean=false
force=false
download_retries=5
retry_wait_time_seconds=30
while (($# > 0)); do
lowerI="$(echo $1 | awk '{print tolower($0)}')"
case $lowerI in
--baseuri)
base_uri=$2
shift 2
;;
--installpath)
install_path=$2
shift 2
;;
--version)
version=$2
shift 2
;;
--clean)
clean=true
shift 1
;;
--force)
force=true
shift 1
;;
--downloadretries)
download_retries=$2
shift 2
;;
--retrywaittimeseconds)
retry_wait_time_seconds=$2
shift 2
;;
--help)
echo "Common settings:"
echo " --baseuri <value> Base file directory or Url wrom which to acquire tool archives"
echo " --installpath <value> Base directory to install native tool to"
echo " --clean Don't install the tool, just clean up the current install of the tool"
echo " --force Force install of tools even if they previously exist"
echo " --help Print help and exit"
echo ""
echo "Advanced settings:"
echo " --downloadretries Total number of retry attempts"
echo " --retrywaittimeseconds Wait time between retry attempts in seconds"
echo ""
exit 0
;;
esac
done
tool_name="cmake-test"
tool_os=$(GetCurrentOS)
tool_folder=$(echo $tool_os | awk '{print tolower($0)}')
tool_arch="x86_64"
tool_name_moniker="$tool_name-$version-$tool_os-$tool_arch"
tool_install_directory="$install_path/$tool_name/$version"
tool_file_path="$tool_install_directory/$tool_name_moniker/bin/$tool_name"
shim_path="$install_path/$tool_name.sh"
uri="${base_uri}/$tool_folder/$tool_name/$tool_name_moniker.tar.gz"
# Clean up tool and installers
if [[ $clean = true ]]; then
echo "Cleaning $tool_install_directory"
if [[ -d $tool_install_directory ]]; then
rm -rf $tool_install_directory
fi
echo "Cleaning $shim_path"
if [[ -f $shim_path ]]; then
rm -rf $shim_path
fi
tool_temp_path=$(GetTempPathFileName $uri)
echo "Cleaning $tool_temp_path"
if [[ -f $tool_temp_path ]]; then
rm -rf $tool_temp_path
fi
exit 0
fi
# Install tool
if [[ -f $tool_file_path ]] && [[ $force = false ]]; then
echo "$tool_name ($version) already exists, skipping install"
exit 0
fi
DownloadAndExtract $uri $tool_install_directory $force $download_retries $retry_wait_time_seconds
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' 'Installation failed'
exit 1
fi
# Generate Shim
# Always rewrite shims so that we are referencing the expected version
NewScriptShim $shim_path $tool_file_path true
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' 'Shim generation failed'
exit 1
fi
exit 0
+3 -3
View File
@@ -69,7 +69,7 @@ tool_name_moniker="$tool_name-$version-$tool_os-$tool_arch"
tool_install_directory="$install_path/$tool_name/$version"
tool_file_path="$tool_install_directory/$tool_name_moniker/bin/$tool_name"
shim_path="$install_path/$tool_name.sh"
uri="${base_uri}/$tool_folder/$tool_name/$tool_name_moniker.tar.gz"
uri="${base_uri}/$tool_folder/cmake/$tool_name_moniker.tar.gz"
# Clean up tool and installers
if [[ $clean = true ]]; then
@@ -101,7 +101,7 @@ fi
DownloadAndExtract $uri $tool_install_directory $force $download_retries $retry_wait_time_seconds
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' 'Installation failed'
echo "Installation failed" >&2
exit 1
fi
@@ -110,7 +110,7 @@ fi
NewScriptShim $shim_path $tool_file_path true
if [[ $? != 0 ]]; then
Write-PipelineTelemetryError -category 'NativeToolsBootstrap' 'Shim generation failed'
echo "Shim generation failed" >&2
exit 1
fi
+5 -7
View File
@@ -46,8 +46,6 @@ Param (
[int] $RetryWaitTimeInSeconds = 30
)
. $PSScriptRoot\..\pipeline-logging-functions.ps1
# Import common library modules
Import-Module -Name (Join-Path $CommonLibraryDirectory "CommonLibrary.psm1")
@@ -95,7 +93,7 @@ try {
-Verbose:$Verbose
if ($InstallStatus -Eq $False) {
Write-PipelineTelemetryError "Installation failed" -Category "NativeToolsetBootstrapping"
Write-Error "Installation failed"
exit 1
}
}
@@ -105,7 +103,7 @@ try {
Write-Error "There are multiple copies of $ToolName in $($ToolInstallDirectory): `n$(@($ToolFilePath | out-string))"
exit 1
} elseif (@($ToolFilePath).Length -Lt 1) {
Write-Host "$ToolName was not found in $ToolFilePath."
Write-Error "$ToolName was not found in $ToolFilePath."
exit 1
}
@@ -119,14 +117,14 @@ try {
-Verbose:$Verbose
if ($GenerateShimStatus -Eq $False) {
Write-PipelineTelemetryError "Generate shim failed" -Category "NativeToolsetBootstrapping"
Write-Error "Generate shim failed"
return 1
}
exit 0
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category "NativeToolsetBootstrapping" -Message $_
Write-Host $_
Write-Host $_.Exception
exit 1
}
+3 -47
View File
@@ -5,14 +5,8 @@
<CliArguments>--dotnet-versions %DOTNET_VERSION% --cli-source-info args --cli-branch %PERFLAB_BRANCH% --cli-commit-sha %PERFLAB_HASH% --cli-repository https://github.com/%PERFLAB_REPO% --cli-source-timestamp %PERFLAB_BUILDTIMESTAMP%</CliArguments>
<Python>py -3</Python>
<CoreRun>%HELIX_CORRELATION_PAYLOAD%\Core_Root\CoreRun.exe</CoreRun>
<BaselineCoreRun>%HELIX_CORRELATION_PAYLOAD%\Baseline_Core_Root\CoreRun.exe</BaselineCoreRun>
<HelixPreCommands>$(HelixPreCommands);call %HELIX_CORRELATION_PAYLOAD%\performance\tools\machine-setup.cmd;set PYTHONPATH=%HELIX_WORKITEM_PAYLOAD%\scripts%3B%HELIX_WORKITEM_PAYLOAD%</HelixPreCommands>
<HelixPreCommands>$(HelixPreCommands);call %HELIX_CORRELATION_PAYLOAD%\performance\tools\machine-setup.cmd</HelixPreCommands>
<ArtifactsDirectory>%HELIX_CORRELATION_PAYLOAD%\artifacts\BenchmarkDotNet.Artifacts</ArtifactsDirectory>
<BaselineArtifactsDirectory>%HELIX_CORRELATION_PAYLOAD%\artifacts\BenchmarkDotNet.Artifacts_Baseline</BaselineArtifactsDirectory>
<ResultsComparer>%HELIX_CORRELATION_PAYLOAD%\performance\src\tools\ResultsComparer\ResultsComparer.csproj</ResultsComparer>
<DotnetExe>%HELIX_CORRELATION_PAYLOAD%\performance\tools\dotnet\$(Architecture)\dotnet.exe</DotnetExe>
<Percent>%25%25</Percent>
<XMLResults>%HELIX_WORKITEM_ROOT%\testResults.xml</XMLResults>
</PropertyGroup>
<PropertyGroup Condition="'$(AGENT_OS)' != 'Windows_NT' and '$(RunFromPerfRepo)' == 'false'">
@@ -30,24 +24,14 @@
<CliArguments>--dotnet-versions $DOTNET_VERSION --cli-source-info args --cli-branch $PERFLAB_BRANCH --cli-commit-sha $PERFLAB_HASH --cli-repository https://github.com/$PERFLAB_REPO --cli-source-timestamp $PERFLAB_BUILDTIMESTAMP</CliArguments>
<Python>python3</Python>
<CoreRun>$(BaseDirectory)/Core_Root/corerun</CoreRun>
<BaselineCoreRun>$(BaseDirectory)/Baseline_Core_Root/corerun</BaselineCoreRun>
<HelixPreCommands>$(HelixPreCommands);chmod +x $(PerformanceDirectory)/tools/machine-setup.sh;. $(PerformanceDirectory)/tools/machine-setup.sh</HelixPreCommands>
<ArtifactsDirectory>$(BaseDirectory)/artifacts/BenchmarkDotNet.Artifacts</ArtifactsDirectory>
<BaselineArtifactsDirectory>$(BaseDirectory)/artifacts/BenchmarkDotNet.Artifacts_Baseline</BaselineArtifactsDirectory>
<ResultsComparer>$(PerformanceDirectory)/src/tools/ResultsComparer/ResultsComparer.csproj</ResultsComparer>
<DotnetExe>$(PerformanceDirectory)/tools/dotnet/$(Architecture)/dotnet</DotnetExe>
<Percent>%25</Percent>
<XMLResults>$HELIX_WORKITEM_ROOT/testResults.xml</XMLResults>
</PropertyGroup>
<PropertyGroup Condition="'$(UseCoreRun)' == 'true'">
<CoreRunArgument>--corerun $(CoreRun)</CoreRunArgument>
</PropertyGroup>
<PropertyGroup Condition="'$(UseBaselineCoreRun)' == 'true'">
<BaselineCoreRunArgument>--corerun $(BaselineCoreRun)</BaselineCoreRunArgument>
</PropertyGroup>
<PropertyGroup Condition="'$(WorkItemCommand)' != ''">
<WorkItemCommand>$(Python) $(WorkItemCommand) --incremental no --architecture $(Architecture) -f $(_Framework) $(PerfLabArguments)</WorkItemCommand>
</PropertyGroup>
@@ -73,49 +57,21 @@
<Partition Include="$(BuildConfig).Partition4" Index="4" />
</ItemGroup>
<PropertyGroup Condition="'$(Compare)' == 'true'">
<FailOnTestFailure>false</FailOnTestFailure>
</PropertyGroup>
<!--
Partition the Microbenchmarks project, but nothing else
-->
<ItemGroup Condition="$(TargetCsproj.Contains('MicroBenchmarks.csproj'))">
<HelixWorkItem Include="@(Partition)">
<PayloadDirectory>$(WorkItemDirectory)</PayloadDirectory>
<PreCommands Condition="'$(Compare)' == 'true'">$(WorkItemCommand) --bdn-artifacts $(BaselineArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(BaselineCoreRunArgument) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</PreCommands>
<Command>$(WorkItemCommand) --bdn-artifacts $(ArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</Command>
<PostCommands Condition="'$(Compare)' == 'true'">$(DotnetExe) run -f $(_Framework) -p $(ResultsComparer) --base $(BaselineArtifactsDirectory) --diff $(ArtifactsDirectory) --threshold 2$(Percent) --xml $(XMLResults);$(FinalCommand)</PostCommands>
<Command>$(WorkItemCommand) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument) --artifacts $(ArtifactsDirectory) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</Command>
<Timeout>4:00</Timeout>
</HelixWorkItem>
</ItemGroup>
<ItemGroup Condition="!$(TargetCsproj.Contains('MicroBenchmarks.csproj'))">
<HelixWorkItem Include="$(BuildConfig).WorkItem">
<PayloadDirectory>$(WorkItemDirectory)</PayloadDirectory>
<PreCommands Condition="'$(Compare)' == 'true'">$(WorkItemCommand) --bdn-artifacts $(BaselineArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(BaselineCoreRunArgument)"</PreCommands>
<Command>$(WorkItemCommand) --bdn-artifacts $(ArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument)"</Command>
<PostCommands Condition="'$(Compare)' == 'true'">$(DotnetExe) run -f $(_Framework) -p $(ResultsComparer) --base $(BaselineArtifactsDirectory) --diff $(ArtifactsDirectory) --threshold 2$(Percent) --xml $(XMLResults)</PostCommands>
<Command>$(WorkItemCommand) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument) --artifacts $(ArtifactsDirectory)"</Command>
<Timeout>4:00</Timeout>
</HelixWorkItem>
</ItemGroup>
<ItemGroup Condition="'$(AGENT_OS)' == 'Windows_NT'">
<HelixWorkItem Include="Crossgen System.Private.Xml.dll">
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen\test.py crossgen --test-name System.Private.Xml.dll --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
</HelixWorkItem>
<HelixWorkItem Include="Crossgen System.Linq.Expressions.dll">
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen\test.py crossgen --test-name System.Linq.Expressions.dll --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
</HelixWorkItem>
<HelixWorkItem Include="Crossgen Microsoft.CodeAnalysis.VisualBasic.dll">
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen\test.py crossgen --test-name Microsoft.CodeAnalysis.VisualBasic.dll --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
</HelixWorkItem>
<HelixWorkItem Include="Crossgen Microsoft.CodeAnalysis.CSharp.dll">
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen\test.py crossgen --test-name Microsoft.CodeAnalysis.CSharp.dll --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
</HelixWorkItem>
</ItemGroup>
</Project>
+7 -30
View File
@@ -1,25 +1,22 @@
Param(
[string] $SourceDirectory=$env:BUILD_SOURCESDIRECTORY,
[string] $CoreRootDirectory,
[string] $BaselineCoreRootDirectory,
[string] $Architecture="x64",
[string] $Framework="netcoreapp5.0",
[string] $Framework="netcoreapp3.0",
[string] $CompilationMode="Tiered",
[string] $Repository=$env:BUILD_REPOSITORY_NAME,
[string] $Branch=$env:BUILD_SOURCEBRANCH,
[string] $CommitSha=$env:BUILD_SOURCEVERSION,
[string] $BuildNumber=$env:BUILD_BUILDNUMBER,
[string] $RunCategories="Libraries Runtime",
[string] $RunCategories="coreclr corefx",
[string] $Csproj="src\benchmarks\micro\MicroBenchmarks.csproj",
[string] $Kind="micro",
[switch] $Internal,
[switch] $Compare,
[string] $Configurations="CompilationMode=$CompilationMode RunKind=$Kind"
[string] $Configurations="CompilationMode=$CompilationMode"
)
$RunFromPerformanceRepo = ($Repository -eq "dotnet/performance") -or ($Repository -eq "dotnet-performance")
$RunFromPerformanceRepo = ($Repository -eq "dotnet/performance")
$UseCoreRun = ($CoreRootDirectory -ne [string]::Empty)
$UseBaselineCoreRun = ($BaselineCoreRootDirectory -ne [string]::Empty)
$PayloadDirectory = (Join-Path $SourceDirectory "Payload")
$PerformanceDirectory = (Join-Path $PayloadDirectory "performance")
@@ -32,34 +29,20 @@ $HelixSourcePrefix = "pr"
$Queue = "Windows.10.Amd64.ClientRS4.DevEx.15.8.Open"
if ($Framework.StartsWith("netcoreapp")) {
$Queue = "Windows.10.Amd64.ClientRS5.Open"
}
if ($Compare) {
$Queue = "Windows.10.Amd64.19H1.Tiger.Perf.Open"
$PerfLabArguments = ""
$ExtraBenchmarkDotNetArguments = ""
$Queue = "Windows.10.Amd64.ClientRS4.Open"
}
if ($Internal) {
$Queue = "Windows.10.Amd64.19H1.Tiger.Perf"
$Queue = "Windows.10.Amd64.ClientRS5.Perf"
$PerfLabArguments = "--upload-to-perflab-container"
$ExtraBenchmarkDotNetArguments = ""
$Creator = ""
$HelixSourcePrefix = "official"
}
# FIX ME: This is a workaround until we get this from the actual pipeline
$CommonSetupArguments="--channel master --queue $Queue --build-number $BuildNumber --build-configs $Configurations --architecture $Architecture"
$CommonSetupArguments="--frameworks $Framework --queue $Queue --build-number $BuildNumber --build-configs $Configurations"
$SetupArguments = "--repository https://github.com/$Repository --branch $Branch --get-perf-hash --commit-sha $CommitSha $CommonSetupArguments"
#This grabs the LKG version number of dotnet and passes it to our scripts
$VersionJSON = Get-Content global.json | ConvertFrom-Json
$DotNetVersion = $VersionJSON.tools.dotnet
$SetupArguments = "--dotnet-versions $DotNetVersion $SetupArguments"
if ($RunFromPerformanceRepo) {
$SetupArguments = "--perf-hash $CommitSha $CommonSetupArguments"
@@ -73,10 +56,6 @@ if ($UseCoreRun) {
$NewCoreRoot = (Join-Path $PayloadDirectory "Core_Root")
Move-Item -Path $CoreRootDirectory -Destination $NewCoreRoot
}
if ($UseBaselineCoreRun) {
$NewBaselineCoreRoot = (Join-Path $PayloadDirectory "Baseline_Core_Root")
Move-Item -Path $BaselineCoreRootDirectory -Destination $NewBaselineCoreRoot
}
$DocsDir = (Join-Path $PerformanceDirectory "docs")
robocopy $DocsDir $WorkItemDirectory
@@ -101,9 +80,7 @@ Write-PipelineSetVariable -Name 'TargetCsproj' -Value "$Csproj" -IsMultiJobVaria
Write-PipelineSetVariable -Name 'Kind' -Value "$Kind" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'Architecture' -Value "$Architecture" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'UseCoreRun' -Value "$UseCoreRun" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'UseBaselineCoreRun' -Value "$UseBaselineCoreRun" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'RunFromPerfRepo' -Value "$RunFromPerformanceRepo" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'Compare' -Value "$Compare" -IsMultiJobVariable $false
# Helix Arguments
Write-PipelineSetVariable -Name 'Creator' -Value "$Creator" -IsMultiJobVariable $false
+23 -70
View File
@@ -2,23 +2,20 @@
source_directory=$BUILD_SOURCESDIRECTORY
core_root_directory=
baseline_core_root_directory=
architecture=x64
framework=netcoreapp5.0
framework=netcoreapp3.0
compilation_mode=tiered
repository=$BUILD_REPOSITORY_NAME
branch=$BUILD_SOURCEBRANCH
commit_sha=$BUILD_SOURCEVERSION
build_number=$BUILD_BUILDNUMBER
internal=false
compare=false
kind="micro"
run_categories="Libraries Runtime"
run_categories="coreclr corefx"
csproj="src\benchmarks\micro\MicroBenchmarks.csproj"
configurations="CompliationMode=$compilation_mode RunKind=$kind"
configurations=
run_from_perf_repo=false
use_core_run=true
use_baseline_core_run=true
while (($# > 0)); do
lowerI="$(echo $1 | awk '{print tolower($0)}')"
@@ -31,10 +28,6 @@ while (($# > 0)); do
core_root_directory=$2
shift 2
;;
--baselinecorerootdirectory)
baseline_core_root_directory=$2
shift 2
;;
--architecture)
architecture=$2
shift 2
@@ -79,10 +72,6 @@ while (($# > 0)); do
internal=true
shift 1
;;
--compare)
compare=true
shift 1
;;
--configurations)
configurations=$2
shift 2
@@ -113,7 +102,7 @@ while (($# > 0)); do
esac
done
if [ "$repository" == "dotnet/performance" ] || [ "$repository" == "dotnet-performance" ]; then
if [[ "$repository" == "dotnet/performance" ]]; then
run_from_perf_repo=true
fi
@@ -125,10 +114,6 @@ if [ -z "$core_root_directory" ]; then
use_core_run=false
fi
if [ -z "$baseline_core_root_directory" ]; then
use_baseline_core_run=false
fi
payload_directory=$source_directory/Payload
performance_directory=$payload_directory/performance
workitem_directory=$source_directory/workitem
@@ -138,19 +123,6 @@ queue=Ubuntu.1804.Amd64.Open
creator=$BUILD_DEFINITIONNAME
helix_source_prefix="pr"
if [[ "$compare" == true ]]; then
extra_benchmark_dotnet_arguments=
perflab_arguments=
# No open queues for arm64
if [[ "$architecture" = "arm64" ]]; then
echo "Compare not available for arm64"
exit 1
fi
queue=Ubuntu.1804.Amd64.Tiger.Perf.Open
fi
if [[ "$internal" == true ]]; then
perflab_arguments="--upload-to-perflab-container"
helix_source_prefix="official"
@@ -160,20 +132,13 @@ if [[ "$internal" == true ]]; then
if [[ "$architecture" = "arm64" ]]; then
queue=Ubuntu.1804.Arm64.Perf
else
queue=Ubuntu.1804.Amd64.Tiger.Perf
queue=Ubuntu.1804.Amd64.Perf
fi
fi
common_setup_arguments="--channel master --queue $queue --build-number $build_number --build-configs $configurations --architecture $architecture"
common_setup_arguments="--frameworks $framework --queue $queue --build-number $build_number --build-configs $configurations"
setup_arguments="--repository https://github.com/$repository --branch $branch --get-perf-hash --commit-sha $commit_sha $common_setup_arguments"
# Get the tools section from the global.json.
# This grabs the LKG version number of dotnet and passes it to our scripts
dotnet_version=`cat global.json | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["tools"]["dotnet"])'`
setup_arguments="--dotnet-versions $dotnet_version $setup_arguments"
if [[ "$run_from_perf_repo" = true ]]; then
payload_directory=
workitem_directory=$source_directory
@@ -191,33 +156,21 @@ if [[ "$use_core_run" = true ]]; then
mv $core_root_directory $new_core_root
fi
if [[ "$use_baseline_core_run" = true ]]; then
new_baseline_core_root=$payload_directory/Baseline_Core_Root
mv $baseline_core_root_directory $new_baseline_core_root
fi
ci=true
_script_dir=$(pwd)/eng/common
. "$_script_dir/pipeline-logging-functions.sh"
# Make sure all of our variables are available for future steps
Write-PipelineSetVariable -name "UseCoreRun" -value "$use_core_run" -is_multi_job_variable false
Write-PipelineSetVariable -name "UseBaselineCoreRun" -value "$use_baseline_core_run" -is_multi_job_variable false
Write-PipelineSetVariable -name "Architecture" -value "$architecture" -is_multi_job_variable false
Write-PipelineSetVariable -name "PayloadDirectory" -value "$payload_directory" -is_multi_job_variable false
Write-PipelineSetVariable -name "PerformanceDirectory" -value "$performance_directory" -is_multi_job_variable false
Write-PipelineSetVariable -name "WorkItemDirectory" -value "$workitem_directory" -is_multi_job_variable false
Write-PipelineSetVariable -name "Queue" -value "$queue" -is_multi_job_variable false
Write-PipelineSetVariable -name "SetupArguments" -value "$setup_arguments" -is_multi_job_variable false
Write-PipelineSetVariable -name "Python" -value "$python3" -is_multi_job_variable false
Write-PipelineSetVariable -name "PerfLabArguments" -value "$perflab_arguments" -is_multi_job_variable false
Write-PipelineSetVariable -name "ExtraBenchmarkDotNetArguments" -value "$extra_benchmark_dotnet_arguments" -is_multi_job_variable false
Write-PipelineSetVariable -name "BDNCategories" -value "$run_categories" -is_multi_job_variable false
Write-PipelineSetVariable -name "TargetCsproj" -value "$csproj" -is_multi_job_variable false
Write-PipelineSetVariable -name "RunFromPerfRepo" -value "$run_from_perf_repo" -is_multi_job_variable false
Write-PipelineSetVariable -name "Creator" -value "$creator" -is_multi_job_variable false
Write-PipelineSetVariable -name "HelixSourcePrefix" -value "$helix_source_prefix" -is_multi_job_variable false
Write-PipelineSetVariable -name "Kind" -value "$kind" -is_multi_job_variable false
Write-PipelineSetVariable -name "_BuildConfig" -value "$architecture.$kind.$framework" -is_multi_job_variable false
Write-PipelineSetVariable -name "Compare" -value "$compare" -is_multi_job_variable false
echo "##vso[task.setvariable variable=UseCoreRun]$use_core_run"
echo "##vso[task.setvariable variable=Architecture]$architecture"
echo "##vso[task.setvariable variable=PayloadDirectory]$payload_directory"
echo "##vso[task.setvariable variable=PerformanceDirectory]$performance_directory"
echo "##vso[task.setvariable variable=WorkItemDirectory]$workitem_directory"
echo "##vso[task.setvariable variable=Queue]$queue"
echo "##vso[task.setvariable variable=SetupArguments]$setup_arguments"
echo "##vso[task.setvariable variable=Python]python3"
echo "##vso[task.setvariable variable=PerfLabArguments]$perflab_arguments"
echo "##vso[task.setvariable variable=ExtraBenchmarkDotNetArguments]$extra_benchmark_dotnet_arguments"
echo "##vso[task.setvariable variable=BDNCategories]$run_categories"
echo "##vso[task.setvariable variable=TargetCsproj]$csproj"
echo "##vso[task.setvariable variable=RunFromPerfRepo]$run_from_perf_repo"
echo "##vso[task.setvariable variable=Creator]$creator"
echo "##vso[task.setvariable variable=HelixSourcePrefix]$helix_source_prefix"
echo "##vso[task.setvariable variable=Kind]$kind"
echo "##vso[task.setvariable variable=_BuildConfig]$architecture.$kind.$framework"
+31 -39
View File
@@ -12,7 +12,6 @@ $script:loggingCommandEscapeMappings = @( # TODO: WHAT ABOUT "="? WHAT ABOUT "%"
# TODO: BUG: Escape % ???
# TODO: Add test to verify don't need to escape "=".
# Specify "-Force" to force pipeline formatted output even if "$ci" is false or not set
function Write-PipelineTelemetryError {
[CmdletBinding()]
param(
@@ -26,55 +25,49 @@ function Write-PipelineTelemetryError {
[string]$SourcePath,
[string]$LineNumber,
[string]$ColumnNumber,
[switch]$AsOutput,
[switch]$Force)
[switch]$AsOutput)
$PSBoundParameters.Remove('Category') | Out-Null
$PSBoundParameters.Remove("Category") | Out-Null
$Message = "(NETCORE_ENGINEERING_TELEMETRY=$Category) $Message"
$PSBoundParameters.Remove("Message") | Out-Null
$PSBoundParameters.Add("Message", $Message)
if($Force -Or ((Test-Path variable:ci) -And $ci)) {
$Message = "(NETCORE_ENGINEERING_TELEMETRY=$Category) $Message"
}
$PSBoundParameters.Remove('Message') | Out-Null
$PSBoundParameters.Add('Message', $Message)
Write-PipelineTaskError @PSBoundParameters
}
# Specify "-Force" to force pipeline formatted output even if "$ci" is false or not set
function Write-PipelineTaskError {
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[string]$Message,
[Parameter(Mandatory = $false)]
[string]$Type = 'error',
[string]$ErrCode,
[string]$SourcePath,
[string]$LineNumber,
[string]$ColumnNumber,
[switch]$AsOutput,
[switch]$Force
)
if(!$Force -And (-Not (Test-Path variable:ci) -Or !$ci)) {
[Parameter(Mandatory = $true)]
[string]$Message,
[Parameter(Mandatory = $false)]
[string]$Type = 'error',
[string]$ErrCode,
[string]$SourcePath,
[string]$LineNumber,
[string]$ColumnNumber,
[switch]$AsOutput)
if(!$ci) {
if($Type -eq 'error') {
Write-Host $Message -ForegroundColor Red
return
Write-Host $Message -ForegroundColor Red
return
}
elseif ($Type -eq 'warning') {
Write-Host $Message -ForegroundColor Yellow
return
Write-Host $Message -ForegroundColor Yellow
return
}
}
if(($Type -ne 'error') -and ($Type -ne 'warning')) {
}
if(($Type -ne 'error') -and ($Type -ne 'warning')) {
Write-Host $Message
return
}
$PSBoundParameters.Remove('Force') | Out-Null
if(-not $PSBoundParameters.ContainsKey('Type')) {
}
if(-not $PSBoundParameters.ContainsKey('Type')) {
$PSBoundParameters.Add('Type', 'error')
}
Write-LogIssue @PSBoundParameters
}
Write-LogIssue @PSBoundParameters
}
function Write-PipelineSetVariable {
@@ -87,7 +80,7 @@ function Write-PipelineTaskError {
[switch]$AsOutput,
[bool]$IsMultiJobVariable=$true)
if((Test-Path variable:ci) -And $ci) {
if($ci) {
Write-LoggingCommand -Area 'task' -Event 'setvariable' -Data $Value -Properties @{
'variable' = $Name
'isSecret' = $Secret
@@ -102,8 +95,7 @@ function Write-PipelineTaskError {
[Parameter(Mandatory=$true)]
[string]$Path,
[switch]$AsOutput)
if((Test-Path variable:ci) -And $ci) {
if($ci) {
Write-LoggingCommand -Area 'task' -Event 'prependpath' -Data $Path -AsOutput:$AsOutput
}
}
@@ -239,4 +231,4 @@ function Write-LogIssue {
}
Write-Host $command -ForegroundColor $foregroundColor -BackgroundColor $backgroundColor
}
}
+2 -9
View File
@@ -2,7 +2,6 @@
function Write-PipelineTelemetryError {
local telemetry_category=''
local force=false
local function_args=()
local message=''
while [[ $# -gt 0 ]]; do
@@ -12,9 +11,6 @@ function Write-PipelineTelemetryError {
telemetry_category=$2
shift
;;
-force|-f)
force=true
;;
-*)
function_args+=("$1 $2")
shift
@@ -26,22 +22,19 @@ function Write-PipelineTelemetryError {
shift
done
if [[ $force != true ]] && [[ "$ci" != true ]]; then
if [[ "$ci" != true ]]; then
echo "$message" >&2
return
fi
message="(NETCORE_ENGINEERING_TELEMETRY=$telemetry_category) $message"
function_args+=("$message")
if [[ $force == true ]]; then
function_args+=("-force")
fi
Write-PipelineTaskError $function_args
}
function Write-PipelineTaskError {
if [[ $force != true ]] && [[ "$ci" != true ]]; then
if [[ "$ci" != true ]]; then
echo "$@" >&2
return
fi
@@ -1,25 +0,0 @@
param(
[Parameter(Mandatory=$true)][string] $PromoteToChannels, # List of channels that the build should be promoted to
[Parameter(Mandatory=$true)][array] $AvailableChannelIds # List of channel IDs available in the YAML implementation
)
try {
. $PSScriptRoot\post-build-utils.ps1
# Check that every channel that Maestro told to promote the build to
# is available in YAML
$PromoteToChannelsIds = $PromoteToChannels -split "\D" | Where-Object { $_ }
foreach ($id in $PromoteToChannelsIds) {
if (($id -ne 0) -and ($id -notin $AvailableChannelIds)) {
Write-PipelineTaskError -Type 'warning' -Message "Channel $id is not present in the post-build YAML configuration!"
}
}
Write-Host 'done.'
}
catch {
Write-Host $_
Write-PipelineTelemetryError -Category 'CheckChannelConsistency' -Message "There was an error while trying to check consistency of Maestro default channels for the build and post-build YAML configuration."
ExitWithExitCode 1
}
@@ -0,0 +1,35 @@
param(
[Parameter(Mandatory=$true)][int] $BarBuildId, # ID of the build which assets should be downloaded
[Parameter(Mandatory=$true)][string] $DropLocation, # Where the assets should be downloaded to
[Parameter(Mandatory=$true)][string] $MaestroApiAccessToken, # Token used to access Maestro API
[Parameter(Mandatory=$false)][string] $MaestroApiEndPoint = "https://maestro-prod.westus2.cloudapp.azure.com", # Maestro API URL
[Parameter(Mandatory=$false)][string] $MaestroApiVersion = "2019-01-16" # Version of Maestro API to use
)
. $PSScriptRoot\post-build-utils.ps1
try {
Write-Host "Installing DARC ..."
. $PSScriptRoot\..\darc-init.ps1
$exitCode = $LASTEXITCODE
if ($exitCode -ne 0) {
Write-PipelineTaskError "Something failed while running 'darc-init.ps1'. Check for errors above. Exiting now..."
ExitWithExitCode $exitCode
}
darc gather-drop --non-shipping `
--continue-on-error `
--id $BarBuildId `
--output-dir $DropLocation `
--bar-uri $MaestroApiEndpoint `
--password $MaestroApiAccessToken `
--latest-location
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
ExitWithExitCode 1
}
+6 -5
View File
@@ -6,19 +6,20 @@ param(
[Parameter(Mandatory=$true)][string] $ToolDestinationPath # Where the validation tool should be downloaded to
)
. $PSScriptRoot\post-build-utils.ps1
try {
. $PSScriptRoot\post-build-utils.ps1
$url = "https://raw.githubusercontent.com/NuGet/NuGetGallery/jver-verify/src/VerifyMicrosoftPackage/verify.ps1"
$url = 'https://raw.githubusercontent.com/NuGet/NuGetGallery/3e25ad135146676bcab0050a516939d9958bfa5d/src/VerifyMicrosoftPackage/verify.ps1'
New-Item -ItemType 'directory' -Path ${ToolDestinationPath} -Force
New-Item -ItemType "directory" -Path ${ToolDestinationPath} -Force
Invoke-WebRequest $url -OutFile ${ToolDestinationPath}\verify.ps1
& ${ToolDestinationPath}\verify.ps1 ${PackagesPath}\*.nupkg
}
catch {
Write-PipelineTaskError "NuGet package validation failed. Please check error logs."
Write-Host $_
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'NuGetValidation' -Message $_
ExitWithExitCode 1
}
+15 -16
View File
@@ -1,17 +1,16 @@
# Most of the functions in this file require the variables `MaestroApiEndPoint`,
# `MaestroApiVersion` and `MaestroApiAccessToken` to be globally available.
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
# `tools.ps1` checks $ci to perform some actions. Since the post-build
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
$disableConfigureToolsetImport = $true
. $PSScriptRoot\..\tools.ps1
function Create-MaestroApiRequestHeaders([string]$ContentType = 'application/json') {
function Create-MaestroApiRequestHeaders([string]$ContentType = "application/json") {
Validate-MaestroVars
$headers = New-Object 'System.Collections.Generic.Dictionary[[String],[String]]'
@@ -51,14 +50,6 @@ function Get-MaestroSubscriptions([string]$SourceRepository, [int]$ChannelId) {
return $result
}
function Assign-BuildToChannel([int]$BuildId, [int]$ChannelId) {
Validate-MaestroVars
$apiHeaders = Create-MaestroApiRequestHeaders -AuthToken $MaestroApiAccessToken
$apiEndpoint = "$MaestroApiEndPoint/api/channels/${ChannelId}/builds/${BuildId}?api-version=$MaestroApiVersion"
Invoke-WebRequest -Method Post -Uri $apiEndpoint -Headers $apiHeaders | Out-Null
}
function Trigger-Subscription([string]$SubscriptionId) {
Validate-MaestroVars
@@ -67,24 +58,32 @@ function Trigger-Subscription([string]$SubscriptionId) {
Invoke-WebRequest -Uri $apiEndpoint -Headers $apiHeaders -Method Post | Out-Null
}
function Assign-BuildToChannel([int]$BuildId, [int]$ChannelId) {
Validate-MaestroVars
$apiHeaders = Create-MaestroApiRequestHeaders -AuthToken $MaestroApiAccessToken
$apiEndpoint = "$MaestroApiEndPoint/api/channels/${ChannelId}/builds/${BuildId}?api-version=$MaestroApiVersion"
Invoke-WebRequest -Method Post -Uri $apiEndpoint -Headers $apiHeaders | Out-Null
}
function Validate-MaestroVars {
try {
Get-Variable MaestroApiEndPoint -Scope Global | Out-Null
Get-Variable MaestroApiVersion -Scope Global | Out-Null
Get-Variable MaestroApiAccessToken -Scope Global | Out-Null
if (!($MaestroApiEndPoint -Match '^http[s]?://maestro-(int|prod).westus2.cloudapp.azure.com$')) {
Write-PipelineTelemetryError -Category 'MaestroVars' -Message "MaestroApiEndPoint is not a valid Maestro URL. '$MaestroApiEndPoint'"
if (!($MaestroApiEndPoint -Match "^http[s]?://maestro-(int|prod).westus2.cloudapp.azure.com$")) {
Write-PipelineTaskError "MaestroApiEndPoint is not a valid Maestro URL. '$MaestroApiEndPoint'"
ExitWithExitCode 1
}
if (!($MaestroApiVersion -Match '^[0-9]{4}-[0-9]{2}-[0-9]{2}$')) {
Write-PipelineTelemetryError -Category 'MaestroVars' -Message "MaestroApiVersion does not match a version string in the format yyyy-MM-DD. '$MaestroApiVersion'"
if (!($MaestroApiVersion -Match "^[0-9]{4}-[0-9]{2}-[0-9]{2}$")) {
Write-PipelineTaskError "MaestroApiVersion does not match a version string in the format yyyy-MM-DD. '$MaestroApiVersion'"
ExitWithExitCode 1
}
}
catch {
Write-PipelineTelemetryError -Category 'MaestroVars' -Message 'Error: Variables `MaestroApiEndPoint`, `MaestroApiVersion` and `MaestroApiAccessToken` are required while using this script.'
Write-PipelineTaskError "Error: Variables `MaestroApiEndPoint`, `MaestroApiVersion` and `MaestroApiAccessToken` are required while using this script."
Write-Host $_
ExitWithExitCode 1
}
@@ -2,26 +2,26 @@ param(
[Parameter(Mandatory=$true)][int] $BuildId,
[Parameter(Mandatory=$true)][int] $ChannelId,
[Parameter(Mandatory=$true)][string] $MaestroApiAccessToken,
[Parameter(Mandatory=$false)][string] $MaestroApiEndPoint = 'https://maestro-prod.westus2.cloudapp.azure.com',
[Parameter(Mandatory=$false)][string] $MaestroApiVersion = '2019-01-16'
[Parameter(Mandatory=$false)][string] $MaestroApiEndPoint = "https://maestro-prod.westus2.cloudapp.azure.com",
[Parameter(Mandatory=$false)][string] $MaestroApiVersion = "2019-01-16"
)
try {
. $PSScriptRoot\post-build-utils.ps1
. $PSScriptRoot\post-build-utils.ps1
try {
# Check that the channel we are going to promote the build to exist
$channelInfo = Get-MaestroChannel -ChannelId $ChannelId
if (!$channelInfo) {
Write-PipelineTelemetryCategory -Category 'PromoteBuild' -Message "Channel with BAR ID $ChannelId was not found in BAR!"
Write-Host "Channel with BAR ID $ChannelId was not found in BAR!"
ExitWithExitCode 1
}
# Get info about which channel(s) the build has already been promoted to
# Get info about which channels the build has already been promoted to
$buildInfo = Get-MaestroBuild -BuildId $BuildId
if (!$buildInfo) {
Write-PipelineTelemetryError -Category 'PromoteBuild' -Message "Build with BAR ID $BuildId was not found in BAR!"
Write-Host "Build with BAR ID $BuildId was not found in BAR!"
ExitWithExitCode 1
}
@@ -39,10 +39,10 @@ try {
Assign-BuildToChannel -BuildId $BuildId -ChannelId $ChannelId
Write-Host 'done.'
Write-Host "done."
}
catch {
Write-Host "There was an error while trying to promote build '$BuildId' to channel '$ChannelId'"
Write-Host $_
Write-PipelineTelemetryError -Category 'PromoteBuild' -Message "There was an error while trying to promote build '$BuildId' to channel '$ChannelId'"
ExitWithExitCode 1
Write-Host $_.ScriptStackTrace
}
@@ -0,0 +1,26 @@
param(
[Parameter(Mandatory=$true)][string] $ReleaseConfigsPath # Full path to ReleaseConfigs.txt asset
)
. $PSScriptRoot\post-build-utils.ps1
try {
$Content = Get-Content $ReleaseConfigsPath
$BarId = $Content | Select -Index 0
$Channels = ""
$Content | Select -Index 1 | ForEach-Object { $Channels += "$_ ," }
$IsStableBuild = $Content | Select -Index 2
Write-PipelineSetVariable -Name 'BARBuildId' -Value $BarId
Write-PipelineSetVariable -Name 'InitialChannels' -Value "$Channels"
Write-PipelineSetVariable -Name 'IsStableBuild' -Value $IsStableBuild
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
ExitWithExitCode 1
}
+36 -66
View File
@@ -1,8 +1,8 @@
param(
[Parameter(Mandatory=$true)][string] $InputPath, # Full path to directory where Symbols.NuGet packages to be checked are stored
[Parameter(Mandatory=$true)][string] $ExtractPath, # Full path to directory where the packages will be extracted during validation
[Parameter(Mandatory=$false)][string] $GHRepoName, # GitHub name of the repo including the Org. E.g., dotnet/arcade
[Parameter(Mandatory=$false)][string] $GHCommit, # GitHub commit SHA used to build the packages
[Parameter(Mandatory=$true)][string] $GHRepoName, # GitHub name of the repo including the Org. E.g., dotnet/arcade
[Parameter(Mandatory=$true)][string] $GHCommit, # GitHub commit SHA used to build the packages
[Parameter(Mandatory=$true)][string] $SourcelinkCliVersion # Version of SourceLink CLI to use
)
@@ -13,12 +13,6 @@ param(
# all files present in the repo at a specific commit point.
$global:RepoFiles = @{}
# Maximum number of jobs to run in parallel
$MaxParallelJobs = 6
# Wait time between check for system load
$SecondsBetweenLoadChecks = 10
$ValidatePackage = {
param(
[string] $PackagePath # Full path to a Symbols.NuGet package
@@ -28,15 +22,15 @@ $ValidatePackage = {
# Ensure input file exist
if (!(Test-Path $PackagePath)) {
Write-Host "Input file does not exist: $PackagePath"
return 1
Write-PipelineTaskError "Input file does not exist: $PackagePath"
ExitWithExitCode 1
}
# Extensions for which we'll look for SourceLink information
# For now we'll only care about Portable & Embedded PDBs
$RelevantExtensions = @('.dll', '.exe', '.pdb')
$RelevantExtensions = @(".dll", ".exe", ".pdb")
Write-Host -NoNewLine 'Validating ' ([System.IO.Path]::GetFileName($PackagePath)) '...'
Write-Host -NoNewLine "Validating" ([System.IO.Path]::GetFileName($PackagePath)) "... "
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$ExtractPath = Join-Path -Path $using:ExtractPath -ChildPath $PackageId
@@ -44,7 +38,7 @@ $ValidatePackage = {
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Directory]::CreateDirectory($ExtractPath) | Out-Null
[System.IO.Directory]::CreateDirectory($ExtractPath);
try {
$zip = [System.IO.Compression.ZipFile]::OpenRead($PackagePath)
@@ -58,7 +52,7 @@ $ValidatePackage = {
$TargetFile = Join-Path -Path $ExtractPath -ChildPath $FakeName
# We ignore resource DLLs
if ($FileName.EndsWith('.resources.dll')) {
if ($FileName.EndsWith(".resources.dll")) {
return
}
@@ -96,7 +90,7 @@ $ValidatePackage = {
$Uri = $Link -as [System.URI]
# Only GitHub links are valid
if ($Uri.AbsoluteURI -ne $null -and ($Uri.Host -match 'github' -or $Uri.Host -match 'githubusercontent')) {
if ($Uri.AbsoluteURI -ne $null -and ($Uri.Host -match "github" -or $Uri.Host -match "githubusercontent")) {
$Status = (Invoke-WebRequest -Uri $Link -UseBasicParsing -Method HEAD -TimeoutSec 5).StatusCode
}
else {
@@ -143,19 +137,17 @@ $ValidatePackage = {
}
if ($FailedFiles -eq 0) {
Write-Host 'Passed.'
return 0
Write-Host "Passed."
}
else {
Write-PipelineTelemetryError -Category 'SourceLink' -Message "$PackagePath has broken SourceLink links."
return 1
Write-PipelineTaskError "$PackagePath has broken SourceLink links."
}
}
function ValidateSourceLinkLinks {
if ($GHRepoName -ne '' -and !($GHRepoName -Match '^[^\s\/]+/[^\s\/]+$')) {
if (!($GHRepoName -Match '^[^\s-]+-[^\s]+$')) {
Write-PipelineTelemetryError -Category 'SourceLink' -Message "GHRepoName should be in the format <org>/<repo> or <org>-<repo>. '$GHRepoName'"
if (!($GHRepoName -Match "^[^\s\/]+/[^\s\/]+$")) {
if (!($GHRepoName -Match "^[^\s-]+-[^\s]+$")) {
Write-PipelineTaskError "GHRepoName should be in the format <org>/<repo> or <org>-<repo>"
ExitWithExitCode 1
}
else {
@@ -163,33 +155,30 @@ function ValidateSourceLinkLinks {
}
}
if ($GHCommit -ne '' -and !($GHCommit -Match '^[0-9a-fA-F]{40}$')) {
Write-PipelineTelemetryError -Category 'SourceLink' -Message "GHCommit should be a 40 chars hexadecimal string. '$GHCommit'"
if (!($GHCommit -Match "^[0-9a-fA-F]{40}$")) {
Write-PipelineTaskError "GHCommit should be a 40 chars hexadecimal string"
ExitWithExitCode 1
}
if ($GHRepoName -ne '' -and $GHCommit -ne '') {
$RepoTreeURL = -Join('http://api.github.com/repos/', $GHRepoName, '/git/trees/', $GHCommit, '?recursive=1')
$CodeExtensions = @('.cs', '.vb', '.fs', '.fsi', '.fsx', '.fsscript')
$RepoTreeURL = -Join("http://api.github.com/repos/", $GHRepoName, "/git/trees/", $GHCommit, "?recursive=1")
$CodeExtensions = @(".cs", ".vb", ".fs", ".fsi", ".fsx", ".fsscript")
try {
# Retrieve the list of files in the repo at that particular commit point and store them in the RepoFiles hash
$Data = Invoke-WebRequest $RepoTreeURL -UseBasicParsing | ConvertFrom-Json | Select-Object -ExpandProperty tree
try {
# Retrieve the list of files in the repo at that particular commit point and store them in the RepoFiles hash
$Data = Invoke-WebRequest $RepoTreeURL -UseBasicParsing | ConvertFrom-Json | Select-Object -ExpandProperty tree
foreach ($file in $Data) {
$Extension = [System.IO.Path]::GetExtension($file.path)
foreach ($file in $Data) {
$Extension = [System.IO.Path]::GetExtension($file.path)
if ($CodeExtensions.Contains($Extension)) {
$RepoFiles[$file.path] = 1
}
if ($CodeExtensions.Contains($Extension)) {
$RepoFiles[$file.path] = 1
}
}
catch {
Write-Host "Problems downloading the list of files from the repo. Url used: $RepoTreeURL . Execution will proceed without caching."
}
}
elseif ($GHRepoName -ne '' -or $GHCommit -ne '') {
Write-Host 'For using the http caching mechanism both GHRepoName and GHCommit should be informed.'
catch {
Write-PipelineTaskError "Problems downloading the list of files from the repo. Url used: $RepoTreeURL"
Write-Host $_
ExitWithExitCode 1
}
if (Test-Path $ExtractPath) {
@@ -197,38 +186,19 @@ function ValidateSourceLinkLinks {
}
# Process each NuGet package in parallel
$Jobs = @()
Get-ChildItem "$InputPath\*.symbols.nupkg" |
ForEach-Object {
Start-Job -ScriptBlock $ValidatePackage -ArgumentList $_.FullName | Out-Null
$NumJobs = @(Get-Job -State 'Running').Count
while ($NumJobs -ge $MaxParallelJobs) {
Write-Host "There are $NumJobs validation jobs running right now. Waiting $SecondsBetweenLoadChecks seconds to check again."
sleep $SecondsBetweenLoadChecks
$NumJobs = @(Get-Job -State 'Running').Count
}
foreach ($Job in @(Get-Job -State 'Completed')) {
Receive-Job -Id $Job.Id
Remove-Job -Id $Job.Id
}
$Jobs += Start-Job -ScriptBlock $ValidatePackage -ArgumentList $_.FullName
}
$ValidationFailures = 0
foreach ($Job in @(Get-Job)) {
$jobResult = Wait-Job -Id $Job.Id | Receive-Job
if ($jobResult -ne '0') {
$ValidationFailures++
}
}
if ($ValidationFailures -gt 0) {
Write-PipelineTelemetryError -Category 'SourceLink' -Message "$ValidationFailures package(s) failed validation."
ExitWithExitCode 1
foreach ($Job in $Jobs) {
Wait-Job -Id $Job.Id | Receive-Job
}
}
function InstallSourcelinkCli {
$sourcelinkCliPackageName = 'sourcelink'
$sourcelinkCliPackageName = "sourcelink"
$dotnetRoot = InitializeDotNetCli -install:$true
$dotnet = "$dotnetRoot\dotnet.exe"
@@ -239,7 +209,7 @@ function InstallSourcelinkCli {
}
else {
Write-Host "Installing SourceLink CLI version $sourcelinkCliVersion..."
Write-Host 'You may need to restart your command window if this is the first dotnet tool you have installed.'
Write-Host "You may need to restart your command window if this is the first dotnet tool you have installed."
& "$dotnet" tool install $sourcelinkCliPackageName --version $sourcelinkCliVersion --verbosity "minimal" --global
}
}
@@ -250,8 +220,8 @@ try {
ValidateSourceLinkLinks
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'SourceLink' -Message $_
ExitWithExitCode 1
}
+45 -44
View File
@@ -4,6 +4,10 @@ param(
[Parameter(Mandatory=$true)][string] $DotnetSymbolVersion # Version of dotnet symbol to use
)
. $PSScriptRoot\post-build-utils.ps1
Add-Type -AssemblyName System.IO.Compression.FileSystem
function FirstMatchingSymbolDescriptionOrDefault {
param(
[string] $FullPath, # Full path to the module that has to be checked
@@ -19,39 +23,39 @@ function FirstMatchingSymbolDescriptionOrDefault {
# checking and which type of file was uploaded.
# The file itself is returned
$SymbolPath = $SymbolsPath + '\' + $FileName
$SymbolPath = $SymbolsPath + "\" + $FileName
# PDB file for the module
$PdbPath = $SymbolPath.Replace($Extension, '.pdb')
$PdbPath = $SymbolPath.Replace($Extension, ".pdb")
# PDB file for R2R module (created by crossgen)
$NGenPdb = $SymbolPath.Replace($Extension, '.ni.pdb')
$NGenPdb = $SymbolPath.Replace($Extension, ".ni.pdb")
# DBG file for a .so library
$SODbg = $SymbolPath.Replace($Extension, '.so.dbg')
$SODbg = $SymbolPath.Replace($Extension, ".so.dbg")
# DWARF file for a .dylib
$DylibDwarf = $SymbolPath.Replace($Extension, '.dylib.dwarf')
$DylibDwarf = $SymbolPath.Replace($Extension, ".dylib.dwarf")
$dotnetSymbolExe = "$env:USERPROFILE\.dotnet\tools"
$dotnetSymbolExe = Resolve-Path "$dotnetSymbolExe\dotnet-symbol.exe"
$dotnetsymbolExe = "$env:USERPROFILE\.dotnet\tools"
$dotnetsymbolExe = Resolve-Path "$dotnetsymbolExe\dotnet-symbol.exe"
& $dotnetSymbolExe --symbols --modules --windows-pdbs $TargetServerParam $FullPath -o $SymbolsPath | Out-Null
& $dotnetsymbolExe --symbols --modules --windows-pdbs $TargetServerParam $FullPath -o $SymbolsPath | Out-Null
if (Test-Path $PdbPath) {
return 'PDB'
return "PDB"
}
elseif (Test-Path $NGenPdb) {
return 'NGen PDB'
return "NGen PDB"
}
elseif (Test-Path $SODbg) {
return 'DBG for SO'
return "DBG for SO"
}
elseif (Test-Path $DylibDwarf) {
return 'Dwarf for Dylib'
return "Dwarf for Dylib"
}
elseif (Test-Path $SymbolPath) {
return 'Module'
return "Module"
}
else {
return $null
@@ -70,7 +74,7 @@ function CountMissingSymbols {
}
# Extensions for which we'll look for symbols
$RelevantExtensions = @('.dll', '.exe', '.so', '.dylib')
$RelevantExtensions = @(".dll", ".exe", ".so", ".dylib")
# How many files are missing symbol information
$MissingSymbols = 0
@@ -78,38 +82,38 @@ function CountMissingSymbols {
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$PackageGuid = New-Guid
$ExtractPath = Join-Path -Path $ExtractPath -ChildPath $PackageGuid
$SymbolsPath = Join-Path -Path $ExtractPath -ChildPath 'Symbols'
$SymbolsPath = Join-Path -Path $ExtractPath -ChildPath "Symbols"
[System.IO.Compression.ZipFile]::ExtractToDirectory($PackagePath, $ExtractPath)
Get-ChildItem -Recurse $ExtractPath |
Where-Object {$RelevantExtensions -contains $_.Extension} |
ForEach-Object {
if ($_.FullName -Match '\\ref\\') {
Write-Host "`t Ignoring reference assembly file " $_.FullName
if ($_.FullName -Match "\\ref\\") {
Write-Host "`t Ignoring reference assembly file" $_.FullName
return
}
$SymbolsOnMSDL = FirstMatchingSymbolDescriptionOrDefault $_.FullName '--microsoft-symbol-server' $SymbolsPath
$SymbolsOnSymWeb = FirstMatchingSymbolDescriptionOrDefault $_.FullName '--internal-server' $SymbolsPath
$SymbolsOnMSDL = FirstMatchingSymbolDescriptionOrDefault $_.FullName "--microsoft-symbol-server" $SymbolsPath
$SymbolsOnSymWeb = FirstMatchingSymbolDescriptionOrDefault $_.FullName "--internal-server" $SymbolsPath
Write-Host -NoNewLine "`t Checking file " $_.FullName "... "
Write-Host -NoNewLine "`t Checking file" $_.FullName "... "
if ($SymbolsOnMSDL -ne $null -and $SymbolsOnSymWeb -ne $null) {
Write-Host "Symbols found on MSDL ($SymbolsOnMSDL) and SymWeb ($SymbolsOnSymWeb)"
Write-Host "Symbols found on MSDL (" $SymbolsOnMSDL ") and SymWeb (" $SymbolsOnSymWeb ")"
}
else {
$MissingSymbols++
if ($SymbolsOnMSDL -eq $null -and $SymbolsOnSymWeb -eq $null) {
Write-Host 'No symbols found on MSDL or SymWeb!'
Write-Host "No symbols found on MSDL or SymWeb!"
}
else {
if ($SymbolsOnMSDL -eq $null) {
Write-Host 'No symbols found on MSDL!'
Write-Host "No symbols found on MSDL!"
}
else {
Write-Host 'No symbols found on SymWeb!'
Write-Host "No symbols found on SymWeb!"
}
}
}
@@ -128,61 +132,58 @@ function CheckSymbolsAvailable {
Get-ChildItem "$InputPath\*.nupkg" |
ForEach-Object {
$FileName = $_.Name
# These packages from Arcade-Services include some native libraries that
# our current symbol uploader can't handle. Below is a workaround until
# we get issue: https://github.com/dotnet/arcade/issues/2457 sorted.
if ($FileName -Match 'Microsoft\.DotNet\.Darc\.') {
if ($FileName -Match "Microsoft\.DotNet\.Darc\.") {
Write-Host "Ignoring Arcade-services file: $FileName"
Write-Host
return
}
elseif ($FileName -Match 'Microsoft\.DotNet\.Maestro\.Tasks\.') {
elseif ($FileName -Match "Microsoft\.DotNet\.Maestro\.Tasks\.") {
Write-Host "Ignoring Arcade-services file: $FileName"
Write-Host
return
}
Write-Host "Validating $FileName "
$Status = CountMissingSymbols "$InputPath\$FileName"
if ($Status -ne 0) {
Write-PipelineTelemetryError -Category 'CheckSymbols' -Message "Missing symbols for $Status modules in the package $FileName"
ExitWithExitCode $exitCode
Write-PipelineTaskError "Missing symbols for $Status modules in the package $FileName"
ExitWithExitCode $exitCode
}
Write-Host
}
}
function InstallDotnetSymbol {
$dotnetSymbolPackageName = 'dotnet-symbol'
function Installdotnetsymbol {
$dotnetsymbolPackageName = "dotnet-symbol"
$dotnetRoot = InitializeDotNetCli -install:$true
$dotnet = "$dotnetRoot\dotnet.exe"
$toolList = & "$dotnet" tool list --global
if (($toolList -like "*$dotnetSymbolPackageName*") -and ($toolList -like "*$dotnetSymbolVersion*")) {
Write-Host "dotnet-symbol version $dotnetSymbolVersion is already installed."
if (($toolList -like "*$dotnetsymbolPackageName*") -and ($toolList -like "*$dotnetsymbolVersion*")) {
Write-Host "dotnet-symbol version $dotnetsymbolVersion is already installed."
}
else {
Write-Host "Installing dotnet-symbol version $dotnetSymbolVersion..."
Write-Host 'You may need to restart your command window if this is the first dotnet tool you have installed.'
& "$dotnet" tool install $dotnetSymbolPackageName --version $dotnetSymbolVersion --verbosity "minimal" --global
Write-Host "Installing dotnet-symbol version $dotnetsymbolVersion..."
Write-Host "You may need to restart your command window if this is the first dotnet tool you have installed."
& "$dotnet" tool install $dotnetsymbolPackageName --version $dotnetsymbolVersion --verbosity "minimal" --global
}
}
try {
. $PSScriptRoot\post-build-utils.ps1
Add-Type -AssemblyName System.IO.Compression.FileSystem
InstallDotnetSymbol
Installdotnetsymbol
CheckSymbolsAvailable
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'CheckSymbols' -Message $_
ExitWithExitCode 1
}
+43 -50
View File
@@ -2,63 +2,56 @@ param(
[Parameter(Mandatory=$true)][string] $SourceRepo,
[Parameter(Mandatory=$true)][int] $ChannelId,
[Parameter(Mandatory=$true)][string] $MaestroApiAccessToken,
[Parameter(Mandatory=$false)][string] $MaestroApiEndPoint = 'https://maestro-prod.westus2.cloudapp.azure.com',
[Parameter(Mandatory=$false)][string] $MaestroApiVersion = '2019-01-16'
[Parameter(Mandatory=$false)][string] $MaestroApiEndPoint = "https://maestro-prod.westus2.cloudapp.azure.com",
[Parameter(Mandatory=$false)][string] $MaestroApiVersion = "2019-01-16"
)
try {
. $PSScriptRoot\post-build-utils.ps1
. $PSScriptRoot\post-build-utils.ps1
# Get all the $SourceRepo subscriptions
$normalizedSourceRepo = $SourceRepo.Replace('dnceng@', '')
$subscriptions = Get-MaestroSubscriptions -SourceRepository $normalizedSourceRepo -ChannelId $ChannelId
# Get all the $SourceRepo subscriptions
$normalizedSourceRepo = $SourceRepo.Replace('dnceng@', '')
$subscriptions = Get-MaestroSubscriptions -SourceRepository $normalizedSourceRepo -ChannelId $ChannelId
if (!$subscriptions) {
Write-PipelineTelemetryError -Category 'TriggerSubscriptions' -Message "No subscriptions found for source repo '$normalizedSourceRepo' in channel '$ChannelId'"
ExitWithExitCode 0
}
if (!$subscriptions) {
Write-Host "No subscriptions found for source repo '$normalizedSourceRepo' in channel '$ChannelId'"
ExitWithExitCode 0
}
$subscriptionsToTrigger = New-Object System.Collections.Generic.List[string]
$failedTriggeredSubscription = $false
$subscriptionsToTrigger = New-Object System.Collections.Generic.List[string]
$failedTriggeredSubscription = $false
# Get all enabled subscriptions that need dependency flow on 'everyBuild'
foreach ($subscription in $subscriptions) {
if ($subscription.enabled -and $subscription.policy.updateFrequency -like 'everyBuild' -and $subscription.channel.id -eq $ChannelId) {
Write-Host "Should trigger this subscription: ${$subscription.id}"
[void]$subscriptionsToTrigger.Add($subscription.id)
}
}
foreach ($subscriptionToTrigger in $subscriptionsToTrigger) {
try {
Write-Host "Triggering subscription '$subscriptionToTrigger'."
Trigger-Subscription -SubscriptionId $subscriptionToTrigger
Write-Host 'done.'
}
catch
{
Write-Host "There was an error while triggering subscription '$subscriptionToTrigger'"
Write-Host $_
Write-Host $_.ScriptStackTrace
$failedTriggeredSubscription = $true
}
}
if ($subscriptionsToTrigger.Count -eq 0) {
Write-Host "No subscription matched source repo '$normalizedSourceRepo' and channel ID '$ChannelId'."
}
elseif ($failedTriggeredSubscription) {
Write-PipelineTelemetryError -Category 'TriggerSubscriptions' -Message 'At least one subscription failed to be triggered...'
ExitWithExitCode 1
}
else {
Write-Host 'All subscriptions were triggered successfully!'
# Get all enabled subscriptions that need dependency flow on 'everyBuild'
foreach ($subscription in $subscriptions) {
if ($subscription.enabled -and $subscription.policy.updateFrequency -like 'everyBuild' -and $subscription.channel.id -eq $ChannelId) {
Write-Host "Should trigger this subscription: $subscription.id"
[void]$subscriptionsToTrigger.Add($subscription.id)
}
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'TriggerSubscriptions' -Message $_
foreach ($subscriptionToTrigger in $subscriptionsToTrigger) {
try {
Write-Host "Triggering subscription '$subscriptionToTrigger'."
Trigger-Subscription -SubscriptionId $subscriptionToTrigger
Write-Host "done."
}
catch
{
Write-Host "There was an error while triggering subscription '$subscriptionToTrigger'"
Write-Host $_
Write-Host $_.ScriptStackTrace
$failedTriggeredSubscription = $true
}
}
if ($subscriptionsToTrigger.Count -eq 0) {
Write-Host "No subscription matched source repo '$normalizedSourceRepo' and channel ID '$ChannelId'."
}
elseif ($failedTriggeredSubscription) {
Write-Host "At least one subscription failed to be triggered..."
ExitWithExitCode 1
}
else {
Write-Host "All subscriptions were triggered successfully!"
}
+10 -9
View File
@@ -1,8 +1,8 @@
[CmdletBinding(PositionalBinding=$false)]
Param(
[string] $configuration = 'Debug',
[string] $configuration = "Debug",
[string] $task,
[string] $verbosity = 'minimal',
[string] $verbosity = "minimal",
[string] $msbuildEngine = $null,
[switch] $restore,
[switch] $prepareMachine,
@@ -32,7 +32,7 @@ function Print-Usage() {
}
function Build([string]$target) {
$logSuffix = if ($target -eq 'Execute') { '' } else { ".$target" }
$logSuffix = if ($target -eq "Execute") { "" } else { ".$target" }
$log = Join-Path $LogDir "$task$logSuffix.binlog"
$outputPath = Join-Path $ToolsetDir "$task\\"
@@ -46,32 +46,33 @@ function Build([string]$target) {
}
try {
if ($help -or (($null -ne $properties) -and ($properties.Contains('/help') -or $properties.Contains('/?')))) {
if ($help -or (($null -ne $properties) -and ($properties.Contains("/help") -or $properties.Contains("/?")))) {
Print-Usage
exit 0
}
if ($task -eq "") {
Write-PipelineTelemetryError -Category 'Build' -Message "Missing required parameter '-task <value>'" -ForegroundColor Red
Write-Host "Missing required parameter '-task <value>'" -ForegroundColor Red
Print-Usage
ExitWithExitCode 1
}
$taskProject = GetSdkTaskProject $task
if (!(Test-Path $taskProject)) {
Write-PipelineTelemetryError -Category 'Build' -Message "Unknown task: $task" -ForegroundColor Red
Write-Host "Unknown task: $task" -ForegroundColor Red
ExitWithExitCode 1
}
if ($restore) {
Build 'Restore'
Build "Restore"
}
Build 'Execute'
Build "Execute"
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Build' -Message $_
ExitWithExitCode 1
}
+92 -107
View File
@@ -1,114 +1,99 @@
Param(
[string] $GuardianPackageName, # Required: the name of guardian CLI package (not needed if GuardianCliLocation is specified)
[string] $NugetPackageDirectory, # Required: directory where NuGet packages are installed (not needed if GuardianCliLocation is specified)
[string] $GuardianCliLocation, # Optional: Direct location of Guardian CLI executable if GuardianPackageName & NugetPackageDirectory are not specified
[string] $Repository=$env:BUILD_REPOSITORY_NAME, # Required: the name of the repository (e.g. dotnet/arcade)
[string] $BranchName=$env:BUILD_SOURCEBRANCH, # Optional: name of branch or version of gdn settings; defaults to master
[string] $SourceDirectory=$env:BUILD_SOURCESDIRECTORY, # Required: the directory where source files are located
[string] $ArtifactsDirectory = (Join-Path $env:BUILD_ARTIFACTSTAGINGDIRECTORY ('artifacts')), # Required: the directory where build artifacts are located
[string] $AzureDevOpsAccessToken, # Required: access token for dnceng; should be provided via KeyVault
[string[]] $SourceToolsList, # Optional: list of SDL tools to run on source code
[string[]] $ArtifactToolsList, # Optional: list of SDL tools to run on built artifacts
[bool] $TsaPublish=$False, # Optional: true will publish results to TSA; only set to true after onboarding to TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaBranchName=$env:BUILD_SOURCEBRANCH, # Optional: required for TSA publish; defaults to $(Build.SourceBranchName); TSA is the automated framework used to upload test results as bugs.
[string] $TsaRepositoryName=$env:BUILD_REPOSITORY_NAME, # Optional: TSA repository name; will be generated automatically if not submitted; TSA is the automated framework used to upload test results as bugs.
[string] $BuildNumber=$env:BUILD_BUILDNUMBER, # Optional: required for TSA publish; defaults to $(Build.BuildNumber)
[bool] $UpdateBaseline=$False, # Optional: if true, will update the baseline in the repository; should only be run after fixing any issues which need to be fixed
[bool] $TsaOnboard=$False, # Optional: if true, will onboard the repository to TSA; should only be run once; TSA is the automated framework used to upload test results as bugs.
[string] $TsaInstanceUrl, # Optional: only needed if TsaOnboard or TsaPublish is true; the instance-url registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaCodebaseName, # Optional: only needed if TsaOnboard or TsaPublish is true; the name of the codebase registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaProjectName, # Optional: only needed if TsaOnboard or TsaPublish is true; the name of the project registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaNotificationEmail, # Optional: only needed if TsaOnboard is true; the email(s) which will receive notifications of TSA bug filings (e.g. [email protected]); TSA is the automated framework used to upload test results as bugs.
[string] $TsaCodebaseAdmin, # Optional: only needed if TsaOnboard is true; the aliases which are admins of the TSA codebase (e.g. DOMAIN\alias); TSA is the automated framework used to upload test results as bugs.
[string] $TsaBugAreaPath, # Optional: only needed if TsaOnboard is true; the area path where TSA will file bugs in AzDO; TSA is the automated framework used to upload test results as bugs.
[string] $TsaIterationPath, # Optional: only needed if TsaOnboard is true; the iteration path where TSA will file bugs in AzDO; TSA is the automated framework used to upload test results as bugs.
[string] $GuardianLoggerLevel='Standard', # Optional: the logger level for the Guardian CLI; options are Trace, Verbose, Standard, Warning, and Error
[string[]] $CrScanAdditionalRunConfigParams, # Optional: Additional Params to custom build a CredScan run config in the format @("xyz:abc","sdf:1")
[string[]] $PoliCheckAdditionalRunConfigParams # Optional: Additional Params to custom build a Policheck run config in the format @("xyz:abc","sdf:1")
[string] $GuardianPackageName, # Required: the name of guardian CLI package (not needed if GuardianCliLocation is specified)
[string] $NugetPackageDirectory, # Required: directory where NuGet packages are installed (not needed if GuardianCliLocation is specified)
[string] $GuardianCliLocation, # Optional: Direct location of Guardian CLI executable if GuardianPackageName & NugetPackageDirectory are not specified
[string] $Repository=$env:BUILD_REPOSITORY_NAME, # Required: the name of the repository (e.g. dotnet/arcade)
[string] $BranchName=$env:BUILD_SOURCEBRANCH, # Optional: name of branch or version of gdn settings; defaults to master
[string] $SourceDirectory=$env:BUILD_SOURCESDIRECTORY, # Required: the directory where source files are located
[string] $ArtifactsDirectory = (Join-Path $env:BUILD_SOURCESDIRECTORY ("artifacts")), # Required: the directory where build artifacts are located
[string] $AzureDevOpsAccessToken, # Required: access token for dnceng; should be provided via KeyVault
[string[]] $SourceToolsList, # Optional: list of SDL tools to run on source code
[string[]] $ArtifactToolsList, # Optional: list of SDL tools to run on built artifacts
[bool] $TsaPublish=$False, # Optional: true will publish results to TSA; only set to true after onboarding to TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaBranchName=$env:BUILD_SOURCEBRANCH, # Optional: required for TSA publish; defaults to $(Build.SourceBranchName); TSA is the automated framework used to upload test results as bugs.
[string] $TsaRepositoryName=$env:BUILD_REPOSITORY_NAME, # Optional: TSA repository name; will be generated automatically if not submitted; TSA is the automated framework used to upload test results as bugs.
[string] $BuildNumber=$env:BUILD_BUILDNUMBER, # Optional: required for TSA publish; defaults to $(Build.BuildNumber)
[bool] $UpdateBaseline=$False, # Optional: if true, will update the baseline in the repository; should only be run after fixing any issues which need to be fixed
[bool] $TsaOnboard=$False, # Optional: if true, will onboard the repository to TSA; should only be run once; TSA is the automated framework used to upload test results as bugs.
[string] $TsaInstanceUrl, # Optional: only needed if TsaOnboard or TsaPublish is true; the instance-url registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaCodebaseName, # Optional: only needed if TsaOnboard or TsaPublish is true; the name of the codebase registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaProjectName, # Optional: only needed if TsaOnboard or TsaPublish is true; the name of the project registered with TSA; TSA is the automated framework used to upload test results as bugs.
[string] $TsaNotificationEmail, # Optional: only needed if TsaOnboard is true; the email(s) which will receive notifications of TSA bug filings (e.g. [email protected]); TSA is the automated framework used to upload test results as bugs.
[string] $TsaCodebaseAdmin, # Optional: only needed if TsaOnboard is true; the aliases which are admins of the TSA codebase (e.g. DOMAIN\alias); TSA is the automated framework used to upload test results as bugs.
[string] $TsaBugAreaPath, # Optional: only needed if TsaOnboard is true; the area path where TSA will file bugs in AzDO; TSA is the automated framework used to upload test results as bugs.
[string] $TsaIterationPath, # Optional: only needed if TsaOnboard is true; the iteration path where TSA will file bugs in AzDO; TSA is the automated framework used to upload test results as bugs.
[string] $GuardianLoggerLevel="Standard", # Optional: the logger level for the Guardian CLI; options are Trace, Verbose, Standard, Warning, and Error
[string[]] $CrScanAdditionalRunConfigParams, # Optional: Additional Params to custom build a CredScan run config in the format @("xyz:abc","sdf:1")
[string[]] $PoliCheckAdditionalRunConfigParams # Optional: Additional Params to custom build a Policheck run config in the format @("xyz:abc","sdf:1")
)
try {
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 2.0
$disableConfigureToolsetImport = $true
$LASTEXITCODE = 0
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
$LASTEXITCODE = 0
# `tools.ps1` checks $ci to perform some actions. Since the SDL
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
. $PSScriptRoot\..\tools.ps1
#Replace repo names to the format of org/repo
if (!($Repository.contains('/'))) {
$RepoName = $Repository -replace '(.*?)-(.*)', '$1/$2';
}
else{
$RepoName = $Repository;
}
if ($GuardianPackageName) {
$guardianCliLocation = Join-Path $NugetPackageDirectory (Join-Path $GuardianPackageName (Join-Path 'tools' 'guardian.cmd'))
} else {
$guardianCliLocation = $GuardianCliLocation
}
$workingDirectory = (Split-Path $SourceDirectory -Parent)
$ValidPath = Test-Path $guardianCliLocation
if ($ValidPath -eq $False)
{
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message 'Invalid Guardian CLI Location.'
ExitWithExitCode 1
}
& $(Join-Path $PSScriptRoot 'init-sdl.ps1') -GuardianCliLocation $guardianCliLocation -Repository $RepoName -BranchName $BranchName -WorkingDirectory $workingDirectory -AzureDevOpsAccessToken $AzureDevOpsAccessToken -GuardianLoggerLevel $GuardianLoggerLevel
$gdnFolder = Join-Path $workingDirectory '.gdn'
if ($TsaOnboard) {
if ($TsaCodebaseName -and $TsaNotificationEmail -and $TsaCodebaseAdmin -and $TsaBugAreaPath) {
Write-Host "$guardianCliLocation tsa-onboard --codebase-name `"$TsaCodebaseName`" --notification-alias `"$TsaNotificationEmail`" --codebase-admin `"$TsaCodebaseAdmin`" --instance-url `"$TsaInstanceUrl`" --project-name `"$TsaProjectName`" --area-path `"$TsaBugAreaPath`" --iteration-path `"$TsaIterationPath`" --working-directory $workingDirectory --logger-level $GuardianLoggerLevel"
& $guardianCliLocation tsa-onboard --codebase-name "$TsaCodebaseName" --notification-alias "$TsaNotificationEmail" --codebase-admin "$TsaCodebaseAdmin" --instance-url "$TsaInstanceUrl" --project-name "$TsaProjectName" --area-path "$TsaBugAreaPath" --iteration-path "$TsaIterationPath" --working-directory $workingDirectory --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Guardian tsa-onboard failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
} else {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message 'Could not onboard to TSA -- not all required values ($TsaCodebaseName, $TsaNotificationEmail, $TsaCodebaseAdmin, $TsaBugAreaPath) were specified.'
ExitWithExitCode 1
}
}
if ($ArtifactToolsList -and $ArtifactToolsList.Count -gt 0) {
& $(Join-Path $PSScriptRoot 'run-sdl.ps1') -GuardianCliLocation $guardianCliLocation -WorkingDirectory $workingDirectory -TargetDirectory $ArtifactsDirectory -GdnFolder $gdnFolder -ToolsList $ArtifactToolsList -AzureDevOpsAccessToken $AzureDevOpsAccessToken -UpdateBaseline $UpdateBaseline -GuardianLoggerLevel $GuardianLoggerLevel -CrScanAdditionalRunConfigParams $CrScanAdditionalRunConfigParams -PoliCheckAdditionalRunConfigParams $PoliCheckAdditionalRunConfigParams
}
if ($SourceToolsList -and $SourceToolsList.Count -gt 0) {
& $(Join-Path $PSScriptRoot 'run-sdl.ps1') -GuardianCliLocation $guardianCliLocation -WorkingDirectory $workingDirectory -TargetDirectory $SourceDirectory -GdnFolder $gdnFolder -ToolsList $SourceToolsList -AzureDevOpsAccessToken $AzureDevOpsAccessToken -UpdateBaseline $UpdateBaseline -GuardianLoggerLevel $GuardianLoggerLevel -CrScanAdditionalRunConfigParams $CrScanAdditionalRunConfigParams -PoliCheckAdditionalRunConfigParams $PoliCheckAdditionalRunConfigParams
}
if ($UpdateBaseline) {
& (Join-Path $PSScriptRoot 'push-gdn.ps1') -Repository $RepoName -BranchName $BranchName -GdnFolder $GdnFolder -AzureDevOpsAccessToken $AzureDevOpsAccessToken -PushReason 'Update baseline'
}
if ($TsaPublish) {
if ($TsaBranchName -and $BuildNumber) {
if (-not $TsaRepositoryName) {
$TsaRepositoryName = "$($Repository)-$($BranchName)"
}
Write-Host "$guardianCliLocation tsa-publish --all-tools --repository-name `"$TsaRepositoryName`" --branch-name `"$TsaBranchName`" --build-number `"$BuildNumber`" --codebase-name `"$TsaCodebaseName`" --notification-alias `"$TsaNotificationEmail`" --codebase-admin `"$TsaCodebaseAdmin`" --instance-url `"$TsaInstanceUrl`" --project-name `"$TsaProjectName`" --area-path `"$TsaBugAreaPath`" --iteration-path `"$TsaIterationPath`" --working-directory $workingDirectory --logger-level $GuardianLoggerLevel"
& $guardianCliLocation tsa-publish --all-tools --repository-name "$TsaRepositoryName" --branch-name "$TsaBranchName" --build-number "$BuildNumber" --onboard $True --codebase-name "$TsaCodebaseName" --notification-alias "$TsaNotificationEmail" --codebase-admin "$TsaCodebaseAdmin" --instance-url "$TsaInstanceUrl" --project-name "$TsaProjectName" --area-path "$TsaBugAreaPath" --iteration-path "$TsaIterationPath" --working-directory $workingDirectory --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Guardian tsa-publish failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
} else {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message 'Could not publish to TSA -- not all required values ($TsaBranchName, $BuildNumber) were specified.'
ExitWithExitCode 1
}
}
#Replace repo names to the format of org/repo
if (!($Repository.contains('/'))) {
$RepoName = $Repository -replace '(.*?)-(.*)', '$1/$2';
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message $_
else{
$RepoName = $Repository;
}
if ($GuardianPackageName) {
$guardianCliLocation = Join-Path $NugetPackageDirectory (Join-Path $GuardianPackageName (Join-Path "tools" "guardian.cmd"))
} else {
$guardianCliLocation = $GuardianCliLocation
}
$ValidPath = Test-Path $guardianCliLocation
if ($ValidPath -eq $False)
{
Write-Host "Invalid Guardian CLI Location."
exit 1
}
& $(Join-Path $PSScriptRoot "init-sdl.ps1") -GuardianCliLocation $guardianCliLocation -Repository $RepoName -BranchName $BranchName -WorkingDirectory $ArtifactsDirectory -AzureDevOpsAccessToken $AzureDevOpsAccessToken -GuardianLoggerLevel $GuardianLoggerLevel
$gdnFolder = Join-Path $ArtifactsDirectory ".gdn"
if ($TsaOnboard) {
if ($TsaCodebaseName -and $TsaNotificationEmail -and $TsaCodebaseAdmin -and $TsaBugAreaPath) {
Write-Host "$guardianCliLocation tsa-onboard --codebase-name `"$TsaCodebaseName`" --notification-alias `"$TsaNotificationEmail`" --codebase-admin `"$TsaCodebaseAdmin`" --instance-url `"$TsaInstanceUrl`" --project-name `"$TsaProjectName`" --area-path `"$TsaBugAreaPath`" --iteration-path `"$TsaIterationPath`" --working-directory $ArtifactsDirectory --logger-level $GuardianLoggerLevel"
& $guardianCliLocation tsa-onboard --codebase-name "$TsaCodebaseName" --notification-alias "$TsaNotificationEmail" --codebase-admin "$TsaCodebaseAdmin" --instance-url "$TsaInstanceUrl" --project-name "$TsaProjectName" --area-path "$TsaBugAreaPath" --iteration-path "$TsaIterationPath" --working-directory $ArtifactsDirectory --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian tsa-onboard failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
} else {
Write-Host "Could not onboard to TSA -- not all required values ($$TsaCodebaseName, $$TsaNotificationEmail, $$TsaCodebaseAdmin, $$TsaBugAreaPath) were specified."
exit 1
}
}
if ($ArtifactToolsList -and $ArtifactToolsList.Count -gt 0) {
& $(Join-Path $PSScriptRoot "run-sdl.ps1") -GuardianCliLocation $guardianCliLocation -WorkingDirectory $ArtifactsDirectory -TargetDirectory $ArtifactsDirectory -GdnFolder $gdnFolder -ToolsList $ArtifactToolsList -AzureDevOpsAccessToken $AzureDevOpsAccessToken -UpdateBaseline $UpdateBaseline -GuardianLoggerLevel $GuardianLoggerLevel -CrScanAdditionalRunConfigParams $CrScanAdditionalRunConfigParams -PoliCheckAdditionalRunConfigParams $PoliCheckAdditionalRunConfigParams
}
if ($SourceToolsList -and $SourceToolsList.Count -gt 0) {
& $(Join-Path $PSScriptRoot "run-sdl.ps1") -GuardianCliLocation $guardianCliLocation -WorkingDirectory $ArtifactsDirectory -TargetDirectory $SourceDirectory -GdnFolder $gdnFolder -ToolsList $SourceToolsList -AzureDevOpsAccessToken $AzureDevOpsAccessToken -UpdateBaseline $UpdateBaseline -GuardianLoggerLevel $GuardianLoggerLevel -CrScanAdditionalRunConfigParams $CrScanAdditionalRunConfigParams -PoliCheckAdditionalRunConfigParams $PoliCheckAdditionalRunConfigParams
}
if ($UpdateBaseline) {
& (Join-Path $PSScriptRoot "push-gdn.ps1") -Repository $RepoName -BranchName $BranchName -GdnFolder $GdnFolder -AzureDevOpsAccessToken $AzureDevOpsAccessToken -PushReason "Update baseline"
}
if ($TsaPublish) {
if ($TsaBranchName -and $BuildNumber) {
if (-not $TsaRepositoryName) {
$TsaRepositoryName = "$($Repository)-$($BranchName)"
}
Write-Host "$guardianCliLocation tsa-publish --all-tools --repository-name `"$TsaRepositoryName`" --branch-name `"$TsaBranchName`" --build-number `"$BuildNumber`" --codebase-name `"$TsaCodebaseName`" --notification-alias `"$TsaNotificationEmail`" --codebase-admin `"$TsaCodebaseAdmin`" --instance-url `"$TsaInstanceUrl`" --project-name `"$TsaProjectName`" --area-path `"$TsaBugAreaPath`" --iteration-path `"$TsaIterationPath`" --working-directory $SourceDirectory --logger-level $GuardianLoggerLevel"
& $guardianCliLocation tsa-publish --all-tools --repository-name "$TsaRepositoryName" --branch-name "$TsaBranchName" --build-number "$BuildNumber" --codebase-name "$TsaCodebaseName" --notification-alias "$TsaNotificationEmail" --codebase-admin "$TsaCodebaseAdmin" --instance-url "$TsaInstanceUrl" --project-name "$TsaProjectName" --area-path "$TsaBugAreaPath" --iteration-path "$TsaIterationPath" --working-directory $ArtifactsDirectory --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian tsa-publish failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
} else {
Write-Host "Could not publish to TSA -- not all required values ($$TsaBranchName, $$BuildNumber) were specified."
exit 1
}
}
+40 -50
View File
@@ -3,12 +3,47 @@ param(
[Parameter(Mandatory=$true)][string] $ExtractPath # Full path to directory where the packages will be extracted
)
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
$ExtractPackage = {
param(
[string] $PackagePath # Full path to a NuGet package
)
if (!(Test-Path $PackagePath)) {
Write-PipelineTaskError "Input file does not exist: $PackagePath"
ExitWithExitCode 1
}
$RelevantExtensions = @(".dll", ".exe", ".pdb")
Write-Host -NoNewLine "Extracting" ([System.IO.Path]::GetFileName($PackagePath)) "... "
$disableConfigureToolsetImport = $true
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$ExtractPath = Join-Path -Path $using:ExtractPath -ChildPath $PackageId
function ExtractArtifacts {
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Directory]::CreateDirectory($ExtractPath);
try {
$zip = [System.IO.Compression.ZipFile]::OpenRead($PackagePath)
$zip.Entries |
Where-Object {$RelevantExtensions -contains [System.IO.Path]::GetExtension($_.Name)} |
ForEach-Object {
$TargetFile = Join-Path -Path $ExtractPath -ChildPath $_.Name
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, $TargetFile, $true)
}
}
catch {
}
finally {
$zip.Dispose()
}
}
function ExtractArtifacts {
if (!(Test-Path $InputPath)) {
Write-Host "Input Path does not exist: $InputPath"
ExitWithExitCode 0
@@ -25,56 +60,11 @@ function ExtractArtifacts {
}
try {
# `tools.ps1` checks $ci to perform some actions. Since the SDL
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
. $PSScriptRoot\..\tools.ps1
$ExtractPackage = {
param(
[string] $PackagePath # Full path to a NuGet package
)
if (!(Test-Path $PackagePath)) {
Write-PipelineTelemetryError -Category 'Build' -Message "Input file does not exist: $PackagePath"
ExitWithExitCode 1
}
$RelevantExtensions = @('.dll', '.exe', '.pdb')
Write-Host -NoNewLine 'Extracting ' ([System.IO.Path]::GetFileName($PackagePath)) '...'
$PackageId = [System.IO.Path]::GetFileNameWithoutExtension($PackagePath)
$ExtractPath = Join-Path -Path $using:ExtractPath -ChildPath $PackageId
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Directory]::CreateDirectory($ExtractPath);
try {
$zip = [System.IO.Compression.ZipFile]::OpenRead($PackagePath)
$zip.Entries |
Where-Object {$RelevantExtensions -contains [System.IO.Path]::GetExtension($_.Name)} |
ForEach-Object {
$TargetFile = Join-Path -Path $ExtractPath -ChildPath $_.Name
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, $TargetFile, $true)
}
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message $_
ExitWithExitCode 1
}
finally {
$zip.Dispose()
}
}
Measure-Command { ExtractArtifacts }
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message $_
ExitWithExitCode 1
}
+13 -32
View File
@@ -1,26 +1,16 @@
Param(
[string] $GuardianCliLocation,
[string] $Repository,
[string] $BranchName='master',
[string] $BranchName="master",
[string] $WorkingDirectory,
[string] $AzureDevOpsAccessToken,
[string] $GuardianLoggerLevel='Standard'
[string] $GuardianLoggerLevel="Standard"
)
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
$disableConfigureToolsetImport = $true
$LASTEXITCODE = 0
# `tools.ps1` checks $ci to perform some actions. Since the SDL
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
. $PSScriptRoot\..\tools.ps1
# Don't display the console progress UI - it's a huge perf hit
$ProgressPreference = 'SilentlyContinue'
# Construct basic auth from AzDO access token; construct URI to the repository's gdn folder stored in that repository; construct location of zip file
$encodedPat = [Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$AzureDevOpsAccessToken"))
$escapedRepository = [Uri]::EscapeDataString("/$Repository/$BranchName/.gdn")
@@ -28,10 +18,11 @@ $uri = "https://dev.azure.com/dnceng/internal/_apis/git/repositories/sdl-tool-cf
$zipFile = "$WorkingDirectory/gdn.zip"
Add-Type -AssemblyName System.IO.Compression.FileSystem
$gdnFolder = (Join-Path $WorkingDirectory '.gdn')
try {
$gdnFolder = (Join-Path $WorkingDirectory ".gdn")
Try
{
# We try to download the zip; if the request fails (e.g. the file doesn't exist), we catch it and init guardian instead
Write-Host 'Downloading gdn folder from internal config repostiory...'
Write-Host "Downloading gdn folder from internal config repostiory..."
Invoke-WebRequest -Headers @{ "Accept"="application/zip"; "Authorization"="Basic $encodedPat" } -Uri $uri -OutFile $zipFile
if (Test-Path $gdnFolder) {
# Remove the gdn folder if it exists (it shouldn't unless there's too much caching; this is just in case)
@@ -39,29 +30,19 @@ try {
}
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipFile, $WorkingDirectory)
Write-Host $gdnFolder
ExitWithExitCode 0
} catch [System.Net.WebException] { } # Catch and ignore webexception
try {
} Catch [System.Net.WebException] {
# if the folder does not exist, we'll do a guardian init and push it to the remote repository
Write-Host 'Initializing Guardian...'
Write-Host "Initializing Guardian..."
Write-Host "$GuardianCliLocation init --working-directory $WorkingDirectory --logger-level $GuardianLoggerLevel"
& $GuardianCliLocation init --working-directory $WorkingDirectory --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Build' -Message "Guardian init failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
Write-Error "Guardian init failed with exit code $LASTEXITCODE."
}
# We create the mainbaseline so it can be edited later
Write-Host "$GuardianCliLocation baseline --working-directory $WorkingDirectory --name mainbaseline"
& $GuardianCliLocation baseline --working-directory $WorkingDirectory --name mainbaseline
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Build' -Message "Guardian baseline failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
Write-Error "Guardian baseline failed with exit code $LASTEXITCODE."
}
& $(Join-Path $PSScriptRoot 'push-gdn.ps1') -Repository $Repository -BranchName $BranchName -GdnFolder $gdnFolder -AzureDevOpsAccessToken $AzureDevOpsAccessToken -PushReason 'Initialize gdn folder'
ExitWithExitCode 0
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message $_
ExitWithExitCode 1
}
& $(Join-Path $PSScriptRoot "push-gdn.ps1") -Repository $Repository -BranchName $BranchName -GdnFolder $gdnFolder -AzureDevOpsAccessToken $AzureDevOpsAccessToken -PushReason "Initialize gdn folder"
}
+2 -2
View File
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.Guardian.Cli" version="0.7.2"/>
<package id="Microsoft.Guardian.Cli" version="0.6.0"/>
</packages>
+39 -57
View File
@@ -1,69 +1,51 @@
Param(
[string] $Repository,
[string] $BranchName='master',
[string] $BranchName="master",
[string] $GdnFolder,
[string] $AzureDevOpsAccessToken,
[string] $PushReason
)
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
$disableConfigureToolsetImport = $true
$LASTEXITCODE = 0
try {
# `tools.ps1` checks $ci to perform some actions. Since the SDL
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
. $PSScriptRoot\..\tools.ps1
# We create the temp directory where we'll store the sdl-config repository
$sdlDir = Join-Path $env:TEMP 'sdl'
if (Test-Path $sdlDir) {
Remove-Item -Force -Recurse $sdlDir
}
Write-Host "git clone https://dnceng:`$[email protected]/dnceng/internal/_git/sdl-tool-cfg $sdlDir"
git clone https://dnceng:$AzureDevOpsAccessToken@dev.azure.com/dnceng/internal/_git/sdl-tool-cfg $sdlDir
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Git clone failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
# We copy the .gdn folder from our local run into the git repository so it can be committed
$sdlRepositoryFolder = Join-Path (Join-Path (Join-Path $sdlDir $Repository) $BranchName) '.gdn'
if (Get-Command Robocopy) {
Robocopy /S $GdnFolder $sdlRepositoryFolder
} else {
rsync -r $GdnFolder $sdlRepositoryFolder
}
# cd to the sdl-config directory so we can run git there
Push-Location $sdlDir
# git add . --> git commit --> git push
Write-Host 'git add .'
git add .
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Git add failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
Write-Host "git -c user.email=`"[email protected]`" -c user.name=`"Dotnet Bot`" commit -m `"$PushReason for $Repository/$BranchName`""
git -c user.email="[email protected]" -c user.name="Dotnet Bot" commit -m "$PushReason for $Repository/$BranchName"
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Git commit failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
Write-Host 'git push'
git push
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Git push failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
# Return to the original directory
Pop-Location
# We create the temp directory where we'll store the sdl-config repository
$sdlDir = Join-Path $env:TEMP "sdl"
if (Test-Path $sdlDir) {
Remove-Item -Force -Recurse $sdlDir
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Sdl' -Message $_
ExitWithExitCode 1
Write-Host "git clone https://dnceng:`$[email protected]/dnceng/internal/_git/sdl-tool-cfg $sdlDir"
git clone https://dnceng:$AzureDevOpsAccessToken@dev.azure.com/dnceng/internal/_git/sdl-tool-cfg $sdlDir
if ($LASTEXITCODE -ne 0) {
Write-Error "Git clone failed with exit code $LASTEXITCODE."
}
# We copy the .gdn folder from our local run into the git repository so it can be committed
$sdlRepositoryFolder = Join-Path (Join-Path (Join-Path $sdlDir $Repository) $BranchName) ".gdn"
if (Get-Command Robocopy) {
Robocopy /S $GdnFolder $sdlRepositoryFolder
} else {
rsync -r $GdnFolder $sdlRepositoryFolder
}
# cd to the sdl-config directory so we can run git there
Push-Location $sdlDir
# git add . --> git commit --> git push
Write-Host "git add ."
git add .
if ($LASTEXITCODE -ne 0) {
Write-Error "Git add failed with exit code $LASTEXITCODE."
}
Write-Host "git -c user.email=`"[email protected]`" -c user.name=`"Dotnet Bot`" commit -m `"$PushReason for $Repository/$BranchName`""
git -c user.email="[email protected]" -c user.name="Dotnet Bot" commit -m "$PushReason for $Repository/$BranchName"
if ($LASTEXITCODE -ne 0) {
Write-Error "Git commit failed with exit code $LASTEXITCODE."
}
Write-Host "git push"
git push
if ($LASTEXITCODE -ne 0) {
Write-Error "Git push failed with exit code $LASTEXITCODE."
}
# Return to the original directory
Pop-Location
+45 -51
View File
@@ -5,69 +5,63 @@ Param(
[string] $GdnFolder,
[string[]] $ToolsList,
[string] $UpdateBaseline,
[string] $GuardianLoggerLevel='Standard',
[string] $GuardianLoggerLevel="Standard",
[string[]] $CrScanAdditionalRunConfigParams,
[string[]] $PoliCheckAdditionalRunConfigParams
)
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
Set-StrictMode -Version 2.0
$disableConfigureToolsetImport = $true
$LASTEXITCODE = 0
try {
# `tools.ps1` checks $ci to perform some actions. Since the SDL
# scripts don't necessarily execute in the same agent that run the
# build.ps1/sh script this variable isn't automatically set.
$ci = $true
. $PSScriptRoot\..\tools.ps1
# We store config files in the r directory of .gdn
Write-Host $ToolsList
$gdnConfigPath = Join-Path $GdnFolder "r"
$ValidPath = Test-Path $GuardianCliLocation
# We store config files in the r directory of .gdn
Write-Host $ToolsList
$gdnConfigPath = Join-Path $GdnFolder 'r'
$ValidPath = Test-Path $GuardianCliLocation
if ($ValidPath -eq $False)
{
Write-Host "Invalid Guardian CLI Location."
exit 1
}
if ($ValidPath -eq $False)
{
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Invalid Guardian CLI Location."
ExitWithExitCode 1
foreach ($tool in $ToolsList) {
$gdnConfigFile = Join-Path $gdnConfigPath "$tool-configure.gdnconfig"
$config = $False
Write-Host $tool
# We have to manually configure tools that run on source to look at the source directory only
if ($tool -eq "credscan") {
Write-Host "$GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args `" TargetDirectory : $TargetDirectory `" $(If ($CrScanAdditionalRunConfigParams) {$CrScanAdditionalRunConfigParams})"
& $GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args " TargetDirectory : $TargetDirectory " $(If ($CrScanAdditionalRunConfigParams) {$CrScanAdditionalRunConfigParams})
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian configure for $tool failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
$config = $True
}
if ($tool -eq "policheck") {
Write-Host "$GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args `" Target : $TargetDirectory `" $(If ($PoliCheckAdditionalRunConfigParams) {$PoliCheckAdditionalRunConfigParams})"
& $GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args " Target : $TargetDirectory " $(If ($PoliCheckAdditionalRunConfigParams) {$PoliCheckAdditionalRunConfigParams})
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian configure for $tool failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
$config = $True
}
$configParam = @('--config')
foreach ($tool in $ToolsList) {
$gdnConfigFile = Join-Path $gdnConfigPath "$tool-configure.gdnconfig"
Write-Host $tool
# We have to manually configure tools that run on source to look at the source directory only
if ($tool -eq 'credscan') {
Write-Host "$GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args `" TargetDirectory < $TargetDirectory `" `" OutputType < pre `" $(If ($CrScanAdditionalRunConfigParams) {$CrScanAdditionalRunConfigParams})"
& $GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args " TargetDirectory < $TargetDirectory " "OutputType < pre" $(If ($CrScanAdditionalRunConfigParams) {$CrScanAdditionalRunConfigParams})
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Guardian configure for $tool failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
Write-Host "$GuardianCliLocation run --working-directory $WorkingDirectory --tool $tool --baseline mainbaseline --update-baseline $UpdateBaseline --logger-level $GuardianLoggerLevel --config $gdnConfigFile $config"
if ($config) {
& $GuardianCliLocation run --working-directory $WorkingDirectory --tool $tool --baseline mainbaseline --update-baseline $UpdateBaseline --logger-level $GuardianLoggerLevel --config $gdnConfigFile
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian run for $tool using $gdnConfigFile failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
if ($tool -eq 'policheck') {
Write-Host "$GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args `" Target < $TargetDirectory `" $(If ($PoliCheckAdditionalRunConfigParams) {$PoliCheckAdditionalRunConfigParams})"
& $GuardianCliLocation configure --working-directory $WorkingDirectory --tool $tool --output-path $gdnConfigFile --logger-level $GuardianLoggerLevel --noninteractive --force --args " Target < $TargetDirectory " $(If ($PoliCheckAdditionalRunConfigParams) {$PoliCheckAdditionalRunConfigParams})
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Guardian configure for $tool failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
} else {
& $GuardianCliLocation run --working-directory $WorkingDirectory --tool $tool --baseline mainbaseline --update-baseline $UpdateBaseline --logger-level $GuardianLoggerLevel
if ($LASTEXITCODE -ne 0) {
Write-Host "Guardian run for $tool failed with exit code $LASTEXITCODE."
exit $LASTEXITCODE
}
$configParam+=$gdnConfigFile
}
Write-Host "$GuardianCliLocation run --working-directory $WorkingDirectory --baseline mainbaseline --update-baseline $UpdateBaseline --logger-level $GuardianLoggerLevel $configParam"
& $GuardianCliLocation run --working-directory $WorkingDirectory --tool $tool --baseline mainbaseline --update-baseline $UpdateBaseline --logger-level $GuardianLoggerLevel $configParam
if ($LASTEXITCODE -ne 0) {
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message "Guardian run for $ToolsList using $configParam failed with exit code $LASTEXITCODE."
ExitWithExitCode $LASTEXITCODE
}
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Force -Category 'Sdl' -Message $_
ExitWithExitCode 1
}
+17 -52
View File
@@ -1,70 +1,35 @@
parameters:
enable: 'false' # Whether the SDL validation job should execute or not
overrideParameters: '' # Optional: to override values for parameters.
additionalParameters: '' # Optional: parameters that need user specific values eg: '-SourceToolsList @("abc","def") -ArtifactToolsList @("ghi","jkl")'
# There is some sort of bug (has been reported) in Azure DevOps where if this parameter is named
# 'continueOnError', the parameter value is not correctly picked up.
# This can also be remedied by the caller (post-build.yml) if it does not use a nested parameter
sdlContinueOnError: false # optional: determines whether to continue the build if the step errors;
continueOnError: false # optional: determines whether to continue the build if the step errors;
dependsOn: '' # Optional: dependencies of the job
artifactNames: '' # Optional: patterns supplied to DownloadBuildArtifacts
# Usage:
# artifactNames:
# - 'BlobArtifacts'
# - 'Artifacts_Windows_NT_Release'
jobs:
- job: Run_SDL
dependsOn: ${{ parameters.dependsOn }}
displayName: Run SDL tool
condition: eq( ${{ parameters.enable }}, 'true')
variables:
- group: DotNet-VSTS-Bot
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
pool:
name: Hosted VS2017
steps:
- checkout: self
clean: true
- ${{ if ne(parameters.artifactNames, '') }}:
- ${{ each artifactName in parameters.artifactNames }}:
- task: DownloadBuildArtifacts@0
displayName: Download Build Artifacts
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
artifactName: ${{ artifactName }}
downloadPath: $(Build.ArtifactStagingDirectory)\artifacts
- ${{ if eq(parameters.artifactNames, '') }}:
- task: DownloadBuildArtifacts@0
displayName: Download Build Artifacts
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
downloadType: specific files
itemPattern: "**"
downloadPath: $(Build.ArtifactStagingDirectory)\artifacts
- task: DownloadBuildArtifacts@0
displayName: Download Build Artifacts
inputs:
buildType: current
downloadType: specific files
matchingPattern: "**"
downloadPath: $(Build.SourcesDirectory)\artifacts
- powershell: eng/common/sdl/extract-artifact-packages.ps1
-InputPath $(Build.ArtifactStagingDirectory)\artifacts\BlobArtifacts
-ExtractPath $(Build.ArtifactStagingDirectory)\artifacts\BlobArtifacts
-InputPath $(Build.SourcesDirectory)\artifacts\BlobArtifacts
-ExtractPath $(Build.SourcesDirectory)\artifacts\BlobArtifacts
displayName: Extract Blob Artifacts
continueOnError: ${{ parameters.sdlContinueOnError }}
continueOnError: ${{ parameters.continueOnError }}
- powershell: eng/common/sdl/extract-artifact-packages.ps1
-InputPath $(Build.ArtifactStagingDirectory)\artifacts\PackageArtifacts
-ExtractPath $(Build.ArtifactStagingDirectory)\artifacts\PackageArtifacts
-InputPath $(Build.SourcesDirectory)\artifacts\PackageArtifacts
-ExtractPath $(Build.SourcesDirectory)\artifacts\PackageArtifacts
displayName: Extract Package Artifacts
continueOnError: ${{ parameters.sdlContinueOnError }}
continueOnError: ${{ parameters.continueOnError }}
- task: NuGetToolInstaller@1
displayName: 'Install NuGet.exe'
- task: NuGetCommand@2
@@ -78,12 +43,12 @@ jobs:
- ${{ if ne(parameters.overrideParameters, '') }}:
- powershell: eng/common/sdl/execute-all-sdl-tools.ps1 ${{ parameters.overrideParameters }}
displayName: Execute SDL
continueOnError: ${{ parameters.sdlContinueOnError }}
continueOnError: ${{ parameters.continueOnError }}
- ${{ if eq(parameters.overrideParameters, '') }}:
- powershell: eng/common/sdl/execute-all-sdl-tools.ps1
-GuardianPackageName Microsoft.Guardian.Cli.0.7.2
-GuardianPackageName Microsoft.Guardian.Cli.0.6.0
-NugetPackageDirectory $(Build.SourcesDirectory)\.packages
-AzureDevOpsAccessToken $(dn-bot-dotnet-build-rw-code-rw)
${{ parameters.additionalParameters }}
displayName: Execute SDL
continueOnError: ${{ parameters.sdlContinueOnError }}
continueOnError: ${{ parameters.continueOnError }}
+64 -79
View File
@@ -1,35 +1,64 @@
# Internal resources (telemetry, microbuild) can only be accessed from non-public projects,
# and some (Microbuild) should only be applied to non-PR cases for internal builds.
parameters:
# Job schema parameters - https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=vsts&tabs=schema#job
cancelTimeoutInMinutes: ''
condition: ''
container: ''
continueOnError: false
container: ''
dependsOn: ''
displayName: ''
pool: ''
steps: []
pool: ''
strategy: ''
timeoutInMinutes: ''
variables: []
workspace: ''
# Job base template specific parameters
# See schema documentation - https://github.com/dotnet/arcade/blob/master/Documentation/AzureDevOps/TemplateSchema.md
artifacts: ''
# Optional: Enable installing Microbuild plugin
# if 'true', these "variables" must be specified in the variables object or as part of the queue matrix
# _TeamName - the name of your team
# _SignType - 'test' or 'real'
enableMicrobuild: false
# Optional: Include PublishBuildArtifacts task
enablePublishBuildArtifacts: false
# Optional: Enable publishing to the build asset registry
enablePublishBuildAssets: false
# Optional: Include PublishTestResults task
enablePublishTestResults: false
enablePublishUsingPipelines: false
mergeTestResults: false
testRunTitle: $(AgentOsName)-$(BuildConfiguration)-xunit
# Optional: enable sending telemetry
enableTelemetry: false
# Optional: define the helix repo for telemetry (example: 'dotnet/arcade')
helixRepo: ''
# Optional: define the helix type for telemetry (example: 'build/product/')
helixType: ''
# Required: name of the job
name: ''
preSteps: []
# Optional: should run as a public build even in the internal project
# if 'true', the build won't run any of the internal only steps, even if it is running in non-public projects.
runAsPublic: false
# Internal resources (telemetry, microbuild) can only be accessed from non-public projects,
# and some (Microbuild) should only be applied to non-PR cases for internal builds.
jobs:
- job: ${{ parameters.name }}
@@ -61,7 +90,7 @@ jobs:
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
variables:
- ${{ if ne(parameters.enableTelemetry, 'false') }}:
- ${{ if eq(parameters.enableTelemetry, 'true') }}:
- name: DOTNET_CLI_TELEMETRY_PROFILE
value: '$(Build.Repository.Uri)'
- ${{ each variable in parameters.variables }}:
@@ -93,12 +122,21 @@ jobs:
workspace: ${{ parameters.workspace }}
steps:
- ${{ if ne(parameters.preSteps, '') }}:
- ${{ each preStep in parameters.preSteps }}:
- ${{ preStep }}
- ${{ if eq(parameters.enableTelemetry, 'true') }}:
# Telemetry tasks are built from https://github.com/dotnet/arcade-extensions
- task: sendStartTelemetry@0
displayName: 'Send Helix Start Telemetry'
inputs:
helixRepo: ${{ parameters.helixRepo }}
${{ if ne(parameters.helixType, '') }}:
helixType: ${{ parameters.helixType }}
buildConfig: $(_BuildConfig)
runAsPublic: ${{ parameters.runAsPublic }}
continueOnError: ${{ parameters.continueOnError }}
condition: always()
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- ${{ if eq(parameters.enableMicrobuild, 'true') }}:
- ${{ if eq(parameters.enableMicrobuild, 'true') }}:
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: MicroBuildSigningPlugin@2
displayName: Install MicroBuild plugin
inputs:
@@ -110,16 +148,6 @@ jobs:
continueOnError: ${{ parameters.continueOnError }}
condition: and(succeeded(), in(variables['_SignType'], 'real', 'test'), eq(variables['Agent.Os'], 'Windows_NT'))
- task: NuGetAuthenticate@0
- ${{ if or(eq(parameters.artifacts.download, 'true'), ne(parameters.artifacts.download, '')) }}:
- task: DownloadPipelineArtifact@2
inputs:
buildType: current
artifactName: ${{ coalesce(parameters.artifacts.download.name, 'Artifacts_$(Agent.OS)_$(_BuildConfig)') }}
targetPath: ${{ coalesce(parameters.artifacts.download.path, 'artifacts') }}
itemPattern: ${{ coalesce(parameters.artifacts.download.pattern, '**') }}
- ${{ each step in parameters.steps }}:
- ${{ step }}
@@ -132,60 +160,20 @@ jobs:
env:
TeamName: $(_TeamName)
- ${{ if ne(parameters.artifacts.publish, '') }}:
- ${{ if or(eq(parameters.artifacts.publish.artifacts, 'true'), ne(parameters.artifacts.publish.artifacts, '')) }}:
- task: CopyFiles@2
displayName: Gather binaries for publish to artifacts
inputs:
SourceFolder: 'artifacts/bin'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/bin'
- task: CopyFiles@2
displayName: Gather packages for publish to artifacts
inputs:
SourceFolder: 'artifacts/packages'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/packages'
- task: PublishBuildArtifacts@1
displayName: Publish pipeline artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/artifacts'
PublishLocation: Container
ArtifactName: ${{ coalesce(parameters.artifacts.publish.artifacts.name , 'Artifacts_$(Agent.Os)_$(_BuildConfig)') }}
continueOnError: true
condition: always()
- ${{ if or(eq(parameters.artifacts.publish.logs, 'true'), ne(parameters.artifacts.publish.logs, '')) }}:
- publish: artifacts/log
artifact: ${{ coalesce(parameters.artifacts.publish.logs.name, 'Logs_Build_$(Agent.Os)_$(_BuildConfig)') }}
displayName: Publish logs
continueOnError: true
condition: always()
- ${{ if or(eq(parameters.artifacts.publish.manifests, 'true'), ne(parameters.artifacts.publish.manifests, '')) }}:
- ${{ if and(ne(parameters.enablePublishUsingPipelines, 'true'), eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: CopyFiles@2
displayName: Gather Asset Manifests
inputs:
SourceFolder: '$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)/AssetManifest'
TargetFolder: '$(Build.ArtifactStagingDirectory)/AssetManifests'
continueOnError: ${{ parameters.continueOnError }}
condition: and(succeeded(), eq(variables['_DotNetPublishToBlobFeed'], 'true'))
- ${{ if eq(parameters.enableTelemetry, 'true') }}:
# Telemetry tasks are built from https://github.com/dotnet/arcade-extensions
- task: sendEndTelemetry@0
displayName: 'Send Helix End Telemetry'
continueOnError: ${{ parameters.continueOnError }}
condition: always()
- task: PublishBuildArtifacts@1
displayName: Push Asset Manifests
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/AssetManifests'
PublishLocation: Container
ArtifactName: AssetManifests
continueOnError: ${{ parameters.continueOnError }}
condition: and(succeeded(), eq(variables['_DotNetPublishToBlobFeed'], 'true'))
- ${{ if ne(parameters.enablePublishBuildArtifacts, 'false') }}:
- ${{ if eq(parameters.enablePublishBuildArtifacts, 'true') }}:
- task: PublishBuildArtifacts@1
displayName: Publish Logs
inputs:
PathtoPublish: '$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)'
PublishLocation: Container
ArtifactName: ${{ coalesce(parameters.enablePublishBuildArtifacts.artifactName, '$(Agent.Os)_$(Agent.JobName)' ) }}
ArtifactName: $(Agent.Os)_$(Agent.JobName)
continueOnError: true
condition: always()
@@ -196,12 +184,10 @@ jobs:
testResultsFormat: 'xUnit'
testResultsFiles: '*.xml'
searchFolder: '$(Build.SourcesDirectory)/artifacts/TestResults/$(_BuildConfig)'
testRunTitle: ${{ parameters.testRunTitle }}
mergeTestResults: ${{ parameters.mergeTestResults }}
continueOnError: true
condition: always()
- ${{ if and(eq(parameters.enablePublishBuildAssets, true), ne(parameters.enablePublishUsingPipelines, 'true'), eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- ${{ if and(eq(parameters.enablePublishBuildAssets, true), ne(variables['_PublishUsingPipelines'], 'true'), eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: CopyFiles@2
displayName: Gather Asset Manifests
inputs:
@@ -209,7 +195,6 @@ jobs:
TargetFolder: '$(Build.StagingDirectory)/AssetManifests'
continueOnError: ${{ parameters.continueOnError }}
condition: and(succeeded(), eq(variables['_DotNetPublishToBlobFeed'], 'true'))
- task: PublishBuildArtifacts@1
displayName: Push Asset Manifests
inputs:
+2 -4
View File
@@ -5,7 +5,6 @@ parameters:
displayName: '' # optional -- display name for the job. Will use jobName if not passed
pool: '' # required -- name of the Build pool
container: '' # required -- name of the container
osGroup: '' # required -- operating system for the job
extraSetupParameters: '' # optional -- extra arguments to pass to the setup script
frameworks: ['netcoreapp3.0'] # optional -- list of frameworks to run against
continueOnError: 'false' # optional -- determines whether to continue the build if the step errors
@@ -45,13 +44,12 @@ jobs:
- HelixPreCommand: ''
- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- ${{ if eq( parameters.osGroup, 'Windows_NT') }}:
- ${{ if eq(variables['Agent.Os'], 'Windows_NT') }}:
- HelixPreCommand: 'set "PERFLAB_UPLOAD_TOKEN=$(PerfCommandUploadToken)"'
- IsInternal: -Internal
- ${{ if ne(parameters.osGroup, 'Windows_NT') }}:
- ${{ if ne(variables['Agent.Os'], 'Windows_NT') }}:
- HelixPreCommand: 'export PERFLAB_UPLOAD_TOKEN="$(PerfCommandUploadTokenLinux)"'
- IsInternal: --internal
- group: DotNet-HelixApi-Access
- group: dotnet-benchview
@@ -37,12 +37,6 @@ jobs:
- name: _BuildConfig
value: ${{ parameters.configuration }}
- group: Publish-Build-Assets
# Skip component governance and codesign validation for SDL. These jobs
# create no content.
- name: skipComponentGovernanceDetection
value: true
- name: runCodesignValidationInjection
value: false
steps:
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
@@ -53,10 +47,6 @@ jobs:
downloadPath: '$(Build.StagingDirectory)/Download'
condition: ${{ parameters.condition }}
continueOnError: ${{ parameters.continueOnError }}
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: NuGetAuthenticate@0
- task: PowerShell@2
displayName: Publish Build Assets
inputs:
@@ -69,7 +59,6 @@ jobs:
/p:Configuration=$(_BuildConfig)
condition: ${{ parameters.condition }}
continueOnError: ${{ parameters.continueOnError }}
- task: powershell@2
displayName: Create ReleaseConfigs Artifact
inputs:
@@ -78,14 +67,12 @@ jobs:
Add-Content -Path "$(Build.StagingDirectory)/ReleaseConfigs.txt" -Value $(BARBuildId)
Add-Content -Path "$(Build.StagingDirectory)/ReleaseConfigs.txt" -Value "$(DefaultChannels)"
Add-Content -Path "$(Build.StagingDirectory)/ReleaseConfigs.txt" -Value $(IsStableBuild)
- task: PublishBuildArtifacts@1
displayName: Publish ReleaseConfigs Artifact
inputs:
PathtoPublish: '$(Build.StagingDirectory)/ReleaseConfigs.txt'
PublishLocation: Container
ArtifactName: ReleaseConfigs
- ${{ if eq(parameters.enablePublishBuildArtifacts, 'true') }}:
- task: PublishBuildArtifacts@1
displayName: Publish Logs to VSTS
+46 -28
View File
@@ -1,10 +1,19 @@
parameters:
# See schema documentation in /Documentation/AzureDevOps/TemplateSchema.md
# Optional: 'true' if failures in job.yml job should not fail the job
continueOnError: false
# Optional: Enable installing Microbuild plugin
# if 'true', these "variables" must be specified in the variables object or as part of the queue matrix
# _TeamName - the name of your team
# _SignType - 'test' or 'real'
enableMicrobuild: false
# Optional: Include PublishBuildArtifacts task
enablePublishBuildArtifacts: false
# Optional: Enable publishing to the build asset registry
enablePublishBuildAssets: false
# Optional: Enable publishing using release pipelines
enablePublishUsingPipelines: false
@@ -14,9 +23,19 @@ parameters:
# Optional: Include toolset dependencies in the generated graph files
includeToolset: false
# Optional: Include PublishTestResults task
enablePublishTestResults: false
# Optional: enable sending telemetry
# if enabled then the 'helixRepo' parameter should also be specified
enableTelemetry: false
# Required: A collection of jobs to run - https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=vsts&tabs=schema#job
jobs: []
# Optional: define the helix repo for telemetry (example: 'dotnet/arcade')
helixRepo: ''
# Optional: Override automatically derived dependsOn value for "publish build assets" job
publishBuildAssetsDependsOn: ''
@@ -43,30 +62,29 @@ jobs:
name: ${{ job.job }}
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- ${{ if or(eq(parameters.enablePublishBuildAssets, true), eq(parameters.artifacts.publish.manifests, 'true'), ne(parameters.artifacts.publish.manifests, '')) }}:
- template: ../job/publish-build-assets.yml
parameters:
continueOnError: ${{ parameters.continueOnError }}
dependsOn:
- ${{ if ne(parameters.publishBuildAssetsDependsOn, '') }}:
- ${{ each job in parameters.publishBuildAssetsDependsOn }}:
- ${{ job.job }}
- ${{ if eq(parameters.publishBuildAssetsDependsOn, '') }}:
- ${{ each job in parameters.jobs }}:
- ${{ job.job }}
pool:
vmImage: vs2017-win2016
runAsPublic: ${{ parameters.runAsPublic }}
publishUsingPipelines: ${{ parameters.enablePublishUsingPipelines }}
enablePublishBuildArtifacts: ${{ parameters.enablePublishBuildArtifacts }}
- ${{ if eq(parameters.graphFileGeneration.enabled, true) }}:
- template: ../job/generate-graph-files.yml
parameters:
continueOnError: ${{ parameters.continueOnError }}
includeToolset: ${{ parameters.graphFileGeneration.includeToolset }}
dependsOn:
- Asset_Registry_Publish
pool:
vmImage: vs2017-win2016
- ${{ if and(eq(parameters.enablePublishBuildAssets, true), eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- template: ../job/publish-build-assets.yml
parameters:
continueOnError: ${{ parameters.continueOnError }}
dependsOn:
- ${{ if ne(parameters.publishBuildAssetsDependsOn, '') }}:
- ${{ each job in parameters.publishBuildAssetsDependsOn }}:
- ${{ job.job }}
- ${{ if eq(parameters.publishBuildAssetsDependsOn, '') }}:
- ${{ each job in parameters.jobs }}:
- ${{ job.job }}
pool:
vmImage: vs2017-win2016
runAsPublic: ${{ parameters.runAsPublic }}
publishUsingPipelines: ${{ parameters.enablePublishUsingPipelines }}
enablePublishBuildArtifacts: ${{ parameters.enablePublishBuildArtifacts }}
- ${{ if and(eq(parameters.graphFileGeneration.enabled, true), eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- template: ../job/generate-graph-files.yml
parameters:
continueOnError: ${{ parameters.continueOnError }}
includeToolset: ${{ parameters.graphFileGeneration.includeToolset }}
dependsOn:
- Asset_Registry_Publish
pool:
vmImage: vs2017-win2016
@@ -1,172 +0,0 @@
parameters:
artifactsPublishingAdditionalParameters: ''
dependsOn:
- Validate
publishInstallersAndChecksums: false
symbolPublishingAdditionalParameters: ''
stageName: ''
channelName: ''
channelId: ''
transportFeed: ''
shippingFeed: ''
symbolsFeed: ''
stages:
- stage: ${{ parameters.stageName }}
dependsOn: ${{ parameters.dependsOn }}
variables:
- template: ../common-variables.yml
displayName: ${{ parameters.channelName }} Publishing
jobs:
- template: ../setup-maestro-vars.yml
- job: publish_symbols
displayName: Symbol Publishing
dependsOn: setupMaestroVars
condition: or(contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], format('[{0}]', ${{ parameters.channelId }} )), eq(dependencies.setupMaestroVars.outputs['setReleaseVars.PromoteToMaestroChannelId'], ${{ parameters.channelId }}))
variables:
- group: DotNet-Symbol-Server-Pats
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Build Assets
continueOnError: true
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
downloadType: 'specific'
itemPattern: |
PdbArtifacts/**
BlobArtifacts/**
downloadPath: '$(Build.ArtifactStagingDirectory)'
# This is necessary whenever we want to publish/restore to an AzDO private feed
# Since sdk-task.ps1 tries to restore packages we need to do this authentication here
# otherwise it'll complain about accessing a private feed.
- task: NuGetAuthenticate@0
displayName: 'Authenticate to AzDO Feeds'
- task: PowerShell@2
displayName: Enable cross-org publishing
inputs:
filePath: eng\common\enable-cross-org-publishing.ps1
arguments: -token $(dn-bot-dnceng-artifact-feeds-rw)
- task: PowerShell@2
displayName: Publish
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishToSymbolServers -restore -msbuildEngine dotnet
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:PDBArtifactsDirectory='$(Build.ArtifactStagingDirectory)/PDBArtifacts/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:SymbolPublishingExclusionsFile='$(Build.SourcesDirectory)/eng/SymbolPublishingExclusionsFile.txt'
/p:Configuration=Release
/p:PublishToMSDL=false
${{ parameters.symbolPublishingAdditionalParameters }}
- template: ../../steps/publish-logs.yml
parameters:
StageLabel: '${{ parameters.stageName }}'
JobLabel: 'SymbolPublishing'
- job: publish_assets
displayName: Publish Assets
dependsOn: setupMaestroVars
timeoutInMinutes: 120
variables:
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
condition: or(contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], format('[{0}]', ${{ parameters.channelId }} )), eq(dependencies.setupMaestroVars.outputs['setReleaseVars.PromoteToMaestroChannelId'], ${{ parameters.channelId }}))
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Build Assets
continueOnError: true
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
downloadType: 'specific'
itemPattern: |
PackageArtifacts/**
BlobArtifacts/**
AssetManifests/**
downloadPath: '$(Build.ArtifactStagingDirectory)'
- task: NuGetToolInstaller@1
displayName: 'Install NuGet.exe'
# This is necessary whenever we want to publish/restore to an AzDO private feed
- task: NuGetAuthenticate@0
displayName: 'Authenticate to AzDO Feeds'
- task: PowerShell@2
displayName: Enable cross-org publishing
inputs:
filePath: eng\common\enable-cross-org-publishing.ps1
arguments: -token $(dn-bot-dnceng-artifact-feeds-rw)
- task: PowerShell@2
displayName: Publish Assets
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:NugetPath=$(NuGetExeToolPath)
/p:AzdoTargetFeedPAT='$(dn-bot-dnceng-universal-packages-rw)'
/p:AzureStorageTargetFeedPAT='$(dotnetfeed-storage-access-key-1)'
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)/PackageArtifacts/'
/p:Configuration=Release
/p:PublishInstallersAndChecksums=true
/p:ChecksumsTargetStaticFeed=$(InternalChecksumsBlobFeedUrl)
/p:ChecksumsAzureAccountKey=$(InternalChecksumsBlobFeedKey)
/p:InstallersTargetStaticFeed=$(InternalInstallersBlobFeedUrl)
/p:InstallersAzureAccountKey=$(InternalInstallersBlobFeedKey)
/p:AzureDevOpsStaticShippingFeed='${{ parameters.shippingFeed }}'
/p:AzureDevOpsStaticShippingFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:AzureDevOpsStaticTransportFeed='${{ parameters.transportFeed }}'
/p:AzureDevOpsStaticTransportFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:AzureDevOpsStaticSymbolsFeed='${{ parameters.symbolsFeed }}'
/p:AzureDevOpsStaticSymbolsFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:PublishToMSDL=false
${{ parameters.artifactsPublishingAdditionalParameters }}
- template: ../../steps/publish-logs.yml
parameters:
StageLabel: '${{ parameters.stageName }}'
JobLabel: 'AssetsPublishing'
- template: ../../steps/add-build-to-channel.yml
parameters:
ChannelId: ${{ parameters.channelId }}
@@ -1,178 +0,0 @@
parameters:
artifactsPublishingAdditionalParameters: ''
dependsOn:
- Validate
publishInstallersAndChecksums: false
symbolPublishingAdditionalParameters: ''
stageName: ''
channelName: ''
channelId: ''
transportFeed: ''
shippingFeed: ''
symbolsFeed: ''
# If the channel name is empty, no links will be generated
akaMSChannelName: ''
stages:
- stage: ${{ parameters.stageName }}
dependsOn: ${{ parameters.dependsOn }}
variables:
- template: ../common-variables.yml
displayName: ${{ parameters.channelName }} Publishing
jobs:
- template: ../setup-maestro-vars.yml
- job: publish_symbols
displayName: Symbol Publishing
dependsOn: setupMaestroVars
condition: or(contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], format('[{0}]', ${{ parameters.channelId }} )), eq(dependencies.setupMaestroVars.outputs['setReleaseVars.PromoteToMaestroChannelId'], ${{ parameters.channelId }}))
variables:
- group: DotNet-Symbol-Server-Pats
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Build Assets
continueOnError: true
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
downloadType: 'specific'
itemPattern: |
PdbArtifacts/**
BlobArtifacts/**
downloadPath: '$(Build.ArtifactStagingDirectory)'
# This is necessary whenever we want to publish/restore to an AzDO private feed
# Since sdk-task.ps1 tries to restore packages we need to do this authentication here
# otherwise it'll complain about accessing a private feed.
- task: NuGetAuthenticate@0
displayName: 'Authenticate to AzDO Feeds'
- task: PowerShell@2
displayName: Enable cross-org publishing
inputs:
filePath: eng\common\enable-cross-org-publishing.ps1
arguments: -token $(dn-bot-dnceng-artifact-feeds-rw)
- task: PowerShell@2
displayName: Publish
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishToSymbolServers -restore -msbuildEngine dotnet
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:PDBArtifactsDirectory='$(Build.ArtifactStagingDirectory)/PDBArtifacts/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:SymbolPublishingExclusionsFile='$(Build.SourcesDirectory)/eng/SymbolPublishingExclusionsFile.txt'
/p:Configuration=Release
${{ parameters.symbolPublishingAdditionalParameters }}
- template: ../../steps/publish-logs.yml
parameters:
StageLabel: '${{ parameters.stageName }}'
JobLabel: 'SymbolPublishing'
- job: publish_assets
displayName: Publish Assets
dependsOn: setupMaestroVars
timeoutInMinutes: 120
variables:
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
- name: ArtifactsCategory
value: ${{ coalesce(variables._DotNetArtifactsCategory, '.NETCore') }}
condition: or(contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], format('[{0}]', ${{ parameters.channelId }} )), eq(dependencies.setupMaestroVars.outputs['setReleaseVars.PromoteToMaestroChannelId'], ${{ parameters.channelId }}))
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Build Assets
continueOnError: true
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
downloadType: 'specific'
itemPattern: |
PackageArtifacts/**
BlobArtifacts/**
AssetManifests/**
downloadPath: '$(Build.ArtifactStagingDirectory)'
- task: NuGetToolInstaller@1
displayName: 'Install NuGet.exe'
# This is necessary whenever we want to publish/restore to an AzDO private feed
- task: NuGetAuthenticate@0
displayName: 'Authenticate to AzDO Feeds'
- task: PowerShell@2
displayName: Enable cross-org publishing
inputs:
filePath: eng\common\enable-cross-org-publishing.ps1
arguments: -token $(dn-bot-dnceng-artifact-feeds-rw)
- task: PowerShell@2
displayName: Publish Assets
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:ArtifactsCategory=$(ArtifactsCategory)
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:NugetPath=$(NuGetExeToolPath)
/p:AzdoTargetFeedPAT='$(dn-bot-dnceng-universal-packages-rw)'
/p:AzureStorageTargetFeedPAT='$(dotnetfeed-storage-access-key-1)'
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)/PackageArtifacts/'
/p:Configuration=Release
/p:PublishInstallersAndChecksums=${{ parameters.publishInstallersAndChecksums }}
/p:InstallersTargetStaticFeed=$(InstallersBlobFeedUrl)
/p:InstallersAzureAccountKey=$(dotnetcli-storage-key)
/p:ChecksumsTargetStaticFeed=$(ChecksumsBlobFeedUrl)
/p:ChecksumsAzureAccountKey=$(dotnetclichecksums-storage-key)
/p:AzureDevOpsStaticShippingFeed='${{ parameters.shippingFeed }}'
/p:AzureDevOpsStaticShippingFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:AzureDevOpsStaticTransportFeed='${{ parameters.transportFeed }}'
/p:AzureDevOpsStaticTransportFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:AzureDevOpsStaticSymbolsFeed='${{ parameters.symbolsFeed }}'
/p:AzureDevOpsStaticSymbolsFeedKey='$(dn-bot-dnceng-artifact-feeds-rw)'
/p:LatestLinkShortUrlPrefix=dotnet/'${{ parameters.akaMSChannelName }}'
/p:AkaMSClientId=$(akams-client-id)
/p:AkaMSClientSecret=$(akams-client-secret)
${{ parameters.artifactsPublishingAdditionalParameters }}
- template: ../../steps/publish-logs.yml
parameters:
StageLabel: '${{ parameters.stageName }}'
JobLabel: 'AssetsPublishing'
- template: ../../steps/add-build-to-channel.yml
parameters:
ChannelId: ${{ parameters.channelId }}
@@ -0,0 +1,147 @@
parameters:
enableSymbolValidation: true
stages:
- stage: IS_Publish
dependsOn: validate
variables:
- template: ../common-variables.yml
displayName: Internal Servicing
jobs:
- template: ../setup-maestro-vars.yml
- job:
displayName: Symbol Publishing
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.InternalServicing_30_Channel_Id)
variables:
- group: DotNet-Symbol-Server-Pats
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Artifacts
inputs:
downloadType: specific files
matchingPattern: "*Artifacts*"
- task: PowerShell@2
displayName: Publish
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishToSymbolServers -restore -msbuildEngine dotnet
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:PDBArtifactsDirectory='$(Build.ArtifactStagingDirectory)/PDBArtifacts/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:Configuration=Release
- job: publish_assets
displayName: Publish Assets
dependsOn: setupMaestroVars
variables:
- group: DotNet-Blob-Feed
- group: AzureDevOps-Artifact-Feeds-Pats
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.InternalServicing_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: current
artifactName: BlobArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Asset Manifests
inputs:
buildType: current
artifactName: AssetManifests
- task: PowerShell@2
displayName: Add Assets Location
env:
AZURE_DEVOPS_EXT_PAT: $(dn-bot-dnceng-unviersal-packages-rw)
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:ChannelId=$(InternalServicing_30_Channel_Id)
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:AzureStorageAccountName=$(ProxyBackedFeedsAccountName)
/p:AzureStorageAccountKey=$(dotnetfeed-storage-access-key-1)
/p:AzureDevOpsFeedsBaseUrl=$(dotnetfeed-internal-private-feed-url)
/p:StaticInternalFeed=$(dotnetfeed-internal-nonstable-feed-url)
/p:NugetPath=$(Agent.BuildDirectory)\Nuget\NuGet.exe
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)\BlobArtifacts'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)\PackageArtifacts'
/p:Configuration=Release
- task: NuGetCommand@2
displayName: Publish Packages to AzDO Feed
condition: contains(variables['TargetAzDOFeed'], 'pkgs.visualstudio.com')
inputs:
command: push
vstsFeed: $(AzDoFeedName)
packagesToPush: $(Build.ArtifactStagingDirectory)\PackageArtifacts\*.nupkg
publishVstsFeed: $(AzDoFeedName)
- task: PowerShell@2
displayName: Publish Blobs to AzDO Feed
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/publish-blobs-to-azdo.ps1
arguments: -FeedName $(AzDoFeedName)
-SourceFolderCollection $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-PersonalAccessToken $(dn-bot-dnceng-unviersal-packages-rw)
enabled: false
- template: ../trigger-subscription.yml
parameters:
ChannelId: ${{ variables.InternalServicing_30_Channel_Id }}
- stage: IS_PublishValidation
displayName: Publish Validation
variables:
- template: ../common-variables.yml
jobs:
- template: ../setup-maestro-vars.yml
- ${{ if eq(parameters.enableSymbolValidation, 'true') }}:
- job:
displayName: Symbol Availability
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.InternalServicing_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: PowerShell@2
displayName: Check Symbol Availability
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/symbols-validation.ps1
arguments: -InputPath $(Build.ArtifactStagingDirectory)/PackageArtifacts/ -ExtractPath $(Agent.BuildDirectory)/Temp/ -DotnetSymbolVersion $(SymbolToolVersion)
- template: ../promote-build.yml
parameters:
ChannelId: ${{ variables.InternalServicing_30_Channel_Id }}
@@ -0,0 +1,148 @@
parameters:
enableSymbolValidation: true
stages:
- stage: Publish
dependsOn: validate
variables:
- template: ../common-variables.yml
displayName: Developer Channel
jobs:
- template: ../setup-maestro-vars.yml
- job:
displayName: Symbol Publishing
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicDevRelease_30_Channel_Id)
variables:
- group: DotNet-Symbol-Server-Pats
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Artifacts
inputs:
downloadType: specific files
matchingPattern: "*Artifacts*"
- task: PowerShell@2
displayName: Publish
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishToSymbolServers -restore -msbuildEngine dotnet
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:PDBArtifactsDirectory='$(Build.ArtifactStagingDirectory)/PDBArtifacts/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:Configuration=Release
- job:
displayName: Publish Assets
dependsOn: setupMaestroVars
variables:
- group: DotNet-Blob-Feed
- group: AzureDevOps-Artifact-Feeds-Pats
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicDevRelease_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: current
artifactName: BlobArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Asset Manifests
inputs:
buildType: current
artifactName: AssetManifests
- task: PowerShell@2
displayName: Add Assets Location
env:
AZURE_DEVOPS_EXT_PAT: $(dn-bot-dnceng-unviersal-packages-rw)
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:ChannelId=$(PublicDevRelease_30_Channel_Id)
/p:ArtifactsCategory=$(_DotNetArtifactsCategory)
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:NugetPath=$(Agent.BuildDirectory)\Nuget\NuGet.exe
/p:AzdoTargetFeedPAT='$(dn-bot-dnceng-unviersal-packages-rw)'
/p:TargetFeedPAT='$(dn-bot-dnceng-unviersal-packages-rw)'
/p:AzureStorageTargetFeedPAT='$(dotnetfeed-storage-access-key-1)'
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)/PackageArtifacts/'
/p:Configuration=Release
- task: NuGetCommand@2
displayName: Publish Packages to AzDO Feed
condition: contains(variables['TargetAzDOFeed'], 'pkgs.visualstudio.com')
inputs:
command: push
vstsFeed: $(AzDoFeedName)
packagesToPush: $(Build.ArtifactStagingDirectory)\PackageArtifacts\*.nupkg
publishVstsFeed: $(AzDoFeedName)
- task: PowerShell@2
displayName: Publish Blobs to AzDO Feed
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/publish-blobs-to-azdo.ps1
arguments: -FeedName $(AzDoFeedName)
-SourceFolderCollection $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-PersonalAccessToken $(dn-bot-dnceng-unviersal-packages-rw)
enabled: false
- stage: PublishValidation
displayName: Publish Validation
variables:
- template: ../common-variables.yml
jobs:
- template: ../setup-maestro-vars.yml
- ${{ if eq(parameters.enableSymbolValidation, 'true') }}:
- job:
displayName: Symbol Availability
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicDevRelease_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: PowerShell@2
displayName: Check Symbol Availability
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/symbols-validation.ps1
arguments: -InputPath $(Build.ArtifactStagingDirectory)/PackageArtifacts/ -ExtractPath $(Agent.BuildDirectory)/Temp/ -DotnetSymbolVersion $(SymbolToolVersion)
- template: ../darc-gather-drop.yml
parameters:
ChannelId: ${{ variables.PublicDevRelease_30_Channel_Id }}
- template: ../promote-build.yml
parameters:
ChannelId: ${{ variables.PublicDevRelease_30_Channel_Id }}
@@ -0,0 +1,147 @@
parameters:
enableSymbolValidation: true
stages:
- stage: PubRel_Publish
dependsOn: validate
variables:
- template: ../common-variables.yml
displayName: Public Release
jobs:
- template: ../setup-maestro-vars.yml
- job:
displayName: Symbol Publishing
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicRelease_30_Channel_Id)
variables:
- group: DotNet-Symbol-Server-Pats
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Artifacts
inputs:
downloadType: specific files
matchingPattern: "*Artifacts*"
- task: PowerShell@2
displayName: Publish
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishToSymbolServers -restore -msbuildEngine dotnet
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:PDBArtifactsDirectory='$(Build.ArtifactStagingDirectory)/PDBArtifacts/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)/BlobArtifacts/'
/p:Configuration=Release
- job: publish_assets
displayName: Publish Assets
dependsOn: setupMaestroVars
variables:
- group: DotNet-Blob-Feed
- group: AzureDevOps-Artifact-Feeds-Pats
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicRelease_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: current
artifactName: BlobArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Asset Manifests
inputs:
buildType: current
artifactName: AssetManifests
- task: PowerShell@2
displayName: Publish
env:
AZURE_DEVOPS_EXT_PAT: $(dn-bot-dnceng-unviersal-packages-rw)
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:ChannelId=$(PublicRelease_30_Channel_Id)
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:AzureStorageAccountName=$(ProxyBackedFeedsAccountName)
/p:AzureStorageAccountKey=$(dotnetfeed-storage-access-key-1)
/p:AzureDevOpsFeedsBaseUrl=$(dotnetfeed-internal-private-feed-url)
/p:StaticInternalFeed=$(dotnetfeed-internal-nonstable-feed-url)
/p:NugetPath=$(Agent.BuildDirectory)\Nuget\NuGet.exe
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)\BlobArtifacts'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)\PackageArtifacts'
/p:Configuration=Release
- task: NuGetCommand@2
displayName: Publish Packages to AzDO Feed
condition: contains(variables['TargetAzDOFeed'], 'pkgs.visualstudio.com')
inputs:
command: push
vstsFeed: $(AzDoFeedName)
packagesToPush: $(Build.ArtifactStagingDirectory)\PackageArtifacts\*.nupkg
publishVstsFeed: $(AzDoFeedName)
- task: PowerShell@2
displayName: Publish Blobs to AzDO Feed
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/publish-blobs-to-azdo.ps1
arguments: -FeedName $(AzDoFeedName)
-SourceFolderCollection $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-PersonalAccessToken $(dn-bot-dnceng-unviersal-packages-rw)
enabled: false
- template: ../trigger-subscription.yml
parameters:
ChannelId: ${{ variables.PublicRelease_30_Channel_Id }}
- stage: PubRel_PublishValidation
displayName: Publish Validation
variables:
- template: ../common-variables.yml
jobs:
- template: ../setup-maestro-vars.yml
- ${{ if eq(parameters.enableSymbolValidation, 'true') }}:
- job:
displayName: Symbol Availability
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicRelease_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: PowerShell@2
displayName: Check Symbol Availability
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/symbols-validation.ps1
arguments: -InputPath $(Build.ArtifactStagingDirectory)/PackageArtifacts/ -ExtractPath $(Agent.BuildDirectory)/Temp/ -DotnetSymbolVersion $(SymbolToolVersion)
- template: ../promote-build.yml
parameters:
ChannelId: ${{ variables.PublicRelease_30_Channel_Id }}
@@ -0,0 +1,99 @@
stages:
- stage: PVR_Publish
dependsOn: validate
variables:
- template: ../common-variables.yml
displayName: Validation Channel
jobs:
- template: ../setup-maestro-vars.yml
- job:
displayName: Publish Assets
dependsOn: setupMaestroVars
variables:
- group: DotNet-Blob-Feed
- group: AzureDevOps-Artifact-Feeds-Pats
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: IsStableBuild
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.IsStableBuild'] ]
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], variables.PublicValidationRelease_30_Channel_Id)
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: current
artifactName: BlobArtifacts
- task: DownloadBuildArtifacts@0
displayName: Download Asset Manifests
inputs:
buildType: current
artifactName: AssetManifests
- task: PowerShell@2
displayName: Add Assets Location
env:
AZURE_DEVOPS_EXT_PAT: $(dn-bot-dnceng-unviersal-packages-rw)
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task PublishArtifactsInManifest -restore -msbuildEngine dotnet
/p:ChannelId=$(PublicValidationRelease_30_Channel_Id)
/p:ArtifactsCategory=$(_DotNetValidationArtifactsCategory)
/p:IsStableBuild=$(IsStableBuild)
/p:IsInternalBuild=$(IsInternalBuild)
/p:RepositoryName=$(Build.Repository.Name)
/p:CommitSha=$(Build.SourceVersion)
/p:NugetPath=$(Agent.BuildDirectory)\Nuget\NuGet.exe
/p:AzdoTargetFeedPAT='$(dn-bot-dnceng-unviersal-packages-rw)'
/p:TargetFeedPAT='$(dn-bot-dnceng-unviersal-packages-rw)'
/p:AzureStorageTargetFeedPAT='$(dotnetfeed-storage-access-key-1)'
/p:BARBuildId=$(BARBuildId)
/p:MaestroApiEndpoint='$(MaestroApiEndPoint)'
/p:BuildAssetRegistryToken='$(MaestroApiAccessToken)'
/p:ManifestsBasePath='$(Build.ArtifactStagingDirectory)/AssetManifests/'
/p:BlobBasePath='$(Build.ArtifactStagingDirectory)\BlobArtifacts'
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)\PackageArtifacts'
/p:Configuration=Release
- task: NuGetCommand@2
displayName: Publish Packages to AzDO Feed
condition: contains(variables['TargetAzDOFeed'], 'pkgs.visualstudio.com')
inputs:
command: push
vstsFeed: $(AzDoFeedName)
packagesToPush: $(Build.ArtifactStagingDirectory)\PackageArtifacts\*.nupkg
publishVstsFeed: $(AzDoFeedName)
- task: PowerShell@2
displayName: Publish Blobs to AzDO Feed
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/publish-blobs-to-azdo.ps1
arguments: -FeedName $(AzDoFeedName)
-SourceFolderCollection $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-PersonalAccessToken $(dn-bot-dnceng-unviersal-packages-rw)
enabled: false
- stage: PVR_PublishValidation
displayName: Publish Validation
variables:
- template: ../common-variables.yml
jobs:
- template: ../setup-maestro-vars.yml
- template: ../darc-gather-drop.yml
parameters:
ChannelId: ${{ variables.PublicValidationRelease_30_Channel_Id }}
- template: ../promote-build.yml
parameters:
ChannelId: ${{ variables.PublicValidationRelease_30_Channel_Id }}
@@ -1,34 +1,14 @@
variables:
- group: AzureDevOps-Artifact-Feeds-Pats
- group: DotNet-Blob-Feed
- group: DotNet-DotNetCli-Storage
- group: DotNet-MSRC-Storage
- group: Publish-Build-Assets
# .NET Core 3.1 Dev
- name: PublicDevRelease_31_Channel_Id
value: 128
# .NET 5 Dev
- name: Net_5_Dev_Channel_Id
value: 131
# .NET Core 3 Dev
- name: PublicDevRelease_30_Channel_Id
value: 3
# .NET Eng - Validation
- name: Net_Eng_Validation_Channel_Id
# .NET Tools - Validation
- name: PublicValidationRelease_30_Channel_Id
value: 9
# .NET Eng - Latest
- name: Net_Eng_Latest_Channel_Id
value: 2
# .NET 3 Eng - Validation
- name: NET_3_Eng_Validation_Channel_Id
value: 390
# .NET 3 Eng
- name: NetCore_3_Tools_Channel_Id
value: 344
# .NET Core 3.0 Internal Servicing
- name: InternalServicing_30_Channel_Id
value: 184
@@ -37,26 +17,14 @@ variables:
- name: PublicRelease_30_Channel_Id
value: 19
# .NET Core 3.1 Release
- name: PublicRelease_31_Channel_Id
value: 129
# General Testing
- name: GeneralTesting_Channel_Id
value: 529
# .NET Core 3.1 Blazor Features
- name: NetCore_31_Blazor_Features_Channel_Id
value: 531
# .NET Core Experimental
- name: NetCore_Experimental_Channel_Id
value: 562
# Whether the build is internal or not
- name: IsInternalBuild
value: ${{ and(ne(variables['System.TeamProject'], 'public'), contains(variables['Build.SourceBranch'], 'internal')) }}
# Storage account name for proxy-backed feeds
- name: ProxyBackedFeedsAccountName
value: dotnetfeed
# Default Maestro++ API Endpoint and API Version
- name: MaestroApiEndPoint
value: "https://maestro-prod.westus2.cloudapp.azure.com"
@@ -69,31 +37,3 @@ variables:
value: 3.0.0
- name: SymbolToolVersion
value: 1.0.1
# Feed Configurations
# These should include the suffix "/index.json"
# Default locations for Installers and checksums
# Public Locations
- name: ChecksumsBlobFeedUrl
value: https://dotnetclichecksums.blob.core.windows.net/dotnet/index.json
- name: InstallersBlobFeedUrl
value: https://dotnetcli.blob.core.windows.net/dotnet/index.json
# Private Locations
- name: InternalChecksumsBlobFeedUrl
value: https://dotnetclichecksumsmsrc.blob.core.windows.net/dotnet/index.json
- name: InternalChecksumsBlobFeedKey
value: $(dotnetclichecksumsmsrc-storage-key)
- name: InternalInstallersBlobFeedUrl
value: https://dotnetclimsrc.blob.core.windows.net/dotnet/index.json
- name: InternalInstallersBlobFeedKey
value: $(dotnetclimsrc-access-key)
# Skip component governance and codesign validation for SDL. These jobs
# create no content.
- name: skipComponentGovernanceDetection
value: true
- name: runCodesignValidationInjection
value: false
@@ -0,0 +1,23 @@
parameters:
ChannelId: 0
jobs:
- job: gatherDrop
displayName: Gather Drop
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], ${{ parameters.ChannelId }})
variables:
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: PowerShell@2
displayName: Darc gather-drop
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/darc-gather-drop.ps1
arguments: -BarBuildId $(BARBuildId)
-DropLocation $(Agent.BuildDirectory)/Temp/Drop/
-MaestroApiAccessToken $(MaestroApiAccessToken)
-MaestroApiEndPoint $(MaestroApiEndPoint)
-MaestroApiVersion $(MaestroApiVersion)
+71 -359
View File
@@ -1,382 +1,94 @@
parameters:
enableSourceLinkValidation: false
enableSourceLinkValidation: true
enableSigningValidation: true
enableSymbolValidation: false
enableSymbolValidation: true
enableNugetValidation: true
publishInstallersAndChecksums: false
SDLValidationParameters:
enable: false
continueOnError: false
params: ''
artifactNames: ''
# These parameters let the user customize the call to sdk-task.ps1 for publishing
# symbols & general artifacts as well as for signing validation
symbolPublishingAdditionalParameters: ''
artifactsPublishingAdditionalParameters: ''
signingValidationAdditionalParameters: ''
# Which stages should finish execution before post-build stages start
validateDependsOn:
- build
publishDependsOn:
- Validate
dependsOn: [build]
# Channel ID's instantiated in this file.
# When adding a new channel implementation the call to `check-channel-consistency.ps1`
# needs to be updated with the new channel ID
NetEngLatestChannelId: 2
NetEngValidationChannelId: 9
NetDev5ChannelId: 131
GeneralTestingChannelId: 529
NETCoreToolingDevChannelId: 548
NETCoreToolingReleaseChannelId: 549
NETInternalToolingChannelId: 551
NETCoreExperimentalChannelId: 562
NetEngServicesIntChannelId: 678
NetEngServicesProdChannelId: 679
Net5Preview2ChannelId: 738
Net5Preview3ChannelId: 739
NetCoreSDK313xxChannelId: 759
NetCoreSDK313xxInternalChannelId: 760
stages:
- stage: Validate
dependsOn: ${{ parameters.validateDependsOn }}
- stage: validate
dependsOn: ${{ parameters.dependsOn }}
displayName: Validate
variables:
- template: common-variables.yml
jobs:
- template: setup-maestro-vars.yml
- ${{ if eq(parameters.enableNugetValidation, 'true') }}:
- job:
displayName: NuGet Validation
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- job:
displayName: Post-build Checks
dependsOn: setupMaestroVars
variables:
- name: InitialChannels
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'] ]
- name: PromoteToMaestroChannelId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.PromoteToMaestroChannelId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: PowerShell@2
displayName: Maestro Channels Consistency
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/check-channel-consistency.ps1
arguments: -PromoteToChannels "$(InitialChannels)[$(PromoteToMaestroChannelId)]"
-AvailableChannelIds ${{parameters.NetEngLatestChannelId}},${{parameters.NetEngValidationChannelId}},${{parameters.NetDev5ChannelId}},${{parameters.GeneralTestingChannelId}},${{parameters.NETCoreToolingDevChannelId}},${{parameters.NETCoreToolingReleaseChannelId}},${{parameters.NETInternalToolingChannelId}},${{parameters.NETCoreExperimentalChannelId}},${{parameters.NetEngServicesIntChannelId}},${{parameters.NetEngServicesProdChannelId}},${{parameters.Net5Preview2ChannelId}},${{parameters.Net5Preview3ChannelId}},${{parameters.NetCoreSDK313xxChannelId}},${{parameters.NetCoreSDK313xxInternalChannelId}}
- task: PowerShell@2
displayName: Validate
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/nuget-validation.ps1
arguments: -PackagesPath $(Build.ArtifactStagingDirectory)/PackageArtifacts/
-ToolDestinationPath $(Agent.BuildDirectory)/Extract/
- job:
displayName: NuGet Validation
dependsOn: setupMaestroVars
condition: eq( ${{ parameters.enableNugetValidation }}, 'true')
pool:
vmImage: 'windows-2019'
variables:
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
artifactName: PackageArtifacts
- ${{ if eq(parameters.enableSigningValidation, 'true') }}:
- job:
displayName: Signing Validation
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: current
artifactName: PackageArtifacts
- task: PowerShell@2
displayName: Validate
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/nuget-validation.ps1
arguments: -PackagesPath $(Build.ArtifactStagingDirectory)/PackageArtifacts/
-ToolDestinationPath $(Agent.BuildDirectory)/Extract/
- task: PowerShell@2
displayName: Validate
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task SigningValidation -restore -msbuildEngine dotnet
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)/PackageArtifacts'
/p:Configuration=Release
- job:
displayName: Signing Validation
dependsOn: setupMaestroVars
condition: eq( ${{ parameters.enableSigningValidation }}, 'true')
variables:
- template: common-variables.yml
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Package Artifacts
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
artifactName: PackageArtifacts
- ${{ if eq(parameters.enableSourceLinkValidation, 'true') }}:
- job:
displayName: SourceLink Validation
variables:
- template: common-variables.yml
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: current
artifactName: BlobArtifacts
# This is necessary whenever we want to publish/restore to an AzDO private feed
# Since sdk-task.ps1 tries to restore packages we need to do this authentication here
# otherwise it'll complain about accessing a private feed.
- task: NuGetAuthenticate@0
displayName: 'Authenticate to AzDO Feeds'
- task: PowerShell@2
displayName: Validate
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/sourcelink-validation.ps1
arguments: -InputPath $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-ExtractPath $(Agent.BuildDirectory)/Extract/
-GHRepoName $(Build.Repository.Name)
-GHCommit $(Build.SourceVersion)
-SourcelinkCliVersion $(SourceLinkCLIVersion)
- task: PowerShell@2
displayName: Enable cross-org publishing
inputs:
filePath: eng\common\enable-cross-org-publishing.ps1
arguments: -token $(dn-bot-dnceng-artifact-feeds-rw)
- ${{ if eq(parameters.SDLValidationParameters.enable, 'true') }}:
- template: /eng/common/templates/job/execute-sdl.yml
parameters:
additionalParameters: ${{ parameters.SDLValidationParameters.params }}
- task: PowerShell@2
displayName: Validate
inputs:
filePath: eng\common\sdk-task.ps1
arguments: -task SigningValidation -restore -msbuildEngine dotnet
/p:PackageBasePath='$(Build.ArtifactStagingDirectory)/PackageArtifacts'
/p:SignCheckExclusionsFile='$(Build.SourcesDirectory)/eng/SignCheckExclusionsFile.txt'
${{ parameters.signingValidationAdditionalParameters }}
- template: ../steps/publish-logs.yml
parameters:
StageLabel: 'Validation'
JobLabel: 'Signing'
- job:
displayName: SourceLink Validation
dependsOn: setupMaestroVars
condition: eq( ${{ parameters.enableSourceLinkValidation }}, 'true')
variables:
- template: common-variables.yml
- name: AzDOProjectName
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOProjectName'] ]
- name: AzDOPipelineId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOPipelineId'] ]
- name: AzDOBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.AzDOBuildId'] ]
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadBuildArtifacts@0
displayName: Download Blob Artifacts
inputs:
buildType: specific
buildVersionToDownload: specific
project: $(AzDOProjectName)
pipeline: $(AzDOPipelineId)
buildId: $(AzDOBuildId)
artifactName: BlobArtifacts
- task: PowerShell@2
displayName: Validate
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/sourcelink-validation.ps1
arguments: -InputPath $(Build.ArtifactStagingDirectory)/BlobArtifacts/
-ExtractPath $(Agent.BuildDirectory)/Extract/
-GHRepoName $(Build.Repository.Name)
-GHCommit $(Build.SourceVersion)
-SourcelinkCliVersion $(SourceLinkCLIVersion)
continueOnError: true
- template: /eng/common/templates/job/execute-sdl.yml
parameters:
enable: ${{ parameters.SDLValidationParameters.enable }}
dependsOn: setupMaestroVars
additionalParameters: ${{ parameters.SDLValidationParameters.params }}
continueOnError: ${{ parameters.SDLValidationParameters.continueOnError }}
artifactNames: ${{ parameters.SDLValidationParameters.artifactNames }}
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
- template: \eng\common\templates\post-build\channels\public-dev-release.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NetCore_Dev5_Publish'
channelName: '.NET 5 Dev'
akaMSChannelName: 'net5/dev'
channelId: ${{ parameters.NetDev5ChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-transport/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-symbols/nuget/v3/index.json'
enableSymbolValidation: ${{ parameters.enableSymbolValidation }}
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net5_Preview2_Publish'
channelName: '.NET 5 Preview 2'
akaMSChannelName: 'net5/preview2'
channelId: ${{ parameters.Net5Preview2ChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-transport/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\public-validation-release.yml
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net5_Preview3_Publish'
channelName: '.NET 5 Preview 3'
akaMSChannelName: 'net5/preview3'
channelId: ${{ parameters.Net5Preview3ChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-transport/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\public-release.yml
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net_Eng_Latest_Publish'
channelName: '.NET Eng - Latest'
akaMSChannelName: 'eng/daily'
channelId: ${{ parameters.NetEngLatestChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net_Eng_Validation_Publish'
channelName: '.NET Eng - Validation'
akaMSChannelName: 'eng/validation'
channelId: ${{ parameters.NetEngValidationChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'General_Testing_Publish'
channelName: 'General Testing'
akaMSChannelName: 'generaltesting'
channelId: ${{ parameters.GeneralTestingChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/general-testing/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/general-testing/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/general-testing-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NETCore_Tooling_Dev_Publishing'
channelName: '.NET Core Tooling Dev'
channelId: ${{ parameters.NETCoreToolingDevChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NETCore_Tooling_Release_Publishing'
channelName: '.NET Core Tooling Release'
channelId: ${{ parameters.NETCoreToolingReleaseChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-internal-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NET_Internal_Tooling_Publishing'
channelName: '.NET Internal Tooling'
channelId: ${{ parameters.NETInternalToolingChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/internal/_packaging/dotnet-tools-internal/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/internal/_packaging/dotnet-tools-internal/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/internal/_packaging/dotnet-tools-internal-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NETCore_Experimental_Publishing'
channelName: '.NET Core Experimental'
channelId: ${{ parameters.NETCoreExperimentalChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-experimental/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-experimental/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-experimental-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net_Eng_Services_Int_Publish'
channelName: '.NET Eng Services - Int'
channelId: ${{ parameters.NetEngServicesIntChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'Net_Eng_Services_Prod_Publish'
channelName: '.NET Eng Services - Prod'
channelId: ${{ parameters.NetEngServicesProdChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-public-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NETCore_SDK_313xx_Publishing'
channelName: '.NET Core SDK 3.1.3xx'
channelId: ${{ parameters.NetCoreSDK313xxChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet3.1-transport/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet3.1/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet3.1-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\generic-internal-channel.yml
parameters:
artifactsPublishingAdditionalParameters: ${{ parameters.artifactsPublishingAdditionalParameters }}
dependsOn: ${{ parameters.publishDependsOn }}
publishInstallersAndChecksums: ${{ parameters.publishInstallersAndChecksums }}
symbolPublishingAdditionalParameters: ${{ parameters.symbolPublishingAdditionalParameters }}
stageName: 'NETCore_SDK_313xx_Internal_Publishing'
channelName: '.NET Core SDK 3.1.3xx Internal'
channelId: ${{ parameters.NetCoreSDK313xxInternalChannelId }}
transportFeed: 'https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal-transport/nuget/v3/index.json'
shippingFeed: 'https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal/nuget/v3/index.json'
symbolsFeed: 'https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal-symbols/nuget/v3/index.json'
- template: \eng\common\templates\post-build\channels\internal-servicing.yml
@@ -0,0 +1,25 @@
parameters:
ChannelId: 0
jobs:
- job:
displayName: Promote Build
dependsOn: setupMaestroVars
condition: contains(dependencies.setupMaestroVars.outputs['setReleaseVars.InitialChannels'], ${{ parameters.ChannelId }})
variables:
- name: BARBuildId
value: $[ dependencies.setupMaestroVars.outputs['setReleaseVars.BARBuildId'] ]
- name: ChannelId
value: ${{ parameters.ChannelId }}
pool:
vmImage: 'windows-2019'
steps:
- task: PowerShell@2
displayName: Add Build to Channel
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/promote-build.ps1
arguments: -BuildId $(BARBuildId)
-ChannelId $(ChannelId)
-MaestroApiAccessToken $(MaestroApiAccessToken)
-MaestroApiEndPoint $(MaestroApiEndPoint)
-MaestroApiVersion $(MaestroApiVersion)
@@ -1,20 +1,11 @@
jobs:
- job: setupMaestroVars
displayName: Setup Maestro Vars
variables:
- template: common-variables.yml
- name: BuildId
value: $[ coalesce(variables.BARBuildId, 0) ]
- name: PromoteToChannelId
value: $[ coalesce(variables.PromoteToMaestroChannelId, 0) ]
pool:
vmImage: 'windows-2019'
steps:
- checkout: none
- task: DownloadBuildArtifacts@0
displayName: Download Release Configs
condition: eq(variables.PromoteToChannelId, 0)
inputs:
buildType: current
artifactName: ReleaseConfigs
@@ -23,59 +14,5 @@ jobs:
name: setReleaseVars
displayName: Set Release Configs Vars
inputs:
targetType: inline
script: |
try {
if ($Env:PromoteToChannelId -eq 0) {
$Content = Get-Content $(Build.StagingDirectory)/ReleaseConfigs/ReleaseConfigs.txt
$BarId = $Content | Select -Index 0
$Channels = ""
$Content | Select -Index 1 | ForEach-Object { $Channels += "$_ ," }
$IsStableBuild = $Content | Select -Index 2
$AzureDevOpsProject = $Env:System_TeamProject
$AzureDevOpsBuildDefinitionId = $Env:System_DefinitionId
$AzureDevOpsBuildId = $Env:Build_BuildId
$PromoteToMaestroChannelId = 0
}
else {
$buildApiEndpoint = "${Env:MaestroApiEndPoint}/api/builds/${Env:BARBuildId}?api-version=${Env:MaestroApiVersion}"
$apiHeaders = New-Object 'System.Collections.Generic.Dictionary[[String],[String]]'
$apiHeaders.Add('Accept', 'application/json')
$apiHeaders.Add('Authorization',"Bearer ${Env:MAESTRO_API_TOKEN}")
$buildInfo = try { Invoke-WebRequest -Method Get -Uri $buildApiEndpoint -Headers $apiHeaders | ConvertFrom-Json } catch { Write-Host "Error: $_" }
$BarId = $Env:BARBuildId
$Channels = 'None'
#TODO: Fix this once this issue is done: https://github.com/dotnet/arcade/issues/3834
$IsStableBuild = 'False'
$AzureDevOpsProject = $buildInfo.azureDevOpsProject
$AzureDevOpsBuildDefinitionId = $buildInfo.azureDevOpsBuildDefinitionId
$AzureDevOpsBuildId = $buildInfo.azureDevOpsBuildId
$PromoteToMaestroChannelId = $Env:PromoteToMaestroChannelId
}
Write-Host "##vso[task.setvariable variable=BARBuildId;isOutput=true]$BarId"
Write-Host "##vso[task.setvariable variable=InitialChannels;isOutput=true]$Channels"
Write-Host "##vso[task.setvariable variable=IsStableBuild;isOutput=true]$IsStableBuild"
Write-Host "##vso[task.setvariable variable=AzDOProjectName;isOutput=true]$AzureDevOpsProject"
Write-Host "##vso[task.setvariable variable=AzDOPipelineId;isOutput=true]$AzureDevOpsBuildDefinitionId"
Write-Host "##vso[task.setvariable variable=AzDOBuildId;isOutput=true]$AzureDevOpsBuildId"
Write-Host "##vso[task.setvariable variable=PromoteToMaestroChannelId;isOutput=true]$PromoteToMaestroChannelId"
}
catch {
Write-Host $_
Write-Host $_.Exception
Write-Host $_.ScriptStackTrace
exit 1
}
env:
MAESTRO_API_TOKEN: $(MaestroApiAccessToken)
filePath: $(Build.SourcesDirectory)/eng/common/post-build/setup-maestro-vars.ps1
arguments: -ReleaseConfigsPath '$(Build.StagingDirectory)/ReleaseConfigs/ReleaseConfigs.txt'
@@ -1,13 +0,0 @@
parameters:
ChannelId: 0
steps:
- task: PowerShell@2
displayName: Add Build to Channel
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/post-build/add-build-to-channel.ps1
arguments: -BuildId $(BARBuildId)
-ChannelId ${{ parameters.ChannelId }}
-MaestroApiAccessToken $(MaestroApiAccessToken)
-MaestroApiEndPoint $(MaestroApiEndPoint)
-MaestroApiVersion $(MaestroApiVersion)
@@ -1,23 +0,0 @@
parameters:
StageLabel: ''
JobLabel: ''
steps:
- task: Powershell@2
displayName: Prepare Binlogs to Upload
inputs:
targetType: inline
script: |
New-Item -ItemType Directory $(Build.SourcesDirectory)/PostBuildLogs/${{parameters.StageLabel}}/${{parameters.JobLabel}}/
Move-Item -Path $(Build.SourcesDirectory)/artifacts/log/Debug/* $(Build.SourcesDirectory)/PostBuildLogs/${{parameters.StageLabel}}/${{parameters.JobLabel}}/
continueOnError: true
condition: always()
- task: PublishBuildArtifacts@1
displayName: Publish Logs
inputs:
PathtoPublish: '$(Build.SourcesDirectory)/PostBuildLogs'
PublishLocation: Container
ArtifactName: PostBuildLogs
continueOnError: true
condition: always()
@@ -23,7 +23,6 @@ parameters:
EnableXUnitReporter: false # optional -- true enables XUnit result reporting to Mission Control
WaitForWorkItemCompletion: true # optional -- true will make the task wait until work items have been completed and fail the build if work items fail. False is "fire and forget."
IsExternal: false # [DEPRECATED] -- doesn't do anything, jobs are external if HelixAccessToken is empty and Creator is set
HelixBaseUri: 'https://helix.dot.net/' # optional -- sets the Helix API base URI (allows targeting int)
Creator: '' # optional -- if the build is external, use this to specify who is sending the job
DisplayNamePrefix: 'Run Tests' # optional -- rename the beginning of the displayName of the steps in AzDO
condition: succeeded() # optional -- condition for step to execute; defaults to succeeded()
@@ -56,7 +55,6 @@ steps:
DotNetCliVersion: ${{ parameters.DotNetCliVersion }}
EnableXUnitReporter: ${{ parameters.EnableXUnitReporter }}
WaitForWorkItemCompletion: ${{ parameters.WaitForWorkItemCompletion }}
HelixBaseUri: ${{ parameters.HelixBaseUri }}
Creator: ${{ parameters.Creator }}
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
condition: and(${{ parameters.condition }}, eq(variables['Agent.Os'], 'Windows_NT'))
@@ -87,7 +85,6 @@ steps:
DotNetCliVersion: ${{ parameters.DotNetCliVersion }}
EnableXUnitReporter: ${{ parameters.EnableXUnitReporter }}
WaitForWorkItemCompletion: ${{ parameters.WaitForWorkItemCompletion }}
HelixBaseUri: ${{ parameters.HelixBaseUri }}
Creator: ${{ parameters.Creator }}
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
condition: and(${{ parameters.condition }}, ne(variables['Agent.Os'], 'Windows_NT'))
+71 -193
View File
@@ -5,7 +5,7 @@
[bool]$ci = if (Test-Path variable:ci) { $ci } else { $false }
# Build configuration. Common values include 'Debug' and 'Release', but the repository may use other names.
[string]$configuration = if (Test-Path variable:configuration) { $configuration } else { 'Debug' }
[string]$configuration = if (Test-Path variable:configuration) { $configuration } else { "Debug" }
# Set to true to output binary log from msbuild. Note that emitting binary log slows down the build.
# Binary log must be enabled on CI.
@@ -24,7 +24,7 @@
[bool]$restore = if (Test-Path variable:restore) { $restore } else { $true }
# Adjusts msbuild verbosity level.
[string]$verbosity = if (Test-Path variable:verbosity) { $verbosity } else { 'minimal' }
[string]$verbosity = if (Test-Path variable:verbosity) { $verbosity } else { "minimal" }
# Set to true to reuse msbuild nodes. Recommended to not reuse on CI.
[bool]$nodeReuse = if (Test-Path variable:nodeReuse) { $nodeReuse } else { !$ci }
@@ -39,25 +39,19 @@
# installed on the machine instead of downloading one.
[bool]$useInstalledDotNetCli = if (Test-Path variable:useInstalledDotNetCli) { $useInstalledDotNetCli } else { $true }
# Enable repos to use a particular version of the on-line dotnet-install scripts.
# default URL: https://dot.net/v1/dotnet-install.ps1
[string]$dotnetInstallScriptVersion = if (Test-Path variable:dotnetInstallScriptVersion) { $dotnetInstallScriptVersion } else { 'v1' }
# True to use global NuGet cache instead of restoring packages to repository-local directory.
[bool]$useGlobalNuGetCache = if (Test-Path variable:useGlobalNuGetCache) { $useGlobalNuGetCache } else { !$ci }
# An array of names of processes to stop on script exit if prepareMachine is true.
$processesToStopOnExit = if (Test-Path variable:processesToStopOnExit) { $processesToStopOnExit } else { @('msbuild', 'dotnet', 'vbcscompiler') }
$disableConfigureToolsetImport = if (Test-Path variable:disableConfigureToolsetImport) { $disableConfigureToolsetImport } else { $null }
$processesToStopOnExit = if (Test-Path variable:processesToStopOnExit) { $processesToStopOnExit } else { @("msbuild", "dotnet", "vbcscompiler") }
set-strictmode -version 2.0
$ErrorActionPreference = 'Stop'
$ErrorActionPreference = "Stop"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
function Create-Directory([string[]] $path) {
if (!(Test-Path $path)) {
New-Item -path $path -force -itemType 'Directory' | Out-Null
New-Item -path $path -force -itemType "Directory" | Out-Null
}
}
@@ -98,10 +92,7 @@ function Exec-Process([string]$command, [string]$commandArgs) {
}
}
# createSdkLocationFile parameter enables a file being generated under the toolset directory
# which writes the sdk's location into. This is only necessary for cmd --> powershell invocations
# as dot sourcing isn't possible.
function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
function InitializeDotNetCli([bool]$install) {
if (Test-Path variable:global:_DotNetInstallDir) {
return $global:_DotNetInstallDir
}
@@ -124,7 +115,7 @@ function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
# Find the first path on %PATH% that contains the dotnet.exe
if ($useInstalledDotNetCli -and (-not $globalJsonHasRuntimes) -and ($env:DOTNET_INSTALL_DIR -eq $null)) {
$dotnetCmd = Get-Command 'dotnet.exe' -ErrorAction SilentlyContinue
$dotnetCmd = Get-Command "dotnet.exe" -ErrorAction SilentlyContinue
if ($dotnetCmd -ne $null) {
$env:DOTNET_INSTALL_DIR = Split-Path $dotnetCmd.Path -Parent
}
@@ -137,13 +128,13 @@ function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
if ((-not $globalJsonHasRuntimes) -and ($env:DOTNET_INSTALL_DIR -ne $null) -and (Test-Path(Join-Path $env:DOTNET_INSTALL_DIR "sdk\$dotnetSdkVersion"))) {
$dotnetRoot = $env:DOTNET_INSTALL_DIR
} else {
$dotnetRoot = Join-Path $RepoRoot '.dotnet'
$dotnetRoot = Join-Path $RepoRoot ".dotnet"
if (-not (Test-Path(Join-Path $dotnetRoot "sdk\$dotnetSdkVersion"))) {
if ($install) {
InstallDotNetSdk $dotnetRoot $dotnetSdkVersion
} else {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "Unable to find dotnet with SDK version '$dotnetSdkVersion'"
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "Unable to find dotnet with SDK version '$dotnetSdkVersion'"
ExitWithExitCode 1
}
}
@@ -151,24 +142,6 @@ function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
$env:DOTNET_INSTALL_DIR = $dotnetRoot
}
# Creates a temporary file under the toolset dir.
# The following code block is protecting against concurrent access so that this function can
# be called in parallel.
if ($createSdkLocationFile) {
do {
$sdkCacheFileTemp = Join-Path $ToolsetDir $([System.IO.Path]::GetRandomFileName())
}
until (!(Test-Path $sdkCacheFileTemp))
Set-Content -Path $sdkCacheFileTemp -Value $dotnetRoot
try {
Rename-Item -Force -Path $sdkCacheFileTemp 'sdk.txt'
} catch {
# Somebody beat us
Remove-Item -Path $sdkCacheFileTemp
}
}
# Add dotnet to PATH. This prevents any bare invocation of dotnet in custom
# build steps from using anything other than what we've downloaded.
# It also ensures that VS msbuild will use the downloaded sdk targets.
@@ -176,7 +149,6 @@ function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
# Make Sure that our bootstrapped dotnet cli is available in future steps of the Azure Pipelines build
Write-PipelinePrependPath -Path $dotnetRoot
Write-PipelineSetVariable -Name 'DOTNET_MULTILEVEL_LOOKUP' -Value '0'
Write-PipelineSetVariable -Name 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE' -Value '1'
@@ -184,54 +156,20 @@ function InitializeDotNetCli([bool]$install, [bool]$createSdkLocationFile) {
}
function GetDotNetInstallScript([string] $dotnetRoot) {
$installScript = Join-Path $dotnetRoot 'dotnet-install.ps1'
$installScript = Join-Path $dotnetRoot "dotnet-install.ps1"
if (!(Test-Path $installScript)) {
Create-Directory $dotnetRoot
$ProgressPreference = 'SilentlyContinue' # Don't display the console progress UI - it's a huge perf hit
$maxRetries = 5
$retries = 1
$uri = "https://dot.net/$dotnetInstallScriptVersion/dotnet-install.ps1"
while($true) {
try {
Write-Host "GET $uri"
Invoke-WebRequest $uri -OutFile $installScript
break
}
catch {
Write-Host "Failed to download '$uri'"
Write-Error $_.Exception.Message -ErrorAction Continue
}
if (++$retries -le $maxRetries) {
$delayInSeconds = [math]::Pow(2, $retries) - 1 # Exponential backoff
Write-Host "Retrying. Waiting for $delayInSeconds seconds before next attempt ($retries of $maxRetries)."
Start-Sleep -Seconds $delayInSeconds
}
else {
throw "Unable to download file in $maxRetries attempts."
}
}
Invoke-WebRequest "https://dot.net/v1/dotnet-install.ps1" -OutFile $installScript
}
return $installScript
}
function InstallDotNetSdk([string] $dotnetRoot, [string] $version, [string] $architecture = '') {
function InstallDotNetSdk([string] $dotnetRoot, [string] $version, [string] $architecture = "") {
InstallDotNet $dotnetRoot $version $architecture
}
function InstallDotNet([string] $dotnetRoot,
[string] $version,
[string] $architecture = '',
[string] $runtime = '',
[bool] $skipNonVersionedFiles = $false,
[string] $runtimeSourceFeed = '',
[string] $runtimeSourceFeedKey = '') {
function InstallDotNet([string] $dotnetRoot, [string] $version, [string] $architecture = "", [string] $runtime = "", [bool] $skipNonVersionedFiles = $false) {
$installScript = GetDotNetInstallScript $dotnetRoot
$installParameters = @{
Version = $version
@@ -242,32 +180,10 @@ function InstallDotNet([string] $dotnetRoot,
if ($runtime) { $installParameters.Runtime = $runtime }
if ($skipNonVersionedFiles) { $installParameters.SkipNonVersionedFiles = $skipNonVersionedFiles }
try {
& $installScript @installParameters
}
catch {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "Failed to install dotnet runtime '$runtime' from public location."
# Only the runtime can be installed from a custom [private] location.
if ($runtime -and ($runtimeSourceFeed -or $runtimeSourceFeedKey)) {
if ($runtimeSourceFeed) { $installParameters.AzureFeed = $runtimeSourceFeed }
if ($runtimeSourceFeedKey) {
$decodedBytes = [System.Convert]::FromBase64String($runtimeSourceFeedKey)
$decodedString = [System.Text.Encoding]::UTF8.GetString($decodedBytes)
$installParameters.FeedCredential = $decodedString
}
try {
& $installScript @installParameters
}
catch {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "Failed to install dotnet runtime '$runtime' from custom location '$runtimeSourceFeed'."
ExitWithExitCode 1
}
} else {
ExitWithExitCode 1
}
& $installScript @installParameters
if ($lastExitCode -ne 0) {
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "Failed to install dotnet cli (exit code '$lastExitCode')."
ExitWithExitCode $lastExitCode
}
}
@@ -288,16 +204,16 @@ function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements =
}
if (!$vsRequirements) { $vsRequirements = $GlobalJson.tools.vs }
$vsMinVersionStr = if ($vsRequirements.version) { $vsRequirements.version } else { '15.9' }
$vsMinVersionStr = if ($vsRequirements.version) { $vsRequirements.version } else { "15.9" }
$vsMinVersion = [Version]::new($vsMinVersionStr)
# Try msbuild command available in the environment.
if ($env:VSINSTALLDIR -ne $null) {
$msbuildCmd = Get-Command 'msbuild.exe' -ErrorAction SilentlyContinue
$msbuildCmd = Get-Command "msbuild.exe" -ErrorAction SilentlyContinue
if ($msbuildCmd -ne $null) {
# Workaround for https://github.com/dotnet/roslyn/issues/35793
# Due to this issue $msbuildCmd.Version returns 0.0.0.0 for msbuild.exe 16.2+
$msbuildVersion = [Version]::new((Get-Item $msbuildCmd.Path).VersionInfo.ProductVersion.Split([char[]]@('-', '+'))[0])
$msbuildVersion = [Version]::new((Get-Item $msbuildCmd.Path).VersionInfo.ProductVersion.Split(@('-', '+'))[0])
if ($msbuildVersion -ge $vsMinVersion) {
return $global:_MSBuildExe = $msbuildCmd.Path
@@ -317,7 +233,7 @@ function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements =
InitializeVisualStudioEnvironmentVariables $vsInstallDir $vsMajorVersion
} else {
if (Get-Member -InputObject $GlobalJson.tools -Name 'xcopy-msbuild') {
if (Get-Member -InputObject $GlobalJson.tools -Name "xcopy-msbuild") {
$xcopyMSBuildVersion = $GlobalJson.tools.'xcopy-msbuild'
$vsMajorVersion = $xcopyMSBuildVersion.Split('.')[0]
} else {
@@ -325,12 +241,9 @@ function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements =
$xcopyMSBuildVersion = "$vsMajorVersion.$($vsMinVersion.Minor).0-alpha"
}
$vsInstallDir = $null
if ($xcopyMSBuildVersion.Trim() -ine "none") {
$vsInstallDir = InitializeXCopyMSBuild $xcopyMSBuildVersion $install
}
$vsInstallDir = InitializeXCopyMSBuild $xcopyMSBuildVersion $install
if ($vsInstallDir -eq $null) {
throw 'Unable to find Visual Studio that has required version and components installed'
throw "Unable to find Visual Studio that has required version and components installed"
}
}
@@ -354,7 +267,7 @@ function InstallXCopyMSBuild([string]$packageVersion) {
}
function InitializeXCopyMSBuild([string]$packageVersion, [bool]$install) {
$packageName = 'RoslynTools.MSBuild'
$packageName = "RoslynTools.MSBuild"
$packageDir = Join-Path $ToolsDir "msbuild\$packageVersion"
$packagePath = Join-Path $packageDir "$packageName.$packageVersion.nupkg"
@@ -365,12 +278,11 @@ function InitializeXCopyMSBuild([string]$packageVersion, [bool]$install) {
Create-Directory $packageDir
Write-Host "Downloading $packageName $packageVersion"
$ProgressPreference = 'SilentlyContinue' # Don't display the console progress UI - it's a huge perf hit
Invoke-WebRequest "https://dotnet.myget.org/F/roslyn-tools/api/v2/package/$packageName/$packageVersion/" -OutFile $packagePath
Unzip $packagePath $packageDir
}
return Join-Path $packageDir 'tools'
return Join-Path $packageDir "tools"
}
#
@@ -387,37 +299,32 @@ function InitializeXCopyMSBuild([string]$packageVersion, [bool]$install) {
# or $null if no instance meeting the requirements is found on the machine.
#
function LocateVisualStudio([object]$vsRequirements = $null){
if (Get-Member -InputObject $GlobalJson.tools -Name 'vswhere') {
if (Get-Member -InputObject $GlobalJson.tools -Name "vswhere") {
$vswhereVersion = $GlobalJson.tools.vswhere
} else {
$vswhereVersion = '2.5.2'
$vswhereVersion = "2.5.2"
}
$vsWhereDir = Join-Path $ToolsDir "vswhere\$vswhereVersion"
$vsWhereExe = Join-Path $vsWhereDir 'vswhere.exe'
$vsWhereExe = Join-Path $vsWhereDir "vswhere.exe"
if (!(Test-Path $vsWhereExe)) {
Create-Directory $vsWhereDir
Write-Host 'Downloading vswhere'
try {
Invoke-WebRequest "https://netcorenativeassets.blob.core.windows.net/resource-packages/external/windows/vswhere/$vswhereVersion/vswhere.exe" -OutFile $vswhereExe
}
catch {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message $_
}
Write-Host "Downloading vswhere"
Invoke-WebRequest "https://github.com/Microsoft/vswhere/releases/download/$vswhereVersion/vswhere.exe" -OutFile $vswhereExe
}
if (!$vsRequirements) { $vsRequirements = $GlobalJson.tools.vs }
$args = @('-latest', '-prerelease', '-format', 'json', '-requires', 'Microsoft.Component.MSBuild', '-products', '*')
$args = @("-latest", "-prerelease", "-format", "json", "-requires", "Microsoft.Component.MSBuild", "-products", "*")
if (Get-Member -InputObject $vsRequirements -Name 'version') {
$args += '-version'
if (Get-Member -InputObject $vsRequirements -Name "version") {
$args += "-version"
$args += $vsRequirements.version
}
if (Get-Member -InputObject $vsRequirements -Name 'components') {
if (Get-Member -InputObject $vsRequirements -Name "components") {
foreach ($component in $vsRequirements.components) {
$args += '-requires'
$args += "-requires"
$args += $component
}
}
@@ -443,27 +350,28 @@ function InitializeBuildTool() {
# Initialize dotnet cli if listed in 'tools'
$dotnetRoot = $null
if (Get-Member -InputObject $GlobalJson.tools -Name 'dotnet') {
if (Get-Member -InputObject $GlobalJson.tools -Name "dotnet") {
$dotnetRoot = InitializeDotNetCli -install:$restore
}
if ($msbuildEngine -eq 'dotnet') {
if ($msbuildEngine -eq "dotnet") {
if (!$dotnetRoot) {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "/global.json must specify 'tools.dotnet'."
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "/global.json must specify 'tools.dotnet'."
ExitWithExitCode 1
}
$buildTool = @{ Path = Join-Path $dotnetRoot 'dotnet.exe'; Command = 'msbuild'; Tool = 'dotnet'; Framework = 'netcoreapp2.1' }
$buildTool = @{ Path = Join-Path $dotnetRoot "dotnet.exe"; Command = "msbuild"; Tool = "dotnet"; Framework = "netcoreapp2.1" }
} elseif ($msbuildEngine -eq "vs") {
try {
$msbuildPath = InitializeVisualStudioMSBuild -install:$restore
} catch {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message $_
Write-PipelineTelemetryError -Category "InitializeToolset" -Message $_
ExitWithExitCode 1
}
$buildTool = @{ Path = $msbuildPath; Command = ""; Tool = "vs"; Framework = "net472" }
} else {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "Unexpected value of -msbuildEngine: '$msbuildEngine'."
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "Unexpected value of -msbuildEngine: '$msbuildEngine'."
ExitWithExitCode 1
}
@@ -472,15 +380,15 @@ function InitializeBuildTool() {
function GetDefaultMSBuildEngine() {
# Presence of tools.vs indicates the repo needs to build using VS msbuild on Windows.
if (Get-Member -InputObject $GlobalJson.tools -Name 'vs') {
return 'vs'
if (Get-Member -InputObject $GlobalJson.tools -Name "vs") {
return "vs"
}
if (Get-Member -InputObject $GlobalJson.tools -Name 'dotnet') {
return 'dotnet'
if (Get-Member -InputObject $GlobalJson.tools -Name "dotnet") {
return "dotnet"
}
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "-msbuildEngine must be specified, or /global.json must specify 'tools.dotnet' or 'tools.vs'."
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "-msbuildEngine must be specified, or /global.json must specify 'tools.dotnet' or 'tools.vs'."
ExitWithExitCode 1
}
@@ -489,9 +397,9 @@ function GetNuGetPackageCachePath() {
# Use local cache on CI to ensure deterministic build,
# use global cache in dev builds to avoid cost of downloading packages.
if ($useGlobalNuGetCache) {
$env:NUGET_PACKAGES = Join-Path $env:UserProfile '.nuget\packages'
$env:NUGET_PACKAGES = Join-Path $env:UserProfile ".nuget\packages"
} else {
$env:NUGET_PACKAGES = Join-Path $RepoRoot '.packages'
$env:NUGET_PACKAGES = Join-Path $RepoRoot ".packages"
}
}
@@ -504,7 +412,7 @@ function GetSdkTaskProject([string]$taskName) {
}
function InitializeNativeTools() {
if (-Not (Test-Path variable:DisableNativeToolsetInstalls) -And (Get-Member -InputObject $GlobalJson -Name "native-tools")) {
if (Get-Member -InputObject $GlobalJson -Name "native-tools") {
$nativeArgs= @{}
if ($ci) {
$nativeArgs = @{
@@ -533,14 +441,14 @@ function InitializeToolset() {
}
if (-not $restore) {
Write-PipelineTelemetryError -Category 'InitializeToolset' -Message "Toolset version $toolsetVersion has not been restored."
Write-PipelineTelemetryError -Category "InitializeToolset" -Message "Toolset version $toolsetVersion has not been restored."
ExitWithExitCode 1
}
$buildTool = InitializeBuildTool
$proj = Join-Path $ToolsetDir 'restore.proj'
$bl = if ($binaryLog) { '/bl:' + (Join-Path $LogDir 'ToolsetRestore.binlog') } else { '' }
$proj = Join-Path $ToolsetDir "restore.proj"
$bl = if ($binaryLog) { "/bl:" + (Join-Path $LogDir "ToolsetRestore.binlog") } else { "" }
'<Project Sdk="Microsoft.DotNet.Arcade.Sdk"/>' | Set-Content $proj
@@ -562,7 +470,7 @@ function ExitWithExitCode([int] $exitCode) {
}
function Stop-Processes() {
Write-Host 'Killing running build processes...'
Write-Host "Killing running build processes..."
foreach ($processName in $processesToStopOnExit) {
Get-Process -Name $processName -ErrorAction SilentlyContinue | Stop-Process
}
@@ -576,21 +484,9 @@ function Stop-Processes() {
function MSBuild() {
if ($pipelinesLog) {
$buildTool = InitializeBuildTool
# Work around issues with Azure Artifacts credential provider
# https://github.com/dotnet/arcade/issues/3932
if ($ci -and $buildTool.Tool -eq 'dotnet') {
dotnet nuget locals http-cache -c
$env:NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS = 20
$env:NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS = 20
Write-PipelineSetVariable -Name 'NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS' -Value '20'
Write-PipelineSetVariable -Name 'NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS' -Value '20'
}
$toolsetBuildProject = InitializeToolset
$path = Split-Path -parent $toolsetBuildProject
$path = Join-Path $path (Join-Path $buildTool.Framework 'Microsoft.DotNet.Arcade.Sdk.dll')
$path = Join-Path $path (Join-Path $buildTool.Framework "Microsoft.DotNet.Arcade.Sdk.dll")
$args += "/logger:$path"
}
@@ -605,12 +501,12 @@ function MSBuild() {
function MSBuild-Core() {
if ($ci) {
if (!$binaryLog) {
Write-PipelineTelemetryError -Category 'Build' -Message 'Binary log must be enabled in CI build.'
Write-PipelineTaskError -Message "Binary log must be enabled in CI build."
ExitWithExitCode 1
}
if ($nodeReuse) {
Write-PipelineTelemetryError -Category 'Build' -Message 'Node reuse must be disabled in CI build.'
Write-PipelineTaskError -Message "Node reuse must be disabled in CI build."
ExitWithExitCode 1
}
}
@@ -620,10 +516,7 @@ function MSBuild-Core() {
$cmdArgs = "$($buildTool.Command) /m /nologo /clp:Summary /v:$verbosity /nr:$nodeReuse /p:ContinuousIntegrationBuild=$ci"
if ($warnAsError) {
$cmdArgs += ' /warnaserror /p:TreatWarningsAsErrors=true'
}
else {
$cmdArgs += ' /p:TreatWarningsAsErrors=false'
$cmdArgs += " /warnaserror /p:TreatWarningsAsErrors=true"
}
foreach ($arg in $args) {
@@ -635,7 +528,7 @@ function MSBuild-Core() {
$exitCode = Exec-Process $buildTool.Path $cmdArgs
if ($exitCode -ne 0) {
Write-PipelineTelemetryError -Category 'Build' -Message 'Build failed.'
Write-PipelineTaskError -Message "Build failed."
$buildLog = GetMSBuildBinaryLogCommandLineArgument $args
if ($buildLog -ne $null) {
@@ -650,12 +543,12 @@ function GetMSBuildBinaryLogCommandLineArgument($arguments) {
foreach ($argument in $arguments) {
if ($argument -ne $null) {
$arg = $argument.Trim()
if ($arg.StartsWith('/bl:', "OrdinalIgnoreCase")) {
return $arg.Substring('/bl:'.Length)
if ($arg.StartsWith("/bl:", "OrdinalIgnoreCase")) {
return $arg.Substring("/bl:".Length)
}
if ($arg.StartsWith('/binaryLogger:', 'OrdinalIgnoreCase')) {
return $arg.Substring('/binaryLogger:'.Length)
if ($arg.StartsWith("/binaryLogger:", "OrdinalIgnoreCase")) {
return $arg.Substring("/binaryLogger:".Length)
}
}
}
@@ -665,14 +558,14 @@ function GetMSBuildBinaryLogCommandLineArgument($arguments) {
. $PSScriptRoot\pipeline-logging-functions.ps1
$RepoRoot = Resolve-Path (Join-Path $PSScriptRoot '..\..')
$EngRoot = Resolve-Path (Join-Path $PSScriptRoot '..')
$ArtifactsDir = Join-Path $RepoRoot 'artifacts'
$ToolsetDir = Join-Path $ArtifactsDir 'toolset'
$ToolsDir = Join-Path $RepoRoot '.tools'
$LogDir = Join-Path (Join-Path $ArtifactsDir 'log') $configuration
$TempDir = Join-Path (Join-Path $ArtifactsDir 'tmp') $configuration
$GlobalJson = Get-Content -Raw -Path (Join-Path $RepoRoot 'global.json') | ConvertFrom-Json
$RepoRoot = Resolve-Path (Join-Path $PSScriptRoot "..\..")
$EngRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
$ArtifactsDir = Join-Path $RepoRoot "artifacts"
$ToolsetDir = Join-Path $ArtifactsDir "toolset"
$ToolsDir = Join-Path $RepoRoot ".tools"
$LogDir = Join-Path (Join-Path $ArtifactsDir "log") $configuration
$TempDir = Join-Path (Join-Path $ArtifactsDir "tmp") $configuration
$GlobalJson = Get-Content -Raw -Path (Join-Path $RepoRoot "global.json") | ConvertFrom-Json
# true if global.json contains a "runtimes" section
$globalJsonHasRuntimes = if ($GlobalJson.tools.PSObject.Properties.Name -Match 'runtimes') { $true } else { $false }
@@ -685,18 +578,3 @@ Write-PipelineSetVariable -Name 'Artifacts.Toolset' -Value $ToolsetDir
Write-PipelineSetVariable -Name 'Artifacts.Log' -Value $LogDir
Write-PipelineSetVariable -Name 'TEMP' -Value $TempDir
Write-PipelineSetVariable -Name 'TMP' -Value $TempDir
# Import custom tools configuration, if present in the repo.
# Note: Import in global scope so that the script set top-level variables without qualification.
if (!$disableConfigureToolsetImport) {
$configureToolsetScript = Join-Path $EngRoot 'configure-toolset.ps1'
if (Test-Path $configureToolsetScript) {
. $configureToolsetScript
if ((Test-Path variable:failOnConfigureToolsetError) -And $failOnConfigureToolsetError) {
if ((Test-Path variable:LastExitCode) -And ($LastExitCode -ne 0)) {
Write-PipelineTelemetryError -Category 'Build' -Message 'configure-toolset.ps1 returned a non-zero exit code'
ExitWithExitCode $LastExitCode
}
}
}
}
+18 -103
View File
@@ -41,14 +41,10 @@ fi
# Configures warning treatment in msbuild.
warn_as_error=${warn_as_error:-true}
# True to attempt using .NET Core already that meets requirements specified in global.json
# True to attempt using .NET Core already that meets requirements specified in global.json
# installed on the machine instead of downloading one.
use_installed_dotnet_cli=${use_installed_dotnet_cli:-true}
# Enable repos to use a particular version of the on-line dotnet-install scripts.
# default URL: https://dot.net/v1/dotnet-install.sh
dotnetInstallScriptVersion=${dotnetInstallScriptVersion:-'v1'}
# True to use global NuGet cache instead of restoring packages to repository-local directory.
if [[ "$ci" == true ]]; then
use_global_nuget_cache=${use_global_nuget_cache:-false}
@@ -77,11 +73,11 @@ function ResolvePath {
function ReadGlobalVersion {
local key=$1
local line=$(awk "/$key/ {print; exit}" "$global_json_file")
local line=`grep -m 1 "$key" "$global_json_file"`
local pattern="\"$key\" *: *\"(.*)\""
if [[ ! $line =~ $pattern ]]; then
Write-PipelineTelemetryError -category 'Build' "Error: Cannot find \"$key\" in $global_json_file"
Write-PipelineTelemetryError -category 'InitializeToolset' "Error: Cannot find \"$key\" in $global_json_file"
ExitWithExitCode 1
fi
@@ -172,7 +168,7 @@ function InstallDotNetSdk {
function InstallDotNet {
local root=$1
local version=$2
GetDotNetInstallScript "$root"
local install_script=$_GetDotNetInstallScript
@@ -191,59 +187,15 @@ function InstallDotNet {
fi
bash "$install_script" --version $version --install-dir "$root" $archArg $runtimeArg $skipNonVersionedFilesArg || {
local exit_code=$?
Write-PipelineTelemetryError -category 'InitializeToolset' "Failed to install dotnet SDK from public location (exit code '$exit_code')."
if [[ -n "$runtimeArg" ]]; then
local runtimeSourceFeed=''
if [[ -n "${6:-}" ]]; then
runtimeSourceFeed="--azure-feed $6"
fi
local runtimeSourceFeedKey=''
if [[ -n "${7:-}" ]]; then
decodedFeedKey=`echo $7 | base64 --decode`
runtimeSourceFeedKey="--feed-credential $decodedFeedKey"
fi
if [[ -n "$runtimeSourceFeed" || -n "$runtimeSourceFeedKey" ]]; then
bash "$install_script" --version $version --install-dir "$root" $archArg $runtimeArg $skipNonVersionedFilesArg $runtimeSourceFeed $runtimeSourceFeedKey || {
local exit_code=$?
Write-PipelineTelemetryError -category 'InitializeToolset' "Failed to install dotnet SDK from custom location '$runtimeSourceFeed' (exit code '$exit_code')."
ExitWithExitCode $exit_code
}
else
ExitWithExitCode $exit_code
fi
fi
Write-PipelineTelemetryError -category 'InitializeToolset' "Failed to install dotnet SDK (exit code '$exit_code')."
ExitWithExitCode $exit_code
}
}
function with_retries {
local maxRetries=5
local retries=1
echo "Trying to run '$@' for maximum of $maxRetries attempts."
while [[ $((retries++)) -le $maxRetries ]]; do
"$@"
if [[ $? == 0 ]]; then
echo "Ran '$@' successfully."
return 0
fi
timeout=$((2**$retries-1))
echo "Failed to execute '$@'. Waiting $timeout seconds before next attempt ($retries out of $maxRetries)." 1>&2
sleep $timeout
done
echo "Failed to execute '$@' for $maxRetries times." 1>&2
return 1
}
function GetDotNetInstallScript {
local root=$1
local install_script="$root/dotnet-install.sh"
local install_script_url="https://dot.net/$dotnetInstallScriptVersion/dotnet-install.sh"
local install_script_url="https://dot.net/v1/dotnet-install.sh"
if [[ ! -a "$install_script" ]]; then
mkdir -p "$root"
@@ -252,19 +204,12 @@ function GetDotNetInstallScript {
# Use curl if available, otherwise use wget
if command -v curl > /dev/null; then
with_retries curl "$install_script_url" -sSL --retry 10 --create-dirs -o "$install_script" || {
local exit_code=$?
Write-PipelineTelemetryError -category 'InitializeToolset' "Failed to acquire dotnet install script (exit code '$exit_code')."
ExitWithExitCode $exit_code
}
curl "$install_script_url" -sSL --retry 10 --create-dirs -o "$install_script"
else
with_retries wget -v -O "$install_script" "$install_script_url" || {
local exit_code=$?
Write-PipelineTelemetryError -category 'InitializeToolset' "Failed to acquire dotnet install script (exit code '$exit_code')."
ExitWithExitCode $exit_code
}
wget -q -O "$install_script" "$install_script_url"
fi
fi
# return value
_GetDotNetInstallScript="$install_script"
}
@@ -273,11 +218,11 @@ function InitializeBuildTool {
if [[ -n "${_InitializeBuildTool:-}" ]]; then
return
fi
InitializeDotNetCli $restore
# return values
_InitializeBuildTool="$_InitializeDotNetCli/dotnet"
_InitializeBuildTool="$_InitializeDotNetCli/dotnet"
_InitializeBuildToolCommand="msbuild"
_InitializeBuildToolFramework="netcoreapp2.1"
}
@@ -296,9 +241,6 @@ function GetNuGetPackageCachePath {
}
function InitializeNativeTools() {
if [[ -n "${DisableNativeToolsetInstalls:-}" ]]; then
return
fi
if grep -Fq "native-tools" $global_json_file
then
local nativeArgs=""
@@ -341,14 +283,14 @@ function InitializeToolset {
if [[ "$binary_log" == true ]]; then
bl="/bl:$log_dir/ToolsetRestore.binlog"
fi
echo '<Project Sdk="Microsoft.DotNet.Arcade.Sdk"/>' > "$proj"
MSBuild-Core "$proj" $bl /t:__WriteToolsetLocation /clp:ErrorsOnly\;NoSummary /p:__ToolsetLocationOutputFile="$toolset_location_file"
local toolset_build_proj=`cat "$toolset_location_file"`
if [[ ! -a "$toolset_build_proj" ]]; then
Write-PipelineTelemetryError -category 'Build' "Invalid toolset path: $toolset_build_proj"
Write-PipelineTelemetryError -category 'InitializeToolset' "Invalid toolset path: $toolset_build_proj"
ExitWithExitCode 3
fi
@@ -375,18 +317,6 @@ function MSBuild {
if [[ "$pipelines_log" == true ]]; then
InitializeBuildTool
InitializeToolset
# Work around issues with Azure Artifacts credential provider
# https://github.com/dotnet/arcade/issues/3932
if [[ "$ci" == true ]]; then
"$_InitializeBuildTool" nuget locals http-cache -c
export NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS=20
export NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS=20
Write-PipelineSetVariable -name "NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS" -value "20"
Write-PipelineSetVariable -name "NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS" -value "20"
fi
local toolset_dir="${_InitializeToolset%/*}"
local logger_path="$toolset_dir/$_InitializeBuildToolFramework/Microsoft.DotNet.Arcade.Sdk.dll"
args=( "${args[@]}" "-logger:$logger_path" )
@@ -398,12 +328,12 @@ function MSBuild {
function MSBuild-Core {
if [[ "$ci" == true ]]; then
if [[ "$binary_log" != true ]]; then
Write-PipelineTelemetryError -category 'Build' "Binary log must be enabled in CI build."
Write-PipelineTaskError "Binary log must be enabled in CI build."
ExitWithExitCode 1
fi
if [[ "$node_reuse" == true ]]; then
Write-PipelineTelemetryError -category 'Build' "Node reuse must be disabled in CI build."
Write-PipelineTaskError "Node reuse must be disabled in CI build."
ExitWithExitCode 1
fi
fi
@@ -417,7 +347,7 @@ function MSBuild-Core {
"$_InitializeBuildTool" "$_InitializeBuildToolCommand" /m /nologo /clp:Summary /v:$verbosity /nr:$node_reuse $warnaserror_switch /p:TreatWarningsAsErrors=$warn_as_error /p:ContinuousIntegrationBuild=$ci "$@" || {
local exit_code=$?
Write-PipelineTelemetryError -category 'Build' "Build failed (exit code '$exit_code')."
Write-PipelineTaskError "Build failed (exit code '$exit_code')."
ExitWithExitCode $exit_code
}
}
@@ -438,7 +368,7 @@ temp_dir="$artifacts_dir/tmp/$configuration"
global_json_file="$repo_root/global.json"
# determine if global.json contains a "runtimes" entry
global_json_has_runtimes=false
dotnetlocal_key=$(awk "/runtimes/ {print; exit}" "$global_json_file") || true
dotnetlocal_key=`grep -m 1 "runtimes" "$global_json_file"` || true
if [[ -n "$dotnetlocal_key" ]]; then
global_json_has_runtimes=true
fi
@@ -458,18 +388,3 @@ Write-PipelineSetVariable -name "Artifacts.Toolset" -value "$toolset_dir"
Write-PipelineSetVariable -name "Artifacts.Log" -value "$log_dir"
Write-PipelineSetVariable -name "Temp" -value "$temp_dir"
Write-PipelineSetVariable -name "TMP" -value "$temp_dir"
# Import custom tools configuration, if present in the repo.
if [ -z "${disable_configure_toolset_import:-}" ]; then
configure_toolset_script="$eng_root/configure-toolset.sh"
if [[ -a "$configure_toolset_script" ]]; then
. "$configure_toolset_script"
fi
fi
# TODO: https://github.com/dotnet/arcade/issues/1468
# Temporary workaround to avoid breaking change.
# Remove once repos are updated.
if [[ -n "${useInstalledDotNetCli:-}" ]]; then
use_installed_dotnet_cli="$useInstalledDotNetCli"
fi
View File
File renamed without changes.
+51
View File
@@ -0,0 +1,51 @@
<Project DefaultTargets="PublishPackages">
<!-- TODO: move properties imported from here into a common props file -->
<!-- <Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), dir.props))\dir.props" /> -->
<!-- Use an explicit SDK import so that arcade uses the build
configuration computed in dir.common.props. -->
<Import Project="Sdk.props" Sdk="Microsoft.DotNet.Arcade.Sdk" />
<Import Project="$(MSBuildThisFileDirectory)\..\.packages\microsoft.dotnet.build.tasks.feed\$(MicrosoftDotNetBuildTasksFeedVersion)\build\Microsoft.DotNet.Build.Tasks.Feed.targets" />
<PropertyGroup>
<!-- Set the TargetFramework just to make the SDK happy -->
<TargetFramework>net462</TargetFramework>
</PropertyGroup>
<Target Name="PublishPackages">
<ItemGroup>
<PackagesToPublish Include="$(MSBuildThisFileDirectory)\..\artifacts\*.nupkg" />
</ItemGroup>
<ItemGroup>
<ItemsToPushToBlobFeed Include="@(PackagesToPublish)" />
</ItemGroup>
<Error Condition=" '$(AzureFeedUrl)' == '' " Text="AzureFeedUrl must be set" />
<Error Condition=" '$(AzureAccountKey)' == '' " Text="AzureAccountKey must be set" />
<Error Condition=" '$(BUILD_REPOSITORY_URI)' == '' " Text="BUILD_REPOSITORY_URI must be set" />
<Error Condition=" '$(BUILD_SOURCEBRANCH)' == '' " Text="BUILD_SOURCEBRANCH must be set" />
<Error Condition=" '$(BUILD_BUILDNUMBER)' == '' " Text="BUILD_BUILDNUMBER must be set" />
<Error Condition=" '$(BUILD_SOURCEVERSION)' == '' " Text="BUILD_SOURCEVERSION must be set" />
<PropertyGroup>
<AssetManifestFilePath>$(ArtifactsLogDir)AssetManifest\$(AssetManifest)</AssetManifestFilePath>
</PropertyGroup>
<Message Importance="Low" Text="BUILD_REPOSITORY_URI: $(BUILD_REPOSITORY_URI)" />
<PushToBlobFeed ExpectedFeedUrl="$(AzureFeedUrl)"
AccountKey="$(AzureAccountKey)"
ItemsToPush="@(ItemsToPushToBlobFeed)"
ManifestBuildData="Location=$(AzureFeedUrl)"
ManifestRepoUri="$(BUILD_REPOSITORY_URI)"
ManifestBranch="$(BUILD_SOURCEBRANCH)"
ManifestBuildId="$(BUILD_BUILDNUMBER)"
ManifestCommit="$(BUILD_SOURCEVERSION)"
AssetManifestPath="$(AssetManifestFilePath)" />
</Target>
</Project>
-1
Submodule external/bdwgc deleted from 6e8db07c8d.
+1 -1
+1 -1
Loaded 100 of 2972 files, more files were not shown because too many files have changed in this diff. Show more