cert-sync already knew how to import into the BTLS cert store,
update it to always import into both stores so we don't need
the btls-cert-sync tool anymore.
(cherry picked from commit 968c60a0f3)
It only looked at the legacy format where certificates had the *.cer
extension, the new BTLS store uses *.0 extension.
(cherry picked from commit 3aa59f2e6b)
This was keeping around potentially many unused gray queue sections, or even constantly leaking in the case of many majors since every time we start a concurrent major we redirect some additional gray queue sections from main to worker thread gray queues, leading to unbounded growth. The gc thread always trims its gray queues at the end of the collection.
On graph8 this reduces memory usage by 5%. Fixes mem leak from #41995
This change implements retrieving a list of DNS server addresses from Android
instead of attempting to parse `/etc/resolv.conf` which doesn't exist on
Android.
The list is retrieved by calling native function in Xamarin.Android that gets
the data from the system.
Part 2 of fix for https://bugzilla.xamarin.com/show_bug.cgi?id=48016
When setting X509CertificateImplBtls.PrivateKey we'd previously just
update the fallback instance. This causes problems if other code
internal to System.dll like MonoBtlsContext.SetPrivateCertificate()
tries to fetch the NativePrivateKey since it'd be null at that point.
This scenario happens e.g. with HttpListener, it supports a special
directory in ~/.config/.mono/httplistener which contains <port>.cer
and <port>.pvk files (public and private key) that will be used
as the server certificate. In that code we're reading the private
key from the file and assigning it to X509CertificateImplBtls.PrivateKey
without ever having a native BTLS handle.
The fix is to create such a native BTLS handle from the managed
private key when needed.
Note that we can't simply always do this e.g. in the PrivateKey
setter because the setter will be used with instances that
don't actually contain a private key. We can't throw in those
cases so instead we do it only when NativePrivateKey is accessed.
To make the code clearer and easier to read, `privateKey` was
renamed to `nativePrivateKey` in X509CertificateImplBtls.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=46602
(cherry picked from commit a9d75b4b66
and commit e135936e2f)
Otherwise we'd not clear the fields like subjectName, issuerName, etc.
and the properties they're backing would continue to return old values.
(cherry picked from commit af85a09321)
From the header docs:
> The |get1| functions return a fresh reference to the underlying
> object or NULL if |pkey| is not of the correct type.
(cherry picked from commit 2cadd74de8)
* Initial version of lldb support code.
* [lldb] Keep all the debug entries in a linked list so lldb can read it after attaching to the process.
* [lldb] Add an --enable-minimal=lldb option to disable the code. Add locking.
* [lldb] Add support for dynamic methods by adding a new protocol entry for unloading code regions.
* [lldb] Add beginnings of a test suite using the lldb python bindings.
* [lldb] Emit line number info for methods, not used yet.
* [lldb] Add dummy fields to structures to make sure 64 bit fields are 64 bit aligned.
* [lldb] Finish emission of line number info.
* [lldb] Use mono_method_full_name () for computing method names.
* [lldb] Emit info for specific trampolines as well.
* [lldb] Add tests for line number info.
* [lldb] Fix unwind info encoding on x86.
* [lldb] Skip seq points with no native offset.
* [lldb] Add tests for breakpoints.
We won't import it into Mono anytime soon and the long paths
cause issues when checking out Mono on Windows.
(cherry picked from commit ca5104094c)
Marek requested this being backported to the 4.8 branch.
Previously we'd just throw the catched exception again
which overwrites its stacktrace, making it less useful
since all you can see is e.g. a NullReferenceException
(re)thrown at MobileAuthenticatedStream:218.
We now capture the original exception information so
the stacktrace is more complete.
(cherry picked from commit 0007e99a87)
The path ".mono/certs/NewTrust" doesn't actually exist,
it's called ".mono/new-certs/Trust". Look it up from
MonoBtlsX509StoreManager instead.
This API isn't used anywhere in the BTLS code, that's
probably why it wasn't noticed.
(cherry picked from commit 6d2420f49a)
CMake doesn't pick up the compiler from the PATH by default,
it only looks in default locations so without this we might
end up using a different compiler for BTLS than for the main
Mono build which we don't want.
BTLS also requires a fairly new GCC which includes C++11
features, added a quick check for stdalign.h which BTLS
requires and disable it if the header isn't found.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=46712
(cherry picked from commit 488b550ac0)
The machineXYZPath fields are initialized with machinePath instead of
userPath which caused the machine stores not to work at all
(cherry picked from commit 10e1dc5314)
This addresses a regression here:
https://bugzilla.xamarin.com/show_bug.cgi?id=45761
The commits
97e51abfcf and
d35b5d2188
changed control flow such that if the host can't be resolved
and host is set to null, then the later check will return null
for the duration that HasTimedOut is false.
Finding a free port on watchOS fails with a PlatformNotSupportedException, but
if the exception is thrown in the constructor, the ExpectedException attribute
isn't taken into account, causing the test to error out.
https://bugzilla.xamarin.com/show_bug.cgi?id=47167
`directory` is already a directory. It's in the name.
`Path.GetDirectoryName ("/path/to/a/thing")` returns `"/path/to/a"`, whether
`thing` is a directory or not. If there's a trailing `/` then it returns
the directory, not the parent - but we don't have a trailing `/`.
This fixes an issue where the path needs to be created by hand before an
SDK can be downloaded.
(cherry picked from commit 031d79c2c4)
First the bug fix.
If we hit a method with a nested type as argument, we would not include the outer types and fail to match.
Now for the big change.
The runtime now includes mvid and aotid in the stacktraces it generates. This makes them symboficatable but
with the side effect that the runtime adopts a more compact and slightly different format. Differences includes:
- short name for primitives (int, object, string, etc)
- <> for generic instances
- / for type nesting
- generics vars are appended with _REF
Commit 41272d634b removed mono/mini/mini-
exceptions-native-unwinder.c from the source tree but didn't update msvc
/libmono-static.vcxproj accordingly causing the VS build to fail.
we will rely on the output of debuggerd in order to get a native stack
trace. The arguments to support that:
(a) native stack trace by debuggerd is at least as good as the one we get via
libunwind (most of the time the trace provided by libunwind is useless?)
(b) we cannot screw up by accident (e.g. we have seen issues with a too small altstack)
(c) we don't need to maintain the libunwind integration into mono (which we
sort of had to do because of the upcoming dlopen limitation in Android 7.0
Nougat).
(d) the libmonosgen-2.0.so had about one megabyte more footprint (16mb ->
17mb). I'm talking about a debug build here though.
See mailing list discussion for more information:
http://lists.dot.net/pipermail/android-devel/2016-December/000026.html
partial cherry-pick of #4124
The existing code for marking custom attributes was only marking the type
and the lowest element type, i.e. marking (1) would also mark (3) but
would miss (2).
1. [System.ServiceModel.ServiceKnownTypeAttribute(typeof(System.Collections.Generic.Dictionary<string, object>[][]))]
2. [System.ServiceModel.ServiceKnownTypeAttribute(typeof(System.Collections.Generic.Dictionary<string, object>[]))]
3. [System.ServiceModel.ServiceKnownTypeAttribute(typeof(System.Collections.Generic.Dictionary<string, object>))]
That's generally not an issue since the type is marked (elsewhere).
However, in the case of custom attributes, we need to update the Scope
to what was resolved (and that was missed).
That lack of updated Scope means that a PCL assembly would still keep
references to System.Runtime, which is something that the linker is
eliminating (causing a TypeLoadException at runtime)
reference:
* https://bugzilla.xamarin.com/show_bug.cgi?id=47064
Note: The linker is not in mono/master and there's a PR in the new
linker repo for the same https://github.com/mono/linker/pull/7
Per .NET documentation, DynamicMethod does not support custom attributes
exception the MethodImplAttribute pseudo-custom attribute.
Also change GetMethodImplementationFlags to include NoInlining as on .NET
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=49686
mono_monitor_ensure_owned used to throw exception directly and not return. After switching the runtime to use pending exceptions, monitor code would continue running assuming we owned to monitor.
Fixes 46994
It needs special logic like the other monotouch profiles in the products
since we don't have a System.Drawing assembly/types for these profiles in Mono
and don't embed the types in the facade anymore after 1074fe91bd.
(cherry picked from commit 1562fc6fe5)
.. `Project.Build(..)`. For example, in:
```
<MSBuild Projects="bar.proj" Targets="foo;foo" />
```
The second invocation of `foo`, in the *same* `Project.Build(..)` call,
would get skipped, but BuildTarget assumed that no previous `outputs`
entry existed in the `targetOutputs` dictionary.
System.Exception: MSBuild operation failed ---> System.ArgumentException: Item has already been added. Key in dictionary: 'foo' Key being added: 'foo'
at System.Collections.Hashtable.Insert (System.Object key, System.Object nvalue, System.Boolean add) [0x00200] in <94fd79a3b7144c54b4cb162b50fc7761>:0
at System.Collections.Hashtable.Add (System.Object key, System.Object value) [0x00000] in <94fd79a3b7144c54b4cb162b50fc7761>:0
at Microsoft.Build.BuildEngine.Project.BuildTarget (System.String target_name, System.Collections.IDictionary targetOutputs) [0x0007e] in <b1524340ac7e4c06b22fac1b2db62c08>:0
at Microsoft.Build.BuildEngine.Project.BuildInternal (System.String[] targetNames, System.Collections.IDictionary targetOutputs, Microsoft.Build.BuildEngine.BuildSettings buildFlags) [0x00134] in <b1524340ac7e4c06b22fac1b2db62c08>:0
at Microsoft.Build.BuildEngine.Project.Build (System.String[] targetNames, System.Collections.IDictionary targetOutputs, Microsoft.Build.BuildEngine.BuildSettings buildFlags) [0x0004b] in <b1524340ac7e4c06b22fac1b2db62c08>:0
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=44549
While CMAKE_POSITION_INDEPENDENT_CODE will be automatically set if
BUILD_SHARED_LIBS is TRUE, it won't if BUILD_DYNAMIC_BTLS is TRUE.
This change sets CMAKE_POSITION_INDEPENDENT_CODE manually in such a case.
.. item.
If we have a project, that sets some properties or items at runtime,
like in a target:
```
<Target Name="Build">
<PropertyGroup>
<Bar>Bar01</Bar>
</PropertyGroup>
```
.. such a property can be accessed as `$(Bar)` subsequently.
Then at later point, a target tries to update metadata on an item group
with existing items, like:
```
<ItemGroup>
<FooItem Include="xyz" />
</ItemGroup>
<Target Name="Foo">
<ItemGroup>
<FooItem>
<SomeMetadata>MetadataValue</SomeMetadata>
</FooItem>
</ItemGroup>
```
.. then it is seen that the value for the earlier created `$(Bar)` disappears!
So, if we try to print the value of `$(Bar)` right after that item
group, it would appear as `''`!
The issue is that the metadata update in target `Foo` caused the project
to get re-evaluated, which meant that items/properites created after the
project load were lost! We should not be reevaluating the project when
setting metadata on the basis of dynamic items, like in the target
`Foo`.
These dynamic item groups in a target are represented as
`BuildItemTask`, and these create BuildItems which have their
`IsDynamic` property set. And this is used to avoid re-evaluating the
project in `BuildItem.SetMetadata`.
But when updating metadata (`BuildItem.UpdateMetadata`), we need to
update metadata on the *existing* items, which might not have been
created from such dynamic item groups! So, their `IsDynamic==false`.
Hence, trying to `SetMetadata` on such items would cause the project to
be reevaluated, and thus properties/items like `$(Bar)` would be lost!
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=45137
In this particular case, `Bar` was the `DesignTimeBuild` property, which
was losing it's value when `ProjectReference`'s `AdditionalProperties`
metadata was set in:
```
<Target Name="BclBuildAddProjectReferenceProperties"
<ItemGroup>
<ProjectReference>
<AdditionalProperties>$(_BclBuildProjectReferenceProperties);%(ProjectReference.AdditionalProperties)</AdditionalProperties>
</ProjectReference>
</ItemGroup>
```
Apparently there's no way to get CMake to just spit out its own uncached variables, so we can't do this programatically.
(cherry picked from commit 6843f6d61d)
Because we wouldn't return the correct ThreadPoolDomain for the requested domain, we would simply wait on the last inserted MonoDomain. This would lead to a hang when unloading the domain, as we wouldn't wait on the correct domain.
If we fail to send the signal, we should not add a pending operation, as the signal handler is never going to answer this operation (via mono_threads_notify_initiator_of_resume).
Because certain platform (such as WatchOS) do not support syscall abort, we need to ensure that it is supported before we call the `mono_threads_suspend_abort_syscall` function.
The SecureStringHelper from referencesource I added in #3887
relies on this to convert from SecureString to plaintext.
It failed on mobile because we always throw a NotImplementedException there,
but we actually have an implementation on the runtime side (even in the DISABLE_COM case).
(cherry-picked from df1475d95e)
It's an optional element according to [1] and we were simply
throwing a NotImplementedException when encountering it.
Fixed by reading the element value instead. I noticed that we
had a 'details' variable that was never assigned, so the check
for `details == null` was always true and can be removed.
This makes inlining the body of the CreateFault() call easier
since we no longer need to choose different overloads and makes
the code consistent with the CreateFault12 method.
Also added a better exception in case of an unrecognized element.
The CreateFault12 method (handles SOAP 1.2) does not need a similar
fix since actor is not an element in that version [2].
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=42226
[1] https://www.w3.org/TR/soap11/#_Toc478383507
[2] https://www.w3.org/TR/soap12/#soapfault
(cherry picked from commit 69b3397c8e)
* [BTLS]: Use `int64_t` instead of `long` in the native API.
* Copy time64.c from Android's source code to get a 64-bit timegm64().
* Rename into btls_timegm64() and remove everything we don't need.
* [aot-compiler] Add generic class and gsharedvt for hybrid AOT mode
* [aot-compiler] Remove dead code
* [hybrid-aot] Add --hybrid-aot for future use
* [aot-compiler] Cleanup GSharedVT check
* [aot-compiler] Enable partial sharing for hybrid AOT
* [aot] Enable GSharedVT for Android
* [aot] Ensure GSharedVT is supported at runtime as well as compile time
* [aot] Factor aot mode setting
* [mkbundle] - Add support for bundling native libraries.
* [mkbundle] add support for bundling native libraries
* [glib] add g_mkdtemp
* mono: Load the various library payloads for bundled executables
* [msvc] Export mono_dl_open and mono_loader_register_module
* [scripts] Add script to package the basic runtime to reuse by mkbundle
* [mkbundle] Add diagnostics, fix crash, fix registered path to match runtime name on embedded MonoImages (#3669)
* Resolve
* Complete the work, it should be working now
* Remove stray code from cherry pick
* [mkbundle] Fix the argument order to put the entry point on argv[1]
What was happening is that we set the stream EOF flag to true when we
could not read any more input data, and that made us early out before
fully finishing the processing of the compressed data.
Now the only EOF flag is set to true is if zlib's inflate() has actually
finished processing the data.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=44994#c2.
[C9][custom attr] Add low-level metadata iterator for assembly custom attributes
And use it to check for the reference assembly attribute.
Using the low level APIs means we don't trigger load of referenced
assemblies and we don't need to take locks, so it is safe to call this
during assembly loading.
Follow-up work for https://bugzilla.xamarin.com/show_bug.cgi?id=42584
This is a followup to e9205142b2.
Monodroid is missing the Cng and CryptoServiceProvider variants of the SHA
algorithms as well so we need to map to the Managed variants there too,
like on monotouch.
(cherry picked from commit e796ded61a)
Now that we're using a metadata-only ReferenceAssemblyAttribute check,
we can treat corlib the same as any other assembly - if we see
ReferenceAssemblyAttribute when loading a candidate assembly, skip it.
And use it to check for the reference assembly attribute.
Using the low level APIs means we don't trigger load of referenced
assemblies and we don't need to take locks, so it is safe to call this
during assembly loading.
This backports the mkbundle support that allows mkbundle to use complete SDKs, as opposed to a single runtime to bundle packages as completing the native library loading support.
* [mkbundle] - Add support for bundling native libraries.
* [mkbundle] add support for bundling native libraries
* [glib] add g_mkdtemp
* mono: Load the various library payloads for bundled executables
* [msvc] Export mono_dl_open and mono_loader_register_module
* [scripts] Add script to package the basic runtime to reuse by mkbundle
* [mkbundle] Add diagnostics, fix crash, fix registered path to match runtime name on embedded MonoImages (#3669)
This race condition has been introduced with 348d793ebb. This is because we would wait on the cleanup_semaphore outside of the threadpool->domains_lock lock.
By using a cond variable instead of a semaphore, we avoid a potential deadlock.
After 2fb07d6c6d BTLS can be used as a shared lib,
but this caused an issue in some monotouch tools which grepped the P/Invokes for `__Internal`
since these icalls wouldn't resolve on monotouch since BTLS is disabled there.
Instead, we now completely leave out building the managed parts of BTLS when
BTLS is not enabled.
(cherry picked from commit db21455da9)
The MacProxy class uses CFNetwork, but since CFNetwork is not a public
framework on watchOS, we can't use it.
So remove MacProxy completely (it only contains internal classes), and throw
PlatformNotSupportedException in any API that used it (the managed networking
stack is not supported on watchOS anyway, so this should be safe).
https://bugzilla.xamarin.com/show_bug.cgi?id=45847
* [btls] Convert BTLS icalls to pinvokes by invoking them using [DllImport("__Internal")], which will make it easier to redirect them to a separate dylib in the future.
* [btls] Add a --enable-dynamic-btls configure flag to enable compiling btls into a separate shared library instead of embedding it into the runtime.
In mono_assembly_load_from_full, the call to
mono_assembly_has_reference_assembly_attribute has to come after
image->assembly is set. This is because image->assembly->base_dir is
used for probing for referenced assemblies, and the reference assembly
attribute check will probe for assemblies referenced by the custom
attributes on the assembly in image being loaded.
Add 'ignore_missing' argument to mono_custom_attrs_from_index_checked:
if it is TRUE, a single custom attribute constructor that can't be
resolved to a MonoMethod won't cause the whole method to return NULL.
if 'ignore_missing' is FALSE, use existing behavior.
Updated all the call sites to pass FALSE, except
mono_custom_attrs_from_assembly_checked which also gets a new
'ignore_missing' arg.
Updated 'mono_assembly_has_reference_assembly_attribute to pass TRUE for
ignore_missing.
SecPolicyCreateSSL() distinguishes between an empty string (validation will fail
if the certificate contains host name constraints) and a null pointer (no host name
checking will be performed) for the hostname.
We incorrectly used `string.IsNullOrEmpty (host)` in a few places.
(cherry picked from commit 00f252d01f)
An assembly with a System.Runtime.CompilerServices.ReferenceAssemblyAttribute
may be loaded in a reflection-only context, but not for execution.
When loading assembly refs, or when loading assemblies by name (as
opposed to file path), a reference assembly will be skipped over and
other assemblies from the GAC and the load path will be tried.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=42584
information about network interfaces from the /sys filesystem. Google
placed restrictions on two files used by the Mono BCL to read
information about the interface:
/sys/class/net/[DEVICE]/flags
/sys/class/net/[DEVICE]/operstate
The information we used to read from `flags` let us determine whether
the interface supports multicast and the `operstate` file provided us
with detailed information on the current device state.
Additionally, it is now impossible to read the hardware MAC address used
by any interface. This bit of information is also unavailable when using
the Java APIs and therefore applications running on devices with API
24 (and possibly newer) will not be able to access this information.
More details about the issue are available from the upstream Android
bug:
https://code.google.com/p/android/issues/detail?id=205565
In order to obtain the required information, we now need to use the Java
APIs on devices with API 24+ and this commit implements the backend to
retrieve this information. It is done in C in order to avoid overhead of
creating mirror .NET objects when calling Java, thus saving time and
memory during an operation which may be performed by some applications
quite frequently.
This patch implements code that calls into the Xamarin.Android
runtime when runnin under API24 or newer to obtain the information.
Part of fix for https://bugzilla.xamarin.com/show_bug.cgi?id=44296
Enviroment::get_bundled_machine_config icall should be ves_icall_System_Environment_get_bundled_machine_config.
Mark all icalls doing get_bundled_machine_config as ICALL_EXPORT so maccore can link them in/out.
When connecting to an FTP server we weren't handling the RFC2428
extension to the FTP protocol which allows connections over IPv6
for the data channel.
We'd try to connect via IPv4 and the server would return an error
saying that this protocol is not supported.
The fix is to add handling for the protocol extension, namely
to use EPSV (for passive FTP) or EPRT (for active FTP) when opening
the data channel and send/parse the IPv6 addresses.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=44025
Note: we're now using the IP address of the the control channel we connected to for the
data channel instead of the address that is returned by the server in the FTP response message.
The reason is that for EPSV there's no IP returned by the server, and it should be
more reliable for normal PASV as well since the server might not know the
correct external address. This is also what MS does in referencesource.
(cherry picked from commit aebed51257)
* [System]: Move MonoBtlsProvider.IsSupported() -> MonoTlsProviderFactory.IsBtlsSupported().
Move the intern-call which is used to determine whether we're using BTLS out of MonoBtlsProvider
to allow the managed linker to eliminate the Mono.Btls.* classes when it's not supported.
* Fix Android build.
Without arguments, 'cert-sync' now operates on the old certificate store
again, like it was before. To test BTLS, you need to explicitly run it
with '--btls'.
We weren't checking whether a PE file is in PE32 or PE32+ format,
causing us to use the PE32 offsets on the newer format as well.
This means that using Process.Start() on a managed assembly didn't
always work when a PE32+ file was in play. Such a file is created
when targeting x64 e.g. via the -platform:x64 csc.exe/mcs option.
The reason why we probably didn't notice until now is that in an
assembly produced by mcs there happens to be some data at the PE32
CLR header offset even in a PE32+ file, causing the check to "work".
This doesn't apply to csc.exe/roslyn though and so we couldn't
execute those assemblies via Process.Start() even though they're
perfectly managed. Since the .NET Core project.json toolchain
produces x64-targeted assemblies by default more people ran into
the issue trying to run those on Mono: https://github.com/cake-build/cake/issues/1247
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=44937
PE/COFF spec at https://msdn.microsoft.com/en-us/library/windows/desktop/ms680547(v=vs.85).aspx
(cherry picked from commit 68bcbb4b77)
The epoch is specified as being in UTC, but the DateTime(int, int, int) constructor
creates a DateTime with an 'Unspecified' timezone, which could lead to problems later on.
(cherry picked from commit 8d1a65128a)
In mono_btls_x509_get_serial_number(), first call i2c_ASN1_INTEGER()
which takes care of padding and computing the 2s complement for negative
numbers, then reverse the buffer.
needs HANDLES
The method (which also goes by
System.Configuration.DefaultConfig:get_bundled_machine_config) was
converted to use coop handles earlier, but we missed this declaration.
RS takes the stance that redirected streams should not be closed by the Process class because they might
be referenced from the outside.
That would be ok if it was true. If you use Process::BeginOutputReadLine, the output stream becomes inaccessible
with no way to dispose them by any mean but GC.Collect. You get an exception when accessing Process::StandardOutput
once async reading starts.
This change addresses the case where the streams are not accessible. This is tracked by the *StreamReadMode variables
that are set on stream getters or Begin*ReadLine.
* [BTLS]: Allow fallback to MX.X509Certificate in X509CertificateImplBtls.
* [BTLS]: Fix X509Certificate2.MonoCertificate.
* X509Certificate2Impl.FallbackImpl: make abstract.
* X509Helper2.GetMonoCertificate(): new internal helper method.
* X509Certificate2.MonoCertificate: use X509Helper2.GetMonoCertificate().
This is only used by X509ChainImplMono; we should investigate whether we
could implement the missing APIs in BTLS.
* [BTLS]: Fully implement X509CertificateImplBtls.Import() and use correct semantics.
.NET's X509Certificate2.Import() uses several implicit fallbacks which we previously
did not handle in BTLS.
* you may provide a (unused) password when importing an unencrypted PKCS#8.
* when using null as password, it will attempt to use string.Empty if importing
without password fails.
* [BTLS]: Implement X509CertificateImplBtls.PrivateKey.
* [BTLS]: Implement X509CertificateImplBtls.Export().
* [BTLS]: Implement X509CertificateImplBtls.Verify().
* [BTLS]: Fix X500DistinguishedName quoting.
* MonoBtlsUtils: don't quote special chars in UTF-16 strings.
* X500DistinguishedName: don't decode raw data when passed in a name.
* [ci] Test System.dll with both the legacy and btls TLS provider
* [ci] Only run the additional btls System.dll test run on OSX for now
Otherwise we might run it on platforms where btls isn't supported yet and get errors.
The MonoBtlsProvider.IsSupported() icalls ves_icall_Mono_Btls_Provider_IsSupported()
needs to be available whether BTLS is available or not because it is used at
runtime to decide whether or not to enable BTLS.
This means we can't build Mono.Data.Tds.dll either, since it uses
Mono.Security.dll extensively. In any case Mono.Data.Tds.dll is useless
without a working networking stack, and since it's a Mono-only dll (i.e. not
part of netstandard or facades), just remove the entire assembly from the
watchOS profile.
System.Security.dll uses Mono.Security.dll as well, but the required classes
are available inside mscorlib.dll (as internal classes), so just open up
mscorlib internals to System.Security.dll.
The -arch option is an Apple extension, it caused failures on i386 Linux: "unrecognized command line option '-arch'"
(cherry picked from commit aad01ff4db)
* Don't invoke undefined behavior if an fd is >= FD_SETSIZE.
* Allow multiple command sockets.
* Close the command socket(s) on shutdown.
* Properly run cleanup logic on shutdown.
We need it in all cases now that we use it to periodically flush all buffers.
The only platforms it was disabled on was NaCl, which we haven't bothered to
support in a while anyway.
This also has the nice effect of reducing the amount of times we attempt to
take the reader lock. Now we take it and release it once for a single event.
Only in very few situations is it okay to access this field without first
taking the reader lock or exclusive lock.
Fixes some crashes observed in the profiler stress tests.
This avoids exposing the SGen worker threads through the profiler APIs. Doing
so had two main problems:
* It's easy for a user of the profiler API to accidentally mess with the state
of the worker thread in all sorts of ways, which could break the GC.
* It's been a long-standing guarantee that any profiler API callback will be
invoked on a thread that is, at the very least, attached to the runtime via
the mono-threads infrastructure. This is not the case for SGen workers.
This fixes a deadlock in the log profiler when using major=marksweep-conc.
* Emit an NUnit test report file.
* Set a timeout of 6 hours per test.
* Do a thread dump for timed-out tests.
* Don't print stdout/stderr for passing tests.
* mono_gc_enable_events: This did nothing for SGen and we called it
unconditionally for Boehm. So no point in having it around.
* mono_gc_enable_alloc_events: The performance difference between checking
a static alloc_events variable versus checking mono_profiler_events is
marginal at best and doesn't justify the indirection, especially now that
we support managed allocators while profiling.
Found by checkbashisms(1) as "unsafe echo with backslash". Some shells don't expand embedded \n in a string by default,
which broke the build on systems like ArchLinux or Alpine where /bin/sh points to those shells because the generated .lo
file contained just a single line instead of two.
Thanks to @JustArchi for researching this on the Mono Gitter channel, it made fixing the issue a lot easier!
(cherry picked from commit 1958bb8b34)
The HttpWebResponse stub that is used e.g. on the monotouch_watch profile was missing an overload that this commit was using.
(cherry picked from commit f4475ff013)
* external/boringssl: unmanaged code from Google (added earlier).
* mono/btls: unmanaged BTLS lives here (added earlier).
* System/Mono.Btls: managed BTLS lives here.
* Mono.Btls.Interface: The purpose of this assembly is to allow test suites
such as xamarin/web-tests to test parts of the BTLS APIs without making
System.dll's internals visible. It is not a stable and maintained API.
(cherry picked from commit 0f8f0c543c)
BTLS uses a different format and naming scheme for the local certificate store.
Before using BTLS for the first time on desktop, you need to convert the local
certificate store by either using the new `btls-cert-sync` tool (which will convert
your existing certificate store to the new new format) or `cert-sync` with the
`--btls` argument.
The new format and naming scheme is identically to what you'll find in
'/system/etc/security/cacerts/' on an Android device. It stores the certificates
in PEM format (with a human-readable text section added) using filenames which
are based on the certificate's subject name hash.
Previously, Mono stored the certificates in binary DER format, using a filename
based on either the Subject-Key-Identifier (if available) or the thumbprint.
We cannot access this from BTLS because computing these filenames requires
reading the entire X509 certificate (including the X509v3 extensions), but
their native lookup code only gives us the SubjectName.
This transitioning won't affect Android users as we're using the system
certificate store (including the user certificate store, via a Java callback).
* Mono.Security.X509.X509Store and X509Stores:
Support both the old and the new format. Using the new format requires BTLS.
* Mono.Security.X509.X509StoreManager:
Add `NewCurrentUserPath` and `NewLocalMachinePath` properties for the
new path names (~/.config/.mono/new-certs).
* Mono.Security.X509.X509StoreManager:
Add `NewCurrentUser` and `NewLocalMachine` properties. Requires BTLS.
* tools/security/cert-sync:
Add new `--btls` argument to use the new store.
(cherry picked from commit fb93be4490)
* [Mono.Security]: Remove Mono.Security.Interface.IMonoTlsContext.
This interface is not actually used anywhere anymore, so we can also
just get rid of it.
* [System]: Removed unused private TLS classes.
* [Mono.Security]: Cleanup the MonoTlsProvider's certificate validation code.
Each TLS Provider now has its own certificate validator and there is no
automatic fallback anymore. We only have three different TLS Providers,
"legacy", "appletls" and "btls" - and legacy is the only one which used
the managed validator. Both AppleTls and BTLS provider their own validator,
which invokes some native API.
All of the following were internal APIs.
* MonoTlsProvider.HasCustomSystemCertificateValidator has been removed.
* MonoTlsProvider.InvokeSystemCertificateValidator() has been renamed
into ValidateCertificate() and is now abstract.
* I got confused between [TestFixtureSetup] and [Setup], so fix tests whose
[Setup] methods were changed/removed to properly clean up after each test
(using a [TearDown] method) so that the next time the on-demand creation of
objects works as expected (i.e. objects created in the [Setup] method should
be created on-demand once for each method instead of once for each class).
* Also fix SocketAcceptAsyncTest, the change to properly catch exceptions on
background threads was incorrect.
LegacyTlsProvider now lives in the 'Mono.Net.Security' namespace
(previously 'Mono.Net.Security.Private') and derives directly from
the public 'Mono.Security.Interface.MonoTlsProvider'.
This allows it to be used by products, for instance replacing
xamarin-macios/src/Security/Tls/OldTlsProvider.cs.
The idea is to have all 'MonoTlsProvider' implementations inside
System.dll and not access any of the private APIs from outside of it.
This API is provided to be used by libraries providing crash reporting, so
that those libraries can install signal handlers and at the same time not
interfere with Mono's signals.
The intended pattern is this:
Mono.Runtime.RemoveSignalHandlers ();
InstallThirdPartySignalHandlers ();
Mono.Runtime.InstallSignalHandlers ();
This ensures that Mono is always notified first of any signals, and will then
chain to the third-party signal handlers if the signal did not originate in
managed code.
Mono's signals must first be removed, so that if third-party signal handler
does signal chaining, we don't end up in Mono's signal handler again.
Mailing-list thread: http://lists.dot.net/pipermail/macios-devel/2016-September/000016.html [Signal-chaining & crash reporters]
Example usage: https://github.com/rolfbjarne/HockeySDK-Xamarin/commit/862721199f1adec210bfc7b1943f975ffb8048a9
When there is an abstract class in the middle of an inheritance
hierarchy that doesn't override a method, get_base_methods needs to search
the parent class instead aborting early.
Example:
```
abstract class Root {
public abstract void Foo ();
}
abstract class Abs : Root { }
class Derived : Abs {
public override void Foo () { }
}
```
If we ask for the base method of Derived.Foo we used to get Derived.Foo
back because Abs did not provide an override. Instead we should skip
abs and look in Root.
When running with async suspend (ie, not coop), a user thread might be
suspended while its manipulating the coop handle stack. In that case,
we want to present a consitent view to mono_handle_stack_scan () by
adding memory fences to code that updates the stack chunks. In
particular we want to ensure that the chunk size is increased after
the new slot has a valid (null) value. Also when creating new chunks
make sure their size is initialized before the chunk stack is updated.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=43921 (at least the
one stack trace observed in [comment 1](https://bugzilla.xamarin.com/show_bug.cgi?id=43921#c1)).
The "global" TLS Provider (returned by MonoTlsProviderFactory.GetProvider())
may only be modified at application startup (before any of the TLS / Certificate
code has been used).
* MonoTlsProviderFactory.HasProvider has been removed.
This property used to initialize the TLS Subsystem without throwing any
exceptions. However, properties with side-effects is a bad design principle.
* Add MonoTlsProviderFactory.IsInitialized.
Does not have any side-effects.
* Add MonoTlsProviderFactory.Initialize() and Initialize(string).
May only be called at application startup and explicitly initializes the
TLS Subsystem.
* Rename MonoTlsProviderFactory.GetDefaultProvider() into GetProvider()
and removed MonoTlsProviderFactory.GetCurrentProvider().
We do not distinguish between a "default" and a "current" provider anymore;
there is only one "global" provider.
* Add MonoTlsProviderFactory.IsProviderSupported(string).
Checks whether a specific TLS Provider is supported, without having side-effects.
* Make the old APIs [Obsolete] until products have been updated.
* Add some documentation to MonoTlsProviderFactory.
* [System] Fix line endings for a few files.
Best reviewed by not showing whitespace changes (should be empty diff in that case).
* [System] Fix potential crash in CookieParserTests.
Socket.EndAccept can throw an exception, which will crash the process if it
happens on a threadpool. So handle any exceptions when calling
Socket.EndAccept, and propagate those exceptions out of any threadpool code.
* [System.Net.Http] Throw a PlatformNotSupported exception when using HttpClientHandler on watchOS.
* [System.Net.Http] Tweak tests after watchOS API change.
Unfortunately the [ExpectedException] attribute can't be put on an entire
class, so remove the [Category ("RequiresBSDSockets")] attribute on the class
and instead decorate each test with an [ExpectedException] attribute.
* [System] Extract the HttpListenerResponseHelper class to a separate file.
Extract the HttpListenerResponseHelper class to a separate file so that it's
easier to replace the HttpListenerResponse implementation by using a different
file.
* [System] Throw a PlatformNotSupported exception when using the networking stack on watchOS.
* [System] Tweak tests after watchOS API changes.
Unfortunately the [ExpectedException] attribute can't be put on an entire
class, so remove the [Category ("RequiresBSDSockets")] attribute on the class
and instead decorate each test with an [ExpectedException] attribute.
* [System] Fix potential crash in SocketAcceptAsyncTest.
Fix potential crash in SocketAcceptAsyncTest by catching any exceptions in
code executed on the threadpool so that we never end up with unhandled
exceptions (which may terminate the process).
* [System] Avoid logic in SetUp attributes in tests.
Any type of exception in [SetUp] attributes causes all tests to fail with that
exception.
This even happens for tests that do not need the logic in the [SetUp]
attribute.
So rewrite tests a bit to not use [SetUp] when we can reasonably avoid it, and
instead instantiate class-level variables on first use. This ensures that any
exceptions that occur when initializing those variables are attributed to the
corresponding test, and not all of them.
* [System] Make exception handler (for test) catch more exceptions to avoid unhandled exceptions causing process termination.
* [System] Remove Obsolete attribute on API that throws PNSE.
Introduction of the structured trace logger accidentally broke Android
logcat logging. The reason for this is that the default logging handler
defined in eglib (in goutput.c) for Android was never called, instead it
was superseeded by the log file logger defined in mono-logger.c
The net effect was that Mono tried to write the messages to some file on
Android leaving logcat devoid of any trace of life :)
This commit fixes the issue by introducing a set of structured logging
functions for Android.
As a side note - file logging on Android is not that useful. Writing to
a file on external storage medium requires app permissions and writing
to a location on internal storage medium, while allowed, might be
subject to space limitations and should not be used for possibly
voluminous data. It's therefore not very practical on Android.
* CertificateValidationHelper.GetDefaultValidator(): rename this internal method into
GetInternalValidator() (keeping the old as [Obsolete] until products have been updated).
* CertificateValidationHelper.GetValidator(): remove the 'provider' argument.
This is part of a set of cleanups and simplifications for the upcoming BTLS integration.
(cherry picked from commit 61d76d9642)
It is possible for SSLHandshake() to return SslStatus.Success while we're still
having a pending write (AsyncOperationStatus.WantWrite).
This is because our managed write callback must never return 'WouldBlock', so
SSLHandshake() things that all data have been sent while we still have them in
our write queue.
When this happens, we currently flush the write queue then call SSLHandshake()
again via AsyncOperationStatus.WantWrite -> AsyncOperationStatus.Continue.
This returns SslStatus.Protocol on iOS 10.
(cherry picked from commit 2cc1b887c1)
* Remove Mono.Security.Providers.NewSystemSource and Mono.Security.Providers.NewTls.
* Remove InternalVisible for "Xamarin.BoringTls".
* Remove Mono.Security.Providers.OldTls and Mono.Security.Providers.DotNet.
Satellite assemblies are special in that they all share the same file
name, but they live in separate directories named after the locale the
satellite assemblies pack. mkbundle currently doesn't support "clashing"
assembly filenames and will not be able to store the satellite
assemblies in the bundle for that reason.
This patch adds support for storing satellite assemblies with their
culture path prefix (e.g. en-US/MyAssembly.resources.dll) in the bundle
and also special code in the assembly loader to support such filenames
in the code which loads assemblies from the bundle.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=27061
The generic setter receives the element by reference which means we will never emit the write barrier since a reference to a reference is not a reference (mini_type_is_reference). Check the actual value instead. Fixes#42903.
Previously the tests were written so that some failures might occur on the
threadpool in methods that did not catch exceptions. In that case the process
would terminate due to an unhandled exception. So rewrite the tests to catch
exceptions in any code that doesn't run on the main thread, and forward those
exceptions to the main thread.
Also wait for async operations to complete on the main thread, and remove
asserts that are not supposed to be hit.
This is a backport of PR #3558.
[arm64] fixes around icache flushing
This fixes https://bugzilla.xamarin.com/show_bug.cgi?id=39859
The problem in `emit_thunk` and the workarounds for the Cortex A53 errata are unrelated to the bug above, but they don't hurt either.
The basic problem that we see is that the Exynos 8890 SoC (shipped for example in the world edition of the Samsung Galaxy S7) is a big.LITTLE architecture with two different CPUs, four of each. `__clear_cache` of GCC [1], the built-in that we use to flush the instruction cache, correctly reads the cache line sizes for data- and instruction cache from the CPU and then caches that result. However, the two CPUs on the Exynos 8890 have two different cache line sizes for the instruction cache (128 bytes vs. 64 bytes): If we happen to initialize the cache line sizes with the larger one, we only flush every other cache line if the code runs on the second CPU model.
Neither V8 [2] and LLVM [3] do caching of the cache line sizes, but just read it every time from the register. We have a slightly different solution, where we cache _and_ read the value every time and try to determine a global minimum across CPUs, in case we get scheduled to a different CPU by the kernel during the flushing loop.
[1] https://android.googlesource.com/toolchain/gcc/+/master/gcc-4.9/libgcc/config/aarch64/sync-cache.c#33
[2] https://github.com/v8/v8/commit/fec99c689b8587b863df4a5c4793c601772ef663
[3] https://github.com/llvm-mirror/compiler-rt/blob/ff75f2a0260b1940436a483413091c5770427c04/lib/builtins/clear_cache.c#L146
This reverts commit 2f88b8eb5a.
There is a subtle problem with turning the monitor thread into a simple
MonoThreadInfo: It creates worker threads via mono_thread_create_internal (),
which allocates the thread object from the GC. No thread can allocate from
the GC unless it's actually attached, which the worker thread is not when
created with just mono_threads_create ().
This handle should have a total of maximum 3 copies:
- in MonoThreadInfo->handle
- in the return value of mono_threads_create_thread
- in MonoInternalThread->handle (if it's a managed thread)
Before, there would always be 2 references (in MonoThreadInfo->handle and another another for MonoInternalThread->handle), even if it would be referenced at one place only (MonoThreadInfo->handle), which would lead to a handle leak.
We also need to duplicate the returned handle, to avoid having a use-after-free of the handle (in case the thread exits before we access that handle)
* [threads] Allocate StartInfo in create_thread
* [threads] Put start_notify semaphore in StartInfo
* [threads] Factor common attach code in mono_thread_attach_internal
* [threads] Use same registered mechanism as mono-threads
The tests that fail like this actually work just fine on Mono on Windows.
Added the NotDotNet category to the CurrentThread_Domains() test since it
fails on .NET.
* Avoid race condition between domain unload and threadpool.
Previous logic only ensured that domain->threadpool_jobs was 0.
There was a race condition as follows:
* worker jobs are queued into ThreadPoolDomain outstanding_request's
* domain unload starts
* domain->threadpool_jobs is 0 so unload proceeds
* queued jobs are processed from the ThreadPoolDomain outstanding_request's
* the domain can be set/entered by worker threads since
mono_domain_set only fails for domains with state of MONO_APPDOMAIN_UNLOADED,
while our domain is in the MONO_APPDOMAIN_UNLOADING state
* jobs are now being unexpectedly run in domain that is being shutdown
* Fix a hang when unloading the domain.
This change applies the diff from ludovic-henry to correct the hang.
https://gist.github.com/ludovic-henry/6c8a4bc8951091c0ef36df2e52d5e32e
For some reason preamble length was asumed to be 3 when cp 65001
instead of always looking att the preamble length of the encoding.
This resulted in responses cropped by 3 chars on windows.
* When doing async connect on windows, connect returns `WSAEINPROGRESS`
which needs to be returned to managed code for correct handling. However
on Windows, last error gets reset before `WSAGetLastError` is called.
This fix moves all `WSAGetLastError` calls up closer to the socket
operation to make sure correct error code is returned to the managed code.
* Making sure `TcpNoDelay` is set to false for IPv4 by default and
that `DualMode` is default for IPv6.
* Making sure option value is set to 0 before calling getsockopt
because on Windows, a `BOOL` option value (eg TcpNoDelay) is returned
as a one byte value and the other bytes are left untouched which resulted
in TcpNoDelay always seemed to be true.
* Fixed so `TF_REUSE_SOCKET` is set only if `reuse` parameter is set
when disconnecting a socket.
* [threads] Remove use of CREATE_SUSPENDED for domain unload thread
* [threads] Remove use of CREATE_SUSPENDED for managed thread
* [threads] Remove CREATE_SUSPENDED
If the sweeping mechanism has been triggered by a major collection, then the iteration will wait for the general sweeping to finish and also for the sweeping of all the blocks. If this isn't the case (we are doing a major collection) we will iterate over the current state of the heap, some objects being bound to die during the collection, and the callback must account for this.
It doesn't make much sense to explicitly attempt to iterate without trying to sweep blocks that can be swept.
We never enter this case since we never iterate over the major objects without requiring to sweep, while a sweep is running. Even if we would be in this case, we directly access the mark bits which can be cleared at the same time by another thread sweeping the block, so we could miss live objects.
It doesn't make sense to check the remset during a major since majors dont use the cardtable and we clear it at the start. So we will obviously have missing remsets. For the debugging concurrent case, we should use the mod union consistency checks instead.
We would need to deal with scanning forwarded objects and dead objects which is not worth it. Only verify the heap at the start of a collection, as indicated by the name of the flag.
When verifying the heap we need to have the set of pinned objects otherwise we can observe missing remsets if suspended between the store and the write barrier. If we don't have the set of pinned objects, allow missing remsets.
After https://github.com/mono/mono/pull/3452 was merged, the LinkSdkRegressionTest.SecurityDeclaration
test in XI started failing:
```
[FAIL] LinkSdkRegressionTest.SecurityDeclaration : FileIOPermissionAccess
Expected: null
But was: <System.Security.Permissions.FileIOPermissionAccess>
at LinkSdk.LinkSdkRegressionTest.SecurityDeclaration () [0x00033] in /Users/builder/data/lanes/1411/2a96404f/source/xamarin-macios/tests/linker-ios/link sdk/LinkSdkRegressionTest.cs:990
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/data/lanes/1411/2a96404f/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/corlib/System.Reflection/MonoMethod.cs:305
```
Due to the added call in the PR the linker preserved FileIOPermissionAccess enum,
which is apparently something we want to avoid.
Since according to https://github.com/mono/mono/pull/3452#discussion_r76384945 the only point of calling the
FileIOPermission is validating that the path doesn't contain invalid characters (mostly an issue on Windows),
I decided to just copy those checks as it's way cleaner.
Reference source of WindowsIdentity is using `MAXIMUM_ALLOWED`
as desired level by default when getting current token.
Changing to same behaviour to enable duplicate and impersonation
to be made on the token retrieved by GetCurrentToken.
* This improves the marshaling for packed classes and class involved in an inheritance hierarchy.
* Similar changes were made in the Unity fork of mono across a number of commits.
* The tests here have been taken from the repo and converted to work as NUnit tests in Mono.
Refcatored Win32RegistryApi.SetValue() to accept uint/byte/etc values for
DWord registry values.
Added handling of ERROR_CHILD_MUST_BE_VOLATILE errors to GenerateException().
Centralized MarkedForDeletion error handling to GenerateException().
Fixed tests which create volatile subkeys in RegistryKeyTest which used the
same key names but didn't clean up properly.
On Windows the TearDown() cannot delete the temp folder since it contains
loaded assemblies. The next time the test is run it will create a temp folder
inside the previous folder. After a number of runs the temp folder path will
approach the ~260 max path supported on Windows and some of the methods will
start failing.
This patch changes the way the temp folder is created and reduces the depth in
the same way as AssemblyTest was patched to fix a similar issue in
b90617e7b7.
Removed the differing code paths in FixtureSetUp() when running on non-Windows
platforms vs when running Mono on Windows or on .NET. The path previously
taken only on non-Windows platforms has been verified to work as expected on
both Mono on Windows and on .NET.
It failed a couple of times according to the babysitter logs with the following error:
```
MESSAGE:
Expected and actual are both <System.Int32[10]>
Values differ at index [1]
Expected: -1
But was: 0
```
Bump timeouts a little bit to see if this helps.
environment variables
Relying on the HOMEDRIVE/HOMEPATH environment variables is fragile as those
variables don't always have to be set. One such scenario is when running on a
Jenkins slave. When not set g_get_home_dir() in eglib would return NULL which
led to a crash in ves_icall_System_Environment_InternalGetHome().
This patch changes g_get_home_dir() to first try to call
SHGetKnownFolderPath() to get the current user's profile directory (i.e. home
dir). If that fails it falls back to the USERPROFILE environment variable. If
that isn't available the function will fallback to the original behavior and
try with HOMEDRIVE/HOMEPATH.
Creating new sockets between each connect instead of
trying to reuse the client socket. The original tests
was not working in the MS .NET runtime and socket reuse
is not recommended to use unless in very special cases.
On Windows the test used URLs like `file://C:\tmp\foo` which the
Mono.Security.Uri class was incapable of parsing. System.Uri from
referencesource parses such URLs just fine. This patch fixes the Uri class to
be able to parse such URLs. It also changes the Zone class to not throw an
exception if the Zone.CreateFromUrl() method is passed an invalid URL. Instead
the zone is set to NoZone. This seems to be how Zone in .NET behaves.
There are a few places in the runtime that should not be aborted as it would render it unusable.
cctor execution was the initial trigger for this changeset, but then edge triggering of TAE turned out
to expose this issue on other places.
This patch introduces those regions and apply them to the following places:
ves_icall_System_Runtime_Remoting_Messaging_AsyncResult_Invoke: Not delivering the completed CB
screws up any completion logic the user setup up.
mono_unhandled_exception: Protect delivery of the unhandled exception. Can't fail during it or we'll
effectively double fault and abort.
A thread abort must not start if the thread is running a .cctor.
OTOH, if the cctor self aborts, it should start.
Threads now track how many cctors they are running in the cctor_exec_depth variable.
The abort machinery (in async_abort_critical) checks that flag and bail from
starting the abort.
The cctor code (in mono_runtime_class_init_full) ensures that cctor_exec_depth is bumped
around execution of the cctor itself. Additionally, error handling was cleaned up as it was pretty
inconsistent.
This callback allows the runtime to query mini on whether there's a block guard in play.
It allows for the runtime to correctly decide when to start an abort.
Call g_log_set_default_handler in mono_trace_set_log_handler and mono_trace_set_log_handler_internal to ensure that g_ logging is properly routed.
Additionally, change the internal logging callback to take the log message as a single string to simplify things.
The patch of mono/metadata/marshal.c makes sure that
Marshal.StringToHGlobalAnsi() allocates a memory area for the returned C
string which is at least as large as the length of the string object. This is
the behavior on non-Windows platforms. Without this patch the
MarshalTest.StringToHGlobalAnsiWithNullValues() test wrote outside of a memory
area and corrupted the heap which later on caused a heap corruption crash when
running the corlib test suite on Windows.
The patch of mono/metadata/mono-security.c fixes the freeing of the memory
returned by GetNamedSecurityInfoW(). When GetNamedSecurityInfoW() is used to
request a DACL one must also pass an argument for the ppSecurityDescriptor
parameter and this is the one that should be freed later on, not the pointer
to the DACL as the original code used to free. Before this patch
DSACryptoServiceProviderTest.UseMachineKeyStore() crashed because of a heap
corruption.
In defer mode, the debugger thread is started as soon as possible before
any clients are attached, but the debugger is_attached flag was being set
immediately. This would cause the runtime (and the System.Diagnostics.Debugger
class) to wrongly report a client as being attached.
Originally fixed in the Unity fork here: https://github.com/Unity-Technologies/mono/commit/797a210e0f2416d779c448bb9e03bb16933e261f
In defer mode, the debugger thread is started as soon as possible before
any clients are attached, but the debugger is_attached flag was being set
immediately. This would cause the runtime (and the System.Diagnostics.Debugger
class) to wrongly report a client as being attached.
Originally fixed in the Unity fork here: https://github.com/Unity-Technologies/mono/commit/797a210e0f2416d779c448bb9e03bb16933e261f
All ia64 compilers have intrinsics that we use. Also, this ia64 code was very
out of date (missing many functions we use now) so it wouldn't compile anyway.
When encoding a UTF-16 C# string into bytes using "latin-1"/"iso-8859-1" encoding we were
not properly taking surrogate pairs into account. MSDN[1] provides a good introduction about the topic.
For example, this string would fail to encode and throw an IndexOutOfRangeException: "\ud83d\ude0a "
However, this string would not: "\ud83d\ude0a" (note missing space at the end). More on that later.
In https://bugzilla.xamarin.com/show_bug.cgi?id=42887 it was noted that encoding the string
didn't cause the exception on Mono 2.10.8.1 and 3.2.8. Indeed, it doesn't happen until Mono 4.0
where we switched most of the string code to referencesources.
While looking at the output of `Encoding.GetEncoding("latin1").GetBytes("\ud83d\ude0a ")` on the
"working" Mono verions I noticed that they essentially returned wrong results, e.g.:
```
csharp> Encoding.GetEncoding("latin1").GetBytes("\ud83d\ude0a ")
{ 63, 32, 0 }
```
This is wrong, because a) there's only one 63 (i.e. the '?' question mark char which is used as
a replacement char whenever the target encoding doesn't support a specific char,
and b) there is a trailing '\0' which should not be there.
Checking on .NET confirms the expected result: { 63, 63, 32 } i.e. '?? '
Back to the IndexOutOfRangeException in Mono 4.0+: digging into Latin1Encoding.InternalGetBytes() it
turns out that when we detect a surrogate pair we were incrementing charIndex, but we didn't decrement count.
This means that after the fallback characters (the two '?') were added we would still try to encode one more
char in the loop even though we're already at the end of the string, resulting in the exception.
For the "\ud83d\ude0a" example we would not hit the `count > 1` code path which made it work in this case.
Adding the decrement fixes the bug and makes the code consistent with MonoEncoding.cs[2].
The reason why it "worked" in earlier Mono versions was that we had a very simple EncoderReplacementFallbackBuffer
implementation[3] which only output a single '?' for the surrogate pair:
```
csharp> var buf = Encoding.GetEncoding("latin1").EncoderFallback.CreateFallbackBuffer()
csharp> buf.Fallback((char)0xD800, (char)0xDC00, 0)
true
csharp> buf.GetNextChar()
'?'
csharp> buf.GetNextChar()
'\x0
csharp> buf.Fallback((char)0xD800, (char)0xDC00, 0)
true
csharp> buf.GetNextChar()
'?'
csharp> buf.GetNextChar()
'?'
csharp> buf.GetNextChar()
'\x0'
```
This means that for the "\ud83d\ude0a " example from above we would only add one '?' character
for the surrogate pair to the result buffer, then the space and then we'd be done, leaving the
uninitialized array value 0 in the buffer on index 2.
I added tests for Latin1Encoding, mostly copied from ASCIIEncoding with additions for these
specific cases.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=42887
[1] https://msdn.microsoft.com/en-us/library/ms404377(v=vs.110).aspx
[2] https://github.com/mono/mono/blob/5b69e839e12b3d3512f310aa5a0e873346d7e50e/mcs/class/I18N/Common/MonoEncoding.cs#L62-L66
[3] https://github.com/mono/mono/blob/mono-3.12.0-branch/mcs/class/corlib/System.Text/EncoderReplacementFallbackBuffer.cs
Previously, we would try to satisfy the heapshot request every single time
process_requests () was invoked, but since heap_walk () would never unset
heapshot_requested, we'd end up doing GCs over and over, causing massive
overhead.
All events now come with a time field right after the event byte, and it is
encoded in the same way for all events (based on LogBuffer.time_base).
This allows readers to queue up events until a sync point event, then sort all
of those events by time, and process them in the order they were actually
written in.
Maintaining it for every format version is getting unwieldy and most readers
likely won't care about ancient format versions. Further, we haven't really
stabilized the format yet, so documenting every iteration of it doesn't make
a whole lot of sense. Finally, if somebody does care about older versions,
they can simply look in decode.c to see how mprof-report deals with them.
* Methods in backtraces are now properly encoded based on the method_base
rather than the ptr_base.
* The unused IL and native offset fields in TYPE_SAMPLE_HIT's backtrace frame
format have been removed.
* The unused flags fields in the backtrace format and in metadata events have
been removed.
* Changed a number of fields to be encoded as a single byte instead of LEB128.
These events basically establish points in the event stream where all previous
events can be processed and a complete view of the profiling state can be
formed. For instance, if an allocation event for object A of type T was seen
before a sync point, then it's guaranteed that the class load event for T also
came before that sync point (although it may have arrived after the allocation
event for A).
Other than timely arrival, there was never really a need to force buffers to be
flushed at these arbitrary points. Now that we flush all buffers periodically,
that reason is no longer relevant.
Heapshots have nothing to do with call depth, so they should be processed in
any case.
Also flush the local buffer in all enter/leave callbacks if needed.
This ensures that even for low-activity threads, it won't take ages before the
events from it arrive. We do this flush every 1sec (as with sampling of perf
counters).
The problem is this: Given an allocation of object O on address A1 in thread T1
and a subsequent GC move of O from A1 to A2 in T2, if T2's event buffer is
flushed before T1's buffer, the reader of the log file will a seemingly
nonexistent object being moved, and then see the actual allocation of that
object later. This makes it extremely difficult to keep track of the managed
heap.
The way we solve this is by flushing all thread-local buffers when the GC stops
the world. This way, we know that all prior allocations in the program have
been committed to the log file before any moves that are about to follow as
part of the collection. Similarly, we flush the GC thread's buffer before
restarting the world to ensure that all moves are committed before allocations
that may happen later (which could use some of the addresses freed up for use
due to the moves).
This is reasonably easy now that we have a global LLS of threads which are
attached to the profiler. However, we have to use some kind of lock to ensure
that buffers aren't being written to while we're trying to flush them, or that
a suspended thread was in the middle of modifying its buffer. For this, we use
a reader/writer spin lock of sorts, somewhat similar to the lock used by the
SGen binary protocol. When threads are modifying their own buffer (whether it
be writing to it or flushing it), they take the reader lock. This means that
many different threads can be operating on their own buffers without any lock
contention (which would kinda defeat the purpose of the thread-local buffers).
Now, when a thread -- such as the thread that initiated a GC -- wants to access
the buffers of other threads, it takes the writer lock. In doing so, all other
threads are prevented from modifying their own buffers. This gives the thread
holding the writer lock an opportunity to flush the buffers of other threads.
* Actual thread-local state is now stored in a MonoProfilerThread struct which
is stored in a single TLS slot per thread. This avoids storing things like
call_depth in LogBuffer. The current LogBuffer of a thread is also stored in
the MonoProfilerThread.
* All MonoProfilerThread structs are stored in a global linked list set. This
LLS is used to manually clean up the thread-local state of threads that, for
whatever reason, don't have the thread_end () callback invoked.
* Lifetime management of thread-local buffers is now much more straightforward.
* The thread-local list of referenced methods is now only allocated if needed.
* Fixed a MethodInfo memory leak in handle_writer_queue_entry ().
* Fixed some class load events that could be lost in the writer thread when
fetching the full name of a method.
* TLS slots are now actually freed on shutdown.
Make GC bridge perform well in the "double fan" scenario
The GC bridge currently performs VERY poorly when the object graph has a "double fan" shape. If M Java objects point to 1 C# object which points to N Java objects, the graph exported to Monodroid will currently drop the C# object node and replace it with (M\*N) edges. It is easy to write code where (M\*N) grows ludicrously big.
In testing on device, this patch solves that problem while leaving other cases unaffected. In testing on device, I found:
* All performance tests except double fan: No discernible performance difference after patch
* Double fan test with M=N=1000: Before the patch this took between three and seven seconds. After the patch this took between 0.2 and 0.3 seconds.
* Double fan test with M=N=4000: Before the patch this took anywhere from one to two minutes. After the patch, this took about 1.2 seconds.
* Double fan test with M=N=6000: Before the patch, this took so long I was not able to get it to ever complete. After the patch, this took about 1.3 seconds.
* Double fan test with M=N=20000: I did not attempt this test pre-patch. After the patch, it took about 5.6 seconds.
The commit messages describe the changes in some detail but the short version is:
* Add a mechanism for exporting non-bridged SCCs to the bridge client
* Turn that mechanism on whenever the product fanin*fanout grows too large
* Make the merge cache more aggressive
To function, this patch requires a corresponding change to monodroid. See https://github.com/xamarin/xamarin-android/pull/154
When running msbuild on mono against an architecture-specific
project that also defines satellite assemblies, the /platform
switch will be passed to al.exe. Mono's implementation of al.exe
does not support this switch, which results in a failed build.
This commit adds the /platform switch, which also required
changing the implementation from System.Reflection to
IKVM.Reflection because Mono's implementation of AssemblyBuilder
does not support the necessary options.
While testing this change, I also noticed a few attributes that
the MS version of al.exe will copy from "template" assemblies,
so I added support for those, as well.
SetUp() already sets up a temp folder for the test methods and TearDown()
handles cleaning up and taking into account that on Windows, loaded assemblies
cannot be deleted. However, some of the tests created temp files outside of
this folder and had their own cleanup code which didn't expect temp files to
be undeletable on Windows.
This patch removes any custom temp file creation/cleanup from individual test
methods. It also slightly changes the way SetUp() created the temp folder for
each test, now the paths will be shallower.
When AppDomain.Unload() is called a new thread is created and attached. When
this thread is done the thread handle is immediately closed using
CloseHandle() but the same handle is also passed to CloseHandle() later on by
the InternalThread destructor. On the second CloseHandle() call Windows will
abort the process with an "Invalid handle" error.
This patch removes the CloseHandle() calls in mono_domain_try_unload() which
is called by AppDomain.Unload(). The patch also wires up some more functions
in mono/utils/mono-threads-windows.c to set the MonoThreadInfo.handle field to
NULL when a thread has exited.
* Get rid of the arch-specific headers.
* Do verbose printing in one place.
* Simplify the cross compilation case to avoid repetition.
* Define all variables with a macro in a single header.
* Print detected hwcap info via g_print () so it's visible on mobile.
* Expose plain bool variables in the s390x implementation.
It doesn't work on nunitlite e.g. on XA:
```
src/corlib/Test/System.Runtime.InteropServices/MarshalTest.cs(268,11): error CS0117: `NUnit.Framework.Assert' does not contain a definition for `Less'
```
Previously every time we add a new runtime setting for the bridge
processor we have added a new callback for it. Because this is getting
unweildy, and also because some of these settings might have been
broken by recent init order changes, I created a new set_config
callback and merged the existing set_dump_prefix and enable_processing
with it.
hash_perturb was being mixed in in the wrong place. Fix that, also
make the hash function take some input from high bits on 64-bit
systems, also make the types around the hash more reasonable.
The names were fixed only for the subset of methods that are included in netstandard,
and in cases where netstandard differs from Desktop .NET we match with .NET.
* [runtime] Remove most of the contents of MonoDynamicGenericClass, the elements of the 'fields' array are not used anywhere, making all the other fields unneccesary too.
* [runtime] Remove the MonoDynamicGenericClass structure, merge it into MonoGenericClass. Add some comments.
On Windows we should check the path for invalid characters. Also expand
relative paths set using file:// URLs like .NET does. This patch also removes
the code which prepends "//" on Windows when a file:// URL is given as .NET
doesn't seem to do that in more recent versions.
When having a float variable in memory, we should always access its contents through the fload_r4/fstore_r4 funcs, which do the conversion between float and double and different memsize vs var size. It is common case, to have a temp float, have its address loaded with ldaddr and later do a load_r4_membase on that address to access the contents. If alias analyis lowers this load_r4_membase to an op_fmove directly on the register pair, this would in turn be decomposed into 2 simple int op_moves on soft float, which is wrong if these op_moves are converted to memory accesses.
Even though we register emulations for these opcodes, the native function doesn't not map to the opcode logic, being used instead as part of the opcode decomposition along with other instructions. Make sure we don't try to emulate these opcodes through the general emulation pass.
Add missing source file, sgen-os-coop.c to project.
Add source files and headers into correct filters (Header/Source Files).
Eliminated Visual Studio linker warning LNK4221 for libmonoruntime project generated
by empty source files under certain configurations, boehm-gc.c, lock-tracer.c,
null-gc.c and sgen-os-coop.c
If we happen to have the debugger or the log profiler enabled, we would run into a use before initialization case for the w32handle. The debugger or profiler would call `mono_gc_base_init` which would attach the current thread, and that would crash in `mono_w32handle_new` at w32handle.c:437 with `scan_mutex` being an invalid value (EINVAL).
FileNotFoundException on Mono on Windows and on .NET when target doesn't exist
FileInfoTest.MoveTo_DestFileName_DirectoryDoesNotExist() expects
FileNotFoundException to be thrown but on Windows a DirectoryNotFoundException
is thrown instead.
This patch changes MoveFile() in mono/io-layer/io.c to set the last error to
ERROR_PATH_NOT_FOUND rather than ERROR_FILE_NOT_FOUND when the destination
path doesn't exist. This results in a DirectoryNotFoundException being thrown
in managed code rather than FileNotFoundException. This is the behaviour
observed in .NET.
Also updated FileInfoTest.MoveTo_DestFileName_DirectoryDoesNotExist() to check
for DirectoryNotFoundException rather than FileNotFoundException.
The methods are already available in our implementations of those types so we can just forward the calls
to the "real" types instead of throwing NotImplementedException.
I audited all code in mcs/class/Facades and the only places where we still throw NotImplementedException
are ones where we really need to add a custom implementation.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=43695
It's unlikely the ServiceController APIs will be supported on mobile since they don't really make sense there,
they were initially designed for controlling Windows Services.
If the declaring type of a nested type marked by an xml descriptor
is not otherwise marked, the members of the nested type will not
be preserved, because the linker will not process the nested type
(since the parent type isn't marked, MarkStep:InitializeAssembly
will skip it, and thus skip akk its nested types as well).
https://bugzilla.xamarin.com/show_bug.cgi?id=43658
If sub == 0, i.e. there are no repeated separators, this method just returns
the input string. On Windows, where DirectorySeparatorChar (\) is different
from AltDirectorySeparatorChar (/), this means that the call
Path.GetDirectoryName ("foo/bar/dingus")
won't return "foo\bar" as .NET would but rather returns "foo/bar".
This patch makes sure the '/' -> '\' conversion is run when
AltDirectorySeparatorChar characters have been seen in the input string, even
if sub == 0.
PathTest.GetDirectoryName() triggered this bug.
When a FileStream is opened with the FileOptions.Asynchronous flag set the
code in file-io.c will pass FILE_FLAG_OVERLAPPED to the CreateFile() function.
On Windows this means the file will be in asynchronous mode and when
ReadFile()/WriteFile() are called an OVERLAPPED struct has to be passed. Mono
doesn't do this which means that when reading/writing asynchronous FileStreams
Windows returns an ERROR_INVALID_PARAMETER and an exception is raised in
managed code.
This patch simply removes the code which passes the FILE_FLAG_OVERLAPPED to
CreateFile(). This will work since asynchronous I/O is handled in managed code
anyway, using threads. On the OS level the I/O is still synchronous. The patch
won't affect other platforms since the implementation of
ReadFile()/WriteFile()/etc in mono/io-layer/, which is used on non-Windows, is
always synchronous and ignores the FILE_FLAG_OVERLAPPED.
This bug was triggered by the
MonoTests.System.IO.BinaryWriterTest.AsynchronousModeWrites test.
This test assumes that `new DirectoryInfo (@"c:").Name == "C:\"` when run on
Windows but the actual value depends on the current directory. If current dir
is C:\temp the Name property will return "temp". Similar problem in the
FullName() and Parent() tests.
This patch sets the current directory to C:\ explicitly before running the
failing tests on Windows. The current dir is reset in TearDown().
In an effort to cut down on the number of different scripts we have that try to output NUnit xml.
It also fixes a bug in emitnunit.pl where it would not output anything when one of the tests crashes
the process, because then the final "Overall results: tests: 22425, 100% pass, opt combinations: 25"
line isn't printed and emitnunit.pl just exits. This happens on Alpine.
DoubleTest.Parse() expects "3 5" to be unparseable in the current culture and
expects an exception to be thrown. In sv-SE however "3 5" parses to 35 and
this test fails.
This patch sets the current culture to en-US when the DoubleTest.Parse() test
is run.
When running DirectoryInfo.EnumerateFileSystemInfos() on an empty directory
MonoIO.FindFirst() will return null and the method returns without closing the
handle which is opened by FindFirst(). This patch makes sure the finally block
which closes the handle is run even when FindFirst() returns null.
Also added a call to FindClose() and some more cleanup in the native
implementation of MonoIO.FindFirst() when an allocation fails.
This bug caused an exception in DirectyoryInfoTest.TearDown() and several
other TearDown() methods in the MonoTests.System.IO tests on Windows due to
RemoveDirectory() failing on an empty directory which still had an open handle
associated with it.
The targets depended on a feature that doesn't exist in
xbuild (https://bugzilla.xamarin.com/show_bug.cgi?id=43645)
and as a consequence would fail to add the generated file to
the build when the GenerateTargetFrameworkMonikerAttribute
target was up to date.
With the .NET 4.6.2 merge in https://github.com/mono/mono/commit/b6a30059db8319348ebb454cc303633dfabd6a29#diff-799d44d25450b5b84906cbe7095708a0L86
the RSAPKCS1SignatureDescription class switched from using SHA1CryptoServiceProvider as the DigestAlgorithm to SHA1Cng.
This uncovered an issue when running SignatureDescriptionTest in FullAOT, as we were missing an entry for SHA1Cng
in CryptoConfig.fullaot.cs and so we didn't find the algorithm.
Additionally, 4.6.2 added new variants of the SignatureDescription class using SHA256, SHA384 and 512 which
had to be added alongside the corresponding digest/signature URLs defined by https://msdn.microsoft.com/en-us/library/windows/desktop/dd979768(v=vs.85).aspx
Note that on mobile/FullAOT we're falling back to the *Managed variants of the SHA algorithms as the *Cng/*CryptoServiceProvider variants
aren't exposed there (except for SHA1CryptoServiceProvider since it was in the framework from the beginning) so we need to special case these in the tests.
They were all using the same .cs source files so just including the generic monotouch .sources
makes it easier to reason about and reduces the risk of forgetting to update one of the lists.
SetUp() calls uname() in libc to check whether the test is running on OS X. On
Windows this fails since libc isn't available.
This patch removes the call to uname() completely since the result wasn't used
by the test anyway.
* Fixing linker warning when building libmono DLL.
After splitting mono runtime into a static library used when building libmono
DLL, we got a linker warning since there were no object files left in the
libmono project. There was also a sematic issue with the current organization
of DllMain since it is located in driver.c that will be compiled into the static
library and then consumed by the linker building the DLL. In cases where the
static library was consumed directly it would still include DllMain entry point.
By splitting the DllMain method implementation into a separate specific windows file
we can both resolve the linker warning and remove the DllMain method implementation from the static
library and only include it when building the libmono DLL.
* Visual Studio projects support to optimally link mono applications using static libmono library.
Added a new property, MONO_USE_STATIC_LIBMONO that can be used to link some
mono applications, for example mono(-sgen).exe, towards static version of libmono.
This gives us the option to build a binary without dependencies on other runtime binaries
minimizing what’s needs to be deployed in order to run mono. This in combination with static linked c-runtime
(another option available through a VS property) will produce a mono runtime binary without
addition dependencies except .NET assemblies deployed by mono runtime users.
The default is still using the dynamic version of libmono as before.
* Visual Studio project to test/debug testdriver tests from within Visual Studio.
Added an additional Visual Studio project preconfigured to run testdriver related tests
directly from within Visual Studio.
Commit also includes an additional configuration script that can be used test projects in order
to setup a mono config file. The testdriver project uses this to create a unique configuration
files pointing to the build libtest.dll for running configuration. The configuration file will then
be used when the tests are executed to make sure the correct libtest.dll for current build configuration
gets used when running test different tests from within the Visual Studio debugger.
The project comes with a number of user macros as well that can be used in order to tailor how the tests are run.
Most of the values are preconfigured and shouldn't need to be changed in order to use
current build. In order to change the test that is executed, use the following user macro
defined in mono-testdriver-test property sheet added to the project:
MONO_TESTDRIVER_RUN_TARGET
and point it to the assembly hosting the tests to execute from within Visual Studio.
* Visual Studio project to test/debug nunit tests from within Visual Studio.
Added an additional Visual Studio project preconfigured to run nunit related tests
directly from within Visual Studio.
The project comes with a number of user macros as well that can be used in order to tailor how the tests are run.
Most of the values are preconfigured and shouldn't need to be changed in order to use
current build. In order to change the test that is executed, use the following user macro
defined in mono-nunit-test property sheet added to the project:
MONO_NUNIT_RUN_TARGET
and point it to the assembly hosting the tests to execute from within Visual Studio.
There is also an additional user macro defined:
MONO_NUNIT_FIXTURE
This can be used to limit the number of nunit test run within a test suite.
The merge_cache previously, if it encountered an SCC ("color") with
many outgoing links, would not attempt to merge that SCC with existing
SCCs and instead would just create a new color. Because unbridged SCCs
are now reported to the bridge client, the cost of creating extra
colors has increased. To mitigate this the merge cache now attempts to
merge objects of all sizes, and has been changed to be able to do this
performantly.
The biggest change is that in the case of a hash collision, SCCs will
sometimes be merged without fully checking if they are exactly equal.
This is considered okay because false merges will be infrequent and
nothing worse will happen than short-term failure to reclaim memory.
The bridge-pathologies double fan test has been amended to verify
large-object merging works.
Augments each tarjan-bridge SCC ("color") with a count of how many
SCCs are pointing to it. The mechanism for exporting non-bridged SCCs
uses this count to only export those SCCs where the "double fan"
scenario appears to be occurring. In testing this eliminates the
performance overhead imposed by the previous commit.
Also renamed/moved some variable declarations because namespace bugs
were creeping in.
Introduces a mechanism for non-bridged SCCs which lay between bridged
SCCs in the SCC forest to be reported to the bridge client. In this
version, such non-bridged SCCs are always reported. The result is that
most GCs get a little slower but GCs in which the "double fan" shape
appears (many bridged objects link to one C# object which links to
many bridged objects) become massively faster.
Because before clients were allowed to assume exported SCCs were
always bridged, SGEN_BRIDGE_VERSION has been incremented.
Windows
The error (ERROR_DIRECTORY) wasn't recognized in MonoIO so the error message
became "Win32 IO returned 267. Path: C:\foo\bar" which includes the path. The
test tests that the path isn't include in the message in the thrown
IOException.
Previously we were wrongly trying to achieve this by doing a load-acq followed by a store-rel. Assuming we have memory access on address A (before) and C (after) while the atomic operation happens on address B, the order of the instructions would look like this :
[A], [B]load, load-acq, store-rel, [B]store, [C]
In this case a reordering of memory accesses like the following is possible, which clearly breaks the desired constraints :
[B]load, load-acq, [C], [A], store-rel, [B]store
In order to provide the sequential memory constraint we could emit the barriers like this instead :
[A], membar, [B]load, [B]store, membar, [C]
We can additionally save an instruction and emit the barriers like this instead (this is also what gcc does) :
[A], [B]load, store-rel, [B]store, membar, [C]
In this case we only need to worry about the relation between memory accesses on A and the loading of B. We need to consider what happens if B is loaded before the access on A is fulfilled. This only matters if we load an older value for B (rather than the one visible by other threads) before A. In that case the erroneous store will fail due to using exclusive stores. At the retry the value read from B will be the right one.
Loads with acquire semantics guarantee that following memory accesses are not reordered with the load. Add full membar before it to ensure full sequencing.
Stores with release semantics guarantee that previous memory accesses are not reordered with the store. Add a full membar after it to ensure full sequencing.
NET_2_1 is also defined whenever we define MOBILE, so they have the same effect. Silverlight was 2.1 but it's been dead for a while now, and we do not even support it.
On Win64 Mono doesn't (yet) implement fast JIT TLS (mono_get_jit_tls_offset()
returns -1). This means that the code in
mono_arch_create_handler_block_trampoline() takes the code path in the "else"
which is the same code path taken when AOT compiling. The code generated in
this code path manipulates the stack pointer to ensure it's aligned properly
according to the ABI (RSP-8 is 16-byte aligned on function entry) for the AOT
case where the mono_handler_block_guard_trampoline() C function is called
directly from the handler_block_trampoline.
This patch unifies the JIT and AOT generated code to be as similar as possible
and gets rid of the fast path. There's no need for this code to be fast. The
handler_block_trampoline will after this patch call
mono_handler_block_guard_trampoline() directly, both in JIT and in AOT mode,
and then jump back to the saved return address itself rather than let the
MONO_TRAMPOLINE_HANDLER_BLOCK_GUARD generic trampoline do this.
The tests mono/tests/finally_block_ending_in_dead_bb.exe and
mono/tests/block_guard_restore_aligment_on_exit.exe trigger this bug.
Split up reflection.c into three separate files:
1. reflection-query.c: This contains System.Type and the interrogative System.Reflection APIs.
2. reflection.c: This contains the System.Reflection.Emit APIs
3. reflection-custom-attrs.c: All about dealing with custom attributes.
Simplify code by always accessing tlab information and inside_critial_region through the SgenThreadInfo pointer. This also fixes potential failure to detect that we are in a critical region within the managed allocator.
MONO_THREAD_VAR_OFFSET definitions don't work when loading mono as a dynamic library. This means that we need to be conservative and don't aot code that contains these tls chunks.
In the future, we should remove HAVE_KW_THREAD altogether and use only pthread since it simplifies the code alot.
Added a couple of new/updated test/debug Visual Studio utility projects.
These projects can be used to quicker get up to speed when there is a need
to run/test/debug mono runtime tests inside Visual Studio debugger.
Added utility projects can be used to run mini regression tests using JIT as
well as compile/run full AOT:ed versions of mini regression tests or other assemblies.
The different .prop files are also organized so they can be reused when adding additional
run/test/debug projects going forward.
The code in mono_arch_create_handler_block_trampoline() in tramp-amd64.c emits
a relative CALL which calls the mono_amd64_handler_block_trampoline_helper()
functions via:
amd64_call_code (code, mono_amd64_handler_block_trampoline_helper)
This macro however only works properly for offsets less than 32-bits. For
64-bit offsets it truncates the offset to a 32-bit number. On Win64 the offset
in this case is always a number larger than 2^32. The emitted CALL will jump
to an illegal address causing a crash.
This patch changes the emitted code sequence to load the address of
mono_amd64_handler_block_trampoline_helper() to RAX and then calls the address
in RAX.
The test mono/tests/finally_block_ending_in_dead_bb.exe triggered this bug.
After this patch the test still crashes but in a new way related to another
bug.
* [corlib] Regression test for non-visible custom attributes.
See https://bugzilla.xamarin.com/show_bug.cgi?id=43291
* [reflection] Don't crash when building dynamic custom attributes on dynamic types (Fixes#43291)
1. Don't dereference a NULL custom attribute ctor, throw a type load
exception.
2. When building MonoCustomAttrInfo from an array of
MonoReflectinoCustomAttr*, compute the number of
non-visible (non-public) attributes correctly, and make sure to
iterate over all the attributes when populating the result array.
Fixes [#43291](https://bugzilla.xamarin.com/show_bug.cgi?id=43291)
* [reflection] Marginally better TLE for custom attrs
At least include a message about what went wrong when trying to
construct a custom attribute from a type that isn't finished yet.
I accidentally added this in 3c3ef17376 because I missed that on
full .NET this class inherits from NameValueCollection which provides `this[string name]` already
(the diff tool showed it as missing because it didn't take this into account).
The test fails due an ArgumentException being thrown when the catch handler in
the test's Run() method tries to write to the console. The ArgumentException
is thrown in EncodingTable.GetCodePageFromName(string) since the encoding name
passed to it is null. Normally this exception would be caught and the default
encoding would fall back to UTF8. However, since the thread is aborted the
ThreadAbortException will be rethrown and terminates the test with an uncaught
exception. I have verified that the same behavior is seen on OS X if I mock up
the default encoding code to throw the same ArgumentException.
This patch adds a call to Thread.ResetAbort() just before the call to
Console.WriteLine() in the catch handler. This removes the abort status of the
thread and avoids the rethrow of the ThreadAbortException.
We need a valid MonoThreadInfo in the TLS key while we run
unregister_thread because we may need to block the thread when it tries
to take the sgen GC mutex.
Fixes [#43099](https://bugzilla.xamarin.com/show_bug.cgi?id=43099)
The mono_threads_platform_get_priority()/mono_threads_platform_set_priority()
functions in mono-threads-windows.c assume the "handle" member has been set in
the MonoThreadInfo struct passed to it but it's always NULL. This patch
implements mono_threads_suspend_register() in the same file which sets the
"handle" member at thread registration time, like it's done in
mono-threads-posix.c.
This patch also makes sure the priority of a new thread is set as expected
when the Thread.Priority property has been changed prior to the thread being
started.
Also added a test method to mono/tests/priority.cs which tests that the
priority of the main thread is Normal by default and that it can be changed as
expected.
The changes in this patch fixes the failures in mono/tests/priority.exe on
Windows i386 and x64.
typeof (int).GetType ().GetType ().Name is now RuntimeType since https://github.com/mono/mono/pull/3045.
We did not notice this since most of our bots don't run with en-US locale set which this test requires.
We don't use svn anymore since a very long time and some diff implementations (e.g. in Alpine) don't support --exclude.
Removing the argument is an easy fix.
Previous implementation laid out the BSTR structure
in directly memory in C# code. This layout was not correct
for windows.
Furthermore memory was allocated using AllocCoTaskMem but later freed
using SysFreeString which on windows are different memory areas.
Third issue is that SecureString asumes big-endian when copying the chars
to its internal byte buffer. So on little-endian environments,
a swap of bytes is needed before the buffer is sent to native side
for BSTR conversion. This should probably be handled internally in
SecureString instead.
In the past there were wars between constified and unconstified
second parameter.. NetBSD went with constifying it. POSIX went without
const and most other OSes standarized. NetBSD is still having
incompatible function prototype.
LIBRARY
Standard C Library (libc, -lc)
SYNOPSIS
#include <iconv.h>
size_t
iconv(iconv_t cd, const char ** restrict src, size_t * restrict srcleft,
char ** restrict dst, size_t * restrict dstleft);
iteritems() was removed in py3, we can just use items() instead since we're iterating
over the list anyway so we don't need the lazy evaluation behavior of iteritems().
TODO: there's a gratuitous call to mono_lookup_internal_call_full().
Should really cache the uses_handle info along with the addr since the
lookup by name is rather involved.
Since UNICODE is defined when Mono is compiled under Windows the
GetLocaleInfo() macro will expand to GetLocaleInfoW() which returns the
current locale identifier in g_win32_getlocale() in 16-bit Unicode.
g_win32_getlocale() however expects an 8-bit encoding. By using
GetLocaleInfoA() explicitly we'll get the identifier back as an ordinary 8-bit
C string.
This patch fixes the mono/tests/threadpool-exceptions5.exe test.
MVID and AOTID metadata is no longer displayed at the end of the stack
trace.
Now, every stack frame contains a MVID and AOTID inlined.
An IL offset stackframe looks like:
at StackTraceDumper.Main () [0x00000] in <a3ec873a838c42c29f4eba7cab7d1dfb>:0
A native offset stack frame looks like:
at StackTraceDumper.<Main>m__0 () <0x100e15d50 + 0x00029> in <a3ec873a838c42c29f4eba7cab7d1dfb#b37a9d11247b811ffffdaff8090d8a59>:0
The crash would come from a use-after-free of stack memory: in coop_cond_timedwait_alertable, we would allocate the struct we pass to the mono_thread_info_install_interrupt function on the stack. But there is no guarantee as to when break_coop_alertable_wait is run. The fix is to allocate the data passed to mono_thread_info_install_interrupt on the heap, and free it in the callback, or if it hasn't been interrupted.
the stack.
Windows uses a guard page before the committed stack memory pages to detect
when the stack needs to be grown. If we suspend a thread just after a function
prolog has decremented the stack pointer to point into the guard page but
before the thread has been able to read or write to that page, starting the
stack scan at the thread's ESP/RSP will raise a STATUS_GUARD_PAGE_VIOLATION and
the process will crash.
This patch uses VirtualQuery() to determine whether the thread's stack_start
value points into the guard page. If it does the stack scan will instead be
started at the following, non-guard page.
Windows x64 only has 1 MB of stack per thread which is less than other x86
64-bit OSes. Using 10000 for "depth" in this test causes a stack overflow on
Windows x64. This patch changes "depth" to 5000 only on Windows x64. 5000 will
fit within the default stack size.
This is the recent behavior in .NET/MSBuild as well [1]:
> Starting in Visual Studio 2013, the MSBuild Toolset version is the same
> as the Visual Studio version number. MSBuild defaults to this Toolset
> within Visual Studio and on the command line, regardless of the
> toolset version specified in project file.
With this change xbuild will always use its own version instead
of the one specified in the project, unlessa ToolsVersion is forced
via the /tv switch on the command line.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=42938
[1] https://msdn.microsoft.com/en-us/library/bb383796.aspx#Anchor_1
For test harnesses which don't support the full babysitter protocol,
the test harness can now report back paths to NUnit-format XML files
containing test results, or the paths to those XML files can be given
by whoever invoked the babysitter script. Failures listed in the XMLs
are included in the babysitter log.
test-runner.cs has been modified to report its XMLs, so we should now
see mono/tests failures in the babysitter log.
The Process.Exited event would be triggered too early, leading to crash in code using this event to access the process exit code. This is because in Process.ExitCode, we first check that the process has really exited.
ABI states that "All memory beyond the current address of RSP is considered
volatile".
On Windows the stack grows dynamically as the thread needs more stack space.
Windows places a guard page before the page at the top of stack and when this
guard page is accessed by the thread Windows will add a new page to the stack
and resume the thread.
Before this fix SGen on Win64 subtracted 128 from the stack pointer of a
suspended thread and started the stack scan from this address. In the case
when the stack pointer would be less than 128 from the start of the memory
page SGen would attempt to read from the previous page. If this happens to be
the guard page a STATUS_GUARD_PAGE_VIOLATION exception is raised and the
process crashes. The sgen-weakref-stress.exe triggered this bug on almost
every invocation.
We split mono-threads-{posix,windows,mach}.c functions in 2 categories:
- platform: depends on the host the runtime is compiled on (possible values: posix or windows)
- suspend: depends on the mono-threads suspend backend in use (possible values: posix, windows or mach)
This involves a few steps.
First define MonoAssemblyName in RuntimeStructs.cs
Then add some helped to decode blob arrays and handle hex digits.
Finally, move the actual implementation from native to managed.
And call it consistently in thread start functions when the managed
thread start function returns with an exception.
(If coop is not enabled, the exception handler unwinds the thread and
when there's no handler invokes the unhandled exception hook via
jit_tls->abort_func, so this is just for the coop case.)
Fixes mono/tests/unhandled-exception-1.cs for coop and bitcode.
Allows safe allocation of native utf8 char* strings from managed
strings.
using (var h = new SafeStringMarshal (str)) {
/* use h.Value IntPtr */
} // g_free'd by h.Dispose ()
On musl __GLIBC__ is not defined, so the conditional logic will
not produce correct result. Add a specific case to handle when
__GLIBC__ is not defined.
It showed the following error before:
```
In file included from wapi.h:23:0,
from wapi.c:2:
../../mono/io-layer/processes.h:218:31: error: unknown type name 'pid_t'
gchar* wapi_process_get_path (pid_t pid);
^
Makefile:603: recipe for target 'wapi.lo' failed
make[3]: *** [wapi.lo] Error 1
```
This is necessary on weird devices where the hwcap code might fail to detect
the ARM version it's running on (usually caused by differences in /proc/cpuinfo
output).
Some Android devices for some reason have a 'model name' line instead of a
'Processor' line. On such devices, we would fail to detect the architecture
version at all, and end up thinking that it's ARMv4 (even if the runtime
was compiled for ARMv7).
When we shutdown we set suspend_finalizers flag in metadata so we no longer run any finalizers. After recent changes, we avoid traversing gc hashes for finalizer enqueuing when this flag is set. The counterpart, popping the queues empty (even without running the finalizers) can still take some time, especially since we lock for each entry. Exit early from this loop if we no longer need to run finalizers.
Achieve this by setting a new flag inside sgen, instead of passing the address of suspend_finalizers all the time to the sgen api.
By asserting on error, we ensure that we return from these functions only when successfull, or when it makes sense to return an error (for example, timedwait return -1 only on timeout or interrupt).
By asserting on error, we ensure that we return from these functions only when successfull, or when it makes sense to return an error (for example, timedwait return -1 only on timeout).
The following scenario might happen:
- thread A: returns from tpdomain D1, then parks
- thread B: returns from tpdomain D1, D1 is unloading so it's removed with `domain_remove`
- thread A: unparks, calls `domain_get_next` with tpdomain D1
- domain_get_next asserts as it can't find tpdomain D1 in the list of domains
To fix that, we simply set the previous_tpdomain to NULL when unparking.
This was from when we did a nursery collection as part of the
finishing collection pause. Now that we don't do that anymore, we
underaccount major collection times.
The store/store fence incurs quite a high cost, so we only want it to
run when absolutely necessary. The most efficient way is to
separately compile nursery object operations for when the concurrent
collector is running, so there has to be no dynamic check.
In the future we might want to try batching updates of references to
moved objects so we have to do only one fence per batch.
The missing remset check uses the binary protocol to log missing
remsets it found. Debugging is easier if we don't have to manually
flush the binary protocol.
If a thread is interrupted after a reference write but before the corresponding
write barrier, the remset will be missing, but the object will be pinned.
After the latest code changes, we may land here with `errors == SslPolicyErrors.None`,
so we need to make sure that we actually set it to `RemoteCertificateChainErrors` if
the system validator returns an error.
(cherry picked from commit b4304354b1)
Alternative implementation to 8f092147bb.
In CI we explicitly pass EMIT_NUNIT=1 and check-seq-points to generate the nunit xml and run the tests,
locally those are omitted. This avoids needing to sniff for Jenkins/Wrench variables.
Same for a9eec21d62, but we already pass V=1 in Jenkins.
When doing the first phase of sweeping we tag the block pointer within the block list with a CAS, after which we own the block and we are free to update the state and do sweep computations on the block. When we finish this phase we directly write the untagged block pointer back in the list, making the block available to inspect by other threads. We need a write barrier before writting the untagged pointer back because, when another thread might acquire the block, it might encounter an invalid block state.
I noticed this while looking at why Mono.CSharp is registered as "crashed" in the the new
fullaot PR job on Jenkins. It turns out all tests were excluded in mobile_static which means
no NUnit xml was produced and that is exactly what the script then interprets as a crash.
I saw that ASTVisitorTest.cs was not included in the net_4_x tests as well, likely inadvertently.
It got bitrotten over the years so I had to fix it up a bit, now it compiles/runs on mobile_static as well.
Since acceptance-tests/Makefile (and llvm/Makefile) include scripts/submodules/versions.mk which evals versions.py which looks
for a SUBMODULES.json file in the current working directory we got confusing error messages if the build directory is not the src dir:
```
Traceback (most recent call last):
File "../../mono/scripts/submodules/versions.py", line 22, in <module>
submodules = json.load(open(CONFIG_FILE))
IOError: [Errno 2] No such file or directory: 'SUBMODULES.json'
```
Instead we now pass the path to SUBMODULES.json as a variable to versions.py and set it based on $(top_srcdir) so it works
even for out-of-tree builds.
In order for our exception handling to work on linux, we need the
unwind-dw2-fde-dip implementation in libgcc to work with our
.o file. This requires that `--eh-frame-hdr` be passed by the linker.
By using clang++ as the linker, we will get this and other flags
necessary for correct interop with c++ libraries.
Fixes issues where ResponseStream.Close would throw an exception when
peer had already closed the connection and ResponseStream.Close would
still try to communicating to peer that it was closing.
The issue was brought to our attention by failures on:
- MonoTests.System.Net.WebRequestTest.TestReceiveCancelation
- MonoTests.System.Net.HttpWebRequestTest.TestLargeDataReading
A common pattern that we use across the test suite is to create objects on a new thread, join the thread, collect then expect certain objects to be dead/finalized since we assumed that thread to be no longer a source of pinning. This is not always the case since, after joining, the thread might still be alive and running unmanaged code related to the thread shutdown, which can reuse the stack that previously contained the managed references.
The workaround is to still allocate objects far down the stack, even if we are on a separate thread.
At the moment the helper can be used in tests which run under the TestDriver (since it's included in TestDriver.dll) and can also be included as a helper source for test bcl classes that need it.
This reverts commit cf94a6fcd4.
It seems this causes something that prevents the chroot from unmounting in the Linux builds on Jenkins,
resulting in an endless "can't unmount" loop and stuck builds. Reverting to see if this fixes it.
For objects that die at the same time, normal finalizers should be run before critical finalizers. When we were unloading a domain we were running finalizers for all objects in that domain without checking whether the finalizer is critical or not. Now we enqueue the objects first in the finalization queues so we can run all the normal finalizers before the critical ones.
Fix for the following failing pinvoke tests in runtime tests suite on win x64:
* pinvoke11.exe
* winx64structs.exe
* pinvoke_ppcs.exe
* pinvoke_ppcc.exe
- Microsoft.WebApplication.targets was inadvertently removed.
- The new Microsoft.NuGet.Build.Tasks assembly sources were not getting dist'ed. Adding to DIST_ONLY_SUBDIRS fixes this for now,
though we can maybe add it to SUBDIRS instead (but neither me nor Marek know for what exactly that's used)
ILASM.exe has features which a mobile_static runtime will not support.
It is invoked with an external mono when used in the runtime.
We skip it here because otherwise it will fail to verify.
Wrapper stack frames don't have MVID or IL associated.
ex: (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
This fixes an inconsistency where xbuild would add System.Core
references to all builds, and MSBuild would only add them for
.NETFramework.
This change could break user XA/XI/XM projects on Mac if the user has
manually removed the explicit System.Core reference from the project,
however such projects would already be broken on Windows.
This is required for interoperating with project.json-based NuGet package configuration files
(not to be confused with DNX or AspNet Core which use it as a project file too) from MSBuild/xbuild.
The task hooks into the build process, resolves assembly references from the nuget package cache
and injects them into the compilation project so the csproj doesn't need to be modified when
adding new references/packages.
It is the first step on our road to PCL5/NuGet3.
Also add Mono.RuntimePropertyHandle struct representing a native MonoProperty pointer.
Also add ves_icall_System_Reflection_PropertyInfo_internal_from_handle_type icall
that wraps mono_property_get_object_checked.
Add new Mono.RuntimeEventHandle struct wrapping a pointer to a native MonoEvent
Add new icall
ves_icall_System_Reflection_EventInfo_internal_from_handle_type that's
mostly a wrapper around mono_event_get_object_checked.
This happens because current implementation is doing a direct call to Win32
GetLastError in order to get the value before it get clobbered by the icall
setting the last error value in mono. When running AOT all constructs that tries
to call a method using a direct address will be dropped since that will not work
in full AOT. This results in incomplete compiled libraries causing runtime exceptions
in full AOT.
The fix on Windows x64 is to introduce a new OP code, GET_LAST_ERROR that will
retrieve the last error without clobbering it in the process. This is done by
emitting code reading the value directly from the TEB (thread environment block)
and will be lowered to just a move from the segment register holding the TEB data
structure on Windows x64. This is how GetLastError is doing this as well, but since
we won’t be able to call the method in a “safe” way in full AOT, using the segment
register directly will solve the issue. We use the segment register for accessing TLS,
so we already relying on the TEB structure.
NOTE, this will only be applied to Windows x64, full AOT for now.
be detected at runtime. Since the process crash, lot of valuable information was never
written out to the console (like the managed stack trace) needed to effectively resolve
the issue.
Current implementation used different value type ABI’s depending on if call
was a managed->native, native->managed or managed->managed call. The implementation
was also overcomplicated since it was a tweaked copy from System V x64 value type
ABI with scenarios and conditions that wouldn’t apply to the Windows x64 value
type ABI. Having a uniform value type ABI when possible regardless of managed->native,
native->managed or managed->managed calls simplifies scenarios and reduce complexity.
While all the above are legitimate reasons the primary driving factor behind
the rewrite was a different issue. There are .NET wrapper libraries around DirectX API
used by game engines and other libraries that uses a pattern using a call into a stub
managed method that gets patched directly in the MSIL to a calli that will end up in a
native DirectX vtable method. In current ABI we can’t detect scenario like this, so arguments
are passed as if the call was a managed->managed call causing issues for value types. Since DirectX
uses value types a lot (vectors, matrixes etc.), this was a blocker getting these technologies
running on Windows using DirectX at all. Another approach could be to change the behavior of the
library, but this would require a major redesign of the library and since several of these libraries
are mature, that wasn’t a realistic option, especially since MS .NET runtime handles the libraries as is
without modifications.
The rewrite makes sure Windows x64 value type ABI follow the platform value type ABI in all scenarios.
Also:
1. MethodBase.GetMethodFromHandleInternalType_native icall
2. reimplemented the other MethodBase GetMethodFromHandleXXX methods
in terms of this new icall.
3. MonoError-ize mono_class_get_methods_by_name
The mutex on pending_bridge_callbacks was overkill because the init
model for the mono runtime already enforces the callbacks being set on
the init thread. Added a couple comments to formalize this.
If we have an invalid SP in the suspended thread, we don't set stack_start in the thread info, expecting it to be 0 in order to signal a resume and suspend retry to the suspender thread. If the thread was already restarted before (from a different reason like IP in critical section), it would have a nonzero stack_start from the previous suspend, leading to an assertion in the previously described case.
This is also what we do on mach.
On Windows, xbuild can fail with the somewhat obscure error message: "error : Error building target GetReferenceAssemblyPaths: UNC paths should be of the form \\server\share." This occurs because a call to Path.GetFullPath(@"\") returns the string "\" on Windows, when it should return a root path that includes the drive letter, like "C:\".
The code in xbuild tries to scan directories for files. Specifically, it looks for "\.*dll". If any .dll files exist in the root directory, each will be returned with the path "/\foo.dll". This path is used to create a FileInfo, and the exception about a UNC path occurs.
So if the path is "\" or "/" Path.GetFullPath should return the correct root path, including the drive letter, on Windows.
Here is the full call stack from xbuild:
Error building target GetReferenceAssemblyPaths: System.ArgumentException: UNC paths should be of the form \\server\share.
at System.IO.Path.InsecureGetFullPath (System.String path) <0x25343f0 + 0x003fb> in <filename unknown>:0
at System.IO.Path.GetFullPath (System.String path) <0x2534378 + 0x00013> in <filename unknown>:0
at System.IO.FileInfo..ctor (System.String fileName) <0x25fae20 + 0x00033> in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.FileInfo:.ctor (string)
at System.IO.DirectoryInfo.GetFiles (System.String searchPattern) <0x25fa7c0 + 0x00087> in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo:GetFiles (string)
input, Int32 ptr, System.IO.DirectoryInfo directory, Boolean recursive) <0x25fa2b0 + 0x0016b> in <filename unknown>:0
input, Int32 ptr, System.IO.DirectoryInfo directory) <0x25fa270 + 0x0002f> in <filename unknown>:0
at Microsoft.Build.BuildEngine.DirectoryScanner.ProcessInclude (ITaskItem include_item, System.Collections.Generic.Dictionary`2 excludedItems, System.Collections.Generic.List`1 includedItems) <0x25d72d0 + 0x002ab> in <filename unknown>:0
at Microsoft.Build.BuildEngine.DirectoryScanner.Scan () <0x25d70e0 + 0x000ff> in <filename unknown>:0
at Microsoft.Build.BuildEngine.BuildItem.Evaluate (Microsoft.Build.BuildEngine.Project project, Boolean evaluatedTo) <0x25d5a50 + 0x005d3> in <filename unknown>:0
at Microsoft.Build.BuildEngine.BuildTaskItem.Execute () <0x25e5ad8 + 0x0002b> in <filename unknown>:0
at Microsoft.Build.BuildEngine.TaskBatchingImpl.Execute (IBuildTask buildTask, TaskExecutionMode taskExecutionMode) <0x25e3698 + 0x0004a> in <filename unknown>:0
at Microsoft.Build. BuildEngine.TaskBatchingImpl.Run (IBuildTask buildTask, TaskExecutionMode taskExecutionMode, System.Boolean& executeOnErrors) <0x25e50e0 + 0x0004f> in <filename unknown>:0
at Microsoft.Build.BuildEngine.TaskBatchingImpl.Build (IBuildTask buildTask, TaskExecutionMode taskExecutionMode, System.Boolean& executeOnErrors) <0x25e21b0 + 0x00097> in <filename unknown>:0
at Microsoft.Build.BuildEngine.TargetBatchingImpl.RunTargetWithBucket (System.Collections.Generic.Dictionary`2 bucket, Microsoft.Build.BuildEngine.Target target, System.Boolean& executeOnErrors) <0x25e14b8 + 0x00213> in <filename unknown>:0
at Microsoft.Build.BuildEngine.TargetBatchingImpl.Run (Microsoft.Build.BuildEngine.Target target, System.Boolean& executeOnErrors) <0x25e1360 + 0x000f7> in <filename unknown>:0
at Microsoft.Build.BuildEngine.TargetBatchingImpl.Build (Microsoft.Build.BuildEngine.Target target, System.Boolean& executeOnErrors) <0x25dfee0 + 0x00053> in <filename unknown>:0
at Microsoft.Build.BuildEngine.Target.DoBuild (System.Boolean& executeOnErrors) <0x25dfcf0 + 0x0004f> in <filename unknown>:0
Enable compact seq points by default so that the stack trace have IL
offsets instead os native ones.
This changes will allow any stacktrace generated from now on to be
symbolicated using mono-symbolicate.
Deprecates options gen-compact-seq-points.
This is cleanup to the previous emergency fix for bug 42469. The goal of that fix was to defer bridge processor initialization until after sgen init, but it introduced a requirement mono_gc_register_bridge_callback be called before sgen init. This breaks Xamarin Designer, which sets up callbacks late.
This patch introduces an sgen_init_bridge () which is called both at startup and after callbacks are set, and performs the initialization once all pieces are in place. Additional concurrency and style issues are also addressed.
* [corlib] Rewrite RuntimeType.GetGenericParameterConstraints
using managed code for managed object manipulation by surfacing the
native MonoGenericParamInfo struct to managed code as Mono.RuntimeStructs.GenericParamInfo
* [corlib] RuntimeType::GetGenericParameterAttributes can be managed code now.
* [corlib] Fix x86 breakage
RuntimeTypeHandle.GetGenericParameterInfo returns a pointer
Not a struct that's the same size as a pointer. On the x86 Linux
calling convention they behave differently.
mono_gc_register_bridge_callbacks is a function which the android implementation calls before initializing the runtime. One of the things this function previously did was initialize the default bridge processor if none has been initialized yet. However, because the function gets called before initializing the runtime, a bridge processor was never initialized. This lead to two bad effects:
- If a bridge processor was ever manually specified, the bridge processor would get initialized twice
- Tarjan crashes if initialized before sgen, so when Tarjan became the default bridge processor Android always crashed on launch
The default bridge processor is now initialized at the end of sgen_gc_init.
MVID and AOTID stacktrace metadata no longer displays guids with '-'.
Metadata is now also using lowercase instead of uppercase.
Changed mono-symbolicate and mono --aot to use the minimalistic GUIDs.
Replaced static constructor with delayed initialization.
Fixed possible undeterministic order of dumped MVIDs.
Used AppendFormat when possible.
Added comment to StackTrace.AddMetadataHandler explaining that it can be
used with reflection by outside code, to avoid removing it or changing
its signature.
Added StringComparer.Ordinal to dictionary.
Extracted Symbolicate.Process stacktrace dump logic into DumpStackTrace
method. Which can also be used to dump the last stack trace after the
process loop.
mono-symbolicate can now be called:
symbolicate store-symbols <msym dir> [<dir>]+
This will store in <msym dir> all the managed symbols found in the
provided directories.
mono-symbolicate now parses stack traces with metadata at their end.
All the stack trace and metada are read, so that symbolicate can use
MVID and AOTID to retrieve the symbols from the msym directory..
The MVID metadata handler displays multiple MVID(s) followed by the
comma separated list of StackFrame lines whose method belongs to the
module with the displayed MVID.
Added a reflectable static method that can be used to customize data
displayed after a stack trace.
Added metadata handler that display the AOTID of the executing assembly.
AOTID is a GUID generated by the AOT compiler and stored along with the
native code.
The AOTID uniquely identifies the generated native assembly.
The AOTID is required when we want to map native offsets to IL offsets
and the sequence points mapping those values need to be determined.
Previously handles were pointer-sized structs. Unfortunately not all
calling conventions treat pointer-sized structs as pointers, so if we
wanted to interact with handles from wrappers, we would have to do
something complicated.
With the new design, `TYPED_HANDLE_DECL(MonoFoo)` will expand to:
```c
typedef struct {
MonoFoo *__obj;
} MonoFooHandlePayload;
typedef MonoFooHandlePayload *MonoFooHandle;
```
- A single dll can be mapped to different targets based on function name.
- The dllentry target can be omitted if it is the same as the source.
This change is released under the MIT license.
Exceptions.exe regression test started to fail on win x64 release builds.
Turns out that the Visual Studio C compiler picked a different optimization
for release builds (due to a change of library type that uses a different default optimization level)
using RSI register in native code calling into JIT:ed code. If the JIT:ed code used reflection
and exceptions, that triggers InternalInvoke and lmf_filter, the code only
stored System V x64 non-volatile registers in the unwind information. On Windows,
RSI is also a non-volatile register (RDI as well) that needs to be preserved and
since the code didn’t include those in the unwind information, the correct RSI value
was not restored during the unwind. Once the control was handled back to the native
code it dereferenced RSI (that was set to 0) and crashed.
The fix is to make sure we store all non-volatile registers on x64 Windows
when we save an lmf so the unwind code can restore them when unwinding through
the lmf.
* Making sure ownerThread is not nulled unless lockCount is zero.
* Fixed so Locked property checks lockCount when determining if locked
instead of _locked field which was never set.
* Added test for setting Locked property.
* Added test capturing the original issue.
I saw this failing occasionally on iOS on Wrench:
```
[FAIL] TimerTest.AutoResetEventFalseStopsFiringElapsed : #2 wrong elapsedCount
Expected: 1
But was: 0
at MonoTests.System.Timers.TimerTest.AutoResetEventFalseStopsFiringElapsed () [0x00091] in /Users/builder/data/lanes/1411/b07703ed/source/xamarin-macios/external/mono/mcs/class/System/Test/System.Timers/TimerTest.cs:327
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/data/lanes/1411/b07703ed/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/corlib/System.Reflection/MonoMethod.cs:309
```
* Fix for non GCC win x86 calling convention when passing empty structs over pinvoke.
Port of https://github.com/mono/mono/pull/3085 to Winwdows x86 fixing failures
in pinvoke2 tests:
test_0_marshal_empty_struct
* Fix for incorrect implementation of small struct (single float/double) on Windows x86.
Failure in pinvoke3 test_0_marshal_small_struct_delegate9 is caused by incorrect
implemented small struct ABI on Windows x86.
Windows x86 ABI for returning structs of size 4 or 8 bytes (regardless of type)
dictates that values are passed in EDX:EAX register pairs,
https://msdn.microsoft.com/en-us/library/984x0h58.aspx.
This is different compared to for example float or double return types
(not in struct) that will be returned in ST(0),
https://msdn.microsoft.com/en-us/library/ha59cbfz.aspx.
Fix disables the alternative to do special handling of structs with 1 float or double
and make sure same logic is always applied to all small structs regardless type
as dictated by Windows x86 ABI.
* mono_test_marshal_delegate_ref_delegate uses incorrect calling convention.
mono_test_marshal_delegate_ref_delegate called by pinvoke3::test_55_marshal_delegate_ref_delegate
uses incorrect calling convention. Mono assumes delegates to be stdcall but
current definitions will be cdecl, this will cause an incorrect stack pointer
on architectures where cdecl and stdcall are different (Windows x86). Test fails
with "corrupt" stack pointer on Windows x86.
* Smaller adjustment based on feedback.
* Changed boolean struct member to bit field (on both x86 and x64)
* Changed compare code style from 0 == x to x == 0 to comply with coding standard.
libmono will use static library when building its dynamic library. This gives
the option to build binaries that could link the static version of mono runtime
or use the dynamic version of the same library. The default build binaries still
link against the dynamic version of the runtime.
The conflict happens when a type is declared like this:
.class value private auto ansi serializable sealed foo extends [mscorlib]System.Enum
The value keyword would make it a value type and ignore the parent declaration.
.net ilasm favors the later, which makes more sense.
I addressed the issue mentioned in the comments of mono_method_can_access_method:
* FIXME:
* with generics calls to explicit interface implementations can be expressed
* directly: the method is private, but we must allow it. This may be opening
* a hole or the generics code should handle this differently.
* Maybe just ensure the interface type is public.
The way to do it is to check access between gmds
Covers #41564
When multiple methods shared soap header a duplicate key exception would
be thrown.
This issue only occurs on XAMMAC and MOBILE where the non reference
source serializer is still used.
.NET supports this, from MSDN: "If an entry with the specified path and name already exists in the archive, a second entry is created with the same path and name.".
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=42219.
* [remoting] Convert mono_load_remote_field_new_checked to managed code.
* [corlib] Expose runtime struct MonoRemoteClass and MonoClass to managed
Also expose mono_class_get_type to managed code.
* [corlib] Add Mono.RuntimeClassHandle to encapsulate MonoClass*
and Mono.RuntimeRemoteClassHandle to encapsulate MonoRemoteClass*
* [runtime] mono_marshal_get_ldfld_remote_wrapper invoke the managed TransparentProxy.LoadRemoteFieldNew method directly
Don't bother going through a native call and back out to managed.
* [runtime] Get rid of MONO_WRAPPER_LDFLD_REMOTE
When the MONO_WRAPPER_LDFLD is operating on a proxy, it always invokes
the TransparentProxy::LoadRemoteFieldNew managed method
* [runtime] Remove dead mono_load_remote_field_new_icall
* [mini] Bump AOT file format version
Because we removed the LDFLD_REMOTE wrapper type so all the
MonoWrapperType enum values shifted.
[utils] Fix retiring of live block for the lock free allocator
Freeing a block descriptor (along with the block) can happen either when freeing a pointer in the block (we do this in order to prevent situations where we need to wait for the next alloc in order to free a block), either when we acquire an empty block while allocating. Either way, in order to free a block descriptor, we need to acquire it first.
When freeing a slot in a block, we are checking if the block is now empty in which case we acquire the block descriptor in order to free it (along with the block). While doing so, we were failing to account for the case where, by the time we make the block descriptor empty and the time we acquire it, another thread would try to allocate from the same descriptor, see that it's empty, free it, allocate a new descriptor (which would happen to have the same address as the retired one) to fulfill the allocation and then set it as the active descriptor for the allocator. When the first thread finally acquires the descriptor, we need to double check that it is still empty since it might be logically a different descriptor than the one we actually freed from. In the unlikely case it's not empty, we need to put it back in the allocator's data structures.
Fixes#41616
ConnectAsyncCallback now throws an exception when called a second time.
ConnectAsync reuses SocketAsyncEventArgs in BeginConnect which
already updates its state and calls ConnectAsyncCallback.
The changes make sure that ConnectAsyncCallback no longer called on
ConnectAsync catch block, while the state is still updated with the
exception.
Covers #41616
Our console runner is currently swallowing this test unhandled
exceptions.
To make it test show the failure I had to set legacyUnhandledExceptionPolicy
to 0 in mcs/class/lib/net_4_x/nunit-console.exe.config
Then running we can observe the crash by running:
make -C mcs/class/System check TESTNAME=System.Net.Sockets.SocketTest.ConnectAsyncUnhandledEx
Other tests appear to crash with legacyUEP set to 0, while the runner
shows no errors.
When we cleanup the gc at shutdown, we no longer care about running the remaining finalizers so we set the suspend_finalizers flag. This flag skipped running the finalizers but we would still traverse and modify the sgen finalizer hashes (on an arm box this could take even longer than 1s and lead to finalizer_thread_exited assertion failure since it's over the 100ms limit).
Avoid traversing sgen finalization hashes as soon as we see that we are shutting down.
libmonosgen-2.0.dll and breaks the build of monograph.exe which links against
it.
Also adds -O2 to CFLAGS on all platforms to align with how releases are built.
This should make it easier to programmatically check for errors, since it will remove the need to test the output string, which is an unreliable way of doing things.
Freeing a block descriptor (along with the block) can happen either when freeing a pointer in the block (we do this in order to prevent situations where we need to wait for the next alloc in order to free a block), either when we acquire an empty block while allocating. Either way, in order to free a block descriptor, we need to acquire it first.
When freeing a slot in a block, we are checking if the block is now empty in which case we acquire the block descriptor in order to free it (along with the block). While doing so, we were failing to account for the case where, by the time we make the block descriptor empty and the time we acquire it, another thread would try to allocate from the same descriptor, see that it's empty, free it, allocate a new descriptor (which would happen to have the same address as the retired one) to fulfill the allocation and then set it as the active descriptor for the allocator. When the first thread finally acquires the descriptor, we need to double check that it is still empty since it might be logically a different descriptor than the one we actually freed from. In the unlikely case it's not empty, we need to put it back in the allocator's data structures.
As described in https://bugzilla.xamarin.com/show_bug.cgi?id=34883 at
least Samsung SG3 doesn't allow access to the local network
interface (in fact, even ifconfig doesn't work on the device). Therefore
the test needs to be disabled on those devices. This is the first part
of the fix, the second one will be implemented in Xamarin.Android test
suite which will provide the AndroidShouldPingWork method which will
check the device id against list of those that do not work for this
test.
Part #1 of fix for https://bugzilla.xamarin.com/show_bug.cgi?id=34883
MXR is not coming back (see https://bugzilla.mozilla.org/show_bug.cgi?id=1279952 and https://github.com/SuaveIO/mono-script-buildpack/issues/9#issuecomment-226749314)
and this broke everyone who still relied on mozroots. Fix by using the version from the hg repository instead.
Starting from Mono 3.12 we've been shipping cert-sync to import CA certificates from the local computer trust store to Mono's trust store,
we strongly encourage everyone to move to that instead. Added a deprecation warning to mozroots to further strengthen this message.
Note that when fetching the CA list over the new https URL we need to skip cert validation errors, as the common use case for mozroots is when
no certs are present on the system (bootstrapping). This enables MITM but we don't really have any other options. Another reason why cert-sync is superior.
In some cases the "git log -1 <remote>" check fails even though the remote is configured
and an unnecessary fetch is done. Adding the branch to the check fixes this.
* An enhancement to the thread priority work added earlier. This will support starting a thread with a specified priority. Currently, the thread will start with the priority of the creator. The semantics of threads allow you to specify the priority of a thread before it has started. Therefore, we take note of this value at thread creation time.
* Missed committing this file the first time
* Correct typo
* Fix creation_flags reference
* MonoThreadParm really should not be in objects-internal
* Place priority in MonoInternalThread where it can be accessed by GetThreadPriority(). This enablses the priority to be set/retrieved before the thread is started. It also ensures that new threads default to a priority of THREAD_PRIORITY_NORMAL rather than inheriting the parent thread's priority.
* Revert changes to MonoInternalThread/Thread.cs. Properly use the handle for thread lookup. Initialize priority at thread creation time. Use stored value of priority if thread has not been started.
* Add code to handle retrieving the thread priority from a started thread that had been set before thread had started. This is necessary as before the thread is started the priority value can be manipulated. This value is stored in the MonoThread structure. Before the thread is started the value can be interrogated so we return this value rather than invoking the GetThreadPriority() API. When a thread is started this value is used to set the threads priority (if scheduling policy is SCHED_FIFO or SCHED_RR). For POSIX threads, after the thread is started we now keep the priority in the _Wapi_thread structure which is located via the thread handle. This structure doesn't exist until after the thread is started so we can't store the value there but after we can cache the priority value there and return it for non-SCHED_[RR|FIFO] threads. This way the expected value will always be set and returned by ThreadPriority property (currently for these threads it will simply return Normal). A test case has been added to the mono/tests directory.
* Use a copy of creation_flags so local modifications aren't propagated
* Use pthread_setschedparam as OSX doesn't support pthread_setschedprio
This fixes the bug where form authentication tickets do not expire until 50 years later.
I bumped into this bug after upgrading mono from 4.2.2 to 4.4.0
Firstly, the authentication ticket's expiry must always equal to the timeout attribute on the <forms/> element. It doesn't matter whether persistent or session cookies are used.
Secondly, if the cookie is persistent, then it's expiry should be set to the same as that of the authentication ticket.
Reference: https://blogs.msdn.microsoft.com/dansellers/2006/02/15/change-to-asp-net-2-0-forms-authentication-persistent-cookies/
This should fix some problem with Nuget archives created by Mono.
No test case for this one because it would involve writing a mini-ZIP file parser to make sure the value is written correctly (or hardcore the value offset in the generated file, which could lead to a very flakey test).
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=41775.
Updated Visual Studio build uses a different default build location of modules
compared to previous version. The tests config file includes a relative
path to libtest used for by all pinvoke tests. Since the module is now
placed at a different location the load failed during the test.
naming for intermediate pdb files for exe/dll's.
All artifacts will now have the same name of their final pdb file as
the library/dll/exe target file name.
The problem with the naming turns out to be a problem in Visual Studio
when a exe/dll use the same name of its intermediate pdb files and final
pdb file. In that case the final pdb file produced by the linker will be
incorrect and not usable by the debugger. This was the reason for the initial
change, giving intermediate pdb files a different name. Moving back to the
default naming for intermediate pdb files for exe/dll's will give the same
effect (a different name) and since libraries are not affected by this
problem, libraries can use the target name for their pdb files as well.
Also added support to not always replace config.h/version.h since that triggers
unnecessary rebuilds during development. New script will check if there is
an actual difference in the config.h/version.h file before doing the copy.
* Added regression test running mono/mini --regression [regression_tests]
* Added regression test running mono/mini --aot [regression_tests]
* Added default debugging options to run regression tests from within visual studio debugger.
* On new build system, setting up include folder if not availalble.
All changes applies to debug/release 32/64 bit.
* Added support for user macros in order to set build/install/sgen.
* Added support for more arguments to bat files + support for quoted paths.
* Added support for verbose/quite output in bat files.
* Moved libmono xcopy post build step into separate bat file.
* Cleaned some sgen configs still left in vcxprojs.
* Enabled build target suffix in order to lable a build (primarly for sgen).
* Enabled separate build tree.
* Dropped all sgen configurations, replaced by properties and target suffix.
* Fixed pdb naming of intermediate pdb files.
* Upgraded all projects to new vs2015 tool chain.
* Used mono.props through property manager instead of import.
* Fixed dependencies to be more explicitly in additional dependecies.
* Fixed error in mono.props, switched $ to % when referencing internal variables.
* Added property variables for includes and libraries, defined in mono.props.
The reason why mkbundle is not working is that for some reason
this constant expression is producing different code than the
-24 that it is supposed to.
What is odd is that the return value from the lseek system
call is the same, but yet, read fails with an end-of-file with
this value, and otherwise it passes.
With this call:
lseek (fd, -(16+sizeof(uint64_t)), SEEK_END)
strace displays:
lseek(0x3, 0xFFFFFFE8, 0x2) = 19375512 0
With this call:
lseek (fd, -24, SEEK_END)
strace displays:
lseek(0x3, 0xFFFFFFFFFFFFFFE8, 0x2) = 19375512 0
Now, while both results from lseek are the same, a subsequent read
with 24 bytes to read returns zero on the former case, and 24 on
the latter case.
When building a mobile_static build, we need to pass a lot of flags
which disable features. These make the net_4_x profile not build
correctly when using the official jenkins script. This helps.
on Windows. Updates the tests-config.in file with Windows specific <dllmap>
elements that map 'libtest' to the MSVC built libtest.dll for the current
architecture. test-runner.cs has been modified to accept a --config option to
be able to use a specific config file. Makefile.am now configures test-
runner.exe to use the tests-config config when running the tests.
Parsing ```default-allowance-ratio``` in ```MONO_GC_PARAMS``` is completely broken and accepts only one value - 1.0, while it should accept range of 1.0-10.0. We can clearly notice that when setting ```default-allowance-ratio``` to e.g. ```2.0``` and trying to run any program through Mono runtime:
```
Warning: In environment variable `MONO_GC_PARAMS': `default-allowance-ratio` must be between 1.00 - 1.00. - Using default value.
```
Therefore, correct this for using proper 1.0-10.0 range.
The do_try_exec_main function is always called in contexts where any
exception from doing the invoke is going to be stored in the exc outarg.
That means any MonoError will be handled locally by do_try_exec_main.
So we don't need a MonoError argument, we just need an inner_error.
We're seeing it randomly fail on Wrench with durations of 1550 - 1800ms.
Talked to Zoltan, this might be normal so bumping the timeout to reduce the failure rate.
The llvmonly backend will use `mono_aot_init_gshared_method_this`
and other initializers to initialize methods. These will transitively call
`mono_icall_get_wrapper_full` when icall wrappers are needed.
The init functions will set the
initialized flag after initialization. They set it in a non-atomic manner. This means that
in a multithreaded environment, it's possible to call the functions
twice.
When `mono_icall_get_wrapper_full` is called without this patch, the
pointers to wrappers it gets from `mono_marshal_get_icall_wrapper`
will not have the same addresses. They will hash to the same value
though.
This will lead to them not being found or inserted by
`find_aot_method_in_amodule`.
The fact that we already have a value with this hash means that
the new wrapper doesn't get inserted. The fact that the pointers differ
mean that we don't say that we've found our desired wrapper.
This leads to an assertion in `mono_icall_get_wrapper_full`
crashing the runtime.
Since we intend to have the semantics of a wrapper for each icall,
this change uses a cache to ensure uniqueness. The observed crashes
do not happen when this change is applied.
MONO_EXECUTABLE env variable to test-runner.exe using the --runtime command
line parameter. Defaults to 'mono' which is also the defult in test- runner.cs
if no --runtime argument is given.
* [runtime] Move 'pedump' to tools/pedump, build tools by default.
* Add missing file.
* [runtime] Mark mono_opcodes[] with MONO_API.
* [runtime] Build monograph when sgen is supported instead of boehm.
* [runtime] Link monograph against the static runtime libraries.
* [runtime] Use the same SUBDIRS/DIST_SUBDIRS for the CROSS_COMPILING case.
We're seeing the following exception on devices in CI:
```
[FAIL] HttpListenerRequestTest.HttpRequestIsLocal : System.Net.Sockets.SocketException : Could not resolve host 'XQA-AppleTV-1'
at System.Net.Dns.Error_11001 (System.String hostName) [0x00000] in /Users/builder/data/lanes/1381/d8287824/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net/Dns.cs:292
at System.Net.Dns.GetHostByName (System.String hostName) [0x00024] in /Users/builder/data/lanes/1381/d8287824/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net/Dns.cs:419
at System.Net.Dns.GetHostEntry (System.String hostNameOrAddress) [0x00061] in /Users/builder/data/lanes/1381/d8287824/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net/Dns.cs:380
at System.Net.Dns.GetHostAddresses (System.String hostNameOrAddress) [0x00065] in /Users/builder/data/lanes/1381/d8287824/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net/Dns.cs:406
at MonoTests.System.Net.HttpListenerRequestTest.HttpRequestIsLocal () [0x00006] in /Users/xamarinqa/QABot/data/lanes/2441/d8287824/source/xamarin-macios/external/mono/mcs/class/System/Test/System.Net/HttpListenerRequestTest.cs:197
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/data/lanes/1381/d8287824/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/corlib/System.Reflection/MonoMethod.cs:309
```
From looking at the code, resolving the hostname to IP addresses via DNS fails so the test doesn't even run.
We don't care where the local IPs come from so we can use an alternative way of getting them - via querying the network interfaces.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=41671
PS: fixed a hardcoded port while I was touching the code
[mini] lower typechecks later
lower typecheck instructions later in the compilation, so we've a chance to do high-level optimization after `method-to-ir`
Rewriite `MonoMethodMessage::InitMessage` in managed code
Change icall
`System.Runtime.Remoting.Messaging/MonoMethodMessage::InitMessage` to
managed code.
Also update `mono_message_init()` to invoke `InitMessage`.
Using mono_marshal_free n mono_string_builder_to_utf8
`mono_string_builder_to_utf16` is allocating using `mono_marshal_alloc` which, on windows, allocates memory using `CoTaskMemAlloc`. This buffer was then released in `mono_string_builder_to_utf8` using `g_free` which resulted in a crash since that's a different memory on windows. The fix in this PR makes sure `mono_marshal_free` is called instead which has the correct behaviour and results in a `CoTaskMemFree` on windows.
[System] Make ServicePointTest.DnsRefreshTimeout test more reliable
We are still seeing quite rare failures of this test:
```
HostEntry should result in the same IPHostEntry object.
Expected: same as <System.Net.IPHostEntry>
But was: <System.Net.IPHostEntry>
+++++++++++++++++++
STACK TRACE:
at MonoTests.System.Net.ServicePointTest.DnsRefreshTimeout () [0x00045] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System/Test/System.Net/ServicePointTest.cs:228
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:309
```
Since the test is only checking if expiring of the cached DNS lookup works, we don't really need to use an external domain name for this as "localhost" works just fine as well and typically doesn't trigger a DNS query which makes the test more reliable against network issues.
@monojenkins merge
We are still seeing quite rare failures of this test:
```
HostEntry should result in the same IPHostEntry object.
Expected: same as <System.Net.IPHostEntry>
But was: <System.Net.IPHostEntry>
+++++++++++++++++++
STACK TRACE:
at MonoTests.System.Net.ServicePointTest.DnsRefreshTimeout () [0x00045] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System/Test/System.Net/ServicePointTest.cs:228
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:309
```
Since the test is only checking if expiring of the cached DNS lookup works, we don't really need to use an external domain name for this
as "localhost" works just fine as well and typically doesn't trigger a DNS query which makes the test more reliable against network issues.
We were not exiting the critical section on this code path. This was leading to hangs at stw since the native code would take locks when compiling the code.
We were seeing intermittent failures on devices like:
```
[FAIL] TimerTest.EnabledInElapsed : #1 loss of events
Expected: True
But was: False
at MonoTests.System.Timers.TimerTest.EnabledInElapsed () <0x33b168 + 0x0014f> in <filename unknown>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object , BindingFlags , System.Reflection.Binder , System.Object[] , System.Globalization.CultureInfo ) <0x66d400 + 0x000b7> in <filename unknown>:0
```
Since OS scheduling and GCs can influence the exact timing of the timer intervals sleeping for 200ms is likely just too short.
Instead of bumping the timeout, I decided to rewrite the test to not rely on Thread.Sleep ().
I also added another test which verifies setting AutoResetEvent=false stops the timer event from firing after the first one,
which was implicitly tested before by the other test.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=41530
Without doing this, the check would consider a valid option name like
IpTimeToLive to be the same as ReuseAddress, as they share the same enum
value.
Fixes SocketTest.TtlChange and SocketTest.UdpMulticasTimeToLive on Android and
other platforms that don't support ReuseAddress.
* Move the versions.mk infrastructure to scripts/submodules so it can be used elsewhere too.
* Add a conditional submodule for llvm.
* Fix commit messages emitted by the bump- targets.
* Use a anonymous git url for llvm.
Before closing a socket we try to interrupt and wait for all socket
pending system calls to return. For that we set the socket as non
blocking which makes the system calls to return with EAGAIN\EWOULDBLOCK.
This issue was caught by System.Net.Sockets.SocketTest.SendAsyncFile
that was intermittently failing, we were only able to reproduce it while
running multiple test suites in parallel. We now can say that it only failed when
the the socket was closed and the sendfile was waiting for the OS to
allocate enough resources to send the file.
This was fixed by changing wapi_sendfile to return on EAGAIN.
Stack traces of blocked threads are now displayed on the exception.
The threads stacktraces are only stored when MONO_TESTS_IN_PROGRESS env
var is set to yes.
When we allocate the complex descriptor for a new type we might have the concurrent marker scan objects. While the concurrent marker scans the bitmap for a complex object, which resides in the complex_descriptor array, another thread might allocate a new complex descriptor, thus reallocate the complex_descriptor array, which would lead to corruption of the bitmap used by the concurrent marker.
We reuse the SgenArrayList which never reallocates, using instead a list of exponentially increasing arrays.
This is the initial code drop of a new handles implementation.
It differs from the previous one is a few ways:
- It keeps a separate stack of handles which is shared by all "arenas".
This eliminates malloc in the majority of the cases without the need for any tuning
- Custom gc scanning code for the handles stack.
The original implementation registered the chunks as a GC roots. Which is a terribly
slow operation as it takes the GC lock.
Instead of allocating a new handle arena for each icall, we compute a stack mark,
which is the position in the handles stack at icall entry. On exit, we simply restore that
position, which is just a couple of stores.
Since the stack is implemented using arraylets, addressed are stable in the face of
expansion.
Only the typed handles function/macros are in as having both typed and untyped handles
adds extra complexity for no clear value.
The macros only handle what was needed to port a sizeable chunk of locales.c.
This commit just sets the stage for the feature, there's quite some work left
before we can start using it, as it can be seen in the big TODO in handle.c
We were seeing an issue where attempts to create the MonoClass for an
exception in a no-exec context was causing failure. No-exec will forbid
the constructor call. Our solution was to realize that there was no
need to allocate an exception, as MonoError could already encode the
errors.
Fixes a bug in exiting modal runloop that caused not really exiting it
(not calling XplatUI.SetModal(..., false).
It caused issues with menu, for example, if you first ran a modal
dialog (a wizard), and then you started the application.
This would lead to a crash when unregistering the thread from the currnt MonoThreadInfo* destructor, as `mono_thread_info_current_unchecked` would return NULL. This would trigger an infinite loop in `mono_threads_enter_gc_safe_region_unbalanced_with_info`.
unregister_thread is the destructor for the current MonoThreadInfo* TLS key, meaning that when in this function, calling mono_thread_info_current_unchecked will always return NULL.
On Unix, errno value might change between the native call and the call to mono_marshal_set_last_error, if we exit the gc safe region. We need to ensure that it's done right after the native call.
[gc] Register a critical region when executing managed allocators.
Previously, if we stopped a thread and it had a stack looking like this:
...
profiler_signal_handler
<signal handler called>
managed_allocator
...
We would fail to identify the fact that we've just interrupted a thread that's
in a managed allocator (uninterruptible code) because we only look at the very
latest instruction pointer of the thread, which will be pointing into
profiler_signal_handler or some other function called by it. So we would
happily continue along with the STW process and proceed to doing the actual GC,
where we would see a broken heap.
We could solve this by unwinding the stack and checking all frames, but that's
complicated and error-prone. Instead, register a critical region while the
managed allocator runs. This way, if we don't identify it by instruction pointer, we
will identify it by the fact that the thread is in a critical region.
This fixes a regression introduced in 123aaa5db0.
We would sometimes see an ObjectDisposedException when calling AcceptAsync ()
because that patch removed the try-catch around it.
Xamarin.Android internal test suite takes advantage of the BCL tests
in Mono, so far their inclusion was a manual process and this commit
fixes it. A set of source inclusion and exclusion lists is added
and a handful of code fixes to make them compile on Xamarin.Android
[System] Fixes socket async exception.
SendAsyncFile uses StartSocketServer to start a server and at the same
time begins to accept a client socket. The issue is that when
the accept callback is called SendAsyncFile already closed the socket.
Fixed race condition by moving StartSocketServer and ReceiveCallback
logic into SendAsyncFile, and by using a new MRE to ensure that the
client receives all the data before the server is closed.
Fixes [#41264](https://bugzilla.xamarin.com/show_bug.cgi?id=41264)
SendAsyncFile uses StartSocketServer to start a server and at the same
time begins to accept a client socket. The issue is that when
the accept callback is called SendAsyncFile already closed the socket.
Fixed race condition by moving StartSocketServer and ReceiveCallback
logic into SendAsyncFile, and by using a new MRE to ensure that the
client receives all the data before the server is closed.
Fixes#41264
Standard C and C++ doesn't allow empty structs, empty structs will always have a
size of 1 byte. GCC have an extension to allow empty structs,
https://gcc.gnu.org/onlinedocs/gcc/Empty-Structures.html. This cause a
little dilemma since runtime build using none GCC compiler will not be
compatible with GCC build C libraries and the other way around. On platforms
where empty structs has size of 1 byte it must be represented in call and cannot
be dropped.
Libraries build using for example MSVC failed pinvoke call when empty
structs where used in pinvoke2.cs tests. The problem was that current
implementation dropped value types of size 0, eliminating them from the
call. Since MSVC implements C/C++ standard regarding empty structs (no extension used),
the function expected the parameter to be present, even if was not used.
This became a problem if the empty struct ended up in one of the 4 first
parameters (backed by registers) since if excluded, wrong parameter gets
into wrong registers (the same will apply to stack slots as well). For example,
a C signature like this, (int a, EmptyStruct es, int b) will get lowered to the
following on MSVC win x64, a -> rcx, es -> rdx and b -> r8, but since caller
eliminated es from the call, the register assignment from the caller was
a -> rcx and b -> rdx causing a violation of the calling convention and
error in callee.
The fix is to make sure that empty structs are still represented on none GCC
compilers on windows x64 (MSVC), since they must be represented in the call
to pinvoked methods. Fix will also use a const value of 0 to represent the value
passed as the struct instead of loading a value from stack into register/stack slot,
this is inline with the value the .NET runtime passes to pinvoked method. The reason
why we use a 0 instead of loading from stack is because a struct of size 0 is not explicitly
represented on stack and will have garbage data if loaded. Since it’s an empty struct value
type there is no need to load anything, just represent it as 0 in the register/stack slot passed
to caller.
There is also a case when returning an empty struct from a pinvoked method. Result will be returned
as 0 in RAX by MSVC compiled libraries (or whatever value stored in the 1 byte representation).
Since we have similar issues as above with empty structs of size 0 not explicitly represented on the
stack, the store will be eliminated and not written back to stack by caller since an empty struct doesn’t
represent any unique data.
Calling convention for managed->managed code is kept intact since it is working as expected
handling the case with empty structs as input and return values.
With fix all pinvoke2.cs tests now pass running on Windows x64 JIT.
This appears to serve no purpose as tlab_next is a thread-local variable. It's
only ever accessed by other threads in order to write to it, not read it. We
also don't do this memory barrier in the native allocation functions.
Previously, if we stopped a thread and it had a stack looking like this:
...
profiler_signal_handler
<signal handler called>
managed_allocator
...
We would fail to identify the fact that we've just interrupted a thread that's
in a managed allocator (uninterruptible code) because we only look at the very
latest instruction pointer of the thread, which will be pointing into
profiler_signal_handler or some other function called by it. So we would
happily continue along with the STW process and proceed to doing the actual GC,
where we would see a broken heap.
We could solve this by unwinding the stack and checking all frames, but that's
complicated and error-prone. Instead, register a critical region while the
managed allocator. This way, if we don't identify it by instruction pointer, we
will identify it by the fact that the thread is in a critical region.
Made breakpoint uniqueness test switch to a hash after 8th item added,
consolidated the code that handles the uniqueness test and makes the
breakpoint in one function, fixed style issues.
* Add TODO list
* Use -useSourcePath when building resources, so we do not need to add the explicit extra path
* More updates, now battling resource generation across the board
* Use newlines in the Unix commands to avoid generating things like cs-parser.cs^M
The tests added during the initial development of the -multiassembly argument failed to test the presence of extension methods;
the result was a crash that occurred when there were extension methods present in multiple assemblies.
This patch addresses the issue by putting a guard clause around the code that adds the extension method to the index.
Tests have been modified accordingly to verify this bug and include extension methods
Additionally, the -multiassembly parameter was removed entirely. This behavior is now triggered by the presence of -apistyle,
which is where support for "multiple assemblies" was meant to work anyways. To review, this is to support cross-platform
development, where you have multiple -- *different* -- assemblies for each platform. These assemblies will share a certain
cross-section of APIs that will be *exactly* the same in each assembly (as opposed to multiple versions of literally the same assembly).
As a result, every member will now contain an individual `AssemblyInfo` tag that also contains the name of the assembly.
The date/time struct in zip.h defines the month value as
uInt tm_mon; /* months since January - [0,11] */
.NET's DateTime object in turn stores months from 1-12. We didn't
take this into account when passing the values from managed->native
so the month ended up being off by one.
We didn't use this codepath before https://github.com/mono/mono/pull/3036
that's why it didn't show up until now.
The test assembly used *.dll.excludes before while the library uses *.dll.exclude.sources.
Make this consistent to reduce confusion.
Also remove corlib_cmp.dll.excludes which is from a profile that no longer exists.
We were seeing timeouts in the runtime step if two builds happen to be scheduled
at the exact same time on the ARM workers on Jenkins, causing increased load and
longer test times. They aborted the runtime step right before it'd finish.
Bump the timeout a bit to avoid this.
If we have binary protocol enabled and the debugged executable will spawn a new mono instance, both of the processes will write to the same debug file, because they share the environment variable, producing invalid results. Add a pid sufix if output file exists in order to separately obtain binary protocol outputs from all mono instances.
When we scan a block we need to have it checked for sweeping. Instead of waiting for all the blocks to be checked for sweeping, we traverse all the blocks (even though they might not have live objects), and do the sweeping check only for blocks that have dirty cards associated with them, without waiting for sweeping to end.
Allocating a major block needs addition to the allocated_blocks array, operation that needs sweep to be finished because sweep moves elements and changes the size when compacting the array. Do this operation in the sweep start instead, when we are not racing.
We had the problem that we were logging at sweep finish which can happen anytime. If we happen to suspend a thread, while doing a GC, in the middle of logging we might deadlock if we wait for sweep to finish (which can be blocked waiting for logging).
When logging multiple collections in the same stw pause, we would not log the real results for both collections since we would compute the major sizes at the very end.
We logged collections in sgen_perform_collection, outside the function calls for specific collections, which makes it cumbersome to add additional log entries.
This refactoring solves these issues by having GC code log info at any time. This log entry is enqueued in a list of pending entries. We flush the list of logs, as before, at the end of the stw, when the world is restarted.
A while ago, we switched our PR builders from using the Github-provided PR branch head (i.e. `pull/merge/XXXX`) to the exact sha1 of the latest commit on the PR (i.e. `abcdefabc`). This change was needed for the performance team, otherwise the changes they wanted to test would be rebased against a new master several times per day, breaking their ability to meaningfully track the changes in a PR.
As a side effect of this, `version.h` went from setting the version from `XXXX/abcdefabc` to `explicit/abcdefabc`, and the Jenkins PR packages use the branch name `explicit` instead of the actual PR number `XXXX`, which is pretty bad from a UX perspective.
This change checks for the `ghprbPullId` environment variable and uses it instead of the actual branch name or `explicit` - it is always set on PR builds.
Support for GSHAREDVT added for Windows x64. Changes needed
(ones gsharedvt concept was understood, crashes debugged and adjustments located)
where quite small. Primarily around shadow stack used by windows + incorrect handling
of none volatile registers on windows when going over the gsharedvt trampolines.
All GSHAREDVT tests, gshared.exe and aot.tests.exe pass in full AOT builds with above adjustments.
Support for DYNCALL added/enabled for Windows x64. Current AMD64 support was working
for basic scenarios but not enabled on Windows. Identified a problem on AMD64
when structs are passed as input parameters or returned by methods for DYNCALL scenarios
in full AOT:ed builds. This scenario doesn't seem to work for any AMD64 platform for
full AOT and the tests in aot-test.exe handling these scenarios are not run on AMD64.
Adjusted categories on test in aot-tests.exe to reflect what’s really needed by the tests
to run correctly. Also added a new category to some DYNCALL tests that uses struct support
currently not implemented on AMD64, !FULLAOT-AMD64, so at least some of the DYNCALL tests
can be run on AMD64 platforms in full AOT:ed builds.
[mini] small improvements around `MONO_IS_CALL` macro and `local-prop`
showed up in vtunes when running
```
./mono/mini/mono-sgen --stats -O=all --aot=threads=1,asmonly ~/work/benchmarker/tests/fsharp/lib/bootstrap/4.0/FSharp.Compiler.dll
```
also, I think when `OP_DYN_CALL` was introduced, it wasn't added to the `MONO_IS_CALL` macro. @vargaz, what do you think?
* Merge System/MonoType.cs into ReferenceSources/RuntimeType.cs, use the latter in the runtime code.
* [sdb] Fix a test which referenced System.MonoType.
MonoError must be explicitly reinitialized if it is used again after
mono_error_cleanup. But it is better to introduce an explicit inner
MonoError when possible.
I had a look in vtunes for
```
$ mono/mini/mono-sgen --stats -O=all --aot=threads=1,asmonly ~/work/benchmarker/tests/fsharp/lib/bootstrap/4.0/FSharp.Compiler.dll
```
and that method showed up as hot.
With this change I get an improvement from ~27.4s down to ~24.3s on my
machine.
Testing with the debugger showed multiple failures around recursive
functions:
1. Step-out breakpoints can trigger on inner frames.
Solution: Extend existing inner-frame test from step-over to step out
also.
2. "Same source line, continuing" test triggers even on wrong frame
(affects both step over and step out)
Solution: Check frame as part of this test
3. When step breakpoints pick next sequence points, they can register
multiple breakpoints at the same place if a recursive call happens to
exit there. (Possibly harmless but may have resulted in duplicate
debugger events being generated in one test.)
Solution: Check for duplicates in step-request breakpoint list before
adding new breakpoints
4. Mono.Debugging.Soft in the debugger-libs repo sometimes resumes
after breakpoints if it decides Mono.Debugger.Soft has not stepped
far enough, but the check to trigger this was not recursion-aware.
Solution: This one is not addressed in this commit, but a fix was
checked in to debugger-libs. Any code using this commit therefore
needs to be run in conjunction with debugger-libs dfbf6d8b13.
Tests were also added for issues 1-3 above.
This change makes it not need to throw exceptions.
In the runtime, use mono_thread_internal_stop instead of mono_thread_stop.
mono_thread_internal_stop only stops other threads, which is what runtime usage calls for.
mono_thread_stop will stop the current thread by raising and exception.
The first argument is whether it's a single or multiple wait. We can infer that from the number of handles.
The second argument is alertable. All waits are alertable, so I baked the constant directly.
They can only ever fail if allocation fails, which we can't recover
from, so let's just assert that we succeed.
This in turn simplifies the error handling in a lot of other thread routines.
Since referencesource is no longer a submodule under external/ after https://github.com/mono/mono/pull/2962
it didn't get included in the tarball anymore.
We need to add it again to fix the tarball build.
In https://github.com/mono/mono/commit/4164d9fbe19a1e6acb3a5ea122891020ff2868d7 the 14.0 target was updated to include
the DependsOnSystemRuntime property. However, it wasn't actually implemented in the code behind. This resulted in the
property not being set/true even if we depend on assemblies that reference System.Runtime, which in turn leads us to not
adding the Facades in ImplicitlyExpandDesignTimeFacades. This then breaks compilation because the System.Runtime lookup fails.
The fix is to actually implement the property.
This change makes it not need to throw exceptions.
In the runtime, use mono_thread_internal_stop instead of mono_thread_stop.
mono_thread_internal_stop only stops other threads, which is what runtime usage calls for.
mono_thread_stop will stop the current thread by raising and exception.
The first argument is whether it's a single or multiple wait. We can infer that from the number of handles.
The second argument is alertable. All waits are alertable, so I baked the constant directly.
When running mono/mini aot-tests.exe on full AOT:ed Windows build the following
warning message is incorrectly written to the console several times:
g_warning ("failed to get 100ns ticks");
This happens since the check triggering this warning is not taking into account
that startup time between first call to mono_ns100_ticks storing the start time
and the launch of the thread pool requesting 100ns ticks and convert it to
elapsed seconds can occur during the first second of the process lifetime.
This is more likely to happen in a full AOT:ed image since nothing needs to
be JIT:ed at startup, making it more likely to exploit the problem reaching this
code within the first second of process lifetime.
The purpose of the check is to validate that mono_ns100_ticks is returning “valid”
data, but currently it checks it after converting it to seconds, meaning that the result
of the division gets truncated to 0 during the first second. The only invalid, deterministic error
from mono_ns100_ticks that could be used to trigger this warning is either 0 or a negative time value,
but currently it just checks for 0 as an error. The fix is to make the check on the return from
mono_ns100_ticks instead of on the result of the division by 10 000 000 as it is today.
On the new Jenkins infrastructure with the new ARM64 bots we're hitting the 2h runtime test
limit right before they'd be finished. Bump the timeout a bit to fix this.
They fail on the new Jenkins ARM64 bots, disable them until we figure out the cause.
It looks like a libgdiplus issue to me which makes deploying a fixed version a bit more difficult.
Filed https://bugzilla.xamarin.com/show_bug.cgi?id=41171 to track.
Due to performance concerns, we don't account for the space wasted in blocks, since it would require dereferencing all vtables. We also don't account for allocation that might happen during sweep.
As with mono_runtime_invoke, we split up into three functions:
1. mono_runtime_invoke_array is external only. Runtime should use
one of the other two.
2. mono_runtime_invoke_array_checked threads exceptions and failures through
MonoError in coop mode. In non-coop managed exceptions unwind the
stack as before.
3. mono_runtime_try_invoke_array sends exceptions from method invocation
through a MonoException* outarg, and other errors through MonoError.
When one of the tests doesn't fail with the expected exception, we'd get just a useless message
like "TtlChangeoverflow #3" instead of the much more useful exception info + stacktrace that
NUnit catches when an unhandled exception is thrown.
During mono shutdown on windows there is, on rare occasions, an exception
in CloseHandle indicating the closure of an invalid handle. This exception
is normally a strong indication of a much more severe underlying problem,
a doubled close on a handle, that could lead to random data corruptions
and very hard to track bugs in random areas of the code. If the system reused
the handle for a different object that silently gets incorrectly closed by above
call, the above bug could manifest itself into some completely different issues,
so getting a consistent repro with the above exception with CloseHandle was a good
point to start tracking down the bug.
The exception throwed in CloseHandle boils down to the fact that threads
in thread pool's that have their finalizer run on shutdown can end up in
a scenario where the thread handle has already been closed elsewhere.
When the threads are finalized in mono_gc_finialize_threadpool_threads,
the handle on the internal thread object, stored in MonoInternalThread,
will be closed by the thread objects finializer in ves_icall_System_Threading_InternalThread_Thread_free_internal.
Under some rare occasions, one of the handles are not valid and will cause
the exception in CloseHandle or even worse, pointing to a complete different
kind of object, potentially closed by the threads finalizer, causing an instable
process with random behavior related to WIN32 objects (files, mutexes, threads, events etc).
The reason why it rarely manifests itself is due to a race between the finializer thread and the
main thread during shutdown. Main thread gives finalizer thread 2000 ms to complete its work and
if it’s time outs, the main thread won’t call the code that will run the finalizers,
mono_gc_finalize_threadpool_threads for the thread objects added to an internal list
by the finalizer thread, so if the finalizer takes more than 2000 ms to complete during shutdown
the bug won’t manifest itself. By increasing the wait time in mono_domain_finalize to for example INFINITE,
the repro of this bug will be 100% and not timing dependent anymore.
Turns out that the offending CloseHandle on the thread object happens at an
earlier point in time during shutdown, when mono_thread_manage tries to wait for
threads to complete manage thread it will do remove_and_abort_threads for background
threads still around. The way this works is that it will setup a list of thread handles
that it will wait for using WaitForMultipleObjects and then close the handles.
It turns out that the implementation of remove_and_abort_threads adds the wrong
handle to the list, not the duplicated handle but the original thread handle,
and that will trigger this bug, since the original thread handle will be closed
(and the duplicated handle will leak). The fix is to use the duplicated thread
handle instead of the original, similar to how it was already done in other similar methods, like build_wait_tids.
Sinces this bug seems to been around for quite some time and is a race related to managed threads and mono
shutdown sequence it could give other weird side effects on windows during shutdown
(since it could have closed a random WIN32 object instead of the intended thread).
Looking at Bugzilla, there is one bug that very well could be related to this specific problem, 10108:
https://bugzilla.xamarin.com/show_bug.cgi?id=10108
This bug should most likely be retested with this fix to see if it still reproducible once merged into master.
* Adds support for a handful of new command line arguments that we now use in the build.
* Always attempts to match to a project name, to get the proper project dependency.
* Update to support .exe and .dll in the generation.
* Remove warnings and some dead code
* Update the resulting csproj files based on running:
make update-csproj
make package-inputs
mono genproj.exe
bridge_processing_in_progress is set by sgen-tarjan-bridge.c at the
start and end of each bridge, although sgen-bridge sets this flag at
the same time.
There is some code that reports the length of a gc bridge operation in
sgen-stw. This is dead code dating from a period when gc bridges
occurred synchronously as part of a stop the world operation. The new
code globally tracks the end of the last stw and reports time since
then when the bridge thread completes.
This allow the removal of MONO_(PREPARE|FINISH)BLOCKING and MONO(PREPARE|FINISH)_RESET_BLOCKING, which were not explicit in their intent, leading to confusion. This also harmonize how the product and the runtime transition between thread states.
`Assertion.Fail` isn't supported by NUnitLite in monodroid, use `Assert` instead to fix the build:
```
[12:29:46] /Users/builder/data/lanes/1845/88dbce4e/source/monodroid/tests/bcl-test/BclTests.csproj (default targets) ->
[12:29:46] /Library/Frameworks/Mono.framework/Versions/4.2.1/lib/mono/4.5/Microsoft.CSharp.Targets (CoreCompile target) ->
[12:29:46]
[12:29:46] src/System/Test/System.Net/IPAddressTest.cs(282,4): error CS0103: The name `Assertion' does not exist in the current context
[12:29:46]
[12:29:46] 14 Warning(s)
[12:29:46] 1 Error(s)
```
This will be used by managed-to-native and native-to-managed wrappers, as well as trampolines, to switch between states. This is necessary as their calls might be unbalanced (a prepare blocking, without the corresponding finish blocking).
If we do not throw it from the finally clause, then in case of a Thread.Abort, we will not rethrow the catched exception, but the ThreadAbortException, and this without using the callback, leading to native stack unwinding.
The support will be broken when we will introduce MonoHandle, as we can't pass the exception naked object ref around, but need to wrap it in a MonoHandle, while still in RUNNING mode.
Fixes needed in order to get full AOT working for mono/mini regression tests on windows x64 builds (currently disabling the gsharedvt and dyncall tests).
Fixes are done in two main areas, exception and thread local storage support.
SEH exception handling enabled in order to catch CPU generated exceptions. All exception related regression tests pass with fix and runtime won't attempt to JIT any code since it is prevented when running with --full-aot startup argument.
Thread local storage (TLS) support was not complete on windows x64. When running full AOT the sgen allocator uses TLS for its allocator. AOT compiler will issues OP_TLS_GET_REG operations when compiling the “Alloc” methods. On win x64 this was not implemented, just the default OP_TLS_GET, meaning that methods compiled using OP_TLS_GET_REG was "silently" dropped when full AOT:ed compiled. At runtime, the sgen allocator would tried to locate the “Alloc” method but fails to find it and since this will cause an exception (that also needs the allocator to be used) and the program entered a loop that will end in a stack overflow crashing the program.
The fix implements OP_TLS_GET_REG and other needed support for win x64. It also fixes the offset calculation needed by full AOT builds (translating to correct runtime TLS key) for keys used by mono/mini regression tests. Since windows TLS support is a little more complex (two different conditions depending on where the TLS key gets allocated) the max size of tls_get_reg for amd64 has been increased from 32 to 64 byte as well.
OpenSSL and the MS RSA CSP (Win7 x64) generate RSA keys with a public exponent of 65537. Smaller public exponents are not allowed by some specifications and may be rejected by other software.
* Start of packager for mkbundle
* [mkbundle] Add fingerprint and directory at end of package
* [mkbundle] Update embedding file format, add support for --options, support for --runtime, chmod +x
* [driver] Add a way to parse arbitrary command line options
* [mkbundle] Align the assemblies on a page boundary, do not release directory buffer
* Work-around odd compiler error by using memcpy
* Update makefile
* Fix the windows build
* [Windows] Export the new mono_parse_options_from API
* Implement the various cross-compiler target fetching commands
* Clean
Marked it external only. Runtime should use mono_string_from_utf16_checked
Also mark mono_string_from_utf32 as external only.
(It's currently unused in the runtime, but in the future
mono_string_from_utf32_checked is available)
The actual meaning is about whether or not the name is allowed to change after
having been set by the mono_thread_set_name_internal () call; it has nothing to
do with whether the call came from managed per se.
If we did consider this, an internal runtime thread that had its name set by a
mono_thread_set_name_internal (..., FALSE /*managed*/, ...) call prior to
starting would not have its native name set nor its name reported to the
profiler.
Process.Exited event can get invoked before the process has really
exited. So, accessing process.ExitCode before the real exit can throw:
Error executing task Exec: System.InvalidOperationException: Process must exit before requested information can be determined.
at System.Diagnostics.Process.EnsureState (System.Diagnostics.Process+State state) [0x000b9] in /Users/ankit/dev/mono/mcs/class/referencesource/System/services/monitoring/system/diagnosticts/Process.cs:1439
at System.Diagnostics.Process.get_ExitCode () [0x00000] in /Users/ankit/dev/mono/mcs/class/referencesource/System/services/monitoring/system/diagnosticts/Process.cs:219
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:get_ExitCode ()
at Microsoft.Build.Utilities.ToolTask.ExecuteTool (System.String pathToTool, System.String responseFileCommands, System.String commandLineCommands) [0x00101] in /Users/ankit/dev/mono/mcs/class/Microsoft.Build.Utilities/Microsoft.Build.Utilities/ToolTask.cs:185
at Microsoft.Build.Tasks.Exec.ExecuteTool (System.String pathToTool, System.String responseFileCommands, System.String commandLineCommands) [0x00026] in /Users/ankit/dev/mono/mcs/class/Microsoft.Build.Tasks/Microsoft.Build.Tasks/Exec.cs:83
at Microsoft.Build.Utilities.ToolTask.Execute () [0x0001c] in /Users/ankit/dev/mono/mcs/class/Microsoft.Build.Utilities/Microsoft.Build.Utilities/ToolTask.cs:128
at Microsoft.Build.BuildEngine.TaskEngine.Execute () [0x00000] in /Users/ankit/dev/mono/mcs/class/Microsoft.Build.Engine/Microsoft.Build.BuildEngine/TaskEngine.cs:134
at Microsoft.Build.BuildEngine.BuildTask.Execute () [0x0008f] in /Users/ankit/dev/mono/mcs/class/Microsoft.Build.Engine/Microsoft.Build.BuildEngine/BuildTask.cs:101
ProcessWrapper.WaitForOutput depends on the
`endEventExit`(ManualResetEvent) to be set, which is done in the event
handler for Process.Exited . So, effectively, WaitForOutput can return
before the process has really exited and ToolTask ends up throwing an
exception when it accesses the ExitCode.
Fix: Add a WaitForExit in WaitForOutput, to be sure!
Some existing tests only works when IPv4 is available, which was fine as
it match how we historically ran those tests.
Apple has a new requirement [1] to test under an "IPv6 only network" and
the change facilitate this.
[1] https://developer.apple.com/news/?id=05042016a
We were seeing compilation errors like this:
```
System.Security.Cryptography.X509Certificates/X509EnhancedKeyUsageExtension.cs(71,20): error CS0135: `oid' conflicts with a declaration in a child block
System.Security.Cryptography.X509Certificates/X509EnhancedKeyUsageExtension.cs(74,17): (Location of the symbol related to previous error)
Compilation failed: 1 error(s), 38 warnings
make[9]: *** [../../class/lib/basic/secxml/System.dll] Error 1
```
The new foreach variables introduced in 4a6e5d24e1 with the same "oid" name as the global constant apparently confused the older mcs compiler.
Rename them to avoid this.
We were seeing random failures of the test in the iOS simulator and devices.
Looking at the test, it seems to me like the 1000ms timeout is just too short. In a single iteration we're
enqueuing 500 threadpool workitems and effectively give each of them just 2ms to run and signal the CountdownEvent.
Indeed I can easily repro this in the simulator on my box if I lower the timeout to 900ms.
This may be a side effect of the new threadpool which starts threads more conservatively afaik.
Bumping the timeout should fix this.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=39305
https://bugzilla.xamarin.com/show_bug.cgi?id=8554
Added unit tests to cover the Asynchronous API and the Task Based API.
Fixed an issue with the Task Based API that would previously throw a
`NullReferenceException` when started as the `CancellationTokenSource`
was not instantiated.
The `CancellationTokenSource` was also not disposed of after use.
It seems that older iOS6 devices don't allow access to /dev, as a DirectoryNotFoundException is
thrown when accessing /dev/zero in the FileStreamTest.OpenCharDeviceRepeatedly test there.
Ignore this exception and the test in those cases.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=40462
AppleTLS supports a lazily-initialized X509Certificate, but not X509Certificate2 so
we need to fall-back to using Mono.Security.X509 whenever we need an X509Certificate2.
To avoid unnecessary fallbacks, the private Mono.Net.Security APIs use X509Certificate
instead of X509Certificate2.
Since 'ExtraStore' returns X509Certificate2Collection, we need to convert these to
X509Certificate2.
(cherry picked from commit 05e2372339)
The sgen tests should run on Windows x86 too (or at least they will in the near future), so we can get rid of all the special casing.
This also fixes an error in the ARM64 test run on Jenkins because test-sgen wasn't defined there.
There would previously be a race between reset blocking and a STW, that would happen as follow:
- T1 goes into blocking - T1 state is BLOCKING(0)
- T2 suspends T1 - T1 state is BLOCKING(1)
- T1 does a reset blocking -> T1 state is SELF SUSPEND REQUESTED
- T2 calls `mono_thread_info_get_suspend_state` on T1 --- SIGABRT as we cannot get suspend state when state is SELF SUSPEND REQUESTED
- T1 does polling -> T1 state is SELF SUSPENDED
The fix is for reset blocking to work like done blocking: switch to BLOCKING AND SUSPENDED.
We were seeing issues on the new Jenkins Azure boxes which had 7GB of physical memory but only about 1GB free at the time of the test, so it'd fail.
Checking the available memory should fix this.
If there's a pending exception and an AbortRequested, the pending
excpetion should be thrown.
This fixes threadpool-exceptions5.cs in coop mode.
There are two situations where this could happen.
1. Like threadpool-exception5.cs we have a threadpool worker thread
request an abort and then its async callback runs and throws an
exception. In this case apparently the async callback's exception
should be allowed to unwind and hit the unhandled exception handler.
2. A thread catches ThreadAbortException and then posts a pending
exception of some other excpetion from the catch clause. In this case
we unwind that other exception, and when it's caught we rethrow the
abort.
When the system certificate validator is invoked with an empty hostname
(caller does not want us to validate the hostname), then we need to pass
null to SecPolicyCreateSSL() and not the empty string.
(cherry picked from commit f29804d149)
Only attempt the automatic PEM->DER conversion when no password was given
and make this version of X509Helper.Import() also take care of the fallback.
(cherry picked from commit 63a214c3f0)
Previously we had separate codepaths in mono_jit_thread_attach for coop
and non-coop modes.
Now there's a common codepath for attaching the thread, and separate
code paths for performing coop state machine transitions and computing
the return value cookie.
Fixes Mono.Debugger.Soft tests under coop.
By skipping thread which have a NULL domain or lmf, we would also skip threads which are in RUNNING mode, but which are still attaching to the runtime. They could have in practice called `mono_thread_info_attach`, without having executed any managed code or set a domain yet. Because we would simply discard the suspending for these threads, we could potentially not suspend them, meaning they would keep running, even potentially run some managed code, while the GC is still running.
To fix that, we need to slightly modify the expectation of sgen STW when using unified suspend. The non-unified expectation of sgen STW would be that even if a thread has a non-initialized domain or lmf, it should still be suspended, but not scanned. The unified expectation of sgen STW would be that if a thread has a non-initialized domain or lmf, then the thread shouldn't be suspended. That second behaviour works well if we only consider that the domain or lmf could be non-initialized when the thread is detaching, but that consideration is just wrong. The main example is the beginning of `start_wrapper_internal` (in threads.c:637), where we have non-initialized domain and lmf, but where the thread is in RUNNING state, and about to access some managed memory.
On coop, an OS thread can go
STARTING->BLOCKING->RUNNING->BLOCKING->DETACHED.
For ordinary managed threads BLOCKING->DETACHED is still a bug like
before, but hopefully the checked build machinery will help us to catch
those situations.
When checked build is running with GC checking, it uses a non-zero
integer cookie instead of the thread info pointer.
This fixes pinvoke3.cs test_43_thread_attach() in checked coop builds.
This is going to be used by Xamarin.iOS to convert a managed exception to an Obj-C exception. We need this as if we don't do that, the managed exception mechanism will unwind native stack, which can lead to undefined behavior. By converting to an Obj-C exception, we let the native exception mechanism do what it's supposed to do.
Because mono_msec_ticks would previously return the number of milliseconds since boot time, it would overflow after ~49 days. That would mean that anywhere you would use it, you would need to be careful of arithmetic overflow.
By simply returning a gint64 we will not overflow before ~30k years.
Fixes bug https://bugzilla.xamarin.com/show_bug.cgi?id=38666
When FileInfo was imported in https://github.com/mono/mono/commit/bdd0de7ee52d9675dada0d19985b9daf4edab5bc the check in Delete()
got removed that raised an UnauthorizedAccessException when the FileInfo object holds a path to a directory instead of a file.
It happened to pass the MonoTests.System.IO.FileInfoTest.Delete_Directory test on OSX because unlink returns EPERM there
which got turned into the expected UnauthorizedAccessException, but on Linux it returns EISDIR so the test would fail.
Fix is to put the explicit check for directory back.
NativeMethods.GetExitCodeProcess would inherently be broken on Unix platforms for non-child processes as it can't succesfully get the exit code. This would in turn break Process.HasExited, which should not break, as it is possible to check if a non-child process is alive or dead.
It's not easy to trigger but following repro works fairly reliably
using System;
using System.Threading;
class MainClass
{
public static void Main (string [] args)
{
var r = new Random ();
for (int i = 0, total = 10000000; i < total; ++i) {
if (i % 50 == 0)
Console.WriteLine ("{0}/{1}", i, total);
var c1 = new CancellationTokenSource ();
var wh = c1.Token.WaitHandle;
GC.Collect ();
c1.CancelAfter (r.Next(5));
c1.Dispose ();
}
}
}
It crashes like
Unhandled Exception:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Threading.ManualResetEvent'.
at System.Threading.WaitHandle.CheckDisposed () <0x1f95be0 + 0x0005c> in <filename unknown>:0
at System.Threading.EventWaitHandle.Set () <0x1f8d320 + 0x0007d> in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Set ()
at System.Threading.CancellationTokenSource.NotifyCancellation (Boolean throwOnFirstException) <0x1d92c70 + 0x00084> in <filename unknown>:0
at System.Threading.CancellationTokenSource.Cancel (Boolean throwOnFirstException) <0x1d92330 + 0x00030> in <filename unknown>:0
at System.Threading.CancellationTokenSource.Cancel () <0x1d92300 + 0x00025> in <filename unknown>:0
at System.Threading.CancellationTokenSource.TimerCallbackLogic (System.Object obj) <0x1d925c0 + 0x0005d> in <filename unknown>:0
This change is fixing the DateTime parser in cases of date/time strings for cultures which has the date and time separators are same. This happen with Norway, Serbia and Finland cultures
[tfs-changeset: 1513801]
The race condition would manifest in the following code:
```
for (int i = 0, total = 100000; i < total; ++i) {
if (i % 50 == 0)
Console.WriteLine ("{0}/{1}", i, total);
var c1 = new CancellationTokenSource ();
var wh = c1.Token.WaitHandle;
c1.CancelAfter (1);
Thread.Sleep (1);
c1.Dispose ();
}
```
And we would observe the following exception:
```
Unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Threading.ManualResetEvent'.
at System.Threading.WaitHandle.CheckDisposed () [0x00016] in /Users/builder/data/lanes/1196/33e00ac6/source/mono/mcs/class/corlib/System.Threading/WaitHandle.cs:426
at System.Threading.EventWaitHandle.Set () [0x0000c] in /Users/builder/data/lanes/1196/33e00ac6/source/mono/mcs/class/corlib/System.Threading/EventWaitHandle.cs:205
at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Set ()
at System.Threading.CancellationTokenSource.NotifyCancellation (Boolean throwOnFirstException) [0x00051] in /Users/builder/data/lanes/1196/33e00ac6/source/mono/external/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:723
at System.Threading.CancellationTokenSource.Cancel (Boolean throwOnFirstException) [0x00006] in /Users/builder/data/lanes/1196/33e00ac6/source/mono/external/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:409
at System.Threading.CancellationTokenSource.Cancel () <0x7d75ada8 + 0x00017> in <filename unknown>:0
at System.Threading.CancellationTokenSource.TimerCallbackLogic (System.Object obj) [0x00012] in /Users/builder/data/lanes/1196/33e00ac6/source/mono/external/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:538
```
This would be a race condition between the TimerCallbackLogic call to Cancel and the call to Dispose on another thread. You could trigger it more reliably by putting a Thread.Sleep at CancellationTokenSource.cs:599.
Mono now uses ExceptionHandlingClauseOptions from methodbody.cs and ExceptionHandler from methobuilder.cs.
Remaining types from thoses files were ignored with ifdef.
Mono's implementation of RSAOAEPKeyExchangeDeformatter did the throwing
but we're now using MS implementation of it - so it make sense to move
the throw (it's needed) in the utils.cs file
MS implementation is incorrect and the .NET framework never hits this code
path as it's RSA implementation calls CryptoAPI (bypassing those managed
classes).
mscorlib/system/security/cryptography/pkcs1maskgenerationmethod.cs
-> use Mono MGF1 implementation
mscorlib/system/security/cryptography/utils.cs
-> remove duplicated code from Mono's PKCS1
-> fix RsaOaepEncrypt and RsaOaepDecrypt to call Mono code
* `ServerCertValidationCallback': Enable this for Mono.
* `ServicePointManager': Use 'MONO_FEATURE_WEB_STACK' here, we'll
eventually bring this together with the web stack.
We want the helper APIs at the top of the file, but not the actual
'ServicePointManager' class.
This is the same issue as in https://github.com/mono/referencesource/pull/7,
i.e. there's an overflow when casting a too large value from double->long
and the referencesource implementation relies on this being a large negative
value in order for the ArgumentOutOfRangeException to be properly thrown.
As Mono's runtime on ARM in this case returns -1, the exception isn't thrown.
The fix is to introduce an explicit overflow check, catch an eventual
OverflowException and raise the proper exception instead. This can be JIT'd
to better code than checking if the value fits into long every time.
It fixes a couple of DateTime unit tests that currently fail on ARM.
When setting Interval = double.MaxValue like in the TimerTest.Interval_TooHigh_ThrowOnEnabled and
TimerTest.Interval_TooHigh_Enabled_Throw Mono unit tests there was a regression on ARM in that
the ArgumentOutOfRangeException wasn't thrown as expected.
Turns out, on ARM the call to (int)Math.Ceiling(double.MaxValue) as used in UpdateTimer() and Enabled setter
doesn't produce -2147483648 (i.e. Int32.MinValue) like on x86/amd64 but rather 2147483647 (i.e. Int32.MaxValue),
which is perfectly valid as the cast overflows and the result is an undefined value.
This means that the value passed to the underlying System.Threading.Timer is not negative
and no ArgumentOutOfRangeException is thrown there.
The fix is to properly check for interval>Int32.MaxValue like is done already in the constructor.
I pulled that logic out and introduced a separate method CalculateRoundedInterval().
Note: The thrown exception changes from an ArgumentOutOfRangeException inside of System.Threading.Timer
to an ArgumentException inside of CalculateRoundedInterval().
I think this has minimal impact as no-one should rely on this exception anyway and it actually matches
what MSDN (https://msdn.microsoft.com/en-us/library/system.timers.timer.interval(v=vs.110).aspx)
says should happen in this case (even though the MS.NET 4.5 implementation doesn't comply).
Mono's implementation also did the same before: https://github.com/mono/mono/blob/4cb3f77b4bbf703b1cda59db2f5aee206e35d31a/mcs/class/System/System.Timers/Timer.cs#L95-L96
Even after we imported System.Xml referencesource, XmlSerializer is still
based on existing code. And we have some special trick for XElement
serialization support in System.Xml, by allowing access from
System.Xml.Linq.dll to use special type converter attribute.
It does not exist in referencesource and thus we lost serializability on
XElement, so we need to add the missing code to bring it back.
The former condition is to disable <msxsl:script> element.
The latter is to disable XslCompiledTransform and co (though they are
mostly removed at *.dll.sources level).
Somewhere in System.Xml requires NameTable to be canonically reused when
deserializing XmlSchema, so that certain deserialization part can check
only reference equality.
Since we have hybrid System.Xml implementation for XmlSerializer, this
is not completely achieved and with reference comparison it causes some
schema deserialization failure. To avoid this problem, just disable
reference comparison. It should not affect outside XmlSchema serialization.
DISABLE_CAS_USE is required to avoid NotImplementedExceptions in CAS land.
MONO_HYBRID_XML is to preserve Mono's XmlSerializer which also allows
interpreter-based XML serialization (as opposed to run-time code generation
based XML serialization, which is not doable in iOS).
Though some Xml.Serialization classes are useful even with mono
implementation, so we partially import them (namely schema importer).
This will let us have both the Mono and the Microsoft ThreadPool at the same time.
To switch to the Microsoft one, set the MONO_THREADPOOL environment variable to "microsoft". It will default to the Mono one.
All changes are conditional to MONO_FEATURE_NEW_TLS.
In addition to that, all changes to their code are also conditional to 'MONO'; this
mostly applies to SecureChannel, which needed several changes to replace their native
SSPI implementation with Mono's new managed version.
* Net/SecureProtocols/SslStream.cs: Make partial and comment out 'SslStream.TransportContext'.
* Net/SecureProtocols/SslState.cs: Make partial and add new internal constructor; pass the
'SSPIConfiguration' argument down to SecureChannel's .ctor.
* Net/SecureProtocols/_SslStream.cs: Make partial.
* Net/SecureProtocols/_SecureChannel.cs: Add Mono-specific SSPI APIs.
* Net/_SslSessionCache.cs: Make conditional.
- SafeRegistryHandle.ReleaseHandle is only called on code that is ignored on Mono.
- Mono is using AppDomain.FriendlyName directly instead of UnsafeNativeMethods.GetModuleFileName.
* Builds without [MethodImpl(MethodImplOptions.AggressiveInlining)]
when target framework version is lower than 4.5.
* Use Thread.Volatile instead of Volatile.Read when target framework
version is lower than 4.5.
* Change location of internal Array constants. Mono runtime cannot handle
array type with constants.
* Removes some warnings from the build for unused fields.
This patch does the following things:
* Adds support to build without Code Access Security, by using the new
define DISABLE_CAS_USE. The reason is that this is both slow, and
Mono does not actually offer a Code Access Secured environment. So
avoid paying the price.
* Removes some warnings from the build for unused variables.
* Allows the code to be bootstrapped with a 2.0 compiler (which Mono
does during the bootstrap phase currently).
Socket.AcceptAsync should use SocketAsyncEventArgs.AcceptSocket
Socket.AcceptAsync shouldn't create new socket if SocketAsyncEventArgs.AcceptSocket isn't null.
The MoveTo target is in the current working directory instead of in the temp dir, so it runs into permission issues.
Use the temp dir instead. Fixes part 2 of #40624
We don't have .NEt 4.6 installed on all of our build machines, to
running mcs.exe on Windows fails, as it was built with .NET 4.6.
Instead, we will run mcs.exe via Mono on Windows (which is what happens
on OSX, although there the low-level Process class handles it).
To accomplish this, we run mcs.bat instead of mcs.exe.
The child process would simpky execute ls, then exiting right away. Writing to its stdin would then crash with "EPIPE: An attempt is made to write to a pipe that is not open for reading by any process." as the child process exited.
Some serialization attributes are unlikely to have direct reference,
from IL, but will be needed at runtime to correctly serialize (or
deserialize) objects.
* OnDeserializedAttribute
* OnDeserializingAttribute
* OnSerializedAttribute
* OnSerializingAttribute
This, along with aad627c5e8, fixed bug
40574. The later part will have new unit tests in XI.
references:
https://bugzilla.xamarin.com/show_bug.cgi?id=40574
The problem is that private fields in described types doesn't start from _ prefix, but it should be so according to reference source.
That's way MONO can't deserialize that types correctly if source stream was created by MS .NET binary serializer.
This change is released under the MIT license.
Switching maccore to use mono-4.5.0-branch introduced this bug where
AesManaged is not available in System.Core for both tvOS and watchOS
profiles (iOS is fine).
This results in unit tests failures on bots building maccore/master.
This fix the problem by importing the "main" monotouch_System.Core.
dll.sources into the other profiles - since they all share the same
feature set (in this case) and make it less likely to forget something
else in the future.
reference:
https://bugzilla.xamarin.com/show_bug.cgi?id=40570
[sgen] Add regression test for the interaction between the bridge and handles.
The bridge code was triggering handles to be considered deallocated thus leading
them to be overwritten.
[corlib] Ifdef out calls to SynchronizationAttribute on monotouch
When WaitHandle was imported from referencesource in f6fa8f0, the code was refactored and basically all WaitHandle.Wait* calls end up in WaitOneNative or WaitMultiple now.
On monotouch those methods threw a NotSupportedException when exitContext == true even before the import, so this behavior got inherited to some other methods that didn't throw after the exception was removed from them in 93e31cb. This broke tests in maccore and other code that passed exitContext=true.
To fix this, we just ifdef out the calls to SynchronizationAttribute on monotouch since they don't make sense there anyway since remoting isn't available. The commit that initially added the exception in 2ce9fc1 only did that to make the behavior consistent when the linker is enabled. By removing the calls we get the same result.
Do the same for Monitor so we're consistent.
commit message from @akoeplinger
[verifier] Do method visibility checks for virtual final methods. Enable type visibility for all methods. Fixes#5644.
I cannot repro the issue mentioned in the comments of mono_method_can_access_method.
* FIXME:
* with generics calls to explicit interface implementations can be expressed
* directly: the method is private, but we must allow it. This may be opening
* a hole or the generics code should handle this differently.
* Maybe just ensure the interface type is public.
@monojenkins merge
This particular piece of documentation is added manually, because it's not in the regular BCL library, but is provided by Xamarin on the iOS platform. Because it was missing an entry in the index file, it was excluded from the published documentation when doing mdoc assemble.
When WaitHandle was imported from referencesource in f6fa8f0, the code was refactored and basically all WaitHandle.Wait* calls end up in WaitOneNative or WaitMultiple now.
On monotouch those methods threw a NotSupportedException when exitContext == true even before the import, so this behavior got inherited to some other methods that didn't throw after the exception was removed from them in 93e31cb. This broke tests in maccore and other code that passed exitContext=true.
To fix this, we just ifdef out the calls to SynchronizationAttribute on monotouch since they don't make sense there anyway since remoting isn't available. The commit that initially added the exception in 2ce9fc1 only did that to make the behavior consistent when the linker is enabled. By removing the calls we get the same result.
Do the same for Monitor so we're consistent.
We load these at runtime so they will naturally have undefined symbols. This is
not allowed under bitcode, though, and we use the static libraries in that case
anyway.
I cannot repro the issue mentioned in the comments of mono_method_can_access_method.
* FIXME:
* with generics calls to explicit interface implementations can be expressed
* directly: the method is private, but we must allow it. This may be opening
* a hole or the generics code should handle this differently.
* Maybe just ensure the interface type is public.
Remove calls to mono_lock_free_queue_node_init for nodes that just came out of mono_lock_free_queue_dequeue.
That call is superfluous as dequeue will set the returned node 'next' field to INVALID_NEXT, which is the
same thing the mono_lock_free_queue_node_init call did.
[utils] Fix MONO_SIGNAL_USE_UCONTEXT_T not being defined on tvOS simulator
In the maccore tvossimulator tests we were seeing the following assertion with the Mono 4.5 preview:
```
error: * Assertion: should not be reached at /Users/builder/data/lanes/1381/0d6923b4/source/maccore/external/mono/mono/utils/mono-context.c:194
0 dont link 0x000000010175702d mono_handle_native_sigsegv + 253
1 libsystem_platform.dylib 0x000000010a0cfeaa _sigtramp + 26
2 ??? 0x0000000001bfc4db 0x0 + 29344987
3 libsystem_c.dylib 0x0000000109e56cbc abort + 129
4 dont link 0x00000001018d1503 _ZL12log_callbackPKcS0_S0_iPv + 67
5 dont link 0x00000001018c26d0 monoeg_assertion_message + 192
6 dont link 0x00000001018af6de mono_sigctx_to_monoctx + 30
7 dont link 0x000000010183f473 sgen_suspend_thread + 323
8 dont link 0x000000010183f58e sgen_thread_handshake + 126
9 dont link 0x000000010183fb20 sgen_client_stop_world + 1296
10 dont link 0x0000000101871283 sgen_stop_world + 67
11 dont link 0x0000000101870d43 sgen_perform_collection + 99
12 dont link 0x0000000101872576 sgen_gc_collect + 102
13 dont link 0x00000001018d1886 _ZL7pump_gcPv + 54
14 libsystem_pthread.dylib 0x000000010a0b7c13 _pthread_body + 131
15 libsystem_pthread.dylib 0x000000010a0b7b90 _pthread_body + 0
16 libsystem_pthread.dylib 0x000000010a0b5375 thread_start + 13
```
This was caused by a4e257a07f, the issue is that the check was changed from UCONTEXT_REG_RAX to MONO_SIGNAL_USE_UCONTEXT_T, but that is only available when HAVE_SIGACTION is defined which it is not in this case.
Talked with Zoltan, fixing this by enabling MONO_SIGNAL_USE_UCONTEXT_T on all apple platforms for now.
/cc @vargaz @xmcclure
@monojenkins merge
1. Mark mono_array_clone external only. Runtime should use
mono_array_clone_checked.
2. Add MonoError* arg to mono_array_clone_in_domain.
3. Separate ves_icall_System_Array_Clone
Mark it external only. Runtime should use
mono_property_set_value_checked.
Note: mono_property_set_value_checked always catches exceptions from
managed code and threads them out via MonoError.
We need to make sure all external references have been resolved so that we
don't invoke the dynamic linker in the mono_sample_hit () callback which is
called in async-signal context.
There is a slight problem when we're using CLOCK_PROCESS_CPUTIME_ID: If we're
shutting down and there's largely no activity in the process other than waiting
for the sampler thread to shut down, it can take upwards of 20 seconds
(depending on a lot of factors) for us to shut down because the sleep
progresses very slowly as a result of the low CPU activity.
We fix this by repeatedly sending the profiler signal to the sampler thread in
order to interrupt the sleep.
We do not need to do this on platforms where we use a regular sleep based on a
monotonic clock. The sleep will return in a reasonable amount of time in those
cases.
The old code had quite a few problems:
1. It would result in stack corruption under heavy load, somehow. This typically
resulted in an assertion when starting the unwinding process because the
unwinding flags would be inconsistent.
2. It sometimes wrote complete garbage into the statistical buffers.
3. As a result of using the helper thread to assemble collected samples into log
buffers, it did not work on platforms where the helper thread is disabled.
4. Performance was poor under heavy load because waking up the helper thread to
assemble events would always trigger a counters sample + dump.
5. The code was very complicated and hard to understand/maintain.
(Points (1) and (2) can probably be attributed to point (5).)
In the new sampling world order, we use a lock-free allocator and a lock-free
reuse queue to acquire memory for sample hit events in the sample hit callback.
We then fill it with the info we collect during the async stack walk. Finally,
we ship it off to a separate dumper thread (via another lock-free queue) where
it gets turned into a log buffer and then shipped off to the writer thread.
This new approach appears very stable; I've yet to make it crash in any of the
stress tests I typically run. In a heavy workload, the helper thread's CPU
usage was previously around 25-30%. It is now down to basically 0% while the
dumper thread is only around 15%. CPU usage by threads that are collecting
samples is a bit higher than before, but only until the reuse queue converges
on a size that is big enough to satisfy all sample event requests. At that
point, no more allocations from the lock-free allocator happen, and performance
improves.
Our previous default (1000 Hz) was quite extreme and resulted in very high
overhead from the profiler for moderate workloads.
Also, fix the profiler to actually treat the 'freq' argument as a frequency
when passing it to the runtime. It was being incorrectly treated as a sleep
interval.
In the maccore tvossimulator tests we were seeing the following assertion with the Mono 4.5 preview:
```
error: * Assertion: should not be reached at /Users/builder/data/lanes/1381/0d6923b4/source/maccore/external/mono/mono/utils/mono-context.c:194
0 dont link 0x000000010175702d mono_handle_native_sigsegv + 253
1 libsystem_platform.dylib 0x000000010a0cfeaa _sigtramp + 26
2 ??? 0x0000000001bfc4db 0x0 + 29344987
3 libsystem_c.dylib 0x0000000109e56cbc abort + 129
4 dont link 0x00000001018d1503 _ZL12log_callbackPKcS0_S0_iPv + 67
5 dont link 0x00000001018c26d0 monoeg_assertion_message + 192
6 dont link 0x00000001018af6de mono_sigctx_to_monoctx + 30
7 dont link 0x000000010183f473 sgen_suspend_thread + 323
8 dont link 0x000000010183f58e sgen_thread_handshake + 126
9 dont link 0x000000010183fb20 sgen_client_stop_world + 1296
10 dont link 0x0000000101871283 sgen_stop_world + 67
11 dont link 0x0000000101870d43 sgen_perform_collection + 99
12 dont link 0x0000000101872576 sgen_gc_collect + 102
13 dont link 0x00000001018d1886 _ZL7pump_gcPv + 54
14 libsystem_pthread.dylib 0x000000010a0b7c13 _pthread_body + 131
15 libsystem_pthread.dylib 0x000000010a0b7b90 _pthread_body + 0
16 libsystem_pthread.dylib 0x000000010a0b5375 thread_start + 13
```
This was caused by a4e257a07f, the issue is that the check was changed from UCONTEXT_REG_RAX to MONO_SIGNAL_USE_UCONTEXT_T, but that is only available when HAVE_SIGACTION is defined which it is not in this case.
Discussed with Zoltan, fixing this by enabling MONO_SIGNAL_USE_UCONTEXT_T on all apple platforms for now.
1. MonoError-ize mono_remoting_wrapper icall.
This is a bit this method is allowed to use
mono_error_raise_exception: it is called from a
MONO_WRAPPER_REMOTING_INVOKE method which is considered a "protected
wrapper" in which the usual pending exception interruption check is a no-op.
2. Use mono_set_pending_exception in mono_marshal_set_domain_by_id icall
fixup f5dc934
The following test would trigger a SIGSEGV during a GC collection:
```
object _Lock;
C5.HashedLinkedList<byte[]> _Set;
List<byte[]> _Candidates;
void Run ()
{
_Lock = new object ();
_Set = new C5.HashedLinkedList<byte[]>(new Core.CommonUtils.ByteArrayComparer());
BuildList ();
for (var i = 0; i < 10; ++i) {
Task.Run (() => WorkOnSet ());
}
}
void BuildList() {
_Candidates = new List<byte[]>();
Random r = new Random();
for(var i = 0; i < 20; ++i) {
var key = new byte[16];
r.NextBytes(key);
_Candidates.Add(key);
}
}
void WorkOnSet() {
Random r = new Random();
for (;;) {
byte[] src = _Candidates [r.Next (_Candidates.Count)];
byte[] key = new byte[src.Length];
for (int i = 0; i < src.Length; ++i)
key [i] = src [i];
lock(_Lock) {
if(r.Next (2) != 0)
_Set.Add(key);
else
_Set.Remove(key);
}
}
}
```
The binary protocol just before the crash would be as follow:
```
collection_requested generation 0 requested_size 4096 force false
collection_begin index 14 generation 0
copy from 0x3150658 to 0x47e0e40 vtable 0x189f1780 size 24
gray_enqueue queue 0xa4a2d0 cursor 0x3b68178 value 0x47e0e40
gray_dequeue queue 0xa4a2d0 cursor 0x3b68178 value 0x47e0e40
scan_begin obj 0x47e0e40 vtable 0x189f1780 size 24
scan_process_reference obj 0x47e0e40 ptr 0x47e0e48 value 0x3150618
scan_process_reference obj 0x47e0e40 ptr 0x47e0e4c value 0x3150638
scan_process_reference obj 0x47e0e40 ptr 0x47e0e50 value 0x0 <----- here the value is 0x0
scan_process_reference obj 0x47e0db0 ptr 0x47e0dc0 value 0x47e0e40
collection_end 14 generation 0 scanned 0 unique 0 0.00%
wbarrier ptr 0x3af8710 value 0x47e0e40 value_vtable 0x189f1780
collection_requested generation 0 requested_size 4096 force false
collection_begin index 15 generation 0
collection_end 15 generation 0 scanned 0 unique 0 0.00%
collection_requested generation 0 requested_size 4096 force false
collection_begin index 16 generation 0
card_scan start 0x47e0df8 size 520
scan_begin obj 0x47e0e40 vtable 0x189f1780 size 24
scan_process_reference obj 0x47e0e40 ptr 0x47e0e48 value 0x47b4d40
scan_process_reference obj 0x47e0e40 ptr 0x47e0e4c value 0x47b03c0
scan_process_reference obj 0x47e0e40 ptr 0x47e0e50 value 0x3100800 <----- here the value is 0x3100800, but no WB in between
```
Object at 0x47e0e40 is a C5.HashSet<T>.Bucket with T a C5.KeyValuePair<byte[],C5.HashedLinkedList<byte[]>.Node>. The memory layout of C5.HashSet<T>.Bucket would be as follow:
class Bucket {
item: offset 8 size 8
overflow: offset 16 size 4
hashval: offset 20 size 4
}
So the bug would manifest itself as a missing write barrier when writing to Bucket.overflow, leading to the premature sweeping of a Bucket object. The Bucket.overflow value would then point to the middle of an object, triggering a SIGSEGV when trying to access its vtable.
When running basic make check in mono/mini there are some hidden errors in the check-seq-points test. Current test compares output between JIT and AOT compile assemblies but doesn't pay attention to the result of the as and ld calls building the AOT:ed modules. It turns out that the linkage of the AOT compiled module on win32 x86 actually fails due to naming of symbols included in .def or .globl assembler expressions. The fix makes sure that local function symbols or global symbol definitions follows the cdecl win32 x86 convention and are prefixed with _. The fix tries to resolve this using two different strategies. Since there are many code paths in the AOT compiler emitting local/global symbols + referencing the emitted symbols, it doesn't look like there is one place to fix this (unless rename the symbol altogether and that could cause regressions elsewhere).
First, local function symbols are emitted using .def expression when used as debug symbols. By prefixing debug symbols with _ will resolve most of the local function symbols. By mangling the debug symbols we can fix both emitted symbols and references to these symbols in two location (get_debug_sym, get_plt_entry_debug_sym).
Second, in order to make sure all other symbols (local/global) ending up in .def or .globl expressions are correctly mangled, the functions to emit local/global symbols in aot-compiler.c have been extended to check mangling on passed in symbols before emitting. If passed in symbol doesn't follow the name mangling rule, the symbols will be mangled and the mangled symbol will be emitted together with a label to the mangled symbol. This is done to make sure we have a correct label for the emitted symbol that can be used by the linker. Since the rest of the code is unaware of the mangled name, it will continue reference the unmangled name and since there is already a label to the unmangled symbol, the mangling will be transparent for the rest of the code.
NOTE, the fix only applies to windows X86 build, other configurations are not affected by this issue.
This PR also includes a fix detected on windows when debugging and fixing above issue. The AOT compiler tries to rename linked module to the final target name, for example from basic.exe.dll.tmp to basic.exe.dll. If the file already exist, the rename will fail on windows causing a silent failure in the AOT compiler and this will "leak" the tmp file + ending up with the old basic.exe.dll. The fix is to check result of the rename function call and if it fails with "file already exist error", unlink target and redo rename operation. The rename can still fail due to other reasons ending up in the same problem, but to make a good fix, the error handling of the method needs to be revisited in order to properly clean up tmp files and allocations in error cases. This will be handled in a different PR.
Fix undefined behaviour in left-shift.
This PR fixes a left-shift that relied on undefined behaviour.
Clang emitted the following warning (prior to this patch):
```
sgen-descriptor.c:189:54: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
return desc | VECTOR_SUBTYPE_REFS | ((gssize)(-1) << 16);
~~~~~~~~~~~~ ^
```
Mark mono_store_remote_field_new external only.
Runtime should use mono_store_remote_field_new_checked.
Change mono_marshal_get_stfld_remote_wrapper to use
mono_store_remote_field_new_icall instead of a native call.
Add mono_load_remote_field_new_checked. Mark mono_load_remote_field_new
external only. Add icall mono_load_remote_field_new_icall and use it in
remoting wrappers.
[appdomain] Cleanup MonoError code
1. Assert in mono_get_corlib_version instead of raising an exn.
2. Don't raise in mono_domain_create_appdomain external API function.
3. Assert in mono_domain_assembly_postload_search instead of raising.
4. Don't raise in mono_domain_fire_assembly_load.
5. Use mono_error_set_pending_exception in
ves_icall_System_AppDomain_InternalGetProcessGuid
6. Dont' raise in unload_thread_main, populate unload_data instead.
Added support for SHA256, 384 and 512 in AuthenticodeDeformatter.
Due to the changes introduced by Microsoft to require signing with stronger algorithms, SHA-256, SHA-384 and SHA-512 support was needed.
This change is released under the MIT license.
There could be a race between System.Net.Sockets.Socket:Close_internal and the io threadpool cleanup, which would lead to a native crash. That's because the io threadpool cleanup would clean the mutex, condition variables as well as close the sockets.
The easiest way to fix that is by cleaning only what's going to stop the runtime from shutting down, and in this case the background selector thread. The same is done for the threadpool where we just shutdown the monitor thread and the worker threads.
Due to the changes introduced by Microsoft to require signing with stronger algorithms, SHA-256, SHA-384 and SHA-512 support was needed.
This change is released under the MIT license.
[marshal] Ensure marshal code emit the right indirect store when dealing with ref types.
The marshal code was using CEE_STIND_I for a lot of ref stores. This would not
generate a WB and is a heap corruption vector.
Part of man-page showing this interface:
PTHREAD_GETNAME_NP(3) Library Functions Manual PTHREAD_GETNAME_NP(3)
NAME
pthread_getname_np - get and set descriptive name of a thread
LIBRARY
POSIX Threads Library (libpthread, -lpthread)
SYNOPSIS
#include <pthread.h>
int
pthread_getname_np(pthread_t thread, char *name, size_t len);
int
pthread_setname_np(pthread_t thread, const char *name, void *arg);
-- pthread_setname_np(3)
The .orig files are traditionally files of patch(1) and diff(1) tools.
Remove stale file Makefile.orig as it is generating false positives for
grabbing patches from a directory tree.
[metadata] Don't require HandleRef in domain.c
In 1da6b76297 loading of System.Runtime.InteropServices.HandleRef
was changed from optional to required. This caused errors in the maccore tests:
```
MONO_PATH=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/64/Build /Users/builder/data/lanes/1411/5ca65b46/source/maccore/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/bin/arm64-darwin-mono-sgen --debug -O=gsharedvt --aot=mtriple=arm64-ios,data-outfile=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/System.arm64.aotdata,static,asmonly,direct-icalls,full,dwarfdebug,no-direct-calls,outfile=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/System.dll.arm64.s "/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/64/Build/System.dll"
Runtime critical type System.Runtime.InteropServices.HandleRef not found
AOT Compilation exited with code 134, command:
```
@monojenkins merge
In 1da6b76297 loading of System.Runtime.InteropServices.HandleRef
was changed from optional to required. This caused errors in the maccore tests:
```
MONO_PATH=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/64/Build /Users/builder/data/lanes/1411/5ca65b46/source/maccore/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/bin/arm64-darwin-mono-sgen --debug -O=gsharedvt --aot=mtriple=arm64-ios,data-outfile=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/System.arm64.aotdata,static,asmonly,direct-icalls,full,dwarfdebug,no-direct-calls,outfile=/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/System.dll.arm64.s "/Users/builder/data/lanes/1411/5ca65b46/source/maccore/msbuild/tests/MyKeyboardExtension/obj/iPhone/Debug/mtouch-cache/64/Build/System.dll"
Runtime critical type System.Runtime.InteropServices.HandleRef not found
AOT Compilation exited with code 134, command:
```
gmcs and ilasm2 are no longer distributed with Mono.
The run-test step had several issues that prevented it from running,
e.g. the comments caused "unexpected end of line" issues on OSX.
Improve shell portability on NetBSD
The "test" command, as well as the "[" command, are not required to know
the "==" operator.
Comparing with empty strings ("") might lead to unspecified results.
Replace unportable shell loops:
for I in {2..6}; do ..; done
with POSIX ones:
I=2; while [ $I -le 6 ]; do ..; I=$((I + 1)); done
The "test" command, as well as the "[" command, are not required to know
the "==" operator.
Comparing with empty strings ("") might lead to unspecified results.
Replace unportable shell loops:
for I in {2..6}; do ..; done
with POSIX ones:
I=2; while [ $I -le 6 ]; do ..; I=$((I + 1)); done
The <sys/errno.h> header on NetBSD declares only a list of ERROR values.
This fixes the following build issue:
In file included from ../../mono/utils/mono-proclib.h:7:0,
from mono-proclib.c:8:
mono-proclib.c: In function 'mono_cpu_usage':
mono-proclib.c:893:54: error: 'errno' undeclared (first use in this function)
g_error ("getrusage() failed, errno is %d (%s)\n", errno, strerror (errno));
^
../../eglib/src/glib.h:600:70: note: in definition of macro 'g_error'
#define g_error(...) do { g_log (G_LOG_DOMAIN, G_LOG_LEVEL_ERROR, __VA_ARGS__); for (;;); } while (0)
^
mono-proclib.c:893:54: note: each undeclared identifier is reported only once for each function it appears in
g_error ("getrusage() failed, errno is %d (%s)\n", errno, strerror (errno));
^
../../eglib/src/glib.h:600:70: note: in definition of macro 'g_error'
#define g_error(...) do { g_log (G_LOG_DOMAIN, G_LOG_LEVEL_ERROR, __VA_ARGS__); for (;;); } while (0)
The NetBSD version of mremap takes additional argument "newp" to hint
new position of a mapped range.
void *
mremap(void *oldp, size_t oldsize, void *newp, size_t newsize,
int flags);
-- NetBSD's mremap(2)
While Linux by default sets old_address and fails if the resize
operation cannot be finished for the original address.
void *mremap(void *old_address, size_t old_size,
size_t new_size, int flags, ... /* void *new_address */);
-- Linux's mremap(2)
NetBSD offers inversed logic with the MAP_FIXED flag to Linux's MREMAP_MAYMOVE:
MAP_FIXED newp is tried and mremap() fails if that address can't
be used as new base address for the range. Otherwise,
oldp and newp are used as hints for the position,
factoring in the given alignment.
-- NetBSD's mremap(2)
MREMAP_MAYMOVE
By default, if there is not sufficient space to expand a mapping
at its current location, then mremap() fails. If this flag is
specified, then the kernel is permitted to relocate the
mapping to a new virtual address, if necessary. If the mapping
is relocated, then absolute pointers into the old mapping
location become invalid (offsets relative to the starting address
of the mapping should be employed).
-- Linux's mremap(2)
The thing that bothers is calling mremap(2) syntax from Linux "posix", but we
cannot do anything with it, for now just simulate the Linux behavior on NetBSD.
The problem here was that the function was not correctly setting is_generic when handling partially inflated methods.
The working scenario was method 'Foo<A>::Bar<B>' inflated with (Something<Else>;)[1] resulting in Foo<Something<Else>>::Bar<B>.
This would work because the is_generic check would verify that the supplied context has no method_inst, thus had to remain a GMD.
The problematic one would be inflating Foo<Something<A>>::Bar<B> with (Else;) resulting too in Foo<Something<Else>>::Bar<B>.
To handle this, we would first extract the root method definition [2] and then inflate the original method context with the supplied
context. So we end up with Foo<A>::Bar<B> and inflate (Something<A>;B) with (Else;) resulting in (Something<Else>;B) *1*.
Finally, we would inflate Foo<A>::Bar<B> with that new context, resulting in Foo<Something<Else>>::Bar<B>.
The resulting method is the same in both scenarios, but context used for inflation is different in that in the second case
it includes the GMD method instantiation (The B in *1*).
This would trip up the computation of whether we ended up with a GMD or not.
The right check, though, is not whether the inflation context has a method instantiation or not, but whether the resulting method
has the same method context as the root GMD[2].
This affected #29585 since disabling some optimizations triggered things to be inflated in different ways.
[1] (Something<Else>;) means a context with a class instantiation and no method instantiation
[2] The GMD defined on the GTD.
<configuration><configSections><section> in a .config file takes a
type name as argument. Some of the allowed types are in System.dll.
The type lookup is done in System.Configuration.dll and does not find
these types. Check System.dll as a fallback to fix this.
This appears to have been copied from Boehm, but here we are not responsible for zeroing.
Also, 's->chars [len]' is not the same as 'align_up (s + size) - 2', so this was corrupting canaries.
The first nursery collection after the start of a concurrent collection allocated mod union card tables for all blocks. Allocate it only for blocks that have dirty cards. Blocks without a mod_union are not scanned in the major collector. This can reduce the pause time of those nursery collections by half.
[sgen] Scan pinned objects in nursery as part of concurrent mark
Objects pinned in the nursery can hide behind them a large graph of objects in the heap which is never scanned because we don't follow references in the nursery. After precleaning, we scan the latest set of pinned objects (while locking in order to make sure that the set doesn't change, so none are moved).
One overlooked aspect of mach is that its calls are interruptible and
we never really accounted for that.
This problem turned up when sampling at high frequency as they would
start to randomly fail.
'make check-seq-points' on Cygwin 2.4.1 64-bit.
* Added removal of *.exe.dll to the clean_aot() function to delete AOT output
from previous 'make check-seq-points' runs.
* Use ';'' as separator in MONO_PATH if running under Cygwin rather than ':'.
Objects pinned in the nursery can hide behind them a large graph of objects in the heap which is never scanned because we don't follow references in the nursery. After precleaning, we scan the latest set of pinned objects (while locking in order to make sure that the set doesn't change, so none are moved).
NativeMethods.GetExitCodeProcess would inherently be broken on Unix platforms for non-child processes as it can't succesfully get the exit code. This would in turn break Process.HasExited, which should not break, as it is possible to check if a non-child process is alive or dead.
Instead of opening a file for the sole purpose of calling
F_GETPATH on the resulting descriptor, we can just use the
native realpath() function; as a side benefit, this also works
on *BSD where F_GETPATH isn't supported.
Per current .NET, exception serialization may be implemented using the
SerializeObjectState event. Use of this approach is mandatory for
certain classes, such as HttpRequestException subclasses. Mono does
not support it. The solution is to switch out our System.Exception
implementation for the referencesource one.
To accommodate this, classes that access System.Exception internal
fields directly must switch to the new field names; a new support
method is added to System.Environment; the "security level" methods
and icall are added to MonoCMethods; and the field order in
_MonoException must be adjusted. A new test is also added in the
System.Runtime.Serialization assembly.
jlorenss/win x64 support
Initial fixes needed in order to get pass on mono regression tests for windows x64 visual studio build,
--regression basic.exe basic-float.exe basic-long.exe basic-calls.exe objects.exe arrays.exe basic-math.exe exceptions.exe iltests.exe devirtualization.exe generics.exe basic-simd.exe
Before fixes there were failures in basic.exe, exceptions.exe and objects.exe for windows x64 Visual studio build.
Failures in basic.exe and exceptions.exe was due to a mix up of signal defines on windows platform in Visual Studio build where SIGILL and SIGFPE was defined differently in different source files causing incorrect signal handler to be invoked at runtime, causing incorrect thrown .NET exception. Fix is to make sure signal.h is always included giving the same value for SIGILl and SIGFPE over all source files.
Failures in objects.exe test_0_mul_ovf_regress_36052 caused a stack overflow crash when running togehter with branch optimization mode. This was due to incorrect lowering of x64 test instruction when optimization moved over to use R8 instead of RAX and current lowering didn't handle that correctly meaning that the actual test was performed against wrong register causing a loop that caused a stack overflow error. The specific amd64 codegen , amd64_test_reg_imm, is currently only used on x64 windows platform for the stackalloc call.
In SSCX.OSX509Certificates.GetCertificate(), we cannot simply use
'X509Certificate.Handle' because it may point to something that's
not a SecCertificate.
(cherry picked from commit 3fc068f7e2)
In the lazy sweep case, the sweep job doesn't sweep the blocks. Instead, the blocks are swept when they are actually needed (when we want to allocate in the block, scan the block etc.). This delays the mutator or increases the nursery pauses. Try to sweep as many blocks as possible concurrently, immediately after we finish the sweep job.
Currently only triggered by objects.exe regression test on windows x64 with branch opt, since stackalloc gets lowered to this test and branch will start using R8 instead of RAX for the test.
Caused incorrect exception behavior and failures in basic.exe, exceptions.exe regression test on x64 windows. This was due to the fact that some source files didn't include signal.h while others did. In case where signal.h was not included incompatible defines were used for SIGILL and SIGFPE in mini-amd64.h.
x86 cleanups for register arguments
Best I can tell, cinfo->args and cfg->args need to be independent before implementing calling conventions where some arguments are in registers, at least if we want to do it cleanly. Also, storing the argument offset in a cfg->args instruction that claims to be for a register seems hacky and confusing to me.
We decompose them as part of the cprop pass to handle more immediates and to further propagate copies.
The native backend of some architectures (for example x86) may support long shifts with specialized instructions. Even there, this still makes immediate shifts 2 times faster because long ops are poorly handled by the jit.
On arm, where we were emulating, this makes long immediate shifts 3-5 times faster.
[sgen] Set ARCH_MIN_MS_BLOCK_SIZE to 64k on both ppc32 and ppc64.
This fixes a problem that was happening on Debian ppc32 build machines. Since
they are set up as a 32-bit userland on a 64-bit kernel, Mono would be compiled
with an ARCH_MIN_MS_BLOCK_SIZE value of 16k. However, at runtime, the system
page size would be 64k (due to the 64-bit kernel with PPC_64K_PAGES set).
Signed-off-by: `Alex Rønne Petersen <[email protected]>`
[metadata] Ifdef out an icall that is not available in the DISABLE_REFLECTION_EMIT case
It caused errors like the following on the maccore builds:
```
CCLD libmonosgen-2.0.la
Undefined symbols for architecture armv7k:
"_mono_reflection_get_custom_attrs_blob_checked", referenced from:
_ves_icall_System_Reflection_Emit_CustomAttributeBuilder_GetBlob in libmonoruntimesgen.a(libmonoruntimesgen_la-icall.o)
ld: symbol(s) not found for architecture armv7k
```
---
This is the last patch required for getting maccore master to build again with mono master.
@monojenkins merge
This is the support that makes it so that mkbundle can create self-contained executables without requiring a native toolchain to be present. Also is the foundation to allow cross-compilation of self-contained executables.
This works by concatenating the runtime with the various dependencies and configuration files, attaching a directory at the end of the file and a signature to flag this as a runtime that has embedded resources in the executable file itself.
This works by modifying the runtime startup to check for a special signature at the end of the main executable, if it is not present, execution resumes as usual. Otherwise a directory is read from the end of the file which contains both the kind of file (assembly, configuration file, and so on), as well as pointer to the contents in the file. Then the regular Mono APIs to load resources are used to load these.This is the support that makes it so that mkbundle can create self-contained executables without requiring a native toolchain to be present. Also is the foundation to allow cross-compilation of self-contained executables.
This works by concatenating the runtime with the various dependencies and configuration files, attaching a directory at the end of the file and a signature to flag this as a runtime that has embedded resources in the executable file itself.
This works by modifying the runtime startup to check for a special signature at the end of the main executable, if it is not present, execution resumes as usual. Otherwise a directory is read from the end of the file which contains both the kind of file (assembly, configuration file, and so on), as well as pointer to the contents in the file. Then the regular Mono APIs to load resources are used to load these.
It caused errors like the following on the maccore builds:
```
CCLD libmonosgen-2.0.la
Undefined symbols for architecture armv7k:
"_mono_reflection_get_custom_attrs_blob_checked", referenced from:
_ves_icall_System_Reflection_Emit_CustomAttributeBuilder_GetBlob in libmonoruntimesgen.a(libmonoruntimesgen_la-icall.o)
ld: symbol(s) not found for architecture armv7k
```
This fixes a problem that was happening on Debian ppc32 build machines. Since
they are set up as a 32-bit userland on a 64-bit kernel, Mono would be compiled
with an ARCH_MIN_MS_BLOCK_SIZE value of 16k. However, at runtime, the system
page size would be 64k (due to the 64-bit kernel with PPC_64K_PAGES set).
Signed-off-by: Alex Rønne Petersen <[email protected]>
Apparently Windows doesn't free this debug info in Vista+ when calling
DeleteCriticalSection [1], so just avoid allocating it in the first place by
calling InitializeCriticalSectionEx with the CRITICAL_SECTION_NO_DEBUG_INFO
flag. This function is only available in Vista+, but we've dropped support for
XP anyway.
[1] http://stackoverflow.com/a/889853
We don't evacuate from full blocks. When it comes to blocks that have free slots, instead of evacuating all of them to a new set of blocks, aim to keep the fuller blocks around and evacuate the emptier ones into them.
[profiler] Fix incorrect initial writer semaphore value.
This was used during debugging. I forgot to revert it back to 0 before posting
the pull request (#2838). This led to ~3-5% more CPU usage by the writer thread
than actually necessary; it now sits around 0-1% for typical workloads.
The overloaded GetValidator() which takes both 'MonoTlsSettings'
and 'MonoTlsProvider' is actually used by the test suite, so let's
keep it public.
(cherry picked from commit 0977d531d2)
babysitter generates some temp paths in python when running under cygwin, which breaks down
when those paths are passed to the native Windows mono.exe during testing:
Exception in TestRunnerThread ---> System.IO.DirectoryNotFoundException: Could not find a part of the path "C:\cygdrive\c\WINDOWS\TEMP\tmpC4K5yK\babysitter_report_current_test_file.txt".
This was used during debugging. I forgot to revert it back to 0 before posting
the pull request (#2838). This led to ~3-5% more CPU usage by the writer thread
than actually necessary; it now sits around 0-1% for typical workloads.
[profiler] Fix usage of the lock-free queue API.
* Free nodes hazardously.
* No need to unpoison nodes since we're not reusing them.
* Fix memory leak when emptying coverage queues.
1. icall `mono_image_module_basic_init`
2. icall `mono_reflection_sighelper_get_signature_local`
3. icall `mono_reflection_sighelper_get_signature_field`
4. API function mono_module_get_object
5. API function mono_type_get_object
6. API function mono_method_get_object
7. API function mono_field_get_object
8. API function mono_property_get_object
9. API function mono_event_get_object
1. static function `create_custom_attr` (set MonoError exception instance)
2. icall `mono_reflection_resolve_custom_attribute_data` (set pending
exception from MonoError). Also rename it to
ves_icall_System_Reflection_CustomAttributeData_ResolveArgumentsInternal
3. icall `mono_reflection_bind_generic_method_parameters` (set pending
exception from MonoError) Not renaming this one because it's actually
used for two icalls:
System.Reflection.Emit.MethodBuilder.MakeGenericMethod and System.Reflection.MonoMethod.MakeGenericMethod_impl
Change icall name to
ves_icall_System_Reflection_Emit_TypeBuilder_create_generic_class
Give mono_reflection_create_generic_class a MonoError argument,
and get rid of the redundant mono_reflection_create_generic_class_checked
Some Jenkins workers (e.g. the s390x one) have an earlier python version installed where the babysitter doesn't work.
Fall back to timeout in those cases.
* Mono.Security.Interface.ICertificateValidator2:
new internal interface, deriving from ICertificateValidator.
* Mono.Security.Interface.CertificateValidationHelper:
the internal GetValidator() now returns ICertificateValidator2.
* Mono.Security.Interface.MonoTlsProvider:
make HasCustomSystemCertificateValidator and InvokeSystemCertificateValidator()
internal and use ICertificateValidator2.
* Mono.Net.Security.ChainValidator:
implement ICertificateValidator2; we may now be called with a
fully-built X509Chain.
* corlib, System, Mono.Security:
add [assembly: InternalsVisibleTo].
[profiler] Use a semaphore to signal the writer thread.
We would previously constantly try to dequeue from the writer queue which would
result in extremely high CPU usage, making many apps (e.g. Xamarin Studio)
nearly unusable.
[corlib] Type.GetType parser and resolver improvements
We have two type parsers the one invoked by `Type.GetType(string)` (and some others) which is implemented in C in `reflection.c`, and the one invoked by `Type.GetType(string, Func<...>, Func<...>)` implemented in managed code as `System.TypeSpec`.
In both cases `string→Type` conversion goes in two stages: parsing and resolution. Parsing just changes the string into an AST. And resolution crawls the AST and constructs the appropriate types.
This fixes [Bugzilla #40060](https://bugzilla.xamarin.com/show_bug.cgi?id=40060)
1. Fix the resolver on the native side for types like `"System.Generic.Collections.Dictionary[System.Int32,MyType]"` where the generic type is from corlib and a type argument is from the current assembly.
2. Fix `get_caller_no_reflection` to work correctly when the call stack has a System.Reflection method as the immediate caller of the icall.
3. Fix the resolver on the native side to work when it's called from corlib's System namespace, not just System.Reflection.
4. Fix the managed parser to work with types like `"MyGeneric[MyType]"` where the generic argument is not an assembly qualified name or there's a mixture of aqn and non-aqn type arguments. Also rationalize how the code handles assembly qualification in general.
5. Test cases
We would previously constantly try to dequeue from the writer queue which would
result in extremely high CPU usage, making many apps (e.g. Xamarin Studio)
nearly unusable.
1. Assert in mono_get_corlib_version instead of raising an exn.
2. Don't raise in mono_domain_create_appdomain external API function.
3. Assert in mono_domain_assembly_postload_search instead of raising.
4. Don't raise in mono_domain_fire_assembly_load.
5. Use mono_error_set_pending_exception in
ves_icall_System_AppDomain_InternalGetProcessGuid
6. Dont' raise in unload_thread_main, populate unload_data instead.
.. attribute for CurrentSolutionConfigurationContents .
<CurrentSolutionConfigurationContents>
<SolutionConfiguration xmlns="">
<ProjectConfiguration Project="{82323650-6C2A-4FAC-91C6-A715C2FE9CB2}" AbsolutePath="/Users/ankit/test/msbuild/bug40094/App1/App1\App1\App1.csproj">
Release|AnyCPU
</ProjectConfiguration>
...
Related to fix for bxc #40094
Previously for types like "Foo[P,Q]" the recursive call of the parser
would incorrectly consume the delimiter after a type argument (ie the
',' or ']') which would leave the caller potentially past the end of the
string which would signal a failed parse. New behavior is to peek at
the delimiter but leave it up to the caller to consume.
Also `Type.GetType(name, null, null)` should behave like
`Type.GetType(name)` (previously it dropped generic type arguments).
type_from_parsed_name is a helper for
ves_icall_System_Type_internal_from_name which is called directly from
System.Type as well as indirectly by System.TypeNameParse. We should
skip over all those classes to find the managed caller.
This fixes the remainder of [40060](https://bugzilla.xamarin.com/show_bug.cgi?id=40060)
In order to find the user code that called some corlib API
that calls an icall, we write:
dest = m = mono_method_get_last_managed ();
mono_stack_walk_no_il (get_caller_no_reflection, &dest);
if (!dest) dest = m;
The get_caller_no_reflection walk works in two stages: in the first
stage we look for the target method (the initial value of dest) when we
find the target we set dest to NULL and then the second stage continues
walking upward past the reflection APIs.
However if the target method is itself in System.Reflection, the
old implementation would skip over it even in the first stage and we'd
never find the target method and exhaust the whole stack.
The fix is to move the check for the target method before the check for
the Reflection API methods.
And in type_from_parsed_name pass the calling image as the metadata
context even if the initial class we're looking for is in corlib.
This lets calls like
Type.GetType ("System.Collections.Generic.Dictionary<int, MyClass<int>>")
succeed. (Previously we would firs try to find
System.Collections.Generic.Dictionary in the calling assembly,
which would fail outside of corlib, and then we'd retry with the
rootimage as corlib which would find Dictionary, but fail to find
MyClass. Note that this was only broken if the outermost type was in
corlib. It always worked deeper in generic type arguments provided
we startd in the calling assmebly:
Type.GetType ("MyClass<System.Collections.Generic.Dictionary<int, MyClass<int>>>")
This fixes part of [40060](https://bugzilla.xamarin.com/show_bug.cgi?id=40060)
According to http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0068b/CIHEEIDJ.html the MRC instruction is documented as:
MRC{cond} coproc, opcode1, Rd, CRn, CRm{, opcode2}
with coproc being "the name of the coprocessor the instruction isfor. The standard name is pn, where n is an integer in the range 0-15."
GCC seemed to accept the operand without the 'p' prefix but clang raised an "invalid operand for instruction" error.
In some platforms, C enums might be considered unsigned, and in that case we'd get the following warning:
mono/mono/metadata/marshal.c:5108:12: warning: comparison of constant -1 with expression of type 'MonoMarshalConv' is always false
[-Wtautological-constant-out-of-range-compare]
if (conv == -1) {
~~~~ ^ ~~
The explicit cast should do the right thing.
Log buffers can end up with a zero thread ID when we synthesize events on the
writer thread, leading to mprof-report thinking a thread exists with that ID.
We simply ignore the "zero thread" in the thread summary.
If we don't do this, mprof-report will pick up on the fact that the helper and
writer threads exist and print them in the thread summary with no names. It's
OK to zero out the thread ID as the kinds of events we write from these threads
cannot be meaningfully associated with the log buffer's thread ID (or in the
case of counters, any thread ID).
We only set the native name as MS.NET leaves the managed name for the main
thread as null. At least by setting the native name, we get a nice name in GDB.
The profiler helper thread does a full attach for on-demand heapshots. Since we
collect no actual data for the helper thread, there's no reason to report it.
A double init could happen when profiling as we'd call mono_gc_base_init ()
from mono_profiler_load () early on, and then later call it again during the
normal startup sequence.
Previously, we would shut down the profiler so early that it would never see
unload events for threads, domains, etc. This sounds harmless at first, but
since the profiler dumps the remaining data in the thread-local log buffer to
disk from within the thread end callback, we would often miss events near the
end of a thread's lifetime.
Shutting down the profiler after the major runtime components doesn't appear to
have any negative effects, which makes sense as all managed threads should be
stopped by this point.
Strong Togglerefs must be considered as roots for the purpose of Ephemeron processing.
Otherwise you make it not possible to use a toggleref object as key.
Fix uninitialized MonoError in mono_basic_block_split
Follow-up on PR #2781 which removed a mono_error_init leading to memory unsafety. This was causing crashes when verifying assemblies in the 4.4 branch although it does not seem to have caused any known problems in master for some reason.
The recent changes to the mobile profiles introduced generated files in their .sources like CommonCrypto/MD5CryptoServiceProvider.g.cs
However, those are not generated when the mobile profiles aren't enabled at configure time.
This caused a problem during "make dist" since it tries to grab the entries in .sources from *all* profiles, causing errors like:
cp: cannot stat ‘CommonCrypto/MD5CryptoServiceProvider.g.cs’: No such file or directory
make[5]: *** [dist-local] Error 1
The easiest fix is to filter out those generated files.
Any native function that can modify errno should have the
SetLastError attribute set to true. Also, no need to specify
EntryPoint if it's the same as the function name.
[corlib] Fix exception in __ComObject finalizer during shutdown
* keep a reference to a __ComObject's proxy so it doesn't get garbage
collected
* don't create __ComObjects unnecessarily
* Fix for bug [24974](https://bugzilla.xamarin.com/show_bug.cgi?id=24974)
We hit this on Jenkins over the weekend:
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/3845/parsed_console/log_content.html#WARNING1
```
Unhandled Exception:
System.InvalidOperationException: Cannot process request because the process (51792) has exited.
at System.Diagnostics.Process.GetProcessHandle (Int32 access, Boolean throwIfExited) <0x10d7ea000 + 0x001b6> in <filename unknown>:0
at System.Diagnostics.Process.GetProcessHandle (Int32 access) <0x10d7f2fd0 + 0x0001e> in <filename unknown>:0
at System.Diagnostics.Process.Kill () <0x10d7f2ee0 + 0x00031> in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Kill ()
at TestRunner+<Main>c__AnonStorey0.<>m__0 () <0x10a9b50b0 + 0x00e30> in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x10c55bc90 + 0x0009a> in <filename unknown>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x10c55a210 + 0x001c6> in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x10c55a1e0 + 0x00020> in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x10c55a130 + 0x00059> in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart () <0x10c55be50 + 0x0002e> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Cannot process request because the process (51792) has exited.
at System.Diagnostics.Process.GetProcessHandle (Int32 access, Boolean throwIfExited) <0x10d7ea000 + 0x001b6> in <filename unknown>:0
at System.Diagnostics.Process.GetProcessHandle (Int32 access) <0x10d7f2fd0 + 0x0001e> in <filename unknown>:0
at System.Diagnostics.Process.Kill () <0x10d7f2ee0 + 0x00031> in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Kill ()
at TestRunner+<Main>c__AnonStorey0.<>m__0 () <0x10a9b50b0 + 0x00e30> in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x10c55bc90 + 0x0009a> in <filename unknown>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x10c55a210 + 0x001c6> in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x10c55a1e0 + 0x00020> in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x10c55a130 + 0x00059> in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart () <0x10c55be50 + 0x0002e> in <filename unknown>:0
make[1]: *** [test-generic-sharing-managed] Error 1
```
We can't do anything about those errors, so just catch and ignore it.
[Profiler] Keep a reference to assemblies while coverage is running
In certain circumstances, for example when running under nunit-console, assemblies could be unloaded while the profiler still had a reference to them in order to generate the coverage data. In these cases, the memory referred to by the profiler would be invalid and would crash.
Fix this by incrementing the reference count of the assemblies preventing them from being unloaded while coverage is being collected.
Fixes BXC #39325
- Back out MONO_PROFILER_LIB_DIR change
- Profiler loading traces should use log mask “dll” not “asm”
- mono_dl_open_runtime_lib should preempt standard paths when loading
profiler
- mono_dl_open_runtime_lib should check in profiler library directory
- mono_dl_open_runtime_lib contained a memory leak
- With above changes, ptestrunner.pl can remove some environment
massaging
Fixes#4242, see bug comments. JsonSerializer seems to have been already
fixed in abe9dc62. The JsonDeserializer number conversion is now much
stricter, allowing only integer number style for integers and float number
style for floats.
[profiler] Use a background thread to send out sampling signals.
Previously, when using an interval timer, the initial profiling signal could be
delivered to *any* thread in the process. This is not normally a problem, but
if the signaled thread has not even finished its initialization inside libc, it
can happen that it hasn't even set up thread-local storage yet. We then blow up
spectacularly when trying to back up errno in the SIGPROF signal handler, as
it's a TLS variable. Even the SIGSEGV handler blows up immediately after as it
can't access JIT TLS data.
Since there appears to be no reliable and portable way for a library like Mono
to control which exact thread gets the initial timer signal, instead switch to
using a background thread that uses a high-resolution sleep and attempts to
switch itself to real time scheduling if possible. This way, we have full
control over which threads we send SIGPROF to, letting us avoid any threads
which aren't in a 'good' state for profiling.
This commit also gets rid of the multiplexing that was going on in the SIGPROF
signal handler, since we now send out the signals from a background thread. As
this was the only use of the async job API, that API has been removed. This
indirectly fixes the signal storming issue that sometimes popped up, where for
some reason multiple threads would think that they're the initiating thread,
resulting in way too many signals going out.
A nice side-effect of doing the signaling ourselves is that we can now use
real-time signals on systems that have them (e.g. Linux), resulting in a nearly
100% signal delivery rate in all cases. Previously, we would lose a tremendous
amount of signals when an application was under heavy load.
Previously, when using an interval timer, the initial profiling signal could be
delivered to *any* thread in the process. This is not normally a problem, but
if the signaled thread has not even finished its initialization inside libc, it
can happen that it hasn't even set up thread-local storage yet. We then blow up
spectacularly when trying to back up errno in the SIGPROF signal handler, as
it's a TLS variable. Even the SIGSEGV handler blows up immediately after as it
can't access JIT TLS data.
Since there appears to be no reliable and portable way for a library like Mono
to control which exact thread gets the initial timer signal, instead switch to
using a background thread that uses a high-resolution sleep and attempts to
switch itself to real time scheduling if possible. This way, we have full
control over which threads we send SIGPROF to, letting us avoid any threads
which aren't in a 'good' state for profiling.
This commit also gets rid of the multiplexing that was going on in the SIGPROF
signal handler, since we now send out the signals from a background thread. As
this was the only use of the async job API, that API has been removed. This
indirectly fixes the signal storming issue that sometimes popped up, where for
some reason multiple threads would think that they're the initiating thread,
resulting in way too many signals going out.
A nice side-effect of doing the signaling ourselves is that we can now use
real-time signals on systems that have them (e.g. Linux), resulting in a nearly
100% signal delivery rate in all cases. Previously, we would lose a tremendous
amount of signals when an application was under heavy load.
Various cleanups on profiler
3 commits:
Assist on profiler module loading
- Add a MONO_PROFILER_LIB_DIR from which profiler modules are
dynamically loading, in case it is not the normal library path
- Add information about profiler loading to mono_trace
Comments and clarity in proflog.c
- Add comments on structures
- Rename 'data' and 'data_end' to 'cursor' and 'buf_end' in LogBuffer/StatsBuffer as the old names were confusing
- Add new assert when processing samples
Improve a safety check when writing data into StatBuffer
- The safety check should occur such that if the new value for
StatBuffer::cursor is beyond the bounds of the StatBuffer, the cursor
is not updated.
Fix hazard free
Replace usage of mono_thread_hazardous_free_or_queue and remove it.
mono_thread_hazardous_free_or_queue has a fundamentally broken design.
It allows arbitrary free code to run in the context of its caller.
The sync/async split on where it's called is not enough to know whether
we can run free'ing code or not.
Since in sync context we called free functions that could take locks,
it happened that those locks conflicted with the ones already taken.
In particular, mono_jit_info_table_free did take a domain lock and
quite a few of the sync context calls would hold locks that must
not be held when taking a domain lock.
This, is practice, means that we'd need to partition the free calls
into 3 groups: async context, reentrant context (no runtime locks held)
and sync context (maybe some locks held).
There was no case where reentrant context would be usable meaning,
in practice, that all calls happens in what effectively is async context
where free functions can't be called.
The new design is a lot more straightforward:
- No implicit free queue pumping
- A pair of free functions with well defined behavior.
mono_thread_hazardous_try_free - to be used where the caller expects the free function to be called
mono_thread_hazardous_queue_free - to be used where the caller don't expect the free function to be called [1]
- Explicit pumping on places that known to be ok
Thread detach
Finalizer thread
[1] This might sound like a weird condition, but a lot of lock-free code
have compensation logic that trigger free'ing during lookups.
We add a callback to the HP so the GC can monitor the queue size and act on it.
The queue size was picked to keep the overhead limited even under high load.
- Add comments on structures
- Rename 'data' and 'data_end' to 'cursor' and 'buf_end' in
LogBuffer/StatsBuffer as the old names were confusing
- Add new assert when processing samples
- Add a MONO_PROFILER_LIB_DIR from which custom profilers are
dynamically loaded, in case it is not the normal library path
- Modify mono/profiler make check to use MONO_PROFILER_LIB_DIR
(this fixes a bug where it would sometimes load the wrong profiler)
- Add information about profiler loading to mono_trace
We used to log the major heap size before the sweep job finished. Remove the major heap logging at collection end time and add a separate log entry at sweep finish time.
mono_thread_hazardous_free_or_queue has a fundamentally broken design.
It allows arbitrary free code to run in the context of its caller.
The sync/async split on where it's called is not enough to know whether
we can run free'ing code or not.
Since in sync context we called free functions that could take locks,
it happened that those locks conflicted with the ones already taken.
In particular, mono_jit_info_table_free did take a domain lock and
quite a few of the sync context calls would hold locks that must
not be held when taking a domain lock.
This, is practice, means that we'd need to partition the free calls
into 3 groups: async context, reentrant context (no runtime locks held)
and sync context (maybe some locks held).
There was no case where reentrant context would be usable meaning,
in practice, that all calls happens in what effectively is async context
where free functions can't be called.
The new design is a lot more straightforward:
- No implicit free queue pumping
- A pair of free functions with well defined behavior.
mono_thread_hazardous_try_free - to be used where the caller expects the free function to be called
mono_thread_hazardous_queue_free - to be used where the caller don't expect the free function to be called [1]
- Explicit pumping on places that known to be ok
Thread detach
Finalizer thread
[1] This might sound like a weird condition, but a lot of lock-free code
have compensation logic that trigger free'ing during lookups.
[sgen] Clean up thread pool on shutdown.
Keeping threads around can cause processes to keep running when they shouldn't, particularly on Windows when the entry point returns.
(The reason this is important for me is that Wine provides _CorExeMain, which is an entry point that does return, unlike normal compiled C programs which appear to call exit() after main() returns. A Windows process doesn't exit until ALL threads have exited, so a thread that never exits means a hung process.)
It doesn't seem like we have other GC-specific cleanup code, so I don't know if this is a good approach. Feedback on that question would be appreciated.
Those two functions are intended to replace mono_thread_hazardous_free_or_queue.
The problem with that function is that it doesn't compose well given it forces
the whole call chain leader to a free call to follow its semantics.
Additionally, it's not enough safety to be in sync context to call the free functions
as they introduce reentrancy - they can be called from any other arbitrary free
call that doesn't claim to be in async context.
Given most data structures, in general, can't guarantee how they are used, they
must make the conservative assumption of being in async context, thus not allowing
any form of locking happening.
Make AOT compilation on 64 bit windows produce executables
This PR contains three commits (see commit descriptions) which:
- Fix 64-bit debug windows builds in MSVC 2015
- Make it possible to use the tool-prefix aot option with mono/mini make fullaotcheck (necessary if you are in cygwin and wish to invoke `x86_64-w64-mingw32-gcc`)
- Change mini so that `--aot=full` successfully emits 64-bit windows executables.
Building 32 bit executables will generally fail because we do not follow cdecl. The 64 bit executables do not run properly when executed.
[runtime] MonoError-ize mono_loookup_dynamic_token{,_class}
Push `MonoError` through `mono_lookup_dynamic_token_class` and `mono_lookup_dynamic_token`
Also simplify the implementations of those functions so that they call `mono_reflection_lookup_dynamic_token` directly, rather than via a function pointer. The old implementation was a holdover from when the runtime was implemented as separate libraries.
Android uses dlmalloc within bionic. For some reason, our x86 build
links against the libc implementation instead of our own.
This leads us allocating memory that is not executable which made
dynamic methods fail when executing.
List of original contributors:
Rodrigo Kumpera <[email protected]>
On some junk Samsung x86 device, pthread_attr_getstack is completely broken.
If the returned value is not sane, we parse /proc/self/maps to find the right
memory regions for the current stack.
Fixes BXC #21550 and DESK #83333.
List of original contributors:
Rodrigo Kumpera <[email protected]>
1) Free() throws on MS.NET when the handle was already freed, copied and adapted the referencesource implementation.
2) Explicit conversion from IntPtr to GCHandle throws InvalidOperationException when value is IntPtr.Zero on MS.NET, not ArgumentException.
This fixes two failing CoreCLR test cases.
With the concurrent collector, we used to reset cementing at the start of the finishing pause (we can uncement since we process all the remsets when the world is stopped, through the mod union entries). This processing means cardtable scan in the finishing pause. In order to reduce pause time and overhead, we extend cementing for the finishing pause. Now we don't uncement objects (during sgen_cement_reset) that are still pinned in the roots at the start of the concurrent collection. We won't mark mod_union entries for these objects and they will survive the collection in the nursery. They can become uncemented at the start of the next major collection, given that they are no longer pinned in the roots.
Call mono_reflection_lookup_dynamic_token directly from
mono_lookup_dynamic_token.
Delete mono_install_lookup_dynamic_token () and the
MonoLookupDynamicToken typedef.
[runtime] MonoError-ize mono_object_isinst{,_mbyref}
Also mark them external only. Runtime should use mono_object_isinst_mbyref_checked and mono_object_isinst_checked
We are quite frequently seeing an error on the ARM workers on Jenkins:
```
Test Case Failures:
1) MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile : System.Exception : Could not abort registered blocking threads before closing socket.
at System.Net.Sockets.SafeSocketHandle.ReleaseHandle () [0x000be] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System/System.Net.Sockets/SafeSocketHandle.cs:77
at System.Runtime.InteropServices.SafeHandle.DangerousReleaseInternal (Boolean dispose) <0x42cb0050 + 0x0020b> in <filename unknown>:0
at System.Runtime.InteropServices.SafeHandle.InternalDispose () <0x446ba758 + 0x00023> in <filename unknown>:0
at System.Runtime.InteropServices.SafeHandle.Dispose (Boolean disposing) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/runtime/remoting/callcontext.cs:804
at System.Runtime.InteropServices.SafeHandle.Dispose () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/runtime/remoting/callcontext.cs:795
at System.Net.Sockets.Socket.Dispose (Boolean disposing) [0x00047] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System/System.Net.Sockets/Socket.cs:3321
at System.Net.Sockets.Socket.Dispose () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System/System.Net.Sockets/Socket.cs:3327
at System.Net.Sockets.Socket.Close () [0x00007] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System/System.Net.Sockets/Socket.cs:3256
at MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile () [0x000e7] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System/Test/System.Net.Sockets/SocketTest.cs:4331
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0003d] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/corlib/System.Reflection/StrongNameKeyPair.cs:103
```
Let's see if implementing the suggestion from https://github.com/mono/mono/pull/1612/files#r26393224
fixes this. I couldn't reproduce the test failure even on the same ARM machine so this is a bit of
an opportunistic change, but it feels like the more "correct" solution anyway.
I decided to inline AbortRegisteredThreads as at the only place we're calling it we already know
blocking_threads can't be null and acquired the lock.
[threadpool-ms] MonoError-ize
1. Convert icalls that throw `NotImplementedException` to do it via mono_error_set_pending_exception
2. Add MonoError outargs to `mono_threadpool_ms_enqueue_work_item` and `mono_threadpool_ms_begin_invoke`
Return TRUE if pending exception was set, FALSE otherwise.
The idea is that you can write:
if (mono_error_set_pending_exception (error)) {
/* cleanup */
return;
}
Krait cpus don't ignore thumb ITT state in ARM mode (it's unspecified CPU behavior).
The linux kernel then didn't bother resetting that state when landing an ARM signal handler.
The end result was that Krait cpus would skip the first few instructions of the signal handler
and things would explode in the most amazing possible ways.
Patch by Mark Probst <[email protected]>.
Don't raise deep in the marshalling code, just assert.
(Raising means we'd have to convert the code to use MonoError and then
thread it out through all of the marshalling code resulting in a lot of
code churn)
When the concurrent mark ends, we have another concurrent phase where we scan the cardtable trying to clean as many cards as possible. The way we achieve this is by first copying all the mod union cards (for each block/los_obj) to a separate temporary card_table. When scanning we will only scan these cards. Therefore we will first preclean these cards in the actual mod_union card_tables, followed by a full memory barrier (which assures us that, when scanning, we will observe the changes that led to the corresponding card marks) and then by a normal mod_union scan, which can remark some of these cards, without any races with the preclean phase.
This way we avoid duplicating cardtable marking code, making sure we have consistent code.
For example, when scanning los objects in the mod union scan step, we didn't check whether the object is cemented or not, leading to additional unnecessary global remsets for the next nursery collection.
We add a new data structure which contains a bucket of entries, exponentially increasing in sizes. The structure is customizable through a few callbacks.
Setting is_slot_set_func enables us to also lookup empty slots, instead of only adding entries at the end of the list.
Setting set_slot_func enables the use of a customized function for setting the slot. If this function is not set, the slots and array fields will be set directly, therefore the structure won't be thread safe.
When doing the mod union scan, we use the block's cardtable (instead of the global one), which we also cannot prefetch, since that would mean dereferencing the block info.
Runtime should use g_assert and g_assert_not_reached or set a MonoError,
as needed.
Also remove REAL_PRINT_REG dead macros from mini-x86.c and mini-amd64.c
- COFF does not use .local or .size
- COFF does use .def to mark functions and .data to mark data
- The seh_handler functions should not be called on full aot builds (Zoltan says they are unnecessary in full aot, and they attempt to jit)
The executables do not yet run without crashing.
* Support MONO_FULLAOT_ADDITIONAL_ARGS env var for make fullaotcheck, appended to --aot parameters
* To support above, permit "blank" parameters (ie trailing commas on parameter list) to --aot=
* --aot tool-prefix parameter now supported for LD as well as AS
Add new internal 'X509Certificate2Impl' and 'X509ChainImpl' classes similar to
'X509CertificateImpl' in corlib and only initialize them when actually needed.
[SRE] MonoError-ize some icalls
* Cleanup some memory that would leak on failure; improve a few error messages
* Add MonoError to `mono_reflection_call_is_assignable_to` and `fieldbuilder_to_mono_class_field`.
* Update several SRE icalls to follow the pattern:
```c
void
the_icall (args... )
{
MonoError error;
(void) the_real_work_function (args..., &error);
mono_error_set_pending_exception (&error);
}
static gboolean
the_real_work_function (args..., MonoError *error)
{
... /* return TRUE on success, FALSE and set error on failure */ ...
}
```
Do this to:
1. `mono_reflection_register_with_runtime`
2. `mono_reflection_setup_internal_class`
3. `mono_reflection_create_internal_class`
4. `mono_reflection_generic_class_initialize`
5. `mono_reflection_event_builder_get_event_info`
6. `mono_reflection_initialize_generic_parameter`
7. `mono_reflection_create_dynamic_method`
It should not prevent the process from terminating. Fixes
issue where the thread is not shut down if ProjectInstances
are built directly instead of via the build manager.
Make `make fullaotcheck` in mono/mini runnable on Windows
Run with
MONO_EXECUTABLE=/path/to/msvc/mono.exe LIST_SEPARATOR=";"
This assumes you are running the test harness in Cygwin and you built mono with MSVC. Also, the test does not currently succeed.
Being able to run the test is a zeroth step toward making the test pass.
It was the only test in the class that required network access (for looking up xamarin.com in the redirect).
Replacing the redirect target with a second local HttpListener should be more resilient to network/DNS issues.
Changes to "checked build" feature
Here are some changes to checked builds I made while trying to debug some coop gc problems. The changes made are:
- Replace the checked build configure flags with a single --enable-checked-build= flag
- Runtime environment variable required for checked build asserts to run
- Report the thread transition history for more kinds of asserts
- For the `mono_threads_reset_blocking` functions, add an assert that guarantees proper nesting
In addition to this, I have documented the checked build feature as implemented here, see my pull request for the website: https://github.com/mono/website/pull/187
Overall my goals with this patch were
- Make it easier to do rapid-fire testing with the checked builds (runtime switch, default-off)
- Make the checked build implementation more internally consistent (more uniform names etc)
Notice I do make the change that instead of checked build sub-features being disabled with a blacklist, you now must enable them with a whitelist. I think this is an improvement because it makes it easier to just casually keep a checked build around (for one thing I would have the option to ship debug builds to users with the checked build modes included but dormant) but I could be talked out of it if this change turns out to be controversial. The only thing I feel strongly about is that if the configure flag is changed back to a blacklist the environment variable should switch to a blacklist also.
If we accept this change the CI bots will need to adopt the new configure flag and environment variable.
[System.Reflection.Emit] MonoError in mono_reflection_create_runtime_class
Depends on #2745
Bubble MonoError up to `mono_reflection_create_runtime_class`. Use `mono_error_set_pending_exception` instead of `mono_error_raise_exception` And try to release resources on the way out.
1. Don't raise while holding locks.
2. Change remove_instantiations_of_and_ensure_contents (called via
g_hash_table_foreach_remove) to populate a MonoError instead of
throwing. If there are multiple errors, we just stash the first one
in user_data and clean up the rest.
3. Use mono_erro_set_pending_exception instead of
mono_error_raise_exception since it's an icall.
[System.Reflection.Emit] Propagate MonoError through some more static functions
Some externally visible changes include:
* Bubble MonoError up to one more SRE icall: `System.Reflection.Emit.CustomAttributeBuilder.GetBlob()`
* `fix_partial_generic_class` and `ensure_generic_class_runtime_vtable` now expect their callers to set the class failure state which they do under slightly more circumstances now than previously (now: whenever there's an error set from the above functions; before: only when we had trouble initializing the generic type definition.)
* Previous API for `mono_reflection_bind_generic_parameters` didn't make it clear whether NULL was a valid return value or not. Now when we return NULL we always set the `error` out-arg. So non-NULL is success and NULL is failure.
The runtime should use mono_reflection_get_custom_attrs_blob_checked.
The icall is now ves_icall_System_Reflection_Emit_CustomAttributeBuilder_GetBlob.
This commit also changes the behavior of
mono_reflection_bind_generic_parameters slightly: if the passed in class
isn't generic, it sets error and returns NULL. (It wasn't clearly
previously whether NULL was an error condition or not - with this commit
we clarify that a NULL result is an error).
Because mono_msec_ticks would previously return the number of milliseconds since boot time, it would overflow after ~49 days. That would mean that anywhere you would use it, you would need to be careful of arithmetic overflow.
By simply returning a gint64 we will not overflow before ~30k years.
Fixes bug https://bugzilla.xamarin.com/show_bug.cgi?id=38666
Due to our usage of chained commands it'd use the exit code of the last command,
which is not what we want. We use chaining because e.g. in the ms-test-suite
case we still want to run the systemruntimebringup suite even if the conformance
suite had a failing test.
It's not supposed to contain a full path to a directory, but rather a relative path
to a subdirectory of the ApplicationBase. However, this isn't enforced in all cases
by Mono so setting a full path worked when the path doesn't contain a '.' because then
the code in [1] doesn't clear the variable and the path is added to the domain->search_path.
This caused the test to fail on Wrench on the mono-4.4.0 lane because the path contains
a dot while confusingly it worked fine on master.
Fixing this by appending the xbuild_14 path via an alternative (albeit slightly hacky)
method related to the standard MONO_PATH env var. Because the variable is only read
once at app startup we need to call into the runtime to explicitly set the new path
before trying to load the MSBuild 14 assemblies. This is fine since we only
want to test that the correct assembly version is loaded and nothing else.
[1] https://github.com/mono/mono/blob/37b2b9fbc25a2199aba1d794117924d4828360a7/mono/metadata/appdomain.c#L1319-L1337
It's flaky on Jenkins, removing the dependency on an external server and instead using a local HttpListener
to answer the requests should hopefully make it more reliable.
Moved StackTrace.GetFullNameForStackTrace into a new class
StackTraceHelper so the code can be reused by mono-symbolicate.
Reusing GetFullNameForStackTrace in mono-symbolicate allows the tool to
easily match a stack trace line with a method without doing a rather
complicate parsing of types and possible generic parameters
combinations.
This was inadvertently removed in https://github.com/mono/mono/commit/611a43ee0f672adbac3e25dc77731843a3d10cf1 from MulticastDelegate.CombineImpl().
Add the type check back in the Delegate base class, we already do the same in Remove().
Add unit tests to cover the behavior to the corlib test suite. (reverted from commit f29878dc713f4e148fe0a1e703fae8929e6fb6e0)
Needed to update some monodoc tests to match that an exception is now thrown again.
fixed replacement of $mono_libdir in config file
strncpy does not NUL-terminate the destination string in this case so one cannot use strcat to append
This fixes the error "malloc: top chunk is corrupt" as reported on mono-devel-list back in 2015-10-20 (subject: "malloc error executing OBS-built mono") at least for me.
This removed the last source of loader error from the runtime
The last source was on the AOT compiler in load_image.
I hooked enough MonoError to bubble the issue to a reasonably higher layer.
@vargaz please review.
This was an incorrect test. Tasks can't really be used like this
especially this once since it tries to Log an error, but it hasn't been
assigned a BuildEngine yet!
Other fixes in TaskLoggingHelper expose this issue.
Fix msbuild remapping
First commit:
```
Revert "Do not remap framework assembly if it's version is higher than the runtime version"
It would cause issues on mobile, cause the runtime version is 2.0.5.0 there and loading
e.g. System.dll v4.0 would no longer be mapped to that version anymore after this change.
```
---
Second commit:
```
[runtime] Stop remapping higher versions of
Microsoft.Build.Framework/Engine and remap Microsoft.Build.Utilities/Tasks
(.v3.5)
The previous remapping was added in 725ed3f824, but it didn't actually
fix the bug in https://bugzilla.xamarin.com/show_bug.cgi?id=32561.
The underlying cause of the bug is that in Mono 4.0 we removed the 2.0/3.5 profiles.
This meant that Microsoft.Build.Utilities.dll and Microsoft.Build.Utilities.v3.5.dll weren't installed
in the GAC anymore, because MSBuild 4.0 uses Microsoft.Build.Utilities.v4.0.dll, so the app failed
to load the 2.0 version of the assembly. The same applies for the .Tasks variants.
We remap earlier versions of the Microsoft.Build.Utilities/Tasks assemblies to the v4.0 version.
I've introduced another item in the AssemblyVersionMap struct which allows remapping to target a different
assembly name, because MSBuild made the questionable decision to put the version number in the assembly name.
While strictly speaking this is not what .NET is doing (it just fails there if .NET 2.0 isn't installed)
on Mono there's no way to install those older profiles anymore, so we remap them to v4.0 to keep apps working.
Newer versions of Microsoft.Build.Framework/Engine (like 12.0, 14.0) however have the same name, so we need to be
more careful here and only do a remapping to the 4.0 version if it's an earlier version (e.g. 2.0 -> 4.0).
This avoids issues with accidentally remapping 14.0 -> 4.0.
I tested the app from the bug report and it now works fine.
```
The previous remapping was added in 725ed3f824, but it didn't actually
fix the bug in https://bugzilla.xamarin.com/show_bug.cgi?id=32561.
The underlying cause of the bug is that in Mono 4.0 we removed the 2.0/3.5 profiles.
This meant that Microsoft.Build.Utilities.dll and Microsoft.Build.Utilities.v3.5.dll weren't installed
in the GAC anymore, because MSBuild 4.0 uses Microsoft.Build.Utilities.v4.0.dll, so the app failed
to load the 2.0 version of the assembly. The same applies for the .Tasks variants.
We remap earlier versions of the Microsoft.Build.Utilities/Tasks assemblies to the v4.0 version.
I've introduced another item in the AssemblyVersionMap struct which allows remapping to target a different
assembly name, because MSBuild made the questionable decision to put the version number in the assembly name.
While strictly speaking this is not what .NET is doing (it just fails there if .NET 2.0 isn't installed)
on Mono there's no way to install those older profiles anymore, so we remap them to v4.0 to keep apps working.
Newer versions of Microsoft.Build.Framework/Engine (like 12.0, 14.0) however have the same name, so we need to be
more careful here and only do a remapping to the 4.0 version if it's an earlier version (e.g. 2.0 -> 4.0).
This avoids issues with accidentally remapping 14.0 -> 4.0.
I tested the app from the bug report and it now works fine.
Saw a few failures on Jenkins on the slower ARM machines over the last days:
```
) MonoTests.System.Collections.Concurrent.BlockingCollectionTests.AddAnyCancellable : System.OperationCanceledException : The operation was canceled.
at System.Collections.Concurrent.BlockingCollection`1[T].TryAddToAnyCore (System.Collections.Concurrent.BlockingCollection`1[] collections, System.Collections.Concurrent.T item, Int32 millisecondsTimeout, CancellationToken externalCancellationToken) [0x00092] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/external/referencesource/System/sys/system/collections/concurrent/BlockingCollection.cs:1041
at System.Collections.Concurrent.BlockingCollection`1[T].TryAddToAny (System.Collections.Concurrent.BlockingCollection`1[] collections, System.Collections.Concurrent.T item, Int32 millisecondsTimeout, CancellationToken cancellationToken) [0x00006] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/external/referencesource/System/sys/system/collections/concurrent/BlockingCollection.cs:971
at System.Collections.Concurrent.BlockingCollection`1[T].AddToAny (System.Collections.Concurrent.BlockingCollection`1[] collections, System.Collections.Concurrent.T item, CancellationToken cancellationToken) <0x4421bf18 + 0x0002b> in <filename unknown>:0
at MonoTests.System.Collections.Concurrent.BlockingCollectionTests.AddAnyCancellable () [0x00031] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/mcs/class/System/Test/System.Collections.Concurrent/BlockingCollectionTests.cs:276
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0003d] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/mcs/class/corlib/System.Reflection/StrongNameKeyPair.cs:103
```
100ms is likely just too short, bumping it.
[runtime] Fix deadlock while unwinding the stack of suspended thread
A suspended thread, in non-coop case, can be holding runtime locks. When the suspender tries to unwind the stack for this thread, it might require taking of runtime locks (ex. in aot case we might search for the jit info in the aot image and then register it, which requires the domain lock). Mark the suspender thread as running in async context while it unwinds other threads.
In addition to --enable-checked-build, it is now mandatory to put the
requested checked-build modes in the environment variable
MONO_CHECK_MODE at runtime.
Also add a MONO_CHECK_THREAD_TRANSITION_HISTORY environement variable
for thread transition history length (instead of hardcoding 3).
This reverts commit 100bd760a4.
It would cause issues on mobile, cause the runtime version is 2.0.5.0 there and loading e.g. System.dll v4.0 would
no longer be mapped to that version anymore after this change.
Testing for the remap message via grep isn't ideal, a better way to verify the assembly
version would be to check asm.GetName().Version, but unfortunately in the test environment
when MONO_PATH is set the assembly loading always fallsback to a simple file name matching
in the directory so we end up with the wrong (v4.0) assembly in that case.
This was revealed by the CoreCLR tests/src/CoreMangLib/cti/system/convert/converttouint3219.exe test.
When we tried to convert a number larger than UInt32.MaxValue (in this case 4933891728) we would cast
the intermediate result to uint which overflows and makes the end result smaller which means we don't throw.
Storing the intermediate result in a long before checking for the correct range fixes this.
Added a few additional tests that I felt were missing and/or only present for other Convert.* methods.
I found two more cases that don't throw OverflowException but fixing it looked more involved so I punted
and just ignored them with a FIXME for now.
ARM platform tends to try very hard to save power by switching off CPUs. This could lead to inconsistencies in the value returned by mono_cpu_count
Recap from Alexander Köplinger <[email protected]>:
When we merged the change from PR #2722, we started seeing random failures on ARM in the MonoTests.System.Threading.ThreadPoolTests.SetAndGetMaxThreads and MonoTests.System.Threading.ManualResetEventSlimTests.Constructor_Defaults tests. Both of those tests are dealing with Environment.ProcessorCount to verify some implementation details.
It turns out that on the Jetson TK1 board we use on public Jenkins and on ARM kernels in general, the value returned by sched_getaffinity (or _SC_NPROCESSORS_ONLN) doesn't contain CPUs/cores that are powered off for power saving reasons. This is contrary to what happens on x86, where even cores in deep-sleep state are returned [1], [2]. This means that we would get a processor count of 1 at one point in time and a higher value when load increases later on as the system wakes CPUs.
Various runtime pieces like the threadpool and also user code however relies on the value returned by Environment.ProcessorCount e.g. for deciding how many parallel tasks to start, thereby limiting the performance when that code thinks we only have one CPU.
Talking to a few people, this was the reason why we changed to _SC_NPROCESSORS_CONF in https://github.com/mono/mono/pull/1688 and why we added a special case for Android in https://github.com/mono/mono/commit/de3addcde5bf875dcd43d3e2c6536d61871e7675 to get the "real" number of processors in the system.
Because of those issues Android/Dalvik also switched from _ONLN to _SC_NPROCESSORS_CONF for the Java API Runtime.availableProcessors() too [3], citing:
> Traditionally this returned the number currently online,
but many mobile devices are able to take unused cores offline to
save power, so releases newer than Android 4.2 (Jelly Bean) return the maximum number of
cores that could be made available if there were no power or heat
constraints.
The problem with sticking to _SC_NPROCESSORS_CONF however is that it breaks down in constrained environments like Docker or with an explicit CPU affinity set by the Linux `taskset` command, They'd get a higher CPU count than can be used, start more threads etc. which results in unnecessary context switches and overloaded systems. That's why we need to respect sched_getaffinity.
So while in an ideal world we would be able to rely on sched_getaffinity/_SC_NPROCESSORS_ONLN to return the number of theoretically available CPUs regardless of power saving measures everywhere, we can't do this on ARM.
I think the pragmatic solution is the following:
* use sched_getaffinity (+ fallback to _SC_NPROCESSORS_ONLN in case of error) on x86. This ensures we're inline with what OpenJDK [4] and CoreCLR [5] do
* use _SC_NPROCESSORS_CONF exclusively on ARM (I think we could eventually even get rid of the PLATFORM_ANDROID special case)
Thoughts?
Helpful links:
[1] https://sourceware.org/ml/libc-alpha/2013-07/msg00383.html
[2] https://lists.01.org/pipermail/powertop/2012-September/000433.html
[3] https://android.googlesource.com/platform/libcore/+/750dc634e56c58d1d04f6a138734ac2b772900b5%5E1..750dc634e56c58d1d04f6a138734ac2b772900b5/
[4] https://bugs.openjdk.java.net/browse/JDK-6515172
[5] https://github.com/dotnet/coreclr/blob/7058273693db2555f127ce16e6b0c5b40fb04867/src/pal/src/misc/sysinfo.cpp#L148
Do not remap framework assembly if it's version is higher than the ru…
…ntime version
Currently, assemblies like Microsoft.Build.Framework and
Microsoft.Build.Engine get remapped to the current runtime version, at
load time. This means that if an assembly references the above
assemblies with a version like `14.1.0.0`, then that would get remapped
to `4.0.0.0`. This is incorrect behavior and breaks msbuild. Instead,
now we do the remapping only if the requested version is lower than the
runtime version.
A suspended thread, in non-coop case, can be holding runtime locks. When the suspender tries to unwind the stack for this thread, it might require taking of runtime locks (ex. in aot case we might search for the jit info in the aot image and then register it, which requires the domain lock). Mark the suspender thread as running in async context while it unwinds other threads.
Currently, assemblies like Microsoft.Build.Framework and
Microsoft.Build.Engine get remapped to the current runtime version, at
load time. This means that if an assembly references the above
assemblies with a version like `14.1.0.0`, then that would get remapped
to `4.0.0.0`. This is incorrect behavior and breaks msbuild. Instead,
now we do the remapping only if the requested version is lower than the
runtime version.
@akoplinger: This will actually impact xbuild too right now because
we're defaulting to xbuild 14.0 in Mono 4.4 now so we'd see messages
like Mono: The request to load the assembly Microsoft.Build.Framework
v14.0.0.0 was remapped to v4.0.0.0 in the assembly loading log, which
will break in very subtle ways.
A few people told me it's surprising that acceptance-tests checks out repos into <mono repo>/../
and it can cause unwanted changes when a repo with the same name already exists there.
While this structure is used in several internal repos I agree for Mono it's better to keep
everything inside the repo itself.
Early ICALL_ADDR lookup on pinvoke to fix checked-build assert
The pinvoke implementation in the failure case was creating a managed
exception object while in blocking mode, a rules violation. This patch
puts a redundant lookup early so that any exceptions occur before
blocking mode is entered. (The "real" lookup is unchanged because the
current pattern is optimized well by the JIT.)
The pinvoke implementation in the failure case was creating a managed
exception object while in blocking mode, a rules violation. This patch
puts a redundant lookup early so that any exceptions occur before
blocking mode is entered. (The "real" lookup is unchanged because the
current pattern is optimized well by the JIT.)
- Add logfile and syslog capability. For Windows, the latter is simulated by writing to a file. Adding Eventlog support is for a future effort. The choice of logging mode is made via MONO_LOG_DEST environment variable.
- Add timestamps and pid to the trace messages to make them more useful
- Add timestamps to SGEN_LOG
- Related to tracing: when tracing and using something like Valgrind the system may be slowed so much that the sleep abort limit may be exceeded for threads. This fix adds an environment variable lookup of "MONO_SLEEP_ABORT_LIMIT" which is a value greater than 40, which defines the number of milliseconds to wait for threads to complete pending operations. If not set, the valus is 200.
[runtime] Set exit code on unhandled exception even if a managed handler is present
In case we would add a AppDomain.CurrentDomain.UnhandledException callback, we would not exit the VM in case of unhandled exception.
This goes against the expected behaviour, as we should indeed call these managed callbacks in all cases, but they should not stop the VM from exiting.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=39077
In case we would add a AppDomain.CurrentDomain.UnhandledException callback, we would not exit the VM in case of unhandled exception.
This goes against the expected behaviour, as we should indeed call these managed callbacks in all cases, but they should not stop the VM from exiting.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=39077
[runtime] MonoError cleanup in mono-security.c
Some easy MonoError cleanup in mono-security.c: use set_pending_exception, don't raise in helper functions, free memory on errors.
[runtime] Use MonoError in mono_runtime_object_init
Mark mono_runtime_object_init checked external only. Runtime should use mono_runtime_object_init_checked.
This is to guarantee that even in the case an exception is thrown between both calls, we switch back and forth correctly from RUNNING to BLOCKING state.
A connect flood will break the async loop and take down the entire HttpListener instance (will stop accepting connections).
To reproduce. Take a HttpListener run say 2 wrk test instances against it (1000 connections each works here). After the run the HttpListener has broken its async loop and will no longer accept any connections after. The reason for this is that it was not previously handling the case that AcceptAsync can return synchronously. In this case event will not be called with args.
Patch handles this case and the above issue is not reproducible with.
[System] Fix more hardcoded test ports
A test failed on Jenkins today with "address already in use".
Opportunistically replaced a few more hardcoded test ports to avoid failures in the future.
@monojenkins merge
A test failed on Jenkins today with "address already in use".
Opportunistically replaced a few more hardcoded test ports to avoid failures in the future.
We'd run into an error before:
```
In file included from ../../mono/utils/mono-context.h:234:0,
from ../../mono/utils/mono-stack-unwinding.h:10,
from ../../mono/utils/mono-threads.h:14,
from mono-mmap.c:36:
../../mono/arch/arm/arm-codegen.h:936:41: fatal error: mono/arch/arm/arm_dpimacros.h: No such file or directory
```
This was done for the other cases in 637ee7cbaa, but not when the monotouch
special case was added in cf754cdb63.
[reflection] MonoError for mono_reflection_type_get_handle
`mono_reflection_type_get_handle` is really a kind of normalization procedure that synthesizes real `MonoType`s from reflection's derived types such as `System.Reflection.Emit.ArrayType, ByRefType`, etc. So we call it all over the place in System.Reflection.Emit runtime code.
That would all be fine, except that users are allowed to subclass `System.Type`, so we have to call the `UnderlyingSystemType()` method to get a real type. Which means that an override can throw. Which in coop mode means we have to thread around a `MonoError`. So that's why this code is uglier.
But really this code is doing a lot of type inspection and high-level work that we should figure out how to migrate to managed.
In certain circumstances, for example when running under nunit-console, assemblies could be unloaded while the profiler still had a reference to them in order to generate the coverage data. In these cases, the memory referred to by the profiler would be invalid and would crash.
Fix this by incrementing the reference count of the assemblies preventing them from being unloaded while coverage is being collected.
Fixes BXC #39325
[utils] Get processor count from scheduler affinity if available
In case the user has set a scheduler CPU affinity, we should use it as the number of available CPU for the runtime. This is particularly important on docket which uses cgroups that set the container a cpu affinity, and by not using that, the runtime report the total number of CPU the machine has, and not the number of CPU the container has.
Fixes bug https://bugzilla.xamarin.com/show_bug.cgi?id=39279
Enable building of monotouch/xammac profiles without mono-extensions and expose mobile_static in configure.ac
Relies on https://github.com/xamarin/mono-extensions/pull/17
This enables us to compile the monotouch, xammac and mobile_static profiles on public Jenkins without mono-extensions.
[corlib] Fix three decimal tests to not rely on current culture for formatting
They'd fail on cultures where "." is not the decimal separator before.
I've also reenabled them on mobile cause the workaround for #27269 is no longer necessary since all bots are now on Mono 4.0+.
@monojenkins merge
We need this because when the runtime is built without --enable-minimal=com the runtime crashes
when trying to AOT because it tries to find the __ComObject (even though nothing in mobile_static
actually uses COM). With this shim we can AOT the mobile_static assemblies even with a "normal"
runtime.
They'd fail on cultures where "." is not the decimal separator before.
I've also reenabled them on mobile cause the workaround for #27269 is no longer necessary since all bots are now on Mono 4.0+.
In case the user has set a scheduler CPU affinity, we should use it as the number of available CPU for the runtime. This is particularly important on docket which uses cgroups that set the container a cpu affinity, and by not using that, the runtime report the total number of CPU the machine has, and not the number of CPU the container has.
Fixes bug https://bugzilla.xamarin.com/show_bug.cgi?id=39279
As it turns out, all these uses of FOREACH_THREAD_SAFE were inside regions
where the suspend lock is held. This effectively meant that no modifications
to the thread list could actually happen while the foreach was executing. Since
the guarantees provided by FOREACH_THREAD_SAFE are not needed in any of these
cases, switch to FOREACH_THREAD. The latter macro also happens to be much more
lightweight, so this should result in an overall (small) performance gain.
The caller of mono_lls_init () can now specify whether the free function can or
cannot lock, rather than the LLS assuming that it can't. The only reason this
limitation existed was because the LLS wasn't extended to allow specifying this
information in/after 56a6ae2bd0.
In addition, all LLS functions are now callable from both safe and async
context; the caller must specify the context as an additional argument. If a
free function is in use and it may lock, we'll simply enqueue it for execution
at a later point if an LLS function is invoked in async context.
1. The function suffered from the "too many bools" problem. Pass enum values
instead so that call sites are clearer.
2. Don't restrict the combination of arguments. This makes it easier to use
the function as part of data structures where the arguments aren't known
ahead of time (e.g. the LLS).
This fixes a bug that would manifest when running the profiler on a program
that rapidly spawns/joins threads.
The problem was this: Given T1 which is joining on T2, T2 which is returning
and removing itself from the thread list, and T3 which is currently in the
SIGPROF signal handler, it was possible for T3 to encounter a pointer to T2's
freed thread info structure in the thread list. This can happen because the
SIGPROF handler doesn't actually acquire the suspend lock (it can't) while
iterating the thread list.
The MONO_LLS_FOREACH_SAFE macro was supposed to protect against this case
specifically and, as a result, only provide a snapshot of the thread list.
However, it was missing some very important bits of the mono_lls_find ()
algorithm. The gory details can be found in the paper that the LLS is based on:
http://www.research.ibm.com/people/m/michael/spaa-2002.pdf
In particular, refer to the "Using Other Memory Management Methods" section
on page 6. It describes why lines E1-2 and E3-4 are necessary in order to
safely iterate the list in an implementation using hazard pointers (such as
ours). Basically, the iteration of the list has to be restarted if any sort of
modification to the list is detected during iteration. Otherwise, it's possible
to read and follow removed/freed memory, leading to all sorts of Bad Things™
happening.
Now, obviously we can't just restart a foreach loop without further thought;
that would cause all sorts of breakage due to the loop body being executed on
the same elements multiple times. As it happens, the LLS is an ordered data
structure: Elements are arranged in ascending order by key. We can exploit
this fact when iterating the list and restarting the loop. By simply storing
the last key we successfully processed, when restarting the loop, we can just
skip all keys that are not greater than the last key we successfully processed.
In addition to this, the MONO_LLS_FOREACH_SAFE macro now assists in freeing
removed elements that it encounters, just like mono_lls_find ().
The end result is that MONO_LLS_FOREACH_SAFE is now truly safe to use at any
point. It does have the potential to be ever so slightly slower, however, since
it does do significantly more work than it used to, and might even have to
restart the loop multiple times if lots of threads are modifying the thread
list at the same time. This doesn't seem to be a problem in practice, though.
Finally, to reduce the maintenance burden, the non-filtered versions of the
macros are now implemented in terms of the filtered versions, using a static
inline filter function that always returns true. When using the non-filtered
macros, the compiler will just eliminate the check entirely through its usual
optimization passes.
When generating varargs signatures for method defs, ilasm would discard the rest of the call conv. Things
like instance or explicit this would be discarded.
CoreClr allows storing a wider primitive byref into a narrower one. int32& -> int16&, for example.
This is a safe weakening as there's no risk of overflows, undefined behavior or unaligned access.
Ass oddly as it sounds, it's valid to emit the catch handler before the try block
and have it be the last sequence of instructions in the method body.
This patch checks for that condition before looking the BB in cil_offset_to_bb.
Even if we are interrupted we must throw the exception only after we have regained the lock. This prevents throwing an SynchronizationLockException in the finally clause of common scenarios like :
lock (obj) {
Monitor.Wait (obj);
}
The downside of this is that, if another thread doesn't give up the lock, we could fail to abort the thread. But this is what MS does.
A monitor enter internal method returns -1 for interrupted, 0 for timed out, 1 for acquired. When doing a monitor wait, we regain the lock using an infinite wait (which can't time out) and retrying in the case of interruption.
We don't register jit infos for tramps that don't have unwind info because we can't unwind from them. Explicitly set uw_info to NULL, instead of pointing to a zero length array in the aot image, to not confuse the runtime when registering the tramp.
Likely to fix#38267
In some cases, the null terminator in the SocketAddress may not be the last
character. This changes the algorithm to ignore the null terminator and
anything after it instead of assuming that the last character is the null
terminator.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=35004
Before when the variable wasn't set (= the default), the mobile_static profile would try to pass
the -O=gsharedvt flag even when gsharedvt is not enabled (e.g. when building without mono-extensions)
which causes AOTing to fail.
A better solution is to export the variable from configure.ac into the mcs build system depending
on whether gsharedvt is enabled at configure time.
The code in ManagedProtection.GetKey() instantiated a new RSACryptoServiceProvider singleton without proper
locking to avoid multiple threads running the initialization concurrently.
Since RSACryptoServiceProvider (or rather KeyPairPersistence underneath) uses an XML file with a name derived
from the parameters to store the data there could be a case where one thread writes while another tries to read
and gets garbage. This results in a CryptographicException later on.
Added locking to prevent this.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=38933
It failed on Jenkins today (https://jenkins.mono-project.com/job/test-mono-mainline/label=debian-amd64/3611/parsed_console/log_content.html#WARNING1)
and I could repro it after running the test in a loop.
While looking at the test, the "sum -= buf [0]" seemed suspicious to me. The final callback invocation after the last byte was read results
in buf [0] being 0 most of the time (because no bytes were read). However, due to the async nature of the test that final callback might happen
before another callback and buf then has a value, resulting in a wrong checksum if that value is substracted.
The fix is to remove the substraction. I have no idea why it was done in the first place, but it's been there since the test was added in 2003...
While at it, I've added another check for the number of bytes read and used proper Interlocked.* operations to ensure atomicity.
By using the <profile>_opt_<assembly>.sources file we can specify files
that are used when mono-extension is not enabled and add shims for the
classes or methods that are missing an implementation in that case.
This allows us to build the monotouch and xammac profiles on public Jenkins.
[threads] Fix mono_thread_info_sleep overflow
This overflow would happen of the machine was up for more than 29 days, leading to an overflow in the value returned by mono_100ms_ticks.
The simplest solution is simply to resort to mono_100ns_ticks which would only overflow after more than 29247 years. So even the platform implement this value as the time since 0000-00-00 00:00:00, we still have more than 27000 years if front of us. It's also highly unlikely that we would overflow the `end' value when adding `ms', as that would mean we are in the last 29 days of that 29247 years.
@monojenkins merge
[reflection] Get rid of loader error in reflection.c
Get rid of (direct and indirect) calls to `mono_loader_set_error_from_mono_error` in reflection.c:
- `mono_custom_attrs_from_index`
- `mono_custom_attrs_from_method`
- `mono_custom_attrs_from_class`
- `mono_custom_attrs_from_assembly`
- `mono_custom_attrs_from_module`
- `mono_custom_attrs_from_property`
- `mono_custom_attrs_from_event`
- `mono_custom_attrs_from_field`
- `mono_custom_attrs_from_param`
In all the functions, add a `_checked` variant that has a `MonoError*` out argument and for the MONO_API functions. The public API versions just clean up the error and return NULL. Loader error isn't touched anymore.
This overflow would happen of the machine was up for more than 29 days, leading to an overflow in the value returned by mono_100ms_ticks.
The simplest solution is simply to resort to mono_100ns_ticks which would only overflow after more than 29247 years. So even the platform implement this value as the time since 0000-00-00 00:00:00, we still have more than 27000 years if front of us. It's also highly unlikely that we would overflow the `end' value when adding `ms', as that would mean we are in the last 29 days of that 29247 years.
[msbuild] Getting build error "Error initializing task XmlPeek: Not registered task XmlPeek".
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=38638
Removes the uses of LogXXXFromResources from XmlPeek because
we do not use string resources in xbuild.
[runtime] Use MonoError for mono_class_vtable_full and mono_runtime_class_init_full
Switch from a boolean `raise_on_error` arg to a MonoError outarg and let
callers decide what to do with the error.
Also mark mono_runtime_class_init external only. Runtime should use
mono_runtime_class_init_full.
Switch from a boolean raise_on_error arg to a MonoError outarg and let
callers decide what to do with the error.
Also mark mono_runtime_class_init external only. Runtime should use
mono_runtime_class_init_full.
FileStream was writing to the wrong position When a value longer than
the FileStream buffer was written after a read.
This was fixed by doing a seek before doing the problematic write.
Fixes#11699
[SafeHandle] Avoid handle leakage in case of ThreadAbortException
If we get a ThreadAbortException while at SafeHandle.cs:124 (before this commit), we would not set release to true, even after incrementing the internal reference count of the SafeHandle. That could lead to potential leak, as the caller (following the common following pattern) would never call DangerousRelase, thus never calling PerformRelease. The finalizer or Dispose would not lead to the call of PerformRelease either, as the DangerousAddRef / DangerousRelase calls are unbalanced.
The common pattern for DangerousAddRef / DangerousRelease is the following:
bool release = false;
try {
safeHandle.DangerousAddRef (ref release);
/* do something with safeHandle.DangerousGetHandle () */
} finally {
if (release)
safeHandle.DangerousRelease ();
}
@monojenkins merge
[runtime] Don't access MonoClass::exception_type directly.
Use `mono_class_get_failure ()` and `mono_class_has_failed ()`
This is mostly just cleanup refactoring at this point that will make it easier to change how exception_type is used.
Also from this exercise we've learned that most of the class handling code just cares whether the class has an error or not, not what sort of error it was.
If we get a ThreadAbortException while at SafeHandle.cs:124 (before this commit), we would not set release to true, even after incrementing the internal reference count of the SafeHandle. That could lead to potential leak, as the caller (following the common following pattern) would never call DangerousRelase, thus never calling PerformRelease. The finalizer or Dispose would not lead to the call of PerformRelease either, as the DangerousAddRef / DangerousRelase calls are unbalanced.
The common pattern for DangerousAddRef / DangerousRelease is the following:
bool release = false;
try {
safeHandle.DangerousAddRef (ref release);
/* do something with safeHandle.DangerousGetHandle () */
} finally {
if (release)
safeHandle.DangerousRelease ();
}
In the case where we would call for example WaitHandle.WaitOne with a timeout, we could run in the case where we would wait way longer than the timeout would allow it (240s instead of 90s for example). This would be due to spurious wake, which would not be considered to be an error, but which would lead calling the `mono_os_cond_timedwait` function with the same timeout every time. That would mean that `mono_os_cond_timedwait` would return that it timedout, if and only if, it would timed out after N milliseconds, with N the millisecondsTimeout parameter to WaitOne in this example. That mean that any spurious wake would effectively lead to rewaiting with a full timeout every time.
That could lead to the following scenario:
- T1 calls waitHandle.WaitOne(timeout = 1000ms) at T=0ms
- T1 calls mono_os_cond_timedwait(timeout = 1000) at T=0ms
- GC suspend T1 at T=500ms <- here it's not necessarily the GC, it can be anything suspending/signaling the thread
- T1 spurious wake, and call mono_os_cond_timedwait(timeout = 1000) at T=500ms
- GC interrupt T1 at T=1250ms
- T1 spurious wake, and call mono_os_cond_timedwait(timeout = 1000) at T=1250ms
- etc.
As you can see, T1 should have returned from waitHandle.WaitOne before the second spurious wake, but as it recalled mono_os_cond_timedwait with a timeout of 1000ms again, it didn't.
The fix consists in keeping track of when the call is supposed to finish, and pass an adaptable timeout to mono_os_cond_timedwait, which decreases as time goes on, to enventually reach 0, and return in a timely manner.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=38382
We do not want to wait on a handle that have been cleaned up, as that will lead to a warning message from the io-layer, without any error in managed code. That mean we could not detect this error case and recover from it.
By doing this DangerousAddRef / DangerousRelease before going to native, we ensure that the WaitHandle is and will stay alive while waiting on it. This same is done in CoreCLR, but in the marshalling code.
When waiting on the current process, we would return WAIT_FAILED, as waitpid would return -1, with errno set to ECHILD.
But the semantic of WaitForSingleObjectEx and WaitForMultipleObjectsEx when waiting on current process is to return WAIT_TIMEOUT. This is then a special case that we need to add to this code.
dmcs was deprecated in favor of mcs a while back, yet people are still creating new scripts with it.
Adding a message that shows up on each dmcs invocation should hopefully make this more obvious.
We can use mono_class_load_from_name in all cases that we previously used to assert after loading.
All optional types have being moved over to use GENERATE_TRY_GET_CLASS_WITH_CACHE.
All but System.Runtime.ConstrainedExecution.CriticalFinalizerObject which is used within
sgen and we must ensure we don't try to lazy load it there.
I'm not 100% sure whether it should be optional, maybe it's a leftover from the 1.0 -> 2.0 timeframe?
The problem is that marking more types as required would be a burden on linkers as they might not force
it to be preserved.
I'm not particularly happy with the current large set of types that are loaded upfront. A significant
number of them could be lazy loaded.
This function generates probing functions that rely on mono_class_try_load_from_name.
In addition replace usage of hand crafted loading code from all around.
Those are variants of mono_class_from_name_checked that implement two common type loading scenarios in the rutime.
The first is mono_class_load_from_name, that loads critical types that the runtime cannot function without.
This means we abort if the type is missing or broken. We should use it in places where today we crash when
either of those would happen.
The other is mono_class_try_load_from_name, which is ok if the type is not found, but will abort if the type is broken.
The reasoning for aborting on a broken type is that it's unclear on whether the runtime should be using the type or not,
specially given most of those optional types change behavior of the application. This, in practice, forces the linker to
not produce a broken corlib.
[System.Drawing.Printing] CUPS default PageSize and InputSlot should …
[System.Drawing.Printing] CUPS default PageSize and InputSlot should override PPD defaults.
There is already a test case for that (PrintingServicesUnixTest.cs, Bug602934_PrinterSettingsReturnActualValues) which fails when user configured options are different from PPD settings. Patch also makes this test not to fail if "Resolution" option is not present in CUPS options.
This is useful for coop mode only, as coop needs additionnal thread state transition from BLOCKING to RUNNING and vice versa, and if both these transitions are not done properly, we finish in a incorrect state.
In non-coop mode, the mono_jit_detach only sets the domain to NULL, to avoid scanning threads unnecessarily, so not actually detaching properly is not hurting anything.
Previously, when we would invoke a native-to-managed wrapper from a thread that has not been attached to the runtime, we would simply ensure to attach this thread to the runtime. The issue arise when using coop and needing to switch a thread which is not executing runtime or managed code to BLOCKING state.
That would arise on iOS when using dispatch_async:
- the user would enqueue a Delegate, triggering a call to mono_delegate_to_ftnptr, generating this native-to-managed wrapper
- the wrapper would be called asynchronously on a thread managed by the OS/dispatch_async
- the thread would attach to the runtime, switching its state from STARTING -> RUNNING
- the thread would finish executing the delegate, thus 'detaching' from the runtime, but without switching its state from RUNNING -> BLOCKING
- during the next garbage collection, the GC would try to suspend this thread as it is still in RUNNING state, but because it's neither executing runtime code, neither managed code, it will never hit a safepoint, leading to a timeouting suspend.
To fix that issue, we need to support the following state switches:
- STARTING -> RUNNING -> BLOCKING: happens when it's the first time this thread calls a native-to-managed wrapper
- BLOCKING -> RUNNING -> BLOCKING: happens when it's not the first time this thread is called from a facility like dispatch_async, or if it is, for example, called as a callback from an external library
- RUNNING -> RUNNING -> RUNNING: if we call this wrapper from runtime or managed code, see for example System.Reflection.Emit.ILGeneratorTest.TestEmitCalliWithNullReturnType
This facility also need to take care of the current domain, to ensure that we switch properly between domains, and set it to the one that was used when creating this delegate, then restoring the original one.
The previous implementation of ProcessWrapper OutputStreamChanged and ErrorStreamChanged would manually launch a background thread to read on the process StandardOutput and StandardError, then simply passing the output to the OutputStreamChanged and ErrorStreamChanged events. This mean that even the newline characters would be passed to these events, leaving these events callbacks deal with splitting the output line by line.
On the other hand, Process.OutputDataReceived and Process.ErrorDataReceived already split the data line by line, and discard the newline character. That implies that, to keep the old ProcessWrapper behaviour, we need to add these newlines character before calling OutputStreamChanged and ErrorStreamChanged events callbacks.
We observe the following error when building XS:
"Threadpool worker" tid=0x0x7f011bfff700 this=0x0x7f01394bfb30 thread handle 0x432 state : not waiting owns ()
at <unknown> <0xffffffff>
at (wrapper managed-to-native) System.Threading.WaitHandle.WaitAll_internal (System.Threading.WaitHandle[],int,bool) <0x0005d>
at System.Threading.WaitHandle.WaitAll (System.Threading.WaitHandle[]) <0x00026>
at Microsoft.Build.Utilities.ProcessWrapper.<Start>m__0 (object,System.EventArgs) <0x0008f>
at System.Diagnostics.Process.OnExited () <0x000ed>
at System.Diagnostics.Process.RaiseOnExited () <0x0009f>
at System.Diagnostics.Process.CompletionCallback (object,bool) <0x00017>
at System.Threading.RegisteredWaitHandle.DoCallBack (object) <0x00088>
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (object) <0x00058>
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001c6>
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00020>
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x0004c>
at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x001d6>
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x00008>
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) <0x0005a>
And the process output would contains a bunch of "_wapi_handle_ref: Attempting to ref unused handle 0x1234" and "_wapi_handle_unref_full: Attempting to unref unused handle 0x1234".
The expected behaviour is for WaitHandle.(WaitAll|WaitAny|WaitOne|SignalAndWait) and EventWaitHandle.Set to throw an ObjectDisposedException in case the WaitHandle (or one of them if calling WaitAll, WaitAny or SignalAndWait) have been disposed.
This error would be due to a race between ProcessWrapper.Dispose, which closes endEventOut and endEventErr, and the above Process.Wrapper.<Start>m__0 (the base.Exited) callback, which waits on endEventOut and endEventErr.
There is already a test case for that (PrintingServicesUnixTest.cs,
Bug602934_PrinterSettingsReturnActualValues) which fails when user
configured options are different from PPD settings. Patch also makes
this test not to fail if "Resolution" option is not present in CUPS
options.
*Update: removed leftover debug Console.WriteLine from patch
The internal GetLegacyCertificatePolicy() now returns null unless this
property has explicitly been assigned to, so the legacy handler will not
be used in ChainValidationHelper.
(cherry picked from commit e69e878405)
Fix mobile/AOT build following System.Diagnostics.Process import
System.Diagnostics.Process was recently added to Mono. This introduced two problems for mobile builds:
- Support classes were not added to the mobile profile .sources.
- The imported referencesource code did not build on fullaot.
This PR fixes both issues.
[Microsoft.Build] Fix Microsoft.Build.Execution.BuildManagerTest.BasicManualParallelBuilds
The previous implementation of ProcessWrapper would launch its own threads to read on the process stdout and stderr. We can more simply use OutputDataReceived and ErrorDataReceived which will do the same under the hood. We use as well the Exited event to track the end of the Process.
The previous implementation of ProcessWrapper would launch its own threads to read on the process stdout and stderr. We can more simply use OutputDataReceived and ErrorDataReceived which will do the same under the hood. We use as well the Exited event to track the end of the Process.
If we construct RegisteredWaitHandle with executeOnlyOnce = true, and never call Unregister, we will keep _waitObject.SafeWaitHandle reference count unbalanced. This will lead to never releasing the waitObject handle.
As we only need to ensure that the waitObject handle is alive while in the Wait method, we use the SafeHandle mechanism only there. The 2 possibles scenarios in the Wait method are:
- waitObject has not been closed: we increment its ref count, so we ensure that it's alive as long as we use it here
- waitObject has been closed: a ObjectDisposedException is thrown, we catch it and keep on without trying to wait on the handle
Operating systems that use linux kernels before 3.9 do not support
binding multiple socket to the same address and port.
In those systems even if it was possible to set
SocketOptionName.ReuseAddress an exception was thrown on the second
bind.
These changes introduce a SocketException that is thrown when
SocketOptionName.ReuseAddress is set on a system that will fail to bind
multiple times to the same address.
- fixes#38818
- setting the *unchanged* scope to non-marked types is harmful,
because in case the type has declaring type, the scope of declaring
type is set as well. see
https://github.com/mono/cecil/blob/master/Mono.Cecil/TypeReference.cs#L132
public virtual IMetadataScope Scope {
get ...
set {
var declaring_type = this.DeclaringType;
if (declaring_type != null) {
declaring_type.Scope = value;
return;
}
scope = value;
}
}
so in our case, we first set correctly scope of
System.Collections.Generic.Dictionary`2 to mscorlib, then (re)set
scope of System.Collections.Generic.Dictionary`2/ValueCollection
and
System.Collections.Generic.Dictionary`2/ValueCollection/Enumerator
to System.Collection (note that these types are not marked and are
swept away by linker), which sets scope of its declaring type
System.Collections.Generic.Dictionary`2 back to System.Collection
and because the System.Collection is linked-out, the typeref is
incorrectly written with the scope set to System later
In the following case we would have a race between Wait and Unregister methods:
- T1 calls ThreadPool.RegisterWaitForSingleObject to wait on waitHandle
- registeredWaitHandle is constructed to wait on waitHandle
- registeredWaitHandle.Wait method is enqueued on the ThreadPool
- T1 calls registeredWaitHandle.Unregister
- T1 calls waitHandle.Close (happens in Process.StopWatchingForExit)
- TTP (the threadpool thread) calls registeredWaitHandle.Wait
- TTP calls WaitHandle.WaitAny (waitHandle, ...)
The last step try to access the waitHandle that has been previously closed, leading to a _wapi_handle_ref / _wapi_handle_unref_full error message.
The fix is to ensure that the RegisteredWaitHandle keeps the WaitHandle alive. To achieve this, we simply use the SafeWaitHandle inside the WaitHandle by: increasing its ref count when constructing the RegisteredWaitHandle, and decrementing it when unregistering it.
We also increase the count when using it in Wait, so if we execute Unregister and Wait concurrently, we guarantee that the WaitHandle is still valid in Wait, or get an exception if not.
FileInfo.MoveTo was not moving a file from one directory to another when
the file name did not change.
Fixes#18361.
An ArgumentException when destFileName is an empty array was added,
otherwise FileInfoTest.MoveTo_DestFileName_Empty would fail. Both
Path.GetFullPath and File.Move throw similar exceptions but
MoveTo_DestFileName_Empty expects the one from File.Move so it was added
to this method.
Mono has a list of assemblies considered "internal", whose versions are automatically remapped to the distro default version.
Since we no longer bundle Npgsql, it is up to individual distributions to package their own Npgsql - but Mono cannot load any version other than 4.0.0.0 from the GAC, due to this remapping:
Mono: Assembly Ref addref npghello[0x17e2fd0] -> mscorlib[0x1789c00]: 2
Mono: The request to load the assembly Npgsql v2.2.7.0 was remapped to v4.0.0.0
Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Npgsql/4.0.0.0__5d8b90d52f46fda7/Npgsql.dll'.
Npgsql, Version=2.2.7.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7
This was added in 2687866e99, but on systems where /bin/sh (which make uses by default) isn't bash this doesn't work.
Fix is to use a standard switch-case which works in all shells.
Do not run 4GB string tests on low memory systems (fixes#38392)
"Low memory" here is defined as 32-bit, less than 6GB of physical RAM, or mobile (since mobile lacks the class for querying physical RAM).
Without this patch, one of the corlib tests allocates 4 GB of RAM, which causes OOM crashes on low-memory Jenkins instances and also mobile phones.
This reverts commit a7955a93f5.
The Process import commit was reverted in f74f703719,
this reverts the added tests as well since they fail on Jenkins now.
The implementation added in b5d0ffe7dd forgot to multiply vmstat.free_count (== number of free pages)
by the page size in the __APPLE__ case, making the reported number much smaller than the reality.
[reflection] Signal pending loader error in mono_reflection_get_custom_attrs_data
Fixes [Bugzilla 38222](https://bugzilla.xamarin.com/show_bug.cgi?id=38222)
While we're at it, convert `mono_reflection_get_custom_attrs_data` to use
`MonoError` and mark it external only. Runtime should use
`mono_reflection_get_custom_attrs_data_checked`
`sgen_ensure_free_space()` assumed that if `size` was larger than
`SGEN_MAX_SMALL_OBJ_SIZE`, it should find free space on the major
heap, so it wouldn't consider triggering a nursery allocation. Making
TLABs larger than that would thus never trigger nursery collections
anymore.
Now we pass in the generation that says where we want space.
The Mono cert store traditionally uses the SKI for the filename of a certificate, but this does not handle cases where a cert has been reissued with the same SKI but different serial (seen in the wild with StartCom certs). This patch appends the serial to all new imported certs, and attempts to interoperate cleanly with existing serial-less certificate stores to avoid duplication.
Closes: #30902
Allows opening a character device or named fifo more than once with code like:
using (var f = new FileStream ("/dev/zero", FileMode.Open))
{
...
}
using (var f = new FileStream ("/dev/zero", FileMode.Open))
{
...
}
[System] Fix a few more hardcoded ports in tests
They'd cause "address already in use" errors when two builds are running concurrently on the same machine on Jenkins.
@monojenkins merge
InternalThread is a fundamentally different type WRT to appdomain. All instances of Thread bound to the same
os thread have a reference to a shared InternalThread.
InternalThread is always allocated in the root domain, so it MUST NOT have references to other domains as
it will lead to crashes on unload.
This was the case with pending_exception. It could point to an exception meant to the domain unloading.
This bug became more and more frequent as we move to use mono_set_pending_exception in more places. I could
reproduce it in 10 minutes of looping appdomain-unload.exe.
Separate the code path for async and self abort/suspend. This make the code intention more clear. It also allows us to remove some unused parameters (can_raise_excetpion which is always TRUE). Add some comments and harmonise locking behaviours : async_suspend_internal and self_suspend_internal now both call UNLOCK_THREAD.
On Windows we saw a failure like the following in mcs tests:
```
cs1589-2.cs... REGRESSION (CORRECT ERROR -> WRONG ERROR MESSAGE)
Exp: Unable to include XML fragment `/foo/bar' of file `there-is-no-such-file'. Could not find file "*PATH*/there-is-no-such-file"
Was: Unable to include XML fragment `/foo/bar' of file `there-is-no-such-file'. Could not find file "C:\j\workspace\z\label\w64\mcs\errors\there-is-no-such-file"
```
Fixing it by replacing the backslash with forward slash before comparing it to the expected value.
When MONO_PATH is set, the corlib lookup returns early on in mono_assembly_load_corlib (),
without adding the Facades dir to default_path.
This causes any app/library that relies on the facades to fail at runtime.
Fixing this by always doing the Facades lookup when returning the found corlib image.
Fixes [Bugzilla 38222](https://bugzilla.xamarin.com/show_bug.cgi?id=38222)
While we're at it, convert mono_reflection_get_custom_attrs_data to use
MonoError and mark it external only. Runtime should use
mono_reflection_get_custom_attrs_data_checked
fix for the following problem:
> [13:33:37] src/System/Test/System.Diagnostics/ProcessTest.cs(1048,19): error CS0103: The name `GetCrossPlatformStartInfo' does not exist in the current context
[threadpool] Improve the monitor thread heuristic
Because the ThreadPool heuristic is optimized for short lived work item, it has more difficulty when executing long lived work item. That case can lead to a starvation of the worker threads: they are all doing work, while there is outstanding request, and these requests are not satisfied because we reached the max number of working thread.
To fix that issue, we have the monitor thread, whose sole job is to unstuck this kind of starvation cases. Unfortunately, it only works when all the worker threads are in the ThreadState.WaitSleepJoin state, which excludes the case of calling an IO operation in a Task. That includes the case of FileStream.BeginRead/Write/... which implementation can be simplified as follows: FileStream.BeginRead(...) -> Task.Run(() => FileStream.Read(...)).
The way we implement it in this commit is: every MONITOR_INTERVAL (500ms here), we check if there is any outstanding request, and if so, we assume that we are in the starvation case, and we simply increase the max number of working thread. Also to reduce the number of false positives, we do that only if there has been no completed work item for more than at least MONITOR_INTERVAL (in case of low CPU usage, more otherwise, see monitor_sufficient_delay_since_last_dequeue). This case is typically the case where we have all working thread stuck in long running work items.
Finally we increase the monitor interval from 100ms to 500ms so we guarantee we do not create more than 2 threads per second in the monitor thread. That is the value used by the old threadpool.
Because the ThreadPool heuristic is optimized for short lived work item, it has more difficulty when executing long lived work item. That case can lead to a starvation of the worker threads: they are all doing work, while there is outstanding request, and these requests are not satisfied because we reached the max number of working thread.
To fix that issue, we have the monitor thread, whose sole job is to unstuck this kind of starvation cases. Unfortunately, it only works when all the worker threads are in the ThreadState.WaitSleepJoin state, which excludes the case of calling an IO operation in a Task. That includes the case of FileStream.BeginRead/Write/... which implementation can be simplified as follows: FileStream.BeginRead(...) -> Task.Run(() => FileStream.Read(...)).
The way we implement it in this commit is: every MONITOR_INTERVAL (500ms here), we check if there is any outstanding request, and if so, we assume that we are in the starvation case, and we simply increase the max number of working thread. Also to reduce the number of false positives, we do that only if there has been no completed work item for more than at least MONITOR_INTERVAL (in case of low CPU usage, more otherwise, see monitor_sufficient_delay_since_last_dequeue). This case is typically the case where we have all working thread stuck in long running work items.
Finally we increase the monitor interval from 100ms to 500ms so we guarantee we do not create more than 2 threads per second in the monitor thread. That is the value used by the old threadpool.
There's no managed code calling mono_domain_try_unload, so raising was
wrong. Thread the error out through exc which gets checked by callers.
Fixup for 45bd9bb5d5
[AsyncResult] Fix unhandled exception behaviour in AsyncCallback
The AsyncCallback is called after the BeginInvoke has completed. In all cases, this AsyncCallback is called on the ThreadPool, meaning that an unhandled exception would lead to a process crash.
The correct behaviour, after verification on .NET, is to not swallow the exception, and to properly crash the process.
@monojenkins merge
The AsyncCallback is called after the BeginInvoke has completed. In all cases, this AsyncCallback is called on the ThreadPool, meaning that an unhandled exception would lead to a process crash.
The correct behaviour, after verification on .NET, is to not swallow the exception, and to properly crash the process.
Recognize OS X file "locked" state (fixes#37834)
Quick fix for https://bugzilla.xamarin.com/show_bug.cgi?id=37834
IOLayer is_file_writable/System.IO.File.GetAttributes on posix systems looks only at the unix file permissions and does not consider special OS flags. This leads to an easily end-user-triggered bug on OS X if a mono app (the bug was observed in Xamarin Studio) uses GetAttribute on a file which has been "Locked" using the checkbox in Finder Get Info. This patch recognizes the "uchg" flag for non-writability which Finder uses.
The performance team discovered this on their bots using the Jenkins snapshot packages. Mono would fail to start with a message like:
```
The assembly mscorlib.dll was not found or could not be loaded.
It should have been installed in the `/opt/mono-2016.02.08+15.25.39/lib/mono/4.5/mscorlib.dll' directory.
```
The one odd thing is that they are using Mono installed in a path which is different from the prefix used during compilation.
It turns out that the runtime has a hardcoded check for lib/mono/2.0 in https://github.com/mono/mono/blob/7d5a7ea7ac324bb4ee63094ddcf8e20ae9ba4322/mono/metadata/assembly.c#L633-L637.
Since we recently stopped shipping 2.0 in the packages (we moved to 2.0-api for the reference assemblies) this directory wasn't there,
so the code in fallback() called into mono_config_get_assemblies_dir() which just returns the MONO_ASSEMBLIES preprocessor symbol.
That preprocessor symbol is defined to the value of --prefix during compilation, so Mono couldn't find corlib there as well since it's installed in another path which ultimately results in the above error message.
The fix is to change the hardcoded value to 4.5, which is what we established as the directory with the "latest" assemblies now.
Ideally we should find a way to stop hardcoding those paths in the runtime.
PS: I grepped through the runtime source to make sure we don't hardcode 2.0/3.5/4.0 elsewhere and only found one instance in a test.
I fixed it there as well, though the test looks suspicious since it hardcodes /usr ...
IOLayer is_file_writable/System.IO.File.GetAttributes on posix systems
looks only at the unix file permissions and does not consider special
OS flags. This leads to an easily end-user-triggered bug on OS X if a
mono app (the bug was observed in Xamarin Studio) uses GetAttribute on
a file which has been "Locked" using the checkbox in Finder Get Info.
This patch recognizes the "uchg" flag for non-writability which Finder
uses.
It failed twice on Jenkins over the weekend:
```
Test Case Failures:
1) MonoTests.System.Runtime.Caching.MemoryCacheTest.TestExpiredGetValues : #EGV4
Expected: 0
But was: 9
at MonoTests.System.Runtime.Caching.MemoryCacheTest.TestExpiredGetValues () [0x0012b] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/mcs/class/System.Runtime.Caching/Test/System.Runtime.Caching/MemoryCacheTest.cs:1304
```
We're sleeping just as long as the expiry time of the cache items (1s) which might not be enough.
Bumping the sleep time should fix this.
Mark mono_assembly_get_object external only.
Runtime should use mono_assembly_get_object_checked.
Also propagated MonoError through a few static functions in reflection:
mono_reflection_get_type_internal () and
mono_reflection_get_type_internal_dynamic ()
The concurrent collector typically uses more memory because more memory is allocated during the concurrent phase, which also leads to more objects surviving the major collections and higher allowances. We improved the memory situation by forcing synchronous finishing of the concurrent mark if too much memory is allocated during the concurrent mark phase. Still, this doesn't tackle the underlying problem. Even if we have less memory allocated during the CMS phase, some is still allocated and leads to increased overall memory usage. And how much memory growth should we allow in the first place ? The even greater problem is that it appears to force a lot of synchronous finishes, significantly increasing finishing pause times.
This commit attempts to solve this problem by following an ideal mathematical approach. We use the following notations : H - heap size at start of major collection, Gs - heap growth during sync collection, Gc - heap growth during conc collection, s - survival rate of the collection, a - allowance ratio (0.33), Ts - trigger for next major sync, Tc - trigger for next major conc.
Assume we have the heap size H at the start of a major collection. For the two types of majors we would have :
Ts = (H + Gs) * s * (a + 1)
Tc = (H + Gc) * s * (a + 1)
Obviously the problem is that Gc > Gs, which leads to Tc > Ts and higher memory usage. In order to have the two triggers equal we need to find an X that we can subtract from Tc to obtain Ts. Considering Gs = 0 (only a few nursery objects are promoted during a major sync collection) and similar survival rates for the two types of collection, we get X = Gc * s * (a + 1). In order to get equal triggers we would need to subtract this value from the allowance computed after a concurrent collection.
The more the heap grows, the more X grows. If X gets bigger than the available starting allowance ((H + Gc) * s * a), it would mean that we are trying to set a trigger that it is lower than the actual heap size at the end of the collection (which means we already exceeded the target memory usage). In order to avoid this problem we need to enforce Tc - X > (H + Gc) * s. Doing the substitutions we get Gc < a * H. When the heap growth exceeds this value it means we need to force the finish of the concurrent collection.
When sample profiling is enabled and we send a SIGPROF signal to a thread that
hasn't yet transitioned to a running state (e.g. STATE_RUNNING), the signal can
sometimes be delivered at such an early point that libc has not even finished
initializing TLS for that thread yet. This is problematic because we then go
right on to crash in the SIGPROF signal handler when we try to save errno. We
then further crash in the SIGSEGV handler because we can't access the jit_tls
variable either.
The fix is fairly simple: Only send SIGPROF to threads that have transitioned
to a running state. Technically, TLS is initialized 'much' earlier than when
this state transition happens, but since the worst thing that can happen is
that we miss one sampling event (i.e. ~1sec by default) right at the beginning
of the thread's lifetime, this simple fix is good enough.
This crash seemingly only happened on OS X.
With GCC 5.x, the compiler optimises away the dummy load used to
estimate the stack pointer address in GC_approx_sp; it returns
zero. This leads to segfaults when embedding Mono.
Instead, use __builtin_frame_address, which GCC's libgc uses for
the same purpose.
[runtime] Use MonoError for mono_string_new
We are refraining from changing runtime callers of `mono_string_new` to `mono_string_new_checked` in one commit. String creation is used in several hundred places and threading error handling through all the callers is very intrusive. Additionally, AFAIK the only error condition from mono_string_new is out-of-memory. In a lot of cases there's no way to meaningfully recover.
We should revisit individual callers when it makes sense, but just mechanically updating everywhere doesn't get us a lot of benefit.
[runtime] Use MonoError for mono_method_get_object
Depends on #2532
Issues:
- [x] ~~depends on #2486~~
- [x] ~~Should use `mono_object_new_checked` #2532~~
- [x] ~~Should use `mono_string_new_checked`~~ (`mono_string_new` will assert instead of raising an exception - #2530)
[threads] avoid race between setting the state flag and suspension
there's a short window where we release the lock and another thread
could modify the state. That is what we saw on the ARM machines for a
while, e.g. in `subthread-exit.cs` there's a race between
`mono_thread_execute_interruption` and `mono_thread_suspend_all_other_threads`:
> Thread 2 (Thread 0x429ff430 (LWP 5189)): // SUB THREAD
> #0 0x40398e30 in nanosleep () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x0028c0ac in monoeg_g_usleep (microseconds=86650) at gdate-unix.c:53
> #2 0x0027b1d8 in suspend_sync_nolock (id=1079603792, interrupt_kernel=1) at mono-threads.c:913
> #3 0x0027b2d4 in mono_thread_info_safe_suspend_and_run (id=1079603792, interrupt_kernel=1, callback=0x1ab109 <suspend_thread_critical>, user_data=0x429fe534)
> at mono-threads.c:935
> #4 0x001ab28a in suspend_thread_internal (thread=0x401c0120, interrupt=1) at threads.c:4807
> #5 0x001a91ec in mono_thread_suspend_all_other_threads () at threads.c:3297
> #6 0x00158d8c in ves_icall_System_Environment_Exit (result=0) at icall.c:6378
> #7 0x400e7d9c in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 1 (Thread 0x40597250 (LWP 5186)): // MAIN THREAD
> #0 0x4039a5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x4039796e in do_futex_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2 0x403979da in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #3 0x002788d8 in mono_os_sem_wait (sem=0x388248, flags=MONO_SEM_FLAGS_NONE) at ../../mono/utils/mono-os-semaphore.h:163
> #4 0x00279722 in mono_thread_info_wait_for_resume (info=0x388210) at mono-threads.c:144
> #5 0x0027ad5e in mono_thread_info_end_self_suspend () at mono-threads.c:700
> #6 0x001ab2d2 in self_suspend_internal (thread=0x401c0120) at threads.c:4822
> #7 0x001aab0a in mono_thread_execute_interruption () at threads.c:4337
> #8 0x001a6a82 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1200) at threads.c:1217
> #9 0x400da75c in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
there's a short window where we release the lock and another thread
could modify the state. That is what we saw on the ARM machines for a
while, e.g. in `subthread-exit.cs` there's a race between
`mono_thread_execute_interruption` and `mono_thread_suspend_all_other_threads`:
> Thread 2 (Thread 0x429ff430 (LWP 5189)): // SUB THREAD
> #0 0x40398e30 in nanosleep () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x0028c0ac in monoeg_g_usleep (microseconds=86650) at gdate-unix.c:53
> #2 0x0027b1d8 in suspend_sync_nolock (id=1079603792, interrupt_kernel=1) at mono-threads.c:913
> #3 0x0027b2d4 in mono_thread_info_safe_suspend_and_run (id=1079603792, interrupt_kernel=1, callback=0x1ab109 <suspend_thread_critical>, user_data=0x429fe534)
> at mono-threads.c:935
> #4 0x001ab28a in suspend_thread_internal (thread=0x401c0120, interrupt=1) at threads.c:4807
> #5 0x001a91ec in mono_thread_suspend_all_other_threads () at threads.c:3297
> #6 0x00158d8c in ves_icall_System_Environment_Exit (result=0) at icall.c:6378
> #7 0x400e7d9c in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 1 (Thread 0x40597250 (LWP 5186)): // MAIN THREAD
> #0 0x4039a5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x4039796e in do_futex_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2 0x403979da in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #3 0x002788d8 in mono_os_sem_wait (sem=0x388248, flags=MONO_SEM_FLAGS_NONE) at ../../mono/utils/mono-os-semaphore.h:163
> #4 0x00279722 in mono_thread_info_wait_for_resume (info=0x388210) at mono-threads.c:144
> #5 0x0027ad5e in mono_thread_info_end_self_suspend () at mono-threads.c:700
> #6 0x001ab2d2 in self_suspend_internal (thread=0x401c0120) at threads.c:4822
> #7 0x001aab0a in mono_thread_execute_interruption () at threads.c:4337
> #8 0x001a6a82 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1200) at threads.c:1217
> #9 0x400da75c in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
The wildcard `$(wildcard ../../../external/binary-reference-assemblies/v4.5/*)` would include the directory entry `../../../external/binary-reference-assemblies/v4.5/Facades/` which returns an error during "make dist".
Fixing this by including only the .dll files in the wildcards.
mono_runtime_invoke might be called by P/Invoked code, which is in blocking mode. Not switching back to running can lead to a crash when hitting a safepoint.
[sgen] Fix register scanning on ARM
This bug would trigger a use after sweep in System.Threading.Tasks.Task.FinishContinuations:3621.
The issue would arise on ARM, as `MonoContext` and `ARCH_NUM_REGS` are defined as followed :
```
typedef struct {
mgreg_t pc;
mgreg_t regs [16];
double fregs [16];
mgreg_t cpsr;
} MonoContext;
\#define ARCH_NUM_REGS 14
```
As you can see, the MonoContext structure is bigger than 14 words, and it does not even covers the last 3 values of `regs`. By using pointer arithmetic, we ensure that we do not miss some parts of the `MonoContext` structure.
The observed behaviour in System.Threading.Tasks.Task.FinishContinuations would be that `continuationObject` would not be marked, and thus freed; `continuationObject` would still contains the pointer to the old location, meaning it wouldn't be null. We would then try to check its type via a call to the `as` operator. This would call the `isinst` IL opcode which would load `continuationObject->vtable->klass->supertypes [...]`, which would trigger a segfault, as `supertypes` would be null.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=38012
Current Debian packaging puts libtool in its own package (libtool-bin)
leaving only libtoolize in the main package. The mono build process
uses libtoolize not libtool, so during autogen check for libtoolize
not libtool.
* only call EndEdit in scrollbar value changed handlers when currently
editing a node
* fix for bug 33183
Change-Id: I54407e00314f4bfbdc175e883199d3d1ef057911
* keep a reference to a __ComObject's proxy so it doesn't get garbage
collected
* don't create __ComObjects unnecessarily
* Fix for bug 24974
Change-Id: Ia09237b7f1af996405134c250bbc29220f81f067
Docker uses aufs or overlayfs to mount the root system in a
container. This change fixes Xamarin-31021 so that the drives
show up properly.
This change is released under the MIT license.
Change-Id: I5150b0e82f26e9dbd07bbcbc50e7ca50d856dde8
This bug would trigger a use after sweep in System.Threading.Tasks.Task.FinishContinuations:3621.
The issue would arise on ARM, as `MonoContext` and `ARCH_NUM_REGS` are defined as followed :
```
typedef struct {
mgreg_t pc;
mgreg_t regs [16];
double fregs [16];
mgreg_t cpsr;
} MonoContext;
\#define ARCH_NUM_REGS 14
```
As you can see, the MonoContext structure is bigger than 14 words, and it does not even covers the last 3 values of `regs`. By using pointer arithmetic, we ensure that we do not miss some parts of the `MonoContext` structure.
The observed behaviour in System.Threading.Tasks.Task.FinishContinuations would be that `continuationObject` would not be marked, and thus freed; `continuationObject` would still contains the pointer to the old location, meaning it wouldn't be null. We would then try to check its type via a call to the `as` operator. This would call the `isinst` IL opcode which would load `continuationObject->vtable->klass->supertypes [...]`, which would trigger a segfault, as `supertypes` would be null.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=38012
In reference source HttpHeaderCollection [1] when Add, Set or Remove is
called and the collection is readonly a PlatformNotSupportedException is
thrown.
Mono HeadersCollection was throwing NotSupportedException and
Microsoft.Owin.Host.SystemWeb.OwinCallContext.RemoveAcceptEncoding was
not catching the exception because it was expecting
PlatformNotSupportedException.
Fixes#33809
[1] http://referencesource.microsoft.com/#System.Web/HttpHeaderCollection.cs,73
We now add data-is-[non-]breaking attributes to the generated
html elements, and then use some javascript logic to show/hide
non-breaking elements (and the javascript also looks in the
container elements show/hide containers that have only
non-breaking descendants).
Also use css classes/styles to colorize.
This allow us catching and rethrowing the exception on the main thread, as well as making sure we wait for the Listen thread to finish in a timely manner.
The use of HttpResponseMessage.ReasonPhrase means:
* marking the HttpListenerResponse.GetStatusDescription method;
* marking the HttpListenerResponse type;
* which implements IDisposable, so it's Dispose method is marked
* marking the Close method...
from there a metric ton of code gets recursively marked to the point
where the whole SSL/TLS server code gets inside the app.
To avoid this we move the GetStatusDescription to a different
type that won't bring the end of the civilized world so fast.
The format of the test assembly name changed in 860334ff45
from System.Web_test_net_4_x.dll to net_4_x_System.Web_test.dll.
This fixes all the places where the old name was used.
[jit] Add a new jit icall mono_interruption_checkpoint_from_trampoline () and use that instead of mono_thread_force_interruption_checkpoint () so the latter can go away eventually.
This macro should be used to decorate functions that are part of the public API but the runtime is
not supposed to call.
This is to allow us to keep them around but not go crazy with commit reintroducing their usage.
I noticed this in https://jenkins.mono-project.com/job/test-mono-pull-request-armhf/1715/ which reported success in the "profiler" test step,
even though the step actually reported errors:
```
MONO_PATH=/media/ssd/jenkins/workspace/test-mono-pull-request-armhf/mcs/class/lib/net_4_x perl ./ptestrunner.pl ../..
Checking test-alloc.exe with report ...
Checking test-busy.exe with report ...
Checking test-busy.exe with report,sample ...
Error: Missing thread Finalizer.
Error: Missing thread Main.
Error: Wrong loaded images 2.
Error: Not enough compiled method.
Error: Not enough compiled code.
Error: Missing thread BusyHelper.
Total errors: 6
Checking test-monitor.exe with report ...
Checking test-excleave.exe with report ...
Checking test-heapshot.exe with report,heapshot ...
Checking test-heapshot.exe with heapshot,output=-traces.mlpd ...
Checking test-traces.exe with output=-traces.mlpd ...
Checking test-traces.exe with nocalls,output=-traces.mlpd ...
make[1]: Leaving directory `/media/ssd/jenkins/workspace/test-mono-pull-request-armhf/mono/profiler'
make: Leaving directory `/media/ssd/jenkins/workspace/test-mono-pull-request-armhf/mono/profiler'
++ bc
+++ date +%s
++ echo 1453874768 - 1453874741
+ echo -e '*** end(27): profiler: \e[42mPassed\e[0m'
*** end(27): profiler: Passed
```
The culprit is that ptestrunner.pl checks whether $total_errors is non-zero to determin the process exit code.
This is wrong, as $total_errors is cleared before each test so unless the last test fails the exit code would always be zero.
Introduced a new $global_errors variable which is only incremented and never cleared to fix this.
[System] Fix error in new BeginConnectToIPV4EndPointUsingDualModelSocket test
It was added with 78c758494a. The test fails on Jenkins:
```
Test Case Failures:
1) MonoTests.System.Net.Sockets.SocketTest.BeginConnectToIPV4EndPointUsingDualModelSocket : System.InvalidOperationException : EndConnect can only be called once per asynchronous operation
at System.Net.Sockets.Socket.ValidateEndIAsyncResult (IAsyncResult ares, System.String methodName, System.String argName) [0x0004a] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:3412
at System.Net.Sockets.Socket.EndConnect (IAsyncResult result) [0x00006] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:1584
at MonoTests.System.Net.Sockets.SocketTest.BeginConnectToIPV4EndPointUsingDualModelSocket () [0x0005d] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/Test/System.Net.Sockets/SocketTest.cs:4365
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
```
The reason is that BCCallback already calls EndConnect() so we need to remove it and just wait for the callback in the test itself.
@monojenkins merge
It was added with 78c758494a. The test fails on Jenkins:
```
Test Case Failures:
1) MonoTests.System.Net.Sockets.SocketTest.BeginConnectToIPV4EndPointUsingDualModelSocket : System.InvalidOperationException : EndConnect can only be called once per asynchronous operation
at System.Net.Sockets.Socket.ValidateEndIAsyncResult (IAsyncResult ares, System.String methodName, System.String argName) [0x0004a] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:3412
at System.Net.Sockets.Socket.EndConnect (IAsyncResult result) [0x00006] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:1584
at MonoTests.System.Net.Sockets.SocketTest.BeginConnectToIPV4EndPointUsingDualModelSocket () [0x0005d] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System/Test/System.Net.Sockets/SocketTest.cs:4365
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
```
The reason is that BCCallback already calls EndConnect() so we need to remove it and just wait for the callback in the test itself.
Failed on Jenkins on ARM today:
```
Tests run: 38, Failures: 1, Not run: 0, Time: 32.132 seconds
Test Case Failures:
1) MonoTests.System.Runtime.Caching.MemoryCacheTest.TimedExpiration : #A4-4
Expected: not null
But was: null
at MonoTests.System.Runtime.Caching.MemoryCacheTest.TimedExpiration () [0x002b6] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/mcs/class/System.Runtime.Caching/Test/System.Runtime.Caching/MemoryCacheTest.cs:902
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armel/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
```
100ms could be too short, bumping the timeout should make it more reliable on the slower ARM machines.
[array] Replace raise exception by pending exception in jit-icalls
The convention for jit-icalls is to set the pending exception, instead of raising it from native.
The following legacy functions are no longer used by the runtime and don't need loader-error conversion:
mono_method_get_signature_full
mono_method_get_signature
mono_field_from_token
May all that is unholy smite upon those attempting to use those functions again.
[runtime] Add MONO_ERROR_EXCEPTION_INSTANCE MonoError
This kind of MonoError can encapsulate a managed MonoException
object. (For example as a result of mono_runtime_invoke ()).
Issues:
- [x] Memory leak correctness.
`MonoError` already leaks memory if you simply swallow it without calling `mono_error_cleanup` or some function that calls it (for example `mono_error_raise_exception` or `mono_error_convert_to_exception`).
Now if we have an encapsulated exception, we'll leak a GC handle unless cleanup happens.
Unlike Linux, OSX does not allow multiple SCM_RIGHTS control messages in
sendmsg().
The SocketTest.ControlMsg() test is split into two tests, one which sends
only a single control message (containing two FDs), and another one which
sends two control messages (each containing one FD). The second test is not
executed on OSX.
Warning: System.Net.Mail is broken by design and sends incorrect emails.
This is not to fix SmtpClient itself but to ensure it remains compatible
(even if incorrect/broken) with older Mono releases.
The commit associated with bug# 33551 [1] can cause issues on computer
and devices that cannot resolve their hostname using the Dns API (which
have their own issues).
This commit catch the exception that occurs when resolution fails and
continue to send an email without a fully qualified domain name - just
like earlier Mono versions did in the past.
This solve bug #32746 [2] where some, but not all, AppleTV devices fails
to execute some of the System.dll unit tests.
Note: Whenever possible stay clear of SmtpClient (and friends) and use
something else like MimeKit/MailKit for your email needs
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=33551
[2] https://bugzilla.xamarin.com/show_bug.cgi?id=37246
[handle] Remove chunk_last field from MonoHandleArena
Remove the `chunk_last` field. This changes MonoHandleArena from a queue of chunks to a stack of chunks. More importantly, the MonoHandleArena has an initial chunk that is allocated on the stack together with the arena. Previously that chunk was at the front of the list of chunks, but now it is at the end.
Added with https://github.com/mono/mono/pull/2097. Fails on Jenkins:
```
Test Case Failures:
1) MonoTests.Mono.Unix.Native.SocketTest.ControlMsg : System.ArgumentException : Invalid argument
----> Mono.Unix.UnixIOException : Invalid argument [EINVAL].
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00000] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Mono.Unix/UnixMarshal.cs:460
at MonoTests.Mono.Unix.Native.SocketTest+<ControlMsg>c__AnonStorey2.<>m__0 (Int32 so1, Int32 so2) [0x0018f] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Test/Mono.Unix.Native/SocketTest.cs:778
at MonoTests.Mono.Unix.Native.SocketTest.WithSocketPair (System.Action`2 f) [0x00025] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Test/Mono.Unix.Native/SocketTest.cs:68
at MonoTests.Mono.Unix.Native.SocketTest.<ControlMsg>m__9 (Int32 inner_so1, Int32 inner_so2) [0x00014] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Test/Mono.Unix.Native/SocketTest.cs:739
at MonoTests.Mono.Unix.Native.SocketTest.WithSocketPair (System.Action`2 f) [0x00025] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Test/Mono.Unix.Native/SocketTest.cs:68
at MonoTests.Mono.Unix.Native.SocketTest.ControlMsg () [0x00000] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/Mono.Posix/Test/Mono.Unix.Native/SocketTest.cs:738
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/jenkins/workspace/test-mono-mainline@2/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
--UnixIOException
```
Because we could sleep for too long, while being in RUNNING state, the cooperative suspend would not finish in a timely manner. That would lead to the abort in `mono_threads_wait_pending_operations`.
When tracing multi-threaded program execution, it is useful to have a
way of separating which events happen on which thread. The
System.Threading.Thread.Name and Thread.ManagedThreadId properties are
useful proxies for tracking program execution.
However, if program execution spans both managed and embedded mono
native code, there is no existing mechanism to read the Thread.Name or
Thread.ManagedThreadId properties from native code without using
delegates, and delegates may not be usable in all circumstances (e.g.
within a mono_gc_register_bridge_callbacks() callback).
Add the following APIs to the embedding API to permit reading the
Thread.Name and Thread.ManagedThreadId properties:
char *mono_thread_get_name_utf8 (MonoThread *thread);
gint32 mono_thread_get_managed_id (MonoThread *thread);
mono_thread_get_name_utf8() returns the Thread.Name value as a UTF-8
string in newly allocated memory; the caller must g_free() this value.
mono_thread_get_managed_id() returns the Thread.ManagedThreadId value.
When the linker is enabled (default for device builds) unused methods
are removed and cannot be reflected.
In this case we do not need reflection as the answer to port reuse is
always true.
https://bugzilla.xamarin.com/show_bug.cgi?id=34133
We would previously open the standard output for the Console.Error StreamWriter. This behavior has never surfaced so far, because if you try to redirect the process standard output through a pipe or to a file, then the condition ConsoleDriver.IsConsole does not apply, and the other path does not have the same bug.
The only way to reproduce it is to redirect the standard error to another file, and observe that you would still have the Console.Error content. For example, with mono/tests/console-output.exe:
Expected output:
$> MONO_PATH=mcs/class/lib/net_4_x mono/mini/mono mono/tests/console-output.exe 2>/dev/null
Out
Actual output:
$> MONO_PATH=mcs/class/lib/net_4_x mono/mini/mono mono/tests/console-output.exe 2>/dev/null
Out
Error
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=36988
Fixes failure, where XmlSerializer would crash while serializing
a list with multiple XmlElement when a sub class of the XmlElement is
used.
Fixes#36829.
[runtime] Add SA_RESTART to SIG{CONT,INT,PROF,QUIT,USR2,WINCH}.
We need this as these signals can be delivered at any point and can interrupt system calls that external (non-Mono) code is executing, which such code may not be written to handle gracefully.
So we were previously taking the set symmetric difference of the
.sources and the .excludes files. This is semantically wrong, and the
change makes us take the difference. What we were doing is concatenating
the two files and taking any lines that are only seen once. If a file is
in excludes and not in sources, then we will include it. This is wrong.
This change always duplicates excludes, so anything present in excludes
is seen at least twice. This has the effect of making sure that only
things in .sources but not in .excludes are in the output file.
We need this as these signals can be delivered at any point and can interrupt
system calls that external (non-Mono) code is executing, which such code may
not be written to handle gracefully.
commit d2325f1483 (only since 4.3.2, not
4.3.1) caused the failure when an application is linked.
The use of `GetOrCreateValue` uses reflection to create an `NtlmSession`
instance so the linker won't see it as used anymore (and removes it).
Fixed by avoiding reflection (which will also perform better).
https://bugzilla.xamarin.com/show_bug.cgi?id=37732
The following exception would arise if we dispose the Process before making sure the AsyncStreamReader finished its work:
Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
at System.Diagnostics.AsyncStreamReader.ReadBuffer (IAsyncResult ar) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/System/services/monitoring/system/diagnosticts/AsyncStreamReader.cs:158
at System.IO.Stream+ReadWriteTask.InvokeAsyncCallback (System.Object completedTask) [0x00015] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/io/stream.cs:670
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) [0x0008d] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/executioncontext.cs:957
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/executioncontext.cs:904
at System.IO.Stream+ReadWriteTask.System.Threading.Tasks.ITaskCompletionAction.Invoke (System.Threading.Tasks.Task completingTask) [0x0005e] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/io/stream.cs:696
at System.Threading.Tasks.Task.FinishContinuations () [0x0009a] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:3633
at System.Threading.Tasks.Task.FinishStageThree () [0x00045] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2366
at System.Threading.Tasks.Task.FinishStageTwo () [0x000f8] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2339
at System.Threading.Tasks.Task.Finish (Boolean bUserDelegateExecuted) [0x00049] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2239
at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task& currentTaskSlot) [0x00079] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2834
at System.Threading.Tasks.Task.ExecuteEntry (Boolean bPreventDoubleExecution) [0x0006f] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2760
at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/Tasks/Task.cs:2707
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/threadpool.cs:857
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/external/referencesource/mscorlib/system/threading/threadpool.cs:1212
This would arise in test-runner.cs when one of the test would timeout, leading to the Process being disposed. This would then dispose the AsyncStreamReader, setting its `stream` value to null. The callback to ReadBuffer would then try to access `stream`, but would throw a NullReferenceException. By not disposing the AsyncStreamReader explicitely, we close the stream first, which will trigger a call to ReadBuffer, thus not triggering the NullReferenceException.
We were using get_runtime_invoke_type () to reduce the number of
wrappers we were using by sharing the wrappers that have MONO_TYPE_I
with that of MONO_TYPE_PTR. They have the same size, but these are not
valid types to share because they have different levels of indirection,
so runtime wrappers that have been aot'ed for the MONO_TYPE_I will be
used for MONO_TYPE_PTR and will dereference the value in the first word
of the pointer. In the case that this is null, we observed a segfault.
Specifically, the wrapper code in marshal.c looks like so:
```
case MONO_TYPE_I:
case MONO_TYPE_U:
case MONO_TYPE_I4:
case MONO_TYPE_U4:
case MONO_TYPE_R4:
case MONO_TYPE_R8:
case MONO_TYPE_I8:
case MONO_TYPE_U8:
mono_mb_emit_byte (mb, CEE_LDIND_I);
mono_mb_emit_byte (mb, mono_type_to_ldind (sig->params [i]));
break;
case MONO_TYPE_STRING:
case MONO_TYPE_CLASS:
case MONO_TYPE_ARRAY:
case MONO_TYPE_PTR:
case MONO_TYPE_SZARRAY:
case MONO_TYPE_OBJECT:
mono_mb_emit_byte (mb, mono_type_to_ldind (sig->params [i]));
break;
```
showing that the two types are not equivalent.
The test was reenabled in 3f3df26945 and we were seeing it randomly fail
on Jenkins on OSX like that:
```
[...]
***** MonoTests.Mono.Unix.UnixSignalTest.TestNoEmitAny
***** MonoTests.Mono.Unix.UnixSignalTest.TestRaise
***** MonoTests.Mono.Unix.UnixSignalTest.TestRaiseAny
***** MonoTests.Mono.Unix.UnixSignalTest.TestRaiseStorm
/bin/sh: line 1: 40164 Hangup: 1 PATH="/Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/runtime/_tmpinst/bin:/opt/mono/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin" MONO_REGISTRY_PATH="/Users/builder/.mono/registry" MONO_TESTS_IN_PROGRESS="yes" MONO_PATH="./../../class/lib/net_4_x::$MONO_PATH" /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/runtime/mono-wrapper --debug ./../../class/lib/net_4_x/nunit-console.exe Mono.Posix_test_net_4_x.dll -noshadow -labels -exclude=NotOnMac,NotWorking,ValueAdd,CAS,InetAccess -labels -xml=TestResult-net_4_x.xml
make[1]: *** [run-test-lib] Error 1
make[1]: Leaving directory `/Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/Mono.Posix'
make: *** [do-run-test] Error 1
```
The problem is that (especially on OSX it seems) the signals raised by the threads in the test can be
delivered after we closed the UnixSignal, which means the default behavior of terminating the program kicks in.
The simplest fix is to just sleep for some time to make sure the signal is delivered and ignored before closing.
This eliminates the builtin 128 bytes char array in MonoError and forces the error message to be always malloc'd.
It makes the code simpler and sacrifices a bit in terms of what we can handle in the case of a malloc OOM.
Malloc OOM is not a solvable problem for a few reasons.
1) Overcommit and virtual memory makes relying on malloc returning zero not an useful thing.
2) Everything goes south when malloc starts to fail. There's nothing much the runtime can do beyond
give up.
OTOH, saving stack space is pretty nice for its own reasons.
1) Clang now generates stack guards by default for frames larger than 120 bytes on amd64. This
means that MonoError has a much bigger overhead than needed.
2) We conservatively scan the stack and, with this change, those are 128 bytes less for the gc to figure out.
On ABI compatibility. We promise backward but not forward compatibility. Meaning someone compiling against 4.4
can't expect it to works against 4.2, but vice versa is fine.
We can handle old users because the struct is now smaller and has always being opaque, meaning callers just
a bit of waste space and that's it.
This means the following:
mono_object_new_mature handles OOM and finalization.
GC glue code returns NULL on failure and doesn't deal with finalization.
Finally, correctly implement mono_gc_alloc_mature in boehm and null-gc instead of the
semantically different implementation we had before.
The current gc interface requires the GC to call mono_gc_out_of_memory on allocation failure.
It won't work if we move away from implementing it with mono_raise_exception as we need to manually bubble
the OOM all the up to the m2n border.
The good thing is that this simplified the gc interface and actually fixes a corner case bug in sgen where
it did not call mono_gc_out_of_memory on alloc failure.
Due to how JI table search works, tombstones must be malloc'd as they must have the same address range
of the JI they are replacing.
Since tombstones are heap allocated, they must be freed together with the chunks they belong to.
The current code free tombstones when the chunk refcount goes to zero. The freeing code identify
tombstones by looping through the chunk entries.
The problem is that, under heavy contention, a domain can have multiple tables live at the same time.
It happens there's another thread using the old table while the first creates the new one.
During that period of multiple live tables, we don't free JIs straight away, we queue them to be
freed together with the next table to go through it.
So far so good. Here's the problematic scenario:
1) We expand the JI table, there's contention and the old old is delay freed.
While doing it, chunk X is split. Meaning its refcount will remain 1.
2) JI Y that belongs to chunk X is freed, the tombstone is installed in a chunk of the new table.
That chunk *is not* X.
Since there was contention in (1), Y is put in jit_info_free_queue.
3) We run mono_jit_info_table_free for the old table.
3.a) We free all delayed JIs. Y is freed.
3.b) The old table is freed, chunk X will be freed too, since its refcount dropped to zero.
We walk X looking for tombstones, which includes a pointer to Y, which is freed memory at this point.
I found this issue to be possible with delegates being marshaled to function pointers.
Part (1) just requires a process with high activity, like XS.
Part 2 & 3 requires really bad scheduling in that once sgen restarts the world, the finalizer thread
runs before mono_thread_hazardous_try_free_some (done right after dropping the GC lock).
An alternate to bad scheduling is that the finalizer thread was suspended holding the domain lock and
now after restart, it forces the GC initiator to deschedule when it tries to get that lock.
It goes without saying that this bug is EXTREMELY hard to reproduce. It required me to change the runtime
to force all the above conditions to happen. Let me mention them in case someone needs it:
- In mono_thread_hazardous_free_or_queue, I commented the calls to try_free_delayed_free_item to avoid
freeing the old table too early. I changed the conditional so free_func is never called straight away.
- In mono_thread_hazardous_try_free_some I added a sleep of 100ms, this gave the finalizer thread enough
time to free some JIs.
Finally, the fix itself, the solution is pretty straightforward. Stop scanning chunks and store a singly
linked list of tombstones installed in a given chunk. We build that list by coopting next_jit_code_hash
for this use.
This is safe because chunks are ref counted so we know nothing else could point to those tombstones and
when we copy a chunk into a new one we remove all tombstones.
[Mono.Posix] Fix OSX build break by checking if accept4() is available
It's a Linux-specific extension and not available on OSX or Linux <2.6.28
/cc @steffen-kiess @jonpryor
@monojenkins merge
The messages contain useful debugging information which is unavailable
unless the mono runtime is recompiled. The debugging information is
particularly useful on mobile platforms and in test runs on bots.
The I/O layer trace messages can be disabled on compile time by
defining the DISABLE_IO_LAYER_TRACE preprocessor symbol.
This changes MonoHandleArena from a queue of chunks to a stack of
chunks. More importantly, the MonoHandleArena has an initial chunk that
is allocated on the stack together with the arena. Previously that
initial chunk was at the front of the list of chunks, but now it is at
the end.
Cprop can convert more binops to their _IMM counterpart, which would otherwise not be detected at method_to_ir time, allowing us to do strength reduction.
We allocate a temporary bb where we can emit the optimization of an instruction. After doing strength reduction, we verify if the bb has instructions, in which case we need to replace the original instruction with the optimized code.
The code in mono_arch_allocate_vars would set cfg->ret->dreg to cfg->ret->inst_c0
regardless. This is ok in some cases, but bad in others.
For OP_REGVAR this would do the right thing, dreg should be register that holds
that argument.
For when vret_addr is used, it would modify a var that is not used, so harmless.
For OP_REGOFFSET, inst_c0 alias with inst_offset, which is set to the offset from
RBP to read the var from. So we'd have a dreg set to, sometimes, a negative number.
This was not a problem as after mono_arch_allocate_vars only mono_spill_global_vars
is called and it only checks the dreg in the case of gsharedvt.
The fix is pretty simple, move setting from happening in all cases to only for OP_REGVAR.
The usage of is_gsharedvt_return_value is easier than having a separate storage as supporting
it would require changing all usage sites and this is only needed by start_call.
All incoming arguments in registers must be saved on the stack as the rest of the JIT
expects to be able to load them from their well known places. Without this change, it would
load a random value instead.
This was enough to get mscorlib to FullAOT with gsharedvt enabled.
Makefile.am.in:
Don't exclude GSHAREDVT from the tests.
Removed DYNCALL too, as it's supported too.
mini-amd64.c (mono_arch_init):
Register mono_amd64_start_gsharedvt_call jit icall.
mini-amd64.c:
Fix a typo in the ifdef guarding inclusion of the gsharedvt extension.
mini-amd64.c (mono_arch_emit_outarg_vt):
Implement decomposing of ArgGSharedVtOnStack to a simple store on the stack.
mini-amd64.h:
define MONO_ARCH_GSHAREDVT_SUPPORTED
Export mono_amd64_start_gsharedvt_call
tramp-amd64.c (mono_arch_create_sdb_trampoline):
Add missing mono_profiler_code_buffer_new call.
Add a couple of functions to !ENABLE_GSHAREDVT
aot-compiler.c (arch_emit_gsharedvt_arg_trampoline):
Implement it for amd64 as a pair of GOT loads (same for other archs).
[corlib] Remove left-over files from long ago (System.PAL/Linux)
They are no longer in use and cause confusion as they show up in search results.
@monojenkins merge
mono-context currently uses a mix of MONO_SIGNAL_USE_SIGACTION and
UCONTEXT_REG_RAX #ifs to determine whether ucontext_t is available.
UCONTEXT_REG_RAX is unreliable, and MONO_SIGNAL_USE_SIGACTION is
misnamed because it is nowhere used to gate use of sigaction.
Standardize on a new MONO_SIGNAL_USE_UCONTEXT_T, replacing SIGACTION.
processes.c contains implementations for module loading for several
platforms. Rather than scattering PLATFORM_MACOSX etc through the
file, pick a loader at the top of the file and base #ifs on that. This
makes BSD code clearer and makes it easier to special-case OS variants
where the normal loader is not available.
Bugfixes in mono-rand.c
2 commits:
* Make mono-rand consistent (fixes threadpool on some UNIX systems)
mono-rand.c defines a function interface and provides three implementations (Windows, modern UNIX, base C standard). The clients of this file (RNGCryptoServiceProvider.cs, threadpool-ms-io.c) appear to be consistent in how they use this interface, but the interface is not documented, and the C-only implementation deviates from it (the return value of init is different). The result is that threadpool-ms-io fails an assert when run on a system using the C-only implementation. I documented the interface and fixed the C-only quirk.
* Fix nonuniform distribution of mono_rand_try_get_uint32
mono_rand_try_get_uint32 returns a value in a range min..max inclusive. Due to a math error the ‘max’ value will be returned only one out of every 2^32-1 calls to the function. This is currently harmless because the only current callers of this function (in threadpool-ms-io) will run fine even if their max value is never returned, but it could become a problem later.
mono_rand_try_get_uint32 returns a value in a range min..max
inclusive. Due to a math error the ‘max’ value will be returned only
one out of every 2^32-1 calls to the function. This is currently
harmless because the only current callers of this function (in
threadpool-ms-io) will run fine even if their max value is never
returned, but it could become a problem later.
mono-rand.c defines a function interface and provides three
implementations (Windows, modern UNIX, base C standard). The clients
of this file (RNGCryptoServiceProvider.cs, threadpool-ms-io.c) appear
to be consistent in how they use this interface, but the interface is
not documented, and the C-only implementation deviates from it (the
return value of init is different). The result is that
threadpool-ms-io fails an assert when run on a system using the C-only
implementation. I documented the interface and fixed the C-only quirk.
We were seeing red builds on Jenkins because the NUnit xml was malformed when something bad happened
during the test_op_il_seq_point tests and Jenkins couldn't parse it.
To fix this, write the results into a temp file and only rename to .xml at the end when everything worked.
Additionally, fixed a small typo in cleaning the .so files.
Emulation used to happen at emitting time (method_to_ir). Now it is done after the copy propagation pass, where we propagate immediates and can do strength reduction on some of these ops. Because the call to the native function might imply an inlined wrapper which compiles more IL instructions, we need to run the decomposition pass again.
Mimic the behaviour of its counterpart, mono_op_to_op_imm, since crashing limits the usage of the function and it is the responsability of the upper layer.
By using the referencesource one, we remove a buggy part of the Process class. It will now simply use Stream.{Begin/End}Read, and not a custom implementation that introduces its own bugs.
[bcl] Deduplicate the SocketResponder test helper class.
A number of fixes were made to the version in System but these fixes were never applied to the System.Web.Services version. Merge the two and place the class in the test-helpers directory to avoid this sort of maintenance problem.
A number of fixes were made to the version in System but these fixes were never
applied to the System.Web.Services version. Merge the two and place the class
in the test-helpers directory to avoid this sort of maintenance problem.
We previously followed the spec semantics for these which meant weaker memory
barriers than MS.NET. However, since we've been importing large amounts of
code from the reference sources, it makes sense to switch these methods to
the MS.NET semantics as we don't generally audit the reference source code for
compatibility with our (spec-compliant) memory model. It's safe to assume that
most MS.NET code is written with the assumption of strong memory barriers in
these methods.
This reverts commit b3fa096e5e.
The Abort () call is problematic on many platforms and is no longer necessary
as we properly close the listening socket since 1103abf383.
Conflicts:
mcs/class/System.Web.Services/Test/System.Web.Services.Protocols/SocketResponder.cs
mcs/class/System/Test/System.Net/SocketResponder.cs
Before we only checked if IntPtr.Size == 8, but this returns false for a 32bit Mono running on a 64bit OS.
Added an icall that gets the correct value from the runtime instead.
Note: this currently only whitelists x86_64, aarch64 and ppc64 as 64-bit platforms.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=37414
This is used to Xamarin.iOS/Mac to wrap threadpool work in an NSAutoreleasePool.
Also open up corlib's internals to Xamarin.iOS/Mac so that the linker
doesn't have to be involved when accessing internal API.
A kevent() syscall may "fail" with EINTR if the process receives
a signal before the timeout expires and before an event is placed
in the kqueue; in this case, rather than erroring out, the
syscall should simply be retried.
The second member of struct timespec is actually called tv_nsec
(nanoseconds), not tv_usec (microseconds); this was likely
confused with struct timeval, where the second member is in fact
called tv_usec.
We were seeing persistent hangs in the threadpool-exceptions5.exe test on the Jenkins ARM machines.
Tracing the execution showed the following exceptions:
```
[0x7f65e3284700:] EXCEPTION handling: System.Threading.ThreadAbortException:
[0x7f65e3284700:] EXCEPTION handling: System.Exception: From OnCBFinished
[0x7f65e3284700:] EXCEPTION handling: System.ArgumentNullException: Value cannot be null. <-- from CultureInfo.CreateSpecificCulture()
Parameter name: name
[0x7f65e3284700:] EXCEPTION handling: System.Threading.ThreadAbortException:
exception inside UnhandledException handler:
... program hangs here ...
```
It turns out that the machines don't have the LANG/LC_ALL env vars set, resulting in a null string returned
from get_current_locale_name(). The code then tries to construct a locale from that and throws an ANE which is
normally catched and turned into the InvariantCulture.
However, in the threadpool-exceptions5 test the first time a CultureInfo is created is in the OnUnhandledException
event handler when calling ToString(). This means that an exception is thrown while we are in the unhandled exception
handler and the runtime apparently just stops there (despite the exception actually being catched, probably because of
the thread abort) so we never get to signaling the monitor object -> we hang in Main().
The fix is to not rely on catching the ArgumentNullException when we get a null locale, which is a better approach anyway.
To easily reproduce the issue before the fix, use the following (on Linux only, we don't use LANG for locale on OSX):
```
~/dev/mono/mono/tests$ LANG=C MONO_PATH=../../mcs/class/lib/net_4_x/ ../mini/mono-sgen --trace=N:nothing threadpool-exceptions5.exe
```
Note: If you pass an invalid LANG=foo, the hang is still there as we throw a CultureNotFoundException and hit the same issue.
As the behavior looks like a runtime bug to me I filed https://bugzilla.xamarin.com/show_bug.cgi?id=37547
Setting the priority before starting a Thread doesn't work right now as there's no handle to the native
thread at that point (the current implementation doesn't store the priority, it just tries setting it
on the native thread) so the test would remember the default 'Normal' priority instead of 'BelowNormal',
causing the test to fail later on.
Added an assert to verify that the thread priority can be set before starting and disable the test for now.
Note: variable names 'before' and 'after' were switched in the test, fixed those as well.
This adds supprot to the getline.cs library to provide popup windows with
completion results. There is a new public string property surfaced,
HeuristicsMode, that if set to "csharp" will trigger the completion window
independently of the "tab" character, based on the user input.
This mode triggers completion after the "." character is entered, and will
continue to trigger it as long as you type, or you backspace and the completion
succeeds. The heuristic also ignores scenarios like "1.2", so there is no
compeltion provided after entering numbers by default.
Perhaps we can later introduce a "OnKeyEntered" command that users could
use to provide their own heuristics to control whether to popup the window
or not.
- fixes#37305
- resource name can be general string and using it as path
might throw
System.ArgumentException: Illegal characters in path.
- it would be possible to check for the path's invalid characters in
the resource name. that would cause crossplatform issues
though.
various platforms have different invalid charactes sets for path
names. for example using assembly built on mac with embedded
linker blacklisting xml file, which's name contains characters in
its filename which are invalid on windows. so the windows built
app using that assembly would fail mysteriously, because the
important parts wouldn't be preserved.
thus we use just string comparison
mono is executed using the SCHED_RR policy, we will map POSIX priorities
to .NET priorities. Under this policy a thread running under this policy
which uses the SetPriority API will have its priority remapped and a
call to pthread_setschedparam() made to change its priority.
The following code fragment may be used to run a mono application
without requiring all root privileges other than the CAP_SYS_NICE
capability which enables priority manipulation:
void
runMonoRT(const char *const name, const int policy, char **argv)
{
const pid_t me = getpid();
struct sched_param param;
param.sched_priority = sched_get_priority_min(policy);
if (sched_setscheduler(me, policy, ¶m) == -1)
fprintf(stderr, "sched_setscheduler(getpid(), %s, { %d }): %s.\n",
name, param.sched_priority, strerror(errno));
else
execvp("mono", argv);
}
int
main(int argc, char **argv)
{
uid_t user;
cap_value_t root_caps[2] = { CAP_SYS_NICE, CAP_SETUID };
cap_value_t user_caps[1] = { CAP_SYS_NICE };
cap_t capabilities;
/* Get real user ID. */
user = getuid();
/* Get full root privileges. Normally being effectively root
* (see man 7 credentials, User and Group Identifiers, for explanation
* for effective versus real identity) is enough, but some security
* modules restrict actions by processes that are only effectively root.
* To make sure we don't hit those problems, we switch to root fully. */
if (setresuid(0, 0, 0)) {
fprintf(stderr, "Cannot switch to root: %s.\n", strerror(errno));
return 1;
}
/* Create an empty set of capabilities. */
capabilities = cap_init();
/* Capabilities have three subsets:
* INHERITABLE: Capabilities permitted after an execv()
* EFFECTIVE: Currently effective capabilities
* PERMITTED: Limiting set for the two above.
* See man 7 capabilities for details, Thread Capability Sets.
*
* We need the following capabilities:
* CAP_SYS_NICE For nice(2), setpriority(2),
* sched_setscheduler(2), sched_setparam(2),
* sched_setaffinity(2), etc.
* CAP_SETUID For setuid(), setresuid()
* in the last two subsets. We do not need to retain any capabilities
* over an exec().
*/
if (cap_set_flag(capabilities, CAP_PERMITTED, sizeof root_caps / sizeof root_caps[0], root_caps, CAP_SET) ||
cap_set_flag(capabilities, CAP_EFFECTIVE, sizeof root_caps / sizeof root_caps[0], root_caps, CAP_SET)) {
fprintf(stderr, "Cannot manipulate capability data structure as root: %s.\n", strerror(errno));
return 1;
}
/* Above, we just manipulated the data structure describing the flags,
* not the capabilities themselves. So, set those capabilities now. */
if (cap_set_proc(capabilities)) {
fprintf(stderr, "Cannot set capabilities as root: %s.\n", strerror(errno));
return 1;
}
/* We wish to retain the capabilities across the identity change,
* so we need to tell the kernel. */
if (prctl(PR_SET_KEEPCAPS, 1L)) {
fprintf(stderr, "Cannot keep capabilities after dropping privileges: %s.\n", strerror(errno));
return 1;
}
/* Drop extra privileges (aside from capabilities) by switching
* to the original real user. */
if (setresuid(user, user, user)) {
fprintf(stderr, "Cannot drop root privileges: %s.\n", strerror(errno));
return 1;
}
/* We can still switch to a different user due to having the CAP_SETUID
* capability. Let's clear the capability set, except for the CAP_SYS_NICE
* in the permitted and effective sets. */
if (cap_clear(capabilities)) {
fprintf(stderr, "Cannot clear capability data structure: %s.\n", strerror(errno));
return 1;
}
if (cap_set_flag(capabilities, CAP_PERMITTED, sizeof user_caps / sizeof user_caps[0], user_caps, CAP_SET) ||
cap_set_flag(capabilities, CAP_EFFECTIVE, sizeof user_caps / sizeof user_caps[0], user_caps, CAP_SET)) {
fprintf(stderr, "Cannot manipulate capability data structure as user: %s.\n", strerror(errno));
return 1;
}
/* Apply modified capabilities. */
if (cap_set_proc(capabilities)) {
fprintf(stderr, "Cannot set capabilities as user: %s.\n", strerror(errno));
return 1;
}
/*
* Now we have just the normal user privileges,
* plus user_caps.
*/
runMonoRT("SCHED_RR", SCHED_RR, argv);
return 0;
}
Fix a NullReferenceException in monop when printing a type name in a
namespace somewhere below System while working on a class which does not
have a namespace.
You must now add the -api-style parameter when running mdoc update for a set of classic or unified assemblies.
This was added in order to properly support types and members that exist only in one or the other set.
Previously, there was an assumption that all types and members would be reflected in both classic and
unified versions of an assembly.
Since we allocate los objects in page_size chunks, we randomly slide the objects within this space to have them start at different offsets. This improves caching behavior, especially when traversing multiple los objects at the same time.
When we concurrently drain the gray stack, we check to see if there are any blocks that need evacuation. If there aren't, we can drain the gray stack using functions that don't check evacuation related block information. As we do with the serial collector.
Keeping threads around can cause processes to keep running when they
shouldn't, particularly on Windows when the entry point returns.
This commit licensed as MIT/X11.
We were sometimes seeing the following failure on Jenkins on OSX:
```
Test Case Failures:
1) MonoTests.System.Net.Sockets.SocketTest.AcceptBlockingStatus : System.Net.Sockets.SocketException : Operation on non-blocking socket would block
at System.Net.Sockets.Socket.Accept () [0x00033] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System/System.Net.Sockets/Socket.cs:896
at MonoTests.System.Net.Sockets.SocketTest.BlockingConnect (Boolean block) [0x0003e] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System/Test/System.Net.Sockets/SocketTest.cs:129
at MonoTests.System.Net.Sockets.SocketTest.AcceptBlockingStatus () [0x0001e] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System/Test/System.Net.Sockets/SocketTest.cs:147
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
```
In non-blocking mode accept() is specified to return this error when no client connection is available right now.
It looks like on OSX we don't immediately get that connection.
To fix this issue, we try to accept in a loop for 100ms and only fail when we don't get a connection in this time.
While serializing operations with XmlSerializerFormatAttribute,
attributes xmlns:xsi and xmlns:xsd were not added to the message body.
XmlBodyWriter now adds the missing attributes.
Fixes#34413.
The CreateEncryptor_KeyNull and CreateDecryptor_KeyNull tests tried to verify the previous Mono
behavior of throwing a CryptographicException when a null key is passed to the methods [1].
However, with the move to referencesource this no longer happens since when a null key is passed
a random one is generated instead [2].
The existing tests passed out of pure luck because a different CryptographicException is thrown later on:
```
System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
at System.Security.Cryptography.RijndaelManagedTransform.DecryptData (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, System.Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast) [0x007b5] in /Users/alexander/dev/mono/external/referencesource/mscorlib/system/security/cryptography/rijndaelmanagedtransform.cs:751
at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) [0x000bb] in /Users/alexander/dev/mono/external/referencesource/mscorlib/system/security/cryptography/rijndaelmanagedtransform.cs:336
at MonoTests.System.Security.Cryptography.RijndaelManagedTest.CreateDecryptor_KeyNull () [0x00066] in /Users/alexander/dev/mono/mcs/class/corlib/Test/System.Security.Cryptography/RijndaelManagedTest.cs:233
```
Unfortunately, sometimes it happens with the random key that this exception *doesn't* get thrown as the padding is valid, which makes the test randomly fail on Jenkins.
Deleting those tests is the simplest fix.
[1] https://github.com/mono/mono/blob/mono-4.0.0-branch/mcs/class/corlib/System.Security.Cryptography/RijndaelManagedTransform.cs#L107
[2] https://github.com/mono/referencesource/blob/33edf60ec4d35aba11850872777c8b3a484ca484/mscorlib/system/security/cryptography/rijndaelmanaged.cs#L66
We sometimes see the following exception in the ServiceModel tests:
```
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.ThrowHelper.ThrowInvalidOperationException (ExceptionResource resource) [0x00000] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/external/referencesource/mscorlib/system/throwhelper.cs:99
at System.Collections.Generic.List`1+Enumerator[T].MoveNextRare () [0x00016] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/external/referencesource/mscorlib/system/collections/generic/list.cs:1180
at System.Collections.Generic.List`1+Enumerator[T].MoveNext () [0x00050] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/external/referencesource/mscorlib/system/collections/generic/list.cs:1174
at System.Linq.Enumerable.FirstOrDefault[TSource] (IEnumerable`1 source, System.Func`2 predicate) [0x00041] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/external/referencesource/System.Core/System/Linq/Enumerable.cs:960
at System.ServiceModel.Channels.Http.HttpListenerManager.TryDequeueRequest (System.ServiceModel.Dispatcher.ChannelDispatcher channel, TimeSpan timeout, System.ServiceModel.Channels.Http.HttpContextInfo& context) [0x00016] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpListenerManager.cs:93
at System.ServiceModel.Channels.Http.HttpListenerManager.TryDequeueRequest (System.ServiceModel.Dispatcher.ChannelDispatcher channel, TimeSpan timeout, System.ServiceModel.Channels.Http.HttpContextInfo& context) [0x000ab] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpListenerManager.cs:104
at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00003] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpReplyChannel.cs:154
at System.ServiceModel.Channels.ReplyChannelBase+<BeginTryReceiveRequest>c__AnonStorey0.<>m__0 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00056] in /var/lib/jenkins/workspace/test-mono-mainline/label/debian-amd64/mcs/class/System.ServiceModel/System.ServiceModel.Channels/ReplyChannelBase.cs:129
```
I managed to repro this consistently on the debian-7-x64-1 Jenkins machine by running the test in a loop with the following code for about 15mins:
while [ $? -eq 0 ]; do make check FIXTURE=System.ServiceModel.Dispatcher; done
The problem happens when RegisterListenerCommon/UnregisterListenerCommon is invoked through HttpChannelListener.OnOpen/OnAbort/OnClose which modifies the 'Entries' collection on another thread.
The fix is to add locking around the collection accesses.
This works by using the imt entries in the vtable to store 'imt thunks', which in llvm-only mode, are C functions
which receive an info structure and the imt method and return the real method which needs to be called. Thus
interface calls are compiled into two calls: one to the imt thunk, and one to the method address returned by
the thunk.
We were sometimes seeing the following test failure on Jenkins:
```
Test Case Failures:
1) MonoTests.System.Web.UI.WebControls.WizardStepBaseTest.WizardStepBase_Theme : System.IO.IOException : Sharing violation on path /var/folders/wm/jvn0yl0d39q9sm9glcrhdrmw0000gn/T/tmp672264ba.tmp/App_Themes/Theme1/WizardTest.skin
at System.IO.File.SetLastWriteTime (System.String path, DateTime lastWriteTime) [0x00029] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.IO/File.cs:474
at MonoTests.SystemWeb.Framework.WebTest.CopyBinary (System.Byte[] sourceArray, System.String targetUrl) [0x00098] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System.Web/Test/mainsoft/NunitWeb/NunitWeb/WebTest.cs:364
at MonoTests.SystemWeb.Framework.WebTest.CopyResource (System.Type type, System.String resourceName, System.String targetUrl) [0x00059] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System.Web/Test/mainsoft/NunitWeb/NunitWeb/WebTest.cs:307
at MonoTests.System.Web.UI.WebControls.WizardStepBaseTest.WizardStepBase_Theme () [0x00000] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/System.Web/Test/System.Web.UI.WebControls/WizardStepBaseTest.cs:311
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:295
```
This happens because System.Web.UI.WebControls/ThemeTest.cs copies the WizardTest.skin file to the same App_Themes/Theme1 path in its setup method.
Since we're not interested in any particular theme in WizardStepBase_Theme we fix this by just using a different theme name (WizardStepBase instead of Theme1) so the two test fixtures don't clash.
scripts/babysitter stdout logging improvements
Always print stdout log statements, even when all babysitter features
are turned off, even when the test suite does not support the protocol.
The babysitter script was failing to act as a GNU timeout replacement
because it was delivering signals to the spawned process only, whereas
GNU timeout would create a process group. This lead to a problem where
timeout kills by the babysitter would leave children of the spawned
process still running. To fix this, the babysitter script now follows
the exact steps that GNU timeout does (or as close as is possible in
Python). An attempt at a Windows process-group implementation is also
included, but I have not tested it.
Also changed the 'retry_support' key to 'babysitter_protocol' in the
JSON logs, because it was pointed out the old key was misleading (it
is true even when retries are deactivated).
- Using “iteration” for # of retries instead of ambiguous “passes”
- More clearly document what environment keys are for in-script
- Remove CRASH_RESUME key, make setting always true
- Print explicit human-readable messages for stdout log
The shim/bridge between Mono and the RS implementation of
XmlObjectSerializer, DataContractSerializer, etc. did not properly
invoke the constructor required by ISerializable, as it completely
omitted passing the populated SerializationInfo object.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=37171
We were seeing the following unhandled exception on Jenkins:
````
Unhandled exceptions:
1) MonoTests.System.ServiceModel.OperationContextTest.Current : System.NullReferenceException: Object reference not set to an instance of an object
at System.ServiceModel.Channels.NetTcp.TcpReplyChannel.OnClose (TimeSpan timeout) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System.ServiceModel/System.ServiceModel.Channels.NetTcp/TcpReplyChannel.cs:141
at System.ServiceModel.Channels.CommunicationObject.Close (TimeSpan timeout) [0x0001c] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:140
at System.ServiceModel.Channels.CommunicationObject.Close () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:131
at System.ServiceModel.Dispatcher.ListenerLoopManager.TryReceiveRequestDone (IAsyncResult result) [0x00027] in /media/ssd/jenkins/workspace/test-mono-mainline/label/debian-armhf/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:583
```
This happens because multiple threads can call OnClose() and so `client` can be null.
Fixed this by taking code from HttpReplyChannel.OnClose which handled this already and also fixed the race condition there (the checks need to happen under a lock).
[symbolicate] Make sure MONO_PATH is quoted
Without it we saw a " ./../../class/lib/net_4_x: Is a directory" error in the cygwin build.
@monojenkins merge
Before this commit string value starting or ending with this char was malformed because Trim removed also part of string value
Also fixed recent regression from 42c0a94445 which failed to split arguments if doubleQuote was inside singleQuoute
This was applied for the x86 build in 398c7fc21f.
Moved the check so that it applies to the x64 MSVC build as well.
We were hitting the same assert in mono_vfree() on Jenkins and this fixes it.
For local handles in coop suspend configuartions we need the idea of a
"critical region": a situation when transitioning from GC
Unsafe (running) to GC Safe (blocking) mode constitutes a bug. In
checked builds, we'd like to assert when that happens.
This PR adds MONO_PREPARE_GC_CRITICAL_REGION,
MONO_FINISH_GC_CRITICAL_REGION and MONO_REQ_GC_NOT_CRITICAL.
In regular build, these are nops. In checked build, they use a new
thread-local flag to prevent transitions from running to blocking.
I see two problems with our current automated PR tests:
- Many false negatives— PR tests often fail due to recurring failures
unrelated to the PR
- Information about what failed is awkward to retrieve, scattered in
multiple places
This commit adds a Python script, `scripts/babysitter`. Currently our
Jenkins test suites are each run wrapped with the `timeout` command.
The babysitter acts as a drop-in replacement for GNU timeout but adds
the following features:
- Logs machine readable output about each test suite (as
line-delimited json)
- If the test suite uses NUnit, can detect if a test case failed or
crashed (terminated mono in mid-test), and retry unsuccessful tests
(up to a limit).
The reasoning here is that tests which fail inconsistently are
currently most likely due to one of our outstanding recurring failures
rather than the change made in the PR. Therefore, if a failing test
succeeds on retry, the PR itself is probably valid (although the
failure should be logged and looked at).
In addition to the script itself, changes to NUnit were required in
order to support the retry feature and allow fine-grained logging for
NUnit suites.
Major TODOs:
- Add retry support for our non-NUnit-based test suites
- Save the XML files NUnit produces (since reruns stomp XMLs from
previous runs)
- Add some kind of sensible feature for dealing with timeouts
which uses gsharedvt out wrappers. We invoke the wrappers, and the
wrappers invoke the methods themselves. The advantage of this approach is
that the set of possible signatures for gsharedvt out wrappers is small,
so the set of runtime invoke wrappers needed to invoke them is also small,
thus only a small number of them need to be generated.
Not yet enabled.
System.Environment.SpecialFolder.CommonApplicationData returns /usr/share on OS X 10.11, which is now not a very good place to put things :) This patch changes this folder to /usr/local/share.
This change is released under the MIT license.
This is implemented by generating two wrappers for each concrete signature found in the assembly:
* an 'in' wrapper translates from the normal to the gsharedvt signature
* an 'out' wrapper translates from the gsharedvt to the normal signature
The wrappers take an extra argument, which points to a function descriptor, which is an <addr, arg> pair. They
make a call to ADDR which ARG as an extra argument. ADDR/ARG is either a gsharedvt method with an rgctx
argument, or another gsharedvt wrapper with a function descriptor argument.
All calls made out of gsharedvt methods with a variable signature use the gsharedvt calling convention. If the
callee is also gsharedvt, no wrapper is needed. If the callee is not gsharedvt, the call goes thought
a gsharedvt out wrapper.
Calls made from normal methods to gsharedvt methods go though a gsharedvt in wrapper.
* mini-generic-sharing.c (mini_get_gsharedvt_in_sig_wrapper): New function to generate gsharedvt in
wrappers as IL code.
(mini_get_gsharedvt_out_sig_wrapper): Ditto for out wrappers.
* method-to-ir.c (emit_llvmonly_calli): New helper function to make a call to a function descriptor.
(mono_method_to_ir): Make some indirect calls to MONO_RGCTX_INFO entries using function descriptors.
* aot-compiler.c (compile_method): Generate gsharedvt wrappers for the signatures used by the method.
* aot-runtime.c (decode_method_ref_with_target): Decode gsharedvt wrappers.
* jit-icalls.c (resolve_vcall): Rewrite this so it can handle gsharedvt as well.
(resolve_iface_call): Ditto.
(mono_init_delegate): Ditto.
(mono_init_delegate_virtual): Ditto.
* mini-trampolines.c (mini_create_llvmonly_ftndesc): New helper function to create a function descriptor.
(mini_add_method_wrappers_llvmonly): New helper function to add gsharedvt in/out wrappers, similar
to mini_add_method_trampolines ().
* mini-generic-sharing.c (mini_get_gsharedvt_wrapper): Return the newly added wrappers in llvmonly mode.
(class_type_info): Insert gsharedvt wrappers if needed. Some MONO_RGCTX_INFO types will now resolve to
function descriptors instead of function addresses.
(instantiate_info): Ditto.
Add mono_signature_full_name () helper function.
* debug-helpers.c (mono_signature_full_name): New helper function to stringify a signature, including
its return types as well.
Add beginnings for support for gsharedvt in llvmonly mode. Gsharedvt methods use a calling convention
where everything is passed and returned by ref, even normal types. I.e.
public int foo<T> (T t, int i);
becomes:
void foo (int *vret, T* t_addr, int *i_addr);
Not enabled yet.
* mini-llvm.c: Implement support for gsharedvt.
* ssa.c (mono_ssa_remove_gsharedvt): Same as mono_ssa_remove (), but only remove the
OP_VPHI nodes referencing gsharedvt variables, since they cannot be handled by the
llvm backend.
* mini-llvm.c (mono_llvm_create_vars): New backend function, similar to mono_arch_create_vars ().
* decompose.c (mono_decompose_vtype_opts): Decompose more opcodes in llvm mode so
opcodes using gsharedvt variables don't reach the llvm backend. Some, like OP_LDADDR and
vcalls, still do.
* method-to-ir.c (mono_allocate_gsharedvt_vars): New function to allocate gsharedvt locals,
extracted from mono_spill_global_vars () so it can be used in llvm mode too.
**The Scenario**: `mono-api-info` and `mono-api-html`, together, are a
useful way to track API changes over time, specifically to check for
accidental API breakage:
# Have a "known good" assembly version;
# keep reference/assembly.xml in version control
$ mono-api-info assembly.dll > reference/assembly.xml
# ...then on every new build, check for breakage:
$ mono-api-info assembly.dll > new-assembly.xml
$ mono-api-html reference/assembly.xml new-assembly.xml > assembly-diff.html
$ if grep '<p>Removed' assembly-diff.html > /dev/null ; then \
echo "ABI BREAK IN assembly.dll" ; \
cat assembly-diff.html ; \
fi
This is a very nice workflow, and has been used in Xamarin.Android to
track accidental API breakage for some time.
**The problem** with the above workflow are *implicit* references:
`mono-api-info` will happily look for dependencies in the same
directory as `assembly`, but it will *also* look for assembly
references in various system-specific directories, which is fine...
...until it *isn't* fine, e.g. if you have your own version of
mscorlib.dll that you need to use, *not* the system one, as your own
version may have API differences from the system mscorlib.dll which
are "percolated" into `mono-api-info` output. For example, on
"desktop" profiles, System.Attribute implements
System.Runtime.InteropServices._Attribute, and thus *all* custom
attribute types will have _Attribute listed as an implemented
interface. The Xamarin.iOS and Xamarin.Android mscorlib.dll,
meanwhile, does *not* have System.Attribute implementing _Attribute.
Consequently, changing the mscorlib.dll which is used while processing
an assembly will change `mono-api-info` output, resulting in "API
breakage" which doens't actually exist, because the wrong mscorlib.dll
was used to generate the original API description in the first place.
We need a way to control which mscorlib.dll/etc. are used. This can be
done by allowing specification of the assembly search directories, so
that we can "override" the default system-specific directory location.
**The solution**: Add a `mono-api-info -L PATH` option which adds PATH
to the list of directories that `mono-api-info` will search when
resolving assembly references. This allows "overriding" the system
directory location, as PATH will be searched for mscorlib.dll before
the system directory location.
Additionally, add a `mono-api-info -r ASSEMBLY` option, which will
pre-process ASSEMBLY (and all dependencies) and add the directory
which contains ASSEMBLY to the assembly search paths.
These two options are conceptually similar to `mcs -L` and `mcs -r`.
Finally, while adding support for these options, add support for
`mono-api-info --help`, so that we don't need to use only the man page
to get option information.
Because mono_thread_info_safe_suspend_and_run is locking the suspend lock, we cannot enter the blocking state in any of its callee, as that could lead to a deadlock.
The mcs/class/Microsoft.Build.Tasks make check contains a test whose
goal is to verify that Microsoft.Build.Evaluation.Project can run a
MSBuild code fragment which has an assembly reference. The assembly
referenced in the test is System.Drawing. However on many systems
System.Drawing does not work without special setup. The test should
reference a different assembly (I picked System.Runtime.Serialization)
because ensuring System.Drawing is working isn't relevant to the test.
Add the new 4.5.2, 4.6 and 4.6.1 versions to Microsoft.Common.targets so xbuild doesn't print "warning : TargetFrameworkVersion 'v4.6.1' not supported by this toolset".
Additionally I found a bug in the target where "and" was used instead of "or" which means the condition was never true, fixed as well.
icall for Remoting.RealProxy.GetTransparentProxy can sometimes attempt
to load an invalid class which is not previously loaded. Currently,
this leads to an assert failure in mono_class_proxy_vtable which does
not expect to be getting an invalid proxy class as input. Fix is for
the icall to load the class early and raise on failure.
A test is included.
[HttpConnection] Bug fix: HttpListener's "IgnoreWriteExceptions" property value is ignored when "Expect: 100-Continue" header set in request
That problem was discussed previously here: https://github.com/mono/mono/pull/2300
I want to offer alternative solution now.
In case all threads would wait on a Sleep or a Wait, the monitor thread would simply increase the number of max working thread, up to the point it would overflow the number of max worker threads.
OpenLiveWriter's build files break due to incorrectly parsing (".",",") as three arguments: a . and two " instead of two: a . and a ,
This is the error:
OpenLiveWriter/writer.build.targets: error : Error executing task WriteLinesToFile: Error converting Property named 'Lines' with value '#define FILE_VERSION $(BuildVersion.Replace(".",","));#define PRODUCT_VERSION $(BuildVersion)' to type Microsoft.Build.Framework.ITaskItem[]: Method 'Replace(., , )' arguments cannot be evaluated'
This patch ignores , as a separator character if it is inside ""
The problem is connected with "Listener" property of "context" object. It is null in "GetResponseStream" method, when it's called internally from "FinishInitialization" method of "HttpListenerRequest" class. This problem is possible only in situation when we have "Expect: 100-Continue" header from http client.
This change is released under the MIT license.
We didn't catch the regression that is fixed in a780c52749
because the tests didn't check the actual string or that an OOM is raised.
Add a new test for creating a large string (which works on 64bit Mono) to test a similar
codepath.
The "entries" collection in CacheEntryCollection can be modified while we're enumerating it,
which is not allowed. Add locking in FlushItems() to prevent this.
* Add explicit casts for void pointers and enums.
* Rename 'class' to 'klass' and 'this' to 'this_obj'.
* Avoid other C++ keywords by appending an underscore.
* Change MonoSeqPointInfo to avoid empty struct size warnings.
When a generic derived class has an open constructed base type, we have
to apply the type arguments in order to find the correct base definition MethodInfo.
When the base type of a generic type is an open constructed generic
type, GetBaseDefinition() must take the instantiation into account while
traversing the class hierarchy.
We just reverted a patch that prevented the FileSystemWatcher from
opening too many file descriptors (200). That was a band-aid at one
of the spots that opened too many file, as we would exhaust the number
of files, and later abort when some other operation needed files.
The problem is that OSX defaults to 256 file descriptors, and .NET
applications are file descriptor hungry, in fact, many applications
increase their file handled limit before starting.
This patch on OSX will attempt to increase the number of handles
available to 1024 if the number of handles avaialble is less than
that.
The commit that removed the limit was:
2af882232c
And it was closed to fix:
#28693
The .NET delegate removal semantic is a bit unpredictable. For example, the following (d1 + d2 + d3) - (d1 + d3) is going to return the d123 delegate, and not the d2 delegate as we could have guessed. This is because removal will remove the `list' of delegates as a whole, and not the elements in the list.
The previous implementation would then have a different behavior than the one observed on .NET.
Fixes bug #36646
We've seen that error for quite some time:
> `* Assertion at ../../mono/utils/mono-os-mutex.h:71, condition `res != EINVAL' not met`
It looks like there's a race between `finalizer_thread()` and `mono_gc_cleanup()` around
`reference_queue_mutex`. The source of the problem is, `finalizer_thread` could end up
being alive, although `mono_gc_cleanup()` made some effort to kill it and then destroys
`reference_queue_mutex`, while the finalizer thread is still using it in
`reference_queue_proccess_all()`.
> $ MONO_GC_DEBUG=bridge=Bridge MONO_GC_PARAMS=minor=split MONO_ENV_OPTIONS=--gc=sgen MONO_PATH=/home/lewurm/monoperf/mono/mcs/class/lib/net_4_x ../../mono/mini/mono sgen-bridge-major-fragmentation.exe --optimize=all --debug
> [...]
> [12/08/2015 01:49:31] done
> Shutting down finalizer thread timed out.
> * Assertion at gc.c:867, condition `finalizer_thread_exited' not met
>
> Stacktrace:
>
>
> Native stacktrace:
>
>
> Debug info from gdb:
>
> [New LWP 7623]
> [New LWP 7622]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
> 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> Id Target Id Frame
> 3 Thread 0x417ff430 (LWP 7622) "mono" 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> 2 Thread 0x42573430 (LWP 7623) "Finalizer" 0x0021482a in finalizers_with_predicate (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64, hash_table=0x338f68 <major_finalizable_hash>) at sgen-fin-weak-hash.c:581
> * 1 Thread 0x406c4250 (LWP 7620) "mono" 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
>
> Thread 3 (Thread 0x417ff430 (LWP 7622)):
> #0 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x404371d8 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2 0x00242266 in mono_os_cond_wait (cond=0x346e98 <work_cond>, mutex=0x346e80 <lock>) at ../../mono/utils/mono-os-mutex.h:105
> #3 0x00242c1a in thread_func (thread_data=0x0) at sgen-thread-pool.c:118
> #4 0x40433fbc in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #5 0x405a0b3c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 2 (Thread 0x42573430 (LWP 7623)):
> #0 0x0021482a in finalizers_with_predicate (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64, hash_table=0x338f68 <major_finalizable_hash>) at sgen-fin-weak-hash.c:581
> #1 0x00214894 in sgen_gather_finalizers_if (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64) at sgen-fin-weak-hash.c:622
> #2 0x001fd16a in mono_gc_finalizers_for_domain (domain=0x37a348, out_array=0x42572c70, out_size=64) at sgen-mono.c:546
> #3 0x001ca2a4 in finalize_domain_objects (req=0x47ba68) at gc.c:678
> #4 0x001ca3b6 in finalizer_thread (unused=0x0) at gc.c:730
> #5 0x001a1392 in start_wrapper_internal (data=0x3b1100) at threads.c:713
> #6 0x001a1430 in start_wrapper (data=0x3b1100) at threads.c:760
> #7 0x0027097a in inner_start_thread (arg=0xbed58de0) at mono-threads-posix.c:92
> #8 0x40433fbc in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #9 0x405a0b3c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 1 (Thread 0x406c4250 (LWP 7620)):
> #0 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1 0x4043a396 in waitpid () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2 0x000d1b24 in mono_handle_native_sigsegv (signal=6, ctx=0xbed58a80, info=0xbed58a00) at mini-exceptions.c:2235
> #3 0x00112424 in sigabrt_signal_handler (_dummy=6, _info=0xbed58a00, context=0xbed58a80) at mini-posix.c:218
> #4 <signal handler called>
> #5 0x405248e6 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> #6 0x405330fe in raise () from /lib/arm-linux-gnueabihf/libc.so.6
> #7 0x40535956 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
> #8 0x00277eaa in monoeg_log_default_handler (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, message=0x47bb28 "* Assertion at gc.c:867, condition `finalizer_thread_exited' not met\n", unused_data=0x0) at goutput.c:233
> #9 0x00277dca in monoeg_g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=0x3089b8 "* Assertion at %s:%d, condition `%s' not met\n", args=...) at goutput.c:113
> #10 0x00277e32 in monoeg_assertion_message (format=0x3089b8 "* Assertion at %s:%d, condition `%s' not met\n") at goutput.c:133
> #11 0x001ca786 in mono_gc_cleanup () at gc.c:867
> #12 0x001c1d24 in mono_runtime_cleanup (domain=0x37a348) at appdomain.c:356
> #13 0x0001dec0 in mini_cleanup (domain=0x37a348) at mini-runtime.c:3560
> #14 0x000a672c in mono_main (argc=5, argv=0x358098) at driver.c:2065
> #15 0x00017c4e in mono_main_with_options (argc=5, argv=0x358098) at main.c:20
> #16 0x00017c7e in main (argc=4, argv=0xbed591b4) at main.c:53
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
When the nursery collector is starting and we have a concurrent collection in progress we need to save the global card table entries to the mod union card tables. When doing so we were marking all the mod union cards with either 0 or 1, so we could lose a mark if the concurrent collector tries to mark the same card at the same time, since the concurrent collector now runs concurrent even with nursery collections. We fix this by never marking with 0.
The concurrent mark no longer follows references into blocks that are evacuated. The cardtable is marked instead and all the objects will be moved during the finishing pause.
A small powershell command grabs the version number from configure.ac and uses it during winsetup.bat,
this way we don't have to manually keep them in sync.
Our implementation for System.Net.WebRequest takes proxy exclusions
which are in fileglob (*.example.com) format, and passes them to
System.Net.WebProxy as a "bypass list", which expects Regex format.
This means immediate failure if you have an exclusion with a fileglob,
because WebProxy will throw an exception for bad Regex format.
mono_os_sem_wait() used mono_os_sem_timedwait() before it was defined in mono-os-semaphore.h.
Moved the function to fix the cygwin build.
Also added a missing return statement in mono-os-mutex.h that was causing a lot of warnings.
This reverts commit 761a84f867, reversing
changes made to d565a22e16.
Corlib compilation was broken with the PR in so I'm reverting it until it can be worked out.
Unify the tls implementation across platform using a TlsImplementation structure which maps a potential pthread_getspecific implementation to the corresponding fastpaths. Some implementations might require an additional check of the kuser_get_tls kernel helper.
With C6 we introduced a regression
https://bugzilla.xamarin.com/show_bug.cgi?id=36100, which is caused by
the new static serialization code that we wrote to reuse Microsoft's
serializer.
The code uses Invoke on a series of MethodInfos, and this obscures the
actual code that will be invoked at runtime to serialize certain
objects, causing the following exception:
Attempting to JIT compile method '(wrapper runtime-invoke) <Module>:runtime_invoke_bool_Nullable`1<Decimal> (object,intptr,intptr,intptr)' while running with --aot-only.
This patch makes the simple code work, but a more thorough review of
the code would allow us to remove many of the slow dynamic codepaths
in the serializer.
We opted for that approach to keep the code as similar as possible to the
original.
But the code basically does this:
* Use the XmlFormatGeneratorStatics property InternalSerializeMethod,
whose sole purpose is to delay computing the MethodInfo for the
XmlFormatWriterGenerator's InternalSerialize.
* Use the resulting method in a few places to call the MethodInfo.
The MethodInfo was used because the original code uses Reflection.Emit
to create fast serializers and it needs handles to internal methods,
but our static rewriter does not need it.
We should replace all of those uses, but for now, this patch at least
deals with the 36100 bug.
Discussed this with @lambdageek, we still see some of the tests timeout on Jenkins on the slow armhf/armel workers.
Let's see if bumping the timeouts helps.
As seen in the stacktraces provided in [1],
a thread could modify the Files hashtable
while other thread is enumerating it, so this
can be prevented by using a lock{} in the parts
of the code that access and modify this field.
Patch licensed as MIT/X11.
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=10205
It sometimes fails on Jenkins and google might be doing unwanted things (like redirecting to https),
so we just use example.org instead which hopefully is more stable in that regard.
When collecting candidate properties, look at the generic definition
when throwing away duplicates (hide-by-name-and-signature for properties
should look at generic definition) because inflating method could
conflate distinct signatures to the same one:
class Foo<T,U> {
public T this[T t] { get { return t; } }
public U this[U u] { get { return u; } }
}
The class Foo<int,int> has 2 indexers, not 1. They just happen to have
the same signature because T and U both got replaced by int.
While there were tests included for the [multiassembly option](https://github.com/mono/mono/pull/2127), unfortunately
the tests did not consider starting from an existing repository. So this patch
rectifies the issue by improving a small function to consider the possibility
of a null result. Test that reproduced, and subsequently proves the resolution
is included here.
Add comments, add assert for edge case around signal registration, and
make consistent *always* using atomic get/set for memory shared
between the signal handler and the threads.
* capget is not part of libc (libSystem.dylib) on iOS (or OSX); and
* There's no working Process class to invoke `ping` by itself leading to
[FAIL] DllImportTest.LackOfCapget : System.InvalidOperationException : Process has not been started.
at System.Diagnostics.Process.get_HasExited () [0x00039] in /Users/poupou/git/xamarin/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Diagnostics/Process.cs:184
It's technically possible to implement ping on iOS [1] so the code is
not totally removed from the build (API remains identical to existing).
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=964
/usr/lib/libsqlite3.dylib: 77 symbols
[NOT FOUND] sqlite3_column_database_name
[NOT FOUND] sqlite3_column_database_name16
[NOT FOUND] sqlite3_column_origin_name
[NOT FOUND] sqlite3_column_origin_name16
[NOT FOUND] sqlite3_column_table_name
[NOT FOUND] sqlite3_column_table_name16
[NOT FOUND] sqlite3_key
[NOT FOUND] sqlite3_rekey
note: XI has unit tests that will detect if those symbols are added in a
future version of iOS.
We would have the problem that array->vector != array + sizeof
(MonoArray), which would waste space and also lead to problems in
accessing the array elements.
This patch fixes rule violations found by the first batch of
mempool-reference audit points. The fix is to remove ambiguities in
what image owns or should own any particular generic parameter object.
Previously, there were four options for specifying a
MonoGenericParam’s image:
1. klass/method union in MonoGenericContainer.
2. image field in MonoGenericContainer.
3. image field in MonoGenericParam.
4. No field; “image” passed in to methods in at time of usage.
This patch removes 3 and 4; it preserves the image field from (2), but
merges it into the union from (1). It adds a new field,
“is_anonymous”, to indicate when the image member of the union should
be used. The single source of the “owner" union now always points to
the generic param’s image, directly or indirectly.
A further change is made to support this: Previously, containers were
optional, and the presence or absence of container was incidentally
used to determine whether an object was a MonoGenericParam or a
MonoGenericParamFull. Now, containers are mandatory for all
MonoGenericParams, so a new “is_small_size” bit in the container
tracks this information.
This change opens some sources of risk:
⁃ Code which creates MonoGenericParams now must make sure to
allocate a container early.
⁃ Code that accesses the MonoGenericContainer::owner field
must all be correctly updated to look for is_anonymous or there will
be a crash. (Much of this code would have previously used a NULL
container as the flag for “anonymous”).
⁃ Code that casts MonoGenericParams to MonoGenericParamFull
must use the is_small_size field.
⁃ Some generic containers will now exist with NULL type_params
and 0 type_argc. Code which reads the type param array must either not
receive objects from these containers, or must handle it.
⁃ If code by users of the embedding API exists which somehow
depends on particular behavior related to the now-unused image
argument of mono_class_from_generic_parameter, it will break.
I have audited code around, at least, the first three of these cases
and fixed what problems I found.
- New audit-point macro for atomic writes
- Check imageset list directly (instead of indirectly through image
list)
- Check dynamic images and have special rules for them (for the time
being we will allow dynamic images to point to non-dynamic ones
without auditing, long-term this is a FIXME)
- Print image names out with more detail (image sets get full names)
- Remove some #if CHECKED_BUILDs
- New g_assert_checked asserts for checked builds only
The following general-purpose enums and exception helpers were moved
here from Mono.Security.NewTls.Interface, which is going away.
* Alert, CipherAlgorithmType, CipherSuiteCode, ExchangeAlgorithmType,
HashAlgorithmType, TlsConnectionInfo, TlsException, TlsProtocolCode,
TlsProtocols.
* TlsConnectionInfo has been merged into MonoTlsConnectionInfo.
Note that you may not use 'using' statements for both 'Mono.Security.Interface'
and 'System.Security.Authentication' due to conflicting types. Use a prefix such as
"using MSI = Mono.Security.Interface;" or "using SSA = System.Security.Authentication;"
if you need both namespaces.
(cherry picked from commit af05134433)
Re-introduce Thread.Abort/Suspend/Resume so that we don't have to
create separate PCL profiles.
Instead make them throw a NotSupportedException, and add an
[Obsolete (error: true)] attribute so that these methods can't
be used from source.
There is a race condition in signal.c where a signal-waiting thread
can close the pipe and then the signal handler can try to write to it.
Introduce a locking mechanism to prevent this. Separate mph_int_set
and mph_int_test_and_set to support the lock code.
Switching to blocking should only be done when actually blocking (ex: mutex
lock, sem wait, cond wait, blocking syscalls), and not around a whole
section. This means that blocking can only be a leaf state, and we
should not switch from blocking to blocking mode again.
Most errors and warnings from c# compiler are reported in one line in format "FILE(LINE, COL): warning CS1234: Text".
But some warning span multiple lines:
/tmp/3fb23cc0/634d2022.0.cs(22,29): warning CS0114: `NS.B.F()' hides inherited member `NS.A.F()' ..
/tmp/3fb23cc0/634d2022.0.cs(16,29): (Location of the symbol related to previous warning)
Previously the second line was considered to be an internal compiler error since it doesn't match the error regex.
This seems to be only type of messages that span multiple lines.
This fixes#35980
If a attacker injects invalid content into the "host" header field like
"><script>alert(123)</script>
this content is now html encoded.
This prevents XSS attacks.
This reduce the size of System.dll and also removes some Win32 specific
p/invokes.
They cause native linking issues for tvOS as Xcode 7.1 does not allow
`-u` in bitcode-enabled builds so we have to call directly the symbols
and it must exists at (native) link time.
**Incomplete** (some remains) but enough so we can run LinkSDK and
System.Data unit tests apps (the managed linker can remove the rest)
In response to various customer queries, wondering why various libraries could not be found in our documentation. The libraries being added to the doc set are:
- System.Json
- System.Net.Http
- System.Runtime.Serialization
- System.ServiceModel
- System.XML.Linq
This reduce the size of System.dll and also removes some Win32 specific
p/invokes.
They cause native linking issues for tvOS as Xcode 7.1 does not allow
`-u` in bitcode-enabled builds so we have to call directly the symbols
and it must exists at (native) link time.
This cause native linking issues for tvOS as Xcode 7.1 does not allow
`-u` in bitcode-enabled builds so we have to call directly the symbols
and it must exists at (native) link time.
Xamarin.iOS and Xamarin.Mac uses this enum, and changing the values
causes us to break.
For Xamarin.Mac apps that do not bundle their mono, this would break
when the user updated their system mono.
This makes it easier to force Cygwin-brokered test suite runs to use MSVC builds of Mono, i.e. by overriding MONO_EXECUTABLE to .../msvc/bin/Win32/mono-sgen.exe
Some of the profiler projects are no longer needed since the corresponding pieces were removed.
Update sln to only build required projects by default and update configurations in projects.
Fixes the Win64 build on Jenkins.
If the reader execution in the try block fails, reader will stay null, but we later try to access reader.RecordsAffected. This raises a null reference exception which hides the original exception, making this particularly hard to detect.
Instead of having to use MONO_PREPARE_BLOCKING/MONO_TRY_BLOCKING in the
user code, we do it directly in this new mutex and semaphore coop API.
Using the mono_os_* locking functions should not be the default case, as
that could lead to longer than necessary time to suspend, and even in
certain cases to deadlocks.
Mono create_allocator accesses libgc private *_freelists directly.
These fields have been grouped together into struct
thread_local_freelists in BDWGC v7.
This commit does code refactoring to match the structure of external
GC regarding *_freelists.
* Move GRANULARITY, NFREELISTS (used by *_freelists) out of
GC_Thread_Rep.
* Move ptrfree_freelists, normal_freelists, gcj_freelists from
GC_Thread_Rep to struct thread_local_freelists.
* Add tlfs field (of type struct thread_local_freelists) to
GC_Thread_Rep.
* Update pthread_support.c and boehm-gc.c code accordingly.
Mono-specific libgc functionality has been recently added to BDWGC.
This commit changes API of included Boehm GC to match that of the
external GC.
* Renaming:
- GCEventType -> GC_EventType
- GC_get_restart_signal -> GC_get_thr_restart_signal
- GC_toggleref_register_callback -> GC_set_toggleref_func
- GC_set_finalizer_notify_proc -> GC_set_await_finalize_proc
* Replace global variable with a setter:
- GC_notify_event -> GC_set_on_collection_event() (acquires GC lock)
* Foreign threads registration:
- add GC_allow_register_threads() (no-op for GC v6, called by
mono_gc_base_init)
- add GC_SUCCESS, GC_DUPLICATE, GC_NO_MEMORY, GC_UNIMPLEMENTED macros
- add struct GC_stack_base
- replace GC_thread_register_foreign(void*) with
GC_register_my_thread(struct GC_stack_base)
- define GC_register_my_thread also for Solaris and Win32
- GC_register_my_thread returns GC_DUPLICATE if already registered
* Toggle-ref callback declaration change:
- add GC_ToggleRefStatus (enum type)
- callback returns GC_ToggleRefStatus instead of int
* Toggle-ref addition status checked:
- GC_toggleref_add return type changed from void to int
- GC_toggleref_add returns GC_SUCCESS or GC_NO_MEMORY
- static ensure_toggleref_capacity() returns FALSE if GC MALLOC failed
- call g_error() if GC_toggleref_add failed
* Conditional public function declaration:
- wrap GC_DllMain declaration into ifdef GC_INSIDE_DLL
- define GC_INSIDE_DLL prior to include gc.h
On OS X and Mobile, the X509Chain will be initialized with all the
certificates from the X509CertificateCollection, but not actually
built.
ServicePointManager.ServerCertificateValidationCallback can use the 'chain' argument
to get all the certificates from the server.
(cherry picked from commit 73e37f0ecf)
This fixes a regression where ServicePointManager.ServerCertificateValidationCallback
was not called when using WebClient.
(cherry picked from commit 5022db1956)
If the SslStreamBase is disposed while a read or write operation is in progress, the asynchronous I/O state is never marked as completed, resulting in synchronous Read/Write calls never returning. To fix this, the internal I/O callbacks should mark the asynchronous operations as completed, possibly with an error, even if the SslStreamBase is disposed.
We were seeing some random test failures on Jenkins with our Dataflow implementation.
Replacing it with Microsoft's CoreFx version fixed those and makes us more compatible.
While we'd ideally not ship this assembly at all with Mono (it doesn't ship with .NET),
we shipped it in the past and as such people might rely on it so we can't remove it.
The CoreFx commit this version was taken is 905a1940bcda0afdca2f14ceb2b0161ebc4d1d02.
* TestDriver.cs Add `TestTimeout` utility to check how much time CI
allocated for the test.
* sgen-domain-unload-2.cs print warning if running low on time.
* sgen-new-threads-dont-join-stw.cs print progress dots and a warning
when running low on time.
* ICertificateValidator.SelectClientCertificate(): return bool, indicating whether
a certificate has been selected or the caller should use its default
implementation (if it has any).
* ICertificateValidator.ValidateCertificate(): new common function for both client
and server certificates; replaces ValidateChain() and ValidateClientCertificate().
* MonoTlsProvider.InvokeSystemCertificateValidator(): return bool, indicating whether
the certificate has been validator or the default system validator should be used.
(cherry picked from commit 42421eec1e)
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=31432
In a few android devices TimeZoneInfo.ParseTZBuffer was failing ito
parse Europe/Moscow with:
at System.ThrowHelper.ThrowKeyNotFoundException ()
at System.Collections.Generic.Dictionary`2[System.Int32,System.String].get_Item (Int32 key)mscorlib/system/collections/generic/dictionary.cs:176
at System.TimeZoneInfo.ParseTimesTypes (System.Byte[] buffer, Int32 index, Int32 count, System.Collections.Generic.Dictionary`2 abbreviations) [0x0002f] in mcs/class/corlib/System/TimeZoneInfo.cs:1293
at System.TimeZoneInfo.ParseTZBuffer (System.String id, System.Byte[] buffer, Int32 length)
It was failing while getting the third index [3] from] from abbrevs.
abbrevs added values were the following:
[0, MMT]
[1, MT]
[2, T]
[4, MST]
[5, ST]
[6, T]
[8, MDST]
[9, DST]
[10, ST]
[11, T]
[13, S]
[15, M]
[17, MSK]
[18, SK]
[19, K]
[21, EET]
[22, ET]
[23, T]
[25, MSD]
[26, SD]
[27, D]
[29, EEST]
[30, EST]
[31, ST]
[32, T]
Index 3 was not added because its values is an empty string, as indexes
with empty values are used in this case we are now adding them to the
abbrevs dictionary.
Fixes#31432
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=31432
Tests TimeZoneInfo.ParseTZBuffer with Europe/Moscow data that thrown
at System.ThrowHelper.ThrowKeyNotFoundException ()
at System.Collections.Generic.Dictionary`2[System.Int32,System.String].get_Item (Int32 key)mscorlib/system/collections/generic/dictionary.cs:176
at System.TimeZoneInfo.ParseTimesTypes (System.Byte[] buffer, Int32 index, Int32 count, System.Collections.Generic.Dictionary`2 abbreviations) [0x0002f] in mcs/class/corlib/System/TimeZoneInfo.cs:1293
at System.TimeZoneInfo.ParseTZBuffer (System.String id, System.Byte[] buffer, Int32 length)
This enables usage of JitHelper.UnsafeEnumCast<> with enums smaller than int.
This changes enables the intrinsic for int conversions of same reg class.
The coreclr-compile-tests would throw a shell error "Argument list too long" on OSX due
to the huge amount of tests passed to make in one go. The same happened for make clean.
We now compile and clean the three sets of coreclr test sources separately to avoid this.
Additionally, bump the coreclr reference and include a new test that was added upstream.
We were seeing random Dataflow tests fail on Jenkins from time to time.
I managed to reproduce the BroadcastBlockTest.CloningTest and WriteOnceBlockTest.CloningTest
fail on my machine by generating CPU load and running those test in a while(true).
Bumping the timeouts from 100ms -> 1s fixed the flakiness for me, so I went ahead and
opportunistically bumped all such small timeouts throughout the testsuite.
The `Sort<TKey, TValue> (TKey [] keys, TValue [] items, IComparer<TKey> comparer)` overload in Array throwed when keys.Length != items.Length.
However, on .NET you can pass in a keys array with fewer elements than in items and it'd sort the elements in items up to that point.
Fixed our implementation by only throwing when keys.Length > items.Length.
This was revealed by the coreclr/tests/src/CoreMangLib/cti/system/array/arraysort12.exe test.
I decided to port the test into our testsuite so we catch regressions earlier.
Roslyn used xunit-style xml by default, we hack the cibuild.sh script to output nunit-style
for now until we get an option upstream.
This allows us to catch those test results in Jenkins.
In addition to fixing the runtime, fix the test suite to correctly detect this problem.
This popped up because, for some reason, those two types have hashcodes that
don't return identity on reference sources.
If we force the finish of the concurrent collection, instead of waiting for the concurrent mark to finish followed by the finishing mark&sweep, we immediately stop the worker thread, change its object operations and drain the remaining gray stack, also following references to the nursery. This way we avoid the cardtable overhead in cases where there are a lot of references from heap to the nursery.
It tried to use CoreCLR but that doesn't work on Jenkins since some of the dependencies are missing there.
We actually want to ensure it's always using Mono so we can catch errors on our side, so we need this
hack until we have an option in the script upstream.
Remove an array idiom which both broke the GCC build (apparently Clang allows some things with variable-length arrays in "gnu99" mode which GCC does not) and wasn't right anyway (there was a typo which caused a read past end of buffer).
We now use our own archive with the few additional pieces that MS added to their copy of
Mono used by Roslyn CI instead of downloading their archive.
Also bump coreclr to get a fix for the GC test that would fail on Jenkins in.
This is intended to be used by MonoTlsProvider implementations; the ChainValidationHelper
now knows about the provider (unless it's invoked directly from user code).
(cherry picked from commit 2dbf0a0ed1)
With this mtouch will show:
> error MT2001: Could not link assemblies. Reason: Failed to process XML description: /work/maccore/master/maccore/tests/link sdk/extra-linker-defs.xml
> Failed to process XML description: /work/maccore/master/maccore/tests/link sdk/extra-linker-defs.xml
> Failed to resolve assembly: 'MonoTouch, Culture=neutral, PublicKeyToken=null'
instead of:
> error MT2002: Failed to resolve assembly: 'MonoTouch, Culture=neutral, PublicKeyToken=null'
which is a lot less helpful.
Mono.Net.Security/LegacySslStream.cs is the only file which should
reference the 'Mono.Security.Protocol.Tls' namespace.
(cherry picked from commit 43307ff086)
This allows us to customize test behavior that is different on Mono on purpose.
Sorting the files provides a much nicer output in the test runner.
Bump CoreCLR to include changes from 20789373370fe8f72dde01ad7ffc0816f618197c
We do not need a 'System.IO.Compression' facade assembly because we
already have a regular assembly with the same name, which contains all
the classes from here - either as actual implementation or as type-forwarders.
(cherry picked from commit fff7805794)
In particular:
- Issues with cref formatting for subtypes.
- Added support for Explicitly implemented members.
- Added support for generic member parameters.
A number of unit tests were added to verify this functionality.
The size of the gc_stackdata buffer was not a multiple of 8 bytes on 64-bits, so sgen_conservatively_pin_objects_from () would address invalid memory on its last iteration, as it progresses down the buffer by pointer size. The function also expects memory references to be aligned.
This is still not a guarantee that we are aligning memory references properly as two misaligns will still give the right size, but we can catch some of them here. Proper sizing is a smaller issue because we could always pad.
The problem happens when a thread is left suspended by mono-threads and sgen tries to suspend it.
The signal mask of sigsuspend call in mono-threads didn't include sgen's suspend signal, making
it impossible for sgen to suspend it.
The fix is to include sgen's suspend signal into mono-threads sigsuspend mask. This way sgen
suspend can nest over mono-thread's.
The opposite is not needed as sgen never leaves threads suspended when it finishes and it holds
the suspend lock the whole time, making it impossible for mono-threads to initiate any suspend operation.
In order to achieve this, when asked to do a major collection, we finish the concurrent collection immediately, regardless if we are explicitely requested or not to wait_to_finish.
On Jenkins we get a "make: execvp: /bin/sh: Argument list too long" error while executing the coreclr-runtest-basic target.
Since the shell doesn't like echoing our (very) big list of CoreCLR tests in one go, we need to split it up into chunks of
100 entries to hopefully avoid that.
This makes serialization consistent with deserialization (where the InnerException is already retrieved as System.Exception explicitly).
Additionally it may allow serializers more options in serializing inner exceptions (i.e. fall back to System.Exception if the specific exception is not serializable itself).
We don't need this here anymore since it moved to System.Runtime.Serialization.dll in
.NET 4.5 and 54f7c82d2b.
This copy is just confusing and not even built anymore, so delete it.
This is currently only supported on iOS and Mac (check 'CertificateValidationHelper.SupportsTrustAnchors')
and will pass a set of trusted root certificates to the OS validation function. Use for self-signed
certificates instead of providing a custom callback.
A bunch of tests were added in the upstream repo, add those to the source listings.
There's a suite of tests in the CoreMangLib folder which are mostly testing corelib,
added a separate target "coreclr-runtest-coremanglib" for it and renamed the existing
target to "coreclr-runtest-basic".
Build and link all cs testcases with the coreclr-testlibrary.dll which enables a
few more tests to run.
The actual validation code now lives in 'SystemCertificateValidator', whereas
'ChainValidationHelper' contains all the glue code which deals with 'ServicePointManager',
user-supplied callbacks, etc.
To do this, set the MONO_TLS_PROVIDER environment variable to either "default",
"newtls", "oldtls" or the full assembly-qualified type name of the MonoTlsProvider
(in this case, the class needs to derive from MonoTlsProvider and have a public
parameterless constructor and it needs to be installed in the GAC).
The idea is to get the address of the top of the stack, so passing a
value is meaninless. This was a typo. It also fixes libtest.c which
used an outdated signature.
This new method lets you create a 'HttpListener' and specify the X509Certificate2 to use
(so you don't have to put it into ~/.config/.mono/httplistener) as well as an optional
'MonoTlsProvider' and optional 'MonoTlsSettings'.
In SslStream.AuthenticateAsServer(), setting 'clientCertificateRequired' to true will ask for a client
certificate and - if the client does not provide one - invoke the 'RemoteCertificateValidationCallback'
with null and 'SslPolicyErrors.RemoteCertificateNotAvailable' - but resume the handshake if this callback
returns true.
The double.Epsilon value represent the smallest value that double is
able to represent. And that is diferent from the machine epsilon, which
is the smallest difference between two double values.
The second is equal to 2 ^ (-52), because there are 52bits to
represent the mantissa in a IEEE 754 double.
The race only happens on Posix with unified suspend.
The chain of event is the following:
- T1 suspend the world (calls `sgen_unified_suspend_stop_world`)
- T2 is async suspended
- T2 execute suspend_signal_handler:
- T2 finish async suspend
- T2->suspend_can_continue is set to FALSE as the thread is detaching
- T2 notify T1 via `mono_threads_notify_initiator_of_suspend`
- T2 is yielded (via the OS scheduler, or forced via a `sleep`)
- T1 returns from `mono_threads_wait_pending_operations`
- T1 does its stuff
- T1 restart the world
- T1 (or any another thread, except for T2) suspend the world (it does not
necessarily uses `sgen_unified_suspend_stop_world`)
- T2 is async suspended: incrementation of T2->suspend_count to 2, but it is
not signalled because it is already in ASYNC_SUSPENDED state
- T2 wakes up (for any reason) and execute compensation <- ERROR: suspend_count != 1
This race can happen because when async suspending, we notify the
suspendor before executing the compensation, in this case for a thread
which is detaching. By simply post-ponning the notification, just after
the compensation, we eliminate this race.
This shaved about 30 seconds off the time until the test-runner started on my machine compared to
passing the tests in the command line directly (we have *lots* of CoreCLR tests).
The test-runner crashed sometimes when running the CoreCLR testsuite with the following exception:
"System.ArgumentException: '', hexadecimal value 0x0F, is an invalid character." when trying to
emit the NUnit xml.
The stacktrace of one of the failing tests contained the character 0x0F which is invalid in XML,
we now filter out those invalid chars.
- this way the following xml will match all the types in the assembly
MyAssembly
<linker>
<assembly fullname="MyAssembly">
<namespace type="*" />
</assembly>
</linker>
before this change it will only match the types defined in main
module and none of the nested types
Some of the tests have both a test.il and test.cs in the same folder, which means we would
generate an .exe with the same name and overwriting one of them.
This is avoided by appending a _il suffix to the IL tests.
CHECKED_METADATA_WRITE_PTR - Use this to assign a pointer when it points from memory hosted by an image or imageset to another.
CHECKED_METADATA_WRITE_PTR_LOCAL - Use this to assign a pointer when it points from memory noted by an image or imageset to itself.
CHECKED_METADATA_WRITE_PTR_EXEMPT - Use this to assign a pointer in memory hosted by an image or imageset, but which the audit feature cannot correctly check at this time (this is the case when an image must contain a pointer to something that is not located in a mempool, such as a constant string). This is so that at least we know where these exceptions to the audits are.
In non-checked mode, each of these is a simple assignment. In checked mode, they walk the graph of references between images to make sure invariants related to ref_count are not violated.
`size_t` is unsigned, so using `-1` by default will always impose a
limit, and make binary protocol file names numbered, which is not
usually what you want or expect.
This would lead to leaked handle, and that would be visible when running
the test suite on Jenkins, with a "Too many open files" exception.
That also reverts the behavior to before 789808b507
This is part of the continued cleanup work for the mempool reference auditing feature. Separating the hashes prevents redundant work when iterating over hashes, something the reference auditor does a lot. External interface to image.c still behaves the same.
The behavior of the WebHttpBinding in case of SecurityMode ==
WebHttpSecurityMode.TransportCredentialOnly must use the http scheme, in
order to allow authentication of the client based on the standard HTTP
headers, like WWW-Authenticate: Scheme realm="" and so on. This behavior
is the default in .net;
We want to use the binary protocol to measure GC pause times in the
benchmarking suite. To make that easier and be able to go back and
(re-)benchmark older versions, we'd like the binary protocol format to
be backwards compatible.
There was still a race for the last processes, where we could not
correctly wait for them to exit, before closing the StreamWriter for
stdout and stderr.
The crash only occurs on Posix platforms.
The assertion error in `mono_threads_begin_global_suspend`
"pending_suspends != 0" arised because of a unproper use of the syscall
abort mechanism.
The signal would have to be send out of a global suspend context, but
in case the call to `mono_threads_pthread_kill` would fail in
`mono_threads_core_abort_syscall`, then we would add a pending
operation, thus increasing the `pending_suspends` counter. But as we
wouldn't call for `mono_threads_wait_pending_operations`, we would never
have a chance to decrement this counter. So next time we would try to
call `mono_threads_begin_global_suspend` we would hit this assertion
error.
The fix is simply to call the abort from inside a global suspend
context, and making sure to wait on pending operations afterwards.
This breaks Linux, because a shutdown will cause a recv/recvfrom on the
remote host to return without an error.
This would trigger in System.Net.Sockets.SocketTest.ConnectedProperty.
mono_threads_prepare_blocking, mono_threads_finish_blocking,
mono_threads_reset_blocking_start, mono_threads_reset_blocking_end all
take a stack pointer argument. Update the native and managed wrappers
to pass it to them.
- Rework in mempool.c: Have USE_MALLOC_FOR_MEMPOOLS work by tweaking existing rather than using an entire different path. Remove unused mempool field “rest”, stop assigning fields “pos” and “end” where unused. Removed unused function mono_mempool_empty. Fix off-by-one bug (>= vs >) in mono_mempool_contains_addr.
- Fix misc bug spotted by Rodrigo in aot-runtime.c: image of MonoGenericParam was always being set to corlib instead of proper image
- Add comments and replace inline constants with named constants in various places
1. Use the System.TypeSpec parser to parse components of the type name,
then call out to native code to apply the modifiers, as before.
2. TypeName and TypeIdentifier abstractions to hide display name / internal name strings
Update ModuleBuilder and TypeBuilder to use TypeIdentifier and TypeName
internally in all places that don't need to talk to native code.
3. Change ModuleBuilder.name_cache from Hashtable to Dictionary<TypeName, TypeBuilder>
This was functionality that changed mid-way through an implementation, and
was never removed. The underlying functionality changed direction and is now
a part of the `mdoc update`, and doesn't work in any way as originally planned ...
so I'm marking it as obsolete, and will completely remove it in a future release.
[merged by rodo from monodroid - the mentioned test is not included here]
Fixes: https://xamarin.desk.com/agent/case/35534
Commit 5ca3cb4b wasn't propertly tested (doh!).
The problem with Aes and AesCryptoServiceProvider is that there's a
cross-assembly dependency: Aes is in mscorlib, while
AesCryptoServiceProvider is in System.Core. The
PreserveCrypto.ProcessSystemCore() method didn't properly handle this
setup.
Fix the linker, and add a test to Hello.
WARNING: The test sucks. In particular, to "test" this fix you need to
run the app and then grep the `adb logcat` output for:
# DownloadDataTaskAsync complete; status=RanToCompletion; exception=
If the status is anything else, or the exception is non-null, it's
busted.
FURTHERMORE, it's only useful in Release apps, not Debug apps.
(No linker in Debug apps.)
The reason for the need to grep is that there's no sane mechanism to
wait until the Task has completed before checking the status of the
Task, and throwing an exception from within the continuation wouldn't
do anything.
FIXME: The real problem is that Task.Wait() blocks forever and never
returns. This needs further investigation.
https://bugzilla.xamarin.com/show_bug.cgi?id=12681
- when more than one "preserve" attribute is set (see IsPreservedAttribute),
make sure we handle all of them and not just last one
- fixes https://bugzilla.xamarin.com/show_bug.cgi?id=19391
- example code which suffered from this issue:
[Preserve(AllMembers=true)]
[DataContract]
public class TestClass
{
...
}
later when a method from this class was queried using the reflection,
it wasn't available as it was linked out
The acceptance-tests folder wasn't getting included in the tarball, adding the folder
to the SUBDIRS in the top-level Makefile.am fixes this. Additionally, EXTRA_DISTFILES
doesn't work outside of mcs/ so rename it to EXTRA_DIST.
Since we don't want a top-level "make check" to recurse into acceptance-tests just yet,
rename the check target to check-full so nothing gets run by default.
This commit integrates the CoreCLR, Roslyn and ms-test-suite (which is Xamarin-internal, but should just print a warning for someone without access).
Running the tests suites can be done with 'make check-coreclr', 'make check-roslyn' and 'make check-ms-test-suite', or all of them with make check in the acceptance-tests folder.
They won't run on a top-level make check though, which is intended until every test suite is fixed.
This uses a custom submodule system that doesn't rely on git submodules (which are annoying when the repo is large and we only need them for tests). Instead we can fetch repositories on demand.
History (before squashing into smaller chunks):
- Rename check-{testsuite} make target to validate-{testsuite}. The former can be easily confused with the "check" target that runs the tests.
- Move acceptance-tests Makefile to automake, so we can avoid hardcoding paths
- Copy System.Console facade to CoreCLR IL test directories. Otherwise most of the IL tests fail as they rely on the System.Console assembly reference.
- Update README.md to mention the tests
- Use $(MAKE) instead of plain make and pass -C instead of cd'ing to directory
- Exit when the repository can't be cloned and print a note about the ms-test-suite repo. The ms-test-suite repo is Xamarin-internal so it'll error out in public.
- Integrate MS CoreCLR runtime tests into Mono's test infrastructure
- Get rid of intermediate msg file and pipe straight to git
- Rename README to CONFIG since we don't store metadata in readme.md anymore
- Fix comparison in versions.mk to be sh compliant. It relied on bashism before.
Build the two libraries Mono.Security.Providers.DotNet and Mono.Security.Providers.NewSystemSource just like
every other class library instead of duplicating the logic into the subfolders.
This also fixes "make dist".
The issue can arise if the Process is going to run faster than the call
between Start and BeginOutputReadLine/BeginErrorReadLine. To increase
the chance of the race happening, insert a Thread.Sleep (1000) between
Start and BeginOutputReadLine/BeginErrorReadLine.
This race exists because the WaitForExit in the background thread in the
Process, will first wait the process to finish, then try to wait for the
output to finish, but it will be null as BeginOutputReadLine has not been
called yet, then try to wait on error (same issue as output), and finally
call the Exited event, which will close the stdout and stderr TextWriter.
The call to BeginOutputReadLine and BeginErrorReadLine will finally happen
(which is still correct as the output and error of the Process has not
been closed yet), then calling the OutputDataReceived and ErrorDataReceived
callbacks which will try to write to the closed stdout and stderr
TextWriter.
Simply enqueuing a ThreadPool work item, which will wait for the process
to exit manually, simply resolve this issue, as we guarantee that the
call to WaitForExit happens after BeginOutputReadLine/BeginErrorReadLine,
and is not racing with it.
Computing the X509Chain is not supported on OS X and iOS because the OS does
not give us access to the root certificates - this would throw an Exception
at runtime.
(cherry picked from commit 44d34a17c1bf305e39bcfc277cc7add48560d899)
(cherry picked from commit 504328ec43d9d917c1f140ecf89f253ab2be39ec)
The original rename in c4aa103e48 was done by accident,
the rest of the codebase still used the old name so disabling AOT didn't actually work.
Renaming this back to the original name and fix a few instances where methods weren't added
in the DISABLE_AOT case.
This allows a `--enable-minmal=aot` build to succeed and actually work.
The problem happens because thread_interrupt runs in the debugger thread (suspend initiator).
thread_interrupt then calls mono_thread_state_init_from_monoctx which collects the state of the current thread
which is definitely not what we wanted.
There used to be some compensation code after calling mono_thread_state_init_from_monoctx to actually init the state properly.
This is why it worked, at all.
Thew problem is that the debugger thread now might not attach at all under some circumstances, which would trigger
mono_thread_state_init_from_monoctx to return FALSE and trip into the assert.
The fix is to simply not use mono_thread_state_init_from_monoctx at all, just copy the state from data. This fix has the
bonus of getting the domain correctly.
Before .NET 4.5 dots were trimmed from the end of uri path segments.
In .NET 4.5 IRI parsing is enabled by default and dot are no longer
trimmed from the end of path segments.
This commit disables dot trimming when Uri.IriParsing is true.
Fixes#29864.
* Removed Mono.Security.Protocol.Tls.ValidationResult, use
Mono.Security.Interface instead. This is an API breaking
change, but you can fix your application by simply adding
the new using directive.
The rationale behind this is to avoid a dependency on the
Mono.Security.Protocol.Tls namespace when building and
related classes when building the new TLS stack.
* Enable ChainValidationHelper in WebConnection, LegacySslStream
and SslStream.
The custom conditionals that were previously required for the
mobile profiles are now gone as ChainValidationHelper already
handles that internally.
* This is an internal API, the public front-end is
Mono.Security.Interface.CertificateValidationHelper.
CertificateValidationHelper can be used to access the system
certificate validator on OS X and iOS; we do not compute the
X509Chain on these platforms as it's not supported by the OS.
(cherry picked from commit 9713f35878dd90fe87423b5052b4f82ddc0b87f0)
This will be removed again laster, but allows us to do a two-step switch
by first switching to the new ChainValidationHelper, then switching to
MonoTlsProvider and friends.
(cherry picked from commit f2e98e82f04dbaa72e48c37d72407000c197fbcb)
When comparing a string with null the other checks in the interpreter for type equality didn't catch this case
and this ultimately resulted in a NotImplementedException in EqualInstruction and NotEqualInstruction.
The fix is to handle the TypeCode.String case in those instructions.
We don't need to do it for CreateLifted() as that method only applies for Nullable<T> types which doesn't make sense for string.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=34334
Make sure we always complete the async operations on error and
also send callbacks outside the lock.
(cherry picked from commit 9027f0ad28bf733ebb8ac9964c78c22853ca93c7)
So, I broke this on Android back in https://github.com/mono/mono/commit/d3726dcde9ef373157cd533ce8091cdde89dfa8e.
Turns out signal() is defined inline in Android:
static __inline__ __sighandler_t signal(int s, __sighandler_t f)
{
return bsd_signal(s,f);
}
So we need to hardcode the define in case we're on an Android platform.
Fixes BclTests on Android:
I/mono-stdout( 1200): UnixSignalTest
F/ ( 1200): signal() is not supported by this platform
F/libc ( 1200): Fatal signal 6 (SIGABRT) at 0x000004b0 (code=-6), thread 1215 (Instrumentation)
This fixes a race condition between the calls to Start and
BeginOutputReadLine/BeginErrorReadLine. This was triggered by the fact
that the call to WaitForExit (in StartBackgroundWaitForExit called by
Start_noshell) could be faster that the call to
BeginOutputReadLine/BeginErrorReadLine, leading to a null async_output,
which would make the WaitForExit call not wait for the output to finish.
This could make the call to Exited happen before a call to
OutputDataReceived/ErrorDataReceived.
To trigger this issue more reliably, you can simply add a sleep at the
beginning of AddInput in ProcessAsyncReader.
When switching to Mono.Net.Security.ChainValidationHelper, we will use
X509Certificate2Collection instead of MSX.X509CertificateCollection; keep
both version for the moment.
(cherry picked from commit 1f004d8e7a8391785d4ae8fcb0270f9be9d522c8)
This is a new internal wrapper class defined in
external/referencesource/System/net/System/Net/ServicePointManager.cs
It wraps the RemoteCertificateValidationCallback, capturing the
current ExecutionContext.
(cherry picked from commit c1695a55dc9a55f6b344bfc4c91991bd1f04c046)
Includes everything from Mono.Security.dll.sources except for the
new Mono.Security.Interface APIs, which are built inside System.dll
on mobile.
(cherry picked from commit df1adee67802417381280c097f3f8f17bbd48f66)
The `monotouch_runtime` profile is exclusively used for the monotouch
REPL assemblies used by Xamarin Sketches and Inspector. In those cases,
Console and Environment should behave just the same as in the
`monotouch` profile.
The only real difference between the two profiles should be the presence
of SRE in `monotouch_runtime`.
The new `multiassembly` option lets you run `mdoc update` on assemblies that
contain the same types. This can come into play when you have a set of assemblies
that run on different platforms, with a slightly differing API surface area on
each platform (for example: tvOS, watchOS, iOS); in this case you can maintain
all of the documentation in a single set of XML documents.
With this option enabled, an `AssemblyInfo` node will be added for every assembly
that a type is found in, in addition to every member. While this may seem verbose,
it's really the only way to capture the information about what assemblies a given
type or member can be found in.
Add a new HttpWebRequest.ServerCertificateValidationCallback property
(new .NET 4.5 / 4.6 API, not enabled yet) and read-only internal
ServerCertValidationCallback accessor (defined in the referencesource's
ServicePointManager.cs).
(cherry picked from commit ea9ed03863c730831c6fac393ed4dad3a94fb149)
The new internal constructor allows a per-request MonoTlsProvider and
related MonoTlsSettings; also expose these as new internal properties.
(cherry picked from commit 31988df75077c11ed87c77ab34e5b77b5913a32b)
The old SetCertificates() will be removed when merging the rest of
the work-newtls changes.
(cherry picked from commit 815731f2b7fd2f7384b8e16fcaf33f54d5ab6cb1)
It's an internal API that's used by the referencesource.
(cherry picked from commit 7fbd50ab52fc4cdf2fa9282ecd9d286ac48d9ba8)
(cherry picked from commit eb2f282d6f)
These should be considered internal APIs for the moment, even where the
classes are public (to avoid making internals visible).
(cherry picked from commit bec0257c97)
This picks up the internal 'ServerCertValidationCallback' class,
not the actual ServicePointManager implementation (we can't switch
to that yet).
(cherry picked from commit f5eeb5597a)
These two internal delegates are defined and used by the referencesource's SslStream
implementation; add them to our SslStream.cs version for compatibility.
(cherry picked from commit 0fbda4c3bd)
These should be used instead of MOBILE / MONOTOUCH / MONODROID to select
extern alias directives to make them work independently from the mobile profile.
* MONO_SECURITY_ALIAS: whether or not to use 'extern alias MonoSecurity'.
* MONO_X509_ALIAS: whether or not to use
extern alias PrebuiltSystem;
using X509CertificateCollection = PrebuiltSystem::System.Security.Cryptography.X509Certificates.X509CertificateCollection;
* MONO_FEATURE_NEW_TLS: ignore, this is not enabled in the default build.
(cherry picked from commit 48b95ec304)
In file included from /Users/joao/Dev/droid/mono/mono/metadata/marshal.c:26:0:
/Users/joao/Dev/droid/mono/mono/metadata/marshal.c: In function 'mono_marshal_get_delegate_invoke_internal':
/Users/joao/Dev/droid/mono/mono/metadata/abi-details.h:29:43: error: 'MONO_OFFSET_MonoDelegate_rgctx' undeclared (first use in this function)
#define MONO_STRUCT_OFFSET(struct,field) (MONO_OFFSET_ ## struct ## _ ## field == -1, G_STRUCT_OFFSET (struct,field))
Roslyn used byref vars whenever possible to store fixed statements. This leads to confusion in
both the full verifier and the JIT built-in one.
Now we skip it in the JIT one and allow it when not in strict mode for the full verifier.
This fixes commit c74c64718d64ecc05ddec21d12313d840842a9c2; these
new internal APIs are required by the referencesource's version of
NetworkStream.
(cherry picked from commit 2905ed96d1)
Linux kernels before 3.9 did not support port reuse.
Added private method System.Net.Sockets.Socket.SupportsPortReuse,
which returns true when the runtime is able to bind to the same address
and port multiple times.
The two known ways to access the environment variables are no longer allowed Apple.
Until something blessed by them is found, we'll provide no way to list all the envvars
of the current process.
This is useful on WatchOS and TvOS as it's the only two platforms that
do not support signal based syscall abort. By calling shutdown, the
blocking syscalls on other threads will be interrupted. All the other
platforms currently support signals, so this will not change anything.
We cannot call Close, as this would let the OS reuse the FD for
something else, and the call to Receive (for example) on another thread
could fail with ENOTSOCK.
Unfortunatly it does not work for UDP based connection, as they are not
stream based, so Shutdown does not have much sense.
A continue statement in a do...while loop was skipping code to read the next line in a connection, causing an infinite loop.
Fixed by refactoring the loop, removing duplicated code.
We have been running into this issue: https://bugzilla.xamarin.com/show_bug.cgi?id=30018 a lot recently, after digging into getaddrinfo a bit we saw that the linked list of addrinfo structs had 1 or more valid ones which can be used and 1 bad one which should just be discarded.
As per RFC2821 section 4.5.2:
- Before sending a line of mail text, the SMTP client checks the
first character of the line. If it is a period, one additional
period is inserted at the beginning of the line.
Fixes [Bug 15153](https://bugzilla.xamarin.com/show_bug.cgi?id=15153) "System.ServiceModel.NetTcpBinding properties are not pre-initialized while they are in .NET causing interoperability problems."
Specifically the `reader_quotas` member of `System.ServiceModel.NetTcpBinding` is not initialized to a new `XmlDictionaryReaderQuotas` object which causes a "NullReferenceException: Object reference not set to an instance of an object." whenever an attempt is made to set any attributes of that property. For example: `using System.ServiceModel; NetTcpBinding binding = new NetTcpBinding(SecurityMode.None); binding.ReaderQuotas.MaxStringContentLength = 8192;` raises "NullReferenceException: Object reference not set to an instance of an object."
When a frame marked with OPCODE 0, it is a continuation frame. This frame does not carry the message type and should be derived from the previous frame.
Without this patch you can't use "WebResource.axd" ressources
in a cluster (each mono instance will get a different link for the same ressource)
Signed-off-by: Etienne CHAMPETIER <[email protected]>
2014-04-01 11:42:09 +02:00
17824 changed files with 3900196 additions and 639878 deletions
This directory contains acceptance tests, which are third party test suites and frameworks that are used to validate Mono against a wider range of test cases that go beyond the Mono unit tests that run as part of CI.
In order to make checking out those test suites optional we don't use traditional git submodules, but instead clone them on demand when needed. The custom submodule repositories are checked out into the acceptance-tests/external/ directory.
## Usage
Running all test suites is possible via "make check-full". There are also targets for running individual test suites, see below.
Some of the test suites require an installed Mono (i.e. they don't work with the in-tree build), those will ask you to pass in the PREFIX variable pointing to the installation directory when invoking make. Note that this directory needs to be writable as we overwrite some files there as part of testing.
## Individual test suites and targets
*`make check-ms-test-suite` - Runs tests that were shared with Xamarin, those are not available publically and will be skipped when the repository is not accessible.
*`make check-roslyn` - Runs the Roslyn test suite.
*`make check-coreclr` - Runs the CoreCLR test suite.
*`make coreclr-runtest-coremanglib` - Runs only the CoreMangLib portion of the CoreCLR tests, those tests mostly target the BCL behavior.
*`make coreclr-runtest-basic` - Runs only the CoreCLR tests that target runtime behavior and stability.
*`make coreclr-compile-tests` - Convenience target that precompiles all the test cases in parallel.
*`make coreclr-gcstress` - Runs the CoreCLR GC stress tests.
## Updating submodules
The SUBMODULES.json file stores information about the submodules, and make targets are used to check out submodules, check their versions, and update the submodule information:
*`make validate-<module>` - Checks whenever `<module>` is checked out and matches the version in the info file.
*`make reset-<module>` - Clones `<module>` if neccesary and checks out the revision specified in the info file.
*`make bump-<module>` - Updates the revision stored in the info file for `<module>`.
*`make bump-current-<module>` - Updates the revision stored in the info file for `<module>` to the current revision.
*`make commit-bump-<module>` - Same as `make bump-<module>`, but commits the change.
*`make commit-bump-current-<module>` - Same as `make bump-current-<module>`, but commits the change.
Example: when making a change in the CoreCLR submodule (like disabling something on Mono), you'd commit the change there and then run `make bump-current-coreclr` or `make commit-bump-current-coreclr` to update the version in SUBMODULES.json.
Loaded 100 of 17824 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.