When using "HTTP CONNECT" to create a tunnel to a proxy server,
check the proxy server's response for the HTTP Version and close
the connection if it's HTTP/1.0.
This patch builds upon commit 96a8500795 which placed parts of Mono.Security
responsible for dealing with SSL/TLS certificates for the purpose of certificate verification. The code enables
validation of the certifcates even if the user does not provide their own validation callback. The changes as related to
the original commit are few, basically consisting of enabling the same code used by MonoTouch in the MonoDroid context with
the only actual code change being the certificate validation bit mentioned above.
Since the change involves adding a subset of Mono.Security files to the monodroid version of System.dll, it is
also necessary to remove Mono.Android's dependence on Mono.Security.dll. The reason is that the code in AndroidPlatform
which looks up TLS support methods in Mono.Android.dll would use a type (X509CertificateCollection) that is now part of
System.dll but is *also* in Mono.Security.dll referenced by Mono.Android.dll and therefore the lookup would fail since
we'd have two distinct types from the reflection point of view.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=18962
* Merged WebConnection.EndWrite() and EndWrite2().
* WebConnectionStream.BeginWrite(): Use the same async-write
callback for both chunked and non-chunked; remove the single-use
small WriteRequestAsync() and inline it here to make the code
more readable.
* WebConnectionStream.WriteRequestAsyncCB(): Now called for all
async writes, call WebConnection.EndWrite() here.
The code is now more readable and easier to understand. This is
the first part of a series of cleanups, which should eventually
lead towards full async and 100-continue.
* ServicePoint now uses an idle timer to close idle connections.
* WebConnectionGroup has been mostly rewritten to use a linked
list of open connections and their idle times.
This fixes both the problem that we were leaking sockets and
that we were not reusing existing connections.
* ServicePoint now correctly keeps track of the number of open
connections.
* ServicePointManager now uses the correct default idle time
of 100 seconds (previously 15 minutes).
* WebConnection uses the new WebConnectionGroup's busy logic
to correctly flag connections as busy / reusable.
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=15857
Scenario (for which I couldn't easily create a testcase, but can
sketch out): Imagine using HttpClient...in a multithreaded manner:
// shared:
static CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource ();
// Thread 1
var request = new HttpRequestMessage(HttpMethod.Post, "https://www.google.com/") {
Content = new StringContent (new string('x', 1024)),
};
using (request)
using (HttpClient httpClient = new HttpClient())
using (HttpResponseMessage response = await httpClient.SendAsync(request,
_cancellationTokenSource.Token)) {
string responseContent = await response.Content.ReadAsStringAsync();
Console.WriteLine ("Read {0} chars", responseContent.Length);
}
// Thread 2
_cancellationTokenSource.Cancel ();
This isn't entirely farfetched; in the case of #15857, Thread 2 was
the Main thread (due to the user clicking a button), while Thread 1
was the ThreadPool (because of async/await).
The _occasional_ result: a TargetInvocationException exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object
at (wrapper delegate-invoke) <Module>:invoke_bool__this___HttpsClientStream (Mono.Security.Protocol.Tls.HttpsClientStream)
at System.Reflection.MonoProperty.GetterAdapterFrame[HttpsClientStream,Boolean] (System.Reflection.Getter`2 getter, System.Object obj)
at System.Reflection.MonoProperty.GetValue (System.Object obj, System.Object[] index)
--- End of inner exception stack trace ---
at System.Reflection.MonoProperty.GetValue (System.Object obj, System.Object[] index)
at System.Net.WebConnection.Write (System.Net.HttpWebRequest request, System.Byte[] buffer, Int32 offset, Int32 size, System.String& err_msg)
at System.Net.WebConnectionStream.WriteHeaders ()
at System.Net.WebConnectionStream.SetHeaders (System.Byte[] buffer)
at System.Net.HttpWebRequest.SendRequestHeaders (Boolean propagate_error)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw ()
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[System.Net.Http.HttpResponseMessage].GetResult ()
at System.Net.Http.HttpClient+<SendAsyncWorker>c__async0.MoveNext ()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw ()
at System.Runtime.CompilerServices.TaskAwaiter`1[System.Net.Http.HttpResponseMessage].GetResult ()
at HttpClientAsyncCancellationRepro.Activity1+<AccessGoogleButton_Click>c__async0.MoveNext ()
Huh?
The reason why is because HttpClientHandler.SendAsync() registers a
Cancel handler with the CancellationTokenSource, and the registered
handler calls HttpWebRequest.Abort() (which eventually calls
WebConnection.Close()). This causes WebConnection.nstream to _change_
while WebConnection.Write() is executing, causing
WebConnection.nstream to go from non-null to null, which
WebConnection.Write() does not expect:
# WebConnection.CreateStream; from 5 Threadpool worker
# WebConnection.Write; from 5 Threadpool worker
# WebConnection.Close; from 1 at at System.Environment.get_StackTrace()
at System.Net.WebConnection.Close(Boolean sendNext)
at System.Net.WebConnection.Abort(System.Object sender, System.EventArgs args)
at System.Net.WebConnection+AbortHelper.Abort(System.Object sender, System.EventArgs args)
at System.Net.HttpWebRequest.Abort()
at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.<>m__0(System.Object l)
at System.Threading.CancellationToken+<Register>c__AnonStorey0.<>m__0()
at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException)
at System.Threading.CancellationTokenSource.Cancel()
at System.Threading.CancellationTokenSource.SafeLinkedCancel()
at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException)
at System.Threading.CancellationTokenSource.Cancel()
at HttpClientAsyncCancellationRepro.Activity1.CancelButton_Click(System.Object sender, System.EventArgs e)
at Android.Views.View+IOnClickListenerImplementor.OnClick(Android.Views.View v)
at Android.Views.View+IOnClickListenerInvoker.n_OnClick_Landroid_view_View_(IntPtr jnienv, IntPtr native__this, IntPtr native_v)
at System.Object.e7b5b870-ea0f-434f-bc31-984f665a44ad(IntPtr , IntPtr , IntPtr )
# WebConnection.Write: checking piTrustFailure; from 5 Threadpool worker
(Yay extra logging. Notice that WebConnection creates a stream from
the ThreadPool, starts WebConnection.Write(), then before it can se
piTrustFailure the instance is Close()d, which nulls out nstream.)
Since WebConnection.Write() attempts to access the (now null)
WebConnection.nstream field, it results in a NullReferenceException,
which is wrapped into a TargetInvocationException by
System.Reflection.
The fix? As usual, "don't do that."
In this case, WebConnection.Write() has already captured
WebConnection.nstream into the local variable `s` (which thus can't be
changed by another thread). Instead of using WebConnection.nstream,
reuse the `s` local variable, which cannot be null, thus avoiding the
NullReferenceException.
(This solution could fail if HttpsClientStream ever throws e.g.
ObjectDisposedException from HttpsClientStream.TrustFailure, but at
present HttpsClientStream.TrustFailure never throws, so this is fine.)
HttpWebRequest is a very tricky piece of code when it comes to its socket, this patch is more like a hack to get something working out of it but it's definitely not the best option.
This fixes #12355/#12395/#5904.
When EndRead() is called with a null "inner" IAsyncResult, then
we should not consider this as an end-of-stream condition.
This happens sometimes when using SSL and chunked encoding.
This reverts commit b190db44cb.
The reason is:
"The HttpWebRequests tests are not enough"
"A quick review of the code shows that the code is taking a lock, while
calling a function that retries connections (which in turn takes another
important lock --for the socket--) which might casue a stack overflow
if someone calls SetWriteStream"
"This also will make the opereation sync when calling Begin/End"
"Finally, this is a patch that improves the use of the threadpool, but
the threadpool still will end up exhausted"
Probably best to just increase the number of threads?
The code path for synchronous requests was the same as the one for
asynchronous ones: the threadpool. It's actually not needed in this
case because the thread is going to wait anyway, so there's no need
to use the threadpool.
This helps in scenarios in which SOA is heavily used:
- The process is hosted under xsp/mono-fastcgi/... which use the threadpool
too.
- If the requests generate other requests to other REST services, the
usage of the threadpool in these cases may interfere with the usage of
the threadpool by xsp/mono-fastcgi/...
- The thread that is serving a request shouldn't request another thread
for a blocking operation, preventing the use of that thread by other
possible incoming requests.
This commit is contributed under the terms of MIT/X11 licence.
(It doesn't cause any breakage in the System test suite.)
ReadDone() now passes the 'WebConnectionData' instance to GetResponse()
and 'readState' has been moved into 'WebConnectionData'.
Abort() can now reset 'Data' while ReadDone() still uses the old
instance, but learns about the abort request from 'data.ReadState'.
* AuthenticationManager.EnsureModules(): Swap the order of the
modules, putting the strongest ones first.
* WebConnection.CreateTunnel(): Pass the proxy's address to
AuthenticationManager to support CredentialsCache. Only attempt
one single authentication method.
I've done some extensive testing with different squid configurations
(digest only, ntlm only, digest and ntlm, ntlm and digest) and the
.NET runtime only attempts one single authentication method.
If you provide credentials like this:
var proxy = new WebProxy (proxyUrl, false);
proxy.Credentials = new NetworkCredential ("user", "pass", "domain");
then these must match the server's first authentication method.
You can also use a CredentialsCache like this:
var ntlm_cred = new NetworkCredential ("u1", "p1", "d1");
var digest_cred = new NetworkCredential ("u2", "u3", "d2");
var cc = new CredentialCache ();
cc.Add (proxy_uri, "NTLM", ntlm_cred);
cc.Add (proxy_uri, "Digest", digest_cred);
var proxy = new WebProxy (proxy_uri, false);
proxy.Credentials = cc;
In WebConnection.CreateTunnel():
* Support multi "Proxy-Authentication" headers from the server.
* Pass a fake HttpWebRequest with method "CONNECT" to
AuthenticationManager.Authenticate(); the HTTP method is part
of the digest response.
* Add "Proxy-Connection: Keep-Alive".
This is a partial revertion of master commit #9069d07, which was
backported into mono-2-10 in commit #bfa23f8.
I only reverted the change in line 716, keeping the rest of the
commit in place.
We occasionally see this happen in the OpenSimulator project when web requests are made (the project makes a lot of these).
I'm reasoning that a -1 from GetResponse() also indicates that the response is badly formatted (wrong number of parts or not properly terminated) as well as catching an Exception.
Making this change removes the issues that we see.
If the transfer encoding was set to chunked, we tried to read even when
the response might known to have no content. This led to timeouts.
Fixes bug #686371.
If there's a POST/PUT request that gets a NTLM 401, we save the data
and don't send it again until after the challenge phase. Ie, instead of
sending all the data 3 times, we only send it at most twice.
There is a possible crash using this profile if the monotouch.dll
isn't loaded into the process space. We will null guard the
type and only call start_wwan if its present.
2010-07-21 Sebastien Pouliot <[email protected]>
* Makefile: Ensure SECURITY_DEP gets defined in our moonlight
raw profile
In System.Net:
2010-07-21 Sebastien Pouliot <[email protected]>
* ServicePointManager.cs: Add special validation for Moonlight
* WebConnection.cs: Change a define to NET_2_1
In System.Net.Security:
2010-07-21 Sebastien Pouliot <[email protected]>
* LocalCertificateSelectionCallback.cs:
* SslStream.cs:
Don't use alias in moonlight build (not needed and breaks
compilation otherwise)
svn path=/trunk/mcs/; revision=160675
* System.Net/WebConnection.cs: attach a validation callback to the new
event in Mono.Security.
* System.Net/ServicePointManager.cs: added a new ValidationHelper
class that takes care of building the certificate chain and perform
all the check that were done in Mono.Security before. This way the 2.0
server certificate validation callback can get all the data it needs.
* System.Net.Security/SslStream.cs: don't do anything if a chain
element has no error.
Fixes bug #584050.
svn path=/trunk/mcs/; revision=152654