I would like to start writing about my latest experience (or should I say, my latest headache). As i said before, most of my company clients are financial institutions or banks.
Due to importance of banking mobile and the growing demand of the use of non-face-to-face services such as mobile apps, cash in and cash out ATMs, Dropbox, contact center and info terminals, my client needed to update their technological platform, which would allow in short term a better utilization of the electronic channels and also will allow to clients mostly private, save time since it will greatly reduce the physical presence of customers in the bank’s branches
We were facing an issue trying to go live with one of our applications, In short words, we needed to connect our website solution with some WCF services. Before going live we did all types of testing ( integration, functional, stress, performance and regression testing). However, one week before we simulated goin-live, the issue was not present.
Here’s the full exception of the issue:
Message action="" exception=" Exception ( System.ServiceModel.CommunicationException ) : The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was &#39;00:00:59.9980000&#39;. Source: mscorlib StackTrace: Server stack trace: at System.ServiceModel.Channels.StreamConnection.Read(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout) at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout) at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout) at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message&amp; message) at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object ins, Object outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at : at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type) InnerException ( System.IO.IOException ) : The read operation failed, see inner exception. Source: System StackTrace: at System.Net.Security.NegotiateStream.ProcessRead(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.NegotiateStream.Read(Byte buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.StreamConnection.Read(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout) InnerException ( System.ServiceModel.CommunicationException ) : The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was &#39;00:00:59.9980000&#39;. Source: System.ServiceModel StackTrace: at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing) at System.ServiceModel.Channels.SocketConnection.Read(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.DelegatingConnection.Read(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.ConnectionStream.Read(Byte buffer, Int32 offset, Int32 count) at System.Net.FixedSizeReader.ReadPacket(Byte buffer, Int32 offset, Int32 count) at System.Net.Security.NegotiateStream.StartFrameHeader(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.NegotiateStream.ProcessRead(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) InnerException ( System.Net.Sockets.SocketException ) : An existing connection was forcibly closed by the remote host Source: System StackTrace: at System.Net.Sockets.Socket.Receive(Byte buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing) "
One of pre-requisites to go live with our application was the upgrade of Microsoft Framework from 3.5 to 4.6.1 https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/versions-and-dependencies
Also, the connection between our services use net tcp with certificates (SSL security). So the configuration in our service and client looks like:
<bindings> <netTcpBinding> <binding name="StandardIncreasedBuffer" closeTimeout="01:10:00" openTimeout="01:10:00" receiveTimeout="01:10:00" sendTimeout="01:30:00" transactionProtocol="OleTransactions" listenBacklog="50" maxBufferPoolSize="52428800" maxBufferSize="65536000" maxConnections="1000" maxReceivedMessageSize="65536000" portSharingEnabled="false" > <readerQuotas maxDepth="100" maxStringContentLength="100000" maxArrayLength="100000" maxBytesPerRead="100000" maxNameTableCharCount="1000" /> <reliableSession ordered="false" inactivityTimeout="00:30:00" /> <security mode="Transport" > <transport clientCredentialType="Certificate" protectionLevel="EncryptAndSign" /> <message clientCredentialType="Windows" /> </security> </binding> </bindings>
<behaviors> <serviceBehaviors> <behavior name="ExposeMetadata" > <serviceMetadata httpGetEnabled="false" httpsGetEnabled="false" /> <serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="10000" /> <dataContractSerializer ignoreExtensionDataObject="true" maxItemsInObjectGraph="2147483640" /> <serviceCredentials> <clientCertificate> <authentication certificateValidationMode="PeerOrChainTrust" trustedStoreLocation="LocalMachine" mapClientCertificateToWindowsAccount="false" /> </clientCertificate> <serviceCertificate findValue= "1010101010101" storeLocation="LocalMachine" storeName="My" X509FindType="FindByThumbPrint"/> </serviceCredentials> </behavior> </serviceBehaviors> </behaviors>
To try to find the issue we decide to not use certificates. Thus, the security mode was set to None.
<security mode="None" > </security>
Immediately after this change we were able to connect our application with no issues. At this time we minimize the range of search to find the causes of the issue. Therefore, we focused in the use of certificates as the possible main cause of the problem.
To have a better idea and track down the root of the issue we checked:
- Windows Event viewer
- Application logs
However, still we did no not have a big clue about the cause. At this point we decided to turn on the tracing inside our config files. Finally we found a different message. One of my favorite application to check scvlogs files is SvcTraceViewer.exe. Once we opened the app, we detected the following:
As you can see in the image above, the channel is using ssl. That simple line, give us a better idea to understand the problem and pointing to the right direction.
Then suddenly we came to the one post from Microsoft regarding possible issue with SSL protocol and certificates and framework 4.5.6
The paragraph that is interesting for us is:
“WCF services that use NETTCP with SSL sercurity and MD5 certificate authentication”
The .NET Framework 4.6 adds TLS 1.1 and TLS 1.2 to the WCF SSL default protocol list. When both client and server machines have the .NET Framework 4.6 or later installed, TLS 1.2 is used for negotiation.TLS 1.2 does not support MD5 certificate authentication. As a result, if a customer uses an MD5 certificate, the WCF client will fail to connect to the WCF service.
You can work around this issue so that a WCF client can connect to a WCF server by doing any of the following:
To overcome this issue you have two solutions:
- Need to force usage of protocol TLS 1.1 via changes in config file. You need to add attribute highlighted to “binding configuration” in web app and service config files, like:
<binding> <security mode= "None|Transport|Message|TransportWithMessageCredential" > <transport clientCredentialType="Certificate" protectionLevel="EncryptAndSign" sslProtocols="Ssl3|Tls1|Tls11" /> </security> </binding>
Don’t forget to restart app pool and the services after changes in configs.
- Update the certificates used in this case from MD5RSA to SHA-1/SHA-2 or some other. The most recommended solution. To check just open mmc.exe see certificate file properties in Windows on tab “Details”. Property is called “Signature algorithm”.
After change the certificates to latest algorithm. everything works perfectly and we were able to go live.