Connections 5.5, TLSv1.2, java.security and the tale of a log day


Let’s set up the background for our story first: Connections 5.5 CR2 on Windows. 3rd party products galore (Docs, Kudos, ProjExec, Text.IO/Ephox), heavy usage and then – above all – the off-and-on problem with the Rich Text widget. As my penchant for acronyms is well known by my friends, so I shall refer to this overall topic as TPP (this pesky problem) – and it kept rearing it’s ugly, mishapen and thoroughly ugly head off and on. We would squash it and then some other config change wold make it come back again.

I wanted to avoid having to switch WAS all the way to TLSv1.2 because of the well documented (potential) fall out for IBM Docs, Text.io and other products. If you want more background on that one, you can read up at the blogs of some of my colleagues – such as Nico, Ben and Robert. There are more, but you can start your education here and branch out.

So, our last defense this time is to enable TLS v. 1.2 ONLY on WebSphere which is a well documented process that actually does not take long – until it turned into the beginning of 8 hours of hell.. All went well until I tried to do a manual sync (syncnode) from any of the Nodes back to the Deployment Manager. I saw errors I had never seen before, all pointing back to SSL and formatting errors. A syncnode with the [-trace] switch wold give me 3000+ lines of juicy gibberish to wade through and no amount of searches on google helped me with anything. It all came back to this errors in the logs:

[Error parsing HTTP status line “\00”: java.util.NoSuchElementException].

After hours of pulling my hair I did what every IT guy does after a while – I looked for somebody to whine to and then beg for help. Multiple people responded, all felt bad for me but nobody was able to assist. In the end, it took my friend Nico going through a list of possible causes for TPP until he hit something that jiggled my memory: [Java Security].

The Cuplprit

This is where we go from prose back to techno talk – I dimply remembered that the install of ProjExec (btw, great project management tool – complicated but really, really good) has a requirement in it’s install documentation to edit the contents of the java.securty file of each node involved – the change is basically to change which SSLServerSocketFactory to use and here the change:

# Default JSSE socket factories

ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl

ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl # WebSphere socket factories (in cryptosf.jar)

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory

#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

 

The above shows what the change looks like, basically you un-comment the first two lines and comment the second two.

I reversed the change and – presto – TLSv1.2 works and the nodes can all talk to each other. We are working with the vendor to figure out if we really still need this change going forward. I am also thinking that this might have something to do with an SSL error on Activities file uploads I saw here and there – not sure.

So, the lessons of this days was:

  • If you are following documentation and other people can get it to work – it’s you, not the documentation
  • Peel back the onion: If you set it all correctly in WebSphere, step one pace back/up the chain of technology – it runs java, is java based -> you need to check up the chain to see what base java settings are in place, other than what you set yourself.
  • Don’t cry, it’s unbecoming
  • When friends who are kind enough to answer your Skype calls, LISTEN TO EACH QUESTION and think the answer through, you might not be seeing the forest because all those damn trees are in the way.
  • Say thank you – publicly. You might still be sitting there all night trying to figure out what went wrong
Advertisements