Connections 5.5, TLSv1.2, java.security and the tale of a log day


Let’s set up the background for our story first: Connections 5.5 CR2 on Windows. 3rd party products galore (Docs, Kudos, ProjExec, Text.IO/Ephox), heavy usage and then – above all – the off-and-on problem with the Rich Text widget. As my penchant for acronyms is well known by my friends, so I shall refer to this overall topic as TPP (this pesky problem) – and it kept rearing it’s ugly, mishapen and thoroughly ugly head off and on. We would squash it and then some other config change wold make it come back again.

I wanted to avoid having to switch WAS all the way to TLSv1.2 because of the well documented (potential) fall out for IBM Docs, Text.io and other products. If you want more background on that one, you can read up at the blogs of some of my colleagues – such as Nico, Ben and Robert. There are more, but you can start your education here and branch out.

So, our last defense this time is to enable TLS v. 1.2 ONLY on WebSphere which is a well documented process that actually does not take long – until it turned into the beginning of 8 hours of hell.. All went well until I tried to do a manual sync (syncnode) from any of the Nodes back to the Deployment Manager. I saw errors I had never seen before, all pointing back to SSL and formatting errors. A syncnode with the [-trace] switch wold give me 3000+ lines of juicy gibberish to wade through and no amount of searches on google helped me with anything. It all came back to this errors in the logs:

[Error parsing HTTP status line “\00”: java.util.NoSuchElementException].

After hours of pulling my hair I did what every IT guy does after a while – I looked for somebody to whine to and then beg for help. Multiple people responded, all felt bad for me but nobody was able to assist. In the end, it took my friend Nico going through a list of possible causes for TPP until he hit something that jiggled my memory: [Java Security].

The Cuplprit

This is where we go from prose back to techno talk – I dimply remembered that the install of ProjExec (btw, great project management tool – complicated but really, really good) has a requirement in it’s install documentation to edit the contents of the java.securty file of each node involved – the change is basically to change which SSLServerSocketFactory to use and here the change:

# Default JSSE socket factories

ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl

ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl # WebSphere socket factories (in cryptosf.jar)

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory

#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

 

The above shows what the change looks like, basically you un-comment the first two lines and comment the second two.

I reversed the change and – presto – TLSv1.2 works and the nodes can all talk to each other. We are working with the vendor to figure out if we really still need this change going forward. I am also thinking that this might have something to do with an SSL error on Activities file uploads I saw here and there – not sure.

So, the lessons of this days was:

  • If you are following documentation and other people can get it to work – it’s you, not the documentation
  • Peel back the onion: If you set it all correctly in WebSphere, step one pace back/up the chain of technology – it runs java, is java based -> you need to check up the chain to see what base java settings are in place, other than what you set yourself.
  • Don’t cry, it’s unbecoming
  • When friends who are kind enough to answer your Skype calls, LISTEN TO EACH QUESTION and think the answer through, you might not be seeing the forest because all those damn trees are in the way.
  • Say thank you – publicly. You might still be sitting there all night trying to figure out what went wrong

Technote: “Freemarker Template files are overwritten during IBM Connections CR2 install”


This happened to me, I was only saved by having a local back-up on my machine … don’t let it hit you!

Technote Link – swg21996243

Hving a good back-up before ANY upgrade, change etc. is important. If nothing else, do a backup config with WebSphere – that will capture all of the important files for you as well!

 

 

 

 

IBM WebSphere / Connections – Performance, Security and the SESN0008E Error


Just found a new issue today that has been vexing me for quite some time, I only found it because of the error [SESN0008E] and the fact that I had to add a whole new WebSphere Nodes to an existing environment so that these errors finally happend with a frequency that I finally noticed it:

logServletError SRVE0293E: [Servlet Error]-[atom-basic]: com.ibm.websphere.servlet.session.UnauthorizedSessionRequestException: SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:defaultWIMFileBasedRealm/CN=Joe Shmoe,OU=MYOU,dc=corp,dc=company,dc=com

We ran function testing on the new Node and performance was horrendous … I mean really horrendous. Performance has been bad overall before that, but at least tings worked. I investigated a few tech notes, there was some mentioning about the LTPA timeout being too short in combination with several other settings. This is an upgrade, same settings as other systems, should not be an issue … so I looked at other sites/pages here, here and here.

All of the tech notes mentioned [Security Integration] being at the heart of it. I checked all servers and noticed – none of the server that the Connections installer created had this settings set, all servers that I created manually had it. I looked into this a bit more and found out that the [Session Management] – [Security integration] is now a default setting for WebSphere and if you create a server manually it is automatically set. I ran a few third party products in separate servers that were all manually created … they probably brought the overall performance down.

So, I went through all servers that I had created, unset the settings (pic below) and then synced and restarted everything and …. voila, speed restored.

sessionManagement

IBM Connections – CCM Folders and File Loss … or not


Thought I’d share this one, it was a bit unique. Anyone out there who has had to do a restore of CCM and CCM files will know it is a pain. I will blog about how to do that separately .. It is not fun.

Here the scenario:

  • Client with IBM Connections 5.0, CR3, CCM – very active site,.
  • Many communities, allot of File Libraries. Many users use the Windows Plug-ins and access files and CCM libraries via the Windows desktop.
  • A user – somehow by mistake – deleted a folder inside a community library (CCM library!) containing a whole bunch of files – and this user did it via the desktop. Apparently what the user did how in Windows Explorer a folder was either moved or deleted (I was never able to exactly find out) and the folder with all files contained in it disappeared. Not to be found.

Long Story Short:

I was in the middle of restoring the Filenet databases from the day before to a separate environment to figure out if I could identify the files and ask the back-up team to restore them for me from tape. I took another look in the system and could not find the files in the acce interface.

But then I had an idea … Connections search indexes it all, no matter where it is.

So, using the Filenet restore I identified the files to look for, did a search for the filenames in Connections … and found them. We then did a “Move to Folder” for all the files (into a new folder in Libraries that we created) and all files were back where they belonged.

So – what this taught us is that deleting folders using the Windows plug-is does nothing to the actual files, it just appears to be removing the pointers to the database that the system needs to display the files … But Connections search still find them all. Like Pokemons … gotta catch’em all!

If I had known this earlier I could have saved myself a day of work …

 

 

 

IBM Connections, Exchange, Kerberos and the Tale of External Non-Collaboration


It is a longer tale, so to make keep it short I decided to busy the lead and give you the synopsis right here:

If you are running IBM Connections integrated with Exchange as your ICMail setup you are using Kerberos. If you want to enable external collaboration by adding another LDAP source for your external users – it will not work.

You can create the repository, add it to WebSphere, you can do all the TDI settings to import the users in it as external users .. but they will not be able to authenticate. The reason is that WebSphere has the authentication mechanism at it’s top level of security (global) and not at the repository level. That means, once you use Kerberos you have to use Kerberos for ALL authentication that happens. Trust me, I have tested. I had PMRs open (with both Connections and WebSphere support). I talked to the IBM Connections Product team and verified that this specific scenario was never actually tested so nobody appears to have known of this, which is also why it never made it’s way into any documentation.

I don’t think there are many clients for whom this might be an issue currently, but I do see many environments wanting more security and wanting to tie in other back-end systems and if that client environment is running AD as their LDAP source , then KERBEROS will be right there as a feature request – or a necessity.

Is External Collaboration Dead when Using Kerberos?

That is an easy answer – No.

But you are now forced to add those external users to your AD forest and either add them to some branch/OU that you can treat as external users or add some AD/LDAP attribute to identify them as external users.

Feature Enhancement Request for WebSphere – PLEASE VOTE!

I entered a feature enhancement request to move the authentication method from a global setting to the repository level – either in general or as art of a security domain setup in WebSphere, thereby allowing non-Kerberos repositories to be used for authentication alongside a KERBEROS enabled repository.

Here is the link to the feature request – the more people look at it, follow it and vote for it the more likely it is to make it’s wat into a future release. you will need to have an IBM website ID to even just look at it but I’d appreciate the effort!