Social Connections Chicago – Here I come!


Had an email in my inbox a short while ago …

Congratulations! Your abstract for Social Connections 11 was accepted!

 

Looks like I will be in Chicago June 1-2 this year! Social Connections is a great conference, you should check it out and attend if you can – there is allot of great content and I am not just talking about my session … and you get to meet and interact with allot of people at the conference and make really, really, really great connections for yourself.

 

Oh yeah .. what will i be speaking on?

IBM Connections – Take Performance Tuning Beyond the Documentation

 

Connections 5.5, TLSv1.2, java.security and the tale of a log day


Let’s set up the background for our story first: Connections 5.5 CR2 on Windows. 3rd party products galore (Docs, Kudos, ProjExec, Text.IO/Ephox), heavy usage and then – above all – the off-and-on problem with the Rich Text widget. As my penchant for acronyms is well known by my friends, so I shall refer to this overall topic as TPP (this pesky problem) – and it kept rearing it’s ugly, mishapen and thoroughly ugly head off and on. We would squash it and then some other config change wold make it come back again.

I wanted to avoid having to switch WAS all the way to TLSv1.2 because of the well documented (potential) fall out for IBM Docs, Text.io and other products. If you want more background on that one, you can read up at the blogs of some of my colleagues – such as Nico, Ben and Robert. There are more, but you can start your education here and branch out.

So, our last defense this time is to enable TLS v. 1.2 ONLY on WebSphere which is a well documented process that actually does not take long – until it turned into the beginning of 8 hours of hell.. All went well until I tried to do a manual sync (syncnode) from any of the Nodes back to the Deployment Manager. I saw errors I had never seen before, all pointing back to SSL and formatting errors. A syncnode with the [-trace] switch wold give me 3000+ lines of juicy gibberish to wade through and no amount of searches on google helped me with anything. It all came back to this errors in the logs:

[Error parsing HTTP status line “\00”: java.util.NoSuchElementException].

After hours of pulling my hair I did what every IT guy does after a while – I looked for somebody to whine to and then beg for help. Multiple people responded, all felt bad for me but nobody was able to assist. In the end, it took my friend Nico going through a list of possible causes for TPP until he hit something that jiggled my memory: [Java Security].

The Cuplprit

This is where we go from prose back to techno talk – I dimply remembered that the install of ProjExec (btw, great project management tool – complicated but really, really good) has a requirement in it’s install documentation to edit the contents of the java.securty file of each node involved – the change is basically to change which SSLServerSocketFactory to use and here the change:

# Default JSSE socket factories

ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl

ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl # WebSphere socket factories (in cryptosf.jar)

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory

#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

 

The above shows what the change looks like, basically you un-comment the first two lines and comment the second two.

I reversed the change and – presto – TLSv1.2 works and the nodes can all talk to each other. We are working with the vendor to figure out if we really still need this change going forward. I am also thinking that this might have something to do with an SSL error on Activities file uploads I saw here and there – not sure.

So, the lessons of this days was:

  • If you are following documentation and other people can get it to work – it’s you, not the documentation
  • Peel back the onion: If you set it all correctly in WebSphere, step one pace back/up the chain of technology – it runs java, is java based -> you need to check up the chain to see what base java settings are in place, other than what you set yourself.
  • Don’t cry, it’s unbecoming
  • When friends who are kind enough to answer your Skype calls, LISTEN TO EACH QUESTION and think the answer through, you might not be seeing the forest because all those damn trees are in the way.
  • Say thank you – publicly. You might still be sitting there all night trying to figure out what went wrong

Technote: “Freemarker Template files are overwritten during IBM Connections CR2 install”


This happened to me, I was only saved by having a local back-up on my machine … don’t let it hit you!

Technote Link – swg21996243

Hving a good back-up before ANY upgrade, change etc. is important. If nothing else, do a backup config with WebSphere – that will capture all of the important files for you as well!

 

 

 

 

IBM WebSphere / Connections – Performance, Security and the SESN0008E Error


Just found a new issue today that has been vexing me for quite some time, I only found it because of the error [SESN0008E] and the fact that I had to add a whole new WebSphere Nodes to an existing environment so that these errors finally happend with a frequency that I finally noticed it:

logServletError SRVE0293E: [Servlet Error]-[atom-basic]: com.ibm.websphere.servlet.session.UnauthorizedSessionRequestException: SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:defaultWIMFileBasedRealm/CN=Joe Shmoe,OU=MYOU,dc=corp,dc=company,dc=com

We ran function testing on the new Node and performance was horrendous … I mean really horrendous. Performance has been bad overall before that, but at least tings worked. I investigated a few tech notes, there was some mentioning about the LTPA timeout being too short in combination with several other settings. This is an upgrade, same settings as other systems, should not be an issue … so I looked at other sites/pages here, here and here.

All of the tech notes mentioned [Security Integration] being at the heart of it. I checked all servers and noticed – none of the server that the Connections installer created had this settings set, all servers that I created manually had it. I looked into this a bit more and found out that the [Session Management] – [Security integration] is now a default setting for WebSphere and if you create a server manually it is automatically set. I ran a few third party products in separate servers that were all manually created … they probably brought the overall performance down.

So, I went through all servers that I had created, unset the settings (pic below) and then synced and restarted everything and …. voila, speed restored.

sessionManagement

IBM Connections – CCM Folders and File Loss … or not


Thought I’d share this one, it was a bit unique. Anyone out there who has had to do a restore of CCM and CCM files will know it is a pain. I will blog about how to do that separately .. It is not fun.

Here the scenario:

  • Client with IBM Connections 5.0, CR3, CCM – very active site,.
  • Many communities, allot of File Libraries. Many users use the Windows Plug-ins and access files and CCM libraries via the Windows desktop.
  • A user – somehow by mistake – deleted a folder inside a community library (CCM library!) containing a whole bunch of files – and this user did it via the desktop. Apparently what the user did how in Windows Explorer a folder was either moved or deleted (I was never able to exactly find out) and the folder with all files contained in it disappeared. Not to be found.

Long Story Short:

I was in the middle of restoring the Filenet databases from the day before to a separate environment to figure out if I could identify the files and ask the back-up team to restore them for me from tape. I took another look in the system and could not find the files in the acce interface.

But then I had an idea … Connections search indexes it all, no matter where it is.

So, using the Filenet restore I identified the files to look for, did a search for the filenames in Connections … and found them. We then did a “Move to Folder” for all the files (into a new folder in Libraries that we created) and all files were back where they belonged.

So – what this taught us is that deleting folders using the Windows plug-is does nothing to the actual files, it just appears to be removing the pointers to the database that the system needs to display the files … But Connections search still find them all. Like Pokemons … gotta catch’em all!

If I had known this earlier I could have saved myself a day of work …

 

 

 

Social Connections – Toronto Jun 6-7, 2016 – I am attending!


The next Social Connections Conference has been announced: June 6-7, 2016 in wonderful and clean TORONTO CANADA

http://socialconnections.info/

 

I already signed up and a submission for an abstract is already in ….

Anybody in the social media/social networking sphere should really attend this conference. Technical and Strategy without the marketing hype, that is why I really like to attend.

 

Go forth and attendeth!

Connections 5.5 – Install Problem for WebSphere Cluster Settings with UNC Shares


I just installed a new Connections V5.5 environment for a new client and came across this issue that I had encountered once before in previous versions when installing the IBM File viewer (look at my presentation from last year at MWLug 2015) .

Scenareo:

  • Connections 5.5,
  • Clustered Windows WebSphere servers (2 nodes on separate Windows server)
  • Windows File Share for shared file services (accessed using a UNC link i.e.: \\[fqhn of server]\[share name])

The Installer will go through and work without a problem, all apps are installed and the clusters in WebSphere created. When you run the WebSphere servers/JVMs for the first time you might notice a new folder created on the same drive as your WebSphere install, the name follows the above UNC naming for the shared file services location. In my case the folder created was [D:\FILESERVER\CnxData\messagestores\xxx).

Messagestores are the way that messaging engines running on WebSphere clustered servers communicate with each other by reading/writing log files (there is much more to it, but let’s keep this lite here …). Both Windows server will create the same folders and you will probably not see a whole lot of errors in the systemout.log files of the WebSphere servers because … those servers can access the files they expect, that they are not getting any inputs from other cluster members is not going to raise any errors inside of WebSphere.

In V5.0 what happens is that the installer creates a WebSphere variable and uses that variable in the cluster settings and then the system works and the UNC drive is read correctly. The V5.5 installer does not do this, it writes the location directly into the sib-engines.xml file of the cluster created and then things fall apart ….

 

What to do:

Basically you have to manually do what the installer should have done:

Create a WebSphere variable

  • I created the same one as V5.0 would have [MESSAGE_STORE_PATH] and gave it the value of the UNC folder location in WINDOWS format (using “\” slashes): i.e. [\\servername\share\messagestores]

Update the sib-engines.xml

  • Search for the sib-engine.cml files  on the Dmgr profile under: ..\WebSphere\AppServer\profiles\Dmgr01\config\cells\[cell name]\clusters\[Cluster Name]
  • Edit the last line in the file for each cluster to look something like this:
<fileStore xmi:id="SIBFilestore_1456105865384" uuid="5976E93BC88E6CB1" logSize="100" minPermanentStoreSize="200" maxPermanentStoreSize="500" minTemporaryStoreSize="200" maxTemporaryStoreSize="500" logDirectory="${MESSAGE_STORE_PATH}/UtilCluster/log" permanentStoreDirectory="${MESSAGE_STORE_PATH}/UtilCluster/store" temporaryStoreDirectory="${MESSAGE_STORE_PATH}/UtilCluster/store"/>

Note the use of “/” in this entry, do it that way!

Do the WAS Thing:

  • You need to then sync the nodes and restart all servers/clusters and then WebSphere will create the folders and subfolders is needs and all will be well ….

 

After a restart you can delete the incorrectly created folders, they do not contain any data you need, the data written into there is transactions and will be re-created when the servers restart.