Connections 5.5, TLSv1.2, java.security and the tale of a log day


Let’s set up the background for our story first: Connections 5.5 CR2 on Windows. 3rd party products galore (Docs, Kudos, ProjExec, Text.IO/Ephox), heavy usage and then – above all – the off-and-on problem with the Rich Text widget. As my penchant for acronyms is well known by my friends, so I shall refer to this overall topic as TPP (this pesky problem) – and it kept rearing it’s ugly, mishapen and thoroughly ugly head off and on. We would squash it and then some other config change wold make it come back again.

I wanted to avoid having to switch WAS all the way to TLSv1.2 because of the well documented (potential) fall out for IBM Docs, Text.io and other products. If you want more background on that one, you can read up at the blogs of some of my colleagues – such as Nico, Ben and Robert. There are more, but you can start your education here and branch out.

So, our last defense this time is to enable TLS v. 1.2 ONLY on WebSphere which is a well documented process that actually does not take long – until it turned into the beginning of 8 hours of hell.. All went well until I tried to do a manual sync (syncnode) from any of the Nodes back to the Deployment Manager. I saw errors I had never seen before, all pointing back to SSL and formatting errors. A syncnode with the [-trace] switch wold give me 3000+ lines of juicy gibberish to wade through and no amount of searches on google helped me with anything. It all came back to this errors in the logs:

[Error parsing HTTP status line “\00”: java.util.NoSuchElementException].

After hours of pulling my hair I did what every IT guy does after a while – I looked for somebody to whine to and then beg for help. Multiple people responded, all felt bad for me but nobody was able to assist. In the end, it took my friend Nico going through a list of possible causes for TPP until he hit something that jiggled my memory: [Java Security].

The Cuplprit

This is where we go from prose back to techno talk – I dimply remembered that the install of ProjExec (btw, great project management tool – complicated but really, really good) has a requirement in it’s install documentation to edit the contents of the java.securty file of each node involved – the change is basically to change which SSLServerSocketFactory to use and here the change:

# Default JSSE socket factories

ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl

ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl # WebSphere socket factories (in cryptosf.jar)

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory

#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

 

The above shows what the change looks like, basically you un-comment the first two lines and comment the second two.

I reversed the change and – presto – TLSv1.2 works and the nodes can all talk to each other. We are working with the vendor to figure out if we really still need this change going forward. I am also thinking that this might have something to do with an SSL error on Activities file uploads I saw here and there – not sure.

So, the lessons of this days was:

  • If you are following documentation and other people can get it to work – it’s you, not the documentation
  • Peel back the onion: If you set it all correctly in WebSphere, step one pace back/up the chain of technology – it runs java, is java based -> you need to check up the chain to see what base java settings are in place, other than what you set yourself.
  • Don’t cry, it’s unbecoming
  • When friends who are kind enough to answer your Skype calls, LISTEN TO EACH QUESTION and think the answer through, you might not be seeing the forest because all those damn trees are in the way.
  • Say thank you – publicly. You might still be sitting there all night trying to figure out what went wrong

Technote: “Freemarker Template files are overwritten during IBM Connections CR2 install”


This happened to me, I was only saved by having a local back-up on my machine … don’t let it hit you!

Technote Link – swg21996243

Hving a good back-up before ANY upgrade, change etc. is important. If nothing else, do a backup config with WebSphere – that will capture all of the important files for you as well!

 

 

 

 

SPNEGO: Map SPNs and Create Combined Keytab Files In One Step


I have been wanting to blog about my SPNEGO install guide for a while but have been just a bit busy lately (my usual excuse). However, I just had to help a client setup SPNEGO for their IBM Connections environment so I decided the time for procrastination is over.

 

If you look at the IBM documentation, the process to create the SPNEGO keytab files and mapping the correct URLs and Fully Qualified Hostnames of servers to the AD account is rather onerous. IBM documentation will have you create separate keytab files for each url/FQHN that you want to include in the SPNEGO config and then merge them. For the normal user that is setting up SPNEGO for the fist time that is painful indeed and confusing. My process below does it all in one step (one step per URL/fqhn) and adds all the settings to ONE keytab file. I am usually done in 5 minutes and then create the config file using wsadmin commands and am up and running in SPNEGO in under an hour.

Note: all commands below have to happen ON AN AD DOMAIN CONTROLLER, running them on your workstation will not work.

 

Environment / Variables:

  • SPNEGOAD account: SPNEGOAccount@DOMAIN.COM – domain\SPNEGOAccount
  • Server FQHN: serverfqhn1.example.com, serverfqhn2.example.com, serverfqhn3.example.com, etc.
  • Connections URL (c-record): connections.example.com



Check Current SPN mappings for SPNEGO AD Account:

  • setspn -l SPNEGOAccount
    (review output)


Step 2: Add SPN mapping to SPNEGOAccount
 and create Keytab files

[setspn -s] or [setspn -a] could be used just to add/map the SPNs to the account, but this does not create the keytab files.

  • setspn -s HTTP/servernew.example.com SPNEGOAccount
  • setspn -s HTTP/newsite.example.com SPNEGOAccount

 

Run commands to create a SINGLE keytab file AND map accounts at the same time:

  • ktpass -princ HTTP/servernew.example.com@example.com -ptype KRB5_NT_PRINCIPAL -mapUser SPNEGOAccount -mapOp set -pass password1A -in C:\Temp\KRB\krb5.keytab-out C:\Temp\KRB\krb5.keytab
  • ktpass -princ HTTP/newsite.example.com@example.com -ptype KRB5_NT_PRINCIPAL -mapUser SPNEGOAccount -mapOp add -pass password1A -in C:\Temp\KRB\krb5.keytab -out C:\Temp\KRB\krb5.keytab

 

Note: the first command has the command [set], all the following commands (one for each url/fqhn you want to add) has the command [add]. If you do not use the [add] command, each of your subsequent commands will override your previous one, leaving your AD account with only one fqhn/URL mapped to it. THIS IS IMPORTANT!
Check whether the SPNS are all correct:

  • setspn -l SPNEGOAccount
    (get output and show it has mappings)
  • ldifde -f c:\temp\new-output1.txt -r “(servicePrincipalName=HTTP/ serverfqhn1.example.com)”
  • ldifde -f c:\temp\new-output2.txt -r “(servicePrincipalName=HTTP/connections.example.com)”
    (Get output files and review)

 

 

Some Gotchas

Which  URLs/c-records and server FQHNs to map:

I map EVERYTHING. The main reason is that often your C-record for the site (our example connections.example.com) will point to the fqhn of a server or a load balancing device. In that case you need BOTH of them mapped. I mal all webservers/HIS, WAS servers and (if existing) the LB address (this s usually overkill and not necessary … but paranoia pays off sometimes).

Command errors:

Depending you your AD forest, the above ktpass command might need the AD account your are mapping to either in the [ACCOUNTNAME@DOMAIN.COM] format or [DOMAIN\ACCOUNTNAME] format. You will see the error right away when you run it for the first time.

SPNEGO setting in WebSphere:

If you go by the IBM documentation (there is allot flying around) you will see they generally tell you to add the fqhn of the Deployment Manager as the HOSTNAME in SPNEGO. Keep in mind that works for them because generally they testers tend to work with single server test installs where ALL the systems run on one server and the Dmgr is also the HIS server and often they don’t bother to change the URL for the Connections setup. What you need in there is the C-Record your users will be putting into their browsers to get to Connections in in our example connections.example.com. Should the C-record point to the FQHN of a web server then you could input that address as well. That is why I generally map EVERYTHING, that way you have maximum flexibility should you need to finagle with your architecture and move functionality around.

Oops, you forgot something …

If you suddenly notice you have to add servers to the SPNEGO setup (maybe you are migrating) – DO NOT ADD MORE MAPPINGS TO THE SPNEGO AD ACCOUNT. That will invalidate the existing keytab files and you will have a n SSO outage. To add additional files you have to stop all WebSphere servers involved , add the mappings with the ktpass command using the [ADD] variable and use the existing keytab file from one of your WebSphere servers. Then recreate the config file using wsdmin and replace the old keytab files with the new one.

WebSphere – The Basics on Security, Directories and Federated Repositories


I had promised earlier this year to post more content (other than opinion and news) so I am now catching up on my promise. This post was inspired by a combined WebSphere – IBM Connections review review I did for a client earlier this year, along with some content from my IBM Connections admin training that I offer and that the same client asked me to give after they read my review of their environment. This is the first in a small series of blog-posts on security and configuration in WebSphere, look forward to some more in the next few weeks.

My Shameless Plug: You can get all of this in one big gulp if you hire me for some admin training for your support staff. I also do really kick-ass reviews of IBM Connections environments and performance tuning . . . .

WebSphere – LDAP / Security / Admin rights … the open door policy

I wrote an article on this webpage back in 2012 – WebSphere: wasadmin – how to recover a lost password – that also has something to do with this topic. This posting is in addition to that and will give you some more background info on how WebSphere keeps it’s security info and LDAP settings. If you read below you can find an even easier way to get that info …

XML – The Language of WebSphere

If you have not yet heard about it, here is the story: just about everything (regarding settings and configuration)in WebSphere is XML based. Yes, there are properties files and basic text files but the most files you will be dealing with are all XML files.

This results in Dr Vic’s first two rules:

Rule#1 – Always use a REAL XML editor program – and notepad.exe or wordpad.exe do not count. I personally have two favorites: Notepad++ on Windows and Geany on Linux (or Bluefish Editor – also awesome).

Rule#2 – Never putz (this is a technical term, I swear) in WebSphere XML files without having a back-up of each and every version of your change. If it gets really bad, you will have to re-install WebSphere and loose allot of work.

Shameless plug: I have more rules … hire me to learn more.

Federated Repository

The majority of my clients set up their LDAP settings in WebSphere by going to [Security – Global Security – Federated Repositories] and then never look at it again after that. They don’t really understand what the back-end is – well, here is a crash course:

Federated – The definition:

From late Latin foederatus, based on foedus, foeder- ‘league, covenant.’

Adj. 1. federated – united under a central government. Federate / united – characterized by unity; being or joined into a single entity; “presented a united front”

OK, what does this mean? When you installed WebSphere you were asked about an admin account and a password to assign to it – by default that account is called [wasadmin] though you can change it to anything you want. That user name and password is saved in a FILE BASED directory structure in the Deployment manager and replicated out to all federated nodes. When you add an LDAP directory then the Files based (the thing you see defined as [defaultWIMFilesBasedRealm] are federated meaning that now they are BOTH together part of a SINGLE directory entity that all WebSphere applications will utilize as a single unit for the purpose of user account look-ups and authentication.

The Files Involved:

wimconfig.xml

Is located in the [deployment manage profile]\config\cells\[cellname]\wim\config folder. This file contains the federated directory setting definitions. So the files based directory (more details below) and the LDAP directory/directories are all defined and configured in this file. As this file is an XML file, each directory is defined inside the <config:repositories and the </comfig:repositoes> items.

Let’s look at the example from my training WebSphere environment:

<config:repositories xsi:type=”config:FileRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.file.was.FileAdapter” id=”InternalFileRepository” supportPaging=”false” messageDigestAlgorithm=”SHA-1″>

<config:baseEntries name=”o=defaultWIMFileBasedRealm”/>

</config:repositories>

<config:repositories xsi:type=”config:LdapRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.ldap.LdapAdapter” id=”TTrainDom01″ isExtIdUnique=”true” supportAsyncMode=”false” supportExternalName=”false” supportPaging=”false” supportSorting=”false” supportTransactions=”false” supportChangeLog=”none”certificateFilter=”” certificateMapMode=”exactdn” ldapServerType=”DOMINO” translateRDN=”false”>

<config:baseEntries name=””/>

<config:loginProperties>uid</config:loginProperties>

<config:loginProperties>mail</config:loginProperties>

<config:loginProperties>cn</config:loginProperties>

<config:ldapServerConfiguration primaryServerQueryTimeInterval=”15″ returnToPrimaryServer=”true” sslConfiguration=””>

<config:ldapServers authentication=”simple” bindDN=”ldapaccess” bindPassword=”{xor}Dz4sLCgwLTtubWx+”

connectionPool=”false” connectTimeout=”20″ derefAliases=”always” referal=”ignore” sslEnabled=”false”>

<config:connections host=”ldap.intranet.toalsys.com” port=”389″/>

</config:ldapServers>

</config:ldapServerConfiguration>

This shows the two entries I have in my environment:

  • The default file based repository identified by the ID <id=”InternalFileRepository”>
  • My Domino based LDAP repository identified by the ID <id=”TTrainDom01″>

Gotcha #1: User Name and Password is Open

This wimconfig.xml contains the user name and encoded password for the LDAP bind account. Note the choice of words … ENCODED, not ENCRYPTED.

If you want to know the password for my training LDAP account copy the encoded password above and go to this link by Andrew Jones: http://www.poweredbywebsphere.com/decoder.html (thanks Andrew, I send all my clients to your site for further info and learning!)

If his is a production environment I have now gained access to an account in your environment, possibly an account that has update/write rights to the LDAP directory ….. all by looking at one file. If you are like 99.9% of my clients you are compromised:

  1. SECURE YOUR SERVER, LOCK DOWN YOUR FILE SYSTEM
  2. DON’T USE ADMIN OR PERSONAL ACCOUNTS TO BIND TO LDAP
  3. DON’T RE-USE THE SAME PASSWORD FOR ALL YOUR ACCOUNTS

Sound obvious, doesn’t it?

 Gotcha #2: Rogue LDAP entries

If you have ever tried to change an LDAP directory, replace and entry in WebSphere you might have run into the issue that you suddenly can’t log into WebSphere anymore after you made the changes. Why? Well, you need to understand that sometimes when you make changes, those old entries don’t disappear totally – they are left behind and impact you.

Remember the part about FEDERATED above? If not ALL directory entries here (in this file, not what shows in the IBM Console) are accessible and functioning, then the federated directory that you are trying to access will not work and you cannot authenticate. It is the Three Musketeer principle: “All for One, One for All”

Gotcha #3:

Some changes can’t be made in the interface. I had a client that mistakenly entered an LDAP directory as Microsoft AD but it was Domino. They tries to clean it up in this file but it still was not working and they could not log in ….. well, the wimconfig.xml contains allot of directory type specific settings which are set by the type: <ldapServerType=”DOMINO” > .. My advice is to remove the incorrect entry and enter a NEW entry at the same time and then make sure the old incorrect one is gone from the wimconfig.xml. DO NOT manually try to clean this up (other than remove the entry) as you might end up destroying the wimconfig.xml and making your environment unusable.

Remember Dr. Vic’s rule #2 above? Make back-ups before any changes to WebSphere security settings.

WebSphere: wasadmin – how to recover a lost password


I had the case recently where was working on a WebSphere 7 environment that was being uncooperative and the previous creator had not taken any backups and the documentation was rather thin . . . one of my most favorite scenarios to walk into.

I found myself with three sets of documents that all stated a different wasadmin password and none of them worked and the client had never bothered to set up any secondary WBM console admins. A dilemma I had to solve.

Encoding vs Encrypting

You might know that all sensitive information about security is entered into the security.xml document that can be found at [$WAS_HOME]/profiles/[profile name]/config/cells/[cell name] folder. In Windows this might equate to:

C:\IBM\Websphere\AppServer\profiles\Dmgr01\config\cells\cell01\security.xml

Linux/AIX would likely be something like:

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/cell01/security.xml

This document contains the name and password information for the primary admin account for the WebSphere cell – in most cases that will be the default account [wasadmin]. The password is, however, not encrypted but rather encoded. Encryption would use an encryption key to hash the password and without that key you would not be able to retrieve it. Encoding however is a whole other deal – the coding/decoding information is integral to WebSphere itself and is the same for any install anywhere in the world. That means if you encode the same password anywhere, the resulting hash will be exactly the same no matter which server you do it on.

Now, this is not great security in and upon itself and I will not go into details on this – other than it is really important to lock down the physical access to to any WebSphere server you are in charge of, all the way down to file rights …. or you might regret it at some later time.

How to Decrypt:

I am not the first blogger out there that is writing about this, but nobody every wrote it out for Windows servers so I am going to concentrate on that OS right now, and most of the blog entries out there are for older versions and the proces has changed since. Here some of the articles that I have read over the last few years Robert Farstad, Robert Maldon,  and a few more . . . . google the conent here and you will find them.

Here some basic details:

  • WebSphere Version: 7.0.0.21 (the process is the same for any V 7.x server)
  • $WAS_HOME=C:\IBM\WebSphere\AppServer

Step 1: find the wasadmin information

Open the security.xml, find the entry for the encrypted password: it always starts with {xor}, in my case it is:

userId=”wasadmin” password=”{xor}LDo8LTor”

Step 2: Find your WAS Version Specific Java Plug-in Folder:

In my case it was:

C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\

Step 3: Find your java home and open a command prompt

In my case this equates to

C:\IBM\WebSphere\AppServer\java\bin\

Change to this folder in the command prompt you opened.

Step 4: Run the Password Encoder/Decoder:

This is where you need the folder location and the encoded password you looked up in the previous steps.

In C:\IBM\WebSphere\AppServer\java\bin\ run the following command

java – java.ext.dirs=C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\wasJars\ -cp securityimpl.jar:iwsorb.jar com.ibm.ws.security.util.PasswordDecoder {xor}LDo8LTor

This above command is one long command string (it might wrap depending on your screen) and it will create the following output in the command prompt:

encoded password == “{xor}LDo8LTor”, decoded password == “secret”

The process for Linux/AIX is basically the same, however the folder structure will be different. The commands are about the same but depending on which version of Linux you are running the Java switches might need some fidlding – though the base does not change.

Security!

Next to being helpful to retrieve lost passwords, this article hopefully also shows you just how important good physical security for your WebSphere servers is – don’t think that just because you have a log-on or locked down root that you are safe.

WebSphere: Errors installing Plug-in fix pack on IHS V7.x


This was a new one, not even the IBMers that I consulted with had run into this before.

I have been working at a client site on a large IBM Connections project since last year – V3.0.0, upgrade to 3.0.1, upgrade to 3.0.1.1 … now multiple code drops for V4 beta installs and preparation to get gold code V4 up as soon as possible (once it is released). In the course of the last year I have probably installed and upgraded more WAS and IHS servers that I have in several years previously – loving every moment of it!

Problem:

Today I had a new V 7.0 IHS on AIX to set up and we were running into issues installing the Plug-in fix for 7.0.0.21. The IHS FPO went without a problem, but the plug-in did not work. Errors, errors, errors:

java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)
java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)         at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)

Nothing was working … I looked at fie permissions, ownership etc. – no change. root, or no root – it failed. I did some searching and after a while came across this tech noteswg21408430.

the errors were close enough – I was using the latest version of the IBM UpdateInstaller (at this time 7.0.0.23) but I decided to wipe out the UDI, install it once more new into a DIFFERENT folder (queue the drum roll) …. that made the difference.

So, sometimes it is not the tool being installed, but rather the tool doing the install that is at fault.  AND – UNIX file permissions are not always at fault either. Poor little Unix – there’s good boy!!

WebSphere – The backupConfig script your friend


This is just a short post – But the built-in utility you get with the backupConfig script is worth looking into for everybody!

if you have worked with WebSphere for any significant time you have come across the built-in backup and restore utility that each WebSphere server has by default: [backupconfig.bat] or on Unix/Linux [backupconfig.sh] and the corresponding restore scripts [restoreConfig.bat] and [restoreConfig.sh]. 

At my current client we are working on application customizations and testing them on new servers. This is where the backupConfig comes in handy as it does not just back-up your application server(s) but the deployment manager and all the node(s) configuration as well – so you can replay a whole server configuration along with the installed applications and any application specific configuration. backupConfig can also be used to migrate servers from one piece of hardware to another (or VM server or, …. any combination is possible).

The process is simple: find the scripts in the [/bin] folder of either the Dmgr or the node,  execute the script and it will check your servers configuration, stop all server instances including nodeagents and then create a zip file of ALL files necessary for the back-up – and all of this wonderfulness is  unencumbered by the human thought process … 🙂

Whenever I am about to install a new application, install any fixes or make configuration changes to a WebSphere  server I run the backupConfig script once first and keep a copy of the zip file it created on a local machine – just to be safe.

Where to run:

Depending on your architecture you can run it on the Deployment Manager and/or all nodes. When you run it on the Deployment manager it will grab all the configuration for the Dmgr, nodes and application servers in one go. This is essential if you have to restore an environment. On a managed node (separate HW) it will capture the configuration and applications installed on that physical node – so you might need to run it on each physical WebSphere server in your environment once to get a total base back-up. Once I have that I usually just run the scripts on the deployment manager as most of the work happens there anyway and all changes are synchronized out to the nodes.

Restores:

On windows it is simple – just run the restoreConfig script and tell it which zip file to use … and presto. On Unix/Linux you have to think a bit more. The backupConfig script does not keep any file rights or ownership information, when restoring it basically sets the file ownership to the account being used to run the script – so make sure you are using the same account and have read/write rights to the folders involved. 

Here is the link to the documentation in the WebSphere Infocenter – I hope you find it useful and make a back-up of your servers soon!