SPNEGO: Map SPNs and Create Combined Keytab Files In One Step


I have been wanting to blog about my SPNEGO install guide for a while but have been just a bit busy lately (my usual excuse). However, I just had to help a client setup SPNEGO for their IBM Connections environment so I decided the time for procrastination is over.

 

If you look at the IBM documentation, the process to create the SPNEGO keytab files and mapping the correct URLs and Fully Qualified Hostnames of servers to the AD account is rather onerous. IBM documentation will have you create separate keytab files for each url/FQHN that you want to include in the SPNEGO config and then merge them. For the normal user that is setting up SPNEGO for the fist time that is painful indeed and confusing. My process below does it all in one step (one step per URL/fqhn) and adds all the settings to ONE keytab file. I am usually done in 5 minutes and then create the config file using wsadmin commands and am up and running in SPNEGO in under an hour.

Note: all commands below have to happen ON AN AD DOMAIN CONTROLLER, running them on your workstation will not work.

 

Environment / Variables:

  • SPNEGOAD account: SPNEGOAccount@DOMAIN.COM – domain\SPNEGOAccount
  • Server FQHN: serverfqhn1.example.com, serverfqhn2.example.com, serverfqhn3.example.com, etc.
  • Connections URL (c-record): connections.example.com



Check Current SPN mappings for SPNEGO AD Account:

  • setspn -l SPNEGOAccount
    (review output)


Step 2: Add SPN mapping to SPNEGOAccount
 and create Keytab files

[setspn -s] or [setspn -a] could be used just to add/map the SPNs to the account, but this does not create the keytab files.

  • setspn -s HTTP/servernew.example.com SPNEGOAccount
  • setspn -s HTTP/newsite.example.com SPNEGOAccount

 

Run commands to create a SINGLE keytab file AND map accounts at the same time:

  • ktpass -princ HTTP/servernew.example.com@example.com -ptype KRB5_NT_PRINCIPAL -mapUser SPNEGOAccount -mapOp set -pass password1A -in C:\Temp\KRB\krb5.keytab-out C:\Temp\KRB\krb5.keytab
  • ktpass -princ HTTP/newsite.example.com@example.com -ptype KRB5_NT_PRINCIPAL -mapUser SPNEGOAccount -mapOp add -pass password1A -in C:\Temp\KRB\krb5.keytab -out C:\Temp\KRB\krb5.keytab

 

Note: the first command has the command [set], all the following commands (one for each url/fqhn you want to add) has the command [add]. If you do not use the [add] command, each of your subsequent commands will override your previous one, leaving your AD account with only one fqhn/URL mapped to it. THIS IS IMPORTANT!
Check whether the SPNS are all correct:

  • setspn -l SPNEGOAccount
    (get output and show it has mappings)
  • ldifde -f c:\temp\new-output1.txt -r “(servicePrincipalName=HTTP/ serverfqhn1.example.com)”
  • ldifde -f c:\temp\new-output2.txt -r “(servicePrincipalName=HTTP/connections.example.com)”
    (Get output files and review)

 

 

Some Gotchas

Which  URLs/c-records and server FQHNs to map:

I map EVERYTHING. The main reason is that often your C-record for the site (our example connections.example.com) will point to the fqhn of a server or a load balancing device. In that case you need BOTH of them mapped. I mal all webservers/HIS, WAS servers and (if existing) the LB address (this s usually overkill and not necessary … but paranoia pays off sometimes).

Command errors:

Depending you your AD forest, the above ktpass command might need the AD account your are mapping to either in the [ACCOUNTNAME@DOMAIN.COM] format or [DOMAIN\ACCOUNTNAME] format. You will see the error right away when you run it for the first time.

SPNEGO setting in WebSphere:

If you go by the IBM documentation (there is allot flying around) you will see they generally tell you to add the fqhn of the Deployment Manager as the HOSTNAME in SPNEGO. Keep in mind that works for them because generally they testers tend to work with single server test installs where ALL the systems run on one server and the Dmgr is also the HIS server and often they don’t bother to change the URL for the Connections setup. What you need in there is the C-Record your users will be putting into their browsers to get to Connections in in our example connections.example.com. Should the C-record point to the FQHN of a web server then you could input that address as well. That is why I generally map EVERYTHING, that way you have maximum flexibility should you need to finagle with your architecture and move functionality around.

Oops, you forgot something …

If you suddenly notice you have to add servers to the SPNEGO setup (maybe you are migrating) – DO NOT ADD MORE MAPPINGS TO THE SPNEGO AD ACCOUNT. That will invalidate the existing keytab files and you will have a n SSO outage. To add additional files you have to stop all WebSphere servers involved , add the mappings with the ktpass command using the [ADD] variable and use the existing keytab file from one of your WebSphere servers. Then recreate the config file using wsdmin and replace the old keytab files with the new one.

WebSphere – The Basics on Security, Directories and Federated Repositories


I had promised earlier this year to post more content (other than opinion and news) so I am now catching up on my promise. This post was inspired by a combined WebSphere – IBM Connections review review I did for a client earlier this year, along with some content from my IBM Connections admin training that I offer and that the same client asked me to give after they read my review of their environment. This is the first in a small series of blog-posts on security and configuration in WebSphere, look forward to some more in the next few weeks.

My Shameless Plug: You can get all of this in one big gulp if you hire me for some admin training for your support staff. I also do really kick-ass reviews of IBM Connections environments and performance tuning . . . .

WebSphere – LDAP / Security / Admin rights … the open door policy

I wrote an article on this webpage back in 2012 – WebSphere: wasadmin – how to recover a lost password – that also has something to do with this topic. This posting is in addition to that and will give you some more background info on how WebSphere keeps it’s security info and LDAP settings. If you read below you can find an even easier way to get that info …

XML – The Language of WebSphere

If you have not yet heard about it, here is the story: just about everything (regarding settings and configuration)in WebSphere is XML based. Yes, there are properties files and basic text files but the most files you will be dealing with are all XML files.

This results in Dr Vic’s first two rules:

Rule#1 – Always use a REAL XML editor program – and notepad.exe or wordpad.exe do not count. I personally have two favorites: Notepad++ on Windows and Geany on Linux (or Bluefish Editor – also awesome).

Rule#2 – Never putz (this is a technical term, I swear) in WebSphere XML files without having a back-up of each and every version of your change. If it gets really bad, you will have to re-install WebSphere and loose allot of work.

Shameless plug: I have more rules … hire me to learn more.

Federated Repository

The majority of my clients set up their LDAP settings in WebSphere by going to [Security – Global Security – Federated Repositories] and then never look at it again after that. They don’t really understand what the back-end is – well, here is a crash course:

Federated – The definition:

From late Latin foederatus, based on foedus, foeder- ‘league, covenant.’

Adj. 1. federated – united under a central government. Federate / united – characterized by unity; being or joined into a single entity; “presented a united front”

OK, what does this mean? When you installed WebSphere you were asked about an admin account and a password to assign to it – by default that account is called [wasadmin] though you can change it to anything you want. That user name and password is saved in a FILE BASED directory structure in the Deployment manager and replicated out to all federated nodes. When you add an LDAP directory then the Files based (the thing you see defined as [defaultWIMFilesBasedRealm] are federated meaning that now they are BOTH together part of a SINGLE directory entity that all WebSphere applications will utilize as a single unit for the purpose of user account look-ups and authentication.

The Files Involved:

wimconfig.xml

Is located in the [deployment manage profile]\config\cells\[cellname]\wim\config folder. This file contains the federated directory setting definitions. So the files based directory (more details below) and the LDAP directory/directories are all defined and configured in this file. As this file is an XML file, each directory is defined inside the <config:repositories and the </comfig:repositoes> items.

Let’s look at the example from my training WebSphere environment:

<config:repositories xsi:type=”config:FileRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.file.was.FileAdapter” id=”InternalFileRepository” supportPaging=”false” messageDigestAlgorithm=”SHA-1″>

<config:baseEntries name=”o=defaultWIMFileBasedRealm”/>

</config:repositories>

<config:repositories xsi:type=”config:LdapRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.ldap.LdapAdapter” id=”TTrainDom01″ isExtIdUnique=”true” supportAsyncMode=”false” supportExternalName=”false” supportPaging=”false” supportSorting=”false” supportTransactions=”false” supportChangeLog=”none”certificateFilter=”” certificateMapMode=”exactdn” ldapServerType=”DOMINO” translateRDN=”false”>

<config:baseEntries name=””/>

<config:loginProperties>uid</config:loginProperties>

<config:loginProperties>mail</config:loginProperties>

<config:loginProperties>cn</config:loginProperties>

<config:ldapServerConfiguration primaryServerQueryTimeInterval=”15″ returnToPrimaryServer=”true” sslConfiguration=””>

<config:ldapServers authentication=”simple” bindDN=”ldapaccess” bindPassword=”{xor}Dz4sLCgwLTtubWx+”

connectionPool=”false” connectTimeout=”20″ derefAliases=”always” referal=”ignore” sslEnabled=”false”>

<config:connections host=”ldap.intranet.toalsys.com” port=”389″/>

</config:ldapServers>

</config:ldapServerConfiguration>

This shows the two entries I have in my environment:

  • The default file based repository identified by the ID <id=”InternalFileRepository”>
  • My Domino based LDAP repository identified by the ID <id=”TTrainDom01″>

Gotcha #1: User Name and Password is Open

This wimconfig.xml contains the user name and encoded password for the LDAP bind account. Note the choice of words … ENCODED, not ENCRYPTED.

If you want to know the password for my training LDAP account copy the encoded password above and go to this link by Andrew Jones: http://www.poweredbywebsphere.com/decoder.html (thanks Andrew, I send all my clients to your site for further info and learning!)

If his is a production environment I have now gained access to an account in your environment, possibly an account that has update/write rights to the LDAP directory ….. all by looking at one file. If you are like 99.9% of my clients you are compromised:

  1. SECURE YOUR SERVER, LOCK DOWN YOUR FILE SYSTEM
  2. DON’T USE ADMIN OR PERSONAL ACCOUNTS TO BIND TO LDAP
  3. DON’T RE-USE THE SAME PASSWORD FOR ALL YOUR ACCOUNTS

Sound obvious, doesn’t it?

 Gotcha #2: Rogue LDAP entries

If you have ever tried to change an LDAP directory, replace and entry in WebSphere you might have run into the issue that you suddenly can’t log into WebSphere anymore after you made the changes. Why? Well, you need to understand that sometimes when you make changes, those old entries don’t disappear totally – they are left behind and impact you.

Remember the part about FEDERATED above? If not ALL directory entries here (in this file, not what shows in the IBM Console) are accessible and functioning, then the federated directory that you are trying to access will not work and you cannot authenticate. It is the Three Musketeer principle: “All for One, One for All”

Gotcha #3:

Some changes can’t be made in the interface. I had a client that mistakenly entered an LDAP directory as Microsoft AD but it was Domino. They tries to clean it up in this file but it still was not working and they could not log in ….. well, the wimconfig.xml contains allot of directory type specific settings which are set by the type: <ldapServerType=”DOMINO” > .. My advice is to remove the incorrect entry and enter a NEW entry at the same time and then make sure the old incorrect one is gone from the wimconfig.xml. DO NOT manually try to clean this up (other than remove the entry) as you might end up destroying the wimconfig.xml and making your environment unusable.

Remember Dr. Vic’s rule #2 above? Make back-ups before any changes to WebSphere security settings.

WebSphere: wasadmin – how to recover a lost password


I had the case recently where was working on a WebSphere 7 environment that was being uncooperative and the previous creator had not taken any backups and the documentation was rather thin . . . one of my most favorite scenarios to walk into.

I found myself with three sets of documents that all stated a different wasadmin password and none of them worked and the client had never bothered to set up any secondary WBM console admins. A dilemma I had to solve.

Encoding vs Encrypting

You might know that all sensitive information about security is entered into the security.xml document that can be found at [$WAS_HOME]/profiles/[profile name]/config/cells/[cell name] folder. In Windows this might equate to:

C:\IBM\Websphere\AppServer\profiles\Dmgr01\config\cells\cell01\security.xml

Linux/AIX would likely be something like:

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/cell01/security.xml

This document contains the name and password information for the primary admin account for the WebSphere cell – in most cases that will be the default account [wasadmin]. The password is, however, not encrypted but rather encoded. Encryption would use an encryption key to hash the password and without that key you would not be able to retrieve it. Encoding however is a whole other deal – the coding/decoding information is integral to WebSphere itself and is the same for any install anywhere in the world. That means if you encode the same password anywhere, the resulting hash will be exactly the same no matter which server you do it on.

Now, this is not great security in and upon itself and I will not go into details on this – other than it is really important to lock down the physical access to to any WebSphere server you are in charge of, all the way down to file rights …. or you might regret it at some later time.

How to Decrypt:

I am not the first blogger out there that is writing about this, but nobody every wrote it out for Windows servers so I am going to concentrate on that OS right now, and most of the blog entries out there are for older versions and the proces has changed since. Here some of the articles that I have read over the last few years Robert Farstad, Robert Maldon,  and a few more . . . . google the conent here and you will find them.

Here some basic details:

  • WebSphere Version: 7.0.0.21 (the process is the same for any V 7.x server)
  • $WAS_HOME=C:\IBM\WebSphere\AppServer

Step 1: find the wasadmin information

Open the security.xml, find the entry for the encrypted password: it always starts with {xor}, in my case it is:

userId=”wasadmin” password=”{xor}LDo8LTor”

Step 2: Find your WAS Version Specific Java Plug-in Folder:

In my case it was:

C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\

Step 3: Find your java home and open a command prompt

In my case this equates to

C:\IBM\WebSphere\AppServer\java\bin\

Change to this folder in the command prompt you opened.

Step 4: Run the Password Encoder/Decoder:

This is where you need the folder location and the encoded password you looked up in the previous steps.

In C:\IBM\WebSphere\AppServer\java\bin\ run the following command

java – java.ext.dirs=C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\wasJars\ -cp securityimpl.jar:iwsorb.jar com.ibm.ws.security.util.PasswordDecoder {xor}LDo8LTor

This above command is one long command string (it might wrap depending on your screen) and it will create the following output in the command prompt:

encoded password == “{xor}LDo8LTor”, decoded password == “secret”

The process for Linux/AIX is basically the same, however the folder structure will be different. The commands are about the same but depending on which version of Linux you are running the Java switches might need some fidlding – though the base does not change.

Security!

Next to being helpful to retrieve lost passwords, this article hopefully also shows you just how important good physical security for your WebSphere servers is – don’t think that just because you have a log-on or locked down root that you are safe.

WebSphere: Errors installing Plug-in fix pack on IHS V7.x


This was a new one, not even the IBMers that I consulted with had run into this before.

I have been working at a client site on a large IBM Connections project since last year – V3.0.0, upgrade to 3.0.1, upgrade to 3.0.1.1 … now multiple code drops for V4 beta installs and preparation to get gold code V4 up as soon as possible (once it is released). In the course of the last year I have probably installed and upgraded more WAS and IHS servers that I have in several years previously – loving every moment of it!

Problem:

Today I had a new V 7.0 IHS on AIX to set up and we were running into issues installing the Plug-in fix for 7.0.0.21. The IHS FPO went without a problem, but the plug-in did not work. Errors, errors, errors:

java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)
java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)         at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)

Nothing was working … I looked at fie permissions, ownership etc. – no change. root, or no root – it failed. I did some searching and after a while came across this tech noteswg21408430.

the errors were close enough – I was using the latest version of the IBM UpdateInstaller (at this time 7.0.0.23) but I decided to wipe out the UDI, install it once more new into a DIFFERENT folder (queue the drum roll) …. that made the difference.

So, sometimes it is not the tool being installed, but rather the tool doing the install that is at fault.  AND – UNIX file permissions are not always at fault either. Poor little Unix – there’s good boy!!

WebSphere – The backupConfig script your friend


This is just a short post – But the built-in utility you get with the backupConfig script is worth looking into for everybody!

if you have worked with WebSphere for any significant time you have come across the built-in backup and restore utility that each WebSphere server has by default: [backupconfig.bat] or on Unix/Linux [backupconfig.sh] and the corresponding restore scripts [restoreConfig.bat] and [restoreConfig.sh]. 

At my current client we are working on application customizations and testing them on new servers. This is where the backupConfig comes in handy as it does not just back-up your application server(s) but the deployment manager and all the node(s) configuration as well – so you can replay a whole server configuration along with the installed applications and any application specific configuration. backupConfig can also be used to migrate servers from one piece of hardware to another (or VM server or, …. any combination is possible).

The process is simple: find the scripts in the [/bin] folder of either the Dmgr or the node,  execute the script and it will check your servers configuration, stop all server instances including nodeagents and then create a zip file of ALL files necessary for the back-up – and all of this wonderfulness is  unencumbered by the human thought process … :)

Whenever I am about to install a new application, install any fixes or make configuration changes to a WebSphere  server I run the backupConfig script once first and keep a copy of the zip file it created on a local machine – just to be safe.

Where to run:

Depending on your architecture you can run it on the Deployment Manager and/or all nodes. When you run it on the Deployment manager it will grab all the configuration for the Dmgr, nodes and application servers in one go. This is essential if you have to restore an environment. On a managed node (separate HW) it will capture the configuration and applications installed on that physical node – so you might need to run it on each physical WebSphere server in your environment once to get a total base back-up. Once I have that I usually just run the scripts on the deployment manager as most of the work happens there anyway and all changes are synchronized out to the nodes.

Restores:

On windows it is simple – just run the restoreConfig script and tell it which zip file to use … and presto. On Unix/Linux you have to think a bit more. The backupConfig script does not keep any file rights or ownership information, when restoring it basically sets the file ownership to the account being used to run the script – so make sure you are using the same account and have read/write rights to the folders involved. 

Here is the link to the documentation in the WebSphere Infocenter - I hope you find it useful and make a back-up of your servers soon!

Get Free Training! New Complimentary IBM WebSphere Education Courses Available from the Global WebSphere Community


I think everybody following me knows that I am telling everybody to learn WebSphere – it will come your way like it or not. Here are some free training courses/videos that you can look at – all you have to do is become a member of the Global WebSphere Community – and PRESTO – free training for you. I will be looking at this later tonight …

Dear WebSpherians,Last year, we polled our Global WebSphere Community members asking you what WebSphere topics you were most interested in having additional training information on from the GWC. The results came back showing a strong interest in WebSphere Application Server and WebSphere MQ. The Global WebSphere Community is pleased to announce the availability of these complimentary IBM WebSphere Education courses to our GWC members.We are pleased to offer the following courses to our members:

  • WebSphere MQ v7 Installation and Configuration
  • WebSphere MQ V7 High Availability Considerations
  • Using WebSphere MQ V7 Traces, Error Logs, and Failure Data Capture Files
  • WebSphere MQ V7 Clustering
  • WebSphere Application Server V8 Architecture
  • WebSphere Application Server v8 Workload Management

Click here for more information or to view one of the courses

Sincerely

The Global WebSphere Community Management Team

Join in the discussion
Follow us on TwitterFacebookFacebook
This email was sent to victor@toalsys.com.If you no longer wish to receive information about the Global WebSphere Community, please inform us here.20 Carematrix Drive, Dedham, MA 02026, USA

N-11691884507-ADA79BCFCF96201E62F8F1BD0CB67ED1.jpg

Java: it’s your fault! Connections on AIX


Just a quick one during my lunch hour …. ran into an issue yesterday at my current client that shows once more that when you do not work with a specific OS for a while, you really loose your touch for the small details.

The Saga of WAS, AIX and the damn Java Cache

We installed iFixes yesterday and that all went well. However the syncing of the nodes (kicked off from the Dmgr console) took forever, and then one of the app clusters on one of the nodes would not restart (it eventually did after 4 hours).

To clean the system and get rid of any old temp files we:

  • Stopped all WebSphere servers:  /WAS_Profile/bin/stopServer.sh xxx -user xxx -password ****
  • Stopped the NodeAgent:  /WAS_Profile/bin/stopNode.sh -username xxxx -password ****
  • Cleaned all temp files /WAS_Profile/temp  and /wstemp (everything inside of both folders)
  • Ran  /WAS_Profile/bin/osgiCfgInit.sh
  • Ran  /WAS_Profile/bin/clearClassCache.sh

Note: you can also use the command “./stopNode.sh -stopservers -username xxxx -password ****” to shut down the node agent AND the servers at the same time. We wanted to see the individual servers come down as we had issues with one of them.

We then tried to restart the node agent ….. and it failed. We found this in the startserver.log for the node agent:

ADMU3011E: Server launched but failed initialization

Damn, nothing worked … re-cleaned, checked, cursed, cried ….. and then opened a Sev 1 ticket with IBM support online. (had a REALLY fast response – thanks guys!)

The Cavalry to the Rescue …

The Connections support guy had a look at the logs and brought in a WAS support specialist who had me repeat the clean-ups steps above AND clean this location as well (everything in this folder, but not delete the folder itself):

/tmp/javasharedresources

The IBM tech thinks we had a corrupted system level java cache that was causing the issue.  After that a ./startNode.sh worked like a charm and the servers started fine as well.

Total Clean-up

Incidentally, we ended up shutting each AIX WAS server (including the Dmgr) down one by one so we do not have a service outage and ran the above maintenance once more. On the nodes we also ran a “./synchNode.sh” with the node agent turned off – just to eliminate any possibility of the nodes maybe being out of synch (thanks for the idea Stuart).

We will also be going through our automated scripts to test adding some more items to them (email notifications when individual steps are done, add the “/temp/javasharedresources” to the list of folders to be cleaned,  etc.).

Lessons to be learned:

  • When you don’t work with an OS for a while you forget the important SMALL stuff (/tmp/javasharedresources) – I had run into this very issue a few years ago and totally forgot about it. I actually did not remember it until this morning, the day after.
  • When in doubt – call support RIGHT AWAY, if for no other reason than to validate your thought process is correct and you are not barking up the wrong tree. We did not wait very long to call, but sometimes even 5 minutes can mean the difference between failure and success.