WebSphere – The Basics on Security, Directories and Federated Repositories


I had promised earlier this year to post more content (other than opinion and news) so I am now catching up on my promise. This post was inspired by a combined WebSphere – IBM Connections review review I did for a client earlier this year, along with some content from my IBM Connections admin training that I offer and that the same client asked me to give after they read my review of their environment. This is the first in a small series of blog-posts on security and configuration in WebSphere, look forward to some more in the next few weeks.

My Shameless Plug: You can get all of this in one big gulp if you hire me for some admin training for your support staff. I also do really kick-ass reviews of IBM Connections environments and performance tuning . . . .

WebSphere – LDAP / Security / Admin rights … the open door policy

I wrote an article on this webpage back in 2012 – WebSphere: wasadmin – how to recover a lost password – that also has something to do with this topic. This posting is in addition to that and will give you some more background info on how WebSphere keeps it’s security info and LDAP settings. If you read below you can find an even easier way to get that info …

XML – The Language of WebSphere

If you have not yet heard about it, here is the story: just about everything (regarding settings and configuration)in WebSphere is XML based. Yes, there are properties files and basic text files but the most files you will be dealing with are all XML files.

This results in Dr Vic’s first two rules:

Rule#1 – Always use a REAL XML editor program – and notepad.exe or wordpad.exe do not count. I personally have two favorites: Notepad++ on Windows and Geany on Linux (or Bluefish Editor – also awesome).

Rule#2 – Never putz (this is a technical term, I swear) in WebSphere XML files without having a back-up of each and every version of your change. If it gets really bad, you will have to re-install WebSphere and loose allot of work.

Shameless plug: I have more rules … hire me to learn more.

Federated Repository

The majority of my clients set up their LDAP settings in WebSphere by going to [Security – Global Security – Federated Repositories] and then never look at it again after that. They don’t really understand what the back-end is – well, here is a crash course:

Federated – The definition:

From late Latin foederatus, based on foedus, foeder- ‘league, covenant.’

Adj. 1. federated – united under a central government. Federate / united – characterized by unity; being or joined into a single entity; “presented a united front”

OK, what does this mean? When you installed WebSphere you were asked about an admin account and a password to assign to it – by default that account is called [wasadmin] though you can change it to anything you want. That user name and password is saved in a FILE BASED directory structure in the Deployment manager and replicated out to all federated nodes. When you add an LDAP directory then the Files based (the thing you see defined as [defaultWIMFilesBasedRealm] are federated meaning that now they are BOTH together part of a SINGLE directory entity that all WebSphere applications will utilize as a single unit for the purpose of user account look-ups and authentication.

The Files Involved:

wimconfig.xml

Is located in the [deployment manage profile]\config\cells\[cellname]\wim\config folder. This file contains the federated directory setting definitions. So the files based directory (more details below) and the LDAP directory/directories are all defined and configured in this file. As this file is an XML file, each directory is defined inside the <config:repositories and the </comfig:repositoes> items.

Let’s look at the example from my training WebSphere environment:

<config:repositories xsi:type=”config:FileRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.file.was.FileAdapter” id=”InternalFileRepository” supportPaging=”false” messageDigestAlgorithm=”SHA-1″>

<config:baseEntries name=”o=defaultWIMFileBasedRealm”/>

</config:repositories>

<config:repositories xsi:type=”config:LdapRepositoryType” adapterClassName=”com.ibm.ws.wim.adapter.ldap.LdapAdapter” id=”TTrainDom01″ isExtIdUnique=”true” supportAsyncMode=”false” supportExternalName=”false” supportPaging=”false” supportSorting=”false” supportTransactions=”false” supportChangeLog=”none”certificateFilter=”” certificateMapMode=”exactdn” ldapServerType=”DOMINO” translateRDN=”false”>

<config:baseEntries name=””/>

<config:loginProperties>uid</config:loginProperties>

<config:loginProperties>mail</config:loginProperties>

<config:loginProperties>cn</config:loginProperties>

<config:ldapServerConfiguration primaryServerQueryTimeInterval=”15″ returnToPrimaryServer=”true” sslConfiguration=””>

<config:ldapServers authentication=”simple” bindDN=”ldapaccess” bindPassword=”{xor}Dz4sLCgwLTtubWx+”

connectionPool=”false” connectTimeout=”20″ derefAliases=”always” referal=”ignore” sslEnabled=”false”>

<config:connections host=”ldap.intranet.toalsys.com” port=”389″/>

</config:ldapServers>

</config:ldapServerConfiguration>

This shows the two entries I have in my environment:

  • The default file based repository identified by the ID <id=”InternalFileRepository”>
  • My Domino based LDAP repository identified by the ID <id=”TTrainDom01″>

Gotcha #1: User Name and Password is Open

This wimconfig.xml contains the user name and encoded password for the LDAP bind account. Note the choice of words … ENCODED, not ENCRYPTED.

If you want to know the password for my training LDAP account copy the encoded password above and go to this link by Andrew Jones: http://www.poweredbywebsphere.com/decoder.html (thanks Andrew, I send all my clients to your site for further info and learning!)

If his is a production environment I have now gained access to an account in your environment, possibly an account that has update/write rights to the LDAP directory ….. all by looking at one file. If you are like 99.9% of my clients you are compromised:

  1. SECURE YOUR SERVER, LOCK DOWN YOUR FILE SYSTEM
  2. DON’T USE ADMIN OR PERSONAL ACCOUNTS TO BIND TO LDAP
  3. DON’T RE-USE THE SAME PASSWORD FOR ALL YOUR ACCOUNTS

Sound obvious, doesn’t it?

 Gotcha #2: Rogue LDAP entries

If you have ever tried to change an LDAP directory, replace and entry in WebSphere you might have run into the issue that you suddenly can’t log into WebSphere anymore after you made the changes. Why? Well, you need to understand that sometimes when you make changes, those old entries don’t disappear totally – they are left behind and impact you.

Remember the part about FEDERATED above? If not ALL directory entries here (in this file, not what shows in the IBM Console) are accessible and functioning, then the federated directory that you are trying to access will not work and you cannot authenticate. It is the Three Musketeer principle: “All for One, One for All”

Gotcha #3:

Some changes can’t be made in the interface. I had a client that mistakenly entered an LDAP directory as Microsoft AD but it was Domino. They tries to clean it up in this file but it still was not working and they could not log in ….. well, the wimconfig.xml contains allot of directory type specific settings which are set by the type: <ldapServerType=”DOMINO” > .. My advice is to remove the incorrect entry and enter a NEW entry at the same time and then make sure the old incorrect one is gone from the wimconfig.xml. DO NOT manually try to clean this up (other than remove the entry) as you might end up destroying the wimconfig.xml and making your environment unusable.

Remember Dr. Vic’s rule #2 above? Make back-ups before any changes to WebSphere security settings.

WebSphere: wasadmin – how to recover a lost password


I had the case recently where was working on a WebSphere 7 environment that was being uncooperative and the previous creator had not taken any backups and the documentation was rather thin . . . one of my most favorite scenarios to walk into.

I found myself with three sets of documents that all stated a different wasadmin password and none of them worked and the client had never bothered to set up any secondary WBM console admins. A dilemma I had to solve.

Encoding vs Encrypting

You might know that all sensitive information about security is entered into the security.xml document that can be found at [$WAS_HOME]/profiles/[profile name]/config/cells/[cell name] folder. In Windows this might equate to:

C:\IBM\Websphere\AppServer\profiles\Dmgr01\config\cells\cell01\security.xml

Linux/AIX would likely be something like:

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/cell01/security.xml

This document contains the name and password information for the primary admin account for the WebSphere cell – in most cases that will be the default account [wasadmin]. The password is, however, not encrypted but rather encoded. Encryption would use an encryption key to hash the password and without that key you would not be able to retrieve it. Encoding however is a whole other deal – the coding/decoding information is integral to WebSphere itself and is the same for any install anywhere in the world. That means if you encode the same password anywhere, the resulting hash will be exactly the same no matter which server you do it on.

Now, this is not great security in and upon itself and I will not go into details on this – other than it is really important to lock down the physical access to to any WebSphere server you are in charge of, all the way down to file rights …. or you might regret it at some later time.

How to Decrypt:

I am not the first blogger out there that is writing about this, but nobody every wrote it out for Windows servers so I am going to concentrate on that OS right now, and most of the blog entries out there are for older versions and the proces has changed since. Here some of the articles that I have read over the last few years Robert Farstad, Robert Maldon,  and a few more . . . . google the conent here and you will find them.

Here some basic details:

  • WebSphere Version: 7.0.0.21 (the process is the same for any V 7.x server)
  • $WAS_HOME=C:\IBM\WebSphere\AppServer

Step 1: find the wasadmin information

Open the security.xml, find the entry for the encrypted password: it always starts with {xor}, in my case it is:

userId=”wasadmin” password=”{xor}LDo8LTor”

Step 2: Find your WAS Version Specific Java Plug-in Folder:

In my case it was:

C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\

Step 3: Find your java home and open a command prompt

In my case this equates to

C:\IBM\WebSphere\AppServer\java\bin\

Change to this folder in the command prompt you opened.

Step 4: Run the Password Encoder/Decoder:

This is where you need the folder location and the encoded password you looked up in the previous steps.

In C:\IBM\WebSphere\AppServer\java\bin\ run the following command

java – java.ext.dirs=C:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.websphere.v7_7.0.2.v20110524_2321\wasJars\ -cp securityimpl.jar:iwsorb.jar com.ibm.ws.security.util.PasswordDecoder {xor}LDo8LTor

This above command is one long command string (it might wrap depending on your screen) and it will create the following output in the command prompt:

encoded password == “{xor}LDo8LTor”, decoded password == “secret”

The process for Linux/AIX is basically the same, however the folder structure will be different. The commands are about the same but depending on which version of Linux you are running the Java switches might need some fidlding – though the base does not change.

Security!

Next to being helpful to retrieve lost passwords, this article hopefully also shows you just how important good physical security for your WebSphere servers is – don’t think that just because you have a log-on or locked down root that you are safe.

WebSphere: Errors installing Plug-in fix pack on IHS V7.x


This was a new one, not even the IBMers that I consulted with had run into this before.

I have been working at a client site on a large IBM Connections project since last year – V3.0.0, upgrade to 3.0.1, upgrade to 3.0.1.1 … now multiple code drops for V4 beta installs and preparation to get gold code V4 up as soon as possible (once it is released). In the course of the last year I have probably installed and upgraded more WAS and IHS servers that I have in several years previously – loving every moment of it!

Problem:

Today I had a new V 7.0 IHS on AIX to set up and we were running into issues installing the Plug-in fix for 7.0.0.21. The IHS FPO went without a problem, but the plug-in did not work. Errors, errors, errors:

java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)
java.lang.NullPointerException
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:245)
        at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.checkFilePermissions(SimVerifyFilePermissionsPlugin.java:317)         at com.ibm.ws.install.ni.framework.simplugins.SimVerifyFilePermissionsPlugin$ValidateFilePermissions.run(SimVerifyFilePermissionsPlugin.java:139)

Nothing was working … I looked at fie permissions, ownership etc. – no change. root, or no root – it failed. I did some searching and after a while came across this tech noteswg21408430.

the errors were close enough – I was using the latest version of the IBM UpdateInstaller (at this time 7.0.0.23) but I decided to wipe out the UDI, install it once more new into a DIFFERENT folder (queue the drum roll) …. that made the difference.

So, sometimes it is not the tool being installed, but rather the tool doing the install that is at fault.  AND – UNIX file permissions are not always at fault either. Poor little Unix – there’s good boy!!

WebSphere – The backupConfig script your friend


This is just a short post – But the built-in utility you get with the backupConfig script is worth looking into for everybody!

if you have worked with WebSphere for any significant time you have come across the built-in backup and restore utility that each WebSphere server has by default: [backupconfig.bat] or on Unix/Linux [backupconfig.sh] and the corresponding restore scripts [restoreConfig.bat] and [restoreConfig.sh]. 

At my current client we are working on application customizations and testing them on new servers. This is where the backupConfig comes in handy as it does not just back-up your application server(s) but the deployment manager and all the node(s) configuration as well – so you can replay a whole server configuration along with the installed applications and any application specific configuration. backupConfig can also be used to migrate servers from one piece of hardware to another (or VM server or, …. any combination is possible).

The process is simple: find the scripts in the [/bin] folder of either the Dmgr or the node,  execute the script and it will check your servers configuration, stop all server instances including nodeagents and then create a zip file of ALL files necessary for the back-up – and all of this wonderfulness is  unencumbered by the human thought process … :)

Whenever I am about to install a new application, install any fixes or make configuration changes to a WebSphere  server I run the backupConfig script once first and keep a copy of the zip file it created on a local machine – just to be safe.

Where to run:

Depending on your architecture you can run it on the Deployment Manager and/or all nodes. When you run it on the Deployment manager it will grab all the configuration for the Dmgr, nodes and application servers in one go. This is essential if you have to restore an environment. On a managed node (separate HW) it will capture the configuration and applications installed on that physical node – so you might need to run it on each physical WebSphere server in your environment once to get a total base back-up. Once I have that I usually just run the scripts on the deployment manager as most of the work happens there anyway and all changes are synchronized out to the nodes.

Restores:

On windows it is simple – just run the restoreConfig script and tell it which zip file to use … and presto. On Unix/Linux you have to think a bit more. The backupConfig script does not keep any file rights or ownership information, when restoring it basically sets the file ownership to the account being used to run the script – so make sure you are using the same account and have read/write rights to the folders involved. 

Here is the link to the documentation in the WebSphere Infocenter - I hope you find it useful and make a back-up of your servers soon!

Get Free Training! New Complimentary IBM WebSphere Education Courses Available from the Global WebSphere Community


I think everybody following me knows that I am telling everybody to learn WebSphere – it will come your way like it or not. Here are some free training courses/videos that you can look at – all you have to do is become a member of the Global WebSphere Community – and PRESTO – free training for you. I will be looking at this later tonight …

Dear WebSpherians,Last year, we polled our Global WebSphere Community members asking you what WebSphere topics you were most interested in having additional training information on from the GWC. The results came back showing a strong interest in WebSphere Application Server and WebSphere MQ. The Global WebSphere Community is pleased to announce the availability of these complimentary IBM WebSphere Education courses to our GWC members.We are pleased to offer the following courses to our members:

  • WebSphere MQ v7 Installation and Configuration
  • WebSphere MQ V7 High Availability Considerations
  • Using WebSphere MQ V7 Traces, Error Logs, and Failure Data Capture Files
  • WebSphere MQ V7 Clustering
  • WebSphere Application Server V8 Architecture
  • WebSphere Application Server v8 Workload Management

Click here for more information or to view one of the courses

Sincerely

The Global WebSphere Community Management Team

Join in the discussion
Follow us on TwitterFacebookFacebook
This email was sent to victor@toalsys.com.If you no longer wish to receive information about the Global WebSphere Community, please inform us here.20 Carematrix Drive, Dedham, MA 02026, USA

N-11691884507-ADA79BCFCF96201E62F8F1BD0CB67ED1.jpg

Java: it’s your fault! Connections on AIX


Just a quick one during my lunch hour …. ran into an issue yesterday at my current client that shows once more that when you do not work with a specific OS for a while, you really loose your touch for the small details.

The Saga of WAS, AIX and the damn Java Cache

We installed iFixes yesterday and that all went well. However the syncing of the nodes (kicked off from the Dmgr console) took forever, and then one of the app clusters on one of the nodes would not restart (it eventually did after 4 hours).

To clean the system and get rid of any old temp files we:

  • Stopped all WebSphere servers:  /WAS_Profile/bin/stopServer.sh xxx -user xxx -password ****
  • Stopped the NodeAgent:  /WAS_Profile/bin/stopNode.sh -username xxxx -password ****
  • Cleaned all temp files /WAS_Profile/temp  and /wstemp (everything inside of both folders)
  • Ran  /WAS_Profile/bin/osgiCfgInit.sh
  • Ran  /WAS_Profile/bin/clearClassCache.sh

Note: you can also use the command “./stopNode.sh -stopservers -username xxxx -password ****” to shut down the node agent AND the servers at the same time. We wanted to see the individual servers come down as we had issues with one of them.

We then tried to restart the node agent ….. and it failed. We found this in the startserver.log for the node agent:

ADMU3011E: Server launched but failed initialization

Damn, nothing worked … re-cleaned, checked, cursed, cried ….. and then opened a Sev 1 ticket with IBM support online. (had a REALLY fast response – thanks guys!)

The Cavalry to the Rescue …

The Connections support guy had a look at the logs and brought in a WAS support specialist who had me repeat the clean-ups steps above AND clean this location as well (everything in this folder, but not delete the folder itself):

/tmp/javasharedresources

The IBM tech thinks we had a corrupted system level java cache that was causing the issue.  After that a ./startNode.sh worked like a charm and the servers started fine as well.

Total Clean-up

Incidentally, we ended up shutting each AIX WAS server (including the Dmgr) down one by one so we do not have a service outage and ran the above maintenance once more. On the nodes we also ran a “./synchNode.sh” with the node agent turned off – just to eliminate any possibility of the nodes maybe being out of synch (thanks for the idea Stuart).

We will also be going through our automated scripts to test adding some more items to them (email notifications when individual steps are done, add the “/temp/javasharedresources” to the list of folders to be cleaned,  etc.).

Lessons to be learned:

  • When you don’t work with an OS for a while you forget the important SMALL stuff (/tmp/javasharedresources) – I had run into this very issue a few years ago and totally forgot about it. I actually did not remember it until this morning, the day after.
  • When in doubt – call support RIGHT AWAY, if for no other reason than to validate your thought process is correct and you are not barking up the wrong tree. We did not wait very long to call, but sometimes even 5 minutes can mean the difference between failure and success.

 

 

Websphere Technote: Application Server does not start – OSGi permission issue


http://www.ibm.com/support/docview.wss?uid=swg21578438&myns=swgws&mynp=OCSSEQTP&mync=R

This one just came in via a feed. I ran into this recently during my first forray into WAS 8 … The errors kicking in were a great way to develop my shaky WAS confidence. I solved it because i deal with Linux allot and access rights/file rights are a commkn problem when a team use different/personal IDs when installing systems instead of a common account. Both are acceptable as long as everybody plays by the rules.

For my friends from the Domino world this is similar to a Domino server not having the correct rights to system databases and therfor crashing. Does not happen often but is can.

WAS is not hard, jump in and learn! 

Domino, WebSphere and why don’t my Friends get it yet


As most people who know me in person can attest to, I usually am not at a loss for words. Some friends of mine actually have made a pact to kill my mouth separately after I die to make sure it goes to rest.  True friends will go to great lengths to take care of you … one way or another.

“The discussion”

I did find my inborn ability to fill the air with words challenged over the holidays when I engaged a good friend and collegue in a discussion about technology, future and IT work in general. We were griping a bit and the topic we were discussed was Domino and how the good paying jobs have become more scarce over the last 2 – 3 years. I imagine this topic gets repeated over and over again among many IT professionals.

To keep the whole thing short: I could not get the point across that as a Domino guy you have to see the writing on the wall and learn Websphere. More and more of Lotus IBM technology is now running on Websphere (Sametime, Connections, Quickr J . . .) and even if you don’t think that Domino will end up on that platform eventually, you need to add to your technology portfolio and make yourself more valuable. With Websphere in your back pocket you can do other projects as well. Add a dash of Unix and Linux experience … and you have the making of a nice Bouillabaisse. That Bouillabaisse will keep you fed for quite some time.

If you know a splatter of WAS – just to be able to integrade Domino with other Lotus IBM products – many of which are free entitlements to Domino licensees now – you have a whole larger pond you can swim in with more work to choose from with better rates and less competition.
I am a freelance IT consultant and started with Domino 18 years ago. I like it and do well and was lucky enough to have a nice year-long project I came off of in December and moved right into a new IBM Connections gig back-to-back.  But if you look out in the market and search for Notes/Domino projects they are getting fewer and fewer and pay less and less each year and the competition for those positions is quite fierce.

Yes, clients want oodles of experience and the knowledge you bring to the table but the price structure is not what it used to be. Look on LinkedIn and check how often you see somebody from India posting a question on how to “quickly learn Domino and maybe that thing called Notes” and you know where allot of the basic support jobs have gone to and what quality clients seem to be willing to live with nowadays. The only IT guys/girls working regularely now are either on the low end of the pay scale or the very top. If you are caught in the middle, the jobs you can find not as plentiful as they used to be and you are constantly searching, right?

Let me spell it out:

  • Change is good
  • Learning is a life-long necessity (damn – after all you did not learn Domino in elementary school, right?)
  • Expecting to do the same thing you are comfortable with for the next 20 years until you retire is not going to happen
  • WAS is not as hard as it looks – if you can take care of a Domino environment with all the moving parts and pieces WAS is not such a leap.

LEARN . IT . NOW .