WebSphere – The backupConfig script your friend


This is just a short post – But the built-in utility you get with the backupConfig script is worth looking into for everybody!

if you have worked with WebSphere for any significant time you have come across the built-in backup and restore utility that each WebSphere server has by default: [backupconfig.bat] or on Unix/Linux [backupconfig.sh] and the corresponding restore scripts [restoreConfig.bat] and [restoreConfig.sh]. 

At my current client we are working on application customizations and testing them on new servers. This is where the backupConfig comes in handy as it does not just back-up your application server(s) but the deployment manager and all the node(s) configuration as well – so you can replay a whole server configuration along with the installed applications and any application specific configuration. backupConfig can also be used to migrate servers from one piece of hardware to another (or VM server or, …. any combination is possible).

The process is simple: find the scripts in the [/bin] folder of either the Dmgr or the node,  execute the script and it will check your servers configuration, stop all server instances including nodeagents and then create a zip file of ALL files necessary for the back-up – and all of this wonderfulness is  unencumbered by the human thought process … :)

Whenever I am about to install a new application, install any fixes or make configuration changes to a WebSphere  server I run the backupConfig script once first and keep a copy of the zip file it created on a local machine – just to be safe.

Where to run:

Depending on your architecture you can run it on the Deployment Manager and/or all nodes. When you run it on the Deployment manager it will grab all the configuration for the Dmgr, nodes and application servers in one go. This is essential if you have to restore an environment. On a managed node (separate HW) it will capture the configuration and applications installed on that physical node – so you might need to run it on each physical WebSphere server in your environment once to get a total base back-up. Once I have that I usually just run the scripts on the deployment manager as most of the work happens there anyway and all changes are synchronized out to the nodes.

Restores:

On windows it is simple – just run the restoreConfig script and tell it which zip file to use … and presto. On Unix/Linux you have to think a bit more. The backupConfig script does not keep any file rights or ownership information, when restoring it basically sets the file ownership to the account being used to run the script – so make sure you are using the same account and have read/write rights to the folders involved. 

Here is the link to the documentation in the WebSphere Infocenter - I hope you find it useful and make a back-up of your servers soon!

Technotics launches new web site – announces new cloud services


http://www.andypedisich.com/blogs/andysblog.nsf/dx/technotics-launches-new-web-site-announces-new-cloud-services.htm

I read this post early this morning in the train on my way to work.

HIGH TIME Andy and Rob did this!

Both Andy and Rob are well known in the Lotus/IBM community for their expertiese, experience, vision and skill. I think with this new offering Technotics is hitting the nerve of the Shakespearian question many clients face: “To go, or not to go.” What I particularly like is that they preface the whole process with a fact finding session/process that first tries to answer the more important queestion: namely where to actually go to (= the goal, the required outcome, the Nirvana destination). Otherwise the experience would be akin to trying to go on vacation without determining where to go before you leave and trying to get all the reservations done on the way to the airport – and changing the departure airport you have to get to while you are sitting in the car.

I wish them the best of luck – though knowing them they probably have this process nailed down to the point where they can get out of bed at 10 AM, have a coffee, do a little bit of magic/pixie dust work and then go fishing in the afternoon followed by some phone calls before dinner (can you hear the envy in my vitual voice?)

If you are a company looking for assistance – go talk to Andy. If you are a tech guy/girl who was aproached by a client – send them to Andy and Rob, you will be pleased with the outcome.

Full Disclosure: no rodents (virtual or real) where harmed in the course of this blog post and I am NOT involved in Technotics in any way – (hint, hint, hint) . . .

Lotus Notes 8.5.2 code error – The Tale of Regression


 

I am part of a Domino migration at a large client in the financial/insurance industry and we have been moving applications from Domino 6.5.5 to version 8.5.2 FP1. I know, there are newer versions out there but you can’t always upgrade blindly – this is going to be one of the stories why IN-DEPTH testing is so important.

The client has an application that has been around since R5 days, migrated and upgraded several times – both client and server versions and never an issue.

Suddenly we get reports (during testing – thank GOD!) that when using the app with the new client (LN 8.5.2 standard) that certain document do not get created – they vanish.

To make a L O N G story short – we opened a ticket with IBM support and after digging and prodding they produced this one for us:

(from the actual ticket)

******************* SUBSEQUENT CALL RECORD TEMPLATE  *******************
ACTION TAKEN:                                                           
Created a test DB in 85 and copied customer form and data.              
Found field “eml” caused the issue: SPR RDJS8APTK6.  

This is a regression error that made its way back into the Lotus notes 8.5.2 code stream. It was fixed in Notes 8.5.3 but if you are on 8.5.2 (no matter the FP level) you are snorked.

What we did

Since it is hard to determine with any certainty how many other applications out there might contain a filed called “eml” the decision was made to do a company wide, mid-project change and instead of rolling out Notes 8.5.2 we will be rolling out 8.5.3 and upgrading all workstations to which 8.5.2 has been deployed previously. A huge undertaking.

Had this not been found during testing it could have been very costly for my client.  As is, the change in scope will be massive and costly as well, but it beats having actual business processes interrupted and clients (actual consumers) be impacted.

So – the lessons everybody should take home from this:

  1. Test – always test even if it is a small point upgrade
  2. Test – with ALL server and client variations that you might have in your environment – do not skip anything
  3. Have test plans that are captured and documented across each scenario and can be compared apples-to-apples.
  4. DO NOT NAME ANY FIELDS, FORMS (or anything else for that matter)  IN YOUR APPLICATIONS “EML” – this error could possibly be re-introduced in a future code stream – you never know.

 

 

Domino 8.5.2 HF1 – the fix for the unread synch server crash


I would like to bring your attention to the SPR# RMAA88UAGF – server crashes when clients synchronize unread marks on Domino 8.5.2 servers.

I ran into this last week when we had some unexplainable server crashes for apparently no good reason – one of them happened when I was showing a help desk tech how to synchronize unread marks between replicas from the client. I investigated and the crash report that Domino gave me pointed me right to this error – now fixed in FP1. It is a regression bug and is not fixed in either of the previous hot fixes for the Domino 8.5.2 code stream. I had servers with both the SMTP patch and the interim fix (IF) as well – they all crashed at one time or another. I installed HF 1 on all servers and now they are safe from this issue – I tested it on all of them just to be on the safe side.

If you don’t have HF 1 installed yet – do it soon, this one can happen at anytime during  the day and there is no warning whatsoever. only a big “boom” when the server disappears from the face of the (electronic) earth. Incidentally, there are allot of other problems that are fixed in that HF as well – I suggest you look into it right away and consider upgrading all your Domino 8.5.2 servers soon.

BES Express for Domino in November – Announcement


This one comes courtesy of Volker Weber. I read one of his tweets about this and searched the web and came up with zilch – I contacted him and it seems it was announced interactively by RIM at an IBM function in Stuttgart that he (I assume) attended.

Exciting news indeed!! Lets see what November brings. I am sure that there will be some official pres release by RIM soon – I can’t wait to ditch my current Small Business BES that is eternally stuck at 4.1.4 and upgrade/migrate to the Express version.

Domino 8.5.2 with SMTP HF is in


Finally! I spent all of 25 minutes yesterday to upgrade my main Domino server to 8.5.2 with the new HF for SMTP that came out yesterday. I waited on this fix and all I can say is that the upgrade (from 8.5.1 FP4) was quick, simple and even the old HW that my server runs on had no issue with it. What took the longest was the download of the Domino install (15 minutes) and everything else was quick.

I do have clients that want to wait until FP1 comes out – somewhat short-sighted but I do understand some hesitation. That is exactly why I did not upgrade my servers immediately when the new version came out – I like to wait and see what issues and downright problems are found by those braver than me. Once I see a lull – then I go ahead and make a decision.

So, if Mr cautious did it last night while watching TV and sipping a nice glass of Cabernet Sauvignon and upgrading Symantec on another server, what are you waiting for?

Upgrading Domino 32 bit to 64 bit


After all the talk of how to decide on what version (32 vs 64 bit) of Domino you want to choose on Windows, I want to blog briefly on some good strategies to migrate a server from 32 to 64 bit.

My Assumptions for this article:

As discussed previously, 64 bit Domino requires a 64 bit OS and for this hypothetical case I am assuming that I am staying with the same OS type and only switching the version (32 -> 64 bit) so I do not have to take any change in disk format into account that is likely if you move from let’s say Windows to AIX.I am also assuming that I am not going to do any major upgrade of the Domino version – that would be a major change and require allot of planning and testing beyond what I want to go into in this short blog post.

Scenario 1: Switch the server

This one is easy – it is basically the same as if you are going to switch out the hardware (in-place upgrade). I would install the new server on a new piece of hardware, bring the old server down, move the data directory to the new server, rename the server, switch IP addresses (or IP reservations, whichever one it is your network does) and then bring the new server up with the original data.

The point in this switch/upgrade is that you have new hardware that you can install at your leisure, burn in and prepare. The only time-sensitive part is when you bring all the servers down, move the data and re-configure the new hardware. Hopefully the data is on a SANS and all you have to do is detach and re-attach the drives. I greatly prefer that to having to copy data from the old server to the server as that can add untold hours to a deployment schedule.

Scenario 2: In-place upgrade

This scenario basically only happens if you have no other piece of hardware that you could use and the in-place is the last straw of yours to try and get past some unspeakable evil from which you need to escape. Therefore I believe a disclaimer is in order:

Do not attempt this at home, don’t do it at the office either and if you are working for a client. Heck, if you value your sanity and don’t want to die from a heart attack don’t do this one at all! Only attempt this option if there is ABSOLUTELY no other way out.

The reason I say this is simple: garbage in = garbage out. Also, it is allot of changes on one physical machine so there is allot of opportunity to fail – two major changes on one physical machine and no good way to back out of the process if it does not work the way you are hoping for . Also, you are unlikely to be able to test this beforehand if something so you really can’t prepare for any eventuality. So again … avoid this one!

Now, if I absolutely had to do this – this is how I would do it: (rough draft)

Prepare:

  1. Have plenty of time available. If you think it will take 6 hours, triple the amount and add 10 hours for good measure. If it take long you will be tired and need rest. also, the server is unlikely to be new so is unlikely to be a fast process even if it goes well
  2. Make sure there is a GOOD, VALID backup and that the tapes are not old and that you can actually restore the server if you need to.
  3. If in any way possible, do a P2V into VMWare and test all of this first … it is insane and crazy, so testing is a must if there is any opportunity for it
  4. Keep a snapshot of that VMWare image around in case all hell breaks loose and it is the only thing you can offer your users to work with.
  5. Have enough free space on the Domino server for temp files, etc. Also make sure that the data partitions have at least 20% free space to accommodate size increases due to design changes

“ACTION!!!” – The upgrade itself

  • Shut down Domino and run a manual compact against all files
  • Run another back-up – just in case
  • Upgrade the OS … pray to whatever Gods you believe in and hope for the best
  • Run Domino, check if it runs and is accessible
  • Take another back-up
  • Pray, sleep, meditate and then … sacrifice a bucket of KFC chicken (like in the movie “Major League”). Also, prepare a bottle of Bourbon, crack it open and sacrifice one glass some alongside the chicken (it can only help)
  • Uninstall Domino – you cannot install 64 bit over 32 … it is a disaster waiting to happen. Save the notes.ini first, you might need it
  • Clean up the install directory, registry AND any temp folders etc. Reboot the server once
  • Install Domino 64 bit – prepare for another sacrificial chicken bucket .. this time extra crispy would be advisable
  • Attach/move/copy the data in and meditate, chant the mighty “OOOHHHMM” if that makes you feel better
  • Start Domino and prepare your sacrifice
  • ……… (waiting for all hell to break loose)

Let’s not go into details what to do if any one of these steps fail … there are too many to details to deal with. I am hoping that the numerous references to sacrifices (non-human) and heavy drink show you that this is more arcane magic than cool, logic guided technology. Again, I would not do this one unless there is absolutely no other way to deal with the issue.

Shameless Plug:

There are many more detailed steps and planning necessary for an upgrade. Hiring somebody who has done it previously would be advisable …. You can hire me or one of my many gifted consulting colleagues to do this scary stuff for you. We already have the gray hair (if we have hair) and our digestive tracts are tough and can take the stress.