Lotus Notes 8.5.2 code error – The Tale of Regression


 

I am part of a Domino migration at a large client in the financial/insurance industry and we have been moving applications from Domino 6.5.5 to version 8.5.2 FP1. I know, there are newer versions out there but you can’t always upgrade blindly – this is going to be one of the stories why IN-DEPTH testing is so important.

The client has an application that has been around since R5 days, migrated and upgraded several times – both client and server versions and never an issue.

Suddenly we get reports (during testing – thank GOD!) that when using the app with the new client (LN 8.5.2 standard) that certain document do not get created – they vanish.

To make a L O N G story short – we opened a ticket with IBM support and after digging and prodding they produced this one for us:

(from the actual ticket)

******************* SUBSEQUENT CALL RECORD TEMPLATE  *******************
ACTION TAKEN:                                                           
Created a test DB in 85 and copied customer form and data.              
Found field “eml” caused the issue: SPR RDJS8APTK6.  

This is a regression error that made its way back into the Lotus notes 8.5.2 code stream. It was fixed in Notes 8.5.3 but if you are on 8.5.2 (no matter the FP level) you are snorked.

What we did

Since it is hard to determine with any certainty how many other applications out there might contain a filed called “eml” the decision was made to do a company wide, mid-project change and instead of rolling out Notes 8.5.2 we will be rolling out 8.5.3 and upgrading all workstations to which 8.5.2 has been deployed previously. A huge undertaking.

Had this not been found during testing it could have been very costly for my client.  As is, the change in scope will be massive and costly as well, but it beats having actual business processes interrupted and clients (actual consumers) be impacted.

So – the lessons everybody should take home from this:

  1. Test – always test even if it is a small point upgrade
  2. Test – with ALL server and client variations that you might have in your environment – do not skip anything
  3. Have test plans that are captured and documented across each scenario and can be compared apples-to-apples.
  4. DO NOT NAME ANY FIELDS, FORMS (or anything else for that matter)  IN YOUR APPLICATIONS “EML” – this error could possibly be re-introduced in a future code stream – you never know.

 

 

Background Agents not running or how shortcuts can cost you in Domino


I’ll keep it brief – short cuts will kill you.

 

I am on a Domino migration project where we are moving the client from 6.5.6 to 8.5.2. We are not upgrading servers but creating a whole new server environment on new virtual servers and will be moving applications over in phases. No mail, they moved to Exchange years ago.

We separated the development and sandbox environments out into new Domino domains (they were in production up until now) and we are in testing mode – each application individually.

That is where we ran into  … “issues” with simple background agents, specifically agents that send e-mail.

“Error connecting to server (servername): The remote server is not a known TCP/IP host,”

Agents fire, things happen and for the most part we are happy, but the above errors pop up again and again and mail is not being created. The mail never makes it into the mail.box, just errors in the logs and no mail at all.

This is where we came across this technote: and while reading it I am also glancing through the notes.ini on one of the servers that is having issues. I notice in there the “Mailserver=” parameter that has a different server name entered – that is where the above technote comes in and then a few Google searches on the [Mailserver] field and we knew where we had managed to trip ourselves in the process.

 

The [Mailserver] field

This is a little knows field since it has no UI equivalent and normally an admin would never have to fiddle with it – ever. However, when you take shortcuts during server setup and configuration though …. maybe you can guess? Yes, we set up one default server document, copied it 20 times, renamed the server name, updated all the server specific addresses and then copied the certificate into it. What that leaves you with is 20 server documents that all have the same one [Mailserver] field value – and the server that it pointed to was a temporary server that was turned off.

It took us a few hours of head scratching to figure this one out but by the end of the day we got it and fixed the field values on all servers. After that restarting the servers (not really necessary but I always feel better that way) took care of the rest and we were back in business.

Lesson learned: if you want to batch create and copy & paste things make sure you take care of ALL server specific information in a server document, not just the ones you can see in the document but the hidden ones as well.

Domino 8.5.2 HF1 – the fix for the unread synch server crash


I would like to bring your attention to the SPR# RMAA88UAGF – server crashes when clients synchronize unread marks on Domino 8.5.2 servers.

I ran into this last week when we had some unexplainable server crashes for apparently no good reason – one of them happened when I was showing a help desk tech how to synchronize unread marks between replicas from the client. I investigated and the crash report that Domino gave me pointed me right to this error – now fixed in FP1. It is a regression bug and is not fixed in either of the previous hot fixes for the Domino 8.5.2 code stream. I had servers with both the SMTP patch and the interim fix (IF) as well – they all crashed at one time or another. I installed HF 1 on all servers and now they are safe from this issue – I tested it on all of them just to be on the safe side.

If you don’t have HF 1 installed yet – do it soon, this one can happen at anytime during  the day and there is no warning whatsoever. only a big “boom” when the server disappears from the face of the (electronic) earth. Incidentally, there are allot of other problems that are fixed in that HF as well – I suggest you look into it right away and consider upgrading all your Domino 8.5.2 servers soon.

Domino 8.5.2 with SMTP HF is in


Finally! I spent all of 25 minutes yesterday to upgrade my main Domino server to 8.5.2 with the new HF for SMTP that came out yesterday. I waited on this fix and all I can say is that the upgrade (from 8.5.1 FP4) was quick, simple and even the old HW that my server runs on had no issue with it. What took the longest was the download of the Domino install (15 minutes) and everything else was quick.

I do have clients that want to wait until FP1 comes out – somewhat short-sighted but I do understand some hesitation. That is exactly why I did not upgrade my servers immediately when the new version came out – I like to wait and see what issues and downright problems are found by those braver than me. Once I see a lull – then I go ahead and make a decision.

So, if Mr cautious did it last night while watching TV and sipping a nice glass of Cabernet Sauvignon and upgrading Symantec on another server, what are you waiting for?

IamLUG: Eileen Fitzgerald – 10 Admin tips


Interesting topic from the point of view of somebody who was once on “my side” of divide (client) and is now with a vendor. Of course, giving some “secrets” away is paramount to shooting herself into the foot ….

Once the presentations become publicly available, I suggest a quick look at it.

Domino 32 bit vs 64 bit – how to decide


I my last post I talked about what the main advantage of going 64 bit Windows for Domino – shared addressable memory – but I really did not go into detail on whether 32 bit or 64 bit Domino is the better choice for an installation.

As I mentioned previously, the main advantage of going 64 bit Windows is that your Domino instance will not have 4 GB of addressable shared memory available. The kicker is that this holds true for both 32 bit and 64 bit Domino on 64 bit Windows. This fact came as quite a surprise to me when I found out, this is due to the way that the Domino memory system and the memory handles are currently designed.

So the question becomes, why would I choose 64 bit Domino if 32 bit does the same for me?

Personally, I believe the main deciding factor is whether you plan to run add-in tasks in your Domino instance or not. To run an add-in task on a 64 bit Domino server the add in task also needs to be 64 bit. To run an add-in task on a 32 bit Domino server the add-in task needs to be 32 bit as well.

The logical conclusion therefore is: if you plan to run ANY add-ins (e.g.: Anti Virus, etc.) on your 64 bit Domino server you need to determine if that program is available in 64 bit as well. If it is and any other products you might consider running are available in 64 bit as well – then you can go ahead (but test first anyway – of course!!) and install 64 bit Domino. If this is not the case your choice will need to be 32 bit Domino.

“Elementary, my dear Watson.”

Domino 32 bit vs 64 bit – what’s the difference?


I have been absent for a while but with good reason – a new project that keeps me VERY busy.

Being busy though also has it’s advantages though, since it also means that I get to be exposed to allot of new things and learn more and more.

So, 32 bit vs 64 bit – what is the difference? Well, first of all let’s state that I am talking about Domino on Windows. To run 64 bit Domino you require 64 bit OS. What that also means is that any add-in tasks that you want to run inside of Domino will also need to be 64 bit – 32 bit 3rd party processes will not run on a 64 bit Domino/Windows server.

What is it that you gain? Well, the main gain you get is shared addressable space. Under 32 bit domino the maximum shared address space you get is 2 GB. that means that any add-in task you run (such as an anti virus product etc.) has a maximum shareable shared address space of 2 GB. On a busy server that can can be tight. The reason I know that is that in the first week I started my present project we saw what happens to a server that runs out of available shared address space – it will will present you with a crash and a wonderful NSD file ….

Now, when you change to 64 bit you now double your shared addressable space to 4 GB. this gives you PLENTY of space for a busy server (user sessions, applications, third party add-ins, etc.).

I will blog more about this topic in the near future and hopefully be able to shed some more light on this interesting subject.