I was part of a panel discussing Domino 9 that included Bruce Elgort of IdeaJam fame, Scott Souder/IBM and Steve Farnsworth. You can watch the recording on YouTube here: http://bit.ly/XaPsu9
This just came in via a feed from IBM support. I hope you guys subscribe to the support feeds as well.
Personally I never came across this specific error but as I do work allot in large, multi nationals this is a good one to squirrel away and keep in my back pocket.
this one promises to be interesting, I am definitely going to listen in. Memory issues are some of the more vexing problems to deal with since they involve more than just the Domino side and Domino knowledge to track and solve.
As most people who know me in person can attest to, I usually am not at a loss for words. Some friends of mine actually have made a pact to kill my mouth separately after I die to make sure it goes to rest. True friends will go to great lengths to take care of you … one way or another.
I did find my inborn ability to fill the air with words challenged over the holidays when I engaged a good friend and collegue in a discussion about technology, future and IT work in general. We were griping a bit and the topic we were discussed was Domino and how the good paying jobs have become more scarce over the last 2 – 3 years. I imagine this topic gets repeated over and over again among many IT professionals.
To keep the whole thing short: I could not get the point across that as a Domino guy you have to see the writing on the wall and learn Websphere. More and more of Lotus IBM technology is now running on Websphere (Sametime, Connections, Quickr J . . .) and even if you don’t think that Domino will end up on that platform eventually, you need to add to your technology portfolio and make yourself more valuable. With Websphere in your back pocket you can do other projects as well. Add a dash of Unix and Linux experience … and you have the making of a nice Bouillabaisse. That Bouillabaisse will keep you fed for quite some time.
If you know a splatter of WAS – just to be able to integrade Domino with other Lotus IBM products – many of which are free entitlements to Domino licensees now – you have a whole larger pond you can swim in with more work to choose from with better rates and less competition.
I am a freelance IT consultant and started with Domino 18 years ago. I like it and do well and was lucky enough to have a nice year-long project I came off of in December and moved right into a new IBM Connections gig back-to-back. But if you look out in the market and search for Notes/Domino projects they are getting fewer and fewer and pay less and less each year and the competition for those positions is quite fierce.
Yes, clients want oodles of experience and the knowledge you bring to the table but the price structure is not what it used to be. Look on LinkedIn and check how often you see somebody from India posting a question on how to “quickly learn Domino and maybe that thing called Notes” and you know where allot of the basic support jobs have gone to and what quality clients seem to be willing to live with nowadays. The only IT guys/girls working regularely now are either on the low end of the pay scale or the very top. If you are caught in the middle, the jobs you can find not as plentiful as they used to be and you are constantly searching, right?
Let me spell it out:
LEARN . IT . NOW .
I am part of a Domino migration at a large client in the financial/insurance industry and we have been moving applications from Domino 6.5.5 to version 8.5.2 FP1. I know, there are newer versions out there but you can’t always upgrade blindly – this is going to be one of the stories why IN-DEPTH testing is so important.
The client has an application that has been around since R5 days, migrated and upgraded several times – both client and server versions and never an issue.
Suddenly we get reports (during testing – thank GOD!) that when using the app with the new client (LN 8.5.2 standard) that certain document do not get created – they vanish.
To make a L O N G story short – we opened a ticket with IBM support and after digging and prodding they produced this one for us:
(from the actual ticket)
******************* SUBSEQUENT CALL RECORD TEMPLATE *******************
Created a test DB in 85 and copied customer form and data.
Found field “eml” caused the issue: SPR RDJS8APTK6.
This is a regression error that made its way back into the Lotus notes 8.5.2 code stream. It was fixed in Notes 8.5.3 but if you are on 8.5.2 (no matter the FP level) you are snorked.
What we did
Since it is hard to determine with any certainty how many other applications out there might contain a filed called “eml” the decision was made to do a company wide, mid-project change and instead of rolling out Notes 8.5.2 we will be rolling out 8.5.3 and upgrading all workstations to which 8.5.2 has been deployed previously. A huge undertaking.
Had this not been found during testing it could have been very costly for my client. As is, the change in scope will be massive and costly as well, but it beats having actual business processes interrupted and clients (actual consumers) be impacted.
So – the lessons everybody should take home from this:
I’ll keep it brief – short cuts will kill you.
I am on a Domino migration project where we are moving the client from 6.5.6 to 8.5.2. We are not upgrading servers but creating a whole new server environment on new virtual servers and will be moving applications over in phases. No mail, they moved to Exchange years ago.
We separated the development and sandbox environments out into new Domino domains (they were in production up until now) and we are in testing mode – each application individually.
That is where we ran into … “issues” with simple background agents, specifically agents that send e-mail.
“Error connecting to server (servername): The remote server is not a known TCP/IP host,”
Agents fire, things happen and for the most part we are happy, but the above errors pop up again and again and mail is not being created. The mail never makes it into the mail.box, just errors in the logs and no mail at all.
This is where we came across this technote: and while reading it I am also glancing through the notes.ini on one of the servers that is having issues. I notice in there the “Mailserver=” parameter that has a different server name entered – that is where the above technote comes in and then a few Google searches on the [Mailserver] field and we knew where we had managed to trip ourselves in the process.
The [Mailserver] field
This is a little knows field since it has no UI equivalent and normally an admin would never have to fiddle with it – ever. However, when you take shortcuts during server setup and configuration though …. maybe you can guess? Yes, we set up one default server document, copied it 20 times, renamed the server name, updated all the server specific addresses and then copied the certificate into it. What that leaves you with is 20 server documents that all have the same one [Mailserver] field value – and the server that it pointed to was a temporary server that was turned off.
It took us a few hours of head scratching to figure this one out but by the end of the day we got it and fixed the field values on all servers. After that restarting the servers (not really necessary but I always feel better that way) took care of the rest and we were back in business.
Lesson learned: if you want to batch create and copy & paste things make sure you take care of ALL server specific information in a server document, not just the ones you can see in the document but the hidden ones as well.
I am not the first one to find this, I actually only noticed after going through my feeds and stumbling on it in Volker Weber’s blog . IT IS FINALLY HERE! We are now officially no longer the unwanted, redheaded stepchildren of RIM – we have our own BES Express server! Go here …
PS: with all this early stuff that VOWE knows, does he ever really work? The eternal questions never stop …. it’s all envy anyway!
I came across this technote yesterday and investigated it right away …
- well, I’ll admit it – I asked somebody else in IBM to clarify some items for me and am now plagiarizing that good deed on my blog – THANKS CAROL!!
First of all – this only applies to Domino 8.5.1 on Windows, and it makes no difference which FP you are running since it was not fixed in FP4 nor will a fix be included in FP5 either, it is fixed in 8.5.2 though – you guys are safe.
The Issue Explained:
The issue CAN be cause by running FIXUP with a combination of either [fixup -F -J] or [fixup -F -J -O]. It will not automatically happen, it has to run into a document in a dB that has ‘invalid items” .. what exactly constitues such “invalid items” is not 100% clear but I am treating it like a 50-50 chance for failure until I have more evidence/experience or a better explanation of what exactly is meant with “invalid items”.
Since the explanation implies that the behavior only happens if you also include the [-J] switch, servers that are not not transaction logged will not be affected, even if they are on Windows. Also, if you are running Domino on any other platform this will not affect you – so my dear Linux buddies are all safe, as are you AIX, iSeries and mainframers as well.
What does this mean in terms of daily admin work?
Well, the [-J] is needed anytime you want to run fixup against transaction-logged databases and [-F] is a very commonly used switch as well. [-O] basically tells fixup to also run against open databases and is the default bahavior if you specify a database name with the fixup command.So the command [load fixup mail/xxx.nsf -J -F] implicitly includes the [-O] switch behavior whether you add it to the command or not.
That leaves you with the option to run Fixup without the [-F] switch to avoid a *possible* kiss of death.
if you are running the fixup command as part of some regular, automated script (I usually advise against it) you need to review the commands to make sure yo do not inadvertently crash your server (possibly, it is not a guaranteed outcome).
Also, when running Fixup against databases during the day to fix database corruption of any kind, I would run it without the [-F] switch during the day and see if your problem is fixed. If that does not fix your database problem, you can either roll the dice or wait until after hours – which is what I will do until I have tried it out a few times and have a better feel whether this actually ever materializes or not.
Another great find (this tech note just came out today) of some improvements that have found their way into DAOS in 8.5.2.
I am really liking DAOS now (I always have, it’s more an increase of like-iness) . It is so much easier to sell a client on DAOS now as the feature has been around for a while and you can show and document the evolution of issues, their resolution and the current state of the product. Along with the DAOS estimator that kind of documentation is a powerful tool that can be used to overcome allot of rather irrational fears.
Many clients are – understandable – hesitant to do anything with attachments … it gets them all nervous. Once you can show them the advantages though it becomes easier – especially if you start talking of performance increases on the server and that even if they get nervous – DAOS is bound to each server individually and that makes them feel safer if some other server has the “full” (a misnomer, but effective) replica.
I like to introduce it on to servers that are used for archiving, back-up and redundancy. Back-up times are quicker, tapes are fewer, AV scans are MUCH quicker and the server (usually not the fasted one in the crowd) performs better. A great way to get a client comfortable and then have them implement it system wide.
You can tell from my posts today that I have been investigating DAOS a bit to get some more under-the-hood info that I might be need. I came across this technote here that specifies how to turn on DAOS on newly created archives when server based archiving is utilized.
The kicker of this is that the new 8.5.2 admin client only is delivered with the 8.5.1 help file (what’s up with that???) and this information is not included in that help file:
Server Based Archiving:
Is run via the compact command using one of the switches [-a/-A/-j] when the option Domino server based archiving is enabled in mail files (best via policies, lets not get into that right now).
Since release 8.5 you could use the following command to enable the newly created archive to be DAOS enabled
load compact -A DAOS on mail\xxxx.nsf
In-place compaction switch [-a] would not work though, only the archiving function as is.
New in 8.5.2 –
In release 8.5.2, you can now use the [-a] switch ion combination with the [-DAOS on] switch to archive, compact the mail file in place and DAOS enable the archive at the same time as follows:
load compact -a -DAOS on mail\xxxx.nsf
*** SUBTLE HINT TO IBM ***
… I hope that IBM will create a new 8.5.1 admin helpfile soon for download. I for myself use that as the first line of investigation whenever looking into some functionality and do nto necessarily always go out onto the Internet for more information. If it ain’t in there I might assume it does not exist . . . . . . the InfoCenter does not appear to be updated either …