The Saga of WAS, AIX and the damn Java Cache

We installed iFixes yesterday and that all went well. However the syncing of the nodes (kicked off from the Dmgr console) took forever, and then one of the app clusters on one of the nodes would not restart (it eventually did after 4 hours).

To clean the system and get rid of any old temp files we:

• Stopped all WebSphere servers:  /WAS_Profile/bin/stopServer.sh xxx -user xxx -password ****
• Stopped the NodeAgent:  /WAS_Profile/bin/stopNode.sh -username xxxx -password ****
• Cleaned all temp files /WAS_Profile/temp  and /wstemp (everything inside of both folders)
• Ran  /WAS_Profile/bin/osgiCfgInit.sh
• Ran  /WAS_Profile/bin/clearClassCache.sh

Note: you can also use the command “./stopNode.sh -stopservers -username xxxx -password ****” to shut down the node agent AND the servers at the same time. We wanted to see the individual servers come down as we had issues with one of them.

We then tried to restart the node agent ….. and it failed. We found this in the startserver.log for the node agent:

ADMU3011E: Server launched but failed initialization

Damn, nothing worked … re-cleaned, checked, cursed, cried ….. and then opened a Sev 1 ticket with IBM support online. (had a REALLY fast response – thanks guys!)

The Cavalry to the Rescue …

The Connections support guy had a look at the logs and brought in a WAS support specialist who had me repeat the clean-ups steps above AND clean this location as well (everything in this folder, but not delete the folder itself):

/tmp/javasharedresources

The IBM tech thinks we had a corrupted system level java cache that was causing the issue.  After that a ./startNode.sh worked like a charm and the servers started fine as well.

Total Clean-up

Incidentally, we ended up shutting each AIX WAS server (including the Dmgr) down one by one so we do not have a service outage and ran the above maintenance once more. On the nodes we also ran a “./synchNode.sh” with the node agent turned off – just to eliminate any possibility of the nodes maybe being out of synch (thanks for the idea Stuart).

We will also be going through our automated scripts to test adding some more items to them (email notifications when individual steps are done, add the “/temp/javasharedresources” to the list of folders to be cleaned,  etc.).

Lessons to be learned:

• When you don’t work with an OS for a while you forget the important SMALL stuff (/tmp/javasharedresources) – I had run into this very issue a few years ago and totally forgot about it. I actually did not remember it until this morning, the day after.
• When in doubt – call support RIGHT AWAY, if for no other reason than to validate your thought process is correct and you are not barking up the wrong tree. We did not wait very long to call, but sometimes even 5 minutes can mean the difference between failure and success.

This technote just came out but we actually had this issue early last month where we noticed it. My current client uses SPNEGO for single sign-on (works GREAT btw) and since moderation is a new feature to Cnnections we did not really notice it in the beginning. After all, if something is new, you are not – yet – aware of what the standard behavior is supposed to be. We have a single-server test environment that does not use SPNEGO on on that we were not forced to log in separately and that difference was what prompted us to ask questions.
Suffices to say, the iFix did rectify the problem and it now works. I urge you to take a look at your system and request the iFix from IBM.

I had not heard of this one yet but I am going to my servers right now and checking for the latest security patches … if your servers do not run x.org server then you have nothing to fear. While you are at it, check your workstations too …..

I came across this one a short while ago. I have not used NGINX myself yet though one of my hosted servers uses it though I have no control over it (cpanel access only) . I have been playing with the idea of configuring some “unusual” systems at home and I think I am going to take a look at how NGINX might integrate with some IBM/Lotus products. I have seen it used as a proxy server (or several servers) in front of IBM web servers but I have not yet seen it used as the actual web server for Connections or Sametime or any other Websphere server.

Anyone out there that has experience in this? I searched online and could not find anything and now I am intrigued - I guess I will have to add this to my long list of things I want to try out.

“Everyone’s always asking me when Apple will come out with a cell phone. My answer is, ‘Probably never.’”—David Pogue, The New York Times, 2006Image: Wikimedia Commons/Linux insidev2

Especially if the gist of your prediction is that something won’t happen or isn’t possible. You wind up with enough egg on your face to make an omelet.

If you’re not convinced, have a look at these whoppers: some of the biggest muffed tech predictions of all time, and spoken by people you’d expect would know better.

* “I predict the Internet will soon go spectacularly supernova and in 1996 catastrophically collapse.”—Robert Metcalfe, founder of 3Com and inventor of Ethernet, writing in a 1995 InfoWorldcolumn

Metcalfe is well aware how silly his prediction came to look. He ate his words—literally. In 1999, addressing the Sixth International WWW Conference, Metcalfe put a copy of his infamous column into a blender, pureed it, and drank it.

* “There is practically no chance communications space satellites will be used to provide better telephone, telegraph, television or radio service inside the United States.”—T.A.M. Craven, Federal Communications Commission commissioner (1961)

Needless to say, Mr. Craven is no longer the commissioner of the FCC.

* “I think there is a world market for maybe five computers.”—Thomas Watson, chairman of IBM, 1943

Of course, Watson was referring to room-size mega-machines filled with vacuum tubes. But still.

* “The Americans have need of the telephone, but we do not. We have plenty of messenger boys.”—Sir William Preece, chief engineer, British Post Office, 1876

How’re the messenger boys working out for you, England?

* “This ‘telephone’ has too many shortcomings to be seriously considered as a means of communication.”—Western Union internal memo, 1876.

Oops! Western Union sent its last telegram in 2006.

* “Television won’t be able to hold on to any market it captures after the first six months. People will soon get tired of staring at a plywood box every night.”—Darryl Zanuck, 20th Century Fox, 1946

He was right. We’ve moved on to aluminum and plastic.

* “Everyone’s always asking me when Apple will come out with a cell phone. My answer is, ‘Probably never.’”—David Pogue, The New York Times, 2006

Yeah, Okay. I’ll admit it. My prediction was wrong—but my thinking was right. I knew that Steve Jobs would never tolerate the micromanagement that the carriers (Verizon, AT&T and so on) then exercised on every aspect of every phone they carried. “I cannot imagine Apple giving veto power to anyone over its software design. It just ain’t gonna happen,” I wrote.

What I didn’t realize, of course, is that Jobs planned an end-run—a deal that Cingular ultimately accepted, which ran like this: “You let us design our phone without your input, and I’ll give you a five-year exclusive.” And the rest is history.

The Bill Gates Collection

Maybe the most quoted bad tech prediction of all time never really happened. It’s the famous 1981 Bill Gates quote: “640K ought to be enough for anybody.” Unfortunately, there’s no evidence that he ever actually said that (and he emphatically denies that he did).

That doesn’t let Mr. Gates off the hook, though—he’s made plenty of doozies.

* World Economic Forum, 2004: “Two years from now, spam will be solved.”

Today, spam accounts for over 90 percent of all e-mail sent.

I read this post early this morning in the train on my way to work.

HIGH TIME Andy and Rob did this!

Both Andy and Rob are well known in the Lotus/IBM community for their expertiese, experience, vision and skill. I think with this new offering Technotics is hitting the nerve of the Shakespearian question many clients face: “To go, or not to go.” What I particularly like is that they preface the whole process with a fact finding session/process that first tries to answer the more important queestion: namely where to actually go to (= the goal, the required outcome, the Nirvana destination). Otherwise the experience would be akin to trying to go on vacation without determining where to go before you leave and trying to get all the reservations done on the way to the airport – and changing the departure airport you have to get to while you are sitting in the car.

I wish them the best of luck – though knowing them they probably have this process nailed down to the point where they can get out of bed at 10 AM, have a coffee, do a little bit of magic/pixie dust work and then go fishing in the afternoon followed by some phone calls before dinner (can you hear the envy in my vitual voice?)

If you are a company looking for assistance – go talk to Andy. If you are a tech guy/girl who was aproached by a client – send them to Andy and Rob, you will be pleased with the outcome.

Full Disclosure: no rodents (virtual or real) where harmed in the course of this blog post and I am NOT involved in Technotics in any way – (hint, hint, hint) . . .

This one just came in via a feed. I ran into this recently during my first forray into WAS 8 … The errors kicking in were a great way to develop my shaky WAS confidence. I solved it because i deal with Linux allot and access rights/file rights are a commkn problem when a team use different/personal IDs when installing systems instead of a common account. Both are acceptable as long as everybody plays by the rules.

For my friends from the Domino world this is similar to a Domino server not having the correct rights to system databases and therfor crashing. Does not happen often but is can.

WAS is not hard, jump in and learn! 

www.ssmcjobs.com

This new job site is dedicated to IT jobs in the social media / social software, collaboration and messaging technology fields. Free for job seekers it is aiming  to give IT staff with “social” in their portfolio a place to showcase their skills and search for new opportunities. If happen to not be actively looking it is still a great place to just “park” your resume and see what kind of offers might be out there.

If you are a consultant who is always looking for your next project in the technology field that is experiencing the largest growth of any segment out there – we have allot of staffing comanies waiting for your resume to pop up on their searches.

SSMCJOBS.COM aims to be technology agnostic, so even though this blog of mine is IBM technology centered SSMCJOBS.COM is not. IBM, MS, Socialtext, Alfresco, Jive … the list is long and they are all looking for resources for their next project.

Go have a look – SSMCJOBS.COM is open for business!

This is quite a problem – install the fix, it worked for my current client and is a quick, simple and effective.

A bored dog .... now happy

But The Empress had an answer for him … he is now duly punished