Oracle troubles

I shouldn’t be writing that all is well on the Oracle front.

Just now two of the Oracle servers froze with database problems. The DBA tells me that they have block corrupts which he hasn’t seen in five years of running the things.

More later.

{edit}

Both instances were terminated by PMON due to unreadable
logfile or spfile. The errors look like this;

ORA-00471: DBWR process terminated with error
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 26973

ORA-00470: LGWR process terminated with error
PMON: terminating instance due to error 470
Instance terminated by PMON, pid = 25260

One instance is a 9.2.0.4 database, the other a 10.2.0.1 database, on separate virtual machines.

For Oracle 9i: Restore from backup.…
For Oracle 10g: Resetlogs by:

startup mount
recover database until cancel;
recover database until cancel; (a message ‘media recovery complete’ will appear)
alter database open resetlogs;

We are viewing this as an unfortunate incident at the moment but will keep monitoring this closely.

Our first production HA failover

A quite unexpected event yesterday was the very first HA failover in production. Although we had tested it and seen it work a number of times in our testing environment it was something else to see it in the production environment. As a result of which we weren’t looking for an HA failover when all of a sudden 14 servers went down.

After reviewing the logs we found out that the servers in question were moved because of a failover. It turned out that one of the hosts lost it’s network connection for three seconds (we still don’t know why) and that HA decided to power off and move all the servers as a precaution.

We can safely say that HA works.

Oracle DB/AS and Grid Control in production

So last weekend we migrated all the Oracle servers. After multiple issues with PowerConvert (it doesn’t clean up after itself properly if a migration fails) we eventually managed to convert an Oracle 10g, Oracle 10g AS and an Oracle 9iAS server. The others were re-installs.

It seems that there still is some ways to go on the Linux conversion front to make it a truly user-friendly application. Although we couldn’t have done it faster by hand it is kind of frustrating to see your migration fail time after time because of problems during copying or the program simply freezing up when it’s trying to take control.

On the Monday it turned out that our estimate for sizing the production 9i databaseserver was a bit on the cheap side (1 vCPU / 3GB memory) so we had to add a second vCPU. Oracle also likes it better when it can run an SMP kernel and it seems to hold up. We may upgrade the memory to 4GB but everything else works fine.