Follow up: Oracle troubles

I will be writing a “lessons learned” post about our project in the near future but for now I will follow up on the Oracle problems we had last month.

What we learned from using Oracle in a VI environment is that memory settings turned out to be highly critical in relation to the performance of the VM. We didn’t have this problem on a physical machine, at least not as intrusive as this, but when Oracle doesn’t have enough memory available it starts very heavy swapping to the virtual disk. This results in a 100% CPU load and processes start to hang with an unresponsive application as the result for the users.

Adding memory to the amount the server needs (indicated by checking the swapsize in the VM) has fixed these problems.

Oracle troubles

I shouldn’t be writing that all is well on the Oracle front.

Just now two of the Oracle servers froze with database problems. The DBA tells me that they have block corrupts which he hasn’t seen in five years of running the things.

More later.

{edit}

Both instances were terminated by PMON due to unreadable
logfile or spfile. The errors look like this;

ORA-00471: DBWR process terminated with error
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 26973

ORA-00470: LGWR process terminated with error
PMON: terminating instance due to error 470
Instance terminated by PMON, pid = 25260

One instance is a 9.2.0.4 database, the other a 10.2.0.1 database, on separate virtual machines.

For Oracle 9i: Restore from backup.…
For Oracle 10g: Resetlogs by:

startup mount
recover database until cancel;
recover database until cancel; (a message ‘media recovery complete’ will appear)
alter database open resetlogs;

We are viewing this as an unfortunate incident at the moment but will keep monitoring this closely.

Oracle DB/AS and Grid Control in production

So last weekend we migrated all the Oracle servers. After multiple issues with PowerConvert (it doesn’t clean up after itself properly if a migration fails) we eventually managed to convert an Oracle 10g, Oracle 10g AS and an Oracle 9iAS server. The others were re-installs.

It seems that there still is some ways to go on the Linux conversion front to make it a truly user-friendly application. Although we couldn’t have done it faster by hand it is kind of frustrating to see your migration fail time after time because of problems during copying or the program simply freezing up when it’s trying to take control.

On the Monday it turned out that our estimate for sizing the production 9i databaseserver was a bit on the cheap side (1 vCPU / 3GB memory) so we had to add a second vCPU. Oracle also likes it better when it can run an SMP kernel and it seems to hold up. We may upgrade the memory to 4GB but everything else works fine.

Oracle 10g Grid Control and AS servers virtualized

Also yesterday we converted our Oracle 10g Grid Control server. This is the server that we use for managing our Oracle databases and to which all these servers write their backup and archiving data (in practice: all servers have their archive and backup folders mounted to the management server).

As this server was still using direct attached storage the bulk of the conversion time went into copying all the backup data to the designated storage tier. Then the machine was PowerConverted and the backup and archive mounts on the database servers refreshed.

This morning we noticed that the machine was using just 11% of the allocated 3GB so we slashed that be 50%.

A much easier conversion was that of an Oracle AS (Application Server) server. This was just PowerConverted without problems. This morning we decided to shrink the memory allocated to that machine from 3GB to 1GB as it was just using 6% of it’s memory.