The Backout Plan

Perhaps this is another legacy of my submarine days, but I always like to have a plan in place to handle the situation wherein problems come up and I have to abandon my upgrade efforts. Perhaps it is just that I do not want to have to “wing it” when the problems come up and everyone is yelling for a solution, or perhaps I just want to assure my customer or boss that I can handle it if the software is bad or some other problem arises. This is especially important when performing upgrades to production systems.

What are some of the items that should be in this backout plan? Well, the first thing is the criteria that would cause you to have to perform the backout. It’s a good idea to get management people to agree to this in advance so that they do not act like armchair quarterbacks the next day and tell you that you should have done it sooner or waited a little longer. State the criteria in terms of time (“If there are still problems at 5 p.m. and there is no immediate solution, begin backout so the system can be operational for nightly batch processing”) and other conditions (“There is a bug that Oracle needs to investigate and it indicates it may be several days”). Again, much like the installation plan, send the backout plan to managers (project managers, DBA group managers, and so forth) so that they can look at it and get their shots in before problems occur. Make them understand that if they do not give you comments, you have their approval to proceed according to the plan.

The main part of the backout plan is the steps needed to recover the instance fully so that it can run under the old software. This usually means restoring a complete, cold backup of the data and resetting parameter files, such as oratab, to point to the old version of the software. List all the files that need to be restored. This serves two purposes. First, you do not have to rely on your memory when it is late, you have been struggling for hours with a particularly annoying bug, and you have to restore the old instance to get production going again. Second, it serves as a good checklist for your backup that is performed as part of the upgrade plan. You really want to ensure that you have backed up all the files that your backout plan will require. This may seem simple on small, single-disk computer systems, but when you have files scattered across 135G of disk or more, it can be a challenge. Remember, Oracle is not forgiving if you remember to restore only 95 percent of the files that it needs.

Lining Up Support

As has been mentioned frequently in this chapter and Chapter 6, there can be problems with the installation and upgrade process. They can be worked through, but it takes time and support from other people to make the installation and upgrade process work. Specifically, there are a few resources that you should consider lining up during the planning stage so that you can have them ready when you actually perform the upgrade:

•  The system administrator should be available to provide support ranging from tape backup restores to simple things such as resetting directory write permissions. If you also act as the system administrator, you can do this yourself, but be careful to not act too quickly and to think things through. The system administrator accounts have a lot of power and therefore can do a lot of damage (recursive deletes of files when you are in the wrong directory, for example).

•  If you and your system administrator are not very familiar with the host operating system, you may consider performing the upgrade during a time when you have access to operating system vendor technical support. This can come in handy when the installer is screaming that it cannot find mk and you do not even know what mk is.

•  You should understand what level of service you have with Oracle technical support and perform your upgrade during the period in which you are allowed to call Oracle. Like most vendors, Oracle provides 24-hour support, but it charges you much more than the 8-hour, weekday support contract. A fair percentage of installation problems can be solved quickly by Oracle over the telephone, so schedule yourself accordingly.

•  It is helpful to have a developer available to test out applications after you have completed your upgrades. In most of the instances that I have worked on, I knew the data structures and basic processing flow, but I was not a user and therefore was not familiar with navigation between panels.

•  Finally, it is helpful to have a shell script programmer available in case you have to perform functions such as manually linking the software. Usually you can do this with support from Oracle over the telephone, but you might need to find someone locally who knows all the tricks you need to do, such as setting up your path and environmental variable to access these development utilities. System administrators are familiar with the general concepts, but they usually do not work with these utilities regularly.