Fundamental Oracle flaw revealed

The combinationBut when you mix the hot backup bug with large numbers of interconnected databases in a massive Oracle implementation, the combination can result in widespread SCN elevation. Some large Oracle customers have hundreds of database servers running hundreds of Oracle instances interlinked throughout the infrastructure. Each one may be tasked with a core service and a few lesser functions -- but like Kevin Bacon and the rest of Hollywood, nearly all are linked together somehow, through one, two, four, or more intermediary servers.

With all of these servers interconnected, their SCNs become synchronized at one point or another. Collectively they might be pushing more than 16,384 commits per second, but certainly not since 01/01/1988, so the SCN soft limit isn't problematic.

But what if a DBA on one part of this Oracle network runs the flawed hot backup routine Suddenly, the SCN on his Oracle instances shoot up by, say, 700 million, and this number soon becomes the new SCN for all interlinked Oracle instances throughout the organization. Some time later, another hot backup command is issued by a DBA on the other side of the company. The SCN shoots up a few hundred million this time, also synchronized across all connected instances over time.

With the issuance of a few commands, the SCN of the entire Oracle database infrastructure has increased by hundreds of millions or even hundreds of billions in a short period. Even database instances that connect only occasionally, or for a weekly or monthly batch run, might see their SCN numbers jump by trillions.

In such a scenario, it may be only a matter of time before enough backup commands are run to cause the SCN to eclipse the soft limit -- at which point every interlinked Oracle database server suffers some significant problem, refuses connections from other servers, or simply crashes.