No matter how secure and well-planned a facility is, there will eventually be some issue that requires recovery operations. These plans should be formulated beforehand so that the time of recovery is restricted simply to following the plan, rather than having to figure out what to do.
Foreplanning
The first part of a disaster recovery plan is to determine what the most likely problems are, and how severely they will affect the functioning of the facility. A common way to determine this is via various metrics, such as
- How long can a network outage be sustained? All outages will impact functioning, but facilities that require constant network connectivity to function and that will lose large amounts of money while being down should be treated differently from a facility for which an outage is a mere inconvenience
- How long will it take to get a facility back online for various outages? What is the cost of this downtime?
- How often can outages be expected? Certain areas are more prone to ISP outages and other types of problems than others. Our more rural clients are used to outages that our big city clients would never tolerate.
These, and other, criteria should be used to formulate a plan that takes into account the most likely problems, the most expensive problems, and the problems that will be the most difficult to mitigate. All of this will combine to determine the best plan of attack in case of emergencies and disasters.
DR Sites
Disaster Recovery Sites come in several different “flavors” – cold, warm, and hot. These relate to the amount of time it will take to move operations to the new site. A “cold” site is available, but will take some time to get setup – perhaps equipment will have to be physically relocated to it, or an ISP-circuit will have to be activated. A “hot” site is, essentially, always ready to go and take over operations. A warm site sits between these two extremes.
Obviously, there is a tradeoff between mitigation time and expense with regards to the type of plan to implement. For our smaller clients, their version of a “cold” site involves us configuring and then shipping/delivering replacement equipment, which can take 1-2 days. More mission-critical clients will have a hot site – there is redundant equipment available in a second location that just needs to be configured to handle the workload. Often, this takes the place of virtual machines and other cloud appliances that just take some DNS updates to begin routing the traffic to them.
High-Availability
Our typical HA approach uses dual firewalls, wherein one is a clone of the other. We also have dual internet circuits running into both machines, with a setting on the firewall designating one firewall as the primary and the other as the HA secondary. If the primary firewall fails, the secondary will begin handling traffic within seconds. If one of the ISP circuits fails, the other will almost immediately begin handling that traffic.
Downstream from that, we have dual switches running into the physical on-prem servers dual (or more) ports; if one fails the other will handle all the traffic.
If the physical machine fails, we can route traffic through the cloud virtual machine (which can be cloned & duplicated from backup if that fails as well).
We haven’t had any equipment failures, fortunately, and ISP outages are handled by the equipment with no interaction required from us. The only disaster recovery we’ve had to use is from power outages, which, if the batteries & UPS begin to alert low capacity available (we have about 1.5 hours battery time), we can manually switch the flow from the primary power circuit to the generator (which will automatically turn on in a power outage).
Testing
Testing is our weak point – our guaranteed uptime is 99.999%, and unfortunately our contract does not make any distinction between testing, actual disasters, software or hardware upgrades. So our 0.001% downtime is almost always consumed by software updates & reboots. But when disasters have struck our DR plan has functioned well. In an ideal scenario, user buy-in would allow testing and even encourage it, as planned downtime is better than unplanned.
Recent Comments