Every analytic application, reporting function, Web service, back-up and disaster recovery system, and client application depends on the rapid, accurate availability of data. Most organizations now need instantaneous access to real-time data, in fact, to meet business goals and maintain high customer service levels. That means batch updating is out. The days of Sunday server shut downs and offline updates are over for companies that traffic in huge volumes of data and real-time transactions. When end-users require data as soon as changes are made, those options are not in play. The article explains what data replication is all about, how it helps all rungs of society starting from students to executives. The article also throws light on key data replication considerations.
What’s needed is a reliable method for delivering up-to-the-minute information – on demand. For many organizations that means getting production data to end-user systems as efficiently and instantaneously as possible. The problem is that a lot of production or transactional data resides in databases that are once removed from the customer-facing application databases. Updates and changes need to happen much faster.
Large, complex organizations typically maintain numerous different databases and connectivity architectures that have evolved over the years. The challenge is to replicate and synchronize data across those architectures in real-time. In order to do that, you have to move beyond the non-real-time, inflexible Extract, Transformation and Load (ETL) methods of the past.
Industries like airlines, educational institutions, and emergency response/law enforcement agencies are using data replication solutions to improve business processes and meet user demands. Airlines, for example, are synchronizing information on numerous flight, crew, maintenance and catering scheduling systems. Government relief agencies are keeping emergency response data updated in real-time, secure, available and backed-up in order to maintain crisis readiness. And, universities are using replication solutions for resource optimization and information reporting.
What is Data Replication?
Data replication propagates exact copies of primary database data without disrupting ongoing operations. It’s a two-part process – replication and synchronization. Database replication software maintains an exact copy of a live database, typically for recoverability, high availability, or non-stop maintenance purposes. Database synchronization software distributes data among similar databases and keeps them in agreement.
The technical benefits are fairly straightforward:
- Committed, changed data is captured from within its operational stream
- Data propagation is based on data “subscriptions” created by the user. Only the data specified is replicated, so CPU and system resources are not wasted on unwanted data replication.
- Cross-platform integration can be achieved easily through replication. The data is sent to any target DBMS or middleware, regardless of platform.
- IT productivity improves because staff doesn’t need to be familiar with multiple data sources, and they don’t need to move data around via FTP and other methods. Replication puts the data right where the business user needs it.
- The replication system is completely recoverable from any point.
The business benefits include:
- Hardware and network costs can be reduced because only relevant data is pushed downstream to the target database(s) or middleware
- Operational information and resources are used more efficiently
- High information availability improves customer service and operational efficiency
- Business analytics, reporting functions and decision making are enhanced by having the most current information at your fingertips whenever there is a change in the source data
Any changes to specified data are automatically pushed out to defined target systems in accordance with pre-defined business rules. If, for example, an airplane delay causes a flight to be scratched, a replication system would update the cancellation to all relevant systems – including flight scheduling, Web ticketing, baggage handling and catering. Once an event is known, there is no delay in the spread of that information. Travelers are updated more quickly, and accurate plans can be made accordingly. Gone are the scenarios where airline personnel tap away at keyboards, trying to get a handle on the situation while the information simply isn’t available. That confusion buffer disappears, and certainty is quickly established.
Flying High, Protecting Citizens, Helping Students
To further illustrate the airline example further, a replication system can be used to proactively push data to their third-party databases and messaging systems. This keeps their flight scheduling and crew scheduling systems in sync and helps the airline manage its schedules to ensure on-time departures and arrivals. Planned or unplanned outages are virtually non-existent since they’ve eliminated batch scheduling. The system also makes it much easier for them to add servers without upsetting maintenance. The overall result—Airline passengers experience fewer delays, flight crews are utilized more efficiently, food is delivered on schedule, and employee and customer satisfaction has improved across the board.
A replication system can also be used in government agencies to ensure citizen safety by keeping their data available and ready for various potential disasters. All the information residing on the customer’s primary database is automatically replicated to another database in a different location, ensuring uninterrupted operations. The standby system in the other location then takes over as the primary production database and continues replication to other standby systems across the globe. This not only eliminates the need for time-consuming database recovery procedures, it ensures the business continuity critical for national security.
Universities can also use an event replicator system to make changes to information resources immediately available to students and faculty. This essentially help to address challenges that universities may face such as heavy database queries and internal requests for ad hoc reports which will slow system-wide performance. The 3-student population can then pull the system down with requests for “on-demand” information.
Users were affected in a number of ways. Sometimes they failed to access systems during peak demand times. Information that they knew existed was often inaccessible due to poor data integration, and the self-serve capability was effectively useless.
Data replication offered them a simple solution. The system they installed includes a second database that is an exact copy of the production database. This duplicate database is available for read-only querying on course schedules, grades, and other static information, while the production database can quickly process online transactions such as registration. Specific information is replicated and synchronized across all of the client/server domains that are exposed to the requests. The system that handles everything from class scheduling to financial aid and housing, automatically publishes the data in real-time to target applications. By propagating changed data on the fly and according to their pre-defined business rules, they easily meet student needs without increasing hardware and maintenance costs.
Data replication helps all these organizations achieve several significant benefits. Legacy data is now much more accessible to client systems and outside users. Productivity increases while hardware and software costs are reduced. And, availability and information reliability improvements enable much more continuous business operations – whether that means saving lives, shuttling passengers around the globe or getting students to class.
A Sound, Low-Risk Solution
Relatively speaking, replication is an elegant solution to a complex problem. Companies have tried migrating complete systems from one DBMS to another – but those projects have proved daunting and extremely risky. There’s also quite a bit of downtime associated with such projects. Another alternative is to hang tight with existing systems. Yet as database sizes and dependencies grow, performance degrades, maintenance costs soar and user malaise grows at a rapid clip. Some companies develop their own homegrown software for updating data. That’s risky, as well, since ad-hoc code can be prone to flaws and the original developers inevitably move on or retire. In either case the resulting costs can be very high.
Data replication solutions are worth much more serious scrutiny. All kinds of organizations – beyond the travel, government and education industries – stand to benefit from dependable real-time data delivery. Manufacturing supply chains are so tight these days that missing data and product parts can bring production to a halt. Web-based businesses live and die by the accuracy and speed with which they summon data and deliver it to customers. Fulfillment companies maintain tight schedules that rely on highly granular inventory data. Advertising and PR agencies rely on very precise and timely reporting data in order to sustain and test campaigns – especially now that highly accountable Internet advertising has become so prevalent.
The time is right to examine data replication alternatives to pressing business data delivery problems. Wholesale database re-alignment or periodic batch updating methods just doesn’t fit the bill these days. Most operations are time-sensitive and require immediate data availability as soon as changes are made. And the status quo is no longer an option.
6 Key Data Replication Considerations
- Early on, establish how quickly you need data delivered: replication solutions record changes at sub-second rates, effectively in real time. This is crucial for financial, airline and emergency response industries, among others. A stock trading institution cannot offer stock prices updates that are five minutes late, for example. There are other options for less critical data and less data availability-intensive industries.
- When your database needs to be upgraded, make sure the base technology still supports your replication tool: you want the vendor's whole family of products and tools to support the underlying system. Ask the vendor specifically -- “Will the current replication tool work with the new version of the database and supporting products?” Nobody can afford downtime due to software incompatibility or an unforeseen software bug. The replication concept is predicated on minimal time loss, so why would you tolerate incompatibility or downtime just because you have to update your system?
- Make sure the replication tool easily maps source tables and columns to their target destinations: most vendors offer easy-to-use GUI mapping tools. Make sure you these are available.
- Consider whether the replication tool takes changed data straight from the database nucleus or from data in the protection logs: 'From the nucleus' is preferable. When data comes from the nucleus, you forgo “puzzle re-assembly” steps, because only the data requested is sent. There's no protection log involved. With a protection log scenario, the replication tool has to sift through the data to find the most recent copy of the update, find the commit that guaranteed the changes, put it all together and then send it to the target.
- Examine real-time data warehousing options: performance and availability suffers when production databases that are constantly hammered by user requests. A real-time data warehouse solves the problem by replicating data to another location. Changed data is updated in real-time, and traffic can be diverted from overwhelmed production systems.
- Determine whether the replication tool replicates utility updates as well as user updates: with like-to-like database updates (for example, Adabas to Adabas) the tool should be able to do both. This is crucial for disaster recovery scenarios, where all data and utility functions need to be replicated in their entirety.
Bruce Beaman is Director of Adabas Product Marketing world-wide. Bruce has been with Software AG for over 20 years and most of that time he’s worked with Adabas and Natural. Many of our long-time customers know Bruce from the seven years he spent in Software AG training as an Adabas instructor. After his stint in Education, Bruce was part of the Technology Marketing Group for Software AG in North America where he teamed with other senior level product specialists. In 2000, Bruce joined Software AG ETS business line and is focused on planning and positioning Adabas and the Adabas family of products for continued growth.
|
