Remedy Archiving Part One - Primer

Over a period of time the amount of data stored in your Remedy system will increase and affect the performance of the application and the user experience. A slow application guarantees unhappy and inefficient users.  Worse still, over time, the accumulated data will cost more to manage and maintain.

 Applying best practice database indexing and regular maintenance can help mitigate the impact on the user experience. However, reducing data volumes in the right areas is certain to improve overall performance. The practice of managing the volume and location of the data that supports a Remedy system is one of the pillars of a performant, cost-effective system.

 A large data set means it costs more and takes longer to perform all database maintenance operations such as database backups, refreshes of development and test environments. Larger hard disks and tapes are required to store the database and its backups. Removing data from the on-line transactional database altogether keeps these maintenance overheads as low as possible.

 This article introduces the topic of Remedy data archiving and explores the out-of-the-box Remedy archiving functionality.

What will Archiving Do?

When we talk of archiving data, we mean the ability to move data from an on-line location to a different location. While Archiving solutions differ as to where archived data is stored and how it is moved, this is the core feature of any solution.

 The key benefits we look for archiving to bring are;

•  Improved Application Performance
•  Reduced System Administration Costs

The most successful solutions will also;

• Enforce Data Consistency
• Respect the Data Retention Policies of the Users
• Be Operationally Flexible

If you’re still reading this then you probably have a Remedy system that has a lot of data, and is starting to have some performance issues and for which you need to find an archiving solution. 

We hope this primer will be useful, however if you’d like to discuss archiving data in more detail please contact info@alderstone.com.

What Do We Archive?

Typically data is moved from the transactional tables where the most data is created and where reductions in data will have the most positive effect on the user experience. For BMC Remedy ITSM this means Incidents, Problems, Changes, Service Requests, Work Orders and Tasks (and all the data related to them), are all candidates for archiving.

The benefits of reduced data volumes must be balanced against the requirements of the business. Although a policy of archiving all Incidents that have been closed for more than 1 week would boost performance it would significantly affect the ability of the teams using the system to deliver effective services to the business.

Drawing up the data retention policy that is right for your BMC Remedy system will involve coordination with the various stakeholders and will include some decisions about the functionality that your archiving implementation will offer.

Requirements, Requirements, Requirements

IT Service Management is fast moving, and much of the data held in a BMC Remedy ITSM system is very time-sensitive. For example, the fact that today the head of purchasing has forgotten their Windows password and cannot run the month end reports, is a very valuable piece of information to the business today. However this fact will not be important three years from now.

By comparison, the fact that on average the Helpdesk were closing 20% more calls as First Time Fixes three years ago than they do today, is very important for the success of IT Service Management within the business.  

Not all data is equal and the requirements for data retention will differ depending on the type of data we’re considering.

A well-defined data retention policy is critical for the success of the archiving solution. Here are a few questions that you should consider as part of your exploration of the right policies;

• Do you need to retain data for auditing purposes?
• Are there key business reports that require longer periods of data to be retained?
• Do you ever need to un-archive data?
• Are there other sources where the data can be found such as a reporting data warehouse?

The time invested into identifying the stakeholders, understanding exactly what data they need and why, will more than pay for itself in achieving the right solution for your business.

The wrong solution can be more expensive to maintain and less effective for the business than keeping the data in the transactional database. In other words, there are just a few ways your archiving solution can make things better but lots of ways it can make things worse!

Take nothing for granted!

You may be told authoritatively; that there is no way that data can be removed from the system as it simply must be maintained on-line for the auditors or for regulatory requirements. Take the simple step of asking the auditors or relevant enforcement bodies. You may find the restrictions to be less severe than believed. Off-line archiving is a far simpler and cheaper solution than on-line archiving.

In summary, approach archiving as you would any other major enhancement to your system; take time to understand requirements and to understand the repercussions of the decisions you make.

Out-of-the-box Remedy Archiving

BMC Remedy provides the ability to create an Archive Form for any Form in the application. For example you can create an Archive Form for the main Incident Form (HPD:Help Desk) and set up a rule to move data from the “on-line” form to the Archive Form. This feature allows the time of the archiving to be scheduled to minimise the performance impact. It is also possible to set this feature to delete data rather than just moving it to the Archive Form.

The diagram on the right illustrates the way in which data is moved between Remedy Forms on the same ARS Server and consequently data is moved between tables in the same underlying database. 

Because Remedy OOTB Archiving only moves data out of the “on-line” tables into “archive” tables within the same database. If accompanied by database maintenance, this can provide performance improvements to the user experience as well as retaining the data in a location where, with bespoke Remedy workflow, data can be easily accessed by users.

Using Remedy OOTB archiving presents the following challenges;

Archiving affects Remedy Performance	Remedy OOTB Archiving uses the AR System Server to check the rules, query the data and move the data between the Forms. This creates a work load on the AR Server which varies depending on the indexes in the database, total volumes of data held in the Remedy Forms, and volume of data being moved. Scheduling the time of an archive run is therefore critical to ensure that end-user performance experience is not adversely affected.
Remedy API is slow	Manipulating bulk data efficiently is not a feature of the Remedy AR System Server. Using the Remedy API to bulk transfer data which is held in the database is far slower than moving data directly using the database. The Remedy OOTB archiving uses the Remedy API.
Scheduling and rules are not flexible	Ideally when managing the archiving of the backlog of data which has no doubt built up in your system you need flexibility to define when and for how long the archiving process runs for. Remedy OOTB Archiving does not allow us to run just in the evening on core business days but all day at weekends. Schedule changes can be made, but result in a large performance hit as the ARS Server re-caches.
Data is held in the on-line transactional database	If data is never moved out of the on-line transactional database then it will never stop growing. Larger databases cost the business more to keep in good working order than a smaller databases. OOTB Remedy archiving does not provide one of the key benefits of Reduced System Administration Costs.
Changes have to be replicated	A Remedy Archive Form must always match original Remedy Form. This means that all changes to the application in the form of BMC patches or major upgrades will need to be manually replicated in the Archive Forms. There is a development cost overhead for this.
Data Relationships are not enforced	If some of the data which makes up an entity is archived and some of the data is not, this can lead to unexpected application behaviour, potentially leading to data corruption and increased support costs. Data consistency is critical. Remedy OOTB Archiving allows a search to be run against just one Form and the data it holds; it cannot consider in data held in other Forms. All BMC Remedy ITSM entities e.g. Incidents/Problems/Changes/Service Requests/Work Orders/Tasks are made up of data held in multiple Forms.
For example, if we want to archive a particular Problem record we also need to archive the SLA Measurements, Work Entries, Tasks and Audit Logs which are associated with that one particular Problem record. Please find below an illustration of some of the relationships for the PBM:Problem Investigation Form. Note: This is for illustration purposes only, is a partial view and may vary on your ITSM application depending on local changes and application version

Bespoke Remedy application enhancements can make Remedy OOTB archiving aware of the relationships between Forms holding ITSM entity data. Unfortunately, this pervasive change modifies a lot of BMC Remedy ITSM Forms. Changes to the out-of-the-box Remedy ITSM Application obviously carry a cost in on-going maintenance and support.

With the challenges posed by the Remedy OOTB Archiving solution, companies are choosing to implement their own bespoke solution for archiving Remedy data that meets their data retention requirements.

Part 2 in this series will look at the design and implementation of archive solutions for BMC Remedy Applications