Roadmap Overview
The intent of this document is to provide guidance in the form of overview of the components of ILM Retention Management, tasks, considerations, resourcing, etc. that can be used for continuous implementation of ILM Retention Management across the various applications in the SAP landscape.
The following aspects of ongoing implementation of data management and ILM Retention Management should be considered and will be addressed in this document.
- Sequence and Priority of objects
- Repeatable procedures
- High level timelines
- Data Quality
- SAP Platform
- Integrated systems / data copies
Archiving and Direct Delete
When the business framework for ILM Retention Management is activated in the SAP system certain enhancements are imported into/alongside the data archiving process. These include the ability to create policies and rules for retention periods that can govern the destruction of the data. With the ability to define the retention rules SAP has provided this enhancement to the standard archiving process. These functions can be used to apply retention rules for data being archived in the current project but can also be applied to historic archive sessions/files through the conversion process.
An additional feature has been provided in the archiving process for direct deletion. This is found as an option in the SAP ILM object variant. This feature uses the standard SAP archiving write process but does not store the data or update archive info structures. It also immediately deletes the archive file when the archiving sessions has been completed.
This feature is more streamlined than the ILM Destruction process using expired resources and the ILM Destruction worklist process. Therefore, it can be a good option for data or information that does not require the formality of the larger process. Technical data that does not have SAP provide direct delete reports would be a good candidate for this process. Other records with a much higher frequency of cleanup (for instance retention periods based on days or weeks) could be run effectively with this process. Overall, when the procedural requirement for execution of data delete can be accomplished with this process it should be considered.
This option is available for all ILM objects. As SAP provided this feature at the same time provided approx. 80 additional ILM objects that are only to be used with direct delete. Those objects can be found on t-code DOBJ.
Destruction Process and Priority
Unlike data archiving where the prioritization of objects to implement is commonly based on the size or growth of tables or in some cases performance bottlenecks or table row counts prioritization of destruction is based on risk exposure. On review of the objects in scope it appears the spectrum of risk
Although archive data files do consume space that space is typically on lower cost storage systems/devices and therefore not the primary driver for destruction.
Priority and process should be driven by the process stakeholders and not a responsibility of the IT department. Business object with a very small data base footprint could be critical to the organization from a risk perspective of maintaining that data longer than the policy retention period.
There is no technical requirement to implement destruction based on archiving sequence. For instance, archiving can require that the purchase req (MM_EBAN) be archived prior to the purchase documents (MM_EKKO). In some case purchase req could have a different retention period than some object categories in purchase documents and therefore the destruction is disassociated unlike in the archiving model.
The following table is a ranking of risk exposure provided as guide of what we have seen at other organizations. The list is sorted in high to lowest with a brief note for each.
Object Implementation
The list of objects below has been grouped into sections that can be considered for prioritizing which areas to implement.Priority and Sequence
The following need to be considered when prioritizing the implementation of ILM RM.
- What is the level of risk of being out of compliance with the data?
- Is there an info series applicable to the data in the SAP object?
- How complex is the application of the record series to the object.
- Are there additional fields required in the value determination tables
- Is there a requirement for additional functionality through development of a BADI
- Testing effort
- What is the status of the data quality for archiving
- Consider the amount of data that needs to be closed
- The complexity of closing that data
- Options such as updating the write job code to ignore the error
- How many resources are required for this object in terms of the ADK file storage and the size of the info structure table(s)
Resourcing and Execution
The following table is a guide to creating a priority for each object. The priority score can then be used to develop the implementation plan. Samples are provided.
Definition of attributes:
- Object – SAP ILM Object
- Risk – exposure of the organization to produce data older than required retention period. Also consider cost of discovery for data in this category
- Policy Mapping – the existence of a policy that maps to the SAP data/information/record and the clarity of the policy to the data within the ILM Object.
- Policy Implementation – the ability to use standard configuration to meet the rule requirements or additional configuration or program requirements to meet the rule requirements.
- Data Quality – the amount of open item data to be closed
- Overall Complexity – combined evaluation of the components
- Priority – Ranking of priority to implement the ILM object
The effort to implement can then be determined from the complexity of the object and additional factors added in the following table. Samples are provided.
Definition of attributes:
- Object – SAP ILM Object
- Archiving Status – what amount of the object has been implemented related to doc types, sales orgs, etc. and is the archiving up to date for residence period
- Amount of Conversion – from conversion run book to determine duration to complete for all files
- Policy Implementation – effort to define, map, config test retention rules
- Data Cleanup – effort to analyze open items and close the items
- Execution – estimate of duration to implement object
Considerations for Objects
For each of the following groups of objects we have provided a brief comment based on our experience or understanding for the objects in the groups.
Warehouse
Warehouse transactions typically have little or no open item considerations. Residence and retention periods are low. Standard ILM attributes may not include Country therefore mapping the policy to warehouse number or plant may be tedious. A destruction frequency of monthly vs yearly depending on the retention period may be appropriate.
Technical
In many SAP installations technical data is deleted based on SAP recommended periods. This process is often referred to as Housekeeping and includes the deletion of job logs and spools, system calls, etc. Archiving of technical data is not always required and in many cases a hybrid approach of prevention, deletion, and archiving is used to manage the data volumes. Given the nature of technical data and its retention requirements we don’t see requirement for technical objects using the ILM RM functionality since there are other ways to delete the data. But in the case of using archiving for technical data within the scope of this project you should consider having an info series that supports deletion.
Mapping of a retention policy to technical objects can be difficult and, in many cases, requires a special info series that can be used for the SAP technical data. Retention policies are typically created to define rules for records. In most cases SAP technical data does not meet the definition of a record and therefore is not included in a record policy. As a result, it might be required to create an info series with input from the IT department to help define the type of data in the ILM object. An info series can then be defined to cover most if not all of the IT technical data whether it’s in an SAP system, network or telephony, infrastructure etc. The following is an example from the US Federal government where the definition of Superseded or Obsolete can be determined by the IT and SAP functional users without further input from Records Management.
“A. [0013-0014] Short-term Information Technology Records These records encompass IT files described above that are not needed extended retention. Records are characterized by being necessary for day-to-day operations but not long-term justification of the bureau/office’s activities. This typically includes all records necessary for the management of a specific system or application, or a related group of the same (e.g. a server).
Disposition: Temporary. Cut off when superseded or obsolete. See records manual for specific criteria that may determine when a record is obsolete or superseded. Destroy no later than 3 years after cut-off.”
A different organization uses a Retention Code of Active or Superseded to be applied for SAP technical data.
Final notes on some of the objects.
- DB_TABLOG typically has the opportunity to stop logging. SAP documentation refers to certain tables that are logging by default and should be deactivated.
- MM_ACCTIT is obsolete functionality and very unlikely to have a use case for archiving. Consider OSS note 48009.
Procurement
Object MM_EKKO has been implemented in this phase of the project and is an example of an advanced object in terms of the complexity of the rule and the enhancements to the standard configuration to accommodate the rule. In the case where the other procurement objects – Purchase Reqs and Purchasing Info Records – are closely associated with the PO then it would be expected that the same level of complexity will apply to these objects.
Pricing/Material Cost
In our experience the material ledger objects have not been a challenge to implement retention management. The rules we have seen to date have been based on country and using time reference of create date. Note that create date needed to be added to the value determination tables.
Material Cost estimates CO_COPC is not ILM enabled in our system. There is no ILM documentation on the SAP site for this object. SAP provides a delete program for Cost Estimates as they are typically not required as a “record” for long term retention. Therefore, this data may be considered as technical data and be mapped to an info series for temporary or active data. See section above on Technical Data. Consider replacing archiving of this object with deletion using SAP t-code CKR1 report SAPRCKR1.
Pricing conditions data model is very complex and difficult to archive. SAP has provided time reference for end of validity which seems to be appropriate except that many organizations have pricing conditions with no end of validity or validity end dates like 9999. Therefore, implementing retention management for SD_COND will likely be challenging. Note that standard configuration does not include country and would need to be added if required.
Plant Maintenance
Records in Plant Maintenance can have exposure product quality from audit and regulatory perspective. In addition, exposure to litigation as a result of injury, vendor relations, etc. is possible. Therefore, maintaining compliance with records series should be considered as primary driver.
The level of support and standard config from SAP for ILM RM config for these objects is inconsistent. Two of the object PM_IMRG and PM_MPLAN are not ILM enabled in our Auritas Lab release. OSS note or message would be required to enhance these objects. Only about half have Country available for determination.
Order To Cash
Objects in OTC have varying degrees of complexity in part due to the breadth of the objects across regions and functionality. For instance, RV_LIKP includes both Inbound and Outbound deliveries. SD_VBAK includes everything from Quotes for Repairs to Cash Sale or Master Contract.
Unlike archiving where there is a sequence of objects based on dependencies the destruction of any object is independent of others from a technical point of view.
In our experience OTC data has been subject to legal hold in most of the cases we have implemented.
Master Data
We have not implemented retention management on the objects in scope – Functional Locations, Equipment, or Projects. Our experience with these objects has been they are difficult to archive as a result of the business complete requirements. Therefore, when evaluating these objects priority of retention management considers the overall archive ability of the object. Otherwise, if
For more information on Master data see the section in this document on general approach to Master data.
Manufacturing
Objects in this group can be difficult to archive and therefore may extend the duration of fully implementing the object from archiving to RM. PR_ORDER (and associated PP_ORDER) do not have robust access to the archive from standard t-codes. As a result, users can be reluctant to allow for aggressive residence periods. On the other hand, the requirement to access historical manufacturing documents may not have a strong use case. Both these conditions can result in residence periods coming near retention periods. If this is the case Direct Deletion should be considered.
The manufacturing logs – object PP_BKFLUSH – are rarely implemented in our archiving projects. Recent experience proved that the archive files contained most of the data from the MM_MATBEL object. The logs for this process may not be easily mapped to an info series since the information seems to be system related only. This object and information could be classified as Technical Data and be governed by an info series for the same.
Finance and Controlling
Implementation of RM for FICO objects – like archiving – is well supported by SAP. Value determination and configuration attributes are provided as standard configuration. Due to the size of the FICO objects primary consideration may be the time to run the conversion process.
Certain objects such as the vendor/customer balances and CO Totals and PCA Sum may have very different retention requirements. The vendor/customer balances rolling forward each year may have no long-term retention requirements. Whereas CO Totals and PCA Sum may have long term business requirements but could be more appropriated maintained in the BW or other reporting system.
Environment Health and Safety
Previous experience with EHS was in regard to where the data was being stored. The companies Records Management team did not let the EHS sheets be stored on cloud system. This was likely due to insufficient contract guidelines about the security of the cloud storage and likely was resolved. It is worth considering if EHS or any other sensitive information has specific requirements regarding where the data can be located.
If the policy trigger for incidents is based on status of Closed or Void, it may be required to have a badi to determine this status based on the last change date.
Custom Objects
Custom archiving objects can be enhanced to include the functionality required for Retention Management. Alan has done this for a number of objects and although the code is similar for all objects the value determinations need to be identified and added to the configuration at multiple points.
Retention Management Set-Up
Creating rules for managing the retention period for each object starts with making the system aware that the object is intended to use the enhanced features of ILM RM (as mentioned in the previous section). This requires the object to be added to the Audit Area. The use of Audit Areas has been reviewed and, in this project, a single audit area has been created. When the archiving object is added to the audit area it takes on the additional features and becomes an ILM object. If this is not immediate for an object as the implementation of 40 or more objects refer to SAP Support for notes on what needs to be done to proceed with ILM enabling to object.
Repeatable Process
There are three repeatable processes in the execution of ILM RM. The first is the setup and testing of the retention policy and the conversion of the archive files. The second is the destruction activity that is carried out based on organization procedures for records management. The third process that is repeatable but not on a regular basis is the legal hold application and removal.
Rule setup and testing. During the initial implementation, the AP team became aware of the different levels of complexity of the retention policy and how to apply the rule in the SAP configuration. Simple rules for MM_MATBEL vs. complex rules for SD_VBAK and MM_EKKO took significantly different levels of effort and time to implement. The process and methods are the same for all objects but as a repeatable process, it is wise to keep in mind that not all objects can be implemented in the same amount of time. Although the core team including Arlene, Travis, and Schanna remain the same, different resources from the organization are required based on the objects. The following chart has been developed based on the implementation to date to provide estimates for the duration of implementation for objects that are scored as High, Med, or Low in complexity to implement.
Note that the above table does not include time and effort to address data quality issues. The duration estimate does not include conversion run time. Refer to Conversion Run Book for execution duration.
Destruction Runs. The destruction process, unlike the rule set up, is the same for each of the objects. The ILM_DESTRUCTION t-code and the procedures that AP develops to execute the destruction within the t-code should remain as these steps should be done by the responsible data governance lead. The creation of worklists using drag and drop, or the Automated Worklist remains the same procedure for each destruction cycle.
Legal Hold. The expectation that legal holds will be part of the ILM Retention Management program at any organization is confirmed by the fact that there is likely a current hold that legal is aware of. Therefore, Legal Hold is included as a repeatable process although every hold will be different and require enhancement of the eDiscovery programs for each case. The primary considerations for a legal hold to be implemented in SAP are 1) the records to be held and 2) for what time period.
If the order to preserve does not identify the type of records to be held then it should be requested of the legal team to do so. For instance, if the hold states deliveries, invoices, returns, etc. then the hold can be reasonably applied. If the order contains no specificity, then it would not be reasonable to apply at hold until further detail is provided. For instance, if the order was simply “all records associated with a certain project or material”
If the order to preserve is for records within a date range for which there is no scheduled destruction activity, then it is recommended to consider the hold to be applied only if/when the destruction of the held data is becoming eligible for destruction. For instance, if the hold is for data in a period of 2-3 years from the current date and the normal retention period for the data in the hold is 10 or 30 years then it would not efficient at the current date to create a hold for a legal action that may be resolved before the normal retention period has been met. But, given the challenges of implementing the legal holds it is recommended that if resources are available model the legal hold in the non-production system so if/when the hold is required the implementation period is reduced since the design and testing would be completed.
Finally, the complexity of the hold and scope of objects to be held needs to be analyzed and a gap review based on current SAP functionality. On a recent project, the hold requirement was for Production and Process Orders. Alan did an analysis and estimated over 120 hours to create and test the eDiscovery programs and he is not able to confirm that the entire process will work. Meaning from eDiscovery to propagation, extraction, and the hold itself all require full testing and no doubt an OSS message or two.
High-Level Timeline
Based on the work done to date an estimate has been provided using the above-mentioned groups to determine a very rough timeline. This does not include project freeze periods or scheduling of releases etc.
The assumption is that the team can do 3 objects concurrently. The order is arbitrary as there has not been a prioritization of objects at this time.
Master Data
For this section, we will consider Customer, Vendor, and Material master data only. Although there are other master data objects, in particular Business Partner, we will reserve that object for follow on discussions around the requirements of privacy regulations.
Master data archiving primary consideration is Data Quality. Because the SAP archiving process requires the complete business status of the object being archived and, in many cases, – especially master data archiving – business complete and/or archived of prerequisite objects these implementations become complex.
Most archiving write or preprocessing programs provide detail on the open item state of the specific objects which allows for direct action to “complete” the business process on that transaction to make it available for archiving. See the following section on Data Quality.
A prerequisite step for archiving master data is to have it marked for deletion. It is recommended that the business process team maintains this status and owns the responsibility of setting delete flags. Certain archiving objects have a preprocessing job that can set the delete flag but these should be used by the archiving team only with direction from the business process owners.
Master Data as candidate for destruction is typically event-based versus time calculated from creation or posting. In addition, master data for destruction may be defined as discrete lists of vendors, materials, customers, etc. and therefore, the RM rule may not be easily established. The RM rules (like archiving) typically are not created based on the individual record number.
Data Quality
Archiving is the process of moving data from the high-cost database space to lower-cost storage. This is a strong value proposition including improvement in performance. Given the goals of data archiving it is not critical that every transaction for any given year has to be archived. In most cases, archive effectiveness of over 90% is acceptable when implementing objects with complete business checks.
When archiving is done as the initial step of data destruction open item records need to be included in the archive files. This presents certain concerns about how to get this done. An example of open records for the object MM_EKKO – Purchasing Docs is displayed below. Note this is not from a previous customer system.
To address some of these issues downstream processes will likely be triggered such as posting in finance. Project teams need to consider the impact – in this example – of creating a financial posting in the current year for a transaction from over 17 years ago. Also worth noting is the residence period configuration which is likely many months if not years. If the project team is to correct open item issues, there may be a requirement to temporarily lower the residence period if that residence period is based on the change date.
Options to continue with the archiving process are to remove the achievability checks from the write programs. This should be done with caution and only for specific periods of time. This is similar in concept to the SAP archiving Snapshot variant with the exception that the records are not removed from the database therefore not meeting the goal of the retention management implementation.
SAP Applications / Platform
SAP has provided the ILM Retention Management components in transactional systems somewhat inconsistently. In CRM we have found that standard retention management through policies and rules works as expected. But CRM does not appear to have the legal hold functionality available. Other systems such as GTS appear to function as expected but given the limited reach of the legal hold capability in ERP, it is unlikely legal hold would be easy to implement in non-ERP systems.
Data deletion in HCM has been available from very early releases. The introduction of ILM HR objects has been somewhat recent. In our experience analysis and consideration of using both historical deletion programs and ILM objects. ILM RM does not have (or did not have) an object for all the personnel info types. A thoughtful approach to data management in HCM, especially in the context of privacy regulations, needs to be planned for and include input from HCM knowledgeable resources.
SAP BW is oftentimes not considered in the scope of retention management due to the nature of the data. Retention management typically applies to records in their original form. Data typically stored in reporting systems do not contain the complete record but only those attributes required for reporting. SAP has not provided ILM Retention Management in the BW reporting system. Due to privacy regulations SAP has recently provided “SAP ILM Notifications” to collect information about archiving and deleting activities in the ECC system and then “inform” other systems. We have not used this functionality and therefore cannot comment on the viability and use cases.
Integrated System
During archiving implementation design and testing of affected integrated systems is typically done and in most cases little if any impact. Some tools like HANA Enterprise SLT Replication may need to be configured to ignore archiving activity but otherwise thorough testing during archiving mitigates any issues.
When executing Retention Management, it is important to keep in mind record copies that may be distributed to other production systems and in the case of privacy regulations the distribution or integration to 3rd party business partners. Certain components of ILM Retention Management can have the capability of evaluating the existence of data such as the block and delete programs for customer, vendor, and business partner EoP and EoB.
As previously mentioned, SAP Notifications have been provided for synchronizing external systems with the results of archiving and destruction.
Conforming to Privacy Regulations
The following section is not an official position of Auritas LLC.
Privacy regulations have been implemented in the EU in the form of the GDPR and in the US primarily in California. These regulations have been in place for a number of years and yet there are a few cases that have been submitted to the various authorities that would seem to involve data in an SAP ERP system. There have been some high-profile cases including a recently filed action against Amazon. The larger cases like Amazon and including British Airways and Marriott as other examples are a result of data breaches. And in some cases, data breaches of companies from before GDPR was implemented.
Through various forums such as SAPPHIRE and working directly with product owners from SAP, it is surprising to see the lack of “activity” around this issue. The SAP Block and Delete capability has been released for supporting a data subject’s right to be forgotten. The Information Retrieval Framework (IRF) was made available for data mining to find the purpose and use of PII. We have worked with the SAP GRC product owner to provide requirements for a privacy compliance tool but that is only for S4, it seems to only address GDPR Article 30 which is to identify Record of Process, and no release date for the completed product has been determined. Where you would seem to find the greatest exposure in the SAP CRM system, we have not seen the development or product maturity of a privacy regulation-compliant solution. We have taken the SAP course and have set up certain scenarios in our lab system anticipating a long list of customers wanting to be ready to address the privacy regulations but it has not materialized. In fact, there have only been a few projects that have done a very modest implementation of “blocking” data and that was not driven by GDPR.
Interpreting the GDPR does give guidance on some measures that the organization should perform to provide a level of compliance to a privacy auditor. One thing is the GDPR, in a sense, mandates the destruction of data after End of Process (EoP) and jurisdictional retention requirements. Regardless of an individual data subject requesting the right to be forgotten, records past the retention policy need to be deleted.
CCPA and other regulations that protect the right to Opt-In or Opt-Out seem to have contradictions in some business models. For instance, I like my loyalty card at my local grocery store. To maintain my loyalty card within the regulation of CCPA if I chose “Opt-Out” doesn’t seem possible. As a result of the difficulty interpreting these regulations – even the definitions of PII seem unreasonable – lack of true caseload, a consideration that a regulation is not legislation until litigated, difficulty in applying these requirements in complex ERP and integrated systems, and other reasons the rush to get Privacy compliant does not seem to have or to be a priority. Other data and information systems are likely far different.
Every organization needs to assess the impact and exposure to privacy regulations and determine the course of action. Is it enough to have SAP ILM RM in place and destroy data after its retention policy? Possibly.
Conclusion
ILM Retention Management and destruction is a technical next step in SAP data volume management. But unlike data archiving which can often be implemented by IT departments with input from business process owners, in our opinion Retention Management is “owned” by other groups within the organization. The primary stakeholders are usually Governance Risk Compliance, Records Management, Legal, Tax, and Audit. IT department has the role of Educate and Facilitate. Educate the stakeholders with regard to the capabilities and requirements of the SAP ILM RM product and process. And facilitate by installing and maintaining the software components.
If you’re interested in learning more about how Auritas products and services can best serve your organization and digital transformation initiatives, schedule a call with our experts here: Select a Date & Time – Auritas Data Health Consultation
Further details on SAP functionality mentioned in this document and any other topics in SAP ILM can be found in the SAP Press publication SAP Information Lifecycle Management The Comprehensive Guide.