By Tara Wedemeyer
Tara Wedemeyer
Business Process Analyst
Farm Service Agency
The MIDAS project has followed a data quality lifecycle model that has many steps data goes through in order to implement a remediation plan. The first steps include profiling and testing data against validation rules to find anomalies. Next, validation workshops are held with Business and IT experts, data stewards, and the data owners who make sure that the profiling report and validation rules are valid. This ensures accurate results are produced when the data tests are run. These validation workshops will be conducted for every process that will be part of MIDAS. The processes that have been profiled so far are: Business Partner, Farm Records, Product Master, and Acreage Reporting. The outcome of the validation workshops is a remediation recommendation report, which holds all rules that pertain to the data object and each individual rule’s description, results, sample data and remediation recommendation. Once finalized, these reports are presented to the MIDAS data governance board to vote on. Once the recommendations of remediation are approved, they can be implemented.
For MIDAS there are three cleansing options that can be done manually or automated:
- Log the error and do nothing
- Cleanse in the source system
- Cleanse in the staging environment as a conversion.
For Business Partner, an example of a validation rule is, “verify the zip code data element adheres to a length of five digits for addresses located in the United States.” SCIMS is the source for zip code and this data element is conversion relevant, interface relevant and SAP system mandatory. The team ran this rule on 11,287,410 records and found some failed due to being blank or null. The recommendation to fix the failings of this rule is to correct data issues in source.
The project prefers to have cleansing done in the source system in order to have both the source and target system (SAP) in synch. This is especially important if interfaces are required back to the source system from SAP. Remediation is not a new idea; reports are given to the field today from the national office to clean up “bad” data. Also, there have been notices released directing clean-up efforts, for example Notices CM -701 and CRP-713. Having clean data is vital to the implementation of the new SAP system because the result of MIDAS is an integrated solution and there are dependencies on the data. For example, if a farm record fails to load correctly in the system, then the Acreage Report attached to that farm record may fail as well. To combat load failures and stay on track with the project timeline, selective cleansing is the path the project is following. “Hard errors” are the priority. A “hard error” is an invalid record that must be valid for SAP to load or a record that will impact business worse than it is today. As you read this article, Remediation Recommendation Reports for Business Partner and Farm Records are being presented to the board for approval, and notices will be written to address the hard errors.
Field offices should complete data cleansing on time. A target date gives the MIDAS data team time to run another profiling report, as well as ensure accuracy of conversions. This target date doesn’t mean cleansing must stop; data cleansing can continue in the source systems until cut-over. Please remember the importance of having clean data in SAP when you are asked to remediate by receiving a notice, a report, or an email calling for action.
