Use Data Prep to filter out unwanted data
Similar to data transformation and enrichment, it’s much cleaner to filter out data that is not needed for your recon by using Data Prep. This gives you clean data in the recon process to work with, while keeping the audit trail of filtered records in the upstream Data Prep process.
Recommendation: Filter unwanted data in Data Prep and keep the logic separate for easier maintenance and improved traceability.
Filter rules execution order
Sometimes you may have filter rules configured in the recon process. Note that they are always applied after the file load and before matching.
While setting filters :
- Please refer to How to setup filter rules.
- Records that meet the filter rule criteria are filtered out from matching. Define precise inclusion/exclusion criteria to avoid over-filtering and missing important data into your controls.
- Decide whether filtered items should remain reported according to your business needs. For example, if filtered records could be useful for exception management, then keep them reported.
- Governance: Review filters periodically to ensure alignment with evolving business needs.
Advantages :
- Improved performance: Reduces unnecessary data entering matching, optimizing run time.
- Cleaner exception sets: Only relevant records reach matching and exception review.
- Audit readiness: Clear documentation of what was filtered, when, and why.
Consequences :
- Performance degradation due to large unfiltered datasets.
- Loss of transparency: Harder to justify which records were processed or ignored.
- Audit risk: Missing evidence of filter intent or inconsistent application across runs.