Duco allows you enable the tracking of records across runs. This means that Duco will recognise, for example, trades that are long-lived over their lifetime and show how their match statuses change from day to day.
When tracking is enabled, Duco is also able to determine the "age" of a partial match. This may be useful information when dealing with breaks. For example, it may be acceptable to have a partially matched record for two days, but not longer than that amount of time.
Why should you enable tracking?
- Because you find it useful to deal with breaks and view results based on how long records have been partially matched.
- By default in Duco, Tracking is disabled. When Tracking is enabled, a limit of 3 previous runs is applied.
Why should you keep tracking disabled?
- Because tracking requires an extra configuration step. If you don't need tracking, you don't have to spend more time than necessary to configure your process.
- Because tracking affects performance. The performance cost associated with tracking is modest, but can become noticeable for big data sets. It is recommended keeping tracking disabled for maximum performance.
- Because tracking may not be possible for your data set. To enable tracking you need to specify a set of key fields that identify a record uniquely. This may not be possible for some data sets. This section explains how to define key fields.
Suppose you want to enable tracking for the following process:
- Click on Settings → Record tracking.
This will take you to the Record tracking screen. Record tracking is disabled by default, as the following image shows.
- Toggle "On" in order to enable tracking.
- For tracking identifiers, enter a set of key fields that identify a record uniquely (more on this in the following section).
- Click on Save changes
To track records across runs, you need to provide Duco with a set of key fields that identify a record uniquely. In practice 1-4 fields are often sufficient. Good candidates are often fields like "Trade ID", "Deal ID", "Transaction ID", etc. Other possibly good candidates are date fields (e.g. trade dates) and fields that tend to be different for each trade (e.g., currency exchange rates or interest rates).
If you are familiar with database terminology, you can think of the tracking identifiers as a "composite key" for a data set. In the image in the previous section the fields "Fixed rate", "Observed floating rate" and "Trade ID" are used as tracking identifiers. They identify records uniquely because, given these three values, Duco can always find only one record.
Choosing an appropriate set of tracking identifiers can take a few tries. If the tracking identifiers don't actually identify records uniquely, Duco will exclude the non-unique records and mark them as duplicates. For example:
The image above shows that the fields "Trade Date" and "Long Lots" are used as tracking identifiers. These fields are clearly a poor choice, because it's likely that in the same data set many trades have the same Trade date and the same long lots. The result is the following:
What are duplicates?
Tracking fields are configured to uniquely identify an item in a data set. They're also used to make sure comments, labels, exception states etc get auto-updated when rows re-appear (often via carry-over).
The Duplicates bucket is displayed when a run contains more than one line item that shares the same values across the tracked fields. For example, Side A submits two line items that have identical values for Currency, Trade Date, Notional, and Counterparty. Duco will move both items to the duplicates bucket.
Something important to be aware of is that duplicate detection is only looking at the tracked fields. This can mean items flagged as duplicates could have differences in fields that are not tracked - which is why correct configuration of the tracking fields is crucially important.
Duplicates may also be captured because of carry-over. In that scenario, your data submitted may only contain a single line item with particular values in the tracked fields, but a carried-over item from a historic run could shares these values, causing a duplicate to be identified.
Duplicates can appear as a result of tracking fields that are not able to uniquely identify the item. These may be individual items that need to be reconciled and investigated. Hence, why choosing proper tracking keys is of utmost importance.
If this happens, you can mark these items as not duplicates. They will then be included in the run and become exceptions. They will be carried-over to the next run if they remain unresolved, and will not be treated as duplicates again in subsequent runs if new records matching the same tracking keys are submitted.
The best way to avoid duplicates is selecting the right amount and right categories for Duco to begin the tracking process. Any changes to the tracking keys after the initial set up, including reordering the keys or adding or removing keys will break the tracking history.
How does Duco select which item becomes a duplicate?
if an item is in the carry over and in the file (a duplicate) the version from the file is used, and the version in the process already is moved to the duplicates bucket. This allows bad data to be updated in a subsequent submission.
Can I turn off duplicates?
In order to keep the history of a record between runs, Duco uses values in the record tracking fields to create a hash which is unique for every record in the run. However, if there are two or more records that have the same values in all the record tracking fields, Duco is not able to create a unique hash. This does not allow Duco to distinguish the records in the run. Since Duco cannot choose which one of those records has to stay, Duco identifies all of them as duplicates, allowing the users to choose the right record.
In order to solve the issues of duplicates, process builders may add additional record tracking keys to help identify the record uniquely. Alternatively, the record tracking capability can be switched off, however in that case several functions of the exceptions workflow will not be available. Because of the technical tracking aspect, there is no way for Duco to not detect duplicates, but you can turn off the view of duplicates in the UI.
You may hide the duplicates bucket by going to results control and turning off duplicates for both sides. This will remove the duplicates bucket from the UI and generated reports.
Once tracking is enabled, the Process overview screen will show also an Ageing box. The Ageing box shows how many partial matches are older than 30 days, between 10 - 30 days old, etc. You can click on the number to view the corresponding partial matches.
In order to view the ageing process in more detail click on the Partial matches and then click on a ID match results to view the details.
When tracking is enabled, a match result contains also a History box. The History box shows a run history bar. You can click on a run number on the run history bar to see the same result on a different run. Click on the text below the run history bar to see the full details. The run history bar is colour-coded as usual (green = match, yellow = partial match, red = unmatched, and grey = excluded).
Record tracking best practices and FAQ:
Which fields should be tracked?
Choosing the right tracking keys is crucial to record tracking.
- Make sure to choose enough tracking keys that would help Duco identify records as unique throughout the history.
- In a case where you don't have enough keys, or keys that are not unique, the records will be sent to the duplicates bucket. For example. If the records have the same values in the record tracking keys, Duco will not be able to identify which is the correct one, so these would end up being registered as duplicates.
- The best tracking keys to choose are unique, such as transaction IDs, dates, or other identifying fields.
What if I want to change the tracking keys?
You can change them at any time, however, if you have already been tracking, changing the tracking keys will break the history of the previous tracked runs. Also, all exceptions investigative work will be lost. That is why it is highly recommended to be mindful when initially setting up the record tracking process.
Will comments and labels carry over between runs?
In addition to record tracking, in order for Duco to apply labels and comments to the latest run, or to copy the comments from the previous record, the previous record has to be closed based on the rules set up in the exceptions workflow.
I use Autoclosure as a part of my workflow. My labels and comments were not copied to the next run- why?
- Do you see the record in the previous run? If autoclosure was switched on and the comments and labels did not copy over, first check if the ID of the record is the same.
- As a best practice, when possible, tie autoclosure rules to record tracking fields. This will provide better consistency.
- If autoclosure rules are tied to the match fields settings, and if something in the match fields changes from the previous run, the exception ID will be different. Although it is the same record, it will not close and nothing will be copied over because Duco is unable to recognize it as the same occurrence.
If you manually match items, will the manually matched items carry over?
Manual matches are force matched. If the same two records show up again, they cannot become matched with anything else, provided that record tracking is enabled. If only one record is present in the run, Duco will not attempt it to match with anything else.
If you manually break apart a partial match in one run, do those unmatched items carry over to the subsequent run with new data?
Duco can carry over items only from the latest run to the new one. If you break apart a match in the latest run, you can expect those records to be present in the new run.
What happens if I manually close a record?
There are two different ways of closing a record. You can close it, which closes the exception in the current run. The second option is you can force-close it, which will close the record, and then keep the record closed in subsequent runs, considering it is the same exception. It will not expire the tracking history, but it will expire the exception. The next time you have the same record, the ageing will start at 0.
How does Duco identify the “new occurrence” of a previously manually force closed item?
There is an option in exceptions workflow to use either record tracking or matching fields to determine whether a previous item is the same item (and should be auto-closed or auto-accepted).
If the same item occurs again “naturally” (e.g. through the user re-submitting the same file), open items in older runs will be auto-closed and accepted items will lead to automatically resolved items.
If items have previously been manually closed, then there should not be an acceptance-style rule in place and the item should be investigated again manually.
Tracking with constituents:
This kind of tracking is only available only when you start using roll-ups in your settings. You can track records based on the values of the parent record or you can track based on the values of the constituents. If you are tracking based on the parent record, Duco will only take the values from the parent record and create an internal ID to track the history. When tracking by constituents is turned on, Duco takes all the values from those constituents and creates an internal ID.
It is important to note that any changes in the number or content of constituents will not affect the tracking as long as the parent record stays the same.
Record tracking has slowed down my run times. Why is this?
Currently, record tracking will look for all historical records that share the same tracking key values. In the future, the plan is to change this to make it more predictable. But currently, it is expected that record tracking will slow run times in larger data sets. This is something to be mindful of when determining if record tracking is needed on a process, especially processes with larger volume.