Once you have set up and fine tuned a process, you may wish to automate certain steps. Doing so allows you to return results with minimal manual intervention.
Duco enables automatic data uploads, removing the need for manual input of files. When this setting is configured, a run is triggered automatically as soon as the data is received. This means results will be available to view as soon as you login.
To set up automated submissions, you'll need to liaise with the technical staff within your organisation and with Duco.
The set-up of automated submissions is a two-part process:
- You need to enable automated submissions in the process configuration menu found in Duco.
- Duco will provide you with SFTP login credentials and an upload location. The way data is uploaded depends on how your organisation operates. The technical staff within your company should be able to help in this respect.
Setting up a simple automatic file submission
Let's say you have a matching process with the following details:
Process Name “Futures & Options”
Input files: “internal.csv”, “exchange.txt”
The Data Inputs screen looks as follows:
To set up automatic submissions we need to:
- Define two data sources: "Internal" and "Exchange"
- Associate each data source with the process
To do this in Duco do as follows:
- Click on the three horizontal line menu and choose Administration (Note: do not select Submissions from the drop-down)
- Click on Submissions routing the menu that then opens.
This will take you to the Submissions routing screen
- Click on Add new data source
- Enter a name for a data source and a file name pattern
The result is as follows:
Now we need to associate the data source with the "Futures & Options" matching process.
- Click on Add
- Select the process "Futures & Options"
- Then select the "Internal" data input
This is the result:
Similarly:
- set up a data source for "Futures and Options_Exchange" using the file name "exchange.txt"
- ... and associate it with the same process, but the data input "Exchange"
This is the result:
Overall effect
The result of the above is that:
- When the file "internal.csv" is uploaded, it is routed as input for "Internal" side of the "Futures & Options" process.
- When the file "exchange.txt" is uploaded, it is routed as input for "Exchange" side of the"Futures & Options" process.
- When both files are uploaded, Duco executes the "Futures & Options" process.
File name patterns
The example above uses the file names "internal\.csv" and "exchange\.txt". This means that the files subsequently uploaded must also use these exact names. If they are not identical then the upload will not work. This requirement could result in the need for manual intervention if, for example, a system produced files with a variable format. For example, where a date was a component of the file name: eg. "internal-2020-01-01.csv" or "exchange-2020-01-01.txt", etc. Here, you would need to rename the files before uploading them.
To avoid this inconvenience, it is also possible to specify a file name pattern. For example, the pattern "internal.*csv" matches any file that starts with "internal", and ends with "csv". The following file names match the pattern: "internal.csv", "internal-full.csv", "internal-partial.csv", "internal-2014-04-18.csv", etc.
The image below shows a data source using a file name pattern.
Submission of ZIP files
It is possible to submit a ZIP file to Duco, containing either a single or multiple files.
For submissions where there are multiple files it is a requirement that they are all in the same format. Duco will automatically unzip the file and submit all of the files contained within as one side of the relevant process. The limit for the number of files that can be zipped for a single submission is 16,383 files.
To make use of this feature, set up a Data source as normal. If the zip file contains only a single underlying file, then from the File matching strategy drop-down select Match only submitted file name or contents when zip. Where multiple files are contained, select Match only submitted file name. This will ensure that the name of the ZIP file (rather than that of the files containing it) will be used to link to the data source to the reconciliation process.
Submit input files as soon as they are available
Duco only runs a process when both input files are available. In other words, it is possible to make asynchronous file submissions. For example, you could have a scenario where the file "internal.csv" is available at 10am and the file "ice.csv" is not available until 5pm. In which case you could simply submit each file as it becomes available. In this example Duco would run the process at 5pm, as soon as it had received both files.
The following diagram shows the sequence of events:
Sticky data sources
A data source can be marked as "sticky" to instruct Duco to hold on to the submitted file after it executes a process. Marking a data source as sticky avoids having to submit the same file multiple times. This should be checked before your file is submitted.
Let's say, referring to the previous example, that you want to match two data inputs: Internal, produced by a system within your organisation, and ICE, which comes from the ICE Exchange. Let's also say that the "Internal" file is available several times during the day and that the "ICE" file is available only once per day.
If you want to run the process each time an "Internal" file is produced, you can mark the ICE data source as "sticky". In this way Duco will hold on to the "ICE" file and use it again when you submit the new "Internal" file. Duco will keep using the same ICE file until a new ICE file is uploaded.
The diagram above illustrates this type of scenario. At 10 am the files internal-01.csv and ice.csv are uploaded. As soon as they are uploaded Duco runs a process using these two files. The ice.csv is a sticky data source. At 5pm a new file internal-02.csv is uploaded. Duco will then run the process using the newly uploaded internal-02.csv file and the ice.csv file, which has been kept because it is a sticky data source.
Submit a file to multiple processes
It is often necessary to use the same input file for more than one matching process. A user could approach this by simply submitting the same file to the multiple processes, multiple times. But this can be inconvenient. To avoid multiple submissions, it is possible to configure Duco to route the same input file to more than one process. This section describes how:
Setting up automatic file submission
Let's say you have two processes with the following details:
Process “Internal vs ICE Exchange”
Process code "INTICE"
Input files: “internal.csv”, “ice.csv”
Process “Internal vs CME Exchange”
Process code "INTCME"
Input files: “internal.csv”, “cme.csv”
Both "“Internal vs ICE Exchange" and "Internal vs CME Exchange" use the same internal data and you want to submit this data only once. This is how you can set up your processes:
- Set up the two processes as usual
We want to define three data sources: "ICE", "Internal", and "CME".
Each data source is associated with a "file pattern" and with one or more processes.
For example, the data source "Internal" is associated with the pattern "internal.*\.csv". This pattern means: a file name that starts with "internal", followed by any character zero or more times, followed by ".csv". For example, the following file names match the pattern: "internal.csv", "internal-full.csv", "internal-partial.csv", "internal-2014-04-18.csv", etc.
The data source "Internal" is also associated with both the "Internal vs CME Exchange" and the "Internal vs ICE Exchange" process. The effect of this is that, when Duco receives a file that matches the pattern, it uses the file as an input for both the "Internal vs CME Exchange" process and the "Internal vs ICE Exchange" process.
To do this in Duco do as follows:
- Click on More and then on Administration
- Click on Submissions routing
This will take you to the Submissions routing screen
- Click on Add new data source
- Enter a name (E.g., "Internal") for a data source and file pattern. To keep things simple we'll use a simple file name, E.g., internal.csv.
The result is as follows:
Now we need to associate the data source with the "ICE" and "CME" process.
- Click on Add
- Select the process "Internal vs ICE Exchange"
- Select the data input "Internal"
Repeat the same actions for the "Internal vs CME Exchange" process.
Here is the result:
- Then create two more data sources. One for "ICE" and one for "CME"
- For "CME", select "Sticky".
Result
The first result of all the above is that the "internal.csv" needs to be submitted only once.
- When Duco receives a file named "internal.csv", it will use it for both the "Internal vs ICE Exchange" and "Internal vs CME Exchange".
- When Duco receives a file named "ice.csv", it will use it for the "Internal vs ICE Exchange"
- When Duco receives a file named "cme.csv", it will use it for the "Internal vs CME Exchange"
Submit one file for multiple runs
When we set up the "CME" data source, we also selected the "Sticky" option. This instructs Duco to "hold on" to a data for the "CME" data source after each run until it is replaced with a new submission. The "Sticky" option is useful when you need to run a process with previously submitted data on one side vs new data on the other side.
Let’s consider the “Internal vs CME Exchange” process. Let’s say that the CME exchange data is available daily at 10am, whereas for the internal a partial data extract is available at 10am and a full data extract is available only at 5pm. For this reason, we want to run the matching process twice a day. Once soon after 10am and once soon after 5pm. This is the sequence of events:
Since we used the sticky option for the "CME" data source, the file "cme.*\.csv" can be submitted only once at 10am.
Searching on Data Routing screen
There is a possibility to look up the Data Source in the Data Routing screen. You can search by:
- The Name of the data source
- The Process Code
- The File Name Pattern
Monitoring submissions
Duco has a screen for this purpose documented in the Monitoring submissions page.
File upload via SFTP
Your technical support team will need the following:
- An SSH encryption key
- Secure FTP server details
- Upload location
SFTP on Amazon Web Services (AWS)
- The submissions folder is not created automatically. Any files submitted to SFTP, into any directory, will be submitted to Duco - there is no need to submit to the submissions folder.
- Submitted files are picked up immediately, rather than polled periodically. Some SFTP clients (eg WinSCP) that attempt to perform actions on the file following upload will, therefore, need to have functionality such as ‘Set Permissions’, ‘Update timestamp’ and ‘Resume Support’ disabled to avoid issues.
- Creating subdirectories is no longer allowed with AWS.
We do not prescribe how to upload files to the secure FTP server. You can use any client of your choice. Organisations typically create a script that runs automatically at pre-defined time intervals, extracts the input data from the organisation's internal systems, creates zip files, and finally uploads zip files to the secure FTP server. The Duco technical team will be happy to answer any queries you may have regarding this.