Human validation is needed for documents that are not fully processed automatically. Depending on your use case, in the beginning of the project this might be more than expected but typically rapidly decreases after promoting production data to training data and re-training. 

In human validation, make sure you always check

• If documents are correctly split and received the correct type. If needed, perform page management and resume the pipeline.
• The issues section (required entities not found or below threshold, aggregated value not defined, parsing problems, ...)
• If all values are found for an entity and not just one. This is not strictly needed for processing most documents but reduces the load on managers performing QA tasks after promoting production data to training data.
• If no false positives are present.
• Business rules output.

By adhering to the labelling guidelines for training data while performing human validation, the load on managers for adding training data and performing QA tasks is reduced.

A manager should go through the production data daily and promote documents that were not fully automated and would be valuable as training data. Bad scans that caused mis-recognised OCR, irrelevant or exceptional documents should not be promoted to training data to make sure the training data stays clean.

Documents that were automated but still contained mistakes - for example when there were false positives - can still be promoted, even though they did not require manual intervention.

There is no use in adding documents that were 100% correct to the training data as it will only slow down training time and the model will not learn anything new.

• adding additional training data or performing new QA checks for specific document types, entities, layouts, suppliers, languages, ...
• reconfiguring business rules
• informing the source of the documents to correct them (for bad scans, irrelevant documents, ...)
• updating the annotation guidelines for human validation or training data. Remember to notify all operators/labellers.

Review and correct all annotations on documents that were promoted from production data to training data to make sure they are 100% perfect quality and do not contain mis-annotations.

It is crucial that this is done as diligently and accurately as during the initial training phase. Bad (or no) QA analysis can easily make models drastically worse.

The annotation guidelines should be revised and updated if necessary.

After the new training data is completely checked, you can retrain the model in model management. After retraining, check the resulting accuracy. If the results are good, you can deploy it to production.

Re-training is what makes models improve over time and makes sure your automation rates increase over time. Typically, the biggest improvements happen during the first weeks of running in production.