Data Cleaner Module

The Data Cleaner is Indicia’s version of the NBN Record Cleaner tool, in other words Data Cleaner provides verification of incoming records.

Note

Validation is the process of ensuring that the format of a record is correct. A valid record might still prove to be a misidenfication or have an incorrect date, but validation ensures that things like the grid references and dates associated with a record are formatted correctly.

Verification is the process of ensuring that a valid record is has a correctly identified species. It is an essential part of the process of biological recording since by verifying a record its potential usages are much greater, e.g. it might then be usable in a planning enquiry or for scientific research.

The Data Cleaner module is optional but must be enabled to perform automated verification checks against incoming data. It does not perform any checks itself but coordinates the other rule modules which are enabled on the warehouse, so you will need to enable at least one of the other associated modules on the warehouse (these are modules whose folders start with data_cleaner_). The Data Cleaner Module

  1. Runs when the scheduled_tasks.php link on the warehouse is called - see Setting up the scheduled_tasks <../scheduled-tasks>.

  2. Finds all new and modified records in websites where the Enable auto-verification checks option for the website is checked.

  3. Runs a series of rule checks against the data. Each rule check is effectively an SQL query statement defined in a sub-module enabled on the warehouse.

  4. When violations are identified, comments are created against the record which detail the violation. The comments are flagged so that it is clear they are auto-generated by the data cleaner sub-module which generated them. Any previous comments from the same module are removed so that comments do not accumulate if a record is re-verified.

Verification checks can also be requested for unsaved data if a recording form calls the web services provided by the Data Cleaner module, e.g. by using the data_entry_helper::verifiation_panel control or by selecting the Include verification precheck button option on a dynamic form. For more information on the web services that support this feature please refer to the developer documentation.