The tests run over single columns (for example, checking for rare, unsually small or large values, etc), pairs of columns (for example checking if one column is consistently larger than the other, contains similar characters (in the case of code or ID string values), etc.), or larger sets of columns (for example, checking if one column tends to be the sum, or mean of another set of columns). And, while this may not be interesting in itself, where rows are flagged multiple times for this or other issues, users can be more confident the rows are somehow different.

In this example, this suggests the identified rows may have been collected or processed in a different manner than the other rows. If the data consistently contains, say, four decimal digits, with no exceptions, this will be reported as a strong pattern without exceptions if the data in this column nearly always has four decimal digits, with a small number of expections (for example having eight decimal digits), this will be reported as a strong pattern with exceptions. For example, one test examines the number of decimal digits typically found in a numeric column. The tool executes a large set of tests over a dataset. The tool allows users to perform this work quicker, more exhaustively, and covering more tests than would normally be done. This does not remove the need to do manual data exploration and, where the goal is identifying outliers, other forms of outlier detection, but does cover much of the work identifying the patterns in the data and the exceptions to these, with the idea that these are two of the principle tasks to understand a dataset. That is, the tool essentially does what a person, studying a new dataset, would do, but automatically. The central idea of the tool is to automate the tests that would be done by a person examining a dataset and trying to determine the patterns within the columns, between the columns, and between the rows (in cases where there is some meaning to the order of the rows). The tool provides useful analysis for any EDA (Exploratory Data Analysis) and may be useful for interpretable outlier detection. It performs approximately 150 tests, identifying patterns in the data and any exceptions to these. A python tool to examine datasets for consistency.

Data dredging eda