\nThis case study is used to teach many data science students today about the WWII Allied statisticians who examined the aircraft that made it back from missions. The statisticians wanted to see where the bullet holes were on the planes to instruct aircraft designers to reinforce those areas.\n
\nMakes sense, right? Except for Abraham Wald, a Jewish mathematician who escaped Nazi advances into Romania and Europe to emigrate to the U.S. He served in the Statistical Research Group, a bunch of egg heads at Columbia University who used math to make the military better at everything from firing rockets to shooting down enemy fighters.\n
\nWald had the epiphany that it was the planes that didn\u2019t make it back that had the most important stories to tell. Their bullet holes were truly indicative of where armor should be added, rather than the ones that returned. This meant there was a survival bias in the data, which would lead to incorrect recommendations. (Here\u2019s a great article<\/a> if you want to read more about this.)\n
\n
Past Relates to Current-Day Project<\/h2>\nNow coming back to more recent times and a similar project\u2026I was presented with data from turbine engines, airframes, and other major aircraft components that had been disassembled. I also reviewed the maintenance logs from normal maintenance modes. But I could not see the data of the equipment that had failed\u2014badly, suddenly or catastrophically\u2014or any of the information leading up to those events. Exceptional maintenance events, especially catastrophic failures, often didn\u2019t make it into the normal log information. We only had the survivor cases. Attempts to integrate data from forensic sources proved difficult and could not be normalized, so we didn\u2019t have a data set for proper inferences.\n
\nTo solve this, we had to get innovative about engineering new data sources. For example, in the area of turbine engines, we had to work with a manufacturer to coat key engine components with different trace elements. Then, we had to run the engines over time, drain the oil periodically, and measure the content of the trace materials in the oil, which would relate to the rate of wear of the specific components.\n
\nNext, we ran the test set under varying conditions to see how it impacted wear and what the precursors of failure looked like. Once sufficient data was acquired, we knew how an engine would age based on time and temperature. We even learned that certain modes of vibration would indicate an impending failure.\n
\nDetermining If Your Data is Biased<\/h2>\nNow to your data science project: Is your data survival biased? Has the culture and discipline around the measurements or metrics been unbiased? Will the data you plan to refine to make smarter decisions give you honest advice?\n
\nFortunately, there are specific industry sectors where this is a known problem, such as hedge funds, where only the funds that have survived are tracked and will likely give an upward bias to returns. Hedge fund companies take many steps to normalize their data so that they can clear any survivor bias.\n
\nThere are other kinds of data bias too:\n
\n\n \t- Sample Selection Bias<\/strong> happens when certain data is excluded; for instance, the specifics around cases that didn\u2019t go as expected or those with a great deal of anomaly edits.<\/li>\n \t
- Look-Ahead Bias<\/strong> occurs when the test parameter uses information that was not available on the test date; for example, in a price-to-book ratio, calculations are made with a mix of real-time data and data that was not available until the end of a quarter.<\/li>\n \t
- Time Period Bias<\/strong> results when a particular trend only happens to the data within a selected time period; this means the period was not statistically significant enough from which to gather a general inference. (Sometimes, external events such as interim government regulations can unknowingly skew the data).<\/li>\n<\/ul>\nThere are even data sets with surprise measurement bias that was acquired by two different generations in which the test or measurement criteria were not the same. This can be caused by the measurement techniques, equipment calibration, training of technicians, and other factors. It\u2019s not apples to oranges\u2014it\u2019s more like Fuji Apples to Pink Lady Apples. They look the same but aren\u2019t.\n
\nTechniques to Test for Bias<\/h2>\nThe good news is, there are techniques that have been developed to test for the different types of data bias. Some use deep statistical tests, and some require testing algorithms against independent data sets to validate similar conclusions are achieved.\n
\nThe selected tests usually map to a particular kind of bias that is suspected. These bias tests are part-and-parcel of a properly-managed data science project. After all, you want to be sure you have \u201cunbiased data\u201d before you allow it to drive your critical business decisions.\n
\nIf you have an internal data science team ready for a project launch, it\u2019s prudent to take a close look at the data prior to commencing. If the data is not what you need, you may have to consider additional methods and costs for obtaining the required data.\n
\nOr, if an outside organization will deliver a turn-key data science driven process or a black-box machine learning algorithm, be sure they take the time to survey your data for bias early in the lifecycle of the project. This might have implications on the data size (storage costs) and the required computational strength to achieve the inferences you are looking for.\n
\nIn the meantime, our sincere best wishes for you to make history with your data science venture!\n
\nIllustration: Martin Grandjean (vector), McGeddon (picture), Cameron Moll (concept)<\/a>, CC BY-SA 4.0<\/a>, via Wikimedia Commons<\/p>\n\n\n\n
\nFortunately, there are specific industry sectors where this is a known problem, such as hedge funds, where only the funds that have survived are tracked and will likely give an upward bias to returns. Hedge fund companies take many steps to normalize their data so that they can clear any survivor bias.\n
\nThere are other kinds of data bias too:\n
\n
- \n \t
- Sample Selection Bias<\/strong> happens when certain data is excluded; for instance, the specifics around cases that didn\u2019t go as expected or those with a great deal of anomaly edits.<\/li>\n \t
- Look-Ahead Bias<\/strong> occurs when the test parameter uses information that was not available on the test date; for example, in a price-to-book ratio, calculations are made with a mix of real-time data and data that was not available until the end of a quarter.<\/li>\n \t
- Time Period Bias<\/strong> results when a particular trend only happens to the data within a selected time period; this means the period was not statistically significant enough from which to gather a general inference. (Sometimes, external events such as interim government regulations can unknowingly skew the data).<\/li>\n<\/ul>\nThere are even data sets with surprise measurement bias that was acquired by two different generations in which the test or measurement criteria were not the same. This can be caused by the measurement techniques, equipment calibration, training of technicians, and other factors. It\u2019s not apples to oranges\u2014it\u2019s more like Fuji Apples to Pink Lady Apples. They look the same but aren\u2019t.\n
\nTechniques to Test for Bias<\/h2>\nThe good news is, there are techniques that have been developed to test for the different types of data bias. Some use deep statistical tests, and some require testing algorithms against independent data sets to validate similar conclusions are achieved.\n
\nThe selected tests usually map to a particular kind of bias that is suspected. These bias tests are part-and-parcel of a properly-managed data science project. After all, you want to be sure you have \u201cunbiased data\u201d before you allow it to drive your critical business decisions.\n
\nIf you have an internal data science team ready for a project launch, it\u2019s prudent to take a close look at the data prior to commencing. If the data is not what you need, you may have to consider additional methods and costs for obtaining the required data.\n
\nOr, if an outside organization will deliver a turn-key data science driven process or a black-box machine learning algorithm, be sure they take the time to survey your data for bias early in the lifecycle of the project. This might have implications on the data size (storage costs) and the required computational strength to achieve the inferences you are looking for.\n
\nIn the meantime, our sincere best wishes for you to make history with your data science venture!\n
\nIllustration: Martin Grandjean (vector), McGeddon (picture), Cameron Moll (concept)<\/a>, CC BY-SA 4.0<\/a>, via Wikimedia Commons<\/p>\n\n\n\n - Look-Ahead Bias<\/strong> occurs when the test parameter uses information that was not available on the test date; for example, in a price-to-book ratio, calculations are made with a mix of real-time data and data that was not available until the end of a quarter.<\/li>\n \t
![\"Henry](\"https:\/\/www.3pillarglobal.com\/wp-content\/uploads\/2024\/07\/leadership-henry-martinez.jpg\")
<\/div>\n\n ![Martin Grandjean (vector), McGeddon (picture), Cameron Moll (concept)<\/a>,](\"https:\/\/commons.wikimedia.org\/wiki\/File:Survivorship-bias.svg\"){kind=link}