Major
Data Science
Research Abstract
As the amount of publicly shared data increases,
developing a robust pipeline to stream, store and process data is
critical, as the casual user often lacks the technology, hardware
and/or skills needed to work with such voluminous data. In
this research, the authors employ Amazon EC2 and EMR,
MongoDB, and Spark MLlib to explore 28.5 gigabytes of CMS
Open Payments data in an attempt to identify physicians who
may have a high propensity to act unethically, owing to significant
transfers of wealth from medical companies. A Random Forest
Classifier is employed to predict the top decile of physicians who
have the highest risk of unethical behavior in the following year,
resulting in an F-Score of 91%. The data is also analyzed by
an anomaly detection algorithm that correctly identified a highprofile
case of a physician leaving his prestigious position, as
he failed to disclose anomalously-large transfers of wealth from
medical companies.
Faculty Mentor/Advisor
Paul Intrevado, Diane Woodbridge
Included in
Predicting Unethical Physician Behavior At Scale: A Distributed Computing Framework
As the amount of publicly shared data increases,
developing a robust pipeline to stream, store and process data is
critical, as the casual user often lacks the technology, hardware
and/or skills needed to work with such voluminous data. In
this research, the authors employ Amazon EC2 and EMR,
MongoDB, and Spark MLlib to explore 28.5 gigabytes of CMS
Open Payments data in an attempt to identify physicians who
may have a high propensity to act unethically, owing to significant
transfers of wealth from medical companies. A Random Forest
Classifier is employed to predict the top decile of physicians who
have the highest risk of unethical behavior in the following year,
resulting in an F-Score of 91%. The data is also analyzed by
an anomaly detection algorithm that correctly identified a highprofile
case of a physician leaving his prestigious position, as
he failed to disclose anomalously-large transfers of wealth from
medical companies.
