Hidden Errors in Big Data: The Case of Property Records

Summary

Big data are the foundation for an increasing share of academic research and AI models deployed in both the public and private sectors, prompting substantial growth over time in reliance on brokered datasets. Brokered property records, which are ubiquitous in studies of gentrification, inequality, and the property tax in the U.S. and serve as inputs to property valuation models, are one notable example. In this paper, we audit two prominent brokered property datasets, finding errors in these data which bias key measures of economic inequality. First, we document that for 1-2% of matched sales in Cook County, IL, from 2018-2021, broker-provided sale prices differ from ground truth sale prices by more than 5%. Moreover, missing data and conceptual differences in the reporting of deed and property characteristics lead to coverage errors ranging from 12 to 15% of transactions. Second, we show that misreporting is highly consistent between brokers: more often than not, brokers make identical reporting errors for the same transactions. Third, to illustrate the significance of these errors, we measure their impact on estimates of property tax regressivity, finding that they drive significant wedges between estimates depending on the data source. These findings generalize to two other large counties in the U.S., and highlight the crucial importance of open administrative data and transparency from brokers regarding data provenance and lineage.