Gaming the Grid: How Sampling Strategies Shape the Fairness of GeoAI
Training a geospatial AI model is like setting up a game—the rules you choose for sampling data determine who wins and who loses. In this thesis, you’ll explore how tweaking those “rules” affects the fairness and accuracy of AI models that rely on satellite or aerial imagery. What happens when you oversample urban areas? Or when you ignore socio-economic variation? By experimenting with different sampling settings, you’ll learn how easy it is to “game” the dataset—and how to catch or correct these biases. The goal is to uncover practical strategies for building GeoAI models that are not just accurate, but also fair and trustworthy across diverse regions and populations. This will be done by exploring different sampling strategies which take into account critical factors such as socio-economic diversity, spatial autocorrelation, or feature distribution and how they influence model performance and fairness in geospatial classification tasks, particularly in low-data or heterogeneous environments.
Objectives
- To investigate how different spatial and socio-economic data layers (e.g. population density, built-up areas, urban/rural classifications) can inform fair and effective sampling strategies.
- To experiment with and evaluate a range of sampling configurations (e.g. stratified by socio-economic variables, feature-space diversity, or spatial distance).
- To assess the impact of these configurations on model performance, spatial generalization, and fairness across different sub-regions.
Methodology
Literature review of sampling strategies in spatial ML, bias in geospatial datasets, and fairness in AI.
Data preparation: Identify gridded datasets that can be used to audit for model fairness (e.g. socio-economic layers depicting relative wealth, urban/rural classifications) and feature diversity (e.g. foundation model embeddings). Determine a study area and classification tasks.
Experimentation: Design an experimental framework consisting of different sampling schemes with different combinations of: socio-economic diversity, feature diversity, and feature distribution. Train a ML model (preferably a deep learning model) based on the different sampling schemes. Evaluate performance across sub-groups (e.g. rural/urban, high/low population density).
Analysis: Compare trade-offs between accuracy, spatial generalization, and fairness metrics.
Gevaert, C. M., Buunk, T. & van den Homberg, M. J. C. Auditing Geospatial Datasets for Biases: Using Global Building Datasets for Disaster Risk Management. IEEE J Sel Top Appl Earth Obs Remote Sens 17, 12579–12590 (2024). doi:10.1109/JSTARS.2024.3422503
Mitchell, M. et al. Model Cards for Model Reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229 (2018) doi:10.1145/3287560.3287596.
Suresh, H. & Guttag, J. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 17, 1-9 (2021). doi.org:10.1145/3465416.3483305