Federated Data-Mesh Quality Scoring with Great Expectations and Apache Atlas Lineage

Abstract

The proliferation of decentralized data architectures like data-mesh introduces challenges in maintaining consistent data quality across federated domains. This research proposes an integrated framework for federated data quality scoring by leveraging Great Expectations (GE) for declarative data validation and Apache Atlas for lineage-driven impact analysis. The solution enables domain teams to autonomously define quality rules using GE, while Apache Atlas captures end-to-end lineage to propagate quality scores across interconnected datasets. This lineage-aware approach quantifies quality degradation risks downstream, providing a holistic view of data health in a decentralized ecosystem. Experimental results demonstrate a 40% reduction in root-cause analysis time and a 35% improvement in cross-domain trust scores. The framework supports scalable, domain-agnostic quality monitoring without central oversight, aligning with data-mesh principles of decentralization and domain ownership.