Architecting data-intensive applications: From architectural design to data quality

Context: Data has become the lifeblood of modern business. However, as the volume of data grows exponentially, managing it has become an increasingly daunting task. The challenge is compounded by data coming from various sources, in different formats, and at different speeds. This is where data architecture provides a roadmap for describing, collecting, storing, processing, and analyzing data to meet business needs. A well-designed data architecture provides an abstract view of data-intensive applications, making it easier to transform data into valuable information. We must take these challenges seriously and invest in robust data architecture to effectively manage and use data to our advantage. Objective: To propose a comprehensive data architecture framework to improve data quality monitoring through automated quality checks. Method: The architecture framework utilizes Model Driven Engineering (MDE) techniques. Its support for data-intensive architecture descriptions enables the automated generation of data quality checks. Result: DAT Framework offers a comprehensive solution for data-intensive applications to model their architecture efficiently and monitor the quality of their data. It automates the entire process and improves precision and consistency in data quality monitoring. With DAT, architects and analysts gain access to a tool that simplifies their workflow and empowers them to make informed decisions based on reliable data insights. Conclusion: We have evaluated the DAT in five cases within various industry domains, demonstrating its effectiveness and efficiency. The evaluation demonstrates that the DAT framework achieves around 96% modeling accuracy and reduces modeling time by 63%, enhancing efficiency. Its automated data quality validation ensures reliable and consistent monitoring in data-intensive applications.