Data meets Compliance:
Cloud Data Architectures & GxP-Compliance

Even in regulated industries such as pharmaceuticals, biotechnology, or medtech, cloud-based platforms and data-processing systems have long become a reality. In particular, data lakes and data warehouses built on automated data pipelines enable the efficient storage, processing, integration, and provision of large volumes of data.

However, when it comes to GxP-relevant decisions, hesitation often prevails. Many life sciences companies are reluctant to use cloud-based information systems as the foundation for quality- and safety-critical processes because they fear loss of control, limited auditability, or insufficient validation capabilities. This article demonstrates how cloud technologies and forward-looking, data-driven concepts can be implemented safely and compliantly in GxP environments.

Building Blocks of Digital Transformation: Lakes, Warehouses, Pipelines…

Data lakes and data warehouses differ mainly in how data is structured, processed, and used. A data lake stores large volumes of structured, unstructured, and semi-structured data in its raw format. It is highly flexible and scalable, allowing new data sources to be integrated easily. As such, data lakes often form the foundation of modern data architectures in which information from various sources (e.g., sensors, LIMS, MES, audit data) is collected and made available.

Data warehouses, by contrast, process primarily structured data that is filtered, consolidated, and prepared for specific analyses based on a predefined schema. Any changes to that schema tend to be time-consuming and costly. However, this makes warehouse data ideally suited for traditional analytics and business intelligence. In practice, the distinction between data lakes and data warehouses is often blurred. Hybrid forms – so-called data lakehouses – are increasingly common.

Data pipelines connect the systems. These automated processing chains collect data from multiple sources, transform and clean it according to predefined rules, and load it into a data lake or data warehouse. A data pipeline typically consists of the steps extraction, transformation, and loading – abbreviated as ETL or ELT – depending on where the data is processed.

Typical cloud pipeline stacks rely on combinations of established tools and platforms. Frameworks such as Apache Spark, Databricks, or AWS Glue are frequently used, supplemented by workflow and integration tools such as Kafka, Airflow, dbt, or Fivetran. Storage generally leverages cloud-native services like Amazon S3, Azure Data Lake Storage, or Google Cloud Storage – optionally combined with platforms such as Snowflake, BigQuery, or Redshift. In the Microsoft ecosystem, Azure Data Factory often takes center stage, combined with OneLake DataLake and Fabric as an analytics solution. Power BI plays a key role as the established standard for business intelligence and, increasingly, AI applications.

Challenges in Regulated Environments

Thanks to data lakes, data warehouses, and data pipelines, growing data volumes can now be captured, stored, transferred, and analyzed almost in real time, regardless of location or system. But how can these flexible, dynamic systems be qualified and validated? Key challenges that need to be addressed include:

• Automation: Many pipelines operate without manual intervention. How can their correctness be demonstrated?
• Scalability: Systems adapt dynamically. What does this mean for infrastructure qualification approaches?
• Transparency: Data flows are complex – data originates from various sources, is transformed in multiple stages, moved across pipelines, and ultimately used in applications such as LIMS or MES. How can traceable documentation and auditability be ensured?
• High rate of change: New data sources, transformations, and infrastructure updates are constantly being introduced. How can compliance be maintained over time?

The solution lies in a risk-based, process-oriented validation approach that takes DevOps and cloud principles into account. The starting point is a risk assessment – typically in the form of a technical risk assessment. Based on the identified risk distribution, qualification and validation priorities are defined. The process begins with qualification of the underlying infrastructure, followed by validation of the specific use cases and data pipelines.

Qualifying Cloud Infrastructure Through Close Collaboration with Providers

Whether the focus lies more on data lakes or data warehousing, successful validation requires prior qualification of the infrastructure – in this case, the cloud services. This demands close cooperation between users and their cloud providers. However, providers often lack experience with regulated environments, as they are not subject to mandatory GxP regulations. In addition, many have limited understanding of life sciences scenarios, and lessons learned from other industries are not always transferable.

User organizations frequently face a dilemma: large providers offer standardized solutions that may not meet specific regulatory needs, while smaller vendors – though typically less familiar with GxP – are often more flexible and willing to develop audit-ready solutions collaboratively.

A central question is whether the platform provider can demonstrate sufficient control and transparency over its infrastructure. Yet many cloud providers do not supply all documentation necessary for qualification. Supplier assessments can help define requirements, such as the submission of additional documentation or certifications. Contractual agreements and service level agreements (SLAs) reduce compliance risks and should include escalation procedures as well as incident and problem management. Furthermore, regular, structured communication with providers is essential to ensure transparency and traceability.

One particular risk arises from platform change processes. Users typically have limited influence over new releases or modules. Traditional installation qualifications (IQs) are therefore often impractical. As an alternative, contractually defined build and release processes (CI/CD – Continuous Integration/Continuous Delivery) combined with continuous monitoring can help keep systems within validated boundaries. Timely review of release notes is a simple yet effective measure for risk mitigation.

The Validation Framework as a Strategic Key

Operating cloud-based data architectures – including data lakes, data warehouses, and data pipelines – in a GxP-compliant manner requires not only structured collaboration with cloud providers but also a robust validation framework. This framework systematically addresses all relevant topics. Its key components are:

1. A systematic, risk-based approach that

• analyzes relevant data flows and identifies GxP-relevant processing steps and systems, and
• qualifies the infrastructure while validating software and pipelines based on specific use cases.

2. Governance and Data Integrity in line with ALCOA+ principles,

• using tools such as lineage tracking, audit trails, and data catalogs,
• complemented by logging, monitoring, and alerting at the pipeline level.

3. A comprehensive test strategy for pipelines 

• automated testing of all processing steps – from extraction through transformation to data provisioning,
• simulation and control of error scenarios, and
• regression testing following code or infrastructure changes.

4. Infrastructure-as-Code and CI/CD as the foundation for technical implementation,

• providing validated deployment processes with tools such as Terraform, GitHub Actions, or Azure DevOps to ensure reproducible results, and
• versioning and approval workflows extending to configurations and metadata.

5. Supplier evaluation, including

• assessment of the cloud services used (SaaS, PaaS, IaaS),
• documentation of SLAs and support levels, and
• clear role and responsibility definitions within the shared responsibility model.

Figure: Validation framework for GxP-compliant data architecture in the cloud © msg industry advisors 

Conclusion: Validation in the Cloud Pays Off

GxP-compliant qualification and validation of cloud-based data lakes, data warehouses, and data pipelines go far beyond regulatory obligation. When implemented correctly, they enhance data quality, simplify audits and inspections, increase the reliability of data-driven decisions, and boost agility – since changes can be made transparently and traceably.

Validation is platform-independent when the architecture is well-documented, automatically testable, and effectively monitored. The key to success lies in the validation framework itself.

The technology is here – the courage to use it wisely will determine success.