Delta Lake is an open source storage layer that brings reliability to data lakes. It provides ACID transactions, data versioning, schema enforcement and/or evolution, and unifies streaming and batch data processing. Through Azure Databricks, we benefit from managed Spark clusters & Delta Lake, as well as the ability to combine this with Azure Data Lake Store (ADLS) gen2. ADLS gen 2 offers file system semantics, directory & file level security and scale, low-cost, tiered storage and high availability/disaster recovery capabilities. Combining compute and storage technology in this way has changed the game of modern data engineering.
Gerard Wolfaardt, Principal Consultant The Data Collective
With over 20 years of experience in the data & analytics space, I’ve helped many customers architect, design and implement modern, cloud data platforms to turn data into a strategic advantage. I specialize in Microsoft’s suite of data technologies, both in Azure and on-premise. I have however worked with many other vendor products over the years. I have successfully delivered multiple solutions leveraging technologies including but not limited to; Azure Data Factory v2, Azure Databricks, Azure Data Lake Storage Gen 2, Azure SQL Data Warehouse, Azure Analysis Services, Power BI, Azure SQL Database, Azure Batch and various flavors of IaaS workloads in hybrid environments.