Have you ever missed the Update or Delete method when working with big data? Do you ever think about how much time has been lost since deploying pipelines using cloud/on premise storage and spark for processing with partition rebuilding strategies? What about how many times you’ve had to reprocess partitions since then?
If you can relate to these situations, then here’s some great news for you: the new transaction storage layer called Delta Lake has the potential to resolve these issues.
What is Delta Lake?
Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, durability) transactions to Apache Spark™ and big data workloads. Users can download open-source Delta Lake and use it on-prem with HDFS. Users can read from any storage system that supports Apache Spark's data sources and write to Delta Lake, which stores data in Parquet format
Delta Lake offers ACID transactions via optimistic concurrency control between writes, snapshot isolation so that readers don't see garbage data while someone is writing, data versioning and rollback, and schema enforcement to better handle schema changes and deal with data type changes.
Although Delta Lake was first introduced in 2017, other companies started to develop their pipelines or companies of a certain maturity using data pipeline with open-source frameworks to start their experiments with this new feature of the open-source universe.
With the use of the Bronze, Silver and Gold layers, it is possible to establish the version control of the data. This is because historical storage of the data makes it possible to identify the data changes, audit it and roll back the data version in the past if necessary.
Key features of Delta Lake
1) ACID Transactions
Data lakes typically have multiple data pipelines reading and writing data concurrently, and data engineers have to go through a tedious process to ensure data integrity, due to the lack of transactions. Delta Lake brings ACID transactions to your data lakes. It provides serializability, the strongest level of isolation level.
2) Scalable Metadata Handling
In big data, even the metadata itself can be "big data". Delta Lake treats metadata just like data, leveraging Spark's distributed processing power to handle all its metadata. As a result, Delta Lake can handle petabyte-scale tables with billions of partitions and files at ease.
3) Time Travel (data versioning)
Delta Lake provides snapshots of data, enabling developers to access and revert to earlier versions of data for audits, rollbacks or to reproduce experiments. It's already designed with GDPR in mind - running deletes concurrently on older partitions while newer partitions are being appended.
4) Open Format
All data in Delta Lake is stored in Apache Parquet format, enabling Delta Lake to leverage the efficient compression and encoding schemes that are native to Parquet.
5) Unified Batch and Streaming Source and Sink
A table in Delta Lake is both a batch table, as well as a streaming source and sink. Streaming data ingest, batch historic backfill, and interactive queries all just work out of the box.
6) Schema Enforcement
Delta Lake provides the ability to specify your schema and enforce it. This helps ensure that the data types are correct and required columns are present, preventing bad data from causing data corruption.
7) Schema Evolution
Big data is continuously changing. Delta Lake enables us to make changes to a table schema that can be applied automatically, without the need for cumbersome DDL.
8) Audit History
Delta Lake transaction log records details about every change made to data providing a full audit trail of the changes.
9) Updates and Deletes
Delta Lake supports Scala / Java APIs to merge, update and delete datasets. This allows you to easily comply with GDPR and CCPA and also simplifies use cases like change data capture.
10) 100% Compatible with Apache Spark API
Developers can use Delta Lake with their existing data pipelines with minimal change as it is fully compatible with Spark, the commonly used big data processing engine.
Delta Lake enables developers to spend less time managing partition control. And, since the data history will be stored, it is much simpler to use the data versioning control layer to apply updates and deletions to the data and support the physical data lineage.
This blog was written by Marcus Azevedo, who worked as a data engineer at Xomnia. We’re on the lookout for data engineers with specific knowledge of Hadoop, Spark and Kafka, and the capability to architect highly scalable distributed systems, using different open-source tools - like Delta Lake. Does this sound like you? Then check out our vacancy and get in touch with us today.