Danilo S Brambila

May 17, 2020

6 min read

Delta Lake in production: a critical evaluation

Trying Delta Lake in production might reserve you quite a few surprises.

I have seen several posts and tutorials on Delta Lake using “Hello World” kind of examples, where everything works wonderfully. However, as most of you know, the performance of data processing technologies changes drastically as the amount of data that it handles increases. That’s why I decided to evaluate Delta Lake in the wild, using a real world in production Spark job that processes around 100GBs of…