Why You Can't Build a Lakehouse Without Spark

The Lakehouse concept is actively promoted as the "golden mean" between a Data Lake and a Data Warehouse: it promises to combine flexible data storage, advanced analytics, and transactional guarantees in a single architecture built on modern open table formats such as Iceberg (which has already become a de facto standard for building Data Lakehouses). This article examines a fundamental question: why can't you build a production-grade Lakehouse without Spark?

We discuss the role Spark plays in the Lakehouse approach, which tasks it handles better than any alternative, its close relationship with Iceberg, and why competing engines often fall short on universality, scalability, and reliability in a large production environment. We also explain why in Alphyn Lakehouse we use Spark as the engine for Iceberg table maintenance and as the primary tool for migrating data into the Lakehouse.

Why Many Engineers Dislike Spark

Nothing in the world is perfect, and Spark is no exception. What are the characteristics that regularly cause frustration in the data community? Let's go through them:

1. Complex session configuration for the environment in which queries and computations run. With engines like Impala, Trino, or StarRocks, the instance is typically already running and pre-configured. All the user needs to do is write a SQL query and get results.

   With Spark, things are different. Every application — whether a scheduled job or a Jupyter session — is a separate instance that must be spun up, and the user must configure a session for it. Questions arise: how many cores does my application need? How much memory? What should I set for spark.dynamicAllocation.maxExecutors? Wait, what is dynamicAllocation? Should I enable the Comet plugin? How much off-heap memory? The natural reaction: "Stop this — I'm an analyst. I want to analyze data and deliver business value, not wrestle with how to efficiently launch my application."

2. Speed. Spark is not the fastest engine for executing analytical queries, and there are many reasons for this.

3. Poor fit for fast, simple queries. In analytics, there are frequent situations where you need to quickly select, review, and draw conclusions from data. By the time a Spark application launches on the cluster, you could have already analyzed the query result on Impala, walked from home to the office, and verbally presented your conclusions to your manager. The root cause is the heavy startup cost of drivers and executors. Each Spark application is a separate instance requiring its own resource allocation on the cluster. While the Cluster Manager in Kubernetes decides which pods to run on which nodes… With other engines, the application instance is shared across all users and already running — just connect and start writing queries.

4. Debugging problematic applications. Spark History Server was built to help with this, and it contains a vast amount of information about Spark applications. Not just a lot — a LOT. On first visit you can get lost for hours, and truly understanding what's there, where it comes from, and what it means requires extended hands-on practice with Spark.

Why Spark Is Essential for Building a Data Lakehouse

A Lakehouse is a modern data management architecture that combines the advantages of a Data Lake and a Data Warehouse and includes business intelligence and machine learning capabilities on stored data. Its key characteristics include:

• Open table data formats (Iceberg, Delta, Hudi)

• Separation of compute and storage resources

• Support for batch and streaming workloads

• Unified data governance

To implement these principles, a reliable and scalable engine is needed. Looking at existing Data Lakehouse solutions, Spark has long been a de facto standard framework for distributed big data processing. Spark is widely used in Data Lake implementations as well as in many Lakehouse platforms. Databricks, Alphyn Lakehouse, AWS Glue, Google Dataproc, and others all use Spark as a core tool for working with data.

It's worth noting that among table formats, Iceberg has become the most widely adopted on the market. Some open table formats (OTFs) are tightly coupled to a specific vendor (hello, Delta from Databricks). Others lag behind Iceberg in performance and applicability. Additionally, Iceberg is developed by the Spark community, which means it has the most complete set of working features on this engine. Since we're talking about Iceberg, let's look at this point in more detail.

The Most Effective Iceberg Table Maintenance

Apache Iceberg is an open table data format that, alongside data files, includes a metadata writing and tracking mechanism enabling:

• ACID transactional compliance

• Schema evolution and hidden partitioning

• Accelerated table scanning (by skipping irrelevant data files)

Every action on a table (creation, data insertion, deletion, etc.) creates a new snapshot in the metadata describing which files represent the table at a specific point in time. As actions accumulate, both data files and metadata files grow in number. To achieve maximum performance, tables must be maintained in a timely and correct manner:

• Remove old, unnecessary data and its associated metadata

• Merge (compact) small data files into larger ones (optionally applying optimization techniques such as z-ordering)

• Find and remove orphan files not referenced by any snapshot

To perform these important procedures, you want maximum control with fine-grained tuning. Spark provides exactly that.

Spark lets you represent maintenance as a set of independent tasks per file group. Instead of a monolithic rewrite of the entire table, you form file groups (e.g., within partitions or by file size/count) and process them in parallel using the max-concurrent-file-group-rewrites setting. This approach offers several advantages:

• You can partition the workload — maintain only a portion of the table per run, limiting the volume of rewritten data or deleted objects

• Flexibly control parallelism to avoid overloading the Lakehouse cluster

• Improve fault tolerance through Spark retries at the individual group level (a single task failure doesn't bring down the entire maintenance job and allows safe restarts), as well as through the partial-progress.enabled configuration, which lets you commit rewrite progress incrementally — per individual file group

No other engine provides this level of fine-grained control over table maintenance.

A Reliable Lakehouse Integration Layer

Integrating data from external systems (such as Postgres, Oracle, Greenplum, Kafka, company portals, and other enterprise sources) into the Lakehouse is a fundamental and important task. A good solution must be able to work with many sources out of the box, add custom data sources from scratch, transform extracted data as needed, and guarantee reliable, repeatable loads.

It's also worth noting separately that it must be possible to work with enormous data volumes — up to terabytes. Among available engines, Spark is the most suitable solution for this task thanks to its architecture, ecosystem of existing connectors, and the ability to add new ones. Among architectural features, the fault-tolerance mechanism deserves special attention — it helps ensure the reliability of extractions and computations. We'll cover this mechanism in more detail in the next section.

Distributed data extraction from sources

For data extraction, Spark uses the standard database interaction interface — JDBC (Java Database Connectivity).

The extraction process works as follows:

1. A JDBC connection is established to the source

2. The query to the source is split into several parallel parts (when query partitioning options are configured)

3. Each Spark application executor runs its own SQL query via JDBC and extracts the results

Because the entire extraction process is broken into independent, unique queries — each executed on a separate executor — true data extraction parallelism is achieved.

Broad connector support and custom connectors

Spark supports a huge number of sources, including:

1. Relational databases (Oracle, Postgres)

2. Object storage (MinIO, AWS)

3. NoSQL stores (Cassandra, Elasticsearch)

4. Streaming sources (Kafka, Kinesis, sockets)

If you have a special source that Spark doesn't natively support, you can write a custom connector. This can be done in Java/Scala, and starting with Spark 4 — in Python as well.

Fault-Tolerant Heavy ETL

When you need not just to extract data but also to transform it further, fault tolerance becomes critically important. Every organization has critical scheduled computations that must meet a specific SLA.

Imagine a process that collects data from multiple sources over several hours and builds a data mart for critical compliance reporting (such as AML/CFT). If a task fails thirty minutes before the application completes, you very much want the application to not crash entirely — and for the restart to resume from where the failure occurred, rather than from the very beginning.

This is precisely the fault tolerance that Spark provides. Yes, it's not the fastest computation tool, but it is unquestionably the most reliable — and also the most transparent (no other engine produces as many varied logs during operation).

Fault tolerance is achieved through several features:

1. Data lineage tracking during computations
   Every DataFrame is a unique object produced by applying certain transformations to other DataFrames (or by reading data from a source, such as a file).

2. Intermediate shuffle file persistence
   If a problem occurs at some stage and the computation attempt fails, Spark doesn't need to restart the entire job from scratch. It can take shuffle files from the previous stage and restart only the last failed stage. This is especially valuable for long-running batch loads or other user computations.

3. Data spill to disk
   When Spark runs out of RAM, data begins to "spill" to disk. While spilling to disk isn't a best practice in itself, it's one of the tools for ensuring extraction reliability. In Trino, for example, any spill is disabled by default (using spill in Trino is considered bad practice, since Trino is primarily designed for fast interactive queries), and all data is stored and processed entirely in memory. If memory runs out — the extraction query fails.

Furthermore, thanks to its architecture, Spark scales its applications more predictably and safely than other engines. Each Spark application deploys as a separate, independent unit on the cluster. If our application fails for some reason — so be it. But only our application failed, not the entire service. There's no situation where a poorly written SQL query in Spark can bring down all running Spark applications on the cluster. Unlike Spark, Trino and Impala function as persistent daemons with many users running simultaneously. There's always the risk that a careless query can take down the entire service. Every query from any user executes within the shared thread and memory pool (or its portion — the resource group) belonging to these persistent processes. If one query consumes too much memory, in the best case it simply consumes all cluster resources, and in the worst case leads to OutOfMemory or an entire node freeze.

In other words, Spark provides a greater degree of isolation for individual applications and queries compared to other engines. And it's precisely this isolation, combined with fault tolerance, that allows us to build truly reliable pipelines for critical and heavy ETL processes.

Additional Advantages of Spark

Spark includes not just batch data processing but also a number of other useful components absent in other engines. For example, with Spark you can tackle machine learning tasks.

Spark MLlib was originally built for distributed data processing (just like Spark's other modules). Using the built-in ML module provides the following advantages:

• The ability to train models on extremely large datasets (hundreds of gigabytes of data)

• Data doesn't need to be transferred to a separate environment — it's stored in the Lakehouse storage, and Spark MLlib works within the same Lakehouse

• Model training is implemented as distributed computation, natively achieving parallelization

Spark ML functions are used for various transformations in standard ETL processes (StandardScaler, MinMaxScaler for feature normalization), or when there's a genuine need to train a model on a massive dataset (for example, training gradient boosting to forecast sales for all SKUs across all stores in a retail chain).

Although Spark ML has fewer features than Python ML libraries and its documentation is far from comprehensive, the ability to train models on enormous datasets is a distinctive advantage of Spark over other data processing engines — one that genuinely proves useful in day-to-day work.

Real-time data processing is a common task today. Therefore, a Data Lakehouse must be capable of handling it. Spark enables near-real-time data processing in a micro-batch manner using the Structured Streaming module. With streaming, data can be loaded from streaming sources such as Kafka or sockets. The streaming module is integrated with other components and uses a unified compute engine. This means streaming and batch data can be processed using the same APIs, reducing the complexity of developing and maintaining applications.

For example, in our Alphyn Lakehouse platform we needed to implement an audit of Spark application execution based on Spark History Server logs stored in S3. Key challenges included the dynamic log structure and non-standard events whose schema was unknown in advance and could change over time. Using Spark, already built into the platform, allowed us to quickly arrive at a working solution without introducing additional components or complicating the architecture.

Addressing Spark's Weaknesses

In the weaknesses section we highlighted slow driver/executor startup. But starting with Spark 3.4, the Spark Connect functionality appeared — a modern client-server architecture. In the standard implementation, the client (e.g., via a Jupyter notebook) connects to Spark directly within the same environment. The client consumes resources locally, and the client and cluster must have identical library versions. With Spark Connect, a thin client is created that allows the user to work with a remote Spark cluster. A logical query execution plan is built from the code, then sent in binary form to the Spark Connect Server. The server interprets the plan and executes it. Results are returned as PyArrow packets. This protocol opens significant possibilities for extending and broadening Spark usage. Beyond the obvious use cases for remote interactive analytics (for analysts or data scientists), it also enables lightweight Spark integration into various microservices.

Spark has a large community that actively develops it and continuously adds new features. For example, Spark 4.0 made a major leap forward in Spark SQL functionality and other components. New features substantially expand the user experience and enable capabilities not available in other engines. Here are just a couple of notable features from the new version:

• PIPE syntax for data queries. Writing queries this way may look unconventional for SQL-savvy users, but the feature can be useful for people who don't know SQL at all.

• Complex SQL scripting in Spark using loops and conditional execution. This capability brings Spark closer to traditional relational databases in terms of user experience. Developers can now implement complex logic in SQL, just as they would in PostgreSQL.

The full list of significant changes in Spark 4.0 is substantial and warrants a dedicated article.

Can You Build a Lakehouse Without Spark — or Only With Spark?

If you really want to, you can of course build a Data Lakehouse without Spark. But in that case you'll need to cover the tasks that Spark handles out of the box with an entire zoo of alternative solutions, since no single engine solves them all individually.

Spark undoubtedly has broad functionality and impressive distinguishing features that make it a reliable, high-performance engine essential for a Data Lakehouse. However, there are use cases where other engines perform much better than Spark.

In particular, this applies to interactive ad hoc queries whose results must be returned to the user as quickly as possible. Here, engines like Impala and StarRocks leave Spark far behind in execution speed — they were specifically designed for these tasks (in-memory data processing, pushing filter and projection operations closer to the data source, aggressive parallelization, etc.). When it comes to Trino, its ability to execute federated queries — where a single script can work with many databases simultaneously — is a clear advantage for analyst work. It's genuinely convenient to access multiple sources from one place without additional overhead.

Spark is a necessary part of Lakehouse platforms, opening the doors to reliable ETL processes, data integration, and a broad spectrum of data operations. But one should not forget: to achieve the most effective solution to diverse data tasks, it's worth taking the best from each engine. This is why in Alphyn Lakehouse we don't force users to use a specific engine for data computations. In our platform, Spark serves as the mandatory engine for Iceberg maintenance and the default engine for data ingestion — one that we continuously develop and enhance.

---

See it on your own data

If you're weighing how this would handle your workloads, we'd be glad to walk you through Alphyn Lakehouse on a real scenario. Book a sovereign-lakehouse walkthrough →

---