I’ve made every dumb Spark mistake at least once. At production scale — real data, real concurrency, real stakeholders yelling in Slack — “it works” and “it works well” are completely different conversations. So I started writing them down.

This is the checklist I wish I had taped to my monitor when I started. Every item comes from a real production screwup — mine or someone else’s.

Before You Write a Single Line

• Use the DataFrame / Dataset API, not RDDs. RDDs are lambda-driven — Spark can’t see inside them, can’t optimize them. DataFrames go through the Catalyst optimizer. You get predicate pushdown, filter reordering, Adaptive Query Execution, and cost-based join reordering for free. The RDD API in MLlib is in maintenance mode. Let it go.

• Pick the right file format. Parquet for analytical queries — column pruning, predicate pushdown, stats-based skipping all work out of the box. Avro for schema-evolution-heavy ingestion. For anything read more than once in the same week, use a table format on top — Iceberg or Delta — so you get ACID, time travel, and stats the query planner can actually use. If you’re reading raw CSV or JSON, always specify the schema explicitly. Inference means a full scan just to figure out types.

• Use splittable compression. Snappy, LZ4, or ZSTD — never GZIP. A 10GB GZIP file can’t be split across executors – one poor node has to decompress the whole thing. Snappy is the reliable default. ZSTD is higher-compression and, as of Spark 4.x, runs in parallel for shuffle (SPARK-46256) — use it for shuffle spill and intermediate files to cut network time.

• Know your Spark version. Spark 4.0 dropped Scala 2.12, JDK 8, JDK 11, Mesos, and Python 3.8. If your platform team is still on any of those, that’s the first fight to pick — the rest of this checklist assumes you’re on JDK 17, Scala 2.13, and Python 3.9+. Don’t tune a job on a platform that’s already a stage behind.

Partitioning — Hidden Architecture

• Tune spark.sql.files.maxPartitionBytes for your storage layout. Default is 128 MB. If your Parquet files are mostly 256 MB+, you’re under-parallelizing reads. If they’re 8 MB, you’ve got too many tasks. Match the split size to your actual file distribution, not the Spark default. This is the input-side knob — it sets your initial partition count before any AQE coalescing kicks in.

• Target 2-4 partitions per available core. Fewer than that and cores sit idle. More than that and the scheduler spends more time tracking tasks than running them. If tasks routinely finish in under 100ms, your partitions are too small. If one task runs 10x longer than the others, you have skew — see the joins section.

• Filter early, filter hard. Push filters as close to the data source as possible. Partition pruning exists for a reason — if your data is partitioned by date and you only need last week, Spark should never touch the other 51 weeks. With Dynamic Partition Pruning (default-on in 4.x), this also works at runtime across joins. Which brings me to the next point.

• [4.x] Use multi-key Dynamic Partition Pruning for compound partitions. SPARK-46946 added multi-key DPP. If your fact table is partitioned by (date, region) and you join it against a small filtered dim, Spark now prunes on both keys at runtime. This unlocks real star-schema performance that was 3.x-impossible. No config needed — it just works if the dim side broadcasts.

• Coalesce after heavy filtering. You just filtered 2 billion rows down to 2 million but still have 10,000 partitions. That’s 10,000 nearly-empty tasks. .coalesce() fixes this without a full shuffle. .repartition() if you need even distribution across the new partition count. AQE’s coalescePartitions handles shuffle-output coalescing automatically, but won’t help on input-side partition count after filter.

• Repartition on join keys before multiple joins. If you’re joining the same DataFrame three times on user_id, repartition on user_id once and cache it. Otherwise you’re shuffling the same data three times. Better: if the table is long-lived, bucket it — see the joins section.

• Repartition after flatMap. flatMap can 10x your row count without touching the partition count. Now you have massively uneven partitions. Repartition explicitly or enjoy your disk spills.

• Use .partitionBy() on writes when downstream jobs filter on those columns. Limit partitioned columns to low-cardinality (≤ a few hundred distinct values). .partitionBy("user_id") on a user table is a disaster — millions of tiny directories. .partitionBy("date") on daily data is the canonical right answer.

Memory

• Know your memory layout. Default: 60% of executor memory for execution + storage (spark.memory.fraction), split 50/50 (spark.memory.storageFraction). The remaining 40% is user memory. Don’t blindly increase executor memory — understand which pool is running out first. The Spark UI’s Storage tab tells you what’s cached; the Executors tab tells you what’s in use.

• For PySpark: bump memoryOverhead to 20–25%. Default is 10% or 384 MB. Arrow and pandas UDFs allocate native memory that doesn’t show up in JVM metrics. Your executor gets killed by YARN or Kubernetes and you have no idea why. This is why.

• Don’t collect on the driver. df.collect() pulls the entire dataset to a single machine. Use .take(), .takeSample(), or .show(). Same goes for .countByKey(), .countByValue(), .collectAsMap() — all driver-side. Spark logs a warning when task serialized size exceeds 1 MB (TASK_SIZE_TO_WARN_KIB = 1000); past spark.driver.maxResultSize (1 GB default), it errors out.

• Watch for disk spills. Check the Spark UI’s Stages tab. If you see “Spill (Memory)” or “Spill (Disk)” on a stage: reduce data per partition (more partitions), increase executor memory, or both. Spills mean your data didn’t fit and Spark wrote intermediate results to disk. That’s 10-100x slower than in-memory.

• Use off-heap memory for big shuffles. Set spark.memory.offHeap.enabled=true and spark.memory.offHeap.size for jobs doing heavy shuffles or joins. Off-heap bypasses GC pressure and is more predictable. But you pay for it with a fixed allocation — but worth it for anything that spills regularly.

Caching — It’s Not Free

• Only cache what you reuse. Caching a DataFrame you touch once just wastes memory that could go to shuffles and joins. Cache when you have branching logic, iterative ML workloads, or multiple actions on the same data.

• Force materialization after caching. .cache() is lazy. Until you trigger an action, nothing is cached. Always follow with .count() or a full action. Otherwise you think you cached it, but you didn’t, and your next job re-computes everything.

• Use MEMORY_AND_DISK as your default storage level. Pure MEMORY_ONLY means data gets evicted silently when memory fills up. MEMORY_AND_DISK spills to disk instead of recomputing. That’s almost always what you want.

• Assume your cache will be evicted. Cache competes with execution memory. Spark uses LRU eviction and won’t warn you when it drops your cached blocks. Design your job so it’s correct even without the cache — cache is for speed, not correctness.

• Beware partial caching. .cache() followed by .take(10) only materializes the partitions Spark touched to get 10 rows. The rest is uncached, and the next action recomputes them without warning. Always cache with a .count() or full action first.

Joins — The Biggest Shuffle of Your Life

• Broadcast small tables. If one side of your join is under spark.sql.autoBroadcastJoinThreshold (10 MB default), Spark broadcasts it — no shuffle, no exchange, just a hash lookup on every executor. For medium dims (10-200 MB), consider raising the threshold or using .broadcast(df) explicitly. Past 200 MB, broadcasting costs more than it saves.

• Diagnose skew before you “fix” joins. Open the Spark UI, go to the Stages tab, look at the task duration distribution. If 99 tasks finish in 2 seconds and 1 takes 40 minutes, you have skew. AQE’s skewJoin handling (default-on in 4.x) covers most cases automatically. If it doesn’t trigger: broadcast the small side, salt the join key, or do an iterative broadcast join.

• [4.x] Use Storage Partition Join for pre-partitioned tables. If both sides of a join come from a DSv2 source (Iceberg, Delta) and are partitioned on the same columns, SPJ (SPARK-51938 improvements in 4.x) skips the shuffle entirely. Set spark.sql.sources.v2.bucketing.enabled=true. This is the single biggest shuffle-elimination feature in Spark 4.x and it’s criminally underused.

• Bucket your tables when SPJ isn’t an option. For Hive-style writes, pre-bucket both tables on the join key with the same bucket count. Spark skips the shuffle on subsequent joins. Older pattern than SPJ but still relevant for non-DSv2 sources.

• Order joins smallest-first when AQE isn’t picking up slack. AQE reorders most joins automatically based on runtime stats, but if you’re outside AQE’s reach (e.g. RDD path, or a shuffle-heavy plan that AQE can’t retry), put the smallest table first in your explicit join chain.

JDBC Sources

• Set numPartitions for parallel reads. Default JDBC reads load everything into a single partition on a single executor. Set .option("numPartitions", N) with .partitionColumn(), .lowerBound(), and .upperBound() to parallelize. This can be the difference between a 2-hour read and a 5-minute read.

• Use predicates for non-numeric partitioning. If your partition key isn’t a clean numeric range, pass an array of SQL WHERE clauses via .option("predicates", ...). One task per predicate, hand-sized ranges. Ugly but effective.

• Push down what you can, don’t trust the driver to do it for you. Spark pushes basic filters (column equality, IN, IS NULL) to JDBC automatically, but anything with a computed column or cast won’t push and will get evaluated in Spark after the full read. For complex predicates, write the filter explicitly in a query option instead of relying on pushdown.

What Spark 4.x Actually Changed

These items used to require config flags. In 4.x they’re on by default. If you’re copy-pasting old configs forward, some are now redundant and a couple are quietly doing the opposite of what you think.

• [4.x] AQE is default-on. Stop toggling it. spark.sql.adaptive.enabled=true is the default since 3.2. If you’re copy-pasting configs that explicitly set it, trim them. What to care about instead: adaptive.coalescePartitions.parallelismFirst (default false — set true if you want AQE to prioritize parallelism over partition size), and the skew join thresholds if your data is unusual.

• [4.x] DPP is default-on, and now multi-key. Same story — stop toggling dynamicPartitionPruning.enabled. The upgrade that matters in 4.x is SPARK-46946: DPP now broadcasts multiple keys, so joins against compound-partition fact tables prune at runtime.

• [4.x] RocksDB is the default shuffle service DB backend (SPARK-45351). If you run external shuffle service with a database backend, this changed under you. Usually a win, but worth checking your ESS metrics after upgrade.

• [4.x] spark.shuffle.service.removeShuffle is default-on (SPARK-47448). Shuffle data is cleaned up automatically when referenced RDDs are GC’d. Your “cluster disk fills up after long jobs” problem from 3.x is probably gone. If it isn’t, check your lineage — something is holding references.

• [4.x] Parallel ZSTD/LZF for shuffle compression. SPARK-46256 and SPARK-48518. If you’re still on default Snappy for shuffle compression, you’re leaving CPU parallelism unused on modern multi-core executors. Set spark.shuffle.compress=true (default) and spark.io.compression.codec=zstd.

• Kryo vs Java serializer — still worth it. Default is still Java, which is 2-10x slower and larger on the wire. spark.serializer=org.apache.spark.serializer.KryoSerializer. You pay this cost on every shuffle. Register your custom classes (spark.kryo.classesToRegister or spark.kryo.registrator) or Kryo silently falls back to Java for unregistered types.

Before You Ship to Prod

• Actually read the Spark UI. Look at the DAG, the stage timeline, the task distribution. Most performance problems are visible in the UI if you bother to look. Uneven task bars = skew. Lots of stages = unnecessary shuffles. Red bars in the stage view = spills. The SQL tab shows you the physical plan with runtime stats — this is where AQE’s decisions become visible.

• Monitor first, tune second. Don’t guess. Don’t pre-optimize. Run the job, look at the metrics, then adjust. Bumping executor memory to 64 GB “just in case” is overprovisioning. You’ll keep paying for it every day until someone audits the bill.

• Use .localCheckpoint() to break lineage. Long chains of transformations build massive execution plans. A checkpoint before a repartition-and-write breaks the plan into manageable stages and can prevent stack overflows on deeply nested DAGs. Also the only way to truncate lineage cheaply when you don’t have reliable distributed storage.

• Turn on Prometheus metrics. spark.ui.prometheus.enabled=true is default-on in 4.x (SPARK-46886). Scrape the executor endpoints into whatever observability stack you have. If you don’t have metrics on executor memory pressure and shuffle throughput, you’re tuning blind.

• Set a job timeout. spark.task.reaper.killTimeout and a driver-level cutoff. Without one, a runaway job will burn cluster cost over the weekend before anyone notices.

TLDR;

If you only remember five things:

1. Use the DataFrame API. Everything else in this list depends on it.

2. Filter and partition-prune as early as possible. Don’t compute on rows you’re going to throw away.

3. Find your skew before you tune anything else. One slow task in 200 is the whole problem 80% of the time.

4. AQE, DPP, SPJ are default-on in 4.x. Know what they do so you stop double-configuring them and start noticing when they don’t fire.

5. Read the UI. The answer is almost always in the UI if you look.

Want this as a printable PDF? Subscribe at luminousmen.substack.com and I’ll send it — plus a new data engineering deep-dive every week.

luminousmen

Helping robots conquer the earth and trying not to increase entropy. Deep-dives on data engineering, career, and industry rants from the big-tech trenches. Author of Grokking Concurrency

Most Spark engineers use caching without actually knowing how it works. Not because they’re lazy, but because Spark makes it look simple. Under the hood, caching is tightly coupled to how Spark handles plans, memory, partitions, and block-level execution. Miss one of those details, and Spark will ignore your cache, drop it, or recompute the entire DAG from scratch without saying a word.

This post isn’t about what .cache() does. It’s about what Spark actually does when you call it, and all the reasons your cache might not behave the way you think.

Recap

Before we touch caching, let’s remind ourselves how Spark thinks.

At its core, Spark builds everything on top of RDDs (Resilient Distributed Datasets) , which are immutable collections of records split across your cluster. Think of RDDs as the assembly language of Spark: everything higher-level (like DataFrames or Datasets) eventually compiles down to RDDs and their operations.

When you write a Spark job, you’re either transforming data (via transformations like map, filter, join) or triggering actions (count, take, collect). Transformations are lazy — they don’t do anything when you define them. Instead, Spark just builds a DAG of steps it could run, like a to-do list for later. It’s only when you hit an action that Spark wakes up and starts actually executing the DAG to produce a result.

But once that action is done, Spark throws away the intermediate results. Poof — gone from memory. If you later run another action on the same logic — say, you do .count() a filtered DataFrame and then .show() it — Spark will re-run the entire DAG from the original source. That might mean fetching a terabyte of data from S3, reading from Kafka, or doing expensive decompression on disk. None of this is free, and if you repeat that logic in multiple places — like in two different branches of your pipeline — you’re basically signing up to recompute everything from scratch multiple times, burning time and cluster resources.

This is where caching enters the picture. Or rather — should. But as we’ll see next, it’s not always as straightforward as calling .cache() and calling it a day.

Caching in Spark

This might sound obvious, but it’s a common source of confusion for folks expecting Spark to “remember” results just because the logic hasn’t changed. It will not. Every time you call an action on an uncached DataFrame or RDD, Spark starts from the source and rebuilds all intermediate steps — even if you already did the exact same thing 5 seconds ago (according to the 5-second rule).

To prevent this waste, Spark offers two tools: .cache() and .persist(). Both let you tell Spark that you are probably going to need this data again in the future. The difference is in how and where the data gets stored.

.cache() is the friendly one-liner. Internally, it’s just shorthand for .persist(StorageLevel.MEMORY_AND_DISK), which means Spark will try to keep the data in RAM. But if that fills up, it’ll spill evicted blocks to disk. You get some performance benefits without risking immediate OutOfMemory errors.

If you need more control, you use .persist() and pick a StorageLevel. There’s a whole zoo of these, and they’re worth understanding — especially when you’re tuning long-running pipelines or squeezing performance out of memory-starved clusters.

At the simple end, you’ve got MEMORY_ONLY, which stores raw, unserialized objects in RAM — fast, but risky. If memory runs out, Spark just evicts blocks, and you’re on your own. Then there’s MEMORY_AND_DISK, which gives you a safety net by backing up evicted blocks to disk. You can also go with the serialized variants (_SER), which store data as byte arrays instead of full JVM objects — reducing memory usage and garbage collection pressure, but increasing CPU cost due to (de)serialization.

💡 You can also add a _2 suffix to any storage level (like MEMORY_AND_DISK_2) to enable replication — useful for fault-tolerant workloads where you’d rather not recompute partitions from scratch if a node goes down.

For cold-storage-style caching, there’s DISK_ONLY, which doesn’t even try to use memory. And if you’re dealing with native code or need to avoid the JVM heap entirely, there’s OFF_HEAP, which pushes data out of the garbage collector’s reach altogether.

💡DISK_ONLY and OFF_HEAP always serialize, whether you like it or not.

Now here’s what this looks like in practice:

df = spark.read.parquet(data_path)
df.cache()  # marks for caching (lazy)
df.count()  # materializes the cache
df.is_cached  # True
df.count()  # uses cached data
df.storageLevel  # StorageLevel(True, True, False, True, 1)
df.unpersist()  # manually clears it

.cache() doesn’t do anything until you trigger an action. It just marks the plan as cacheable. Only when Spark actually executes a full pass over the data — for example, during a .count() action — will it start caching partitions as they’re computed.

💡 Once you persist an dataframe with a given storage level, you can’t change it. You’d have to unpersist and re-persist it.

You can do the same with temporary views, which is sometimes cleaner when working with Spark SQL-style APIs:

df.createOrReplaceTempView("df")

spark.catalog.cacheTable("df")
spark.catalog.isCached("df")  # True
spark.catalog.uncacheTable("df")
spark.catalog.clearCache()

So, does caching make things faster?

The answer, of course, is: sometimes.

When used correctly, caching can be a massive win — especially in iterative jobs (like ML workflows), interactive analysis, or when you’re branching logic and reusing intermediate results. But misuse it — say, by caching everything just because it “seems safe” — and you’ll tank memory, increase garbage collector (GC) time, and possibly make things worse.

The trick isn’t just to use caching. It’s to know when it helps and what is actually being cached. Spark will happily let you believe something is cached, and then silently recompute everything anyway if you break its assumptions. We’ll get to that next.

Caching Behavior

So you’ve called .cache(), maybe even .persist(), and you’re expecting Spark to behave like a good little compute engine and reuse your precious data instead of recomputing it. Seems simple, right? Just stash the data and read it back later.

Well… not quite.

Let’s talk about what actually happens under the hood — starting with the thing that controls all of this: CacheManager.

When you mark a DataFrame to be cached, what Spark really does is modify the logical plan of your query by wrapping it in a special logical operator called InMemoryRelation. This isn’t some background process — it’s part of Spark’s planning pipeline. Specifically, this wrapping happens after the analyzer phase, but before optimization. In other words, the cache is tied to the analyzed logical plan, not the final optimized one. And this is where things get tricky.

Let’s walk through a real example:

df = spark.read.parquet(path)
df.select("col1", "col2").filter("col2" > 0).cache()
df.filter("col2" > 0).select("col1", "col2")

Now, if you’ve worked with Spark optimizations before, you’re probably thinking: “Hey, those two queries are semantically the same. The optimizer will push down the filter either way. The final execution plan should be identical”.

And you’d be right — but that doesn’t matter.

Because again: caching happens before optimization. The second query has a different analyzed plan than the first, even if the optimizer will eventually produce the same physical plan. Spark will look at the analyzed plan of your current query, compare it to the analyzed plan that was cached, and if they don’t match exactly — it will skip the cache and recompute everything.

This is why you can “cache something” and then later run what looks like the same query — and Spark still redoes the work. And unless you know this, it’s incredibly easy to shoot yourself in the foot (or other sensitive areas).

Block-Level Reality

So far, we’ve been talking about caching at the level of DataFrames, queries, logical plans — the kind of stuff you see in code. But underneath all that, Spark doesn’t think in terms of “DataFrames”. It thinks in blocks. And understanding how block-level caching works will save you from some very expensive surprises.

Every RDD (and by extension, every DataFrame) is made up of partitions. Each partition maps to a block, and these blocks are the actual units of caching in Spark.

Let’s say you run a .cache() followed by a .count(). Spark will compute every partition, and as each partition is computed on a given executor, the resulting block is stored by that executor’s local BlockManager. That BlockManager decides where and how to store the block — in memory, on disk, serialized, off-heap — depending on the StorageLevel you set.

But the important part is this: caching happens locally and incrementally.

There’s no centralized controller orchestrating “cache this entire dataset”. Each executor only caches the blocks it computes. And it only caches them when they’re computed. No data is prefetched or broadcasted unless you explicitly tell Spark to do so.

This has two major implications:

First, if a block is computed on Executor A, then cached, and Executor A goes down before you reuse that block — Spark will have to recompute it. Unless you used a replicated storage level like MEMORY_AND_DISK_2, the data’s just gone. This is why block replication exists — not for performance, but for fault tolerance.

Second, the only blocks that get cached are the ones Spark was forced to touch. If your action only scans a subset of the dataset — think .take(10), .limit(100), or any filtered scan — then only those blocks will be materialized and cached. The rest? Still lazily waiting. So unless you run an action that touches every partition (like .count() or a full write), you’re working with a partial cache at best.

Even worse, Spark won’t warn you about any of this. There’s no built-in visibility into how much of your DataFrame is actually cached, or how much has already been evicted. You’re flying blind unless you hook into metrics yourself or use third-party observability tools.

💡 This is why experienced Spark users follow .cache() with a full action like .count() — not because they’re curious about how many rows there are, but because they want to force Spark to scan all partitions and trigger caching across the whole dataset.

Eviction

Now, let’s talk memory. The BlockManager doesn’t get to use the whole heap. Spark allocates a chunk of the JVM memory for execution and storage via spark.memory.fraction, and then further splits that chunk between caching and temporary execution data using spark.memory.storageFraction. We talked about that in the previous post so check it out.

And that means one thing: you’re always fighting for memory.

That means your cache competes directly with joins, aggregations, and shuffles for memory. And Spark will steal from the cache if an active task needs more execution memory. Spark will happily start evicting cached blocks to make room. Your nicely cached block you spent 5 minutes reading that data from S3? Gone. Evicted mid-job. No warning.

Spark uses a Least Recently Used (LRU) eviction strategy. If memory gets tight, it starts dumping the oldest or least-used blocks to make room. If you’re using a disk-backed storage level (like MEMORY_AND_DISK), Spark will spill evicted blocks to disk. If you’re not — say, you used MEMORY_ONLY — those blocks are just gone. And Spark will recompute them later if needed.

To make this worse, Spark doesn’t wait until your next action to evict things — blocks can be kicked out of memory before you ever reuse them, especially on busy clusters or long-running jobs. So just because you called .cache() doesn’t mean your data is still around.

In the best-case scenario, if you’re using something like MEMORY_AND_DISK_SER, Spark will spill those evicted blocks to disk, and pull them back in later. You’ll pay the disk I/O cost, but avoid recomputing from source.

In the worst case — say, you used MEMORY_ONLY — those blocks are gone for good. The next time you need them, Spark has to go all the way back to the source and rebuild the full DAG.

Here’s the kind of warning you might see when that happens:

WARN org.apache.spark.sql.execution.datasources.SharedInMemoryCache: 
Evicting cached table partition metadata from memory due to size constraints 
(spark.sql.hive.filesourcePartitionFileCacheSize = 262144000 bytes).

Translation: something else needed the memory more than you did, and your cached data got sacrificed.

So if you’re running a big job on a small cluster, or working with long pipelines where memory pressure fluctuates — don’t assume your cache is still intact. Spark gives you no guarantees. It’s a best-effort system. Useful? Absolutely. Predictable? Only if you know what’s happening behind the curtain.

To Wrap It Up

Caching is one of those Spark features that everyone uses, few people understand, and most overuse. It’s easy to reach for it — because at a glance, it feels like a no-brainer. Why wouldn’t you want to avoid recomputation? Why wouldn’t you want your pipeline to go faster?

But caching isn’t about calling .cache() and moving on. It’s about systems thinking — knowing what actually happens under the hood when you ask a cluster to keep data around for you. You have to know how it works. You have to test under pressure. You have to assume the cache will disappear at the worst possible time and make sure your system handles that gracefully. If caching is part of your performance strategy, make sure it’s backed by either disk or replication — and design your jobs assuming eviction will happen. Because sooner or later, it will.

Additional materials

• Spark: The Definitive Guide by Bill Chambers, Matei Zaharia

• Learning Spark by Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee

Apache Spark is a distributed computing engine. Its main feature is the ability to perform computations in memory.

That’s what the brochure says.

And while that’s technically true — Spark can crunch terabytes in-memory across a cluster of machines — what that brochure is not saying is that the exact same job that ran fine yesterday can blow up with an OutOfMemoryError today, and you won’t even know what changed.

Welcome to the wild west of memory management in Spark!

The concept of memory management is quite complex. Python has its arenas and pools, C has malloc/free hell, and Java has garbage collectors and metaphysical tuning flags. Spark adds another abstraction layer on top of all that, and most engineers — myself included (back when I was young and hopeful) — treat it like magic and hope for the best.

For better use of Spark and to achieve high performance with it, however, a deep understanding of its memory management model is important.

Hell yeah, we’re diving into Apache Spark memory management!

High-level Overview

The memory management system in Spark is built on layers of abstraction stacked on top of each other like a bad lasagna.

Here’s the rough breakdown of Spark’s memory management layers that you should keep in the back of your head:

• OS level — your Linux kernel doesn’t care about Spark. It just sees processes with memory requests.

• JVM level — Java heap, off-heap, metaspace, GC overhead… the classics.

• Cluster manager level — YARN, Kubernetes, or Mesos enforces resource limits.

• Spark application level — where things get weird.

We’re not going to deep dive into all of them, but you need to know that these layers exist because Spark memory errors often come from the boundaries between them, not just from the application layer

💡 For example: you can allocate a giant heap in Spark, and still get killed by YARN for exceeding container memory.

Let’s rewind for a second. What actually happens when you launch a Spark app?

When you submit a Spark job, the cluster spins up two kinds of processes — Drivers and Executors:

The Driver is the control tower. It creates the SparkContext, parses your code, splits it into stages, schedules tasks, tracks metrics, fails things, retries things … you get the idea. The Driver is just a regular JVM process (usually with way too little memory), and its memory model isn’t special — so we’ll mostly ignore it here.

Executors are the actual workers. They run your tasks, spill your data, and eat your memory. Each executor runs as a JVM inside a container (YARN, Kubernetes, etc), and this JVM is where Spark does both storage and compute — shuffles, caches, broadcasts, UDFs, Arrow memory, native code, GC, off-heap buffers… all fighting for the same scraps of RAM.

Spark tries to manage all of it with some hardcoded fractions and a few memory pools. It’s amazing that everything works at all despite that. Let’s go deeper into the Executor memory.

Executor Container

When you submit a Spark job in a cluster — whether it’s YARN, Kubernetes, or something weird and/or custom — Spark spins up Executor containers on worker nodes. These are the JVMs that will actually do the work: running your tasks, spilling data, caching results, and shuffling data back and forth across the network.

Now, let’s talk memory. Who’s in charge of giving it out?

Well, that depends on the cluster manager.

YARN

YARN has two components that control resource allocation:

ResourceManager — the cluster-level boss, it decides how much memory you’re allowed to ask for based on yarn.scheduler.maximum-allocation-mb. If you ask for more than that, you get nothing or worse, a silent cap and unexpected crashes.

On a single node, it is done by NodeManager. It manages memory allocation on each individual node, based on yarn.nodemanager.resource.memory-mb, which sets the upper physical memory per node.

Kubernetes

On Kubernetes, Spark uses the pod model:

• Each executor is a pod.

• Pod memory is controlled via resource requests/limits, which map to container-level cgroups.

• If your process (heap + off-heap + Python + native) exceeds the pod limit, Kubernetes won’t hesitate — OOMKilled.

So, when Spark talks about an Executor, it’s really talking about:

A single JVM process running inside a container (YARN or Kubernetes pod), allocated a fixed amount of memory.

That container is carved up into three main memory areas:

• Heap memory — the JVM heap, controlled via --executor-memory.

• Off-heap memory — for native buffers like Arrow or Tungsten, outside the JVM.

• Memory overhead — for JVM internals, Python workers, and native memory.

We’ll walk through each of these in the following sections.

Dive Into The Heap

This is the big one — the memory most people think of. This is the memory size specified by --executor-memory during submission of the Spark application or by setting spark.executor.memory.

Internally, this becomes your JVM heap size (-Xmx). Spark uses this memory to hold most of your cached data, shuffle buffers, task results, UDF outputs, etc and yes, it’s subject to garbage collector (GC), which means you’ll spend a lot of time debugging mysterious GC pauses if you’re not careful.

Spoiler: it’s not all available for your code. Spark slices and dices it further.

To understand this madness, we’ll need to look into Spark’s MemoryManager (source: org/apache/spark/memory). That’s the interface Spark uses to track and control heap usage.

UnifiedMemoryManager is the default MemoryManager in Spark since 1.6 (StaticMemoryManager has been deleted in 3.0). UnifiedMemoryManager manages the executor heap in three main regions, plus a hard-reserved chunk that Spark needs for internal bookkeeping.

Let’s break this down.

Reserved Memory

Before Spark gives your code anything, it quietly grabs 300MB from the executor heap just for itself — logs, metrics, internal structures, and whatever else it thinks is important.

This is hardcoded (RESERVED_SYSTEM_MEMORY_BYTES in the source) and not configurable in production. You can only tweak it in tests via spark.testing.reservedMemory, but that’s irrelevant for real workloads.

Usable memory = Executor memory - 300MB

Everything else — execution, storage, user memory — comes out of this usable heap.

User Memory

This is the wild west of Spark memory. No tracking, no boundaries, no safety rails. Spark calls it, “User Memory”, but it’s really just a chunk of heap left over for:

• UDFs doing who-knows-what

• Python wrappers and runtime glue code

• Native ML libs like XGBoost allocating their own buffers

• Arrow buffers (which Spark ignores, but GC doesn’t)

It calculated as:

User memory = Usable memory * (1 - spark.memory.fraction)

By default, spark.memory.fraction = 0.6 — so User Memory = 40%

Spark does not manage this memory. It doesn’t track what you put here. If you go overboard here, you’ll get GC pressure or random OOMs. Spark won’t even warn you, it just dies.

Execution & Storage Memory

After Spark subtracts Reserved and User Memory, the remaining 60% of heap becomes a shared pool for:

• Execution Memory — used for shuffle, joins, aggregations, sorts.

• Storage Memory — used for caching, broadcasts, and unrolled RDDs.

They’re split like this:

Execution memory =
  Usable memory
  * spark.memory.fraction
  * (1 - spark.memory.storageFraction)

Storage memory =
  Usable memory
  * spark.memory.fraction
  * spark.memory.storageFraction

By default:

• spark.memory.fraction = 0.6

• spark.memory.storageFraction = 0.5

Which gives you:

• Execution = 30% of total heap

• Storage = 30% of total heap

Execution and Storage live in a shared memory pool. They can borrow memory from each other. This is what Spark calls -

Dynamic Occupancy Mechanism

With UnifiedMemoryManager, execution and storage don’t have a hard wall between them. Their sizes are elastic, but not equal.

The creators of this mechanism decided that Execution memory always has priority over Storage memory. They had reasons to do so — the execution of the task is generally more important than the cached data, and the whole job can crash if there is an OOM in the execution.

So the rules are simple:

• If Execution needs memory, it takes it.

• If Storage is using that space (cached RDDs, broadcasts), Spark starts evicting blocks.

• If Execution is idle, Storage can grow into that space — until Execution comes back.

Internally, Spark tracks the maximum Storage usage with a threshold called onHeapStorageRegionSize. By default, it’s equal to the full Storage region. The moment Execution asks for memory though, Spark shrinks that region and pushes Storage out of the way.

This leads to an important consequence. Your cache is not safe.

If you persist something with .persist(StorageLevel.MEMORY_ONLY) and later run a wide transformation — shuffle, join, aggregation — Spark may silently evict cached blocks to feed Execution memory. And yes, you’ll pay the cost of recomputation later. We will talk about that in detail in the next blog post.

💡 AQE shifts the failure mode. Adaptive Query Execution has been on by default since Spark 3.2 spark.sql.adaptive.enabled=true). It coalesces small post-shuffle partitions and splits skewed ones at runtime — so the classic “one task OOMs on a billion-row key” used to be the failure mode, and now it’s “AQE flips a sort-merge into a broadcast join you didn’t expect, and Storage memory eats it”. If you tuned spark.memory.storageFraction upward in Spark 2.x to give caching more room, revisit it. AQE may have shifted the balance under your feet. Partition-side details in Spark Tips: Partition Tuning.

💡 Everything in Spark is physically grouped and stored as blocks — serialized chunks of data. These can live:

• in memory (on-heap or off-heap),

• on disk,

• or on remote executors (for broadcast).

Blocks are managed by BlockManager, which acts as a distributed key-value store. BlockManager works as a local cache that runs on every node of the Spark application, i.e. driver and executors. It handles:

• Caching

• Shuffle inputs/outputs

• Broadcasts

• Temporary files

This block abstraction allows Spark to move, cache, and evict data efficiently. And more importantly: multiple executors can fetch the same block concurrently, boosting throughput.

But blocks aren’t small. And their memory footprint isn’t always predictable. So caching a large dataset means betting on your eviction policy not screwing you over.

Off-heap Memory

Most Spark jobs live happily inside the JVM heap, running under the watchful eye of the GC. But Spark also has the ability to allocate memory off the heap, bypassing the JVM entirely.

Why would anyone want to do that?

Simple: GC can be the problem. Especially when you’re working with:

• Giant datasets

• Millions of tiny objects

• Low-latency jobs

• Python UDFs backed by Arrow

• Or you’re just tired of tuning G1GC with 18 flags

GC is expensive. When you’re dealing with large datasets, the overhead of object tracking, allocation, and collection inside the JVM can kill performance. Spark’s answer is to use off-heap memory to store raw data buffers and skip the JVM’s babysitting.

Under the hood, Spark does this using sun.misc.Unsafe, a low-level Java API that lets you manually allocate and free memory — kind of like malloc() in C. It’s fast. It’s risky. It’s entirely unmanaged. Which is why it’s usually disabled unless you know what you’re doing. (On JDK 17+ this routes through jdk.internal.misc.Unsafe, which is why some Spark builds need --add-opens flags at startup).

To enable it, you’d set:

spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=1g

Now your executors will be running with two memory pools: the traditional JVM heap, and a chunk of raw memory Spark can directly write into — no GC involved. But don’t get too excited: YARN (or Kubernetes) doesn’t care where you allocate memory. If your total memory usage (heap + off-heap + overhead) exceeds the container limit, the executor gets killed. Period.

But! Off-heap memory isn’t just some exotic optimization. It’s actually core to how modern Spark works, thanks to a project you might have heard of ...

Tungsten

Tungsten is Spark’s long-running effort to optimize physical execution with a heavy focus on memory. The idea is simple: Spark jobs shouldn’t pay the tax of Java objects. They’re heavy, they fragment the heap, and they make GC unpredictable. So Tungsten introduced:

• Custom encoders/decoders for rows (for tight binary layout)

• Explicit memory management

• CPU cache-aware execution

• Vectorized processing

• And yes — off-heap memory

Now, when you turn on off-heap memory, you’re telling Tungsten to go full native. It will allocate all its memory pages directly off-heap. That gives you better GC performance and tighter control over memory — but it also means you’re now responsible for not blowing up your container limits. Spark won’t warn you and GC won’t save you. Dramatic pause.

💡 Even when running in on-heap mode, Tungsten still tries to minimize object allocation. It uses byte arrays and memory pages to hold data in a compact format, allocating large chunks of memory only when needed. This is why Spark’s performance jumped massively around versions 1.6–2.0 — it stopped creating millions of tiny Java objects and started acting more like a proper database engine.

💡 If you’re on Databricks, Photon (their vectorized C++ engine) lives entirely off-heap — your spark.executor.memory mostly does bookkeeping while the real work happens in native land. The catch: Photon’s allocator sits in memoryOverhead, not heap. The default 10% overhead is almost never enough — Databricks K8s runtimes default spark.executor.memoryOverheadFactor to 0.40 for exactly this reason. Open-source equivalent: Apache Gluten (Velox/ClickHouse backends). Same advice: bump overhead.

Furthermore, container managers like YARN or Kubernetes don’t know how much off-heap memory you’re using. They monitor process-level memory (RSS), not JVM heap. So if you go crazy with spark.memory.offHeap.size and forget to increase spark.executor.memoryOverhead, your job will be randomly killed with a vague container memory error.

Welcome to production.

💡 Quick tip: If you’re using Arrow (e.g. in Pandas UDFs), you’re already in off-heap territory, even if you never explicitly enabled it. Arrow allocates native memory behind the scenes, and if your executor’s memoryOverhead is too low — 💥.

Off-heap memory is conceptually simpler than on-heap: there’s no user memory, no GC, and Spark only divides it into Execution and Storage, just like it does for the heap. In UnifiedMemoryManager, the same borrowing rules apply — Execution has priority, Storage gets evicted.

But don’t let that simplicity fool you. The second you turn it on, you’re on your own. No GC means no safety net. Which is great — until it’s not.

Overhead Memory

Let’s talk about the part of memory no one thinks about — until their executors start randomly dying.

When you set --executor-memory, you’re only specifying the JVM heap size. But Spark executors also need memory for everything outside the heap — thread stacks, JIT buffers, metaspace, JNI, Python processes, native libs, and whatever else the JVM decides to hoard.

That’s what Overhead Memory is for.

By default, Spark sets this via spark.executor.memoryOverhead. If you don’t specify it, Spark will default to 10% of executor memory, or 384 MB, whichever is greater.

Since Spark 3.3, you can also use spark.executor.memoryOverheadFactor instead of an absolute size — 0.10 by default on YARN, 0.40 on Kubernetes. It scales with executor size, so set it once and forget it. The deprecated spark.yarn.executor.memoryOverhead was removed in Spark 3.0 — if you still have it lying around in old configs, delete it.

Now, you might be thinking: “Fine, that’s just some buffer zone. I don’t need to touch it.” Not true, lemme give you an example.

Let’s say you set --executor-memory=8G. Spark will default to 10% overhead, so:

spark.executor.memoryOverhead = max(0.1 * 8192MB, 384MB) = 819MB

Total memory requested from YARN or K8s:

Executor memory + Memory overhead = 8192MB + 819MB = 9011MB

Okay, not too bad.

But now, you enable off-heap memory:

spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=1g

Guess what? That off-heap 1GB is not included in executor memory or overhead. So now your actual memory usage is:

8192 MB (heap)
+ 819 MB (default overhead)
+ 1024 MB (off-heap)
= 10,035 MB

But your container still only got 9011 MB, because Spark didn’t include off-heap in the overhead by default. It will be killed by YARN or OOMKilled by the K8s kubelet. Not a Spark error. No nice logs. Just dead.

PySpark makes it worse

And if you’re using PySpark, it’s even worse.

PySpark spins up a separate Python process per task, and it uses memory outside the JVM. If you don’t explicitly allocate space for it with spark.executor.pyspark.memory.

...then Spark will stuff it inside the overhead memory too. Which means now that 819MB has to cover:

• JVM internals (metaspace, thread stacks, JIT)

• Off-heap allocations

• Arrow buffers

• Python workers

That won’t end well.

So here’s how Spark actually calculates container memory:

val totalMemMiB =
    executorMemoryMiB
  + memoryOverheadMiB
  + memoryOffHeapMiB
  + pysparkMemToUseMiB

If you’re running on YARN, that’s the amount YARN allocates. If you’re on Kubernetes, this becomes your pod memory limit.

💡 If your executors are randomly getting killed — especially with vague YARN “container killed” or K8s OOMKilled — don’t just throw more memory at it, check these first:

• Are you using PySpark, Pandas UDFs / Arrow, GPU libraries?

• Did you enable off-heap memory?

• Did you bump spark.executor.memoryOverhead accordingly?

If the answer to any of these is “no” — that’s probably why you’re getting OOMed.

Real clusters often use 20–25% of executor memory as overhead for PySpark workloads. Monitor first, then adjust

TaskMemoryManager

By now, we know that Spark’s MemoryManager slices executor memory into regions — execution, storage, user, and so on. But what happens when multiple tasks are running on the same executor and they all want a slice of execution memory?

They fight over it, and Spark plays referee using a component called TaskMemoryManager.

Tasks don’t talk to MemoryManager directly. Instead, each task works with its own TaskMemoryManager instance, which acts as a middleman for memory allocation — both for on-heap and off-heap memory. It keeps track of how much memory each task has used, how much it’s allowed to request, and whether it needs to wait or fail.

Why does this matter?

Because multiple tasks in the same executor share the execution memory pool. And there’s no hard isolation between them. That means:

• Task A can show up early and grab a big chunk of memory.

• Task B shows up a few milliseconds later and finds the fridge empty.

• Task B either blocks, spills to disk, or in worst cases, just dies.

So how does Spark try to make this fair?

Spark tries to be “fair-ish” with memory by enforcing soft limits. The logic looks like this: if n tasks are running concurrently on an executor, each task is allowed to allocate between 1/2n and 1/n of the total execution memory.

That range is fuzzy by design. Spark starts by allowing each task up to 1/n of the memory. But if multiple tasks start at different times, the first one gets a little more breathing room (up to 1/2n for others). In practice, the first task often grabs more than its fair share, especially in uneven workloads or poorly partitioned data.

It’s not a strict cap either — Spark doesn’t stop a task immediately if it goes over. It just starts blocking other tasks, triggers spills, or (worst case) throws an OOM.

TaskMemoryManager doesn’t prevent these issues — it just tries to contain the blast radius. And that’s where tuning comes in:

• You can reduce spark.task.cpus to run fewer concurrent tasks per executor.

• Or allocate more spark.executor.memory to give the execution pool some breathing room.

• Or go distributed and shrink partition sizes — fewer records per task = less memory pressure.

But if you ignore this, your “optimized” job will randomly choke on 1 out of 200 stages — and then you’ll be back here, re-reading this post with tears in your eyes.

Conclusion

Today, with Spark 3.x and beyond, memory is more dynamic than ever — but that doesn’t mean it’s safer. Adaptive execution, columnar formats, off-heap buffers, native UDFs — all of it still shares the same physical RAM. If you don’t plan for that, you will get burned.

If you’re tired of manually tuning configs, tools like Spark Configuration Optimizer can give you a decent starting point. But remember: no tool will save you if you don’t understand the system it’s tuning.

Spark is not magic. It just looks like it — right up until the OOM.

Additional materials

• Apache Spark codebase

• Apache Spark documentation

• Spark: The Definitive Guide by Bill Chambers, Matei Zaharia

• Learning Spark by Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee

• High Performance Spark by Holden Karau, Adi Polak, Rachel Warren

P.S. I think this also is a cool infographic to see the full picture of Spark memory management:

Most engineers, even experienced ones, treat partitioning as a default — something that Spark, “just handles”. And when it breaks, it doesn’t throw a clean error. It just gets … slow. Or starts spilling. Or OOMs, or hits the network harder than it should. Or, it burns 5x the compute for the same workload.

Partitioning isn’t just a Spark concept, it’s fundamental to data engineering in general. If you’re using Spark, Flink, Dask, or even writing your own distributed system — how you split the work always matters.

If you’re building data pipelines in Spark and you’re not controlling your partitions, you’re not in control of your performance. Spark is.

Let’s fix that.

A Reminder on Spark and Partitions

Let’s rewind for a second.

You may know this already, but Apache Spark is a distributed data processing engine. It lets you run computations across a cluster in parallel, splitting up work so you can chew through terabytes in minutes — not hours. But the real power here isn’t just Spark’s distributed magic, it’s how Spark structures distributed work.

When you launch a Spark job, you’re building a DAG — a Directed Acyclic Graph of transformations. Transformations such as filter(), join(), groupBy(), and write(). Spark compiles this logical graph into a physical plan. That plan gets broken down into stages, which are then sliced into tasks.

No matter how complex your logic is, at the bottom of the stack, Spark is just executing tasks, and each task works on one partition. Period.

Once your job starts running, each partition gets assigned to an executor. Executors run tasks one at a time. So if a task gets a huge partition, it’ll hog that executor for a while. If your partitions are too tiny, your cluster gets flooded with overhead. If they’re too big, a few slow tasks end up dragging down the whole job — causing memory pressure, long runtimes, or worse.

💡 Spark partitions have nothing to do with Hive partitions. Hive partitions are about how files are laid out on disk. Spark partitions are about how data is cut up in memory during execution. Same word, totally different game.

That means your partition count directly controls how much parallelism Spark can squeeze out of your cluster.

Spark redefines partitions multiple times throughout the job. Read from files? That’s one way of partitioning. Shuffle after a .groupBy()? That’s another.

So let’s break it down.

Input partitions

As we already know, when you initially read data into Spark, the engine quietly breaks it up into partitions. A partition is a chunk of the dataset that Spark processes as a single unit of parallelism. Each task works on one partition, and together those partitions make up the full dataset. This is the bottom layer of Spark’s partitioning story.

Now, how Spark does this depends on a mix of file format and config values. And yeah, it’s a bit of a circus.

For structured formats like Parquet, ORC, or Avro, Spark actually reads the metadata — footers, row groups, that kind of thing — and tries to slice the file in a way that makes sense. Often you’ll see one partition per row group, though Spark may merge or split depending on file sizes and configs. Makes sense, right?

For plain-text formats like CSV, JSON, or raw text, Spark partitions data based on file size and the spark.sql.files.maxPartitionBytes setting (128MB by default). It won’t split a row in half, but it does chunk files using that size threshold. JSON can be tricky: if it’s multi-line JSON, each file becomes a single partition (since it’s not splittable). But line-delimited JSON is splittable.

If you’re dealing with storage backends like HDFS or S3, you get even more layers of complexity. Block size starts playing a role (fs.blocksize), which you can’t always control, especially if you’re on managed cloud infra. So now your partitions are dictated by whatever the storage layer felt like doing that day.

Let me throw a couple examples at you. Say you’re reading a single Parquet file — 30GB total, and it’s got 300 row groups. Spark will split that into exactly 300 partitions. Predictable.

Now imagine you’re reading 100 tiny JSON files, each around 2MB. Spark’s going to turn that into over a thousand partitions. Why? Because it won’t split those files — one file, one partition, every time. Congratulations, you just bought yourself some overhead.

And then there’s Kafka and Cassandra, which are different beasts entirely. Partitioning here follows their own logic: Kafka will give you one partition per topic-partition, Cassandra will do it based on token ranges.

💡 For example if you have, say topic: user_events with 12 Kafka partitions. Cool, now you connect Spark to this topic. You get... 12 Spark tasks, one per Kafka partition. Doesn’t matter if your Spark job has 100 executors or 500 cores available. You’re reading from 12 partitions? You’re doing 12 parallel reads.

None of that matches how you probably want to process the data in Spark.

So, how Spark handles input partitioning depends on the data source. And if you don’t control this layer, you might end up with 3 partitions for your entire dataset — or 10,000. Either way, don’t expect efficient CPU usage.

Shuffle partitions

Now let’s talk about that glorious moment in Spark when everything slows down, fans spin up, and your cluster suddenly turns into a toaster. That moment is called the shuffle.

Shuffles happen during wide transformations — things like groupBy(), join(), distinct(), or orderBy(). Basically, whenever Spark needs to move data across executors so that rows with the same key land on the same node. It can’t just power through — it has to reshuffle everything. That’s when new partitions are born.

And guess what the default is? 200 shuffle partitions.

Why? Because someone, somewhere at Spark HQ, once set the default:

spark.sql.shuffle.partitions = 200

It doesn’t matter if you’re dealing with 20MB or 500GB — Spark will happily slice your post-shuffle data into exactly 200 chunks unless you tell it otherwise. Hilarious.

If your data is tiny, say a few megabytes, then those 200 partitions will each get like ten rows. Tasks become microscopic. Most of your CPUs will just be sitting there doing nothing, wasting cluster hours while you pretend it’s “distributed computing”.

On the other hand, when you’re working with hundreds of gigabytes, and still only have 200 shuffle partitions it causes fewer tasks at a time to be processed in executors, but it increases the load on each individual executor and often leads to memory errors.

Also, if you increase the size of the partition larger than the available memory in an executor, you will get disk spills. Spills are the slowest thing you can probably be able to do. Essentially, during disk spills Spark operations place part of the data into a disk if it does not fit in memory, allowing the Spark job to run well on any size of dataset. But, even though it won’t break your pipeline, it still makes it super inefficient because of the additional overhead of disk I/O and increased garbage collection.

💡 Sure, AQE (Adaptive Query Execution) helps a bit. It can adjust the number of shuffle partitions after Spark starts executing — but it’s not magic. And it won’t save you if you never tuned your config in the first place. We will talk about that a bit later.

Output partitions

So you’re done with all your joins and filters. Now, it’s time to finally write the results somewhere. But what happens when you write? Spark always writes one file per output partition. Period.

Got 200 partitions? You’re getting 200 files.

Got 3000? Congratulations, you just wrote 3000 files.

Even if half of those files contain like, three rows — Spark doesn’t care. It’ll dump them all anyway.

You think you just saved your dataset. In reality, you might have just killed your S3 bucket. Or your BigQuery load job. Or your downstream Spark job that now has to read ten thousand tiny files scattered like confetti across cloud storage.

So yeah — writing is not the end of the pipeline. It’s part of the pipeline.

How to Actually Control Them

Spark defaults to 200 shuffle partitions. Not because that’s a good number — but because it had to pick something. Because Spark has no clue what your data looks like. It doesn’t know that your customer_id is heavily skewed. It doesn’t know your files are a mess. It doesn’t know that your filters drop 95% of the rows, or that one of your columns is 99% null and turns into a bomb when you group by it.

All it sees is your code.

You write groupBy(”country”), and Spark nods along: “HashPartitioner. 200 partitions. Carry on.” Except 90% of your data is US, and now one task is hauling a boulder while the rest twiddle their thumbs. That’s not parallelism. That’s just dumb.

And that’s why you have to take control.

Shuffle partitions

spark.sql.shuffle.partitions, this little config controls how many partitions Spark creates during wide stages — join(), groupBy(), aggregations. If you don’t touch it, Spark sticks with 200. Doesn’t matter if you’re joining 5MB or 5TB.

You can crank it up or down with:

spark.conf.set(”spark.sql.shuffle.partitions”, “600”)

How do you pick the right number? There’s no magic formula, but if your tasks are processing 3GB each and spilling all over the place, you probably need more partitions. If they’re running for 2 seconds each and writing tiny files, probably less. Somewhere around 100–200MB per task tends to work well — but again, you know your data. Spark doesn’t.

Repartition

This is Spark’s “start over” button. It forces a full-on shuffle and redistributes the data however you tell it to — by number of partitions, or by key. It’s a full-blown network operation, with serialization, temp files, and a whole lot of I/O. You’re throwing the data into the air and letting Spark reshuffle the deck.

It’s expensive. It’s also sometimes necessary.

For example, after a heavy filter.

Let’s say you start with a massive dataset — Spark gives you 2000 partitions. Then you apply a filter and keep 5% of the rows. Spark doesn’t care. It keeps the same 2000 partitions, now mostly empty. You end up with thousands of tiny tasks doing nothing, or thousands of micro-files dumped into S3.

Fix it:

val cleaned = df.filter(...).repartition(100)

Now you have real partitions again — enough to parallelize, not enough to waste resources.

Another popular example, when you’re sending data downstream — to a model trainer, a multi-node write, a batch load.

If your partitioning is uneven, downstream parallelism collapses. One node gets 10GB, others get peanuts. Spark can’t fix this for you. If you want balance, you have to force it:

val prepared = df.repartition(200)

In both cases, repartition() gives you control where Spark won’t. You’re paying for predictability — and that’s usually a fair trade.

Coalesce

At the other end of the scale is coalesce().

Where repartition() is a full reshuffle, coalesce() is a quiet merge. It takes existing partitions and fuses them — no data movement, no shuffle. It’s almost always used at the end of a pipeline, right before writing to disk.

Say Spark left you with 3000 partitions. You write that to S3, and congrats — now you’ve got 3000 tiny files. Great if your goal is to make downstream analysts hate you.

So you do this:

df.coalesce(100).write.parquet(”s3://bucket/output/”)

Simple. Now you’ve got 100 reasonably sized files. S3 thanks you. Presto thanks you. Your cloud bill thanks you.

But don’t push it too far. People see coalesce(1) and think, “Nice — one output file”. Except now you’ve killed parallelism — one executor is doing all the work. The rest are idle.

Even worse, coalesce() doesn’t rebalance data. It just stacks partitions on top of each other. So if your original partitions were uneven, they still are — just lumped together.

There is a trick though:

df.coalesce(50, shuffle=True)

This acts more like a repartition, but leaning toward reduction. It’s not free — you pay for the shuffle cost — but you get better distribution and fewer partitions.

Custom Partitioning

Now, there’s also partitionBy() — but don’t confuse it with actual Spark partitions. This one is about physical layout on disk. It’s what decides how your data gets saved into folders.

If you write:

df.write.partitionBy(”region”).parquet(...)

Spark splits the output into folders like region=US/, region=EU/, and so on. But this has nothing to do with memory partitions or parallelism. If you want those files to be cleanly separated — one file per region — you still need to do:

df.repartition(”region”).write.partitionBy(”region”).parquet(...)

Otherwise, you’ll get a mess: dozens of tiny files per region, randomly scattered. Which defeats the whole point.

What about Adaptive Query Execution?

Since Spark 3.0, there’s been this thing called Adaptive Query Execution (AQE) — a smarter layer on top of the regular SQL planner. It can change the physical plan while your query is running, based on what’s actually happening.

Sounds amazing, right?

It is — up to a point.

Well, it can be. But you still have to know how it works, and when it doesn’t.

AQE can merge tiny shuffle partitions, split skewed ones, and even switch join strategies on the fly. It’s Spark learning as it goes. For open-source Spark ≤3.3, you need to turn it on explicitly; for newer versions and most Databricks environments, it’s already enabled:

spark.conf.set(”spark.sql.adaptive.enabled”, “true”)
spark.conf.set(”spark.sql.adaptive.coalescePartitions.enabled”, “true”)
spark.conf.set(”spark.sql.adaptive.skewJoin.enabled”, “true”)
spark.conf.set(”spark.sql.adaptive.shuffle.targetPostShuffleInputSize”, “64MB”)

Once enabled, Spark starts adapting at runtime — not during planning, but while the job is running.

You filter 2TB down to 50MB, and instead of blindly running 200 tasks and writing 5000 files, Spark notices and gives you 3 tasks. You group by a skewed key and Spark slices that one overloaded partition into smaller pieces so your stage doesn’t hang for an hour. It’s the first time Spark feels like it’s paying attention.

But AQE only kicks in after the first shuffle. It won’t help you with garbage input partitioning, bad file layouts, or anything you’ve explicitly told Spark to do. If your source data has wrong stats — say, compressed JSON or Kafka streams — AQE might even make things worse. And since it adjusts plans at runtime, explain() might show you one thing, and the Spark UI might do something else entirely.

So no, AQE isn’t magic. It’s a smart assistant, not a mind reader.
It helps polish your plan — it doesn’t design it for you. You still need good partitioning fundamentals. AQE just makes the edges smoother.

Wrapping it up

• Input partitions: Spark guesses how to split files based on format, size, and config. Sometimes it gets it right. Often it doesn’t. Especially with JSON, Kafka, and weird file layouts. You need to know what’s happening under the hood.

• Shuffle partitions: 200 is not a sacred number, it’s a default. And like all defaults, it’s wrong 80% of the time.

• Output partitions: Every partition = one output file. You write 3000 partitions? You get 3000 files. Coalesce or repartition before writing.

• Repartition(): Use it when your data got filtered/skewed and you want a fresh, even split. It’s expensive but sometimes necessary.

• Coalesce(): Use it when you want fewer files, and you trust your data is already nicely distributed. Use it before writes. Don’t overdo it.

• partitionBy(): Controls how data is written to disk. Not how it’s processed in memory. Don’t confuse the two.

• AQE: It’s smart. Smarter than the old planner. But it’s not a mind reader. Think of it like auto-correct — helpful, but you still need to know how to spell.

Materials

• Apache Spark docs

• Spark: The Definitive Guide by Bill Chambers, Matei Zaharia

• Learning Spark by Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee

Where are we?

Let's pause and take stock of where we are in this series.

In the first post we talked about the playground Spark runs on — clusters, resource managers - all that infrastructure you need before a single line of code moves data.

In the second post we talked about how Spark thinks — it's core abstractions like RDDs (Resiliant Distributed Dataset) , DataFrames, transformations, and actions. Basically, the theory.

Now we finally get to the fun part: what actually happens when your job runs: Spark job's life cycle. This is where all that abstract stuff suddenly becomes very real. Tasks start flying, executors spin up, and data shuffles across the cluster.

Spark Application Architecture Essentials

If you strip Spark down to its moving parts, it's basically a distributed to-do‑do list with a coordinator. Your code says, "I want to crunch this giant pile of data", and Spark figures out who does what, where, and when.

If you've ever stared at the Spark UI wondering why stage 2 has 400 tasks and stage 3 has 12, these are the components that are making that happen:

• Spark Driver — the brain of the operation. It turns your nice, innocent map and reduceByKey calls into a DAG (Directed Acyclic Graph, if you don't recall) of tasks, sends them to the cluster, and keeps tabs on who's still alive.

• Executors — the workers. They're JVM processes living on the cluster nodes, running tasks in parallel, caching RDDs, and shuffling data around. Spark scales because you can have hundreds of these chewing through partitions at the same time.

• Cluster Manager / Resource Manager — the landlord that owns the machines (the land of the cluster). Popular ones are YARN, Kubernetes, or Spark's standalone manager.

• Spark Context — the entry point to all of Spark's features. It's the Spark Driver's (see below) hotline to the cluster, letting it request executors, create RDDs, and manage shared variables. You only get one per JVM.

Each of these pieces is part of the pipeline that turns your "just run df.groupBy().count()" code into a swarm of distributed tasks.

Let's dive deeper into these components to understand their roles.

Spark Driver

The Driver is where a Spark application truly begins. Once the Driver spins up and takes charge, everything that happens in the cluster is dictated by this single process.

In the hierarchy of a Spark application (application → job → stage → task), the Driver is at the top — it manages the entire process from the top down in real time, orchestrating how work flows down from jobs to individual tasks on executors. Everything else in Spark exists to serve the plan that the Driver creates.

As your code runs and you define transformations, the Driver incrementally assembles them into a logical DAG of operations. Spark is lazy — these transformations only describe how to compute the data, not execute it. No actual computation happens until you call an action like collect or count. At that point, the Driver turns the logical DAG into a physical execution plan and starts running tasks on executors.

Execution isn't a single step. The Driver cuts the DAG into stages wherever a wide dependency appears. Each stage is then broken into tasks, one per data partition. These tasks are the actual work units that will eventually land on executors somewhere in the cluster.

Once the tasks exist, the Driver takes on the role of a scheduler. It talks to the Cluster Resource Manager to request resources, hands batches of tasks to available executors, and keeps a close eye on their progress. If an executor dies, the Driver knows which tasks were running there and can resubmit them elsewhere. This constant awareness is what allows Spark to tolerate node failures without losing your entire job.

At the same time, the Driver is the historian of the application. It keeps lineage information for RDDs and partitions, maintains the logical-to-physical mapping of operations, and exposes this through the Spark UI.

💡 If you've ever stared at a "Jobs" tab watching stages light up in green or red, that's the Driver narrating the story of your computation in real time.

Everything in your application — jobs, stages, and tasks — flows downward from the Driver. It is the single point from which high‑level logic becomes executable reality, quietly controlling the chaos of a distributed system.

In material terms, the Driver is just a JVM process. Where the Driver lives depends on how you launch the job. In client mode, it stays on your laptop or submission machine, orchestrating everything remotely. In cluster mode, it moves into the cluster itself, running on a dedicated node. In both cases, its existence is non‑negotiable — kill the Driver, and the application is gone. There's no Spark without the Driver.

How can the Driver can do so much? The Driver itself consists of several managers. Let's describe them one by one.

DAGScheduler

If the Driver is the brain, the DAGScheduler is the part that thinks in big pictures. It doesn't care about CPUs, threads, or executors. Instead, it stares at your logical plan — the DAG of RDD transformations — and figures out how to break it into stages.

One stage can run as long as its input partitions are available locally. The DAGScheduler's job is to translate logical operations into stage boundaries. It looks at your RDD lineage, identifies wide dependencies like groupByKey or join, and decides where data shuffles must occur. Every shuffle creates a stage boundary because data has to move between nodes.

From there, the DAGScheduler breaks each stage into a set of tasks literally called TaskSets, one per partition, and sends them to the TaskScheduler. But it doesn't just generate tasks — it also tracks their lineage. This lineage allows Spark to recompute lost partitions in case of failure instead of starting your job from scratch.

DAGScheduler is also the one who decides job success or failure. If a stage can't make progress after repeated attempts — say, due to corrupted input data — it's the DAGScheduler that ultimately declares the job to have failed.

It is important to note that, the DAGScheduler makes dynamic decisions. It doesn't pre-schedule all stages at once. Instead, it reacts to stage completions and unlocks child stages on-the-fly.

TaskScheduler

Once the DAGScheduler has carved the DAG into stages and handed off a TaskSet for the next stage, the TaskScheduler. This is the part of Spark that stops thinking in lineage and starts thinking in terms of machines and slots.

The TaskScheduler doesn't care about RDDs or shuffles. Its job is mechanical:

• Take each TaskSet from the DAGScheduler

• Work with the SchedulerBackend to request executors and figure out where tasks can run

• Assign tasks to executors, trying to keep them close to their data (see below) whenever possible.

• Retry failed tasks when necessary

If an executor dies or a task throws an exception, the TaskScheduler resubmits the task to another executor, respecting Spark's task locality preferences — keeping tasks close to where the data resides whenever possible.

This handoff — DAGScheduler to TaskScheduler — is what converts a logical plan into a physical reality running across the cluster. Where the DAGScheduler thinks in stages, the TaskScheduler works in terms of executors and available CPU cores — the slots where tasks will run. It speaks the language of threads, resources, and task placement.

💡 A slot is not a class, API, or config setting. It's just a term for a single task execution thread on an executor. If an executor has 4 cores, it has 4 slots — meaning it can run 4 tasks in parallel. Slots are Spark's way of describing how many tasks can run concurrently across the cluster. More cores = more slots = more parallelism. It's that simple.

All that being said, TaskScheduler itself does not launch tasks. It hands them to the SchedulerBackend.

SchedulerBackend

Underneath the TaskScheduler lives the SchedulerBackend. The SchedulerBackend is the messenger between Spark and the cluster manager. It doesn't understand DAGs or stages — it just deals in resource requests and executor lifecycle management.

Its responsibilities include:

• Talking to the cluster manager to request new executors

• Launching and killing executors as needed

• Reporting executor availability back to the TaskScheduler

When you see Spark dynamically scale executors up and down during a job, that's the SchedulerBackend negotiating with the cluster manager behind the scenes.

BlockManagerMaster

While above schedulers focus on when and where tasks run, the BlockManager is all about the data those tasks need.

Spark processes everything as partitions, and each partition becomes a block stored either in memory or on disk. And every RDD partition, every shuffle file, every broadcast variable goes through the BlockManager. When an executor needs a block of data, the BlockManager either hands it over locally or fetches it from a neighbor.

💡 You might be wondering, are we talking about a shuffle operation here? Mostly, yes — this is what happens during a shuffle read. When a wide transformation like `reduceByKey` runs, tasks often need to fetch intermediate data (shuffle blocks) produced by other tasks on different executors. The BlockManager handles these lookups and transfers. But we're not just talking about shuffle here — the same mechanism is also used when tasks fetch cached RDDs or broadcast variables from remote nodes. If data isn't local, the BlockManager knows where to find it, and how to pull it across the network.

Every executor has its own BlockManager, but the Driver hosts the BlockManagerMaster, which keeps a global map of all block locations across the cluster. Whenever a task needs a block, the BlockManager can serve it directly if it's local, or fetch it from a remote executor if necessary. This is the mechanism that makes shuffles, caching, and recomputation possible.

BlockManager handles:

• Storage and retrieval of blocks in memory or on disk

• Replication for fault tolerance, so losing one executor doesn't mean losing the only copy of a partition

• Data exchange between executors during shuffles or when fetching cached partitions

It also powers Spark's caching and persistence. When you call cache or persist, the BlockManager decides whether the block stays in memory, spills to disk, or is evicted to make room for new data. This constant juggling is what keeps future stages fast without running out of memory.

If a BlockManager disappears, Spark may need to recompute lost partitions using RDD lineage or fetch replicas from surviving executors. Without this system, Spark would have no awareness of where its data lives, and every shuffle or cached computation would fall apart. Sadly.

Together, these four components turn the Driver from a passive coordinator into a full blown distributed engine. And when they all do their jobs well, you never have to think about them — you just see your Spark job finish.

Spark Context

The Spark Context serves as the primary entry point for all Spark operations, acting as a bridge between the Spark Driver and the cluster's resources. Through the Spark Context, the Driver communicates with the Cluster Resource Manager, requests executors, and coordinates the execution of distributed computations. It's also the mechanism for creating RDDs, managing shared variables and tracking the status of executors via regular heartbeat messages.

Every Spark application operates with its own dedicated Spark Context, instantiated by the Driver when the application is submitted. This context remains active throughout the application's lifecycle, serving as the glue that holds together the distributed components. Once the application completes, the context is terminated, releasing the associated resources.

💡 A critical limitation of Spark Context is that only one active context is allowed per JVM. If you need to initialize a new Spark Context, you must explicitly call `stop` on the existing one. That way we have only one resource management entity which prevents conflicts within the single application.

The modern way to start a Spark application is through a SparkSession, which wraps around the lower-level SparkContext. While the SparkSession gives you access to DataFrames, SQL, and config management in a friendly API, it's the SparkContext underneath that actually drives the core engine — requesting executors, creating RDDs, and coordinating cluster resources. You can think of SparkSession as the front desk, and SparkContext as the operations center.

Cluster Resource Manager

In a distributed Spark application, the Cluster Resource Manager (or Resource Manager for short) is the component responsible for allocating compute resources across the cluster. Depending on your deployment, this could be YARN, Kubernetes, or Spark's own standalone cluster manager.

When a Spark application starts, the Driver requests executor resources via the SchedulerBackend, which then communicates with the Cluster Resource Manager. Based on available capacity and scheduling policies, the cluster manager launches executor processes on worker nodes — in containers, pods, or raw processes, depending on the setup.

Once executors are up, they register back with the Driver, and the Driver begins assigning tasks to them. From that point on, the cluster manager's job is mostly done — it keeps the executors running and watches for node failures, while Spark handles the actual data processing.

The SparkContext is the Driver's interface to the cluster manager. It abstracts the differences between backends and lets your application scale across environments without changing your code.

Executors

Executors are the backbone of a Spark application, running on worker nodes to execute tasks assigned by the Driver. These processes handle the actual data processing and data storage.

By default, executors are allocated statically, meaning their number remains fixed for the duration of the application. However, Spark also supports dynamic allocation, where executors can be added or removed to adapt to workload changes. While this flexibility can optimize resource utilization, it may affect other running applications and introduce contention with other applications sharing the same cluster.

In addition to task execution, executors manage data storage. They use the BlockManager to store intermediate RDD data, which can be cached in-memory for quick access or spilled to disk when memory is insufficient (using localCheckpoint).

Spark Application Running Steps

Read more

If you've spent any time wrangling data at scale, you've probably heard of Apache Spark. Maybe you've even cursed at it once or twice — don't worry, you're in good company.

Spark has become the go-to framework for big data processing, and for good reason: it's fast, versatile, and (once you get the hang of it) surprisingly elegant. But mastering it? That's a whole other story. Spark is packed with features and an architecture that feel simple on the surface but quickly grow complex. If you've ever struggled with long shuffles, weird partitioning issues, or mysterious memory errors, you know exactly what I mean.

This article is the second in a series on Apache Spark, created to help you get past the basics and into the real nuts and bolts of how it works — and how to make it work for you.

Apache Spark Key Abstractions

At its core, Spark is built on two fundamental abstractions:

1. Resilient Distributed Dataset (RDD)

2. Directed Acyclic Graph (DAG)

These are the gears under the hood of everything Spark does. Let's unpack each one.

RDD — The Core of Spark

The beating heart of Apache Spark is the Resilient Distributed Dataset (RDD) — a collection of objects spread across cluster nodes for parallel processing. RDDs were Spark's original abstraction and still remain central to its design, even as DataFrames and Datasets have taken center stage.

💡 While RDDs are still there under the hood — especially when it comes to low-level fault-tolerant mechanics — you shouldn't write modern Spark code directly with RDDs. Unless you're doing something super low-level (like custom serialization or working with graphs), stick with DataFrames or Datasets. New releases of Apache Spark pushes even harder on DataFrames and Datasets.

Physically, an RDD lives as an object in the JVM, pointing to data from external sources (HDFS, S3, Cassandra, etc). Every RDD carries its own metadata to enable fault tolerance and distributed execution. Key components include:

• Partitions — chunks of data distributed across the cluster nodes. One partition = one unit of parallelism.

• Dependencies — lineage information, a list of parent RDDs and transformation history, forming a lineage graph. This lets Spark recompute lost data.

• Computation — the computation function applied to parent RDDs.

• Preferred Locations — hints where partitions are stored, enabling data-local execution.

• Partitioner — defines how data is split into partitions (like default HashPartitioner or RangePartitioner, etc).

RDDs are immutable — you can never change an RDD in place, you simply derived from an existing RDD using transformation. That new RDD contains a pointer to the parent RDD and Spark keeps track of all the dependencies between these RDDs via lineage graph. In case of data loss, Spark replays the lineage to regenerate it from the original data source making the pipelines fault tolerant.

Here's a quick example:

>>> from pyspark import SparkContext
>>> sc = SparkContext.getOrCreate() # sc is the SparkContext, the entry point to Spark
>>> rdd = sc.parallelize(range(20)) # Create RDD with numbers from 0 to 19
>>> rdd
ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:195
>>> rdd.collect() # Collects all data from nodes and returns it to the driver node
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

💡 The Spark Driver orchestrates Spark jobs, manages tasks, and collects results from executors (like with collect()). For a deeper content, check out the next blog post.

RDDs can be cached to optimize repeated computations. This is huge when your pipeline hits the same RDD multiple times. Without caching, Spark must traverse the lineage graph to recompute RDDs from scratch every time they used in a computation.

You can control partitioning manually to balance workloads or rely on Spark's defaults (typically based on cluster configuration). More, smaller partitions often mean better resource utilization (though there's a point of diminishing returns).

Let's inspect the number of partitions and their contents:

>>> rdd.getNumPartitions() # Returns the number of partitions (default behavior depends on cluster settings)
4
>>> rdd.glom().collect() # Collects partitioned data as lists back to the driver node (this is Spark's standard behavior)
[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]]

All operations on RDDs fall into two categories:

• Transformations (lazy)

• Actions (eager)

Transformations

Transformations are the backbone of RDD processing in Spark. They're how you take a dataset and do something meaningful with it — filter it, map over it, join it, whatever. But there's a catch: transformations are lazy.

When you call a transformation (e.g. map or filter), Spark doesn't immediately execute it. It just takes notes — "Okay, when someone asks for the result, I'll know what to do". And actual execution only kicks in when you trigger an action like collect or count. This approach — known as lazy evaluation — lets Spark analyze your entire workflow, optimize execution plans, and minimize data movement before doing any heavy lifting.

⚠️ Side effects in transformations? Bad idea. Since transformations are lazy, they might get evaluated multiple times or in weird orders. If you're relying on side effects (like writing to a file in map()), you're setting yourself up for trouble. Just don't.

But not all transformations are created equal. Spark draws a very important line in the sand.

Narrow vs Wide Transformations

• Narrow transformations (e.g. map, filter, flatMap) don't require moving data between partitions. Each partition works with its own local data, independently, making these operations fast and highly parallelizable (aka embarrassingly parallel).

• Wide transformations (e.g. groupByKey, reduceByKey, join) require shuffling — a fancy way of saying Spark needs to move data between executors to regroup things to meet the transformation's requirements. These are slow, expensive, and should be treated with suspicion unless absolutely necessary.

💡Note on shuffling (aka data shuffle)

A shuffle happens when Spark has to move data between partitions across the cluster — usually to regroup, sort, or aggregate it. This is required when the output of a transformation depends on records from multiple partitions — for example, summing all values or grouping records by some key.

During a shuffle, Spark performs a full-on data exchange: it reads from all partitions, redistributes data according to some partitioning logic, and writes new partitions — often on different executors. That's expensive, as it involves network I/O, data serialization, and disk writes.

Minimizing unnecessary shuffling is key to optimizing Spark jobs.

Narrow transformations operate entirely within a single partition, avoiding the overhead of data shuffling.

For example:

>>> filteredRDD = rdd.filter(lambda x: x > 10) # Filter elements greater than 10
>>> print(filteredRDD.toDebugString()) # Print out the lineage/DAG
(4) PythonRDD[1] at RDD at PythonRDD.scala:53 []
|  ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:195 []
>>> filteredRDD.collect() # Only local partition filtering, no shuffle
[11, 12, 13, 14, 15, 16, 17, 18, 19]

There is no shuffle here — Spark doesn't move any data — each partition filters its own elements. That's what makes narrow transformations cheap and parallelizable. In fact, Spark often fuses a chain of narrow transformations into a single block (a stage), optimizing them behind the scenes.

Wide transformations by contrast involve dependencies across multiple partitions, requiring Spark to shuffle data around. For example:

>>> groupedRDD = filteredRDD.groupBy(lambda x: x % 2) # Group data based on mod
>>> print(groupedRDD.toDebugString()) # Triggers shuffle, costly but necessary for grouping
(4) PythonRDD[6] at RDD at PythonRDD.scala:53 []
|  MapPartitionsRDD[5] at mapPartitions at PythonRDD.scala:133 []
|  ShuffledRDD[4] at partitionBy at NativeMethodAccessorImpl.java:0 []
+-(4) PairwiseRDD[3] at groupBy at <ipython-input-5-a92aa13dcb83>:1 []
|  PythonRDD[2] at groupBy at <ipython-input-5-a92aa13dcb83>:1 []
|  ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:195 []

Here, Spark had to shuffle data based on the grouping key (in this case, odd/even values). Notice the ShuffledRDD? That's your performance tax right there.

💡 So when do you have to use wide transformations? If you're grouping, aggregating, or joining across partition boundaries or your logic inherently needs a global view of the dataset. That's fine — just be intentional about it. Know when you're paying the shuffle tax, and make sure it is worth it.

Actions

Unlike transformations, actions are eager. They will compute things. If transformations are the instructions, actions are the "Go!" button. They're what finally force Spark to roll up its sleeves, stop being lazy, and actually execute the computation.

Calling an action triggers execution of the entire DAG you've been building with transformations. Spark spins up tasks, shuffles data if needed, and materializes the result — whether that means dumping it to disk, loading it into a database, or just printing a sample to console.

Common actions include:

• collect: Pulls all the data to the driver. Great for debugging, terrible for large datasets. Don't use it unless you know your data fits in memory.

• reduce: Aggregates all elements using a user-defined function.

• count: Counts all the elements in the RDD.

For example:

>>> filteredRDD.reduce(lambda a, b: a + b)
135

This line kicks off a full DAG execution. All previous transformations get evaluated, partitioned tasks are distributed across the cluster, and the final result comes to the Spark driver.

Beyond RDDs: DataFrames and Datasets

While RDDs form the foundation of Spark, modern APIs like DataFrames and Datasets are preferred for handling structured data due to their optimizations and simplicity. Here's a quick comparison:

• RDDs: Provide low-level control over distributed data but lack query optimizations. Best for unstructured data and fine-grained transformations.

• DataFrames: Represent data in a table-like format with named columns (like a SQL table). They leverage Spark's Catalyst Optimizer for efficient query planning and execution.

• Datasets: Offer the benefits of DataFrames with type safety, making them ideal for statically typed languages like Scala or Java.

Since Spark 3.x, DataFrames aren't just preferred — they're practically mandatory for performance-sensitive or production Spark workloads. While RDDs give more control, if you're using RDDs for anything besides niche low-level operations, you're fighting the engine.

Directed Acyclic Graph

One of the biggest innovations that sets Apache Spark ahead of Hadoop MapReduce — aside from being 100x faster and 10x less painful — is the way it models your computation. Instead of rigid, hardcoded pipelines of Map → Shuffle → Reduce, Spark builds a Directed Acyclic Graph (DAG) as it ingests your transformations: a blueprint of operations and dependencies that spells out exactly how your job will execute.

What Is a DAG, Anyway?

At its core, a DAG in Spark is just a graph with two key properties:

• Directed: every edge points one way — there's a clear "before" and "after"

• Acyclic: there are no cycles — once you move forward, you cannot loop back

Why does this matter? Because it guarantees that your data processing pipeline has a clear beginning and end. There's no chance of infinite loops, no ambiguous backtracking — just a straight‐line (albeit branching and merging) flow from raw data to final results.

• Nodes in this graph are your RDD transformations and actions: map, filter, flatMap, reduceByKey, and so on.

• Edges represent the dependencies between those operations: "I can't run this reduceByKey until the map and filter that precede it have finished."

That's it. Simple and elegant.

Logical vs Physical Plans

Before we get into the DAG construction, let's clarify two concepts:

• Logical DAG: The abstract plan Spark builds when you define transformations but haven't yet called an action. It's a high‐level blueprint that says "do map, then filter, then reduceByKey", but it hasn't decided how or where to execute anything.

• Physical execution plan: The concrete realization of that DAG once you call an action. Here, Spark chooses the actual algorithms, the number of partitions, which nodes to run on, and so forth.

Spark keeps these two worlds separate to maintain laziness — it only pays the execution cost when you truly need a result.

DAG Construction

When you define a chain of transformations nothing executes yet. Spark's driver program records each call, building up that logical DAG in memory. These nodes and edges stack up until you finally call an action - the green light for Spark to convert the logical DAG into a physical execution plan and start processing.

But before Spark can start processing, the logical DAG must be split into stages. A stage is a sequence of operations that can be executed without a shuffle. Narrow transformations like map, filter, and flatMap — process data locally on each partition, so Spark fuses them into the same stage. The moment a wide transformation appears - such as reduceByKey, groupByKey, or join — a shuffle is required, and Spark cuts the DAG, starting a new stage.

Check out this classic word-count example:

sc = SparkContext.getOrCreate()
text = sc.textFile("hdfs://...") # Read text file
words = text.flatMap(lambda line: line.split(r"\s+")) # Split lines into words
pairs = words.map(lambda word: (word, 1)) # Map each word to (word, 1)
counts = pairs.reduceByKey(lambda a, b: a + b) # Reduce by key (word) to get counts

Here:

• Stage 1 will contain: textFile → flatMap → map → filter

• Stage 2 will contain: reduceByKey

That's two stages, each of which Spark can schedule independently, as soon as its dependencies are met.

Why Spark Builds a DAG

Imagine you're using classic Hadoop MapReduce. You write one Map job, it writes to disk, then a Reduce job reads from disk, does its work, and writes to disk again. On each step, the framework forces you to materialize data to HDFS, incurring heavy I/O and disk seeks. To make matters worse, you have zero understanding into the global structure of your computation — you only see one stage at a time.

Spark's approach — don't run a thing until you have to. Record your intentions, compile them into a DAG, then optimize across the whole graph before you fire off any tasks. That gives Spark two massive advantages:

1. Global optimization. The DAG allows Spark to treat your code as a full graph, not just a sequence of steps. It sees the entire dependency chain and can make both global and local optimizations — like reordering operations, collapsing stages, or adjusting partitioning dynamically (especially with AQE enabled).

2. Fault tolerance via lineage. Because Spark knows the entire history of your operations, if a partition goes missing, it only recomputes the lost data by re‐applying the minimal needed transformations — not full job restarts.

The DAG-based execution model is a big reason why Spark can handle complex workloads across massive datasets while minimizing the overhead typically associated with distributed processing. Without the DAG, Spark would lose much of its flexibility, speed, and fault tolerance.

Conclusion

By now you've seen how Spark's core abstractions — RDDs, transformations and actions, and the DAG execution model — work together to deliver both performance and resilience. You know now that lazy evaluation lets Spark optimize across your entire workflow, that narrow transformations avoid costly shuffles, and that lineage‐based fault tolerance keeps your jobs reliable.

Additional materials

• Spark: The Definitive Guide by Bill Chambers, Matei Zaharia

• Learning Spark by Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee

If you've spent any time wrangling data at scale, you've probably heard of Apache Spark. Maybe you've even cursed at it once or twice — don't worry, you're in good company. Spark has become the go-to framework for big data processing, and for good reason: it's fast, versatile, and (once you get the hang of it) surprisingly elegant. But mastering it? That's a whole other story.

Spark is packed with features and an architecture that feels simple on the surface but gets complex real quick. If you've ever struggled with long shuffling, weird partitioning issues, or mysterious memory errors, you know exactly what I mean.

This article is the first in a series on Apache Spark, which I put together: to help you get past the basics and into the real nuts and bolts of how it works and how to make it work for you.

What is Spark?

Think of splitting your laptop into a thousand tiny replicas, each with its own RAM, then running .groupBy().sum() across billions of events.

That, in spirit, is Apache Spark.

Apache Spark is an open‑source cluster‑computing engine built to handle both batch and streaming workloads with an unapologetic focus on speed. It keeps data in memory whenever possible, spilling to disk only when datasets outgrow RAM. Add concise APIs in Scala, Python, Java, and SQL, and engineers get a tool that feels as productive as a notebook but scales to a data‑center.

Beyond language support and in‑memory execution, Spark brings two other pillars that matter day‑to‑day:

1. Lazy evaluation. Transformations are recorded as a directed acyclic graph (DAG) and executed only when an action forces a result, letting Spark optimize the entire plan.

2. Resilience. The same lineage graph allows partitions to be recomputed on another machine if a node dies, keeping long jobs from collapsing.

Yet for all its intelligence, Spark intentionally stays out of one debate: where your code should run. It simply declares, “Give me 80 executors, each with 4 vCPU and 8 GB RAM, plus a driver”. Choosing the actual hosts, allocating resources, and recovering from failure is the job of a cluster manager. But before we meet those managers, let’s rewind to the platform that made space for them.

The Evolution of Hadoop

In 2006, Yahoo! turned Google’s MapReduce research project into an open-source reality and called the result Hadoop. Release 0.x shipped two core pieces:

• HDFS (Hadoop Distributed File System) – a block-based, write-once distributed file system tuned for high-throughput, sequential I/O on clusters of commodity servers.

• MapReduce – a batch-processing engine that runs every job through three canonical phases: map → distributed shuffle → reduce.

For the Big Data world, this was a revelation — terabytes could now be processed on racks of cheap servers instead of specific premium hardware.

The next major revision, Hadoop 2 introduced a third component: YARN (Yet Another Resource Negotiator). YARN split resource management out of MapReduce and became a cluster‑wide operating system, handing out CPU and memory to any engine that asked politely. With YARN in place, processing engines like Spark, Tez, and Flink could run and flourish side-by-side on the same platform without touching Hadoop’s storage layer. That decoupling is what let Spark catch fire inside existing Hadoop deployments.

Why Spark Won

So, why did Spark become such a go-to tool for big data processing?

One word: speed.

While MapReduce relies heavily on disk I/O — writing data to disk between each processing phase — Spark uses in-memory computing to keep data in RAM throughout the computation and spill to disk only when necessary. By explicitly caching or persisting an RDD/DataFrame, Spark dramatically increased processing speed, especially for iterative algorithms that need to access the same data multiple times. This shift from disk-based to in-memory processing makes Spark not just faster, but also more suitable for applications that require near real-time analytics or iterative machine learning models, which MapReduce struggles to handle efficiently.

On top of that, Spark’s friendy API (all cool stuff like Scala’s chained functions, Python’s lazy DataFrames, ANSI‑compliant SQL) makes it a framework that’s not just useful, but one data engineers want to use. We prefer tools that minimize cognitive overhead so we can focus on solving problems, not wrestling with syntax, right?

Lastly, Spark is poly-modal — batch ETL is built in, real-time pipelines can ride on Spark Structured Streaming, and machine-learning workloads plug straight into MLlib — all powered by the same execution engine.

Power and usability made engineers reach for Spark — as long as their jobs could find a slot to run. Leaving the comfort of a single machine means you still need a data‑center operating system that decides where each executor lands and how to restart it after failure. That’s the job for cluster managers — YARN in the Hadoop world, Kubernetes in the container era, and a few others along the way.

Cluster Managers 101

To understand where YARN — and later Kubernetes — fit into the picture, let’s borrow a metaphor from everyday computing.

A single machine’s operating system does roughly two big jobs:

• File System: Manages data storage and retrieval (FAT32, ext4, NTFS, etc).

• Kernel & Scheduler: Decide which processes gets CPU, RAM, and I/O, and when.

When we scale this concept to a cluster level and implement it within Hadoop, we see a similar division. But instead of a single-node file system, Hadoop uses HDFS (Hadoop Distributed File System). YARN provides cluster-level resource negotiation (CPU cores and memory) and launches containers for the application execution.

So Spark outsources the “OS”-type management tasks — it’s focusing only on the actual data processing.

Out of the box Spark can speak to:

• Standalone — lightweight scheduler for laptops, CI pipelines, and small proofs‑of‑concept

• Apache Mesos — influential but largely retired now

• YARN — the default in Hadoop‑centric stacks thanks to tight HDFS and Kerberos integration.

• Kubernetes — the default in container‑first organizations which as of now is probably everybody

💡 We used “container” term a lot here, let me clarify if there is a confusion. In YARN terminology, a container is simply a slice of resources — CPU cores, memory, and (optionally) GPUs — allocated on a worker node. It shares the host’s operating system and filesystem. Kubernetes, on the other hand, schedules Docker/OCI containers, which bundle an application’s entire filesystem, runtime libraries, and process isolation via namespaces and cgroups. Same word, two very different abstractions.

We deep dive into YARN because it paved the runway for Spark’s early success — and still powers petabytes of production data where HDFS and Hadoop security are non‑negotiable.

Deep Dive into YARN

At the heart of YARN sits the ResourceManager. Think of it as the cluster’s traffic controller: every request for CPU or memory, from every application, flows through the ResourceManager first. Nothing starts, scales, or restarts without its say-so.

ResourceManager consists of two key services:

• ApplicationManager:

  • Accepts and registers new applications

  • Manages the application state‑machine (NEW → RUNNING → FINISHED/FAILED/KILLED)

  • Requests the first container to launch the ApplicationMaster (described below)

• Scheduler:

  • Maintains an up‑to‑date view of free CPU, memory, and GPUs on every node (via heartbeats)

  • Decides where and when to place containers

  • Enforces multi‑tenant policies (Capacity, Fair, or FIFO)

  • Preempts lower‑priority work

Each physical or virtual host in the cluster runs a NodeManager daemon, which reports to the ResourceManager.

Its jobs are straightforward but crucial:

• Launch containers on local Linux cgroups/namespaces

• Monitor the health and resource usage of those containers

• Report status back to the ResourceManager every few seconds (heartbeats)

Each cluster node hosts several Containers (YARN containers), a Container is YARN’s atomic chunk of compute: a slice of memory, a share of vCPUs, plus the environment variables and files the task needs.

Containers are allocated to execute specific tasks for user applications. Their size and configuration can be adjusted based on workload requirements. The size of each container is negotiated at run time.

💡 Going a bit deeper — when a Spark job starts, the ResourceManager asks a NodeManager to create a container; the NodeManager forks a JVM, attaches it to the cgroup, and hands the container ID back. If that executor tries to exceed its quota, the kernel’s cgroup OOM-killer terminates it, protecting the rest of the node.

The NodeManager daemon oversees the containers on its respective node. When a new application is submitted to the cluster, the ResourceManager allocates a container for the ApplicationMaster. The ApplicationMaster job is to negotiate resources with the ResourceManager and coordinates with the NodeManagers to execute and monitor tasks.

Once launched, the ApplicationMaster takes charge of the application lifecycle. Its first task is to send resource requests to the ResourceManager to acquire additional containers for running the application's tasks. Each resource request typically includes:

• The amount of resources needed, specified in terms of memory and CPU shares

• Preferred container locations (like hostname, rack name)

• Priority within the application

I want to hightlight — the ApplicationMaster runs in a container just like other components of the application. If it crashes or becomes unavailable, the ResourceManager can allocate a new container and restart it, ensuring high availability. The ResourceManager maintains metadata about running applications and task progress in HDFS, so when the ApplicationMaster restarts, it has all the context it needs. This allows the system to recover and restart only incomplete tasks if the application fails.

In this architecture, the ResourceManager, NodeManagers, and containers are framework-agnostic — they don't need to understand the specifics of the application. Instead, the ApplicationMaster handles all framework-specific logic, enabling any distributed framework to run on YARN as long as it provides a suitable ApplicationMaster. We are focused on Spark, but other frameworks can run with YARN as well. Spark executors, Flink tasks, even legacy MapReduce tasks — all of them can run inside YARN containers. This design allows YARN to support a diverse workloads efficiently and flexibly.

Submitting an Application to YARN

When you spark-submit --master yarn, three daemons and a handful of containers jump into action. Here’s the workflow:

1. Client (your laptop or CI agent) sends a tiny “launch request” to the cluster’s ResourceManager. The payload says, essentially, “Please start Spark’s ApplicationMaster — here’s the JAR, here’s the memory budget”.

2. ResourceManager scans its bookkeeping tables, picks a free host, and asks that host’s NodeManager to carve out one fresh container for the ApplicationMaster.

3. Inside that container, Spark’s ApplicationMaster starts the JVM, loads your job’s classpath, and registers itself back with the ResourceManager. From this point on, the ApplicationMaster is the single source of truth for everything your job will do.

4. Next, the ApplicationMaster requests executor containers from the ResourceManager: “Give me 50 containers, each 4 vCPU/8 GB, preferably close to data on rack A or B”. The request can contain multiple sizes and priorities if your job mixes lightweight and heavyweight tasks which is kinda cool.

5. The ResourceManager grants containers as capacity frees up. Each chosen NodeManager spins up a new process under cgroups/namespaces, sets environment variables, and hands control to Spark’s executor script.

6. Every executor reports back to the ApplicationMaster (which is also the Spark driver) to say, “I’m alive; send work”. The ApplicationMaster runs tasks — map, shuffle, reduce, whatever — while the NodeManagerMs monitor CPU, RAM, and exit codes.

7. Meanwhile, your spark-submit process (or a web UI) polls the ApplicationMaster for progress: completed stages, failed tasks, shuffle size, you name it. The ApplicationMaster keeps the ResourceManager updated with heartbeat traffic so the job stays visible in cluster dashboards.

8. When the application’s last task finishes, the ApplicationMaster tells the ResourceManager, “All done”. It writes final counters to HDFS, de-registers, and exits. The ResourceManager then marks remaining containers for garbage collection, and NodeManagers delete logs once retention rules say it’s safe to do so.

In roughly a dozen round-trips and a few seconds of coordination, your Spark job goes from a CLI command to a fully managed, distributed application — ready to process terabytes until the last partition is joined, aggregated, or filtered away all with the help of YARN. Sweet.

Deep Dive into Kubernetes

A decade ago YARN was the obvious companion for Spark — both lived inside the Hadoop ecosystem, both assumed Java everywhere, and both were happiest when HDFS was the only distributed storage. Since then, the center of gravity has shifted. Modern platforms package everything — front-end apps, REST gateways, model servers — into containers and schedule them with Kubernetes.

Read more