Apache Spark Scala Interview Questions- Shyam Mallesh Apr 2026

import org.apache.spark.sql.types._ val schema = StructType(Seq( StructField("name", StringType), StructField("age", IntegerType), StructField("address", StructType(Seq( StructField("city", StringType), StructField("zip", LongType) ))) ))

val rdd = sc.textFile("data.txt") // nothing read yet val words = rdd.flatMap(_.split(" ")) // transformation val counts = words.map(w => (w, 1)).reduceByKey(_ + _) // transformation counts.saveAsTextFile("output") // 🔥 Action triggers job | Operation | Shuffle Behavior | Performance | |----------------|------------------|--------------| | groupByKey | Sends all values for a key across the network → high shuffle I/O | Slower, risks OOM | | reduceByKey | Combines values locally (map-side reduce) before shuffle → reduces data transfer | Faster, memory efficient | Apache Spark Scala Interview Questions- Shyam Mallesh

val df = spark.read.option("inferSchema", "true").json("data.json") import org

breaks long lineages by saving RDD to reliable storage (HDFS/S3). ✅ 3. What is the difference between cache() , persist() , and checkpoint() ? | Method | Storage Level | Purpose | |--------------|------------------------------|---------| | cache() | MEMORY_ONLY (default) | Speed up repeated actions | | persist() | Choose level (MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, etc.) | Fine-grained control over eviction | | checkpoint() | Saves to HDFS/S3 (reliable storage) | Break lineage, reduce driver memory, avoid recomputation chain | 💡 Use persist when memory is limited. Use checkpoint for long iterative algorithms (ML, GraphX). ✅ 4. Explain how Spark evaluates transformations and actions. Spark uses lazy evaluation – transformations build DAG but no data is processed until an action ( count , collect , save , show , etc.) is called. | Method | Storage Level | Purpose |

⚠️ coalesce(1) avoids shuffle but may cause data skew. Only safe if current partitions are small. With schema inference (slow but automatic):