Apache Spark Training

Curriculum:

Module 1: Introduction to Big Data and Apache Spark

• Big Data ecosystem overview and limitations of MapReduce
• What is Apache Spark?
• Spark ecosystem and its components (Spark Core, SQL, MLlib, GraphX, Streaming)
• Spark architecture: Driver, Executors, Cluster Manager
• Spark vs Hadoop MapReduce – Performance and Use Cases
• Setting up Spark environment (Standalone, YARN, Databricks)

Module 2: Spark Core and RDD Programming

• Understanding RDDs (Resilient Distributed Datasets)
• Creating RDDs from collections and external data sources
• RDD operations: Transformations and Actions
• Lazy evaluation and Spark job execution process
• Persisting and caching RDDs for performance
• Hands-on: RDD transformations for data cleaning and aggregation

Module 3: Spark DataFrames and Datasets

• Introduction to Spark SQL and DataFrame API
• Creating and transforming DataFrames
• Schema definition and inference
• Differences between RDD, DataFrame, and Dataset
• Type-safe operations with Datasets (Scala/Java)
• Hands-on: Working with structured data (CSV, JSON, Parquet)

Module 4: Spark SQL and Query Optimization

• Introduction to Spark SQL and Catalyst optimizer
• Writing SQL queries on DataFrames
• Integrating Spark with Hive Metastore
• User-defined functions (UDFs) and User-defined aggregate functions (UDAFs)
• Query plan analysis and optimization
• Hands-on: Running analytical queries with Spark SQL

Module 5: Spark Streaming and Structured Streaming

• Introduction to Spark Streaming
• Difference between DStreams and Structured Streaming
• Event-time vs Processing-time concepts
• Integrating Spark with Kafka for real-time ingestion
• Windowed aggregations and watermarking
• Hands-on: Building a real-time dashboard using Spark and Kafka

Module 6: Machine Learning with Spark MLlib

• Overview of Spark MLlib architecture
• Data preprocessing, feature extraction, and transformation
• Supervised vs Unsupervised learning
• Implementing classification, regression, and clustering models
• Model persistence and evaluation
• Hands-on: Building a customer churn prediction model

Module 7: Graph Processing with GraphX

• Introduction to GraphX and its use cases
• Graph operators and parallel graph computations
• PageRank and connected components
• Integration of GraphX with DataFrames
• Hands-on: Social network graph analysis

Module 8: Spark Integration with Hadoop, Hive, and Kafka

• Reading/writing data from HDFS and NoSQL databases
• Connecting Spark with Hive for advanced analytics
• Streaming data integration with Kafka
• Working with Amazon S3, Azure Data Lake, and Google Cloud Storage
• Hands-on: Building an ETL pipeline using Spark and Hive

Module 9: Performance Tuning and Optimization

• Understanding Spark job execution and DAG visualization
• Best practices for partitioning and shuffling
• Memory and garbage collection tuning
• Broadcast variables and accumulators
• Data serialization and I/O performance improvement
• Hands-on: Tuning a large-scale Spark job for faster execution