Apache Spark and Scala Certification Training Course

Module 1: Introduction to Spark & Scala Setup

⏳ 1 week

• Topics: Spark ecosystem, driver vs. executor, setting up Scala IDE or IntelliJ with sbt

• Hands-on: Launch a local Spark shell and write your first RDD operations in Scala

Module 2: RDDs & Core Transformations

⏳ 1 week

• Topics: RDD creation methods, transformations (map, filter), actions (collect, count)

• Hands-on: Build a word-count pipeline and analyze logs using RDD APIs

Module 3: DataFrames & Spark SQL

⏳ 1 week

• Topics: DataFrame vs. RDD, schema inference, SparkSession, SQL queries on structured data

• Hands-on: Load JSON and CSV into DataFrames, register temp views, and run SQL aggregations

Module 4: Dataset API & Typed Transformations

⏳ 1 week

• Topics: Strongly-typed Datasets, encoder usage, mapping to case classes

• Hands-on: Convert DataFrames to Datasets and perform type-safe transformations

Module 5: ETL & Data Processing Patterns

⏳ 1 week

• Topics: Joins, window functions, complex types (arrays, maps), UDFs in Scala

• Hands-on: Cleanse and enrich a sales dataset, then compute moving averages with windowing

Module 6: Machine Learning with MLlib

⏳ 1 week

• Topics: Pipelines, feature transformers, classification models, clustering algorithms

• Hands-on: Implement a full ML pipeline (e.g., Logistic Regression) and evaluate model performance

Module 7: Performance Tuning & Optimization

⏳ 1 week

• Topics: Partitioning strategies, broadcast variables, caching, shuffle avoidance, resource configs

• Hands-on: Profile a slow job in the Spark UI and apply tuning to reduce runtime

Module 8: Deployment & Cloud Integration

⏳ 1 week

• Topics: spark-submit, YARN vs. standalone clusters, Databricks notebooks, integrating with HDFS/S3

• Hands-on: Deploy an end-to-end ETL Spark job on a Hadoop cluster and monitor via the Spark UI

Module 9: Capstone Project & Best Practices

⏳ 1 week

• Topics: End-to-end pipeline design, code modularization, logging, error handling

• Hands-on: Build a complete real-world data pipeline: ingest raw logs, transform, analyze, and persist results

Get certificate