CMPT 732 Lecture Notes

  1. Course Introduction [“Course Introduction” slides]
    1. Us [Us slides]
    2. This Course [This Course slides]
    3. What is Big Data? [What is Big Data? slides]
    4. How big is “Big Data”? [How big is “Big Data”? slides]
    5. “Big data” isn't always big. [“Big data” isn't always big. slides]
    6. None [None slides]
    7. Clusters [Clusters slides]
    8. Hadoop [Hadoop slides]
    9. Our Environment [Our Environment slides]
    10. Things you will do [Things you will do slides]
    11. Lecture and Labs [Lecture and Labs slides]
    12. Assignments [Assignments slides]
    13. Quizzes [Quizzes slides]
    14. Expectations [Expectations slides]
    15. Course Topics [Course Topics slides]
  2. Hadoop Concepts [“Hadoop Concepts” slides]
    1. Our Cluster [Our Cluster slides]
    2. Hadoop Pieces [Hadoop Pieces slides]
    3. HDFS [HDFS slides]
    4. YARN [YARN slides]
    5. (Simplified) Cluster Overview [(Simplified) Cluster Overview slides]
    6. Work on Hadoop [Work on Hadoop slides]
    7. MapReduce [MapReduce slides]
    8. MapReduce Stages [MapReduce Stages slides]
    9. Example: word count [Example: word count slides]
    10. Diagram: Fall 2024 [Diagram: Fall 2024 slides]
    11. MapReduce Anatomy [MapReduce Anatomy slides]
    12. Hadoop MapReduce Details [Hadoop MapReduce Details slides]
    13. Summary Output [Summary Output slides]
    14. MapReduce Parallelism [MapReduce Parallelism slides]
    15. Writables [Writables slides]
    16. Example: word count [Example: word count slides]
    17. About MapReduce [About MapReduce slides]
    18. MapReduce: One more way [MapReduce: One more way slides]
    19. MapReduce Data Flow [MapReduce Data Flow slides]
  3. Python Preliminaries [“Python Preliminaries” slides]
    1. About Python [About Python slides]
    2. Data Types [Data Types slides]
    3. Unpacking Tuples [Unpacking Tuples slides]
    4. First-Class Functions [First-Class Functions slides]
    5. Lambda Functions [Lambda Functions slides]
    6. Iterators and Generators [Iterators and Generators slides]
    7. Imperative vs declarative [Imperative vs declarative slides]
  4. Spark Concepts [“Spark Concepts” slides]
    1. Spark [Spark slides]
    2. An Example [An Example slides]
    3. RDDs [RDDs slides]
    4. RDD Operations [RDD Operations slides]
    5. Operations and Partitions [Operations and Partitions slides]
    6. Partitions [Partitions slides]
    7. Lazy Evaluation [Lazy Evaluation slides]
    8. Chaining Calculations [Chaining Calculations slides]
    9. Combining Calculations [Combining Calculations slides]
    10. Shuffle Operations [Shuffle Operations slides]
    11. Drivers & Executors [Drivers & Executors slides]
    12. Controlling Executors [Controlling Executors slides]
    13. Spark Web Frontend [Spark Web Frontend slides]
    14. Spark vs MapReduce [Spark vs MapReduce slides]
    15. Spark DAG [Spark DAG slides]
    16. Stages [Stages slides]
    17. Job, Stages, Tasks [Job, Stages, Tasks slides]
    18. RDD Methods [RDD Methods slides]
  5. Spark DataFrames Concepts [“Spark DataFrames Concepts” slides]
    1. Working With Data [Working With Data slides]
    2. Spark DataFrames [Spark DataFrames slides]
    3. DataFrames [DataFrames slides]
    4. Column Expressions [Column Expressions slides]
    5. SQL Syntax [SQL Syntax slides]
    6. The Optimizer [The Optimizer slides]
    7. Input/Output [Input/Output slides]
    8. Parquet [Parquet slides]
    9. Partitioning [Partitioning slides]
    10. Limitations [Limitations slides]
    11. UDFs [UDFs slides]
    12. Python ↔ JVM [Python ↔ JVM slides]
  6. Cloud & Data Management [“Cloud & Data Management” slides]
    1. Scalable Storage [Scalable Storage slides]
    2. Scalable Compute [Scalable Compute slides]
    3. Cloud Services [Cloud Services slides]
  7. NoSQL & Cassandra [“NoSQL & Cassandra” slides]
    1. The Problem [The Problem slides]
    2. Some DB Operations [Some DB Operations slides]
    3. Non-Relational Databases [Non-Relational Databases slides]
    4. NoSQL Limitations [NoSQL Limitations slides]
    5. Cassandra [Cassandra slides]
    6. Cassandra Data Model [Cassandra Data Model slides]
    7. CQL [CQL slides]
    8. Fault Tolerance [Fault Tolerance slides]
    9. Consistency [Consistency slides]
    10. Relational Data [Relational Data slides]
    11. Denormalizing Data [Denormalizing Data slides]
  8. Data Management [“Data Management” slides]
    1. Databases [Databases slides]
    2. Data Lake [Data Lake slides]
    3. Extract-Transform-Load [Extract-Transform-Load slides]
    4. Less-Structured Data [Less-Structured Data slides]
    5. Data Warehouse [Data Warehouse slides]
    6. Demo: Weather ETL [Demo: Weather ETL slides]
    7. Clickhouse [Clickhouse slides]
    8. Data Lakehouse [Data Lakehouse slides]
    9. Summary [Summary slides]
  9. Spark Machine Learning [“Spark Machine Learning” slides]
    1. Recap: Machine Learning [Recap: Machine Learning slides]
    2. Spark ML [Spark ML slides]
    3. Pipelines [Pipelines slides]
    4. Models [Models slides]
    5. Evaluation [Evaluation slides]
    6. ML Algorithms [ML Algorithms slides]
    7. More Topics [More Topics slides]
    8. Demo [Demo slides]
  10. Spark Streaming [“Spark Streaming” slides]
    1. The Purpose [The Purpose slides]
    2. Options [Options slides]
    3. RDD-Based: The Idea [RDD-Based: The Idea slides]
    4. DStreams [DStreams slides]
    5. Structured Streaming [Structured Streaming slides]
  11. Why MapReduce?
  12. Small Data
  13. Other DataFrame Tools
  14. Other Big Data Tools
  15. NumPy/Pandas Speed

Course home page.

Schedule

Week Deliverables (*) Lecture Date First Slide
1 Assign 0 in lab Sep 5
2 Assign 1 Sep 9
3 Assign 2 Sep 16
4 Assign 3 Sep 23
5 Assign 4 Sep 30
6 Assign 5 Oct 7
7 Assign 6 Oct 14
8 Assign 7, Quiz 1 Oct 21
9 Assign 8 Oct 28
10 Assign 9, Quiz 2 Nov 4
11 Assign 10 Nov 11
12 Nov 18
13 Quiz 3 Nov 25
14 Project

* Check CourSys for the actual due dates and times.