CMPT 732 Lecture Notes

1. Course Introduction [“Course Introduction” slides]
   1. Us [Us slides]
   2. This Course [This Course slides]
   3. What is Big Data? [What is Big Data? slides]
   4. How big is “Big Data”? [How big is “Big Data”? slides]
   5. “Big data” isn't always big. [“Big data” isn't always big. slides]
   6. None [None slides]
   7. Clusters [Clusters slides]
   8. Hadoop [Hadoop slides]
   9. Our Environment [Our Environment slides]
   10. Things you will do [Things you will do slides]
   11. Lecture and Labs [Lecture and Labs slides]
   12. Assignments [Assignments slides]
   13. Quizzes [Quizzes slides]
   14. Expectations [Expectations slides]
   15. Course Topics [Course Topics slides]
2. Hadoop Concepts [“Hadoop Concepts” slides]
   1. Our Cluster [Our Cluster slides]
   2. Hadoop Pieces [Hadoop Pieces slides]
   3. HDFS [HDFS slides]
   4. YARN [YARN slides]
   5. (Simplified) Cluster Overview [(Simplified) Cluster Overview slides]
   6. Work on Hadoop [Work on Hadoop slides]
   7. MapReduce [MapReduce slides]
   8. MapReduce Stages [MapReduce Stages slides]
   9. Example: word count [Example: word count slides]
   10. Diagram: Fall 2025 [Diagram: Fall 2025 slides]
   11. MapReduce Anatomy [MapReduce Anatomy slides]
   12. Hadoop MapReduce Details [Hadoop MapReduce Details slides]
   13. Summary Output [Summary Output slides]
   14. MapReduce Parallelism [MapReduce Parallelism slides]
   15. Writables [Writables slides]
   16. Example: word count [Example: word count slides]
   17. About MapReduce [About MapReduce slides]
   18. MapReduce: One more way [MapReduce: One more way slides]
   19. MapReduce Data Flow [MapReduce Data Flow slides]
3. Python Preliminaries [“Python Preliminaries” slides]
   1. About Python [About Python slides]
   2. Data Types [Data Types slides]
   3. Unpacking Tuples [Unpacking Tuples slides]
   4. First-Class Functions [First-Class Functions slides]
   5. Lambda Functions [Lambda Functions slides]
   6. Iterators and Generators [Iterators and Generators slides]
   7. Imperative vs declarative [Imperative vs declarative slides]
4. Spark Concepts [“Spark Concepts” slides]
   1. Spark [Spark slides]
   2. An Example [An Example slides]
   3. RDDs [RDDs slides]
   4. RDD Operations [RDD Operations slides]
   5. Operations and Partitions [Operations and Partitions slides]
   6. Partitions [Partitions slides]
   7. Lazy Evaluation [Lazy Evaluation slides]
   8. Chaining Calculations [Chaining Calculations slides]
   9. Combining Calculations [Combining Calculations slides]
   10. Shuffle Operations [Shuffle Operations slides]
   11. Drivers & Executors [Drivers & Executors slides]
   12. Controlling Executors [Controlling Executors slides]
   13. Spark Web Frontend [Spark Web Frontend slides]
   14. Spark vs MapReduce [Spark vs MapReduce slides]
   15. Spark DAG [Spark DAG slides]
   16. Stages [Stages slides]
   17. Job, Stages, Tasks [Job, Stages, Tasks slides]
   18. RDD Methods [RDD Methods slides]
5. Spark DataFrames Concepts [“Spark DataFrames Concepts” slides]
   1. Working With Data [Working With Data slides]
   2. Spark DataFrames [Spark DataFrames slides]
   3. DataFrames [DataFrames slides]
   4. Column Expressions [Column Expressions slides]
   5. SQL Syntax [SQL Syntax slides]
   6. The Optimizer [The Optimizer slides]
   7. Input/Output [Input/Output slides]
   8. Parquet [Parquet slides]
   9. Partitioning [Partitioning slides]
   10. Limitations [Limitations slides]
   11. UDFs [UDFs slides]
   12. Python ↔ JVM [Python ↔ JVM slides]
6. NoSQL & Cassandra [“NoSQL & Cassandra” slides]
   1. The Problem [The Problem slides]
   2. Some DB Operations [Some DB Operations slides]
   3. Non-Relational Databases [Non-Relational Databases slides]
   4. NoSQL Limitations [NoSQL Limitations slides]
   5. Cassandra [Cassandra slides]
   6. Cassandra Data Model [Cassandra Data Model slides]
   7. CQL [CQL slides]
   8. Fault Tolerance [Fault Tolerance slides]
   9. Consistency [Consistency slides]
   10. Relational Data [Relational Data slides]
   11. Denormalizing Data [Denormalizing Data slides]
7. Cloud & Data Management [“Cloud & Data Management” slides]
   1. Scalable Storage [Scalable Storage slides]
   2. Scalable Compute [Scalable Compute slides]
   3. Cloud Services [Cloud Services slides]
8. Data Management [“Data Management” slides]
   1. Databases [Databases slides]
   2. Data Lake [Data Lake slides]
   3. Extract-Transform-Load [Extract-Transform-Load slides]
   4. Less-Structured Data [Less-Structured Data slides]
   5. Data Warehouse [Data Warehouse slides]
   6. Demo: Weather ETL [Demo: Weather ETL slides]
   7. Clickhouse [Clickhouse slides]
   8. Data Lakehouse [Data Lakehouse slides]
   9. Summary [Summary slides]
9. Spark Machine Learning
10. Spark Streaming
11. Small Data
12. Other DataFrame Tools
13. Other Big Data Tools
14. NumPy/Pandas Speed

Course home page.

Schedule

Week	Deliverables (*)	Lecture Date	First Slide
1	Assign 0	in lab Sep 4	Intro
2	Assign 1	Sep 8	Hadoop Concepts
3	Assign 2	Sep 15	MapReduce Parallelism
4	Assign 3	Sep 22	Operations and Partitions
5	Assign 4	Sep 29	Job, Stages, Tasks
6	Quiz 1, Assign 5	Oct 6	Quiz 1, The Optimizer
7	Assign 6	Oct 13	Thanksgiving holiday 🦃
8	Assign 7	Oct 20	NoSQL & Cassandra
9	Assign 8	Oct 27
10	Quiz 2, Assign 9	Nov 3
11	Assign 10	Nov 10
12		Nov 17
13	Quiz 3	Nov 24
14	Project	Dec 1

* Check CourSys for the actual due dates and times.