Big Data

Revision · Topics 1–5
Lectures and tutorials

Topics and tutorials

Follow one progression: distribute and store records, express and execute queries, improve the pipeline, build a model, and analyse stored data with SQL.

TopicCore knowledgeTutorial connection
1 · HadoopHDFS, YARN and MapReduce1 · Word count on EMR
2 · SparkData models and distributed execution2 · Interactive word count
3 · Spark in depthCorrect pipelines, movement and storage3 · Adult data analysis
4 · Spark MLFeatures, fitting and honest evaluation4 · Customer churn
5 · HiveMetadata, SQL and storage layout5 · MovieLens ratings
02 / 97

Data to a checked result

A useful result depends on every preceding step: the files must be right, the schema must interpret them correctly, and the computation must answer the intended question.

Acquire + storeKeep originalrecordsRecord source +scopePrepare +computeValidate types +rowsJoin and aggregateAnalyse orpredictSQL result or MLoutputCheck against abaselineVerify +preserveRead back savedoutputKeep evidence offcluster
03 / 97

Hadoop and distributed data

Before a distributed program can calculate anything, its records need storage and its tasks need resources.

FilesSplit into blocksHDFS stores replicasResourcesYARN allocates capacityWorkers launch containersProcessingMap emits keyed valuesReduce combines groups
04 / 97

The five Vs

Big Data is relative to the task and available capacity. Diagnose the actual pressure before choosing a larger cluster or a new tool.

PressureMeaningConsequence
VolumeAmount of dataStorage and scan cost
VelocityArrival and processing rateBacklog grows if arrivals exceed capacity
VarietyDifferent structures and formatsParsing and integration work
VeracityTrustworthiness of recordsCheck missing, duplicate and invalid values
ValueUseful outcome for a questionA correct calculation still needs a purpose
05 / 97

Scale up and scale out

Scaling up adds capacity to one machine; scaling out divides work among machines. Distributed work also needs coordination, data movement and recovery.

ChoicePotential gainLimit or cost
Scale upMore CPU, memory or storage per machineHardware limits; one machine remains a failure point
Scale outMore workers can process independent dataNetwork, partitioning and coordination costs
ExampleTwo workers scan separate log partitionsA grouped total still has to combine their results
DecisionMeasure the limiting resource firstMore nodes do not repair duplicate or invalid rows
06 / 97

Hadoop component roles

Hadoop Distributed File System (HDFS) stores bytes, YARN allocates resources, and MapReduce processes records. These responsibilities cooperate within one job.

HDFS · storageNameNode: names + locationsDataNodes: replicated bytesYARN · resourcesResourceManager: allocationNodeManagers: containersMapReduce · workMap: emit key–value pairsReduce: combine grouped valuesStorage, resource allocation and computation have separate responsibilities.
07 / 97

HDFS blocks and metadata

HDFS splits files into blocks and stores replicated block bytes on DataNodes. The NameNode maintains the file namespace and block-location metadata.

NameNode/logs/a → blocks A, BA → workers 1, 2; B → 2, 3DataNode 1Block ADataNode 2Blocks A + BDataNode 3Block BLocations / reports
08 / 97

HDFS read and write paths

For a read, the client asks the NameNode for block locations and then reads bytes from DataNodes. A write uses selected DataNodes in a replication pipeline.

OperationMetadata pathData path
ReadClient → NameNode: find block locationsClient ← DataNode: read block bytes
WriteClient asks NameNode to create the fileClient → DataNode → replica DataNode
ResponsibilityNameNode tracks names and locationsDataNodes hold the actual block bytes
Workload fitLarge sequential scans and writesMany tiny files add metadata overhead
09 / 97

Replication and recovery

Replicas let a client read another copy after a DataNode fails. The system can replace missing replicas while surviving copies remain available.

Before failureBlock A on workers 1 + 2Metadata records bothWorker 1 failsIts copy unavailableOther files may be affectedRead surviving copyClient reads worker 2Restore desired replication
10 / 97

YARN resources and tasks

YARN means Yet Another Resource Negotiator. Its ResourceManager handles cluster resource allocation; each application has an ApplicationMaster, and NodeManagers manage containers on workers.

ResourceManagerCluster-wide resource allocationApplicationMasterCoordinates one applicationResource grantsWorker 1 · NodeManagerContainer: allocated CPU + memoryRuns the application taskWorker 2 · NodeManagerContainer: allocated CPU + memoryRuns the application taskLaunch / track work
11 / 97

MapReduce

A mapper emits key–value pairs. The shuffle routes and groups matching keys; reducers combine their values into output records. A key identifies the group, while a value supplies the data to combine.

MapInput: red blue redEmit (word, 1)Shuffle + groupred → [1, 1]blue → [1]Reducered → 2blue → 1
12 / 97

Keys, reducers and output records

A reducer task can process several distinct keys. Partitioning determines which reducer receives a key; it does not create one reducer per word.

QuantityIllustrative caseInterpretation
Mapper output(a,1), (b,1), (a,1), (c,1), (b,1), (a,1)Six emitted records
Distinct keysa, b, cThree groups
Reducer task 0a → [1,1,1]; c → [1]Two groups in one task
Reducer task 1b → [1,1]One group in another task
Final output(a,3), (b,2), (c,1)Three output records; two reducers
13 / 97

Combiner correctness

A combiner optionally aggregates mapper output locally before the shuffle. The final answer must remain correct if combining happens zero, one or several times.

OperationSafe intermediate stateWhy
SumPartial sumsSums can be merged in any grouping
CountPartial countsAdding counts preserves total records
Mean(sum, count), then final sum / countUnweighted means lose group sizes
Counterexample[10,20] and [90]Mean of means = 52.5; true mean = 40
Correct merge(30,2) + (90,1) → (120,3)120 / 3 = 40
14 / 97

Blocks and input splits

A block is a storage unit; an input split describes logical work for a mapper. Their boundaries often relate, but the input format determines how records become mapper input.

TermRoleAvoid confusing it with
HDFS blockReplicated bytes in storageA record or a reducer group
Input splitLogical portion assigned to a mapperA permanently stored new file
RecordInput key and value read by a taskAn entire storage block
Map taskProcesses an input splitOne task for each distinct output key
15 / 97

EMR node roles and data lifetime

An EMR primary node coordinates cluster services; core nodes provide processing and HDFS storage; task nodes add processing capacity. Closing a terminal or notebook does not terminate the cluster.

ObjectWhat it doesLifetime consequence
Primary nodeCoordinates services and applicationsIts local files belong to that cluster
Core nodesRun work and hold HDFS blocksCluster HDFS is lost on cluster termination
Task nodesAdd processing capacityDo not provide persistent HDFS storage
Independent storageApproved S3 or other external locationCan preserve outputs beyond the cluster
ShutdownSave, verify, then terminate in EMRConfirm the cluster reaches its final state
16 / 97

Tutorial 1 · Hadoop word count

The handout is Lab 06. It stages Sherlock Holmes text in HDFS, runs the supplied MapReduce word-count JAR, inspects counters and reads reducer output.

hadoop fs -mkdir -p /user/wordcount/input
hadoop fs -put sherlock.txt /user/wordcount/input/
hadoop jar /usr/lib/hadoop-mapreduce/\
hadoop-mapreduce-examples.jar wordcount \
  /user/wordcount/input /user/wordcount/output
hadoop fs -cat /user/wordcount/output/part-r-00000 | head

What to explain

-put copies a local file into HDFS.

Map emits words; reduce adds counts.

The output path must be fresh for a normal rerun.

17 / 97

Tutorial 1 · Evidence and cleanup

A successful launch is only the start. Connect the input file, completed job and actual output using paths, counters and result inspection.

StageList HDFS inputCheck file and sizeExecuteJob reachescompletionInspect input/outputcountersInspectList part filesRead actual wordcountsPreserve + stopSave requiredevidenceTerminate cluster
18 / 97

Hadoop responsibilities

Which statement correctly assigns the responsibilities of HDFS, YARN and MapReduce?

A
HDFS schedules tasks; YARN stores blocks; MapReduce defines schemas.
B
HDFS stores blocks; YARN allocates resources; MapReduce processes records.
C
HDFS stores blocks; YARN groups keys; MapReduce allocates all cluster resources.
D
HDFS defines SQL tables; YARN trains models; MapReduce stores replicas.
19 / 97

Combining averages

Mapper 1 reads values 10 and 20; mapper 2 reads 90. Which intermediate output allows a reducer to calculate the correct overall mean?

A
Emit 15 and 90, then take their unweighted mean.
B
Emit 30 and 90, then divide their sum by two mappers.
C
Emit only each mapper’s largest value, then average them.
D
Emit (sum, count) pairs (30, 2) and (90, 1), then merge them.
20 / 97

Read the reducer output

The word-count job has completed. Which command displays the records in its HDFS output file?

A
hadoop fs -cat /user/wordcount/output/part-r-00000
B
hadoop fs -ls /user/wordcount/output
C
cat /user/wordcount/output/part-r-00000
D
hadoop fs -mkdir -p /user/wordcount/output
21 / 97

Spark execution

The next program answers the same kind of grouped question, but Spark builds a connected plan and executes it across partitions.

DescribeRDD functions orDataFrame expressionsRequestAn action needs outputDriver schedules workExecuteTasks process partitionsShuffle combines keys
22 / 97

RDDs and DataFrames

RDD means Resilient Distributed Dataset: distributed records transformed with functions. A DataFrame provides distributed rows with named, typed columns that Spark can use to plan structured queries.

PropertyRDDDataFrame
Example record("North", 10)region = North; amount = 10
OperationFunctions inspect each recordExpressions use named columns
StructureApplication controls record interpretationSchema defines column names and types
Use hereTrace low-level word recordsExpress sales totals and analytical queries
Common behaviourLazy transformations and distributed actionsLazy transformations and distributed actions
23 / 97

Schema and data validity

A schema states column names and types. Parsing a file successfully does not prove its values satisfy the analytical rules.

CSV textregion,amountNorth,10 · South,?Declared schemaregion STRINGamount DOUBLEInspect valuesNorth | 10.0Investigate invalid amount
24 / 97

Transformations and actions

Transformations describe a new distributed result; actions request computation. Creating a DataFrame variable does not mean Spark has materialised all its rows.

a = sales.filter("amount > 0")
b = a.select("region", "amount")
c = b.groupBy("region").sum("amount")
c.show()

Trace the four lines

a, b and c describe DataFrame plans.

show() requests the result.

The input sales DataFrame is unchanged.

25 / 97

Regional totals

Use three rows: North 10, South 20 and North 5. The required result is one summed amount per region: North 15 and South 20.

from pyspark.sql import functions as F
valid = sales.filter(F.col("amount") > 0)
totals = valid.groupBy("region").agg(
    F.sum("amount").alias("total"))
totals.show()

Expected rows · order unspecified

North | 15

South | 20

3 input sales → 2 groups; total = 35.

26 / 97

Driver, executors and manager

The driver plans and schedules work. The cluster manager allocates resources for executor processes; executors process partitions and access storage. An executor is a process, not an entire machine.

DriverPlans + schedules tasksCluster managerAllocates executor resourcesExecutor AProcesses partition tasksExecutor BProcesses partition tasksTasksStorage: executors read input and write output
27 / 97

Partitions and tasks

A partition is a slice of distributed data. In a stage, one task processes one partition; rows, partitions, tasks and machines are different counts.

Partition 0North 10South 20One taskKeep amount > 0Output partitionNorth 10South 20Partition 1North 5One taskKeep amount > 0Output partitionNorth 5
28 / 97

Shuffle and stage boundaries

A narrow operation can work locally within partitions. A shuffle redistributes data across partitions; a dependent stage uses that shuffle output to combine matching keys.

Stage 1 · local tasksStage 2 · mergeInput partition 0North 10 · South 20Input partition 1North 5North partition10 + 5 = 15South partition20North 10South 20North 5Shuffle by region
29 / 97

Lineage and recomputation

Lineage records how a computed partition was derived. Spark can recompute lost work from available inputs and resources; caching alone is not a durable backup.

Available sourcePartition 1: North 5Recorded filter operationExecutor failureComputed partition lostSource remains accessibleReplacement taskRepeat the filterNorth 5 restored
30 / 97

Word count with DataFrames

Tutorial 2 uses the same Sherlock text as Tutorial 1. Lowercase and tokenise lines, expand words into rows, remove empty words, then group and count.

from pyspark.sql import functions as F
lines = spark.read.text("/user/hadoop/input/sherlock.txt")
words = lines.select(F.explode(
    F.split(F.lower("value"), r"\W+")).alias("word"))
counts = words.filter("word != ''").groupBy("word").count()
counts.orderBy(F.desc("count"), "word").show(20)

Meaning

explode changes one line into many word rows.

groupBy(...).count() builds a lazy aggregation.

show() requests the ordered result.

31 / 97

Word count with RDDs

The RDD version exposes record functions directly. flatMap emits several tokens per input line; reduceByKey merges counts for the same word.

import re
lines = spark.sparkContext.textFile(
    "/user/hadoop/input/sherlock.txt")
words = lines.flatMap(lambda s: re.split(r"\W+", s.lower()))
counts = (words.filter(lambda w: w != "")
    .map(lambda w: (w, 1))
    .reduceByKey(lambda a, b: a + b))
counts.take(5)

Same operation, different API

map creates one pair per word.

reduceByKey sums values by key.

take requests a sample; it does not rank words.

32 / 97

Tutorial 2 · Notebook and data paths

The handout is Lab 07. A local browser connects through an SSH tunnel to Zeppelin; Spark then reads the staged file from HDFS. These are distinct connections.

Local computerBrowser: localhost:8890SSH forwards port 8890EMR primaryZeppelin notebookSpark application driverCluster storage +workersHDFS input pathExecutor tasks processrecords
33 / 97

Spark trace exercise

Start with North 10, South 20 and North 5. Predict the type of a, b, c and d, identify the action, and state whether b controls the final answer.

a = sales.filter("amount > 0")
b = a.limit(2)
c = a.groupBy("region").count()
d = c.count()

Distinctions to apply

A grouped count describes a DataFrame.

A DataFrame count action returns an integer.

Only dependencies of the action need execution.

34 / 97

Spark trace answer

a, b and c are DataFrames. d is the integer 2 because c has one row for North and one for South; c counts sales within each region, while d counts those grouped rows.

ExpressionMeaningResult
a = sales.filter("amount > 0")Keep positive salesPlan for 3 rows
b = a.limit(2)Bound a separate resultPlan for up to 2 unspecified rows
c = a.groupBy("region").count()Count sales per regionNorth 2; South 1
d = c.count()Count grouped rows2; b is not used
35 / 97

Lazy evaluation

Which statement about Spark DataFrames is correct?

A
Assigning a transformation to a variable materialises all output rows.
B
Every transformation changes the original DataFrame in place.
C
Transformations describe a result; an action requests its computation.
D
A transformation automatically saves the result outside the cluster.
36 / 97

Grouping across partitions

Partition 0 holds North 10 and South 20; partition 1 holds North 5. A query sums sales by region. Why is data movement needed for the North total?

A
Each original sale must be collected on the driver before any sum can run.
B
North contributions from different input partitions must reach the same final group.
C
Every input partition already contains the complete North total.
D
Grouping by region creates one executor process for every sale.
37 / 97

Tokenise the text

lines is a DataFrame with one text line in column value. Which expression produces one lowercase token per output row?

from pyspark.sql import functions as F
A
lines.select(F.split(F.lower("value"), r"\W+").alias("word"))
B
lines.select(F.lower("value").alias("word"))
C
lines.groupBy("value").count()
D
lines.select(F.explode(F.split(F.lower("value"), r"\W+")).alias("word"))
38 / 97

Correct and efficient pipelines

Once the query meaning is fixed, improve the work Spark performs while preserving the same answer.

CorrectnessTypes, missing valuesJoin multiplicityExecution costScan, shuffle, skewTask and memory evidenceSaved outputSuitable format + layoutRead back and verify
39 / 97

Cleaning without changing the question

NULL means a value is missing; it does not mean zero. A filter keeps only true conditions, so amount > 0 excludes both NULL and negative amounts.

Input amountamount > 0Consequence
10TrueKeep 10
NULLUnknown / NULLExcluded by this filter
−4FalseExcluded by this filter
Business meaningPositive sales vs net salesRemoving returns changes the requested quantity
ValidationCount rejected and missing rowsDo not hide lost records
40 / 97

SQL and DataFrame equivalence

A temporary view lets SQL refer to a DataFrame in the current session. Equivalent APIs must use the same population, grouping, aggregation and ordering.

sales.createOrReplaceTempView("sales")
totals_sql = spark.sql("""
  SELECT region, SUM(amount) AS total
  FROM sales
  WHERE amount > 0
  GROUP BY region
""")
totals_sql.show()

Check equivalence

Expected North 15 and South 20.

COUNT(*) would count sales, not add amounts.

A temporary view is not a saved durable table.

41 / 97

Join multiplicity

A join creates one output row for each matching pair. A non-unique lookup can multiply fact rows and inflate a later sum, even if the join code executes successfully.

SalesNorth | 10North | 5Two lookup matchesNorth | ANorth | BFour joined rows10 × A; 10 × B5 × A; 5 × BSum = 30, not 15Lookup uniqueness controls whether a join preserves or multiplies fact rows.
42 / 97

Physical plans and evidence

A physical plan shows operators Spark proposes to use. Look for scans, filters, exchanges and join strategies; runtime measurements establish what actually took time.

EvidenceWhat it revealsWhat to do with it
explain("formatted")Scan, filter, Exchange, join operatorsLocate data movement and planned work
Spark UI stage metricsTask duration, input and shuffle bytesIdentify costly stages
Task duration spreadA few tasks dominateCheck skew or resource differences
Spill metricsIntermediate work exceeds available memoryReduce data or adjust layout/resources
Broadcast joinSmall side distributed to executorsVerify size and plan; memory is finite
Execution hierarchyApplication → jobs → stages → tasksActions request jobs; shuffle boundaries separate dependent stages
43 / 97

Parallelism and partition sizing

More partitions can expose more work, but available task slots limit simultaneous tasks. Very small partitions add overhead; very large partitions can create long tasks and memory pressure.

Six tasks · two available slotsSlot 1Task 0Task 2Task 4Slot 2Task 1Task 3Task 5Three waves of work if task durations are similar; times are illustrative.
44 / 97

Skew

Skew means work is unevenly distributed. A hot grouping key can leave one task processing far more data after most other tasks have finished.

ObservationInterpretationTargeted response
One reducer has most bytesA frequent key dominates a partitionInspect key frequencies and query logic
Many tiny tasksScheduling overhead may dominateUse a more suitable partition size
Large shuffle after a joinData is redistributed widelyFilter/project early; inspect join strategy
Uniformly slow tasksCould be scan or resource limitsCheck input, CPU, I/O and spill evidence
45 / 97

Cache, spill and durability

Caching can reuse a computed dataset across actions; it costs memory and may spill or be evicted. A write to independent storage serves a different purpose: preservation.

prepared = sales.filter("amount > 0").cache()
prepared.count()
prepared.groupBy("region").sum("amount").show()
prepared.groupBy("region").count().show()
prepared.unpersist()

When reuse helps

The first action can populate the cache.

Later actions may reuse cached partitions.

One use may not repay caching cost.

46 / 97

Parquet and output layout

Parquet is column-oriented: readers can select needed columns and may skip row groups using statistics. Partition directories can exclude unrelated paths; many tiny files add overhead.

ChoiceBenefitQualification
Column selectionRead fewer fieldsUse only needed columns in the query
Directory partition predicateSkip unrelated partition locationsFilter must match a useful partition key
Row-group statisticsSkip impossible matchesReader support and data layout matter
repartition(n)Redistribute data and change partitioningNormally adds a shuffle
coalesce(n)Often reduces partitions with less movementCan reduce parallelism; usually no rebalance
47 / 97

Fair performance comparisons

Compare the same input, output rules and measured work. Changing the answer, using a warm cache in only one run, or skipping output verification invalidates a speed claim.

RecordReason
Input paths, bytes and row countsEstablish the same workload
Configuration and cache treatmentExplain resource and reuse differences
Action that triggers the full workLazy plan construction is not execution time
Repeated times and their spreadOne run may be noisy
Stage metrics and checked outputConnect speed changes to work and correctness
48 / 97

Tutorial 3 · Adult data pipeline

The handout is Lab 08. Read Adult records with an explicit schema, register adult_data, compare SQL with DataFrames, chart aggregates, then save Parquet.

Read + inspectTyped adult.csv rowsCheck occupation +hoursClean +aggregateNormalise missingmarkerAverage hours byoccupationCompare + chartMatch population +calculationPlot the aggregatedresultWrite + verifyParquet datasetdirectoryRead back schema +rows
49 / 97

Tutorial 3 · Normalise before filtering

The lab reader ignores leading whitespace. Trim occupation and compare with the normalised marker "?"; a literal " ?" test can miss the intended missing-value rule.

from pyspark.sql import functions as F
adult = spark.table("adult_data")
clean = adult.withColumn("occupation", F.trim("occupation"))
result = (clean.filter("occupation IS NOT NULL AND occupation != '?'")
    .groupBy("occupation")
    .agg(F.avg("hours_per_week").alias("avg_hours")))
result.write.mode("overwrite").parquet(
    "/user/hadoop/results/occupation_hours")

Check the numerical rule too

The lab SQL truncates to two decimal places.

The DataFrame mean keeps full precision.

Compare equal rules, or state the rounding difference.

50 / 97

Tutorial 3 · Verify saved data

A directory listing proves that files exist; reading them back tests whether the saved dataset is interpretable and has the expected structure and values.

saved = spark.read.parquet(
    "/user/hadoop/results/occupation_hours")
saved.printSchema()
saved.count()
saved.orderBy("occupation").show()

Evidence to retain

Check one row per retained occupation.

Inspect averages and excluded records.

Preserve required output outside cluster HDFS.

51 / 97

Cached and durable data

Which statement correctly distinguishes a cached DataFrame from a durable saved dataset?

A
Caching supports reuse, but output must be independently preserved to survive cluster termination.
B
Calling cache immediately writes a permanent copy to object storage.
C
Cached partitions cannot be lost when an executor fails.
D
Any successful write to cluster HDFS survives cluster termination.
52 / 97

Join multiplicity

Sales contains North 10 and North 5. A lookup contains two matching North rows. An inner join uses region, then sums amount. What total does it produce?

A
15, because a join always preserves the original total.
B
20, because only the first North sale is duplicated.
C
30, because both sales match both lookup rows.
D
The join must fail because lookup keys are duplicated.
53 / 97

Normalise missing markers

An occupation value may be "?", " ?" or SQL NULL. Which filter keeps known occupations after leading and trailing spaces are removed?

from pyspark.sql import functions as F
clean = adult.withColumn("occupation", F.trim("occupation"))
A
clean.filter("occupation != ' ?'")
B
clean.filter("occupation IS NOT NULL AND occupation != '?'")
C
clean.filter("occupation IS NULL OR occupation = '?'")
D
clean.filter("occupation IS NOT NULL")
54 / 97

Spark ML and evaluation

Prepared rows become a model only after a prediction target, feature contract and evaluation boundary are defined.

DefineOne prediction per entityFeatures available in timeLearnFit preparation + modelUse training dataEvaluate + reuseChoose with validationTest once; save fittedworkflow
55 / 97

Learning tasks and prediction unit

Classification predicts a category; regression predicts a number; clustering groups similar records without target labels. Define what one row and one prediction represent before selecting features.

TaskExample outputRequired interpretation
ClassificationChurn: yes / noWhich event and positive class?
RegressionPredicted future spendWhich units and prediction horizon?
ClusteringSimilarity-group identifierNo target label used for fitting groups
Prediction unitOne customer at a defined timePrevent repeated-entity or future-data leakage
56 / 97

Training, validation and test

Training learns parameters, validation chooses settings, and test estimates performance after those choices are fixed. The split must represent deployment: random, chronological or separated by entity.

BoundaryUseMistake to avoid
TrainingFit imputation, encoding, scaling and modelLearning preparation from the full dataset
ValidationChoose features, model and thresholdRepeatedly tuning to test outcomes
TestEvaluate the frozen final workflowCalling tuned results an untouched test
Time / entity splitKeep future or related records outRandomly mixing records that share information
Very large dataChoose enough representative holdout casesFixed percentages are not mandatory; rare cases still matter
57 / 97

Leakage

Leakage occurs when training or selection uses information unavailable for a real prediction. High scores can result from future outcomes, overlapping customers or preparation fitted across the split.

Leakage routeExampleCorrection
Future informationCancellation reason predicts churnUse only fields available before prediction
Repeated entitySame customer appears in train and testSplit by customer when predicting new customers
Fitted preparationScaler learns statistics from all rowsFit on training; transform held-out rows
Model selectionChoose the model with the best test scoreSelect on validation; reserve the test
58 / 97

Features and labels

Spark ML commonly expects one numeric features vector and a numeric label. Feature order, category mapping and missing-value treatment form a contract shared by training and inference.

Raw customertenure = 12charges = 70contract = MonthPreparationCast + handle missingEncode categoryAssemble fixed feature orderModel inputfeatures = [12,70,1,0,…]label: defined outcome
59 / 97

Estimator, Transformer and Pipeline

An Estimator learns from data through fit() and returns a fitted Model. A Transformer applies transform() to rows; a Pipeline preserves the ordered sequence of preparation and model stages.

ObjectMethodEffect
StringIndexer / StandardScalerfit(training)Learn category order or statistics
Fitted indexer / scalertransform(new_rows)Apply the learned mapping or statistics
VectorAssemblertransform(rows)Build a vector in the specified column order
LogisticRegressionfit(training)Learn weights from features and labels
PipelineModeltransform(test)Run the complete fitted workflow without refitting
60 / 97

Categorical encoding

StringIndexer assigns category indices; these indices are labels, not meaningful distances. OneHotEncoder gives categories separate vector positions; choose an explicit policy for unseen values.

InputIntermediate representationStudy point
Contract = MonthIndex such as 0Mapping depends on fitted indexer
Contract = YearIndex such as 11 is not twice as much contract as 0
One-hot representationSeparate category positionsVector size depends on encoder settings
Unknown contractConfigured error or retained unknown categoryUse the same fitted policy at inference
Churn labelExplicit No → 0, Yes → 1Never assume automatic index 1 means Yes
61 / 97

Numeric and text preparation

Fit numeric statistics and text vocabularies on training data, then reuse them for held-out rows. Inverse document frequency (IDF) reduces the weight of terms that occur in many documents.

Input issuePreparationConsequence
Missing numeric valuesCount and investigate; reject or fit an imputerDropping rows changes the evaluated population
Different numeric scalesFit a StandardScaler on training vectorsApply those same statistics to held-out rows
Text fieldsTokenise and build term vectors; optionally weight with IDFVocabulary and IDF statistics belong inside the training boundary
New prediction rowsApply the saved fitted preparationPreserve feature order, types and unknown-category policy
62 / 97

Fit the complete workflow

Split rows before learning preparation. This condensed numeric example assumes clean tenure and MonthlyCharges columns and explicitly maps Yes to positive class 1.

from pyspark.sql import functions as F
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import LogisticRegression
labelled = clean.withColumn("label",
    (F.col("Churn") == "Yes").cast("double"))
train, test = labelled.randomSplit([0.7, 0.3], seed=1234)
assembler = VectorAssembler(inputCols=["tenure", "MonthlyCharges"],
                            outputCol="features")
pipeline = Pipeline(stages=[assembler, LogisticRegression()])
fitted = pipeline.fit(train)
predictions = fitted.transform(test)

Pipeline boundary

Learned preparation also belongs inside the pipeline.

Use separate validation data when choosing settings.

A seed helps repeatability; it does not prevent leakage.

63 / 97

Logistic regression outputs

Logistic regression converts a weighted feature score into a class probability. A decision threshold converts a score into a predicted class; changing the threshold changes errors without retraining the weights.

Column or quantityMeaningInterpretation
rawPredictionModel score representationUsed by a suitable evaluator
probabilityEstimated class-probability vectorCheck which position is the positive class
predictionClass chosen by the decision ruleDepends on the threshold
Lower positive thresholdMore cases predicted positiveRecall can rise while false positives rise
Class probabilityA model estimateDoes not establish causation or certainty
64 / 97

Confusion matrix

Use class 1 = churn. True positives are actual churners predicted positive; false negatives are actual churners missed by the model.

Six customers · threshold 0.5 · positive = churnPredicted churnPredicted no churnActual churnActual no churnTP = 2Churners foundFN = 1Churners missedFP = 1False alarmsTN = 2Non-churners correct
65 / 97

Accuracy, precision and recall

For TP = 2, FP = 1, FN = 1 and TN = 2, accuracy, precision and recall are all 2/3. Their equal numbers here hide different meanings.

MetricFormulaWhat it asks
Accuracy(TP + TN) / all = 4 / 6What fraction of all decisions were correct?
PrecisionTP / (TP + FP) = 2 / 3Among predicted churners, how many churned?
RecallTP / (TP + FN) = 2 / 3Among actual churners, how many were found?
Imbalance example95 non-churners out of 100Always predicting no churn gives 95% accuracy and zero churn recall
ChoiceMatch costs and positive classA high overall accuracy can hide poor minority detection
66 / 97

ROC AUC

ROC AUC measures how well scores rank positives above negatives across thresholds. It is not the fraction of correct classifications at a single threshold.

Scores in a small exampleInterpretation
Positive scores: 0.9, 0.4Two actual positives
Negative scores: 0.8, 0.2Two actual negatives
Correctly ordered pairs0.9 beats both; 0.4 beats 0.2 → 3 of 4
ROC AUC = 0.75Ranking quality; ties contribute half credit
At threshold 0.5One positive missed and one negative flagged
Precision–recall viewOften informative when positives are rare; check deployment prevalence
67 / 97

Model and threshold selection

Choose settings with validation data, then evaluate the fixed choice on test data. Cross-validation repeats fitting across folds; cost grows with parameter choices, folds and pipeline stages.

BaselineSimple referencemodelSame rows + targetValidationCompare settingsChoose metric +thresholdTestFreeze all decisionsReport honest held-out resultReuseSave fittedPipelineModelCheck reloadedpredictions
68 / 97

Tutorial 4 · Customer churn

The handout is Lab 09. Cast numeric fields, investigate missing TotalCharges, build features, train logistic regression, and evaluate predictions on held-out customers.

Lab stepMeaningRevision clarification
Numeric castingConvert tenure and charge fieldsCount invalid / excluded rows; assess resulting population
Feature preparationfeatures vector + labelDefine positive label and preserve feature order
Split and fitLearn using training customersFit learned preparation after splitting
EvaluationBinaryClassificationEvaluatorareaUnderROC is ROC AUC, despite the lab variable accuracy
ReuseApply fitted preparation and modelSave the whole workflow, not only classifier weights
69 / 97

ML boundary exercise

A team fits an imputer and category indexer on every customer, selects a threshold using test scores, and reports areaUnderROC = 0.91 as 91% accuracy. Identify the three problems.

PreparationFull dataset usedBefore the splitSelectionThreshold chosenUsing test outcomesReportingROC AUC = 0.91Called 91% accuracy
70 / 97

ML boundary answer

Preparation leaked held-out information, test data was used for selection, and a ranking metric was mislabelled as classification accuracy. Each flaw changes how the result can be trusted.

ProblemRepair
Preparation fitted on all rowsSplit first; fit learned stages only on training
Threshold chosen on testChoose on validation; evaluate the frozen choice on untouched test
ROC AUC called accuracyReport ROC AUC as ranking quality; compute accuracy separately if needed
Saved classifier onlyPersist the fitted preparation and classifier together
71 / 97

Learning and applying parameters

Which statement correctly distinguishes an Estimator from a fitted Transformer in Spark ML?

A
An Estimator only changes column names; a Transformer always learns new parameters.
B
Both must be fitted again whenever a held-out dataset is scored.
C
An Estimator is a metric; a Transformer is a train/test split.
D
An Estimator learns through fit; its fitted model applies learned parameters through transform.
72 / 97

Selecting a threshold

A churn team compares ten thresholds on the test set and reports the best test F1 score. Which correction best protects the final evaluation?

A
Choose the threshold on validation data, freeze it, then evaluate once on an untouched test set.
B
Keep choosing on test data, but rename the score ROC AUC.
C
Choose the threshold with the highest training F1 and report that as the test score.
D
Combine training and test rows so that the final score uses more data.
73 / 97

Name the evaluation metric

In the churn tutorial, evaluator.evaluate(predictions) returns 0.91 with the configuration below. What does 0.91 represent?

from pyspark.ml.evaluation import BinaryClassificationEvaluator
evaluator = BinaryClassificationEvaluator(metricName="areaUnderROC")
A
Exactly 91% of customers have the correct predicted class.
B
Precision is 91% at the current classification threshold.
C
ROC AUC, which measures ranking performance across thresholds.
D
The model predicts that every customer has a 91% churn probability.
74 / 97

Hive and analytical SQL

The final topic exposes stored records as tables. Correct metadata and query semantics determine whether the SQL result represents the intended data.

FilesTyped fields need a formatStorage location mattersCatalogTable schema + locationsPartitions are metadata tooSQL + executionFilter rows; group; joinCheck and preserve results
75 / 97

Hive, Spark SQL and storage

Hive and Spark SQL can query distributed data through SQL, but have separate planning and execution paths. The Hive execution engine depends on release and configuration; a SQL query does not imply Hive runs on Spark.

ComponentRoleVerification
Metastore / catalogNames, schemas, formats and locationsDESCRIBE FORMATTED; inspect table properties
Hive planner + driverResolve and plan HiveQLInspect the query plan
Configured Hive engineExecute distributed operatorsSET hive.execution.engine; record release
Spark SQLSpark planning and executionInspect its own plan and catalog configuration
StorageContains the record bytesCheck files and interpreted rows
76 / 97

External table definition

Schema-on-read interprets existing records using a declared schema and format. The MovieLens file has four tab-separated columns; CREATE TABLE registers how to read them.

CREATE EXTERNAL TABLE ratings (
  user_id INT,
  movie_id INT,
  rating INT,
  rating_ts INT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LOCATION '/user/hive/data/movielens/';

Four fields in order

user_id, movie_id, rating, rating_ts.

LOCATION is a dataset directory.

Creating metadata does not copy or validate every record.

77 / 97

Metadata, validity and lifecycle

A table can exist while its delimiter, column order or location is wrong. Managed and external table types have different lifecycle defaults, but release and purge properties can change deletion behaviour.

CheckReason
DESCRIBE FORMATTED ratingsConfirm schema, format, type and location
SELECT * FROM ratings LIMIT 5Inspect interpreted values, not only metadata
COUNT(*), null counts, rating rangeDetect missing fields, duplicates and invalid values
Directory contentsUnrelated or duplicated files can change counts
Before DROP TABLEInspect type, purge settings and release; external is not a backup
78 / 97

Rows, groups and ordering

WHERE filters input rows, GROUP BY forms groups, aggregate functions calculate group values, and HAVING filters those groups. ORDER BY defines ranking; LIMIT alone does not mean top.

RowsFROM + JOINWHERE keeps truerowsGroupsGROUP BY movie_idCOUNT(*) +AVG(rating)Group selectionHAVING testsaggregatesMinimum count /averageRank + returnORDER BY count DESCLIMIT requestednumber
79 / 97

Movie ratings example

Use six ratings: movie 10 has 5 and 3; movie 20 has 4, 5 and 3; movie 30 has 5. These values distinguish popularity, average score and sample size.

MovieRatingsCountAverage
105, 324.0
204, 5, 334.0
30515.0
InterpretationMost rated: movie 20Highest mean: movie 30Different questions
80 / 97

WHERE and HAVING

Removing low ratings before averaging changes the quantity. To keep movies whose average is at least 4, aggregate all intended rating rows first and test the group with HAVING.

SELECT movie_id, COUNT(*) AS n, AVG(rating) AS avg_r
FROM ratings
GROUP BY movie_id
HAVING COUNT(*) >= 2 AND AVG(rating) >= 4
ORDER BY movie_id;

Six-row example result

10 | 2 | 4.0

20 | 3 | 4.0

Movie 30 fails the minimum count.

WHERE rating >= 4 would remove movie 10's rating 3.

81 / 97

COUNT and NULL

COUNT(*) counts rows; COUNT(column) counts only rows with a non-null value in that column. AVG also ignores null inputs, so report the denominator behind an average.

Rating inputCOUNT(*)COUNT(rating)AVG contribution
5115
NULL10Ignored
3113
Result3 rows2 values(5 + 3) / 2 = 4
82 / 97

Left joins and right-side filters

An inner join removes unmatched left rows; a left join keeps them with null right fields. A right-side WHERE condition can then remove those retained rows.

Six ratings; lookup has 10 and 20Output consequence
Inner join on movie_id5 rows; movie 30 disappears
Left join with unique lookup6 rows; movie 30 has NULL title
Left join with two lookup rows for 108 rows; movie 10 ratings are duplicated
WHERE m.title = 'Alpha' after LEFT JOINKeep only matching Alpha rows; null titles do not pass
AND m.title = 'Alpha' inside ONAll left ratings remain; other right titles become NULL
83 / 97

Storage partitions and pruning

A Hive partition maps a partition-key value to a location. Register the locations correctly; a predicate on that key can exclude other directories before data-level filtering.

Query predicatedt = '2026-10-01'Read selected columnsPartition dt = Oct 01Read matching locationPartition dt = Oct 02Skip this locationPartition dt = Oct 03Skip this locationCatalog partition values must match stored directories.
84 / 97

Nested data and shared catalogs

A STRUCT contains named fields; an ARRAY contains elements. A compatible SerDe, or serializer/deserializer, decodes stored records so nested fields can be queried.

SELECT text, author.name, author.followers
FROM posts
WHERE author.followers > 1000000;

Interpretation

author is a STRUCT with name and followers.

A matching JSON SerDe and record layout are required.

Shared catalogs need compatible types, formats and permissions.

85 / 97

Tutorial 5 · MovieLens analysis

The handout is Lab 10. Stage one intact MovieLens 100K ratings file, register the external table, verify COUNT(*) = 100000, and answer the specified grouped questions.

SELECT movie_id, COUNT(*) AS n, AVG(rating) AS avg_r
FROM ratings
GROUP BY movie_id
HAVING COUNT(*) >= 100 AND AVG(rating) > 4.0
ORDER BY avg_r DESC, movie_id
LIMIT 5;

Match the stated task goal

At least 100 ratings, despite the “few reviews” heading.

Average strictly greater than 4.0.

For most-rated movies, order by count instead.

86 / 97

Tutorial 5 · Checks and interpretation

One query counts rating events; another counts events within each rating value. Keep rows, movie groups and result limits distinct, and preserve query evidence before shutdown.

SELECT COUNT(*) FROM ratings;

SELECT rating, COUNT(*) AS n
FROM ratings
GROUP BY rating
ORDER BY rating;

Checks

100000 is expected for one intact MovieLens 100K file.

Sum of per-rating counts should equal total rows.

Unexpected counts may signal duplicate files or parsing errors.

87 / 97

Schema-on-read

What does creating an external Hive table over existing files establish?

A
Proof that every stored record is valid and unique.
B
Metadata describing how to interpret records at the declared location.
C
A mandatory copy of every record into a new managed directory.
D
Proof that the query will execute using Spark.
88 / 97

Filtering rows or groups

Movie 10 has ratings 5 and 3. The task is to keep movies with at least two ratings and an average of at least 4. Which approach gives the intended decision?

A
Apply WHERE rating >= 4, then require COUNT(*) >= 2.
B
Apply LIMIT 2 before grouping, then keep every resulting movie.
C
Keep the movie because MAX(rating) >= 4.
D
Group all intended ratings, then apply HAVING COUNT(*) >= 2 AND AVG(rating) >= 4.
89 / 97

Select qualifying movies

For the MovieLens task, which clause completes this query to retain movies with at least 100 ratings and an average strictly greater than 4.0?

SELECT movie_id, COUNT(*) AS n, AVG(rating) AS avg_r
FROM ratings
GROUP BY movie_id
A
HAVING COUNT(*) >= 100 AND AVG(rating) > 4.0
B
HAVING COUNT(*) > 100 AND AVG(rating) >= 4.0
C
HAVING COUNT(*) >= 100 OR AVG(rating) > 4.0
D
HAVING COUNT(*) >= 100 AND MAX(rating) > 4.0
90 / 97

Connections across the five topics

The same evidence chain applies to every tool: define the data, interpret it correctly, execute the intended work, then verify and preserve the result.

Storage + metadataHDFS blocks / table schemaFind and interpret recordsExecutionYARN / Spark / Hive engineResources and data movementCorrect resultSQL rules / ML boundaryCounts, metrics and savedevidence
91 / 97

Keep the quantities separate

Many errors come from using the same word for different levels. State exactly what is being counted, stored, learned or returned.

DistinctionMeaning
Row / group / partition / taskRecord; matching-key set; data slice; unit of stage work
Block / split / output fileStorage unit; logical mapper input; written dataset part
Schema / validationHow values are interpreted; whether they satisfy rules
Transformation / actionDescribe a result; request its computation
Cache / durable saveReusable computed partitions; independently preserved output
fit / transform / evaluateLearn parameters; apply parameters; measure outcomes
92 / 97

Pipeline exercise

Sales are North 10, South 20, North 5 and South −4. The required answer is net revenue per region, including returns. A lookup has two rows for North. The team filters amount > 0, joins the lookup, caches the result and reports success after writing to HDFS. Diagnose the result.

Requested quantityNet revenueReturns remain in scopeImplemented pipelinePositive filterDuplicate-key joinReported evidenceCache + HDFS writeNo read-back or independentcopy
93 / 97

Pipeline exercise answer

Correct net totals are North 15 and South 16. The positive filter incorrectly removes the return, and the duplicate North lookup doubles its contributions.

IssueConsequenceRequired correction
amount > 0South becomes 20 instead of 16Keep returns under the net-revenue rule
Two North lookup matchesNorth becomes 30 after the joinResolve lookup multiplicity or define intended one-to-many logic
CacheReuse only; not independent preservationWrite to approved independent storage
Successful HDFS writeFiles exist, but values may be wrongRead back; verify 2 rows and total 31
Cluster terminationCluster HDFS disappearsPreserve and verify required results first
94 / 97

What each tutorial demonstrates

The tutorial result should establish both execution and meaning. A screenshot of a ready cluster or a directory alone does not prove the calculation is correct.

TutorialPractical resultKnowledge it demonstrates
1 · HadoopMapReduce word counts + countersHDFS paths, keyed grouping, reducer output
2 · SparkInteractive ordered word countsTunnel, DataFrame plan, action, shuffle
3 · PipelineAdult aggregates saved as ParquetSchema, cleaning, API equivalence, read-back
4 · MLChurn predictions + named metricFeatures, fit/transform, split and evaluation
5 · HiveMovie counts, averages and HAVINGSchema-on-read, SQL groups, query meaning
95 / 97

A reliable analytical result

Preserve the chain from source records to the final claim. Correctness, execution evidence and interpretation need to agree.

DefineOne row / target /scopeTypes and businessrulesComputeSuitable engine +resourcesControl joins andmovementCheckExpected counts andvaluesHonest held-outmetricsPreserveCode + data + fittedworkflowVerified independentoutput
96 / 97

Lecture and tutorial references

This revision consolidates the current lecture decks and the five supplied lab handouts. Tutorial numbering 1–5 on the contents page corresponds to original handouts Lab 06–10.

ReferenceOriginal resource
Topic 1Big Data, Hadoop and MapReduce · 80 slides
Topics 2 + 3Spark · 60 slides
Topic 4Spark ML · 62 slides
Topic 5Hive · 61 slides
Tutorials 1–506-Lab-Hadoop; 07-Lab-Spark; 08-Lab-Pyspark; 09-Lab-ML; 10-Lab-HIVE