This chapter reviews how to organize, access, and transform data using the built-in Python structures covered in STAT 201. It builds on Python Basics. Use your existing Python setup; the executable examples need only Python and its standard library.
Our guiding question is: What information does this structure preserve, and what might an operation change or lose? This question matters when preparing data for exploration and analysis.
Predict the answers before running any code. Use the answers to decide which sections need your attention.
values[-1] and values[1:3] for values = [10, 20, 30, 40]?[20, 20, 18]?a = [1, 2], b = a, and b.append(3), what is a?result after values = [3, 1, 2] and result = values.sort()?list(zip(["A", "B"], [10])) contain?dict([("A", 10), ("A", 20)])?40 and [20, 30]. Review Accessing and Updating Values.[1, 2, 3], because both names refer to the same list. Review Shared References and Copying.None; values itself becomes [1, 2, 3]. Review Sorting and Key Functions.("A", 10). Review Iteration and Unpacking.10 for key "A" is overwritten. Review Working with Structured Records.Choose a structure according to the operations and information you need, rather than memorizing syntax alone.
| Structure | Order | Mutable? | Duplicates | Access | Typical use |
|---|---|---|---|---|---|
list |
Positional order | Yes | Allowed | Index or slice | A sequence of readings or records |
tuple |
Positional order | No | Allowed | Index or slice | A record with a fixed arrangement of fields |
str |
Character order | No | Allowed | Index or slice | Text |
set |
No guaranteed order | Yes | Removed | Membership; no indexing | Unique identifiers and overlap checks |
dict |
Insertion order | Yes | Keys unique; values may repeat | Key | Labeled fields or identifier-based lookup |
A tuple’s immutability prevents replacing its elements; it does not make a nested mutable object immutable. Dictionary keys and set elements must be hashable. Strings and integers are common choices. A tuple is hashable only when all its elements are hashable: (101, "STAT303") works, but ([101], "STAT303") does not.
readings = [20, 20, 18]
station_location = (42.05, -87.68)
station_name = "North"
unique_readings = set(readings)
station = {"id": "N01", "name": station_name}
print(readings)
print(sorted(unique_readings)) # Sort for a predictable display, not arrival order.
print(station_location)
print(station)[20, 20, 18]
[18, 20]
(42.05, -87.68)
{'id': 'N01', 'name': 'North'}Choose and explain: Which structure would you use for an ordered measurement history, a collection of unique station IDs, and a station record with named fields?
Use a list for the history, a set for unique IDs, and a dictionary for named fields. Converting the history to a set would discard repeated measurements and their original order.
Sequence indexes start at zero; negative indexes count from the end. A slice excludes its stop position. Out-of-range indexing raises IndexError, whereas a slice can safely extend beyond the end.
values = [10, 20, 30, 40]
print(values[0])
print(values[-1])
print(values[1:3])
print(values[:2])
print(values[::2])
print(values[1:100])10
40
[20, 30]
[10, 20]
[10, 30]
[20, 30, 40]append() adds one object. extend() adds each item from an iterable. These methods change the list in place and return None.
appended = [10, 20]
append_result = appended.append([30, 40])
extended = [10, 20]
extended.extend([30, 40])
print(appended)
print(extended)
print(append_result)
extended[0] = 11
print(extended)[10, 20, [30, 40]]
[10, 20, 30, 40]
None
[11, 20, 30, 40]Predict: What would extended.extend("50") add? Why is that different from extended.append("50")?
extend("50") adds two separate strings, "5" and "0", because a string is iterable. append("50") adds one string, "50". Neither operation converts text into a number.
String methods return new strings rather than changing the original string. strip() removes leading and trailing whitespace; split() separates text into pieces; a separator’s join() method joins an iterable of strings.
raw_label = " north, temperature "
parts = raw_label.strip().split(",")
clean_parts = []
for part in parts:
clean_parts.append(part.strip())
print(raw_label)
print(clean_parts)
print(" | ".join(clean_parts)) north, temperature
['north', 'temperature']
north | temperaturesplit(",") is suitable for this simple label. CSV files can contain quoted commas and other rules, so use the standard-library csv module or a data-reading library for actual CSV data.
record[key] retrieves a value or raises KeyError if the key is absent. record.get(key, default) returns the default for an absent key; without a supplied default it returns None. Membership in a dictionary tests keys, not values.
station = {"id": "N01", "reading": None}
station["unit"] = "C"
station["unit"] = "degrees C" # Update the existing key.
print(station["id"])
print("reading" in station)
print("N01" in station)
print(station.get("reading", "not recorded"))
print(station.get("height", "not recorded"))N01
True
False
None
not recordedA present key whose value is None is different from an absent key. .get() does not replace an existing None with its default.
Use sets to find unique values and compare groups. Union finds values in either set; intersection finds shared values; difference finds values in one set but not the other.
expected_ids = {"N01", "S01", "E01"}
received_ids = {"N01", "S01", "W01"}
print("All IDs:", sorted(expected_ids | received_ids))
print("Shared IDs:", sorted(expected_ids & received_ids))
print("Missing IDs:", sorted(expected_ids - received_ids))
print("Unexpected IDs:", sorted(received_ids - expected_ids))All IDs: ['E01', 'N01', 'S01', 'W01']
Shared IDs: ['N01', 'S01']
Missing IDs: ['E01']
Unexpected IDs: ['W01']Dictionary key lookup and set membership are typically fast on average; list membership searches through values. Do not confuse membership with positional indexing: values[0] is direct list access, while sets do not support indexing. For this course, choose the correct representation before considering performance.
An iterable can supply an iterator to a loop. An iterator is also iterable: it remembers its position and becomes exhausted after producing its values. A list can supply fresh iterators for repeated passes; an already exhausted iterator does not restart itself.
readings = [18.5, 21.0]
reading_iterator = iter(readings)
print(next(reading_iterator))
print(list(reading_iterator))
print(list(reading_iterator))
print(list(readings))
print(iter(reading_iterator) is reading_iterator)18.5
[21.0]
[]
[18.5, 21.0]
TrueCalling next() on an exhausted iterator raises StopIteration; a for loop handles that signal automatically. next(iterator, default) can provide a fallback instead.
enumerate()Unpacking assigns items to names. Ordinary unpacking needs the same number of names and items; starred unpacking collects remaining items into a list. enumerate() supplies a counter and an item. Its start argument changes the counter, not the underlying sequence indexes.
station_id, reading = ("N01", 18.5)
print(station_id, reading)
first, *rest = [10, 20, 30]
print(first, rest)
for row_number, label in enumerate(["North", "South"], start=1):
print(row_number, label)N01 18.5
10 [20, 30]
1 North
2 SouthFor dictionaries, use for key, value in record.items() when you need both labels and values.
zip()zip() returns an iterator of tuples paired by position. It does not match identifiers or verify that two inputs refer to the same entities. By default, it stops at the shortest input.
station_ids = ["N01", "S01", "E01"]
readings = [18.5, 21.0]
pairs = list(zip(station_ids, readings))
print(pairs)
print("Lengths match:", len(station_ids) == len(readings))
paired_ids, paired_readings = zip(*pairs)
print(paired_ids) # A tuple, not a list.
print(list(paired_readings)) # Convert explicitly if a list is needed.[('N01', 18.5), ('S01', 21.0)]
Lengths match: False
('N01', 'S01')
[18.5, 21.0]Here E01 has no partner and is omitted. If equal-length lists are required, check their lengths before pairing. Python 3.10 and later also support zip(..., strict=True), which raises ValueError for mismatched lengths when consumed. Neither approach detects IDs in different orders. The unzip pattern above also assumes there are pairs to unpack.
Predict: Pair ["S01", "N01"] with [18.5, 21.0], assuming the readings originally belonged to North then South. Will zip() detect the mismatch?
No. It produces ("S01", 18.5) and ("N01", 21.0). Equal lengths do not establish correct alignment. Preserve IDs with their readings, or explicitly match records by ID.
| Tool | Purpose | Important detail |
|---|---|---|
len() |
Count items in a sized object | An arbitrary iterator may have no length |
min(), max() |
Find extrema | Empty input raises ValueError unless an appropriate default is supplied |
sum() |
Total numerical values | An empty input returns 0 with the default starting value |
sorted() |
Create a sorted list | Consumes an iterator supplied as input |
reversed() |
Iterate in reverse | Requires a reversible object; not every iterable supports it |
all(), any() |
Combine truth tests | all([]) is True; any([]) is False |
separator.join() |
Join text items | All items must be strings |
Use a comprehension for a simple transformation or filter. Start with an ordinary loop, then compare the equivalent expression. Use a loop when several steps or branches make the comprehension difficult to read.
readings = [18.5, None, 21.0, -2.0, 0.0]
valid_readings = []
for value in readings:
if value is not None and 0 <= value <= 100:
valid_readings.append(value)
valid_comprehension = [
value for value in readings
if value is not None and 0 <= value <= 100
]
print(valid_readings)
print(valid_comprehension)[18.5, 21.0, 0.0]
[18.5, 21.0, 0.0]The expression comes first, then the iteration, then an optional filter. Square brackets build a list, braces around a single expression build a set, and braces around key: value build a dictionary.
squares = [number ** 2 for number in range(5, 9)]
labels = ["north", "south", "north"]
unique_labels = {label.upper() for label in labels}
label_lengths = {label: len(label) for label in labels}
print(squares)
print(sorted(unique_labels))
print(label_lengths)[25, 36, 49, 64]
['NORTH', 'SOUTH']
{'north': 5, 'south': 5}The set loses repetitions. The dictionary has only one value per key; repeated keys replace earlier values. Neither structure can preserve repeated records under the same key without additional design.
Write: From [" North ", "", "SOUTH", " "], create a list of stripped, lowercase, nonempty labels.
raw_labels = [" North ", "", "SOUTH", " "]
clean_labels = [label.strip().lower() for label in raw_labels if label.strip()]
print(clean_labels) # ['north', 'south']sorted() returns a new list. A list’s .sort() changes that list and returns None. If you pass an iterator to sorted(), the iterator is consumed even though a new list is returned.
values = [3, 1, 2]
ordered = sorted(values)
print(values)
print(ordered)
result = values.sort()
print(values)
print(result)
print(sorted(values, reverse=True))[3, 1, 2]
[1, 2, 3]
[1, 2, 3]
None
[3, 2, 1]The key argument accepts a function that extracts each item’s sorting value. Pass the function itself; do not call it at the argument position. Equal keys retain their original relative order: Python sorting is stable.
words = ["pear", "apple", "banana", "kiwi"]
print(sorted(words, key=len))
def last_character(word):
return word[-1]
print(sorted(words, key=last_character))
print(sorted(words, key=lambda word: word[-1]))['pear', 'kiwi', 'apple', 'banana']
['banana', 'apple', 'kiwi', 'pear']
['banana', 'apple', 'kiwi', 'pear']pear stays before kiwi when sorting by length because both have four letters and pear came first. The last-character example assumes nonempty strings.
A lambda creates a function from a single expression. It returns that expression’s value. Use it for a short key such as lambda record: record["reading"]; use def when a meaningful name or multiple steps improve clarity.
number_text = ["10", "2", "1"]
print(sorted(number_text))
print(sorted(number_text, key=int))
people = [("Chen", 22), ("Bob", 19), ("Ann", 22)]
print(sorted(people, key=lambda person: person[1]))
print(sorted(people, key=lambda person: (person[1], person[0])))['1', '10', '2']
['1', '2', '10']
[('Bob', 19), ('Chen', 22), ('Ann', 22)]
[('Bob', 19), ('Ann', 22), ('Chen', 22)]The first people sort preserves Chen-before-Ann for equal ages. The second explicitly breaks age ties alphabetically by name. Say which behavior you intend in a task.
Practice: Sort ["data", "python", "AI", "science"] alphabetically ignoring case. Then sort [-3, 1, -2, 5, 0] by absolute value, smallest first.
print(sorted(["data", "python", "AI", "science"], key=str.lower))
# ['AI', 'data', 'python', 'science']
print(sorted([-3, 1, -2, 5, 0], key=abs))
# [0, 1, -2, -3, 5]Neither task needs a lambda because an existing function already supplies the desired key.
map() transforms items and filter() selects items; both return iterators. A simple comprehension often expresses the same task more directly. functools.reduce() repeatedly combines values; prefer a clear built-in operation when one exists.
from functools import reduce
values = [1, 2, 3, 4]
print(list(map(lambda value: value ** 2, values)))
print([value ** 2 for value in values])
print(list(filter(lambda value: value % 2 == 0, values)))
print([value for value in values if value % 2 == 0])
print(reduce(lambda left, right: left * right, values)) # 24Recognizing these tools is useful; mastering them is not a prerequisite for starting the data science libraries.
A list of dictionaries keeps each observation’s fields together and gives the fields names. Nested access combines sequence indexing with dictionary lookup.
records = [
{"station": "N01", "reading": 18.5},
{"station": "S01", "reading": 21.0},
{"station": "N01", "reading": 19.0},
]
print(records[0]["reading"])
print([record["reading"] for record in records if record["station"] == "N01"])
print(sorted(records, key=lambda record: record["reading"]))
latest_by_station = {record["station"]: record["reading"] for record in records}
print(latest_by_station)18.5
[18.5, 19.0]
[{'station': 'N01', 'reading': 18.5}, {'station': 'N01', 'reading': 19.0}, {'station': 'S01', 'reading': 21.0}]
{'N01': 19.0, 'S01': 21.0}The dictionary retains only the last encountered reading per station, not every observation. Calling it “latest” is justified only if records are in time order. Here we know only their list order.
A set can track names already seen while a list preserves the desired output order.
names = ["Alice", "Bob", "Alice", "Charlie", "Bob", "Dave"]
seen = set()
unique_names = []
first_positions = []
for index, name in enumerate(names):
if name not in seen:
seen.add(name)
unique_names.append(name)
first_positions.append((index, len(name)))
print(unique_names)
print(sorted(first_positions, key=lambda pair: pair[1], reverse=True))['Alice', 'Bob', 'Charlie', 'Dave']
[(3, 7), (0, 5), (5, 4), (1, 3)]The positions refer to the original list, not positions after deduplication. For equal lengths, stable sorting preserves encounter order. In real data, identical names do not necessarily identify the same person; use a suitable identifier when available.
The following illustrative course-enrollment dataset has one tuple per enrollment:
(student_id, student_name, course_code, grade)
A student can appear in several courses. Repeated student IDs are therefore expected; they do not automatically indicate duplicate records. Keep the dataset below as the input for all tasks.
# Illustrative course-enrollment records (not real student data).
students = [
(101, "Alice", "STAT303", 85),
(102, "Bob", "STAT301", 92),
(103, "Charlie", "STAT303", 78),
(104, "Diana", "STAT301", 95),
(105, "Eve", "STAT303", 88),
(106, "Frank", "STAT304", 82),
(107, "Grace", "STAT301", 90),
(108, "Hank", "STAT304", 87),
(109, "Ivy", "STAT303", 91),
(101, "Alice", "STAT301", 88),
(101, "Alice", "STAT304", 82),
(101, "Alice", "STAT302", 86),
(102, "Bob", "STAT303", 79),
(102, "Bob", "STAT304", 85),
(102, "Bob", "STAT302", 83),
(103, "Charlie", "STAT301", 91),
(103, "Charlie", "STAT304", 84),
(103, "Charlie", "STAT302", 80),
(104, "Diana", "STAT303", 89),
(104, "Diana", "STAT304", 93),
(104, "Diana", "STAT302", 96),
(105, "Eve", "STAT301", 87),
(105, "Eve", "STAT304", 90),
(105, "Eve", "STAT302", 85),
(106, "Frank", "STAT303", 76),
(106, "Frank", "STAT301", 81),
(106, "Frank", "STAT302", 79),
(107, "Grace", "STAT303", 84),
(107, "Grace", "STAT304", 88),
(107, "Grace", "STAT302", 91),
(108, "Hank", "STAT303", 80),
(108, "Hank", "STAT301", 83),
(108, "Hank", "STAT302", 82),
(109, "Ivy", "STAT301", 94),
(109, "Ivy", "STAT304", 89),
(109, "Ivy", "STAT302", 92),
(110, "Jack", "STAT303", 83),
(110, "Jack", "STAT301", 86),
(110, "Jack", "STAT304", 79),
(110, "Jack", "STAT302", 81),
(111, "Karen", "STAT303", 96),
(111, "Karen", "STAT301", 92),
(111, "Karen", "STAT304", 94),
(111, "Karen", "STAT302", 98),
(112, "Leo", "STAT303", 72),
(112, "Leo", "STAT301", 75),
(112, "Leo", "STAT304", 78),
(112, "Leo", "STAT302", 74)
]Work without the solutions first. Use only built-in structures and ordinary Python.
maximum - minimum).(student_id, course_code) always be enough if students could retake courses?The outer structure is a list, each record is a tuple, and the fields have types int, str, str, and int.
student_ids = {student_id for student_id, name, course, grade in students}
course_codes = {course for student_id, name, course, grade in students}
print(len(students), len(student_ids), len(course_codes))
# 48 12 4Sets are appropriate for counting unique identifiers, but not for preserving all enrollments.
grades = [grade for student_id, name, course, grade in students]
if grades:
low, high = min(grades), max(grades)
print(low, high, high - low)
print(sum(grades) / len(grades))
else:
print("No enrollment grades")The minimum is 72, the maximum is 98, the range is 26, and the enrollment-level mean is 85.6875. Each enrollment contributes one grade. If students had different enrollment counts, this would give more weight to students with more enrollments than a mean of student-level means would.
stat303_records = [record for record in students if record[2] == "STAT303"]
ranked = sorted(stat303_records, key=lambda record: (-record[3], record[0]))
for student_id, name, course, grade in ranked:
print(student_id, name, grade)There are 12 STAT303 records. Karen (111) has the highest grade, 96; Leo (112) has the lowest, 72. Negating the grade sorts grades downward while the second key keeps student IDs ascending for ties.
Student ID alone repeats, so later records overwrite earlier records for the same student. A (student_id, course_code) key works for this dataset, but retakes would require another distinguishing field, such as term or attempt ID. Alternatively, a dictionary could map each student ID to a list of enrollment records.
Before moving on, confirm that you can:
You have completed the STAT 201 foundation review. Continue to Jupyter Notebooks in VS Code to begin the setup activities, then create a project environment for third-party libraries.