7  NumPy and pandas in a Real Workflow

In NumPy Fundamentals, you learned to reason about array shapes, broadcasting, and axes, and in the pandas chapters before it you learned to build, transform, and summarize labeled tables. In practice you rarely use one library alone. A working analysis reads and cleans a labeled table with pandas, hands a well-defined block of numbers to NumPy when an array operation expresses the calculation better, and then returns the result to pandas so it can be labeled, checked, and reported.

This chapter walks that path one stage at a time. Each stage gets its own small, self-contained example, so you can see what each library is responsible for and what has to be true at the two moments where data crosses between them.

By the end of this chapter you should be able to:

7.1 Set Up Your Chapter Files

Download the NumPy and pandas Workflow practice kit. Extract stat303-pandas-numpy-workflow inside the chapters folder of your stat303-1 course folder, open stat303-1 in VS Code, and select the course environment. The terminal opens in stat303-1, so move it into the kit with cd chapters and then cd stat303-pandas-numpy-workflow before you render the activity.

stat303-1/
├── .venv/
├── assignments/
└── chapters/
    └── stat303-pandas-numpy-workflow/
        ├── workflow_examples.ipynb
        ├── activity07.ipynb
        └── README.md

Run workflow_examples.ipynb for the lesson and complete activity07.ipynb for your own activity report. Use stat303-pandas-numpy-workflow as the notebook working folder. This chapter ships no data files: the worked examples write the small messy CSV they then clean, and the activity notebook builds its own inputs in its first cell.

import numpy as np
import pandas as pd

print("NumPy version:", np.__version__)
print("pandas version:", pd.__version__)
NumPy version: 2.5.3
pandas version: 3.0.5

Environment check: This chapter uses NumPy and pandas together. If an import fails, check the selected notebook kernel first. If a package is missing, activate your project environment and run this command in its terminal:

python -m pip install numpy pandas

7.2 One Stack, Two Layers

From pandas: Positions versus Labels in the NumPy chapter drew the contrast between the two libraries one operation at a time. The relationship behind that contrast is simpler than it looks: a pandas Series is a NumPy array plus an index. It is worth stating as a picture, because it explains every recommendation in this chapter:

                       your analysis
                             │
  ┌──────────────────────────▼──────────────────────────┐
  │  pandas                                             │
  │  labels, one dtype per column, missing values,      │
  │  dates, text, reading and writing files             │
  │  → load, clean, and report                          │
  └──────────────────────────┬──────────────────────────┘
                             │  each numeric column is one of these
  ┌──────────────────────────▼──────────────────────────┐
  │  NumPy                                              │
  │  one contiguous block of same-typed numbers,        │
  │  addressed by position                              │
  │  → arithmetic, broadcasting, axes, matrix math      │
  └─────────────────────────────────────────────────────┘

The two libraries are not competitors, and “NumPy or pandas?” is not really a question about speed. It is a question about what your data looks like at this moment in the workflow. Data arriving from a file is labeled and mixed: that is pandas-shaped. Data in the middle of a calculation is a block of numbers: that is NumPy-shaped. Results going to a human need names back on them: pandas-shaped again.

Let’s confirm the “a column is an array” claim rather than taking it on faith. np.shares_memory reports whether two objects are looking at the same block of memory:

scores = pd.Series([88, 92, 79, 95, 84])
arr = scores.to_numpy()

print("type of the Series :", type(scores).__name__)
print("type of .to_numpy():", type(arr).__name__)
print("same memory block? :", np.shares_memory(scores, arr))
type of the Series : Series
type of .to_numpy(): ndarray
same memory block? : True

True means no copying happened — .to_numpy() handed back the very array that was already holding the Series’ numbers. This is why df["revenue"] + df["tax"] is not a Python loop: pandas passes those two existing arrays straight to NumPy’s compiled addition routine.

It also means vectorized pandas code is already running at NumPy speed. Keep that sentence in mind for When Conversion Actually Buys Speed, because it is the reason converting to NumPy often buys you far less than people expect.

7.3 The Workflow, Stage by Stage

A small analysis project has a predictable shape. Here is which layer owns each stage:

Stage What you are doing Library Why that one
1. Load read a CSV or Excel file pandas NumPy has no column names and no way to store a different type per column
2. Inspect .shape, .dtypes, .head(), .describe() pandas the labels are the whole point of inspecting
3. Clean repair dtypes, handle missing values, parse dates pandas NumPy has no missing-value marker for integers and no date parser
4. Derive compute new columns from existing ones pandas, vectorized already delegates to NumPy; no conversion needed
5. Compute broadcasting, matrix products, per-axis statistics, sorting along an axis NumPy this is exactly what arrays are built for
6. Report put labels back on, round, display, save pandas numbers need names before anyone can read them

Notice the shape of that column: pandas at both ends, NumPy in the middle. Most real workflows spend most of their lines of code in pandas and most of their CPU time in NumPy. The rest of this chapter walks one dataset through all six stages.

7.4 Stages 1-3: Load, Inspect, Clean (pandas)

Real files are messy. To make that concrete, the cell below writes out a small CSV with three problems that show up constantly in practice: a price column with currency symbols, a word (unknown) where a count should be, and a genuinely empty field. Normally you would receive a file like this from someone else; here we create it so the notebook is self-contained.

raw_csv = """date,store,product,units,unit_price
2026-01-02,North,Notebook,12,$5.00
2026-01-02,North,Pen,20,$2.00
2026-01-02,South,Notebook,unknown,$5.00
2026-01-03,South,Pen,15,$2.00
2026-01-03,West,Notebook,12,$5.00
2026-01-03,West,Pen,,$2.00
2026-01-04,North,Notebook,9,$5.00
2026-01-04,South,Pen,22,$2.00
"""

with open("sales_raw.csv", "w") as f:
    f.write(raw_csv)

raw = pd.read_csv("sales_raw.csv")
raw
date store product units unit_price
0 2026-01-02 North Notebook 12 $5.00
1 2026-01-02 North Pen 20 $2.00
2 2026-01-02 South Notebook unknown $5.00
3 2026-01-03 South Pen 15 $2.00
4 2026-01-03 West Notebook 12 $5.00
5 2026-01-03 West Pen NaN $2.00
6 2026-01-04 North Notebook 9 $5.00
7 2026-01-04 South Pen 22 $2.00 
raw.dtypes
date          str
store         str
product       str
units         str
unit_price    str
dtype: object

Read those dtypes carefully, because they are the reason Stage 3 exists:

  • unit_price is str, not float64. The $ made every entry text, so pandas kept the column as text. You cannot do arithmetic on it at NumPy speed — raw["unit_price"] * 2 would either fail or duplicate the characters.
  • units is also str, because the word unknown is not a number. One bad cell in 100,000 is enough to demote an entire column.
  • date is str too — just characters that happen to look like dates.

A detail worth knowing: pandas does recognize some spellings of “missing” on its own. The empty field in row 5 became NaN automatically (you can see it in the table above), and so would n/a, NA, null, or NaN — read_csv treats those as missing by default. What it cannot guess is an arbitrary word like unknown, which is exactly why pd.to_numeric(..., errors="coerce") exists.

A text column is the enemy of fast computation. Its values are Python strings rather than one contiguous block of numbers, so NumPy cannot run a compiled loop over them. Repairing dtypes is not cosmetic tidying; it is what makes the fast path available.

pandas 3.0 labels text columns str. Earlier versions labeled them object, the general-purpose dtype for anything pandas cannot store as numbers, and you will still meet object in older notebooks and in most answers online. The two names describe the same problem here: the column holds text, not numbers, and arithmetic on it is either an error or nonsense. object has not disappeared — you will see it again in Trap 1, when a whole mixed table is converted to one array.

sales = raw.copy()

# strip the currency symbol, then convert the text to real numbers
sales["unit_price"] = pd.to_numeric(sales["unit_price"].str.replace("$", "", regex=False))

# errors="coerce" turns anything unparseable (like "unknown") into NaN instead of raising
sales["units"] = pd.to_numeric(sales["units"], errors="coerce")

sales["date"] = pd.to_datetime(sales["date"])

sales.dtypes
date          datetime64[us]
store                    str
product                  str
units                float64
unit_price           float64
dtype: object
print("missing values per column:")
print(sales.isna().sum())

sales
missing values per column:
date          0
store         0
product       0
units         2
unit_price    0
dtype: int64
date store product units unit_price
0 2026-01-02 North Notebook 12.0 5.0
1 2026-01-02 North Pen 20.0 2.0
2 2026-01-02 South Notebook NaN 5.0
3 2026-01-03 South Pen 15.0 2.0
4 2026-01-03 West Notebook 12.0 5.0
5 2026-01-03 West Pen NaN 2.0
6 2026-01-04 North Notebook 9.0 5.0
7 2026-01-04 South Pen 22.0 2.0

Now decide what to do about the two missing units values. That is a judgment call about your data, not a technical one — drop the rows, fill with zero, or fill with a typical value. Here, a missing unit count most likely means the sale was not recorded, so dropping is defensible and we will say so out loud:

sales = sales.dropna(subset=["units"]).copy()
sales["units"] = sales["units"].astype("int64")   # no NaN left, so int is safe again

print("rows remaining:", len(sales))
print(sales.dtypes)
sales
rows remaining: 6
date          datetime64[us]
store                    str
product                  str
units                  int64
unit_price           float64
dtype: object
date store product units unit_price
0 2026-01-02 North Notebook 12 5.0
1 2026-01-02 North Pen 20 2.0
3 2026-01-03 South Pen 15 2.0
4 2026-01-03 West Notebook 12 5.0
6 2026-01-04 North Notebook 9 5.0
7 2026-01-04 South Pen 22 2.0

Every single operation in this section — read_csv, .str.replace, to_numeric(errors="coerce"), to_datetime, .isna(), .dropna(subset=...) — is pandas-only. NumPy has no equivalent of any of them, because none of them makes sense for a homogeneous block of numbers. There is no decision to make at Stages 1-3: you stay in pandas.

One detail worth noticing: units could only go back to int64 after the NaN values were gone. NumPy integer arrays have no way to represent “missing”, which is Trap 2 in Four Conversion Traps.

7.5 Stage 4: Derive Columns — Vectorized, Not Row-by-Row

Stage 4 is where the workflow’s real performance decision lives, and it is not “pandas or NumPy”. It is “vectorized or row-by-row”.

Eight rows will not tell us anything about speed, so let’s build a realistic 200,000-row table and give ourselves a rule with branches in it — the kind of rule that makes .apply() tempting:

Gross revenue is units x unit_price. Orders of 50+ units get 10% off, orders of 20-49 get 5% off, smaller orders get nothing. Members get an extra 5% off. Orders whose net total is under $500 pay $19.99 shipping.

rng = np.random.default_rng(303)
n = 200_000

orders = pd.DataFrame({
    "store": rng.choice(["North", "South", "West"], size=n),
    "units": rng.integers(1, 80, size=n),
    "unit_price": rng.choice([2.0, 3.0, 4.0, 5.0], size=n),
    "is_member": rng.random(size=n) < 0.30,
})

print(orders.shape)
print(orders.dtypes)
orders.head()
(200000, 4)
store             str
units           int64
unit_price    float64
is_member        bool
dtype: object
store units unit_price is_member
0 South 46 5.0 False
1 North 66 3.0 False
2 West 60 5.0 False
3 South 53 4.0 False
4 South 1 5.0 True

7.5.1 Version A: row-by-row with .apply(axis=1)

The natural first draft. Write the rule for one row, let pandas run it for every row.

%timeit itself you met in the setup chapter: it runs a line repeatedly and reports the average, because one run of fast code measures mostly noise. The timing lines here add three options to that basic form. -o returns the measurement as an object instead of only printing it, so it can be stored in a variable — t_apply.average later gives the average in seconds, which is how the comparison table at the end of this section is built. -r 3 -n 1 says “repeat 3 times, one run each” and is there for patience: this single .apply() call takes about half a second, so the default of 7 repeats of many runs apiece would keep you waiting for minutes. The faster versions below need no such limit and use the default. The IPython documentation lists every option.

def order_total_row(row):
    gross = row["units"] * row["unit_price"]
    if gross >= 250:
        rate = 0.10
    elif gross >= 100:
        rate = 0.05
    else:
        rate = 0.0
    if row["is_member"]:
        rate += 0.05
    net = gross * (1 - rate)
    return net + (0.0 if net >= 500 else 19.99)

t_apply = %timeit -o -r 3 -n 1 orders.apply(order_total_row, axis=1)
429 ms ± 1.97 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)

7.5.2 Version B: vectorized, still entirely in pandas

The same rule, expressed as whole-column operations. np.select handles the discount tiers and np.where handles the shipping rule — both from Advanced Selection Methods in the NumPy Fundamentals chapter. Note that we never call .to_numpy() here; the Series are already NumPy arrays underneath.

def order_total_pandas(df):
    gross = df["units"] * df["unit_price"]
    rate = np.select([gross >= 250, gross >= 100], [0.10, 0.05], default=0.0)
    rate = rate + df["is_member"] * 0.05          # True/False arithmetic -> 0.05 or 0.0
    net = gross * (1 - rate)
    return net + np.where(net >= 500, 0.0, 19.99)

t_pandas = %timeit -o order_total_pandas(orders)
1.16 ms ± 13.7 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

7.5.3 Version C: pull the columns out as plain NumPy arrays first

units_arr = orders["units"].to_numpy()
price_arr = orders["unit_price"].to_numpy()
member_arr = orders["is_member"].to_numpy()

def order_total_numpy(units, price, member):
    gross = units * price
    rate = np.select([gross >= 250, gross >= 100], [0.10, 0.05], default=0.0)
    rate = rate + member * 0.05
    net = gross * (1 - rate)
    return net + np.where(net >= 500, 0.0, 19.99)

t_numpy = %timeit -o order_total_numpy(units_arr, price_arr, member_arr)
997 μs ± 5.1 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
# Never trust a speedup you have not verified.
result_apply = orders.apply(order_total_row, axis=1).to_numpy()
result_pandas = order_total_pandas(orders).to_numpy()
result_numpy = order_total_numpy(units_arr, price_arr, member_arr)

print("apply vs pandas-vectorized match:", np.allclose(result_apply, result_pandas))
print("apply vs numpy-vectorized  match:", np.allclose(result_apply, result_numpy))

print()
print(f"A. .apply(axis=1)      {t_apply.average*1e3:9.2f} ms   1.0x")
print(f"B. pandas vectorized   {t_pandas.average*1e3:9.2f} ms {t_apply.average/t_pandas.average:6.0f}x faster than A")
print(f"C. numpy vectorized    {t_numpy.average*1e3:9.2f} ms {t_apply.average/t_numpy.average:6.0f}x faster than A")
print(f"                                    and {t_pandas.average/t_numpy.average:6.1f}x faster than B")
apply vs pandas-vectorized match: True
apply vs numpy-vectorized  match: True

A. .apply(axis=1)         428.66 ms   1.0x
B. pandas vectorized        1.16 ms    371x faster than A
C. numpy vectorized         1.00 ms    430x faster than A
                                    and    1.2x faster than B

Read those three numbers in the right order, because the lesson is in the gaps, not the totals:

  • A to B is an enormous jump — two or three orders of magnitude — and it required no conversion at all. We stayed in pandas the whole time.
  • B to C is a small jump. Converting to plain arrays removes some per-operation bookkeeping (index alignment and dtype checks that pandas performs on every operation), but the heavy arithmetic was already NumPy’s either way.

So the headline is: vectorizing is the win; converting is a tune-up.

7.5.4 Why .apply(axis=1) is so slow

It is worth seeing where .apply’s time actually goes, because the answer is surprising: mostly not in your function.

t_empty = %timeit -o -r 3 -n 1 orders.apply(lambda row: 0, axis=1)

print()
print(f"apply doing your real rule : {t_apply.average*1e3:8.1f} ms  ({t_apply.average/n*1e6:5.2f} us per row)")
print(f"apply doing NOTHING at all : {t_empty.average*1e3:8.1f} ms  ({t_empty.average/n*1e6:5.2f} us per row)")
print(f"fraction of apply's time that is pure overhead: {t_empty.average/t_apply.average:.0%}")
124 ms ± 2.37 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)

apply doing your real rule :    428.7 ms  ( 2.14 us per row)
apply doing NOTHING at all :    123.6 ms  ( 0.62 us per row)
fraction of apply's time that is pure overhead: 29%

A function that does literally nothing (lambda row: 0) still costs a large share of the total. That overhead is what pandas must do per row before your code runs: hand you a Series object representing the row, and then resolve each row["units"] label lookup into a position, a bounds check, and a boxed Python number. Four column lookups per row, 200,000 rows, is 800,000 label lookups — and that, not the multiplication, is the bill.

The vectorized version replaces all of it with roughly eight passes over whole arrays, each pass a single compiled loop with one dtype check for the entire column.

If you ever hit a rule that genuinely cannot be vectorized — one where each row depends on the result of the previous row, say — two cheap improvements exist, and it is worth knowing they are there:

  • df.apply(func, axis=1, raw=True) hands your function a plain NumPy array instead of a Series, skipping the label lookups. It only works when the columns involved share one numeric dtype.
  • df.itertuples() is several times faster than df.iterrows(), because it yields a lightweight tuple per row instead of building a Series.

Beyond that, libraries such as Numba can compile a Python loop down to machine code, which is a topic for a later course. Neither is a substitute for vectorizing when vectorizing is possible.

7.6 Stage 5: Hand the Numbers to NumPy

Now the part of the workflow where converting genuinely pays. Some calculations are natural on a rectangle of numbers and awkward on a labeled table. For those, .to_numpy() is not a performance trick — it is a clarity trick.

Here is a small weekly table: one row per day, one column per product, values are units sold. Every column is the same dtype, so it forms a clean numeric block.

days = ["Mon", "Tue", "Wed", "Thu", "Fri"]
products = ["Notebook", "Pen", "Folder", "Marker"]

weekly = pd.DataFrame(
    rng.integers(20, 200, size=(5, 4)),
    index=days,
    columns=products,
)
weekly
Notebook Pen Folder Marker
Mon 120 37 36 30
Tue 59 160 150 44
Wed 83 60 100 168
Thu 136 186 41 198
Fri 117 86 148 20 
mat = weekly.to_numpy()

print("dtype:", mat.dtype, "| shape:", mat.shape)
mat
dtype: int64 | shape: (5, 4)
array([[120,  37,  36,  30],
       [ 59, 160, 150,  44],
       [ 83,  60, 100, 168],
       [136, 186,  41, 198],
       [117,  86, 148,  20]])

A clean (5, 4) integer array — no labels, just numbers addressed by position. Now four calculations that are one line each in NumPy.

(a) Totals along each axis. axis=1 collapses the columns (one total per day); axis=0 collapses the rows (one total per product).

per_day = mat.sum(axis=1)       # shape (5,)  -- one number per row
per_product = mat.sum(axis=0)   # shape (4,)  -- one number per column

print("units per day    :", per_day, per_day.shape)
print("units per product:", per_product, per_product.shape)
units per day    : [223 413 411 561 371] (5,)
units per product: [515 529 475 460] (4,)

(b) Each product’s share of its own day’s total — the broadcasting pattern from the NumPy Fundamentals chapter. per_day has shape (5,), which will not line up against a (5, 4) matrix; reshaping it to (5, 1) makes it a column vector, and NumPy then stretches it across the four product columns, one factor per row.

share = mat / per_day.reshape(-1, 1)     # (5, 4) / (5, 1) -> (5, 4)

print("shapes:", mat.shape, "/", per_day.reshape(-1, 1).shape, "->", share.shape)
print("every row sums to 1?", np.allclose(share.sum(axis=1), 1.0))
share.round(3)
shapes: (5, 4) / (5, 1) -> (5, 4)
every row sums to 1? True
array([[0.538, 0.166, 0.161, 0.135],
       [0.143, 0.387, 0.363, 0.107],
       [0.202, 0.146, 0.243, 0.409],
       [0.242, 0.332, 0.073, 0.353],
       [0.315, 0.232, 0.399, 0.054]])

mat.sum(axis=1, keepdims=True) is an equivalent and slightly tidier way to get that (5, 1) shape without a separate reshape. Use whichever reads better to you.

(c) Revenue per day with a matrix product. With one price per product, revenue per day is “multiply each day’s units by the price vector and add them up” — which is exactly what the @ operator does. One symbol replaces a nested loop.

prices = np.array([5.0, 2.0, 3.0, 4.0])    # Notebook, Pen, Folder, Marker

revenue_per_day = mat @ prices             # (5, 4) @ (4,) -> (5,)

print("shapes:", mat.shape, "@", prices.shape, "->", revenue_per_day.shape)
print(revenue_per_day)

# the long way round, to show they agree
print("matches an explicit sum?", np.allclose(revenue_per_day, (mat * prices).sum(axis=1)))
shapes: (5, 4) @ (4,) -> (5,)
[ 902. 1241. 1507. 1967. 1281.]
matches an explicit sum? True

(d) Drop each day’s weakest product and average the rest. Sorting each row puts the smallest value in column position 0 every time, so slicing off that column drops the weakest product per day, with no loop anywhere.

Hint — the method you need. np.sort(a, axis=1) returns a new array in which the values of each row have been rearranged into ascending order, leaving the original array untouched; axis=0 sorts down each column instead. It is the value-returning companion of the np.argsort you used in Finding the Top-k with np.argsort(), which returns positions rather than values, and the numpy.sort reference documents every argument.

sorted_rows = np.sort(mat, axis=1)      # each row ascending
trimmed_mean = sorted_rows[:, 1:].mean(axis=1)

print("sorted rows:")
print(sorted_rows)
print()
print("mean after dropping each day's weakest product:", trimmed_mean.round(2))
sorted rows:
[[ 30  36  37 120]
 [ 44  59 150 160]
 [ 60  83 100 168]
 [ 41 136 186 198]
 [ 20  86 117 148]]

mean after dropping each day's weakest product: [ 64.33 123.   117.   173.33 117.  ]

Each of those four calculations is positional, homogeneous and numeric — the NumPy shape exactly. Writing (b), (c) or (d) as a .apply(axis=1) over the DataFrame would be slower and, more importantly, harder to read: the broadcasting and the axis are right there in the code instead of hidden inside a method name.

7.7 Stage 6: Back to pandas for the Report

Raw arrays are unreadable to anyone but the person who just computed them. Stage 6 puts the labels back on. When you rebuild a DataFrame, pass the original index and columns so the numbers land under the right names:

share_report = pd.DataFrame(share, index=weekly.index, columns=weekly.columns)
share_report.round(3)
Notebook Pen Folder Marker
Mon 0.538 0.166 0.161 0.135
Tue 0.143 0.387 0.363 0.107
Wed 0.202 0.146 0.243 0.409
Thu 0.242 0.332 0.073 0.353
Fri 0.315 0.232 0.399 0.054 
# One value per day -> a new column on the existing table.
daily_report = weekly.copy()
daily_report["units_total"] = per_day
daily_report["revenue"] = revenue_per_day
daily_report["trimmed_mean_units"] = trimmed_mean.round(2)

daily_report
Notebook Pen Folder Marker units_total revenue trimmed_mean_units
Mon 120 37 36 30 223 902.0 64.33
Tue 59 160 150 44 413 1241.0 123.00
Wed 83 60 100 168 411 1507.0 117.00
Thu 136 186 41 198 561 1967.0 173.33
Fri 117 86 148 20 371 1281.0 117.00

That assignment worked because per_day, revenue_per_day and trimmed_mean were all computed from weekly itself and nothing reordered the rows in between — so position i in each array still refers to row i of the table.

That “nothing reordered the rows” condition is not automatic, and pandas will not check it for you. It is Trap 4, and it is the one that produces wrong answers instead of error messages.

7.8 Choosing: A Decision Table

Put the whole chapter into one table. Read the middle column as “where this task belongs”, not “which library is better”.

The task in front of you Choose Why
Read or write a file pandas NumPy cannot hold a different type per column, and has no CSV/Excel reader worth using
Repair dtypes, parse dates, clean text, handle missing values pandas none of these operations exist for a homogeneous numeric block
Elementwise math or comparisons on numeric columns either — pandas is fine pandas already delegates to NumPy; converting rarely changes the timing much
A .apply(..., axis=1) doing arithmetic or branching rewrite as vectorized removes the per-row Python loop; by far the biggest win available
Broadcasting, matrix products, reshaping, sorting along an axis NumPy positional, homogeneous work — both faster and clearer as arrays
Data with more than two dimensions NumPy a DataFrame is two-dimensional by construction
The same computation repeated hundreds of times NumPy arrays pandas’ small per-operation overhead adds up once you multiply it by hundreds
Data contains NaN and you care about the answer pandas, or be careful Series.mean() skips NaN; array.mean() returns nan (Trap 2)
Output that a human will read pandas labels, alignment, rounding, and a readable display

A useful default: stay in pandas unless the data in front of you is genuinely a rectangle of same-typed numbers, in which case reach for NumPy and enjoy it.

7.9 When Conversion Actually Buys Speed

We have one measurement already: converting bought a modest improvement for a single vectorized pass (B to C in Stage 4). Let’s measure the two other things you would want to know before converting — what the conversion itself costs, and when the small win becomes a real one.

(a) How expensive is .to_numpy()?

t_conv_num = %timeit -o orders[["units", "unit_price"]].to_numpy()
t_conv_mixed = %timeit -o -r 3 -n 1 orders.to_numpy()

print()
print("numeric columns only:", orders[["units", "unit_price"]].to_numpy().dtype)
print("whole mixed frame   :", orders.to_numpy().dtype)
print()
print(f"converting 2 numeric columns : {t_conv_num.average*1e3:8.2f} ms")
print(f"converting the mixed frame   : {t_conv_mixed.average*1e3:8.2f} ms"
      f"   ({t_conv_mixed.average/t_conv_num.average:.0f}x slower, and the result is unusable)")
122 μs ± 342 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
3.74 ms ± 92.4 μs per loop (mean ± std. dev. of 3 runs, 1 loop each)

numeric columns only: float64
whole mixed frame   : object

converting 2 numeric columns :     0.12 ms
converting the mixed frame   :     3.74 ms   (31x slower, and the result is unusable)

Converting the numeric columns is cheap — often a small fraction of the computation you are about to run. Converting the whole mixed frame is both much slower and self-defeating: the object dtype means you have built a slow array of boxed Python objects, throwing away the very speed you were converting to get.

Always select the numeric columns you intend to compute with. This is the single most common conversion mistake.

(b) When does the small win become a real one?

The B-to-C gap in Stage 4 was modest because we ran the calculation once. pandas’ overhead is a small fixed cost per operation — so it disappears into the noise on one pass, and becomes significant when you multiply it by a few hundred. A parameter sweep is the classic example: try many discount thresholds and record the total revenue each one produces.

thresholds = np.arange(20, 220, 4)     # 50 candidate thresholds
print("number of thresholds:", len(thresholds))

def sweep_pandas(df, thresholds):
    gross = df["units"] * df["unit_price"]
    return [ (gross * np.where(gross >= t, 0.90, 1.0)).sum() for t in thresholds ]

def sweep_numpy(units, price, thresholds):
    gross = units * price
    return [ (gross * np.where(gross >= t, 0.90, 1.0)).sum() for t in thresholds ]

t_sweep_pd = %timeit -o -r 3 -n 1 sweep_pandas(orders, thresholds)
t_sweep_np = %timeit -o -r 3 -n 1 sweep_numpy(units_arr, price_arr, thresholds)

print()
print("results identical:", np.allclose(sweep_pandas(orders, thresholds),
                                        sweep_numpy(units_arr, price_arr, thresholds)))
print(f"pandas Series sweep : {t_sweep_pd.average*1e3:8.1f} ms")
print(f"numpy array sweep   : {t_sweep_np.average*1e3:8.1f} ms"
      f"   ({t_sweep_pd.average/t_sweep_np.average:.1f}x faster)")
number of thresholds: 50
9.19 ms ± 207 μs per loop (mean ± std. dev. of 3 runs, 1 loop each)
6.61 ms ± 68.9 μs per loop (mean ± std. dev. of 3 runs, 1 loop each)

results identical: True
pandas Series sweep :      9.2 ms
numpy array sweep   :      6.6 ms   (1.4x faster)

Same arithmetic, same answers, but the array version skips 50 rounds of index alignment and dtype checking. That is the profile of a computation worth converting for: many operations on the same numbers, with the conversion paid once, outside the loop.

7.9.1 The summary you should remember

Change you make Typical payoff When to bother
Row-by-row (.apply(axis=1)) to vectorized very large (often 100x or more) always — this is the real optimization
pandas vectorized to NumPy arrays, one pass small not worth the loss of labels on its own
pandas vectorized to NumPy arrays, inside a loop or sweep moderate and real yes, convert once before the loop
Converting the whole mixed DataFrame negative never — select the numeric columns instead

Measure first, in that order. If %timeit says your bottleneck is already a vectorized pandas operation, converting to NumPy is not the fix you are looking for; a different algorithm, or simply processing less data, is.

7.10 Four Conversion Traps

Every one of these produces a wrong answer or a silent surprise rather than a helpful error, which is why they are worth practising deliberately.

7.10.1 Trap 1: converting a mixed table gives you an object array

Covered above, but worth restating as a habit: convert df[["a", "b"]], never df, unless every column already shares one numeric dtype. A table that mixes store names with quantities has no single dtype that fits everything, so NumPy falls back to object — an array of pointers to Python objects, with none of the speed you converted to get. Check with .dtype immediately after converting — if it says object, stop and fix the selection.

7.10.2 Trap 2: NaN means different things in pandas and NumPy

with_missing = pd.Series([10.0, 20.0, np.nan, 40.0])
as_array = with_missing.to_numpy()

print("pandas  .mean() :", with_missing.mean(), "   <- skips NaN, averages 3 values")
print("numpy   .mean() :", as_array.mean(), "   <- NaN contaminates the whole result")
print("numpy nanmean() :", np.nanmean(as_array), "   <- matches pandas")
print()
print("pandas  .sum()  :", with_missing.sum(), " | numpy .sum():", as_array.sum(),
      " | np.nansum():", np.nansum(as_array))
pandas  .mean() : 23.333333333333332    <- skips NaN, averages 3 values
numpy   .mean() : nan    <- NaN contaminates the whole result
numpy nanmean() : 23.333333333333332    <- matches pandas

pandas  .sum()  : 70.0  | numpy .sum(): nan  | np.nansum(): 70.0

pandas skips missing values by default; NumPy propagates them by default. Neither is wrong, but they answer different questions. After converting, either check for NaN first or switch to the np.nan* family (np.nanmean, np.nansum, np.nanstd, np.nanmax). A whole analysis coming out as nan is usually this trap.

7.10.3 Trap 3: .to_numpy() can hand you a read-only view, not a copy

demo = pd.DataFrame({"a": [1.0, 2.0, 3.0]})
view = demo["a"].to_numpy()

print("shares memory with the DataFrame?", np.shares_memory(demo["a"], view))
print("writeable?                       ", view.flags.writeable)

try:
    view[0] = 999.0                  # writing to the array...
except ValueError as err:
    print("writing into it fails    :", err)
shares memory with the DataFrame? True
writeable?                        False
writing into it fails    : assignment destination is read-only
# Ask for a copy when you intend to modify the array.
demo2 = pd.DataFrame({"a": [1.0, 2.0, 3.0]})
safe = demo2["a"].to_numpy(copy=True)
safe[0] = 999.0

print("shares memory?", np.shares_memory(demo2["a"], safe))
print("array     :", safe)
print("DataFrame :", demo2["a"].to_numpy(), " <- untouched")
shares memory? False
array     : [999.   2.   3.]
DataFrame : [1. 2. 3.]  <- untouched

For a single numeric column, .to_numpy() usually hands back a view onto the DataFrame’s own memory rather than a copy, and np.shares_memory confirms it. pandas marks that view read-only, so an attempt to write into it raises instead of quietly editing the table behind your back. Reading is always safe; pass copy=True whenever you plan to write, as the cell above does.

Do not assume that converting several columns at once escapes this. df[["a", "b"]].to_numpy() on two float columns also shares memory and also comes back read-only. Be explicit instead of relying on when a copy happens to be made.

Before pandas 3.0 the same view was writable, so arr[0] = 999 silently changed the DataFrame it came from. That is the version of this trap you will find described in older tutorials, and the reason copy=True is worth writing even when the current version would stop you.

small = pd.DataFrame({"units": [10, 50, 30], "price": [2.0, 5.0, 4.0]})
totals = (small["units"] * small["price"]).to_numpy()    # [20., 250., 120.]

reordered = small.sort_values("units")                   # row order is now 10, 30, 50
reordered["total_WRONG"] = totals                        # pandas aligns by POSITION

reordered["total_right"] = reordered["units"] * reordered["price"]
reordered
units price total_WRONG total_right
0 10 2.0 20.0 20.0
2 30 4.0 250.0 120.0
1 50 5.0 120.0 250.0

The total_WRONG column is nonsense — 30 units at $4.00 is labelled $250.00 — and pandas raised no error, because an array has no index for pandas to align against. It simply dropped the values in row by row.

Two habits prevent this entirely:

  1. Do the extract, compute, assign back sequence without reordering or filtering the DataFrame in between.
  2. If you must reorder, re-extract the arrays afterwards rather than reusing the old ones.

And one cheap check before any positional assignment: len(arr) == len(df) catches the length mismatch case, though not the reordering case — only the habits above catch that one.

7.11 Summary Cheat Sheet

Task Code
Stage 1-3: pandas
Read a CSV pd.read_csv("file.csv")
See what you actually got df.dtypes, df.shape, df.head()
Text to numbers, bad values to NaN pd.to_numeric(df["col"], errors="coerce")
Strip a character out of a text column df["col"].str.replace("$", "", regex=False)
Parse dates pd.to_datetime(df["col"])
Count / drop missing values df.isna().sum(), df.dropna(subset=["col"])
Restore an integer dtype after cleaning df["col"].astype("int64")
Stage 4: vectorize, do not loop
Branching without .apply np.where(cond, a, b), np.select(conds, choices, default=...)
Time one line %timeit expr (add -o to capture the result)
Check two float results agree np.allclose(a, b)
Stage 5: NumPy
Numeric columns to a 2-D array df[["a", "b"]].to_numpy()
Confirm you did not get object arr.dtype
Per-row / per-column totals arr.sum(axis=1) / arr.sum(axis=0)
One factor per row (broadcasting) arr / arr.sum(axis=1, keepdims=True)
Matrix product mat @ vec
Sort along each row np.sort(arr, axis=1)
NaN-safe statistics np.nanmean(arr), np.nansum(arr)
Stage 6: pandas
Array back to a labeled table pd.DataFrame(arr, index=df.index, columns=df.columns)
One value per row as a new column df["new"] = arr (positional — check the row order)
Copy instead of a live view df["col"].to_numpy(copy=True)

7.12 Practice Activity: From Messy File to Labeled Report

Goal: Run one small dataset through all six stages of the workflow and defend each handoff between pandas and NumPy.

File: activity07.ipynb from the practice kit.

Submit: activity07.html through the final upload question in the NumPy and pandas in a Real Workflow Canvas quiz.

This section contains the complete activity instructions. The starter supplies every input and spaces to record your work. Open activity07.ipynb from the folder prepared in Set Up Your Chapter Files, using the same project environment. The worked examples and the Extended Practice below are preparation, not additional submission requirements.

The setup cell writes the messy inventory_raw.csv used in Part A, builds the large orders table used for the Part B timings, and defines the warehouse table and the unit_costs vector used in Parts C and D.

A. Load, Inspect, and Clean (Stages 1-3)

  • Replace Your Name in the opening Raw cell’s author field. Run the setup cell; it must report that inventory_raw.csv was written.
  • Read inventory_raw.csv and display .dtypes. For each column that came in as text, say in one sentence why it did.
  • Repair all three problem columns so that quantity is an integer dtype, cost is a float dtype, and restocked is a datetime dtype.
  • Report how many missing values existed and what you chose to do about them, with a one-sentence justification.

B. Derive a Column Two Ways (Stage 4)

  • Now, use the orders DataFrame. We will apply the rule: value = quantity * cost, then a bulk discount of 15% when quantity >= 40 and 8% when quantity >= 15, with no discount below that, and finally a $12.50 handling fee added to any item whose discounted value is under $200.
  • Write it first as a row function with .apply(axis=1) on the large orders table, and time it with %timeit.
  • Write it again vectorized, using np.select and np.where, and time that.
  • Verify that the two agree with np.allclose, report the speedup ratio, and say which change actually produced the speedup.

C. Hand Off to NumPy (Stage 5)

  • Convert the four numeric warehouse columns of the supplied warehouse table to a single 2D array. Confirm that the dtype is not object.
  • Calculate each warehouse’s total (one number per row) and each item’s total (one number per column), naming the axis you used for each and why.
  • Using broadcasting, calculate each cell’s share of its own row total, and verify that every row of the result sums to 1.
  • Using the @ operator and the supplied unit_costs vector, calculate the total inventory value per warehouse. Show the check that unit_costs is in the same order as your array’s columns.

D. Return to pandas, and Demonstrate One Trap (Stage 6)

  • Rebuild your share matrix as a DataFrame carrying the original row and column labels, rounded to three decimals.
  • Add your per-warehouse value as a new column on the warehouse table, and explain in one or two sentences why the positional assignment is safe in this specific case.
  • Choose either Trap 2 (NaN semantics) or Trap 4 (positional alignment) from Four Conversion Traps. Show the wrong answer and the corrected version side by side, and explain what a reader should check and why no error appears.

Render and Submit

Restart the kernel, run all cells in order, resolve unexpected errors, and save. Include your predictions, outputs, and explanations for A-D, and a short Markdown completion note. From stat303-pandas-numpy-workflow in the terminal, run:

quarto render activity07.ipynb --to html

Follow the Quarto refresher: inspect the HTML and a copy opened outside the project folder, and confirm that every cell ran without errors, every output is visible, and your name is on the report. Upload only activity07.html to the NumPy and pandas in a Real Workflow Canvas quiz; keep your notebook locally.

HTML grading (16 points): dtype repairs, the missing-value count, and a justified choice (3); both versions of the derived column with their timings, the np.allclose check, and the speedup ratio (4); the non-object conversion, both axis totals with their axes named, broadcasting with the row-sum check, and the matrix product (4); the labeled DataFrame, the column assigned back with its safety explanation, and the demonstrated trap with its fix (4); name, readable report, and completion note (1).

7.13 Extended Practice

Use a separate notebook beside your activity notebook. These longer exercises are additional practice, not requirements for the Canvas quiz. Include your code, outputs, and explanations.

Start that notebook with the cell below. It rebuilds the 200,000-row orders table and the order-total rule from Stage 4, so your notebook runs correctly from a restart without borrowing anything from the chapter notebook.

import numpy as np
import pandas as pd

rng = np.random.default_rng(303)
n = 200_000

orders = pd.DataFrame({
    "store": rng.choice(["North", "South", "West"], size=n),
    "units": rng.integers(1, 80, size=n),
    "unit_price": rng.choice([2.0, 3.0, 4.0, 5.0], size=n),
    "is_member": rng.random(size=n) < 0.30,
})

gross = orders["units"] * orders["unit_price"]
rate = np.select([gross >= 250, gross >= 100], [0.10, 0.05], default=0.0)
rate = rate + orders["is_member"] * 0.05
net = gross * (1 - rate)
orders["total"] = net + np.where(net >= 500, 0.0, 19.99)

7.13.1 Per-Store Summaries with Boolean Masks

Compute the mean order total for each store two ways: once with a pandas Boolean mask per store, and once by converting total and store to arrays and masking those. Compare the results with np.allclose, then explain which version you would put in a report and why. Which parts of the bookkeeping did you have to do by hand in the array version?

7.13.2 Per-Record Thresholds

Flag every order whose total is in the top 20% for its own store. Build one array of per-record thresholds by computing the 80th percentile within each store’s mask, then do the comparison itself as a single NumPy comparison of two arrays. Confirm that the row order of the threshold array still matches the row order of the totals array, and explain what would go wrong if it did not.

Hint — the method you need. np.percentile(values, 80) returns the value below which 80% of the observations fall, so an order is in the top 20% of its store when its total is at least that store’s own 80th percentile. The function takes an array and one or more percentages between 0 and 100; it returns nan if any input value is missing, so use np.nanpercentile() when that is a possibility; and it accepts an axis argument when you want one threshold per row or per column rather than one for the whole array. NumPy Fundamentals introduces it briefly under More Summaries: Median, Percentiles, and Range, and the numpy.percentile reference documents every argument.

7.13.3 A Conversion That Does Not Pay

Find a calculation on the 200,000-row orders table where converting to NumPy makes the complete workflow slower rather than faster, and demonstrate it with %timeit. Candidates worth trying: an operation over the store text column; converting the whole mixed DataFrame rather than selected columns; or converting, computing one cheap thing, and converting straight back. Explain where the time actually went.

7.14 Before You Move On

The mental model to carry forward is the six-stage pipeline: pandas at both ends, NumPy in the middle. You load and clean in pandas because only pandas understands messy, labeled, mixed data. You compute in NumPy when the data has become a rectangle of same-typed numbers, because that is what arrays are for. You come back to pandas to put names on the results.

And you now have the two performance facts, both measured rather than assumed: the large win is always row-by-row to vectorized, and the smaller win from pandas to NumPy arrays only becomes worth having when you operate on the same numbers many times over. Everything else is a trap to check for — object dtype, NaN semantics, shared memory, and positional alignment.

Continue to Data Visualization I, where these labeled, validated results become figures.

References: pandas and NumPy interoperability, to_numpy(), to_numeric(), NumPy broadcasting, np.select, and IPython %timeit.