Appendix C — Assignment C: Pandas

Instructions

  1. The assignment is worth 100 points. See Canvas for the due date. There is an optional bonus question worth 10 points, so you can score a maximum of 110 (out of 100) points.

  2. You may discuss approaches with classmates, but write your own code and explanations. Put Python computations in Code cells and write the final answer to each question in a Markdown cell.

  3. Download the Assignment C project kit. Extract it and move the extracted assignment-c folder into stat303-1/assignments/, next to assignment-a and assignment-b. Check that you did not create a nested assignment-c/assignment-c/ folder. Open stat303-1 in VS Code, open assignments/assignment-c/assignment_c.ipynb, and select the course .venv as its kernel. Pandas is already installed there from Assignment B.

  4. This page contains the complete assignment instructions. The starter notebook contains numbered answer spaces and supplied code. Keep those sections in order, answer every numbered part below, and add your name to the author line at the top of the notebook.

  5. The data files are in the kit’s data/ folder. Read them with relative path strings such as "data/social_indicator.txt"; the notebook runs from the assignment-c folder, so these paths work without changes.

  6. You are not allowed to use a for loop or any other kind of loop in this assignment.

  7. Submit one file, assignment_c.html, to the Assignment C submission in Canvas. Prepare and Submit the HTML at the end of this page explains how to render it.

  8. Three of the 100 points are for properly formatting the assignment. The breakdown is as follows:

    • Must be an HTML file rendered using Quarto (1 pt).
    • There aren’t excessively long outputs of extraneous information (e.g. no printouts of entire data frames without good reason, there aren’t long sections of commented-out code, etc.) (1 pt).
    • Final answers of each question are written in Markdown cells, and there is no unnecessary / redundant code or text (1 pt).

  9. Generative AI. You may use generative AI tools (ChatGPT, Claude, GitHub Copilot, and similar) to understand concepts, interpret error messages, and debug code you have written. You may not use them to produce the code or written answers you submit. The test: you should be able to explain every line you submit and reproduce it without assistance. Disclose any use in the AI Use Disclosure section (Question 4). Honest disclosure earns points and carries no penalty on its own; undisclosed use is an academic integrity violation under the course syllabus.

When you place the kit, your assignments folder will look like this:

stat303-1/
├── .venv/
└── assignments/
    ├── assignment-a/
    ├── assignment-b/
    └── assignment-c/         ← the extracted kit
        ├── assignment_c.ipynb
        ├── README.md
        └── data/
            ├── conditions.csv
            ├── patients.csv
            ├── social_indicator.txt
            └── stat303_survey.csv

Points Summary

Section Points
1. GDP per capita and social indicators 34
2. GDP per capita vs social indicators 49
3. Medical data 12
4. AI Use Disclosure 2
5. Formatting (instruction 8 above) 3
Total 100
Bonus: survey data cleaning (optional) 10

1. GDP per Capita and Social Indicators (34 points)

1.1 Read the data (4 points)

Read the file data/social_indicator.txt with Python. Set the first column as the index when reading the file. How many observations and variables are there in the data?

1.2 Strongest and weakest correlations (4 points)

Which variables have the strongest and weakest correlations with GDP per capita? Note that lifeFemale and lifeMale are the female and male life expectancies respectively.

Judge a correlation as strong or weak by its magnitude only.

1.3 Male economic activity (4 points)

Does the male economic activity (in the column economicActivityMale) have a positive or negative correlation with GDP per capita? Did you expect the positive/negative correlation? If not, why do you think you are observing that correlation?

1.4 Rank of the US (4 points)

What is the rank of the US amongst all countries in terms of GDP per capita? Which countries lie immediately above, and immediately below the US in the ranking in terms of GDP per capita? The country having the highest GDP per capita ranks 1.

Note that:

  1. The US is mentioned as United States in the data.

  2. The country with the highest GDP per capita will have rank 1, the country with the second highest GDP per capita will have rank 2, and so on.

Hint: rank()

1.5 Top 20 on every indicator (6 points)

Which country or countries rank among the top 20 in terms of each of these social indicators - economicActivityFemale, economicActivityMale, gdpPerCapita, lifeFemale, lifeMale? For each of these social indicators, the country having the largest value ranks 1 for that indicator.

Hint:

  1. Use rank(). Note that this method is different from the method given in the hint of the previous question. This method is of the DataFrame class, while the one in the previous question is of the Series class. This part of the hint is just for your understanding. You don’t need to write any code in this part.

  2. Using rank(), get the DataFrame consisting of the ranks of countries on each of the relevant social indicators (one line of code).

  3. In the DataFrame obtained in (2), filter the rows for which the maximum rank is less than or equal to 20 (one line of code).

1.6 Worst ranking of the US (6 points)

On which social indicator among economicActivityFemale, economicActivityMale, gdpPerCapita, lifeFemale, lifeMale, illiteracyFemale, illiteracyMale, infantMortality, and totalfertilityrate does the US have its worst ranking, and what is the rank? Note that for illiteracyFemale, illiteracyMale, and infantMortality, the country having the lowest value will rank 1, in contrast to the other social indicators.

1.7 Countries ahead of the US on every indicator (6 points)

Find all the countries that have a lower GDP per capita than the US, despite having lower illiteracy rates (for both genders), higher economic activity (for both genders), higher life expectancy (for both genders), and lower infant mortality rate than the US.

2. GDP per Capita vs Social Indicators (49 points)

This section continues with the data from Section 1. For the questions below, assume that all numeric columns, except GDP per capita, are social indicators.

2.1 Continent column (8 points)

Use the column geographic_location to create a new column called continent. Merge the values of the geographic_location column appropriately to obtain 6 distinct values for the continent column – Asia, Africa, North America, South America, Europe and Oceania. Drop the column geographic_location. Print the first 5 observations of the updated DataFrame.

Hint:

  1. Use value_counts() to see the values of geographic_location. The code if 'Asia' in 'something' will return True if ‘something’ contains the string ‘Asia’, for example, if ‘something’ is ‘North Asia’, the code with return True. This part of the hint is just for your understanding. You don’t need to write any code in this part.

  2. Apply a lambda function on the Series geographic_location to replace a string that contains ‘Asia’ with ‘Asia’, replace a string that contains ‘Europe’ with ‘Europe’, and replace a string that contains ‘Africa’ with ‘Africa’. This will be a single line of code.

  3. Rename the column geographic_location to continent.

2.2 Sort and drop columns (4 points)

Sort the column labels lexicographically. Drop the columns region and contraception. Print the first 5 observations of the updated DataFrame.

Hint: sort_index()

2.3 Countries in each continent (4 points)

Find the percentage of the total countries in each continent.

Hint: One line of code with value_counts() and shape

2.4 Highest GDP per capita (4 points)

Which country has the highest GDP per capita? Let us call it country \(G\).

2.5 African country closest to country \(G\) (19 points)

We need to find the African country that is the closest to country \(G\) with regard to social indicators. Perform the steps in 2.5(a)–2.5(c).

2.5(a) Standardize the social indicators (6 points)

Standardize each of the social indicators to a standard normal distribution so that all of them are on the same scale (remember to exclude GDP per capita from social indicators).

Hint:

  1. For scaling a random variable to standard normal, subtract the mean from each value of the variable, and divide by its standard deviation.

  2. Use the apply method with a lambda function to scale all the social indicators to standard normal.

  3. The above (1) and (2) together is a single line of code.

2.5(b) Manhattan distances (8 points)

Compute the Manhattan distance between country \(G\) and each of the African countries, based on the scaled social indicators.

Hint:

  1. Broadcast a Series to a DataFrame

  2. The Manhattan distance between two points \((x_1, x_2, ..., x_p)\) and \((y_1, y_2, ..., y_p)\) is \(|x_1 - y_1| + |x_2 - y_2| + ... + |x_p-y_p|\), where \(|.|\) stands for absolute value (for example, \(|-2| = 2; |3| = 3\)).

2.5(c) Closest African country (5 points)

Identify the African country, say country \(A\), with the least Manhattan distance to country \(G\).

2.6 Distance and GDP per capita (6 points)

Find the correlation between the Manhattan distance from country \(G\) and GDP per capita for African countries.

2.7 Emulating country \(G\) (4 points)

Based on the correlation coefficient in 2.6, do you think African countries should try to emulate the social characteristics of country \(G\)? Justify your answer.

3. Medical Data (12 points)

Read the data sets data/conditions.csv and data/patients.csv. Suppose we are interested in studying patients with prediabetes condition. Do not drop or compute any missing values. In conditions.csv, the patient IDs are stored in column PATIENT, and the medical conditions are stored in column DESCRIPTION. In patients.csv, the patient IDs are stored in column Id.

3.1 Prediabetes patient IDs (4 points)

Print the patient IDs of all the patients with prediabetes condition.

3.2 Prediabetes patients (4 points)

Make a subset of the data with only prediabetes patients. How many prediabetes patients are there?

Hint: .isin()

3.3 Share of healthcare expenses (4 points)

What proportion of the total HEALTHCARE_EXPENSES of all the patients correspond to the HEALTHCARE_EXPENSES of prediabetes patients?

4. AI Use Disclosure (2 points)

In the AI Use Disclosure section of your notebook, state whether generative AI tools were used to complete any part of this assignment.

  • If no AI was used, write: “I did not use generative AI tools on this assignment.”

  • If AI was used, you must specify:

    1. Tool(s) used (e.g., ChatGPT, GitHub Copilot, Claude, etc.)
    2. How they were used (e.g., explaining a concept, interpreting an error message, debugging code you wrote)
    3. Where they influenced your work (which questions/sections)
    4. Edits & verification you made (how you checked correctness, what you modified)

AI Use Template

  • Used AI? Yes / No
  • Tool(s):
  • Purpose / How used (1–3 sentences):
  • Scope (which questions/sections):
  • Edits & verification (what you changed and how you checked correctness):

Example (for illustration only): - Used AI? Yes - Tools: ChatGPT (free) - How used: Asked what the axis argument does in a pandas method, and for help debugging a KeyError when selecting columns. - Scope: 1.5, 2.2. - Edits & verification: Rewrote the selection in my own style; checked the result by comparing row counts before and after.

Grading (2 points): - 2 points = a clear statement that you did not use AI, or a complete, specific and honest disclosure (tools named, usage and verification described) - 1 point = a disclosure that is vague or missing required details - 0 points = no statement at all, or a disclosure that is misleading

Bonus: Clean the Survey Data (10 points, optional)

This is an optional question with no partial credit. You will get points only if your solution is completely correct. We advise you to attempt it only when you are done with the rest of the assignment.

The file data/stat303_survey.csv holds responses to a survey of STAT 303-1 students. In this question, we’ll work to clean this data a bit. As with every question, you are not allowed to use a for loop or any other loop.

The starter notebook supplies this code, which reads the data and cleans the column names:

survey_data = pd.read_csv("data/stat303_survey.csv")
new_col_names = ['parties_per_month', 'smoke', 'weed', 'introvert_extrovert', 'love_first_sight', 'learning_style', 'left_right_brained', 'personality_type', 'social_media', 'num_insta_followers', 'streaming_platforms', 'expected_marriage_age', 'expected_starting_salary', 'fav_sport', 'minutes_ex_per_week', 'sleep_hours_per_day', 'how_happy', 'farthest_distance_travelled', 'fav_number', 'fav_letter', 'internet_hours_per_day', 'only_child', 'birthdate_odd_even', 'birth_month', 'fav_season', 'living_location_on_campus', 'major', 'num_majors_minors', 'high_school_GPA', 'NU_GPA', 'age', 'height', 'height_father', 'height_mother', 'school_year', 'procrastinator', 'num_clubs', 'student_athlete', 'AP_stats', 'used_python_before', 'dominant_hand', 'childhood_in_US', 'gender', 'region_of_residence', 'political_affliation', 'cant_change_math_ability', 'can_change_math_ability', 'math_is_genetic', 'much_effort_is_lack_of_talent']
survey_data.columns = list(survey_data.columns[0:2])+new_col_names

Check the datatype of the variables using the dtypes attribute of the Pandas DataFrame object. You will notice that only two variables are numeric. However, if you check the first few observations of the data with the function head() you will find the there are several more variables that seem to have numeric values.

Write a function that accepts a Pandas Series (or a column of a Pandas DataFrame object) as argument, and if the datatype of the Series is non-numeric, does the following:

  1. Checks if at least 10 values of the Series contain a digit in them.

  2. If at least 10 values are found to contain a digit, then:

    A. Eliminate the characters ~, +, and , from all the values of the Series.

    B. Convert the Series to numeric (with coercion if needed).

  3. If at least 10 values are NOT found to contain a digit, then:

    A. If the values of the Series are ‘Yes’ and ‘No’, then replace ‘Yes’ with 1 and ‘No’ with 0. The Series datatype must change to numeric as well.

    B. If the values of the Series are ‘Agree’ and ‘Disagree’, then replace ‘Agree’ with 1 and ‘Disagree’ with 0. The Series datatype must change to numeric as well.

Apply the function to each column of survey_data using the apply() method. Save the updated DataFrame as survey_data_clean. Then run the supplied check:

survey_data_clean.describe().loc['mean',:]

It should print out the mean values of 28 numeric columns in the data survey_data_clean. The variables that you should see as numeric are given in the supplied list numeric_columns (this is just to check your work).

Note that your function must be general, i.e., it must work for any other dataset as well. This means, you cannot hard code a column name, or anything specific to survey_data in the function.

Prepare and Submit the HTML

Restart the kernel, run all cells in order, resolve unexpected errors, and save. Render from the assignment folder. The terminal starts in stat303-1, so first move it there with cd assignments and then cd assignment-c, and check the current directory. Then run:

quarto render assignment_c.ipynb --to html

Open assignment_c.html and check that your code, output, and written answers are visible. Upload it to Assignment C in Canvas.

This step is graded through the formatting points in instruction 8.