Data Wrangling with tidyverse: A Practical Guide

R
Tidyverse
Data Cleaning
Learn how to efficiently wrangle messy datasets using dplyr, tidyr, and readr with real-world survey data examples.
Author

Pawan Jangra

Published

June 15, 2025

Introduction

After years of working with messy survey data — from PLFS datasets to field-collected ODK forms — I’ve come to appreciate how the tidyverse ecosystem makes data wrangling not just efficient, but almost enjoyable. This post shares the patterns I use most often.

Setting Up

Code 
library(tidyverse)
library(knitr)

# Simulate a messy survey dataset
set.seed(2025)
n <- 200

raw_survey <- data.frame(
  id         = 1:n,
  age        = sample(c(18:80, NA), n, replace = TRUE, 
                      prob = c(rep(1, 63), rep(5, 1))),
  income     = paste0("₹", round(runif(n, 5000, 150000), -2)),
  gender     = sample(c("M", "F", "male", "female", "MALE", "1", "2", NA), 
                      n, replace = TRUE),
  state      = sample(c("HR", "PB", "UP", "BR", "MH", NA), n, replace = TRUE),
  employment = sample(c("Employed", "Unemployed", "Self-Employed", 
                         "employed", "NA", NA), n, replace = TRUE)
)

glimpse(raw_survey)
Rows: 200
Columns: 6
$ id         <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, …
$ age        <int> 64, 47, 21, 48, 68, 21, 76, 23, 58, 53, 44, 79, 59, NA, 46,…
$ income     <chr> "₹114500", "₹10500", "₹62900", "₹31200", "₹128400", "₹13450…
$ gender     <chr> "F", "M", NA, "1", "female", NA, "1", NA, "F", "female", "f…
$ state      <chr> "PB", "MH", "UP", "MH", "UP", NA, "MH", "UP", NA, "UP", "UP…
$ employment <chr> "Employed", "NA", "employed", "Employed", "employed", "Empl…

Step 1: Standardise Text Variables

One of the most common issues in field surveys is inconsistent coding. Here’s how to fix it cleanly:

Code 
clean_survey <- raw_survey |>
  # Standardise gender
  mutate(
    gender_clean = case_when(
      str_to_lower(gender) %in% c("m", "male", "1") ~ "Male",
      str_to_lower(gender) %in% c("f", "female", "2") ~ "Female",
      .default = NA_character_
    ),
    # Clean income: remove ₹ symbol and convert to numeric
    income_num = as.numeric(str_remove(income, "₹")),
    # Standardise employment
    employment_clean = case_when(
      str_to_lower(employment) == "employed"      ~ "Employed",
      str_to_lower(employment) == "unemployed"    ~ "Unemployed",
      str_to_lower(employment) == "self-employed" ~ "Self-Employed",
      .default = NA_character_
    )
  )

# Check results
clean_survey |>
  count(gender_clean) |>
  kable(caption = "Gender distribution after cleaning")
Gender distribution after cleaning
gender_clean n
Female 68
Male 102
NA 30

Step 2: Handle Missing Values

Code 
# Check missingness
miss_summary <- clean_survey |>
  summarise(across(everything(), ~sum(is.na(.)))) |>
  pivot_longer(everything(), names_to = "variable", values_to = "missing") |>
  mutate(pct_missing = round(missing / n * 100, 1)) |>
  arrange(desc(missing))

kable(miss_summary, caption = "Missing value summary")
Missing value summary
variable missing pct_missing
employment_clean 67 33.5
state 42 21.0
employment 38 19.0
gender 30 15.0
gender_clean 30 15.0
age 10 5.0
id 0 0.0
income 0 0.0
income_num 0 0.0

Step 3: Create Summary Tables

Code 
# Age-Income analysis by gender
summary_table <- clean_survey |>
  filter(!is.na(gender_clean), !is.na(income_num)) |>
  group_by(gender_clean, state) |>
  summarise(
    n          = n(),
    mean_age   = round(mean(age, na.rm = TRUE), 1),
    mean_income = round(mean(income_num, na.rm = TRUE)),
    median_income = median(income_num, na.rm = TRUE),
    .groups = "drop"
  ) |>
  arrange(state, gender_clean)

kable(summary_table, 
      caption = "Summary statistics by gender and state",
      format.args = list(big.mark = ","))
Summary statistics by gender and state
gender_clean state n mean_age mean_income median_income
Female BR 13 52.2 74,515 70,000
Male BR 18 56.2 83,744 86,600
Female HR 6 52.0 84,517 77,550
Male HR 21 49.7 90,119 101,100
Female MH 9 53.2 65,900 52,600
Male MH 10 64.4 78,220 85,850
Female PB 11 53.0 65,036 64,900
Male PB 19 48.9 81,774 74,800
Female UP 16 50.2 79,781 80,450
Male UP 11 45.5 86,891 81,400
Female NA 13 49.8 60,362 37,600
Male NA 23 50.3 77,752 86,600

Step 4: Visualise the Clean Data

Code 
clean_survey |>
  filter(!is.na(gender_clean), !is.na(employment_clean), !is.na(income_num)) |>
  ggplot(aes(x = income_num, fill = gender_clean)) +
  geom_density(alpha = 0.6, adjust = 1.5) +
  facet_wrap(~employment_clean) +
  scale_x_continuous(labels = scales::label_number(prefix = "₹", big.mark = ",")) +
  scale_fill_manual(values = c("Male" = "#2563eb", "Female" = "#e11d48")) +
  labs(
    title = "Income Distribution by Gender and Employment Status",
    x = "Monthly Income (₹)", y = "Density", fill = "Gender"
  ) +
  theme_minimal(base_size = 11) +
  theme(
    strip.background = element_rect(fill = "#f1f5f9", color = NA),
    strip.text = element_text(face = "bold"),
    panel.grid.minor = element_blank(),
    legend.position = "top"
  )

Income distribution by gender and employment status

Key Takeaways

The tidyverse pipeline (|>) allows you to chain data transformations in a readable, logical sequence. The key principles I follow:

  1. Never modify raw data — always create a new clean_ object
  2. Standardise early — fix encoding, case, and symbols before any analysis
  3. Document decisions — use # comments to explain why you made choices
  4. Check your work — use count() and glimpse() frequently

In my next post, I’ll cover how to handle survey weights using the survey package in R.