Web Scraping

Much of empirical social science now depends on data collected from the web. This workshop uses correspondence experiments, which send fictitious job applications to measure discrimination in hiring (Bertrand & Mullainathan 2004; Kline, Rose & Walters 2022), as a running example of large-scale automated data collection.

Topics

1. Correspondence experiments as a running example
2. Browser automation
3. Handling failures: site changes, slow pages, tracking progress, and site security
4. Scraping job portals
5. Cleaning job titles and matching them to O*NET occupations
6. Analyzing voicemail responses

Materials

• Lecture slides (PDF)
• Hands-on exercises on GitHub

Past Workshop Offerings

• UC Love Data Week, UCLA 2026
• Summer Institute of Computational Social Science (SICSS), UCLA 2025