Big Data Analysis in Social Sciences

Exciting breakthrough for social science: Simulating behavior with AI agents. It might not be perfect, but it's an interesting and scalable way to pilot ideas and experiments before rolling out to the world. A recent study presents a generative agent architecture, powered by LLMs, capable of replicating the attitudes and behaviors of over 1,000 individuals. By anchoring simulations in in-depth interviews, this approach offers unprecedented precision in understanding human responses across social, political, and economic domains. Key Advancements: 1) Realistic Simulation: The generative agents are based on transcripts from two-hour semi-structured interviews with participants selected to represent the U.S. population. 2) Comprehensive Evaluation: The agents were tested using established metrics such as the General Social Survey, the Big Five Personality Inventory, and well-known behavioral economic games like the dictator game and public goods game. 3) Dynamic Modeling: These agents leverage full interview transcripts, enabling contextually rich responses in forced-choice surveys and multi-stage decision-making tasks. 4) Normalization of Accuracy: By comparing agents’ predictions to individuals' own response consistency over two weeks, the study establishes robust benchmarks for measuring accuracy. Findings: 1) High Predictive Accuracy: Agents demonstrated the ability to predict individual responses with accuracy normalized to the consistency of the participants' own behavior over time. 2) Population-Level Insights: Beyond individual accuracy, agents replicated population-level treatment effects and effect sizes observed in large-scale social science experiments. 3) Adaptive Interaction: The architecture supports diverse applications, ranging from social policy testing to modeling organizational dynamics. 4) Applications and Access: To facilitate broader research while safeguarding privacy, the study introduces a two-pronged access system: aggregated responses for general research and restricted individual data for approved studies. Not perfect, but helps generative agents be accessible and responsibly used. By blending qualitative richness with quantitative rigor, this generative agent architecture is a major advancement in social science research, opening doors to predictive simulations that could revolutionize how we study and influence human behavior. #ArtificialIntelligence #SocialScience #HumanBehavior #GenerativeAI #LLMs #ResearchInnovation #BehavioralEconomics

I did an analysis on 17 years of U.S. data to understand a pressing social issue: teenage pregnancy. The result is a visual, insight-driven story that reveals not just national trends but the disparities still affecting many communities. As a data scientist passionate about real-world impact, I examined teenage birth rates across more than 3,000 U.S. counties from 2003 to 2020. I used techniques in data wrangling, exploratory data analysis, and correlation analysis and created clear, colorblind-friendly visualizations to communicate the findings. In this project, I uncovered: ✅ A consistent national decline in teen births, especially after 2010 ✅Regional disparities, with Southern states lagging behind ✅County-level extremes that point to areas needing targeted intervention ✅Patterns in data uncertainty, revealed through credible intervals This project is more than a portfolio piece. It's a demonstration of how data can guide smarter decisions, inform public health efforts, and tell stories that matter. If you're interested in data science that goes beyond algorithms to create awareness and drive change, you’ll enjoy this read. Here’s the full blog post: https://lnkd.in/dQWzBAFJ #datascience #teenpregnancy #publichealth #eda #datavisualization #socialimpact #python #womenindatascience #machinelearning #mediumblog

Survey-based data collection is becoming increasingly difficult, and many of us are turning to passively collected big data (e.g., GPS trajectories, Uber/Lyft or bikeshare trips) as alternatives. In fact, I’ve heard that many transportation agencies have stopped conducting household travel surveys, relying instead on big data products (e.g., StreetLight, Replica) to guide decision-making. But what can these big datasets truly offer—and what risks should we be mindful of? In two recently published articles with my students and collaborators, we examine how big data can open new research opportunities while also introducing biases. https://lnkd.in/e45ChVjK This Health&Place paper focuses on using GPS data for understanding food access: ✅ GPS data capture food acquisition patterns at unprecedented spatiotemporal resolution. ⚠️ But they show coverage/representation biases, significantly undercount trips, and are highly sensitive to algorithm design choices. https://lnkd.in/eibmjP6D This Transportation Research Part D paper focuses on using large-scale micromobility trip data to understand transit and micromobility integration: ✅ Inferring first-/last-mile trips from large-scale micromobility data enables citywide FM/LM analysis and modeling. ⚠️ But inference assumptions inevitably introduce biases, which can distort findings and policy insights. I will always take findings based solely on opportunistically collected big data with a grain of salt, and I believe evidence #triangulation is essential to ensure robustness and accuracy. To me, the most promising path forward is combining big data with traditional small-data approaches to balance between breadth and depth. 🙏 Grateful to my wonderful students and collaborators for conducting the research together: DUANYA LYU, Yiheng Qian, Luyu Liu, Catherine Campbell, Yuxuan Zhang