Does GPT-3 predict votes better than last election's survey?
Argyle et al. (2023) showed that GPT-3, given a respondent's background, predicts how that person voted. Using their replication data, I compare GPT-3 with a logistic regression fitted on the previous election's survey.
library(tidyverse)
library(psych)
library(knitr)
# Theme settings
primary <- "#16a34a"
theme_set(theme_minimal(base_size = 16))
Argyle et al. (2023) gave GPT-3 the background characteristics of real survey respondents and asked it how each of them voted. Its predictions agreed with how those people said they voted for 85–89% of respondents in three US presidential elections, with tetrachoric correlations of 0.90 to 0.94 (a correlation between two binary variables that treats each as a cut of an underlying continuous one). The authors conclude that a language model, conditioned this way, can stand in for groups of human respondents. They call such a simulated sample a silicon sample. The term comes from this paper, and since it’s cited so much, it might be a good paper to look at critically.
I think the evidence for this conclusion is weaker than it looks, because the paper never compares GPT-3 with a simpler way of making the same predictions. In this post I make that comparison with the authors’ replication data of Study 2.
How silicon sampling works
In Study 2, Argygle et al. uses the American National Election Studies (ANES) of 2012, 2016 and 2020. For each respondent, Argyle et al. turned their answers on ten variables into a short first-person backstory. These variables are race and ethnicity, gender, age, ideology, party identification, political interest, church attendance, whether they discuss politics, how they feel about the American flag, and their state. The flag item and state were not available for 2020. GPT-3 then completed the sentence “In [year], I voted for…”, and the authors recorded the probability that the completion was the Republican candidate. A probability above 0.5 counts as a predicted Republican vote. That prediction is compared with the vote the respondent reported.
The missing baseline
Argyle et al. picked the ten variables because decades of political science show that they predict vote choice. That same research also gives a simpler way to predict votes. If we know that party identification, ideology and the other variables predict vote choice, we can fit a logistic regression with those variables and use it to predict the outcome. If that would lead to similar predictions, one can question the value of silicon sampling (at least in this context).
A regression needs data with both the predictors and the outcome. Fitting it on the same respondents it is tested on would give it an unfair advantage, because it could learn patterns specific to that sample. A more informative version fits the model on the previous election’s survey and uses it to predict the next one. The regression then never sees the outcomes it is tested on, just like GPT-3.
Argyle et al. compare GPT-3 with other language models, but not with a statistical model. So I made that comparison myself.
Data
The authors shared their data on the Political Analysis Dataverse. For each election, the vote study’s data file contains the ANES variables used in the backstories, the reported vote, and GPT-3’s probability of a Republican vote.
data_2012 <- read_csv("data/full_results_2012_2.csv")
data_2016 <- read_csv("data/full_results_2016_2.csv")
data_2020 <- read_csv("data/full_results_2020_2.csv")
The three waves use different variable names, and some variables are coded differently. I gave the variables the same names in each wave and harmonised two of them. Race and ethnicity uses a different order of categories in 2020, and 2020 combines Native American with other races. I therefore recoded race into five categories (White, Black, Hispanic, Asian, Other) in every wave. Ideology has a “haven’t thought much about this” answer, coded −2 in 2012 and 99 in 2016 and 2020; I gave it code 99 in every wave.
anes_2012 <- data_2012 |>
mutate(
year = 2012,
p_gpt3 = p_romney,
vote = presvote2012_x,
race = case_when(
dem_raceeth_x == 1 ~ "White",
dem_raceeth_x == 2 ~ "Black",
dem_raceeth_x == 5 ~ "Hispanic",
dem_raceeth_x == 3 ~ "Asian",
dem_raceeth_x %in% c(4, 6) ~ "Other"
),
discuss = discuss_disc,
ideology = if_else(libcpre_self == -2, 99, libcpre_self),
party_id = pid_x,
church = relig_church,
age = dem_age_r_x,
gender = gender_respondent_x,
interest = paprofile_interestpolit,
flag = patriot_flag,
state = sample_stfips,
.keep = "none"
)
anes_2016 <- data_2016 |>
mutate(
year = 2016,
p_gpt3 = p_trump,
vote = V162062x,
race = case_when(
V161310x == 1 ~ "White",
V161310x == 2 ~ "Black",
V161310x == 5 ~ "Hispanic",
V161310x == 3 ~ "Asian",
V161310x %in% c(4, 6) ~ "Other"
),
discuss = V162174,
ideology = V161126,
party_id = V161158x,
church = V161244,
age = V161267,
gender = V161342,
interest = V162256,
flag = V162125x,
state = V161010d,
.keep = "none"
)
anes_2020 <- data_2020 |>
mutate(
year = 2020,
p_gpt3 = p_trump,
vote = V202110x,
race = case_when(
V201549x == 1 ~ "White",
V201549x == 2 ~ "Black",
V201549x == 3 ~ "Hispanic",
V201549x == 4 ~ "Asian",
V201549x %in% c(5, 6) ~ "Other"
),
discuss = V202022,
ideology = V201200,
party_id = V201231x,
church = V201452,
age = V201507x,
gender = V201600,
interest = V202406,
.keep = "none"
)
Next, I kept the respondents who reported voting for one of the two major-party candidates, as the authors did, and coded the vote as 1 for the Republican candidate. Negative values are ANES codes for missing answers. GPT-3 still got a backstory for respondents with a missing answer, so I kept them and gave every predictor a “missing” category instead of dropping those respondents. I also grouped age into the four groups the paper uses and treated all predictors as categorical.
anes <- bind_rows(anes_2012, anes_2016, anes_2020) |>
filter(vote %in% c(1, 2)) |>
mutate(
vote_rep = if_else(vote == 2, 1, 0),
pred_gpt3 = if_else(p_gpt3 > 0.5, 1, 0),
across(
c(discuss, ideology, party_id, church, gender, interest, flag, age),
\(x) if_else(x < 0, NA, x)
),
gender = if_else(gender == 3, NA, gender),
age = cut(
age,
breaks = c(17, 30, 45, 60, Inf),
labels = c("18-30", "31-45", "46-60", "60+")
),
across(
c(race, discuss, ideology, party_id, church, age, gender, interest),
\(x) fct_na_value_to_level(factor(x), "missing")
),
flag = fct_na_value_to_level(factor(flag), "missing"),
state = factor(state)
)
Before fitting anything, I checked that these data reproduce the whole-sample results in the paper’s Table 1.
anes |>
group_by(year) |>
summarise(
n = n(),
agreement = mean(vote_rep == pred_gpt3),
tetrachoric = tetrachoric(cbind(vote_rep, pred_gpt3))$rho[2]
) |>
kable(digits = 2)
| year | n | agreement | tetrachoric |
|---|---|---|---|
| 2012 | 4188 | 0.85 | 0.90 |
| 2016 | 2609 | 0.87 | 0.92 |
| 2020 | 3904 | 0.89 | 0.94 |
The agreement (0.85, 0.87, 0.89) and tetrachoric correlations (0.90, 0.92, 0.94) match the paper.
Models
I fitted two logistic regressions per election. The first uses the backstory variables that both the training and the test wave share. The flag item is coded on a different scale in 2012 and 2016, and 2020 has neither the flag item nor state. The 2012 model therefore leaves out the flag item, and the 2016 model leaves out the flag item and state. GPT-3 doesn’t need an earlier wave to learn from, so its 2016 backstories did include the flag item. For 2016, that leaves the regression with one variable fewer than GPT-3; for 2020, both use the same eight variables. The second regression uses party identification only, as a check on how much of the prediction comes from that one variable.
I fitted the models on 2012 to predict 2016, and on 2016 to predict 2020. In 2016, 30 respondents have no answer on gender, a category that doesn’t occur in the 2012 data, so the 2012 models can’t make a prediction for them. I left those 30 out of the 2016 comparison for all three models. There is no earlier wave in the data to fit a model for 2012, so 2012 serves only as training data.
train_2012 <- filter(anes, year == 2012)
train_2016 <- filter(anes, year == 2016)
model_all_2016 <- glm(
vote_rep ~ race +
discuss +
ideology +
party_id +
church +
age +
gender +
interest +
state,
family = binomial,
data = train_2012
)
model_pid_2016 <- glm(
vote_rep ~ party_id,
family = binomial,
data = train_2012
)
model_all_2020 <- glm(
vote_rep ~ race +
discuss +
ideology +
party_id +
church +
age +
gender +
interest,
family = binomial,
data = train_2016
)
model_pid_2020 <- glm(
vote_rep ~ party_id,
family = binomial,
data = train_2016
)
test_2016 <- anes |>
filter(year == 2016, gender != "missing") |>
mutate(
p_all = predict(model_all_2016, pick(everything()), type = "response"),
p_pid = predict(model_pid_2016, pick(everything()), type = "response")
)
test_2020 <- anes |>
filter(year == 2020) |>
mutate(
p_all = predict(model_all_2020, pick(everything()), type = "response"),
p_pid = predict(model_pid_2020, pick(everything()), type = "response")
)
predictions <- bind_rows(test_2016, test_2020) |>
mutate(id = row_number()) |>
select(id, year, party_id, vote_rep, p_gpt3, p_all, p_pid) |>
pivot_longer(
c(p_gpt3, p_all, p_pid),
names_to = "model",
values_to = "p"
) |>
mutate(
model = case_when(
model == "p_gpt3" ~ "GPT-3",
model == "p_all" ~ "Regression, all variables",
model == "p_pid" ~ "Regression, party ID only"
),
pred = if_else(p > 0.5, 1, 0)
)
Results
I compared the three models on four measures. Agreement is the share of respondents whose reported vote the model predicts correctly, and the tetrachoric correlation is the measure the paper reports. Both use the 0.5 cutoff. The Brier score uses the probabilities themselves: it is the mean squared difference between the predicted probability and the reported vote (0 or 1), so lower is better. Finally, I compared the mean predicted probability with the share of respondents who reported a Republican vote, the aggregate comparison the paper makes.
predictions |>
group_by(year, model) |>
summarise(
agreement = mean(vote_rep == pred),
tetrachoric = tetrachoric(cbind(vote_rep, pred))$rho[2],
brier = mean((p - vote_rep)^2),
predicted_share = mean(p),
observed_share = mean(vote_rep),
.groups = "drop"
) |>
kable(digits = 3)
| year | model | agreement | tetrachoric | brier | predicted_share | observed_share |
|---|---|---|---|---|---|---|
| 2016 | GPT-3 | 0.866 | 0.923 | 0.093 | 0.434 | 0.480 |
| 2016 | Regression, all variables | 0.903 | 0.954 | 0.077 | 0.488 | 0.480 |
| 2016 | Regression, party ID only | 0.885 | 0.941 | 0.083 | 0.466 | 0.480 |
| 2020 | GPT-3 | 0.890 | 0.940 | 0.116 | 0.472 | 0.412 |
| 2020 | Regression, all variables | 0.899 | 0.953 | 0.075 | 0.449 | 0.412 |
| 2020 | Regression, party ID only | 0.883 | 0.956 | 0.080 | 0.465 | 0.412 |
The regression with all variables does better than GPT-3 on every measure in both elections. In 2016 it predicts 90.3% of votes correctly, against 86.6% for GPT-3. In 2020 the gap in agreement is smaller, 89.9% against 89.0%, but the gap in Brier score is larger (0.075 against 0.116). GPT-3’s probabilities are further from the reported votes, even for respondents where they fall on the right side of 0.5. The regression’s predicted Republican share is also closer to the observed share in both years. That includes 2020: GPT-3’s training data end in 2019, so 2020 is the one election whose outcome it can’t have seen, and there it overestimated the Republican share by six percentage points.
Party identification on its own gets almost as far. With that one variable, the regression agrees with 88.5% of reported votes in 2016 and 88.3% in 2020. That is better than GPT-3 in 2016 and about the same in 2020, and its Brier score is lower than GPT-3’s in both years.
All three models predict the same respondents, so I compared their agreement with a McNemar test, which looks only at the respondents on whom two models disagree.
correct <- predictions |>
mutate(correct = pred == vote_rep) |>
select(id, year, model, correct) |>
pivot_wider(names_from = model, values_from = correct)
correct |>
group_by(year) |>
summarise(
p_all_vs_gpt3 = mcnemar.test(
table(`GPT-3`, `Regression, all variables`)
)$p.value,
p_pid_vs_gpt3 = mcnemar.test(
table(`GPT-3`, `Regression, party ID only`)
)$p.value
) |>
kable(digits = 3)
| year | p_all_vs_gpt3 | p_pid_vs_gpt3 |
|---|---|---|
| 2016 | 0.000 | 0.003 |
| 2020 | 0.015 | 0.131 |
The regression with all variables does better than GPT-3 in both years. The party ID model does better in 2016, and the 2020 difference is too small to tell apart.
The partisanship groups show where the predictions come from. The party identification scale runs from 1 (strong Democrat) to 7 (strong Republican), and I grouped it the way the paper does: strong partisans (1 and 7), weak partisans (2 and 6), leaners (3 and 5) and independents (4).
by_group <- predictions |>
filter(party_id != "missing") |>
mutate(
group = case_when(
party_id %in% c("1", "7") ~ "Strong partisans",
party_id %in% c("2", "6") ~ "Weak partisans",
party_id %in% c("3", "5") ~ "Leaners",
party_id == "4" ~ "Independents"
),
group = fct_relevel(
group,
"Strong partisans",
"Weak partisans",
"Leaners",
"Independents"
)
) |>
group_by(year, group, model) |>
summarise(
n = n(),
agreement = mean(vote_rep == pred),
.groups = "drop"
)
by_group |>
pivot_wider(names_from = model, values_from = agreement) |>
kable(digits = 3)
| year | group | n | GPT-3 | Regression, all variables | Regression, party ID only |
|---|---|---|---|---|---|
| 2016 | Strong partisans | 1185 | 0.974 | 0.976 | 0.976 |
| 2016 | Weak partisans | 633 | 0.736 | 0.829 | 0.829 |
| 2016 | Leaners | 565 | 0.871 | 0.912 | 0.913 |
| 2016 | Independents | 192 | 0.620 | 0.672 | 0.427 |
| 2020 | Strong partisans | 1894 | 0.975 | 0.977 | 0.977 |
| 2020 | Weak partisans | 807 | 0.822 | 0.844 | 0.840 |
| 2020 | Leaners | 891 | 0.899 | 0.896 | 0.906 |
| 2020 | Independents | 300 | 0.530 | 0.577 | 0.357 |
ggplot(by_group, aes(x = agreement, y = fct_rev(group), color = model)) +
geom_point(size = 3) +
facet_wrap(~year) +
scale_x_continuous(labels = scales::label_percent()) +
scale_color_manual(values = c("gray50", primary, "#86efac")) +
labs(x = "Agreement with reported vote", y = NULL, color = NULL) +
theme(
legend.position = "top",
legend.direction = "vertical",
panel.spacing = unit(2, "lines")
)
All three models predict 97–98% of strong partisans correctly. For them, the vote follows almost directly from the party identification in the backstory. Agreement is lower for weak partisans and leaners, and lowest for independents, between 53% and 67%. For independents the full regression agrees more often than GPT-3: 67% against 62% in 2016, and 58% against 53% in 2020. With 192 and 300 independents, though, a McNemar test can’t tell these differences apart from chance.
correct |>
left_join(distinct(predictions, id, party_id), by = "id") |>
filter(party_id == "4") |>
group_by(year) |>
summarise(
n = n(),
p_all_vs_gpt3 = mcnemar.test(
table(`GPT-3`, `Regression, all variables`)
)$p.value
) |>
kable(digits = 3)
| year | n | p_all_vs_gpt3 |
|---|---|---|
| 2016 | 192 | 0.289 |
| 2020 | 300 | 0.202 |
The party ID model does worst for independents, because it predicts the same vote for all of them, based on how independents voted in the previous election.
Conclusion
The high correlations that Argyle et al. report likely reflect, for the most part, partisanship that was already in the backstory: party identification alone predicts votes about as well as GPT-3 does. Where the vote doesn’t follow from party identification, all three models do much worse, and GPT-3 does no better than the regression. A regression on the previous election’s survey, given the same or less information than GPT-3 got, predicts votes at least as well and gets the probabilities and the aggregate share closer.
This points to a related problem with how silicon samples are validated. Argyle et al. could choose the backstory variables because prior research says what predicts vote choice, and they could check the result because they had the human data. For a question with little prior research, you wouldn’t know what to put in the backstory, and the only way to find out is to compare the silicon sample with human responses. Once you have those, you can also fit a simple model, and in this case that model did at least as well as GPT-3.
These results come from one model, GPT-3, and one outcome, the US presidential vote, which is strongly tied to party identification. Within those limits, the results show that the paper’s vote study doesn’t support using a silicon sample instead of a statistical model. Whether that holds for other outcomes is open, but the comparison is cheap to make, and it should be part of any study that proposes silicon samples as a stand-in for human respondents.
References
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337–351. https://doi.org/10.1017/pan.2023.2