PPBS 2016
The following describes the survey methodology of the 2016 iteration of The Psychology Political Behavior Studies (PPBS). This report is meant to detail the utilized survey methodology and relevant, general purpose characteristics of both exploratory and confirmatory samples, as well as any extra samples and its combined version. The text is in APA Style and is akin to an APA-like ‘Methods Section’.
PPBS datasets are designed to have an exploratory, quota-based Nationally Representative sample (on Age, Education, Income and Sex), and a Confirmatory (Replication) convenience sample, from the same data source to avoid false positives. Some PPBS studies also have recontacts or extra-samples to answer additional specific research questions (e.g., 2nd Sample in 2016). All data’s coodbooks can be found at the bottom of the page, with a searchable feature allowing to find metadata within and between columns.
Taken together, there were three independent samples collected in the months preceding the 2016 U.S. Presidential Election (from August 16th to September 19th, 2016) and amount to 4039 online interviews of American adults, lasting ~55 minutes (Median) for those successfully passing the variety of data quality checks (87%). The 2016 attrition rate was pretty constant at ~22%.
For more details, see PBBS’s Motivation.
| Sample 1 | Sample 2 | Sample 3 | |
|---|---|---|---|
| Election cycle | Pre-Election 2016 | Pre-Election 2016 | Pre-Election 2016 |
| Type | Nationally Representative | Confirmatory (Replication) | Extra (2nd Study) |
| Survey Period (Start) | August 16th, 2016 | August 20th, 2016 | September 15th, 2016 |
| Survey Period (End) | September 16th, 2016 | September 13th, 2016 | September 19th, 2016 |
| Country | United States | United States | United States |
| Attention Checks | Yes (11; various types) | Yes (11; various types) | Yes (11; various types) |
| Time Checks | Yes | Yes | Yes |
| CAPTCHA | Yes (begining of survey) | Yes (begining of survey) | Yes (begining of survey) |
| Sample Size (N) | 1500 | 2119 | 420 |
| Length of Interview (MD) | 51.28 | 57.77 | 55.54 |
| Attrition (%) | 0.22 | 0.22 | 0.23 |
| Data Quality checks (%) | 0.16 | 0.16 | 0.07 |
| Note. The Overall column displays the simple, non-weighted on sample size, average between the preceeding columns. If interested in the weighted averages, see section ‘Combined Samples’ below. |
Nationally Representative Sample (N=1500)
Sample Description
We hired Survey Sampling Incorporated (SSI; www.surveysampling.com), a market research firm that recruits participants from a panel of over 7 million U.S. citizens, to recruit a nationally representative sample of 1500 Americans (50.67% women) in the months preceding the 2016 U.S. Presidential Election (from August 16 to September 16, 2016).
The quotas were designed to match the 2014 American Community Survey (ACS)1 from the US Census Bureau on age, income, education and gender. The representativeness of the collected sample is presented in the table below, which shows an average absolute deviation of 10.38% points (MD = 6.67% points) from the desired quotas, with age showing an an average absolute deviation of 0.88% points (MD = 0.91% points), gender 2.94% points (MD = 2.94% points), income 8.78% points (MD = 8.12% points), and education, 27% points (MD = 12% points), which had the largest spread on the bracket “Less than high school/No high school diploma”, for which we could only arrive at ~25% of the expected frequency, indicating the sample has achieved a moderate level of national representativeness.
In addition to administering a much greater number and variety of political and psychological instruments (including full scales) than in other nationally representative surveys (such as ANES, GSS, and WVS), we took a number of steps to insure that the quality of the data would be especially high. These included following professional recommendations to minimize problems of careless responding and satisficing behavior in online survey studies (Meade & Craig, 2012). Specifically, we employed 11 random attention questions, as well as page-time, survey-total, and click count controls. A total of 2424 participants were directed to the survey, and 1885 of them finished the survey (attrition rate 22%). There were 385 (16%) participants who failed more than two attention checks or finished the survey in under ~22 minutes and were therefore excluded. For the final sample of 1500, participants who successfully completed all study materials had a completion time of 69.29 minutes on average (MD: 51.28min).
The age distribution of our sample was as follows: 18–24 years (12.87%), 25–34 (17.6%), 35–44 (17.53%), 45–54 (19.47%), 55–65 (15.6%), and older than 65 (16.93%). The ethnic breakdown was: White (82.47%), Black/African American (7.67%), Latino (5.87%), Asian/Pacific Islander (1.93%), Native American (0.87%), Middle Eastern (1.2%). In terms of religion, (67.6% identified as Christian, 0.6% as Muslim, 3.47% as Jewish,15.33% as either Atheist or Agnostic, 13% and responded they are not sure, religion not listed, or refused to answer. With respect to education, 3.4% declared their highest educational achievement to be less high school, 31.67% indicated High school graduate (includes equivalency), 31.4% chose some college ( or associate degree) or no higher-education degree, 20.67% indicated having a Bachelor’s degree, and finally, 12.87% indicated having received a Graduate or professional degree. The median income category was $35,000 to $49,999. The exact distribution of Income is as follows: Less $15,000 (11.87%), $15,000 to $24,999 (12%), $25,000 to $34,999 (11.73%), $35,000 to $49,999 (15.13%), $50,000 to $74,999 (19.47%), $75,000 to $99,999 (12.8%), $100,000 to $149,999 (10.67%) and $150,000 more (6.33%).
| Demographic | Brackets | Census ACS % | Expected Sample Frequencies | Observed Frequencies | Expected vs. Observed Frequencies | Expected vs. Observed % | Expected vs. Observed % [Census 2010]* |
|---|---|---|---|---|---|---|---|
| Age | 18 to 24 | 0.13 | 192 | 193 | 1 | 0.52 | -1.53 |
| Age | 25 to 34 | 0.18 | 269 | 264 | -5 | -1.86 | 0.38 |
| Age | 35 to 44 | 0.18 | 265 | 263 | -2 | -0.75 | 0.00 |
| Age | 45 to 54 | 0.19 | 290 | 292 | 2 | 0.69 | 1.39 |
| Age | 55 to 64 | 0.15 | 232 | 234 | 2 | 0.86 | 0.43 |
| Age | 65 to 80+ | 0.17 | 253 | 254 | 1 | 0.40 | -1.55 |
| Education | Less than high school | 0.14 | 204 | 51 | -153 | -75.00 | -75.71 |
| Education | High school graduate (includes equivalency) | 0.28 | 420 | 475 | 55 | 13.10 | 9.20 |
| Education | Some college (associate degree) or no degree | 0.29 | 436 | 471 | 35 | 8.03 | 8.28 |
| Education | Bachelor’s degree | 0.18 | 274 | 310 | 36 | 13.14 | 14.81 |
| Education | Graduate or professional degree | 0.11 | 165 | 193 | 28 | 16.97 | 28.67 |
| Gender | Female | 0.52 | 773 | 760 | -13 | -1.68 | 2.98 |
| Gender | Male | 0.48 | 727 | 740 | 13 | 1.79 | -2.89 |
| Income | Less than $15,000 | 0.12 | 188 | 178 | -10 | -5.32 | -8.72 |
| Income | $15,000 to $24,999 | 0.11 | 160 | 180 | 20 | 12.50 | 0.00 |
| Income | $25,000 to $34,999 | 0.10 | 153 | 176 | 23 | 15.03 | 6.67 |
| Income | $35,000 to $49,999 | 0.14 | 202 | 227 | 25 | 12.38 | 8.10 |
| Income | $50,000 to $74,999 | 0.18 | 267 | 292 | 25 | 9.36 | 8.15 |
| Income | $75,000 to $99,999 | 0.12 | 183 | 192 | 9 | 4.92 | 6.67 |
| Income | $100,000 to $149,999 | 0.13 | 195 | 160 | -35 | -17.95 | -11.11 |
| Income | $150,000 or more | 0.10 | 150 | 95 | -55 | -36.67 | -20.83 |
1 Note. An important note is that the US Census Bureau makes minor adjustments to its data over time and is not averse to closing tables and/or redesigning them. This is not particular of the US. Hence, the numbers were correct at date of access, which was Monday 1st August, 2016. Furthermore, due to this, we also provide representation in relation to the 2010 US Census, which is shown in the last column. Specifically, we take Gender and Age from U.S. Census Bureau, 2010 Census and Income and Education from 2010-2014 American Community Survey 5-Year Estimates, which at the time, were the most accurate data available.
Regional Representation
As shown below, the distribution of data points per state tracks well with state population. The date has not been designed to be regionally representative, nor it claims to be, but results are not bad (cf. US Decennial Census Tables). Please note that frequencies add to 1389, not to 1500, due to non-responses.
Convenience Confirmatory (Replication) Sample (N=2119)
Sample Description
Also through SSI, we also administered the same survey to a large convenience sample of 2119 American adults (21.47% women) in the months preceding the 2016 U.S. Presidential Election (from August 20 to September 13, 2016).
We applied the same quality-control criteria as explained in the Nationally Representative sample. Specifically, we followed recommendations to minimize the problem of careless responding in online studies (Meade & Craig, 2012). A total of 3425 participants were directed to the survey, and 2662 of them finished the survey (attrition rate 22%). There were 543 (16%) participants who failed more than two attention checks or finished the survey in under ~22 minutes and were therefore excluded. For the final sample of 2119, participants who successfully completed all study materials had a completion time of 92.01 minutes on average (MD: 57.77min).
The age distribution of our sample was as follows: 18–24 years (9.06%), 25–34 (13.83%), 35–44 (11.42%), 45–54 (2.74%), 55–65 (3.63%), and older than 65 (59.32%). The ethnic breakdown was: White (85.89%), Black/African American (5.05%), Latino (4.11%), Asian/Pacific Islander (2.17%), Native American (0.94%), Middle Eastern (1.84%). In terms of religion, (70.65% identified as Christian, 0.52% as Muslim, 5.57% as Jewish,13.69% as either Atheist or Agnostic, 9.58% and responded they are not sure, religion not listed, or refused to answer. With respect to education, 1.04% declared their highest educational achievement to be less high school, 15.15% indicated High school graduate (includes equivalency), 40.4% chose some college ( or associate degree) or no higher-education degree, 23.88% indicated having a Bachelor’s degree, and finally, 19.54% indicated having received a Graduate or professional degree. The median income category was $50,000 to $74,999. The median income category was $50,000 to $74,999. The exact distribution of Income is as follows: Less $15,000 (10.15%), $15,000 to $24,999 (8.07%), $25,000 to $34,999 (11.09%), $35,000 to $49,999 (14.39%), $50,000 to $74,999 (21.24%), $75,000 to $99,999 (14.91%), $100,000 to $149,999 (11.89%) and $150,000 more (11.89%).
Combined Samples (N=3619)
Sample Description
We combined and analyzed data from two large surveys conducted before the 2016 U.S. General Election (from August 16 to September 16, 2016), including a nationally representative sample (N = 1500) and a large convenience sample (N = 2119; see Azevedo, Jost, & Rothmund, 2017; Azevedo, Jost, Rothmund, & Sterling, 2019). We hired SSI (SSI; www.surveysampling.com), a survey research firm that recruits participants from a pool of over 7 million U.S. citizens. We took a number of steps to insure that the quality of the data would be especially high. These included following professional recommendations to minimize problems of careless responding and satisficing behavior in online survey studies (Meade & Craig, 2012). Specifically, we employed 11 random attention questions, page-time and number of clicks controls. A total of 5849 participants were directed to the survey, and 4547 of them finished the survey (attrition rate 22%). There were 928 (16%) participants who failed more than two attention checks or finished the survey in under ~22 minutes and were therefore excluded. For the final sample of 3619, participants who successfully completed all study materials had a completion time of 82.6 minutes on average (MD: 55.15min).
The age distribution of our sample was as follows: 18–24 years (10.64%), 25–34 (15.39%), 35–44 (13.95%), 45–54 (9.67%), 55–65 (8.59%), and older than 65 (41.75%). The ethnic breakdown was: White (84.47%), Black/African American (6.13%), Latino (4.84%), Asian/Pacific Islander (2.07%), Native American (0.91%), Middle Eastern (1.58%). In terms of religion, (69.38% identified as Christian, 0.55% as Muslim, 4.7% as Jewish,14.37% as either Atheist or Agnostic, 11% and responded they are not sure, religion not listed, or refused to answer. With respect to education, 2.02% declared their highest educational achievement to be less high school, 22% indicated High school graduate (includes equivalency), 36.67% chose some college ( or associate degree) or no higher-education degree, 22.55% indicated having a Bachelor’s degree, and finally, 16.77% indicated having received a Graduate or professional degree. The median income category was $50,000 to $74,999. The exact distribution of Income is as follows: Less $15,000 (10.86%), $15,000 to $24,999 (9.7%), $25,000 to $34,999 (11.36%), $35,000 to $49,999 (14.7%), $50,000 to $74,999 (20.5%), $75,000 to $99,999 (14.04%), $100,000 to $149,999 (11.38%) and $150,000 more (11.38%).
2nd Study (N=420)
Sample Description
Also through SSI, we also administered the same survey to a large convenience sample of 420 American adults (61.43% women) in the months preceding the 2016 US Presidential Election (from September 15 to September 19, 2016).
This study is characterized by having the same questions as PPBS2016 samples, with the inclusion of four additional instruments: Stealth Democracy, Direct Democracy Attitudes and Citizen Participation items, as well as additional Conspiracy theories, all of which had been cut from the survey instrument due to concerns over the length of interview.
We took a number of steps to insure that the quality of the data would be especially high. These included following professional recommendations to minimize problems of careless responding and satisficing behavior in online survey studies (Meade & Craig, 2012). Specifically, we employed 11 random attention questions, as well as page-time, survey-total, and click count controls. A total of 598 participants were directed to the survey, and 463 of them finished the survey (attrition rate 23%). There were 43 (7%) participants who failed more than two attention checks or finished the survey in under ~22 minutes and were therefore excluded. For the final sample of 420, participants who successfully completed all study materials had a completion time of 90.8 minutes on average (MD: 55.54min).
The age distribution of our sample was as follows: 18–24 years (4.29%), 25–34 (16.43%), 35–44 (20%), 45–54 (20%), 55–65 (38.33%), and older than 65 (0.95%). The ethnic breakdown was: White (82.62%), Black/African American (9.29%), Latino (3.33%), Asian/Pacific Islander (2.62%), Native American (0.71%), Middle Eastern (1.43%). In terms of religion, (69.29% identified as Christian, 0.24% as Muslim, 5.71% as Jewish,14.29% as either Atheist or Agnostic, 10.48% and responded they are not sure, religion not listed, or refused to answer. With respect to education, 4.29% declared their highest educational achievement to be less high school, 11.43% indicated High school graduate (includes equivalency), 24.52% chose some college ( or associate degree) or no higher-education degree, 41.19% indicated having a Bachelor’s degree, and finally, 18.57% indicated having received a Graduate or professional degree. The median income category was $50,000 to $74,999. The exact distribution of Income is as follows: Less $15,000 (9.29%), $15,000 to $24,999 (9.05%), $25,000 to $34,999 (10%), $35,000 to $49,999 (12.14%), $50,000 to $74,999 (21.67%), $75,000 to $99,999 (14.52%), $100,000 to $149,999 (16.43%) and $150,000 more (6.9%).
Codebook
Search box on the right finds items in all columns. Search boxes at the column level finds matches in their specific column.