Survey researchers and pundits alike frequently lament the fact that citizens know too little about politics. Furthermore, public opinion polling consistently suggests that women appear to be less knowledgeable about politics than men. But how can we measure political sophistication in the first place? And more importantly, how does our measurement approach impact our conclusions about citizen competence about politics? The conventional approach to measure political sophistication in surveys focuses on multiple-choice questions assessing people’s ability to recall information about political institutions and officeholders. (For instance, the American National Election Study routinely asks questions such as, “Do you happen to know which party currently has the most members in the U.S. Senate?”) These types of factual knowledge questions—which have been used in scores of surveys for academic research and polling—have important limitations. For instance, determining a comprehensive set of knowledge questions is far from trivial since it involves strong assumptions about what information is necessary for people to be politically competent. Furthermore, knowledge questions vary in difficulty across demographic segments and can therefore introduce systematic measurement error when comparing groups of respondents.
One important consequence of these measurement issues is the apparent gender gap in political sophistication. Traditionally, women tend to score lower on factual knowledge assessments than men. While this may partly stem from genuine differences in political interest due to gendered socialization, a substantial portion of this gender gap can be attributed to the measurement issues associated with conventional knowledge metrics. For instance, men’s higher willingness to guess instead of admitting that they don’t know an answer inflates their aggregate knowledge levels compared to women. Furthermore, research indicates that gender differences lessen when focusing on gender-relevant political knowledge or in contexts with balanced gender representation.
Introducing Discursive Sophistication
To overcome these limitations, I propose an alternative measure called discursive sophistication. Instead of focusing on factual knowledge questions, discursive sophistication evaluates how people discuss their political preferences in open-ended survey responses using computational text analysis. Specifically, I develop a framework to assess whether beliefs and attitudes in a given political domain are expressed in a more elaborate manner—a question that is not directly discernible from off-the-shelf factual knowledge items. This approach holds two key advantages over conventional metrics: first, it captures the extent to which political beliefs are grounded in elaborate reasoning, and second, it can pinpoint competence in specific domains through targeted open-ended queries. The resulting measurement therefore more closely matches our theoretical conceptualization of political sophistication.
I validate the new measure across diverse surveys encompassing multiple languages, revealing its effectiveness in predicting various indicators of civic competence and engagement. While closely related to traditional metrics, the two measures are distinct. For example, both discursive sophistication and factual knowledge independently predict political engagement and efficacy, implying their complementary nature. Importantly, this new measure disrupts the conventional gender gap narrative. While women might lag men in factual political knowledge, there’s no discrepancy in the complexity of their expressed political attitudes across various surveys (see Figure 1).
Note: The figures display distributions of political sophistication using open-ended or conventional measures comparing women and men (including 95% confidence intervals around the means). Gender differences are statistically significant at * p< 0.05, ** p< 0.01, and *** p< 0.001.
Explaining the (Lack of a) Gender Gap
Intriguingly, there is evidence that this divergence between both measures can be attributed to women focusing on different issues in open-ended responses. Figure 2 shows gender differences in topics raised in the open-ended responses themselves. The results indicate that the gender gap in conventional knowledge metrics is at least partly driven by the fact that the issues women care about are not represented in standard item batteries. For discursive sophistication on the other hand, any evidence for systematic differences between women and men disappears since open-ended questions allow respondents to focus on specific considerations that are most relevant to them.
Note: Figure is based on the structural topic model used to compute discursive sophistication (Including 95% Confidence Intervals). Coefficients indicate the difference in predicted topic prevalence among women and men; positive values indicate higher prevalence among women. Labels are based on the five most frequent and exclusive (FREX) terms associated with each topic.
Looking Ahead: Political Sophistication Reimagined
My findings suggest that while conventional knowledge measures and the novel open-ended approach share commonalities, they offer distinct insights into political sophistication. This distinction is vital, as both metrics independently influence political engagement and efficacy. Notably, the gender gap typically observed in factual knowledge disappears when scrutinized through the lens of discursive sophistication. The absence of gender differences in this measure could stem from the opportunities open-ended responses offer for women to focus on distinct considerations.
Past survey research has almost exclusively relied on conventional knowledge metrics to assess political sophistication. My study calls for a broader approach, highlighting the potential of open-ended responses to help us better understand the structure of people’s political belief systems. In other words, the focus of political research needs to shift from a mere emphasis on factual knowledge to a deeper understanding of how individuals think about their political preferences. Discursive sophistication provides such an alternative that can help us gain new insights into how people navigate the political landscape. This method is easily adaptable to diverse decision-making contexts through tailored open-ended questions. Moreover, researchers and policymakers alike can harness a user-friendly software package in the R programming environment to implement this framework.
Read more
Patrick Kraft. “Women Also Know Stuff: Challenging the Gender Gap in Political Sophistication” in American Political Science Review 2023.
image: European Parliament via Flickr (CC BY-NC-ND 2.0)
This article was originally published on the USAPP – American Politics and Policy blog of the London School of Economics.
No Comments