Methodological notes to the Motivation Barometer

Within the Motivation Barometer, waves of measurements are regularly set out in function of the evolution of the pandemic. Even though for each wave of measurement there are many thousands of respondents from Dutch-speaking and French-speaking Belgium, the results are not representative for the entire population. The main reason for this is that participation is based on the respondent's own decision. As a result, self-selection occurs. This means that certain answering patterns may occur relatively more often because people with certain characteristics (rather than a representative sample of the population) participate in the survey. This self-selection can be driven by situational, psychological or sociodemographic elements. Respondents are, for instance, people with a computer, tablet or smartphone and an internet connection, with an interest in (aspects of) the COVID-19 policy, with motivation to fill in the list, with a certain conviction for or against certain measures, with an understanding of the questions asked, etc. Note that such self-selection also occurs when representative samples are sampled via a panel study, as psychological or situational factors can also influence the intake of participants. On the one hand, this self-selection can be corrected to a certain extent by statistical methods, but on the other hand, it also imposes limitations on what we can conclude from this survey study. We will discuss both aspects here.

Correction possibilities

Distribution

To avoid self-selection as much as possible, it is important that people from all walks of life come into contact with the invitation to participate and respond. For this reason, invitations are distributed through as many channels as possible, including news websites, a wide range of newspapers, and internet channels such as Facebook and Twitter. Unfortunately, people who do not follow "mainstream" news channels and/or do not follow social media will not be reached. Because the same dissemination channels are used throughout the study, the sociodemographic composition remains broadly stable across the waves of measurement.

Weighting

The non-representative character of the respondents is expressed, among other things, in their sociodemographic characteristics such as age, gender, level of education, and country region. Since the sociodemographic composition of the entire Belgian population is well known (stabel.fgov.be), the answers of certain types of respondents can be given more or less weight in the analyses in order to approach the real sociodemographic characteristics of the population. Such weighting procedures are used within the Motivation Barometer. However, such weighting does not correct for possibly relevant variables of which the distribution is less well known across the different segments of the population (e.g. percentage of parents with young, school-going children, vaccination status of the respondents, etc.) or for the fact that psychological characteristics drive the self-selection (e.g. motivation or annoyance of participants).

Conclusions

As the possibilities for correcting self-selection bias are limited, it is important to clearly delineate the type of conclusions that can be drawn with greater or lesser certainty.

Relatively certain conclusions

Statements about the structural relationships between measured or manipulated psychological variables (e.g. communication style, behaviour, motivation, vaccination, well-being, and trust in policy) or between sociodemographic and psychological variables (e.g. age and motivation) are less influenced by the non-representative nature of the data. This concerns:

• Testing the internal consistency (i.e., reliability) and validity (e.g., internal and construct validity) of constructs;
• Cross-sectional (dynamic) relationships between variables;
• Longitudinal relationships in the same people;
• Relative differences between manipulated variables in experimental designs;

Examining these structural relationships between different variables allows us to test hypotheses inspired by strongly validated theoretical frameworks. On this basis, we can develop a meaningful psychological narrative (e.g. about the role of risk perception in motivation, about changes in support for certain measures, etc.) that provides interpretation and guidance for the population and the policy and that, because of its empirical basis, transcends anecdotal impressions.

Conclusions that require caution

The non-representative nature of the samples makes it difficult to make reliable statements about

• The (absolute) degree to which certain characteristics are present in the population as a whole (e.g. percentage of support for compulsory vaccination; the percentage of participants who are positive towards the coronapas (CST), etc.).
• The relative extent to which certain characteristics (e.g. political trust, well-being) are present in subpopulations (e.g. vaccinated vs. unvaccinated; younger vs. older participants).

Thus, certain figures may over- or underestimate what is happening in reality. We therefore exercise caution when making such statements:

• Draw attention to evolutions over time in the characteristics measured (e.g. decrease/increase in motivation) rather than to the percentage occurrence of these characteristics in isolation.
• Present the results separately for relevant characteristics that are not included in the weighting (e.g. vaccination status). The focus is on the difference and the effect size of this difference rather than on the differential occurrence of certain characteristics.
• In our communication of the results in the media, to highlight the psychological interpretation as much as possible (e.g. a decrease in motivation can be attributed to a decrease in risk awareness).