To cite this report, please use the following:
Bjørnebekk, O.,
Wettergreen, J., Grendal, O., & Stokke, G. (2024). Norwegian Citizen
Panel methodology report, wave 30 [Produced by Ideas2evidence].
This report describes the procedures of data collection in the 30th wave of The Norwegian Citizen Panel. Technical aspects of data collection are discussed, along with the representativity of the panel, and how survey weights are calculated.
The Norwegian Citizen Panel (NCP) is one of the main components of Digital Social Science Core Facility (DIGSSCORE) at the University of Bergen. NCP is as a collaboration between several departments at the Faculty of Social Sciences at the University of Bergen and NORCE.
ideas2evidence is responsible for the panel recruitment, the administration of the panel, and the technical solutions regarding data collection and computing.
The surveys are administrated through the web-based survey software Confirmit (now part of the company Forsta). Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.
The survey went through small-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.
The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members was therefore invited shortly.
Throughout the field period respondents was randomly allocated into one of five subgroups, all of which were exposed to different questions sets. Group 6 is reserved for newly recruited panel members. However, due to a technical fault, 11 respondents in group 6 are not newly recruited panel members. The issue was addressed during the field period. Respondents can be identified by cross-tabulating the variable r30_group with the variable recruited. Respondents in group 6 was randomly allocated a set of question which were exposed to respondents in group 2, 3, 4, and 5. Group 1 is therefore technically reserved for existing panel members.
Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary1, but they all share some common characteristics that will be described below.
All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.
Randomisation functions are written in JavaScript. Math.random()2 is widely used in combination with Math.Floor()3. These functions are deployed to achieve:
The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.
Here is an example of the JavaScript code executed in Confirmit:
var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
var precodes = x1.domainValues(); //copies the length of x1
var randomNumber: float = Math.random()*precodes.length;
var randomIndex: int = Math.floor(randomNumber);
var code = precodes[randomIndex];
form.set(code);
}
The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example4:
function shuffle(array){
var currentIndex = array.length, temporaryValue, randomIndex;
//While there remain elements to shuffle
while (0 != currentIndex) {
randomIndex = Math.floor(Math.random() * currentIndex);
currentIndex -= 1;
// And swap it with the current element
temporaryValue = array[currentIndex];
array[currentIndex] = array[randomIndex];
array[randomIndex] = temporaryValue;
}
return array;
}
Existing panel members were recruited in wave 1, wave 3, wave 8, wave 11, wave 14, wave 16, wave 18, wave 22, and wave 25. All samples were drawn from the National Population Registry of Norway. This registry holds information on everyone born in Norway, as well as former and current inhabitants. The Norwegian Tax Administration holds the formal responsibility for the registry.
Samples consist of people over the age of 18 who were randomly drawn from the registry. The extracted information was a) last name, b) first name, c) address, d) gender, e) year of birth, and f) phone number (the latter was not included in wave 1). Samples exclude people without a permanent address in Norway.
Table 1 outlines a short summary of previous recruitment efforts. Note that there are some differences between the recruitment processes. For a detailed description of each recruitment process, please refer to the respective methodology reports. A detailed description of the recruitment in wave 30 follows in the next section.
Event | Sample size | Mode | Contacts | Returned letters | Response rate (%) |
---|---|---|---|---|---|
Recruitment 1 (wave 1) | 25 000 | Postal | 2 | 546 | 20.1 |
Recruitment 2 (wave 3) | 25 000 | Postal, phone/SMS | 4 | 543 | 23.0 |
Recruitment 3 (wave 8) | 22 000 | Postal/SMS | 3 | 479 | 19.4 |
Recruitment 4 (wave 11) | 14 000 | Postal/SMS | 2 | 334 | 15.1 |
Recruitment 5 (wave 14) | 14 000 | Postal/SMS | 2 | 389 | 15.0 |
Recruitment 6 (wave 16) | 34 000 | Postal/SMS | 2 | 994 | 14.9 |
Recruitment 7 (wave 18) | 15 000 | Postal/SMS | 2 | 381 | 14.0 |
Recruitment 8 (wave 22) | 23 000 | Postal/SMS | 2 | 623 | 14.5 |
Recruitment 9 (wave 25) | 18 000 | Postal/SMS | 2 | 562 | 13.9 |
Recruitment 10 (wave 30) | 25 000 | Postal/SMS | 4 | 989 | 15.0 |
The response rate of recruitment in recruitment 4 and onwards were lower than recruitment in previous waves. The most important explanation were restrictions enforced by the Norwegian Tax Administration with regards to how many times the Citizen Panel can contact the invitees. Respondents in recruitments 4-9 were contacted twice at most. Recruitment 1 also had a maximum of two contact points, but achieved a response rate of 20 percent. One explanation for why we cannot replicate a response rate of 20 percent in recruitments 4-10 might be that NCP did a lot of promotion of the panel through media outlets leading up to and during recruitment 1. Additional promotion of the panel was carried out in relation to the Norwegian Parliamentary election the same fall. We also observe a slow, declining, recruitment rate after the fourth recruitment until recruitment 10 where the restriction on maximum contact points was lifted.
In wave 30, The Norwegian Citizen Panel recruited new panel members. This section gives a detailed description of the sample frame, recruitment process, and the results. Compared to previous waves of recruitment, NCP was no longer limited by only making a maximum of two points of contact towards attempted recruited panel members.
NCP attempts to recruit persons age 18 or above to the panel regularly. The probability to be attempted recruited varies between the persons in Norway, where younger persons have a lower probability to be recruited as many of them have not been eligible for participation in previous rounds of recruitment, and thus have a lower overall probability of having been invited. To account and correct for that, gradual stratified sampling has been used to recruit persons in wave 30. The fewer chances a person has had to be attempted recruited previously, the larger the probability to be attempted recruited with the gradual stratified approach. People aged 18-29 were oversampled to correct for the recruitment probability. Within that group, a further oversampling was done for the younger persons. As recruitment is not necessarily carried out every calendar year, this is not a linear oversampling year-for-year from 18-29, but the younger people within this group generally enjoyed greater probability to be attempted recruited when compared to the older persons within the group, and more so when compared to the the part of the sample which were sampled of the general population as long as the person is 18 or older.
A stratified sample gross sample was drawn from the population registry. Approximately 17 500 people aged 18 - 29 were drawn from the register, while approximately 7 500 people over the age of 18 were randomly drawn. The extracted information included a) surname, b) first name, c) address, d) gender, and e) year of birth. The sample excluded individuals without a current home address in Norway. Telephone numbers were attached to the sample by using Data Factory. The availability of phone number varied with age groups within the sample. The younger people had a lower coverage in general. Table 2 below sums up the coverage ratio of phone number by age groups within the sample.
Age group | Registered with phone number (%) | Not registered with a phone number (%) |
---|---|---|
18 - 24 | 38.6 | 61.4 |
25 - 34 | 49.3 | 50.7 |
35 - 44 | 49.2 | 50.8 |
45 - 64 | 61.6 | 38.4 |
65 - 84 | 75.6 | 24.4 |
85 and over | 79.1 | 20.9 |
Information on the different points of contact can be found below. If the recipient was under 30 years of age the content of the letter, postcard, and SMS varied textually from the letter, postcard, and SMS sent to recipients aged 30 and above. If the recipient was not registered with a phone number they received the letter of invitation and postcard(s). The remainder of the sample also received one or multiple SMS.
Date | Type of Contact |
---|---|
4th of June | Letter of invitation |
11th of June | First postcard |
14th of June | First SMS |
20th of June | Second SMS |
24th of June | Second postcard |
27th of June | Third SMS |
Initially, letters were sent to everyone in the sample. The letters contained the following information: a) a description of the project, b) the Citizen Panel’s policy on privacy and measures taken to protect the anonymity of the participants, c) the time-frame of the project, d) the participants’ rights to opt out of the panel at any time in the future, e) contact information for the people responsible for the project, f) a unique log-in id and the web address to the panel’s web site and g) the estimated time required to complete the survey.
In order to maximize the response rate, an incentive in the form of three gift cards is included in the project. The values of the gift cards are 8 000 NOK. To enter the lottery respondents were required to join the panel and provide their email addresses. Respondents were asked to register on the panel’s website and log into the survey using the unique id-code provided in their personal letter.
All reminders were sent to respondents who a) had not logged into the survey, or b) had not completed the survey. Respondents were encouraged to join the panel, with reference to the invitation letter. The unique log-in id provided in the original letter was included in the postcard and the SMS. The SMS reminder(s) also included a direct link to the survey.
It is necessary to make a distinction between panel members and survey respondents. Panel members are defined as respondents who register their e-mail address in the survey, regardless of whether they have completed the questionnaire or not. Survey respondents are respondents who have completed a certain share of the questionnaire, regardless of whether they have entered their e-mail or not.
Of the 24 917 letters that were sent out, 989 were returned, and 14 respondents opted out. 22.4 percent (5 364) of the remaining 23 914 logged on and accessed the survey. 3 466 individuals completed the questionnaire, while 1 898 exited before completion. 7.7 percent of these responses are kept as a part of the survey data as these respondents completed a certain amount of the questionnaire before exiting. The remaining 1 751 incomplete responses were excluded from the data set, due to a lack of data.
In sum, after subtracting a few cases where the credentials of the respondent did not match the credentials of the invited, this recruitment wave resulted in 3 576 new survey respondents, a recruitment rate of 15 percent. This is higher that what was achieved in recruitment 9 (13.9%). 98.3 percent of the respondents who completed the survey submitted their e-mail address. 3 746 new panel members were recruited to the Norwegian Citizen Panel, resulting in a panel recruitment rate of 15.7 percent.
Further discussions in this report about respondents recruited in wave 30 are based on survey respondents. As survey respondents and panel members closely overlap, these descriptions can be assumed valid for the panel members as well.
In previous waves of NCP where recruitment has been carried out, it has been fairly trivial in understanding what mode of contact (letter, postcard, or SMS) made the respondent enter the survey. However, wave 30 used more points and modes of contact that heavily overlapped during the field period, and trying to construct a table as done in previous waves might introduce errors and give an erroneous understanding of who answered when, and as a response to what mode of contact. Instead, a simplified table can be found below, strictly differentiating between those respondents who likely answered in response to an SMS, and those who likely did not. In short, if the respondent received an SMS and started the survey on the same day as the SMS was distributed, the table assume that the respondent answered due to the SMS. If they did not receive an SMS, or answered on a different date than the SMS was sent out, the table assumes that the respondent answered due to the letter of invitation or subsequent postcards.
Mode of contact | Response | Response rate (%) | Cumulative responses | Cumulative response rate (%) |
---|---|---|---|---|
Likely a response to the letter of invitation or subsequent postcards | 2 651 | 11.1 | 2 651 | 11.1 |
Likely a response to SMS-reminder(s) | 925 | 3.9 | 3 576 | 15.0 |
To understand when the newly recruited answered in relation to the different reminders, the figure below outlines when the questionnaire was started among the newly recruited respondents.
Figure 1: Responses counted day by day among respondents recruited in wave 30 Figure 1 outlines both the gradual incline of responses assumed created by the letter of invitation, and by comparison, the steep incline in number of responses when SMS is used as mode of contact.
Given the stratified approach to sampling, it is assumed to the level of response is influenced by the age or age group of the persons attempted recruited. Table 5 shows the response rate according to the groups in which the sample was stratified into: 18-29 and 30 and above. In line with the expectation that it is harder to recruit younger people, table 5 shows that the response rate among the stratified sampling group was lower when compared to the older people.
Age group | Sample | Number of respondents | Responserate (%) |
---|---|---|---|
18 - 29 | 17 475 | 2 224 | 12.7 |
30 or older | 7 525 | 1 352 | 18.0 |
The survey was distributed to 26 497 panel members on the 3rd of June 2024 for the soft launch and 4th of June for the main launch. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.
The invitation, first reminder, and third reminder were all distributed by e-mail. The second reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 30, 51.5 percent of the panel members were registered with a mobile phone number.
Event | Response | Cumulative responses | Response rate | Cumulative response rate |
---|---|---|---|---|
Invitation (3rd/4th of June) | 5 085 | 5 085 | 39.6 % | 39.6 % |
First reminder (11th of June) | 2 651 | 7 736 | 20.6 % | 60.2 % |
Second reminder - email (19th of June) | 314 | 8 050 | 2.4 % | 62.6 % |
Second reminder - SMS (19th of June) | 841 | 8 891 | 6.5 % | 69.1 % |
Third reminder (27th of June) | 1 042 | 9 933 | 8.1 % | 77.2 % |
In total 9 933 existing panel members filled out the questionnaire. A response rate of 39.6 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 6 for further details on number of respondents after reminders.
Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 12 853 eligible respondents. The overall response rate, as reported in table 6, is 77.2 %.
Approximately 1 600 of the initial invitations were reported as not delivered by Confirmit, which rounds to 3 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.
Comparing the number of wave 30 respondents (9 933) to the number of respondents in the previous wave 29 (10 099), gives an overall wave-to-wave retention rate of 98 percent. Figure 2 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 31 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 31th data collection (t31). For respondents recruited in wave 22, the current wave is the ninth data collection (t9).
The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent. Across five out of ten waves of recruitment, the current wave has a retention rate of more than 100 percent. In other words, more respondents participated in wave 30 compared to wave 29 in these.
Figure 2: Wave-to-wave retention rate
The questionnaire was prepared for input via smart phones, tablets, and other units capable of running web-browsers. In order to enhance the respondents’ experience, the questionnaire is responsive. Respondents on smaller devices, measured in pixels per inch (PPI), are exposed to slightly different visual representations of some questions. For instance are question grids presented as a set of individual questions on the same page, which is different from the desktop presentation where it would be presented in a table. 49 percent of all survey respondents that opened the questionnaire used a mobile phone.
A set number of survey questions must be answered for a person to be included as a respondent. 12 percent of the mobile users did not reach this minimum requirement, compared to 28 percent for non-mobile users.
The share of mobile users is high among respondents between 18 and 45 years of age. As shown in figure 3, the share of mobile users decline with age.
Figure 3: Share of mobile users by gender and age
The average respondent used 19.1 minutes to complete the questionnaire. Measuring average time usage is a challenge, as respondents may leave the questionnaire open in order to complete the survey later. This idle time causes an artificially high average for completing the survey. The average therefore includes only the respondents that spent 60 minutes or less completing the survey.
Figure 4: Time usage distribution of survey respondents
Type | All | Group 1 | Group 2 | Group 3 | Group 4 | Group 5 | Group 6 |
---|---|---|---|---|---|---|---|
All users | 19.1 | 20.1 | 18.2 | 18.2 | 18.5 | 19.6 | 19.6 |
Non-mobile users | 20.3 | 20.7 | 19.4 | 19.2 | 19.7 | 20.5 | 21.8 |
Mobile users | 18.0 | 19.4 | 17.0 | 17.2 | 17.3 | 18.6 | 18.3 |
In this section, we describe the representativity of the panel as a whole. First, we will discuss factors explaining representativity. Thereafter we apply demographic variables to present data on representativity by different strata. The data on representativity is the foundation for the section on weighting.
There are two main points that can serve as explanations to non-response and lack of representativity when recruiting and maintaining panel members:
The first challenge is strongly related to the age composition of the survey respondents. Although Norway has a very high computer and internet density, the probability of having an e-mail address, and the skills required to access and fill in an online questionnaire, normally decrease with increasing age. The second challenge, motivation and interest, is often explained by the respondents’ level of education. In addition to age and education, we added the variables of geography and gender in order to test the representativity of the survey respondents. The variables have the following categories:
Please note that starting wave twenty-one, the former county of Akershus is part of Eastern Norway, rather than being part of the traditional Akershus/Oslo stratum.
The sampling frame of the survey equals to the Norwegian population above the age of 18, comprising a population of approximately 4.4 million individuals. Earlier reports have documented a systematic underrepresentation of respondents belonging to the two lowest educational groups, independent of gender and age. The underrepresentation is particularly strong for young men. As expected, individuals with education from universities or university colleges are overrepresented. All of these observations hold true for wave 30.
Type | 18 - 29 years | 30 - 59 years | 60 years and above |
---|---|---|---|
Population | 18.8 % | 49.7 % | 31.5 % |
Net sample | 18.8 % | 35.6 % | 45.7 % |
From the age distribution presented in table 8, we see that those between 30 - 59 years are underrepresented in the net sample of wave 30 when compared to the distribution we find in the population. Inversely, respondents age 60 years and more are overrepresented in the net sample when compared to the population, while the youngest age group is on par with the population as a whole.
Over time, the panel has exhibited a drift away from perfect representativity with regard to age until wave 30. As figure 5 shows, the oldest respondents started out as underrepresented in wave 1, but have become increasingly overrepresented over time. The youngest respondents, on the other side, have become increasingly underrepresented. This has previously been explained by a difference in panel membership loyalty; younger panel members are more likely to stop responding to new NCP waves after having been an active member of the panel. The level of under- and overrepresentation of age groups has been reduced with wave 30, as can be seen in figure 5 due to recruitment with the stratified sampling approach.
Figure 5: Representativity of age groups
Table 9 breaks down population and the net sample by age and gender. This reveals a slight gender-age interaction in the panel representativity. Younger men are underrepresented, while younger women are slightly overrepresented. Among the oldest in the panel men are more overrepresented than women in the same age bracket.
Type | Men | Women | Men | Women | Men | Women |
---|---|---|---|---|---|---|
Population | 9.7 % | 9.1 % | 25.2 % | 24.5 % | 15.1 % | 16.5 % |
Net sample | 8.1 % | 10.6 % | 17.2 % | 18.3 % | 24.8 % | 20.9 % |
The inclusion of educational level in table 10 reveals a systematic underrepresentation of respondents with little or no education, independent of age and gender. The underrepresentation is present in all age brackets, but is especially strong for young respondents.
Men | Women | Men | Women | ||
---|---|---|---|---|---|
18 - 29 years | No education/elementary school | 3.8 % | 2.9 % | 0.7 % | 1 % |
Upper secondary education | 4 % | 3.2 % | 4.8 % | 5.3 % | |
University/university college | 1.9 % | 3.1 % | 2.3 % | 3.9 % | |
30 - 59 years | No education/elementary school | 5.1 % | 4 % | 0.4 % | 0.4 % |
Upper secondary education | 10.8 % | 7.6 % | 5.7 % | 3.7 % | |
University/university college | 9.3 % | 12.8 % | 11.3 % | 14.3 % | |
60 years and above | No education/elementary school | 3.3 % | 4.5 % | 1.6 % | 1.1 % |
Upper secondary education | 7.5 % | 7.7 % | 8.7 % | 6.4 % | |
University/university college | 4.2 % | 4.2 % | 14.9 % | 13.5 % |
Respondents who have completed upper secondary education or have not completed any level of education are in general underrepresented, while respondents with university/university college as their highest level of education are overrepresented. Those who have university or university college education are clearly overrepresented in the two oldest age brackets, irrespective of gender.
Figure 6: Representativity of education groups
Figure 6 illustrates the representation of education groups since wave one. The general trend is that the highly educated are overrepresented compared to those with less or no education. Except for slight improvements in representativity of the education groups when new respondents are recruited (wave 1, 3, 8, 11, 14, 16, 18, 22, 25 and 30), the overall pattern has remained stable throughout all waves. This pattern is now broken, as effects from the stratified approach to sampling with regard to age is seemingly correcting for skewness in education-representativity as well.
Men | Women | Total | Men | Women | Total | ||
---|---|---|---|---|---|---|---|
Oslo | 18-29 years | 1.3% | 1.5% | 2.8% | 1.2 % | 1.8 % | 2.9 % |
30-59 years | 3.6% | 3.5% | 7.1% | 3 % | 3.4 % | 6.4 % | |
60 years and above | 1.4% | 1.6% | 3% | 2.9 % | 2.9 % | 5.9 % | |
Total | 6.3% | 6.6% | 12.9% | 7.1 % | 8.1 % | 15.2 % | |
Eastern Norway | 18-29 years | 3.4% | 3.1% | 6.4% | 2.9 % | 3.4 % | 6.3 % |
30-59 years | 9.4% | 9.3% | 18.7% | 5.8 % | 6.2 % | 12 % | |
60 years and above | 6.1% | 6.8% | 12.9% | 9.8 % | 8.1 % | 17.9 % | |
Total | 18.9% | 19.2% | 38% | 18.5 % | 17.8 % | 36.3 % | |
Southern Norway | 18-29 years | 0.6% | 0.5% | 1.1% | 0.5 % | 0.6 % | 1.1 % |
30-59 years | 1.4% | 1.4% | 2.8% | 0.8 % | 0.9 % | 1.7 % | |
60 years and above | 0.9% | 1% | 1.8% | 1.3 % | 1.1 % | 2.4 % | |
Total | 2.9% | 2.9% | 5.7% | 2.6 % | 2.7 % | 5.3 % | |
Western Norway | 18-29 years | 2.6% | 2.4% | 4.9% | 2.2 % | 2.9 % | 5.1 % |
30-59 years | 6.4% | 6.1% | 12.6% | 5.1 % | 4.7 % | 9.8 % | |
60 years and above | 3.8% | 4.1% | 7.9% | 6.9 % | 5.5 % | 12.4 % | |
Total | 12.8% | 12.6% | 25.4% | 14.2 % | 13.1 % | 27.4 % | |
Trøndelag | 18-29 years | 0.9% | 0.9% | 1.8% | 0.8 % | 1 % | 1.8 % |
30-59 years | 2.2% | 2.1% | 4.2% | 1.4 % | 1.7 % | 3 % | |
60 years and above | 1.3% | 1.4% | 2.8% | 1.9 % | 1.6 % | 3.6 % | |
Total | 4.4% | 4.4% | 8.8% | 4.1 % | 4.3 % | 8.4 % | |
Northern Norway | 18-29 years | 0.9% | 0.8% | 1.7% | 0.6 % | 0.9 % | 1.4 % |
30-59 years | 2.2% | 2.1% | 4.3% | 1.1 % | 1.4 % | 2.6 % | |
60 years and above | 1.5% | 1.6% | 3.1% | 1.9 % | 1.6 % | 3.5 % | |
Total | 4.6% | 4.5% | 9.1% | 3.6 % | 3.9 % | 7.5 % |
We observe that the representation of panel members living in Trøndelag are nearly on level with the population, while respondents from Northern Norway are underrepresented. Respondents from Western Norway and Oslo are overrepresented. Respondents aged 60 years and above are overrepresented in all parts of the country, but only slightly so in Trøndelag, Southern and Northern Norway. We find the biggest overrepresentation in Oslo and Eastern Norway. Respondents aged 18-29 years are mostly close to the general population in all regions.
Figure 7: Representativity of regions
For wave twenty-one, population data stratified on the new regions was available for the first time since the regional reform of 2020. While this data eliminates some small uncertainty in the representativity analyses5, it also introduces a break in time series for Oslo (previously including Akershus) and Eastern Norway (now including Akershus). In the 2024 regional reform, Akershus, among other former counties, were reinstituted as an independent county. In order to preserve the regional division established in 2020, Akershus remains a part of Eastern Norway in our analysis. Compared to age and education, geography does, however, not seem to be a strong determinant of survey participation. Apart from effects from the regional reform, the geographic representativity is more or less stable over time.
A weight has been calculated to compensate for observed biases. The weight is equal to the relation between a given strata in the population and the total population, divided by the relation between a given strata in the net sample and the total net sample6. This procedure returns values around 1, and always more than 0. Respondents belonging to a stratum that is underrepresented will receive a weight above 1 and respondents belonging to an overrepresented stratum will receive a weight below 1. We have listed the weights of the different strata in the table in the appendix.
When calculating the weight, information regarding the respondents’ geographical location, gender, and age is based on registry data. Information on these variables was included in the sample file received from the Norwegian National Population Registry upon recruitment. Information on the level of education is collected from NCP surveys. 2 percent of the 30th wave net sample have not answered the question about level of education.
The following demographic variables are applied in weight4:
When applied, the weight will provide a weighted N equal to the number of cases in the dataset. In other words, the weight is calculated using the whole dataset. NCP has an extensive use of (randomized) sub-groups, which might alter the demographic profile of the sub-group compared to the whole dataset.
Consequently, the weight might be less precise for some sub-groups. Note that the dataset is provided with necessary information7 to calculate custom weights if needed, following the procedure described above.
Table 12 shows the effects of weight 4 on the distribution of self-reported level of education in the net sample. As we can observe, the weight gives the sample a distribution closer to the population. It is, however, important to stress that the unweighted distribution is not ideal, with a clear underrepresentation of people with low levels of education.
Sample - standard | Sample - weighted | Population | Difference between sample and population | Difference between weighted sample and population | |
---|---|---|---|---|---|
Upper secondary school and lower | 39.9 % | 64.4 % | 64.4 % | -24.5 | 0 |
University/university college | 60.1 % | 35.6 % | 35.6 % | 24.5 | 0 |
Age | Education | Men | Women | Age | Education | Men | Women | ||
---|---|---|---|---|---|---|---|---|---|
Oslo | 18-29 | Upper secondary and lower | 1.6 | 1 | Western Norway | 18-29 | Upper secondary and lower | 1.3 | 0.9 |
University/university college | 0.8 | 0.8 | University/university college | 0.8 | 0.8 | ||||
30-59 | Upper secondary and lower | 3.2 | 3 | 30-59 | Upper secondary and lower | 2 | 2.8 | ||
University/university college | 0.8 | 0.8 | University/university college | 0.7 | 0.9 | ||||
60+ | Upper secondary and lower | 1.1 | 1.3 | 60+ | Upper secondary and lower | 0.9 | 1.4 | ||
University/university college | 0.3 | 0.3 | University/university college | 0.3 | 0.3 | ||||
Eastern Norway | 18-29 | Upper secondary and lower | 1.4 | 1 | Trøndelag | 18-29 | Upper secondary and lower | 1.5 | 0.9 |
University/university college | 0.8 | 0.8 | University/university college | 1 | 0.8 | ||||
30-59 | Upper secondary and lower | 2.8 | 3 | 30-59 | Upper secondary and lower | 2.9 | 2.6 | ||
University/university college | 0.9 | 1 | University/university college | 0.8 | 0.8 | ||||
60+ | Upper secondary and lower | 1.1 | 1.6 | 60+ | Upper secondary and lower | 1.1 | 2 | ||
University/university college | 0.3 | 0.3 | University/university college | 0.3 | 0.3 | ||||
Southern Norway | 18-29 | Upper secondary and lower | 1.3 | 0.9 | Northern Norway | 18-29 | Upper secondary and lower | 1.8 | 1.2 |
University/university college | 0.8 | 0.8 | University/university college | 1.5 | 0.7 | ||||
30-59 | Upper secondary and lower | 2.5 | 2 | 30-59 | Upper secondary and lower | 3.8 | 2.6 | ||
University/university college | 1 | 1.1 | University/university college | 0.9 | 1 | ||||
60+ | Upper secondary and lower | 1.3 | 1.9 | 60+ | Upper secondary and lower | 1.3 | 2.2 | ||
University/university college | 0.3 | 0.3 | University/university college | 0.3 | 0.3 |
Some examples: sorting respondents in different thematic subsets, randomly allocate treatment value in experiments, randomize order of an answer list/array, order a sequence of questions by random, ask a given question to a subset of the respondents.↩︎
Please see following resource: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/random↩︎
Please see following resource: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/floor↩︎
Collected from Mike Bostocks visualisation: https://bost.ocks.org/mike/shuffle/↩︎
Note that Oslo (including Akershus), and Eastern Norway diverge in wave eighteen, due to the regional reform implemented 1st of January 2020.↩︎
The applied formula for weight wi for element i, in strata h is: \[ wi=\frac{N_h/N}{n_h/n} \]↩︎
See columns r30_Weight4_stratapop and r30_generic_stratapop↩︎