logo

To cite this report, please use the following:
Skjervheim, Ø., Bjørnebekk, O., Wettergreen, J., Grendal, O., & Stokke, G. (2024). Norwegian Citizen Panel methodology report, wave 29 [Produced by Ideas2evidence].

Background

This report describes the procedures of data collection in the 29th wave of The Norwegian Citizen Panel. Technical aspects of data collection are discussed, along with the representativity of the panel, and how survey weights are calculated.

The Norwegian Citizen Panel (NCP) is one of the main components of Digital Social Science Core Facility (DIGSSCORE) at the University of Bergen. NCP is as a collaboration between several departments at the Faculty of Social Sciences at the University of Bergen and NORCE.

ideas2evidence is responsible for the panel recruitment, the administration of the panel, and the technical solutions regarding data collection and computing.

Technical Aspects of the Survey

Software

The surveys are administrated through the web-based survey software Confirmit (now part of the company Forsta). Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.

Pilot, Soft Launch, and Distribution

The survey went through small-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.

The pilot testing was regarded as successful, and no major technical revisions were deemed necessary.

The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members was therefore invited shortly.

Randomisation Procedures

Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary1, but they all share some common characteristics that will be described below.

All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.

Randomisation functions are written in JavaScript. Math.random()2 is widely used in combination with Math.Floor()3. These functions are deployed to achieve:

  • Randomly select one value from a vector
  • Randomly shuffle the contents of an array

The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.

Here is an example of the JavaScript code executed in Confirmit:

var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
  var precodes = x1.domainValues(); //copies the length of x1
  var randomNumber: float = Math.random()*precodes.length;
  var randomIndex: int = Math.floor(randomNumber);
  var code = precodes[randomIndex];
  form.set(code);
}

The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example4:

function shuffle(array){
  var currentIndex = array.length, temporaryValue, randomIndex;
  
  //While there remain elements to shuffle
  while (0 != currentIndex) {
    randomIndex = Math.floor(Math.random() * currentIndex);
    currentIndex -= 1;
    
    // And swap it with the current element
    temporaryValue = array[currentIndex];
    array[currentIndex] = array[randomIndex];
    array[randomIndex] = temporaryValue;
  }
  return array;
}

Previous Waves of Recruitment

Existing panel members were recruited in wave 1, wave 3, wave 8, wave 11, wave 14, wave 16, wave 18, wave 22, and wave 25. All samples were drawn from the National Population Registry of Norway. This registry holds information on everyone born in Norway, as well as former and current inhabitants. The Norwegian Tax Administration holds the formal responsibility for the registry.

Samples consist of people over the age of 18 who were randomly drawn from the registry. The extracted information was a) last name, b) first name, c) address, d) gender, e) year of birth, and f) phone number (the latter was not included in wave 1). Samples exclude people without a permanent address in Norway.

Table 1 outlines a short summary of previous recruitment efforts. Note that there are some differences between the recruitment processes. For a detailed description of each recruitment process, please refer to the respective methodology reports.

Table 1: Summary of recruitment
Event Sample size Mode Contacts Returned letters Response rate (%)
Recruitment 1 (wave 1) 25 000 Postal 2 546 20.1 %
Recruitment 2 (wave 3) 25 000 Postal, phone/SMS 4 543 23.0 %
Recruitment 3 (wave 8) 22 000 Postal/SMS 3 479 19.4 %
Recruitment 4 (wave 11) 14 000 Postal/SMS 2 334 15.1 %
Recruitment 5 (wave 14) 14 000 Postal/SMS 2 389 15.0 %
Recruitment 6 (wave 16) 34 000 Postal/SMS 2 994 14.9 %
Recruitment 7 (wave 18) 15 000 Postal/SMS 2 381 14.0 %
Recruitment 8 (wave 22) 23 000 Postal/SMS 2 623 14.5 %
Recruitment 9 (wave 25) 18 000 Postal/SMS 2 562 13.9 %

The response rate of recruitment in wave 11 and onwards were lower than recruitment in previous waves. The most important explanation is new restrictions enforced by the Norwegian Tax Administration with regards to how many times the Citizen Panel can contact the invitees. Respondents in recruitments 4-9 were contacted twice at most. Recruitment 1 also had a maximum of two contact points, but achieved a response rate of 20 percent. One explanation for why we cannot replicate a response rate of 20 percent in recruitments 4-9 might be that NCP did a lot of promotion of the panel through media outlets leading up to and during recruitment 1. Additional promotion of the panel was carried out in relation to the Norwegian Parliamentary election the same fall. We also observe a slow, declining, recruitment rate after the fourth wave of recruitment.

Data Collection

Responses by Method of Data Collection

The survey was distributed to 26 665 panel members on the 15th of January 2024 for the softlaunch and main launch. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.

The invitation, first reminder, and third reminder were all distributed by e-mail. The second reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 29, 55.3 percent of the panel members were registered with a mobile phone number.

Table 2: Responses and response rate for panel members during data collection
Event Response Cumulative responses Response rate Cumulative response rate
Invitation (15th of January) 5 287 5 287 38.4 % 38.4 %
First reminder (19th of January) 2 306 7 593 16.7 % 55.1 %
Second reminder - email (24th of January) 452 8 045 3.3 % 58.4 %
Second reminder - SMS (24th of January) 1 176 9 221 8.5 % 66.9 %
Third reminder (29th of January) 878 10 099 6.4 % 73.3 %

In total 10 099 existing panel members filled out the questionnaire. A response rate of 38.4 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 2 for further details on number of respondents after reminders.

Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 13 770 eligible respondents. The overall response rate, as reported in table 2, is 73.3 %.

Approximately 1 600 of the initial invitations were reported as not delivered by Confirmit, which rounds to 6 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.

Contact with panel members associated with an @online.no-email

During data collection in wave 27, it became evident that fewer panel members registered with an e-mail domain to @online.no responded to the survey. Telenor, the owner of the email service, made changes to their service on the 1st of June 2023, by removing a free alternative in using the service and enhancing their spam-filter. We found it likely that these changes reduced our ability to reach these panel members. During wave 28, it was attempted to counter this change by sending both letters and text messages to the relevant respondents. In wave 29, only text messages were used to try to reach inactive panel members with an @online.no email domain. For the second reminder, a specifically tailored text message was sent to these, providing information on how the panel member could change their registered e-mail address in case they had not received the initial invitation by mail. 390 such text messages were sent. Overall, 1163 panel members with an @online.no email-address responded to the survey, with 808 of these responding before receiving a text message reminder.

Table 3: Number of respondents with an email associated with the @online.no-domain
Wave Number of respondents
Wave 24 1 683
Wave 26 1 842
Wave 27 1 030
Wave 28 940
Wave 29 1 163

Response of Existing Panel Members Over Time

Comparing the number of wave 29 respondents (10 099) to the number of respondents in the previous wave 28 (10 242), gives an overall wave-to-wave retention rate of 99 percent. Figure 1 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 30 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 30th data collection (t30). For respondents recruited in wave 22, the current wave is the eighth data collection (t8).

The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent. Across five out of nine waves of recruitment, the current wave has a retention rate of more than 100 percent. In other words, more respondents participated in wave 29 compared to wave 28 in these.

Figure 1: Wave-to-wave retention rate

Platforms

The questionnaire was prepared for input via smart phones, tablets, and other units capable of running web-browsers. In order to enhance the respondents’ experience, the questionnaire is responsive. Respondents on smaller devices, measured in pixels per inch (PPI), are exposed to slightly different visual representations of some questions. For instance is a question grid presented as a set of individual questions on the same page, which is different from the desktop presentation where it would be presented in a table. 43 percent of all survey respondents that opened the questionnaire used a mobile phone.

A set number of survey questions must be answered for a person to be included as a respondent. 10 percent of the mobile users did not reach this minimum requirement, compared to 24 percent for non-mobile users.

The share of mobile users is high among respondents between 18 and 45 of age. As shown in figure 2, the share of mobile users decline substantially with age.

Figure 2: Share of mobile users by gender and age

Time Usage

The average respondent used 17.4 minutes to complete the questionnaire. Measuring average time usage is a challenge, as respondents may leave the questionnaire open in order to complete the survey later. This idle time causes an artificially high average for completing the survey. The average therefore includes only the respondents that spent 60 minutes or less completing the survey.

Figure 3: Time usage distribution of survey respondents

Table 4: Average time usage in minutes for each subset
Type All
All users 17.4
Non-mobile users 18.5
Mobile users 16.2

Representativity

In this section, we describe the representativity of the panel as a whole. First, we will discuss factors explaining representativity. Thereafter we apply demographic variables to present data on representativity by different strata. The data on representativity is the foundation for the section on weighting.

Factors Explaining Representativity

There are two main points that can serve as explanations to non-response and lack of representativity when recruiting and maintaining panel members:

  • access to and familiarity with the internet (given that a web-based questionnaire was the only response mode made available)
  • the motivation and interest of the respondents

The first challenge is strongly related to the age composition of the survey respondents. Although Norway has a very high computer and internet density, the probability of having an e-mail address, and the skills required to access and fill in an online questionnaire, normally decrease with increasing age. The second challenge, motivation and interest, is often explained by the respondents’ level of education. In addition to age and education, we added the variables of geography and gender in order to test the representativity of the survey respondents. The variables have the following categories:

  • Age: 18-29 years, 30-59 years, 60 and above.
  • Highest completed education: no education/elementary school, upper secondary, university/university college.
  • Geography: Oslo, Eastern Norway, Southern Norway, Western Norway, Trøndelag, Northern Norway.

Please note that starting wave twenty-one, the former county of Akershus is part of Eastern Norway, rather than being part of the traditional Akershus/Oslo stratum.

Representativity of the Norwegian Citizen Panel

The sampling frame of the survey equals to the Norwegian population above the age of 18, comprising a population of approximately 4.3 million individuals. Earlier reports have documented a systematic underrepresentation of respondents belonging to the two lowest educational groups, independent of gender and age. The underrepresentation is particularly strong for young men. As expected, individuals with education from universities or university colleges are overrepresented. All of these observations hold true for wave 29.

Table 5: Age distribution in the population and the net sample
Type 18 - 29 years 30 - 59 years 60 years and above
Population 18.8 % 49.7 % 31.5 %
Net sample 3 % 40.3 % 56.7 %

From the age distribution presented in table 5, we see that the two youngest age groups are underrepresented in the net sample of wave 29 when compared to the distribution we find in the population. Inversely, respondents age 60 years and more are overrepresented in the net sample when compared to the population.

Over time, the panel has exhibited a drift away from perfect representativity with regard to age. As figure 4 shows, the oldest respondents started out as underrepresented in wave 1, but have become increasingly overrepresented over time. The youngest respondents, on the other side, have become increasingly underrepresented. This can be explained by a difference in panel membership loyalty; younger panel members are more likely to stop responding to new NCP waves after having been an active member of the panel. The rate of misrepresentation of age groups has peaked with wave 29.

Figure 4: Representativity of age groups

Table 6 breaks down population and the net sample by age and gender. This reveals a slight gender-age interaction in the panel representativity. Younger men are slightly more underrepresented than younger women, while among the oldest in the panel men are more overrepresented than women in the same age bracket.

Table 6: Combined distribution of age and gender in the population and the net sample
18 - 29 years
30 - 59 years
60 years and above
Type Men Women Men Women Men Women
Population 9.7 % 9.1 % 25.2 % 24.5 % 15.1 % 16.5 %
Net sample 1.1 % 1.9 % 19.7 % 20.6 % 31.1 % 25.6 %

The inclusion of educational level in table 7 reveals a systematic underrepresentation of respondents with little or no education, independent of age and gender. The underrepresentation is present in all age brackets, but is especially strong for young respondents.

Table 7: Combined distribution of age, gender and education in the population and the net sample
Population
Net sample
Men Women Men Women
18 - 29 years No education/elementary school 3.8 % 2.9 % 0.1 % 0 %
Upper secondary education 4 % 3.2 % 0.4 % 0.6 %
University/university college 1.9 % 3.1 % 0.6 % 1.3 %
30 - 59 years No education/elementary school 5.1 % 4 % 0.4 % 0.4 %
Upper secondary education 10.8 % 7.6 % 6.1 % 4.1 %
University/university college 9.3 % 12.8 % 13.3 % 16.1 %
60 years and above No education/elementary school 3.3 % 4.5 % 1.9 % 1.3 %
Upper secondary education 7.5 % 7.7 % 10.5 % 7.7 %
University/university college 4.2 % 4.2 % 18.7 % 16.4 %

Respondents who have completed upper secondary education as their highest completed level of education are underrepresented in all groups, except for those with upper secondary education aged 60 years or above. Those who have university or university college education are clearly overrepresented in the two oldest age brackets, irrespective of gender.

Figure 5: Representativity of education groups

Figure 5 illustrates the representation of education groups since wave one. The general trend is that the highly educated are overrepresented compared to those with less or no education. Except for slight improvements in representativity of the education groups when new respondents are recruited (wave 1, 3, 8, 11, 14, 16, 18, 22 and 25), the overall pattern has remained stable throughout all waves.

We observe that the representation of panel members living in Trøndelag are nearly on level with the population, while respondents from Northern and Southern Norway are underrepresented. Respondents from Eastern Norway, Western Norway, and Oslo are overrepresented.

Table 8: Combined distribution of age, gender and geography in the population and the net sample
Population
Net sample
Men Women Total Men Women Total
Oslo 18-29 years 1.3% 1.5% 2.8% 0.3 % 0.5 % 0.8 %
30-59 years 3.6% 3.5% 7.1% 3.6 % 3.7 % 7.3 %
60 years and above 1.4% 1.6% 3% 3.7 % 3.7 % 7.4 %
Total 6.3% 6.6% 12.9% 7.6 % 7.8 % 15.4 %
Eastern Norway 18-29 years 3.4% 3.1% 6.4% 0.2 % 0.5 % 0.7 %
30-59 years 9.4% 9.3% 18.7% 6.3 % 7 % 13.3 %
60 years and above 6.1% 6.8% 12.9% 12.3 % 9.9 % 22.2 %
Total 18.9% 19.2% 38% 18.8 % 17.4 % 36.2 %
Southern Norway 18-29 years 0.6% 0.5% 1.1% 0 % 0.1 % 0.1 %
30-59 years 1.4% 1.4% 2.8% 1 % 1 % 2 %
60 years and above 0.9% 1% 1.8% 1.5 % 1.2 % 2.8 %
Total 2.9% 2.9% 5.7% 2.5 % 2.4 % 4.9 %
Western Norway 18-29 years 2.6% 2.4% 4.9% 0.3 % 0.3 % 0.6 %
30-59 years 6.4% 6.1% 12.6% 5.9 % 5.5 % 11.4 %
60 years and above 3.8% 4.1% 7.9% 8.8 % 7 % 15.8 %
Total 12.8% 12.6% 25.4% 15.1 % 12.7 % 27.8 %
Trøndelag 18-29 years 0.9% 0.9% 1.8% 0.1 % 0.3 % 0.5 %
30-59 years 2.2% 2.1% 4.2% 1.6 % 1.9 % 3.4 %
60 years and above 1.3% 1.4% 2.8% 2.4 % 1.9 % 4.2 %
Total 4.4% 4.4% 8.8% 4.1 % 4 % 8.1 %
Northern Norway 18-29 years 0.9% 0.8% 1.7% 0.1 % 0.2 % 0.2 %
30-59 years 2.2% 2.1% 4.3% 1.3 % 1.5 % 2.8 %
60 years and above 1.5% 1.6% 3.1% 2.6 % 1.9 % 4.5 %
Total 4.6% 4.5% 9.1% 3.9 % 3.6 % 7.5 %

Respondents aged 60 years and above are overrepresented in all parts of the country, which is especially so for Oslo, Eastern Norway, and Western Norway. Conversely, respondents aged 18-29 years are underrepresented in all regions.

Figure 6: Representativity of regions

For wave twenty-one, population data stratified on the new regions was available for the first time since the regional reform of 2020. While this data eliminates some small uncertainty in the representativity analyses5, it also introduces a break in time series for Oslo (previously including Akershus) and Eastern Norway (now including Akershus). In the 2024 regional reform, Akershus, among other former counties, were reinstituted as an independent county. In order to preserve the regional division established in 2020, Akershus remains a part of Eastern Norway in our analysis. Compared to age and education, geography does, however, not seem to be a strong determinant of survey participation. Apart from effects from the regional reform, the geographic representativity is more or less stable over time.

Weighting

A weight has been calculated to compensate for observed biases. The weight is equal to the relation between a given strata in the population and the total population, divided by the relation between a given strata in the net sample and the total net sample6. This procedure returns values around 1, and always more than 0. Respondents belonging to a stratum that is underrepresented will receive a weight above 1 and respondents belonging to an overrepresented stratum will receive a weight below 1. We have listed the weights of the different strata in the table in the appendix.

When calculating the weight, information regarding the respondents’ geographical location, gender, and age is based on registry data. Information on these variables was included in the sample file received from the Norwegian National Population Registry upon recruitment. Information on the level of education is collected from NCP surveys. 0 percent of the 29th wave net sample have not answered the question about level of education.

The following demographic variables are applied in weight4:

When applied, the weight will provide a weighted N equal to the number of cases in the dataset. In other words, the weight is calculated using the whole dataset. NCP has an extensive use of (randomized) sub-groups, which might alter the demographic profile of the sub-group compared to the whole dataset.

Consequently, the weight might be less precise for some sub-groups. Note that the dataset is provided with necessary information7 to calculate custom weights if needed, following the procedure described above.

Table 9 shows the effects of weight 4 on the distribution of self-reported level of education in the net sample. As we can observe, the weight gives the sample a distribution closer to the population. It is, however, important to stress that the unweighted distribution is not ideal, with a clear underrepresentation of people with low levels of education.

Table 9: Effect of weight4 on self-reported level of education
Sample - standard Sample - weighted Population Difference between sample and population Difference between weighted sample and population
Upper secondary school and lower 33.6 % 61.1 % 64.4 % -30.8 -3.3
University/university college 66.4 % 38.9 % 35.6 % 30.8 3.3

Appendix

Table 10: Weight4 applied to different strata
Age Education Men Women Age Education Men Women
Oslo 18-29 Upper secondary and lower 5 5 Western Norway 18-29 Upper secondary and lower 5 5
University/university college 1.9 2.3 University/university college 3.6 4.8
30-59 Upper secondary and lower 3.2 3.2 30-59 Upper secondary and lower 2.2 2.5
University/university college 0.7 0.8 University/university college 0.6 0.8
60+ Upper secondary and lower 1 1.2 60+ Upper secondary and lower 0.8 1.3
University/university college 0.2 0.3 University/university college 0.2 0.3
Eastern Norway 18-29 Upper secondary and lower 5 5 Trøndelag 18-29 Upper secondary and lower 5 5
University/university college 5 2.6 University/university college 4.5 1.4
30-59 Upper secondary and lower 2.8 3 30-59 Upper secondary and lower 3 2.9
University/university college 0.9 0.9 University/university college 0.8 0.8
60+ Upper secondary and lower 1 1.5 60+ Upper secondary and lower 1.1 2
University/university college 0.3 0.3 University/university college 0.3 0.3
Southern Norway 18-29 Upper secondary and lower 5 5 Northern Norway 18-29 Upper secondary and lower 5 5
University/university college 5 2.9 University/university college 5 2.9
30-59 Upper secondary and lower 2.7 2.5 30-59 Upper secondary and lower 3.7 2.7
University/university college 0.8 1 University/university college 0.9 1.1
60+ Upper secondary and lower 1.2 1.9 60+ Upper secondary and lower 1.1 2
University/university college 0.3 0.3 University/university college 0.3 0.3

  1. Some examples: sorting respondents in different thematic subsets, randomly allocate treatment value in experiments, randomize order of an answer list/array, order a sequence of questions by random, ask a given question to a subset of the respondents.↩︎

  2. Please see following resource: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/random↩︎

  3. Please see following resource: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/floor↩︎

  4. Collected from Mike Bostocks visualisation: https://bost.ocks.org/mike/shuffle/↩︎

  5. Note that Oslo (including Akershus), and Eastern Norway diverge in wave eighteen, due to the regional reform implemented 1st of January 2020.↩︎

  6. The applied formula for weight wi for element i, in strata h is: \[wi=\frac{N_h/N}{n_h/n}\]↩︎

  7. See columns r29_Weight4_stratapop and r29_generic_stratapop↩︎