Norwegian Citizen Panel
Methodology Report
31th wave

Software

The surveys are administrated through the web-based survey software Confirmit (now part of the company Forsta). Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.

Pilot, Soft Launch, and Distribution

The survey went through small-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.

The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members were invited shortly after.

Throughout the field period respondents was randomly allocated into one of five subgroups, all of which were exposed to different questions sets.

Randomisation Procedures

Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary¹, but they all share some common characteristics that will be described below.

All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.

Randomisation functions are written in JavaScript. Math.random()² is widely used in combination with Math.Floor()³. These functions are deployed to achieve:

• Randomly select one value from a vector
• Randomly shuffle the contents of an array

The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.

Here is an example of the JavaScript code executed in Confirmit:

var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
  var precodes = x1.domainValues(); //copies the length of x1
  var randomNumber: float = Math.random()*precodes.length;
  var randomIndex: int = Math.floor(randomNumber);
  var code = precodes[randomIndex];
  form.set(code);
}

The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example⁴:

function shuffle(array){
  var currentIndex = array.length, temporaryValue, randomIndex;
  
  //While there remain elements to shuffle
  while (0 != currentIndex) {
    randomIndex = Math.floor(Math.random() * currentIndex);
    currentIndex -= 1;
    
    // And swap it with the current element
    temporaryValue = array[currentIndex];
    array[currentIndex] = array[randomIndex];
    array[randomIndex] = temporaryValue;
  }
  return array;
}

Responses by mode of contact

The survey was distributed to 29 759 panel members on the 4th of November 2024 for the softlaunch and on the 5th of November for the main launch. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.

The invitation, first reminder, and third reminder were all distributed by e-mail. The second reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 31, 53.4 percent of the panel members were registered with a mobile phone number.

In total 11 515 existing panel members filled out the questionnaire. A response rate of 34.6 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 2 for further details on number of respondents after reminders.

Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 16 054 eligible respondents. The overall response rate, as reported in table 2, is 71.8 %.

Approximately 1 700 of the initial invitations were reported as not delivered by Confirmit, which rounds to 5 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.

Response of Existing Panel Members Over Time

Comparing the number of wave 31 respondents (11 515) to the number of respondents in the previous wave 30 (13 509), gives an overall wave-to-wave retention rate of 85 percent. Figure 1 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 32 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 32th data collection (t32).

The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent.

Figure 1: Wave-to-wave retention rate

Platforms

The questionnaire was prepared for input via smart phones, tablets, and other units capable of running web-browsers. In order to enhance the respondents’ experience, the questionnaire is responsive. Respondents on smaller devices, measured in pixels per inch (PPI), are exposed to slightly different visual representations of some questions. Question grids are presented as a set of individual questions on the same page, which is different from the desktop presentation where it would be presented in a table. 50 percent of all survey respondents that opened the questionnaire used a mobile phone.

A set number of survey questions must be answered for a person to be included as a respondent. 7 percent of the mobile users did not reach this minimum requirement, compared to 20 percent for non-mobile users.

The share of mobile users is high among respondents between 18 and 45 years of age. As shown in figure 2, the share of mobile users decline with age.

Figure 2: Share of mobile users by gender and age

Time Usage

The average respondent used 18.4 minutes to complete the questionnaire. Measuring average time usage is a challenge, as respondents may leave the questionnaire open in order to complete the survey later. This idle time causes an artificially high average for completing the survey. The average therefore includes only the respondents that spent 60 minutes or less completing the survey.

Figure 3: Time usage distribution of survey respondents

Factors Explaining Representativity

There are two main points that can serve as explanations to non-response and lack of representativity when recruiting and maintaining panel members:

• access to and familiarity with the internet (given that a web-based questionnaire was the only response mode made available)
• the motivation and interest of the respondents

The first challenge is strongly related to the age composition of the survey respondents. Although Norway has a very high computer and internet density, the probability of having an e-mail address, and the skills required to access and fill in an online questionnaire, normally decrease with increasing age. The second challenge, motivation and interest, is often explained by the respondents’ level of education. In addition to age and education, we added the variables of geography and gender in order to test the representativity of the survey respondents. The variables have the following categories:

• Age: 18-29 years, 30-59 years, 60 and above.
• Highest completed education: no education/elementary school, upper secondary, university/university college.
• Geography: Oslo, Eastern Norway, Southern Norway, Western Norway, Trøndelag, Northern Norway.

Please note that starting wave twenty-one, the former county of Akershus is part of Eastern Norway, rather than being part of the traditional Akershus/Oslo stratum.

Representativity of the Norwegian Citizen Panel

The sampling frame of the survey equals to the Norwegian population above the age of 18, comprising a population of approximately 4.4 million individuals. Earlier reports have documented a systematic underrepresentation of respondents belonging to the two lowest educational groups, independent of gender and age. The underrepresentation is particularly strong for young men. As expected, individuals with education from universities or university colleges are overrepresented. All of these observations hold true for wave 31.

From the age distribution presented in table 4, we see that both the youngest age groups are underrepresented in the net sample of wave 31 when compared to the distribution we find in the population. Inversely, respondents age 60 years and more are overrepresented in the net sample when compared to the population.

Over time, and until wave 30, the panel exhibited a drift from perfect representativity with regard to age. As figure 4 shows, the oldest respondents started out underrepresented in wave 1, but have become increasingly overrepresented over time. The youngest respondents have become increasingly underrepresented at the same time. This has been explained by a difference in panel membership loyalty; younger panel members are less likely to participate in new waves of NCP after having been an active member of the panel. Recruitment in wave 30 introduced stratified sampling, and the youngest group was no longer underrepresented. Figure 4 shows a steep drop in representativity for the youngest age group from wave 30 to wave 31, highlighting how difficult it is to activate this age group.

Figure 4: Representativity of age groups

Table 5 breaks down population and the net sample by age and gender. This reveals a slight gender-age interaction in the panel representativity. Across the two youngest age groups, women are less undererepresented than men. Among the oldest in the panel men are more overrepresented than women in the same age bracket.

The inclusion of educational level in table 6 reveals a systematic underrepresentation of respondents with little or no education, independent of age and gender. The underrepresentation is present in all age brackets, but is especially strong for young respondents.

Respondents who have completed upper secondary education or have not completed any level of education are in general underrepresented, while respondents with university/university college as their highest level of education are overrepresented. Those who have university or university college education are clearly overrepresented in the two oldest age brackets, irrespective of gender.

Figure 5: Representativity of education groups

Figure 5 illustrates the representation of education groups since wave one. The general trend is that the highly educated are overrepresented compared to those with less or no education. Except for slight improvements in representativity of the education groups when new respondents are recruited (wave 1, 3, 8, 11, 14, 16, 18, 22, 25 and 30), the overall pattern has remained stable throughout all waves.

We observe that the representation of panel members living in Trøndelag are nearly on level with the population, while respondents from Northern and Eastern Norway are underrepresented. Respondents from Western Norway and Oslo are overrepresented. Respondents aged 60 years and above are overrepresented in all parts of the country, but only slightly so in Southern and Northern Norway. We find the biggest overrepresentation in Oslo.

Figure 6: Representativity of regions

For wave twenty-one, population data stratified on the new regions was available for the first time since the regional reform of 2020. While this data eliminates some small uncertainty in the representativity analyses⁵, it also introduces a break in time series for Oslo (previously including Akershus) and Eastern Norway (now including Akershus). In the 2024 regional reform, Akershus, among other former counties, were reinstituted as an independent county. In order to preserve the regional division established in 2020, Akershus remains a part of Eastern Norway in our analysis. Compared to age and education, geography does, however, not seem to be a strong determinant of survey participation. Apart from effects from the regional reform, the geographic representativity is more or less stable over time.