logo

To cite this report, please use the following:
Skjervheim, Ø., Bjørnebekk, O., Wettergreen, J., Grendal, O., & Stokke, G. (2024). Norwegian Citizen Panel methodology report, wave 29 [Produced by Ideas2evidence].

Background

This report describes the procedures of data collection in the 29th wave of The Norwegian Citizen Panel. Technical aspects of data collection are discussed, along with the representativity of the panel, and how survey weights are calculated.

The Norwegian Citizen Panel (NCP) is one of the main components of Digital Social Science Core Facility (DIGSSCORE) at the University of Bergen. NCP is as a collaboration between several departments at the Faculty of Social Sciences at the University of Bergen and NORCE.

ideas2evidence is responsible for the panel recruitment, the administration of the panel, and the technical solutions regarding data collection and computing.

Technical Aspects of the Survey

Software

The surveys are administrated through the web-based survey software Confirmit (now part of the company Forsta). Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.

Pilot, Soft Launch, and Distribution

The survey went through small-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.

The pilot testing was regarded as successful, and no major technical revisions were deemed necessary.

The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members was therefore invited shortly.

Randomisation Procedures

Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary1, but they all share some common characteristics that will be described below.

All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.

Randomisation functions are written in JavaScript. Math.random()2 is widely used in combination with Math.Floor()3. These functions are deployed to achieve:

  • Randomly select one value from a vector
  • Randomly shuffle the contents of an array

The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.

Here is an example of the JavaScript code executed in Confirmit:

var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
  var precodes = x1.domainValues(); //copies the length of x1
  var randomNumber: float = Math.random()*precodes.length;
  var randomIndex: int = Math.floor(randomNumber);
  var code = precodes[randomIndex];
  form.set(code);
}

The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example4:

function shuffle(array){
  var currentIndex = array.length, temporaryValue, randomIndex;
  
  //While there remain elements to shuffle
  while (0 != currentIndex) {
    randomIndex = Math.floor(Math.random() * currentIndex);
    currentIndex -= 1;
    
    // And swap it with the current element
    temporaryValue = array[currentIndex];
    array[currentIndex] = array[randomIndex];
    array[randomIndex] = temporaryValue;
  }
  return array;
}

Previous Waves of Recruitment

Existing panel members were recruited in wave 1, wave 3, wave 8, wave 11, wave 14, wave 16, wave 18, wave 22, and wave 25. All samples were drawn from the National Population Registry of Norway. This registry holds information on everyone born in Norway, as well as former and current inhabitants. The Norwegian Tax Administration holds the formal responsibility for the registry.

Samples consist of people over the age of 18 who were randomly drawn from the registry. The extracted information was a) last name, b) first name, c) address, d) gender, e) year of birth, and f) phone number (the latter was not included in wave 1). Samples exclude people without a permanent address in Norway.

Table 1 outlines a short summary of previous recruitment efforts. Note that there are some differences between the recruitment processes. For a detailed description of each recruitment process, please refer to the respective methodology reports.

Table 1: Summary of recruitment
Event Sample size Mode Contacts Returned letters Response rate (%)
Recruitment 1 (wave 1) 25 000 Postal 2 546 20.1 %
Recruitment 2 (wave 3) 25 000 Postal, phone/SMS 4 543 23.0 %
Recruitment 3 (wave 8) 22 000 Postal/SMS 3 479 19.4 %
Recruitment 4 (wave 11) 14 000 Postal/SMS 2 334 15.1 %
Recruitment 5 (wave 14) 14 000 Postal/SMS 2 389 15.0 %
Recruitment 6 (wave 16) 34 000 Postal/SMS 2 994 14.9 %
Recruitment 7 (wave 18) 15 000 Postal/SMS 2 381 14.0 %
Recruitment 8 (wave 22) 23 000 Postal/SMS 2 623 14.5 %
Recruitment 9 (wave 25) 18 000 Postal/SMS 2 562 13.9 %

The response rate of recruitment in wave 11 and onwards were lower than recruitment in previous waves. The most important explanation is new restrictions enforced by the Norwegian Tax Administration with regards to how many times the Citizen Panel can contact the invitees. Respondents in recruitments 4-9 were contacted twice at most. Recruitment 1 also had a maximum of two contact points, but achieved a response rate of 20 percent. One explanation for why we cannot replicate a response rate of 20 percent in recruitments 4-9 might be that NCP did a lot of promotion of the panel through media outlets leading up to and during recruitment 1. Additional promotion of the panel was carried out in relation to the Norwegian Parliamentary election the same fall. We also observe a slow, declining, recruitment rate after the fourth wave of recruitment.

Data Collection

Responses by Method of Data Collection

The survey was distributed to 26 665 panel members on the 15th of January 2024 for the softlaunch and main launch. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.

The invitation, first reminder, and third reminder were all distributed by e-mail. The second reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 29, 55.3 percent of the panel members were registered with a mobile phone number.

Table 2: Responses and response rate for panel members during data collection
Event Response Cumulative responses Response rate Cumulative response rate
Invitation (15th of January) 5 287 5 287 38.4 % 38.4 %
First reminder (19th of January) 2 306 7 593 16.7 % 55.1 %
Second reminder - email (24th of January) 452 8 045 3.3 % 58.4 %
Second reminder - SMS (24th of January) 1 176 9 221 8.5 % 66.9 %
Third reminder (29th of January) 878 10 099 6.4 % 73.3 %

In total 10 099 existing panel members filled out the questionnaire. A response rate of 38.4 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 2 for further details on number of respondents after reminders.

Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 13 770 eligible respondents. The overall response rate, as reported in table 2, is 73.3 %.

Approximately 1 600 of the initial invitations were reported as not delivered by Confirmit, which rounds to 6 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.

Contact with panel members associated with an @online.no-email

During data collection in wave 27, it became evident that fewer panel members registered with an e-mail domain to @online.no responded to the survey. Telenor, the owner of the email service, made changes to their service on the 1st of June 2023, by removing a free alternative in using the service and enhancing their spam-filter. We found it likely that these changes reduced our ability to reach these panel members. During wave 28, it was attempted to counter this change by sending both letters and text messages to the relevant respondents. In wave 29, only text messages were used to try to reach inactive panel members with an @online.no email domain. For the second reminder, a specifically tailored text message was sent to these, providing information on how the panel member could change their registered e-mail address in case they had not received the initial invitation by mail. 390 such text messages were sent. Overall, 1163 panel members with an @online.no email-address responded to the survey, with 808 of these responding before receiving a text message reminder.

Table 3: Number of respondents with an email associated with the @online.no-domain
Wave Number of respondents
Wave 24 1 683
Wave 26 1 842
Wave 27 1 030
Wave 28 940
Wave 29 1 163

Response of Existing Panel Members Over Time

Comparing the number of wave 29 respondents (10 099) to the number of respondents in the previous wave 28 (10 242), gives an overall wave-to-wave retention rate of 99 percent. Figure 1 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 30 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 30th data collection (t30). For respondents recruited in wave 22, the current wave is the eighth data collection (t8).

The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent. Across five out of nine waves of recruitment, the current wave has a retention rate of more than 100 percent. In other words, more respondents participated in wave 29 compared to wave 28 in these.

Figure 1: Wave-to-wave retention rate