Norwegian Citizen Panel
Methodology
Report
28th wave
ideas2evidence
December, 2023
To cite this report, please use the following:
Skjervheim, Ø.,
Bjørnebekk, O., Wettergreen, J., Grendal, O., & Stokke, G. (2023).
Norwegian Citizen Panel methodology report, wave 28 [Produced by
Ideas2evidence].
Background
This report describes the procedures of data collection in the 28th wave of The Norwegian Citizen Panel. Technical aspects of data collection are discussed, along with the representativity of the panel, and how survey weights are calculated.
The Norwegian Citizen Panel (NCP) is one of the main components of Digital Social Science Core Facility (DIGSSCORE) at the University of Bergen. NCP is as a collaboration between several departments at the Faculty of Social Sciences at the University of Bergen and NORCE.
ideas2evidence is responsible for the panel recruitment, the administration of the panel, and the technical solutions regarding data collection and computing.
Technical Aspects of the Survey
Software
The surveys are administrated through the web-based survey software Confirmit. Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.
Pilot, Soft Launch, and Distribution
The survey went through small-N and large-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.
The pilot testing was regarded as successful, and no major technical revisions were deemed necessary.
The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members was therefore invited shortly.
Randomisation Procedures
Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary1, but they all share some common characteristics that will be described below.
All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.
Randomisation functions are written in JavaScript. Math.random()2 is widely used in combination with Math.Floor()3. These functions are deployed to achieve:
- Randomly select one value from a vector
- Randomly shuffle the contents of an array
The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.
Here is an example of the JavaScript code executed in Confirmit:
var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
var precodes = x1.domainValues(); //copies the length of x1
var randomNumber: float = Math.random()*precodes.length;
var randomIndex: int = Math.floor(randomNumber);
var code = precodes[randomIndex];
form.set(code);
}The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example4:
function shuffle(array){
var currentIndex = array.length, temporaryValue, randomIndex;
//While there remain elements to shuffle
while (0 != currentIndex) {
randomIndex = Math.floor(Math.random() * currentIndex);
currentIndex -= 1;
// And swap it with the current element
temporaryValue = array[currentIndex];
array[currentIndex] = array[randomIndex];
array[randomIndex] = temporaryValue;
}
return array;
}Previous Waves of Recruitment
Existing panel members were recruited in wave 1, wave 3, wave 8, wave 11, wave 14, wave 16, wave 18, wave 22, and wave 25. All samples were drawn from the National Population Registry of Norway. This registry holds information on everyone born in Norway, as well as former and current inhabitants. The Norwegian Tax Administration holds the formal responsibility for the registry.
Samples consist of people over the age of 18 who were randomly drawn from the registry. The extracted information was a) last name, b) first name, c) address, d) gender, e) year of birth, and f) phone number (the latter was not included in wave 1). Samples exclude people without a permanent address in Norway.
Table 1 outlines a short summary of previous recruitment efforts. Note that there are some differences between the recruitment processes. For a detailed description of each recruitment process, please refer to the respective methodology reports.
| Event | Sample size | Mode | Contacts | Returned letters | Response rate (%) |
|---|---|---|---|---|---|
| Recruitment 1 (wave 1) | 25 000 | Postal | 2 | 546 | 20.1 % |
| Recruitment 2 (wave 3) | 25 000 | Postal, phone/SMS | 4 | 543 | 23.0 % |
| Recruitment 3 (wave 8) | 22 000 | Postal/SMS | 3 | 479 | 19.4 % |
| Recruitment 4 (wave 11) | 14 000 | Postal/SMS | 2 | 334 | 15.1 % |
| Recruitment 5 (wave 14) | 14 000 | Postal/SMS | 2 | 389 | 15.0 % |
| Recruitment 6 (wave 16) | 34 000 | Postal/SMS | 2 | 994 | 14.9 % |
| Recruitment 7 (wave 18) | 15 000 | Postal/SMS | 2 | 381 | 14.0 % |
| Recruitment 8 (wave 22) | 23 000 | Postal/SMS | 2 | 623 | 14.5 % |
| Recruitment 9 (wave 25) | 18 000 | Postal/SMS | 2 | 562 | 13.9 % |
The response rate of recruitment in wave 11 and onwards were lower than recruitment in previous waves. The most important explanation is new restrictions enforced by the Norwegian Tax Administration with regards to how many times the Citizen Panel can contact the invitees. Respondents in recruitments 4-9 were contacted twice at most. Recruitment 1 also had a maximum of two contact points, but achieved a response rate of 20 percent. One explanation for why we cannot replicate a response rate of 20 percent in recruitments 4-9 might be that NCP did a lot of promotion of the panel through media outlets leading up to and during recruitment 1. Additional promotion of the panel was carried out in relation to the Norwegian Parliamentary election the same fall. We also observe a slow, declining, recruitment rate after the fourth wave of recruitment.
Data Collection
Responses by Method of Data Collection
The survey was distributed to 26 944 panel members on the 23rd and 24th of October 2023 for the softlaunch and main launch respectively. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.
The invitation, first reminder, and second reminder were all distributed by e-mail. The third and last reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 28, 54.3 percent of the panel members were registered with a mobile phone number.
| Event | Response | Cumulative responses | Response rate | Cumulative response rate |
|---|---|---|---|---|
| Invitation (23rd/24th of October) | 5 292 | 5 292 | 35.1 % | 35.1 % |
| First reminder (30th of October) | 2 329 | 7 621 | 15.5 % | 50.6 % |
| Second reminder (3rd of November) | 1 376 | 8 997 | 9.1 % | 59.7 % |
| Third reminder - email (9th of November) | 321 | 9 318 | 2.1 % | 61.8 % |
| Third reminder - SMS (9th of November) | 924 | 10 242 | 6.1 % | 67.9 % |
In total 10 242 existing panel members filled out the questionnaire. A response rate of 35.1 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 2 for further details on number of respondents after reminders.
Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 15 061 eligible respondents. The overall response rate, as reported in table 2, is 67.9 %.
Approximately 1 500 of the initial invitations were reported as not delivered by Confirmit, which rounds to 5 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.
Contact with panel members associated with an @online.no-email
During data collection in wave 27, it became evident that fewer panel members registered with an e-mail domain to @online.no responded to the survey. Telenor, the owner of the email service, made changes to their service on the 1st of June 2023, by removing a free alternative in using the service and enhancing their spam-filter. We found it likely that these changes reduced our ability to reach these panel members. Before and during wave 28 was fielded, attempts to counter these changes have been made.
A letter was distributed to the affected panel members a week before wave 28 was fielded. The letter provided information about the changes in the e-mail service, along with a link to a form where the panel memebr could change their registered e-mail address. In case the respondent wished to keep their original e-mail address, the letter also provided a step-by-step guide on how to address as a safe sender in order to avoid the spam-filter.
3 334 letters were distributed. Panel members who previously had indicated that they had moved (question fielded in wave 14 and wave 27) were exluded as address information stems from each panel members respective wave of recruitment.
424 letters were returned, in that they couldn’t be delivered to the recipient. An additional 26 panel members were deregistered for various reasons. 148 panel members accessed the form to register a new e-mail address, and 130 of those registered a new e-mail address. Of the 3 334 respondents who were contacted due to Telenor’s change in service, 940 participated in wave 28. By comparison 1 030 participated in wave 27. In wave 28 approximately 50 percent of them respondend after receiving a reminder by text message, rather than the email. Contacting the respondents by text proves to be an important tool in order to meet the issues cause by the change in Telenor’s change in email service.
| Wave | Number of respondents |
|---|---|
| Wave 24 | 1 683 |
| Wave 26 | 1 842 |
| Wave 27 | 1 030 |
| Wave 28 | 940 |
Reactivation of inactive panel members
Most respondents fail to actively opt out of the panel before they stop answering surveys. Therefore, there are more than 10 000 inactive panel members in NCP. An inactive panel member is understood as a member who has not participated in any of the last three waves. There are many reasons why respondents become inactive. The respondent might have passed away, or they have changed their contact information and failed to update it. However, completing three surveys a year creates attrition, and inactivity can also be understood as an implicit opt-out. During wave 28, reactivating a selection of inactive panel members has been attempted.
Reactivating inactive panel members is a different approach to recruitment: it yields information on whether reactivation might cause a similar increase in active panel members to normal recruitment-processes, and how hard it is to come in contact postally with inactive panel members.
In order to test the potential in reactivation, 1 000 inactive panel members were randomly selected. None of these used the e-mail domain @online.no, or had indicated that they had moved in previous waves (r14_Pmoved and r27_Pmoved). Instead of the normal e-mail invitation, they received a postal invitation to wave 28, which was distributed at the start of the field period. Subsequent reminders were distributed by email and/or text message.
| Male | Female | No education/primary school | Upper secondary | University College/University | <= 1959 | 1960-1989 | >= 1990 | |
|---|---|---|---|---|---|---|---|---|
| Sample | 47.5 % | 52.5 % | 10.7 % | 33.8 % | 45 % | 20.5 % | 55.8 % | 22.9 % |
| Respondents | 54 % | 46 % | 12.7 % | 28.6 % | 55.6 % | 23.8 % | 57.1 % | 19 % |
281 letters were returned as they failed to reach their intended recipient. 63 inactive panel members became respondents as a result of reactivation. Table below outlines characteristics of the sample and the reactivated respondents.
Although only 63 respondents, when comparing the characteristics of the respondents to the random sample of panel members, the distribution in age is fairly similar. It also seems that panel members with higher education is easier to reactivate, a fairly common issue in representativeness. Gender skews differently when compared to the sample. Comparisons between the two groups should be made with caution, given the relative low number of reactivated panel members.
The reactivation effort yields a reactivation rate of 8.7 % (63/(1000-281)*100). The reactivation rate is similar to the recruitment rate in the last three waves where recruitment was carried out: wave 25 8.3 %, wave 22 8.4 %, and wave 18 8.0 %, when comparing to the first point of contact, which all were letters in the aforementioned waves5. It is also similar when compared to recruitment attempts further back in time.
Response of Existing Panel Members Over Time
Comparing the number of wave 28 respondents (10 242) to the number of respondents in the previous wave 27 (9 997), gives an overall wave-to-wave retention rate of 102 percent. Figure 1 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 29 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 29th data collection (t29). For respondents recruited in wave 22, the current wave is the seventh data collection (t7).
The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent. Across all waves of recruitment, the current wave has a retention rate of more than 100 percent. In other words, more respondents participated in wave 28 compared to wave 27.
Figure 1: Wave-to-wave retention rate