Norwegian Citizen Panel
Methodology
Report
30th wave - revised release
ideas2evidence
September, 2024
To cite this report, please use the following:
Bjørnebekk, O.,
Wettergreen, J., Grendal, O., & Stokke, G. (2024). Norwegian Citizen
Panel methodology report, wave 30 [Produced by Ideas2evidence].
Background
This report describes the procedures of data collection in the 30th wave of The Norwegian Citizen Panel. Technical aspects of data collection are discussed, along with the representativity of the panel, and how survey weights are calculated.
The Norwegian Citizen Panel (NCP) is one of the main components of Digital Social Science Core Facility (DIGSSCORE) at the University of Bergen. NCP is as a collaboration between several departments at the Faculty of Social Sciences at the University of Bergen and NORCE.
ideas2evidence is responsible for the panel recruitment, the administration of the panel, and the technical solutions regarding data collection and computing.
Technical Aspects of the Survey
Software
The surveys are administrated through the web-based survey software Confirmit (now part of the company Forsta). Confirmit is a “Software-as-a-Service” solution, where all software runs on Confirmit’s continuously monitored server park, and where survey respondents and developers interact with the system through various web-based interfaces. This software provides very high data security and operational stability. The security measures are the most stringent in the industry, and Confirmit guarantees 99.7 percent uptime. ideas2evidence does the programming of the survey in Confirmit on behalf of The Norwegian Citizen Panel.
Pilot, Soft Launch, and Distribution
The survey went through small-N pilot testing before data collection. In addition, the survey was tested extensively during the development phase by ideas2evidence and the researchers involved in the project.
The field period started by inviting a random sample of high participation respondents (soft launch). This was done in order to minimize the consequences if the questionnaire contained technical errors. No such errors were located/reported, and remaining panel members was therefore invited shortly.
Throughout the field period respondents was randomly allocated into one of five subgroups, all of which were exposed to different questions sets. Group 6 is reserved for newly recruited panel members. However, due to a technical fault, 11 respondents in group 6 are not newly recruited panel members. The issue was addressed during the field period. Respondents can be identified by cross-tabulating the variable r30_group with the variable recruited. Respondents in group 6 was randomly allocated a set of question which were exposed to respondents in group 2, 3, 4, and 5. Group 1 is therefore technically reserved for existing panel members.
Randomisation Procedures
Each wave of NCP has an extensive use of randomisation procedures. The context of each randomisation procedure may vary1, but they all share some common characteristics that will be described below.
All randomisation procedures are executed live in the questionnaire. This means that the randomisation takes place while the respondent is in the questionnaire, as opposed to pre-defined randomisations that are uploaded to the questionnaire. All randomisations are independent from another, unless the documentation states otherwise.
Randomisation functions are written in JavaScript. Math.random()2 is widely used in combination with Math.Floor()3. These functions are deployed to achieve:
- Randomly select one value from a vector
- Randomly shuffle the contents of an array
The first procedure is typically used to determine a random sample of respondents to separate groups, for instance a treatment group within an experiment. As an example, consider an environment where we want to separate all respondents in two groups: group 1 and group 2. All respondents are randomly assigned the value 1 or 2, where each randomisation is independent from respondent to respondent. When N is sufficiently large, the groups will be of equal size.
Here is an example of the JavaScript code executed in Confirmit:
var form = f("x1");
if(!form.toBoolean()) { //If no previous randomisation on x1
var precodes = x1.domainValues(); //copies the length of x1
var randomNumber: float = Math.random()*precodes.length;
var randomIndex: int = Math.floor(randomNumber);
var code = precodes[randomIndex];
form.set(code);
}The second procedure is typically used when defining the order of an answer list as random. This can be useful when asking for the respondent’s party preference or in a list experiment. However, as a party cannot be listed twice, the procedure must take into account that the array of parties is reduced by 1 for each randomisation. Here is an example4:
function shuffle(array){
var currentIndex = array.length, temporaryValue, randomIndex;
//While there remain elements to shuffle
while (0 != currentIndex) {
randomIndex = Math.floor(Math.random() * currentIndex);
currentIndex -= 1;
// And swap it with the current element
temporaryValue = array[currentIndex];
array[currentIndex] = array[randomIndex];
array[randomIndex] = temporaryValue;
}
return array;
}Previous Waves of Recruitment
Existing panel members were recruited in wave 1, wave 3, wave 8, wave 11, wave 14, wave 16, wave 18, wave 22, and wave 25. All samples were drawn from the National Population Registry of Norway. This registry holds information on everyone born in Norway, as well as former and current inhabitants. The Norwegian Tax Administration holds the formal responsibility for the registry.
Samples consist of people over the age of 18 who were randomly drawn from the registry. The extracted information was a) last name, b) first name, c) address, d) gender, e) year of birth, and f) phone number (the latter was not included in wave 1). Samples exclude people without a permanent address in Norway.
Table 1 outlines a short summary of previous recruitment efforts. Note that there are some differences between the recruitment processes. For a detailed description of each recruitment process, please refer to the respective methodology reports. A detailed description of the recruitment in wave 30 follows in the next section.
| Event | Sample size | Mode | Contacts | Returned letters | Response rate (%) |
|---|---|---|---|---|---|
| Recruitment 1 (wave 1) | 25 000 | Postal | 2 | 546 | 20.1 |
| Recruitment 2 (wave 3) | 25 000 | Postal, phone/SMS | 4 | 543 | 23.0 |
| Recruitment 3 (wave 8) | 22 000 | Postal/SMS | 3 | 479 | 19.4 |
| Recruitment 4 (wave 11) | 14 000 | Postal/SMS | 2 | 334 | 15.1 |
| Recruitment 5 (wave 14) | 14 000 | Postal/SMS | 2 | 389 | 15.0 |
| Recruitment 6 (wave 16) | 34 000 | Postal/SMS | 2 | 994 | 14.9 |
| Recruitment 7 (wave 18) | 15 000 | Postal/SMS | 2 | 381 | 14.0 |
| Recruitment 8 (wave 22) | 23 000 | Postal/SMS | 2 | 623 | 14.5 |
| Recruitment 9 (wave 25) | 18 000 | Postal/SMS | 2 | 562 | 13.9 |
| Recruitment 10 (wave 30) | 25 000 | Postal/SMS | 4 | 989 | 15.0 |
The response rate of recruitment in recruitment 4 and onwards were lower than recruitment in previous waves. The most important explanation were restrictions enforced by the Norwegian Tax Administration with regards to how many times the Citizen Panel can contact the invitees. Respondents in recruitments 4-9 were contacted twice at most. Recruitment 1 also had a maximum of two contact points, but achieved a response rate of 20 percent. One explanation for why we cannot replicate a response rate of 20 percent in recruitments 4-10 might be that NCP did a lot of promotion of the panel through media outlets leading up to and during recruitment 1. Additional promotion of the panel was carried out in relation to the Norwegian Parliamentary election the same fall. We also observe a slow, declining, recruitment rate after the fourth recruitment until recruitment 10 where the restriction on maximum contact points was lifted.
Data Collection
Recruiting a new set of panel members
In wave 30, The Norwegian Citizen Panel recruited new panel members. This section gives a detailed description of the sample frame, recruitment process, and the results. Compared to previous waves of recruitment, NCP was no longer limited by only making a maximum of two points of contact towards attempted recruited panel members.
The recruitment process
NCP attempts to recruit persons age 18 or above to the panel regularly. The probability to be attempted recruited varies between the persons in Norway, where younger persons have a lower probability to be recruited as many of them have not been eligible for participation in previous rounds of recruitment, and thus have a lower overall probability of having been invited. To account and correct for that, gradual stratified sampling has been used to recruit persons in wave 30. The fewer chances a person has had to be attempted recruited previously, the larger the probability to be attempted recruited with the gradual stratified approach. People aged 18-29 were oversampled to correct for the recruitment probability. Within that group, a further oversampling was done for the younger persons. As recruitment is not necessarily carried out every calendar year, this is not a linear oversampling year-for-year from 18-29, but the younger people within this group generally enjoyed greater probability to be attempted recruited when compared to the older persons within the group, and more so when compared to the the part of the sample which were sampled of the general population as long as the person is 18 or older.
A stratified sample gross sample was drawn from the population registry. Approximately 17 500 people aged 18 - 29 were drawn from the register, while approximately 7 500 people over the age of 18 were randomly drawn. The extracted information included a) surname, b) first name, c) address, d) gender, and e) year of birth. The sample excluded individuals without a current home address in Norway. Telephone numbers were attached to the sample by using Data Factory. The availability of phone number varied with age groups within the sample. The younger people had a lower coverage in general. Table 2 below sums up the coverage ratio of phone number by age groups within the sample.
| Age group | Registered with phone number (%) | Not registered with a phone number (%) |
|---|---|---|
| 18 - 24 | 38.6 | 61.4 |
| 25 - 34 | 49.3 | 50.7 |
| 35 - 44 | 49.2 | 50.8 |
| 45 - 64 | 61.6 | 38.4 |
| 65 - 84 | 75.6 | 24.4 |
| 85 and over | 79.1 | 20.9 |
Information on the different points of contact can be found below. If the recipient was under 30 years of age the content of the letter, postcard, and SMS varied textually from the letter, postcard, and SMS sent to recipients aged 30 and above. If the recipient was not registered with a phone number they received the letter of invitation and postcard(s). The remainder of the sample also received one or multiple SMS.
| Date | Type of Contact |
|---|---|
| 4th of June | Letter of invitation |
| 11th of June | First postcard |
| 14th of June | First SMS |
| 20th of June | Second SMS |
| 24th of June | Second postcard |
| 27th of June | Third SMS |
Initially, letters were sent to everyone in the sample. The letters contained the following information: a) a description of the project, b) the Citizen Panel’s policy on privacy and measures taken to protect the anonymity of the participants, c) the time-frame of the project, d) the participants’ rights to opt out of the panel at any time in the future, e) contact information for the people responsible for the project, f) a unique log-in id and the web address to the panel’s web site and g) the estimated time required to complete the survey.
In order to maximize the response rate, an incentive in the form of three gift cards is included in the project. The values of the gift cards are 8 000 NOK. To enter the lottery respondents were required to join the panel and provide their email addresses. Respondents were asked to register on the panel’s website and log into the survey using the unique id-code provided in their personal letter.
All reminders were sent to respondents who a) had not logged into the survey, or b) had not completed the survey. Respondents were encouraged to join the panel, with reference to the invitation letter. The unique log-in id provided in the original letter was included in the postcard and the SMS. The SMS reminder(s) also included a direct link to the survey.
Results of the recruitment process - survey respondents and panel members
It is necessary to make a distinction between panel members and survey respondents. Panel members are defined as respondents who register their e-mail address in the survey, regardless of whether they have completed the questionnaire or not. Survey respondents are respondents who have completed a certain share of the questionnaire, regardless of whether they have entered their e-mail or not.
Of the 24 917 letters that were sent out, 989 were returned, and 14 respondents opted out. 22.4 percent (5 364) of the remaining 23 914 logged on and accessed the survey. 3 466 individuals completed the questionnaire, while 1 898 exited before completion. 7.7 percent of these responses are kept as a part of the survey data as these respondents completed a certain amount of the questionnaire before exiting. The remaining 1 751 incomplete responses were excluded from the data set, due to a lack of data.
In sum, after subtracting a few cases where the credentials of the respondent did not match the credentials of the invited, this recruitment wave resulted in 3 576 new survey respondents, a recruitment rate of 15 percent. This is higher that what was achieved in recruitment 9 (13.9%). 98.3 percent of the respondents who completed the survey submitted their e-mail address. 3 746 new panel members were recruited to the Norwegian Citizen Panel, resulting in a panel recruitment rate of 15.7 percent.
Further discussions in this report about respondents recruited in wave 30 are based on survey respondents. As survey respondents and panel members closely overlap, these descriptions can be assumed valid for the panel members as well.
Responses by mode of contact
In previous waves of NCP where recruitment has been carried out, it has been fairly trivial in understanding what mode of contact (letter, postcard, or SMS) made the respondent enter the survey. However, wave 30 used more points and modes of contact that heavily overlapped during the field period, and trying to construct a table as done in previous waves might introduce errors and give an erroneous understanding of who answered when, and as a response to what mode of contact. Instead, a simplified table can be found below, strictly differentiating between those respondents who likely answered in response to an SMS, and those who likely did not. In short, if the respondent received an SMS and started the survey on the same day as the SMS was distributed, the table assume that the respondent answered due to the SMS. If they did not receive an SMS, or answered on a different date than the SMS was sent out, the table assumes that the respondent answered due to the letter of invitation or subsequent postcards.
| Mode of contact | Response | Response rate (%) | Cumulative responses | Cumulative response rate (%) |
|---|---|---|---|---|
| Likely a response to the letter of invitation or subsequent postcards | 2 651 | 11.1 | 2 651 | 11.1 |
| Likely a response to SMS-reminder(s) | 925 | 3.9 | 3 576 | 15.0 |
To understand when the newly recruited answered in relation to the different reminders, the figure below outlines when the questionnaire was started among the newly recruited respondents.
Figure
1: Responses counted day by day among respondents recruited in wave
30
Figure 1 outlines both the gradual incline of responses assumed created
by the letter of invitation, and by comparison, the steep incline in
number of responses when SMS is used as mode of contact.
Given the stratified approach to sampling, it is assumed to the level of response is influenced by the age or age group of the persons attempted recruited. Table 5 shows the response rate according to the groups in which the sample was stratified into: 18-29 and 30 and above. In line with the expectation that it is harder to recruit younger people, table 5 shows that the response rate among the stratified sampling group was lower when compared to the older people.
| Age group | Sample | Number of respondents | Responserate (%) |
|---|---|---|---|
| 18 - 29 | 17 475 | 2 224 | 12.7 |
| 30 or older | 7 525 | 1 352 | 18.0 |
Responses by Existing Panel Members
The survey was distributed to 26 497 panel members on the 3rd of June 2024 for the soft launch and 4th of June for the main launch. The invitation contained information on the Norwegian Citizen Panel, unique URLs for each panel member that led to the questionnaire, and unique access code which the panel members could use to log in to the survey by accessing a link on www.uib.no/medborger.
The invitation, first reminder, and third reminder were all distributed by e-mail. The second reminder was, depending on whether the panel member had a registered mobile phone number or not, distributed via SMS or e-mail. Prior to wave 30, 51.5 percent of the panel members were registered with a mobile phone number.
| Event | Response | Cumulative responses | Response rate | Cumulative response rate |
|---|---|---|---|---|
| Invitation (3rd/4th of June) | 5 085 | 5 085 | 39.6 % | 39.6 % |
| First reminder (11th of June) | 2 651 | 7 736 | 20.6 % | 60.2 % |
| Second reminder - email (19th of June) | 314 | 8 050 | 2.4 % | 62.6 % |
| Second reminder - SMS (19th of June) | 841 | 8 891 | 6.5 % | 69.1 % |
| Third reminder (27th of June) | 1 042 | 9 933 | 8.1 % | 77.2 % |
In total 9 933 existing panel members filled out the questionnaire. A response rate of 39.6 % was achieved between the invitation and the first reminder. Following a pattern observed in previous waves, the initial invitation produced a higher number of respondents than subsequent reminders. See table 6 for further details on number of respondents after reminders.
Using the same methodology as in previous waves for calculating response rate, respondents who have not participated in any of the last three waves are excluded. This leaves us with 12 853 eligible respondents. The overall response rate, as reported in table 6, is 77.2 %.
Approximately 1 600 of the initial invitations were reported as not delivered by Confirmit, which rounds to 3 percent. Measures are taken to ensure email deliverability, but are unable to accurately estimate how many of the delivered emails ended up as spam with the recipient.
Response of Existing Panel Members Over Time
Comparing the number of wave 30 respondents (9 933) to the number of respondents in the previous wave 29 (10 099), gives an overall wave-to-wave retention rate of 98 percent. Figure 2 illustrates each wave of recruitment by individual lines, and shows how many respondents that are preserved for each data collection. NCP has carried out 31 waves of data collection. Depending on when the respondents were recruited, the current wave is highlighted with a red circle. For respondents recruited in wave 1, the current wave is the 31th data collection (t31). For respondents recruited in wave 22, the current wave is the ninth data collection (t9).
The wave-to-wave retention rate increases substantially after the first three waves (t1 - t3), until it stabilizes around a mean of 95 percent. Across five out of ten waves of recruitment, the current wave has a retention rate of more than 100 percent. In other words, more respondents participated in wave 30 compared to wave 29 in these.
Figure 2: Wave-to-wave retention rate