2 Sampling methodology

Sample surveys such as the NZCASS offer a highly cost-effective alternative to censuses. Confidence in their results rests on the sample design, which ideally aims to produce a probability sample or a close approximation to one. In a probability sample, every item in the population has a known, non-zero chance of being selected. Sampling theory provides a strong foundation for the estimation of population characteristics from a probability sample. Deviations from a pure probability sample are hard to avoid in practice, however, as are measurement problems, and these can affect the accuracy of survey estimates. We attempt to highlight the most significant issues in this report, except where these have already been dealt with in the Key Findings.

Perhaps the simplest probability sampling method is a simple random sample, in which every possible sample is equally likely to be selected. However, this method is not practical for face-to-face surveys of the general New Zealand resident population. The NZCASS has used a multistage design, almost identical to that used in the 1996 and 2001 NSCVs. Areas were selected first. Then households within areas were selected. After this, one respondent within each household was selected. Each of these steps forms a distinct sampling stage.

The final stage in the sample design is the selection of a small number of incidents from those experienced by respondents. Each of these four stages will be detailed further below.

To give more reliable results for Māori, a Māori booster sample was selected independently[2] of the main sample of the general population. The booster sample used a similar multistage sample design, while screening out non-Māori by asking the initial contact whether anyone in the household aged 15 or more would identify as Māori. The target sample size for the booster sample was 1,600 interviews; the main sample was designed to achieve 4,000 interviews.

Survey population

The survey population was the total usually resident, non-institutionalised, civilian population of New Zealand aged 15 years and over. As such it excludes long-term residents of old peoples’ homes, hospitals and psychiatric institutions; inmates of penal institutions; those members of the New Zealand armed forces who live in institutional settings; non-New Zealand diplomats and their non-New Zealand staff; members of non-New Zealand armed forces stationed in New Zealand; overseas visitors stationed in New Zealand for less than twelve months; and residents of offshore islands, except Waiheke.

The ABS methodological review of the 1996 NSCV recommended that the official definition of "usually resident" be adopted. This was used for the 2001 sweep, and has been used again in 2006. It mandates respondent self-definition of "usually resident", with a number of exceptions. These are defined in Statistics New Zealand’s "Statistical Standard for Usual Residence 1999".

Area selection

ACNielsen maintains a sampling frame that divides New Zealand into agglomerations of meshblocks[3] called Nielsen Area Units (NAUs). This frame was used for both the 1996 and 2001 surveys, and was used again for the 2006 NZCASS. Details of the frame are given in Appendix A3. Offshore islands (except Waiheke Island) were excluded from the survey.

A stratified systematic sample was selected with probability proportional to size, with replacement, for both the main and booster samples using Gambino’s (2003) pps package in R (R Development Core Team, 2006). The size measure used for the main sample was the number of households in the Nielsen Area Unit according to the 2001 census. For the booster sample, an estimate of the number of households where at least one Māori person usually lived was used.

Growth or decline in area unit populations since the 2001 census was not accommodated in the sample design or weighting. Although this introduced some potential bias, this is believed to be quite small, as explained later in Appendix A4. Accepting some bias to avoid the fieldwork complications of such accommodations is often felt to be a worthwhile trade-off in face-to-face surveys.

A cluster of interviews was conducted in each selected Nielsen Area Unit. As in the 2001 survey, the number of designated dwellings to be approached per cluster was determined by a combination of anticipated response rate and the desirability of spreading the sample geographically as widely as possible, while retaining the cost efficiencies that cluster interviewing provides. With a main sample size of 4,000, an average of 5 completed interviews per cluster was targeted, providing a good compromise between sample spread and cost efficiency. This meant 800 areas were needed for the main sample, and 320 for a Māori booster sample with a target of 1,600 interviews.

The stratification system employed split the country into both geographical regions and levels of urban/rural density, based upon Statistics NZ definitions. The following tables show the resulting region-by- level-of-urbanisation grid. The sample size given in each row is directly proportional to that row’s share of the total number of households in occupied private dwellings (or of Māori households, for the booster sample).

Table 2.1 Stratification of the main sample

Because the proportion of Māori people is relatively small, and they tend to be clustered residentially, some NAUs contained very few Māori people. For the booster sample, trying to interview in these areas can be very unproductive. Having recruited ethnic booster samples on many occasions, ACNielsen have addressed the problem of maximising both cost efficiency and sample representativeness, given the "needle in a haystack" nature of the sampling task.

The solution here was to delete from the sampling frame those NAUs with a low Māori density. As in the 2001 survey, NAUs where less than 5% of dwellings contain Māori were removed from the sampling frame for the booster sample. This accounted for 3% of NAUs, but only 0.2% of Māori households.[4]

Table 2.2 Stratification of the Māori booster sample

Household selection

The cluster sampling procedure used for this survey (and in the 2001 survey) involved providing the interviewer with a randomly selected start point. This was a numbered house on a named street, and interviewers then needed to call on every fourth dwelling from that point (following a pre-determined walk pattern). This was until an outcome was obtained from every designated dwelling in the cluster.

This dwelling interval reduced the potential "word of mouth" effect from interviewing at adjacent addresses, as was done in 1996, and criticised in the ABS review. It also minimised the clustering effect to some extent. In rural areas consecutive dwellings were approached to minimise travel costs. Based upon a target response rate of 65% and a targeted average number of achieved interviews of five per cluster, this initially meant that eight designated dwellings per cluster were to be approached. However, based on the pilot feedback and response rates, this was revised to nine designated dwellings for the final survey.

Māori booster sample

For the Māori booster sample, the number of designated dwellings approached per cluster was 40. This was based on the assumption that Māori incidence and response rates would be broadly similar to the 2001 NSCV (or a little lower following the 49% response rate in our 2005 pilot). Because of the screening design used, a dwelling interval of four was not necessary here. Even after eliminating those Nielsen Area Units with very small Māori populations, the great majority of dwellings would still be ethnically ineligible. Thus, every second dwelling was approached in urban areas for the booster sample.[5]

Non-private dwellings were disregarded when following the walk pattern, while empty dwellings were recorded but not counted towards the required number of dwellings to be approached.

Respondent selection

Because many types of victimisation are household-based, only one respondent per dwelling was selected. This provides efficient measurement of household victimisation, and avoids potential contamination effects that might arise if more than one person in a household was interviewed. As will be discussed later in Chapter 8, weights for person-based estimates will therefore need to incorporate the number of residents aged 15+ per household to remove any household size biasing effect. This is a routine statistical procedure in household-based survey research.

To select the respondent, we asked for a list of the names and birth month of every household resident aged 15+ from the person who answers the door and then selected for interview the one who has the next birthday. For situations where the next birthday procedure was not workable, the contact sheet included an alternative procedure based on the alphabetical order of first names.

For the Māori booster sample, only Māori aged 15+ were eligible for selection. In the 2001 survey, the booster sample Contact Sheet had the following question after a general introduction: “……Is there anyone in this household who is Māori and aged 15 years or older?” In the 2006 survey we changed the wording to be more consistent with the Census-style ethnicity question in the body of the questionnaire as follows: “... Is there anyone usually living here aged 15 years or older who might consider themselves Māori? That is, if asked which ethnic group or groups they belong to, they would include Māori.”

Selection of incidents

At four points during the interview (namely at the main screener questions, and at each of the three sets of self-completion screener questions), respondents are asked how many incidents they had experienced since 1 January 2005 of various types of crime. More detailed information is then requested through a "victim form" for some of these incidents. (See Box 3.1 in the next chapter for an outline of the questionnaire.)

A feature of the New Zealand victimisation surveys is the substantial proportion of the reported incidents that were experienced by heavily victimised respondents. For instance, half of all victimisations were experienced by the 6% of people who experienced five or more offences, and several people experienced more than fifty incidents.[6] Since completing a victim form takes about 10 minutes, it would not be feasible to get heavily victimised respondents to fill in a form for each of the incidents they experienced. Instead, if a respondent recorded more than three incidents at the main interviewer-administered victimisation screener questions, victim forms were completed for three incidents randomly selected by the CAPI software. (If there were three incidents or fewer, victim forms were completed for them all.) The victim forms collect detailed information about people’s experience of and feelings about victimisation, and provide critical inputs to the calculation of victimisation rates.

The sample design for selecting incidents aimed to enhance the accuracy of incidence and prevalence rates for key offence types, and provide sufficient victim form information on the characteristics of major offence types, while maintaining some consistency with the approach used in the 2001 survey. Simulations based on 2001 survey data were run for various candidate designs to explore how these affected the number of victim forms completed for various offence types. Designs producing over 400 victim forms for most offence types were preferred.

Incidents were selected independently, without replacement, with selection probabilities proportional to the weight given to the incidents' screener questions. Screener questions fell into three priority categories (low, medium and high, as shown below), with corresponding selection weights 1, 2 and 3.[7] The probability of selection for a particular incident depended on both the extent of competition from other incidents, and the screener question that the incident was recorded at.

Low priority (weight=1):

• Q29 (things stolen from/off vehicle)
• Q30 (vehicle tampering/damage)
• Q32 (successful burglary)
• Q34 (theft from property – outside, >$10)
• Q43 (other)

Medium priority (weight=2):

• Q31 (unsucessful burglary)
• Q35 (theft of household property – inside)
• Q37 (other theft)
• Q39 (threatened to damage)
• Q35.416 (damage – household property)

High priority (weight=3):

• Q28 (vehicle stolen)
• Q36 (theft from person)
• Q38 (damage – personal property)
• Q35.40 (assault)
• Q35.41 (threatened to assault)

Denote the selection weight for incident i by w_i. Then the probability of selection for incident j for a particular victim form is w_j / sum (w_i), where the sum is taken over all incidents available for selection. (Incidents that have already been selected would not be included in the sum.)

Once this incident selection design was implemented in CAPI, extensive simulation tests were conducted (along with code review) to ensure that it worked as intended.

Questions analogous to those in the victim form were asked about the most recent incident recorded at each of the three self-completion questionnaire sections. Some of the heaviest victimisation is recorded in these sections (as would be expected from their coverage of victimisation within ongoing relationships). Because only one "victim form" was allocated to each section, the probability of selecting incidents experienced by these heavily victimised respondents was very low. This results in highly variable incident weights, and may mean that the incidents with missing data are not similar to other incidents, so the accuracy of victimisation estimates could probably be improved by reallocating some of the interview time from interviewer-administered victim forms to self-completion incidents.

A similar approach was taken in the 2001 survey, although randomised incident selection was conducted manually,[8] with a simpler sample design involving just high and low priority incidents. The approach in 1996 was different again, selecting four incidents deterministically[9] from the main questionnaire. The approach taken for self-completion incidents was the same in all three surveys.

One planned enhancement that was not implemented was the introduction of a mini victim form for respondents reporting more than 3 incidents. This would gather some information about another randomly selected incident; not every question in the full victim form, but just what was needed to determine whether the incident should count towards 2005 victimisation rates. While this proposal survived to an advanced stage of planning, it was eventually dropped because the addition of new questions on other topics meant that questionnaire length had become excessive.

Footnotes

2 Areas for the booster sample were selected independently from the main sample areas, with replacement. If an area was selected for both samples, or twice within one sample, the households to be approached for the second selected area were obtained by following ACNielsen’s usual procedure of setting the "start point" for the second area shortly after the last household selected for the first area.

3 Meshblocks are the smallest geographical unit used by Statistics New Zealand, used in the Census and many other surveys.

4 Consistent with this proportion, only one of the 511 Māori respondents in the main sample lived in an NAU outside the sampling frame for the Māori booster sample.

5 Two urban NAUs were selected for the Māori booster sample that contained fewer than 80 dwellings, and were thus too small to allow this procedure to be used. Consecutive dwellings were approached in these areas instead. (Both contained at least 40 dwellings.)

6 While the victimisation reported in the survey was highly concentrated among a small proportion of respondents, these reports generally did not seem implausible. There were no obvious gaps in the distribution of incidents experienced, and even the largest numbers of incidents reported seem feasible in the context of an ongoing abusive relationship, for instance, over a reference period lasting a year to 18 months.

7 These weights were chosen partly for simplicity, and partly to maintain some consistency with the 2001 survey (in which weights of 1 and 3 were used).

8 An automatic selection process was originally planned for the 2001 survey, but a manual randomised process had to be adopted due to limitations of the CAPI software used then.

9 In the 1996 survey, incidents were selected based on a ranking of offence types, with the most recent incident or incidents being selected when choosing between offences of the same type.