To improve the measurement of women’s work and provide guidance on good measurement practices, the International Labour Organization (ILO) and the World Bank collaborated with the Sri Lanka Department of Census and Statistics (DCS) between 2017 and 2019 on a multi-stage pilot study. The study compared the outcomes of two types of household surveys: the labor force survey (LFS) and a multi-topic living standards survey (MLSS). LFSs are the primary data source for labor market indicators, including the unemployment rate and labor force participation rate. MLSSs are designed to allow broader measurement and analysis of living standards and poverty. Due to these different primary objectives, the two survey types differ in questionnaire length, structure, sample size, and other elements, meaning they do not always yield comparable results.
This difference in outcomes can be concerning for users who can often assume that an indicator carrying the same label – such as unemployment – should amount to approximately the same thing regardless of the source.
The Sri Lanka study, which was supported by Hewlett Foundation and Data2X under the Women’s Work and Employment Partnership, yielded key findings (now published) that could help solve persistent measurement problems and generally improve our understanding of the issue of comparability across sources. In this blog, we highlight three findings:
Different types of surveys won’t always give the same results, but gaps between them can be reduced
An important insight from the Sri Lanka pilot study is that the size of measurement gaps can be surprisingly large.
Researchers completed two rounds of interviews for the LFS and MLSS to discover any key differences between the two survey types and test whether changes in questionnaire design and implementation could yield improvements. In the first wave of data collection, the MLSS questionnaire captured less employment than the LFS – with a gap of about 5.5 percentage points overall and 8.1 percentage points for women in the employment-to-population ratio (see Figure 1). The size of this gap could have important implications for the use of the data for countries that use different sources over time.
In the second wave of the study, a few additional questions were added to the MLSS labor module, for example: a question on contributing family workers in agriculture and “recovery questions” for small/casual jobs (i.e., additional questions worded specifically to target such jobs if not reported by respondents with the original set of questions). As a result of these and a few other changes, the gap between the two surveys was substantially reduced, with the MLSS now only reporting a 3.5 percentage points lower employment-to-population ratio than the LFS overall, and 2.5 percentage points for women.
Whether differences of this scale remain of concern depends on the intended use of the data. It’s important to remember that the source of statistics does make a difference, something that should be kept in mind when planning and analyzing surveys.
-
Women’s work is at greater risk of mismeasurement
While the wave 1 gap between the LFS and MLSS in the measured employment-to-population ratio was 8.1 percentage points for women, it was only 2.4 percentage points for men (Figure 1). Furthermore, both surveys identified essentially the same number of people engaged in full-time employment.
In general, measurement difficulties were concentrated among people in casual, low-hours jobs or helping on family farms or in businesses — activities more common among women respondents. A possible explanation for this gap may be that, due to traditional gender role beliefs, women working in family businesses self-identify as housewives and therefore do not answer affirmatively when asked about their engagement in work for pay or profit, a pattern that has also been observed in various studies, including a recent one in Honduras.
-
Good translation matters
Some of the difficulties in the measurement of farming work in wave 1 of data collection in the MLSS were a result of not using everyday Sinhalese terminology for some of the farming-related questions. In wave 2, the MLSS sought to avoid abstract terminology (e.g., “own account crop farming”) and instead opted for simpler terms with examples (e.g., “work on a family farm to prepare or maintain the land, or to plant, grow or harvest any crops vegetables or fruits”). The change likely contributed to closing the gap with the LFS.
These and other findings from the Sri Lanka pilot study have been used to update the guidance, tools, and support of the ILO for LFSs and of the World Bank for multi-topic household surveys. These model questionnaires, guidance, and tools are excellent reference points for those facing the task of designing a questionnaire to capture work- and labor-related issues through a household survey in line with the latest statistical standards.
This blog is based on a presentation delivered by the authors at the Jobs and Development Conference in September 2021.
