How reliable is survey data? Many surveys do meet the necessary robustness criteria, in terms of being composed of a suitable sample size, drawn from representative population, and subject to significance testing against an appropriate probability or confidence interval.
Other surveys are less reliable. The ones that all into this camp are small sample sizes, unrepresentative and with no attempt made to assess the significance of the data. Some also carry very large percentages attempting to support a particular position. Such surveys also lack any presentation of the central tendency (like the mode, the most common; or the median, the point where 50 percent of the responses are in one direction and 50 percent of the responses are in another direction).
Such surveys are often biased or at least contain leading questions. Or selective bits are extracted from the survey to convey points that are convenient to the public relations company or to they are selective picked by the journalist penning an article (a form of reporting bias). We can define bias as “selective revealing or suppression of information”, and this is sometimes too apparent.
Let’s take an example, in terms of an employment survey that came across the Digital Journal desk. This concerned the place of work.
The differences of opinion between the work-from-home and the need-to-be-at-work contingent is fairly large. In the middle of this are the hybrid advocates. The question of what is best for the employee cannot be easily answered as social and psychological factors come into play, as well as work metrics. Furthermore, there are many jobs where working from home is simply not an option. This creates challenges in workplaces where there are office and non-office jobs, in relation to equitable working practices.
As to what workers make of these issues, surveys tend to have differing results. The survey in question finds that 94 percent of employees a believe they should be able to work from anywhere, so long as they get their work done.
The survey also suggests that a ‘failure’ to provide that flexibility by the employer makes hiring and retention more difficult.
The 94 percent is a very large figure and it perhaps highlights issues with survey responses to statements that contain ‘agree/disagree’ scales. The primary concern is that such approaches to data gathering tend to lead to a positive bias.
Moreover, for those who neither agree nor disagree with the statement put in front of them, this can be interpreted in different ways. For example, this may represent either be a “hidden don’t know” (the respondent has no opinion) or it can mean a neutral opinion (the respondent is somewhere between agreeing and disagreeing) or it can simply be that the respondent is not inclined to think about the answer.
These types of questions are often ‘leading’ or ‘loaded’ in a particular way where there are either insufficient options or the question is hard to disagree with. For example:
- Leading question: Would you say working from home is better where it cuts out the commute?
- Loaded question: Do you enjoy working from home?
Surveys based on structured or semi-structured interviews tend to elicit data that is more meaningful, and they also provide opportunity for the interviewer to probe or to ask follow-up questions, especially where more extreme claims are made. These types of surveys cost money and they may not lead to the ‘headline grabbing’ soundbite beloved of public relations agencies and, it must be said, a sizable splattering of journalists.
The survey also singles out office workers and it finds favourably towards flexible work arrangements. The survey also infers that people will seek to change jobs in order to secure a job where the commute is rarely required.
The study was also focused on workers in the U.S. and the U.K. (countries with similar forms of capitalist work relations), made up of around 1,500 full-time office workers. Each of the firms was trading as an international company. Hence, the survey data needs to be looked at as applying to a narrow subpopulation of workers.
Furthermore, the timeframe over which data was gathered is very narrow: the data was collected between December 22, 2021 and January 11, 2022. This time of year is generally associated with workers looking to change jobs, sometimes as a way to battle the ‘January blues’.
Hence, this particular survey had indications of bias and it would not have been a reliable one to present to readers of Digital Journal.
