The appropriate statistical test, scales and satisfaction in customer surveys

Citation

Ole Pors, N. (2001), "The appropriate statistical test, scales and satisfaction in customer surveys", The Bottom Line, Vol. 14 No. 1. https://doi.org/10.1108/bl.2001.17014aab.013

The appropriate statistical test, scales and satisfaction in customer surveys

The appropriate statistical test, scales and satisfaction in customer surveys

Keywords: Statistics, User studies, Customer surveys, Libraries

As so many other persons in the field of library and information science, I have been involved in quite a lot of user surveys, satisfaction surveys and other types of investigation into the mediation between library systems and users.

Many of these studies have been based on questionnaires, which simply means that they are using a quantitative approach. There seems, at the moment, to be a trend to employ qualitative methods to a higher degree. In relation to the traditional survey, qualitative methods have the possibility to give more in-depth information about how customers really feel about information services. This is not the place to go into the merits and demerits of the different methodologies, but it will suffice to emphasise that the Servqual method is an attempt to use a quantitative approach that elicits, at least, some of same type of information it is possible to elicit employing a qualitative approach.

I guess that one of the things that have embarrassed quite a lot of both professionals and researchers has been that so many surveys show a rather high degree of satisfaction with services. Some have formulated that a library user will always express satisfaction independent of the objective quality of the service he or she has received.

The Servqual methodology tries to solve this problem by relating perceptions to experiences and analyse a possible gap. The methodology goes one step further, because it also measures the users perception or judgement of importance of the different service dimensions. Service standards, benchmarking and the like are other tools to change the perceptions of the users. These techniques are also used simply to raise the awareness of the potential of the library in relation to the user.

Survey example

Let me illustrate by an example. During summer 2000, I participated in the IFLA Conference in Jerusalem. I had the task to make the official evaluation of the participants' satisfaction in relation to different dimensions of the conference. The evaluation consists of a rather traditional questionnaire and includes focus group-like interviews. The evaluation is not finished yet, but quite a lot of the material has been analysed and it shows characteristics that I believe are transferable to user studies in libraries.

First of all, in the questionnaire, the scale used to express satisfaction or dissatisfaction is a five-point Likert-like scale or a numerical ordinal scale. The last one is often treated as an linear numeric scale. The scale goes from one to five and the average of most variables will be over three. Only very few people use the 1 as the lowest rating. It is exactly the same in user studies. The focus group-like interviews, four in total, all started the same way. A verbal expression with some kind of satisfaction, but after a while the participants expressed rather strong feelings with unsatisfactory aspects of the conference. They influenced each other both in negative and positive comments. I felt the impression from the focus group in total was a more negative evaluation than the evaluation conducted through the questionnaire. The other characteristics the survey forms show are a kind of response set and in some cases number giving in contrast to the written verbal explanation. It is not uncommon to see a rather harsh remark and the number three given to the same variable.

This experience indicates to me that most people are very wary of using the especially negative extreme of a scale if it is concerned with a topic that is not extremely important for them.

Translating practice

What we normally do in user surveys is to confront people with symmetrical answering options on a five- or seven-point scale. They are then asked to express their satisfaction with a service or offer an opinion on the quality of a service. There is no doubt that many questions or statements in user surveys in reality are of little importance for the users. And we also ask them questions about topics where their knowledge or awareness is rather low.

All in all this could mean that people use an answering scale differently than intended. Respondents are not inclined to make very negative statement about questions they do not know much about and which do not have a great importance for them. In other words, they use the answering scale in asymmetrical way. They tend to use the middle-point of the scale to express indifference or lack of knowledge. Some even use the positive point to express a general positive feeling about services in general. Such behavior could be some of the reasoning many surveys come out with rather positive results.

Data representation

We will often see the results from surveys expressed in means and standard deviations. We are treating people's answers on an ordinal scale as if they were expressed on an interval scale. It gives the possibility to use more advanced and precise statistical analysis, but surely some of the fundamental assumptions of statistical testing are violated.

Constructing appropriate scales is more an art than a science. We have suggested some problems using scales. Especially the middle point of a scale raises considerations, because respondents can treat it as stated or as the point they cross if the awareness or importance are low. Some simply skip the question. The wording in relation to the middle-point of a scale has always given rise to a certain controversy among researchers. It is difficult on a scale supposed to elicit information about behavior and attitudes, to design a wording that creates a consensus among respondents about the exact meaning. Often you will only word the extremes of a scale and let people interpret numbers. Then, the researcher, when writing the report has the job to word or interpret what the numbers signify.

It is evident that scaling an individual's perceptual mapping of the underlying values cause some problems when the analysis has to start. Parametric statistics are based on interval data and assumptions about an approximately normal distribution of the phenomenon one measures.

Closing cautions on statistics

I have indicated that the distribution of perceptions, attitudes and meanings are distributed asymmetrically, at least, in cases where importance and awareness are factors that influence the response pattern. You may ask yourself why the author feels that it is important to draw attention to these rather simple research questions. I do think that some people in the library and information profession have what could be labelled as a kind of numerical illiteracy. It does not, and should not, keep them from interpreting figures. But some caution would often be suitable.

In teaching, it is often impossible to cover all aspects of research in a methodology module. We now teach quite a lot of skills based on statistical packages. The price we pay for these skills is often an inadequate understanding of the underlying, more fundamental, research questions.

In the profession, benchmarking based on numbers, comparing figures and satisfaction surveys becomes more and more important. Importance is connected to both self evaluations, taking the temperature of the services and applications for funding. A sound understanding of what we do when we measure and analyse numbers would not hurt the profession.

Niels Ole PorsAssociate Professor, The Royal School of Library and Information Science, Denmark