Interrogating Methods

  1. Selecting the Method
  2. Thematic Interview
  3. The Questionnaire
  4. The Problem of No-reply
En Español  In Finnish   Contents

The production and use of artifacts involve several groups of people - producers, present and potential customers, buyers and users of the products - whose experiences, knowledge or wishes are valuable data which you can collect by interrogating.

Before starting to put questions to people, it is advisable to define exactly the extent of your interrogations. There are two decisions that you have to make:

  1. Demarcate the study by asking yourself which is the population or the entire multitude of those people whose knowledge and opinions you want to capture? It is often more or less identical to the set of people where your findings shall be applied if such application is foreseeable.
  2. Select the sample to whom the questions are finally addressed, if you cannot or do not want to ask the entire population defined above. Random sample is best in principle, but in practice it is often difficult to reach and persuade randomly selected people to participate, so it is usual that various kinds of convenience sampling are used instead, with the result that it becomes more difficult to generalize the findings.

Once you have settled the two questions above, you can proceed to define which kind of response you wish to get from these people. It can be either

The above division is not absolute. You can gather subjective opinions in normative purpose and try to change things according to them, but it is also possible to study them in descriptive and objective style, by using concepts like:

It is also possible to mix descriptive and normative approaches into one questionnaire or interview, as long as the researcher keeps the difference clear to himself or herself. The methods of studying either one are very similar.

Selecting the Method

In the interview, responses are given orally, and in the questionnaire in writing. This division seems superficial and indeed it is. A more important difference between the methods concerns the amount of knowledge and theoretical basis that you have when starting your inquiry, as compared to the amount of additional data you need to collect. Sometimes the missing pieces of the puzzle are relatively small when compared to the mass of knowledge (i.e. theory) that you have when starting the investigation. Sometimes there are large gaps: you know almost nothing of the object in advance. Each of these cases calls for a different method of research where you focus your inquiry to just the missing pieces of information.

We often discern three possible typical situations regarding the relative weights of existing theory and the amount of data to be gathered (which are more fully explained under the title Models in the Research Process):

As the table below indicates, each of the three conditions above calls for a specific style of survey, though it is, of course, also permissible to combine elements from these three methods. If you are not sure which one of the three is best for your situation, you may want to look at a few other characteristics that distinguish the available variants of survey:

Style of research: Exploratory research Research as Revision of a Model Hypothesis based Study
Style of questions:Mostly openMostly fixed-choiceMostly fixed-choice
Possible to add new questions?YesNo No
What to do with unforeseen findings?Register and study themRegister
or exclude
Exclude
Work hours and cost per answerHighLow High
Is quantitative analysis possible?SeldomOftenAlways
Other viewpoints:Impossible to answer anonymouslyRequires some writing abilities from the respondent Difficult to arrange for anonymous answering
Method: Thematic Interview Questionnaire Experiment with oral reaction

When planning an inquiry, one should keep in mind that opinion polls and marketing research are subject to certain ethical considerations, especially if the individuals can later be recognized on the basis of the given information. The respondents should be informed about the purpose of the poll, the name of the researcher and also about the fact that participation is voluntary. The respondent has also the right to know who has commissioned the poll except when it is obvious that this information would decrease the validity of the results. The name of the commissioner does affect the results: most people are reluctant to express such opinions which they think are in contrast to the intentions of the commissioner. In order to diminish this effect, it would be advantageous to disclose the commissioner only afterwards.

Thematic Interview

The interview is a good method, if:

Interviews take a great deal of time: about one day for each interview and the accompanying paperwork, while you can handle dozens of questionnaires a day (even more if you can use a scanner for the pick-up of the responses). This does not mean that a questionnaire would be ten times more "productive" than an interview; on the contrary the insights that you get in the interview may be ten times more valuable to you.

In the following, we discuss the method of the unstructured, or more aptly, thematic interview. It resembles normal discussion: the respondent has the right to add any comments that she/he finds relevant, and if the interviewer finds these new topics interesting he/she takes the hint and may present additional questions based on the new viewpoint. But if the interviewer thinks the digression is unnecessary, he/she directs the conversation back to the originally selected themes.

The thematic interview is a suitable method especially when you have no exact theory about the issue while on the contrary you are eager to learn about new viewpoints you had not anticipated. If you select this method, the respondents will often produce more new viewpoints than you can use.

Individual or Group Interviews?

Earlier, most researchers thought that a third person always biases the responses and therefore superfluous persons, such as other members of the family or colleagues, should never be allowed in interviews. Others have pointed out that personal values and attitudes are created while the individual lives in his or her social group, they do not effectually exist when the person is isolated from the group and in such an isolation they cannot be studied, either.

Group interview means discussion among an existing social group like a workplace team or unit, or a family. It can be effective when you need to gather information on collective ways of living, working and recreation, and also on prevailing usages, evaluations and meanings related to products. It tends to neglect personal differences, topics of dispute and sensitive questions. Moreover, in a large group usually some members talk most of the time and others are more subdued. If the goal of the study is to describe the factual dynamics of the group, you may choose to accept and record all these asymmetries in the conversation. If the goal, instead, is to gather views on a given topic, you should steer the discussion, prevent it from straying away from the theme, and take care that all the participants are being heard.

Practical Arrangements

The setting. In interviews, like in other research, all arrangements which aim at facilitating research can influence the respondent. To minimize the influence, it is advisable to select a place that is neutral and familiar to the respondent: his/her home, a meeting room, a cafeteria or another peaceful environment where you can converse undisturbed and in no hurry. When gathering opinions about a product or its use, the best setting often is the place where this product is customarily being used.

The interviewer is always a foreign and possibly disturbing element to the normal environment of the respondent. To minimize the effect, he or she should dress and behave inconspicuously.

The responses can be recorded by an assistant or with a tape or video recorder. Note that the playback always takes a lot of time, often more than three times as much as the time spent on recording.

Material for demonstration. When the questions pertain to a given product or its use, answering can be easier and fuller if the product itself is present and it can be used. If this is not possible, the researcher can provide suitable substitutes or mock-ups of the product (see Method of presentation). For example, in the "puzzle interview" method the researcher makes available a selection of prefabricated picture cards which symbolize various elements of the product, its use, different users, etc. Besides, the researcher places on the table or on the wall a few named frames or trays which represent different modes of using the product in various contexts or circumstances. By placing one or more cards into suitable frames the respondent can illustrate situations and problems related to the use of the product.

The agenda. The interviewer usually begins by stating the responsible organization, the purpose of the study, and how its results will be used. These small pieces of information may already somewhat influence the respondent's attitudes, but it is difficult to see how that could be avoided. Moreover, it is often necessary to explain to which degree the respondent's statements can be kept confidential.

The first questions on the issue of study are to be worded in general terms. The questions are "open ended", and you often encourage the respondent to explain and amplify his/her responses. To avoid biasing the responses, the interviewer must never reveal his/her own opinions on the discussed topics. In order to encourage the respondent, you can indicate (by nodding, for example) that you approve his or her opinions, but then you should be careful and avoid showing approval to just some of the opinions.

When the respondent elaborates his/her statement, he/she does not know which new viewpoints will interest the researcher. That is why the interviewer must somehow direct the responses. Stopping the unnecessary digressions might be tactless and it can often be avoided by simply waiting until the respondent has finished. Positive encouragement which aims at directing the respondent (back) to the interesting issues is more useful. Such positive prompts are called probes. Examples:

Another type of probe is called for when the respondent says something that you suspect to be an overstatement which he/she might perhaps reconsider. In such a situation you simply ask: Are you saying that...?, Do you really mean that ...? and rephrase the statement.

At the end of the interview you could ask if the respondent is willing to see and correct your transcription of the statements later on, or if he/she is willing to continue with another interview if you later find some of the topics worth amplifying.

Conversation sampling is a special method which aims at eliminating fully the influence of the researcher's presence. It means eavesdropping the talk of private persons in public places like shops and parks. Some researchers have used this method to gather general public opinions concerning new public buildings; it could perhaps also be used to collect viewpoints on articles on sale in a shop. A weakness of the method is that it takes often very much time before people turn to topics that interest the researcher. Besides, it is usually impossible to restrict the study to any definite population.

Before using the method, you may want to consult the chapter on ethical considerations. An unsolved question is, have you to ask for the consent of people before publishing their statements, even in the case that these people remain incognito?

Reporting

Reporting thematic interviews is often complicated, even when you have the responses recorded on tape. Sometimes you have to listen to them several times, trying to understand the meaning. Once you have understood it, you just paraphrase the statements in writing. If the question is very important, you may wish to verify your interpretation in a renewed interview.

The opinions of interviewees are to be recorded as faithfully as possible, but factual statements are another thing. For the facts presented by the respondent you should keep in mind the possibilities of:

In the critical appraisal of factual truthfulness you can perhaps make use of an extensive list of typical questions used in source criticism of textual sources.

Methods of analyzing the results of interviews are discussed under the title Analysis.

The Questionnaire

The questionnaire is a popular method of collecting responses to simple questions. The respondents can live far away, as the questionnaire can be mailed to them and they can mail it back to the researcher. To be able to ask exact questions, you must have an exact idea of what you want to know. You often have quantitative hypotheses with arithmetic variables.

Questionnaire is a good method if:

Note that when using a questionnaire, you will only get the answers to your questions; you will have no means of asking additional questions (which you will have in an interview). This difference may be crucial in such research projects where you know the issue only superficially in the beginning and those additional questions might be essential. On the other hand, there are certainly projects where no additional questions are needed.

Modern informatics has brought to us new channels for questionnaires. In one of them the answers are given on telephone by pressing the number keys according to given code (yes=1, no=2 etc). Another method is to place the questionnaire, with the help of HTML language, on a www page as a form. The answers will be automatically transferred to the researcher's server where he then can analyze them statistically. For these tasks there are also special programs like 'Infopoll' which was used by Jääskeläinen (2001 p. 112) when gathering expert views on interactive TV.

A disadvantage of many modern techniques is that you have to restrict your sample to people who have the right type of connection (like telephone or email) and who can and want to use it. This restriction can cause bias in the results. A remedy could be to arrange simultaneously another questionnaire with traditional technology.

A method which is basically similar to the questionnaire is the structured interview, in which the interviewer just asks the prespecified questions and records the responses. Usually almost all of the questions are of the fixed-choice type. The method seems to combine the disadvantages of the interview and the questionnaire while giving no special advantages other than the results being easy to analyze statistically.

Whatever the method of asking, you must honour the respondents' right to know why they are questioned. That is why at the beginning of the form, you usually put an introduction where you state your organization and how the report of the enquiry is going to be used. Moreover, you should give a deadline for the return, and it is also advisable to include a self addressed, postage paid envelope.

In many cases, it would be proper to state to which degree the respondent's statements will be kept confidential. It is often best not to include the respondent's signature or other identification in the form itself. However, you would probably want to put an identification symbol on the envelope. In the next phase, this symbol will enable you to detect the non-respondents and to mail the reminders.

Asking about Facts

A fact is something that is not affected by an attitude or by an opinion. You can include factual questions in interviews or questionnaires. Make sure not to join two topics in one question.

You can often make fixed-choice questions. Examples:

Year of birth:________

Marital status: single married divorced widowed

When presenting fixed-choice questions, you must make sure that all the possible alternatives are included. Just to be on the safe side, you can add an extra box:

other

or:
Other, what _____________

The questions must be absolutely unambiguous because the respondent has no means to ask for clarifications. Therefore, the sentences should be simple and contain neither double negations nor rare words. Sometimes a question can be clarified by emphasizing important words or by adding drawings or photos.

The researcher has made a serious mistake if the respondent is obliged to reflect on what the question really means. Ambiguous questions produce no useful responses; on the contrary, they irritate respondents and increase the risk of not returning the response. That is why each questionnaire should be tested in advance. In the trial, the researcher observes a tester filling out the form. If the writing slows down, the researcher asks why.

The responses to factual statements are not always exactly truthful. People often wish to give a "good" picture of their incomes, education and other attributes connected to their social status. In some questions you have the possibility of comparing the data from your sample to official records from the area. Sometimes, you could ask the same thing once more at some other point of the questionnaire and in different wording.

Asking about Opinions and Attitudes

When testing attitudes, the statements in the questionnaire should be formulated in the following way:

When forming the statements and looking at the results, you should bear in mind the biasing effect of the questions. Most people would rather answer a question positively than negatively, especially if they think that the researcher himself supports the statement. Thus it would be better to alternate the statements in such a way that they reflect both positive and negative attitudes.

A typical tool is the Osgood scale (or "the Likert scale") also called "the semantic differential". Examples:
How do you evaluate the design of this telephone?
Tick one box on each line.
Light _ _ _ _ _ _ _ Heavy
Solemn _ _ _ _ _ _ _ Cheerful
Handy _ _ _ _ _ _ _ Awkward
Classical _ _ _ _ _ _ _ Modern

An alternative:

Do you agree with the following claim: This telephone is beautiful. Tick one of the boxes below.

Later, you can transform the answers into mathematical frequencies, and analyse them statistically. 

To measure attitudes, a Guttman scale is sometimes devised. It is a battery of statements with increasing rigidity of attitudes. The following is an example of a Guttman scale measuring the discriminative attitude:

  1. Would you marry a refugee?
  2. Would you accept a refugee as a close friend?
  3. Do you find refugees living in you neighbourhood acceptable?
  4. Should refugees be allowed to live in the same neighbourhood as others?

Devising a good Guttman scale is cumbersome because it should be tried out and polished before use. Ready made, tried batteries of questions are available for many central concepts of sociology and psychology, see for example the list made by the American Psychological Association. These usually go by the rather inappropriate name of (psychological) test. The word meter would be a better name.

The Role Playing Method is sometimes used to study attitudes and behaviour norms. In role playing, the researcher shows a written or pictorial story about an invented social situation, and the respondent is asked to continue it in whatever way he finds appropriate. Alternatively, the respondent is asked to imagine the events preceding the ones in the picture. The researcher usually presents the story under two slightly different variations, thus forming essentially an experimental design where the different stories function as stimuli and the respective reactions indicate the dependent variable.
It is convenient to place the stimulus story at the top of the page which is handed out to the respondents. You should not try to send this kind of questionnaire by post because the respondents would hardly be motivated enough to complete it.

Intentions and wishes often have to be surveyed in market research. Unfortunately the wishes people indicate in polls reflect their future behaviour rather badly. Many young people intend to do or get something they will never do or acquire; on the other hand, old people may not even require such things that an outsider would deem absolutely necessary for their well-being.

The Problem of No-reply

Most people gladly agree to be interviewed although not as eagerly as earlier on because surveys have become so very common nowadays. The ideal situation for the researcher is one in which the subject matter is genuinely important or interesting to the respondent. If this is not the case, the researcher should perhaps somehow motivate the respondents.

Positive motivation is the most effective kind. It can be accomplished by explaining the purpose and importance of the survey before presenting the actual questions. The researcher could perhaps also emphasize the fact that these interviews are valuable to him because he cannot obtain this information anywhere else.
The researcher should think what the respondents will find motivating; the fact that carrying out a survey is required for university studies may motivate the researcher, but not necessarily the respondent.

Another way of motivating people would be to point out that very little trouble will be caused to them if they fill in the questionnaire. This is why a questionnaire should not look lengthy or intimidating, and the questions should be short. Answering them should be made as easy as possible. A self-addressed, postage paid envelope should be enclosed with the questionnaire.

Several other tactics, too, have been devised to improve the answering rate of mail surveys. Jobber (1995 p. 177) summarizes their tested effect as follows:
Tactic Effect on response rate
Prior notification by mail Increase in consumer research but not for commercial populations
Prior notification by telephone Increased response rate
Monetary and non-monetary incentives Increased response rate
Stamped return envelopes Increased response rate
Personalization The effect varies
Granting anonymity to respondents Higher response rates when the issue is sensitive
Questionnaire length Only slight effect on response rate
Coloured questionnaire No effect on response rate
Deadline No effect on response rate
Type of questions Closed-ended questions get higher response than open-ended
Follow-ups Follow-up telephone calls and mailings increase response rates

If, nevertheless, a great number of responses never turn in, the researcher should not automatically replace them with answers from other people who are more willing to cooperate. If this was done, that would easily distort the results, because the absentees may differ from those who handed in an answer, being perhaps also deviant with respect to the important variables. Patrick W. Jordan (p.158) gives an example of this:

"Consider ... a scenario whereby the manufacturer of software for use with home computers decided to survey its customers to check their levels of satisfaction with their products. ... Those who would take the trouble to respond would most likely be those who had a particularly strong opinion about the software. ... The conclusion [might wrongly be] that the customers were firmly divided into two camps - those who loved the software and those who hated it."

To rectify the distortion caused by absentees, the following procedure is sometimes used:

  1. At first, the researcher keeps separate the answers obtained without any claiming (group A, say 50 % in this example), and those obtained after claiming (group B, say 30 % in this example). The final absentee rate (group C) in this example will thus be 20 %.
  2. After this, the researcher has to see if group B differs from group A with respect to the variables to be studied. A statistical t-test is appropriate for this.
  3. If there is any deviation, we can assume that the group of definite absentees, group C, will deviate from A in the same direction as group B. From now on, the answers given by group B will be used as substitutes for the missing respondents in group C. Mathematically, this will be done by increasing the weight of group B so that it equals the sum of groups B and C, i.e., 50 per cent in our example.
  4. If groups A and B do not considerably differ from each other, we can expect that the absence of group C will have no great influence on the results, and thus no corrections will be needed. Groups A and B are combined, and the total results from them are assumed to be true for the whole of the population.

Once the empirical measurements or estimations have been recorded the study can proceed to the phase of analysis.

WWW Sites

En Español  In Finnish   Contents

August 3, 2007.
Comments to the author:

Original location: http://www2.uiah.fi/projects/metodi