Registering Facts

  1. Concepts and definitions
  2. Scales and other languages of description
  3. How to minimize errors
En Español  In Finnish   Contents

Concepts and definitions

Arteology According to the idea of scientific realism, the object of study exists in the empirical (tangible) world but theory belongs to the conceptual world of thinking. The goal of research is to create a theoretical picture of the object of study which resides in the empirical world. The reason behind the researcher's efforts can be pure inquisitiveness (descriptive study), but it is also possible that this picture shall be later used for improving the object of study or other similar objects (normative study). All the theoretical knowledge that we have concerning empirical things make up a more or less complete picture of the empirical world.

In the diagram on the right, you can see on green background a fragment of the empirical world which includes an object of a study, and on the right half of the diagram the corresponding theoretical model.

The notion of theory as an image of empiria is generally a fruitful one when studying products, their creators and users. A similar vista is also used by designers though there is the difference that the designers proceed in the opposite direction, i.e. from theory to empiria, starting their work in the world of concepts and then proceeding to realize their images in the empiria.

A researcher (and a designer as well) normally constructs the image of his object of study by using two types of elements:

  1. Concepts. Examples of concepts are the names of objects, groups or components of them, their properties like sizes, weights or materials.
  2. Relations between concepts, such as comparisons, correlations, structures, evolutions, reasons and consequences. The methods for presenting relationships are discussed elsewhere, see Modelling languages.

Concepts serve not only as building blocks of theory, but also as connecting links between an empirical object and its theoretical image. These links are necessary when registering data, and when a future user of the report applies the researcher's findings to his own purposes and problems. When you are studying products and services which relate to the lives and activities of people you have to deal with innumerable phenomena and factors which these people are handling on the basis of their sometimes vague tacit knowledge, and often it would be impossible for a researcher to explicate precisely every such factor, but in any case the most central concepts should have unequivocal correspondence with empirical objects or phenomena.

The most usual and elementary indication of a link between a theoretical concept and its empirical counterpart is given by the name of the concept. Moreover, there is the more precise method of presenting a definition for the concept, but normally it can be used only for a few key concepts of the project, because we cannot count on the user of the report remembering more than a handful unique definitions, nor can we expect him to peruse the list of definitions repeatedly.

The researcher has to select for his task appropriate concepts, their names and definitions, but this does not mean that he personally needs create all these things. On the contrary, it is usually advantageous to adopt as many concepts as possible, with names and definitions, from earlier treatises in the field. In fact, every established branch of science has an ample assortment of cardinal concepts which reiterate in numerous studies. Handbooks and textbooks in the field of science usually include lists of such concepts together with their definitions. If these established concepts can be used in a novel research project, unnecessary work can be avoided and the new project can profit from existing theory including tested methods of measurement and analysis and data about previous comparable cases. Besides, using generally known vocabulary brings the findings of the new project into close contact with already existing theory and it thus enlarges effectively the range of knowledge in the field.

Though it is generally advisable to avoid creating too many novel concepts when launching a new research project, it must be admitted that the researcher not always has free choice in the matter because of the lack of suitable established concepts. Such is the case in exploratory research where the project deals with questions that have not been studied earlier.

Moreover, it often happens that the researcher, after doing intensive studies on a topic, concludes that one or more concepts around the topic have traditionally been ill defined and need revision.

In the case that a novel concept has to be created, two aspects of the concept need consideration: its name, and its definition.

Naming a novel concept. Usual methods for creating new scientific terms are, among others:

Definitions. The name of a concept can normally give just an approximate idea for the content, just as the title does for a book. To give a more precise account, definitions are used so that both the researcher and the reader-user of the study report can have the same notion of the object. There are two types of definitions:

  1. An empirical definition announces how the concept is to be observed or measured in the empirical world. As the definition thus explains the operations of the observation or measurement it is sometimes called operational definition. At least a few empirically defined concepts are always needed to link a scientific model with empiria.
  2. A nominal definition describes the meaning of the concept by using other concepts which have already been adequately defined (empirically or nominally). Below are some logical structures that are often used in nominal definitions:

In the diagram below, yellow backgroud marks concepts (in italics) for which empirical definitions are self-evident or easily given, and nominally defined concepts are shown in red color in the lower part of the diagram.

Production

For example, when studying the productivity of a given workshop the 'output' can be measured as number of finished items per month, and 'work input' can be measured in man-hours per month. Now we can use these operationally defined variables for defining nominally a third concept:
'productivity' = 'output of products' / 'work input'.

Other nominally defined variables in the diagram are cost per unit, profitability etc.

In many fields of research the concepts and theory can be arranged in layers so that there is a layer of empirically defined concepts and on this basis one or more layers which consist of successively nominally defined, more and more abstract concepts. An example is shown below.

Usability The empirically measurable concepts are on the right: 'speed of operating' an instrument, and 'errors in operation'. They are then used on a higher level of abstraction for defining 'effectiveness' of operation. Effectiveness and 'learnability' are used on a third level to define 'usability', and so on. Rest of the figure, made by Shackel (1991, 24), is explained under Usability of Interactive Products.

It is by no means necessary to construct an all-encompassing model which contains every concept that the researcher has worked with, with definitions. A useful rule of thumb says that the researcher should try to operate with as few theoretical variables or concepts as possible. This recommendation goes by the name of "Occam's razor" dating back to the 14th century and attributed to Wilhelm Occam.

Besides, it would be impossible to give a definition for each word in the report. It is not necessary to define everyday words like 'human being', 'day' and 'buy' if these are used in their usual meanings given by basic dictionaries.

If devising new definitions is necessary, the following four requirements are often given as a target, though they are usually easier to meet with when studying physical objects than in the study of people or products of culture:

  1. Validity means that your definition matches the concept. It should refer to just that concept and not to something similar. If your definition is valid, you are registering exactly what you intended to register and not something else. Two levels of validity are often differentiated:
  2. Reliability means that if you, or other people, repeat your measurement or registration, the result will always be the same. This goal is, indeed, often practically feasible in physical sciences where reliability therefore often is taken as synonymous with repeatability. However, when you are studying unique human experiences of, for example, creating or enjoying art, replicating the experience will not be possible. Sometimes you might instead consider registering the phenomenon by several independent observers. Good match between their recordings then means good reliability. In any case, the criterion of reliability purports to promote the precision, trustworthiness, credibility and dependability of your results and thus their usability for other scientists and practitioners.
  3. Figurative or obscure language should not be used in definitions or recordings.
  4. The definition must not be a "vicious circle".

A vicious circle usually consists of two nominal definitions referring to each other while the concepts used in these definitions have no real definitions to link them to empiria. In the following, there is such a link between two nominal definitions:

Note that it is all right if you use just one of the definitions above. The vicious circle is created first when both are used simultaneously with no additional connection to empiria.

However, the mark of a real master is his/her ability to break any rule. Thus, MacIntyre (1985, p.219) defines in cold blood: "The good life for man is the life spent in seeking for the good life for man." This is formally a vicious circle but no less it transmits a message.

Validity and reliability can be illustrated by imagining a study whose purpose was to measure the length of objects. A faulty yard stick which had shrunk by 10 percent, was used. The real length of the stick was thus 90 cm although it had a normal looking scale from 0 to 100 cm.
The measurements made with this stick were reliable: the measured objects could be placed in an exact order of length and objects of the same length were properly recognized; if the measurements were repeated, the same results were always obtained. The measurements were not, however, valid in spite of their good repeatability: they did not indicate the real lengths but something else.

Other examples:

Phenomena that are studied in natural sciences and technology can usually be measured and expressed as quantitative variables and models. In such research, it is usually easy to fulfil all the four requirements we discussed above. However, the definitions of concepts that interest researchers studying people and cultures are not as easy. Often the most natural way to describe them in qualitative words, and it is often more difficult to meet all the four requirements. The researcher could consider which requirement he could compromise with in such research. What is often compromised on is validity. If a concept is very difficult to be measured, you could consider replacing it by another close concept which is easier to measure. "Intelligence tests" in psychology are one example of this: different tests yield different results when testing the same person, which fact indicates that they are measuring different things.

Can definitions be altered as the work proceeds? Yes, and in fact, in qualitative research this is usual because the researcher's understanding of what he is studying usually deepens when the research proceeds. When studying human culture it sometimes happens that an entire study contains nothing more than an attempt to define a few stubborn concepts. Especially when studying works of art you have to take into account that their content quite often varies depending on the context where they are studied, and often the work is intentionally made to elude fixed definitions.

On the other hand, if the definitions which have served as a basis for measurements or registration are changed after gathering empirical data, the gathered data which corresponds to the old definition is wasted.

In the final report definitions are often listed in an appendix or at the beginning of text. Another possibility is to give the definition in the text when starting to discuss a new concept.

Scales and other languages of description

Every word you can think of belongs to a language, and similarly every definition for a theoretical concept belongs to a specific language or system of coding. This language identifies the phenomenon or the object and its relevant attributes in two quite different "worlds" each of which have distinct requirements that the definition has to fulfil:

These two levels of presentation are quite different, and that can make the selection of language problematic. Sometimes you have found a tentative model in earlier treatises, and you can adopt its language; if not, you must try to anticipate the type of model that you are aiming at. Nothing prevents the researcher to develop a special language for handling his specific problem, but there are also many proven techniques of description that have been used in earlier studies. Note, however, that quantitative, qualitative and other descriptions of things do not mean that the properties of things would really and inherently be of one or the other sort. These words just indicate our usual method of recording the property. In the following are listed some of the most common alternative languages of description:

1. Individual naming of specific objects or events does not much promote the building of general theory, but in case studies and documentation it is sometimes used: "The Paris Exhibition 1900", "Parthenon", "Le Corbusier".

2. Verbal description is suited to almost any phenomenon of human culture. This type of study is often called qualitative because adjectives are usually prominent in the description.

Boat types 3. Presenting the visual form or pattern can be important when studying products. It is easy to make a photograph, but another thing is how you analyze the patterns in the picture. Earlier, it was a little difficult to analyse images and the normal method was first to introduce each pattern in one picture, give this pattern a name (e.g. "schooner" and "yawl" are the type names of the upper boats on the right) and then continue the analysis verbally.
It is also possible to analyse forms and patterns directly, with no translation to words. Examples of simple visual recordings of individual objects can be found in Measuring Physical Objects, and examples of general conceptual presentations of patterns under the title Icon Models.

4. Classification or "coding" compresses information effectively when only one or only a few attributes of the phenomenon are of interest. What is of interest, is a thing that you must conclude on the basis of your problem and the proposed model. If the interesting attribute is present in the object or phenomenon, you mark it with the predefined code, usually a number or a letter.

Classification is discontinuous in nature, which means that the variable has only certain discrete values, but no intermediate values between them. Classification is also called nominal scale if one wants to emphasize that it is a kind of measurement. After classification, the number of the individuals or measurements designed to each class will be known, and this is how qualitative data can be made quantitative.

Thus, if you are studying boats of different types, but their exact shapes are not of great interest to you, you could translate the boat types to codes, for example:

schooner = 1
yawl = 2
cutter = 3
gaff sloop = 4
Bermuda sloop = 5

Some classifications of everyday life and administration are suited to research purposes as such, for example sex and nationality.

On the other hand, there are numerous properties of humans (and of other objects of study), the values of which do not follow any clear-cut classification. If you, for example, have measured how tall and intelligent are the members of a group of people and what are their preferences of taste, you will usually find that all these values are distributed more or less like the upper diagram on the right, which researchers therefore often call as "normal distribution". Distributions like the lower diagram on the right are quite exceptional in empirical measurements. To be sure, it is always possible to classify even normally distributed measurements, but the question is whether it is reasonable.

5. Ordinal scale puts the individuals in a line, for example in the order of size, without paying any attention to how large the individual differences are. On this scale, we could for example indicate preferences of taste: "I'd rather live in a terraced house than in a block of flats." "This fabric is more beautiful than that one." Worded scales used in opinion polls must also be seen as some type of ordinal scale:

beautiful / - / - / - / - / - / ugly

In the scale above, a qualitative concept (beauty) which is usually evaluated verbally has been operationalized to be measured and analysed numerically. It is up to the researcher to decide if this procedure is fair toward the concept or if it discards some important points.

Most (but not all) ordinal scales are discontinuous in nature, which means that they contain only discrete values, but no intermediates.

6. Arithmetical scale. Its spacing is uniform, that is, the intervals between the markings are equal. These scales are often continuums, so the number of possible values is infinite. Hence these scales allow very precise measurements of the variables. Most physical quantities are measured this way. The following subcategories exist, though in practice there is little difference between them:

Operationality of the scales in analysis. When selecting the language of presentation, the researcher should consider not only the facility and accuracy of registering facts in the empiria, but also the practical procedures of analysis and the requirements of data processing, if that is used. In that, arithmetic scales are unbeatable: the relations between numbers are exact and known to everyone, it is easy to feed them into the computer and there are excellent mathematical methods available for analysing them. Some of the most powerful methods of analysis include divisions and can consequently only be applied to data measured with a ratio scale. From the point of view of analysis, the ratio scale would thus be preferable. Indeed, it often happens that a researcher "operationalizes" one of the attributes of his object of study: he deliberately constructs a never-heard-of ratio scale for measuring it because he wants to use statistical analysis.

Remember, on the other hand, that the success of analysis depends also on validity: how faithfully the language can present the thing that is of interest in the study? In this respect verbal or other non mathematical languages can often be superior after all. There is a discussion on the format of data in the various stages of analysis under Tools for Analysis.

How to minimize errors

An important goal of empirical research is to create a theoretical picture of the object of study which resides in the empirical world. However, we have to admit that this picture never can be absolutely faultless or reliable. The reasons are many: every registering of data can contain an inadvertent error; it is not possible to know all relevant cases that should be measured, or approach them; the number of cases can be so great that we can afford to study only a sample of them, and this sample can be biased; analysis of the data can contain errors or misinterpretation. Many of these sources of error can be eliminated by careful preparation of the project.

When preparing definitions of concepts it is thus worth the while to deliberate how their empirical recording shall be made, and which kind of difficulties it might entail. Sometimes a slight adjustment of the concept definition can help to avoid errors and disturbing variation of data, not to speak of unnecessary field work.

A researcher is usually searching invariable structures in the object of study, which structures usually become visible as some kind of systematic variation in the measurements and recordings. Beside this interesting variation, the data usually contain other, disturbing variation which the researcher usually wants to minimize. Its most usual sources are:

Each of these three types of error will be discussed in the following.

Errors in measurement

A meter includes the apparatus and the arrangement with which the researcher registers whatever interests him in the object. Also non-mechanical methods for gathering data can function as meters, such as psychological tests and the scales used in questionnaires, e.g.:

pleasant unpleasant

The worst defect that a meter can have is that it measures the wrong thing. In other words, a good meter must be valid. Moreover, it should be reliable, giving always the same results even when used by different people. Finally, a sufficient resolution will be needed. Resolution (or least count) is equal to the smallest measurable difference between the observations.

In practice, no meter is totally reliable; measuring always includes a slight error of measurement, i.e., a difference between the measurement and the real value. No absolutely correct value can be ascertained in practice, nor is there any way of knowing exactly the error of measurement.
The error can, however, be minimized, and to this end, you have to look at the two components of the error.

Let us say that we have an object that is difficult to measure, and we have made not less than 172 measurements of it. These are represented by small squares in the figure below. The smallest measured value is =5, the largest =34, and the average of all the results is 17.

The random error equals the difference between one measurement and the mean of all the measurements. For example, the random error of the measurement at the value=32 is 32-17=15. The random error can be measured in the same way as any other dispersion of values, for example by calculating the standard deviation. A small dispersion also means good reliability.
The random error is easily eliminated by calculating the arithmetic mean of the results.

A systematic error equals the difference of the mean of all measurements from the real value of the variable (which is normally unknown in the study). In our figure, it is 24-17=7. The systematic error will normally remain the same when the measurement is repeated. Hence, it is difficult to detect it in a study. A systematic error also indicates that the meter is not quite valid.

Sometimes it is possible to detect a systematic error if the same object is measured with two different methods. If a systematic error is discovered, it is eliminated by correcting the measurements (e.g. by normalizing them) or by calibrating the scale of the meter.

Manufacturers of meters often guarantee that the total error (random + systematic) of their product will be lower than a certain limit, provided that the instrument is used properly.

Disturbances

NoiseWhen studying a phenomenon, the researcher usually wishes to find out how it is caused or influenced by certain "explaining factors". However, normally empirical objects are influenced by many factors beside those that the researcher is interested in.

On the right is an example of a typical research model (an experimental design, in this case) for studying the relationship of a certain stimulus and the expected reaction to it. In this example the research project gets spoiled because the object of study receives, instead of the intended stimulus, unexpected "noise" or disturbance which causes the wrong type of reaction.

Disturbances can occur in any type of study, such as observation, interview, filling in a questionnaire, etc.

Obviously data which are corrupted by noise are of small value in research. To avoid them, there are two possible approaches: either minimizing the disturbances themselves, or eliminating their effect from recordings.

Preventing or diminishing the disturbances that are present when recording data can be accomplished, for example, by choosing a quiet and secluded environment for observations or interviews.

Note that even the activities of research itself, like posing questions, measuring etc. can cause some disturbance. For eliminate it, special "unobtrusive" research methods have been devised.

Shielding Sometimes it is possible to shield the research object, thus keeping out the disturbing influences. This is common practice in experiments set up in laboratories.

It is to be noted, however, that shielding, arranging the recording to an environment other than where the activity normally takes place, or any other manipulation of the genuine circumstances of the phenomenon can alter the object or procedure in unexpected ways. In other words, it can lessen the validity of the recorded data.

Eliminating the effects of disturbances after the data have been recorded is usually an uncertain method, because these effects are seldom exactly known. The procedures are in principle similar to the corrections that are used for erroneous measurements:

Subjectivity of Recording

Source: Habitat Forum News no 3. Jan 1976If the beliefs and attitudes of the researcher affect the observations, these are called subjective; if not, objective. Subjectivity can play a role even later in the project, when analysing and interpreting data, but that problem is not discussed on this page which regards the questions of registering facts.

Because subjectivity is caused by personal beliefs and attitudes which are relatively constant, it causes a systematic error in recording. It can affect both descriptive and normative data.

In descriptive research which aims at recording the state of the object as it is (or as it was, in historical study), subjectivity often causes errors when studying people's or animals' actions and their motives. The researcher often relies too much on his facility of empathy, looks at the activity from his own point of view only, and fails to perceive the real purpose of the endeavor.

One more example of descriptive studies which often suffer from subjectivity is when people who produce works of art and other phenomena of human culture are studying their own or their colleagues' works. To study one's own activity was earlier wholly disapproved by many scientists, but today many think instead that the maker of a product is the best expert on it and can perhaps do useful work as a researcher as well.

As a contrast, the ideal of objectivity is often easy to achieve in natural sciences, when studying objects to which the researcher has no personal attachment nor sympathy.

In normative research projects which are aimed at improving the object of study (or other similar objects) the preferences which shall guide the project should normally be collected from those people who are going to encounter the result of the project or use it. These groups of interest are sometimes classified as 'agents', 'beneficiaries' and 'victims' of the project, cfr. Point of view. Subjectivity when surveying these people's preferences concerning the desirable improvements to the object means that the researcher mixes in his own opinions and interprets other people's suggestions on the basis of his personal preferences. Another usual example of subjectivity is that the researcher simply forgets some interest groups, for example those people that are going to suffer from the secondary consequences of development, or he forgets the environmental aspects.

In normative projects of research and development the target of impartiality is sometimes difficult to achieve completely. The researchers that have been hired to the project are often more or less obliged to share its values and promote its goals, i.e. to improve the object. What "improving" means, is usually preliminarily defined in the initial plans for the project. Besides, the research team often includes one or more planners or designers who are expected to continue the project by preparing proposals for accomplishing the necessary improvements in reality, and these people often have strong opinions on the future development already at the outset.

Despite the difficulties, in most cases there is no reason to abandon the traditional general rule to scientists which says simply that they should aim at objectivity, that is, they should not let their opinions affect the recording of facts (and of opinions of interest groups, when applicable). Using the metaphor of the "two worlds" of empiria and theory, researchers should thus try to stay, as personally not interested spectators, outside of both of them.

The methods available for a researcher for deliberately counteracting his subjectivity are slightly different depending on whether the researcher is working alone, or as a member of a team.

A researcher working alone can try to unearth his personal attitudes by putting to himself questions such as the following:

Besides, the researcher can try to detect outside forces which could bias his observations, like:

Attitudes and outside influences which may be revealed with the help of above questions normally belong to human life and are often quite legitime and understandable. Nevertheless, the researcher should try to make them explicit in order to be able to analyze them and decide whether they are likely to influence his work too much. It would be no bad idea to put a few words about your analysis in the report as well.

For a team of researchers, a collective discussion is an efficient method to clarify everyone's personal evaluations and perhaps arbitrate between them.
Also such trivial reasons as slightly different methods of collecting data can lead to interpreting the observations differently, which can cause some systematic variation in the data. This annoying variation can be minimized by giving exact instructions to the assistants, by rehearsing the procedures, and by assigning the cases randomly to the assistants.

Note, finally, that there are approaches of study which do not condemn empathy and co-operation between researchers and the people that are being studied but instead make use of mutual compassion because it helps the researcher to understand the situation and actions of people. Methods which take advantage from it include action research and participating observation.

En Español  In Finnish   Contents

August 3, 2007.
Comments to the author:

Original location: http://www2.uiah.fi/projects/metodi