Evaluating Normative Proposals

  1. Collective Evaluation
  2. Theoretical Evaluation
  3. Testing

Assessing the findings of descriptive study are discussed on another page.

En Español  In Finnish   Contents

When the purpose of the study is normative, i.e. defining how the state of the study object or other similar objects could be improved in the future, the end product of the project will be one or more proposals which explain in detail the desired improvements and how they are best achieved. These proposals are published in the final report of the project, after which the work sometimes continues as a development project which takes care of the practical execution of the proposals in the form of, for example, Developing an Industrial Product or Developing an Activity.

Iterative study Because the proposals are the most important content of a normative research project, especially when the project leads to practical operations of development, the researcher should try to make them as effective and optimized as possible. In most normative projects the proposals receive their content gradually (see Methods of Normative Analysis) and each of their successive drafts are evaluated in their turn, which often reveals new need to revision and adjustment of the proposal, as is shown in the diagram on the right.

The final and best possibility for assessing and fine-tuning the proposals is just before the report will be printed in its final version. In this draft the proposals, their starting points, practical possibilities to their achievement and other related factors are summed up, the benefits and expenditures are precise and their mutual relationships are definite, all of which facilitates giving a total judgment of them.

The key question when assessing a normative proposal is: Is it an effective, practical and economically optimized instrument for attaining the desired improvement?

In the following are presented few usual methods that are often used in evaluating proposals, either with or without the collaboration of the users of the desired improvement.

Beside making the proposals, a normative study project often has included gathering descriptive data from empiria, to be used as a basis of the proposals. When evaluating this information, the main criterion is their reliability. This mode of evaluation is discussed on another page: Assessing the Findings.

Collective Evaluation

Every normative study project is made from start to end from somebody's point of view, and this is the case even in the assessment of its final proposals. Often the same group of people that had defined the targets of the projects in its beginning phase can also take care of the final evaluation of the proposals.

It often happens that during the project the point of view gets wider and is at the end not exactly the same as in the beginning. New people and interest groups have perhaps joined the project, for example the manufacturing department or other such people who shall take care of the realization of the proposals. Sometimes questions of side effects - concerning raw material, wastes and pollution, for example - have emerged that nobody had thought of at the initial meetings of the project. All this means that disagreements of opinion have become more likely in the final stage than they were at the outset of the project. However, the methods explained in Compressing the opinions are always available for arbitrating the differences of opinion. Most of these function on the basis of discussions between the interest parties. In any case, it is the task of researchers to visualize and explain the alternative proposals comprehensibly to all participants so that a meaningful discussion of the various options and their advantages and weaknesses becomes possible. An example of a simplified "design language" for houses.

Another responsibility of the researcher is to consider whether all relevant interest groups will be heard in the evaluation. Sometimes one or more of the alternatives can affect the lives of future generations which cannot be present at the meeting. Besides, there are people that seldom get their voice heard at meetings, such as visually impaired or hearing-impaired people, people with wheelchairs or otherwise reduced ability of motion, and children (see a list of important aspects when designing for these). At a meeting the researcher can remind people of the needs of these people, or afterwards he can try to modify the selected alternative so that it would be acceptable for even those that were not present at the evaluation.

When it is difficult to gather people from the interest groups into a meeting, the interrogating methods of interview or questionnaire are available for gathering the assessments. Their disadvantage is the difficulty of arbitrating the opinions that are in contrast.

Theoretical Evaluation

There are also methods that can be used when it is impossible to gather any opinions at all from the interest groups. In such an event, the researcher can try to assume an objective role of arbitrator and simulate a general and impartial viewpoint. When no direct evaluation from the interested people is available, the researcher normally makes use of the reports published in the fields of research that have relevance with the normative target.

Acceptability or attractiveness is a characteristic of proposals or products not easy to study, because it is fairly general and vague, too. The same is true for related concepts like "desire", "need" and "pleasure". The content of any of these depends quite much on the situation, which makes it difficult to try to create a general theory of them. Such an attempt was, nevertheless, made already in antiquity by Epicurus (342/1 - 270/1), who thus became the father of the hedonistic school of philosophy (hedone = 'pleasure' in Greek).

In early 19 C, so called utilitarian philosophers like Jeremy Bentham and John Stuart Mill studied various components of pleasure, mapped out their relationships and tried to create models for them, or for utility in general. Mathematical models for the same purpose, or the theory of utility were a little later presented by Daniel Bernoulli, Vilfredo Pareto and others.

The first researchers tried to analyze theoretically the concept of attractiveness. Empirical studies on these questions started in late 19 C in the laboratories of psychology, notably by Georg Th. Fechner who experimented with various geometric patterns and tried to find their relationship with aesthetic pleasure. He laid the foundations for the modern paradigm of perception psychology.

Perception psychology deals with human senses, each of which can generate a feeling of pleasure and attraction. Below is a list of human senses and examples of products which are particularly concerned with the mentioned sense.

Channels, possible for a solitary person, of getting pleasure from a product:
Channel of perceiving attraction Examples of products where this type of attractiveness is important
Vision Pictorial works of art. Refined products. More examples: see Beauty Of products.
Hearing Drama, music, and the instruments involved thereby. More examples, see Semiotics.
Senses of taste and smell Culinary art and instruments.
Senses of touch and movement Sports equipment. Clothing. Vehicles. Furniture, especially for relaxing.

Beside the mother sciences of physiology and perception psychology the table above mentions some now established paradigms i.e. traditions of study, each of which specializes in a certain mode of sensory pleasure. First to appear among them is the science of beauty, aesthetics.

Sensory pleasures are mostly if not wholly innate mechanisms of man. They work on a physiological basis which does not much vary between people, their tribes or their spheres of culture. When selecting a method for their study, researchers therefore seldom bother doing laborious surveys in the field but instead count on experiments in a laboratory.

However, there are also other pleasures of man which do not emanate simply from our senses but from a more complex system: from living in society. Below are examples of these.

In society possible channels of getting pleasure from a product:
Channel of perceiving attraction Examples of products where this type of attractiveness is important
Desire to learn, know and comprehend. TV. Telephone. PC. Instruments for education. See also Beauty of Discovery.
Sexuality. Clothing. Perfumery.
Co-operation. Telephone. TV. Tools. Playground equipment. See also: Developing an Activity.
Competition. Desire to get appreciation. Clothing. Dwellings. Vehicles. Sports equipment.

Mother sciences for this group are sociology and psychology which have given birth, during the past century, to an extensive series of more specialized paradigms of research. Some of these, having some relevance with products, are mentioned in the table.

Because the modes of pleasure and attraction listed in the latter table are born in the complex system of society, they are often understood quite differently in dissimilar societies and spheres of culture. If a researcher wants to discover any general and invariant structures in them, valid in several or all spheres of culture, he has to gather material from more than one country or community of people, i.e. field studies are necessary.

First field studies of socially originated needs and pleasures were made in early 20 C in the field of anthropology. Bronislaw Malinowski studied the succession of needs which he found emerging gradually in evolving, primitive societies. Abraham Maslow thought that the same principle of successive needs governs even our behaviour in contemporary societies.

Later researchers have proposed a great number of different models for describing the relations or hierarchies between the various pleasures of using products.
Shackel's modelTurkka Keinonen has paraphrased some of these recent proposals, which review can be found elsewhere under the title Usability and emotions.

An example of the models where pleasures and needs related with products are arranged as a hierarchy can be seen in Brian Shackel's (1991, above) diagram depicting the principal dimensions of product acceptance. Its explanation can be found under the title Usability as a measurement.

Nielsen's modelAnother similar example is Jacob Nielsen's hierarchy of product acceptability, on the right. Its explanation. Also Patrick Jordan has lately (2000) developed a theory which purports to include all the "pleasurable" aspects which can be fulfilled in industrial products.

And finally there is the structure of typical goals of product design, used as the arrangement of the pages belonging to the Arteology site. Theories about these include theory of production, theory of design of the type of product that shall be created, and a few goal-specific theories, i.e. reports on the general targets that the operations in question normally have. These can concern, for example, the economical aspects in manufacture and marketing, and also some ecological aspects like the amount of raw material and waste. The product user's view is typically concentrated in the theories of usability and the long-term ecology of use of the product. It can, however, be difficult to simulate fully the user's viewpoint in evaluation, because it usually includes simultaneously many different product properties which can be in conflict.

Procedures of Theoretical Evaluation

When designing a new product all the characteristics of the future product are elective and you are free to choose among them the most attracting combination that you can find. In other words, you can and you should consider simultaneously as many characteristics of the future product as possible. The task becomes complicated because higher attractiveness usually means also higher costs, and you have to choose the right level for them.

Quite often many characteristics of attractiveness are interdependent and you cannot manage them without first finding out the nature of the relationship. The descriptive models, some of which were discussed above, can alleviate this problem if they are available. Moreover, there are strategies which can help you to manage a great number of requirements, like:

Typical situations in a product development project where you have to think about several different goals, are Strategic Design, the task of creating a Product Concept, then the physical Design or shaping the product, and finally Evaluating a Design Proposal.

The paradigm of evaluation is also discussed by King et al. (1987) and in the journal Evaluation Review. More sources about it can be found in literature or internet when using the keyword 'evaluation'.

Time perspective. Products are things which exist for a time, but the pleasures or attractions concerning them seldom stay constant during this time. The life span of a product often includes several distinct phases, which perhaps should be studied separately:

Value Engineering

Value engineering is a method of summing up the significant utility values of the available alternatives and finding the best alternative. In the case that also cost or price of each alternative is included in the comparison the method can be called cost benefit analysis. It is definitely a quantitative tool, and it necessitates the measurement of all the components to be analysed. The analysis is carried out in distinct and logical steps:

  1. Before you can start the analysis you need a clear idea about how the utility value or acceptability of each important property of the alternative can be assessed. This can be given, for example, as a prefabricated table of satisfaction for each property.
  2. Moreover, you will need to define the weights of all the important properties of the object of study.
  3. Cross tabulate all the available alternatives and all the attributes to be appraised. Assign to each alternative two columns: one for the utility values for each property, and another column for the final evaluation of this property which you get by multiplying the utility value with the weight of the property.
  4. The next step is to add together all the evaluations for one alternative. In the example below, the sums are written (in italics) into the red boxes on the bottom line. The best alternative will be the one with the highest sum. This is not necessarily the most profitable alternative yet, because the prices and inputs are not included in this stage.

  5. Attribute,
    or property of
    the alternative
    Weight
    W
    Alternative 1 Alternative 2
    Utility
    value
    U
    WxU Utility
    value
    U
    WxU
    Capacity 40 2 80 5 200
    Ease of use 40 3 120 4 160
    Design, appearance 10 5 50 2 20
    Materials, recycling 10 3 30 2 20
    Total 100 -- 280 -- 400
  6. If the analysis is made in the above fashion and only the utility values are considered, a final step in the analysis will be, for each alternative, to compare its total utility value (from the bottom line of the above table) to the price (or other input) of the same alternative. This is done simply by comparing the ratios between total utility and input (which often includes both investment and annual cost, cf. Expenditures). The highest ratio indicates the optimal alternative.

    An alternative method is to include the costs in the table as an additional row. Its weight in the final evaluation is then set at 40% or 50%. An example of such a table is given elsewhere.

Testing

Testing a proposal is a procedure where the presumably best alternative for a proposal, or a few of the best alternatives, are put in practical operation in a small scale and for a limited period of time. On the basis of the results the tested alternative then will be either rejected, modified, or accepted into continuous operation.

Usually the goal is to invite a few future users of the proposal to participate in testing, though this cannot always be achieved and a convenience sample of people is used instead. The researcher asks them to act in the test in a similar way than they would do when the final accepted proposal is in operation, though often part of their operation has to be more or less simulated because a full-fledged operation had been impractical to arrange. The method is sometimes called "pilot project".

Testing is very usual in industrial new product development, especially for measuring and improving the usability of products. It is normally arranged with more or less functional types of prototypes of the product, see Evaluation in Product Development. Some arrangements and procedures can be adopted from the method of scientific experiment with the exception that usually it is unnecessary to include the existing product or the present state of things as an alternative because it is already well known to the testers.

Another variant of practical testing of new products or services is market testing where a restricted series of the new product or service is being marketed to genuine consumers living in a chosen area.

En Español  In Finnish   Contents

August 3, 2007.
Comments to the author:

Original location: http://www2.uiah.fi/projects/metodi