Developing an Industrial Product, page 4:

Evaluation in Product Development

En Español  In Finnish   Contents

When the product development cycle nears its end, the cost of making changes increases. Changes to design should be done early in the process in order to make the most impact and to cause less financial stress. Therefore, the project should be submitted for evaluation as early as is practical in the design process. It is then easy to alter the project designs on the basis of the tests.

The first evaluator of the project is the designer himself. However, he cannot know all the situations where the future project will be used, so additional estimations should also be asked of others. Nowadays, appraisals preferably include the whole cycle of life of the product, that is, also its use, transportation and final rejection with its ecological consequences. Often the most valuable suggestions come from either the production people or from the future users of the product. The skills of a researcher are useful when gathering such a wide range of opinions, and when summing them up.

Presenting the Draft and Prototype

The method of presentation is important when the designers demonstrate their proposals to be evaluated by people who are not accustomed to those conventions of drawing that the designers normally use. The goal is to make the viewer's responses to the presentation the same as to the real thing. If this is the case, the presentation is said to have good fidelity.

On the other hand, making very realistic physical models or prototypes can be quite expensive and time consuming, so the choice of method needs some consideration.

The method of presentation or "prototyping" should be selected on the basis of the purpose. The presentation can be intended just to help the designer in his work by showing whether a design is feasible, practical and good looking. It can be used to plan the manufacturing process. It is also possible to ask tens or hundreds of users to evaluate the new product. (On the question of selecting the users, see Sampling.) And finally, a prototype can be tested in real environments. Often it is the only effective way of evaluating the user interface of the product, or the ergonomic factors in its use.

For all these people who are not familiar with the product, it becomes easier to understand a concept if there is a realistic presentation or a physical model to see. There are many possible ways of presenting design proposals. A comprehensive list of them is given in Säde (2001). Some of the most common are presented below, starting from the most symbolic methods and ending in the most realistic ones. Realistic presentations are usually more expensive, so they are seldom used in the early phases of design.

Testing the Prototypes

Beside the method of presentation, you will have to consider the environment where the presentation takes place, the method of observing the behavior of test persons, and the method of gathering their views. In a laboratory people do not always behave naturally, which may risk the validity of the results. An alternative might be organizing the testing as normative observation in a natural environment; however, this method is more expensive and involves more disturbances.

If the product design can be presented through internet that gives a handy channel also for selecting and contacting the evaluators. This is the normal method for evaluating "beta-versions", i.e. fully functional early versions of computer programs, but it can be used for physical products, too. For example, Nokia selected early in 2005 some fifteen well-known blog writers and sent the latest model of portable phones to them for testing. The comments were given in public on the blog page, and they thus functioned simultaneously as publicity.

Testing can comprise one or more alternative proposals for the product which the test persons are asked to use in the normal way. Sometimes the target can also be to submit it to more difficult circumstances than normally.

Before the test the researcher explains the purpose of the test and shows the apparatus. This is done in order to avoid later distracting the subject's attention. It can be appropriate also to explain that the researcher will provide no help in the decisive test situation, and that anybody can quit at will.

Expectation. Note that in the test the people are not evaluating the product in itself but primarily its relation to the person's expectation of what the product should be. A layman's expectations are often simple and easy to satisfy, while an expert has perhaps ostentatious expectations and few products can fulfill them. In other words, a layman can estimate the a product as 'excellent' while an expert judges the same product as 'poor'. This means that you will get a lot of random variation into the results if you do not remove it by registering beside the opinion, also an indication of each individual's level of expectations. This could be done by asking, for example: "Have you used a similar product before? Which model was it?" or "If you were intending to buy a product like this, are there other models that you might select? Alternatively, you could before showing the product present a questionnaire where the test persons can define what they expect of the product.

Points of View in Evaluation

Parties that often contribute to the evaluation of a product proposal include:

  1. future users of the product,
  2. manufacturing people and technology,
  3. the marketing people and organization,
  4. outsiders and the environment,
  5. the school or association of professional designers, that the designer is a member of.

Some points of view that are typical of the above parties are discussed below. Note that it will often be impossible to fulfil simultaneously all the requirements, and an arbitration between them may be necessary, for which the methods of Summing Up the Evaluations can be used.

1. The point of view of the user. Important aspects include the following, most of which have been assigned specific www-pages elsewhere:

A competent designer can perhaps divine some of these points of view, but a more accurate method is that people, selected among the genuine users, evaluate the products. Ideally these people should have similar opinions to the intended future segment of customers, which means that they should be a representative sample of the targeted segment of customers.
On the other hand, products that are intended for everybody should be evaluated also by people who have well grounded special requirements, such as:

User Group: Viewpoints important to this group:
Visually impaired users: Is it possible to use the product without seeing its controls (e.g. by touching and feeling knobs which have three-dimensional identification symbols)?
Is the instruction booklet available in other formats such as cassette, Braille printing or synthesized speech?
Hearing-impaired users: Is it possible to dispose of audible signals or interchange them with other types of signal?
People with reduced ability of motion or muscular strength: Does use of the product require considerable strength or agility?
When used, does the product remain steady in place?
Are the movable parts too heavy?
Are the control knobs clumsy or too small?
People bound to wheelchair: Can the product be used when seated?
Epileptics: Has the product sharp corners or protruding parts where the user can hurt himself when falling?
Children as intended users of the product, and other people that are mentally not quite mature: Are the controls of the product intelligible and logically grouped?
Are the instructions clearheaded?
Small children who should not use the product: Are there efficient barriers which protect dangerous components?
Are these barriers removable when they are not needed?

The table above can help you in selecting which special groups need to be represented in the evaluation team. The team becomes thus either a weighted random sample or a non-random sample of the whole population. The total number of test persons is usually between 5 and 15, but it is also possible to start testing with a smaller number.

2. The point of view of manufacturing is easily examined by consulting the people from the production plant, if the manufacturer has already been defined. If not, you can study the available general theory of manufacture, or you can try to find competent evaluators in the field, among specialists of the pertinent new technology. The difficulty is that the specialists of a given technology are not always known. They almost never belong to a definite population that you could enumerate and select a proper sample of it, not to speak of posting questionnaires to their addresses. Some possible approaches when searching experts in a developing field of technology are given under Populations of evaluators.

Sometimes a competent methods engineering specialist can find ways of saving time and materials by developing not only the work itself but also the product. The picture below is an example of the improvements made to an engine governor lever. The original lever (1) was cast steel, and it required 9 working phases. The new, simplified lever (2) was made by bending a steel plate in just 3 phases (Barnes p. 52).

LeverImproved lever

How well adapted the product is to manufacture, can be evaluated by considering the following questions:

3. The viewpoint of marketing typically includes the following themes:

Chair packageFrom the viewpoint of logistics the packaging of the product is important. Sometimes it is possible to design the product and its package so that it saves a lot of space and cost. For example, furniture items often enclose much empty space which could be disposed of by packing the constituents of the product into a compact bundle to be later expanded and assembled into the final shape, perhaps by the customer. On the right is an example of squeezing the components of several chairs into a small flat box (from Berglund, 1976 p. 58).

4. The viewpoint of outsiders and of the environment. If the project or product is likely to incorporate additional, not wanted or detrimental effects to other people, you should consider acquiring evaluations also from these "stakeholders" of the project. These are persons who have some relation to the project and are likely to benefit and/or suffer losses because of it. See a general list of stakeholders of a development project. The inconveniences to outsiders caused by research and development are also discussed under the title Ethical Considerations.

Effects on the environment are the theme on the page Ecology of Products.

5. The designer's professional targets. The designer is expected to take into account and arbitrate all the requirements that originate from the parties that have been enumerated above. This arbitration is often slow and laborious, and it is only natural if a designer instead prefers to give the impression of having found his proposal with masterful ease and speed. A simple, clear, precise and powerful shape of the designed object can uphold such an impression, and indeed such a shape is taken as a target by many a designer.

A cardinal question in modern design is the relation to exemplars, i.e. to earlier works of other artists, and to traditions, either vernacular or classical ones. There are several approaches that a designer can choose to follow:

5.1. One possible approach is to exploit fully a suitable earlier model, modifying it only in one or two small details when the new situation so requires. This has always been the normal approach in traditional handicrafts and even today there are fields of design where it is important to follow closely the latest trend or the "industry standard". Moreover, an exemplar can be a convenient instrument when defining a product concept, though today there are often so many new requirements for the new product that no suitable exemplar can be found. Many professional designers regard the exemplar method as inferior because it is said to restrain innovation.

5.2. The opposite approach is to avoid totally depending on tradition or on earlier works of other artists, in other words taking care that only novel and original structures and shapes are used in the design. This technique is more difficult, but when it succeeds, it certainly demonstrates the artist's mastery among his colleagues. Novelty and originality are, indeed, among the highest values among design professionals today. For example, originality has long been one of the principal criteria when selecting products for the professionally important Design in Finland Suomi muotoilee exhibitions, where it has been defined as follows:

"Originality means that the product or its component is a result of reasoning or innovation which has helped to find a novel or improved solution to a problem or situation" (quoted from Takala-Schreib, 2000, 86)

An alternative definition for originality comes from perception psychology, see Expectation and Distinction on the page Beauty of Products.

5.3. A third, intermediate strategy for dealing with exemplars is often adopted by capable designers who keep themselves well aware of current trends, fashions and recent works of colleagues and occasionally take a feature from these as a source of inspiration and to be modified thoroughly and more or less creatively and ingeniously.

A similar approach can be used with the models taken from traditions: it is often possible to cultivate them creatively, either respectfully or ironically (which latter trick was often done by post-modern architects). Appreciating such a witticism is possible only for a professional who knows thoroughly the pertinent traditions.

The above mentioned professional targets and criteria for the art of design, such as originality, novelty and strength of shape, do not derive from the clients of designers. They are mostly results of a social evolution inside of the pertinent group of professionals, and thus they vary with time and in different countries. They are seldom well known or understood outside of the profession, though it is, of course, possible for any interested layman to study and learn the values of the professionals, to appreciate them and to become thus a "connoisseur" of the art.

Summing Up the Evaluations

If the appraisals come simultaneously from several people and if they also include evaluation of several attributes of the product, it will often be difficult to get a total view of the analysis. In this situation it can be advisable to consider whether it is possible to organize a collective discussion between the evaluators where they perhaps could settle at least some of their differences of opinion. Some available methods are:

Cost Benefit Analysis is a method for locating the best alternative among several proposals. It is definitely quantitative in nature, so that for each property of the alternatives that shall be evaluated, the degree of satisfaction must be given with numbers. The degrees can be defined with the help of a table, see example.

The analysis is started by making a table which displays all the important attributes of the proposals, and their weights. As a basis you can use the table of weights that was set up when the desirable product attributes were first time defined for the product concept. However, if the criteria of evaluation have changed much, you can as well make a new table from these.

In order to simplify the calculations, it is permissible to exclude those properties of the products which are equal in all the alternatives.

In any case, the table must include two new columns for each proposal to be evaluated. The first of them (yellow in the table below) gets the scores (on e.g. a scale of 0 to 5) that this proposal merits for each of the attributes.

In the second new column (below: W x U) the product of weight and score is calculated. Finally all these products are summed up on the bottom line (red cells). The largest sum indicates the best alternative.

Attribute,
or property of
the product
Weight
W
Alternative 1 Alternative 2
Utility
value
U
WxU Utility
value
U
WxU
Capacity 40 2 80 5 200
Ease of use 40 3 120 4 160
Design, appearance 10 5 50 2 20
Materials, recycling 10 3 30 2 20
Total 100 -- 280 -- 400

If the analysis is made in the above fashion and only the useful properties of the proposals are considered, a final step in the analysis will be, for each alternative, to compare its total utility value (in the red cell) to the price (or other input) of the same alternative. The highest ratio between utility value and input points out the optimal alternative.

Note that the inputs should be measured from the same point of view than the utility values. That is, if the analysis is made from the manufacturer's point of view, inputs consist of the costs of production. If instead, the customer's point of view is used, the inputs should include beside the price of purchase, also the annual costs of using the product. These cannot, however, simply be added to the price of purchase because later costs are not equal to present costs. To compensate for the difference, you should discount the future yearly costs, translating their value to the same year as the product's price was paid, or will be paid.

For the payments, the same procedure as for the benefits is available for simplifying the calculations: those payments that are equal for all the alternatives can be excluded from the calculations.

Market testing

Market testingIf the product is to be produced in large quantities it can be advisable first to test the consumer response in the marketplace so that people have the option of choosing the product among the rivals and having to pay for it with their own money.

The theoretical counterpart of market testing is the marketing mix which provides the basis for market testing, and reciprocally the test indicates in which respects the marketing mix can be improved. It contains several decisions by the management of the company, on at least the following topics (the 4Ps, cf. Jobber 1995, p.15):

For a realistic market testing you will need a certain quantity of the product so that you can start marketing it, for a limited period, in one or more chosen geographical areas. In the same time you have the option of testing advertising, too (cf. Information content in advertisements).
Experimental designIf you do the testing in two or more places or periods and vary the price, product properties or other parameters, testing resembles an experiment where the product variants act as "stimulus", customers of the shop as "object" and the amount of sales is "reaction".

A methodological problem in market testing is the difficulty of selecting a representative area, or a few of them, cf. Non-Random Sampling. There may be practical problems, too: it can be difficult to persuade a local distributor to cooperate. Remember that rivalling companies should be kept unaware of the operation so long as possible, otherwise they will do their best to spoil your testing.

From the results of market testing usually the following statistics are calculated:

On the basis of these statistics it will be possible to calculate the most profitable combination of marketing decisions, product properties and price. A prerequisite is, of course, that all the tested alternatives fulfil all the requirements for usability, ease of manufacture etc., enumerated earlier.

Feedback and Critique

Every organization sometimes makes mistakes. The mistakes will be repeated if there is no feedback on them. Consequently, if your business intends to stay long in the market, it is advisable to collect and stimulate feedback and criticism on your operations and products. In the long run, such companies that recognize and satisfy the needs and wishes of the customers better than the competitors will do well.

There are two approaches when gathering feedback:

Passive Feedback Reception

One problem in modern industry is that nowadays the producer seldom has a direct customer relationship with the consumers of the product. Wholesalers and retailers now serve as intermediaries. The voice of the consumer reaches the growing and improving production plants with increasing difficulty if special channels are not created for it.

Probably the most common method to obtain customer feedback is complaint management. Besides treating the original problem of the customer by e.g. replacing the faulty product, the enterprise can collect the information content of the complaints from those sources where they have been directed:

  1. the customer is not satisfied, but he does not complain
  2. the customer complains orally to people close to him
  3. the customer complains to the retailer / manufacturer
  4. the customer appeals to an authority, e.g. to a Consumer Complaint Board, if available in the country, or sues the retailer or the manufacturer in court.

The best category to investigate among these will be the complaints made to the company itself; it will also be easy to gather the complaints made to the retailers of the product. The other groups are seldom investigated.

Complaint management is, however, a crude method for gathering customer feedback because it is involved mostly with complaints and costs and lacks completely all positive feedback, ideas for developing the quality of existing products and ideas for better products.

There should be a parallel channel for not only complaints but for any comments that a customer might wish to give to the company. The customers have, of course, the option of sending a letter, but in practice this is seldom done. Instead, it might be worthwhile for a company to complement their existing www-pages with the option of transmitting customers' comments to the company. This can be done with HTML forms which can accept feedback either as fixed-choice alternatives or as opinions in free form, or both. In either case, the forms can be made so that the feedback arrives conveniently pigeonholed for each different product or service of the company so that they are easy to file and analyze.

Lately have appeared a growing number of WWW-sites for general feedback concerning any product that the public wishes to discuss, i.e. anybody who wants to say something about a new product can start a specific line of discussion for it. These www-pages are independent, i.e. they are not operated by the makers of the products which shall be discussed on the page. In the future, if the discussion becomes lively on these pages, it might be a good idea for a manufacturing company to study now and then the responses.

When analyzing the voluntary feedback that people send in as their opinions you should keep in mind that these people are far from being a random sample of all the customers (in fact, they are a "sample of volunteers") and therefore their opinions may be biased, i.e. differ from those of other customers who did not give feedback. Beside other possible types of bias, there is normally the phenomenon of the "absent middle": those people who have no decisive opinions on your product will send no feedback, which means that in the feedback that arrives two sorts of opinion predominate: the distinctly positive and the clearly negative ones.

Failure rate If you wish to compare the amounts of complaints concerning the different products of the company, you should keep in mind that they are usually most frequent in the running-in phase of a new product when the novelty still suffers from "children's diseases". Another rise in complaints can be expected when the useful life of the product turns to its end and failures become more frequent. The "bathtub curve" on the right is from Abbott, 1989 p. 127.

Feedback from manufacturing. Suggestion Scheme is an arrangement that has long existed at many industrial plants. It has mostly been used for gathering ideas from workers and employees for improving the production process, but it could assist in improving the product, as well, though this option has seldom been used. The system usually consists of the following parts:

Public critique, when it exists, could be a valuable source of feedback to a manufacturer, although it is, per definition, normative and thus the judgments depend on the values and goals of the critic or of his organization (see Point of view of normative study). Usual points of departure for public critique include:

User point of view. Consumer organizations and their journals, and lately a growing number of technological journals are continually testing cars, boats, computers, cameras, hifi and other new products. The purpose is to help the customer to select the right product for his needs, taking into account all the aspects that can be relevant to its owner and long-time user. These aspects can include all the usual goals of product design which normally are measured and assessed on prefabricated scales, weighted and summed up as value engineering tables so that the result as accurately as possible agrees with the average user's point of view. Tests can include both scrupulous laboratory testing and prolonged intense practical operation in order to uncover potential weaknesses in the product. Pertinence and reliability of such tests are usually good.

Views of the designer's colleagues. A competent colleague's critique of a proposed or executed design could often be invaluable, because he can profit from his experience of the requirements that similar products normally are expected to fulfil. He is thus often able to simulate the customer's preferences quite successfully in the evaluation.

However, the content of professional critique depends much on its purpose and address. In a country where the profession is young, the designers need to get a footing for it by convincing manufacturers and customers on the usefulness of their services. As a contrast, when the profession is well established, the discussion usually turns more inwards, to professional topics, which can degrade its relevance to the users of the products and to other outsiders. This can happen especially at the time of a paradigm change in design style, i.e. when the younger generation of designers is struggling for influence in the ranks of the metier. Such a situation is described quite acidly by Bourdieu (1994, 62):

The history of a field [of art or science] advances in the fights between those that have a name (as a writer, a philosopher, a scientist, etc.) and their challengers, as they are called in the sport of boxing. In these fights writers, schools and works of them get older. Those that already have created masterpieces (and have won a standing in the field) fight in order to keep it, to immortalize themselves as classics, and to stop all development. Their challengers in turn can make a masterpiece only by thrusting into the past these established predecessors. ... The climbing new avantgarde tries to regenerate the foundations of the genre. They insist on returning to the roots and to the original purity.

Good professional practice of design, i.e. the merits (if any) in the product's design itself, are understood differently depending on place and time, but often mentioned merits include the skill of handling contrasting requirements in a balanced way; clarity, precision and strength of the result, novelty and originality of the form, taking tradition in account skillfully, which tradition can be either classical or vernacular. What is "good" design, was already discussed earlier, in the paragraph Professional designer's view.

Generally designers do not often want to put on paper their opinions about the works of their colleagues. One reason probably is that it really can be difficult to analyze designs in writing, because a great part of the competence is learned from masters to pupils, the skill exists mostly in tacit form and cannot be fully clarified as explicit theory. Many concepts are used without clear definitions, and in fact an important share of collegial evaluations appears not as written criticism but by selecting (by a jury composed of eminent designers) produced works to a national exhibition or to a professional publication.

Views of exhibition visitors. One strong paradigm of art criticism is that the critic just describes his impressions and reflections during the short time that he inspects each work. Critiques in newspapers on exhibitions of designed products often follow this "impressionistic" tradition. Such writings thus contain just one person's opinions, albeit that the writer usually is an expert in the field, and they normally contain only a few aspects of the object. Sometimes an object whose only merit is that it deviates from the common line in the exhibition gets good marks in such an assessment. Impressionistic critique is therefore only occasionally useful for product development purposes. However, this style of quick and superficial inspection of objects perhaps a little resembles the situation in a business shop, when a client is choosing a product for himself, which means that this type of criticism can sometimes help designing products intended for impetuous clients. The questions around immediate appeal of products are discussed further on the page Evaluating Normative Proposals.

Public criticism often examines a large number of products in parallel. Consequently it often fails to focus on the products' specific strengths and weaknesses themselves, and instead simply puts the question: which product is best in this exhibition? Answering such a question is much easier than laborious speculation on all the aspects of a prolonged use of a product, but the resulting critique will be much less useful for product development.

Active Feedback Collection

When a company has long been selling the same product, it often turns out that the sales slow down, and it becomes necessary to find out in which respects the product could be improved. Receiving passively complaints (see above) and other spontaneous feedback can give some useful hints, but often their sporadic content does not give answers to exactly those specific questions that the company would want to ask, for example, concerning potential improvements to the product that technological development has recently made possible. In such a situation the company needs to consider a project of active gathering of feedback where exactly the right questions could be put to the right people.

Active feedback collection does not much differ from typical survey research with questionnaires or interviews. Per definition, the population consists normally of the users of the product in question, but there is often the practical difficulty that the names and addresses of these people are not known. Many companies have started building up a customer register by by enclosing in the package of each sold product a prepaid return envelope in which the client can give his or her name and address. For products that are sold on the internet, customer registration with a HTML form is common and easy.

In the case that there is no customer file nor time to build up one, sometimes a sufficient number of product users can be registered when they are, for example, visiting the product after-sales service or a spare parts shop. Note that the samples that depend on the customer's willingness to co-operation are often a little biased, see Non-random Samples.

Once a contact to a product user has been established, the normal method is to give them a questionnaire with return envelope. A modern alternative (beside the internet form, discussed above) is an arrangement where the client can call, with the fee of a local call, to a number where a machine records the feedback. The client gives his response by following the instructions of the machine, either by pushing the buttons of the phone, or by dictating an oral message.

When planning the feedback questionnaire you should note that the evaluations in itself can be difficult to decipher without some additional information. A layman can estimate as 'excellent' the same product that an expert with higher expectations judges as 'poor'. Therefore you should try to register beside the opinion, also an indication of the person's level of expectations, cf. Expectation.

Note that both customer satisfaction and dissatisfaction consist of various components such as the price of the product, the usefulness of the product in different situations, its symbolism and status, durability and maintenance. Some of these components may be unsuitable from the customer's point of view, while others may be perfectly in order. Therefore, it is seldom very useful to get a total utility value of the product - instead the customer should be allowed to differentiate his answer.

As a medium of reciprocal discussion questionnaires and structured interviews are insufficient in the sense that when it is the company that asks the questions and the customer who answers them, the company will get no new ideas. Therefore, the customer should be allowed and encouraged to put forward such ideas that nobody in the company has come to think of. In this respect, a thematic interview, a discussion over the phone, is better than a questionnaire because it allows the consumer to present his latent information and wishes while it also makes additional questions possible and allows the parties to discuss new ideas and develop them together. If the customer is really interested in finding improvements to the product, he/she could be asked to participate in brainstorming or testing of the next model of the product.

Note finally, that to ensure a sufficient answering percentage, you have to reward the respondents for their trouble by, for example, giving them useful information about the use or care of the product they have bought.

The www-pages on the development of an industrial product:

  1. Logic of Product Development
  2. Analysis in Product Development
  3. Synthesis in Product Development
  4. Evaluation in Product Development (this page)

En Español   In Finnish   Contents

August 3, 2007.
Comments to the author:

Original location: http://www2.uiah.fi/projects/metodi