Portfolio Tests : Are They Here to Stay?
The author challenges the usefulness of the recall data produced by portfolio tests as manufactures of the performance of print advertising. The differences between the effectiveness of these recall measures and other recall measures are considered in a new light. PORTFOLIO test has been THE most widely used approaches another only the “recall score” aspects of portfolio the to tests are questioned here. Other findings pre-testing of print advertising.
This method, which involves exposure of the “test ad” in a folder which also contains number of “control ads,” yields various and sundry “playbacks” or recall data as the principal criteria of advertising performance. Such recall measures are typically expressed as a percentage of respondents who can recall the brand and/or product advertised, or pictorial or copy aspects of an advertisement after the folder of advertisements has been viewed in the interviewer’s presence, for a brief time interval.
These playbacks are usually based on unaided recall of the ad or ads in question, although standard “prompts” are often used to produce aided-recall playbacks to supplement those obtained from unaided recall. The portfolio method may also involve ratings of the ads or products advertised in many instances. The playback scores are the main quantitative derivatives of portfolio tests per SE, however. Note that ; ABOUT THE AUTHOR: John C. Maloney is Manager of Research Development at Leo Burnett Company, Inc. , Chicago.
He is a graduate of the University of Nebraska and holds M. S. And Ph. D. Degrees in psychology from Purdue University. In addition to broad experience in marketing and consumer research work, Dry. Maleness’s activities have provided him with an unusual opportunity to compare wide range of advertising-research techniques. He is a lecturer in Psychology of Advertising and Selling at Northwestern University. 32 from tests which may involve the presentation of more than one advertisement to a respondent in the same folder or “portfolio” are not considered in this article.
POPULARITY OF PORTFOLIO TESTS Portfolio tests have a high face validity. They seem more realistic than the protesting methods which “rub respondents’ noses in the test ad”-?without the benefit of control ads as a background. They are straight-forward, easily administered, easily analyzed, and produce results easily communicated to the non-researcher. Moreover, they are highly reliable, in the sense that very comparable results are obtained from repetitions of portfolio tests using the same control ads. The author has found the average rank-order correlations of unaided, correct brand recall scores to be in excess of . 90 for all possible interrelations of seven different ten-ad tests using the same nine control ads. ) It is little wonder, then, that portfolio tests have been so popular. How does one dare to question whether or not they are here to stay? Why would they not be here to stay? THE CASE AGAINST PORTFOLIO TESTS Yet portfolio tests, as they are commonly used today, are not here to stay.
The reason is simply that they very seldom work for the purpose for which they are intended. PORTFOLIO TESTS – ARE THEY HERE TO STAY? Brand and/or product being advertised. These include variations which result from the fact that people are most likely to notice and least likely to forget ads for those brands and Common Purposes of Portfolio Tests products in which they are most Nina common purpose for which portfolio trusted or which they know the most sets (or any other print advertising breakout. sets) are used is to help the advertiser It can be seen that variations of the to decide whether to run Ad A or B as latter sort could account for considerable his next insertion in a given magazine “reliability” or stability in recall score or newspaper, when Ad A and Ad B are measures. If the same ads are involved in advertisements of the same size, are com- repeated tests, the general level of responsible as to black-white or amount of dents’ interest in, or awareness of, the color, and are for the same brand and product being advertised would naturally rotund. End to be comparable from test to test. In other words, portfolio tests are often If the scores in question are more subject used to test headline, copy, or graphic to such influence than they are subject to variations when all other factors are held the influence of the differences among ads, constant and the advertiser wishes to SE- the scores would tend to be stable but could elect the best ad from among several hardly be regarded as meaningful measures possibilities. Of the performance of the ads themselves.
Another use for portfolio tests-?the one The point is that portfolio test playbacks which accounts for the majority of re- are more subject to the non-valid, “underestimating instances of the use of such tests-? or-awareness” influence than they are sub’s to determine how well a new ad “stacks Eject to valid, “goodness-of-the- ad” influence. Up” in comparison with previous ads for the same brand of the same product or for evidence Against Portfolio Tests comparable brands of closely related The indictment of portfolio-test recall products. Cores rests on the assumption that if the same control ads are used for tests Factors Which Can effluence Recall Scores f different “test” ads for the same brand For present purposes it is convenient to of the same product, then the “test” ads’ classify the innumerable factors which can recall scores should vary more from test to influence recall scores in three ways: test than do the recall scores for the control 1 . Sources of “error variance. ” These ads. Include the troublesome variations If this is not found to be true, it can in interviewers’ motivations and ability- be assumed that either (a) the scores were ties, variations in respondents’ ability- unstable or unreliable (a possibility overtires to member the ads for ruled by a superabundance of evidence), interviewers, respondents’ varying or that (b) the scores are more subject inclinations to co- operate with the to the aforementioned, non-valid, “interventionists, seasonal factors, prop- or-awareness” influence than to the valid elms of “response set,” prior exposure “goodness-of-the-ad” influence.
Either of biases, and other non-valid influences. These two conclusions would clearly jugs. Variations due to the differences gets that portfolio-test playbacks are among ads themselves. These Include suited for the purposes for which they are the valid influences of better or poorer commonly used. Use. Oaf headline, copy, graphics, and In Table 1, playback tabulations are brother elements. Ended for six different ten-ad portfolio 3. Variations due to respondents’ inhere- tests. These are expressed both in terms of .NET interest in, or awareness of, the percentage recalling the correct brand and Consider the purposes for which portfolio tests are ordinarily used. At the same time, consider those factors which might influence portfolio-test recall scores.
