A Conceptual Framework
Philip McGee, Ed.D.
In 1959, Donald Kirkpatrick outlined his now famous four levels of evaluation for training programs: reaction, learning, behavior and results. Building upon this foundation, Jack Phillips has since added a fifth level, that being the Return On Investment (ROI) produced by a training program using the financial formula:
Unfortunately, both the Kirkpatrick model and Phillip’s ROI formula may not provide the type of information needed by trainers and instructional designers to evaluate and improve training systems. To address this need a new model has been developed.
A New Training Systems Evaluation Model
First, let’s recognize that all systems produce results. Albeit, the results they produce may at times be difficult to predict, but nevertheless, systems produce results. Secondly, to achieve results, something must happen; something must be done. In other words, activity must take place. Third, in order to have activity, resources must be used. These three elements of a system hold true whether the system is mechanical, electrical, biological, financial, social or educational.
The next step to understanding this new evaluation model is to examine the dynamic relationships that exist between the elements of a system. We will start with the relationship between results and activity.
Effectiveness is concerned with "how well" something works (produces predetermined, desired results). This concern for "how well" is the basis for another concept known as quality. As designers of systems, we must strive first for effectiveness. For without it, there is little reason to proceed with the design of any system.
Efficiency asks the question, "How much?" However, this is a dangerous question to ask. Dangerous in that if we cut resources too much, we run the risk of producing poor results.
This is not to say that we should not be concerned with resources and their associated costs. We should, because there is an optimum balance within every system: a balance between resources and activity, and activity and results. This optimum balance is known as productivity.
Applying the Model to Improving the Productivity of Training Systems
Let me expand on this concept. Instructional quality is the major concern of curriculum development. Curriculum development answers the question, “What should be taught?” It is during the development of the instructional curriculum that a performance standard for the curriculum is established and program content is identified. Instructional quality is often measured by how well participants are able to predictably achieve the objectives of the program. This data is most often gained through traditional testing procedures. For example, a particular curriculum may produce results, wherein an average participant achieves a 95% level of competency on the material presented.
Instructional quantity, on the other hand, falls into the domain of instructional strategy, which is the process of determining and selecting the most efficient method and media for delivering a program of instruction (curriculum). The goal of instructional strategy is to answer the question, “By what means should the curriculum be taught?” Again, a performance standard must be established by which to measure this dimension. Common standards are money, time, instructional staff, equipment required, i.e., instructional resources.
In order to determine the productivity of an instructional system, we must consider both the results produced by the curriculum and the instructional resources required to deliver the curriculum. Keep in mind that productivity is a ratio or composite measure of both the effectiveness and efficiency of a system.
For example, let’s say that we have an instructional system in which 95% of the participants achieve competency using a media/method to deliver the training which has a per participant cost of $125.
Taking Corrective Action
In other situations, given different data, trainers and designers may want to improve their curriculums. Which brings us to the following.
Factors that Influence the Effectiveness of an Instructional System
Factors that Influence the Efficiency of an Instructional System
Comparing Two or More Instructional Systems
Let us say that we have two instructional systems that deliver the same
information and skills. 85% of the trainees, who use Instructional System
A, achieve competency for an average delivery cost of $100 per participant.
However, 95% of the trainees who use Instructional System B achieve competency
using media and methods which have a per participant cost of $125. Now
then, which is the most productive instructional system of the two being
considered? To determine this, we must graph out the two instructional
It can be seen in the above graph, that Instructional System B falls higher and closer to the line we call the "Slope of Productivity," and therefore is the more productive choice.
A Grounded Model
The basic elements of a system: resources, activity and results, are described throughout the literature on systems theory and are sometimes referred to simply as an input/output model.
Accountants and financial people, who often speak of return on investment, have known the relationship between resources and results: (ROI). ROI is where returns are results and investments are costs, and in this way, are able to determine the health of a business enterprise (activity).
Physicists and engineers recognized long ago that in order to describe the performance of various phenomena and systems, they had to be described in terms of dynamic relationships between two variables. For example, miles per gallon or feet per second.
Thomas Gilbert, a founding father of the human performance field, developed the First Leisurely Theorem, which says that worth is equal to value divided by costs. In other words, activity adds value to resources (cost) and results in something of greater worth.
Peter Drucker pointed out in the 1970s that effectiveness was doing right things, while efficiency was doing things right. In the field of education, these concepts were expanded upon by Ivor Davies and applied to decision-making concerning instructional methods.
Davies, Ivor K. (1981). Instructional Techniques. New York: McGraw-Hill Book Company.
Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers.
Gilbert, Thomas F. (1978). Human Competence. New York: McGraw-Hill Book Company.
Kirkpatrick, Donald (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.
Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods, 3rd Edition. Houston: Gulf Publishing Company.
Rothwell, William J. and Kazanas, H. C. (1998). Mastering the Instructional Design Process, 2nd Edition. San Francisco: Jossey-Bass Publishers.