|
EVALUATION
OF FORMAL, EMPLOYER-SPONSORED TRAINING IN
THE U. S. HEALTHCARE INDUSTRY Executive
Summary Dianne R. Hill, Ph.D. Performance
by Design Austin, Texas INTRODUCTION Training
is an important aspect of the Human Resource Development (HRD) field as
organizations seek increased efficiency and effectiveness in a complex,
changing operating environment (Stewart, 1997; Phillips, 1997b, 1997c).
Each year some 58.6 million employees receive formal, employer-sponsored
training (Phillips, 1997c). Training magazine’s Industry
Report 1998 (17th annual) indicates that $60.7 billion was budgeted
for training in U.S. organizations in 1998.
Within the same month, USA Today reported that a Rutgers
University study estimated that companies waste between $5.6 and $16.8
billion every year on ineffective training programs (Armour, S., 1998).
Training and development is big business; therefore it should be evaluated
in the same way as other large investments, in terms of costs and
benefits” (Mosier, 1990). This
trend toward accountability emerges as one of the most significant and
visible developments in the Human Resource Development field in recent
years (Ulrich, 1997; Phillips, 1997c). However, despite rising emphasis on
establishing accountability in human resource management systems,
evaluation of HRD programs, including training, faces significant
challenges in implementation. HRD professionals cite numerous barriers to
evaluation; among the most frequent of which are difficulty and cost
(Phillips, 1997b, 1997c). Still, management wants to know the
organizational impact of HRD training and development programs, and
employees want to know that their own investments in such programs produce
results. . This executive summary presents important results on the
percentages of evaluation at different levels, characteristics of survey
respondents, and the impact of management involvement on training
evaluation. BACKGROUND
AND METHOD This
study was designed as my dissertation for a Ph.D. in Education
Administration at The University of Texas at Austin. I was curious about
how training is evaluated and placed the study in healthcare because I was
aware of the cost constraints and challenges faced by the healthcare
industry. I believed that my research could assist the healthcare industry
by providing data regarding current training evaluation practices. Using
my baseline research, my hope is that other research would be follow that
will assist the healthcare industry in making better training decisions. The
primary purpose of this research was to examine how formal,
employer-sponsored training is evaluated in the U. S. healthcare industry.
The current general state of training evaluation in the healthcare
industry is largely unknown, especially in the area of assessing the
return on training investments (Barrett, 1997; Palladoro, 1997). While
researchers in the Human Resource Development (HRD) field have identified
a trend toward increased accountability (Phillips, 1997c), evaluation of
HRD training programs faces significant barriers in implementing training
evaluation including difficulty, cost, and time (Phillips, 1997b, 1997c;
Carnevale & Schultz, 1990). The results from this study provide data
that help answer questions regarding current training evaluation practice
and the influence of those barriers on training evaluation in the
healthcare industry. Survey
research methodology was used to collect data from members of the American
Society for Training & Development Healthcare Forum. Membership in a
training/education and development professional association was believed
to be a strong indicator of both interest and likelihood of receiving
informed responses concerning evaluation practices within members'
organizations. A total of 244 responses from a sampling frame of 894
individuals was received and analyzed using descriptive and inferential
statistics. KEY
FINDINGS A
key way of thinking about evaluation is the Kirkpatrick model of four
levels of evaluation. This study added a fifth level which is return on
investment of training. Table 1 below provides an overview of each
evaluation level and the percentage of programs evaluated at the
respective levels as reported by the survey respondents. Standard
deviation reflects variability from mean of responses.
The top four barriers to ttraining evaluation were cost, difficulty,
and lack of training (and/or experience). Table
1 Percentage
of Programs Evaluated at Different Levels of Evaluation (N=244)
MANAGEMENT
INVOLVEMENT IN TRAINING EVALUATION Barriers
to Training Organizations
with a training evaluation policy had fewer barriers to evaluation. In
particular, for levels 2, 3, 4, and ROI, organizations with an evaluation
policy reported (statistically significant) fewer barriers at these
levels. There appears to be a gap between management’s reported emphasis
on evaluation and the actual evaluation-in-use (Argyris & Schon, 1974)
which could remain low if management continues to believe in the value of
evaluation but does not require feedback of results of evaluation. Differences
in barriers were associated with the level of evaluation conducted.
Barriers to evaluation at Levels 1 and 2 were that evaluation has little perceived value to the organization and that it is not
required by the organization. Barriers for Levels 3 and 4 (plus ROI)
were cost, lack
of training, and not required by
the organization (order of barriers varied between Levels 3 and 4).
Overall past research regarding barriers to evaluation is consistent in
naming cost, lack of training, and the organization’s failure to require
evaluation (Phillips, 1997a, 1997b, 1997c; Twitchell, 1997. Reporting
to Management Organizations
where training evaluation results were routinely reported to management
conducted a higher percentage of evaluation. Further analysis using an
independent samples t-test was conducted using evaluation results reported
to executive management and percentage of programs evaluated at various
levels. In organizations where the evaluation is reported to executive
management, the percentage use of the evaluation levels 2 through ROI is
statistically significantly greater than in those organizations that do
not routinely report evaluation information to executive management. WHO
RESPONDED TO THE SURVEY Looking
at who responded to the survey will help us better understand the impact
of management on evaluation. Types of healthcare organization structures
represented in the study are listed in Table 2, and types of healthcare
organizations are presented in Table 3. Table 4 provides information about
the individual respondents: title, job function, number of years in
training, and gender. Table 2Type
of Healthcare Organization Structure
1Note:
Some respondents reported more than one organization structure so the
total N=292. Table
3
Types
of Healthcare Organization
2Note:
Some respondents reported more than one organization type so the total N=
265
Table
4
Respondent
Demographics
CONCLUSIONS Conclusion
One Most
training programs in the healthcare industry receive minimal evaluation,
typically at Level 1. Level 1 reaction questionnaires are an immediate,
relatively low-cost way of monitoring a training program. Although this
level of evaluation may continue to be sufficient due to some of the
barriers identified in this study, growing interest by top-level
executives will increase the likelihood that more extensive evaluation is
required. Although evaluation at Level 1 may be a gateway to these higher
levels, further research is needed on how to help organizations move
beyond the gateway and use higher levels of evaluation. Conclusion
Two When
training evaluation becomes a part of the healthcare organization's
infrastructure or culture, necessary resources are made available to
support training evaluation. Several things are linked to increased use of
higher levels of evaluation. First, a written, implemented policy is an
example of a visible sign of organizational support for training
evaluation. Second, when management requires reporting of evaluation
training, more evaluation is conducted. Evaluation using ROI appears to be
the most difficult to implement, perhaps because there was no identified
clear method to use. In addition, some of the open-ended comments suggest
more training is needed on what types of ROI models can be used. Conclusion
Three Variations
exist in relation to organization characteristics. There is minimal
integration of financial analysis in training evaluation as evidenced by
the lack of responses to survey items regarding Level 4 methods of
evaluation including ROI. It is possible that the resources required to
evaluate training at higher levels do not exist in most organizations, or
perhaps not in the training area. Evaluation at Levels 4 and ROI,
especially, require financial analysis and statistical skills (or access
to an effective resource). Even if these tools are learned in an academic
setting, actual application may not be simple. Conclusion
Four Barriers
to training evaluation (cost, not required by the organization, and lack
of training and/or experience) are actually symptoms and evidence of the
perceived low need for evaluation among organizational leadership in
healthcare. The barriers indicate that evaluation is not a critical need
of the organization, since these barriers could be overcome with
additional resources (Twitchell, 1997; Gutek, 1988). Organizations commit
resources to what they value or as insurance against what they fear (risk
management). Each barrier could be reduced or eliminated based on how
training evaluation is positioned in the marketplace of the organization's
needs. The idea of using evaluation to provide greater accountability,
particularly showing the ROI of the training activities could provide the
impetus for healthcare organizations to invest more in evaluation (Hill
& Phillips, 1997). RECOMMENDATIONS
FOR PRACTICE Practice
Recommendation 1:
Plan evaluation of training by involving consumers of training (trainees)
as well as internal and external customers. Encouraging the perspectives
of other functional areas, hierarchical levels, and geographical locations
can provide valuable, comprehensive input to the process. Inviting 'user'
input to evaluation helps determine if evaluation is appropriate and, if
so, what is needed in the evaluation. "Evaluation, as a tool, rather
than an end product, is a means to understanding" (Gutek, 1988, p.
116). For example, respondents reported barriers to evaluation, such as
cost and difficulty that could potentially be reduced through early
collaboration in program development. Understanding what is important to
the stakeholders can help clarify what is needed in training evaluation. Practice
Recommendation 2:
Audit current training evaluation methods on a regular basis. Determining
what data is collected and how the data is used invites continuous
improvement to the evaluation process. As with many procedures in an
organization, training evaluation methods can become an unexamined habit.
Creating more effective evaluations could free time for other priorities.
Evaluating training for the purpose of producing better training is a
narrow and dangerous focus and could easily lead to doing well that which
is not worth doing at all (Gutek, 1988). Just as training must be
need-driven, so must be evaluation. Another method for Level 4 evaluation,
for example, was the use of patient satisfaction surveys. Audits provide
not only the opportunity to examine what data is collected, why it is
collected, and how it is used, but also often uncovers new data and
methods of collection that could prove more valuable. Practice
Recommendation 3:
Determine if a training evaluation policy could be valuable to the
organization. A written policy on training evaluation that is implemented
provides an ongoing opportunity to support the organization's strategic
and operational goals and issues through training. For example, the
relationship between the existence of a training evaluation policy and
increased percentage of programs evaluated at various levels encourages,
at a minimum exploration of the need for a written policy. Practice Recommendation 4: Establish criteria for evaluation of training programs. Achieving agreement among the stakeholders helps determine what evaluation needs exist (based on the stakeholders’ expectations). Due to the frequency that respondents reported the low perceived value of training evaluation | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||