EVALUATION OF FORMAL, EMPLOYER-SPONSORED TRAINING

IN THE U. S. HEALTHCARE INDUSTRY 

Executive Summary 

Dianne R. Hill, Ph.D.

Performance by Design

Austin, Texas

INTRODUCTION 

Training is an important aspect of the Human Resource Development (HRD) field as organizations seek increased efficiency and effectiveness in a complex, changing operating environment (Stewart, 1997; Phillips, 1997b, 1997c). Each year some 58.6 million employees receive formal, employer-sponsored training (Phillips, 1997c). Training magazine’s Industry Report 1998 (17th annual) indicates that $60.7 billion was budgeted for training in U.S. organizations in 1998.  Within the same month, USA Today reported that a Rutgers University study estimated that companies waste between $5.6 and $16.8 billion every year on ineffective training programs (Armour, S., 1998). Training and development is big business; therefore it should be evaluated in the same way as other large investments, in terms of costs and benefits” (Mosier, 1990). 

This trend toward accountability emerges as one of the most significant and visible developments in the Human Resource Development field in recent years (Ulrich, 1997; Phillips, 1997c). However, despite rising emphasis on establishing accountability in human resource management systems, evaluation of HRD programs, including training, faces significant challenges in implementation. HRD professionals cite numerous barriers to evaluation; among the most frequent of which are difficulty and cost (Phillips, 1997b, 1997c). Still, management wants to know the organizational impact of HRD training and development programs, and employees want to know that their own investments in such programs produce results. . This executive summary presents important results on the percentages of evaluation at different levels, characteristics of survey respondents, and the impact of management involvement on training evaluation. 

BACKGROUND AND METHOD 

This study was designed as my dissertation for a Ph.D. in Education Administration at The University of Texas at Austin. I was curious about how training is evaluated and placed the study in healthcare because I was aware of the cost constraints and challenges faced by the healthcare industry. I believed that my research could assist the healthcare industry by providing data regarding current training evaluation practices. Using my baseline research, my hope is that other research would be follow that will assist the healthcare industry in making better training decisions.

The primary purpose of this research was to examine how formal, employer-sponsored training is evaluated in the U. S. healthcare industry. The current general state of training evaluation in the healthcare industry is largely unknown, especially in the area of assessing the return on training investments (Barrett, 1997; Palladoro, 1997). While researchers in the Human Resource Development (HRD) field have identified a trend toward increased accountability (Phillips, 1997c), evaluation of HRD training programs faces significant barriers in implementing training evaluation including difficulty, cost, and time (Phillips, 1997b, 1997c; Carnevale & Schultz, 1990). The results from this study provide data that help answer questions regarding current training evaluation practice and the influence of those barriers on training evaluation in the healthcare industry. 

Survey research methodology was used to collect data from members of the American Society for Training & Development Healthcare Forum. Membership in a training/education and development professional association was believed to be a strong indicator of both interest and likelihood of receiving informed responses concerning evaluation practices within members' organizations. A total of 244 responses from a sampling frame of 894 individuals was received and analyzed using descriptive and inferential statistics.  

KEY FINDINGS 

A key way of thinking about evaluation is the Kirkpatrick model of four levels of evaluation. This study added a fifth level which is return on investment of training. Table 1 below provides an overview of each evaluation level and the percentage of programs evaluated at the respective levels as reported by the survey respondents. Standard deviation reflects variability from mean of responses.  The top four barriers to ttraining evaluation were cost, difficulty, and lack of training (and/or experience).  

Table 1

 Percentage of Programs Evaluated at Different Levels of Evaluation (N=244)  

MANAGEMENT INVOLVEMENT IN TRAINING EVALUATION 

Barriers to Training 

Organizations with a training evaluation policy had fewer barriers to evaluation. In particular, for levels 2, 3, 4, and ROI, organizations with an evaluation policy reported (statistically significant) fewer barriers at these levels. There appears to be a gap between management’s reported emphasis on evaluation and the actual evaluation-in-use (Argyris & Schon, 1974) which could remain low if management continues to believe in the value of evaluation but does not require feedback of results of evaluation.  

Differences in barriers were associated with the level of evaluation conducted. Barriers to evaluation at Levels 1 and 2 were that evaluation has little perceived value to the organization and that it is not required by the organization. Barriers for Levels 3 and 4 (plus ROI) were cost, lack of training, and not required by the organization (order of barriers varied between Levels 3 and 4). Overall past research regarding barriers to evaluation is consistent in naming cost, lack of training, and the organization’s failure to require evaluation (Phillips, 1997a, 1997b, 1997c; Twitchell, 1997. 

Reporting to Management 

Organizations where training evaluation results were routinely reported to management conducted a higher percentage of evaluation. Further analysis using an independent samples t-test was conducted using evaluation results reported to executive management and percentage of programs evaluated at various levels. In organizations where the evaluation is reported to executive management, the percentage use of the evaluation levels 2 through ROI is statistically significantly greater than in those organizations that do not routinely report evaluation information to executive management. 

WHO RESPONDED TO THE SURVEY 

Looking at who responded to the survey will help us better understand the impact of management on evaluation. Types of healthcare organization structures represented in the study are listed in Table 2, and types of healthcare organizations are presented in Table 3. Table 4 provides information about the individual respondents: title, job function, number of years in training, and gender.

Table 2

Type of Healthcare Organization Structure 

¹Note: Some respondents reported more than one organization structure so the total N=292.

 Table 3

Types of Healthcare Organization 

²Note: Some respondents reported more than one organization type so the total N= 265

Table 4

Respondent Demographics  

CONCLUSIONS 

Conclusion One 

Most training programs in the healthcare industry receive minimal evaluation, typically at Level 1. Level 1 reaction questionnaires are an immediate, relatively low-cost way of monitoring a training program. Although this level of evaluation may continue to be sufficient due to some of the barriers identified in this study, growing interest by top-level executives will increase the likelihood that more extensive evaluation is required. Although evaluation at Level 1 may be a gateway to these higher levels, further research is needed on how to help organizations move beyond the gateway and use higher levels of evaluation.  

Conclusion Two 

When training evaluation becomes a part of the healthcare organization's infrastructure or culture, necessary resources are made available to support training evaluation. Several things are linked to increased use of higher levels of evaluation. First, a written, implemented policy is an example of a visible sign of organizational support for training evaluation. Second, when management requires reporting of evaluation training, more evaluation is conducted. Evaluation using ROI appears to be the most difficult to implement, perhaps because there was no identified clear method to use. In addition, some of the open-ended comments suggest more training is needed on what types of ROI models can be used.  

Conclusion Three 

Variations exist in relation to organization characteristics. There is minimal integration of financial analysis in training evaluation as evidenced by the lack of responses to survey items regarding Level 4 methods of evaluation including ROI. It is possible that the resources required to evaluate training at higher levels do not exist in most organizations, or perhaps not in the training area. Evaluation at Levels 4 and ROI, especially, require financial analysis and statistical skills (or access to an effective resource). Even if these tools are learned in an academic setting, actual application may not be simple. 

Conclusion Four 

Barriers to training evaluation (cost, not required by the organization, and lack of training and/or experience) are actually symptoms and evidence of the perceived low need for evaluation among organizational leadership in healthcare. The barriers indicate that evaluation is not a critical need of the organization, since these barriers could be overcome with additional resources (Twitchell, 1997; Gutek, 1988). Organizations commit resources to what they value or as insurance against what they fear (risk management). Each barrier could be reduced or eliminated based on how training evaluation is positioned in the marketplace of the organization's needs. The idea of using evaluation to provide greater accountability, particularly showing the ROI of the training activities could provide the impetus for healthcare organizations to invest more in evaluation (Hill & Phillips, 1997). 

RECOMMENDATIONS FOR PRACTICE 

Practice Recommendation 1: Plan evaluation of training by involving consumers of training (trainees) as well as internal and external customers. Encouraging the perspectives of other functional areas, hierarchical levels, and geographical locations can provide valuable, comprehensive input to the process. Inviting 'user' input to evaluation helps determine if evaluation is appropriate and, if so, what is needed in the evaluation. "Evaluation, as a tool, rather than an end product, is a means to understanding" (Gutek, 1988, p. 116). For example, respondents reported barriers to evaluation, such as cost and difficulty that could potentially be reduced through early collaboration in program development. Understanding what is important to the stakeholders can help clarify what is needed in training evaluation.  

Practice Recommendation 2: Audit current training evaluation methods on a regular basis. Determining what data is collected and how the data is used invites continuous improvement to the evaluation process. As with many procedures in an organization, training evaluation methods can become an unexamined habit. Creating more effective evaluations could free time for other priorities. Evaluating training for the purpose of producing better training is a narrow and dangerous focus and could easily lead to doing well that which is not worth doing at all (Gutek, 1988). Just as training must be need-driven, so must be evaluation. Another method for Level 4 evaluation, for example, was the use of patient satisfaction surveys. Audits provide not only the opportunity to examine what data is collected, why it is collected, and how it is used, but also often uncovers new data and methods of collection that could prove more valuable.  

Practice Recommendation 3: Determine if a training evaluation policy could be valuable to the organization. A written policy on training evaluation that is implemented provides an ongoing opportunity to support the organization's strategic and operational goals and issues through training. For example, the relationship between the existence of a training evaluation policy and increased percentage of programs evaluated at various levels encourages, at a minimum exploration of the need for a written policy. 

Practice Recommendation 4: Establish criteria for evaluation of training programs. Achieving agreement among the stakeholders helps determine what evaluation needs exist (based on the stakeholders’ expectations). Due to the frequency that respondents reported the low perceived value of training evaluation to the organization, it is possible that stakeholders outside of training have different perspectives on determining the worth of training. Reaching agreement on criteria for evaluation could surface unexamined assumptions as to what is important, thereby streamlining the training evaluation process.

Practice Recommendation 5: Explore alternative ways of conducting evaluation. It is possible that helpful data for an evaluation has already been collected in another part of the organization or available as benchmark data from a similar organization. It is possible that the low frequency of use of various evaluation methods signals not only a limited knowledge of what data is available in other departments (e.g., accounting, management information services, etc.), but also ways in which to use the information.  

Practice Recommendation 6: Evaluation may not be required by the organization currently, but could be required in the future. The quality initiatives that are taking hold in healthcare will create opportunities for increased training evaluation as organizations begin looking more critically at internal processes and activities. One of the most frequently reported barriers to evaluation was that it was ‘not required by the organization.’ Several respondents wrote, however, that they were not currently required to evaluate training, but that ‘it was coming.’  

Practice Recommendation 7: Determine what is important to the healthcare organization and its members. Improvement in human performance to achieve the goals of the organization is the actual end product of training. Training directors must relate their activities to organizational achievement (goals, objectives, and measures) or risk being discarded as a tool of unknown worth (Gutek, 1988). Trainers must continuously scan the policy environment to learn what information key stakeholders view as valuable. Recognition of how evaluation spans organizational boundaries (e.g., internally among departments and externally with clients) is an antidote to 'silo' management of training and creates linkages for organizational effectiveness. What senior management values in evaluation may differ from what the training/education department values (Palladoro, 1997; Gutek, 1988; McGough, 1998). For example, if only half of the respondents indicated that evaluation results were reported to executive management, what are reasons for not reporting results? 

Practice Recommendation 8: Explore ways to expand evaluation resources. Rather than trying to conduct training evaluation activities by themselves, training/education staff might use trainees' supervisors and managers or other interested parties to design evaluation measures and gather useful data (Phillips, 1997a). Serious consideration should be given to training audits conducted by someone outside the training department as a way of operationalizing and valuing concept of evaluation (Gutek, 1988). 

RESEARCH RECOMMENDATIONS 

Research Recommendation 1: Healthcare organizations conducting more frequent Level 3, 4 and ROI levels of evaluation with their programs should be part of a separate study to determine what supports their efforts. If training evaluation is not mandated by upper level managers, as the Gutek (1988) study suggests, then further research should be conducted to determine how the training director's environment provides reinforcement for the time and resource investment required to demonstrate the value in training evaluation. 

Research Recommendation 2: Evaluation and needs assessments are interrelated. Research conducted in healthcare organizations whose training is driven by formal needs assessment would provide a different perspective from this study where such information was not collected. Needs assessments can be useful in determining that training is the answer to a business challenge. In addition, the needs assessment drives decisions concerning the training content, delivery, and desired outcomes. A training program designed to address a particular need can yield desired outcomes. Training evaluation research conducted in organizations that base training on effective needs assessment may yield different and important results. 

Research Recommendation 3: Research placed in functional areas other than training and at various hierarchical areas of healthcare organizations could provide rich data concerning how other organizational members view training evaluation. Some research using interviews has been conducted from the CEO and senior management perspective. The equivalent of a 360-degree perspective around training evaluation within one organization would invite multiple perspectives in considering the need for and contribution of training evaluation. This could be done in a case-study format. 

Research Recommendation 4: It is possible that a limited understanding of both financial analysis and the language of finance and accounting may contribute to undervaluing financial analysis as a resource to training evaluation. It appears that the financial analysis academic preparation is present to some degree, both by courses required in a business degree and courses often required in educational administration. If the financial analysis skills are not used routinely, however, the skills may not be considered an active tool in evaluation of training/education. Further research is needed to explore academic preparation in both financial analysis and evaluation, especially in the area of education programs.  

Research Recommendation 5: Due to the perceived difficulty of training evaluation, more research is needed to explore how the finance/accounting areas of healthcare organizations are working through the challenges of the intangible assets where traditional accounting for tangible assets may not be effective. More research is needed in areas such as research and development and intellectual capital to provide possible models for evaluation of training and other performance improvement initiatives. 

Research Recommendation 6: Similar research in other types of healthcare organizations is needed. This study may be limited in generalizability to all healthcare organizations. The sampling frame included only organizations with membership in the American Society for Training & Development. Healthcare organizations without training staff, training activities, or training budgets (for professional association membership) would represent examples of organizations that may not have been included in this research. This research is largely confined to responses from hospitals (particularly not-for-profit) and may represent a bias. Furthermore, due to the sample size, this study can only be considered as preliminary to a larger study; conclusions may not generalize to the healthcare industry as a whole.  

CONCLUSION 

While the focus is evaluation, it is important to remember that evaluation is a starting point, not the ending point. Evaluation provides a context for exploration and experimentation by stakeholders in an organization. Training is also a starting point, not an ending point. Training is one of many responses to a business need. What is needed is neither more evaluation nor training, but a thoughtful openness to examining the worth of an activity by the stakeholders in the need, the activity, and the outcome.  

ADDITIONAL RESOURCES  

Several summary tables of data are available to assist you in presenting the data to your organization.  If a Microsoft Powerpoint file would be helpful to you, please send your request to infopbd@aol.com and specify the desired version of the Powerpoint file.  

A list of evaluation literature (articles and books) can be provided to you in Microsoft Word, or Rich Text Format (.rtf). Please send your request to infopbd@aol.com and specify the desired format for the listing of evaluation resources. A copy of this information has been provided with this executive summary. 

The complete dissertation can be ordered from Bell+Howell Information and Learning (formerly UMI) by calling 1-800-521-0600 (or 734-761-4700) or writing to 300 North Zeeb Road, P. O. Box 1346, Ann Arbor, MI 48106-1346. The Website address is www.umi.com.

References 

Argyris, C. & Schon, D. A. (1974). Theory in practice: Increasing professional effectiveness. San Francisco: Jossey-Bass.

Armour, S. (1998, October 7). Big lesson: Billions wasted on job-skills training. USA Today, p. B1.

Barrett, S. (1997). Employees perceptions of quality improvement initiatives in healthcare settings. (Doctoral dissertation, University of Illinois at Chicago, 1997).

Carnevale, A.P., & Schulz E. R. (July 1990). Return on investment: Accounting for training. Training and Development Journal 44(7), S1-S32.

Gutek, S.P. (1988). Training program evaluation: An investigation of perceptions and practices in nonmanufacturing business organizations. (Doctoral dissertation, Western Michigan University, 1988).

Hill, D. & Phillips, J. J. (1997). Preventing sexual harassment. In Phillips, J. J. (Ed.), In Action: Measuring Return on Investment, Vol. 2, pp. 17-35. Alexandria, VA: American Society for Training and Development.

McGough, D. R. (1998). Evaluation and support of training by senior executives. (Doctoral dissertation, The Union Institute Graduate School, 1998).

Mosier, N.R. (1990, Spring). Financial analysis: The methods and their application to employee training. Human Resource Development Quarterly, 1(1), 45-63.

Palladoro, V. A. (1997). An investigation of continuous improvement in healthcare: Is there an evaluation strategy? (Doctoral dissertation, University of Rochester, 1997).

Phillips, J. J. (Ed.). (1997a). In action: Measuring return on investment, Vol. 2. Alexandria, VA: American Society for Training & Development.

Phillips, J. J. (1997b). Return on investment in training and performance improvement programs. Houston: Gulf.

Phillips, J. J. (1997c). Handbook of training evaluation. Houston: Gulf.

Stewart, T. A. (1997). Intellectual capital: The new wealth of organizations. New York: Doubleday/Currency.

Twitchell, S. (1997) Technical training program evaluation: present practices in United States’ business and industry. (Doctoral dissertation, Louisiana State University and Agricultural and Mechanical College, 1997).

Ulrich, D. (1998 January-February). A new mandate for human resources. Harvard Business Review, 76(1), 124-134.