The evidence-based approach of our model requires a stringent program evaluation for the purposes of quality management, research for future program development, and accountability towards our donors. With program evaluation, we plan to research the short and long term influences of our program on the  participants' professional development, as well as on their experience of empowerment and organizational/community involvement. At the same time, our research will track changes in the client base and general situations of the regions in which we work, in order to adapt our programs accordingly. The leadership program should be relevant for short and long term goals of organizations, communities and individuals, and specifically applicable in work AND everyday life situations. For this purpose, we define core competencies that we have determined to be essential for applying leadership, facilitation and coaching skills. These competencies are to be used as criteria by which to measure the pedagogic effectiveness of our programs. We have chosen the most accepted leadership development evaluation methods, but expanded upon them with our own research approach. All components, including defined competencies and evaluative methods, are under our ongoing scrutiny and in the process of continuous, updated improvement based upon feedback from the process of research, evaluation and training, as the programs unfold.

Methodological background

The evaluation of the Process Oriented Leadership Development Program will include what we consider the two most established and respected training evaluation models, Kirkpatrick’s (Kirkpatrick 1994) Four Level Model and Phillips (Phillips & Phillips 2005) ROI Model. Data that is generated from Kirkpatrick’s model will provide general, essential information about the quality of learning in our program, based on direct participant feedback. Questions on each of the four levels address essential aspects of any training program, and will help us to gauge our progress based on our own standards as well as on those of other programs. General questions enable participants to provide personal feedback data. Data will be fed back into the system to provide categories and directions for adjusting training strategies.
The ROI model will help us to determine categories for evaluation other than financial gain, or directly measurable outcomes. This is essential for our program, as many of the benefits of training manifest in changed attitudes and perceptions that lead to changed actions, which finally bring greater results. However, the most direct measurement of these benefits is through the evaluation of these initial subjective qualities. This model considers alternate methods for measuring such essential qualities.

Kirkpatrick’s Four Level Model

This model proposes evaluation of four levels. Our evaluations will address questions that correspond to each of these levels:

1.    Participant reaction: How did program participants feel about the training content and setting?

2.    Participant learning: To what degree were the participants able to achieve the skill, knowledge and attitude objectives of the program?

3.    Participant behaviour: Transfer of learning: To what extent were participants able to transfer the learned leadership skills, attitudes and mind-sets into professional and private life settings during and at the completion of the leadership program?

4.    Results: Did the skills and mindsets taught in the program result in an increase in participants’ personal access to socio-economic resources? Have UN millennium development goals been impacted?

Phillips ROI Model

The process of identifying and defining training benefits is complex, and does not always relate directly to identifying monetary benefits. Although financial benefits may be a secondary result of identified intangibles, this causal relationship is difficult to prove and relatively insignificant. Phillips (2006) recommends a financial evaluation step for only 5 to 10 percent of all programs. However, intangibles, those aspects that are difficult to measure monetarily but significant components of a well functioning workplace, are one of six types of data named in the ROI methodology, described as follows: “Intangibles are those measures that are not converted to monetary values, and usually include other hard-to-value measures. If these measures cannot be converted to money credibly and with minimum resources, they are identified as intangibles. Intangibles are very important because they represent the human dynamics elements in the work environment, such as commitment to organizational goals, teamwork, and communication. That they cannot be feasibly or credibly converted to monetary values does not undermine their significance in the workplace.” (Phillips & Phillips 2006, p. 160).
We have defined the following intangibles as significant standards of measurement for the Process Oriented Leadership Development Program: centeredness, coherence, ability to support various belief systems, ability to manifest creativity in the world, and conflict awareness.

Our Own Research on Evaluation Methods

We are also working on our own evaluation methods, in conjunction with European and U.S. colleagues, measuring parameters such as organizational field awareness, presence of meta-communicator, ability to maintain an overview, and eldership.  We will continue to bring into operation other aspects of the multi-dimensional leadership core competencies based on results and onsite research. Please visit us again at this website for current updates on this front!

References

Behrens, T., & Benham, M. (2006). Evaluating Community Leadership Programs. In Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.

Kirkpatrick, D.L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.

Phillips J. (2003). Return on Investment in Training and Performance Improvement Programs (2nd ed.). Woburn, Mass.: Butterworth-Heinemann.

Phillips J. J. & Phillips P. (2006). Measuring Return on Investment in Leadership Development.  In: Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.

This is an interesting application of program evaluation in practice that is governed by the cited theory. It gives a good comparison of the Kirkpatrick and Phillips models and utilizes them together to achieve evaluation goals.

dear friend,
thanks so much for your note and encouragement, and for taking the time to study our evaluation and to share your expertise, we so appreciate it, and especially now as we are in the midst of further developing our evaluation methods, it's a big gust of wind on our backs. Thanks again so much for this!
with all the best to you,
the ddi team

The application of the program evalution seems like it would be very effective. It would be a great may to make sure that the program is always making improvements when needed.

we so appreciate your interest with ours in making our programs as effective as possible, we love to improve whatever we can and in this spirit are also working on our evaluation methods. Your interaction and support means a lot, thanks so much, from thd DDI team!