At Langevin, we teach a 3-day workshop called Evaluation of Training where we lay out a user-friendly plan of attack to conduct all four levels of evaluation. I find that most trainers are surprised at how easy it is to measure the effectiveness of their programs, once they understand what I refer to as “PDA”! And, by the way, this PDA doesn’t mean “personal digital assistant” or “public display of affection!”

PDA is an acronym that represents three major phases of the evaluation process. Let’s take a closer look:

P is for Planning

Recently I saw a quote by HRD Blogger Patti Phillips that read, “Plan your work, work your plan.” In the evaluation process, the more time you spend in the planning phase, the simpler the execution. The planning phase, if done correctly, will make the rest of the process less daunting. In my opinion, the planning phase offers the most potential for improvement in the entire evaluation process. When planning an evaluation remember to:

• Determine the purpose of the evaluation (revise an existing course or secure more funding?).
• Determine expected and baseline performance.
• Determine the timing of the evaluation.

D is for Design

Design instruments, that is. In the evaluation process, “Instrument” is another word for data collection method. We teach a total of 27 instruments for all four levels in our Evaluation of Training workshop. Once you’ve determined the levels at which your training program will be evaluated, the instruments can be identified. Instruments can be traditional or not-so-traditional in nature. A few examples of evaluation instruments include: surveys, observations, focus groups, and tests/work samples.

A is for Analyze

Once the instruments have been designed and implemented, the analysis begins. Without proper analysis, it’s tough to explain the results. I refer to this phase as “the heart of evaluation.” This is the phase where you analyze tangible and intangible data and calculate an ROI if applicable. What you do, and when you do, it are dependent on the purpose of the evaluation, the type of data, time to analyze, cost, your corporate culture, and practicality.

PDA is an abbreviated version of the evaluation process we teach in our 3-day Evaluation of Training workshop. Hopefully this brief synopsis is enough to get you thinking about it. Evaluation is a must if you want to know whether or not your training programs are successful!

I’d love to hear how you are using measurement & evaluation in your organization.

Melissa

4:31

What do the surveys (aka smiley sheets, happy sheets, level 1 evaluations) really mean in the world of training – to the participants, the course leader, and the course leader’s supervisor?

The end-of-course survey is an important document, used by almost all training departments, but it isn’t scientific evidence of the effectiveness of the event. It is essentially a customer satisfaction survey – often filled out at the end of a long training session where the main concern of the learner is to get out of the room. The survey may ask about the learners’ reaction to the course leader, content, and/or facilities. While it is commonly done in hard copy at, or near the end of, the training session, some organizations are moving to an e-solution through their LMS or a third party solution such as Zoomerang.com or surveymonkey.com. Organizations (Langevin included) often quantify the results down to the hundredths of a percent. However, what’s being quantified is someone’s opinion which, by its nature, must be very subjective.

What does it mean to the participants?

For most, it’s just a chance to say “well done!” Most people are receptive to training either as time away from the workplace with pay or as something that genuinely makes their lives easier somehow. They appreciate the efforts of the instructor. Of course, by the same token, if any participants have some residual resistance to the training, it is a chance to voice that resistance. In either case, the overall scores of the evaluation tend to reflect those feelings (the “halo effect”). For example, if the learners like the content, they like the way the course leader conducted the course. If they didn’t like the content, the course leader may take a hit. Additionally, the end of course survey really does give the person who feels strongly about something – one way or another – an opportunity to be heard.

What does it mean for the course leader?

I call it a chance to polish the diamond. Most course leaders’ skills are solid. So, what the end-of-course survey provides is a way to make their skills even better. This is not to say that when the course leader considers the suggestions made on the evaluation sheet, he or she should blindly incorporate everything said on the sheet. Sometimes the suggestions involve things that are a characteristic of the design of the training or of a classroom constraint. For example, a participant may complain about “too much practice,” yet if the design of the training is highly interactive and a high degree of proficiency is required as they leave the room, there needs to be a lot of practice in the training. However, some suggestions provide an easy way to improve. For example, one of my learners commented that he couldn’t tell if I had good eye contact or not as the light reflected off the lenses of my glasses. I found that I could do fine without the glasses. Today, I don’t wear them in the classroom – Polishing the diamond!

Here’s a tip for instructors. Many instructors enter the training field because they are high performers – on the job and in the classroom. As such, they hold themselves to a very high standard. That’s great, but when adverse comments are made on an end-of-course survey, there is a tendency to take those comments personally. It is especially a problem when the instructor is exhausted physically and psychologically – like immediately after finishing a class. A suggestion given to me by one of my early Langevin coaches was to put off reviewing the end of course surveys for a couple of days. It’s a lot easier to be objective with a little “distance” from the training.

What does it mean for the supervisor?

One tool a course leader supervisor has for instructor development is the end-of-course survey. An occasional critical comment by a participant may mean little. People come into the classroom with the baggage of the day. If they are having a bad day outside of the classroom, the end-of-course survey may just as easily reflect that day as what occurred in the classroom. While critical comments should be followed up on with both the instructor and, as a customer service issue, with the learner, they are seldom a make-or-break issue for the course leader. What the supervisor looks for is trends.  When overall ratings drop over a period of time or when a number of learners make the same comment, there is a problem with the facility, or the training design, and/or the delivery.

The end-of-course survey is one instrument to gauge the success of training. Important? You bet. Especially in this economy, listening to the voice of the customer is a key to survival. While there are other ways to see how effective the training is, the end of course survey is the most common way of listening to the voice of the customer, your learners.

Paul

Another January, another new year, another decade, all of which means only one thing: It’s time to make a new years’ resolution. Yay!

We’re all too familiar with January, a time for a new start for many people where the goal becomes getting healthier and maybe even losing a few unwanted pounds gained during the holiday season. Health clubs, gyms, and YMCA’s look forward to January each year because new memberships spike in response to everyone’s longing to achieve their goals. Unfortunately, come March or earlier, at the first scent of warmer weather, going to the gym becomes as stale as those holiday cookies buried in the cupboard.

Well, do not get discouraged because I have a better alternative this January, and it is all about gaining – and I do not mean weight! You see, having been with Langevin for over 10 years, I can easily say that I have never had one class where there was someone who  did not struggle with gaining management support and buy-in for their training. So, as we start this New Year, I have some simple, yet effective, steps you can take to help you gain management support and buy-in for your training programs – and they are guaranteed not to go stale on you in a few months!

1. Be visible in your organization.

People do not tend to work freely with, or support, people they don’t know. Additionally, if you are not visible, then you are easy to forget. Attend key management meetings, go to the company picnic, pop your head into the VP’s informal birthday party for a small slice of cake (but remember our new years’ resolution!) Make a weekly plan of key activities you will attend, so that you are visible to the organization and making connections with your colleagues.

2. Always maintain a positive attitude and a smile.

People will also not tend to work freely with someone who is grumpy.

3. Train yourself, and your staff on how to conduct ROI’s and Level 3 and 4 evaluations.

If you really want management support and buy-in, you must show the WIIFM (What’s in it for me) or benefit to the company of how your training impacts the bottom-line. Remember this powerful phrase: If you cannot measure it, then you should not train it. Use metrics in your training. Unsure of how to do this?  The good news is that we offer several programs to help you over this hurdle. Now we are ready for the final step.

4. Add to that weekly plan by allotting some time to spend with other departments.

You can have lunch with management, job shadow, or even try doing a task for someone else here or there. I suggest doing these activities because, not only will they continue to give you visibility; they will provide you with the skills that you need to speak the language of each department and the organization. Becoming a strategic partner, where you help solve job performance problems, rather than a program peddler, will create management support and buy-in for your training.

Our mission at Langevin is to provide trainers with the skills, knowledge, and materials you need to succeed in your careers – powerful tools that can help you gain ground with management for your training programs this New Year.

Happy New Year!!!

Ron

Phase I:  Planning & Analysis

E-learning, like any training strategy, takes time and money to build. You either build it yourself or you hire a consulting firm to build it for you. Before you spend money on any e-learning project, it is wise to do some advance research. If you understand, and apply, the planning and analysis steps in instructional design, your e-learning will:

• Meet a legitimate business need.
• Stay on time and on budget.
• Include content targeted appropriately for your audience.
• Contain only relevant content for your learners.
• Produce desired performance outcomes.

Phase II: Design & Development

Depending on your learning goals and circumstances, e-learning can be more cost-effective than traditional classroom methods. How do you guarantee that e-learning will save money and meet your learning objectives? Start with good design. If you understand, and apply, formal design and development steps, your e-learning will:

• Be the appropriate strategy to use to teach your content.
• Contain assessment tests to measure knowledge, skill, and retention.
• Be structured properly to maximize learning success.
• Include feedback mechanisms and any additional job aids.

Phase III: Validation & Evaluation

e-Learning is no different from other methods of training in that it needs to be validated and evaluated. Does the e-learning course need any final revisions before implementation? Are the employees performing their jobs effectively after completing the e-learning course? If you understand, and apply, formal design validation and evaluation steps, your e-learning will:

• Incorporate Adult Learning Principles in its design.
• Be targeted to the appropriate experience level of your learner population.
• Run properly without any typographical or grammatical errors.
• Enable you to evaluate your training effectiveness.

So pull that design manual off the shelf or take a workshop to learn the basics of design and then enjoy playing with your new e-learning toys!

Lynne

So what can you do to convince your company that the hours employees spend in training and the dollars it invests in its people are time and money well spent? Start by evaluating your training programs! I’m not talking about the Level 1 “smile sheets” you hand out at the end of your trainings. I’m talking about the deeper levels of evaluation that increase your department’s credibility and justify your worth to the “powers that be.” These evaluation tools can save your department.

The Kirkpatrick Model gives us 4 levels at which to evaluate the effectiveness of our training programs. This article will explore levels 3 and 4; what they are and when to use them. In my next post, I will outline the “how-to’s” (guidelines for designing evaluation instruments) for both levels.

Level 3:

What happens when employees leave the classroom and return to their jobs? How much of what they learned in training actually transfers to the job site? In other words, what change in job behavior occurred because people attended a training program? This is what a level 3 evaluation attempts to evaluate. One of the more common ways to conduct a level 3 evaluation is observation. An observation involves either a Manager or Trainer observing employees perform the tasks (just trained on) according to standard or performance checklist (a performance checklist ensures objectivity on behalf of the observer). The challenge with this method is getting management’s support and commitment to implement it or do the observation themselves. Many managers feel it’s disruptive and/or don’t have time to observe their employees. Creating a partnership with managers and communicating the benefits (of employees performing to standard) is crucial to getting buy-in and cooperation.

Level 4:

This is the most important and perhaps the most difficult of all the levels – what final results occurred because the employees attended a training program. Results can include increased production, improved quality, decreased costs, reduced frequency and/or severity of accidents, or increased sales. Consider the following 5 guidelines when conducting a level 4 evaluation.

Assemble a control group if practical (a control group refers to a group of employees that does not receive the training. The group that receives the training is an experimental group). Any difference between the control group and the experimental group can be explained by the results achieved because of the training program. I liken it to using a placebo group in drug trials/testing.

1. Allow time for results to be achieved. Time has to elapse before the evaluation can be done. Each situation is different, so timing depends on the skills taught in training.
2. Measure both before and after the program, if practical. Fairly easy to do at this level, as records are usually available to determine the situation before the training program took place.
3. Repeat the measurement at appropriate times.  Each organization must decide how often and when to evaluate.
4. Consider cost vs. benefits. Generally, it isn’t nearly as costly to evaluate at this level as it is other levels.

Implementing a level 4 evaluation provides the greatest challenge to training professionals as some training programs show tangible results, some don’t.

In summary, most of us are concerned with our department’s and our own credibility. We want to be accepted, trusted, and respected by senior executives. When we’re accepted and respected, lots of wonderful things happen – not to mention you sleep better, worry less, and enjoy life more. The best way to earn this trust and respect is by evaluating and reporting the worth of our training – now more than ever! Be sure to watch for my next posting on how to design traditional and alternative evaluation instruments.

Melissa

Here are some tips that have served me well:

Be Fair and Stay Objective

I’ve found that the first step to dealing with a negative course evaluation is to get comfortable with the fact that everyone is entitled to their perspective and opinion and to view any experience through their own set of glasses.  After all, each evaluation is, in essence, an opinion.  In the end, it boils down to whether you can honestly answer the following questions with a solid “YES.”

Did I…

• …follow the lesson plan for the course?
• …do everything possible to address that person’s needs during the course?
• ….do my best in establishing a healthy learning climate?
• …receive a substantial number of positive evaluations from other attendees?
• …make sure everyone felt included and supported by me during the course?
• …create a fun healthy environment for all?
• …clearly show the course benefits to each attendee?
• …show learners how the training directly related to the day-to-day job?
• …bank on each participant’s experiences to add value to the course?
• …spend some one-on-one time with each participant?

Allowing yourself to sulk over one bad evaluation when all the other ones are stellar is not the way to deal with the issue.  Rather, read the positive evaluations in detail before you go back to reviewing the “bad” one. It will give you a fair perspective with which to accept any constructive criticism offered in the latter.  After all, we ALL have the chance to learn something new from time to time.

Coping and Moving Forward

You may, however, have answered, “NO” or “Not to the extent that we usually do” to any of the questions listed above.  One of the most rewarding experiences that I have had as a Langevin Course Leader is that every time I teach a course, I am given the opportunity to learn as much as my participants – and maybe even more!  As part of our course summary process, we are responsible for completing and submitting a document, along with the individual participant evaluations, to our supervisor.  In that document there are two questions that I find are key to assessing both positive and negative evaluations:  “What will I do differently the next time I teach this course?” and “What was one new thing I learned about how to train people this week?”

I find that it automatically makes it easier for me to be objective about any comment made on an evaluation when I take the time to explore these two questions and I give myself permission to be human rather than “the perfect trainer.”  This helps me accept that there is always something to learn from every experience and some way to improve what I have done.

I would be remiss if I did not also share with you that I have had to learn how to take advantage of a secret and very powerful weapon when I have to deal with tough evaluations:  my boss.  My supervisor plays a key role in helping me stay objective and consider the evaluation comments.  She is essential in my mentoring process and helps me discover what I have learned from the experience and how to make the best of it.  Someone like a supervisor, mentor, or a trusted team member is able to offer valuable perspective, as our opinions of ourselves are often askew – if not too harsh – when it comes to dealing with comments on an evaluation.

But enough about what I think…now it’s your turn.  You need to give me a hand here and send in your comments to this blog.  Tells us about how you deal with the impact of rough or negative evaluations.  Feel free to also post questions related to the topic or to recommend any favorite columns, articles, or books that may help.  I look forward to hearing from you.

José

Soft Skills

When I researched the term “soft skills” I discovered it no longer means what it once did.  Years ago it was assumed that “soft skills” referred to skills that were more or less intangible – person-to person.  However, the current definition in Wikipedia states: “Soft skills” is a sociological term for a person’s EQ (Emotional Intelligence Quotient), which refers to the cluster of personality traits, social graces, communication, language, personal habits, friendliness, and optimism that mark us. Soft skills complement hard skills (part of a person’s IQ), which are the technical requirements of a job.

Is this what most trainers think of when they hear this term?  Should trainers continue to use a term that does not adequately describe the training taking place?  Perhaps calling them “interpersonal skills” is more accurate.  Wikipedia provides the following definition: “Interpersonal skills” refers to mental and communicative algorithms applied during social communications and interactions in order to reach certain effects or results.  The term “interpersonal skills” is used often in business contexts to refer to the measure of a person’s ability to operate within business organizations through social communication and interactions.  Interpersonal skills are how people relate to one another.

Smiley Sheets

Another term that may minimize the validity of our training is “smiley sheet.”  The term suggests a light, happy, or fluffy evaluation of a course.  When we use the term “smiley sheet” do we unconsciously devalue the importance and effectiveness of a “Level One Eval?”  The information gained in these evaluations shows us the areas for improvement that will make the courses more meaningful for our clients and ultimately increase their satisfaction.  Perhaps we should be using the more accurate and descriptive term like “Level One Evaluation” or “Course Evaluation” rather than “Smiley Sheet.”

As trainers, we sell ideas to our learners in the classroom as well to those in management who fund our training initiatives.  Shouldn’t we be aware of what these terms are saying about the quality and effectiveness of training? Can you think of any other words that minimize our effectiveness and the validity of our training?  Let me know.

Langevin Team