Like Bugs Drawn to a Bug Zapper - The Attraction of Unstructured Comments on Employee Surveys...

Capitalist Note: I'm at the Aberdeen Human Capital Management Summit in Manhattan this week, where I'm getting my professional development game on, as well as speaking and participating in a panel.  Check out the hashtag for the conference at #hcm2010nyc...

Employee Surveys:  You know them, you love them.  You gotta do them.  What's the one piece of the employee survey your executives turn immediately to upon receipt?

Wait for it...

No really...Wait for it...

That's right, the comments section.  You collect reams of data and if you're good and committed to the survey craft, you've got years of trending data on specific questions to check the pulse of your workforce.  Good work, quant girl/boy.  I'm sure it's pretty satisfying when your CFO turns first to the comments section and blurts out, "The marketing team is taking 3 hour lunches - says so right here (referring to an anonymous rant that's a half page long).  We need to fix that"...

Sigh... And so it goes with the employee survey.

Why is this on my mind?  I was taking in the first speaker at the Aberdeen Human Capital Management Summit in Manhattan this week, guy named Steve Church who serves as the Chief Operational Excellence Officer for Avnet.  One of Steve's first comments was this (paraphrased):

"you learn more from the comments of the employee survey than from the data, because that's where the emotion is."

No doubt that's true.  There's just one problem with that reality - lots of managers aren't capable of putting the employee comments in context.  When reading the employee comments, three things can happen, two of which are bad.  Let's break them down:

1.  Managers read the comments, mix it with the trends of the data and naturally make solid assumptions. 

2.  Managers read the comments, are immediately put off by the rants from 15% of your employee base, and tag the whole process as being overrun by jaded employees.  You never really get them back... especially if they are named...(yikes)

3.  Managers read the comments and form emotional calls for action on items that really aren't germane (that's right, I'm using the word GERMANE, look it up...).  See the CFO call for action on marketing lunches above.

So, I'm giving you a free best practice here.  Next time you distribute employee survey results to your management team, have two meetings - one to cover the data results and trends.  That one comes first, because you want them to trend the results and form their own top line conclusions based on what they see.  Don't give in to the call for comments, because you need their focus on data first.

Once you've got that in the bag, distribute the comments, allow the managers to soak on them, then get together to talk about how, if at all, the managers' view of the data trends changed based on access to the comments. 

Do this and thank me later.  You'll have a much more meaningful conversation, and you'll be less likely to focus an hour of your time on stuff that doesn't really matter in the big scheme of things.  You're after the big things when it comes to your survey...

Comments

[Cathy Martin]

Well as a self proclaimed employee survey queen, I think I may have to disagree on the issue of open ended comments on surveys. Depending on the question, and it usually is something about "how would you improve things around here", you will need to take a quantitative approach to these comments just as you would the top box (percentage favorables). By conducting a theme analysis you can assign comments to "buckets" and do a frequency analysis therefore minimizing, the comments that are extraneous. (outliers). We have found that many of the comments go a long way to explain the numbers in the survey itself, if the comment questions are structured and timed appropriately. We rarely do a survey without open ended questions. It is a timely analysis but well worth it.

[Rob Orr]

As a guy involved in doing employee surveys internally for more that 20 years, I agree with Kris... my first-hand experience mirrors his, except that I have never seen outcome 1.
Yes the "verbatim comments are important to gain insight into the quantitative data, but I think Kris's approach makes a lot of sense - 2 meetings the first of which deals with the data... I might even suggest that the comments be distributed in the 3rd meeting. The 2nd meeting should present the data again, this time to a group of managers the next level down who have been given the charge to address the situation represented by the data. The 3rd meeting shows the qualitative stuff, but the marching orders will have already be given. This group will need the insights of the comments to focus their efforts.

[Eve Stranz]

I'm happy to say that while I've seen a ranting comment in every survey that included comments verbatim, I've always had some quant analysis or grouping of comments as part of the deal.

I'd like to offer a few other best practice alternatives:

* Skip the open ended questions. Facilitated focus groups can give you balanced and deeper context to the data. This is especially helpful when dealing with large employee populations (where you would go cross-eyed before you could finish reading all of the comments.) Of course, you need to apply quant analysis to the input you get in the focus group.

* Regardless whether you have open ended questions or not, have a team of employees recommend action plans to management, and make management wait to hear what action plans the team recommends before implementing any sweeping action plans of their own. You need to get the right mix of employees on the team and you may need to have someone guide them through the process the first time, but it can be done fairly painlessly.

[mike cook]

Kris, sounds like you hit a bit of a nerve here, lots of passion. I confess to being pretty mixed on surveys, mainly because I don't always trust the motives of senior managers(my problem). What I have generally found with both data and comments is that managers like data that agrees with them, challenge numbers that don't, grab on to comments that prick a matter of personal(rarely business) interest and generally wind up sweeping the whole event under the rug because they are just confused by anyone who does not think the way they do or requires anything other than what the paternalistic view would deem "wantable."

That being said (whew!) I like your recommended approach but am as interested in hearing what you may have to say about having HR types see this survey process as an opportunity to inform, influence and lead, no matter what the data.

MFC

[David Bowles Ph.D.]

Kris, this is very interesting and I have seen all three of those reactions. First though I would say that your Avnet guy is totally wrong is saying that you learn more from comments than the data: that's a terrible mistake to make. In my experience the comments are completed by those who are more negative. Maybe its just human nature that people who are content don't need to vent (is that like if the glove don't fit....etc?) Secondly the comments are not completed by as many as take the survey; would you rather generalize from a 90% sample which did the survey or the 50% (or less) who completed the comments? As for "that's where the emotion is, how would your Avnet guy explain some of the quantitative data I have collected from disgruntled people over the years...does he think a 90%+ negative rating of a boss isn't "emotion"...??

In think having a comments section is at least 50% process vs. content: its just a much more "open" survey that way. Having it there also gives people a chance to say what might not be in the questions, and yes if they want to vent and feel safe venting, fine. So tricky to manage whether people get identified though, and that's the big big downside.

Enjoy your posts...thanks.

David