August 2001

Evaluating by Comparisons: Watch Out for the Traps!

By Susan J. Ellis
Subscribe: |

A few months ago, I received the following question in my e-mail:

We have just completed our year end report for 2000 and I am looking for information which would help us better evaluate our volunteer program. Can you help me find information on number/percentage "norms" for: 1) number of people who contact us about volunteering, compared to number who actually attend an introductory meeting; 2) number of those who attend an introductory meeting and actually become volunteers; 3) yearly rate of increase in volunteer numbers; and 4) yearly attrition rate for a volunteer program.

This was not the first time I've been asked about an external "standard" against which a specific agency can measure the effectiveness of its volunteer effort. Here is how I answered:

Your question - while certainly seeking useful information - unfortunately presumes a far better database of statistics about volunteering than exists! I do not know of any national database that would help you. On the other hand, I think you are looking for 'generic' comparative data, which may not really be what you need.

The variety of things that volunteers do is so enormous that it would be hard to create measurements relevant to all. For example, in the paying work world, would you think of comparing attrition/retention rates of nurses to, say, ditch diggers, or to astronauts? Yes, there are Labor Department data, but no one tries to find a 'standard' answer for every category of employees.

Also, things like whether or not the program 'increased' may not measure effectiveness! I know of many programs that would be more successful if they cut their volunteer corps in half and retained only those volunteers qualified to help! Further, 'retention' is measured not by longevity but by how long each volunteer told you s/he was going to stay.

So here are my suggestions:

1. Stop looking for outside validation of your program. Instead, articulate reasonable, measurable goals and objectives for the work you need volunteers to do. For example, rather than some arbitrary, 'we want to grow 15% next year,' how about: 'Next year we'd like to have enough qualified volunteers to respond to 90% of requests made of the department.' See the difference?

2. If you do feel you want to compare yourself to others, then focus on other organizations similar to yours, perhaps through a professional society for your field or setting. Then contact those types of organizations and see if you can get their statistics as a comparison to yours.

3. Perhaps you would be better off comparing internal data from one year to the next. So, if 60% of volunteers remained committed after training in 1998, what's the percentage in 2000 and why?"

As I sent this response, I knew that I wanted to re-visit this in a Hot Topic at some point. It sounds so reasonable for someone to ask, "what happens elsewhere?" Sometimes this makes sense, but not when the question being studied is fundamental to the provision of services integral to a specific setting. Some more things to consider are:

  • Did you begin a period by articulating what you want volunteers to accomplish? Too many evaluations start at the end, which is too late. It's only possible to analyze data against initial desired goals. You may discover you doubled the volunteer workforce this year, but did you need to triple it?
     
  • As important as collecting data is interpreting it correctly. For example, if you discover that 50% of volunteers drop off after 6 months, you need to look for clues as to why. Asking what the "norm" is for volunteer retention in other settings is irrelevant. In fact, it focuses the attention (blame) on volunteers - trying to find a cause inherent in them - rather than considering what is going on in your setting.
     
  • After you have assessed your own situation, some things might, in fact, be useful to know about trends or issues faced by others - at least those who are dealing in a similar context. Some examples: Are others finding that high school students are most likely to cite transportation problems as a reason they can't volunteer? How are others dealing with volunteers who are over age 85 and perhaps losing some abilities? These types of questions allow you to distinguish what is and isn't the result of your managerial actions. But no external data is going to mean anything in deciding whether the attrition rates of volunteers in YOUR organization are acceptable or not.

The only other thing I want to say about comparisons is the old problem of colleagues asking: "What do other volunteer program managers earn as a salary?" This is a great object lesson in the fallacy of general comparisons. If we look at the volunteer management field as a whole, salaries range from maybe $12,000 to $80,000. So what? Maybe (but only maybe) it might be valid to see what volunteer program managers in similar settings earn, so that at least you aren't equating the organizational staff budgets of a huge health care system with a one-room rape crisis center. But isn't the REAL question: "How does my salary compare to the other people on staff in MY setting who work at the department head level?"

If we can see this problem with comparisons on salary, it ought to be clear why a similar, internal approach is needed in assessing the accomplishments of volunteers.

What has been your experience? Are there external comparisons that you found useful?

Responses from Readers

Posted 28Aug01
Submitted by Denni Wentzell, Coordinator of Volunteer Services with Community Information & Referral Society, Alberta Canada
I am the Coordinator of Volunteer Services for the city of Red Deer, Alberta Canada. In my position, I work closely with many voluntary organizations in the city by chairing a monthly meeting for volunteer managers to share information, ideas and resources. We develop committees to do the actual hands-on-work for events related to National Volunteer Week. One of the many conflicts we continually face is the workload for each manager as they continue to work in their paid position with their respective organizations. Many comments are received regularly stating that much time is spent on things aside from their regular jobs. This is true; however, NOT as a volunteer - these duties always fall within working hours for which each manager is being paid by the organization. My thought is that when we are asking others to volunteer their time, we, as managers, must be willing to set the example by volunteering our time - either for our organization or for another organization in the community. But, as with this situation regarding wage/salary comparisons, the primary concern is the wage to be earned. I am finding that many potential funders are not pleased to see so much of the money raised for the organization to be going to pay an individual to oversee community volunteers.

Posted 08Aug01
Submitted by Lisa Coble, Manager of Volunteer Services, Newport Hospital, Newport, RI USA
What an excellent "hot topic" this month. Susan had some excellent thoughts on the subject of evaluation and measurement. This year I've documented statistics on number of inquiries resulting in placements, number of volunteers completing their minimum 3 month commitment and number of hours served by month. Now I will at least have a base line of information for comparison from year to year. I think having an Exit Survey process is important to collect information on why people are leaving their volunteer position. Any evaluations that can be done to capture "what's working" and "what needs fixing" from volunteers and staff is what you need to make improvements.

Posted 08Aug01
Submitted by Laurie Eytel, Volunteer Manager, Prevention Services, Fairfax County Dept. of Family Services, VA, USA
I like the suggestion about not measuring volunteer "success" by looking at % increases. However, how does one go about measuring success then? For example, "enough qualified volunteers respond to 90% of the requests made by the department" - does this mean if you put out 100 requests and fill them with 90 volunteers you are then successful (assuming they are qualified?) Or would retention rates be better? I am trying to establish a new & improved method for measuring the success of our programs (social service volunteer opportunities.)

Posted 06Aug01
Submitted by Colette Mandin-Kossowan, Volunteer Program Coordinator, The Support Network, Edmonton, Alberta, Canada
How timely for this discussion to come up now...right when I am posing these questions about our agency's volunteer program. I agree with all three suggestions. Measuring numbers of volunteers is about inputs and outputs. Numbers, though, may not be the only factor in the actual outcome of the program and the benefits to our clients. I also agree that comparing unlike programs is an inefficient way to evaluate your own. I work in a suicide prevention agency with a crisis line. It would be ludicrous for me to compare my volunteer program with that of a child care facility. Thank your for this topic, and the thoughts on volunteer manager salaries.

Posted 06Aug01
Submitted by Louise Shivers, Coordinator, Volunteer Services at Monmouth Medical Center, New Jersey, USA
You are so right! We had a vice-president who insisted we up our volunteer force by 20% during a particular year and I found myself looking for "warm bodies" instead of qualified volunteers. I am suffering now with inappropriate volunteers and he is long gone! I also agree with the comparison between "like" organizations. I coordinate a program at a 526-bed hospital and in our system there are 11 hospitals. All of the volunteer managers in the system get together and compare all sorts of info, but it works because we all have very similar programs.

Posted 06Aug01
Submitted by Judy Rust, Director of Volunteers at Minnetrista Cultural Center, Muncie, In
I agree with your statements about outside comparisons for volunteer programs. But I do think that it is good to look to national figures for general comparison. I did this and found that as the "national average hours per week" declined ours did also. At the same time, the national figures on number of people volunteering increased as did hours. All this leads me to believe that more people are volunteering for shorter amounts of time. For this reason, we need to do more focusing on the short term volunteer and their needs. These revelations meant a lot to our Board members. On the other hand, comparing us to another organization was not possible. As with people, no two organizations are alike.

Posted 03Aug01
Submitted by Joan Cardellino, Director, Volunteer Resources, Abington, Pennsylvania
This is one of my favorite "hot topics". My salary grade is determined (and has been for the past thirty years) by surveys of similar positions/job duties OUTSIDE of my institution such as "Hay" salary studies. From my own personal experiences, the figures that come back on volunteer director positions are not pretty and force my salary (no matter how extensive or meaningful the range of my duties are) into the lowest upper level management pay scale in the nonprofit industry. I feel it is unrealistic for volunteer directors to expect the upper level nonprofit management to be grateful for all we do and then pay us accordingly. That has never happened to me no matter how many times I explain the variety of wonderful things I do for my individual organization. Upper management's typical response to me has been "Show us the money/results/impact on organization goals in upper management language" not "volunteerese." I've actually begun to appreciate the wisdom of that response. Volunteer directors are perceived and paid within the industry as secretaries/administrative assistants. I can't help but wonder if this perception isn't actually based in reality when I examine the uneven range of volunteer directors' training, education and job duties in relation to standards for most other nonprofit upper level management positions. The fact that volunteer directors embrace such a such a diverse list of duties/titles under the volunteer department is in reality hurting us, not helping us. I feel we must we crack the upper management salary ceiling by reinforcing a standard, upper management level set of Volunteer Director job duties/training/education across the non-profit industry. {Editor's Note: For a discussion on salaries, see the March 1999 hot topic.}

Posted 02Aug01
Submitted by Cheryl Morehouse, Manager, Volunteer, Community & Senior Services, Saint Joseph Hospital, Nebraska
I agree whole-heartedly with the suggestions you offered to the initial e-mail question, in particular suggestion number 2. I am very fortunate to have an EXCELLENT group of peers in my field who have been a wonderful resource and support. We meet on a monthly basis and usually have a roundtable discussion about issues pertaining to our field.

Your suggestion number 3 is also a great practice to follow, because no matter how similar your volunteer program may be to others in your city, state, or nation, there will always be unique circumstances and situations particular exclusively to your agency, facility or program. "Amen" to your comment and observations that "increase" or "numbers" doesn't always measure success in a volunteer program. I would rather stay "lean and mean" with accurately placed and skilled volunteers than deal with the unpleasant aftermath of bored, unfulfilled, or difficult volunteers accumulated merely for the sake of numbers!

Add Your Comment