A Proposal for Evaluating the Reference Service at the Hamilton Public Library

A Proposal for Evaluating Reference Services at the Hamilton Public Library
By Marcel H. Faulkner
ABSTRACT: This research proposal describes a bipartite plan for evaluating the reference service at the Central Branch of the Hamilton Public Library (HPL). The proposal contains a review of pertinent studies of reference service evaluation, as well as a detailed methodology for evaluating reference staff knowledge and behaviours, and a second methodology for surveying library customers about reference service. The purpose of the study is to gauge how well HPL reference staff are coping with today’s rapid atmospheric shifts in reference practices, and will provide HPL management with vital information about the sort of staff retaining will be required in the future.

Marcel H. Faulkner is a librarian assistant at the Hamilton Public Library.







INTRODUCTION
This paper is a proposal for a study of the general effectiveness of reference service at the Central Branch of the Hamilton Public Library (HPL). The HPL is a medium-large library system that has undergone significant expansion and reorganization in the last few years. Once restricted only to the city of Hamilton, the HPL service area now includes Dundas, which has had its own library since 1896,[1] and the outlying rural areas that were once served by the Wentworth Libraries system. In 2001,[2] the Wentworth and Dundas libraries amalgamated with the HPL into a single system with 24 branches, 34 bookmobile stops, and a visiting library service for the homebound.[3] The system now serves a population of more than 650,000.[4] The downtown Hamilton Central Branch is by far the largest HPL branch, with four full floors of public access shelving areas, with materials arranged mostly by Dewey number.
As with all library reference workers in this era of mercurially burgeoning information technology, the HPL’s reference staff has had to adapt to the volatile timbre of the times. The HPL’s Central Branch reference desks are staffed by twenty-five regulars, including two department managers who take occasional desk shifts, all of whom are either librarians or librarian assistants. There is also a varying number of irregulars, who are called in as needed. All reference staff members have had to become adept at the use of online resources in addition to the traditional print sources that not so long ago comprised almost the entirety of library reference source materials. It is undeniable that online resources often make reference work easy by providing instantaneous answers: for example, if someone wants to contact the Canadian Olympic Committee, a simple Google search will provide the information in seconds, or if a patron wants to see a current version of the Controlled Drugs and Substances Act, we know that the online version is easy to find and more likely than any library’s print version to be completely up-to-date.[5]
But as inestimably useful as online resources may be, they can also be overwhelming for the reference professional, who now has to know not only what is in the library but also what is on the World Wide Web, and on top of this must also have a good working knowledge of the numerous specialized databases the library subscribes to. This may in fact be the most difficult time in history to be a reference professional, because so much is new and so much is changing, and there is every indication that the changes are far from over. Will reference work someday require the use of 100% online sources and 0% print sources? It is beyond the scope of this research proposal to delve into this possibility, but it is clear that the library profession today is no place for those who become squeamish in the face of rapid and constant change.
As so many libraries do nowadays, the HPL attempts to strike a balance between traditional client demand for print publications and the new demands for easy-to-access full-text online sources, and must do so with a materials budget that is not large enough to completely satisfy both sets of demands. There is simply not enough money to please everyone. Over the past few years, the trend at the HPL has been to gradually discontinue subscriptions to some print publications and apply the savings to online subscription databases. Some patrons are understandably unhappy about losing treasured magazines and journals, but many of the discontinued periodicals, such as Canadian Journal of Criminology and Railway Age, are accessible online for HPL cardholders, free of charge, through the HPL’s subscription databases.[6] The HPL subscribes to thirty-one databases, with specialization in subjects as diverse as health information, auto repair, literary reviews, current events, and local business contact information, as well as general information databases such as MasterFile, Expanded Academic ASAP and eLibrary Canada, which cover a stunning pastiche of subject areas.[7]
In the midst of this information inundation, how well are HPL reference staff coping with changes? Are they adapting smoothly to the new forms of reference materials, and using them to effectively serve library customers, who themselves are often struggling to change with the times? Do they know when to use Google and when not to? If staff relied in the past on print versions of directories, reports, or journals that the HPL no longer subscribes to, how well are they handling the migration to online sources? If a patron comes to them bemoaning the library’s cancellation of Railway Age, can they show the patron the steps involved in accessing MasterFile’s online version of Railway Age? One study has shown that reference personnel are generally not completely aware of the coverage and scope of their own libraries’ online databases.[8] The same study also uncovered evidence that patron use of online databases may actually increase the number of reference enquiries a library receives.[9] The purpose of this research proposal is to provide HPL management with the method and means to evaluate how their reference professionals are serving their public, and enable them to identify specific problem areas that can be bolstered by additional training or workshops.
RESEARCH PROBLEM
HPL managers are confident in the competence and adaptability of their librarians and librarian assistants, and have a general idea of how well they are holding up in the midst of saluki-quick change, but would nonetheless welcome a method of evaluating reference staff that would pinpoint staff strengths and weaknesses. Such an evaluation would identify what future training needs.
As crucial as staff adaptability to technical change may be, it is not the sole defining characteristic of good reference service, and therefore should not be the only quality evaluated when assessing staff performance. Staff should be knowledgeable, of course, but reference service is just that, a service to a clientele, and a proper and thorough evaluation of reference staff performance should incorporate the views and values of the customers in some way. Was the reference person approachable and helpful, or did the customer get the impression that answering the reference enquiry was an unwelcome imposition on the reference person’s time? Did the reference person make the effort to understand exactly what it was the customer wanted? Was the reference person friendly, courteous, and interested in the customer’s reference enquiry, or did the customer get the impression that the reference person was only going through the motions? And perhaps most importantly, was the customer satisfied with the reference person’s handling of the reference enquiry?
It can be argued that library customers lack the expertise to make accurate critical assessments of reference service, and even though this is true to some degree, the customers form the public perception of the library in any community, and public perception is and has always been critical to a library’s operations. Librarians and managers can be absolutely gaga about the reference service they provide, but what good is this if the customers do not share this opinion? We must never forget that if the library needs a budget increase, it may require extra municipal taxation; library customers will resent this increase, and they will not defend the usefulness of library, if they do not have a gut-level belief in the efficacy of library services. And while the public may lack the critical expertise to properly evaluate reference services, librarians lack the objectivity to assess how their reference services are being perceived by their customers. However much we may think we know about what constitutes good reference service, and however much we may think that the customers should be happy with the service we provide, only the customers can tell us if they are truly satisfied with what we do. And we must also remember that different patrons may be satisfied with vastly different levels of service: some will want you to give them the moon in a mule-wagon, while others will be completely satisfied if all you do is find a book on the shelf for them.[10]
It is therefore essential for any evaluation of the HPL’s reference service to include an assessment of customer satisfaction. The reference evaluation proposed in these pages is a bipartite program in which library professionals assesses HPL reference service, and another assessment that focuses on the customers’ unprofessional but nonetheless important opinions about the service they receive. Either assessment without the other would be incomplete. Failing to consult the customers when assessing reference service is akin to shooting at a target with a pistol, and then looking only at the pistol to see whether if you’ve hit the target.
LITERATURE REVIEW
Studies that evaluate library reference service can be roughly divided into two categories, those that examine reference staff and those that concentrate on customers. Studies rarely if ever do both.
Of the studies that evaluate reference staff, Hernon and McClure did what is arguably the most famous one: in their highly controversial “Unobtrusive reference testing: the 55 percent rule,” they used the “secret shopper” technique to test the ability of reference staff to answer sets of pre-determined questions.[11] The study was fairly intensive: it was conducted over a four-month period in 1985, and involved the reference staff of 13 public libraries and 13 academic libraries; in all, 390 questions were asked of reference staff at various locations throughout the U.S.[12] As the article title suggests, Hernon and McClure found reference staff performance to be mediocre: library patrons have little better than a 50-50 chance of getting their reference enquiries answered correctly. Librarians were generally appalled by this figure, and both the methodology and the results of Hernon and McClure’s study have been heavily criticized since its publication in the late 1980’s. For one thing, the study used set questions with right or wrong answers, which is artificial and unrealistic, since library patrons frequently ask questions that are not resolutely formed “discrete enquiries” and do not have answers that can be easily recognized as right or wrong.[13] Also, Hernon and McClure’s professional questioners were unlikely to ask questions the same way that real library customers would. Nor did Hernon and McClure’s study account in any way for how the question was answered or how the person asking the question felt about the reference person’s mannerisms and overall demeanour (i.e., friendly or standoffish, helpful or minimally communicative, polite or abrupt, etc.).[14] Other studies using the same methodology as Hernon and McClure produced widely varying results, suggesting that reference service is difficult if not impossible to quantify precisely.[15]
Baker and Field also used a secret shopper methodology when they sent library sciences students out to evaluate reference staff at various public and academic libraries.[16] Unlike the work of Hernon and McClure, this study focused less on questions being answered and more on physical observations, such as how the reference person reacted to the student, and how easy it was to identify the reference person as the one who should be consulted.[17] The study found that 42% of the students were dissatisfied with the reference service they received,[18] but this exact number should not be taken to heart because of a flaw in the methodology used: library sciences students are not representative of typical library users, and as prospective librarians are not without bias.
In “Reference is better than we thought,” Richardson states that the 55 percent figure is not particularly useful for evaluating reference service.[19] According to Richardson, some studies came close to duplicating Hernon and McClure’s results of approximately 55 percent accuracy rate, but exit surveys of patrons conducted at the same libraries indicated much higher user satisfaction with reference services.[20] In general, users were almost always directed at least something they wanted from the reference transaction, and usually expressed satisfaction, rather than dissatisfaction, with the service they received.[21] This suggests that Hernon and McClure’s study, and others that used similar methods, were not measuring what they should have been measuring. Richardson also criticizes the consistency of most studies that used unobtrusive testing, citing a lack a clear definition of good reference service and “inconsistent operational definitions of both the independent and outcome variables,” as well as failures to eliminate bias, samples that were too small and insufficiently random, and other questionable statistical and methodological practices.[22] Hubbertz is also highly critical of unobtrusive testing studies, and dismisses them as worthless, not in the least because the uniform testing requirement within individual studies is generally ignored or disregarded.[23] As questionable as unobtrusive testing can be when different reference people are asked the same question, the reliability of the results are even more doubtful if different questions are asked.[24] Hubbertz concludes that unobtrusive testing has produced too few worthwhile results to be a reliable barometer of reference service,[25] and that the 55 percent rule is a fairly worthless generalization.[26]
When Durrance conducted a study reference interviews, she evaluated not only the accuracy of the reference response but also the physical environment where the interviews took place.[27] One of Durrance’s key observations was that her secret shoppers, library science students who are more library-savvy than the general public, sometimes had difficulty finding the reference desks.[28] This important environmental aspect is rarely addressed in reference evaluation literature. If customers are confused or visually overwhelmed, which can happen easily in an unfamiliar and busy place with lots of signage, the reference transaction starts behind the eight-ball. Durrance’s study noted that the reference desk was often difficult to identify, and customers also did not know the status or role of the person sitting there.[29] Durrance concludes that if libraries seek to improve reference service, they must also be willing to improve the ambience of the reference desk.[30]
Dewdney and Ross used a methodology similar to Durrance’s, and the title of their study article, “Flying a light aircraft,” refers to a laughably sad instance when a reference person found a book on how to fly an airplane when the customer really wanted to know about jet lag.[31] This is a single instance, and should not be regarded as typical: the customer here was a library sciences student, a secret shopper who deliberately asked for books on flying to see if the reference person would conduct a proper interview to get at what the customer really wanted. As with other studies mentioned here, the artificiality of the reference questions must be considered. Even so, the findings of Dewdney and Ross should not be ignored: the study confirmed rather than disputed the 55 percent rule, and some of the interview results suggest that some reference desk personnel need to improve their basic social skills.[32] The authors concur with Durrance’s idea that good reference service should induce the customer to return to the same reference person the next time they need help,[33] a test which 40% of the reference staff in this study failed.[34]
Secret shopper testing helped the Stanislaus County Free Library system in Modesto, California improve its service.[35] Unlike other unobtrusive testing studies, this one was conducted by the local Better Business Bureau (BBB), rather than by anyone affiliated with the library profession. Once the BBB agreed to do the test, library administration did not know when the test would by conducted or by whom, and even after the test did not know it had been done until they received an evaluation letter from the BBB a few weeks later.[36] Secret shoppers were sent to each of the system’s 13 branches and reported only on how they were treated at each branch, and were evaluating general customer service rather than reference service specifically. This was an informal study, and some of the evaluation criteria, which were devised to evaluate businesses or retail outlets, were unfair when applied to libraries. For example, customer service clerks were judged on whether or not they greeted the secret shoppers on entering the building, which would be impossible in a busy library, but the evaluation form did not account for this in any way.[37] Ultimately, the secret shopper testing did bring to light several areas where improvements were needed, and enabled management to know what kind of refresher training to use in the future.[38] This suggests that unobtrusive testing can be useful for evaluating customer service, and this can by extension be applied to reference services, as long as the aims of the testing are general impressions rather than statistical certitudes.
According to the American Library Association’s Guidelines for Behavioral Performance of Reference and Information Service Professionals, “the positive or negative behavior of the reference staff person…becomes a significant factor” in the success or failure of the reference transaction, and will impact customer satisfaction.[39] Quite simply, the customer must not find the person behind the reference desk off-putting or intimidating if the reference transaction is to succeed. Moysa conducted an evaluation of reference service at an academic library using these guidelines as a starting point, and agreed with the ALA’s guidelines that the three most important behaviours for reference staff are approachability, interest in the customer’s enquiry, and a positive attitude.[40] Moysa devised a customer service checklist which focuses on the behaviour of the reference desk staff, but does not assess staff knowledge of reference materials in any way. [41] In this study, staff members were evaluated three times. They were initially given the checklist to evaluate their own performance, which familiarized service expectations, and also permitted them to prepare for their second and third evaluations, which would be done by other staff members using the same checklist.[42]
Studies that focus on the customer rather than library staff comprise the second broad category of reference evaluation studies. Radford conducted a study that fits into both categories, because the reference personnel were evaluated by library professionals and customers were surveyed about the reference service they received.[43] The nonverbal communication by reference professionals to customers was evaluated by a third party, who was a library staff member.[44] This atypical study observed customers when they approached the reference desk and had a choice between two reference staff members to ask for help. The reference staff’s nonverbal communications were noted, and customers were afterwards asked why they chose the person they did.[45] There are a couple of questions about the methodology used: for one thing, how could the evaluator be absolutely certain about noting the same non-verbal communication cues that the customer saw? It is possible for the evaluator to see many nonverbal cues that the customer missed, and vice versa. The follow-up interview would help to alleviate this problem, as long as the customers were conscious of the effect of reference personnel’s non-verbal cues. In real life libraries, some customers will simply not admit why they chose one reference person over another: they may simply have chosen the more attractive of the two, or perhaps wanted to discuss a sensitive topic only with someone of their own gender. Overall, however, Radford was able to conclude that customers react to numerous nonverbal communications by reference staff.[46]
Whitlach points out that before any evaluation is begun, management should have a clear and precise definition of satisfactory reference service, and should also know exactly what it intends to do with the evaluation results afterwards.[47] Whitlach also gives some good pointers on getting customer-based evaluations. Questionnaires and surveys are useful for gathering data directly from the customers, and have the advantages of being reasonably efficient, relatively easy to create, and not labour-intensive; the disadvantages are low response rates and the possibility of self-selected samples, particularly with online surveys.[48] For both online and paper questionnaires, respondents tend to leave open-ended questions unanswered.[49] One-on-one customer interviews and focus groups, on the other hand, are more effective for getting detailed customer feedback, but have the disadvantage of being both labour- and time-intensive, which in turn makes them relatively expensive.[50]
Jacoby and O’Brien surveyed undergraduate students on their perceptions of reference staff at a university library. Approachability was highly valued here, but so was the educational aspect of reference work: students were asked whether they learned anything from the reference transaction about finding material for themselves.[51] None of the other studies discussed here made teaching the customer a high priority. This study did exit surveys, and found that the behaviour, mannerisms, and general demeanour of staff had a huge effect on reference transaction outcomes.[52] Jacoby and O’Brien ensured the validity of their findings by doing this study thoroughly: confounding variables were minimized, the surveyed students were randomly selected, and surveying was done at different times of day and on different days of the week to ensure a suitable cross-section of both students and reference staff.[53]
Norlin devised a tripartite study that used customer questionnaires for demographic information, exit surveys for immediate feedback, and focus groups to get more in-depth information.[54] This thorough study emphasizes that reference service is best evaluated by several methods, rather than by any single one. Even so, Norlin consulted only the customers, without having library professionals rate the reference staff. An over-reliance on customer surveys will produce biased results. Studies that do no more than ask customers in libraries what they think will almost always rate reference service highly.[55] This is in fact what Norlin’s test subjects did.[56] If the local library is the only game in town, customers have nothing to compare it to, and will adapt to a gradual lowering of reference service standards simply because they have no choice.[57] And if they are interviewed at the library, odds are they are already library keeners and hardly an unbiased, random sample. Anyone who has become fed up with ineffective reference service will simply stop using it, and will therefore not be surveyed.[58] This may be an unsolvable paradox: how would you go about surveying people who no longer use the library about what they think of reference service? Their reasons for library apostasy would be useful criticisms, but where would you find these people?
The point here is that customer surveys are useful but should not be the sole method of evaluating reference services, simply because they will tell you only part of what you need to know. Similarly, studies that focus only on the reference staff are also unlikely to give you the complete picture of reference staff effectiveness. The only conclusion you can draw from this is that evaluations of reference staff effectiveness must assess the ability and demeanour of reference persons, and must also get input from customers about the service they have received.
RESEARCH QUESTION OR HYPOTHESIS
Based on the literature reviewed above, an evaluation of a library’s reference service should include assessments of both the reference staff and customer opinions. Of the studies reviewed in the Literature Review above, only Radford did this. (For a table of what each of the studies covered, see Appendix A, page 25 of this paper).
The research question here is pretty basic: how good is the reference service at the HPL? These questions can be answered by means of a bipartite program. In the first part of the program, reference staff members are evaluated by library professionals; in the second part, reference customers are surveyed about the reference service. The author of this proposal has no pretensions of methodological originality, and has devised evaluation methods that are conceptually similar to ones described in the Literature Review above.
For the staff evaluation, a designated library staff member will evaluate each reference person during two-hour reference desk shifts. There are twenty-five people who regularly work on the HPL’s Central Library subject floor reference desks, including the two departmental managers who are qualified librarians and take once- or twice-weekly desk shifts.[59] Each reference person will be evaluated twice, each time on a different day. These evaluations will all be done by the same person to ensure consistency. The math here is that each of the twenty-five people will be evaluated for a total of 4 hours, making 100 evaluation hours in total. Accounting for staff vacations and other absences, the evaluations can be done over a five-week period.
There is no getting around the fact that staff evaluations make people uncomfortable, and these ones are exacerbated by the fact that they will not take place out of public view in a manager’s office or meeting room, which can be stressful enough, but on the job. It would be the rare individual who would be unaffected by this. But whether we like it or not, staff evaluations are part of modern working life, and librarians and librarian assistants must survive them somehow.
There are several ways to lessen staff stress. Trust levels between managers and reference workers are definitely a factor here. It will be imperative for the HPL departmental managers to assure staff that the evaluations are intended to improve service: anyone who shows weaknesses in any area will be not be disciplined or penalized but given refresher training. In the unionized HPL, salaries or job status will not be affected by minor performance imperfections. Keep in mind also that all HPL reference staff will be evaluated, including the two departmental managers, who will also be subject to refresher training if their evaluations deem it necessary. This will be good for morale, and will help staff understand that everyone will be held to the same standard.
Susan Moysa also has an ingeniously simple way to help staff cope with evaluation anxiety: give copies of the evaluation checklist to staff members in advance and allow them to evaluate themselves before the two evaluations that will be done by the designated staff member.[60] The checklist below is not intended to be statistical: reference performance is not an exact science and is nearly impossible to quantify mathematically, so no method of evaluation will produce statistical exactitude. This pre-informing process has the advantage of letting staff improve their own performances before being formally evaluated, and lets them know exactly what is expected of them.[61] This evaluation methodology has several distinct advantages: every staff member gets personally evaluated and knows precisely what is being evaluated, and the entire evaluation process is made less mysterious and fearsome.
REFERENCE SERVICE EVALUATION CHECKLIST

Name:

REFERENCE SERVICE STANDARD
Almost always
Most of the time
Some of the time
Almost never
Comments

1. APPROACHABILITY

(a) Breaks away from other tasks to give customer priority






(b) Initiates eye contact with customer






(c) Greets customer






(d) Offers assistance to customers before they ask for it






(e) Reassures customers who are unfamiliar with library methods






(f) Acknowledges waiting customers






(g) Leads customers to materials when necessary






2. INTEREST

(a) Faces customer when speaking, listening






(b) Listens to customer carefully








(c) Maintains appropriate eye contact during transaction






(d) Verbally or nonverbally communicates understanding of customer’s reference needs






3. POSITIVE ATTITUDES

(a) Overall professional demeanour






(b) Respectful, courteous to all customers & staff






(c) Overall positive attitudes towards library, services, policies








4. USE OF RESOURCES





(a) Knowledge of HPL print sources






(b) Instructs customers in use of OPAC’s & other library apparatus when necessary






(c) Advises customers to use HPL electronic databases






(d) Instructs customers in use of HPL electronic databases when necessary







The checklist above is an adaptation of one used by Moysa.[62] Some of Moysa’s evaluation criteria were not terribly relevant to the HPL’s staff and were therefore excluded. For example, her item “1.10: is well-dressed and groomed appropriately” is unnecessary; the HPL dress code for reference staff is not particularly strict, but anyone who failed to adhere to it would simply be taken aside by the departmental manager and told to dress properly. Such instances are rare, probably because this is a matter of elementary common sense. There is no reason to wait for staff evaluations to address such a matter.
Moysa was not evaluating how staff members were using electronic resources, so her checklist did not include this; for the HPL, however, this is an important feature of the evaluation, so three reference service standards relating to online sources, including the HPL onsite OPAC’s, have been added to the checklist. This checklist is for general impressions rather than statistical precision. The evaluator must keep in mind that not every item on the list is necessary for every single customer, and some of the items on the list become unnecessary if other items are checked off. For example, if the reference person asks the customer if he or she needs assistance (1d), it not necessary to also greet the customer (1c), and it would probably be ludicrous and awkward to do so. At the HPL reference desks, it is also possible for customers to approach the reference desks from the back or side, which makes it impossible for reference staff to initiate eye contact (1b). Nor is the checklist intended to be absolutely perfect. For example, it may not always be possible for a third party to accurately observe if and when the reference desk patron makes eye contact with the customer.
The second part of the HPL reference service evaluation process is the customer exit survey. The survey itself will be a simple questionnaire: customers will be asked by an HPL staff member to participate in the survey, and will then be asked to fill out the questionnaire. A prototype of the four-part questionnaire is included below.[63] Since this survey will be filled out by customers, the use of professional jargon has been judiciously avoided.

REFERENCE SERVICE CUSTOMER SURVEY
Part 1: Why did you come to the library today?
Check off each item that applies.
Looked for magazine/journal article(s) on a specific topic

Looked for/reserved book(s) on a specific topic

Looked for/reserved newly arrived items

Read newspapers or magazines

Used Employment Centre resources

Worked or studied

Looked for/reserved DVD’s/videos

Looked for/reserved music CD’s/cassettes

Used library computers

Used microfilm readers/printers

Used photocopiers

Used piano practice room

Went to ESL tutorial session

Went to Homework Help Club session

Played chess or other game

Other (please specify):



Part 2: What did the reference staff member help you do today?
Check off each item that applies.
Find articles in magazines/journals

Use HPL online databases

Use other online sources (search engines, government websites, etc.)

Find/reserve books

Use public catalogue terminals

Find information in print directories

Find information in online directories

Sign up for and use library computers

I’m new to this library and needed to ask about membership, procedures, computer use, etc.

Other (please specify):

Part 3: How helpful was the reference/information desk person you spoke to?
Check off each item that applies.
Stopped other tasks & acknowledged me immediately

Made effort to understand what I was looking for

Was approachable and cheerful

Did not condescend or talk down to me

Was interested in what I was searching for

Was professional in every way

Part 4: Overall, how good was the assistance you received from reference/information desk staff?
Check one only.
Excellent

Good

Adequate

Less than adequate

Terrible

In your own words, describe what you liked about the service you received?


What do you think should be improved in the future?





Respondents can choose more than one item in Parts 1-3, so eventual percentage totals will exceed 100%. A test trial would determine if this questionnaire is hitting the mark, and is highly recommended. It must also be noted that both the questionnaire and the Reference Service Evaluation Checklist would need minimal adaptation for use at the other twenty-three HPL branches.
CONCLUSION & DISSEMINATION OF FINDINGS
As a tool for determining what aspects of HPL reference service need improvement, the research proposal above would be feasible, easy to implement, and relatively cost-effective. As a formal scientific study, however, it has a flaw that you have probably already noticed. For the reference staff evaluations, informing the test subjects (the reference staff) in advance of the evaluation criteria would taint the eventual study results: this Hawthorne Effect would almost certainly affect HPL staff performances, because there is every possibility that staff will improve on their own after they first read the checklist. This would render any statistical results invalid. But keep in mind that the ultimate purpose of this evaluation is to seek out reference staff strengths and weaknesses, for the purpose of encouraging the former and correcting the latter. If the process of the study itself helps improve staff performance, how much does it matter that the results are statistically invalidated? Another advantage of the pre-informing process is that it helps to reduce evaluation anxiety. This methodological quirk lessens this study’s additive value to reference evaluation literature, but at the same time makes it more useful to HPL reference staff managers, whose primary interest is service improvement rather than precise statistical analysis. The interests of the library for which this study is being conducted must take precedence over the study’s ability to further enhance reference evaluation literature. Another consequence of this study’s built-in Hawthorne Effect is that it will not measure reference staff improvement; it will, however, gauge reference staff ability and customer satisfaction: if both of these are affected by the study itself, the affect will be for the better, which is more than acceptable to HPL departmental managers.
There are, of course, biases on the part of the reference staff evaluator. Even when the stated and agreed-upon goal is objective evaluation, there is always the possibility that the evaluator will intensely dislike one of the reference people being evaluated. This is unavoidable human nature. There are 25 reference desk staff members at the HPL, and it is highly unlikely that every single one of them absolutely loves the other 24. Even using someone from outside the department as an evaluator will not eliminate the problem, because in the HPL system reference staff members often change jobs between HPL departments and branches, which in effect means that everybody pretty much knows everybody else, regardless of where they may work in the HPL system. One possible way to solve this problem would be to have the evaluations done by someone who does not know the staff personally, either because he or she has been hired on a contract basis specifically to do these evaluations or because he or she has been newly hired on a permanent basis but has not yet had a chance to form biases about staff members one way or the other.
The Hawthorne Effect that makes this study less scientifically valid would also affect the publication value of this study’s findings. The editors of a journal such as Reference & User Services Quarterly, which mostly publishes scholarly articles rather than opinion pieces or more general news stories for librarians, would not likely be interested in an article about this study. On the other hand, publications like Feliciter and Library Journal, which publish scholarly articles but also print more general library-related news and articles, might very well be interested. This study’s methodology, and particularly its methods of minimizing the trauma of staff evaluations, would be of great interest to working librarians and library managers, and might be something they could use in their own libraries. The evaluation-and-survey process described in these pages will indirectly help to improve reference service by revealing to HPL management what is working and what needs to be improved. When this happens, the greater library community would undoubtedly love to hear about it.





















REFERENCE LIST

Articles:
Baker, Lynda M., and Field, Judith J. “Reference success: what has changed over the past ten years?” Public Libraries, v. 39, no. 1, Jan/Feb 2000; pp. 23-7. HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=224055.

Cotter, Rosemary. “Performance indicators for reference and information services, Part 1,” IATUL Proceedings [paper presented at IATUL Conference, 1996]; HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=225117.

Czopek, Vanessa. “Using mystery shoppers to evaluate customer service in the public library,” Public Libraries, v. 37, no. 6, Nov/Dec 1998; pp. 370-3. HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=32EC0N1V5S2IDQA3DIMSFF4ADUNGIIV0?_requestid=128660.

Dewdney, Patricia, and Ross, Catherine Sheldrick. “Flying a light aircraft: reference evaluation from a user’s viewpoint,” RQ, v. 34, no. 2, Winter 1994; pp. 217-31. HTML version online from Expanded Academic ASAP: http://infotrac.galegroup.com.myaccess.library.utoronto.ca/itw/infomark/161/818/83899491w5/purl=rc1_EAIM_0_A16442115&dyn=17!xrn_38_0_A16442115?sw_aep=utoronto_main.

Durrance, Joan C. “Reference success: does the 55 percent rule tell the whole story?” Library Journal, v. 114, no. 7, Apr 15, 1989; pp. 31-6.

Green, Denise D., and Peach, Janis K. “Assessment of reference instruction as a teaching and learning activity,” College & Research Library News, v. 64, no. 4, Apr 2003; pp. 256-8.

Hernon, Peter, and McClure, Charles R. “Unobtrusive reference testing: the 55 percent rule,” Library Journal, v. 111, no. 7, Apr 15, 1986; pp. 37-41.

Hubbertz, Andrew. “The design and interpretation of unobtrusive evaluations,” Reference & User Services Quarterly, v. 44, no. 4, Summer 2005; pp. 327-35.

Jacoby, JoAnn, and O’Brien, Nancy P. “Assessing the impact of reference services provided to undergraduate students,” College & Research Libraries, v. 66, no. 4, Jul 2005; pp. 324-40.

Moysa, Susan. “Evaluation of customer service behaviour at the reference desk in an academic library,” Feliciter, v. 50, no. 2, 2004; pp. 60-3.

Murray, Janet, and Tschernitz, Cindy. “The Internet myth: emerging trends in reference enquiries,” Australasian Public Library & Information Services, v. 17, no. 2, June 2004; pp. 80-8.

Norlin, Elaina. “Reference evaluation: a three-step approach: surveys, unobtrusive observations, and focus groups,” College & Research Libraries, v. 61, no. 6, Nov 2000; pp. 546-53. HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=32EC0N1V5S2IDQA3DIMSFF4ADUNGIIV0?_requestid=128214.

Radford, Marie L. “Approach or avoidance? The role of nonverbal communication in the academic library user’s decision to initiate a reference encounter,” Library Trends, v. 46, no. 4, Spring 1998; pp. 699-717.

Richardson, John V. “Reference is better than we thought,” Library Journal, v. 127, no. 7, Apr. 15, 2002; pp 41-2.

Wallace, Danny P., and Van Fleet, Connie. “Strange bedfellows: evidence of accuracy in professional performance,” Reference & User Services Quarterly, v. 43, no. 2, Winter 2003; pp. 109-10.

White, Herbert S. “public library reference services: expectations and reality,” Library Journal, v. 124, no. 11, June 15, 1999; pp. 56-7. HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=223315.

Whitlach, Jo Bell. “Evaluating reference services in the electronic age,” Library Trends, v. 50, no. 2, Fall 2001; pp. 207-17.

Websites:
American Library Association website, Reference and User Services Association “Guidelines for Behavioral Performance of Reference and Information Services Professionals” page: http://www.ala.org/ala/rusa/rusaprotools/referenceguide/guidelinesbehavioral.htm.

Hamilton Public Library home page: http://www.myhamilton.ca/myhamilton/LibraryServices/.

Libraries Today website, Ontario Library Picture Gallery section, Dundas Library page: http://www.uoguelph.ca/~lbruce/photogallery.shtml.

Statistics Canada website, Census Metropolitan Area section, Hamilton page: http://www40.statcan.ca/l01/cst01/demo27l.htm.

Works Consulted But Not Cited:
Creswell, John W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 2nd edition (Thousand Oaks, CA: SAGE Publications, 2003).

Ford, Charlotte E. “An exploratory study of the differences between face-to-face and computer-mediated reference interactions,” College & Research Library News, v. 65, no. 11, Dec. 2004; p. 645.

Locke, Lawrence F., Spirduso, Waneen Wyrick, and Silverman, Stephen J. Proposals That Work: A Guide for Planning Dissertations and Grant Proposals, 4th edition (Thousand Oaks, CA: SAGE Publications, 2000).

Punch, Keith F. Developing Effective Research Proposals (London: SAGE Publications, 2000).

Salant, Priscilla, and Dillman, Don A. How To Conduct Your Own Survey (New York: Wiley & Sons, 2000).




























APPENDIX A:
Table of Studies Examined in Literature Review





TABLE OF STUDIES EXAMINED IN LITERATURE REVIEW

AUTHOR(S)
FOCUS OF STUDY

METHODS
LIBRARY TYPE(S)
Baker & Field
Reference staff, library environment
Secret shopper
Academic, public
Czopek
All library staff
Secret Shopper
Public
Dewdney & Ross
Reference staff
Secret shopper
Academic, public
Durrance
Reference staff, library environment
Secret shopper
Academic, public, special
Green & Peach
Reference staff
Survey
Academic
Hernon & McClure
Reference staff
Secret shopper
Academic, public
Moysa
Reference staff
Checklist evaluation by other staff
Academic
Murray & Tschernitz
Reference staff
Interviews
Public
Norlin
Customers
Questionnaires, surveys, focus groups
Academic
Radford
Customers, reference staff
Observation, interviews
Academic
Whitlach
Customers
Questionnaires, surveys, interviews
Academic









APPENDIX B:
Moysa’s customer service checklist for reference desk staff.
(Resized slightly to fit this document but otherwise unaltered.)



APPENDIX C:
Jacoby and O’Brien’s survey tables, Parts 1 & 2.
Resized slightly to fit this document but otherwise unaltered.

[1] Libraries Today website, Ontario Library Picture Gallery section, Dundas Library page: http://www.uoguelph.ca/~lbruce/photogallery.shtml.
[2] HPL website, Meeting User Needs 1: A Model for the Delivery of Information/Reference Service, November 2001: http://www.myhamilton.ca/NR/rdonlyres/4E5AE266-BE17-4E78-AC51-AE99BC1D0AB3/16341/meetinguserneeds.pdf; p. 1.
[3] Hamilton Public Library (HPL) website, Locations and Hours page: http://www.myhamilton.ca/myhamilton/LibraryServices/AboutTheLibrary/LocationsAndHours/.
[4] Statistics Canada website, Census Metropolitan Area section, Hamilton page: http://www40.statcan.ca/l01/cst01/demo27l.htm.
[5] Both of these examples are actual patron reference questions asked at the HPL in March, 2006. The Canadian Olympic Committee contact information is available on the Canadian Olympic Committee website, Contact Us page, http://www.olympic.ca/EN/organization/contact.shtml; the text of the Controlled Drugs and Substances Act is available at the Canada Department of Justice website, Laws section, Controlled Drugs and Substances Act page, http://laws.justice.gc.ca/en/c-38.8/229593.html. Neither search was actually timed, but in both cases the information was found quickly and easily by the writer of this proposal.
[6] The HPL subscribed to the print editions of both Canadian Journal of Criminology and Railway Age, but recently cancelled both subscriptions. Current and back issues of both titles are available online to HPL members from the MasterFile database on the HPL’s Search Online Databases page: http://www.myhamilton.ca/myhamilton/LibraryServices/WorkFromHome/Search+Online+Resources.htm.
[7] HPL website, Search Online Resources page: http://www.myhamilton.ca/myhamilton/LibraryServices/WorkFromHome/Search+Online+Resources.htm. Accessed Mar. 24, 2006. This page shows only twenty-six of the subscription databases: five others can only be accessed from within HPL branches and are not shown on this website page.
[8] Murray, Janet, and Tschernitz, Cindy. “The Internet myth: emerging trends in reference enquiries,” Australasian Public Library & Information Services, v. 17, no. 2, June 2004; p. 82.
[9] Ibid., p. 84.
[10] Cotter, Rosemary. “Performance indicators for reference and information services, Part 1,” IATUL Proceedings [paper presented at IATUL Conference, 1996]; p. 2. Page numbers herein refer to Print Preview pages of HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=225117.
[11] Hernon, Peter, and McClure, Charles R. “Unobtrusive reference testing: the 55 percent rule,” Library Journal, v. 111, no. 7, Apr 15, 1986; p. 41.
[12] Ibid., p. 38.
[13] Green, Denise D., and Peach, Janis K. “Assessment of reference instruction as a teaching and learning activity,” College & Research Library News, v. 64, no. 4, Apr 2003; pp. 256.
[14] Ibid., pp. 257-8.
[15] Wallace, Danny P., and Van Fleet, Connie. “Strange bedfellows: evidence of accuracy in professional performance,” Reference & User Services Quarterly, v. 43, no. 2, Winter 2003; p. 109.
[16] Baker, Lynda M., and Field, Judith J. “Reference success: what has changed over the past ten years?” Public Libraries, v. 39, no. 1, Jan/Feb 2000; pp. 23-7. Pages cited herein refer to Print Preview pages from the HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=224055.
[17] Ibid., p. 4.
[18] Ibid., p. 7.
[19] Richardson, John V. “Reference is better than we thought,” Library Journal, v. 127, no. 7, Apr. 15, 2002; pp 41-2.
[20] Ibid., p. 41.
[21] Ibid.
[22] Ibid., pp. 41-2.
[23] Hubbertz, Andrew. “The design and interpretation of unobtrusive evaluations,” Reference & User Services Quarterly, v. 44, no. 4, Summer 2005; pp. 328-9.
[24] Ibid., p. 331.
[25] Ibid., p. 335.
[26] Ibid., p. 331.
[27] Durrance, Joan C. “Reference success: does the 55 percent rule tell the whole story?” Library Journal, v. 114, no. 7, Apr 15, 1989; pp. 31-2.
[28] Ibid., p. 32.
[29] Ibid., p. 36.
[30] Ibid.
[31] Dewdney, Patricia, and Ross, Catherine Sheldrick. “Flying a light aircraft: reference evaluation from a user’s viewpoint,” RQ, v. 34, no. 2, Winter 1994; pp. 3-4. Pages cited herein refer to Print Preview of HTML version online from Expanded Academic ASAP: http://infotrac.galegroup.com.myaccess.library.utoronto.ca/itw/infomark/161/818/83899491w5/purl=rc1_EAIM_0_A16442115&dyn=17!xrn_38_0_A16442115?sw_aep=utoronto_main.
[32] Ibid., p. 20.
[33] Durrance, p. 34.
[34] Dewdney and Ross, p. 20
[35] Czopek, Vanessa. “Using mystery shoppers to evaluate customer service in the public library,” Public Libraries, v. 37, no. 6, Nov/Dec 1998; pp. 370-3. Page numbers cited hereinafter refer to Print Preview pages from the HTML version, online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=32EC0N1V5S2IDQA3DIMSFF4ADUNGIIV0?_requestid=128660.
[36] Ibid., p. 2.
[37] Ibid., p. 4.
[38] Ibid.
[39] American Library Association website, Reference and User Services Association “Guidelines for Behavioral Performance of Reference and Information Services Professionals” page: http://www.ala.org/ala/rusa/rusaprotools/referenceguide/guidelinesbehavioral.htm.
[40] Moysa, Susan. “Evaluation of customer service behaviour at the reference desk in an academic library,” Feliciter, v. 50, no. 2, 2004; p. 61.
[41] Ibid., p. 62. For a copy of the checklist, see Appendix B on p. 26 of this document.
[42] Ibid., p. 61.
[43] Radford, Marie L. “Approach or avoidance? The role of nonverbal communication in the academic library user’s decision to initiate a reference encounter,” Library Trends, v. 46, no. 4, Spring 1998; pp. 699-717.
[44] Ibid., p. 703.
[45] Ibid., pp. 704-5.
[46] Ibid., p. 713-4.
[47] Whitlach, Jo Bell. “Evaluating reference services in the electronic age,” Library Trends, v. 50, no. 2, Fall 2001; pp. 207-8.
[48] Ibid., p. 209.
[49] Ibid., p. 211.
[50] Ibid., p. 214.
[51] Jacoby, JoAnn, and O’Brien, Nancy P. “Assessing the impact of reference services provided to undergraduate students,” College & Research Libraries, v. 66, no. 4, Jul 2005; p. 324.
[52] Ibid., pp. 325-6.
[53] Ibid., pp. 326-7.
[54] Norlin, Elaina. “Reference evaluation: a three-step approach: surveys, unobtrusive observations, and focus groups,” College & Research Libraries, v. 61, no. 6, Nov 2000; p. 1. Page numbers cited herein refer to Print Preview pages from the HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=32EC0N1V5S2IDQA3DIMSFF4ADUNGIIV0?_requestid=128214.
[55] White, Herbert S. “Public library reference services: expectations and reality,” Library Journal, v. 124, no. 11, June 15, 1999; p. 1. Page numbers cited herein refer to Print Preview pages from the HTML version online from WilsonWeb: http://vnweb.hwwilsonweb.com.myaccess.library.utoronto.ca/hww/shared/shared_main.jhtml;jsessionid=Q4GE53BJD0GOPQA3DILCFGOADUNGIIV0?_requestid=223315.
[56] Norlin, p. 4.
[57] White, p. 1.
[58] Ibid.
[59] The current HPL departmental managers, who also take reference desk shifts, are Yvonne Patch and Darcy Glidden.
[60] Moysa, p. 61.
[61] Ibid., p. 62.
[62] Ibid. For a copy of Moysa’s checklist, see Appendix B on p. 26 of this paper.
[63] A significant portion of the criteria used in this prototype is drawn from survey tables used by Jacoby and O’Brien, pp. 330-1. Copies of these tables are included in Appendix C on p. 27 of this paper.