Conducting Pan-institutional and cohort studies
One of the most beneficial elements of an evaluation network is the opportunity to conduct network-wide studies, either in the form of pan-institutional or smaller cohort studies. During these studies all organizations work collaboratively to implement the same study at each institution. There are many benefits of participating in a network-wide study.
Evaluation Capacity Building (ECB)
Doing evaluation is one of the best ways to learn about evaluation. When done in a larger team, the learning can be exponentially greater.
What DEN Does
DEN members use the pan-institutional studies as an opportunity to learn together.
Physical Resource Sharing
Some instruments may call for a tool, such as a set of cards, a recording device, or a questionnaire, but not all partners have the equipment needed for data collection. Through a network the instruments may be created (or already exit) at one site and shared with the others.
What DEN Does
Human resource sharing.
Increased knowledge of community. Because the studies are designed, implemented, and shared across the network, members learn much more about the museum environment outside of their own organizations.
Cautionary Advice: A Case for Cohort Studies
Pan-institutional studies are not always the best method for collecting useful data. With members varying in size, budget, audiences, disciplines, and programs, it becomes increasingly difficult to find one study that will fit everyone equally. The benefits of the pan-institutional studies are listed above. Here are some of the challenges:
Evaluation Capacity Building (ECB)
Doing evaluation is one of the best ways to learn about evaluation. When done in a larger team, the learning can be exponentially greater.
What DEN Does
DEN members use the pan-institutional studies as an opportunity to learn together.
- Member institutions or outside professionals conduct group training on the use of a particular method or instrument.
- Member organizations become familiar with an instrument and the ECB is an internal group commodity.
- Working as part of the Studies Committee, members with little evaluation capacity work alongside more seasoned professionals, thus learning through doing. Participating on this committee is particularly helpful in learning how to draft good questions and protocols.
Physical Resource Sharing
Some instruments may call for a tool, such as a set of cards, a recording device, or a questionnaire, but not all partners have the equipment needed for data collection. Through a network the instruments may be created (or already exit) at one site and shared with the others.
What DEN Does
Human resource sharing.
- There are often occasions when an outside consultant is needed at an institution, such as an interviewer for a visitor panel or focus group. Through DEN, members can easily trade this service.
- Because individual members are required to assist with data collection, institutional members have access to a pool of free, trained, research assistants on an as-needed basis.
Increased knowledge of community. Because the studies are designed, implemented, and shared across the network, members learn much more about the museum environment outside of their own organizations.
Cautionary Advice: A Case for Cohort Studies
Pan-institutional studies are not always the best method for collecting useful data. With members varying in size, budget, audiences, disciplines, and programs, it becomes increasingly difficult to find one study that will fit everyone equally. The benefits of the pan-institutional studies are listed above. Here are some of the challenges:
- Many organizations will have small staff, low visitation or fewer resources, and may not be able to participate at the same level as other members.
- Organizations with a strong evaluation program in place may find that some studies are redundant to recent activity.
- Because pan-institutional studies need to be designed for the lowest common denominator, they can be too broad, not relevant to organizational activities, or provide data that are not helpful to everyone.
- It is not always possible to use the same tool the same way at all institutions. Logistically there may be too few available participants, or the layout of a facility may not attract the same traffic flow or concentration points (i.e. some may have multiple entrances/exits).
- It may not be possible to customize methods in such a way that the instrument intent/model remains recognizable and true.
- The more member organizations, the greater the challenge in finding a study that has mutual benefit to all. Additionally, partners that are not interested in a method or topic may be less likely to proceed with the same degree of rigor. At best this could lead to partial or incomplete data; at worst it may dissuade the organization from continuing with the network because they find that the activities don’t apply to their needs.
