This Families Matter! article was developed through an interview process between Lisa Conlan, TA Partnership Co-Family Resource Specialist, and Hillsborough Community Evaluation Team members Lyn Casseday, Dan Casseday, Norín Dollard, Paula Thomas, and Kristy Williams. Other members of the Community Evaluation Team who have been very involved, but unavailable for the interview process are Marisol Salgado, Caryn Santiago, Lydia Santiago, Delbra Stowe, and Jan Voyer.

The Hillsborough Community Evaluation team is a group of community members who are analytical advocates trained to evaluate systems and services and to make data useful to families and the organizations that serve them. This team provides technical assistance and evaluation services to ensure that the family perspective is part of evaluation and quality improvement for organizations that serve children and families.

The Hillsborough Community Evaluation team was established by a group of families and partners who developed the idea to form a team after attending the World of Evaluation (WOE) courses offered by the Federation of Families for Children's Mental Health. The Federation of Families developed the World of Evaluation courses based on their recognition of the importance of bringing the unique, experience-based perspectives of families into the realm of evaluation. Families can bring powerful and strategic improvements to systems through the use of evaluation. The Federation of Families worked in partnership with diverse family members, family organizations, the Research and Training Center on Family Support and Children's Mental Health at Portland State University, ORC Macro International, the Florida Mental Health Institute at the University of South Florida, and the National Technical Assistance Center for Children's Mental Health at Georgetown University to provide input and expert consultation related to the development of these courses. The World of Evaluation series includes the following three courses:  

• Level 1, The World of Evaluation: How to Understand It, helps families to understand the major types of evaluation, key terms, visual displays, and research articles. Families also learn how to use evaluation information to improve services and the relationship between evaluation and advocacy. The goal of the first course is to prepare family members to use evaluation results for advocacy purposes.
• Level 2, The World of Evaluation: How to Work in It , builds off of the experiences in Level 1. The course teaches families about how an evaluation question is created; the design of an evaluation project; measurement selection; collection, analysis, and interpretation of data; dissemination of findings; and the influences on each phase in the process. Family members who complete this course should be able to effectively participate in and make strategic decisions about their involvement in evaluation projects.
• Level 3, The World of Evaluation: How to Lead It , requires completion of Levels 1 and 2 and will offer skills to family members who chose to lead evaluation projects from start to finish.

The Hillsborough group originally met in August, 2003 during the Level 1 World of Evaluation training. To maintain the enthusiasm and forward momentum initiated during this training, the group continued to meet once or twice a month to continue and expand on the training. These monthly meetings have continued ever since, and the core membership has remained constant over time. The group, originally funded through the local Comprehensive Community Mental Health Services for Children and Their Families Program grant, sought other support as their grant community approached graduation, and they ultimately found a home in the local chapter of the Federation of Families.

The individual members of the Hillsborough Community Evaluation team have diverse experiences and backgrounds. The team is diverse in terms of culture and ethnicity, age and gender, and can therefore offer many different perspectives that enhance the evaluation process. Collectively, they have a depth of knowledge and training in the areas of EQUIPO (a parent professional training curriculum), system of care and wraparound values, outcome measures used locally, and outcome measures required by the national evaluation for the Comprehensive Community Mental Health Services for Children and Their Families Program. The team has a combined experience of over one hundred years in caring for children and youth with serious emotional disturbance. All team members are trained evaluators, experienced professionally and personally as parents and as advocates, and are diverse members of the community. The team prides itself on continually learning, taking an active role in the community, and problem solving.

The team members are parents, biological and adoptive, and relative caregivers of children and youth with serious emotional disturbance. The group includes African-American, Caucasian, and Latino families. The group is diverse in terms of age, including both young parents with young families and more experienced parents who have older youth and young adults. Three team members are Spanish-speaking, which increases the team's capacity to reach out to and engage other families.

The team has discussed the issues of race, class, and power and feels that, although they have not experienced such issues within the Community Evaluation Team itself, they certainly have seen them elsewhere among family members in their local community. The team recognizes that racism, classism, and other unpleasant ‘isms' are very real and will keep them in mind as they continue to work together. The team's common goal of helping kids and families have a good life in the community has helped to keep them focused and get past these types of divisions.

The group expressed a sense that the World of Evaluation courses had caused them to look at the work differently, both in the context of the Community Evaluation Team and in their personal lives. One team member reported that, although her typical approach to the world is at the emotional–or “gut instinct”—level, she now tries to hold her emotions off a little bit more and wait for “what the data tell us”. Another team member expressed a similar opinion. In order to be the best possible advocate for families, it is not enough to have the passion--you must have some sense of “what the numbers mean” and understand that data can add the evidence you need to support your opinion. Another team member commented that, “Advocates provide the impetus for change. Numbers provide the evidence for change.” Finally, in addition to the technical elements of evaluation that the team learned in training (e.g., what a sample is, quantitative and qualitative methods, etc.), the group felt that it had the tools to turn data (e.g., the cold hard facts) into information—that is, the tools that team members and others can use on behalf of families and kids.

The group is still developing several new projects. The following are examples that the group undertook as part of their learning while attending the WOE courses.

Feedback and Refinement Customer Survey for ASO

The local system of care community established an administrative service organization (ASO) to allocate funding for services and supports that are consumer-directed and based on each family's needs and strengths. The Community Evaluation Team reviewed the survey and provided feedback to the ASO manager on how to make the survey more reflective of families' perceptions of what constitutes quality service as well as on how to articulate questions in a more family-friendly way.

Review of Measures Required in the National Evaluation

As an extension of the WOE training, the Community Evaluation Team reviewed specific instruments required by the national evaluation for SOC communities. As part of this review, they examined how to write questions, how to interpret responses to questions, and how to present information obtained from the evaluation instruments. Training also included how to code responses to open-ended questions and other more technical aspects of the evaluation process.

The Community Evaluation Team officially became part of the local Federation of Families chapter in October, 2004. The team is the “evaluation arm” of the local Federation of Families chapter and is responsible for internal evaluation activities to improve the Federation's services and supports to its members. The team has one active contract with a local mental health provider to conduct telephone interviews with families who have left the program. They also have a pending contract with a local collaborative of provider agencies to provide technical assistance in survey design, development of a data collection plan that is sensitive to the families served, data analysis, and report writing.

The Hillsborough Community Evaluation Team is composed of the constituents of the local system of care, and team members have had experience with all child and family serving systems (e.g., child welfare, schools, juvenile justice, Medicaid, etc.). However, the team intends to continually expand the feedback loops from families in the community. To meet this goal, the team must first focus on the design and implementation of family involvement in the local Federation of Families chapter. The local Federation chapter wants to provide the best supports to families and to engage family members to become active participants in the organization. The Community Evaluation Team's task is to help accomplish this goal. The first step will likely consist of forming focus groups comprised of current Federation members in order to understand their perceptions and experiences as Federation members. A survey that will be disseminated to the general Federation membership will be created based on focus group results.

Currently, confidentiality training has taken place, but creating policy related to confidentiality has not occurred at the organizational level of the local Federation chapter. Procedures for ensuring confidentiality are being taken on a project-by-project basis. The topic of developing a confidentiality policy that applies to the local Federation of Families as a whole will be discussed with the Executive Director and interested board members.

The Hillsborough Community Evaluation Team has a great deal of advice for other families. They encourage families to contact them for guidance. The following list contains important pieces of advice for families interested in starting a family driven evaluation team.

• Have a unifying vision. The most important thing for the Community Evaluation Team is that they work together to help the children in their community. Keeping this in mind often helps them get through the “rough patches”.
• Be flexible. Be flexible when scheduling team meetings to allow for maximum participation. This may mean scheduling meetings during the evenings, on weekends, or before the kids get off the school bus on weekdays. No one can make it to every meeting, but people who are interested in keeping up with the community evaluation team will make concerted efforts to attend meetings as often as possible. Provide food at meetings whenever possible, and either assist team members in finding child care, or allow them to bring their children with them.
• Be patient and persistent. Everyone learns in different ways and at different paces. Hang in there, and take the time needed to help each group member to fully understand the concept on which you are working.
• Keep a sense of humor and be willing to laugh at yourself.
• Pay families for their efforts. Families who have completed the WOE training have already made a large commitment (nine days) to acquire a very specialized and valuable set of skills. Providing compensation honors this commitment and dedication, and it shows families that you respect their work. Having said this, there also has to be some kind of mechanism in place to ensure that people who are being paid are contributing to the group's efforts.
• Be professional. Part of being professional means learning to live with disagreement among group members. You must understand that even if your idea or opinion didn't “win” in a given situation it doesn't mean that your voice, ideas, and opinions are not important. People have to comfortable with continuing to speak up, even if the topic of the discussion causes them to be self-conscious. Learning from and adjusting to one another's approach to problem-solving, work ethic, communication style, and mannerisms is an important part of teamwork. Sometimes, team members need to help each other learn how to assume the role of a professional.
• It's a team effort. Although it is important to keep the team mentality in mind, there also must be a leader or a spokesperson since every team member can't be everywhere all the times. Important decision-making, however, is a group effort, and cannot be made by individuals alone. The team as a whole needs to reach consensus and make important decisions. It is the leader or spokesperson's job to communicate the team's decision to others outside of the group.
• Be inclusive, be diverse. The team should represent the many voices of families in the community, including young families, experienced caregivers, members of all ethnic groups, and members of different social classes. Having said this, the group also needs to establish minimum standards for group membership (e.g., completion of all three levels of the WOE training). Implementing standards for group membership will ensure that team members appear credible to those outside the group and will establish team members' strong commitment to the cause. Group members should be ready to help prospective members to complete minimum requirements so that the group can keep growing.
• Continuous learning is key. The evaluation process will be new to your team, so you will all learn from each other on a regular basis. Sometimes you will learn technical aspects of evaluation, and sometimes the learning will involve discovering the group process that works best for the team for successful learning and interaction.
• Take advantage of the World of Evaluation training. The WOE training should be a required foundation course for groups like the Hillsborough Community Evaluation Team. However, it should be considered a starting point for a community evaluation team , rather than just a “one shot deal”. As one group member expressed it, “the more I learn [about evaluation], the more I find I need to learn.”

To learn more about the Hillsborough Community Evaluation Team, please contact Norín Dollard at dollard@fmhi.usf.edu or Dan Casseday at dcasseday@childrensboard.org . To learn more about family leadership in systems evaluation and the Federation of Families for Children's Mental Health's World of Evaluation training courses, please contact Elaine Slaton at Eslaton@ffcmh.org .