The Circles of Care initiative provides funding to plan, design, and assess the feasibility of implementing a culturally appropriate mental health service model for American Indian/Alaska Native children with Serious Emotional/Behavioral Disturbances (SED) and their families. This program is funded by the Federal Center for Mental Health Services (CMHS, part of the Substance Abuse and Mental Health Services Administration, or SAMHSA) with additional support from the Indian Health Service and the National Institute of Mental Health.
It was out of the growing appreciation for the high prevalence of mental health problems among American Indian and Alaska Native children and adolescents, the inadequacy of existing services for these children, adolescents, and their families, and the need for community-based strategic planning to improve these services that Circles of Care was born. CMHS set the following goals for the initiative:
- to support the development of mental health service delivery models that are designed by American Indian/Alaska Native communities to achieve outcomes for their children that they chose for themselves;
- to position tribal and urban Indian organizations advantageously for future service system implementation and development;
- to strengthen tribal and urban Indian organizations’ capacity to evaluate their own service system’s effectiveness; and
- to develop a body of knowledge to assist tribal and urban Indian organizations, and other policy makers and program planners from all child-serving systems in improving systems of care for the American Indian/Alaska Native population overall.
As a part of this effort, CMHS funds two organizations to provide technical assistance to the Circles of Care grantees. The National Indian Child Welfare Association (NICWA) is charged with providing technical assistance in regard to the program development activities of the grantees. The Division of American Indian and Alaska Native Programs (Department of Psychiatry, University of Colorado Denver) is charged with providing technical assistance in regard to the program evaluation activities of the grantees.
The Circles of Care initiative requires an Evaluation Plan that both informs and assesses the strategic planning process. The overarching goals of the Circles of Care Evaluation Plan are as follows:
- to provide a knowledge base for the planning effort;
- to facilitate the process for developing the capacity for ongoing evaluation efforts;
- to examine the feasibility of the service system models; and
- to document and disseminate the results of the initiative.
Thus, evaluation activities are designed to assure that the final service delivery models developed through the Circles of Care initiative are consistent with community needs, developed through community consensus building, and practical and feasible given the resources available.
The Circles of Care evaluation consists of six specific components designed to achieve the four overarching goals noted above: 1) to describe the need for services within each participating community; 2) to develop a locally-relevant definition of SED; 3) to describe the current services available in these same communities; 4) to design a plan for measuring outcomes for the new service delivery model; 5) to assess the feasibility of this model system of care; and 6) to describe and assess the strategic planning process. These components are presented schematically in Figure 1.
Evaluation Component 1: Assessment of Community Needs
In this component of the evaluation, the Circles of Care grantees attempt to answer the following questions. How many children and adolescents suffer from SED in the community? What specific types of difficulties do these children, youth, and families struggle with? What strengths do these children, youth, families, and community at large possess that can be mobilized to address these difficulties? What are community members’ perceptions regarding the availability, accessibility, and acceptability of services?
Evaluation Component 2: Definition of SED
The definition of SED as outlined in the [first] Circles of Care GFA allows grantees to define what kind and level of emotional, behavioral, or mental disability is required for eligibility for services under their strategic plans. This is important for several reasons. First, the GFA anticipated that the term “SED” might be perceived as stigmatizing, and that some communities could be concerned that such labels might impact negatively the future opportunities for children and families who would be served in these new systems of care. Second, some communities might prefer strength-based conceptualizations of need in place of the deficit-based concepts used in standard definitions of SED. Third, it was anticipated that DSM-IV conceptualizations of dysfunction might map poorly onto local conceptualizations of health and illness. Fourth, it was also clear that the definition of the target population would have significant implications for the design of a model system of care. For example, a broadly defined target population would likely require greater capacity within the model systems of care and might require a greater array of services.
Because of the critical importance of characterizing each community’s concept of SED as well as defining the target population for the model system of care, each grantee develops a process for examining the standard definition of SED as described in the GFA and accepting, modifying, expanding, or replacing this definition as appropriate to the community to be served.
Evaluation Component 3: Description and Assessment of the Current Service System
Another critical issue for developing a locally relevant system of care is a careful description and assessment of existing services within the community. In this component of the Evaluation, grantees attempt to answer the following questions: What are the components of the System of Services? What are characteristics of each component? How do the different components of the System of Services interact with each other? How available, accessible, and acceptable are these services? How effective are they? What are the gaps in the existing service system?
Evaluation Component 4: Plan for Measuring Outcomes
While the development of the model system of care is the responsibility of each grantee’s program staff, each grantee’s evaluation staff plays a major role in developing the plan for measuring outcomes for the model system. These plans are designed to enable the grantees to identify the key domains – in terms of individual children, families, and the community – that would be impacted by the model system of care as well as the methods for measuring them.
Evaluation Component 5: Feasibility Assessment of the Service Delivery Model
In this component of the Evaluation, the Circles of Care grantees assess the feasibility of their model systems of care. The goal of the Feasibility Assessment is to assure that each model system of care is well designed with careful consideration of project goals, community resources, and measurable outcomes. The Feasibility Assessment is designed to answer the following questions: Are the needs for services in the community matched with model system of care? Are there adequate human and other resources to bring the plan to fruition? Is the management system appropriate to the service system design? Is the service system design financially sound? Is it economically justified?
Evaluation Component 6: Assessment of the Planning Effort
In this final component of the Evaluation, the Circles of Care grantees monitor, record, and assess the strategic planning effort itself. In completing this assessment, grantees attempt to answer the following questions: What are the accomplishments of the Circles of Care initiative in our community? What steps have we taken to achieve these accomplishments? What types of barriers or obstacles have we encountered? How did we overcome them (or fail to overcome them)? Were people in our community satisfied with our Circles of Care initiative?
As the Circles of Care initiative took form, CMHS sought consultation from the Division of American Indian and Alaska Native Programs (DAIANP, Department of Psychiatry, University of Colorado Denver) regarding the conceptualization and design of the initiative. This led eventually to the DAIANP becoming the national evaluation technical assistance provider to the Circles of Care grantees through an agreement between SAMHSA and the National Institute of Mental Health.
Once charged with providing evaluation technical assistance to the Circles of Care grantees, the DAIANP formed the Circles of Care Evaluation Technical Assistance Center (CoCETAC) and gathered an experienced, interdisciplinary staff of doctoral-level professionals to achieve these goals.
Prior to the awarding of the Circles of Care grants, CoCETAC staff designed an evaluation framework for the initiative as well as a set of specific technical assistance activities to assist the Circles of Care grantees in their evaluation activities.
The CoCETAC Evaluation Framework
CoCETAC developed an evaluation framework based on both the strategic vision described in the Circles of Care GFA as well as the DAIANP’s considerable experience in working with American Indian and Alaska Native communities. First, by carefully reviewing the GFA and considering the principles of strategic planning and program evaluation, CoCETAC identified the six components that would make up the evaluation. Second, CoCETAC developed a set of six goals that guided its evaluation technical assistance activities. These goals are:
- To provide a clear framework for the grantees to use in designing their evaluation efforts. This framework includes the Evaluation Components previously described as well as a concrete set of tasks (including draft and final reports to CoCETAC) and timelines for completing these tasks.
- To encourage the grantees to design an evaluation effort that is most consistent with the priorities of their communities. While the above framework provides for a clear set of tasks and timelines, the specific focus of these tasks is left up to the individual grantees. For example, the grantees are expected to complete a Needs Assessment, but the specific foci for this activity, the methods used to complete it, and the avenues for analyzing and reporting the results are left to the individual grantees. This flexibility is particularly important given the diversity of the participating American Indian and Alaska Native communities and the stated goals of their programs. It is also most consistent with both the Systems of Care philosophy and the Circles of Care GFA that emphasize the importance of community-driven strategic planning.
- To facilitate a process for identifying common domains for each of the evaluation components. The principle of community-driven evaluation plans has to be balanced with CMHS’ charge to DAIANP to pursue a cross-site evaluation of the initiative. There is significant danger that the evaluation efforts of the grantees might be so diverse as to preclude meaningful cross-site analysis. However, an evaluation plan dictated by CoCETAC would have been both risky in terms of its acceptance by the Circles of Care grantees and would likely have stifled potential innovations in evaluation approaches. [To balance these conflicting forces, CoCETAC convened an Evaluator’s Meeting in October, 1999 to discuss the Needs Assessment, definition of SED, the Process Evaluation (Describing the Planning Efforts), and to identify a set of common domains that each grantee would be responsible to address.] For the other evaluation components, CoCETAC provides templates for organizing and reporting key cross-site information that are also discussed at Grantee and Evaluator Meetings and revised per grantee suggestions. Thus, while each grantee is obligated to identify barriers to accessing services in their community as part of their Needs Assessment, the method for identifying these barriers is left to the discretion of the individual programs. Indeed, Circles of Care grantees utilize a wide variety of methods in addressing this domain, ranging from surveys, focus groups, and case studies.
- To assist the grantees in identifying the locally relevant and feasible methodologies to use in their evaluations. The Circles of Care strategic planning process demands an evaluation effort that is substantial and complex. CoCETAC staff believe that there is a considerable risk that grantees will struggle with their evaluation efforts because of its size and intricacy. For example, some grantees might have underestimated the level of effort necessary to complete these activities and failed to allocate the necessary time and resources. In contrast, other grantees might have developed evaluation plans that were too ambitious given the short timeline and limited resources at their disposal. Therefore, CoCETAC’s technical assistance needs to include thoughtful advice regarding the specific nature of each grantee’s evaluation efforts.
- To provide a clear delineation of CoCETAC and grantee roles and responsibilities in the evaluation effort. Given the DAIANP’s reputation for conducting outstanding research with American Indian and Alaska Native communities, another concern was that the grantees would expect CoCETAC to actually conduct their evaluations of and for their programs, which was well beyond its Scope of Work. Therefore, the CoCETAC provides a clear delineation of CoCETAC and grantee roles and reinforces these guidelines in technical assistance activities.
- To provide consistently high quality technical assistance through a specific set of activities. The DAIANP shares the excitement of CMHS, its partner Federal agencies, and its Advisory Board regarding the significance of the Circles of Care initiative. CoCETAC staff are dedicated to providing outstanding technical assistance to the Circles of Care grantees. This includes building a staff with the necessary expertise to provide high quality technical assistance, an intention to be readily available to the grantees to assist them as needed, and a deep sense of commitment to the goals of the initiative.
CoCETAC Technical Assistance Activities
CoCETAC utilizes a variety of technical assistance activities that are aimed at achieving the goals outlined above. As anticipated in the Scope of Work, the complex nature of the evaluation effort requires careful orientation to the evaluation goals and components, attention to each grantee’s priorities, coordination of efforts across grantee communities, and ongoing feedback and troubleshooting. Rather than rely on a single activity to achieve the goals for the Circles of Care evaluation, CoCETAC employs the following menu of activities.
- Grantee Meetings. In many ways, the Grantee Meetings are the core of CoCETAC’s technical assistance activities. CoCETAC uses didactic sessions to orient the grantees to each of the evaluation components as well as evaluation-related tasks (such as working with Institutional Review Boards). These didactic sessions are invariably followed by a series of facilitated large and small group discussions (which include program staff, evaluators, and family representatives) aimed at increasing the grantees’ appreciation of the nature of these evaluation efforts, including both the specific expectations and individual flexibility embedded in these activities. CoCETAC also facilitates special sessions for evaluators that allow the grantees to share their experiences with each other and problem solve as a group around difficult issues that arise during their work. The Grantee Meetings are convened jointly by CoCETAC and NICWA, the program development technical assistance provider, with the goal of demonstrating and conveying the important relationships between evaluation and program development for this initiative.
- Evaluator Meetings. While not envisioned in CoCETAC’s Scope of Work, the need for meetings focusing specifically on evaluation became quite apparent. [These meetings consist mostly of large-group discussions that focus on developing a common set of domains that each of the grantees is expected to address in their evaluation efforts.] These meetings also provide additional opportunities for discussion of exemplary evaluation practices and group problem solving. While intended primarily as an opportunity for the evaluators to meet as a group, the vast majority of grantees send at least one representative from their program staff to participate. Representatives from CMHS, IHS, and NICWA participate in these meetings as well.
- Site Visits. While the Grantee and Evaluator meetings are the primary avenues for group-oriented technical assistance to the grantees, the annual site visits to the grantees is the key activity for engaging the grantees in intensive, individualized discussions. Topics covered in these 1½ day visits include survey development, conducting focus groups and case studies, database development, data analysis and interpretation, as well as strategies for building support for the initiative and its evaluation effort among families, youth, service providers, and tribal leadership.
- Scheduled Technical Assistance Conference Calls. Every month, CoCETAC’s Associate Director conducts scheduled conference calls with each grantee’s Program Director and Evaluator. These calls are another opportunity for intensive, individualized discussions with grantees, and assists the grantees in addressing specific evaluation issues as they arise, tracking progress, and identifying issues requiring further discussion.
- Ad Hoc Conference Calls and Email Exchanges. In addition to the scheduled conference calls, CoCETAC staff conduct a number of ad hoc conference calls and email exchanges to address a variety of issues that arise during the grantees’ evaluation activities.
- Detailed Reviews and Critiques of Evaluation Reports. During the course of their programs, all of the grantees submit draft reports that are reviewed in detail by CoCETAC staff. This provides the opportunity for CoCETAC to help the grantees further refine their evaluations and tailor their reports for both local and national use.
- Referral to and Provision of Key Reference Materials. CoCETAC assists the grantees in identifying reference materials that support their evaluation activities. In addition, CoCETAC provides copies of many key reference materials that are highly relevant to all grantees.
This wide variety of evaluation activities allows CoCETAC to meet the diverse evaluation technical assistance needs of the Circles of Care grantees and supports their efforts to produce results relevant to their strategic planning efforts.