EVALUATION OF THE IMPLEMENTATION OF THE TRIPLE P POSITIVE PARENTING PROGRAM IN PITT COUNTY by Melissa Francine Nolan November, 2014 Director of Thesis: Eboni Baugh, PhD Major Department: Child Development and Family Relations Participants were 14 mid-level supervisors from agencies who currently have providers trained in the Triple P Positive Parenting Program (Triple P). The present study evaluated the implementation of the Triple P in Pitt County, North Carolina, as well as assessed whether or not an adapted framework could be used to evaluate the implementation of the Triple P Positive Parenting Program. Data were collected using Qualtrics an online survey program. Using an adapted framework a self-created survey was distributed to the participants through email. Overall results suggest the implementation of Triple P was a successful initiative in Pitt County. Results also suggest providers may benefit from flexibility and integration workshops to increase implementation. The findings also suggest the adapted framework can be used to evaluate effective implementation of the Triple P Positive Parenting Program. Evaluation of implementation is essential in establishing key factors that contribute to effective implementation, as well, as finding ways to achieve sustainability of an intervention. Keywords: triple p, effective implementation, preventative interventions, parenting programs, behavioral interventions, evidence-based, ecological theory, evaluation, public health EVALUATION OF THE IMPLEMENTATION OF THE TRIPLE P POSITIVE PARENTING PROGRAM IN PITT COUNTY A Thesis Presented To the Faculty of the Department of Child Development and Family Relations East Carolina University In Partial Fulfillment of the Requirements for the Degree Master’s of Science Child Development and Family Relations by Melissa Francine Nolan November, 2014 © Melissa Francine Nolan, 2014 EVALUATION OF THE IMPLEMENTATION OF THE TRIPLE P POSITIVE PARENTING PROGRAM IN PITT COUNTY by Melissa Francine Nolan APPROVED BY: DIRECTOR OF THESIS: ______________________________________________________________________ (Eboni Baugh, PhD) COMMITTEE MEMBER: ________________________________________________________ (Sharon Ballard, PhD) COMMITTEE MEMBER: _______________________________________________________ (Lisa Tyndall, LMFT) COMMITTEE MEMBER: _______________________________________________________ (Kerry Littlewood, PhD) CHAIR OF THE DEPARTMENT OF Child Development and Family Relations: _________________________________________ (Sharon Ballard, PhD) DEAN OF THE GRADUATE SCHOOL: _________________________________________________________ Paul J. Gemperline, PhD ACKNOWLEDGEMENTS I would like to thank Lynne Carter from the Pitt County Health Department for her assistance in contacting the participants for the current study. I would also like to thank Dr. Eboni Baugh, Dr. Sharon Ballard, Dr. Kerry Littlewood, and Dr. Lisa Tyndall for their guidance and unwavering support through this process. The example you set for me has carried me through this journey. Finally, I would like to thank my parents for supporting my decision to continue my education and encouraging me to be the best I can be. TABLE OF CONTENTS TITLE PAGE .................................................................................................................................. i COPYRIGHT .................................................................................................................................. ii SIGNATURE PAGE ..................................................................................................................... iii ACKNOWLEDGEMENTS ........................................................................................................... iv LIST OF TABLES ....................................................................................................................... viii LIST OF FIGURES ....................................................................................................................... ix CHAPTER 1: INTRODUCTION ....................................................................................................1 CHAPTER 2: LITERATURE REVIEW .........................................................................................4 What is Preventative Intervention ........................................................................................4 Triple P Positive Parenting Program....................................................................................5 Framework for the Evaluation of Implementation ...............................................................8 Systems Affecting Implementation....................................................................................12 Prevention Delivery System ..............................................................................................12 General Organizational Factors .............................................................................13 Specific Practices and Processes ............................................................................15 Specific Staffing Concerns ....................................................................................16 Prevention Support System ................................................................................................17 Training ..................................................................................................................18 Technical Support ..................................................................................................19 Characteristics of the Intervention .....................................................................................20 Compatibility .........................................................................................................20 Adaptability............................................................................................................21 Provider Characteristics .....................................................................................................22 Recognized Need and Benefit of Preventative Intervention ..................................22 Self-Efficacy and Skill Proficiency of Provider ....................................................23 CHAPTER 3: METHODS .............................................................................................................27 Participants/Sample............................................................................................................27 Procedure ...........................................................................................................................30 Measures ............................................................................................................................30 CHAPTER 4: RESULTS ...............................................................................................................32 Data Analysis .....................................................................................................................32 Provider Characteristics .....................................................................................................32 Characteristics of the Intervention .....................................................................................32 Prevention Delivery System ..............................................................................................32 Prevention Support System ................................................................................................33 Correlation of Variables .....................................................................................................34 CHAPTER 5: DISCUSSION .........................................................................................................36 Provider Characteristics .....................................................................................................36 Characteristics of the Intervention .....................................................................................37 Prevention Delivery System ..............................................................................................38 Prevention Support System ................................................................................................40 Correlation between Systems .............................................................................................41 Limitations .........................................................................................................................42 Future Research .................................................................................................................43 Conclusion .........................................................................................................................44 REFERENCES ..............................................................................................................................46 APPENDIX A: INSTITUTIONAL REVIEW BOARD APPROVAL ..........................................50 APPENDIX B: IMPLEMENTATION SURVEY .........................................................................52 LIST OF TABLES 1. Systems of Implementation, Corresponding Factors of implementation, and Triple P Characteristics ........................................................................................................... 26 2. Participant Demographics .......................................................................................... 29 3. Correlation Table of Intervention Characteristics, Administrative Support, and Provider Characteristics ............................................................................................................ 35 LIST OF FIGURES 1. Adapted Ecological Framework for Evaluating Implementation .............................. 11 2. Agencies Administrative Hierarchy Including Mid-Level Supervisors .................... 28 CHAPTER 1: INTRODUCTION Public health preventative interventions have been widely used to improve health outcomes for individuals and families. There are a number of studies that aim to evaluate the effectiveness of implementation of these public health preventative interventions. (Kallestad & Olweus, 2003; Riley, Taylor, & Elliot, 2003). These evaluations of found these programs effective as a whole. However, the public health approach has only recently been studied with preventative parenting interventions (Prinz & Sanders, 2007). A public health approach to child neglect and maltreatment may be the most effective approach in preventative parenting interventions because it increases the reach of the intervention to ensure total community access (Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009; Sanders, 2008). Positive outcomes in these parenting interventions are influenced by the implementation of these programs making it imperative to evaluate the implementation process (Durlak & DuPre, 2008). The evaluation of implementation can determine factors that contribute to effective implementation and sustainability of a preventative intervention, as well as barriers that may impede the process (Shapiro, Prinz, & Sanders, 2014). The purpose of the current study is to evaluate the factors that influenced the implementation of the Triple P Positive Parenting Program (Triple P) in Pitt County, NC. Studies Triple P have been mostly targeted the perspective of providers (Shapiro, Prinz, & Sanders, 2012; Shapiro, et al., 2014). The current study is the one of the few known quantitative evaluation of factors contributing to effective implementation of Triple P using an adapted framework for preventative interventions from the perspective of mid-level supervisors (Durlak & DuPre, 2008). Surveying mid-level supervisors provides a better idea of the organizational processes that comprise effective implementation. 2 The Ecological Framework (Brofenbrenner, 1977) is often used and adapted to help explore the implementation and adoption of preventative intervention programs (Ballard & Taylor, 2012; Durlak & DuPre, 2008). Using this framework, Durlak and DuPre (2008) reviewed 500 quantitative studies evaluating preventative intervention programs. As a result of the review, the researchers suggested 23 contextual factors determined to significantly influence the implementation of a prevention program. Using 19 of the 23 contextual factors suggested by Durlak and DuPre (2008), the researcher used an adapted framework to create a quantitative measure to assess the implementation of the Triple P Positive Parenting Program in Pitt County, North Carolina (NC). The Durlak and DuPre (2008) framework examines implementation and adoption of a prevention program as an organizational process that requires all levels of a system to function efficiently. These systems include: Provider Characteristics, Intervention Characteristics, the Prevention Delivery System, and the Prevention Support System (Durlak & DuPre, 2008). The methodology used for the current study was developed using items which represented each of the contextual factor that contribute to four systems. With the intent of understanding implementation of Triple P in Pitt County, NC factors that affected implementation were examined using an adapted framework for evaluation of implementation of preventative interventions. While previous research has focused primarily on factors affecting implementation from the perspective of the provider, the current study focused on factors from the perspective of the mid-level supervisor. A mid-level supervisor allows the researcher to get a view from the “battlefield”. To use supervisors higher than mid-level (i.e. CEOs, presidents, board members, etc.) would be an unrealistic view of everyday agency functioning. By taking the perspective of the supervisor, the goal is to achieve a richer 3 understanding of all systems that contribute to the organizational processes which aid in successful implementation. CHAPTER 2: LITERATURE REVIEW This is based on a systematic review of the literature related to evaluation and implementation of public health interventions, the Triple P Positive Parenting Program, and the ecological framework applied to evaluation. The factors influencing implementation are identified within the systems they represent. Databases including Google Scholar, JStor, ERIC, PsychINFO, MEDLINE, SocINDEX and EBSCO were searched (1981 to 2014) using the following key words: public health, Triple P, implementation, evaluation, dissemination, parenting programs, prevention, intervention, systems contextual approach, ecological framework, parenting education and evidence-based. What is a Preventative Intervention? A preventative intervention is characterized as an intervention designed to intervene before a full-blown problem exists (Durlak & Wells, 1998). Preventative interventions exist on a continuum and are placed in one of three categories: universal, selective, and indicated (Mrazek & Haggerty, 1994). A universal preventative intervention is for everyone in a population regardless of risk or protective factors that may or may not be present (Mrazek & Haggerty, 1994). A selective preventative intervention is when a subgroup of a population has above average risk for development of a specific disorder or problem (Mrazek & Haggerty, 1994). Finally, an indicated preventative intervention is for the subgroup of a population who are at high risk for development of a disorder or problem (Mrazek & Haggerty, 1994). In any form, preventative interventions are intended to be implemented in populations before the development of problems or disorders. The goal of a preventative intervention is to identify protective factors and risk factors in order to prevent a fully developed epidemic or problem or to reduce the duration and impact of the developing disorder (Durlak & Wells, 1998). 5 A preventative intervention takes a population perspective; the needs of a particular population are assessed using previously identified risk and protective factors for that specific population. Once the need is determined, it is rapidly followed by implementation of a preventative intervention (Durlak & Wells, 1998). The expectation is that implementing an intervention during the early stages of a developing problem can result in preventing later more serious dysfunction (Durlak & Wells, 1998). The Triple P Positive Parenting Program is a current preventative intervention being offered in all three categories (universal, selective, and indicated) to parents in Pitt County with the goal of reducing child behavior problems, child maltreatment, and increasing parental self-efficacy. Triple P Positive Parenting Program The Triple P Positive Parenting Program (Triple P) is a tiered, multilevel evidence-based program (EBP) designed to aid parents in preventing severe emotional, developmental, and behavioral problems in children (Sanders, 1999). EBP refers to programs that are well defined, have been peer-reviewed for efficacy, and have been endorsed by a government agency or other established, respected organizations (Small, Cooney, & O’Connor, 2009). Triple P takes a population level, public health approach to dissemination or distribution. The intervention builds on the knowledge and skills of parents with the goal of increasing their confidence in their own parenting ability and subsequently reducing child maltreatment and child discipline issues (Sanders, 1999). Triple P programming is available for all parents with children ages 2-12, on five levels which increase in intensity with each level (Sanders, 1999). Sanders (1999) recognized that families have differing needs; this ranges from the challenges children may present to the knowledge and skills parents already possess. Triple P also acknowledges that the most effective 6 delivery method for parents can also vary (Sanders, 1999). From this idea, a tiered, five level programmatic approach evolved. From the perspective of agency supervisors, a multi-level approach allows for efficiency while reducing cost and ensuring the intervention is appropriate for all types of families (Sanders, 1999). Triple P providers are able to assess parents’ needs and determine what level would be most effective in achieving parents’ goals. As previously stated, Triple P is disseminated with a multi-level approach to ensure families receive the appropriate dosage. Level 1 of Triple P is a universal marketing campaign to provide parents with parenting information that would be of use to them through tip sheets, social media, TV, radio, and print ads (Sanders, 1999). The goal of Level 1 is to create awareness in the community of the available resources for parents as well as gauge the need of the parents in the area. An additional goal of Level 1 is to begin to “soften” parents to the idea that there are easy solutions to their children’s common behavior and development issues (Sanders, 1999). As designed, Level 2 (selective) increases in strength by offering strategies for parents that guide the development of children who exhibit minor behavior issues (Sanders, 1999). Primary health care providers typically conduct one to two 2-hour sessions with a group of parents in the form of a seminar. Level 3 (selective), delivered individually, increases to four sessions and is designed to meet the needs of parents with children that have mild to moderate behavior challenges (Sanders, 1999). Level 4 (indicated) is a more intensive approach, requiring anywhere from eight to ten sessions. The sessions can be conducted one-on-one or in groups, and is best suited for children who are exhibiting severe behavior problems. Finally Level 5 (indicated), the most intensive of all, is offered to families whose parenting challenges are further complicated by other sources of family dysfunction such as, problems with marital 7 communication, lack of stress-coping skills for parents, and issues with mood management (Sanders, 1999). In 2005, the Centers for Disease Control and Prevention (CDC) and the National Center for Injury Prevention and Control funded a trial study of the outcomes from Triple P implementation for eighteen counties in South Carolina (Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009). The results of the trial found that after the implementation of Triple P, the participating counties reported lower rates of substantiated child abuse cases, out-of-home child placements, and reductions in hospitalizations and emergency room visits for childhood injuries (Prinz, et al., 2009). Given the positive results from this trial, the Centers for Disease Control and Prevention (CDC) set out to answer three additional questions: Do agencies have the ability to implement Triple P through partnerships? What is the most effective way to do so? And is the target population reached through these partnerships? In August 2011, Pitt County, NC was one of two counties in the U.S. to be awarded a grant for the implementation of Triple P. The Pitt County Health Department (PCHD) in conjunction with the James D. Bernstein Health Center in Pitt County, NC were responsible for managing the grant. The grant, funded by a partnership between the Centers for Disease Control and Prevention (CDC) and the National Association of County and City Health Officials (NACCHO), was to enable investigation and trial implementation of Triple P to answer the three questions previously posed regarding partnership implementation, effective ways to implement through partnership, and assessment of population reach through a partnership. The project rolled-out over a three-year period with the first year dedicated to training the maximum number of providers in the appropriate underserved areas. Years two and three focused on public 8 awareness and implementation, as well as, data collection and evaluation. At the time of the current research, the Triple P implementation project is in its final months and reports of implementation reach are being disseminated. East Carolina University (ECU) was contracted to assist in evaluating the reach of the Triple P Positive Parenting Program in Pitt County. The researcher of the current study was employed as the Graduate Assistant for the evaluation team and was responsible for obtaining data from the providers implementing Triple P and then entering the data into statistical software. During this process, it became evident to the researcher that there was a need to examine what facilitated and prohibited the implementation process. Framework for the Evaluation of Implementation Durlak and DuPre (2008) suggested a framework for effective implementation when evaluating the capacity of a preventative intervention. Capacity is defined as the overall process of dissemination; motivation behind choosing an intervention, the ability to choose, plan, and implement the intervention, and the skills to evaluate and sustain the chosen intervention. A population-level approach involves multiple environments in which the program is implemented, suggesting a multi-level ecological perspective is necessary when evaluating effective implementation related to capacity (Altschuld, Kumar, Smith & Goodway, 1999; Durlak & DuPre, 2008; Richard, Lehoux, Breton, Denis, Labrie & Leonard, 2004; Riley, 2003; Shediac- Rizkallah & Bone, 1998; Sanders & Kirby, 2012; Sanders & Prinz, 2008; Shapiro, et al., 2012; Turner & Sanders, 2006; Wandersman & Florin, 2003). The ideas and concepts informing the suggested framework have their roots in the Ecological Theory (Brofenbrenner, 1977). The Ecological Theory, proposed by Urie Brofenbrenner (1977), is traditionally used to examine the effect between a developing human and the ever-changing systemic environment in 9 which he/she exists, in addition to the relationships that are present within and between each system. The systems in the Ecological Theory are suggested to be arranged in a nested structure within each other moving from the “innermost” system to the “outermost” system. The “innermost” system is proposed to have a most significant or direct effect on development. In the context of programming aimed at family needs, the ecological theory for effective implementation is further supported by Ballard and Taylor’s (2012) proposed framework for best practices in family life education. Ballard and Taylor’s framework recognizes the provider, the program content and program design are interrelated and interdependent and a balance of all components is required to be effective in implementation (Ballard & Taylor, 2012). This framework suggests it is necessary to consider all variables present in the environment to be effective when implementing family life education programs, not unlike Durlak and DuPre’s (2008) framework. To determine the factors associated with each of the five systems identified, Durlak and DuPre (2008) conducted a review of articles that contained data on factors affecting implementation. Common factors found in at least five articles were investigated further and if they were still found to be consistent they were suggested to be related to effective implementation (Durlak & DuPre, 2008). Durlak & DuPre (2008) used the studies with larger sample sizes and more psychometrically sound measures to confirm the inclusion factors of themes in the review of quantitative studies. For qualitative studies the researchers used the articles that included multiple cases studies, prospective design, and multiple methods of data collection to confirm factors. Through this process, the researchers were able to identify factors affecting the implementation. The researchers then placed the suggested factors within 5 systems: 10 Community Factors, Provider Characteristics, Characteristics of the Intervention, Prevention Delivery System, and the Prevention Support System. For the purpose of the current study, the researcher will be examining 4 of the 5 systems: Intervention Characteristics, Provider Characteristics, Prevention Delivery System, and Prevention Support System. The fifth system, Community Level, will not be assessed because the researcher is only focusing on factors that are within the agency’s control. Figure 1 depicts the adapted framework in the context of program implementation. 11 Figure 1. Adapted ecological framework for evaluating implementation. Adapted from “Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation” by J. Durlak and E. DuPre, 2008, American Journal of Community Psychology, 41, p. 335. Provider Characeristics Internvetion Characteristics Prevention Support System Prevention Delivery System Effective Implementation 12 The systems described by Durlak and DuPre (2008) are Provider Characteristics (skill proficiency, self-efficacy, etc.), Intervention Characteristics (adaptability, compatibility), Prevention Delivery System (organizational capacity, staffing, etc.), and Prevention Support System (in-service training, resource access, etc.).The Prevention Delivery System (organizational capacity) and the Prevention Support System (training & technical assistance) have the most direct influence on effective implementation. In order to implement an intervention successfully, there must be some form of organizational structure responsible for facilitating implementation, as well as, support in place to assist with training and technical barriers (Durlak & DuPre, 2008). In addition to the direct effect the Prevention Delivery System and the Prevention Support System have on effective implementation, factors in three additional systems interact to influence effective implementation: Intervention Characteristics, Provider Characteristics, and Community Factors (Durlak & DuPre, 2008; Breitkreuz, McConnell, Savage & Hamilton, 2011; Shapiro, et al., 2012; Turner & Sanders, 2006). The inclusion of the three additional systems provides the basis for the suggested extended ecological framework (Durlak & DuPre, 2008). Systems Affecting Implementation Prevention Delivery System The Prevention Delivery System refers to factors related to organizational capacity such as the infrastructure in which the intervention is going to exist. In combination with the Prevention Support System, the Prevention Delivery System has the most direct effect on implementation (Durlak & DuPre, 2008). General organizational factors, specific practices and processes, and specific staffing considerations are included in the Prevention Delivery System. Implementation of a preventative intervention should be a collaborative effort encouraged and 13 supported by the agency environment and staff in order to increase effectiveness. The Triple P model has a structure as to allow flexibility while maintaining fidelity (Sanders & Kirby, 2014). The flexibility of Triple P allows for the program to be easily adapted according to the agencies organizational structure. General Organizational Factors. When addressing general organizational factors the intent is to assess work climate, organizational norms towards change, history of integrating interventions, and the shared vision between the goal of the intervention and the mission/goal of the agency (Durlak & DuPre, 2008). Agencies see more success with implementation when they are well established, have a prior history with programming, and are embedded within their community (Breitkreuz, McConnell, Savage & Hamilton, 2011). When an agency is a community fixture, clients have had an opportunity to develop a sense of trust with the agency and more than likely have seen success come out of the agency. A strong pre-existing provider- client relationship allows providers to justify Triple P to clients more effectively (Breitkreuz, et al., 2011). Having a history with programming in general can also help with program implementation. The familiarity with the process of implementing new programming may result in increased confidence with implementation. In addition, providers have an ability to anticipate some of the barriers they might face when implementing an intervention. When Asgary-Eden and Lee (2012) evaluated implementation of Triple P in Canada they found providers who perceive that their agency encourages a supportive environment for implementation, as well as those who view the supportive environment as an asset, are more likely to implement an intervention with their clients. This idea was further supported by a study exploring the implementation of a tobacco preventative intervention in schools. The results 14 showed that teachers who reported a warm, supportive work environment were less likely to resist the adoption of the intervention in their classroom (Barr, Tubman, Montgomery, & Soza- Vento, 2002). Furthermore, if during implementation providers have the ability to discuss clients with other providers, receive consultations from outside collaborators, and have the option for supervisors to sit in on sessions, program use increases (Breitkreuz, et al., 2011; Shapiro, et al., 2012). These findings suggest that as agencies adopt new programming, providers benefit from various levels of support to help integrate the intervention into already existing programming. Having the support of an administrator helps providers develop a sense of security needed to try a new program or intervention. This may be because providers feel more comfortable and welcome to go to their superiors for help and/or they may have felt encouraged to voice their concerns and problems faced during implementation. Durlak and DuPre (2008) also suggested agencies that are forward thinking and consistently trying new approaches are more likely to be effective in implementing an intervention. Agencies who are consistently attempting new approaches may have an infrastructure that is conducive to integrate new programming. These agencies already have a system in place that assists providers in implementing new programming (Shapiro, et al., 2010). The agencies may have already addressed all the barriers that arise when implementing new programming. When evaluating a preventative intervention program on bullying, the feeling of open communication accompanied with an organization’s positive attitude towards change increased likelihood of implementation (Kallestad & Olweus, 2003). This further supports the idea of the strong role that flexibility in leadership plays in effective implementation. Previous research suggested that if a new intervention cannot be easily integrated into an agency’s already heavy caseload, then it is less likely to be used (Breitkreuz, et al., 2011; Durlak 15 & DuPre, 2008; Sanders & Turner, 2005; Shapiro, et al., 2012). This idea emphasizes the importance of assessing all necessary components of implementation before choosing an intervention. Agencies must take into account the time and resources it will require to incorporate a new intervention, as well as planning and preparation for success (Shapiro, et al., 2012). If agencies are unable to provide the time and work force then implementation may consequently suffer. In addition to easy integration, if an agency or organization is already using other resources similar to the intervention being introduced, then the agency was more likely to refer to the intervention as a viable option (Breitkreuz, et. al., 2011; Shapiro, et al., 2012). When an intervention is similar to what agencies are already doing, there is not much change that is required in order to implement, leading providers to feel more comfortable implementing. In addition to similarity, it is easier to implement if the provider already has positively established a relationship with the potential client (Breitkreuz, et al., 2011; Shapiro, et al., 2012). Specific Practices and Processes. In order to determine factors related to specific practices and processes, Durlak and DuPre (2008) examined decision-making processes, coordination with other agencies, communication, and delegation of responsibilities related to implementation. By looking closer at the preceding areas, the researchers determined that shared decision making, collaboration, communication, and formulation of tasks were the specific practices and processes of most importance when determining effective implementation. Research suggests that by collaborating with all relevant parties, including community members, agencies encourage an environment of collaboration within themselves (Durlak & DuPre, 2008; Shapiro, et al., 2010). Collaboration with outside agencies and community members presents the advantage of bringing in other perspectives and skills. As a result of 16 collaborating with individuals outside of their own agency, providers and community members may feel a sense of ownership towards the intervention resulting in a higher possibility of effective and sustained implementation (Durlak & DuPre, 2008). Employees’ confidence in implementation also increases when collaboration between agencies is present. Along with shared decision making and collaboration with other agencies, introducing a proper procedure to encourage open communication among the providers, further assists with effective implementation (Durlak & DuPre, 2008). This open communication includes confirmation that providers understand that their roles and responsibilities in the implementation process contributes positively to successful implementation (Durlak & DuPre, 2008). In addition, this provides an opportunity for administrators to make any clarifications needed for the provider to be successful in implementation. The designation of roles and responsibilities also can allow for strategic planning in terms of current procedures and protocols that may need to be adjusted during adaption of a new intervention. Specific Staffing Concerns. When implementing a new preventative intervention, it is important to evaluate the amount and quality of staff within the agency. Prior to adopting the preventative intervention, evaluation of staff must occur as it ensures the agency has an adequate number of people in place to allow for flexibility and selectiveness when choosing who is involved in implementation. It also allows for more effective implementation by identifying which employees have greater potential for success when implementing (Asgary-Eden & Lee, 2012; Shapiro, et al., 2014). Evaluating the staff will allow the agency to appoint a program champion and maintain workload after integrating the intervention (Durlak & DuPre, 2008; Sanders & Kirby, 2014). The program champion is defined as the internal advocate for the new intervention (Durlak & DuPre, 2008). This individual should be well-liked by their peers and 17 administrators. They also should be highly visible within the agency, so they are able to encourage support for the intervention, as well as, troubleshoot problems that arise when leadership is not present (Turner & Sanders, 2006). Durlak and DuPre (2008) suggested that effective leadership is one of the most important predictors of effective implementation. Leadership determines priorities within the agencies, gives providers’ opportunities to share experiences with one another, and motivates and encourages the staff to implement and try new ideas and strategies. Organizational leaders oversee the entire process of implementation, thereby having one of the most powerful voices for change (Durlak & DuPre, 2008). If an agency is struggling to retain or hire staff, then they are going to be less likely to implement a newly introduced intervention (Breitkreuz, et al., 2011; Durlak & DuPre, 2008). Breitkreuz et al., (2011) suggested that staffing concerns seemed to be the biggest struggle to implementation with sites in remote locations. In instances where staff turnover rate is high, agencies are losing qualified providers and are not able to train new employees. This forces the agency to put the intervention aside to focus on staffing concerns. Consequently, it becomes too costly for agencies to continue to offer specific interventions. The struggling agencies continuously have to pay to have more employees trained who may or may not remain with the agency. Prevention Support System Once an agency has begun the process of implementation of a preventative intervention, post-training and follow-up education should be made available. The availability of these support resources contribute to achieving effective implementation (Durlak & DuPre, 2008). Training and technical assistance holds importance for a number of reasons. First, the intent of the support system is to not only give the provider the knowledge and strategies to implement the 18 intervention, but also to allow for the trainer to address issues regarding expectations of the intervention during the implementation process (Durlak & DuPre, 2008). Training also allows for an outside individual to motivate providers to continue implementing and to foster a sense of self-efficacy (Durlak & DuPre, 2008). Training. The goal of training is to educate the providers and equip them with the skills to effectively implement the intervention. Training should also address motivation, expectations, and self-efficacy therefore continued training after implementation is imperative. Shapiro, Prinz, and Sanders (2012) suggests that as time passes from initial training, the rate of use of an intervention may decrease, especially if continued training does not occur, because of a decrease in confidence of the provider’s ability. Implementation of Evidence-Based Programs require a significant amount of time spent on learning and reinforcing new skills, therefore if outside training is not available to providers, implementation can suffer (Scherer, Kohl & Bellamy, 2010). In an examination of the implementation of parenting interventions in a mid-western town, researchers found that even though providers were reporting frequent training, the training was internal and the quality of the training was unknown which brought into question fidelity (Schurer, et al.2010). When implementing an evidence based program doing so with fidelity, ensures the model is not compromised and predicted outcomes can be achieved. Durlak and DuPre’s (2008) framework suggests continued training during implementation increases effectiveness, however, it is important to note the training must be of high quality. In addition to the passage of time, the quality of training or trainer prior to implementation can determine the use of an intervention. One study found that when providers described their pre-implementation trainer using words like, “rigid” and “inflexible”, they were less likely to implement the intervention with their clients when they returned to their agency 19 (Breitkreuz, et al., 2011). A possible explanation for this could be that the providers associated those same characteristics of the trainer with the characteristics of the intervention itself. Technical Support. Technical support refers to all the resources that are available to agencies once implementation begins. The intent of providing technical assistance is to maintain motivation and commitment in providers, to provide opportunities to improve upon the providers’ skill level, and to assist in solving problems that may arise during the implementation process (Durlak & DuPre, 2008). After training and during implementation, interventions were more likely to be implemented consistently if providers felt they were supported and had access to resources (Sanders & Prinz, 2008; Shapiro, et al., 2014). When discussing implementation of Triple P, providers expressed concern with the process and procedure of securing tip sheets for their clients (Breitkreuz, et al., 2011). Inaccessible or hard to come by resources were found to intimidate providers enough to reduce the use of the intervention even if they thought it was helpful, while user-friendly materials and quick access increased ease of implementation (Breitkreuz, et al., 2011). Not only was it helpful to know resources were easily accessible, but if providers felt the support staff at the agency were also knowledgeable in the intervention, the implementation was positively influenced (Shapiro, et al., 2010). Providers felt more supported and the staff was better equipped to answer potential client questions thereby referring them to the appropriate provider. Ensuring the entire agency is knowledgeable regarding the intervention also may increase community awareness (Durlak & DuPre, 2008). This is important for potential clients who may call the agency inquiring about said intervention. If support staff is not properly 20 trained in the basic information for the intervention then this may lead to a client receiving inaccurate information causing the intervention to not be implemented to the right population. Characteristics of the Intervention Intervention Characteristics take into account the adaptability or flexibility of an intervention. When examining intervention characteristics, the degree of compatibility an intervention has with an agency is essential to effective implementation (Durlak & DuPre, 2008). Interventions that take the as-is approach are less likely to be implemented than those that leave room for modification to fit the needs of the client, provider, and community (Durlak & DuPre, 2008). Also taking into the account the mission and goal of the agency when choosing an interventions will promote effective implementation. Compatibility. When assessing for compatibility, determining the levels of contextual appropriateness, fit, match, and congruency is the intent (Durlak & DuPre, 2008). These factors indicate that the perception of working towards a common goal can benefit agencies when choosing, implementing, and sustaining an intervention. Providers reported difficulty adapting to an intervention because their previous training and education contradicted with the theory of the intervention they were asked to implement (Breitkreuz, et al., 2011). Specifically, behavioral modification strategies that were suggested for use conflicted with what they were taught regarding theories of attachment and development (Breitkreuz, et al., 2011). These particular providers were uncomfortable using parenting techniques such as time-outs or allowing a child to cry, resulting in them being less likely to use the particular intervention. When strategies presented in an intervention are not in line with the theoretical framework practiced by the provider, then the provider may not be able to effectively implement. 21 Some providers found it easier to integrate the program if they were able to combine Triple P material with other parenting interventions already used in their agency (Breitkreuz, et al., 2011). When providers are able to combine an intervention with other practices already being used it allows them to view the program as yet another “tool” in their “toolbox”. The perception for the provider is they are just adding to what they already do as opposed to changing their current practices. Adaptability. Adaptability (flexibility) is controversial because EBPs require an adherence to a specific script using specific resources, processes, and strategies. This concept is commonly referred to as program fidelity. However, Durlak and DuPre (2008) proposed that interventions that allow providers to make minor adjustments to fit the needs of the agency and the needs of their clients provide better outcomes and were more likely to be implemented. Effective implementation is increased when an intervention takes into account the cultural norms of the population being served (Ballard & Taylor, 2012; Durlak & DuPre, 2008). The implementation of preventative programming can be more effective when you start building on skills that are already present. Specific cultural groups and populations have strengths unique to their group. What individuals find acceptable and not acceptable is determined by their family and their cultural norms. Taking into account these strengths and making adjustments accordingly may result in increased success of an intervention because the participants feel the intervention is tailored towards them (Ballard & Taylor, 2012). For example, simplifying the language in Triple P helped meet the needs of English as a second language clients (Breitkreuz, et al., 2011). This flexibility can allow for more effective implementation. According to Mazzucchelli and Sanders (2010) one of the most common reported reasons for not implementing an intervention is the providers’ resistance to following a 22 script and a set manual. The Triple P model is designed to allow for flexibility to ensure providers can meet the felt needs of their population. Some examples of modifications that can be made to Triple P while maintaining fidelity are, providing more sessions than originally designated, conducting sessions over the phone or at the parents workplace if needed, modifying examples of parenting strategies to apply to individual families circumstances, and adjusting parenting plans from what is suggested on the tip sheet in collaboration with the parents (Mazzucchelli & Sanders, 2010). Provider Characteristics Provider characteristics refer to the elements that are most closely related to perceptions held by the practitioner. The perceptions determine whether or not the provider feels the intended population is in need of the intervention. Durlak & DuPre (2008) suggest Provider Characteristics are the least influential factor in determining effective implementation, but this does not imply that Provider Characteristics are of least importance. The factors that contribute to Provider Characteristics are perceived need for the intervention, perceived benefits of the intervention, self-efficacy for the provider, and skill proficiency related to implementation (Durlak & DuPre, 2008). Recognized need and benefit of preventative intervention. Once a provider recognizes a need for an intervention and buys-in to the intervention, the likelihood of implementation increases (Shapiro, et al., 2014). However, even when the need is recognized other challenges may exist. An anti-tobacco intervention found that providers who believed it to be the norm for adolescents to try tobacco were less likely to implement or show enthusiasm for implementation (Barr, et al., 2002). These findings suggest that even though tobacco use was high in adolescence, if providers’ view it as normal for an adolescent to try cigarettes, they were less 23 likely to implement the preventative intervention effectively or correctly. These providers may believe that adolescents are going to try cigarettes regardless of the measures put in place. The more a provider perceives evidence-based programming to be effective and believes an intervention can produce results at a local level, the more likely they will implement an intervention effectively and successfully (Durlak & DuPre, 2008; Asgary-Lee & Eden, 2012). If the providers believe the reason for the change in the family is because of the success of the intervention or their own effort, they were more likely to implement the intervention with excitement (Shapiro et al., 2012). This would suggest collecting feedback from the client and charting the degree of change would be important to showing the practitioner the effect the implementation is having on their client base. The importance of recognizing the need of an intervention and the clinical benefit of the intervention go hand in hand. This was demonstrated by a pilot-study conducted by Shurer (2010) in a mid-size Midwestern town, which examined the use of parenting interventions. Results suggested the agencies were open to change and were willing to change, but did not see the benefit of implementation from a business standpoint so they were reluctant to implement a new program (Schurer, et al., 2010). Agencies may understand the need for parenting programming however, adopting a newly introduced intervention does not seem as imperative when agencies are already seeing the outcomes they want with the programming already in place. Self-Efficacy and Skill Proficiency of Provider. In terms of program implementation and Provider Characteristics, self-efficacy refers to the confidence a provider has in their own ability to deliver a program effectively (Durlak & DuPre, 2008). In Evidence Based Programs, self-efficacy is one of the best predictors of successful implementation (Durlak & DuPre, 2008). 24 It is important to note that in evidence-based practices, when program outcomes fall short of expectation some providers may view the research and evidence-based practices as a way to attack their own credibility and their ability to affect change. This can have a negative effect on the provider and their willingness to try interventions (Sanders & Turner, 2005). This further reinforces the need for post-implementation training to increase efficacy. When a provider is confident in their skills they possess they feel more capable of meeting the requirements to deliver an interventions which otherwise may be intimidating (Shapiro, et al., 2014). In a recent study, researchers evaluated factors that increased the use of the Triple P program (Shapiro, et al., 2012). The researchers found that increased provider self-confidence regarding ability to consult with parents significantly and provider perception of their own knowledge of behavioral family intervention skills, increased the rate of use (Shapiro, et al., 2012). The same study found that provider self-efficacy was also a significant predictor in completion of training requirements. Additionally, provider education and the level of Triple P provider training did not predict use of the interventions (Shapiro, et al., 2012). These findings suggest that the instruction providers receive on behavioral counseling and parent consultations during Triple P training is more meaningful than the education they received prior to Triple P training. This is assuming providers do not receive pre-service or in-service training on different methods of consultations. Guiding this evaluation is the theoretical framework suggested by Durlak and DuPre (2008) for preventative interventions. Many of the factors suggested by the framework align with the innate characteristics of Triple P. This suggests the framework will be a helpful tool in evaluating the implementation of the program. Table 1 depicts the four systems in the adapted 25 framework along with corresponding factors and the characteristics of Triple P which align with each factor. The review of the literature revealed a focus on providers’ perspective when assessing implementation. The current study focuses on the perspective of mid-level supervisors because one of the keys of sustainability is getting buy-in from the key stakeholders who have the power to make decisions and change policy. The current study sets out to answer the following questions: 1) What factors facilitated and impeded effective implementation of the Triple P Positive Parenting Program in Pitt County? 2) Can the framework adapted from Durlak and DuPre (2008) be used to evaluate effective implementation of the Triple P Positive Parenting Program? 26 Table 1. Systems of implementation, corresponding factors of implementation, and Triple P Characteristics Systems Factor Triple P Provider Characteristics Perceived Need for Intervention Perceived Benefits for Intervention Self-efficacy Skill Proficiency Multi-level approach; Varying delivery methods; Wide reach potential Application of pre and posttest; Wide reach potential Employee accredited through accreditation process; Peer support meetings; Triple P provider network Peer support meetings; Self- regulatory model Characteristics of Intervention Compatibility Adaptability Tiered multi-level approach; Flexibility Flexibility with fidelity Prevention Delivery System (Organizational Capacity) Positive work climate Organizational norm regarding change Integration of new programming Shared vision Shared decision-making Coordination with other agencies Communication Formulation of tasks Leadership Program Champion Managerial/supervisory/administrative support Level of resistance; Effects of staff turnover Level of resistance; Flexibility with fidelity Ability to provide high quality material on specific topics; Tip sheets Flexibility for identified risk and protective factors; Level of resistance Steering committee; Multi- disciplinary Steering committee; Multi- disciplinary Steering committee; Multi- disciplinary Peer support within agencies; Number of providers in agency trained Supervisor accredited in intervention Peer support; Triple P provider network Steering committee; Peer support Prevention Support System Training Technical Assistance Supervisor accredited in intervention Navigating Website; Ordering tip sheets CHAPTER 3: METHODS Participants/Sample The population included 21 mid-level supervisors from the various agencies with employees trained in Triple P in Pitt County, North Carolina. The mid-level supervisors were emailed a survey which they had three weeks to complete. There were five participants who did not respond to the survey and two participants who gave consent but did not complete any of the questions resulting in a final sample of 14. Mid-level supervisors were defined as those who are one level higher than the employee or provider who delivers any Triple P service. One level higher is defined as the person responsible for the provider’s evaluations and performance reviews (Figure 2). Figure 2 depicts the hierarchy of administration in possible agencies providing Triple P. The majority of the sample was trained in Triple P, had education beyond a high school diploma, and held a degree in a social service discipline. Table 2 provides details regarding the characteristics of the sample. 28 Figure 2. Agencies administrative hierarchy inlcluding mid-level supervisors. Executive Director Top-Level Supervisor Health Educaton Director Mid-Level Supervisor Education Officer Provider Health Care Provider Provider Case Manager Mid-Level Supervisor Counselor Provider Social Worker Provider Interns Volunteer Department Head Mid-Level Supervisor Teacher/ Instructor Provider Students Assistant Program Coordinator Provider Business Director High-level Supervisor 29 Table 2. Participant Demographics N % Race/Ethnicity Black or African American White Asian 14 3 10 1 21.4 71.4 7.1 Education Level Associate’s Degree Bachelor’s Degree Master’s Degree Doctoral Degree Juris Doctorate 14 1 2 8 1 1 7.1 14.3 57.1 14.3 7.1 Education Degree Child Dev. &Family Relations Social Work Public Health Education Sociology Nursing Law 14 3 4 2 2 1 1 1 21.4 28.6 14.3 14.3 7.1 7.1 7.1 7.1 Trained in Triple P Yes No 14 9 5 64.3 35.7 Level Trained In 3 4 14 6 3 42.8 4 30 Procedure Permission to conduct this research was approved by the Institutional Review Board (IRB) at East Carolina University (Appendix A). Participants were recruited based on the position they held in agencies with trained providers of The Triple P Positive Parenting Program. An email was distributed to all mid-level supervisors using email addresses provided by the Pitt County Triple P coordinator and those found listed on each agency’s website. Included in the email was a link to an online survey created through Qualtrics. Qualtrics is a web-based application offered through East Carolina University that allows the researcher to create and distribute web surveys. Once the participant was on the Qualtrics website he/she was able to either give consent or exit the survey. By checking yes the participants gave signature consent. The total number of questions was 31 including demographic information. The participants were not able to see all the questions at once and had the option to skip any question they did not want to answer. The participants were given three weeks to complete the survey with a reminder email distributed one week before the deadline. Measures The measure used was created using an adaption of the ecological framework suggested by Durlak and DuPre (2008). The measure contains a total of 31 questions (Appendix B) to address factors within the four systems: 1. Provider Characteristics – e.g., “to what extent do you agree Triple p will achieve positive results if implemented properly with your current client base?”; 2. Characteristics of the Intervention – e.g., “to what extent do you agree Triple P can easily be adapted to fit your agencies organizational practices?”; 31 3. Prevention Delivery System – e.g., “to what extent was your agency able to integrate Triple P with the interventions already being used?”; and 4. The Prevention Support System – e.g., “do you regularly attend steering committee meetings?”. The measure also contained five questions to obtain background information including how many people are trained and how they are implementing. Demographic data (e.g. race, education level) were also collected from the participant. CHAPTER 4: RESULTS Data analysis The data were analyzed using SPSS version 20. Descriptive statistics were used to summarize data frequency and measures of central tendency. Frequency analyses were conducted to show the number and percentage of occurrences in responses. Correlational analyses were used to examine the relationships between variables. Provider Characteristics Seventy-one percent (n=10) of participants reported that providers in their agency were implementing Triple P, 21% (n=3) reported they were not implementing, and 7% (n=1) reported they were not aware if providers in their agency were implementing or not. Thirty-seven percent (n=6) of participants were implementing using the seminar delivery method, 50% (n=8) were implementing solely using tip sheets, 31% (n=5) were implementing Level 3, 31% (n=5) were implementing Level 4 standard, and 31% (n=5) reported implementation of Level 4 group. There was no reported implementation of Level 5. One person reported Level 1 media implementation. Characteristics of the Intervention Of the 14 participants, 14% (n=2) reported no non-English speaking clients, 31% (n=5) reported less than 25% of the client-base was non-English speaking, 21.4% (n=3) reported 25% of their client-base was non-English speaking, and 7% (n=1) reported that 50% of their client- base was Non-English speaking. Prevention Delivery System Out of 14 participants, 79% (n=11) reported that they were aware of Triple P peer support meetings, while 21% (n=3) were not aware. Of the 79% who reported being aware of peer 33 support meetings, all reported encouraging their employees to attend the meetings. Of the 14 valid responses, 50% (n=8) reported that they were a member of the steering committee, and 38% (n=6) were not. Thirty-eight percent (n=6) regularly attended steering committee meetings and 13% (n=2) did not. Of the 14 mid-level supervisors who responded to the questionnaire, 69% (n=11) have designated a Triple P liaison or program champion within their agency. With regard to offering opportunity for employees to share their experiences with each other, 38% (n=6) reported less than once a month, 31% (n=5) reported once a month, and 3 remaining respondents reported (1) 2-3 times a month, (1) 2-3 times a week, and (1) daily. For staff turnover, 25% (n=4) reported it was not a concern for their agency at all in the past year, 50% (n=8) reported it was a very small concern, and 13% (n=2) reported it was a great concern for their agency. Fifteen percent (n=2) reported not being able to integrate Triple P at all with other interventions already being used; 8% (n=1) reported very little integration and 46% (n=6) reported some integration. The remaining four (31%) reported they were able to integrate to a great extent. One participant did not answer this question. Prevention Support System Seven percent (n=1) of the supervisors’ reported being comfortable in assisting employees in ordering materials necessary to implement Triple P, 7% (n=1) reported that they were not comfortable, 7% (n=1) reported they were somewhat comfortable, 36% (n=5) reported they were comfortable, and 50% (n=7) reported they were extremely comfortable. Twenty-one percent (n=3) of supervisors reported feeling somewhat comfortable assisting employees to find ways to implement Triple P. Thirty-six percent (n=5) reported they were comfortable and 43% (n=6) reported they were extremely comfortable helping providers find ways to implement Triple P. Fourteen percent (n=2) of supervisors reported feeling somewhat comfortable assisting 34 employees to problem solve issues that arise when implementing Triple P and 50% (n=7) reported they were comfortable doing so. The remaining 35% (n=5) reported they were extremely comfortable helping providers in their agency problem solve. Correlation of Variables The relationships between each of the factors were investigated using a Pearson product- movement correlation (Table 3). The relationship between Intervention Characteristics (M = 3.53; SD = .80 and Administrative Support (M = 3.24; SD=.68) was a strong, positive correlation, r = .701, n = 14, p < .001 The results suggest that the more the Characteristics of the Intervention fit with the agency the more Administrative Support is reported. There is a shared variance of 49% (r2 = .49). The relationship between the Intervention Characteristics and Provider Characteristics (M = 4.40; SD = 1.03) was determined to be a strong, positive correlation, r = .981, n = 14, p < .001. The results suggest the more the Characteristics of an Intervention are reported to fit with an agency, the more Provider Characteristics are reported favorably. There is a shared variance of 96% (r2 = .96). The relationship between Administrative Support and Provider Characteristics was determined to be a strong, positive correlation, r = .622, n = 14, p < .05, with higher reported Administrative Support associated with higher levels of favorable provider characteristics. There is a shared variance of 38% (r2 = .38). 35 Table 3. Pearson Correlation of Intervention Characteristics, Administrative Support, and Provider Characteristics Measure Intervention Characteristics Administrative Support Provider Characteristics M SD Intervention Characteristics 1 .701** .981** 3.54 .80 Administrative Support .701** 1 .622** 3.24 .68 Provider Characteristics .981** .622* 1 4.4 1.03 **Correlation is significant at the 0.01 level (2-tailed) *Correlation is significant at the .05 level (2-tailed) CHAPTER 5: DISCUSSION The purpose of this study was to investigate factors that influenced the implementation of the Triple P Positive Parenting Program (Triple P) in Pitt County, NC and to assess whether or not the adapted framework could be used to evaluate effective implementation of Triple P. There were a number of useful pieces of information learned through the current study that may be helpful in evaluating effective implementation of Triple P. There were also good indicators that suggest using the adapted framework is effective in evaluation of implementation of Triple P. Provider Characteristics The adapted framework suggests provider characteristics may have more of a direct influence on effective implementation than previously suggested. A key finding involving Provider Characteristics was the majority of the participants reported providers in their agency were implementing various levels of Triple P. By reporting the use of several different delivery methods and various levels being implemented it suggests, the providers and agencies know there are varying levels of need with the families they serve and it also suggests they understand the dosage of their clients. By choosing the level and delivery method that best fits their individual clients’ needs, providers are acknowledging the need for the intervention within their client base and the benefit of using Triple P with their clients. According to the framework, recognizing the perceived need for the intervention and the perceived benefit of the intervention contribute to effective implementation. The framework also suggests a high rate of self-efficacy and skill proficiency contribute to effective implementation within the system of provider characteristics (Durlak & DuPre, 2008). The training process in Triple P is designed to encourage self-efficacy of the provider as well as skill proficiency. During training, an opportunity to create a support network among the 37 attendees is encouraged through peer support. Peer support provides an opportunity for providers to gather and share stories of success and road bumps when actively implementing. This allows providers to ask questions and receive advice from their peers. They can then apply the new strategies in their own work to build confidence in their ability to successfully implement. This may build their confidence and aid the provider in becoming more proficient in the skills required to implement Triple P. One of the keys to increasing effective implementation is building provider confidence. When program outcomes fall short of expectations this can result in negative effects and the provider may show a decrease in their willingness to try an intervention because they lack the confidence in their skills and ability (Sanders & Turner, 2005). Results from the current study show no variability between number of supervisors reporting awareness of peer support meetings and those encouraging providers to attend the meetings. The supervisors who reported being aware of peer support meetings also reported encouraging their providers to attend these meetings, which may indicate an increased likelihood of effective implementation in Pitt County. However, because participant numbers were small there was no way to statistically explore how peer support influenced implementation. Future research should include a larger sample size to allow for further exploration of the degree of influence of peer support on implementation. Characteristics of the Intervention The characteristics of an intervention can contribute or hinder the process of implementation (Durlak & DuPre, 2008). The current literature suggests the level of compatibility of a new intervention with the current offerings of an agency increasing the likelihood of effective implementation. The flexibility of a preventative intervention to adapt to 38 fit an agency’s organizational practices also contributes to effective implementation (Breitkreuz, et al., 2011; Durlak & DuPre, 2008). All but one participant of the current study reported perceiving Triple P to fit with other behavioral interventions already being used within their agency. The participants also reported Triple P to be easily adaptable to their agencies current organizational practices. Triple P is unique in that by design, it is a multi-level approach with varying delivery methods allowing for an easier fit with other interventions and more flexibility during implementation (Mazzuchelli & Sanders, 2010; Sanders, 1999). Providers are able to choose what specific components of the program best supplement what they are already doing or choose what best fills a void currently being experienced by the agency. Prevention Delivery System Within the adapted framework used to evaluate implementation of Triple P in Pitt County, the Prevention Delivery System, along with the Prevention Support System was suggested to have the most direct influence on effective implementation. The Prevention Systems are the infrastructure of an agency to support the implementation of a preventative intervention. Within this system a missing link was identified. Though most of the supervisors reported that Triple P fit with other behavioral interventions already being used and it could easily be adapted to fit with organizational practices, the majority of them reported that they were struggling with integrating the program into their current work. Previous research has suggested that the ability to integrate an innovation is a key indicator for effective implementation (Breitkreuz, et al., 2011; Durlak & DuPre, 2008). One explanation for the current findings of difficulty with integration is that the majority of participants reporting that they were not updating agency practices often. If agencies are not in the practice of adopting new 39 interventions it may become harder to find the most efficient ways to integrate. Agencies that update practices typically have the infrastructure in place to support the adoption of a new program (Breitkreuz, et al., 2011; Shapiro, et al., 2014). Although supervisors reported difficulty with integration, they did report minimal resistance from staff when implementing Triple P. This finding suggests that agency staff, namely providers, could have been open and willing to try Triple P, but may have required more guidance on to how to do so. It would beneficial for future researchers to investigate the relationship between peer support and level of integration. Future research should also focus on the level of program champion involvement and its influences on integration. Future Triple P trainings could benefit by adding a component on successfully integrating the intervention with the providers’ current workload. Triple P conducting the integration workshops could increase the quality of training with regard to flexibility and fidelity (Sanders & Kirby, 2014). An additional aid in successful integration of a new intervention is through the peer support meetings mentioned earlier that are supported by Triple P. The supervisors in this study reported encouraging their providers to attend peer support meetings with providers outside of their agencies; however the majority reported offering limited opportunities for providers to gather and discuss their experiences within their agency. This could have possibly hindered the providers’ ability to integrate Triple P into their current workload. It would be beneficial for future research to explore the relationship between the number of opportunities offered within an agency to share implementation experiences and their influence on integration. Most of the supervisors in the current study were able to report a designated Triple P program champion within their agency. According to the literature appointing a program champion who offers ongoing support when implementing Triple P increases the likelihood of 40 effective implementation of the program (Shapiro, et al., 2010). In addition to being aware of a Triple P program champion within their agency, respondents were also able to specifically identify that person. The high rate of reported program champions suggests agencies seem to be taking some ownership over the implementation of Triple P. While this study did not examine whether or not program champions were offering ongoing support, future research should explore if level of program champion support has a relationship with effective implementation. The reported staff turnover in the participating agencies was extremely low suggesting the agencies were not short employees in the last year. However, the training for Triple P providers in agencies around Pitt County begun two years ago so, it would be valuable in future research to assess how many employees remain at an agency who were originally trained in Triple P and what the relationship is with effective implementation. Prevention Support System The implementation framework suggests the Prevention Support System is comprised of training available during and after the initial accreditation process and technical assistance provided by the administration of an agency. Sustainability of a newly adopted program can be influenced by providers increased confidence and level of comfort with a new program (Shapiro, et al., 2014). The ability of administrators to help problem solve implementation issues, as well as the ability to access program resources contributes to providers’ confidence and effective implementation. The findings of the current study suggested administrators had the ability to assist providers within their agencies when faced with bumps. The majority of the participants reported a high level of assisting providers with issues that may come up while implementing Triple P. The majority reported feeling comfortable assisting in ordering additional resources for 41 their providers. As most participants were members of the steering committee, they may have had additional knowledge that contributed to their comfort level with assisting in these areas. The steering committee provided an opportunity for supervisors to come together with other key stakeholders to discuss victories and struggles with the implementation of Triple P. This collaboration may have increased the supervisors’ ability to assist providers when issues with Triple P arose. As the majority of respondents were trained in some level of Triple P, they had working knowledge of the intervention which is suggested by Durlak & DuPre (2008) to increase effective implementation. However five supervisors were not trained in Triple P. These results bring up many organizational issues that need to be examined more thoroughly in future research: 1. How effective is a supervisor who is not trained in Triple P? 2. How much can a supervisor assist their employees with barriers and integration if not familiar with the intervention? 3. How does a supervisor not trained handle client referrals and collaboration? 4. Does being supervised by someone unfamiliar with the Triple P program, limit implementation? Correlation between Systems The results of the current study suggest there is a significant correlation between factors contributing to Provider Characteristics and factors contributing to the Intervention Characteristics. The strong relationship between the two systems is interesting. According to the framework suggested by Durlak and DuPre (2008), Provider Characteristics is the least 42 influential system when evaluating effective implementation. The results from the current study, however, suggest otherwise. There are many possibilities as to why that is. With the implementation in Pitt County, supervisors played a key role in the selection of employees who received training in Triple P. By allowing supervisors to choose the employees who received training it may have allowed a better assessment of who would be successful and who possessed the skills to effectively implement Triple P (i.e. history of implementation, established credibility in the community, etc.) (Shapiro, et al., 2014). An agency or provider who already has creditability in the community increases the likelihood of effective implementation (Breitkreuz, et al., 2011; Shapiro, et al., 2012). The ability to identify the individuals to be trained supports the importance of evaluating the skills and ability of agency staff before choosing an intervention to adopt. It may be beneficial for future research to examine the criteria administrators used when choosing employees to receive Triple P training. Examining the strengths of successful providers may give insight as to specific traits necessary to be effective when implementing Triple P or any preventative intervention. Limitations For the current study, there were a number of limitations. First, the data were collected from a small population of mid-level supervisors in Pitt County, NC. Every effort was made to identify supervisors for all trained providers, however, not all trained providers had identifiable supervisors monitoring their implementation. There are a number of trained providers who are categorized as independent providers and are not affiliated with an agency. Therefore, the implementation of this population of providers is not represented by the current results. With regard to who was considered a mid-level supervisor, only those people who were one level 43 above the provider were contacted. This did not include upper management and key stakeholders in the agencies (i.e. agency directors, CEOs, monetary contributors, board members, etc.). Another limitation was in the measure used to assess implementation. The measure was a self-created instrument which was developed for the purposes of this study. Therefore, revisions will be necessary and additional data will need to be collected in order to establish reliability and validity of the instrument. After data analysis, it was determined that a number of questions should be added to reflect the organizational structure of the systems involved in effective implementation. Adding content area that investigates how the option to bill for services influences implementation of the interventions may be beneficial. Additionally, questions should be added to reflect employee performance reviews and whether or not the implementation of Triple P is included in these reviews. The researcher replicating the study should also add questions regarding in-service trainings in Triple P beyond initial certification and whether or not they are familiar with the Triple P website and how to navigate the website. All the recommended additional questions would contribute richer data aimed at better understanding what factors influence effective implementation of the Triple P program. During data collection, the researcher realized two demographic questions were missing from the measure: gender and age. Adding these two questions could provide additional information about the perspective of those involved in implementation. Future Research Future studies should compare reported organizational processes (i.e. decision-making process, administration support, staff turnover, placement of a program champion, etc.) to 44 program implementation data to identify those agencies that are reporting high implementation, and highlight the key processes that are found to lead to effective implementation. To gain more in-depth information on specific factors influencing implementation, additional research should compare the perspective of the supervisors of each agency to the perspective of the providers from the same agency. This would provide a comparison of factors affecting multiple systems involved in the implementation process. Future studies could benefit from including upper management and key stakeholders in the evaluation. Including these individuals may allow the researcher to better understand how to obtain more funding and achieve buy-in by the key players who control policy and decision making. Understanding how to get more top-down support is one key component to achieving sustainability of any intervention (Shapiro, et al., 2014). Future studies should increase the number of participants to allow for further exploration as to how peer support influences implementation. Conclusion Keeping in mind no definitive answer can be given because the small sample size, it appears the implementation of Triple P in Pitt County has been, a successful initiative that is beginning to pick up momentum in the area. Overall, supervisors’ responses indicated enthusiasm and buy-in for the innovation. Their responses also indicated the supervisors’ willingness to offer support to their employees during implementation. More importantly, the results of this study give an indication that Pitt County has the key components in place for effective implementation. It is now a matter of assisting agencies in integration with services currently offered. As indicated by the adapted framework, by design Triple P already possess the key components for effective implementation of a preventative intervention but it may be 45 beneficial for Triple P to include a section on integration and flexibility during the training and accreditation process for both providers and their supervisors. After training, it would be beneficial for local agencies or program champions to offer in-service training pertaining to further integration and flexibility. Finally, participation in peer support seemed to be a possible protective factor in effective implementation. Implementing various methods to increase participation in peer support would greatly benefit the implementation of Triple P. Implementing a new preventative intervention at a population level is a large undertaking that can take years to embed in the community. Through this study it is evident that more effort should be placed on integration strategies and techniques when adopting a new program. When seamless integration takes place, agencies are well on their way to embedding the program in the community REFERENCES Asgary-Eden, V., & Lee, C. (2012). Implementing an evidence-based parenting program in community agencies: What helps and what gets in the way? Administration, Policy, and Mental Health, (39), 478-488. doi:10.1007/s10488-011-371-y Ballard, S. M., & Taylor, A. C. (2012). Family life education with diverse populations. Los Angeles: SAGE Barr, J., Tubman, J., Montgomery, M., & Soza-Vento, R. (2002). Amenability and implementation in secondary school antitobacco programs. American Journal of Health Behavior, 26(1), 3-15. Breitkreuz, R., McConnell, D., Savage, A., & Hamilton, A. (2011). Integrating triple p into existing family support services: A case study on program implementation. Prevention Science, (12), 411-422. doi:10.1007/s11121-011-0233-6 Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32(7), 513-531. doi:10.1037/0003-066X.32.7.513 Durlak, J. A., & Wells, A. M. (1998). Evaluation of indicated preventive intervention (secondary prevention) mental health programs for children and adolescents. American Journal of Community Psychology, 26(5), 775-802. Retrieved from http://search.proquest.com.jproxy.lib.ecu.edu/docview/205344125?accountid=10639 Durlak, J., & DuPre, J. (2008). Implementation matters: A review of the research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, (41), 327-350. doi: 10.1007/s10464-008-9165-0. 47 Kallestad, J., & Olweus, D. (2003). Predicting teachers' and schools' implementation of the olweus bullying prevention program: A multilevel study. Prevention & Treatment, 6(1), 21. doi:10.1037/1522-3736.6.1.621a Mazzucchelli, T. G., & Sanders, M. R. (2010). Facilitating practitioner flexibility within an empirically supported intervention: Lessons from a system of parenting support. Clinical Psychology: Science and Practice, 17(3), 238-252. doi: 10.111/j.1468-2850.2010.01215.x Mrazek, P. J., & Haggerty, R. J. (Eds.). (1994). Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research. National Academies Press. Prinz, R., & Sanders, M. (2007). Adopting a population-level approach to parenting and family support interventions. Clinical Psychology Review, (27), 739-749. doi:10.1016/j.cpr.2007.01.005 Prinz, R., Sanders, M., Shapiro, C., Whitaker, D., & Lutzker, J. (2009). Population-based prevention of child maltreatment: The u.s. triple p system population trial. Prevention Science, (10), 1-12. Doi: 10,1007/s11121-009-0123-3. Richard, L., Lehoux, P., Breton, E., Denis, J., Labrie, L., & Leonard, C. (2004). Implementing the ecological approach in tobacco control programs: results of a case study. Evaluation and Program Planning, (27), 409-421. doi: 10.1016/j.evalprogplan.2004.07.004 Riley, B. L., Taylor, S. M., & Elliott, S. J. (2003). Organizational capacity and implementation change: a comparative case study of heart health promotion in Ontario public health agencies. Health Education Research, 18(6), 754-769. doi: 10.1093/her/cyf051 48 Sanders, M. (1999). Triple p-positive parenting program: Towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clinical Child and Family Psychology Review, 2(2), 71 90. doi: 10.1023/A:1021843613840 Sanders, M., & Turner, K. (2005). Reflections on the challenges of effective dissemination of behavioral family intervention: Our experience with the Triple P–Positive Parenting Program. Child and Adolescent Mental Health, 10(4), 158-169. doi: 10.1111/j.1475 3588.2005.00367.x Sanders, M. (2010). Adopting a public health approach to the delivery of evidence-based parenting interventions. Canadian Psychology, 51(1), 17-23. doi: 10.1037/a0018295 Sanders, M., & Kirby, J. (2012). Consumer engagement and the development, evaluation and dissemination of evidence-based parenting programs. Behavioral Therapy, 43(2), 236- 250. doi: 10.106/j.beth.2011.01.005 Sanders, M., & Kirby, J. (2014). Surviving or Thriving: Quality Assurance Mechanisms to Promote Innovation in the Development of Evidence-Based Parenting Interventions. Prevention Science, 15(6), 1-11. doi: 10.1007/s11121-014-0475-1. Schurer, J., Kohl, P. L., & Bellamy, J. L. (2010). Organizational context and readiness for change: A study of community-based parenting programs in one Midwestern city. Administration In Social Work, 34(2), 178-195. doi:10.1080/03643101003609487. Shapiro, C., Prinz, R., & Sanders, M. (2010). Population-based provider engagement in delivery of evidence-based parenting interventions: Challenges and solutions. Juvenile Primary Prevention, (31), 223-234. doi:10.1007/s10935-010-0210-z 49 Shapiro, C., Prinz, R., & Sanders, M. (2012). Facilitators and barriers to implementation of an evidence based parenting intervention to prevent child maltreatment: The triple-p positive parenting program. Child Maltreatment, 17, 86-95. doi: 10.1177/1077559511424774 Shapiro, C., Prinz, R., & Sanders, M. (2014). Sustaining use of an evidence-based parenting intervention: Practitioner perspectives. Journal of Child and Family Studies, 1-10. doi: 10.1007/s10826-014-9965-9. Small, S. A., Cooney, S. M. and O’Connor, C. (2009), Evidence-informed program improvement: Using principles of effectiveness to enhance the quality and impact of family-based prevention programs. Family Relations, 58, 1–13. doi: 10.1111/j.1741- 3729.2008.00530.x Turner, K., & Sanders, M. (2006). Dissemination of evidence-based parenting and family support strategies: Learning from the triple p-positive parenting system approach. Aggression and Violent Behavior, (11), 176-193. doi: 10.1016/j.avb.2005.07.005 Wandersman, A., & Florin, P. (2003). Community interventions and effective prevention. American Psychologist, 58(6-7), 441-448. doi:10.1037/0003-066X.58.6-7.441 50 APPENDIX A: IRB APPROVAL EAST CAROLINA UNIVERSITY University & Medical Center Institutional Review Board Office 4N-70 Brody Medical Sciences Building· Mail Stop 682 600 Moye Boulevard · Greenville, NC 27834 Office 252-744-2914 · Fax 252-744-2284 · www.ecu.edu/irb Notification of Amendment Approval From: Social/Behavioral IRB To: Eboni Baugh CC: Date: 6/2/2014 Re: Ame2_UMCIRB 13-000329 UMCIRB 13-000329 Triple P Evaluation Your Amendment has been reviewed and approved using expedited review for the period of 6/2/2014 to exempt no expiration. It was the determination of the UMCIRB Chairperson (or designee) that this revision does not impact the overall risk/benefit ratio of the study and is appropriate for the population and procedures proposed. Please note that any further changes to this approved research may not be initiated without UMCIRB review except when necessary to eliminate an apparent immediate hazard to the participant. All unanticipated problems involving risks to participants and others must be promptly reported to the UMCIRB. A continuing or final review must be submitted to the UMCIRB prior to the date of study expiration.The investigator must adhere to all reporting requirements for this study. Approved consent documents with the IRB approval date stamped on the document should be used to consent participants (consent documents with the IRB approval date stamp are found under the Documents tab in the study workspace). The approval includes the following items: Document Description Informed Consent - Triple P Implementation.docx(0.01) Consent Forms Qualtrics Survey.doc(0.01) Interview/Focus Group Scripts/Questions Qualtrics Survey.doc(0.01) Data Collection Sheet Qualtrics Survey.doc(0.01) Surveys and Questionnaires The Chairperson (or designee) does not have a potential for conflict of interest on this study. IRB00000705 East Carolina U IRB #1 (Biomedical) IORG0000418 IRB00003781 East Carolina U IRB #2 (Behavioral/SS) IORG0000418 51 EAST CAROLINA UNIVERSITY University & Medical Center Institutional Review Board Office 4N-70 Brody Medical Sciences Building· Mail Stop 682 600 Moye Boulevard · Greenville, NC 27834 Office 252-744-2914 · Fax 252-744-2284 · www.ecu.edu/irb Notification of Amendment Approval From: Social/Behavioral IRB To: Eboni Baugh CC: Date: 5/12/2014 Re: Ame1_UMCIRB 13-000329 UMCIRB 13-000329 Triple P Evaluation Your Amendment has been reviewed and approved using expedited review for the period of 5/12/2014 to no expiration. It was the determination of the UMCIRB Chairperson (or designee) that this revision does not impact the overall risk/benefit ratio of the study and is appropriate for the population and procedures proposed. Please note that any further changes to this approved research may not be initiated without UMCIRB review except when necessary to eliminate an apparent immediate hazard to the participant. All unanticipated problems involving risks to participants and others must be promptly reported to the UMCIRB. A continuing or final review must be submitted to the UMCIRB prior to the date of study expiration.The investigator must adhere to all reporting requirements for this study. Approved consent documents with the IRB approval date stamped on the document should be used to consent participants (consent documents with the IRB approval date stamp are found under the Documents tab in the study workspace). The approval includes the following items: Document Description Sub Investigator Added: Melissa Nolan The Chairperson (or designee) does not have a potential for conflict of interest on this study. IRB00000705 East Carolina U IRB #1 (Biomedical) IORG0000418 IRB00003781 East Carolina U IRB #2 (Behavioral/SS) IORG0000418 52 APPENDIX B: IMPLEMENTATION SURVEY 1. What is your official job title within your agency? 2. Please check your race/ethnicity. Black or African American White Asian American Indian or Alaska Native Native Hawaiian or other Pacific Islander Hispanic Multiracial Other ____________________ 3. Highest level of education attained or completed. High School Diploma or equivalent Some college, no degree Associate's degree Bachelor's degree Master's degree Professional degree Doctoral Degree Refused Other ____________________ 4. What is your area of specialization? Child Development and Family Relations Social Work Public Health Psychology Education Sociology Other ____________________ 5. Are you trained in the Triple P Positive Parenting Program (Triple P)? Yes No 53 6. How many employees are currently trained in Triple P at your agency? 0-4 4-8 8-12 13+ 7. How many clients in your agency are non-English speaking or English as a second language? 25% 50% 75% 100% 8. Are providers in your agency currently implementing The Triple P Positive Parenting Program (Triple P)? Yes No 9. How are they implementing? Check all that apply. Seminar Tip Sheets Level 3 Level 4 Level 5 Group Other ____________________ 10. Please indicate to what extent you agree with the following statement Triple P is relevant to the needs of my agency’s client base Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 54 11. Please indicate to what extent you agree with the following statement Triple P will achieve positive results if implemented properly with our current client base. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 12. Please indicate to what extent you agree with the following statement Triple P fits well with my agencies mission and goals. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 13. Please indicate to what extent you agree with the following statement Triple P can easily be adapted to fit the needs of my agencies client base. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 14. Please indicate to what extent you agree with the following statement Triple P can easily be adapted to fit my agencies organizational practices. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 15. Does Triple P fit with the other behavioral interventions you are currently using within your agency? Yes No 55 16. To what extent was your agency able to integrate Triple P with the interventions already being used? Not at all Very Little Somewhat To a great extent 17. To what extent did you face resistance from your staff in implementing Triple P with their clients? None at all Very Little Somewhat To a great extent 18. Rate your comfort level with the following, assisting employees in finding ways to implement Triple P with their current client base. Not at all Somewhat Comfortable Comfortable Extremely Comfortable 19. Rate your comfort level with the following, assisting employees to problem solve challenges faced during implementation of Triple P. Not at all Somewhat comfortable Comfortable Extremely Comfortable 20. Rate your comfort level with the following, assisting employees in ordering materials and resources needed for implementing Triple P. Not at all Somewhat comfortable Comfortable Extremely Comfortable 56 21. In the past year has staff turnover been an issue within your agency? Yes No 22. To what extent did you feel a part of the decision making process when your agency decided to adopt the Triple P program? Not at all Very little Somewhat To a great extent 23. Are you aware of peer-support meetings offered to providers certified in Triple P? Yes No 24. Do you encourage your providers to attend peer support meetings? Yes No 25. Are you a member of the steering committee? Yes No 26. How often does your agency update and implement new practices and techniques to use when working with families? Never Rarely Sometimes Quite Often Very Often 57 27. Do you provide opportunities for Triple P providers within your agency to meet and discuss their experiences implementing Triple P? Yes No 28. Within your agency, has someone been designated the Triple P liaison or program champion? Yes No 29. Do you report to anyone within your agency in regards to the implementation of Triple P? Yes No 30. What level are you trained in? Check all that apply. Seminar Level 1 Level 2 Level 3 Level 4 Group Level 5 31. Do you regularly attend steering committee meetings? Yes No