Organizational characteristics that influence capacity building in evaluation in community organizations in Quebec: a review of the literature

Authors

  • David Buetti École interdisciplinaire des sciences de la santé
  • Isabelle Bourgeois École nationale d'administration publique
  • Sébastien Savard Université d'Ottawa

DOI:

https://doi.org/10.29173/cjnser.2019v10n2a287

Keywords:

Community organizations, Organizational evaluation capacity, Evaluation capacity building, Program evaluation / Organismes communautaires, Capacités organisationnelles en évaluation, Renforcement des capacités en évaluation, Évaluation des programmes

Abstract

Few studies have conceptualized evaluation capacity for community organizations (COs), and even fewer in the specific context of COs in Québec. The objective of this article, therefore, is twofold: 1) to identify barriers to, and opportunities for, building evaluation capacity among OCs and 2) to verify the extent to which an existing conceptual framework of organizational evaluation capacity established by Isabelle Bourgeois and J. Bradley Cousins could be useful in its current form for understanding the evaluation capacity needs of COs in Québec. We first conducted a review of the scientific literature to identify the barriers and opportunities for evaluation capacity building in OCs, and analyzed the results of the literature review against the conceptual framework. Our results point to the fact that Bourgeois and Cousins’ framework is appropriate for understanding the key factors influencing the capacity to do and to use evaluations among OCs. Additional efforts would, however, be necessary to improve the framework’s compatibility with the practices and values of COs in Québec.

Les capacités en évaluation sont peu étudiées et encore moins conceptualisées en fonction du contexte particulier des organismes communautaires (OC) du Québec. Ainsi, l’objectif de cet article est double : 1) identifier chez les OC les barrières et les éléments qui facilitent le renforcement des capacités en évaluation et 2) vérifier dans quelle mesure un cadre conceptuel établi par Isabelle Bourgeois et J. Bradley Cousins pourrait s’avérer propice dans sa forme actuelle pour l’analyse des capacités en évaluation du milieu communautaire québécois. Nous avons d’abord identifié les barrières et les éléments qui facilitent le renforcement des capacités en évaluation des OC à partir d’une recherche documentaire d’articles scientifiques pertinents pour ensuite effectuer une analyse à partir du cadre conceptuel choisi. Nos résultats démontrent que le cadre conceptuel de Bourgeois et Cousins est pertinent pour répertorier les facteurs clés influençant la capacité à effectuer et à utiliser l’évaluation chez les OC. Des efforts supplémentaires seraient toutefois nécessaires pour améliorer la compatibilité du cadre conceptuel avec les pratiques et valeurs du milieu communautaire québécois.

Downloads

Download data is not yet available.

Author Biographies

David Buetti, École interdisciplinaire des sciences de la santé

École interdisciplinaire en sciences de la santé, Candidat au doctorat en santé des populations

Isabelle Bourgeois, École nationale d'administration publique

Professeure agrégée

Sébastien Savard, Université d'Ottawa

École de service social, Professeur agrégé

References

Adams, A. E., Nnawulezi, N. A., & Vandenberg, L. (2015). "Expectations to Change" (E2C): A Participatory Method for Facilitating Stakeholder Engagement With Evaluation Findings. American Journal of Evaluation, 36(2), 243-255. doi:10.1177/1098214014553787

Akintobi, T. H., Yancey, E. M., Daniels, P., Mayberry, R. M., Jacobs, D., &

Berry, J. (2012). Using Evaluability Assessment and Evaluation Capacity-building to Strengthen Community-based Prevention Initiatives. Journal of Health Care for the Poor and Underserved, 23(2), 33-48.

Alaimo, S. P. (2008). Nonprofits and evaluation: Managing expectations from the leader's perspective. New Directions for Evaluation, 2008(119), 73-92. doi:10.1002/ev.269

Andrews, A., Motes, P. S., Floyd, A., Flerx, V. C., & Fede, A. L.-D. (2005). Building Evaluation Capacity in Community-Based Organizations: Reflections of an Empowerment Evaluation Team. Journal of Community Practice, 13(4), 85-104. doi:10.1300/J125v13n04_06

Bourgeois, I. (2008). Understanding the dimensions of organizational evaluation capacity (Doctoral dissertation). Retrieved from Library and Archives Canada Database. (Accession No. 9780494483893).

Bourgeois, I., & Cousins, J. B. (2013). Understanding Dimensions of Organizational Evaluation Capacity. American Journal of Evaluation, 34(3), 299-319. doi:10.1177/1098214013477235

Bourgeois, I., Hotte, N., Simmons, L., & Osseni, R. (2016). Measuring evaluation capacity in Ontario public health units. Canadian Journal of Program Evaluation, 31(2), 165-183. doi:10.3138/cjpe.306

Bourgeois, I., Simmons, L., & Buetti, D. (2018). Building evaluation capacity in Ontario's public health units: promising practices and strategies. Public Health. doi:10.1016/j.puhe.2018.01.031

Bourgeois, I., Whynot, J., & Theriault, E. (2015). Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned. Evaluation and Program Planning, 50, 47-55. doi:10.1016/j.evalprogplan.2015.01.004

Bourque, M., Grenier, J., Pelland, D., & St-Germain, L. (2016). L'action communautaire autonome, mouvance des contextes, place et autonomie des acteurs : quelles perspectives ? Sciences et actions sociales, 4, 1-20.

Carman, J. G. (2007). Evaluation Practice among Community-Based Organizations: Research into the Reality. American Journal of Evaluation, 28(1), 60-75. doi:10.1177/1098214006296245

Carman, J. G., & Fredericks, K. A. (2010). Evaluation Capacity and Nonprofit Organizations: Is the Glass Half-Empty or Half-Full? American Journal of Evaluation, 31(1), 84-104. doi:10.1177/1098214009352361

Carnochan, S., Samples, M., Myers, M., & Austin, M. J. (2014). Performance Measurement Challenges in Nonprofit Human Service Organizations. Nonprofit and Voluntary Sector Quarterly, 43(6), 1014-1032. doi:10.1177/0899764013508009

Cheverton, J. (2007). Holding Our Own: Value and Performance in Nonprofit Organisations. Australian Journal of Social Issues, 42(3), 427-436.

Cousins, J. B., Goh, S. C., Elliott, C., Aubry, T., & Gilbert, N. (2014). Government and voluntary sector differences in organizational capacity to do and use evaluation. Evaluation and Program Planning, 44, 1-13. doi:10.1016/j.evalprogplan.2013.12.001

Daniel, C. (1999). L'évaluation d'un organisme social et communautaire. Montréal: Centre de formation populaire.

Despard, M. R. (2016). Strengthening Evaluation in Nonprofit Human Service Organizations: Results of a Capacity-Building Experiment. Human Service Organizations Management Leadership & Governance, 40(4), 352-368. doi:10.1080/23303131.2016.1140101

Ducharme, É., Lesemann, F., Charbonneau, J., Martin, C., & Lefèvre, S. (2011). Les fondations et la « nouvelle philanthropie » : un changement de paradigmes scientifiques et politiques. Lien social et Politiques(65), 203-224.

Fonds, M. (2012). Milieu communautaire : à quoi sert l’évaluation ? À babord: revue sociale et politique. Retrieved from https://www.ababord.org/Milieu-communautaire-a-quoi-sert-l

Garcia-Iriarte, E., Suarez-Balcazar, Y., Taylor-Ritzler, T., & Luna, M. (2011). A Catalyst-for-Change Approach to Evaluation Capacity Building. American Journal of Evaluation, 32(2), 168-182. doi:10.1177/1098214010387114

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies (Vol. 26, pp. 91-108). Oxford, UK.

Harper, G. W., Contreras, R., Bangi, A., & Pedraza, A. (2003). Collaborative Process Evaluation: Enhancing Community Relevance and Cultural Appropriateness in HIV Prevention. Journal of Prevention & Intervention in the Community, 26(2), 53-69. doi:10.1300/J005v26n02_05

Houlbrook, M. (2011). Critical perspectives on results-based accountability: Practice tensions in small community-based organizations. Third Sector Review, 17(2), 45-68.

Janzen, R., Ochocka, J., Turner, L., Cook, T., Franklin, M., & Deichert, D. (2017). Building a community-based culture of evaluation. Evaluation and Program Planning, 65, 163-170. doi:10.1016/j.evalprogplan.2017.08.014

Kapucu, N., Healy, B. F., & Arslan, T. (2011). Survival of the fittest: Capacity building for small nonprofit organizations. Evaluation and Program Planning, 34(3), 236-245. doi:10.1016/j.evalprogplan.2011.03.005

Kegeles, S. M., Rebchook, G. M., & Tebbetts, S. (2005). Challenges and Facilitators to Building Program Evaluation Capacity among Community-Based Organizations. AIDS Education and Prevention, 17(4), 284-299.

Kelly, C. M., LaRose, J., & Scharff, D. P. (2014). A Method for Building Evaluation Competency Among Community-Based Organizations. Health Promotion Practice, 15(3), 431-437. doi:10.1177/1524839913496427

Kitchenham, B. (2004). Procedures for Performing Systematic Reviews Keele: Keele University

Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A Research Synthesis of the Evaluation Capacity Building Literature. American Journal of Evaluation, 33(3), 307-338. doi:10.1177/1098214011434608

Larose, F., Bédard, J., Couturier, Y., Grenon, V., Béland, S., Larivée, S., & Vincent, F. (2018). Étude évaluative des interventions éducatives et socioéducatives de La Relance, jeunes et familles. Rapport final (2014-2017) Version courte (sans annexes).

Lemay, J.-C., Leclair, L., Bélanger, S., & Messikh, D. (2013). Les particularités de l’évaluation pour les groupes communautaires : réflexions et enjeux. Bulletin de la Société québécoise d’évaluation de programme, 25(1), 8-10.

Lum, J., Evans, B., & Shields, J. (2016). Co-constructing Performance Indicators in Home and Community Care: Assessing the Role of Nongovernmental Organizations in Three Canadian Provinces. Canadian Journal of Nonprofit and Social Economy Research, 7(1), 46-67. doi:10.22230/cjnser.2016v7n1a215

Mayne, J. (2017). Building Evaluative Culture in Community Services: Caring for Evidence. Evaluation and Program Planning. doi:10.1016/j.evalprogplan.2017.05.011

McKegg, K., Wehipeihana, N., & Pipi, K. (2016). The Evaluation Capacity Building Project Developing an evaluation capacity assessment tool and process for New Zealand NGOs. Wellington: The Social Policy Evaluation and Research Unit’s Retrieved from http://www.superu.govt.nz/sites/default/files/Evaluation%20Capacity%20report.pdf.

Mitchell, L., & Fiellin, D. A. (2014). Assessing Research Activity and Capacity of Community-Based Organizations: Development and Pilot Testing of an Instrument. Progress in Community Health Partnerships: Research, Education, and Action, 8(4), 413-414. doi:10.1353/cpr.2014.0067

Pierson, J. (2008). Going local : working in communities and neighbourhoods. London : Routledge.

Poth, C., Lamarche, M., Yapp, A., Sulla, E., & Chisamore, C. (2014). Toward a Definition of Evaluation Within the Canadian Context: Who Knew This Would Be So Difficult? The Canadian Journal of Program Evaluation, 29(1), 87. doi:10.3138/cjpe.29.1.87

Potvin, P. (2010). Éléments et critères d'évaluabilité d'un programme d'intervention psychosociale. In M. Alain & D. Dessureault (Eds.), Élaborer et évaluer les programmes d’intervention psychosociale (pp. 195-216). Québec: Presses de l’Université du Québec.

Preskill, H. (2014). Now for the Hard Stuff: Next Steps in ECB Research and Practice. American Journal of Evaluation, 35(1), 116-119. doi:10.1177/1098214013499439

Preskill, H., & Boyle, S. (2008). A Multidisciplinary Model of Evaluation Capacity Building. American Journal of Evaluation, 29(4), 443-459. doi:10.1177/1098214008324182

SacAction. (2015). L'évaluation : pour aller plus loin! Québec: Gouvernement du Québec Retrieved from https://www.mtess.gouv.qc.ca/publications/sacaction/no81/historique.asp.

Schalock, R. L., Lee, T., Verdugo, M., Swart, K., Claes, C., van Loon, J., & Lee, C.-S. (2014). An evidence-based approach to organization evaluation and change in human service organizations evaluation and program planning. Evaluation and Program Planning, 45, 110-118. doi:10.1016/j.evalprogplan.2014.03.012

Sébastien, S., & Jean, P. (2012). Les organismes communautaires au Québec : De la coexistence à la supplémentarité. Canadian Journal of Nonprofit and Social Economy Research, 3(2), 24-32.

Service aux collectivités de l'UQAM. (2005). Analyse des pratiques d'évaluation dans les organismes communautaires : rapport de recherche. Montréal: Université du Québec à Montréal.

Simmons, L. (2015). Building Evaluation Capacity In Ontario’s Public Health Units: A Locally Driven Collaborative Project. Ontario: Eastern Ontario Health Unit.

Simmons, V. N., Klasko, L. B., Fleming, K., Koskan, A. M., Jackson, N. T., Noel-Thomas, S., . . . Gwede, C. K. (2015). Participatory evaluation of a community-academic partnership to inform capacity-building and sustainability. Evaluation and Program Planning, 52, 19-26. doi:10.1016/j.evalprogplan.2015.03.005

Sobeck, J. L. (2008). How cost-effective is capacity building in grassroots organizations? Administration in Social Work, 32(2), 49-68. doi:10.1300/J147v32n02_04

Société canadienne d'évaluation. (2010). Qu'est-ce que l'évaluation? Ottawa: Société canadienne d'évaluation Retrieved from https://evaluationcanada.ca/fr/quest-ce-que-levaluation.

Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a Definition of the ECB Process: A Conversation with the ECB Literature. New Directions for Evaluation(93), 1-25.

Weir, A., & Fouche, C. (2016). “Dancing with data”: Investing in capacity building for non-government organizations (NGOs). Evaluation Matters—He Take Tō Te Aromatawai, 2, 131-160.

Yu, S., & McLaughlin, D. (2013). Program Evaluation and Impact Assessment in International Non-Governmental Organizations (INGOs): Exploring Roles, Benefits, and Challenges. Canadian Journal of Nonprofit and Social Economy Research, 4(2), 23-38.

Yung, B., Leahy, P., Deason, L. M., Fischer, R. L., Perkins, F., Clasen, C., & Sharma, M. (2008). Capacity-building needs of minority health nonprofits. Evaluation and

Downloads

Published

2019-11-28