self-evaluation.info

Back into the field work

admin — Mon, 30 Jul 2007 22:30:35 +0000

Kia ora koutou to our NGO partners

It has been a while since we saw anyone in the field (June) and we are now planning the next lot of visits to complete work started with Jigsaw affiliated agencies and to begin with Child-Youth Advocate host agencies.

Here is a “map” showing NGOs we have met with, and in most cases have begun to work with. NB. Click on the image below to open a larger view, and click on it again if you still can’t read it properly.

We have had some changes in our team - Joseph Schumacher has just left to get married in the US and then to work in Iraq. Also we have been focused upon analysing information for the formative (beginning) stage of the Child-Youth Advocates evaluation, and ensuring we have the resources we need to continue with the self-evaluation project, more or less as planned.

Our hope is see you based on the schedule below:

2nd week August (6-10th) - Wanganui Family Trust, CAPS Hauraki, Te Korowai Aroha O Ngati Whatua, Hestia Women’s Refuge, North Harbour Living Without Violence

3rd week August (13-17th) - Te Ruruhau O Meri, Parent Trust, Tu Wahine, Pacific Island Safety & Prevention, Barnados - Waitakere, Ngati Porou Hauora, Te Whare Tu Wahine, Barnardos - Gisborne

End of August - Early September - Wellington-based meetings with National Collective of Independent Womens Refuges, National Network of Stopping Violence Services, Jigsaw / National Infrastructure for Child-Youth Advocates, Barnardos NZ

Clearly these visits will be split across our self-evaluation project team, which currently includes Kathy Fielding, Jane Renwick and Geoff Stone. We look forward to seeing you soon.

Next we’ll post a reminder and summary of the key ideas and tools that we work with on the self-evaluation project.

You can subscribe to this weblog, that is get an email bulletin letting you know of new posts, by clicking on the Feedblitz button on the right hand side of this page, or here.

Posted by Geoff Stone

Project update - July

admin — Wed, 04 Jul 2007 02:53:33 +0000

Greetings everyone

This project update covers the period from late May through June and focuses on:

Engagement with Child-Youth Advocate NGOs
Consolidating the project and working with MSD-IT
Connecting with Stopping Violence Services
Linking with other initiatives (ACC, Pathways to Partnership, Vulnerable Families)

Child-Youth Advocate Workshop
An evaluation workshop for Child-Youth Advocates held in Waitakere on the 28-29 May provided an opportunity to introduce 10 NGOs to the FVPS-E project (two of whom are already involved. They all agreed in principle to participating in the self-evaluation project, with a number indicating it may take four to six weeks to gain the endorsement of their boards. It is also important to secure the involvement of the collaborative that sponsored the child advocate position/s, and the national body they affiliate to.

Workshop participants had the immediate benefit of being able to share experiences and swap ideas about establishing a Child-Youth Advocate position. Also, they were able to provide feedback to MSD on the set-up phase of Child-Youth Advocate initiative as part of a CSRE led formative evaluation. The CSRE evaluation is being led by Kathy Fielding and involves colleagues, Joseph Schumacher and Jane Renwick. I’ve begun to talk with Barnardos National Office and the National Collective of Independent Women’s Refuge to secure their involvement in the project alongside their local agencies. Jigsaw (Child Protection Services) is already participating.

CAPS Hauraki (Thames), Te Korowai Aroha O Ngati Whatua (Wellsford), Ngati Porou Hauora (Gisborne), Te Whare Tu Wahine – Gisborne Women’s Refuge, and Hestia Women’s Refuge (Rodney) are the agencies that have so far have confirmed their interest intention to participate in the FVPS-E project (at least in the pre-trial phase). We are expecting to begin site visits this month to tease out how the Child-Youth Advocates positions link (or not) with the other family violence prevention-related work done by host agencies and the collaboratives.

Consolidating the project and working with MSD-IT
June was a month mostly occupied with the ordinary tasks associated with the end of the financial year, and with ensuring the project has the resources it needs for 2008 and beyond. FVPS-E is one of five initiatives that are covered by what is known as the MSD Non-government Organisations Information Technology Applications project (NGO-IT Apps.). We have only recently heard that funds are now available to build the FVPS-E online database, web-screens for inputting information, and reporting templates. While we have been working with members of the MSD-IT group for some time on aspects of database design, there have been no funds to create the IT infrastructure for FVPS-E. The issue now is being able to move sufficiently quickly to build a “prototype” for the self-evaluation project that NGOs can begin to use, starting in October or November.

MSD-IT is concerned with data standards - trying to ensure that the way we think within FVPS-E about basic programme information, and about outcomes or results isn’t out of step with the way that NGOs who deliver Family Start, Social Workers in Schools, Strengthening Families and Enterprising Communities, think about these things. These are the other initiatives for which MSD provides databases. We are positioning the FVPS-E to be an “advanced guard” within the NGO-IT Apps. project because it seeks to do more than traditional programme monitoring. The use of a theory of change approach to understanding programme conditions, and FVPS-E’s emphasis on measuring results, and linking these to programme performance through a dashboard style report makes it somewhat unique.

Working with stopping violence services
We have been in discussion with the National Network of Stopping Violence Services (NNSVS) since mid last year about their potential involvement in FVPS-E. However, the first opportunity for FVPS-E to link with NGOs delivering stopping violence services came only very recently, on the 28^th and 29^th of June. An Appreciative Inquiry workshop was held in Masterton to identify good, and promising, practices based on the experiences of NNSVS’s lower North Island network. It was coordinated by Brian Gardener, NNSVS, facilitated by Liz Kinley, and funded by FaCS who are keen to have good practice captured in integrated contracts. It was also attended by a representative from FaCS, Mike Grace (Funding for Outcomes-Integrated Contracting) and me (FVPS-E). The NGO’s included Whanganui Living Without Violence Trust, DOVE Hawkes Bay, Horowhenua Family Violence Intervention Project, Hutt Valley Stopping Violence Services, Kapiti Living Without Violence, Stopping Violence Services Wairarapa, and Porirua Living Without Violence (Wellington Ending Abuse and Violence was not able to come).

Again, an immediate benefit was the chance to share approaches, ideas and methods and to identify common values and themes in the work, particularly, but not exclusively with men. FVPS-E, FaCS and NNSVS will now work together to compile some principles and tools identified from the workshop that can then be:

discussed, added to and promoted across the NNSVS network
compiled into a set of good practice guides and checklist for use within the self-evaluation project, and beyond. There is already a draft guide as a starting point.
used to refine the integrated contracting process. This is concerned with how to best to describe and support high quality programme implementation (or performance) and the achievement and reporting of programme results, agreed across numerous funders.

All the NGO representatives expressed an interest in participating in FVPS-E in several months time. However as a start, the FVPS-E project will work with several agencies to enhance the way they currently think about and capture programme information. They were particularly keen to try the scoring methods proposed by FVPS-E (i.e. the use of Goal Attainment Scaling and modified (strengths-based) versions of Global Assessment Scales to measure “client” (individual, couple or family) progress, or even organisational or community development, as these are a good fit with current and developing practices.

Linking with other initiatives

ACC
Discussions have also been going on with ACC’s Lou Renner regarding that agency’s support for the development of a “process-outcome”, or as we have more recently described it, an “implementation-results” monitoring system. ACC is considering contributing funds to the FVPS-E project on the grounds that a comprehensive monitoring system for family violence prevention providers will contribute to injury prevention, particularly for children, and help government decision-making regarding service planning. FVPS-E also recognises ACC conceptual contribution as we draw on a previous evaluation that ACC conducted of its Child Advocates pilot that ran from 2004 to 2006.

Pathways to Partnership
Also, the FVPS-E project may connect, or cooperate with the evaluation of MSD’s Pathways to Partnership policy initiative. This would consolidate a “theory of change” approach within the ministry to understanding the vital components to policy and programme success – the community and operational context, the nature and quality of implementation, and understanding / identifying results. The evaluation of Pathways to Partnership would also build upon and refine the web-based monitoring and reporting system initiated by FVPS-E.

Vulnerable families – monitoring and evaluation
Finally CSRE is developing a web-based case management system for Work and Income’s Vulnerable Families initiative modeled on a successful system for helping long-term unemployed - Weka. There are strong similarities in our thinking about standard ways of capturing and reporting the most important programme information for individuals and families. Justine O’Reilly, the Principal Analyst responsible for Weka, and currently leading CSRE’s Vulnerable Families work, will be collaborating with the FVPS-E project. We expect this will speed up the development of the FVPS-E database and enhance the project generally.

Geoff Stone
For FVPS-E

Project update - May

admin — Tue, 22 May 2007 05:17:05 +0000

Udpdate of project progress so far

Site visits with Jigsaw-affiliated NGO’s began in February-March after several initial workshops in October and February. These included Family Help Trust (Christchurch), Family Support Services (Wanganui), Te Korowai Aroha O Ngati Whatua (Wellsford), CAPS Hauraki (Thames), Te Whare Ruruhau o Meri (Auckland) and Jigsaw (Wellington). My thanks again to all those that hosted the daylong visits and contributed to the process.

The purpose of visiting NGOs is to “map” or “model” the delivery of family violence prevention initiatives/services. I am now into making sense of the information provided by NGOs so their particular patterns of service delivery are reflected in the database and reporting templates MSD will creates for their use. Parent Help Trust (Auckland) is due for a visit on the 24^th of May.

Progress with this “sense-making” part of the work has been held up a little over April/May due to:

attendance of the Social Policy Research and Evaluation conference
the Easter break
taking some leave during the school holidays
working on the child advocates evaluation – we are planning to include the first 10 host agencies/advocates in the self-evaluation trial
continued development of the support website for the project – needing to progress this at the same time as everything else
planning with NNSVS to run a workshop on emerging “good practice” in stopping violence programmes and measuring change – this is in advance of involving some stopping violence agencies in the trial
working with MSD-IT on how best to manage the database build for the project
securing ongoing funding (including from other departments such as ACC) and ensuring MSD stakeholder awareness and support (especially CYF and Social Service Policy).
other analysts initially allocated to the FVPS-E - Joseph Schumacher and Kathy Fielding - project needing to work on other priorities

While things have been busy, it feels like everything is coming together nicely.

I now expect to follow-up Jigsaw agencies over May and June to:

complete maps/models of service delivery to our mutual satisfaction
agree on NGOs’ “theories of change” – how you think about what you do (or will do) makes a difference for particular people and communities
agree on the most important information to be collected on NGO programme delivery for the database and reporting - data categories and types

At the end of May, we plan to meet and begin to work with NGO’s that are hosting Child/Youth Advocates (witnesses of family violence), starting with a workshop on 28^th and 29^th.

After beginning with Jigsaw-affiliated agencies, and then those NGOs hosting the first 10 Child-Youth Advocate positions, we expect to involve other clusters of family violence prevention providers in the trial - NNSVS agencies (July-August) and NCIWR agencies (around October-November). Engaging NGO “clusters” in this stepped way allows us to keep the self-evaluation trial manageable while recognising the high level of cooperation these different providers have at a local and national level.

As mentioned, we are currently re-negotiating the scope of the database build and timelines with MSD-IT. We will keep you updated about timing for the database launch.

For further information please don’t hesitate to give me a call on (04) 916-3613

Kia ora - goodbye for now.

Geoff

Short project description

admin — Wed, 18 Oct 2006 15:06:50 +0000

Family Violence Prevention Self-evaluation Project

Introducing the project
The Family Violence Prevention Self-evaluation Project involves the development and trialling of an Internet-based self-evaluation toolkit for NGOs working in the family violence prevention area. This will enable NGOs to evaluate their family violence prevention initiatives themselves, using well-established and consistently applied methods to provide useful information. The project will also build upon the good evaluation practices already in place. The trial itself is expected to last 12 to 24 months beginning in 2007, with a range of benefits anticipated for NGOs and the Ministry of Social Development (MSD) – see below.

The project is sponsored by the MSD and is being run by the research and evaluation division of the Ministry, in association with Family and Community Services (FaCS) and Child Youth and Family (CYF).

Reasoning behind the project
Family violence is considered a significant social problem in New Zealand and the government has made reducing family violence a priority. Both government agencies and non-government organisations provide initiatives designed to prevent family violence. However, the government faces the problem of having insufficient information to work out how well it is doing in addressing family violence (either directly through providing services, or through funding and supporting for NGOs working in this area). Also, some NGOs have indicated that they would like to be clearer about the impact of their initiatives for the people they are trying to help, and to better understand what is needed to do their best work.

Lastly, FaCS is adopting a results-based accountability approach to its initiatives to ensure that they do in fact help people, rather than just provide (or fund) activities. This kind of thinking is connected with attempts across government to improve NGO capacity, to provide better support and coordination of effort through integrated contracts, and to increase the sharing of information via mediums such as the Family Violence Clearinghouse.

Project aims and method
The project therefore aims to improve the quality of information available to NGOs and government about the impact of family violence prevention initiatives, and which also indicates how we might do better. Improved information and analysis is expected to improve the design and delivery of family violence prevention services and campaigns, which should in turn enhance the safety and quality of life of people who are, or would be, affected by family violence. Recognising the need for a partnership approach, MSD anticipates providing extra financial support to a group of NGOs to collaborate in a trial to test the most promising self-evaluation processes and tools. The trial is to ensure that doing more internal evaluation does in fact result in more useful programme and social impact information.

Specific goals
A goal of the project is to test (and further develop) useful methods for evaluating Family Violence initiatives using the proposed two-year trial. As part of the trial MSD expects to provide participating NGOs with:

some evaluation processes, templates and report formats that can easily fit in with their everyday programme implementation;
a means of systematically gathering programme information and reporting results through an online database developed and maintained by MSD (NB. MSD won’t access this data);
coaching and support regarding the use of the evaluation processes and tools through training workshops, field visits and through the use of the Internet; and
opportunities to share their learning with other NGOs

The other main goal of the project is for NGOs to inform MSD, and government more generally, about the “real world” issues (and options) associated with running effective family violence prevention initiatives.

Project stages
The project involves three stages, which are outlined in the table below. These include:

negotiating the participation of NGOs or clusters of NGOs (i.e. national office and 4-6 affiliated organisations), and finding out in detail what each group/service does, what information it collects and what other information it needs to gather for programme improvement and reporting (2006-2007);
trialling the use of particular evaluation processes/tools and using an online database to routinely gather data and report achievements and results (2007-2009); and
evaluating the overall project – what has been the benefit to participant NGOs and “end-users” and how can this be sustained and spread throughout the sector (mid 2008 and mid 2009).

Year 1. 2006-07	Year 2. 2007-2008		Year 3. 2008-2009
Negotiate NGO partnerships and institutional reference group	Training in use of evaluation tools/processes, data management and reporting.		Evaluate for key learnings upon conclusion of Year 2. trial – evaluate key learnings gleaned over 12 months – make necessary changes to tools and database structure.
Service modelling – identifying the essential components of the FV prevention initiatives	Trial self-evaluation and monitoring tools/processes over a 12 month period		Expand trial – run for a second year with some NGO clusters and/or include other NGOs to increase reach
Explore evaluation and monitoring processes and reporting requirements – “what works” and “doesn’t work” so well	Fine-tune evaluation tools/processes based on three monthly check-ins and structured reflection with NGOs. Also record wider learning
Develop customised data gathering & analysis processes that meet organisational needs (service development / external accountability)		Extensive dissemination of evaluation tools/processes to wider FV prevention providers and social service sector. Formalising guidelines and planned changes to MSD capacity development, contracting, reporting and external evaluation protocols (enabling “results”)

The project will be organised so that clusters of participating NGOs will enter the trial in a sequenced way, perhaps three months apart. This will allow the evaluation project team to give each cluster of NGOs the right amount of attention to ensure things go as smoothly as possible.

Liaison
The project manager and main contact for the project is Geoff Stone, Senior Analyst, CSRE.
Email: geoff.stone002@msd.govt.nz DDI 04 916-3613

Summary of intended benefits
Intended benefits of the self-evaluation toolkit project are that it:

Clarifies relationships and expectations between agencies that collaborate and between agencies and funders
Gathers evidence of benefit or risk for each main organisational activity.
Creates or refines systems for review and internal learning. This can build understanding, morale, purposeful direction and (re)commitment.
Provides evidence for annual and strategic planning.
Provides evidence for supporting funding proposals and (re)negotiating contracts, including the cost of built-in monitoring and evaluation.
Covers existing reporting requirements while also providing a basis for programme and sector advocacy. This would draw upon data related to important changes in local conditions, unexpected programme demands or impacts.

These project benefits are all expected to contribute to improved programme information, programme delivery and social results – for both MSD and partner NGOs.