This positioning paper is the first in what will be a series of papers, blogs and insights as part of our commitment to understanding innovation and systems change within a te ao Māori paradigm. This paper outlines our rationale for using a DE approach within our overall organisational structure and how we are aligning our core values with our performance and value measurement.
Developmental Evaluation (DE) is an evaluation approach ideally suited to complex or uncertain environments, especially where the focus of the evaluation is to inform social change. It is similarly useful where the solutions to the problems are unknown, previous approaches to the problem unsuccessful, and the landscape of the problem is dynamic and changing. It supports social innovation as its focus is to provide opportunities, insights and learnings, (OIL) to decision-makers as initiatives develop. The use of the acronym also refers to the Kohia – to draw oil (in this sense reflections and insights) nourishes organisational social health. As social change solutions are almost always dynamic, complex and developed over time DE supports this emergent learning approach. Further it enables rapid feedback to inform social change initiatives in a continuous feedback loop process. In its simplest form DE is the fuel for decision-making.
Within the TTR organisational framework DE is a scalable tool that is used for quarterly evaluations of living projects, wānanga, future-making cohorts (labs), leadership groups and innovations. Based on the Tokona Te Raki ‘Case for Change’ (www.maorifutures.co.nz) our framework has been designed to simultaneously capture insight, real-time data and provide staff with a simple but effective tool to reflect on their own practice. It seeks to enable staff to uncover underpinning assumptions and beliefs (noted in the theory of change: www.maorifutures.co.nz) and works to turn staff inputs into actionable items for organisational leaders. Importantly, our organisational DE approach provides a means to understand to what extent our guiding principles are showing up in our practice. This will help us to articulate if we are ‘walking our talk’ and if what we espouse or say we do, is reflected in our practice in the community. Measuring how well and in what ways our principles are showing up in our key priority areas will inform our strategic direction and support us as a team to refine and innovate our practice considering the integrity of our core organisational values.
A key component of the DE approach to evaluation is not only measuring the impact of living projects, future making (labs) and wānanga initiatives in light of the long-term change theories and goals, but to also capture innovation and creativity across the organisation. It provides a basic outline that has been designed to be a pragmatic and living document.
The more evaluators try to play it safe and be value-neutral, the more fuel we give to those peddling toxic misinformation in the post-truth ērā”Jane Davidson (Tweeted by Adriene Field, 17th September 2019)
Our core business is to mediate and facilitate Te Rūnanga o Ngāi Tahu (TRoNT) tribal focus on advancing Māori and Māori achievement outcomes in the education and vocational pathway sector. TTR achieves its goals by creating broad, collectivist support systems that lead capability building within the social innovation and design space, and that nurtures cultural change in industry, teaching and learning environments and policies that embrace te ao Māori principles, values, and epistemologies. TTR innovates to bring about social and policy changes, by ‘doing things differently’. Generating new solutions requires a disciplined approach to change. Our role is to support TRoNT to revision ‘Business as Usual’ (BAU) and align its core mahi with future-focused systemic change initiatives. A key premise of this work is that building capability across diverse teams supports innovation which in turn leads to positive educational and vocational outcomes for Māori.
DE In practice
At an operational level, once a fortnight our team participate in DE meetings (hui) to reflect on their practice, articulate their impact, insight and engage in shared data analysis processes that seek to foster collective vision and meaning. Quarterly these insights are analysed as a collective and shared back via full staff engagement at data parties. Once a month using our Te korekorenga theory (abundance mindset kaupapa Māori DE model see Tarena et al.,2019 article in press for full description of Te korekorenga model) staff engage in collective reflections on their Te Po (planning) and Te Ao (doing) activities. This provides us with an opportunity to reflect as a team and understand each other’s mahi in terms of the big picture of the organisation and our overall vision. As well as developing an internal learning culture through the DE approach we are working to capture tangible data to ascertain the value of initiatives to whānau in alignment with our key principles.
Timely data collection at pivotal points in our projects will be planned ahead for 2020 to ensure we are collecting the right amount and variety of data to give a thorough overview and to answer our research questions. Understanding and articulating the impact of TTR in achieving its goals is key to our DE approach to learning. We also seek to highlight areas for further growth and focus as we move forward over the coming two years. More information on the data collection, analysis and matrix used to measure tangible and intangible outcomes will be posted in the next paper.
The DE framework seeks a dynamic and holistic review of outcomes, specifically with regard to what’s working, what is innovative and what is having a meaningful impact on whānau lives. Note that ‘innovative’ is not intended here in a global sense, ie, that we’re undertaking completely novel approaches. Rather, TTR is looking at local and contextual innovations: ‘What TTR has done in the context of projects, capability building and kaihautū leadership advocacy, that was innovative. It also brought about systemic and/or mind-set, behaviour change where there was stagnation’? An example of innovative practice was captured recently when a project manager applied new tools and techniques to the way she ran her consortia meetings, and these approaches were taken on board and applied in other organisations by leaders attending these meetings. Another example includes a leadership team we are working with who recently adopted our organisational vision and have included this in their strategic goals as part of their commitment to this social movement. An important question underpinning the organisational approach to evaluation is understanding ‘how can this innovation be replicated in the future’? ‘How does this innovation reflect our core principles’? and ‘Is It having an impact on whānau’? Other examples of innovation include: ‘What value has the rangatahi future-making collective had in building capability and working towards the long-term goal of training the future generation up to lead in innovation’?
The Systems we occupy
Following Michael Quinn Patton’s principles in terms of ‘sensitising concepts’ throughout our organisational approach we are hoping to include the eight principles below:
• Developmental Purpose
• Evaluation Rigour
• Utilisation Focus
• Innovation Niche
• Complexity perspective
• Systems Thinking
• Timely Feedback
(Patton, Mckegg, wehipeihana, 2015)
As noted by Patton et al., (2015), utilising a DE framework includes reframing to maximise the learning based on the actors who are doing the interventions. This suggests that those who are part of the intervention are part of the system in which they are operating, and therefore we are most interested to learn how our role supports the shifting of mental models, and to learn more about the systems in which we occupy.
The DE encourages staff to be participatory in their engagement with their own, and the organisations learning processes, as when they enter the system they become part of the system. Staff are tasked with reporting on key information distilled from project reporting documents; stakeholder organisations’, namely, key personnel that liaise with TTR; TTR staff; programme champions; and, where appropriate, programme participants. More importantly, however, staff are prompted to think and comment on how their projects successes and failures inform how TTR innovates, changes, and adapts to meet the changing context of the sector. In this way we are able to identify processes that support innovations. With a focus on organisational wide DE, we are in a stronger position to replicate new ideas in projects as a way to identify and assess strategies, activities, and champions of good practice that are effective toward achieving the organisation’s aims and goals.
Why Organisational Based Evaluation?
By focusing on an organisational approach to evaluation there is greater capacity to develop tight feedback mechanisms that will support staff capacity to understand the impact of the project at a macro level while keeping grounded in micro activities. The methodology of this approach is designed to build transformative solutions to redressing inequity by focusing on what’s working well, especially in frames of testing assumptions in contexts where change is stagnated. We also hypothesize that as we are in the emergent stage of our organisational development, taking an organisational approach will support us to understand how we are practicing our principles and at the same time articulate our value both in intangible (unmeasurable) and tangible ways (Measurable). As an organisation we are in our first year of growth and as Dr Ingrid Burkett (social innovation expert) so eloquently stated “At the moment, everybody’s obsessed with measurement… nottelling us anything about: Are we doing the right things? Are we doing things right? And are we really making a difference across systems? That, to me, is evaluation. That’s the big picture ‘box’ (Cited by King, September 26th, 2019).
Within our mahi the fundamental purpose of measurement in evaluation changes from using measurement to show change to using measurement within the DE framework to answer these deeper more meaningful questions. We are eager to learn how what we are doing is making a difference and if we are doing the right things in the right way.
Teaching social systems and how to seek solutions
The Developmental Evaluation approach is about building capacity across the system to engage with systems change; philosophy and approaches to become a more responsive. In a similar way to the saying, ‘if you give a man a fish you feed him for a day, if you teach a man to fish he will be fed for life’. The mahi of TTR is not to give social systems a solution, but to teach social systems how to seek solutions. As such, DE not only becomes essential for our own mahi, but looking forward, our ability to empower the use of DE to support social change in the wider rōpū is an even more important goal. Prioritising how the TTR collaborative is able to innovate in order to envision new and unexpected ways forward to shape practice will support TTR to meet complex and varying demands of the cross-sectors we work within.
Why Mixed Methods?
Social systems that underpin the work of TTR are complex and dynamic. By their very nature of involving social interaction, social systems are organic, changing as contexts change, social actors come and go, and values and perceptions change. As such, no one method can capture the dynamic nature of a social system, it takes a range of methods that ‘point’ toward the answers we seek. TTR uses this range of methods, not only in parallel to provide multiple sources of evidence, but also over time as the social context changes. It is also true that to make informed decisions around changes to improve social outcomes change agents require more understanding of the situation than purely how many or who are.
Questions such as: how many, what proportion, and how long TTR seeks answers from quantitative methods. These focus on the use of numbers to understand situations. For example, measuring quantitatively pass rates, or income levels, can identify lower levels of achievement for Māori. When we use quantitative methods, we focus on the reliability (will we get the same result if we measured it again and again?) and validity (are we actually measuring what we claim?) Exploring why these may be the case, however, is also very important to our mahi.
Qualitative methods are used by TTR when we are seeking to explore or describe something. Exploring why Māori may have lower completion rates, may require exploring people’s perceptions, understandings or beliefs. Qualitative methods are often better when exploring the unknown, especially when it is a complex social situation as constitutes much of the context of TTR mahi. To ensure the trustworthiness of qualitative data, TTR focuses on the credibility, dependability, transferability and confirmability of the data. As we note, one approach is not better than the other, they support each other in exploring new ideas and innovations and then measuring the impact of these. Layered on top of this will be the need to use qualitative methods to provide on-going insight and transformative potential of innovation across the organisation while quantitative data will provide evidence of demonstrable outcomes.
Principles, Research Questions and Matrix
|Organizational Principles||Exemplar of Research Questions|
How well do our priorities align with our guiding principles?
|Exemplar of evidence we will collect|
|Rangatahi are the solution (How are we |
investing in future makers and building capability
through knowledge, skills and tools).
|How well are we realising the potential of |
our growing rangatahi capability movement? How well and in what ways are rangatahi driving
our solutions for future-making?
|Rangatahi lab feedback; surveys, focus groups |
Rangatahi capability building – accreditation,
completion of facilitation qualifications
Rangatahi pass rates of course material
|Change starts with us (Speaks to systems |
change – connection to ourselves and our
partners. How we are in relationship with partners strengths and challenges. How we understand
what is going on at a systems level).
|How well and in what ways are we recognising our own part in the ‘system’? In what ways are |
we developing skills, dispositions, capabilities that enable us to apply DE learning principles to our
How well is Te Korekorenga supporting our personal and professional practice? E.g. How are we
working with whānau, funders?
How well are we communicating our key
learning to our wider network?
How are we driving shifts in narrative?
How are we measuring our impact on whānau future-making beliefs, values and practices?
What are we doing to perpetuate success and / or challenges?
How are we growing our own practice to achieve
How are we tracking as an organisation?
|DE – Individual and |
collective Insights (Qualitative interviews).
Data Parties – reflections, team learning.
Using evidence to point to innovation and solutions.
Increase in skills, knowledge and learning of DE,
design theory and social innovation in team.
Change stories, blogs and learning reports.
Theory of change – whole organisation and
|We meet whānau where they are and support |
them to reach where they want to be
|How well and in what ways are we working |
with whanau to drive whānau-centric solutions?
Understanding whānau perspectives of the future – what information are they receiving? What is
informing their view? Where do they want to be?
|Living examples – Community projects.|
Qualitative feedback from whānau (engagement
and regular testing of facilitation process)
|Whānau are at the heart of all we do||How are whānau reflected in all of our mahi?|
In what ways do whānau see, feel and hear
themselves in TTR?
What is our role in unleashing whānau potential?
How can we work with community
organisations already doing great practice to
maximise whānau reach?
How well do we capture diversity of whānau
|Whānau insights through in depth interviews.|
Other exemplars include 2uantifiable statistics suchas Māori in meaningful careers, qualifications, work readiness and employableskills. Support into higher level training and qualifications. Growth in
Quantitative future reports.
|Treaty partners own their power and bring their |
strengths to the kaupapa.
|In what ways are we working to develop our treaty partnership relationships? Strengths/Challenges.|
How are our partners demonstrating their power,
advocacy and making changes?
What’s shifted for them in terms of power and how they use this to advocate?
|Cross-sector partner surveys.|
In-depth interviews with Kaihautū group building
cross-sector range of knowledge, skills and
experience to lead out change in their own
networks through our wānanga series.
Ideally, we are seeking to maximise the learning across the organisation by having teams share learning to increase innovation, collective learning, and to build synergy between the living labs and advocacy mahi. We anticipate that once all teams are working to individually collect, analyse and reflect on data this will help facilitate sharing across teams and greater overarching learning will emerge.
We will be collecting data from each of the initiatives over the next 12 months to explore if what we are doing is living our values and making a difference in the lives of rangatahi and/or effecting systems level changes?
The next blog in this series will discuss our rubric to measure change and how this aligns with our Te Korekorenaga innovation and practice model.
For more information please contact Porsha: firstname.lastname@example.org