SWiS seminar - video transcript

(Moira Wilson - Principal Analyst, Resarch and Evaluation, Ministry of Social Development)

Great to be here today to talk about some of the work that's been going on and is coming up around the SWiS service.  The presentation Min and I have for you nests quite well with AL's and that SWiS is one of the preventive services for children that's intended to help support school engagement and retention.


I need to open with the same disclaimer as AL in broad terms. So the results I'm going to talk about today are not official statistics, access to the de-identified data used was provided by Stats New Zealand in accordance with security and confidentiality provisions in Stats Act and results have been confidentialised. And I also need to acknowledge our co-authors. The work was a lovely collaboration with Dean Heslop from (inaudible) Professor Michael Belgrave from Massey University and Pete McMillan. We also had a lot of help along the way and advice from others and I'd like to gratefully acknowledge that too. It's useful to briefly start with the context for our study. When we started in 2016 government agencies were facing increased demand for evidence that investment in social services was making a positive difference for the families that were being served and the children and the communities and at the same time the Stats New Zealand IDI was opening up new opportunities for impact evaluation that could potentially provide some of that evidence.  The data available relating to children in particular was being greatly expanded at that time. Those data were opening up new opportunities but also posing some challenges. The admin data and the IDI missed many of the important outcomes that are sought by social programmes, where measures of that are available they might provide imperfect proxies for the outcomes that are sought or changes over time might have more to do with changes in administrative processes or service access than real change in outcomes for children and their families.  And they've got the potential for telling a biased story about the prevalence of the outcome we're concerned with, depending on who has service access or who's targeted for a service.  They have important sources of error(?) and even when good measures are available there's only some cases in which robust impact evaluation as possible.  So it's important to view these data as just one possible part of the evaluative story and to take quite a lot of care in interpreting what we see happening in the data.


It was what that context that we went into the IDI to examine the impact of SWiS on children's outcomes.   SWiS is a government funded community social work service available in low decile primary and intermediate schools and kura kaupapa Māori services are provided by social workers employed by contracted social service providers and what the social workers provide as individual casework for children and their families and whanau, group programs.  There's been a reduction in funding for broke group programs over time but those are still part of the mix and community liaison and service coordination for the children and the whanau.  Participation in the service by the children and their families and whānau is voluntary. The evolution of the service is really interesting and we were very fortunate to have Michael Belgrave as part of the team. He was closely involved in the early development of SWiS and in our report, which is available through the Oranga Tamariki and MSD websites, Michael provides a historical overview which traces the origins of the service back to the calls that were being made by iwi in the 1980s for government to resource Māori to provide services to Māori and efforts in the late 1990s to improve the coordination of services for families and whānau under Strengthening Families. So SWiS was part of a new set of interventions contributing to strengthening families which included Family Start and tangentially resource teachers of learning and behaviour.


Schools welcomed the service as a result of the upheavals and the economic circumstances of families that were happening around that time. They felt they had few other options to address what they saw as ever more complex needs of some of their students. After its introduction the service was rapidly expanded and there were a few periods of expansion. The latest expansion was in 2012/13 which took the service into all schools and kura that were decile 1 to 3 at the time and took the number of schools and kura served to 700.  


Most of the children in these schools are Māori and Pacific or Māori or Pacific. While the service is a high intensity one at the individual level its low intensity at the school level so each social worker works with around 16 children at any time. But it's working in a cluster of schools or a school with a large role that might be around 400 to 700 students.  So not a lot of resource is going into schools relative to the needs that might exist in some cases.


Just to quickly take you through some of the past studies that had been done at the time that we started our study. There was some really rich and indepth evaluations done in the early 2000s which found strong support for the service and were really important in shaping its early development.  Kaupapa Māori studies by Rachel Selby, Athena Hollis English and Hayley Bell undertaken some years ago now have affirmed the value of SWIS as a part of a package of programmes delivered by Māori providers.  Māori providers and Māori social workers were able to deliver the service with in a kaupapa Māori framework and that kind of spread to mainstream provision of services as well through a very strong network of Māori social workers and social work conferences.


So SWiS was an important service in broadening out the role of social work in New Zealand.  All of the earlier studies found that many clients and families had deeply held negative views about social workers and that these attitudes could be turned around by good quality and responsive engagement by the school social workers in a voluntary context.  In terms of quantitative estimates of the difference the programme makes there was a study in 2015 which provided some encouraging results. But that study had some limitations and further work was recommended.


So what did our study do?  It estimated the impact of that very large expansion in 2012 on selected outcomes using the IDI. We examined whether there were indications of improvements and outcomes for students by looking at whether they were less likely to have stand downs or suspensions from school, care and protection notifications to what was then Child, Youth and Family and police apprehensions for alleged offending. We used a difference and differences estimation approach and here's the basic outline of that in words. We're looking at differences over time for students in schools that were affected and looking to see if those differences were different to the differences for students in comparison schools over the same period.


And if you like pictures more than equations this one's for you. What we did was compare students in three groups of schools. Schools that were already served by SWIS before the expansion, these were mainly decile 1 3 schools and interestingly they were more likely to be decile 1 to 2 schools than the newly served schools. Our main group of interest were the decile1 to 3 schools that were newly served and we also used as a comparison group decile 4 to 5 schools that had never received SWIS. We were looking to see if there was a reduction in the rates of the adverse outcomes that we were looking at following the introduction of SWIS that occurred in the newly served schools but not the comparison schools.  We would expect students who were the direct recipients of individual casework to have shown the largest difference in outcomes if there was an effect on outcomes. And data from the early implementation of the service had suggested that boys and Māori students, for example, were more likely than average to receive the individual casework.  


So we examined the possibility of these differential effects by interacting the variable that was intended to capture the effects of this SWiS expansion, if it was there, with observable characteristics like being Māori, like being a boy and other characteristics that might proxy for a greater likelihood of being a direct recipient.


We also use that same interaction approach to look to see if the effect was greater in schools that were the base schools for the social workers, where they might have had a greater presence in the daily lives of the students and in kura kaupapa Māori compared with mainstream schools.


We spent a bit of time trying to understand what was going on in terms of services and schools that might have also influenced student outcomes at about around the same time as the expansion. If another programme with similar aims was expanding with a similar patterning across schools then this would be important to know.


We concluded that the expansion of SWiS was the biggest shift in services and schools at that time, but you'll see here that a lot is happening in the school environment over this period and we were only able to capture data at the school level for a subset of the programmes that were expanding.  So we had health promoting schools, milk for schools and a number of other initiatives also ramping up over this time.  These ones on this graphic other ones that we were able to take account of in the modelling.


So what where are estimated impacts for subgroups of students who would be the most likely to be the direct recipients of individual casework provided by the social workers. We found a general pattern of improvements in outcomes in the schools including newly served relative to outcomes for students and the comparison schools. Sixteen of the twenty one interaction effects that we looked at were negative so that's suggesting an improvement in outcomes and five of these were significant at the 1 or 5 per cent level.


So if that the expansion had had no effect we would might have expected ten or eleven of the effects to be negative and we might have expected one of the effects to be statistically significant by chance.


Relative to trends the similar students in comparison schools and kura for Māori boys enrolled in the base schools with the social workers the SWIS expansion was associated with lower rates of police apprehensions for alleged offending and for Pacific students there were lower rates of care or protection notifications to CYF.  So although these results weren't strong statistically we saw them as really encouraging.


So those were results for the subgroups we thought would be the most likely to be the direct recipients of SWIS.  When we looked at all the children on the role without considering the subgroup effects we found no statistically significant evidence that the SWIS expansion improved outcomes for students overall.


We did see a tiny shift immediately the year that the expansion happened and notifications to Child, Youth and Family but that was very short lived.  So there may have been a sort of an anticipatory effect, "Here comes the social worker. We won't need to refer to CYF" then but that was quite short lived.


Before wrapping up I just want to draw your attention to some really interesting results for students enrolled in kura kaupapa Māori. We found no difference in the impact of the SWIS expansion even with those subgroup effects, comparing kura and mainstream schools, but we found associations between enrolling in a kura and large improvements in each of the outcomes we examined and these were all highly significant.  After controlling for other observed differences between students enrolled in the two groups of schools, including some measures that we thought might pick up early adversities and early socio economic disadvantage in the lives of the students.  So these results really clearly invite some further study if you know anyone who's interested in a project, we're really willing to share where we got to as the basis for further study.


It's useful just to circle back round to some of the limitations of this kind of study. Although the large scale expansion of SWIS provided a good opportunity for quasi experimental impact evaluation in this case, the complexity of the school service environment and the low intensity of SWIS at the school level made in estimating impacts difficult, given the wide variation in student needs and in the services that would have been provided in response.  We probably wouldn't have expected large effects on single outcomes in any case.  And again the administrative data has limitations. It did not enable us to look at a range of outcomes for which strong perceptions of positive change in the lives of students and their whānau have been reported in other studies. So when the research that Athena Hollis English and colleagues did, among the indicators that Māori social workers they interviewed were looking for when they were thinking about whether the whānau they were working with were seeing improvements where things like living more positively as a cohesive unit and a willingness to engage in activities both within the school and after school.


Our study could never have got at any of that sort of granular improvement. The other the other thing to say is that in some of the limited overseas literature its impacts on these sorts of outcomes, these kind of less tangible, less academic, less traditional school outcomes that seem to have been the ones that have been moved by these sorts of caseworkers in schools.


So against these limitations - I probably should have called this "strengths and limitations" - against these limitations the strengths of linked data, linked administrative data, is it offers the ability to examine outcomes across domains that we've traditionally looked at in silo.  In our case education, child welfare and justice.  It also offers a long and growing longitudinal data source unaffected by non-response bias.  A large, and in our case, comprehensive sample of the populations of interest allowing us to look at some relatively rare outcomes, so stand downs and suspensions and police apprehensions for offending, for example, for the age group of children that we were looking at are extremely rare.  However, it can only tell us part of the evaluative story and it's important to maintain the flow of qualitative insights as the context in which SWIS operates changes.  For example, all of the qualitative studies to date predate the emergence of Whānau Ora as a programme.


So to conclude, based on the past studies SWiS offers a preventive social works service that's acceptable to families and whānau and experienced as helpful by schools and kura and is seen as having a wide range of important benefits.  In this study there were indications of some encouraging impacts on outcomes that could be measured using linked administrative data for the students most likely to have been the direct recipients of the service.  I'll hand over to Min.


(Min Vette - Team Leader, Services in Schools, Oranga Tamariki)

Ā tēnā tātou.  Tēnā koe, Moia, mo tō kōrero.  So thank you very much for getting us to this point.  I've got five minutes to finish us off.  So, with all those linked data, you know, we still ask the question, so there are some encouraging results there.  And so the question is so how does this happen and what's missing in this data that tells us the story.  Something that we can do something about.  


So what we want to do is to be able to do another study.  We've been asked by prime ministers to add another study to this.  So we're going to go ahead and do a qualitative study.  So there's a new SWIS evaluation that we're starting to get underway and this one is going to look at implementing (inaudible) methodology right from its beginning. The reason for this is because of the encouraging data that we have through the IDI and the majority of Māori clients who are accessing the service.  And wanting to ensure that we tell those stories in the way that they are, in terms of Māori stories of them as Māori. And we want to ensure that the methodology used for that can ensure that we get those stories through as best as we can.


So (inaudible) methodology is something that we've used before.  And it did show some slightly different findings.  And that's the Family Start quasi if you're willing to have a look at that.  It's a method that was developed by Angus McFarland in 2011 as a model for reconciling the western science and teo Māori research and evaluation perspectives.


So I wanted to just for those who are unfamiliar with this method, the analogy used or the metaphor that's used is the braided rivers of the South Island.  Really important rivers for South Islanders. You only get these kind of rivers down the south, we don't have up in the north here.


But I was told by a Ngai Tahu elder about these braided rivers and that they have different streams of water coming through them. One stream comes from the alpines as the snow melts, and so it was once a solid and the other's from the rain it hits the land and then soaks down through into the tributaries and those two streams of water come together.  And chemically they don't merge very well but they do merge when they come into to the awa and then as they flow on they will naturally separate themselves. And then as they hit more tributaries it will merge again. So that's the kind of metaphor or analogy that we'll be using for our (inaudible) methodology is to bring the teo Māori and the Western theory through together, at stages they will merge and then they'll separate and be their own.


So we feel that we will get a perspective that is a joint perspective that will ensure that both of those views can be represented. So in that as well we'll be drawing on other Māori programmes or services that have an effect for families and children as well. And one of those Whānau Ora.  So we'll be looking at how that connects with SWIS. And also looking at international indigenous social workers in schools, they're all over the world.  So these indigenous ones in Toronto, as an example, and Australia. And so bringing those international indigenous services and having a look at some of that alongside what we have here.


This methodology will allow us to bring that through in another way as well.  So this research that we've got underway, it hasn't quite got underway, we're still in the stage of confirming who will be doing it, but it is to follow this quasi IDI evaluation that we've just done and it will align that with a qualitative, so you'll have the quantitative and qualitative sitting together.


So to understand the perspectives of whānau and tamariki. So this is specifically going to get the voices of the children and the families that use the service, that have access to, what it was for them, how they felt about it through what they did and also through their social workers.


So really excited about that, about hearing from the tamariki.  We have heard their voice once before in the research that Athena and Rachel Selby did.  I want to revisit that SWiS model, what it looks like and how it works for schools and kura and ensuring that we capture the view of the Māori tamariki whānau using that kaupapa Maori research. We will be doing quite a lot of work and interviews within kura kaupapa so we want to be able to ensure that that approach is also appropriate for that environment.


And using cultural appropriate responsive methods that we do.  So that's the evaluation going forward, a new one which we're really excited about and we have some good support there for that.  He anoi ra tātou.  That was my little bit.  I'm going to hand over to Davina.


(Davina Jones - Evidence Centre, Oranga Tamariki)  

Well I don't really have a lot to add to what Min said because I think the case has already been made for why we need to continue and move forward. The qualitative studies that we have had, while valuable, are a little out of date and when we had the quasi experimental design ministers found that really valuable. But they also wanted to unpack the black box to know how and why, as Min has said, those results were achieved and to understand how people were experiencing the social workers in schools programme on the ground in both by Māori for Māori settings and mainstream settings.


So really I have one further slide that just talks about all the different sort of methods that we're considering at the moment. We haven't fully landed on exactly what the whole study will look like, but as Min said they'll be both by Māori for Māori streams that we look to dig into and unpack and examine as well as looking at how the model is working generally. So we're going to have a literature review there's going to be a survey and of course we want to carry out interviews to actually hear from people themselves about their experiences and look at SDQ scores as well which is a validated instrument. The strengths and difficulties questionnaire that's also used in the before school check.


So that's a very brief overview, but as I said, we're just shaping that study at the moment with potential partners.