Not for Profit: Catholic Education Diocese Parramatta
The challenge of how to assess and improve student performance at school has been a vexing one since time immemorial.
These days, there’s a fair degree of consensus that things like ethnicity, geography, socio economic status and language play a much bigger role than many of us might be comfortable admitting.
However, for most students – and indeed their teachers – such factors are usually only considered after the fact; once a student has completed, say their final school exams and received results that might actually impede them moving to the next level of education or in entering the workforce.
It’s a problem that Dr Raju Varanasi, chief information officer and director data intelligence with Catholic Education Diocese of Parramatta and his team have been wrestling with for quite some time, but especially since the onset of the pandemic.
This year’s winners of the inaugural ‘CIO50 Best Project in Not For Profit’ category, CEDP, led by Varanasi and his team started with a simple yet critical question: If there are clearly understood factors that can predict how a student will perform, why not try to employ predictive analytics to better understand those factors and then take action to mitigate against them before condemning them to a lower result than they deserve and which the data already show they’re destined to receive?
While student performance is important at all levels of school, CEDP decided to focus on its more ‘high-stakes’ HSC – or final year – students.
Still, Varanasi and his team working with the organisation’s Data Intelligence Unit were given the massive task of analysing an eye-popping 43,000 HSC students and more than 5,000 teaching staff across 82 schools.
The result was a predictive analytics engine capable of guessing with an impressive 93.6% degree of accuracy what an individual HSC student would score in their final school exams.
Key to the success of the initiative, which went live in 2021, was the use of automated machine learning systems that interfaced with CEDP information services to develop a workable platform, not only to extract the data, but also to present it visually in the hopes of yielding further intelligence and insight.
“CEDP schools can now use AML to foreshadow final HSC scores one full academic year ahead of the exam across various subjects such as English, mathematics, ancient history and biology,” Varanasi tells CIO Australia. “CEDP has enabled its schools to move from a retrospective mindset to a predictive and supportive work program”.
Teachers are now able to link multiple data sources using desktop “dashboards” revealing trends and patterns for each student across numerous data points.
Schooling the AI
Naturally, assessing reasons for why one student performs better or worse than the next involves many different variables.
As mentioned above these include things like ethnicity, religion, language, geography and socioeconomic circumstances. Then there’s a student’s experiences of learning as well as the many other activities across the broader school environment. Added to that there’s actual student attendance, behaviour, literacy, numeracy, choice of subjects, the school itself and the quality – or effectiveness – of teaching provided.
Fortunately, Varanasi and his team were able to hit the ground running with an existing bank of 10 years’ worth of CEDP data on 20,000 students between 2009 and 2020. These data were used to train the AML algorithms in the CEDP DataRobot platform.
Varanasi notes that merely understanding these data sets required ”much technical knowledge”, let alone what’s needed to ensure the accuracy of predictions.
“[We] quickly realised that the linear, statistical approach would not suffice with so much data available. Instead, predictive analytics was used and expanded to machine learning to enable staff to predict which students are most vulnerable and most likely to benefit,” he explains.
He and his team settled on a model that considered 41 input variables to identify relative importance and influence to predict the target variable – in this case HSC score – for different subjects.
“Blending internal captured data (demographics, attendance, socio-economic and student performance) with externally-generated data (teacher certifications, HSC, NAPLAN, community growth, parent perception surveys) was challenging but provided actionable insights for executives.”
More than just numbers
Getting into the ‘math’, Varanasi notes that accuracy is consistently over 90% across the score bins from the 60-90 score range, and the error distribution is a normal curve, mainly in the -5 to 5 error range.
This was especially important, he notes, if the data were to be “confidently” shared with various stakeholders.
Furthermore, it needed to be more than just ‘numbers’.
“The challenge was getting the model right, which involved many internal discussions and work to make the statistics easy to visualise and use," he says.
The CEDP approach is unique in that it presents data through easy-to-use visualisations available to every principal which enables them to make evidence-based decisions and allows schools to make real-time interventions regarding student learning.
“Data is not just about raw percentages; the 41 input variables spring to life visually on a dashboard, delivering an organic feel for the student's past and present capabilities,” Varanasi adds.
“Numbers become story points and weave a narrative contextual to the school and the student. Predictive analytics is not about measurement just for the sake of comparison but for the judgement and quality of decisions made to support students.”
This provides a better way for staff to interact with students and parents when speaking about course results.
There are myriad benefits of doing this, not least of which being reducing uncertainty about exam performance, the enhanced ability for teachers to focus on students’ pain points and reducing, even eliminating, the need for school leaders to conduct a retrospective analysis of exam results.
The predictive power of the solutions means teachers are spared onerous, time consuming administrative work with more freedom to explore the possibilities for intervention based on data.
“By reducing the burden on teachers and principals, the CEDP is enabling them to move away from administrative tasks, allowing them to spend time on the more critical matter of educating students,” Varanasi notes.
Meanwhile, at an organisational level there were significant time savings, as well as reduced administrative and data analytics costs compared with more traditional, manual processes for acquiring insights. Further, having established processes and technologies to ensure protection of these data from the outset meant that critical aspect of the current project was always under control.
“The analytics strategy has resulted in better student outcomes and school performance and significantly improved every student's experience,” Varanasi says.
And while the scale of what Varanasi and his team were able to achieve from a data analytics perspective is undeniable, he stresses the importance of not losing sight of the core stated objective, that being to help students develop a more personalised education journey, get better results and ultimately live better lives.
“The most important beneficiaries of the digital transformation are the students,” he says.
The upshot is that “with better insight, students and teachers don’t need to wait until after the HSC to determine where more support is required”.