AI Assessment with Educ8
Educ8, a leading South Wales-based training provider, partnered with CEMET to assess the potential of integrating Large Language Models (LLMs) into their platform to support Training Coaches in evaluating student coursework.
Educ8 delivers high-quality vocational training across South Wales, offering programmes such as Jobs Growth Wales Plus, ReAct, traineeships, and apprenticeships. The organisation supports learners and employers of all sizes—from micro-businesses to global corporations—through its commitment to accessible, career-focused education.
As AI technologies rapidly evolve, Educ8 was keen to investigate how LLMs like ChatGPT, Gemini, and Llama might be integrated into their platform to enhance the assessment process. However, before committing to development, Educ8 engaged CEMET to lead a structured research project that would explore the feasibility, risks, and longer-term implications of adopting such tools.
Educ8 wanted to understand whether LLMs could assist their Training Coaches by partially automating the evaluation of student coursework. However, the team recognised the need to thoroughly assess the implications of this move—particularly around cost, data privacy, environmental concerns, ethical responsibility, and long-term risks. The challenge was to gain a comprehensive picture of how LLMs could be deployed responsibly, effectively, and sustainably within their platform and workflows.
What Did We Do?
CEMET undertook a research project focused solely on investigation rather than development. The initial phase explored the capabilities of three key LLMs—OpenAI’s ChatGPT, Google’s Gemini, and Meta’s Llama—benchmarking their ability to assess coursework, interpret grading criteria, and respond accurately to training-relevant prompts.
In parallel, our team examined each model’s associated data privacy risks, particularly in relation to GDPR and user data retention. A full cost analysis compared token-based API billing with on-premise hosting requirements, including the expense of high-performance hardware. Environmental and ethical implications were also reviewed, particularly those related to energy usage, potential bias, and algorithmic transparency.
CEMET supported Educ8 in designing UX research tools, including feedback forms for both Training Coaches and learners. These tools were designed to gather insight into user trust, perceived usefulness, and the broader impact of AI in the learning environment. Our final deliverable was a detailed report summarising the findings and offering a series of strategic recommendations.
The Results
CEMET’s research provided Educ8 with a comprehensive understanding of the practical, financial, ethical, and technical considerations surrounding the integration of Large Language Models into their assessment process. While the technology shows promising potential for enhancing the efficiency and consistency of coursework evaluation, a number of important challenges were identified—particularly relating to data privacy, long-term costs, environmental impact, and implementation risks.
The final Knowledge Transfer Document delivered clear, evidence-based guidance on how Educ8 could proceed, including recommendations for pilot testing, data protection strategies, cost optimisation, staff training, and future scalability. With these insights, Educ8 is equipped to make a confident, well-informed decision about whether and how to integrate AI tools into their platform in a way that aligns with their values and strategic goals.
CEMET’s research has empowered Educ8 with the insight needed to explore emerging technologies responsibly, building the foundations for a considered and sustainable approach to innovation.. To find out more about their latest work visit their website.
This project is funded by the UK government through the UK Shared Prosperity Fund and supported by Caerphilly County Borough Council.