Normalized claim
Quantified impact: 76%
76% of students preferred access to the chatbot over no AI practice option.
Last evidence check: Jun 1, 2026
Australian law students gain instant feedback with AI-powered chatbot
Use case typeStudent supportUpdated Jun 13, 2026
Researchers at the University of Wollongong developed and trialed SmartTest, a custom chatbot platform built with Microsoft Azure OpenAI technologies, to provide instant, formative feedback in criminal law classes. The chatbot, using Socratic dialogue and model answers, allowed students to practice answering legal questions and receive guidance on their learning gaps. Multiple cycles of real-world use revealed that while students appreciated the immediate feedback and conversational experience, significant error rates were observed, especially on complex scenario tasks. These reliability issues required rigorous manual oversight by educators. Student surveys confirmed a strong preference for access to immediate AI feedback, but also highlighted ongoing trust and quality concerns. The project underlines the labor-intensive nature of responsible AI deployment in education and the necessity of robust human supervision.
- Organization
- University of Wollongong
- Industry
- Education
- Location
- Australia
- Published
- May 2025
Reported outcomes
76%
quantified impactOther quantified impact
40-54%accuracy
Strategic outcomes
New product / capabilityBuilt an AI tutoring chatbotCustomer experience & trustEnabled instant formative feedbackCustomer experience & trustImproved student practice opportunitiesBetter decisions & insightIdentified AI reliability limitations
Why do we believe this?Outcome claims, sources, and evidence checks
Normalized claim
Accuracy: 40-54%
Error rates of 40-54% in complex scenarios and 6-27% in simple answers highlighted AI limitations.
Last evidence check: Jun 1, 2026
Why do we believe this deployment?Customer identity, provider attribution, maturity, and source checks
- Customer
- University of Wollongong
- Provider
- Microsoft
- Maturity
- Scaled Production
- Linked source
- theconversation.com
Manual marking and feedback processes were time-consuming and unsustainable at scale
Customer identity supportedSource describes one deploymentMaturity supported
Primary read
Use case focus
Showing 3 of 3
- 1AI-Powered Student Feedback Agent
- 2Chatbot for Interactive Legal Education
- 3Automated Socratic Tutoring
- Faculty workload was high due to delivering personalized, formative feedback to law students.
- Delayed feedback diminished learning effectiveness.
- Manual marking and feedback processes were time-consuming and unsustainable at scale.
- Built a custom AI chatbot (SmartTest) using Azure OpenAI Service (ChatGPT-4/4.5).
- Enabled Socratic, real-time feedback on legal questions during tutorials.
- Iteratively improved prompts and evaluated different model versions for reliability.
- Educators maintained manual review and quality control over chatbot outputs.
Technologies
- 76% of students preferred access to the chatbot over no AI practice option.
- Error rates of 40-54% in complex scenarios and 6-27% in simple answers highlighted AI limitations.
- Increased formative engagement and practice opportunities for students.
- AI-driven approach saved some labor but required extensive oversight due to reliability issues.
Sources & evidence1
Evidence: High75/100Evidence strength
- Customer explicitly identified
- Deployment status explicitly supported
- Independent source available
- Quantified outcome available
- Technical implementation details available
- Recent evidence check available
- Last evidence check: Jun 1, 2026.
Live sourceStill referenced
The case's original source is still reachable.
- Cited source last checked Jun 1, 2026 — ok (0/1 broken).
Measures whether this deployment's public evidence persists — not whether the system is still in production.
Type: Research ReportPublished: May 29, 2025Publisher: theconversation.comEvidence: SecondaryConfidence: Low
AI-generated summary. Verify important details with the linked sources before relying on this case.
Explore related AI use cases
Customer evidence
Provider evidence
AI patterns
Browse the catalog
Was this useful?
Community
Comments
No published comments yet.