Building Evaluation and Feedback agents

This paper explores the enhancement of large language models (LLMs) for personalized education, specifically in dialog tutoring. Current LLMs struggle to accurately detect student errors and tailor feedback. The authors introduce error detection verifiers grounded in real-world teaching practices and demonstrate, through empirical evaluations, that these verifiers improve the accuracy and relevance of tutor responses,

The paper introduces PROMETHEUS, an open-source large language model (LLM) designed to evaluate long-form responses with customizable criteria, addressing limitations of proprietary LLMs like GPT-4. Trained on a new dataset, FEEDBACK COLLECTION, PROMETHEUS achieves a Pearson correlation of 0.897 with human evaluators, matching GPT-4’s performance. The model also outperforms other open-source models on human preference