Paper: Surprisal as a Predictor of Essay Quality

ACL ID W14-1807
Title Surprisal as a Predictor of Essay Quality
Venue Innovative Use of NLP for Building Educational Applications
Year 2014

Modern automated essay scoring systems rely on identifying linguistically-relevant features to estimate essay quality. This paper attempts to bridge work in psy- cholinguistics and natural language pro- cessing by proposing sentence process- ing complexity as a feature for automated essay scoring, in the context of English as a Foreign Language (EFL). To quan- tify processing complexity we used a psy- cholinguistic model called surprisal the- ory. First, we investigated whether es- says? average surprisal values decrease with EFL training. Preliminary results seem to support this idea. Second, we in- vestigated whether surprisal can be effec- tive as a predictor of essay quality. The results indicate an inverse correlation be- tween surprisal and essay scores. Overall, the results are prom...