SWAG

Software / App

Another benchmark that appears to suffer from the 'first character equals final answer' issue, similar to issues found in the MMLU.

Mentioned in 1 video

Videos Mentioning SWAG

SmartGPT: Major Benchmark Broken - 89.0% on MMLU + Exam's Many Errors

SmartGPT: Major Benchmark Broken - 89.0% on MMLU + Exam's Many Errors

AI Explained

Another benchmark that appears to suffer from the 'first character equals final answer' issue, similar to issues found in the MMLU.