The study by UC Riverside sheds light on the complex relationship between confidence and correctness in large language models (LLMs). Researchers discovered that these models can express confidence in their answers even when incorrect, and be uncertain yet correct. This challenges the conventional belief that confidence is a reliable indicator of accuracy.
Key findings include:
-
Internal Features Dissonance: The study identifies distinct internal features linked to confidence and correctness in LLMs, suggesting that developers can modify models for better performance without complete retraining.
-
Experimental Approach: Researchers utilized two open-weight models, Meta’s Llama-3.1-8B and Google’s Gemma-2-9B, to assess how internal features affected model behavior through isolated feature manipulation.
-
Impact of Feature Suppression:
- Suppressing features related solely to uncertainty decreased accuracy.
- Altering “confounded” features, linked to both uncertainty and incorrectness, improved accuracy by up to 1.1% while reducing uncertainty.
-
Practical Applications: The study suggests these internal features could help AI systems abstain from responding when likely to be wrong, significantly raising accuracy.
-
Generalizability: The findings indicate that the identified features could be applicable across various benchmarks, hinting at a broader underlying mechanism in LLMs.
Overall, this research has profound implications for developing more reliable AI systems while suggesting that improving specific internal features could lead to better models. The study will be presented at an upcoming conference, indicating its relevance in the AI community.
