What are the risks of using AI for investment analysis?
The primary risks of using AI for investment analysis are overconfidence without calibration, hallucinated or fabricated supporting evidence, sycophantic reasoning that confirms rather than challenges the analyst's hypothesis, and data sovereignty exposure when sensitive deal materials are processed through shared infrastructure. Research shows that AI systems trained through standard methods are systematically overconfident and that users cannot reliably distinguish between correct and incorrect AI outputs.
Overconfidence and calibration failure
<p>AI systems trained through reinforcement learning from human feedback are systematically overconfident. Research demonstrates that the training process encodes human annotators' bias against uncertainty expressions — annotators consistently prefer confident-sounding answers, which trains the model to present uncertain conclusions with unwarranted certainty.</p> <p>In investment analysis, this means an AI system may present a revenue projection assessment with high confidence even when the underlying evidence is weak or contradictory. The analyst sees a definitive-looking output and has no reliable way to distinguish genuine confidence from trained-in overconfidence.</p>LLMs are systematically overconfident because RLHF training encodes human annotators' bias against uncertainty expressions. Zhou et al. (2024), arXiv:2401.06730