Analytic Confidence

The need for analytic confidence ratings arise from analysts' imperfect knowledge of a conceptual model. An analytic confidence rating pairs with a statement using a word of estimative probability to form a complete analytic statement. Scientific methods for determining analytic confidence remain in infancy.

In an effort to apply more rigorous standards to National Intelligence Estimates, the National Intelligence Council includes explanations of the three levels of analytic confidence made in estimative statements.

• High confidence generally indicates judgments based on high-quality information, and/or the nature of the issue makes it possible to render a solid judgment. A “high confidence” judgment is not a fact or a certainty, however, and still carries a risk of being wrong.
• Moderate confidence generally means credibly sourced and plausible information, but not of sufficient quality or corroboration to warrant a higher level of confidence.
• Low confidence generally means questionable or implausible information was used, the information is too fragmented or poorly corroborated to make solid analytic inferences, or significant concerns or problems with sources existed.

Analytic confidence beginnings coincide with the cognitive psychology movement, especially in psychological decision theory. This branch of psychology did not set out to study analytic confidence as it pertains to intelligence reporting. Rather, the advances in cognitive psychology established a groundwork for understanding well calibrated confidence levels in decision making.

Early accounts of explaining analytic confidence focused on certainty forecasts, as opposed to the overall confidence the analyst had in the analysis itself. This highlights the degree of confusion among scholars about the difference between psychological and analytic confidence. Analysts often lessened certainty statements when confronted with challenging analysis, instead of proscribing a level of analytic confidence to explain those concerns. By lessening certainty levels due to a lack of confidence, a dangerous possibility of misrepresenting the target existed.

The Intelligence Reform and Terrorism Prevention Act of 2004 establishes some guidelines for conveying the analytic confidence in an intelligence product. The summary document states each review should include, among other things, whether the product or products concerned were based on all sources of available intelligence, properly describe the quality and reliability of underlying sources, properly caveat and express uncertainties or confidence in analytic judgments, and properly distinguish between underlying intelligence and the assumptions and judgments of analysts.

 

Mercyhurst College students use the Peterson Table of Analytic Confidence Assessment to determine the level of analytic confidence in their estimative statements. The table outlines certain areas in the intelligence cycle important to determining analytic confidence. The key areas of the table include the use of a structured method, overall source reliability, source corroboration and agreement, level of expertise on the subject or topic, amount of peer collaboration, task complexity, and time pressure.