Human Judgment in A.I.-Assisted Management Decisions
Know When to Rely on AI Advice, and When Not To
A calibrated-judgement framework for managers in nonprofit and non-governmental organisations who receive advice from AI tools.
Managers increasingly receive AI advice when they forecast demand, allocate staff and budgets, screen applications and assess risk. The research points in two directions: people can abandon a good algorithm after seeing it err, and they can follow advice selectively when it matches their expectations. This working paper argues that the goal is neither trust nor distrust, but calibration.
What you will take from it
- See why reliance on AI advice should match its reliability for the decision at hand, rather than a general attitude towards technology.
- Understand two experimental findings: algorithm aversion after observed errors, and selective adherence to stereotype-aligned advice, with no evidence of automation bias in the studies concerned.
- Connect these with research on accountability and advice rationales, conditions for intuitive expertise, complacency under workload and the automation–augmentation paradox.
- Apply six testable propositions on advice rationales, track records, independent first judgements, workload, decision structure and preserving skills over time.
- Take practical steps for leaders, such as recording an independent judgement before viewing AI advice and comparing AI and human track records on recurring decisions.
The central claim is that good AI-assisted management depends less on trusting or distrusting tools than on designing the conditions under which managers can judge when to rely on them.
Summary
Managers in nonprofit and non-governmental organisations increasingly receive advice from artificial intelligence tools when they forecast demand, allocate staff and budgets, screen applications, prioritise programmes and assess risk. The central management question is no longer whether such advice is available but how human judgement should be combined with it.
This conceptual paper develops a calibrated-judgement framework for AI-assisted management decisions. It builds on two experimental literatures that point in different directions. Dietvorst, Simmons and Massey report that people became less likely to rely on an algorithm after seeing it err, even when it outperformed a human forecaster.
Alon-Barkat and Busuioc, in three experiments in the Netherlands, found no evidence of automation bias but found stronger adherence to advice aligned with group stereotypes in one study, with no significant difference between algorithmic and human-expert advice. These findings are connected to research on accountability and advice rationales, conditions for intuitive expertise, complacency and automation bias, decision structures and the automation–augmentation paradox.
The framework holds that reliance on AI advice should be calibrated to the predictability of the decision environment, the reasons given for advice and the accountability that attaches to the decision. Six propositions address advice rationales, track records, selective adherence, workload, decision structure and skill preservation.
No original empirical data are presented.
Frequently asked questions
- Is this an empirical study?
- No. It is a conceptual paper. No original empirical data are presented. The six propositions are theoretical and require empirical testing.
- What does "calibrated judgement" mean?
- Reliance on AI advice that matches how reliable that advice is for the decision at hand, taking account of how predictable the decision environment is, the reasons given for the advice and the accountability attached to the decision.
- Does the paper argue for or against using AI?
- Neither. It argues that organisations should design the conditions under which managers can judge when to rely on AI advice and when to question it.
Publication details
- Author: Thomas A. Guerriero (Thomas A. Guerriero, PhD; ORCID 0009-0001-9456-8654)
- Published: 2026-11-21
- Type: Working paper (conceptual; no original empirical data)
- DOI: https://doi.org/10.5281/zenodo.24398787
- LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:7529552491407917056/
- Academia.edu: https://www.academia.edu/181477965/Human_Judgment_in_A_I_Assisted_Management_Decisions
- ResearchGate: https://www.researchgate.net/publication/418141043_Human_Judgment_in_AI-Assisted_Management_Decisions
- Zenodo record (DOI target): https://zenodo.org/records/24398787
- PDF: human-judgment-ai-assisted-decisions.pdf
More from Thomas Guerriero
See all research at Research, and the main hub at ThomasGuerriero.com. Also: ThomasAGuerrieroPhD.com · ThomasAGuerriero.com.
This page summarises a working paper by Thomas A. Guerriero and provides the PDF.
Don't get left behind. Chase greatness.
If you want help growing your business, team, or just want to pursue greatness yourself, book a free 15 minute call at the link below.
cal.com/thomas-guerriero-gvx0zs/15min