Aggregating expert judgement

• Autores: S. French
• Localización: Revista de la Real Academia de Ciencias Exactas, Físicas y Naturales. Serie A: Matemáticas ( RACSAM ), ISSN-e 1578-7303, Vol. 105, Nº. 1, 2011, págs. 181-206
• Idioma: inglés
• DOI: 10.1007/s13398-011-0018-6
• Texto completo no disponible (Saber más ...)
• Resumen

  • In a paper written some 25 years ago, I distinguished three contexts in which one might wish to combine expert judgements of uncertainty: the expert problem, the group decision problem and the textbook problem. Over the intervening years much has been written on the first two, which have the focus of a single decision context, but little on the third, though the closely related field of meta-analysis has developed considerably. With many developments in internet technology, particularly in relation to interactivity and communication, the textbook problem is gaining in importance since data and expert judgements can be made available over the web to be used by many different individuals to shape their own beliefs in many different contexts. Moreover, applications such as web-based decision support, e-participation and e-democracy are making algorithmic 'solutions' to the group decision problem attractive, despite many results showing we know that such solutions are, at best, rare and, at worst, illusory. In this paper I survey developments since my earlier paper and note some unresolved issues. Then I turn to how expert judgement might be used within web-based group decision support, as well as in e-participation and e-democracy contexts. The latter points to a growing importance of the textbook problem and suggests that Cooke's principles for scientific reporting of expert judgement studies may need enhancing for such studies to be used by a wider audience. © 2011 Springer-Verlag.

• Referencias bibliográficas
  • Adhikari, S., Bayley, C., Bedford, T., Busby, J.S., Cliffe, A., Devgun, G., Eid, M., Wu, S., (2008) Human Reliability Analysis: A Review...
  • Armstrong, J.S., Combining forecasts (2001) Principles of Forecasting: A Handbook for Researchers and Practitioners, pp. 417-439. , In: Armstrong,...
  • Arnell, N.W., Tompkins, E.L., Adger, W.N., Eliciting information from experts on the likelihood of rapid climate change (2005) Risk Anal.,...
  • Arrow, K.J., (1963) Social Choice and Individual Values, , New York: Wiley
  • Ashby, D., Smith, A.F.M., Evidence-based medicine as Bayesian decision-making (2000) Stat. Med., 19, pp. 3291-3305
  • Aspinal, W., A route to more tractiable expert advice (2010) Nature, 463, pp. 294-295
  • Bacharach, M., Group decisions in the face of differences of opinion (1975) Manag. Sci., 22 (2), pp. 182-191
  • Bayley, C., French, S., Public participation: Comparing approaches (2010) J. Risk Res., , (in press)
  • Bedford, T., Cooke, R., (2001) Probabilistic Risk Analysis: Foundations and Methods, , Cambridge: Cambridge University Press
  • Bedford, T., Quigley, J., Walls, L., Expert elicitation for reliable system design (2006) Stat. Sci., 21 (4), pp. 428-450
  • Beierle, T., Cayford, J., Democracy in Practice: Public Participation in Environmental Decisions (2002) Resources for the Future
  • Bennett, P.G., Calman, K.C., (1999) Risk Communication and Public Health: Policy Science and Participation, , Oxford: Oxford University Press
  • (2010) Risk Communication and Public Health, , 2nd edn., P. G. Bennett, K. C. Calman, S. Curtis, and D. Fischbacher-Smith (Eds.), Oxford:...
  • Bier, V.M., Implications of the research on expert overconfidence and dependence (2004) Reliab. Eng. Syst. Saf., 85, pp. 321-329
  • Bolger, F., Wright, G., Assessing the quality of expert judgment: Issues and analysis (1994) Decis. Support Syst., 11 (1), pp. 1-24
  • Booker, J.M., Meyer, M.A., Sources and effects of interexpert correlation: An empirical study (1988) IEEE Trans. Syst. Man Cybern., 18, pp....
  • Boone, I., van der Stede, Y., Bollaerts, K., Messens, W., Vose, D., Daube, G., Aerts, M., Mintiens, K., Expert judgement in a risk assessment...
  • Bordley, R.F., Combining the opinions of experts who partition events differently (2009) Decis. Anal., 6 (1), pp. 38-46
  • Borenstein, M., Hedges, L.V., Higgins, J.P.T., Rothstein, H.R., (2009) Introduction to Meta-Analysis, , Chichester: Wiley
  • Burns, W.J., Clemen, R.T., Covariance structure models and influence diagrams (1993) Manag. Sci., 39 (7), pp. 816-834
  • Clemen, R.T., Calibration and the aggregation of probabilities (1986) Manag. Sci., 32 (3), pp. 312-314
  • Clemen, R.T., Combining forecasts (1989) Int. J. Forecast., 5, pp. 559-583
  • Clemen, R.T., Winkler, R.L., Aggregating point estimates: A flexible modelling approach (1993) Manag. Sci., 39, pp. 501-515
  • Clemen, R.T., Reilly, T., Correlations and copulas for decision and risk analyses (1999) Manag. Sci., 45 (2), pp. 208-224
  • Clemen, R.T., Winkler, R.L., Combining probability distributions from experts in risk analysis (1999) Risk Anal., 19
  • Clemen, R.T., Fischer, G.W., Winkler, R.L., Assessing dependence: Some experimental results (2000) Manag. Sci., 46 (8), pp. 1100-1115
  • Clemen, R.T., Lichtendahl, K.C., (2002) Debiasing expert overconfidence: A Bayesian calibration model, , PSAM6, San Juan, Puerto Rico
  • Clemen, R.T., Winkler, R.L., Aggregating probability distributions (2007) Advances in Decision Analysis: From Foundations to Applications,...
  • Clemen, R.T., Comment on Cooke's classical method (2008) Reliab. Eng. Syst. Saf., 93 (5), pp. 760-765
  • Cleveland, W.S., (1994) The Elements of Graphing Data, , New Jersey: Murray Hill
  • Conitzer, V., Rothe, J., (2010) COMSOC2010: Proceedings of the Third International Workshop on Computational Social Choice, , September 13-16,...
  • Cooke, R.M., (1991) Experts in Uncertainty, , Oxford: Oxford University Press
  • Cooke, R.M., Bedford, T., Kraan, B., (1997) Report on the Benchmark Workshop on Uncertainty/Sensitivity Analysis Codes, , European Safety...
  • Cooke, R.M., Goossens, L.H.J., Expert judgement elicitation for risk assessments of critical infrastructures (2004) J. Risk Res., 7 (6),...
  • Cooke, R.M., Probst, K.N., (2006) Highlights of the Expert Judgement Policy Symposium and Technical Workshop, , Resources for the Future,...
  • Cooke, R.M., Expert judgement studies (2007) Reliab. Eng. Syst. Saf., 93, pp. 655-777
  • Cooke, R.M., Elsaadany, S., Xinzheng, H., On the performance of social network and likelihood based expert weighting schemes (2007) Reliab....
  • Cooke, R.M., Goossens, L.H.J., TU delft expert judgement database (2007) Reliab. Eng. Syst. Saf., 93 (5), pp. 657-674
  • Cooke, R.M., (2008) Response to the Reliability of Aggregated Probability Judgments Obtained through Cooke's Classical Model, , Delft:...
  • Dalkey, N., Helmer, O., An experimental application of the Delphi method to the use of experts (1963) Manag. Sci., 9 (3), pp. 458-467
  • Dawid, A.P., Degroot, M.H., Mortera, J., Coherent combination of experts' opinions (with discussion) (1995) Test, 4 (2), pp. 263-313
  • de Finetti, B., (1974) Theory of Probability, , Chichester: Wiley
  • Degroot, M.H., Bayarri, M.J., What Bayesians expect of each other (1991) J. Am. Stat. Assoc., 86 (416), pp. 924-932
  • (1998) Department of Health: Communicating About Risks to Public Health: Pointers to Good Practice, , HMSO, London
  • Dewispelare, A.R., Herren, L.T., Clemen, R.T., The use of probability elicitation in the high-level nuclear waste regulation program (1995)...
  • Dryzek, J.S., List, C., Social choice theory and deliberative democracy: A reconciliation (2003) Br. J. Political Sci., 33 (1), pp. 1-28
  • Eden, C., Radford, J., (1990) Tackling Strategic Problems: The Role of Group Decision Support, , C. Eden and J. Radford (Eds.), London: Sage
  • Ehrhardt, J., Brown, J., French, S., Kelly, G.N., Mikkelsen, T., Muller, H., RODOS: Decision making support for off-site emergency management...
  • Ehrhardt, J., Weiss, A., (2000) RODOS: Decision Support for Off-Site Nuclear Emergency Management in Europe. EUR19144EN, , Luxembourg: European...
  • Faria, A., Smith, J.Q., Conditionally externally Bayesian pooling operators in chain graphs (1997) Ann. Stat., 25 (4), pp. 1740-1761
  • Franco, A., Shaw, D., Westcombe, M., Problem structuring methods I (2006) J. Oper. Res. Soc., 57, pp. 757-878
  • Franco, A., Shaw, D., Westcombe, M., Problem structuring methods II (2007) J. Oper. Res. Soc., 58, pp. 545-682
  • Franke, G.R., Applications of meta-analysis for marketing and public policy: A review (2001) J. Public Policy Market., 20 (2), pp. 186-200
  • French, S., Updating of belief in the light of someone else's opinion (1980) J. R. Stat. Soc. A, 143, pp. 43-48
  • French, S., Consensus of opinion (1981) Eur. J. Oper. Res., 7, pp. 332-340
  • French, S., Group consensus probability distributions: A critical survey (1985) Bayesian Statistics, 2, pp. 183-201. , In: Bernardo, J. M.,...
  • French, S., Calibration and the expert problem (1986) Manag. Sci., 32, pp. 315-321
  • French, S., Conflict of belief: When advisers disagree (1987) Analysing Conflict and its Resolution: Some Mathematical Contributions, pp....
  • French, S., Cooke, R.M., Vogt, F., The use of expert judgement in the context of a physical model (1991) Bayesian Statistics, IV, pp. 617-624....
  • French, S., Rios Insua, D., (2000) Statistical Decision Theory, , London: Arnold
  • French, S., Maule, A.J., Mythen, G., Wales, C., (2002) Trust and Risk Communications, , Manchester: Manchester Business School
  • French, S., Modelling, making inferences and making decisions: The roles of sensitivity analysis (2003) Top, 11 (2), pp. 229-252
  • French, S., Battson, A., (2003) Review and implications of research on the cognitive understanding of uncertainties relating to spatial and...
  • French, S., Bayley, C., (2003) An analysis of the ENSEMBLE exercises from the perspective of decision making, , ENSEMBLE Project Report,...
  • French, S., Rios Insua, D., Ruggeri, F., e-Participation and decision analysis (2006) Decis. Anal.
  • French, S., Web-enabled strategic GDSS, e-democracy and Arrow's Theorem: A Bayesian perspective (2007) Decis. Support Syst., 43, pp....
  • French, S., Carter, E., Niculae, C., Decision support in nuclear and radiological emergency situations: Are we too focused on models and...
  • French, S., Rios Insua, D., Ruggeri, F., e-participation and decision analysis (2007) Decis. Anal., 4 (4), pp. 1-16
  • French, S., Maule, A.J., Papamichail, K.N., (2009) Decision Behaviour, Analysis and Support, , Cambridge: Cambridge University Press
  • French, S., Bedford, T., Pollard, S., Soane, E., Human Reliability Analysis: A critique and review for the management of risk (2010) Saf....
  • Galmarini, S., Bianconi, R., Klug, W., Mikkelsen, T., Addis, R., Andronopoulos, S., Astrup, P., Zelazny, R., Can the confidence in long range...
  • Garthwaite, P.H., Kadane, J.B., O'Hagan, A., Statistical methods for eliciting probability distributions (2005) J. Am. Stat. Assoc.,...
  • Genest, C., Zidek, J.V., Combining probability distributions: A critique and annotated bibliography (1986) Stat. Sci., 1, pp. 114-148
  • Genest, C., Wagner, C.G., Further evidence against independence preservation in expert judgement synthesis (1987) Aequationes Mathematicae,...
  • Geyskens, I., Krishnan, R., Steenkamp, J.-B.E.M., Cunha, P.V., A review and evaluation of meta-analysis practices in management research...
  • Gigerenzer, G., (2002) Reckoning with Risk: Learning to Live with Uncertainty, , Harmondsworth: Penguin Books
  • Gilovich, T., Griffin, D., Kahneman, D., (2002) Heuristics and Biases: The Psychology of Intuitive Judgment, , Cambridge: Cambridge University...
  • Givens, G.H., Tweedie, Smith, D.D., Publication bias in meta-analysis: A Bayesian data-augmentation approach to account for issues exemplified...
  • Goldstein, M., Rougier, J.C., Reified Bayesian Modelling and inference for physical systems (with discussion) (2009) J. Stat. Plan. Inference,...
  • Goldstein, M., External Bayesian analysis for computer simulators (2010) Bayesian Statistics, 9. , In: Bernardo, J. M., Bayarri, M. J., Bergeret,...
  • Goossens, L.H.J., Kelly, G.N., Special issue: Expert judgement and accident consequence uncertainty analysis (2000) Radiat. Prot. Dosim.,...
  • Gregory, R.S., Fischhoff, B., McDaniels, T., Acceptable input: Using decision analysis to guide public policy deliberations (2005) Decis....
  • Hartung, J., Knapp, J., Sinha, B.K., (2008) Statistical Meta-Analysis with Applications, , Hoboken: Wiley
  • Hastie, R., Experimental evidence of group accuracy (1986) Information Pooling and Group Decision Making, pp. 129-157. , In: Grofman, B.,...
  • Hockey, G.R.J., Maule, A.J., Clough, P.J., Bdzola, L., Effects of negative mood on risk in everyday decision making (2000) Cogn. Emot., 14,...
  • Hodge, J.K., Klima, R.E., (2005) The Mathematics of Voting and Elections: A Hands-On Approach, , Rhode Island: American Mathematical Society
  • Hora, S., Eliciting probabilities from experts (2007) Advances in Decision Analysis: From Foundations to Applications, pp. 129-153. , In:...
  • Hora, S.C., Probability judgements for continuous quantities: Linear combinations and calibration (2004) Manag. Sci., 50 (5), pp. 597-604
  • Hora, S.C., Expert judgement (2008) Encylcopedia of Quantitative Risk Analysis and Assessment, pp. 667-676. , In: Melnick, E. L., Everitt,...
  • Jacobs, R.A., Methods for combining experts' probability assessments (1995) Neural Comput., 7, pp. 867-888
  • James, A., Low-Choy, S., Mengersen, K., Elicitator: An expert elicitation tool for regression in ecology (2010) Environ. Model. Softw., 25...
  • Jeffreys, H., (1961) Theory of Probability, , Oxford: Oxford University Press
  • Jouini, M.N., Clemen, R.T., Copula models for asggregating expert opinions (1996) Oper. Res., 44, pp. 444-457
  • Kadane, J.B., Schervish, M.J., Seidenfeld, T., (1999) Rethinking the Foundations of Statistics, , Cambridge: Cambridge University Press
  • Kahneman, D., Slovic, P., Tversky, A., (1982) Judgement under Uncertainty: Heuristics and Biases, , Cambridge: Cambridge University Press
  • Kallen, M.J., Cooke, R.M., Expert aggregation with dependence (2002) Proabilistic Safety Assessment and Management, pp. 1287-1294. , In:...
  • Karacapilidis, N.I., Pappis, C.P., A framework for group decision support systems: Combining AI tools and OR techniques (1997) Eur. J. Oper....
  • Keeney, R.L., Raiffa, H., (1976) Decisions with Multiple Objectives: Preferences and Value Trade-Offs, , New York: Wiley
  • Keith, D.W., When is it better to combine expert judgements? (1996) Clim. Chang., 33, pp. 111-143
  • Kelly, F.S., (1978) Arrow Impossibility Theorems, , New York: Academic Press
  • Kennedy, M.C., O'Hagan, A., Bayesian calibration of computer models (2001) J. R. Stat. Soc. B, 63, pp. 425-464
  • Koning, J.-L., How electronic voting can escape Arrow's Impossibility Theorem (2003) e-Government and e-Democracy: Progress and Challenges,...
  • Kurowicka, D., Cooke, R.M., (2006) Uncertainty Analysis with High Dimensional Modelling, , Chichester: Wiley
  • Langford, I., Marris, C., O'Riordan, T., Public reactions to risk: Social structures, images of science and the role of trust (1999)...
  • Leach, M., Scoones, I., Wynne, B., (2005) Science and Citizens, , London: Zed Books
  • Leamer, E.E., (1978) Specification Searches, , Chichester: Wiley
  • Lehmann, H.P., Goodman, S.N., Bayesian communication: A clinically significant paradigm for electronic publication (2000) J. Am. Med. Inf....
  • Lichtendahl, K.C., (2005) Bayesian Models of Expert Forecasts, , PhD Thesis. Department of Business Administration, Duke University, Durham,...
  • Lichtendahl, K.C., Winkler, R.L., Probability elicitation, scoring rules, and competition among forecasters (2007) Manag. Sci., 53 (11),...
  • Lichtenstein, S., Fischhoff, B., Phillips, L.D., Calibration of probabilities: The state of the art to 1980 (1982) Judgement Under Uncertainty,...
  • Lin, S.-W., Bier, V.M., A study of expert overconfidence (2008) Reliab. Eng. Syst. Saf., 93, pp. 711-721
  • Lin, S.-W., Cheng, C.-H., The reliability of aggregated probability judgments obtained through Cooke's classical model (2009) J. Model....
  • Lindley, D.V., Tversky, A., Brown, R.V., On the reconciliation of probability judgements (with discussion) (1979) J. R. Stat. Soc. A, 142,...
  • Lindley, D.V., The improvement of probability judgements (1982) J. R. Stat. Soc. A, 145, pp. 117-126
  • Lindley, D.V., Reconciliation of probability distributions (1983) Oper. Res., 31 (5), pp. 866-880
  • Lindley, D.V., Reconciliation of discrete probability distributions (1985) Bayesian Statistics, 2, pp. 375-390. , In: Bernardo, J. M., DeGroot,...
  • Linstone, H.A., Turoff, M., (1978) The Delphi Method: Techniques and Applications, , London: Addison-Wesley
  • Lipscomb, J., Parmigiani, G., Hasselblad, V., Combining expert judgment by hierarchical modeling: An application to physician staffing (1998)...
  • Low-Choy, S., Rousseau, J., Mengersen, K., (2010) Poster on 'Balancing expert consensus and diversity', , Bayesian Statistics 9....
  • Madansky, A., (1964) Externally Bayesian groups, , RM-4141-PR. RAND
  • Maule, A.J., Translating risk management knowledge: The lessons to be learned from research on the perception and communication of risk (2004)...
  • Maule, A.J., Risk communication and organisations (2008) The Oxford Handbook of Organizational Decision Making, , In: Starbuck, W., Hodgkinson,...
  • Merrick, J.R.W., Getting the right mix of experts (2008) Decis. Anal., 5 (1), pp. 43-52
  • Merrick, J.R.W., Bayesian simulation and decision analysis: An expository survey (2009) Decis. Anal., 6 (4), pp. 222-238
  • Mikkelsen, T., Galmarini, S., Bianconi, R., French, S., (2003) ENSEMBLE: Methods to reconcile disparate national long-range dispersion forecasts,...
  • Morris, C.N., Normand, S.L., Hierarchical models for combining information and meta-analyses (1992) Bayesian Statistics, 4, pp. 321-344....
  • Morton, A., Ackermann, F., Belton, V., Technology-driven and model-driven approaches to group decision support: Focus, research philosophy,...
  • Mumpower, J.L., Stewart, T.R., Expert judgement and expert disagreement (1996) Thinking Reason., 2 (2-3), pp. 191-211
  • Murray, J.V., Goldizen, A.W., O'Leary, R.A., McAlpine, C., Possingham, H.P., Low-Choy, S., How useful is expert opinion for predicting...
  • Newbold, P., Granger, C.W.J., Experience with forecasting univariate time series and the combination of forecasts (1974) J. R. Stat. Soc....
  • O'Hagan, A., Eliciting expert beliefs in substantial practical applications (1998) Statistician, 47 (1), pp. 21-35
  • O'Hagan, A., Kennedy, M.C., Oakley, J.E., Uncertainty analysis and other inference tools for complex computer codes (1998) Bayesian Statistics,...
  • O'Hagan, A., Forester, J., (2004) Bayesian Statistics, , London: Edward Arnold
  • O'Hagan, A., Buck, C.E., Daneshkhah, A., Eiser, R., Garthwaite, P.H., Jenkinson, D., Oakley, J.E., Rakow, T., (2006) Uncertain Judgements:...
  • O'Neill, S.J., Osborn, T.J., Hulme, M., Lorenzoni, I., Watkinson, A.R., Using expert knowledge to assess uncertainties in future polar...
  • Ortiz, N.R., Wheeler, T.A., Breeding, R.J., Hora, S., Meyer, M.A., Keeney, R.L., Use of expert judgement in NUREG 1150 (1991) Nucl. Eng....
  • Phillips, L.D., Decision conferencing (2007) Advances in Decision Analysis: From Foundations to Applications, pp. 375-399. , In: Edwards,...
  • Predd, J., Osherson, S., Kulkarni, H., Aggregating probabilistic forecasts from incoherent and abstaining experts (2008) Decis. Anal., 5...
  • Raiffa, H., (1982) The Art and Science of Negotiation: How to Resolve Conflicts and Get the Best out of Bargaining, , Cambridge: Belknap...
  • Raiffa, H., Richardson, J., Metcalfe, D., (2002) Negotiation Analysis: The Science and Art of Collaborative Decision Making, , Cambridge:...
  • Ranjan, R., Gneiting, Combining probability forecasts (2010) J. R. Stat. Soc. B, 72 (1), pp. 71-91
  • Reagan-Cirincione, P., Combining group facilitation, decision modelling and information technology to improve the accuracy of group judgement...
  • Renn, O., Webler, T., Rakel, H., Dienel, P., Johnson, B., Public participation in decision making: A three step-procedure (1993) Policy Sci.,...
  • Renn, O., Webler, T., Wiedermann, P., (1995) Fairness and Competence in Citizen Participation: Evaluating Models and Environmental Discourse,...
  • Renn, O., The role of risk communication and public dialogue for improving risk management (1998) Risk Decis. Policy, 3, pp. 3-50
  • Renn, O., (2008) Risk Governance, , London: Earthscan
  • Renooij, S., Probability elicitation for belief networks: Issues to consider (2001) Knowl. Eng. Rev., 16 (3), pp. 255-269
  • Rios Insua, D., Ruggeri, F., (2000) Robust Bayesian Analysis, , New York: Springer
  • Rios Insua, D., French, S., (2010) e Democracy: A Group Decision and Negotiation Perspective, , Dordrecht: Group Decision and Negotiation....
  • Rosenthal, R., Dimatteo, M.R., Meta-analysis: Recent developments in quantitative methods for literature reviews (2001) Annu. Rev. Psychol.,...
  • Rosqvist, T., Bayesian aggregation of experts' judgements on failure intensity (2000) Reliab. Eng. Syst. Saf., 70 (3), pp. 283-289
  • Rougier, J.C., Guillas, S., Maute, A., Richmond, A.D., Expert knowledge and multivariate emulation: The thermospher-ionosphere electrodynamics...
  • Rowe, G., Wright, G., The Delphi technique as a forecasting tool: Issues and analysis (1999) Int. J. Forecast., 15, pp. 353-375
  • Savage, L.J., (1972) The Foundations of Statistics, , New York: Dover
  • Scapolo, F., Miles, I., Eliciting experts' knowledge: A comparison of two methods (2006) Technol. Forecast. Soc. Chang., 73 (6), pp....
  • Shanteau, J., Expert Judgment and Financial Decision Making (1995) Risky Business: Risk Behavior and Risk Management, , In: Green, B. (ed.)...
  • Skjong, R., Wentworth, B.H., (2001) Expert Judgement and Risk Perception, , In: Proceedings of the Eleventh (2001) International Offshore...
  • Slovic, P., Trust, emotion, sex, politics and science: Surveying the risk-assessment battlefield (1997) Environment, Ethics and Behaviour,...
  • Slovic, P., (2001) Perceptions of Risk, , London: Earthscan Library
  • Smith, J.Q., Faria, A., Bayesian Poisson models for the graphical combination of dependent expert information (2000) J. R. Stat. Soc. B,...
  • Smith, J.Q., (2010) Bayesian Decision Analysis: Principles and Practice, , Cambridge: Cambridge University Press
  • Smith, L., (2007) Chaos: A Very Short Introduction, , Oxford: Oxford University Press
  • Stanley, T.D., Wheat from chaff: Meta-analysis as quantitative literature review (2001) J. Econ. Perspect., 15 (3), pp. 131-151
  • Suantak, L., Bolger, F., Ferrell, W.R., The hard-easy effect in subjective probability calibration (1996) Organ. Behav. Human Decis. Process.,...
  • Surowiecki, J., (2004) The Wisdom of Crowds, , New York: Random House
  • Sutton, A.J., Abrams, K.R., Bayesian methods in meta-analysis and evidence synthesis (2001) Stat. Methods Med. Res., 10 (4), pp. 277-303
  • Szwed, P.S., van Dorp, J.R., (2002) A Bayesian Model for Rare Event Risk Assessment Using Expert Judgement About Paired Scenario Comparisons,...
  • Szwed, P.S., van Dorp, J.R., Merrick, J.R.W., Mazzuchi, T.A., Singh, A., A Bayesian paired comparison approach for relative accident probability...
  • Taylor, A.D., (2005) Social Choice and the Mathematics of Manipulation, , Cambridge: Cambridge University Press
  • Turner, R.M., Speigelhalter, D.J., Smith, G.C.S., Thompson, S.G., Bias modelling in evidence synthesis (2009) J. R. Stat. Soc. A, 172 (1),...
  • Winkler, R.L., Combining probability distributions from dependent information sources (1981) Manag. Sci., 27, pp. 479-488
  • Wiper, M.W., (1990) Calibration and use of expert probability judgements, , PhD. School of Computer Studies, University of Leeds, Leeds
  • Wiper, M.W., French, S., Cooke, R.M., Hypothesis test based calibration scores (1994) Statistician, 43, pp. 231-236
  • Wiper, M.W., French, S., Combining experts' opinions using a normal-Wishart model (1995) J. Forecast., 14, pp. 25-34
  • Wisse, B., Bedford, T., Quigley, J., Expert judgement combination using moment methods (2007) Reliab. Eng. Syst. Saf., 93, pp. 675-686
  • Wright, G., Ayton, P., (1994) Subjective Probability, , Chichester: Wiley
  • Zacharakis, A.L., Shepherd, D.A., The nature of information and overconfidence on venture capitalist's decision making (2001) J. Bus....