Note to users. If you're seeing this message, it means that your browser cannot find this page's style/presentation instructions -- or possibly that you are using a browser that does not support current Web standards. Find out more about why this message is appearing, and what you can do to make your experience of our site the best it can be.

Science 14 March 1997:
Vol. 275. no. 5306, pp. 1593 - 1599
DOI: 10.1126/science.275.5306.1593
Prev | Table of Contents | Next

Articles

A Neural Substrate of Prediction and Reward

Wolfram Schultz, Peter Dayan, P. Read Montague *

The capacity to predict future events permits a creature to detect, model, and manipulate the causal structure of its interactions with its environment. Behavioral experiments suggest that learning is driven by changes in the expectations about future salient events such as rewards and punishments. Physiological work has recently complemented these studies by identifying dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events. Taken together, these findings can be understood through quantitative theories of adaptive optimizing control.

W. Schultz is at the Institute of Physiology, University of Fribourg, CH-1700 Fribourg, Switzerland. E-mail: Wolfram.Schultz{at}unifr.ch  P. Dayan is in the Department of Brain and Cognitive Sciences, Center for Biological and Computational Learning, E-25 MIT, Cambridge, MA 02139, USA. E-mail: dayan{at}ai.mit.edu  P. R. Montague is in the Division of Neuroscience, Center for Theoretical Neuroscience, Baylor College of Medicine, 1 Baylor Plaza, Houston, TX 77030, USA. E-mail: read{at}bcm.tmc.edu
*   To whom correspondence should be addressed.

Read the Full Text


THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES:
A neurochemical approach to valuation sensitivity over gains and losses.
S. Zhong, S. Israel, H. Xue, P. C. Sham, R. P. Ebstein, and S. H. Chew (2009)
Proc R Soc B 276, 4181-4188
   Abstract »    Full Text »    PDF »
Reinforcement learning, conditioning, and the brain: Successes and challenges.
T. V. Maia (2009)
Cogn Affect Behav Neurosci 9, 343-364
   Abstract »    PDF »
Aging and the neuroeconomics of decision making: A review.
S. B. R. E. Brown and K. R. Ridderinkhof (2009)
Cogn Affect Behav Neurosci 9, 365-379
   Abstract »    PDF »
Caudate Nucleus Is Critically Involved in Trace Eyeblink Conditioning.
L. C. Flores and J. F. Disterhoft (2009)
J. Neurosci. 29, 14511-14520
   Abstract »    Full Text »    PDF »
How Humans Integrate the Prospects of Pain and Reward during Choice.
D. Talmi, P. Dayan, S. J. Kiebel, C. D. Frith, and R. J. Dolan (2009)
J. Neurosci. 29, 14617-14626
   Abstract »    Full Text »    PDF »
Neural Components Underlying Behavioral Flexibility in Human Reversal Learning.
D. G. Ghahremani, J. Monterosso, J. D. Jentsch, R. M. Bilder, and R. A. Poldrack (2009)
Cereb Cortex
   Abstract »    Full Text »    PDF »
Pharmacological modulation of subliminal learning in Parkinson's and Tourette's syndromes.
S. Palminteri, M. Lebreton, Y. Worbe, D. Grabli, A. Hartmann, and M. Pessiglione (2009)
PNAS 106, 19179-19184
   Abstract »    Full Text »    PDF »
Neural representation of time in cortico-basal ganglia circuits.
D. Z. Jin, N. Fujii, and A. M. Graybiel (2009)
PNAS 106, 19156-19161
   Abstract »    Full Text »    PDF »
Disrupted Effective Connectivity Between the Medial Frontal Cortex and the Caudate in Adolescent Boys With Externalizing Behavior Disorders.
K. E. Shannon, C. Sauder, T. P. Beauchaine, and L. M. Gatzke-Kopp (2009)
Criminal Justice and Behavior 36, 1141-1157
   Abstract »    PDF »
Transient Firing of Dorsal Raphe Neurons Encodes Diverse and Specific Sensory, Motor, and Reward Events.
S. P. Ranade and Z. F. Mainen (2009)
J Neurophysiol 102, 3026-3037
   Abstract »    Full Text »    PDF »
Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values.
S. J. Gershman, B. Pesaran, and N. D. Daw (2009)
J. Neurosci. 29, 13524-13531
   Abstract »    Full Text »    PDF »
Neural mechanisms for learned birdsong.
R. Mooney (2009)
Learn. Mem. 16, 655-669
   Abstract »    Full Text »    PDF »
Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions.
L. K. Krugel, G. Biele, P. N. C. Mohr, S.-C. Li, and H. R. Heekeren (2009)
PNAS 106, 17951-17956
   Abstract »    Full Text »    PDF »
The Importance of Failure: Feedback-Related Negativity Predicts Motor Learning Efficiency.
J. van der Helden, M. A. S. Boksem, and J. H. G. Blom (2009)
Cereb Cortex
   Abstract »    Full Text »    PDF »
Corticostriatal Interactions during Learning, Memory Processing, and Decision Making.
C. M. A. Pennartz, J. D. Berke, A. M. Graybiel, R. Ito, C. S. Lansink, M. van der Meer, A. D. Redish, K. S. Smith, and P. Voorn (2009)
J. Neurosci. 29, 12831-12838
   Abstract »    Full Text »    PDF »
Neural computations underlying action-based decision making in the human brain.
K. Wunderlich, A. Rangel, and J. P. O'Doherty (2009)
PNAS 106, 17199-17204
   Abstract »    Full Text »    PDF »
Do Substantia Nigra Dopaminergic Neurons Differentiate Between Reward and Punishment?.
M. J. Frank and D. J. Surmeier (2009)
J Mol Cell Biol 1, 15-16
   Abstract »    Full Text »    PDF »
Adenylyl Cyclase Type 5 Contributes to Corticostriatal Plasticity and Striatum-Dependent Learning.
M. A. Kheirbek, J. P. Britt, J. A. Beeler, Y. Ishikawa, D. S. McGehee, and X. Zhuang (2009)
J. Neurosci. 29, 12115-12124
   Abstract »    Full Text »    PDF »
Reward-learning and the novelty-seeking personality: a between- and within-subjects study of the effects of dopamine agonists on young Parkinson's patients.
N. Bodi, S. Keri, H. Nagy, A. Moustafa, C. E. Myers, N. Daw, G. Dibo, A. Takats, D. Bereczki, and M. A. Gluck (2009)
Brain 132, 2385-2395
   Abstract »    Full Text »    PDF »
Restriction of dopamine signaling to the dorsolateral striatum is sufficient for many cognitive behaviors.
M. Darvas and R. D. Palmiter (2009)
PNAS 106, 14664-14669
   Abstract »    Full Text »    PDF »
Uncertainty during Anticipation Modulates Neural Responses to Aversion in Human Insula and Amygdala.
I. Sarinopoulos, D. W. Grupe, K. L. Mackiewicz, J. D. Herrington, M. Lor, E. E. Steege, and J. B. Nitschke (2009)
Cereb Cortex
   Abstract »    Full Text »    PDF »
Validation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia.
M. Ito and K. Doya (2009)
J. Neurosci. 29, 9861-9874
   Abstract »    Full Text »    PDF »
Which Way Do I Go? Neural Activation in Response to Feedback and Spatial Processing in a Virtual T-Maze.
T. E. Baker and C. B. Holroyd (2009)
Cereb Cortex 19, 1708-1722
   Abstract »    Full Text »    PDF »
Controls of Tonic and Phasic Dopamine Transmission in the Dorsal and Ventral Striatum.
L. Zhang, W. M. Doyon, J. J. Clark, P. E. M. Phillips, and J. A. Dani (2009)
Mol. Pharmacol. 76, 396-404
   Abstract »    Full Text »    PDF »
Single-Cell and Population Coding of Expected Reward Probability in the Orbitofrontal Cortex of the Rat.
E. van Duuren, G. van der Plasse, J. Lankelma, R. N. J. M. A. Joosten, M. G. P. Feenstra, and C. M. A. Pennartz (2009)
J. Neurosci. 29, 8965-8976
   Abstract »    Full Text »    PDF »
The Computation of Social Behavior.
T. E. J. Behrens, L. T. Hunt, and M. F. S. Rushworth (2009)
Science 324, 1160-1164
   Abstract »    Full Text »    PDF »
Role of 5-Hydroxytryptamine2C Receptors in Ca2+-Dependent Ethanol Potentiation of GABA Release onto Ventral Tegmental Area Dopamine Neurons.
J. W. Theile, H. Morikawa, R. A. Gonzales, and R. A. Morrisett (2009)
J. Pharmacol. Exp. Ther. 329, 625-633
   Abstract »    Full Text »    PDF »
Near-Term Liability of Exploitation: Exploration and Exploitation in Multistage Problems.
C. Fang and D. Levinthal (2009)
Organization Science 20, 538-551
   Abstract »    PDF »
Dysconnection in Schizophrenia: From Abnormal Synaptic Plasticity to Failures of Self-monitoring.
K. E. Stephan, K. J. Friston, and C. D. Frith (2009)
Schizophr Bull 35, 509-527
   Abstract »    Full Text »    PDF »
The Dopamine Hypothesis of Schizophrenia: Version III--The Final Common Pathway.
O. D. Howes and S. Kapur (2009)
Schizophr Bull 35, 549-562
   Abstract »    Full Text »    PDF »
A Dual Role for Prediction Error in Associative Learning.
H. E.M. den Ouden, K. J. Friston, N. D. Daw, A. R. McIntosh, and K. E. Stephan (2009)
Cereb Cortex 19, 1175-1185
   Abstract »    Full Text »    PDF »
Functional Dissociations of Risk and Reward Processing in the Medial Prefrontal Cortex.
G. Xue, Z. Lu, I. P. Levin, J. A. Weller, X. Li, and A. Bechara (2009)
Cereb Cortex 19, 1019-1027
   Abstract »    Full Text »    PDF »
Different Pedunculopontine Tegmental Neurons Signal Predicted and Actual Task Rewards.
K.-i. Okada, K. Toyama, Y. Inoue, T. Isa, and Y. Kobayashi (2009)
J. Neurosci. 29, 4858-4870
   Abstract »    Full Text »    PDF »
Dopamine Signaling Differences in the Nucleus Accumbens and Dorsal Striatum Exploited by Nicotine.
T. Zhang, L. Zhang, Y. Liang, A. G. Siapas, F.-M. Zhou, and J. A. Dani (2009)
J. Neurosci. 29, 4035-4043
   Abstract »    Full Text »    PDF »
Human Substantia Nigra Neurons Encode Unexpected Financial Rewards.
K. A. Zaghloul, J. A. Blanco, C. T. Weidemann, K. McGill, J. L. Jaggi, G. H. Baltuch, and M. J. Kahana (2009)
Science 323, 1496-1499
   Abstract »    Full Text »    PDF »
Activity of Neurochemically Heterogeneous Dopaminergic Neurons in the Substantia Nigra during Spontaneous and Driven Changes in Brain State.
M. T. C. Brown, P. Henny, J. P. Bolam, and P. J. Magill (2009)
J. Neurosci. 29, 2915-2925
   Abstract »    Full Text »    PDF »
Distinct Subtypes of Basolateral Amygdala Taste Neurons Reflect Palatability and Reward.
A. Fontanini, S. E. Grossman, J. A. Figueroa, and D. B. Katz (2009)
J. Neurosci. 29, 2486-2495
   Abstract »    Full Text »    PDF »
Synaptic Overflow of Dopamine in the Nucleus Accumbens Arises from Neuronal Activity in the Ventral Tegmental Area.
L. A. Sombers, M. Beyene, R. M. Carelli, and R. Mark Wightman (2009)
J. Neurosci. 29, 1735-1742
   Abstract »    Full Text »    PDF »
Striatal Dopamine Predicts Outcome-Specific Reversal Learning and Its Sensitivity to Dopaminergic Drug Administration.
R. Cools, M. J. Frank, S. E. Gibbs, A. Miyakawa, W. Jagust, and M. D'Esposito (2009)
J. Neurosci. 29, 1538-1543
   Abstract »    Full Text »    PDF »
Encoding of Probabilistic Rewarding and Aversive Events by Pallidal and Nigral Neurons.
M. Joshua, A. Adler, B. Rosin, E. Vaadia, and H. Bergman (2009)
J Neurophysiol 101, 758-772
   Abstract »    Full Text »    PDF »
CNTRICS Final Task Selection: Long-Term Memory.
J. D. Ragland, R. Cools, M. Frank, D. A. Pizzagalli, A. Preston, C. Ranganath, and A. D. Wagner (2009)
Schizophr Bull 35, 197-212
   Abstract »    Full Text »    PDF »
Complementary roles for amygdala and periaqueductal gray in temporal-difference fear learning.
S. Cole and G. P. McNally (2008)
Learn. Mem. 16, 1-7
   Abstract »    Full Text »    PDF »
Changes in Control of Saccades during Gain Adaptation.
V. Ethier, D. S. Zee, and R. Shadmehr (2008)
J. Neurosci. 28, 13929-13937
   Abstract »    Full Text »    PDF »
Anticipatory affect: neural correlates and consequences for choice.
B. Knutson and S. M Greer (2008)
Phil Trans R Soc B 363, 3771-3786
   Abstract »    Full Text »    PDF »
The role of the striatum in aversive learning and aversive prediction errors.
M. R Delgado, J. Li, D. Schiller, and E. A Phelps (2008)
Phil Trans R Soc B 363, 3787-3800
   Abstract »    Full Text »    PDF »
Explicit neural signals reflecting reward uncertainty.
W. Schultz, K. Preuschoff, C. Camerer, M. Hsu, C. D Fiorillo, P. N Tobler, and P. Bossaerts (2008)
Phil Trans R Soc B 363, 3801-3811
   Abstract »    Full Text »    PDF »
Neuroethology of reward and decision making.
K. K Watson and M. L Platt (2008)
Phil Trans R Soc B 363, 3825-3835
   Abstract »    Full Text »    PDF »
The Neuromodulatory System: A Framework for Survival and Adaptive Behavior in a Challenging World.
J. L. Krichmar (2008)
Adaptive Behavior 16, 385-399
   Abstract »    PDF »
Conceptual representations in goal-directed decision making.
N. Shea, K. Krug, and P. N. Tobler (2008)
Cogn Affect Behav Neurosci 8, 418-428
   Abstract »    PDF »
Decision theory, reinforcement learning, and the brain.
P. Dayan and N. D. Daw (2008)
Cogn Affect Behav Neurosci 8, 429-453
   Abstract »    PDF »
A Role for Dopamine in Temporal Decision Making and Reward Maximization in Parkinsonism.
A. A. Moustafa, M. X. Cohen, S. J. Sherman, and M. J. Frank (2008)
J. Neurosci. 28, 12294-12304
   Abstract »    Full Text »    PDF »
From Fear to Safety and Back: Reversal of Fear in the Human Brain.
D. Schiller, I. Levy, Y. Niv, J. E. LeDoux, and E. A. Phelps (2008)
J. Neurosci. 28, 11517-11525
   Abstract »    Full Text »    PDF »
Midbrain Dopaminergic Neurons and Striatal Cholinergic Interneurons Encode the Difference between Reward and Aversive Events at Different Epochs of Probabilistic Classical Conditioning Trials.
M. Joshua, A. Adler, R. Mitelman, E. Vaadia, and H. Bergman (2008)
J. Neurosci. 28, 11673-11684
   Abstract »    Full Text »    PDF »
The Spatial Attention Network Interacts with Limbic and Monoaminergic Systems to Modulate Motivation-Induced Attention Shifts.
A. Mohanty, D. R. Gitelman, D. M. Small, and M. M. Mesulam (2008)
Cereb Cortex 18, 2604-2613
   Abstract »    Full Text »    PDF »
A cAMP Pathway Underlying Reward Prediction in Associative Learning.
M. A. Kheirbek, J. A. Beeler, Y. Ishikawa, and X. Zhuang (2008)
J. Neurosci. 28, 11401-11408
   Abstract »    Full Text »    PDF »
Relation Between Obesity and Blunted Striatal Response to Food Is Moderated by TaqIA A1 Allele.
E. Stice, S. Spoor, C. Bohon, and D. M. Small (2008)
Science 322, 449-452
   Abstract »    Full Text »    PDF »
Moment-to-Moment Tracking of State Value in the Amygdala.
M. A. Belova, J. J. Paton, and C. D. Salzman (2008)
J. Neurosci. 28, 10023-10030
   Abstract »    Full Text »    PDF »
A Local Circuit Model of Learned Striatal and Dopamine Cell Responses under Probabilistic Schedules of Reward.
C. O. Tan and D. Bullock (2008)
J. Neurosci. 28, 10062-10074
   Abstract »    Full Text »    PDF »
Tripartite Mechanism of Extinction Suggested by Dopamine Neuron Activity and Temporal Difference Model.
W.-X. Pan, R. Schmidt, J. R. Wickens, and B. I. Hyland (2008)
J. Neurosci. 28, 9619-9631
   Abstract »    Full Text »    PDF »
Reward Processing in Schizophrenia: A Deficit in the Representation of Value.
J. M. Gold, J. A. Waltz, K. J. Prentice, S. E. Morris, and E. A. Heerey (2008)
Schizophr Bull 34, 835-847
   Abstract »    Full Text »    PDF »
Reinforcement and Reversal Learning in First-Episode Psychosis.
G. K. Murray, F. Cheng, L. Clark, J. H. Barnett, A. D. Blackwell, P. C. Fletcher, T. W. Robbins, E. T. Bullmore, and P. B. Jones (2008)
Schizophr Bull 34, 848-855
   Abstract »    Full Text »    PDF »
Distinctive Roles for the Ventral Striatum and Ventral Prefrontal Cortex during Decision-Making.
L. T. Hunt (2008)
J. Neurosci. 28, 8658-8659
   Full Text »    PDF »
Dynamic changes in accumbens dopamine correlate with learning during intracranial self-stimulation.
C. A. Owesson-White, J. F. Cheer, M. Beyene, R. M. Carelli, and R. M. Wightman (2008)
PNAS 105, 11957-11962
   Abstract »    Full Text »    PDF »
Influence of Neuronal Nicotinic Receptors over Nicotine Addiction and Withdrawal.
M. De Biasi and R. Salas (2008)
Experimental Biology and Medicine 233, 917-929
   Abstract »    Full Text »    PDF »
Opioid Receptor PET Reveals the Psychobiologic Correlates of Reward Processing.
M. Schreckenberger, A. Klega, G. Grunder, H.-G. Buchholz, A. Scheurich, R. Schirrmacher, E. Schirrmacher, C. Muller, G. Henriksen, and P. Bartenstein (2008)
J. Nucl. Med. 49, 1257-1261
   Abstract »    Full Text »    PDF »
Influence of Reward Delays on Responses of Dopamine Neurons.
S. Kobayashi and W. Schultz (2008)
J. Neurosci. 28, 7837-7846
   Abstract »    Full Text »    PDF »
Evidence for Segregated and Integrative Connectivity Patterns in the Human Basal Ganglia.
B. Draganski, F. Kherif, S. Kloppel, P. A. Cook, D. C. Alexander, G. J. M. Parker, R. Deichmann, J. Ashburner, and R. S. J. Frackowiak (2008)
J. Neurosci. 28, 7143-7152
   Abstract »    Full Text »    PDF »
What's in a Smile? Maternal Brain Responses to Infant Facial Cues.
L. Strathearn, J. Li, P. Fonagy, and P. R. Montague (2008)
Pediatrics 122, 40-51
   Abstract »    Full Text »    PDF »
Preferential Reactivation of Motivationally Relevant Information in the Ventral Striatum.
C. S. Lansink, P. M. Goltstein, J. V. Lankelma, R. N. J. M. A. Joosten, B. L. McNaughton, and C. M. A. Pennartz (2008)
J. Neurosci. 28, 6372-6382
   Abstract »    Full Text »    PDF »
Methylphenidate Has Differential Effects on Blood Oxygenation Level-Dependent Signal Related to Cognitive Subprocesses of Reversal Learning.
C. M. Dodds, U. Muller, L. Clark, A. van Loon, R. Cools, and T. W. Robbins (2008)
J. Neurosci. 28, 5976-5982
   Abstract »    Full Text »    PDF »
Neurocomputational mechanisms of reinforcement-guided learning in humans: A review.
M. X. COHEN (2008)
Cogn Affect Behav Neurosci 8, 113-125
   Abstract »    PDF »
Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors.
T. A. Hare, J. O'Doherty, C. F. Camerer, W. Schultz, and A. Rangel (2008)
J. Neurosci. 28, 5623-5630
   Abstract »    Full Text »    PDF »
Opposing Patterns of Signaling Activation in Dopamine D1 and D2 Receptor-Expressing Striatal Neurons in Response to Cocaine and Haloperidol.
J. Bertran-Gonzalez, C. Bosch, M. Maroteaux, M. Matamales, D. Herve, E. Valjent, and J.-A. Girault (2008)
J. Neurosci. 28, 5671-5685
   Abstract »    Full Text »    PDF »
Reward-Dependent Modulation of Neuronal Activity in the Primate Dorsal Raphe Nucleus.
K. Nakamura, M. Matsumoto, and O. Hikosaka (2008)
J. Neurosci. 28, 5331-5343
   Abstract »    Full Text »    PDF »
Disconnecting force from money: effects of basal ganglia damage on incentive motivation.
L. Schmidt, B. F. d'Arc, G. Lafargue, D. Galanaud, V. Czernecki, D. Grabli, M. Schupbach, A. Hartmann, R. Levy, B. Dubois, et al. (2008)
Brain 131, 1303-1310
   Abstract »    Full Text »    PDF »
Attaching Values to Actions: Action and Outcome Encoding in the Primate Caudate Nucleus.
C. H. Donahue and H. Seo (2008)
J. Neurosci. 28, 4579-4580
   Full Text »    PDF »
Extracellular Signal-Related Kinase Activation During Natural Reward Learning: A Physiological Role for Phasic Nucleus Accumbens Dopamine?.
J. J. Day (2008)
J. Neurosci. 28, 4295-4297
   Full Text »    PDF »
Distinct error-correcting and incidental learning of location relative to landmarks and boundaries.
C. F. Doeller and N. Burgess (2008)
PNAS 105, 5909-5914
   Abstract »    Full Text »    PDF »
Drosophila Egg-Laying Site Selection as a System to Study Simple Decision-Making Processes.
C.-h. Yang, P. Belawat, E. Hafen, L. Y. Jan, and Y.-N. Jan (2008)
Science 319, 1679-1683
   Abstract »    Full Text »    PDF »
Motor Adaptation as a Process of Reoptimization.
J. Izawa, T. Rane, O. Donchin, and R. Shadmehr (2008)
J. Neurosci. 28, 2883-2891
   Abstract »    Full Text »    PDF »
BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area.
K. D'Ardenne, S. M. McClure, L. E. Nystrom, and J. D. Cohen (2008)
Science 319, 1264-1267
   Abstract »    Full Text »    PDF »
Decoding of Temporal Intervals From Cortical Ensemble Activity.
M. A. Lebedev, J. E. O'Doherty, and M. A. L. Nicolelis (2008)
J Neurophysiol 99, 166-186
   Abstract »    Full Text »    PDF »
Cerebrovascular responses to incremental exercise during hypobaric hypoxia: effect of oxygenation on maximal performance.
A. W. Subudhi, M. C. Lorenz, C. S. Fulco, and R. C. Roach (2008)
Am J Physiol Heart Circ Physiol 294, H164-H171
   Abstract »    Full Text »    PDF »


To Advertise     Find Products

Science. ISSN 0036-8075 (print), 1095-9203 (online)