Note to users. If you're seeing this message, it means that your browser cannot find this page's style/presentation instructions -- or possibly that you are using a browser that does not support current Web standards. Find out more about why this message is appearing, and what you can do to make your experience of our site the best it can be.

Science 16 April 2004:
Vol. 304. no. 5669, pp. 452 - 454
DOI: 10.1126/science.1094285
Prev | Table of Contents | Next

Reports

Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning

John O'Doherty,1* Peter Dayan,2 Johannes Schultz,1 Ralf Deichmann,1 Karl Friston,1 Raymond J. Dolan1

Instrumental conditioning studies how animals and humans choose actions appropriate to the affective structure of an environment. According to recent reinforcement learning models, two distinct components are involved: a "critic," which learns to predict future reward, and an "actor," which maintains information about the rewarding outcomes of actions to enable better ones to be chosen more frequently. We scanned human participants with functional magnetic resonance imaging while they engaged in instrumental conditioning. Our results suggest partly dissociable contributions of the ventral and dorsal striatum, with the former corresponding to the critic and the latter corresponding to the actor.

1 Wellcome Department of Imaging Neuroscience, Institute of Neurology, University College London, London WC1N 3BG, UK.
2 Gatsby Computational Neuroscience Unit, University College London, London WC1N 3BG, UK.

* To whom correspondence should be addressed. E-mail: j.odoherty{at}fil.ion.ucl.ac.uk

Read the Full Text


THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES:
Reinforcement learning, conditioning, and the brain: Successes and challenges.
T. V. Maia (2009)
Cogn Affect Behav Neurosci 9, 343-364
   Abstract »    PDF »
How Humans Integrate the Prospects of Pain and Reward during Choice.
D. Talmi, P. Dayan, S. J. Kiebel, C. D. Frith, and R. J. Dolan (2009)
J. Neurosci. 29, 14617-14626
   Abstract »    Full Text »    PDF »
Immaturities in Reward Processing and Its Influence on Inhibitory Control in Adolescence.
C.F. Geier, R. Terwilliger, T. Teslovich, K. Velanova, and B. Luna (2009)
Cereb Cortex
   Abstract »    Full Text »    PDF »
Brain Hemispheres Selectively Track the Expected Value of Contralateral Options.
S. Palminteri, T. Boraud, G. Lafargue, B. Dubois, and M. Pessiglione (2009)
J. Neurosci. 29, 13465-13472
   Abstract »    Full Text »    PDF »
Ventral Striatal Neurons Encode the Value of the Chosen Action in Rats Deciding between Differently Delayed or Sized Rewards.
M. R. Roesch, T. Singh, P. L. Brown, S. E. Mullins, and G. Schoenbaum (2009)
J. Neurosci. 29, 13365-13376
   Abstract »    Full Text »    PDF »
Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions.
L. K. Krugel, G. Biele, P. N. C. Mohr, S.-C. Li, and H. R. Heekeren (2009)
PNAS 106, 17951-17956
   Abstract »    Full Text »    PDF »
Corticostriatal Interactions during Learning, Memory Processing, and Decision Making.
C. M. A. Pennartz, J. D. Berke, A. M. Graybiel, R. Ito, C. S. Lansink, M. van der Meer, A. D. Redish, K. S. Smith, and P. Voorn (2009)
J. Neurosci. 29, 12831-12838
   Abstract »    Full Text »    PDF »
Neural computations underlying action-based decision making in the human brain.
K. Wunderlich, A. Rangel, and J. P. O'Doherty (2009)
PNAS 106, 17199-17204
   Abstract »    Full Text »    PDF »
Do Substantia Nigra Dopaminergic Neurons Differentiate Between Reward and Punishment?.
M. J. Frank and D. J. Surmeier (2009)
J Mol Cell Biol 1, 15-16
   Abstract »    Full Text »    PDF »
Stable Encoding of Task Structure Coexists With Flexible Coding of Task Events in Sensorimotor Striatum.
Y. Kubota, J. Liu, D. Hu, W. E. DeCoteau, U. T. Eden, A. C. Smith, and A. M. Graybiel (2009)
J Neurophysiol 102, 2142-2160
   Abstract »    Full Text »    PDF »
Right Ventromedial and Dorsolateral Prefrontal Cortices Mediate Adaptive Decisions under Ambiguity by Integrating Choice Utility and Outcome Evaluation.
A. Christakou, M. Brammer, V. Giampietro, and K. Rubia (2009)
J. Neurosci. 29, 11020-11028
   Abstract »    Full Text »    PDF »
Restriction of dopamine signaling to the dorsolateral striatum is sufficient for many cognitive behaviors.
M. Darvas and R. D. Palmiter (2009)
PNAS 106, 14664-14669
   Abstract »    Full Text »    PDF »
Validation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia.
M. Ito and K. Doya (2009)
J. Neurosci. 29, 9861-9874
   Abstract »    Full Text »    PDF »
Encoding of Marginal Utility across Time in the Human Brain.
A. Pine, B. Seymour, J. P. Roiser, P. Bossaerts, K. J. Friston, H. V. Curran, and R. J. Dolan (2009)
J. Neurosci. 29, 9575-9581
   Abstract »    Full Text »    PDF »
Adaptive Coding of Action Values in the Human Rostral Cingulate Zone.
G. Jocham, J. Neumann, T. A. Klein, C. Danielmeier, and M. Ullsperger (2009)
J. Neurosci. 29, 7489-7496
   Abstract »    Full Text »    PDF »
Opposing Influences of Affective State Valence on Visual Cortical Encoding.
T. W. Schmitz, E. De Rosa, and A. K. Anderson (2009)
J. Neurosci. 29, 7199-7207
   Abstract »    Full Text »    PDF »
Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women.
K. N. Spreckelmeyer, S. Krach, G. Kohls, L. Rademacher, A. Irmak, K. Konrad, T. Kircher, and G. Grunder (2009)
Soc Cogn Affect Neurosci 4, 158-165
   Abstract »    Full Text »    PDF »
Neural correlates of social exclusion during adolescence: understanding the distress of peer rejection.
C. L. Masten, N. I. Eisenberger, L. A. Borofsky, J. H. Pfeifer, K. McNealy, J. C. Mazziotta, and M. Dapretto (2009)
Soc Cogn Affect Neurosci 4, 143-157
   Abstract »    Full Text »    PDF »
Reduced Caudate and Nucleus Accumbens Response to Rewards in Unmedicated Individuals With Major Depressive Disorder.
D. A. Pizzagalli, A. J. Holmes, D. G. Dillon, E. L. Goetz, J. L. Birk, R. Bogdan, D. D. Dougherty, D. V. Iosifescu, S. L. Rauch, and M. Fava (2009)
Am J Psychiatry 166, 702-710
   Abstract »    Full Text »    PDF »
Near-Term Liability of Exploitation: Exploration and Exploitation in Multistage Problems.
C. Fang and D. Levinthal (2009)
Organization Science 20, 538-551
   Abstract »    PDF »
A Dual Role for Prediction Error in Associative Learning.
H. E.M. den Ouden, K. J. Friston, N. D. Daw, A. R. McIntosh, and K. E. Stephan (2009)
Cereb Cortex 19, 1175-1185
   Abstract »    Full Text »    PDF »
Functional Dissociations of Risk and Reward Processing in the Medial Prefrontal Cortex.
G. Xue, Z. Lu, I. P. Levin, J. A. Weller, X. Li, and A. Bechara (2009)
Cereb Cortex 19, 1019-1027
   Abstract »    Full Text »    PDF »
Activity in the Superior Temporal Sulcus Highlights Learning Competence in an Interaction Game.
M. Haruno and M. Kawato (2009)
J. Neurosci. 29, 4542-4547
   Abstract »    Full Text »    PDF »
The medial prefrontal cortex exhibits money illusion.
B. Weber, A. Rangel, M. Wibral, and A. Falk (2009)
PNAS 106, 5025-5028
   Abstract »    Full Text »    PDF »
The Neurobiology of Reference-Dependent Value Computation.
B. De Martino, D. Kumaran, B. Holt, and R. J. Dolan (2009)
J. Neurosci. 29, 3833-3842
   Abstract »    Full Text »    PDF »
Neural Response to Reward Anticipation under Risk Is Nonlinear in Probabilities.
M. Hsu, I. Krajbich, C. Zhao, and C. F. Camerer (2009)
J. Neurosci. 29, 2231-2237
   Abstract »    Full Text »    PDF »
Differential Effect of Reward and Punishment on Procedural Learning.
T. Wachter, O. V. Lungu, T. Liu, D. T. Willingham, and J. Ashe (2009)
J. Neurosci. 29, 436-443
   Abstract »    Full Text »    PDF »
CNTRICS Final Task Selection: Long-Term Memory.
J. D. Ragland, R. Cools, M. Frank, D. A. Pizzagalli, A. Preston, C. Ranganath, and A. D. Wagner (2009)
Schizophr Bull 35, 197-212
   Abstract »    Full Text »    PDF »
Neural correlates of economic game playing.
F. Krueger, J. Grafman, and K. McCabe (2008)
Phil Trans R Soc B 363, 3859-3874
   Abstract »    Full Text »    PDF »
Neurobiological studies of risk assessment: A comparison of expected utility and mean-variance approaches.
M. d'Acremont and P. Bossaerts (2008)
Cogn Affect Behav Neurosci 8, 363-374
   Abstract »    PDF »
Conceptual representations in goal-directed decision making.
N. Shea, K. Krug, and P. N. Tobler (2008)
Cogn Affect Behav Neurosci 8, 418-428
   Abstract »    PDF »
Decision theory, reinforcement learning, and the brain.
P. Dayan and N. D. Daw (2008)
Cogn Affect Behav Neurosci 8, 429-453
   Abstract »    PDF »
A neural basis for the effect of candidate appearance on election outcomes.
M. L. Spezio, A. Rangel, R. M. Alvarez, J. P. O'Doherty, K. Mattes, A. Todorov, H. Kim, and R. Adolphs (2008)
Soc Cogn Affect Neurosci 3, 344-352
   Abstract »    Full Text »    PDF »
Functional Connectivity of Human Striatum: A Resting State fMRI Study.
A. Di Martino, A. Scheres, D.S. Margulies, A.M.C. Kelly, L.Q. Uddin, Z. Shehzad, B. Biswal, J.R. Walters, F.X. Castellanos, and M.P. Milham (2008)
Cereb Cortex 18, 2735-2747
   Abstract »    Full Text »    PDF »
Neural Substrates for Reversing Stimulus-Outcome and Stimulus-Response Associations.
G. Xue, D. G. Ghahremani, and R. A. Poldrack (2008)
J. Neurosci. 28, 11196-11204
   Abstract »    Full Text »    PDF »
Striatal outcome processing in healthy aging.
K. M. COX, H. J. AIZENSTEIN, and J. A. FIEZ (2008)
Cogn Affect Behav Neurosci 8, 304-317
   Abstract »    PDF »
Abnormal temporal difference reward-learning signals in major depression.
P. Kumar, G. Waiter, T. Ahearn, M. Milders, I. Reid, and J. D. Steele (2008)
Brain 131, 2084-2093
   Abstract »    Full Text »    PDF »
Pediatric Bipolar Disorder.
E. Leibenluft and B. A. Rich (2008)
Focus 6, 331-347
   Abstract »    Full Text »    PDF »
Understanding the Neural Computations of Arbitrary Visuomotor Learning through fMRI and Associative Learning Theory.
A. Brovelli, N. Laksiri, B. Nazarian, M. Meunier, and D. Boussaoud (2008)
Cereb Cortex 18, 1485-1495
   Abstract »    Full Text »    PDF »
Neurocomputational mechanisms of reinforcement-guided learning in humans: A review.
M. X. COHEN (2008)
Cogn Affect Behav Neurosci 8, 113-125
   Abstract »    PDF »
Low-Serotonin Levels Increase Delayed Reward Discounting in Humans.
N. Schweighofer, M. Bertin, K. Shishida, Y. Okamoto, S. C. Tanaka, S. Yamawaki, and K. Doya (2008)
J. Neurosci. 28, 4528-4532
   Abstract »    Full Text »    PDF »
Language processing within the striatum: evidence from a PET correlation study in Huntington's disease.
M. Teichmann, V. Gaura, J.-F. Demonet, F. Supiot, M. Delliaux, C. Verny, P. Renou, P. Remy, and A.-C. Bachoud-Levi (2008)
Brain 131, 1046-1056
   Abstract »    Full Text »    PDF »
Expected Value, Reward Outcome, and Temporal Difference Error Representations in a Probabilistic Decision Task.
E. T. Rolls, C. McCabe, and J. Redoute (2008)
Cereb Cortex 18, 652-663
   Abstract »    Full Text »    PDF »
Focal basal ganglia lesions are associated with impairments in reward-based reversal learning.
C. Bellebaum, B. Koch, M. Schwarz, and I. Daum (2008)
Brain 131, 829-841
   Abstract »    Full Text »    PDF »
BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area.
K. D'Ardenne, S. M. McClure, L. E. Nystrom, and J. D. Cohen (2008)
Science 319, 1264-1267
   Abstract »    Full Text »    PDF »
Action and Outcome Encoding in the Primate Caudate Nucleus.
B. Lau and P. W. Glimcher (2007)
J. Neurosci. 27, 14502-14514
   Abstract »    Full Text »    PDF »
Altered Reward Processing in Women Recovered From Anorexia Nervosa.
A. Wagner, H. Aizenstein, V. K. Venkatraman, J. Fudge, J. C. May, L. Mazurkewicz, G. K. Frank, U. F. Bailer, L. Fischer, V. Nguyen, et al. (2007)
Am J Psychiatry 164, 1842-1849
   Abstract »    Full Text »    PDF »
History- and Current Instruction-Based Coding of Forthcoming Behavioral Outcomes in the Striatum.
H. Yamada, N. Matsumoto, and M. Kimura (2007)
J Neurophysiol 98, 3557-3567
   Abstract »    Full Text »    PDF »
Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making.
T. Schonberg, N. D. Daw, D. Joel, and J. P. O'Doherty (2007)
J. Neurosci. 27, 12860-12867
   Abstract »    Full Text »    PDF »
Temporal isolation of neural processes underlying face preference decisions.
H. Kim, R. Adolphs, J. P. O'Doherty, and S. Shimojo (2007)
PNAS 104, 18253-18258
   Abstract »    Full Text »    PDF »
Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point.
A. Johnson and A. D. Redish (2007)
J. Neurosci. 27, 12176-12189
   Abstract »    Full Text »    PDF »
Insights from spatially mapped gene expression in the mouse brain.
S. M. Sunkin and J. G. Hohmann (2007)
Hum. Mol. Genet. 16, R209-R219
   Abstract »    Full Text »    PDF »
Blunted response to feedback information in depressive illness.
J.D. Steele, P. Kumar, and K.P. Ebmeier (2007)
Brain 130, 2367-2374
   Abstract »    Full Text »    PDF »
The Role of the Dorsal Striatum in Reward and Decision-Making.
B. W. Balleine, M. R. Delgado, and O. Hikosaka (2007)
J. Neurosci. 27, 8161-8165
   Abstract »    Full Text »    PDF »
Understanding Neural Coding through the Model-Based Analysis of Decision Making.
G. Corrado and K. Doya (2007)
J. Neurosci. 27, 8178-8180
   Abstract »    Full Text »    PDF »
Neural Responses to Taxation and Voluntary Giving Reveal Motives for Charitable Donations.
W. T. Harbaugh, U. Mayr, and D. R. Burghart (2007)
Science 316, 1622-1625
   Abstract »    Full Text »    PDF »
Neural signature of fictive learning signals in a sequential investment task.
T. Lohrenz, K. McCabe, C. F. Camerer, and P. R. Montague (2007)
PNAS 104, 9493-9498
   Abstract »    Full Text »    PDF »
How the Brain Translates Money into Force: A Neuroimaging Study of Subliminal Motivation.
M. Pessiglione, L. Schmidt, B. Draganski, R. Kalisch, H. Lau, R. J. Dolan, and C. D. Frith (2007)
Science 316, 904-906
   Abstract »    Full Text »    PDF »
Gene gene interaction associated with neural reward sensitivity.
J. Yacubian, T. Sommer, K. Schroeder, J. Glascher, R. Kalisch, B. Leuenberger, D. F. Braus, and C. Buchel (2007)
PNAS 104, 8125-8130
   Abstract »    Full Text »    PDF »
Determining the Neural Substrates of Goal-Directed Learning in the Human Brain.
V. V. Valentin, A. Dickinson, and J. P. O'Doherty (2007)
J. Neurosci. 27, 4019-4026
   Abstract »    Full Text »    PDF »
The Nucleus Accumbens and Pavlovian Reward Learning.
J. J. Day and R. M. Carelli (2007)
Neuroscientist 13, 148-159
   Abstract »    PDF »
Neural Coding of Reward-Prediction Error Signals During Classical Conditioning With Attractive Faces.
S. Bray and J. O'Doherty (2007)
J Neurophysiol 97, 3036-3045
   Abstract »    Full Text »    PDF »
Dynamics of Prefrontal and Cingulate Activity during a Reward-Based Logical Deduction Task.
C. Landmann, S. Dehaene, S. Pappata, A. Jobert, M. Bottlaender, D. Roumenov, and D. Le Bihan (2007)
Cereb Cortex 17, 749-759
   Abstract »    Full Text »    PDF »
Individual differences and the neural representations of reward expectation and reward prediction error.
M. X Cohen (2007)
Soc Cogn Affect Neurosci 2, 20-30
   Abstract »    Full Text »    PDF »
Decoding the neural substrates of reward-related decision making with functional MRI.
A. N. Hampton and J. P. O'Doherty (2007)
PNAS 104, 1377-1382
   Abstract »    Full Text »    PDF »
Facilitation of Saccadic Eye Movements by Postsaccadic Electrical Stimulation in the Primate Caudate.
K. Nakamura and O. Hikosaka (2006)
J. Neurosci. 26, 12885-12895
   Abstract »    Full Text »    PDF »
Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain..
J. Yacubian, J. Glascher, K. Schroeder, T. Sommer, D. F. Braus, and C. Buchel (2006)
J. Neurosci. 26, 9530-9537
   Abstract »    Full Text »    PDF »
Structural Brain Changes in Tinnitus.
M. Muhlau, J. P. Rauschecker, E. Oestreicher, C. Gaser, M. Rottinger, A. M. Wohlschlager, F. Simon, T. Etgen, B. Conrad, and D. Sander (2006)
Cereb Cortex 16, 1283-1288
   Abstract »    Full Text »    PDF »
Amygdala response to facial expressions reflects emotional learning..
C. I. Hooker, L. T. Germine, R. T. Knight, and M. D'Esposito (2006)
J. Neurosci. 26, 8915-8922
   Abstract »    Full Text »    PDF »
The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans..
A. N. Hampton, P. Bossaerts, and J. P. O'Doherty (2006)
J. Neurosci. 26, 8360-8367
   Abstract »    Full Text »    PDF »
Volumetric Analysis and Three-Dimensional Glucose Metabolic Mapping of the Striatum and Thalamus in Patients With Autism Spectrum Disorders.
M. M. Haznedar, M. S. Buchsbaum, E. A. Hazlett, E. M. LiCalzi, C. Cartwright, and E. Hollander (2006)
Am J Psychiatry 163, 1252-1263
   Abstract »    Full Text »    PDF »
Striatal functional alteration in adolescents characterized by early childhood behavioral inhibition..
A. E. Guyer, E. E. Nelson, K. Perez-Edgar, M. G. Hardin, R. Roberson-Nay, C. S. Monk, J. M. Bjork, H. A. Henderson, D. S. Pine, N. A. Fox, et al. (2006)
J. Neurosci. 26, 6399-6405
   Abstract »    Full Text »    PDF »
Ventrolateral Prefrontal Cortex Activation and Attentional Bias in Response to Angry Faces in Adolescents With Generalized Anxiety Disorder.
C. S. Monk, E. E. Nelson, E. B. McClure, K. Mogg, B. P. Bradley, E. Leibenluft, R. J. R. Blair, G. Chen, D. S. Charney, M. Ernst, et al. (2006)
Am J Psychiatry 163, 1091-1097
   Abstract »    Full Text »    PDF »
Neural Coding of Distinct Statistical Properties of Reward Information in Humans.
J.-C. Dreher, P. Kohn, and K. F. Berman (2006)
Cereb Cortex 16, 561-573
   Abstract »    Full Text »    PDF »
Basal Ganglia Orient Eyes to Reward.
O. Hikosaka, K. Nakamura, and H. Nakahara (2006)
J Neurophysiol 95, 567-584
   Abstract »    Full Text »    PDF »
Different Neural Correlates of Reward Expectation and Reward Expectation Error in the Putamen and Caudate Nucleus During Stimulus-Action-Reward Association Learning.
M. Haruno and M. Kawato (2006)
J Neurophysiol 95, 948-959
   Abstract »    Full Text »    PDF »
Human Neural Learning Depends on Reward Prediction Errors in the Blocking Paradigm.
P. N. Tobler, J. P. O'Doherty, R. J. Dolan, and W. Schultz (2006)
J Neurophysiol 95, 301-310
   Abstract »    Full Text »    PDF »
Neural Systems Responding to Degrees of Uncertainty in Human Decision-Making.
M. Hsu, M. Bhatt, R. Adolphs, D. Tranel, and C. F. Camerer (2005)
Science 310, 1680-1683
   Abstract »    Full Text »    PDF »
Representation of Action-Specific Reward Values in the Striatum.
K. Samejima, Y. Ueda, K. Doya, and M. Kimura (2005)
Science 310, 1337-1340
   Abstract »    Full Text »    PDF »
Rat Nucleus Accumbens Neurons Predominantly Respond to the Outcome-Related Properties of Conditioned Stimuli Rather Than Their Behavioral-Switching Properties.
D. I. G. Wilson and E. M. Bowman (2005)
J Neurophysiol 94, 49-61
   Abstract »    Full Text »    PDF »
Reward, Motivation, and Emotion Systems Associated With Early-Stage Intense Romantic Love.
A. Aron, H. Fisher, D. J. Mashek, G. Strong, H. Li, and L. L. Brown (2005)
J Neurophysiol 94, 327-337
   Abstract »    Full Text »    PDF »
Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network.
W.-X. Pan, R. Schmidt, J. R. Wickens, and B. I. Hyland (2005)
J. Neurosci. 25, 6235-6242
   Abstract »    Full Text »    PDF »
Electrophysiological correlates of reward prediction error recorded in the human prefrontal cortex.
H. Oya, R. Adolphs, H. Kawasaki, A. Bechara, A. Damasio, and M. A. Howard III (2005)
PNAS 102, 8351-8356
   Abstract »    Full Text »    PDF »
Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats.
M. Khamassi, L. Lacheze, B. Girard, A. Berthoz, and A. Guillot (2005)
Adaptive Behavior 13, 131-148
   Abstract »    PDF »
Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange.
B. King-Casas, D. Tomlin, C. Anen, C. F. Camerer, S. R. Quartz, and P. R. Montague (2005)
Science 308, 78-83
   Abstract »    Full Text »    PDF »


To Advertise     Find Products

Science. ISSN 0036-8075 (print), 1095-9203 (online)