Delay aversion but preference for large and rare rewards in two choice tasks: implications for the measurement of self-control parameters

Lack of self-control abilities is an important symptom of many psychiatric disorders, notably in the attention deficit/hyperactivity disorder (Sagvolden and Sergeant, 1998; Swanson et al., 1998; Sonuga-Barke, 2003). There is indeed a growing interest in the study of impulsive decision in humans (Krawczyk, 2002; Bechara, 2004). As for animal models, many different aspects of impulsivity have been studied with operant-behavior paradigms. In laboratory settings, impulsive behavior can be defined in terms of "poor" decision making, based on anomalous processing of actual incentive values of the two alternatives (Evenden, 1999; Ho et al., 1999; Monterosso and Ainslie, 1999). In one of the most widely adopted paradigms, the intolerance-to-delay (ID) protocol, slightly food-restricted animals are tested in operant-behavior cages, where they are provided with choice between either one immediate pellet of food (small-and-soon, SS) or five pellets of food after a delay (large-but-late, LL). Animals classically shift their preference from the LL to the SS reward as delay increases. Since impulsive subjects are intolerant to the forced waiting for a delayed reward, a flatter or a steeper shift towards SS choice are a classical index of reduced or increased impulsivity, respectively (Evenden and Ryan, 1996, 1999).

It has been suggested that impulsive subjects may also avoid conditions where the reward income is made uncertain. Such suggestion implies that both "uncertainty" and delay of reward will elicit the same underlying form of intolerance (Mazur 1995; Rachlin et al. 1991). This is based on the assumption that any randomly omitted reward will force animals to wait for one of the next trials (the "lucky" one) to get actual reinforcement, and that they need to cope with this unpleasant but forced procrastination. It has been demonstrated that the perceived value of a given reinforcer is discounted similarly following both "delay" and "probabilistic gaps" in the delivery of rewards (Richards et al., 1999; Ho et al., 1999). Moreover, the detection of reward value across its delayed or uncertain delivery has been suggested to require a common neural substrate, namely an intact nucleus accumbens (Cardinal and Cheung 2005; Cardinal and Howes 2005). However, there is evidence that time- and probability-induced discounting are different and dissociable processes (Green et al. 1999; Green and Myerson 2004). This notion should be kept in mind, since a shift from a small-and-sure (SS) towards a probabilistic (large-but-luck-linked, LLL) reward is often discussed in terms of impulsivity-driven aversion for uncertainty in classical literature studies that used probabilistic-discount (PD) protocols (Mobini et al., 2000, 2002).

A crucial point in intolerance-based tasks is to consider whether the actual parameters make the shift in preference (from large reward in either protocol, termed LL/L, to SS) economically convenient or not. In the ID task, delay duration is always imposed in a range of values which render any SS shift overtly sub-optimal, and this assumption is actually confirmed by present data. Conversely, within uncertainty-based tasks, probability ("p") values to be imposed can be divided into two distinct fields, separated by the indifference-point "p" value (calculated as "small reward size"/"large reward size", e.g. 20% in our present work, where SS was 1 and LL/L was 5). In the range of "p" values before the mathematical indifference point (100% > p > 20%), the risk of large reinforcement loss is mild relative to its size. Under these conditions, it is still "economically" convenient for rats to choose the LLL (average outcome being still more than one pellet per nose-poking) over SS reward (only one pellet per nose-poking). Hence, if animals display a shift towards SS, this would be interpreted as an intolerance reaction against uncertainty. In the range of "p" values beyond the mathematical indifference point (20% > p > 0%), it becomes "economically" convenient for rats to choose SS (one certain pellet per nose-poking) over LLL reward (the average outcome being less than one pellet per nose-poking). In classical literature studies (Mobini et al., 2000, 2002), untreated control animals showed a shift from the probabilistic towards the certain reward, but this happened under conditions that rendered such a shifting actually optimal in terms of food gain. This consideration raises the possibility that the observed shift rather represented the obvious consequence of the natural drive towards food maximization. By definition, impulsive decisions are taken "without consideration of (negative) consequences". Hence, only those behavioral responses, which can be demonstrated to be still adopted despite adverse consequences on total foraging, can be defined as truly impulsive.

In the present experiment, we wished to directly compare the ID and PD protocols, to evaluate whether or not adolescent rats would display an equivalent intolerance towards delay vs uncertainty of reward delivery. The specific age-period for testing was chosen mainly because the peculiar behavioral instability, which is typical of adolescence as demonstrated by previous work (Laviola et al., 1999, 2003; Spear, 2000), might be pushing animals towards extreme drives at this age, such as novelty- and risk-oriented behavior. We employed a range of delay/probability values under which the "optimal" choice would always be the large reward. In this way, any shift towards the SS reward would be a "true" index of impulsivity-driven intolerance, since it would be anti-economical from the point of view of foraging optimization. As expected, in the ID protocol, animals shifted towards the SS reward, a classical index of impulsive choice. Conversely, in the PD protocol, animals did not shift against economical convenience. Rather, their preference for the LLL reward was very stable, being maintained even when its delivery was made very infrequent, so that it would be economically convenient to shift to the SS one, at least in theory.

As a final remark, we wish to underline that the indifference point is separating, in both protocols, the field where LL/L choice is always optimal from the field where the SS choice would be optimal. Such turning point (i.e. odds against obtaining the large reinforcer being 4 in our experiment) is mainly based on abstract mathematical calculation and comparison of the final payoff obtained with a clear-cut (i.e. 100%) SS vs LL/L choice. However, the behavioral strategy adopted by rats is rather complex, consisting of preferential choice at a given hole plus a lower but constant patrolling at the other hole. Under these conditions, rats are unlikely to detect very slight differences between the two alternative outcomes, and may hence be under quasi-indifference conditions. Thus, rather than being pushed by overt economical convenience, animals might express their hole-preference based on the emotional evaluation of rewarding features issuing from either alternative option. A further aim of the present work was to compare the economical contingencies of the two procedures around the indifference point. Specifically, we compared actual foraging with theoretical food gain, ideally obtained by maintenance of LL choice in ID task and by a shift to SS reward in PD task. Specifically, we calculated what amount of food rats could have eaten if they behaved just as their siblings in the other protocol did. Such a calculation allowed to assess what economical conditions rats were actually facing. Such analysis was focused on the specific window around the mathematical indifference point, which represents a sort of upper limit in intolerance-based tasks. Indeed, to avoid the bias of an economically-driven preference shift as outlined above, task contingencies shall be able to generate phenomena of behavioral intolerance by this point.

This study compared the behavioral reaction to rarefaction of large reinforcement in a pair of similar two-choice operant-behavior tasks, where preference for large or small rewards is assessed in food-restricted rats. We wish to underline that the present animals were trained to consume their daily meal during the testing session and shortly when placed back into the home cage. In this way, they were facing a 23 h/day feeding absence, and were hence motivated to work for food during operant testing. While undergoing this procedure, which is largely adopted in the literature, rats are not under starvation nor malnutrition, since they fully express a normal behavioral repertoire in the home cage, including a playful social behavior. The attributes of food reward in either test share some similarities and some key differences. Namely, the small reward (SS) is always for sure and comes immediately, whereas the large reward delivery may be either delayed (ID protocol) or uncertain (PD protocol). Specific values, to be run for each daily session, were selected after an a priori calculation of the correspondence between delays and odds/probability, aimed to produce a similar level of rarefaction for LL/L delivery in both protocols. This was to be sure that the striking differences in behavior, generated by the two protocols, could not be explained by gross differences in the maximal frequency of large rewards potentially obtained, and should be ascribed to other intrinsic key features. An a posteriori re-estimation of equivalence between odds/probability and delays was conducted, which scaled the whole curve generated by the ID task towards the left. As a result, differences between choice profiles in the two tasks were even strengthened. Indeed, testing of animals until a delay of 150 s within the ID task, which would be substantially equivalent to odds = 6 reached in the PF task, likely would produce a further enhancement of the dissociation observed between the two profiles.

The experimental observations reported here demonstrate that the profile shown in PD protocols may differ substantially from classical intolerance/aversion observed in ID protocols. As classically reported, animals in the ID protocol then developed a robust aversion for the longest delays, and showed a marked shift from LL to SS choice. At this point, animals are likely to have realized that there is nothing to do to avoid the elapsing of delay intervals, and are therefore likely to develop an anti-economical intolerance (Evenden, 1999; Ho et al., 1999; Monterosso and Ainslie, 1999). Conversely, animals tested in the PD protocol did not show the same profile. As a reaction to an increasing proportion of omitted reinforcement, animals developed and maintained a robust preference for LLL reward. Such behavior may in part be explained by the fact that chamber and magazine lights, required to signal starting and ending of the time-out period, were also turned on when food delivery was omitted. These lights may indeed be acting as a conditioned reinforcers, and sustain LLL-seeking behavior until the primary reinforcer eventually comes (as in second-order schedules, see Everitt and Robbins, 2000). These considerations may help formulating a possible explanation, namely that training with the uncertainty challenge was gradual enough for rats to realize that large reinforcement eventually comes, despite the repeated and unpredictable omissions. Preference for LLL until odds of 5 and 6 is striking, since over 80% of these nose-poking demands did not trigger any food delivery. It is intriguing to note that the average waiting-time for the eventual LLL reward was 60 seconds or more, a delay which produced a considerable aversion for LL reward when assessed in the ID protocol.

It is noteworthy that animals faced with either delays or omissions reacted in two distinct ways. After a substantial similarity at the mildest levels of LL/L rarefaction, choice curves began to diverge beyond odds = 1, which was substantially equivalent to delay = 30 s. In our opinion, the crucial difference between LL and LLL rewards is that the former is delayed in a signaled and predictable manner, whereas delay in obtaining the latter is completely unpredictable and luck-linked (Cardinal et al., 2000). Indeed, once triggered, the delay in the ID protocol must follow its scheduled duration, and there is nothing that animals can do. This situation is likely to induce a state of frustration, thus generating intolerance/aversion for delays. Conversely, after each omitted large-reward delivery in the PD protocol, animals have the possibility to express another nose-poking choice. In general, there are more opportunities of nose-poking choices in the PD protocol, and this may help animals to feel that the situation is more "under control". A great number of studies have shown that controllability of stress sources helps animals to cope with adverse contingencies, and hence uncertainty of LLL reward should be not necessarily expected to produce aversion.

From our data, it seems that LLL rewards are actively preferred by rats. Animals apparently adopted the strategy to wait for a "lucky" but "rare" event, rather than collecting many smaller reinforcement. There are two possible explanations for this finding. One is that animals developed a "habit", the other is that rats were perhaps attracted by "binge" reinforcement, without being affected by its uncertainty neither by its rarefaction. As for the first possible explanation, the finding of minor effects on choice behavior in the PD protocol may indicate that the choice of nose-poking hole somewhat continued to occur independently from the outcome. This could suggest that rats developed a compulsive and perseverant hole preference, possibly due to the establishment of a behavioral habit. In other words, normal rats seem to express a sort of fixed habit-based responding, rather than being open to an evaluation of the actual outcome. Accordingly, recent findings (Adriani et al., 2004) raise the question of behavioral inflexibility (with a tendency to perseveration) shown during preference shifts, and suggest that some individuals may be less flexible than others (Evenden, 2002; Laviola et al., 2003). However, whilst PD rats were quite insensitive to LLL rarefaction, their ID-protocol siblings quickly displayed intolerance against large-reinforcement delay. ID subjects showed a shift to SS choice, a finding fully consistent with impulsivity-based responding, according to previous work (Evenden and Ryan 1996, 1999). It seems unlikely that PD rats were characterized by a lack of flexibility when their ID siblings were able to shift in their choice. Thus, we propose that the PD protocol generates a true "instinctive" preference for LLL rewards. Noteworthy, our present findings unequivocally suggest that rats prefer a random-coming reward, delivered eventually and all-at-once, rather than a similar overall gain, collected by slow accumulation of smaller unitary bits. In PD rats, the salient cue to decision seems to reside into the size attractivity of unitary reward, and not in frequency of its delivery nor in the total amount gained (or lost) over time.

Classical definitions of impulsivity predict that a given factor (delay, uncertainty) shall discount large-reward value, and that a choice shift is then observed against economic convenience of the outcome (Evenden, 1999; Ho et al., 1999; Monterosso and Ainslie, 1999). The phenomenon of uncertainty-produced discounting did not occur in our hands. It is likely that animals would eventually show a shift towards SS, at least for "p" values much lower than those presently tested. However, a shift to SS reward would be economically convenient in these conditions, and cannot therefore be used as an index of impulsive behavior. Indeed, only anti-economical choices can be considered an unbiased and reliable index in the field of impulsive decision making (Evenden, 1999; Ho et al., 1999; Monterosso and Ainslie, 1999). Such kind of considerations also apply to interpretation of data by Mobini and colleagues (2000, 2002). These authors employed a "small"/"large" reward ratio of 0.5 (Mathematical indifference point at "p" = 50%) and tested rats under a range of "p" values well beyond this point (Mobini et al., 2000, 2002). In our opinion, by promoting a preference shift according to (rather than against) economical convenience, these features do bias the preference shift as a parameter for impulsive decision making. Abnormalities in the process of large-reward discounting, observed by these authors in rats after various manipulations, might well be discussed in terms of altered perception of reward magnitudes and/or deficits in comparing global payoff of either choice option. In other words, their protocol contingencies may be useful to evaluate abnormalities in the emergence of an economically-forced shift in choice behavior, being conversely not suitable to measure impulsive choice, by definition.

Two considerations must be highlighted here regarding models of reward discounting. It is assumed that, when animals discount the value of a delayed or uncertain reward, then the value of the SS reward is compared against the discounted value of the LL/L reward to take a decision. These considerations are put forward by also assuming that perception of reward size is equivalent in both paradigms and that odds against rewarding may be compared across the two protocols. A first consideration is that delay-induced discounting is apparently a more reliable phenomenon than uncertainty-induced discounting. In other words, rats poorly cope with delay and inescapable waiting constraints, whereas they accept unpredictable omissions in reward-delivery. Moreover, under the present conditions, where the final net foraging was not affected, rats preferred to work for "binge wins", even at high levels of rarefaction, rather than shifting towards a SS-seeking strategy. The latter, made of nose-poking for a lower and constant outcome, would imply a flatter distribution of food gains. Such observation again suggests that very similar temporal distributions, made of bouts of five pellets coming far apart each other, are potentially generated in both tasks. However, such kind of distribution can be avoided, when the distance between bouts is an inflexible constraint (ID task), and can conversely be preferred, when its length varies in an unpredictable way (PD task). One possible explanation for the preference, displayed in the latter task, is that the LLL-choice behaviour may be reinforced by the contrast between many no-food trials and the eventual five-pellet one. Such kind of contrasts have been shown to increase dopaminergic activity in the midbrain (Fiorillo et al., 2003), and may support risk-taking behaviour (Van den Bos et al., 2005).

A second consideration is that there might be a pitfall in those models, where criteria for decision reside in comparison between unitary value of SS versus unitary discounted value of LL/L reward. This assumption may stand valid for a sort of "instinctive" decision process, where two alternative unitary values are first discounted and then compared. But more "evolved" decision processes may occur, where animals may be capable to take into account the global convenience of a choice strategy, by considering delivery failures and food amount earned (or lost) across multiple choices. This kind of process requires the ability to consider, beyond each single choice, the estimated quantity of food gained having access to many choices during a given time interval. Without taking into account this process, the conceptual models describing decision making may be limited. In this light, the PD task adopted here is suggested to unveil attraction for "binge and rare" over "low and constant" reinforcement, when rarefaction is obtained by random omissions (rather than a constant delay) of reinforcer delivery. This behavioral drive, consisting of willingness to seek for a highly rewarding sensation despite association with some negative features (such as random rarefaction in the present case), is not surprising. Much effort has been devoted to study determinants of risk-seeking drives, in terms of neurobiological substrates and adaptive function (Bardo et al., 1996; Laviola et al., 1999). Present observations may open new perspectives for studies in the field of peculiar (patho)physiological conditions, such as sensation seeking, reckless behavior, and gambling.

There is no doubt that behavioral output in a two-choice operant paradigm provides precious information about decision-making strategies. However, we showed quite clearly that the decision taken by animals, in terms of choice between a larger (but less frequent) versus a smaller reinforcer, provides insights that do not necessarily reside within the "impulsivity" dimension. Depending on protocol contingencies, and specifically in the absence of the strong aversion otherwise elicited by inflexible delays, other dimensions emerge, such as the natural attraction for "binge" reinforcement. Under conditions of economical quasi-indifference, i.e. when there were no overt adverse consequences on the total foraging, such attraction was able to sustain the LLL preference, despite a considerable rarefaction in actual food delivery. Under these contingencies, animals can freely choose a temporal profile of reward distribution, ranging between two possible extremes: a "sharper" one, made of random-coming "lucky-strike" episodes, separated by time intervals of uncertain duration, or a "smoother" one, obtained by a more frequent alternation of either choice. The latter strategy would possibly allow to dilute the frustration, associated with the unlucky attempts to obtain more food, with a more frequent nose-poking for the single and certain pellet. However, not always laboratory animals are driven by simplistic cost/benefit rules. It is known that some strategies, being apparently less rewarded and/or requiring more effort, are often adopted and perhaps more adaptive, a phenomenon called "contrafreeloading" (Inglis et al., 1997). In the present specific case, rats may be specifically motivated by the "temptation to gamble" and may conversely find more "boring" a monotonous collection of small bits. In this frame, we propose that such a behavioral dimension might more closely resemble features that are typical of a "gambling" trait. The present study may thus provide novel useful insights for the interpretation of PD tasks.

It should be underlined that these two protocols may be used to probe animals for the balance between "primordial" and "evolved" drives. In other words, a similar process is likely to be required to overcome aversion to delay and attraction for "binge" rewarding, in order to produce more self-controlled individuals. In theory, it should be possible to modulate the weight of primordial "delay aversion" and "binge-reward attraction" drives, and/or the ability to inhibit these primordial drives to express an evolved and truly self-controlled response. Such ability is known to require intact serotonergic activity (Wogar et al., 1993; Harrison et al., 1997; Puumala and Sirvio, 1998; Dalley et al., 2002), expecially within the pre-frontal cortex (McClure et al., 2004; Ridderinkhof et al., 2004) and the cortico-striatal projections (Cardinal et al., 2004; Christakou et al., 2004). We have recently obtained some data, showing that exposure to methylphenidate during adolescence resulted in enduring changes, adult rats being more self-controlled in both tasks. Specifically, methylphenidate-exposed rats were less impulsive in the ID protocol (Adriani et al., in preparation), thus increasing their overall food gain, and showed an increase of low-risk certain-payoff nose-poking in the PD protocol, thus smoothing the temporal distribution of foraging and diluting the uncertainly-rewarded periods (Adriani et al., 2005). These animals were apparently not driven by "delay-avoidance" nor by "binge-attraction". Rather, they seemed to overcome such instinctive drives, by looking beyond single-choice outcome and by taking into account the long-term payoff.

A final remark shall underline that the present data were obtained on adolescent rats, which are known to differ substantially from adults in reward-related manifestations (Spear, 2000; Laviola et al., 1999, 2003). Whether these findings can be extended to adult rats remains an open question, deserving further work. We demonstrated previously that adolescent mice are more impulsive than adult ones (Adriani and Laviola, 2003) but, to the best of our knowledge, there are no rat studies comparing age-dependent profiles for either ID or PD task. Present data are however relevant in view of recent literature, suggesting that animal models of adolescence may be useful to evaluate age-related physiological trajectories that may, in some cases, progress into psycho-pathological processes (Andersen, 2003). In this frame, elaboration of correct expectations from two-choice protocols is essential for a deeper psycho-biological investigation of reward-seeking abnormalities.

Experimental protocols were approved by institutional authorities and are in close agreement with European Community Directives and with the Italian Law. All efforts were made to minimize animal suffering, to reduce the number of animals used, and to use alternatives to in vivo testing.