7#,}}B   "vtx *IG " >- made some changes -> APA - we can keep paragraph numbering, Cognition uses it - I have collapsed a few short paragraphs into a single larger one - Added footnote on non-symbol architectures, e.g. neural nets - in section 2.7, I propose to cut some redundant sentences. I still have difficulties parsing the first of section 2. Maybe we could get rid of the letters and italicize the to show it is the essence of our proposal: A way to keep the multiple-constraint advantage offered by UTCs while making it tractable is as follows: By gathering a large number of empirical/experimental observations on a single subject (or few subjects analysed individually) and using a variety of tasks exercising multiple abilities (e.g. perception, memory, problem solving), to develop a detailed computational model of the subjects behaviour in these tasks that is able to learn while performing the tasks. The model will be (e)updated and checked as new data is added and the model refined. The combination of both UTCs and using individual subject's data is the key, as we shall argue below. We believe that this approach solves the problem of between-subject variability, so detrimental if one wants to use the constraints offered by the data to their full strength. After elaborating on the main advantage offered by this approach, we discuss several potential difficulties of our approach, and show that none of them is critical. Individual Data Analysis and Unified Theories of Cognition: A Methodological Proposal Fernand Gobet & Frank E. Ritter ESRC Centre for Research in Development, Instruction and Training and Institute for Applied Cognitive Science University of Nottingham Address of correspondence: Dr. Fernand Gobet ESRC Centre for Research in Development, Instruction and Training Department of Psychology University of Nottingham Nottingham NG7 2RD England + 44 (115) 951 5402 (phone) + 44 (115) 951 5324 (fax) Fernand.Gobet@nottingham.ac.uk Running head: Individual Data Analysis and Unified Theories of Cognition Abstract Unified theories regularly appear in psychology (e.g. Skinners, Hull's, and Piaget's theories). They also regularly fail to fulfill all of their goals. Newell (1990) has called for a revival of unified theories, using computer modelling as a way to avoid the pitfalls of previous attempts. His call, embodied in the Soar project, has however so far failed to produce the breakthrough it promised. We will argue that one of the reasons for the lack of success of Newell's approach is that the methodology commonly used in psychology, based on controlling potentially confounding variables by using group data, is not is not the best way forward for developing unified theories of cognition. Instead, we will propose an approach where (a) the problems related to group averages are alleviated by analysing subjects individually; (b) there is a close interaction between theory building and experimentation; and (c) computer technology is used to routinely test versions of the theory on a large set of data. We show that the advantages of this approach overwhelmingly dominate its disadvantages. Individual Data Analysis and Unified Theories of Cognition: A Methodological Proposal 1. Introduction What is the best way to make theoretical progress in the study of behaviour in general and of cognition in particular? To develop micro-theories explaining a small domain, or to aim at a higher goal, and develop an overarching theory covering a large number of domainsa unified theory? Modern psychology, as a field, has tended to prefer the former approach. It is true that unified theories have regularly appeared in psychologythink of Hulls (1943), Piagets (1954) or Skinners (1957) theoriesbut is generally admitted that such unified theories have failed to offer a rigorous and testable picture of the human mind. Given this relatively unsuccessful history, it was with excitement that cognitive science has observed Newell's (1990; see also Newell, 1973) call for a revival of unified theories in psychology. Newell, who focused on cognition, was quite aware of the problems that plagued previous attempts: vagueness, lack of specificity, and untestability, to cite the most damaging. He was also aware that psychology now has a tool that was not available to Hull, Piaget or Skinner to avoid these problems: computers. Newell embodied his call for unified theories of cognition (UTCs) in the Soar project, where computer modelling plays a preponderant role. In the ten years or so the Soar project has been running, the verdicts by observers have gone from mild support (e.g. Norman, 1991; the commentaries following Newell, 1992; Stefik et al., 1993) to highly negative (Cooper & Shallice, 1995). We believe that one of the reasons for the lack of complete success of Newells own brand of UTC is that the methodology commonly used in psychology, based on controlling potentially confounding variables by using group data, is not the best way forward for developing UTCs. Instead, we proposed an approach, which we call individual data analysis, where (a)the problems related to group averages are alleviated by analysing subjects individually; (b) there is a close interaction between theory building and experimentation; and (c)computer technology is used to routinely test versions of the theory on a large set of data. The discussion of the advantages and disadvantages of this approach will show that there are significant advantages and that this approach will help traditional approaches progress as well. The main potential disadvantagelack of generalitymay be taken care of by adequate testing procedures. 1.1. Unified Theories A common criticism of unified theories of the past (e.g. see Chomsky, 1959, for a criticism of Skinners theory) is that they were formulated in rather vague terms, and that, as a consequence, both their internal consistency and their testability were open to serious doubts. This criticism also applies to some extent to theories such as Piagets and Hulls, which, although formulated formally (logic for the former, mathematics for the latter), were too unspecified and awkward to make direct, testable empirical predictions. The strength of Newells (1990) argument for UTCs is that it avoids the danger of lack of specificity by showing that we have now the necessary technology (the computer) to build up theories complex enough to match human intelligence. The most advanced computational theories of cognition, including Newells (1990) Soar and Anderson and Lebires (1998) ACT-R, are an existence proof that psychological theories can be formulated in a way that satisfies the exigencies of scientific rigor and specificity while predicting intelligence by exhibiting it themselves. Newell (1990) proposed a special type of UTC as a new methodological way of studying cognition. The key idea is that a single architecture should be used to account for as many regularities in empirical data as possible, and that this architecture should be implemented in a computer programboth to avoid the vagueness of verbal theories and to make it possible to simulate complex behaviour. Newells insight, already anticipated in Newell (1973), is that multiple constraints are brought to bear with UTCs, allowing one to limit the number of degrees of freedom in the theory and hopefully to converge to a theory that accounts satisfactorily for most of the regularities of human behaviour. Newell is clear (1990, p. 16-17) that this approach does not imply that a single mechanism must unify, but that the set of mechanisms must work and exist together in a unified whole. To make Newells ideas more concrete, let us consider a simple, idealised example. Researcher A develops a theory of memory, and manages to estimate two parameters: capacity of working memory (WM), and time to create a new node in long-term memory. On her own, Researcher B develops a theory of problem solving in arithmetic, and uses a parameter for the capacity of WM. This is essentially a free parameter that can be adjusted to fit the data. Now consider Researcher UTC, who is simultaneously interested in both domains, memory and problem solving in arithmetic. For her, the capacity of WM is not a free parameter anymore, as it was "set with simulations on memory. The parameter estimated for WM has constrained the space of possible theories for researcher UTC. Were researcher UTC to change values of the WM parameter, perhaps because its current value does not allow problem solving in arithmetic to be carried out at all, she would have to retest, and perhaps revise her theory of memory with the new value. Without a UTC, it is much harder to propagate the constraints across subtheories, both because of the amount of information to be processed by the theorists (i.e. translating the restrictions between formalisms), and because of the biases that they may hold for or against features of their theories. UTCs are, therefore, the necessary vehicle to propagate constraints across subtheories. In fact, attempts to propagate constraints across subtheories start to create unified theories. 1.2. Soar as a Candidate UTC Newell illustrates his UTC methodology with Soar, a candidate Unified Theory of Cognition." Soar, which is both a cognitive theory and an AI system, represents intelligence as a function of problem solving and learning, and essentially describes cognition as search in problem spaces. In Soar, all knowledge is encoded as productions and all learning is done by chunking. (See Newell, 1990; Newell, 1992; or Baxter & Ritter, 1996, for more detailed descriptions of Soar). How does Soar fare with empirical data? Reasonably well, as it has been tested in detail against various domains and types of data. A partial list includes reaction tasks, typing, skill acquisition, problem solving (e.g. cryptarithmetic), reasoning (syllogisms), development (balance-beam task), sentence comprehension, human-computer interaction, reflection, reasoning with syllogisms, episodic memory, categorical learning, and driving (Aasman & Michon, 1992; Altmann & John, in press; Howes & Young, 1996; Lewis, 1996; Miller & Laird, 1996; Newell, 1990; Polk & Newell, 1995). In particular, the chunking mechanism used by Soar offers a parsimonious explanation of the ubiquitous power law of learning (Nerb et al., 1993; Newell, 1990; Rosenbloom & Newell, 1987). In spite of this long list of achievements, which even its stark opponents acknowledge (e.g. Cooper & Shallice, 1995), Newells UTC approach has been subjected to a tide of criticisms, which may be classified into two categories: criticisms against Soar, and criticisms against UTC as a general research methodology. Norman (1991) is a good example of a set of criticisms aimed at Soar as a cognitive theory. Norman finds implausible the level of theoretical unification proposed by Soar: a single learning mechanism, a single knowledge representation, and a uniform problem state. The lack of capacity limit in working memory is also seen as a problem. Norman also regrets that Soar does not take into account more neuropsychological evidence, and notes that there may be non-symbolic intelligence, which is not captured by Soar because it only uses symbolic mechanisms. A different line of attack against Newells project is to identify the methodological difficulties faced by UTCs in general. This line is adopted, among others, by Cooper and Shallice (1995). They assert that any theory can be implemented in Soar, and that, as a consequence, Soar can be seen as just a powerful computer language. They also deplore the gap between theoretical descriptions and computational implementations of the theory, and consider that UTCs do not adequately address the so-called Reitmans (1965) irrelevant specification problem (what aspects of a program make psychological claims, and what aspects are present just to have the program run?). They also note that start-up assumptions can be tailor-made for each task and that Soar modellers may use different assumptions, thus destroying the potential benefit of UTCsbringing to bear multiple constraints. Even though Cooper and Shallice (1995) identified some genuine difficulties faced by UTCs and illustrated the uneasiness that traditional approaches in psychology have with Newells approach to a UTC, many of their criticisms are unwarranted (see Young, Ritter & Gobet, in preparation). However, the focus of this paper is to address a set of other difficulties not addressed by Cooper and Shallice that seriously hamper the usefulness of UTCs, and, in particular, the supposed strength of bringing together multiple constraints. These difficulties have to do with the way constraints are (or are not) efficiently used to prune the search space of possible theories. We can identify four difficulties that we will attempt to address with our new approach: (a) How to deal with between-subject variability? Is averaging the data a good solution? (b) How to deal with between-task variability? (c) How to control for subjects strategies? (d) How to estimate quantitative parameters and qualitative parameters (e.g., strategies) using psychological data? A non-negligible part of these difficulties come from using group data. The pitfalls associated with such groupings have been known for a long time, but are rarely dealt with satisfactorily in current research. It has been noted several times that data averaged over people may not accurately reflect the behaviour of any person (Estes, 1956; Newell & Simon, 1972; Siegler, 1987). The same point has been made for data averaged over a task, where subjects may use different strategies (Delaney, Reder, Staszewski, & Ritter, 1998; Newell, 1973; Siegler, 1987). Finally, what is the meaning of cognitive parameters estimated from the average subject, and what constraining power do they have? A solution may be to use ranges of parameter values. For example, Card, Moran, and Newell (1983) estimated the range of the possible values associated with the capacity and the decay rate of various information stores, such as visual short-term memory. Using ranges has, however, the disadvantage that UTCs may lose one of their strongest aspects (using multiple constraints), just because ranges do not constrain the theory as well as point measurements. Another solution is to dispose with group data completely, and to turn to Individual Data Analysis. We will explore the second solution here. 1.3. Individual Data Analysis (IDA) The obsession of modern psychology with statistical testing has led its practitioners to hold strong prejudices against IDA. However, research using IDA has a long history in psychology (see Dukes, 1968, for a review), including Freuds efforts to develop psychoanalysis and Piagets clinical method used to understand childrens development. Even now, this methodology is not uncommon in neuropsychology, where the rarity of patients with specific brain damages almost compels IDA, in clinical psychology, which actually has developed methodological tools based on IDA to study the effect of therapies, and in psychophysics, where two subjects are typically needed, the experimenter and a naive subject to test for idiosyncrasies of the experimenter. IDA has had a lasting impact on cognitive psychology (broadly defined) as well. Ebbinghaus (1885) started the field of verbal learning by experimenting on himself. Empirical studies of the acquisition of language tend to focus on very few, if not single, subjects (e.g. Brown, 1973). In developmental psychology, Siegler has several times (e.g. Siegler, 1987) warned about the dangers of averaging data, and has developed techniques to study the development of each child separately (microgenetic methodology). The information-processing approach to problem solving (Newell & Simon, 1972) has also tended to focus on subjects individually. For example, in a review of process models that had their sequential predictions tested with verbal and non-verbal protocols, 11 out of the 22 studies used single subjects (Ritter & Larkin, 1994). Finally, Lovett, Reder, and Lebire (1997) and Miwa and Simon (1993) have investigated ways of using computer modelling to estimate individual parameters. Within cognitive psychology, research on skill acquisition has made heavy use of IDA. Bryan and Harter (1899), studying how Morse code is learnt, or Seibel (1963), analysing a choice reaction-time task, have studied single subjects tested for long period (for up to 70,000 trials) on a given task. De Groot (1946/1978), has launched the modern study of expertise using analyses that focused on the detailed description of individual subjects. (Ironically, De Groots book is remembered best for its quantitative analyses, using group data, which is covered in only a handful of pages.) This influence remains in current experiments on expertise, where a single subjects development is studied for a long period of time, using intensive data collection, or where the same (few) subjects are observed in a variety of tasks. As examples, we may mention the digit-span task (Chase & Ericsson, 1982) recently simulated by Richman et al. (1995) or chess (Chase & Simon, 1973; Gobet & Simon, 1996). 2. A Proposed Solution to Some UTC Difficulties A way to keep the multiple-constraint advantage offered by UTCs while making it tractable is (a)by gathering a large number of empirical/experimental observations on a single subject (or few subjects analysed individually) and (b)using a variety of tasks exercising multiple abilities (e.g. perception, memory, problem solving), to (c)develop a detailed computational model of the subjects behaviour in these tasks that (d)is able to learn while performing the tasks. The model will be (e)updated and checked as new data is added and the model refined. The combination of both UTCs and using individual subject's data is the key, as we shall argue below. We believe that this approach solves the problem of between-subject variability, so detrimental if one wants to use the constraints offered by the data to their full strength. After elaborating on the main advantage offered by this approach, we discuss several potential difficulties of our approach, and show that none of them is critical. 2.1. Rapid Interaction between Theory Development and Data Collection We believe that the main advantage offered by this methodology is to allow hypotheses to be generated and tested in a rapid cycle, and therefore to rapidly improve the theory. This methodology also allows one to collect converging evidence, and to rapidly test and retesting theoretical mechanisms and parameters. Interacting with the model in such a way will suggest new experiments or new variations of old experiments. The experiments may then been carried out rapidly (there is only a single subject to arrange and run!), and the feedback may be compared swiftly to the theory. This rapid collection of data, which is similar to other fields of research such as biochemistry (see Jacob, 1980, for a breathtaking description of research in this field), allows both rapid feedback and a close interaction of theory building and data collection. This pace is in stark contrast with the relatively slow collection of data typical for psychology. Because they become usable, data constraints become real. The use of previous values immediately informs and constrains the theory that is being developed rapidly. 2.2. Practicality of the Approach The first obvious objection to the proposed approach is that it is not implementable in practice. Gathering a large variety of data is difficult, and having a subject participate in multiple experiments is costly. Can we hope to gather the amount and variety of data necessary? We believe it to be possible. Many experiments on perception, memory, and problem solving are computer-based (e.g. the PsyScope software described by Cohen et al. (1993); also see the suite of software referenced in Anderson & Lebire, 1998). Computer-based display of experiments speeds up acquisition of data and allows collection of detailed data (e.g. reaction times). It also makes it possible to interface the UTC model with the software used to run experiments. There are now at least two psychologically-plausible perceptual interfaces that can be used for this purpose: the Nottingham Hand-Eye Architecture (Baxter, Ritter, Jones & Young, submitted) and ACT-R/PM (Byrne & Anderson, 1998). As the theory may guide the selection of further experiments, the subject(s) should be kept available so that he or she can participate in new experiments. We contend that the financial cost of reusing a subject is less than running dozens of subjects on various experiments. A good example that this approach is practicable is offered by the research on the digit-span task (Chase & Ericsson, 1982; Staszewski, 1990). 2.3. Controlling for Strategies In most current theories, strategies are essentially free parameters, and as such impede the effective use of multiple constraints. It is one of the strongest features of our approach that it offers a solution to this problem. It is now possible to capture changes of strategies within a task by a single subject due to the collection of data that are detailed and varied enough (e.g. eye movements, verbal protocols, reaction times, and so on) and that allow cross-validation between data types, and due to the presence of a simulation model that allow fine-grained predictions. Experiments can also be carried out where strategies are systematically and specifically varied (Medin & Smith, 1981; Gobet et al., 1996). In the long term, this set of converging evidence will constrain strategies into fixed parameters for each subject, therefore reducing the number of degrees of freedom of the theory. 2.4. Controlling for Learning and Other Confounds A legitimate worry is that the single subject's learning that occurs during the sessions will corrupt the data of later sessions. But here, because current UTCs include mechanisms for learning, the effect of learning is no longer a confound. Learning can be the object of theoretical investigation and simulation, for example, to analyse how the estimated parameters are affected by practice. Another advantage of our approach, in particular its emphasis of coupling the simulation model to the same experimental apparatus as human subjects, is that experiments can be modelled in detail (we know exactly what was done). Interestingly, this includes experiments which would not seen as perfect from a traditional methodological point of view. Consider the example of an experiment where the order of stimuli presented to a subject has not been randomised optimally. This infelicity in the design does not matter within our approach, because the model can be subjected to the same tasks as the human subjects, and we may actually predict, or postdict, what is the effect of the order of presentation confound. 2.5. Requirements for Software Development An important aspect of our approach is that the (computational) theory must be regularly tested against the empirical data, to make sure that any change in theory (both with mechanisms and parameters) does not vitiate previous simulations. This was one of the ideas behind Newells program, but it has not been carried out systematically within the Soar community (Cooper & Shallice, 1995; Ritter, 1995). This regular testing of previous simulations, that the ACT-R and EPAM communities have now started to implement, is far from being a trivial task, as various technical problems (changes in the programming language, in the hardware, and so on...) conspire against it. Two measures seem imperative: (a)to develop programs to carry out simulations in batch; and (b) to interface the task simulations and the cognitive UTC model in a way that is robust against modifications of the model. Ideally, the simulations programs should be reusable for another theory. Finally, the search through the space of possible theories should be made more palatable through the use of optimisation techniques (e.g. genetic algorithms) to search the set of parameters to better fit the data (e.g. Ritter, 1991). 2.6. Averaging over Theoretical Parameters The methodology proposed here is not antithetical to group summaries or aggregates. However, instead of computing aggregate values using observed data (such as reaction times, errors, etc.), we propose first to estimate UTC-parameters over tasks, and then to compute aggregate values over these parameters. We believe that this between-subject analysis of theoretical parameters offers a method to estimate aggregate values that is more robust and theoretically more meaningful than the traditional way of aggregating data (see Table 1). --------------------------- Insert Table 1 around here --------------------------- Note that UTC-parameters need not be necessarily numeric. For example, they may be strategies or other types of knowledge. In the latter cases, summaries may take the form of probability distributions over the possible strategies or productions achieving the same goal. If one wishes to extract one single value from the distribution, one may, for example, take the parameters that occur most often for a given goal (modal strategy or production). Clearly, for some parameters, taking a summary value could be meaningless. For example, subjects may use, for the same goal, strategies or productions so idiosyncratic as to make overlap between subjects non-existent, and averaging strategies does not make sense. 2.7. Difficulties In spite of these advantages, of perhaps because of them, there are a few difficulties that this approach may face. First, it can be argued that the experiments that can be used both by subjects and by the model currently represent only a subset of human activities, typically activities that can be hosted by a computer. This is not an important problem, we think, because this subset contains quite a large number of behaviours; in addition, the advances in virtual reality and robotics may extend the range of activities that can be simulated. Second, the theory may be overfit to a single subject (or to a few subjects), and therefore not be generalisable (Spada & Pltzner, 1994). This is probably true to some extent (in particular the knowledge and the strategies used by the subject), but we believe that the fundamental parameters constraining cognition do not vary to the point that they make generalisation (within certain bounds) impossible. On the other hand, if the results of fitting a single subject are not generalisable across subjects, creating models of average data is futile for these models correspond to no natural phenomena. The within-subject danger of overfitting may be alleviated by the large number of experiments done for each subject. In addition, and perhaps most tellingly, the resulting theory can simply be tested with other subjects taken individually, with the advantage that only a subset of the parameters have to be tested. Finally, as noted above, a theory may later be tested with larger samples, where key parameters are estimated with a subset of the tasks. Hopefully, our approach will make it clear which experiments will yield the most information with respect to the key parameters of a theory, and therefore allow one to develop an optimal subset of tests. This reduction in the number of experiments will make it easier to test further subjects, and therefore show whether and how parameter vary across a given population. This, of course, is a more powerful way of studying cognition than to limit oneself to average values. Another concern is that the data may be difficult to analyse because of their density. This is certainly a cost, but it should be born in mind that this density carries more information than a low density of data, and represents a savings in time running subjects. Current technology (e.g. Soar/Protocol Analysis: Ritter & Larkins, 1994, or the ACT-R eye tracing tools) facilitates this analysis to a certain extent. It is also necessary to consider the difficulty of how to estimate the fit of the model to the data (a classical problem in computer simulation). A practical approach is to use a convergence of measures of fit, such as the amount of variance accounted for or the mean squared error explained. 3. Conclusion The approach we have outlined in this paper contrasts with the traditional hypothesis-testing, Popperian bent of psychology, and emphasises theory building and refining, almost taking an engineering stance (Meehl, 1967; Grant, 1962). It recognises the great insights historically gained by IDAeven though these analyses are criticised by the dominant, statistics-driven, empirical Anglo-Saxon tradition in psychology. It also recognises the power of using computers to help build psychological theories, as exemplified in Anderson's ACT-R and Newells Soar projects. And it aims at taking the best of these two methodologies IDA and computer modelling by combining them. 4. References Aasman, J., & Michon, J. A. (1992). Multitasking in driving. In J. A. Michon & A. Akyrek (Eds.), Soar: A cognitive architecture in perspective (p. xxx-xxx). Dordrecht, The Netherlands: Kluwer. Altmann, E. M. & John, B. E. (in press) Modeling episodic indexing of external information. Cognitive Science. Anderson, J. R., & Lebire, C. (1998). The atomic components of thought. Mahwah, NJ: LEA. Baxter, G. D., Ritter, F. E., Jones, G., & Young, R. M. (submitted). Extending user interface management systems to support cognitive models as users. Baxter, G. D., & Ritter, F. E. (1996). The Soar FAQ, http://www.psyc.nott.ac.uk/users/ritter/soar-faq.html (1.0). Nottingham: Psychology Department, U. of Nottingham. Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University Press. Bryan, W.L., & Harter, N. (1899). Studies on the telegraphic language. The acquisition of a hierarchy of habits. Psychological Review, 6, 345-375. Byrne, M. D., & Anderson, J. R. (1998). Perception and action. In J. R. Anderson & C. Lebire (Eds.), The atomic components of thought (p.xxx-xxx). Mahwah, NJ: LEA. Card, S., Moran, T., and Newell, A., (1983).The psychology of human-computer interaction. Hillsdale, NJ: LEA. Chase, W.G., & Ericsson, K.A. (1982). Skill and working memory. The psychology of Learning and Motivation , 16, p xxx. Chase, W.G., & Simon, H.A. (1973). The minds eye in chess. In W.G. Chase (Ed.) Visual information processing (p. xxx) New York: Academic Press. Chomsky, N. (1959). A review of Skinner's Verbal Behavior. Language, 35, 26-58. Cohen, J., MacWhinney, B., Flatt, M., & Provost, J. (1993). PsyScope: An interactive graphical system for designing and controlling experiments in the Psychology laboratory using Macintosh computers. Behavioral Research Methods, Instrumentation, and Computation, 25, 257-271. Cooper, R., & Shallice, T. (1995). Soar and the case for unified theories of cognition. Cognition, 55, 115-149. de Groot, A.D. (1978). Thought and choice in chess. The Hague: Mouton Publishers. (Original in Dutch, 1946). Delaney, P. F., Reder, L. M., Staszewski, J. J., & Ritter, F. E. (1998). The strategy specific nature of improvement: The power law applies by strategy within task. Psychological Science, 9, 1-8. Dukes, N. F. (1968). N=1. Psychological Bulletin, 64, 74-79. Ebbinghaus, H. (1964/1885). Memory, a contribution to experimental psychology. New York: Dover. (Original in German, 1885). Estes, W. K. (1956). The problem of inference from curves based on group data. Psychological Bulletin, 53, 134-140. Gobet, F., & Simon, H.A. (1996). Templates in chess memory: A mechanism for recalling several boards. Cognitive Psychology, 31, 1-40. Gobet, F., Richman, H., Staszewski, J., & Simon, H. A. (1997). Goals, representations, and strategies in a concept attainment task: The EPAM model. The Psychology of Learning and Motivation, 37, 265-290. Grant, D. A. (1962). Testing the null hypothesis and the stategy and tactic of investigating theoretical models. Psychological Review, 69, 54-61. Howes, A. & Young, R. M. (1996). Learning consistent, interactive, and meaningful task-action mappings: A computational model. Cognitive Science, 20, 301-356. Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century. Jacob, F. (1987). La statue intrieure. Paris : Seuil. Lewis, R. L. (1996). Interference in short-term memory: The magical number two (or three) in sentence processing. Journal of Psycholinguistic Research, 25, 93-115. Lovett, M. C., Reder, L. M., & Lebire, C. (1997). Modeling individual differences in a digit working memory task. In M. G. Shafto & P. Langley (Eds.), Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society (p. 460-465). Mahwah, NJ: LEA. Medin, D. L. & Smith, E. E. (1981). Strategies and classification learning. Journal of Experimental Psychology: Human Learning and Memory, 7, 241-253. Meehl, P. E. (1967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 103-115. Miller, C. S., & Laird, J. E. (1996). Accounting for graded performance within a discrete search framework. Cognitive Science, 20, 499-537. Miwa, K., & Simon, H. A. (1993). Production system modeling to represent individual differences: Tradeoff between simplicity and accuracy in simulation of behavior. In A. Sloman, D. Hogg, G. Humphreys, A. Ramsay, & D. Partridge (Eds.), Prospects for artificial intelligence: Proceedings of AISB'93 (p. 158-167). Amsterdam: ISO Press. Nerb, J., Krems, J., & Ritter, F. E. (1993). Rule learning and the power law: A computational model and empirical results. In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society (p. 765-770). Hillsdale, NJ: LEA. Newell, A. (1973). You can't play 20 questions with nature and win: Projective comments on the papers of this symposium. In W.G. Chase (Ed.), Visual information processing. New York: Academic Press. Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press. Newell, A. (1992). Prcis of Unified theories of cognition and responses. Behavioral and Brain Sciences, 15, 425-492. Newell, A., & Simon, H.A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall. Norman, D. A. (1991). Approaches to the study of intelligence. Artificial Intelligence, 47, 327-346. Polk, T. A. & Newell, A. (1995). Deduction as verbal reasoning. Psychological Review, 102, 533-566. Piaget, J. (1954). The construction of reality in the child. New York, NY: Basic Books. Reitman, W. (1965). Cognition and thought. New York: Wiley. Richman, H.B., Staszewski, J., & Simon, H.A. (1995). Simulation of expert memory with EPAM IV. Psychological Review, 102, 305-330. Ritter, F. E. (1991). Towards fair comparisons of connectionist algorithms through automatically generated parameter sets. In Proceedings of the Thirteenth Annual Conference of the Cognitive Science Society (p.877-881). Hillsdale, NJ: LEA. Ritter, F. E. (1995). Review of "Soar: An architecture in perspective". Philosophical Psychology, 8, 301-305. Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human-Computer Interaction, 9, 345-383. Rosenbloom, P.S., & Newell, A. (1987). Learning by chunking, a production system model of practice. In D. Klahr, P. Langley, & R. Neches (Eds.).Production system models of learning and development (p. 221-286). Cambridge: MA MIT Press. Seibel, R. (1963). Discrimination reaction time for a 1,023-alternative task. Journal of Experimental Psychology, 66, 215-226. Siegler, R. S. (1987). The perils of averaging data over strategies: An example from children's addition. Journal of Experimental Psychology, xxx 115, 250-264. Skinner, B. F. (1957). Verbal behavior. Englewood Cliffs, NJ: Prentice-Hall. Spada, H., & Pltzner, R. (1994). Multiple mental representations of information. In R. Lewis & P. Mendelsohn (Eds.), Lessons from learning (p. 1-11). North-Holland: Elsevier. Staszewski, J.J. (1990). Exceptional memory: The influence of practice and knowledge on the development of elaborative encoding strategies. In F.E. Weinert & W. Schneider (Eds.), Interactions among aptitudes, strategies, and knowledge in cognitive performance, (pp. 252-285). New York: Springer. Stefik, M. J., (eds), S. W. S., & others, a. (1993). Eight reviews of "Unified Theories of Cognition" and a response. Artificial Intelligence, 59, 261-413. Young, R. M., Ritter, F. E., & Gobet, F. (in preparation). A reply to Cooper and Shallice (1995). bones Note that it is not necessary to develop the theory on half of the data and test on the other half, a methodology often advocated in model building (e.g. De Groot, 1969), but also criticised for logical reasons (Simon, 1977). We can safely avoid the arguments around splitting data because new data keep being gathered for the same subject. Finally, considerations should be made on what set of experiments can be carried out on a subject without infringing on ethical grounds. [FER sez: frg to fill it in.]  In this paper, we will focus on symbolic cognitive architectures, but our proposal applies to non-symbolic architectures as well.  While our approach makes good use of some advantages of a rapid interaction between theory and data, in a way comparable to research done in biochemistry, it should be kept in mind that it does not inherited of all of its strengths. In particular, the experiments described by Jacob were done using a sample of petri dishes (each containing a large population of bacteria), while our approach is limited to one, or at best a few subjects. Individual Data Analysis and Unified Theories of Cognition  Individual Data Analysis and Unified Theories of Cognition       ux{| x{|}'v'f' '""## #######, Geneva .+ fr2dod ANOTKsV fr2do=cite scott, but papers are lost in stacks, carry on till thene tasks. The model will be - Interestingly, single-subject or individual data doesnt appear in the index of the     _ ` j  Yopuv %']))4855L5f:=??EI|IJLLN!N"NDOSU U WpWY1acf_hjjkllloor(r)t|tttttu9uJutu@ @ J Z    Puv<v=vwwwx=x^xjxxy+yUy[yayyyyyz-z;zC{ {N{{{{||}}.}S}~~~~Ang#dSh_??\@kLgd}VKh9Inru@@auCY02/0'(*+,-hijklmnoprsuvxy|}  !"#$'BCTUVWXYZ[\]^_`abcȿ @ @ @ H   Z  J@KP_c3W{         ` i j Žз hh#(#(#( #(ph #( @hh #(hhh hh#(h#(h8  Y {#~%e%']+-1A484445f8:y:=AyE^EI|IJN"NDORSSعΪ⯈}rgb#( #(hh #(hh #(hh#( #(hh #(hh #(#( #( #(hh#( #( #(#( #(#( #(hh#(#(h%SWpWY1\\/`ac-cIcdcfMf_hnp.qTqbt ttuMuv>vwDwx|xybyzC{W{|4|}7}~'~y *nsϾ媥 #( h#( #( #( #( #( #(#(#( #(hh #(hh #(#(#(#(h#( #(hh4xI qxs`}B/'(,-hinotuz{|}!#$%0CUpqtz~ɿɺɿɺɺɲ#(h#(h#( #( hh#( #( #( #( C refpara7  hP(hPhhhh UTC book. modelling 5 - that... Ive added p.15. This should do it ? (d) it with EPAM now .,1993 what the subject learns bying individu As noted above, , productions, he dominant, statistics-driven cdefghijklmnopqstxyz{}~ =>]^_@ @?167-200 .1-58 The P 215-281 al of Experimental Psychology, al of Experimental Psychology, D bN@}@J b$0=,ISj`m9suNW&+"@  R P obW>^_L[PADAAAAAAD bP@}@J b$0=,ISj`m9swPW&+"  R P obW&%%-'-_034(4X44::2AIMNQSjWWIX[``c>fhVq4wxy5OJxP#(#(#(#( #( #(@#( #( #(#(#( #( #( #(#(#(#(#(#(#(@ #( #(#(#( #(#(#(#(#(#(#(#(#(#(#(@#(#( #( #( #( #( #( @#( <A}uuc_LMNT S_OPQRV  &p&r:hhhHH /1A{[,-^&'dAA.|FPR@H -:LaserWriter ;ChicagoNew YorkGenevaMonaco Zapf DingbatsPalatinoTimes HelveticaCourierSymbolReference Phonetic MishawakaMishawaka Bold Aldenderfer18 Aldenderfer24ThaiSkiaOld English Text Wingdings Arial NarrowArial Book AntiquaBookman Old StyleCentury Gothic Courier NewMonotype CorsivaMonotype SortsCentury SchoolbookTimes New RomanApple Chancery Hoefler TextHoefler Text Ornaments BraggadocioBritannic Bold DesdemonaImpact Wide LatinMistralPlaybillStencil MT Extra eWorld Tight Arial Rounded MT Bold Bauhaus 93 Bookman Old Style Bold Colonna MT Footlight MT Light Kino MT Matura MT Script Capitals Vivaldi Brush Script MT SmartChessFont MS LineDraw&p Espy Sans&rEspy Sans Bold:Tekton Plus Regularh KrungthephThonburihAyuthaya ICEmbdObject4oP(oPElE<>  h9$>GWdnq.J !!!!%a%b%%'(M(X(Z....114'44447w7x77;O;P;_;`<<<<<<<<<DDO^O_ObOcOOOORRRRW=W>WGWHWdWWWXXXXZZZZbbb4b<b?bIbJc>cLcNccccklzm/m0m7m8rrrx/x2x9x<xyy*y-y.y/y3y5yyyyop&E|}NOP>=G $<!<<'C==S<k<!<<U!V"W%X%(e(<q(Y.Z.24284tx45f[7\7];^;<z{<<_=T`=Z~EO<OPaRbRcWdWWWWXXeZXfZ`gbYb~<bhbcc<ck<lim^jmfr׀rx^xhxky0yYy\ y`ybyЀy׀ykl>rmn ^{|>CREDIT-talk-UTC-methodology Fernand Gobet Fernand Gobet