Approved for Computational Minor as an Additional course
Approved for Quantitative Course for an IST PhD student
TABLE OF CONTENTS
Please note, this is a live document. Changes announced in class and on the list server will be incorporated from time to time. Announcements in class and their mirror here are the definitive version.
This course examines the theory and application of human performance models as simulations. In addition to discussions and readings on the methodology of simulation, students will gather human data (both qualitative and quantitative), consult relevant published reports, and build human models for a simple simulation (game) environment (or a topic related to their thesis). A prototypical final project is to build a model of your own play in this game (model interacts with simulation, human interacts with simulation, comparison, insights), but students will be able to focus on topics such as interaction, memory, problem solving, arousal, behavioral moderators, or other related topics.
Grading in the course will be on a series of model building homeworks designed to get students ready to do a project (individually or in teams), and on the project presentation and report.
As a prerequisite, students need to be interested in programming, and have some sympathy for psychology. Graduate students interested in simulation of human behavior, such as those in IST, Psychology, Computer Science, and Industrial Engineering, should find it interesting and rewarding, and have taken this course.
To explore this new technology and its science base, this class has four components. The first, short, component examines the application of models of human performance firstly as scientific theories, but also their use in training environments, in operations research, in design, and as opponents in computer games. This section is based on reading and discussion.
The second component examines how to create such models. Students will study how to create such models using an existing cognitive and AI architecture such as Soar or ACT-R. Students will be introduced to the ideas of using published descriptions of human behavior (i.e., journal articles and handbooks), as well as how data is gathered when published materials are not available. Tools for analysing the data to help create models will also be covered. This component will typically result in a model created by the student or a team within a somewhat constrained task, dTank. A prototypical final project is to build a model of your own play in this game. PhD students will be encouraged to do projects related to their thesis. Previous projects in courses like this have led to publications 
The third component examines the current state of the art. We will be reading recent articles and new books edited by Wayne Gray on Integrated Models of Cognition or by John Anderson on ACT-R, or other current readings.
The final component examines how such models in general and the student's model in particular can be tested, validated, and thus can be believed and improved. If the dTank environment is used, a competition between models will be staged, both as models of intelligence (do they win?), and as models of humans (do they act like humans?). Thus questions related to the philosophy of science are also discussed.
This course requires an Instructor technology room. You should also be able to install the software on your home machine.
This course should change the way you think about human behaviour, providing a theory of cognition that can be extended to help predict human behaviour for use in applications such as video games and interface testing.
This course provides a balance between theory and practice, which are tightly intertwined in this area. Basic and more advanced readings will introduce the student to current thinking about facts, theories, and ways to model data. A project drawing on the different backgrounds students bring to the program, done by individually or in very small groups, will support integrating these various pieces of knowledge and applying them. The course includes working in small groups, speaking, and other non-traditional processes and outputs, but processes and outputs that researchers use in this area.
At the conclusion of this course, students will be able to:
3.1 The IST 597 Web Site. This course has an active web page that contains the syllabus, assignments, links to useful sites, and other valuable material (such as how to correctly prepare assignments, citations templates, and other academic and recreational information). We will post late-breaking information and updates to the web page. This page can currently be found at acs.ist.psu.edu/ist597, and later will be available through links from the IST home page via course listings.
3.2 The Listserv. The course will use a hand-built mailing list to post course and class information, conduct on-line discussions, and share information. You are encouraged to use your PSU account, and not a hotmail or yahoo account that cannot receive attachments.
If you send mail to me about this course, please include "ist597" in the subject, as this will help a filter bring it to my attention more quickly.
Materials on Soar (read, in order, until you know Soar)
Ritter, F. E. (2003). Soar. In L. Nadel (Ed.), Encyclopedia of cognitive science. vol. 4, 60-65. London: Nature Publishing Group. [A006.pdf]
Lehman, J. F., Laird, J. E., & Rosenbloom, P. S. (1996). A gentle introduction to Soar, an architecture for human cognition. In S. Sternberg & D. Scarborough (Eds.), Invitation to cognitive science, vol. 4 Cambridge, MA: MIT Press.
Newell, A. (1992). Précis of "Unified theories of cognition". Behavioral and Brain Sciences, 15, 425-492. Responses Introduction to Newell
The Psychological Soar Tutorial, available online, latest slides
Soar 9 (perhaps really it is Soar 10)
The Soar Video
Laird's online tutorial notes, chapter 1
Materials on ACT-R (read, in order, until you know ACT-R)
ACT-R web site (includes tutorial, system, example papers, example models)
PSU Hot4 ACT-R tutorial
Materials on the software (you will need all of these, and all are free)
Herbal [a Description of Ontologies, if you need help with ontologies, and XML]
Herbal Design . . . . . Herbal implementation
The Soar 9 distribution.
Those who like SoarceForge, can also find Soar there
You may find VisualSoar helpful, as well as lots of things in the Soar FAQ
dTank-Soar game, available locally (source code is in the jars!)
Stuff on psychology and writing [one copy per group]
(ABCS) The ABCS of HCI. Ritter, F. E, Gilmore, D., & Churchill, E. 2010. Available from Kinko's (on the corner-ish of Atherton and College, ph. 238-2679) at cost, for about $38.00. approximately 400 pages.
Publication Manual of the APA (available at the PSU Bookstore) as a guide to referencing, citing, and the formatting of papers and manuscripts in general. Each pair should have access to a copy of this, and to Strunk and White's "The elements of style."
3.4.1 The importance and applications of models (read two)
Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(4), 467-505.
Laird, J. E., & van Lent, M. (2001). Human-level AI's killer application: Interactive computer games. AI Magazine, 22(2), 15-26.
Hudlicka, E., & McNeese, M. D. (2002). User affective and belief states: Assessment and user interface adaptation. Journal of User Modeling and User Adapted Interaction, 12, 1-47.
St. Amant, R., Horton, T. E., & Ritter, F. E. (2007). Model-based evaluation of expert cell phone menu interaction. Transactions on CHI. 14(1), Article 1 (May 2007), 24 pages.
Booher, H. R., & Minninger, J. (2003). Human systems integration in Army systems acquisition. In H. R. Booher (Ed.), Handbook of human systems integration (pp. 663-698). Hoboken, NJ: John Wiley.
3.4.2 Testing and validating models (read all)
Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human-Computer Interaction, 9(3), 345-383.
Grant, D. A. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 69(1), 54-61.
Paschler on model testing
Schunn and Wallach on goodness of fit
Ritter, F. E. (2003). Comments on Grant and Roberts & Paschler, social processes in validation, part of a Symposium on Model fitting and parameter estimation, notes included in the Proceedings of the ACT-R Workshop. 129-130.
Taatgen, N., & van Rijn, H. (2010). Nice graphs, good R2, but still a poor fit? How to be more sure your model explains your data. In D. D. Salvucci & G. Gunzelmann (Eds.), Proceedings of ICCM - 2010- Tenth International Conference on Cognitive Modeling (pp. 247-252).
3.4.3 Creating and building models (read all)
Ritter, F. E. (2004). Choosing and getting started with a cognitive architecture to test and use human-machine interfaces. MMI-Interaktiv-Journal's special issue on Modeling and Simulation in Human-Machine Systems. 7. 17-37. [in English, abstract in German]
Nuxoll, A., & Laird, J. (2003). Soar Dogma. (how to write better, safer, faster Soar code.)
Nerb, J., Ritter, F. E., & Langley, P. (2007). Rules of order: Process models of human learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to learn: How the sequences of topics affect learning. New York, NY: Oxford.
Ritter, F. E., Lehtinen, E., & Nerb, J. (2007). Putting things in order: Collecting and analysing data on learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to learn: How the sequences of topics affect learning. New York, NY: Oxford
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press. Their appendix on collecting protocols. Their 1980 paper as Precis of book [Read appendix, rest is optional]
The Soar Frequently Asked Questions list (acs.ist.psu.edu/soar-faq) and the ACT-R FAQ (acs.ist.psu.edu/actr-faq).
3.4.4 Current topics in modeling
Selections from Wayne Gray's book on Integrated Models of Cognitive Systems will be made available as preprints. Or from John Anderson's book, How can the mind exist in the physical universe? Or from the Cognitive Science or BRIMS or ICCM conferences.
Paik, J., Kim, J. W., Ritter, F. E., Morgan, J. H., Haynes, S. R., & Cohen, M. A. (2010). Building large learning models with Herbal. In D. D. Salvucci & G. Gunzelmann (Eds.), Proceedings of ICCM - 2010- Tenth International Conference on Cognitive Modeling (pp. 187-191).
How to write an abstract by Mary-Claire Van Leunen
Writing mistakes that I hate
Copy editing marks
Pew, R. W., & Mavor, A. S. (Eds.). (1998). Modeling human and organizational behavior: Application to military simulations. Washington, DC: National Academy Press. nap.edu/catalog/6173.html (click on the cover of the book on the left to read it online.)
Ritter, F. E., Shadbolt, N. R., Elliman, D., Young, R. M., Gobet, F., & Baxter, G. D. (2003). Techniques for modeling human performance in synthetic environments: A supplementary review. Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center (HSIAC), formerly known as the Crew System Ergonomics Information Analysis Center (CSERIAC). (click on the cover PDF file to read it online, or on Table of contents to get individual chapters.)
Common Lisp the Language online manual
Practical Common Lisp, Peter Seibel; Apress, 2005, ISBN 1590592395, and web site
On Lisp, by Paul Graham. Out of print.
Lisp in a box
You earn your grade but it will be assigned by me. The criteria for each assignment will be discussed in detail, as will the grading scheme. Each written assignment will be evaluated on how well it addresses the questions posed, the clarity of thinking, the organization and presentation of the material, the quality of writing, and its timeliness.
Your grade will be based on 100 possible points. You earn points with each assignment (see below). As a maximum scale (i.e., cutoffs may be lowered): A: 100-74, A-: 73-70, B+ 69- 67, B: 66- 64, B-: 63- 60, C+: 59- 57, C: 56- 50, D: 49- 40, F: 39- 0. (The cutoffs for each grade is the lower number, without rounding.)
Your learning will be assessed in several ways.
Please consult the schedule to see when papers/ assignments are due
and exams scheduled. You will receive more written instructions for
each assignment well in advance of the due date. Here is a brief
summary of each:
You will do a variety of labs/homeworks with the modeling language. Each lab writeup is nominally 5-10 points, 66 points total including an extra 5 extra credit points and the initial 1 point project writeup. 60 points will be taken to be the maximum lab grade (i.e., you can miss 6 points and get a perfect score). This score may be modified/moderated/adjusted by self and team evaluations.
Discuss a book, book section, or journal article in class.
14 December 2010
The meetings will be divided in 3 parts. The first hour will be lecture/discussion about modeling topic. The second hour is a discussion about general readings, and the third hour is spent working with Soar (or ACT-R) in a lab setting.
|2||30-Aug-10||Applications||two application papers, Ritter & Shadbolt||Herbal|
|3||13-Sep-10||Herbal and Soar and ACT-R||Architecture papers||PST|
|5||27-Sep-10||Model development||Laird & Nuxoll, Herbal, Ritter04, ritter&nerb07|
|6||4-Oct-10||Testing||Ericsson & Simon 83||Herbal|
|7||11-Oct-10||Testing||Ritter & Larkin/Grant||vacuum|
|8||18-Oct-10||Data gathering||ABCS ch.5/Rui/dTank||dTank|
|14||6-Dec-10||Future directions||Ritter & Shadbolt||slack|
Model the effect of your learning, starting with a naive model, and then a less naive model.
Create a dTank model that learns.
Model the Seibel task.
Model how you learn, starting with a naive model, and then with the learning component turned on, show that the model learns like you do (considerably harder).
Extend the dTank interface in some interesting way, and show how it is easier for you and for the model.
Set up two opponents for people to train against. Show how one is a better sparing partner, either for some of your friends, or for your model to learn from.
Example project reports: Sickel, 2007 Walsh, 2007 Paik, 2007