In this session, there was a call for converging language and approaches to design for game designers and instructional designers. Pure game (entertainment only) designers find adding the instruction/assessment pieces to games challenging and pure instructional designers find challenges in making content fun.

Dr. Kirkley reviewed several projects his company has developed. In all, the challenges of managing stakeholders (who want to have a say in the project’s direction) as well as the external vendors/developers is greater than the challenges of actually making the game.

One example he discussed was the Virtual Astronaut project which is a 10-year project and they have completed the first year (it’ll be multiple modules added to the core game over time).

It is designed for middle school students and is based on problem-based learning using a first person explorer of STEM curricula. They collaborated with Virtual Heroes on this project.

What I took away from this presentation are the following:

1. The struggles we have right now for simple instructional multimedia are likely to only increase with games (bring it on!). One reason is that funding (especially iterative funding) can force problematic processes. We’ve already seen this one…
2. A well-defined process (including detailed game design documents) are essential once the project officially begins. See Gamasutra’s article on this topic.
3. Authentic assessment within the game is the goal. If we can be smart about assessment up front, we will have a good chance of being successful.
4. The design team needs to articulate their own theories of learning and ideas about what makes games fun — at the outset — in order to minimize conflict later and better communicate from the start.

I couldn’t help but think of the book Persuasive Technology which is really at the core of all of these gaming sessions. I’m a bit surprised that no one has mentioned it or cited it…?

In any case, her site www.healthgamesresearch.org will soon send out a RFP (January 2009) that we should check out.

The next paper was given by John Richardson, a college senior and video game designer/researcher who has a form of cerebral palsy. He made a strong case for the need to ensure all of our games are accessible for people with disabilities: it serves as a means for empowerment — for gaining control over one’s body. This is complex and challenging because a disability for one person may change over time (congenital disabilities) and the discussion about disabilities tends to focus on inputs rather than a standard platform.

Is there an organization dedicated to accessibility in games? We need to do more testing; seek out feedback from our current designs…

The last presenter was Moses Silbiger whose paper and research focused on using games to help people move through the human development/self-actualization process. (I didn’t really understand this one…?)

Wow! How fun does this game look? Made by two former students (see 2D Boy) in Carnegie Mellon’s Entertainment Center — on a shoestring budget with smart marketing.  I’ll buy it!

Drew Davidson, the Director of the ETC walked us through parts of this game showing us how the attention to detail in this physics puzzle game including graphic style, sound effects, and smart, satiric writing all combine to make a great gameplay experience.

The Tower of Goo came out of the Experimental Gameplay Project in which each game must be made in less than 7 days by 1 person, and show off something never seen before. Another game from this project is Crayon Physics.

The World of Goo should be available on Wii Ware and PC on 10.13.08. It is available for pre-order at Amazon.

Selling is the hardest part of game development. Hilleman’s advice: Begin by figuing out how to sell what you’re making. Be clever! The better the narrative (about what you’re making) and the more provacative the content, the easier it is to sell. For those of us providing fee-based development services, this is useful advice…

Developing creative leaders is essential to success. At EA they have developed internal education programs designed to prepare employees to take on greater leadership roles — if they show the aptitude.  Programs are selective, and people can fail. But overall, they have proven to be a resounding success to the company. Here are some of the first programs Hilleman developed:

1. The Gong Show – a 2-day rapid-prototyping program where the team composes a game, the financial model, marketing strategy, and pitch. It not only brings to light new ideas and solidifies team collaboration, but also gives team members the confidence that game development doesn’t have to take years…
2. Seeing is Believing – a workshop focused on managing Art Directors. Too often Executive Producers instruct the Art Director (i.e., “use this image and turn it blue”) rather than giving Art Directors the creative freedom and support they need to make the game successful.
3. Roaring Silence – a workshop focused on game audio and audio design
4. Listening Out Loud – Listening to audiences and opinion leaders
5. It’s All Nuts & Bolts – Game design skills workshop targeted to non-programmers. In these sessions, teams use Lego Mindstorms robots to complete challenging tasks

What qualities does EA look for in candidates? For Designers, they look for people who like people. Empathy and respect for the player is key. For Producers, they look for selflessness; the ability to amplify the efforts of others.

Parting comment: Flash is the most ubiquitous game platform. As of the end of September 2008, there were over 1 BILLION Flash installations. Yowza.

In thinking about ways that we can evaluate the effectiveness of learning via our multimedia course redesigns, the following comes to mind:

1. Web analytics is an excellent way to unobtrusively learn more about our students’ learning that is particularly suited to the medium.
2. Analytics can tell us what and how: what students are doing online (while logged into the content) and how they are using that content. We can “see” without actually observing them while they click.
3. Because our work is for required courses and students have an incentive to master the content in order to pass the course as well as end-of-program qualifying/certifying exams, “intent” is not an issue – as is currently discussed in the analytics literature. And “conversion rates” are more about achievement of learning outcomes on short and longer term exam/program outcomes – not on short-term purchasing decisions.
4. We can use the Trinity framework to assess our multimedia course designs and possibly (I hope) student learning outcomes.

Other ideas:

• If we can provide students with the ability to monitor their learning — through analytics reporting –   they may be more open to the idea of pre- and post- “quiz” assessments. In order to be credible, this research must have pre/post measurements as baseline evidence of learning.
• Qualitative data is essential to complement analytics. We need to determine the best strategy for collecting students’ reactions to the multimedia as well as their own assessments of their learning. Would prefer to capture that feedback anonymously…and use in aggregate.

In summary, clickstream, time-on-task and other data gleaned from logs will tell us how students are using the multimedia (which we need to know). Linking behavior to assessment scores (learning outcomes) will help us understand correlations between behavior and achievement (maybe).  Connecting behavior, outcomes, and students’ own assessments of the impact of multimedia on their learning can help us develop “actionable insights” to change/improve the media and educational impact for future students.

Next step to figure out: how to design reports for students and reports for instructors — with the aim of helping them target studying/improvement in master and changes/improvements in classroom instruction, respectively.

The clickstream is one <easy> way to implement an imperfect method of assessment that aligns with the technology used — but it seems that the true power comes from:

1. the meaning individuals (faculty) or groups (faculty, students, and support staff) attribute to the data and the action that results, and
2. the ability to display trends in behavior over time

It seems that we are on a useful path but would be wise to clearly articulate what data is tracked, why, and how it is used. Data to help students reflect on their learning (metacognition) is one of the best immediate uses of the data.

In the end, it’s about trust—-just like any instructor/student interaction.

We need to conduct experimental research in Fall 2008 classes and try to identify and publish results and best practices associated with this methodology — as pertains to education. Bring on the IRB (institutional review board) paperwork…

Caveats:

• Students log in to the module outside normal class time with their unique ID
• The module supports and enhances in-class interactions between students and with the instructor
• Only students’ behavior inside the instructional module is tracked

Why use clickstream data? What are the drivers?

• Accountability (accreditation; institutional accountability efforts)
• Affordances of the technology (easy to implement, unobtrusive)
• Research (scholarship of teaching and learning)

What factors raise concerns?

• Privacy issues. Do students know that their behavior online is tracked (similar to Blackboard page tracking)?
• The overall weight given to clickstream data as an assessment form
• How the resulting data will be used (high-stakes decision-making regarding individual students v. assessing overall class progress at a point in time)

What opportunities are presented?

• Improving individual and group learning outcomes by immediately identifying misconceptions and problems, and addressing them in class
• Modifying in-class instruction and activities to meet individual and class needs at the point in time in which they arise
• Discovering how students actually use interactive content (research)
• Improving the instructional impact of interactive content & online learning environments based on students’ behavior, feedback, and other assessments
• Mapping content, activities, and assessments into a connected whole rather than disparate parts
• Co-opting a method typically used by advertising/marketing to predict who will buy something and instead using it to better understand learners’ behavior and potentially, predict how people with different learning styles will use online environments (research)
• Providing students’ own clickstream data to help them reflect on their learning, progress, and strategies

Questions:

• Can a student turn off tracking or is it “always on”? Can a student choose to have his or her tracking be anonymous?
• How long is data stored? Where? What trends can reasonably be deduced over time?
• What is the best that tracking can tell us about learning? Where are the boundaries and what are the limits?

Next steps:

• Reports (sense-making from raw data to inform instructors, students, and designers)
• Visualization (graphs, charts, the elusive “dashboard” of learning outcomes)

Resources:

• Wikipedia: Clickstream

• Connectivism
• George Siemens‘ blog & his ELI slideshow
• Henry Jenkins‘ blog
• MacArthur Series on Digital Media & Learning (emerging research)

Complexity, education, networks, participation & learning ecologies — loved it!

We also had the opportunity to meet Lance Speelmon, IU’s Manager Online Development / Sakai Release Manager. He was gracious enough and crazy enough to answer our 400 Sakai/LMS questions.

And the Penn State contingency arrived in full force. Don’t ask about “group hugs.”

Today the conference officially begins…more later.

• NMC Pageflakes