Flipping assessment

I've been exploring the idea of flipping assessment for a while. It started with my work as a TA during my M.Ed graduate studies, also explored it during my time with WikiEducator, it was confirmed as I reflected upon my graduate studies and every community of practice since, and I designed it into a language learning portal I architected a while back. Why do we have people who have already completed the work (and possibly long since forgotten the details) assess those currently working on building their understanding. Why not have the cohort of learners most current with the subject provide the bulk of the assessment. Why not flip the assessment and have the novice learners assess those a few steps ahead on the same learning journey? I believe the cohort of novice learners (working with master learners) would be the most motivated to develop understanding and invest the most time in the assessment of another learners work. Particularly, if the assessment event is a formative event and just a part of the overall assessment.

This is what I envision for flipping assessment. This wouldn't be a complete flip, where novice learners do all of the assessment, but they would have an increased role in the assessment of the learning cohort. As I see it the elements of the diagram are as follows;

• Working from left to right a person masters a subject.
• A subject within a knowledge domain is exactly that, a complete subject that fits within a larger domain of knowledge. An example would be baking bread is a subject within the food preparation (or cooking, or baking) knowledge domain. And the subject of bread baking is large enough to be considered a subject within itself where someone would develop an expertise.
• An Interested learner is a person with an interest in the subject, who never actually studies the subject, but they will read about it and not shy away from a discussion.
• A Novice learner is a person committed to learning a subject. They actively seek out information, read and consume information about the subject. They participate in learning events (online and otherwise) to increase their understanding. They would even take a course from a traditional institution.
• A Master learner is someone nearing completion of their studies of a subject. They could be considered to have "almost" mastered the subject.
• Teaching assistants are people who are rewarded for providing mentorship to learners. This role is a part of traditional institutional learning, and plays a role in helping learners master a subject. Teaching assistants are often much closer to a subject than a professor as their learning was more recent and, given their role, they are closer to the learning cohort due to their frequency of personal interaction with the learners.
• A Seasoned learner is someone who has mastered a subject, and is not actively involved in learning the subject. They do continue to read about the subject and actively seek out new and related information. They are members of communities of practice regarding a subject and occasionally engage and/or contribute to the community. They are very current with the subject and provide feedback and guidance to other community members.
• The Professors are well, professors. They are most often people with Ph.D`s who are very active in furthering and adding to a domain of knowledge. Their focus is not always on teaching or the details of a specific subject within their chosen domain of knowledge, unless that is their area of research. I guess what I want to say is that a professor won`t be focused on the subject matter (and the learners study) of an undergraduate level course and its specific subjects. This is not to say professors can`t be an amazing resource for someone learning a new subject. A professors time should be used very wisely, but finding ways to encourage their engagement is good.
• The Experts are people who, through a number of different methods, have become experts in a subject. They continue to be experts by ongoing reading, personal research tasks, and involvement with online communities. They sometimes lurk (even engage) in online learning communities. Getting experts to contribute to assessment would be great, and finding ways to include them is time well spent.

My plan to implement flipped assessment
I want to experiment with the idea of having novice learners assess the work of master learners. Both the novice and master learners need to be active within the same learning journey, where the master learner is only a few lessons (or tasks) ahead of the novice. The idea being the novice is beginning their study of the subject and the master is almost finished.  I want the novice learners to read and understand the work of a master learner and give it an assessment. If the novice has questions, they ask them of the master learner. The master learner also has to encourage the novice learners to assess their work. Without this assessment neither the novice or the master will progress to completion. If novice learners want to team up to assess a master learners work, they can. This flipped assessment will not be the only assessment the learners will receive as a peer based summative assessment will occur when a learner has finished their study of the subject.

I plan to implement this flipped assessment in the Badge System Design challenge I am creating in P2Pu. The design of the challenge is iterative in that we will be covering the topics more than once with increasing depth as the challenge progresses. This iterative approach allows for a task earlier in the challenge to include the assessment of another learners completed works later in the challenge. Essentially, a novice learner is assessing the work of a master learner. Flipped Assessment.

If you are interested in the previous background chat discussing this idea it can be found in Google+.

formative and summative events, popcornjs, and screencasts

Another round of enhancements and the addition of summative events completes step 0 of my open badges step by step guide. I have been building a step by step guide for open badges where I have been creating a number of open educational resources. All this work (and other peoples contributions) is destined to become a part of the School of Open Badges on the P2PU. One of the main elements I have been working on is how to enhance screencasts for learning. Just watching a video screencast can be good for learning, particularly when you think about having the ability to stop and start the screencast to view something more than once. The main enhancement I have been working on is to add formative events to the screencast, so the learner can focus on important information and have immediate links to the subject being described. This allows the learner to pause, read deeper, reflect, and then continue with the screencast.

click this image to view the enhanced screencast.

Formative events:
Formative events enhance learning by providing further opportunities to learn during the overall learning experience. The formative events are immediate and occur as skills and knowledge are forming. The idea of formative events is often used with the idea of a formative assessment, that is in-lined with the learning module and the assessment become a part of the learning. Within my enhanced screencasts I have used popcorn.js to inject links, popups and learning outcomes while the screencast is running. When the learner clicks the link it is opened in a new tab or window, this allows the screencast to continue (or be paused) and the learner to further explore the concept via the linked information (this is not possible with current screencasting technology). The popups highlight important pieces of information. The learning outcomes identify during the screencast.

Summative events:
The summative event(s) are via a link once the screencast is complete. It provides a complete summary of the screencast with links to the following;

1. the youtube instance of the screencast
2. the popcorn.js enhanced version of the screencast
3. a listing of all this links presented during the screencast
4. a listing of the learning outcomes identified during the screencast
5. a reflective activity to deepen learning

The idea of the summative event is to provide something to go back to and use as an educational resource to reflect and deepen learning. Feel free to enjoy and learn from the growing repository of these enhanced screencasts made specifically to learn about and implement Mozillas Open Badges.

OER Assessment

I believe new approaches to open assessment need to be built to handle the growing global demand for education and learning. I am not alone in my thinking. I believe there is no shortage of great OER and a growing number of approaches toward open accreditation. These approaches to accreditation follow both traditional models and those more innovative. The point I am wanting to make is; there is no shortage of good open educational materials and there are some solid open accreditation services emerging. What I believe is missing is open assessment, for assessment closes the gap between studying educational materials and receiving accreditation for that study.

Inaugural ICT4D event participants.

This post is about open assessment and the approaches that I believe will work toward creating open assessment for the masses. I also think some background on myself and why I have developed my way of perceiving open assessment is important. I haven't taken the traditional route in obtaining an education; yet, in the end I have received a Bachelors of Technology from the Open Learning Agency of BC (OLA) and a Masters of Education from Memorial University of Newfoundland (MUN). But to think it was a straight line toward these credentials would be wrong. My OLA B.Tech degree was by cobbling together a couple of technology diplomas, with University credits from three different institutions. My MUN M.Ed IT was all online, even though I spent a year on campus, all my coursework was done online. Needless to say I have often taken alternate routes toward a traditional destination.

I know one of the biggest influences in my adult life was my participation at the inaugural ICT4D event at Royal Holloway in 2006. Ever since then I have been focusing a large part of my working and volunteer efforts in blending technology and education for the greater good. The paper I presented was about what I called a "critical technologist". It is really about the role someone could take in assisting to develop capacity with ICT in developing countries. The "critical technologist" is a play on Paulo Friere's Critical Pedagogy. I am strong believer in Paulo Friere's work. The three areas of work I have done within this capacity since 2006 have been;

• WikiEducator (2007 - 2010) - I will always look fondly upon my work with WikiEducator. I was a large  contributor at its early stages and wrote a few key pieces that are still in use today. To see my participation, review my profile; http://wikieducator.org/User:Prawstho. I was honoured and fortunate to be nominated to WikiEducator council, and in 2010 we parted ways. On occasion I still contribute to WikiEducator, though my focus is more in Wikiversity.
• Continuing Legal Education of BC (2008 - 2011) - I was Senior Project Manager / Lead Technical Architect for a large grant to increase access to education for the lawyers of British Columbia. We used a lot of open source software and open approaches to provide a services orientation for building learning communities; This 2011 post titled "Increasing access to education", does good job describing what we did.
• Progressive Video Assessment - I came up with this learning systems architecture in 2008 and had the opportunity to implement it in a couple of projects; AIM Language Professional and CLE Engagements Questions.

How does my educational and work experience influence how I believe OER assessment should be built? I believe it can be distilled into eight important points;

1. It should be peer-based
2. It should be scalable
3. It should be fun and enjoyable
4. It should provide many reusable instruments
5. It should play well with others
6. It should be reference-able
7. It should support open data
8. It should feed into open credentialing

I believe open assessment should be easily integrated with other approaches of assessment and credentialing. It should assist learners who are attending traditional institutions as well as alternative approaches like OERu and P2Pu. It should also support those who want to learn independently without ever stepping foot in a traditional institution (online, virtual or otherwise) or those independent learners who dont seek credentials. I really like the parallel universe approach being offered by OERu; yet, I am also concerned about the need for fees and my internal critical pedagogue is tingling. I have extended the parallel universe approach being proposed by OERu.

OERu parallel learning universe with an additional dimension.

Approaches to Open Assessment:

• In the immediate term we should build peer-assessment to utilize and promote the use of badges (see wikieducator: http://wikieducator.org/User:Prawstho or Mozilla badges for example). This will get us going quickly and then build upon this with greater automation and other assessment approaches.
• In the near term we should build rubrics to allow people to perform self-assessments and then submit portfolio for a peer-review. Peer reviewers should also be given badges for executing N peer reviews. This does not require a lot of automation, can be implemented quite quickly.
• In the close term we should build formative and summative assessment instruments that can be baked into websites, apps and tools. People should be able to edit / improve on instruments in a community kind of way.
• Beginning immediately and through the next year we should start building apps and experimenting with mass collaboration for assessment.
• In the medium term we should build a repository of guidelines, templates, assessment instruments and approaches.
• In the long term we should build a fun community around assessment and broadening peoples skills and knowledge regarding assessment and accreditation.
• As an ongoing initiative we should encourage research and centers of excellence around open assessment and its relationship with open accreditation.