Progressive Video Assessment

A recent project has taken me down the path of designing a peer based assessment surrounding a learners uploaded videos of their displaying skills in action. All the technologies and approaches are now available at a reasonable cost to enable video based peer assessment or IMHO better titled as "progressive video assessment". The main ideas being leveraged are;

1. Learners progressing through different sets of competencies grouped into levels.
   
2. Learners are grouped into virtual cohorts, that change depending on pace, availability and competencies.
   
3. As learners complete competency sets they also provide feedback to those who have not yet completed the competencies.
4. A part of a learners own assessment is providing an amount of feedback to their peers, this follows more of a master-apprentice type model. The master is only one or two levels ahead of the apprentice.
5. Feedback is provided in private discussions triggered by an uploaded video demonstrating competencies and the learners self-assessment.
6. Competencies are described in rubrics.
   

Utilizing a Progressive Inquiry type approach provides a good foundation for it encourages an iterative process for building understanding and deepening meaning. It also acknowledges the importance of peers and mentors in the learning process.

The aim of a video based peer assessment approach is to bring people to a level of mastery within a given subject area. The subject area needs to be appropriate for video as a medium for capturing demonstrated skills. We want people who already have basic competency in an area wanting to improve through peer review and feedback from others with greater expertise.

Step 1. Learners provide evidence of subject knowledge (traditional assessment)
Step 2. Learners review rubric(s) describing competencies for level being attempted
Step 3. Learner video is recorded and upload (privacy ensured as videos are only available to reviewer cohort and can be deleted once review is complete)
Step 3.1. Uploaded video is immediately assigned a cohort of reviewers
Step 4. Learner kicks of discussion by answering a number of system assigned self-assessment questions.
Step 5. A small private cohort engages in facilitated discussions giving feedback focused by the respective rubric and learner answered self-assessment questions
Step 6. Reviewers also get credit toward their progression through the levels for providing the peer review
Step 7. For quality, reviews and videos are randomly / periodically critiqued by experts

In two coming posts I'll be describing the approach (or theory) behind the peer assessment and the technology infrastructure to support video based peer assessment.

Ubuntu Drupal and multiple Virtual Hosts

A few months back I completed a small project to enable a simple single sign-on approach to product level information access. This wasn't a replacement for OpenID or anything like that, for that would be a crazy thing to take on. What I did was to build a way to assess whether a user had subscribed to one or more information resources within a site that contains many resources. What I created leveraged cookies, an expiry system, and the restful API. This was all good.

To develop all this I used an old Dell Inspiron laptop running a UAMP (Ubuntu, Apache, MySQL, PhP) stack. The project is now running in DEV and is moving into QA so I still need my development environment. I've got another development project where I want to evaluate the quiz / question modules available within Drupal and I want to create another virtual host on the UAMP development server. This is what I did to set up another virtual host in addition to the localhost already being utilized by the restful API project;

1) added a drupal to the hosts file with the IP 127.0.0.2

127.0.0.1 localhost
127.0.0.2 drupal

2) added a new file to the /etc/apache2/available-sites folder
3) edited the new file to include the correct virtual host entry pointing toward the drupal directory

<VirtualHost 127.0.0.2>
DocumentRoot /var/www/drupal/
</VirtualHost>

4) enable the site using the a2ensite command with the new file as the parameter
5) restart apache

All good. Now in the local browser I can run an instance of drupal being hosted at http://drupal

How public education should provide ROI on my tax dollars

A month ago I was questioning the value of a University Education. Don't get me wrong, I think higher education is an awesome thing and I believe everyone should (and needs to) be a life long learner. I was just questioning the value of getting higher education at a University. I think there are better ways, particularly when you consider the cost (personal and through taxes) of attending a University. To me the better ways are through what is now available via the Internet (asynchronously 7 x 24). And considering the body of work regarding the different learning approaches (constructivism, connectivism and progressive inquiry apply very well, IMHO) and how these approaches are being incarnated on the Internet it becomes a compelling alternative to traditional higher education.
So all my questioning of the value of higher education was initiated from a few tweets with someone from the PR dept. of a local university and they asked me some great questions about where I was coming from on all this. The question that really stood out for me became;

"What I deem to be a worthy ROI on my tax funded educational investment?"

1. free access to research data - As a taxpayer I shouldn't have to pay for data or research that was created because of grants or the funding coming from tax dollars. There is great progress being made here as evidenced by much activity (a couple of examples; open access at Concordia, and the recent open access week at SFU). I look forward to all tax payer funded data and related publishing being available as open data.
   Suggestion: all grants, funded research, etc. should have openness written into their terms.
   
2. pedagogically aware professors - modern academic institutions haven't traditionally focused on the faculty having strong pedagogical skills. There is work going on here, though I do not feel it is enough. My experiences in working for the Instructional Development office of a major Canadian University provided me insights into how graduate, post-graduate and new faculty are developing pedagogical skills, but existing faculty are slow on the uptake. I do think a younger faculty population will make a big difference here, but weve got 10 to 20 years before the scale tips in the students favor. Do I believe my academically very successful daughter should attend a University in the next decade? Considering the Higher Education (HE) faculty pedagogical skills and the use of technology to support learning by HE institutions I believe she may better spend her time elsewhere... time will tell with this; it is good to see the Society for Teaching and Learning in Higher Education (STLHE) has published ethical principles in university teaching and they include a principle on pedagogical competence.
   Suggestion: all faculty should have mandatory pro-d with focus on pedagogy.
   
3. improved assessment techniques - assessment still seems like a game show to me - executing exams, writing papers, completing projects and their subsequent grading doesn't capture what a person has learned during a period. Using these "traditional" approaches as a way to assess peoples knowledge on a subject is disconnected. When I think of the explicit knowledge vs. tacit knowledge that a person acquires (needs) to be successful. And when tacit knowledge is considered a greater part of our knowing I just don't see strong evidence that traditional assessment methods capture a persons knowledge on a subject. I believe that much better assessment techniques need to be developed in determining a persons capabilities and knowledge. A greater amount of effort needs to be focused upon assessment within academia.
   Suggestion: all institutions should have or participate in a center for assessment excellence. Active research should be going on here.
   
4. transparency and reduction of institutional boundaries - it seems to me that academic institutions are internally focused and overly competitive with each other. In a time where it is apparent that our future is knowledge based and our national (global) economic success is dependent upon our "national intelligence" our tax dollars should go to the greater good instead of funding intra-institutional dysfunction. As an example, why every time I apply to a new institution should I have to pay a fee to get all my transcripts transferred to the new institution? Shouldn't a national database of my academic accomplishments be available? Wouldn't it be nice to have a publicly available transcript served up by the institutions I have attended. As we move into a time where learning is more self-directed, we should be able to cobble together educational activities (courses, workshops, groups, etc.) without having to be conscious of which institution they are being made available. Should we leverage the course materials created by all Canadian institutions in similar subject areas? How different are the learning outcomes for all the same undergraduate courses across all institutions? How much tax payer $$$ are spent in duplicating the same course materials across Canadian institutions. If I was getting good ROI, institutional boundaries would have been eliminated years ago and I would have open access to my educational accomplishments and the accomplishments of the institution and its faculty.
   Suggestion: national online registry for transcripts (funded by students/alumni requesting transcripts); Incentives for more frequent publishing of OER.
   
5. cross institutional shared services - almost every public institution has a their own registrar, network and server farm, library, bookstore, publishing abilities, website, mail server, etc... why? I do not feel this is a good return on my tax dollars. All public institutions should be using our tax dollars more wisely by administratively getting together and creating shared services in all (or at least most) of these areas. There exist many examples of all these services available online. I'd like to challenge all publicly funded institutions of learning to work together to make use of shared services. We do have examples of this in BC with BCCampus... somehow we need to provide incentive to public institutions to adopt a shared services approach.
   Suggestion: reallocate funding toward shared infrastructure and provide the resources to make it really easy for institutions to adopt a shared services approach.
   
6. increase in blended approaches (reduce physical travel) - A lot of the content that makes up a course does not need to be consumed on campus. A lecture can easily be put online (either live streamed or archive), and if discussion is required a companion chat could be available for the live streaming and an asynchronous discussion can be available for the archive. Readings, self-assessment, discussion groups, podcasts, webinars, etc. can all be facilitated online. Therefore, when students do show up on campus this should be honored with pedagogically exceptional face-to-face learning experiences all supported by the online materials. All the lecture halls could be converted into hives of learning activities. If each HE course had > 50% of their required content online, the institutions could double their capacity without having to create new physical seats / classrooms. That creates a lot of capability without a large investment. Particularly if the institution uses the shared services approach mentioned in item 5. Then what they really need is an increase in their online faculty.
   Suggestion: increase focus on international, remote and adult students need for education (7x24), recognize revenue potential. Increase number of teacher faculty to take on super-mentor role.
   
7. reasonable priced graduate tuition - inspiring minds and emptying wallets; > $48,000 to get an Ed.D; > $28,000 to get a professional level PhD; that's a lot of money! When you think about the importance of highly skilled and knowledgable people are to a knowledge based economy, shouldn't my tax dollars be going toward easing our best and brightest to complete all the higher education they can get. Therefore, increasing our nations competative advantage and increase global investment due to our highly skilled workforce. nuf said!
   Suggestion: find ways to give it ALL back! Tax credits for completed graduate studies.
   
8. greater completion rates - a 2004 study found that both graduation rates and times to completion are problematic within graduate study programs across Canada. Upon further review it would seem that completion rates are lower in undergraduate programs. As a taxpayer I would like to see greater focus on completion. The amount of resources used in getting people into higher education (HE) and the costs that go into every year they stay is a poor return when over 30% don't complete.
   Suggestion: have faculty based super-mentors ensure greater completion rates; follow up, collaborate, encourage and cajole.
   

Given my potential direct investment of > $30,000 and who knows what from taxes, do I believe my daughter should spend six years in a University environment acquiring a Masters level credential? Given the age of my daughter I've got four years to decide, and unless I see changes toward what I have written above I honestly believe putting her in such an environment may be a disservice to her. IMHO it would be better for me to use the > $30k to support her self directed pursuits, help her create an effective PLE and find her some great super-mentors.

What I also find interesting is the potential positive impact what I am requesting would have on our economy and resilience as a nation. I also believe I am not alone in my thinking...