Manifesto for Teaching Online – Aphorism No. 10 – “New forms of writing make assessors work harder: they remind us that assessment is an act of interpretation. ”

“To the extent that rhetoric is conceived as an art, capable of practical refinement, phronēsis, or practical wisdom, is often considered to be one of the by-products or relational ‘goods’ enhanced and cultivated through rhetorical conduct. For Aristotle, practical wisdom was one of the rhetorical constituents of ethos. But perhaps most important, this overriding intellectual virtue was also cultivated in audiences through the practice of deliberation. In fact, the methods of invention and argument, along with the vast array of commonplaces and topoi, may all be conceived as devices for the enhancement of phronēsis in speakers and audiences.”

(Thomas B. Farrell, “Phronēsis.” Encyclopedia of Rhetoric and Composition: Communication from Ancient Times to the Information Age, 1996)

“Everybody experiences far more than he understands. Yet it is experience, rather than understanding, that influences behavior, especially in collective matters of media and technology, where the individual is almost inevitably unaware of their effects upon him.” (McLuhan, 1994, ;p.318)

“Depth of understanding involves something which is more than merely a matter of deconstructive alertness; it involves a measure of interpretative charity and at least the beginnings of a wide responsiveness.” ― Stefan Collini

Assessment is an old bugbear for students, staff and institution alike. Nobody enjoys them, unless they are a masochist, or the exams are over and we have come out on top. They put us on the line and it often seems as if our very existence, our future depends upon passing, not just our statement of knowledge being assessed. This makes assessments sobering affairs, rather like looking for work after we pass them. This is the first of three aphorisms from the series which addresses assessment. I aim to keep this one brief whilst picking up some of its points in more depth in subsequent posts.

So what kind of skills are required for today, and what should we show evidence of having attained at the end of the proscribed period of study? Moving between the two worlds of [business and education], I have come to understand that there is a core set of survival skills for today’s workplace beyond subject knowledge that also prepare us for lifelong learning and active citizenship.

1) critical thinking and problem-solving;

2) collaboration across networks and leading by influence;

3) agility and adaptability;

4) initiative and entrepreneurialism;

5) effective oral and written communication;

6) accessing and analyzing information;

7) curiosity and imagination.

To what extents do assessments address these points? I mean how would one measure  ‘curiosity and imagination’? Exams for instance, and essay questionnaires can illustrate some aspects of critical thinking and problem-solving within certain parameters. But would they show collaboration and leading by influence?  But it is quite easy and very procedural, marking multiple choice questionnaires, as opposed to exploring the internal integrity of anything like an ‘original’ work, or a set of interlocking reports on how a group performed. It seems that the reason we have a capacity called bias is that, like forms and questionnaires of various hues, it reduces just what we have to contend with in the world, in administration. It prevents us being overwhelmed. That my bias will be different from yours, or my form contains question items that are different from yours,  is just in every way just as relevant as we being differentiated as the knowledge we may be ascertained to possess.

Reading well written essay work, where the grammar is good, the sentences well-formed and the vocabulary not overly difficult, lets us easily move through just what is being said or stated. Whether it makes it any more relevant or profound is another matter.

Writing in such a manner could be said to have good ‘usability’. This means that everything is nice and consistent, flows well; it can enter and leave the mind without causing friction. It’s not difficult like making sense of Joyce’s Finnegan’s Wake, or reading philosophical the philosophical passages of Heidegger or the French social theorists. Here, most mortals have to pause and think a bit, and even if it does make some sort of impressionistic sense, you must go back  and trace your steps, and confirm that is what you picked up. You have to engage in close reading. This is in a manner that most undergraduate student essay markers know well, and those assessors who are in locations where the language of instruction is a second, or even third, language.

First year essays vary considerably and even though everyone on the course has met with the stipulated entry requirements to gain admission. There are those who have been clearly tutored on essay writing [and other things like presenting an argument] at a sophisticated level in the 6th form, and others who have not.  This latter group may even spell ‘psychology’ with a ‘k’. The minute you hit this as an assessor, there is the innate  tendency to expect or actively search for further weaknesses.

There is a great deal of mystique attached to assessment because at school and in the past the establishment wished to suggest or attach some sort of scientific relevance to it. The “dependent variable” was the paper, thesis or exam, the presentation or exhibition, supposed to show that learning actually took place during the time one was attending the course. The “independent variables” represent the inputs or causes, how you came to be able to answer the question, your innate capacities, your previous experience of the topic and subject, your motivations and interests in studying it, the difficulties you face in addressing it, how you set around surpassing these difficulties, how the institution supports you or hinders you in development, did you study at home, make consultations with tutors or others, or learn from colleagues and friends and so forth.  None of these are typically addressed, expect perhaps as part of the ‘pastoral role’ of the tutor.

I think this is important and I trust there is a literature that addresses the extent to which students learn subjects independently of lectures and tutorials. I wonder also to which extent students understand the real world relevance of that which they study so as to not only make informed choices in career but to make informed criticisms and interrogations in their researches and acceptance of that material which they are exposed to.

They are often never tested to find out if you spent your time in class in a trance thinking of football, girlfriends/boyfriends, and what you were going to have for dinner that evening. This is not unreasonable is it? I have ‘dinner party with friends’ offered as an excuse for a late essay, and ‘we would love to have you at the next one’ offered as kind of bribe. I have also had “my daughter has been diagnosed with Leukaemia, and my husband has left me.” All together quite different, but real, reason for late submission.

Do people really comprehend much in lectures, or you comprehended the material more independently, in class or in the library or at home? There is a huge debate going on right now regarding the implementation of quasi-managerial accounting used by put governmental authority regarding ‘performance’. This invariably means performance at exams and tests aimed at ascertaining not only the status of the absorption of facts and ability to perform certain operations and tasks on behalf of individual students, but the quality of the teacher, school, region and even country.

Schools are pitted against each other, as are regions and school districts, and even countries are viewed as competing. In the United States they have been considerable fears raised with respect to student performance when placed against other nations such as Finland, South Korea and Singapore. That they are falling behind on global league tables such as the OECD’s PISA studies which test 15 years olds in reading, maths and science, is seen as the fault of individual schools, teachers and their unions. The emphasis has been to out in place a greater emphasis on testing, which has led opponents to venture that this only breeds a learning culture where students ‘learn to the test’ that is they learn more about the mechanics and methods of passing exams than a truly in-depth understanding of the subject at hand.

The basic model is that there are certain standards which can be objectively measured and which are not open to interpretation. Items on the questionnaire are not ambiguous and neither should be the answers. They should reflect the individuals’ absorption of key facts, and ability to perform operations and thus act as a benchmark illustrating where the student is at in terms of what is designated that they should know on completion of the course. The latter is of course ‘the learning outcomes’, a defined set of statements which unambiguously detail a range of actions, operations and attributes which should be forthright and apparent in the work produced by the student to be marked positively.

But clearly this is a case where the map is certainly not the territory. It is simply not possible in a single essay paper, or in one set of exam questions, or one presentation, to cover what was detailed in the literature or even a single source on the subject, ten or eleven lectures, let alone all that was covered in the lecture, or the actual realm of experience and practice of the subject and all its associated practices and techniques in the real world.

One can say with a certain degree of certainty that the first 13 letters of the alphabet and their correct sequence has been remembered by the student, that they can be recited verbally, heard comprehended and understood, and can be written in lower and upper case correctly. The same with basic numeracy, that 5 + 6 = 9 and not 8, and so forth. I know my ABCs far better than I know my ZYXs. The fact that I know my 987s as well as I know my 123s tells me that I know my numbers better than I know my letters.

But as the work develops from mechanical remembering of facts and basic operations and progresses to ever more sophisticated combinations and choices of approach or method, other rubrics must be used. As problems progress in complexity to ‘wicked’ status, so there is more onus to define in in more than one set of terms, causes and effects, relationship between issues integral to it and so forth. In some sense to use the computer metaphor it has to be treated as a ‘cloud’ than a discrete singular PC and hard drive.

In art and design education there may be a part of the assessment which is project-based. In these cases the student will answer a brief to create a work which uses some of the skills or techniques they have developed during their time on the course. Performance-based assessments, also called ‘authentic assessments’, are a form of testing that requires students to perform and demonstrate a task rather than write or select an answer. In a sense such assessments simulate a working environment where designers, writers, animators, film and video makers, web site and graphic designers, must respond to briefs given by managers or clients directly. Their interpretation of the brief will manifests in an articulation of its components such as “shoot an interview with several martial arts practitioners, each interview should be a ‘talking head’ with some additional footage where necessary” There will be a mark for the overall work in its effectiveness to ‘convey its message’ but also there will be a mark with respect to the basic operations involved i.e. lighting, set up of shots, panning, zooming, inclusion of special effects and so forth. There will be a relationship set up between the rudimentary elements and the overall effect of the piece taken as a whole.

Continuous assessment strategies such as the portfolio provide a way to illustrate improvement over time. Student journals [or blogs] detailing something of their thoughts, also fall in this category. It may be expected that learning as a process may have intense periods where much insights are developed is made and less intense periods where practice, consolidation and reflection may dominate. It would be useful to see and track competencies as they assimilated.

The dialectic of creativity and reality-testing has taken us far beyond other animals and can take us farther. The next step is to dump our most natural and mistaken metaphor — education as the filling of empty minds – and recognize that we learn by extrapolating, testing, modifying and recombining mental models of the world. Interestingly, a recent study from Cornell University states that “Anti-creativity bias is so subtle that people are unaware of it, which can interfere with their ability to recognize a creative idea.” In other words, our aversion to uncertainty means we find it difficult to even recognize a creative idea when we see it, focused as we are on removing the risky, uncomfortable strain on the status quo.

Consequently, new ideas are often rejected out-of-hand in favor of the tried and trusted at times when we need new ideas the most. This resistance is so strong at times that even supporting objective evidence may not help break down barriers. The study concludes, “Our results show that regardless of how open minded people are, when they feel motivated to reduce uncertainty either because they have an immediate goal of reducing uncertainty, or feel uncertain generally, this may bring negative associations with creativity to mind which result in lower evaluations of a creative idea.”

1. People with a fixed mindset believe that talents and abilities are fixed. You are rationed an amount and those are the cards you are dealt.

2. People with the growth mindset believe that talents and abilities are developed through education, work, passion, and persistence.

Memorization is downplayed in favor of weighing evidence, reasoning, and analysis. Research, writing, and effective oral communication matter far more than performance on multiple-choice tests

I wonder regarding the new forms of writing are emerging. It seems the university, in terms of its own assessments, and appointments remains focused upon publications, and publications that accord with a hit parade – tier one, tier two, tier three and so forth. there has been some criticism that like ‘ teaching to the test’  has its correlate in ‘ researching to the paper’.  I have mentioned elsewhere that skews the scope and scale of what can be, and what will be taught or researched, opposed to what could be or should be researched. While people criticise the value of the Ph.D. it is its openness to consider what could be or should be that finds value in a climate where professional academics must show and sometimes struggle to show, ‘impact’. 

It will remain that ‘facts’ must remain in the public domain, and to be facts must remain fairly stable. ‘Opinions’ are more ephemeral and while having the potential to become more ‘factual’ remain subjective and arguable. Any interpretation of scholarly work will need to form a rubric tho9rugh which to temper and critique whether learning has taken place. Consider the difficualty faced in critiquing radical work such as John Cage’s 4′33″of silence. The work to have any meaning has to be placed within a larger picture of socio-economic, technical and cultural frameworks as well as the historiography of music and its criticism. In other words it had to provoke a reaction and mythology such s the removal of the artist and performer from the piece itself. It stands in relief from the music of Motzart, Beethoven, and all other romantic classical composers, and sets the tone for further explorations into the meaning as well as the product of the ‘text’ within society and the artform. This was how it came to be interpreted and this still  related to the intentions involved in how it was created.

The question arises to just how much you read the lines and that which lies between the lines? This is the dichotomy between ‘writerly’ and ‘readerly’ texts. It also relates to McLuhan’s notion of ‘hot’ and ‘cool’ media.  Assessments in this sense not only include what is presented but what it means, how it is experienced which is difficult to articulate in a quasi-scientific sense.

“The new electric structuring and configuring of life more and more encounters the old lineal and fragmentary procedures and tools of analysis from the mechanical age. More and more we turn from the content of messages to study total effect. . . . Concern with effect rather than meaning is a basic change of our electric time, for effect involves the total situation, and not a singe’ level of information movement.” (McLuhan, 1994; p.26) [all emphases in original]

There is real danger in losing this ‘quasi-scientific’ aspect of assessment.  Working in academic environments where English is a second language, even where administratively there may be antagonism towards its hegemony as a language of instruction globally, there circulates the idea that one can ‘read between the lines’ to uncover truths that ‘ lie beyond ‘ the slaughtered language, impossible grammar and  non-existent sense. In a certain sense the exercise becomes totally ‘readerly’, completely interpretative, subjective to the point of the assessor using their own imagination to the point of totally inventing meaning and sensibility.  It is hard to consolidate hard nosed ‘rigour’ with ‘interpretation’ given that interpretation is an active process and never disinterested, and subject o educational and cultural capital, tastes, distinction and bias.  The question then becomes ‘who has done the work?’.  In 4′33″ the piece was aimed at illustrating that silence in life is a misnomer, that their is always sound. What the piece reproduces is the sound of an audience anticipating performance, and consists of the sounds of couching, shuffling and other ambient sounds including those internal sounds of body functions. In some sense it is the sound of an embodied existential reality. 

The question then remains precisely where the emphasis of interpretation is to be in assessment. Is it in some sport of appraisal of how much interpretation of the subject matter has been made by the student? And to what extent can they reproduce the techniques, knowledge, operation and rationales? This would place it firmly in the didactic or’ teacherly’ level. Conversely, in a facilitative or student manged learning perspective, a ‘ studently’ orientated position they would go so far as to lay down their own criteria or rubrics for the success or failure of their project and any interpretation of this would be with respect to how they met this and the wider context and ‘effects’ of the work within the appraiser’s wider knowledge of the field.

Since this is the first of three aphorism on the subject of assessment and in accordance with my bad habit of interpolating between the themes I am going to finish here with the promise of taking u some of these points later. Suffice to say here that the difference between ‘hot’ and ‘cool’ media may be useful in defining the nature of ‘new forms of writing’ and the possible assessment strategies for dealing with them. the table below lifts some of the

Hot media

Cool media

extends single sense in high definition

low definition (less data)

low in audience participation

high in audience participation

engenders specialization/fragmentation

engenders holistic patterns

detribalizes

tribalizes

excludes

includes

uniform, mechanical

organic

extends space

collapses space

horizontally repetitive

creates vertical associations

Photograph, radio, phonetic alphabet, print, lecture, teaching, film, books, exams, rigour

 

Cartoon, telephone, ideographic/pictographic writing, speech (orality), seminar, discussion, facilitation, final product, thesis, essay, television, comics, interpretation

 

 

Advertisements

About this entry