Thursday, 15 October 2009

SOLO applied to programming

Over the last five years, I have been involved in the BRACELet project in which we have endeavoured to learn something about novice programmers from data gathered primarily through exams (Whalley and Lister 2009, Lister 2008). this work has involved an iterative cycle of data gathering and analysis very much like an action research project but utilising different groups of students around the world.

One of the key tools in analysing our data has been the SOLO taxonomy (Biggs and Collis 1982, Biggs 1999). Our initial use of the SOLO taxonomy was to analyse student responses to a code summarisation question (Whalley et al. 2006). That has now been repeated in a number of studies revealing a very similar pattern of results (Sheard et al. 2008, Clear et al. 2008). There have also been some discoveries that have been a little like side journeys (Lister et al. 2006, Thompson et al. 2006). The work has also been extended in an attempt to find a correlation between reading, tracing, and writing (Lopez et al. 2008). A paper accepted for publication later this year draws together all of these studies and extends the work into new areas including using SOLO to classify code writing exercises (Clear et al. 2009).

As the research has progressed, the use of the SOLO taxonomy for analysing reading responses has been revised as we have learnt more about the classification issues. A summary of the principles applied is:.

SOLO categoryDescription
PrestructuralSubstantially lacks knowledge of programming constructs or is unrelated to the question
UnistructuralProvides a description for one portion of the code (i.e. describes the if statement)
MultistructuralA line by line description is provided of all the code. Summarisation of individual statements may be included
RelationalProvides a summary of what the code does in terms of the code’s purpose. Provides a summary of the code that recognises applicability of the code segment to a wider context

This hierarchy is based on the degree of integration and abstraction from the code. At the lowest level, the student has no or little comprehension of the code. At the next level, they are looking at individual constructs. At the next, they are working with multiple lines but still not seeing the whole as a unit. It isn't until the relational response that they are seeing the whole as a unit..

The translation to writing isn't quite as simple. Clear et al. (2009) will describe an initial attempt at using SOLO for analysing code writing responses. It is based on the following principles..

SOLO categoryDescription
PrestructuralSubstantially lacks knowledge of programming constructs or is unrelated to the question
Unistructural – direct TranslationRepresents a direct translation of the specifications. The code will be in the sequence of the specifications.
Multistructural - RefinementRepresents a translation that is close to a direct translation. The code may have been reordered to make a valid solution.
Relational - EncompassingProvides a valid well structured program that removes all redundancy and has a clear logical structure. The specifications have been integrated to form a logical whole.
Extended Abstract - ExtendingUsed constructs and concepts beyond those required in the exercise to provide an improved solution

This draws on an example of translation presented by Hattie and Purdie (1998).

Since the work done for Clear et al. (2009), we have done more work to verify the SOLO classification for writing. This work applies the analysis to further questions and seeks to clarify the principles for the analysis of code writing questions. This type of replication is important to ensure that an analysis tool achieves its objectives..

In the code reading exercises, we realised that come code snippets provided a better question base for summarisation questions than others. Code that had difficult to identify coding tricks presented particular problems (Lister et al. 2006). In code writing questions can be written that require little interpretation in order to be able to perform a direct translation. Other questions require considerable interpretation before a line of code can be written. The relationship between the wording of the question and the written code becomes more critical..

There is another element that these code writing questions raise. The different solutions represent different levels of integration of the code segments to form a comprehensive whole. This idea of integration was used for assessing major programming assignments (Thompson 2004, 2007). applying these principles to smaller code segments may appear to present some additional options for presenting code variations in teaching. This is a direction of research that I am keen to pursue..


Biggs, J. B. (1999). Teaching for quality learning at University. Buckingham: Open University Press..

Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). New York: Academic Press..

Clear, T., Lister, R., Simon, Bouvier, D., Carter, P., Eckerdal, A., Jacková, J., Lopez, M., McCartney, R., Robbins, P., Seppälä, O., & Thompson, E. (2009). Naturally occurring data as research instrument: Analyzing examination responses to study the novice programmer. Inroads - The SIGCSE Bulletin..

Clear, T., Whalley, J., Lister, R., Carbone, A., Hü, M., Sheard, J., Simon, B., & Thompson, E. (2008). Reliably classifying novice programmer exam responses using the SOLO taxonomy. In S. Mann & M. Lopez (Eds.), 21st Annual conference of the National Advisory Committee on Computing Qualifications (NACCQ 2008). Auckland: NACCQ.

Hattie, J., & Purdie, N. (1998). The Solo model: Addressing fundamental measurement issues. In B. Dart & G. M. Boulton-Lewis (Eds.), Teaching and learning in higher education (pp. 145-176). Camberwell, Vic: Australian Council of Educational Research..

Lister, R. (2008). After the gold rush: toward sustainable scholarship in computing. In Simon & M. Hamilton (Eds.), Tenth Australasian Computing Education Conference (ACE 2008) (Vol. 78, pp. 3-18). Wollongong, NSW, Australia: ACS.

Lister, R., Simon, B., Thompson, E., Whalley, J., & Prasad, C. (2006). Not seeing the forest for the trees: Novice programmers and the SOLO taxonomy. In M. Goldweber & P. Salomoni (Eds.), Innovation and Technology in Computer Science Education (ITiCSE 2006) (pp. 118-122). Bolonga, Italy: Association for Computing Machinery..

Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008). Relationships between reading, tracing and writing skills in introductory programming, Proceeding of the fourth international workshop on Computing education research (pp. 101-112). Sydney, Australia: ACM..

Sheard, J., Carbone, A., Lister, R., Simon, B., Thompson, E., & Whalley, J. L. (2008). Going SOLO to assess novice programmers, Proceedings of the 13th annual conference on Innovation and technology in computer science education. Madrid, Spain: ACM..

Thompson, E. (2004). Does the sum of the parts equal the whole? In S. Mann & T. Clear (Eds.), Proceedings of the seventeenth annual conference of the National Advisory Committee on Computing Qualifications (pp. 440-445). Christchurch, New Zealand: National Advisory Committee on Computing Qualifications..

Thompson, E. (2007). Holistic assessment criteria - Applying SOLO to programming projects. In S. Mann & Simon (Eds.), Proceedings of the Ninth Australasian Computing Education Conference (ACE2007) (Vol. 66, pp. 155-162). Ballarat, Victoria, Australia: Australian Computer Society Inc..

Thompson, E., Whalley, J., Lister, R., & Simon, B. (2006). Code Classification as a Learning and Assessment Exercise for Novice Programmers. In S. Mann & N. Bridgeman (Eds.), The 19th Annual Conference of the National Advisory Committee on Computing Qualifications: Preparing for the Future — Capitalising on IT (pp. 291-298). Wellington: National Advisory Committee on Computing Qualifications..

Whalley, J., & Lister, R. (2009). The BRACELet 2009.1 (Wellington) specification. Paper presented at the Proc. 11 Australasian Computing Education Conference (ACE 2009), Wellington, NZ..

Whalley, J., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, A., & Prasard, C. (2006, 16 - 19 January). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Paper presented at the Eighth Australasian Computing Education Conference (ACE2006), Hobart, Tasmania, Australia..

No comments: