Untitled Document

 

Introduction

The ARC Centre of Excellence in Creative Industries and Innovation (CCi) commissioned two jams –online collaborative events – to discover new ideas for improving outcomes in two important aspects of the Centre’s work.

The first jam – known as the CCi Mainstreaming Challenge - was held for two weeks from 8th to 23rd June 2009 and focused on findings ways to improve the mainstream impact of the Centre’s research. The event involved 112 participants from the CCi’s nodes around Australia and internationally, CCi Advisory Board members and a select group of external ‘critical friends’ (see Appendix A for a full participant list). Participants posted their own ideas, and commented and voted on the ideas of others, on a challenge question related to improving the mainstream impact of the Centre’s research. In total the jam generated 22 substantial new ideas, 103 votes and 96 comments.

The second jam – known as the CCi RHD Challenge – was held from June 8th to 24th 2009 to elicit ideas and opinions from the Centre’s Research Higher Degree (RHD) community on how to improve their professional development experience and outcomes during their student tenure. The event invited 75 RHD students to participate and was kept exclusive to this group (i.e. no involvement from supervisors or Centre management). In total the community generated 17 ideas, 26 comments and 78 votes

This report summarises the key results and insights generated from both Challenges, and is divided into three parts: Section One describes the challenges around which these two events were focused and participation rates achieved, while Section Two summarises the outcomes in terms of statistics and trends by theme, as well as the value of the tool for ‘capturing the long tail’ of collaborative processes.  Section Three concludes by presenting the top ideas by popularity and activity, as well as identifying the ideas that most polarised opinion and emergent themes.

Section One: Challenge Summaries

1.1a     Context and question for CCi Mainstreaming Challenge
The challenge context and question for the CCi Mainstreaming Challenge were developed in close consultation with Centre management and Terry Cutler, Chair of the CCi Advisory Board, and were designed to reflect the fact that there exists considerable diversity amongst Centre participants in what constitutes ‘impact’.

The primary challenge question posed by the jam was thus kept deliberately broad:

How can we increase the CCI’s mainstream impact?

The challenge question was accompanied by a short context-setting text to provide background and guidance on responding to the challenge question:

Terry Cutler, Chair of the CCI’s Advisory Board, wants to hear your ideas for how the Centre can have greater mainstream impact.

This follows from observations that collectively, the CCI has a unique set of perspectives on social, technological, institutional and cultural change, but these are not always sufficiently focused on mainstream issues in our respective fields.

What should we do to ensure that our work at the Centre has the greatest impact possible, within our own disciplines, within business and policy worlds, and in media and public perception? How can we better leverage the assets and capabilities of the CCI? Are we doing everything we can to take advantage of cross-disciplinary synergies?

Increasing the CCI's mainstream impact might involve looking to international models for inspiration, or adopting some of the practices of think tanks or the private sector. It might mean developing brand new ‘offerings’ from the Centre. Or it may mean simply getting better at celebrating what we already do. Importantly, we want to think about what kind of legacy the Centre leaves and what kind of springboard it provides for all of us involved as we move on to other endeavours in the future.

This challenge is broad in remit because we recognise that creating greater impact on mainstream issues means such different things to each of us. That’s why we’re interested to hear your ideas, and your opinions on the ideas of others. We’ve also invited a number of trusted friends and collaborators into our discussion to hear their thoughts as well.

The challenge context and question were consciously positioned as widely and openly as possible in order to stimulate a broad range of responses and minimise the risk of disenfranchising particular cohorts within the participant group.

1.1b     Context and question for CCi RHD Challenge
The RHD Challenge was structured around three key themes reflecting current areas of concern in the experience of Research Higher Degree students at the Centre. The wording of the themes was designed to be as meaningful as possible to the RHD participants in the event:

  • The CCI's interdisciplinary research agenda, including what you understand it to be and whether it has helped you in your own program of work.

  • What you need in regards to professional development, above and beyond disciplinary scholarship and direct contact with supervisor(s). This could include:
    • skills to disseminate work to different audiences
    • conferences, networking and profile-building opportunities
    • access to jobs both during and after life as a research student
    • taking advantage of formal courseware

  • Your hopes and expectations for future work – specifically the career position you hope to reach after graduation and how your degree will help you get there.

Participants were also provided with a ‘free space’ area where any other ideas not captured in the above listed themes could be posted.

1.2a     Participation in CCi Mainstreaming Challenge
Of the 112 people invited to participate in the CCi Mainstreaming Challenge, 31.25% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard ‘ideation’ events which rarely achieve participation rates of more than 20%.

Of those who did participate in the CCi Mainstreaming Challenge, just over ten percent contributed by vote only (i.e. did not post any ideas or comments). This highlights that the jam process provides two distinct ways for individuals to participate – through written contributions and through expressing opinions in the forms of votes. The CCi Mainstreaming community generated a vote-to-idea ratio that approximately matches the average of other jam events, voting at a rate of about 5 to 11 .  

Figures 1 and 2 below show that participation over the duration of the event was strongly influenced by the communications programme designed to stimulate an increase in participation rates. Votes and comments in particular spiked significantly after the mid-point email reminder sent on 15-06-09.

Figure 1 also shows that posting of ideas started high at the commencement of the event but tailed off over time – this is a very typical pattern for idea contribution held over a multi-day period. However it is worth noting that in this case the communications programme was sufficiently effective to cause a small spike in new ideas posted on 15-06-09. It is typically very difficult to achieve this spike in idea contributions in the second week of a jam.

Figure 1: Ideas and comments by date, with email reminder sent on 15-06-09

Figure 2: Votes by date, with email reminder sent on 15-06-09

1.2b     Participation in CCi RHD Challenge
Of the 75 people invited to participate in the CCi RHD Challenge, 32% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard ‘ideation’ events which rarely achieve participation rates of more than 20%.

Of those who did participate in the Challenge, more than 45% contributed by vote or comment only, with 8% contributing by comment only and 37.5% contributing by vote only. This highlights that voting was an extremely important mechanism for expressing opinions for this particular community. However the CCi RHD community generated a vote-to-idea ratio that is slightly lower than the average of other jam events, voting at a rate of about 4.5 votes per idea on average.  

Figures 3 and 4 below show that participation over the duration of this event was also strongly influenced by email reminders. Votes and comments spiked significantly after the mid-point email sent on 16-06-09, albeit with a delayed response of about 24 hours. It is also worth noting that the commenting rate in particular stayed high in the second week of this event, which may be attributed to the effectiveness of the reminder email or to the characteristic behaviours of the group involved2 .

Figure 3: Ideas and comments by date, with email reminder sent on 16-06-09

Figure 4: Votes by date, with email reminder sent on 15-06-09

Section Two: Outcomes

In total, the CCi Mainstreaming Challenge generated 22 ideas, 103 votes and 96 comments, while the RHD Challenge generated 17 ideas, 78 votes and 26 comments. These are particularly strong results considering that no face-to-face events, such as workshops, were held to augment the online jams prior to or during their execution (although results for both were presented and discussed at a research symposium held after both jams had closed).

2.1       Outcomes by theme
Figure 5 below shows the distribution of ideas, comments and votes contributed in the CCi RHD Challenge grouped by theme. (Note that this breakdown is not possible for the CCi Mainstreaming Challenge as it involved a single challenge question.)

The ‘future work opportunities and challenges’ and ‘free space’ themes clearly captured the most activity in terms of ideas, comments and votes. It is interesting to note that no ideas – and therefore no comments or votes – were posted against the theme related to the Centre’s interdisciplinary research agenda. This indicates that this particular theme may have been mismatched with the actual concerns of the RHD community.

Figure 5: RHD Challenge ideas, comments and votes by theme

As shown in Figures 6, 7 and 8 below, the dominance of the ‘free space’ or ‘miscellaneous’ theme in attracting ideas, comments and votes indicates that the challenge themes were not particularly well matched to the types of concerns and opportunities perceived by the group. This may be due to the fact that this group had previously been engaged by the Centre via surveys, and some of the categorisation of ‘issues’ identified through that mechanism may have inadvertently bled into the design of the jam process. This highlights that survey/questionnaire activities are fundamentally different to ideation initiatives, in that the former counts and categorises responses to known questions whereas the latter seeks potential solutions to problems or challenges broadly defined.

A recommendation arising from this outcome is that a single challenge (i.e. non-themed) jam design should be used unless there is strong and accurate knowledge of the topic categories that are relevant for the particular combination of challenge and participant group.

Figure 6: Ideas by theme

Figure 7: Votes by theme

Figure 8: Comments by theme

2.2       Long tail of contributions
Contributions to both Challenges conformed to a “long tail” distribution (also known as a power law distribution). This means that a small number of people were responsible for the majority of the activity in each event. This pattern of contribution is seen in many online and interactive phenomena such as Wikipedia, social networking and internal firm collaboration processes.

In the CCi Mainstreaming Challenge, the top 3 individual contributors (in terms of the number of comments and ideas submitted) contributed just under 40% of the total ideas and comments generated by the jam. This is illustrated in Figure 9 below.

Figure 9: Proportion of contributions (ideas and comments) from most to least prolific contributor in CCi Mainstreaming Challenge

It may be the case that those contributors who are relatively prolific in an online context are much less likely to contribute in traditional face-to-face settings, so that their ‘voices’ can be more clearly ‘heard’ through a web-based process such as the jam. It is only possible to demonstrate this conclusively, however, by comparison to data on face-to-face contribution rates, which are not currently known for these groups.

It is possible however to map the source of the most popular ideas onto the long tail of contributions. For the CCi Mainstreaming Challenge this mapping is shown in Figure 10, with corresponding mappings shown for the RHD Challenge in Figure 11.

Figure 10: Location in the long tail of most popular ideas - CCi Mainstreaming Challenge

Figure 11: Location in the long tail of most popular ideas - CCi RHD Challenge

The concentration of popular ideas in the long tail of the distribution shows that the jam provides an effective mechanism for capturing valuable ideas from individuals who are relatively less prolific in their contributions compared to others. For example, the second most popular idea in the CCi Mainstreaming Challenge was submitted by one of the least prolific contributors in the event.

Section Three: Ideas

This section presents the top ideas by popularity and activity, as well as identifying the ideas that most polarised opinion and emergent themes.

3.1       Top ideas by popularity
Votes cast during a jam can be negative (a thumbs-down or ‘Demote’) or positive (a thumbs-up or ‘Promote’ for the idea), resulting in an overall net score per idea. The top ideas by popularity (i.e. with the highest net score) for each Challenge are summarised in Tables 1 and 2 below.

Rank By Popularity

Net Score

Idea Description

Contributed By

4th 
(joint) 

4

National Broadband Network - NBN- Leadership

Brian Fitzgerald

4th 
(joint) 

4

Launching special issues in journals cross-disciplinary

Carsten Herrmann-Pillath

4th 
(joint) 

4

Broadband Services Advisory Councils

Trevor Barr

3rd 
(joint) 

5

Australian Culture Online

Brian Fitzgerald

3rd 
(joint) 

5

CCI consulting

Jason Potts

3rd 
(joint) 

5

Hold specialist 1 or 2 year forums funded by participants

John Howkins

2nd
(joint) 

7

Mainstreaming through the media – Media Training Workshops

Carrie Miller

2nd 
(joint) 

7

The next stage of innovation research, thinking, and policy

Stuart Cunningham

1st 

9

An increased focus on creative production processes

Phil Graham

Table 1: Top ideas in order of popularity – CCi Mainstreaming Challenge

Rank By Popularity

Net Score

Idea Description

Contributed By

4th 
(joint) 

5

Networking

Wen Wen

4th 
(joint) 

5

themes, space, and beer

Jaz Choi

4th 
(joint) 

5

Dedicated budgeting and dedicated staffing

Linda Watterson

3rd 
(joint) 

6

more hands-on experience to meet professional development needs

Henry Li

3rd 
(joint) 

6

Job Postings for Positions in the CCI network

Krystina Benson

3rd 
(joint) 

6

Let's show off at the CCI conference!

Woitek Konzal

2nd 

10

Learning to play the game from the outset

Mark Ryan

1st 

11

Who needs a Doctoral Programme?

Thomas Petzold

Table 2: Top ideas by popularity – CCi RHD Challenge

Prizes were awarded to the contributor of the most popular idea in each event – a bottle of fine wine in the case of the CCi Mainstreaming Challenge; and a $1,000 travel bursary in the case of the RHD Challenge.

3.2       Top ideas by activity
A key principle in making the most of a jam process is to recognise that the most popular ideas are not necessarily the most valuable to the organisation.
We can also rank ideas by the volume of overall activity they attracted in terms of both votes and comments. In Table 3 below, ideas are ranked by the number of votes they received, whether negative, positive or indifferent (i.e. ‘No Opinion’), as shown in the column ‘All Votes’, plus the number of comments generated, as shown in the column ‘Comments’.

Rank

Idea

All Votes
(+ and -) 

Net Score

Comments

Contributed By

1st

CCI Consulting

9

5

9

Jason Potts

2nd

An increased focus on creative production processes

11

9

7

Phil Graham

3rd

The next stage of innovation research, thinking, and policy

7

7

8

Stuart Cunningham

4th

Mainstreaming through the media – Media Training Workshops

7

7

6

Carrie Miller

5th

join CSIRO

7

1

7

Jason Potts

6th

Launching special issues in journals cross-disciplinary

8

4

5

Carsten Herrmann-Pillath

7th

Hold specialist 1 or 2 year forums funded by participants 

7

5

6

John Howkins

8th

Short Courses?

5

3

3

John Banks

9th

Australian Culture Online

5

5

4

Brian Fitzgerald

10th

NBN- Leadership

4

4

5

Brian Fitzgerald

Table 3: Top ideas in terms of activity generated – CCi Mainstreaming Challenge

3.3       Controversial ideas
The jam voting system allows participants to express three types of opinion about an idea: positive (‘Promote’), negative (‘Demote’) or indifferent (‘No Opinion’).  The total number of votes received by an idea may not correlate with its popularity if the idea receives a relatively large number of negative or indifferent votes. This is demonstrated in the discrepancy between the values in the columns ‘All Votes’ and ‘Net Score’ in Table 3 above.

Table 4 below shows the three ideas that most polarised opinion during the CCi Mainstreaming Challenge.

No of Promotes

No of Demotes

No of No Opinions

Net Score

Idea Description

Contributed By

4

3

0

1

Join CSIRO

Jason Potts

7

2

0

5

CCi Consulting

Jason Potts

6

2

0

4

Launching special issues in journals cross-disciplinary

Carsten Herrmann-Pillath

Table 4: Ideas that polarised opinion – CCi Mainstreaming Challenge

The above breakdown of activity by idea emphasises that it is important to take a holistic view of each idea – and the qualitative and quantitative responses that each has generated – rather than focusing solely on relative popularity as given by the net voting score.

3.4       Emergent Themes
As discussed, the Mainstreaming Challenge used a single question design due to the wide divergence of views known to be held on the jam topic amongst the invited participants. While the three specific themes originally chosen for the CCi RHD Challenge were successful in eliciting a high response rate and strong voting levels, the relatively dominant use of the ‘free space’ or ‘miscellaneous’ theme indicates that the three original themes were a less than perfect match for the ideas and discussion subsequently generated – although all clearly fell under the umbrella theme of ‘improving the RHD experience at CCi’.

In both cases there was a strong rationale for conducting a ‘meta-analysis’ to identify emergent themes that collectively describe all of the ideas put forward during the events, and assist with deciding on specific actions going forward. These emergent themes are based on ‘bottom-up’ analysis of the jam content itself rather than on the ‘top-down’ themes that were originally conceived for structuring the discussion (in the case of the RHD Challenge) or the wide scope created by the deliberately broad challenge question set for the Mainstreaming jam.

To demonstrate this approach, six ‘emergent themes’ from the Mainstreaming Challenge are identified:

  • NBN: ideas related to how the CCi can take a leadership role in the emerging debate on the potential role and impact of the national broadband network;
  • Web applications/repositories/tools: ideas related to the use or creation of web-based mechanisms to create or aggregate content and/or generate new content related to the fields in which CCi has specialist knowledge;
  • Consulting: ideas related to how CCi could deploy its research and expertise in an applied context to third parties in the private and government sectors;
  • Academic: ideas related to building CCi’s profile and reputation in traditional academic outlets such as journals and cross-institutional alliances;
  • House-in-order: ideas that have a clear mandate from the participant group (i.e. are popular and non-controversial) - can be considered as ‘things we should already be doing’;
  • Long-term positioning: ideas related to how the Centre and its work might be best positioned for longer term sustainability and maximum impact.

All of the 22 ideas generated by the CCi Mainstreaming Challenge can be categorised under these six emergent themes, as tabled below:

NBN:

Idea

From

Broadband Services Advisory Councils  

Trevor Barr

Leadership in mapping NBN policy framework

Brian Fitzgerald

Web applications/repositories/tools:

Idea

From

Lightweight content generation tools

Andrew Brown

Australian Culture Online

Brian Fitzgerald

Wikimedia/pedia

John Quiggin

Creative economy policypedia

Julian Thomas

Consulting:

Idea

From

CCI Consulting

Jason Potts

Specialist 1-2 year participant funded forums

John Howkins

Short courses

John Banks

Convergence Culture Consortium Model 

John Banks

Academic:

Idea

From

Building international alliances with similar institutions

Carsten Herrmann-Pillath

Standardise teaching materials/case studies for business schools

Carsten Herrmann-Pillath

Launching special issues in cross-disciplinary journals

Carsten Herrmann-Pillath

House-in-order:

Idea

From

Use CCI website to publish open access research results

Amanda Lawrence

Media training workshops

Carrie Miller

Long-term positioning:

Idea

From

Increased focus on creative production processes

Phil Graham

The next stage of innovation research, thinking and policy

Stuart Cunningham

Inter-disciplinary knowledge advancement

Rachel Parker

Creative Industries ministerial responsibility

Ellissa Nolan

Harnessing the creative arts to the innovation agenda 

Lelia Green

Join CSIRO

Jason Potts

Establish and define current impact

Jo Ann Tacchi

Emergent themes such as these allow jam sponsors to organise action planning activities around the topics that have actually arisen from the jam process rather than those that may have been erroneously perceived as being relevant by management prior to the event.

This exhaustive approach to categorising ideas – i.e. providing a ‘home’ for every proposal that is put forward - also highlights that it is crucial that all forms of response to ideas be taken into account when formulating action plans, and in particular that popularity votes as articulated through the net score per idea are balanced by the qualitative information in the discussion threads.

Finally, it is strongly recommended that the participant group be given clear information as to how future actions will be decided and implemented as a result of the jam, and that reporting back to the group on these initiatives is made a priority. Failure to do so will jeopardise the effectiveness of subsequent jam - or broader ideation - events.

 


1For example it may be tempting to infer that students (even at the RHD level) have a tendency to procrastinate, reflected in this case in the much higher voting and commenting rates observed in the second week of the event.

 


2It can be noted that other jam events have generated a much higher vote-to-idea ratio, with a recent challenge achieving over 10 votes per idea on average. Further research is needed to conclusively identify the group characteristics that correlate with intensive voting patterns, however anecdotally it can be observed that the group involved is highly internally competitive and operates within a strongly hierarchical organisation compared to the collegiate structure of the CCi groups.