The Cathedral and Assignment Zero, or what Eric Raymond Has to Teach Us About Crowdsourcing

It's been almost ten years since Eric Raymond presented his paper, "The Cathedral and the Bazaar" to the 1997 Linux Kongress. In that remarkable work Raymond compares two methods of software development. The "cathedral" approach requires a program to be "carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time." To contrast this he "anatomizes" the approach taken by Linus Torvalds in developing the Linux OS:

Linus Torvalds's style of development - release early and often, delegate everything you can, be open to the point of promiscuity - came as a surprise. No quiet, reverent cathedral-building here - rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches ... out of which a coherent and stable system could seemingly emerge only by a succession of miracles.

Raymond makes an astonishingly lucid and persuasive case for open source's raw, hurly-burly horsepower, which "not only did not fly apart in confusion but seemed to go from strength to strength at a speed barely imaginable to the cathedral builders."

The Cathedral and the Bazaar was required reading for any aspiring (or in my case, accidental) technology journalist back in the late '90s, and it's surely old hat for some of my readers. Yesterday I re-discovered it. I'm currently researching the third chapter of my book, in which I'll argue that crowdsourcing would never have emerged without the methodology, philosophy and digital architecture of open source software. To this end I re-read "Cathedral and the Bazaar," and was amazed by its prescience, and the degree to which it anticipated the challenges of crowdsourcing in general and Assignment Zero in particular.

In recent years the 10,000-odd words in Raymond's essay have been been boiled down to a single catchphrase: "Given enough eyeballs all bugs are shallow," which is to say, a diverse and large enough network can solve nearly any problem. It's the centerpiece of  Raymond's essay—what he calls "Linus' Law—and with good reason. It explains why a crowd can be wiser than the smartest individual within it in a pithy six words.

The problem is that like a lot of catchphrases, "All bugs are shallow" has entered the realm of the shibboleth—a way to flash one's Web 2.0 credentials without displaying any actual understanding of the underlying concept. I worry that the crowdsourcing community—which as far as I can tell does not contain many programmers—is largely unaware of all the other valuable lessons Raymond has to teach us. Raymond proposes 19 lessons in his paper. Most of them apply equally to all of us—Pros and Ams alike—at AZ. Below I highlight the most salient lessons, and how they apply to our little open source journalism project:

Every Good Work of Software Starts By Scratching a Developer's Personal Itch

Replace "Work of Software" with "Crowdsourcing Project" and you've got the reasoning behind Assignment Zero. Why did we decide to focus on crowdsourcing? Because we had an itch: We wanted to know more about  the phenomenon. But this lesson applies to our contributors as well. The greatest asset citizen journalists possess is passion. Find your itch on our Assignment Desk and scratch it.

Plan to Throw One Away. You Will Anyhow.

If you read yesterday's post by AZ Project Manager Steve Fox, you're already aware that Assignment Zero is—get this—a work in progress. But then, that's always been the plan.  (See "Release Early, Release Often below). We launched AZ knowing we'd probably have to scrap the first iteration to make way for the second iteration. "There's the way we want things," Steve told me last night. "And there's reality." How will we reconcile the two? "If people are dissatisfied, then the best way to make them satisfied is to bring them into the process of helping fixing it." Well put, Steve. And thanks for the segue to the next lesson:

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Whence springs "Given enough eyeballs, all bugs are shallow." This has implications for you, our users and contributors. Assignment Zero isn't our project — It's your project, and it's important to recognize that, well, with great power comes great responsibility. We don't need peanut gallery criticisms, we need constructive advice, hands-on help and most of all, a little bit of patience and the understanding that this is how open source projects develop, through trial and error and a great deal of collaboration from everyone concerned. For our part the implications are obvious: We should welcome participation from as wide—and crucially, as diverse—a group of contributors as possible, in order to leverage the infinite range of experience and expertise out there.

 Release early. Release often. This is what we learned this week. In Steve's words, we need to let everyone see each other's work, not hide it away until the day of the big release. In open source terms this means one doesn't try to write a perfect bit of code before letting ones fellow programmers start hacking away at it. For AZ, it means we should publish the results of our reporting and writing as we compile it. This allows our contributors to avoid conducting redundant research, but even more importantly, it allows for cross-fertilization of ideas.

Good Programmers Know What to Write. Great Ones Know What to Rewrite (And Reuse).

As is obvious now, I'm arguing that Assignment Zero has more in common with an open source project than it does a newspaper or even conventional news site. We're not writing poetry, we're writing mash-up journalism. This doesn't mean we have a license to plagiarize. On the contrary, as with open source software, proper attribution is even more important because it takes the place of more liquid currencies like, you know, actual money. This does mean all of us should be casting the broadest net possible and incorporating good ideas wherever we find them.

Too Much Collaboration Can Be a Bad Thing.

Okay, you won't find this one word-for-word in Raymond's essay. I extrapolated it from his discussion of what sociologists once called the "Delphi Effect" and what we now call the Wisdom of the Crowds. To wit,  in Raymond's words: "The average opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than that of a single randomly-chosen one of the observers." All well and good, but what's often misunderstood about the wisdom of the crowds is that it's the wisdom derives from a crowd of individuals thinking and working independently, not from a crowd thinking as a single unit, which leads to mobocracy. (See the jelly bean jar example in my post on the subject from a few days ago.)

What's this mean for AZ? Well, if all we wanted was the crowd to come to a consensus on crowdsourcing, we would have stopped after building our forums (we call it The Exchange.) But we have much more ambitious goals — We want you out there digging up research independently. Confer, by all means, but also be careful to come to your own conclusions. The wealth of the network lies in diversity and discourse, not from sermonizing amongst the choir. Besides, time on the forum is time you could be reporting!

It's been a long post, so congratulations to those of you who made it through. I hope you learned as much reading it as I did writing it. You'll learn much more by reading Raymond's original essay, which I love so much I'm going to link to it again.

Comments

[Alan]

Interesting post Jeff. Whilst looking at his work a while back I started to read some of his essays and came to the conclusion that his genius, clarity and insight must surely spring from a sense of purpose that is uniquely individualized. His point of view places him well outside of any even remotely traditional one. A man after my own heart! Alan.
http://www.catb.org/~esr/writings/

[Shazz]

Brings to mind the corporate vs. skunkworks teams in Soul of New Machine - they pre-dated opensource of course, but a similar contrast. Jeff, love the last point summary in particular, harnessing the power and intelligence of the individual to ultimately benefit and guide the group. Sounds democratic! :D

[Daren C. Brabham]

We need to be careful about the democratic veneer of crowdsourcing as it exists right now. Crowdsourcing is certainly democratic in the sense that everyone's voice gets a chance to be heard in the creative/problem solving process. However, it's also 1) extremely Darwinistic, 2) there are technology access issues that are barriers to true democratic engagement (the "digital divide"), and 3) businesses are making a ton of money off the backs of the crowd, paying them far less than the labor is worth in the market today. None of this is what critical theorists would call democratic. As crowdsourcing goes forth and we begin to refine a definition/model/best practices, we should also be very aware of ethics and purpose. What good is crowdsourcing if the model will be completely co-opted by big business and people will be fired left and right so that the crowd can take a stab at doing the same work for less pay? And what good is a diversity of ideas if the crowd is not diverse? We should begin to posit ways for crowdsourcing to benefit the REAL public good: health, environment, peace, safety. It cannot remain solely a business venture. Also, we should develop some ethical standards for the model as it evolves. I suppose there is always crowdslapping to counteract when injustice has been done, but businesses who employ the model should be aware of how they craft their proposal will affect and potentially marginalize people.

[LukePDQ]

There's no easy way to say this. However, as one of your regular original readers, I am puzzled how this fine crowdsourcing project has metamorphosed into an AZ project with a current $450,000 budget (looking for $1.5 million)to pay a large team of salaried professionals to plan and execute Assignment Zero; which apparently wants hundreds of amateurs to do the grunt work in providing free grist for the pro's to chew over and declare what?
That crowdsourcing adds speed, range, depth and value to published reportage?
Quelle surprise!

Come on, guys! How hard is it to posit that a select team of salaried pro's can moderate, adjudicate and polish the grunt work of the masses (am's) to produce a fine body of professionally edited published work.

I really don't want to sound this negative, because crowdsourcing is a good model; so please tell me I've missed the point somewhere along the line.

I say this as one who contributes and benefits from open source software; so citizen journalism as a win/win is a given for me - if the citizen journo gets fee payments back as well.

Now, if the AZ project was to define how the quality mix of pro's and am's could produce hot news and quality work in a sustainable business model that rewarded the citizen journo's as well. That would be a good outcome.

[alan]

Long time no hear Shazz and Daren, hope all is well.
What’s the Darwinist angle Daren? Alan

[Daren C. Brabham]

All is indeed well, it's just busy on my end.

I view crowdsourcing as very Darwinist at this point because it really is survival of the fittest. The "best" ideas find their way to the top, the "bad" ideas are weeded out, and only the strong ones are mass produced and turn a profit for the crowdsourcing company. Applying Darwinistic biological principles to the social realm (Social Darwinism) means that we begin to look at human traits, successes, and behaviors and reason that societal norms got to where they are through some sort of natural selection. Critics of Social Darwinism cite eugenics and "designer babies" as the end result of this line of thinking.

I see crowdsourcing as a model where ideas are thrown into a pot and the "best" ones rise to the top, but I worry whether those ideas are "good" on their own or whether we have adjusted to a white, male, heterosexual, Protestant, etc. "normal" way of viewing the world, and our collective judgment as the crowd will always tend to value the idea that is inherently more conservative to those ends than more creative ideas or ideas that do not fit into the WASP framework.

In other words, as a critical theorist, I am always questioning power dynamics, and always skeptical of models that rely on survival of the fittest rather than on considerations for diverse perspectives. I've written a post on diversity in the crowd which may or may not end up posted here on the blog soon. In it I posit three considerations for diversity and their value in producing a good idea from the crowd...but ultimately I wonder how truly diverse the crowd is to begin with, given issues of the "digital divide," access to education, and so on.

I am cautious of market economics, generally. Hell, I'm really a socialist at heart. I just think that when you turn human creativity into a survival-of-the-fittest game without thinking how and why certain ideas become privileged over others--possibly due to cultural difference (race, class, gender, sexuality, etc.)--you wind up reinstituting the same old power dynamic and the same old privileges in the dominant class.

Many states have instituted ways to provide incentive for diverse perspectives in business contexts, particularly because all the big firms would always win the bid. In architecture firms, for instance, historically underutilized businesses (HUBs, which include minority-owned businesses and tiny firms) are given a first nod in a bid for a project because the state mandates that the firm issuing the call for proposals must give HUBs a fair shot and document that fact. This is like opening a door to a firm that would have otherwise never been able to get their idea into the mix. It's analogous to the affirmative action debate: either you think we should be colorblind and look only at talent in college admissions processes or you think we should take race into consideration. The former stance--pure in its outcome-based assessment--ignores the macro-level reasons why people of color generally do worse on exams: lower test scores from worse schooling due to impoverished economics and the inability to retain good teachers. The latter stance, admittedly, can at times donwplay the importance of talent and intelligence and focus too much on identity politics. Still, though, it is an effort to address the macro-level cultural problems we are often too unwilling to notice and address.

The question for crowdsourcing is this: Given that it is now basically a market based on talent and not diverse identity, what is being lost? How much creativity is being shut out in favor of an idea that seems familiar and comfortable to the majority? This is especially problematic when it's a board (of upperclass, white, male businessmen) who select winners from the crowd rather than the crowd selecting them. How should we intervene, if at all, in the crowdsourcing process to make sure ideas get equal footing?

[Shazz]

Hey Daren and Alan!
Daren, quick question: would you say the Darwinist effects and oppressive power structures, which you attribute above to CS, are significantly different from standard capitalist big-business practices?

In response to your question: "How much creativity is being shut out in favor of an idea that seems familiar and comfortable to the majority?" ... I think the "maintained individuality" aspect of tapping the individual intelligence of crowd members, as described in the previous post by Jeff, is an important and essential way to avoid this groupthink comfort factor.

cheers!

[Daren C. Brabham]

Shazz - I don't think big business is all that different. Capitalism relies on a Darwinistic mentality to a large degree. I think that's why crowdsourcing is so appealing to for-profits right now. I just hope we can shift crowdsourcing into non-profit purposes soon.

I also think your comment that maintained individuality will prevent that kind of oppressive groupthink is right from one perspective. But from another perspective you could argue--as some cultural theorists have--that we are so plugged into mainstream ideas and so servile to dominant interests on a daily basis that even as individuals, when we think we're being really independent in our thinking, we end up replicating previous ideas and behaviors and policies. I see it as analogous to internalized racism/homophobia/sexism/etc.: if you ask a bunch of little kids to write a story about a doctor, for instance, odds are their doctor character will be a white male, no matter if the kid was white or black. Since we are immersed in a culture and media environment that stresses that only white males are doctors, we instinctively think this is the norm and replicate that norm when we have the chance to do otherwise. My question, then, is given far-flung, independent individuals asserting their own "new" creative solution to a problem, will they really put forth the creativity and out-of-the-box thinking we think they will? And when the crowd assesses what is "good" and "bad," is their measuring stick the same hegemonic old stick they grew up with? That's my concern, I think: we cannot blindly assume that this is a wholly democratic process in its purest sense.

I like this discussion, though--let's keep it going.

[Jeff Howe]

Now THIS is what comment boards are for. Daren, Alan, Shazz, Luke, there are so many interesting points here I don't know where to start. I'm really beginning to feel like the debate's beginning to sharpen. I'm just going to jump in:
@Daren: I'll start with your first comment. Your 2nd point is indisputable. Not only do we have a digital divide, but a cultural one as well. The crowd is self-selecting, and any consensus reached (even without information cascades playing a role) is representative of that fairly narrow demographic (fairly tech savvy; substantial leisure time; presumably well-educated etc.) I have some problems with 1 and 3. It's reductivist to say companies are making their money "off the backs" of the crowd. In the most oft-cited cases it's a highly symbiotic arrangement. It's difficult to equate the flickr and de.li.cio.us user with, say, a 19th Century Russian serf (or 20th Century Columbian Narco-peasant, take your pick). Even in a case like Amazon, in which the crowd is conducting labor that would have been the province of an employee only a few short years ago, it's hard to fault employer. If people are willing to trade tangible compensation for reputation points, the satisfaction of having a forum for their opinions, or what have you, then that changes the market for this particular commodity. (I happen to traffic in this particular commodity, so I don't come to this conclusion lightly.) I generally believe that the market (I'm not a socialist) will serve our (as explained above, I'll admit that we're talking about a limited first-person plural) best interests. If a company seeks to fulfill labor without offering ample rewards, they won't find any takers.

This isn't to say I'm not somewhat aghast at the situation. Why are people willing to transcribe podcasts for less than minimum wage? Or help Google label images? I just don't think it does any good to kvetch about it. Observe, adapt, survive. Stock photos were once a scarce resource. Digital photography came along, and now they're not.

But that's also not to say that I don't encourage a robust ethical critique. If bloggers replace trained journalists, should we as a society collectively pay a tax to support the continuation of investigative journalism? Not every public good has a commensurate market (I'm not a full-fledged libertarian either.) Not only do I not have all the answers to these questions, I'm just beginning to ferret out the questions. Props to you all for doing much of the heavy lifting in this regard. I regard Social Darwinism as dangerous as both practice and critique, so I'll save that for below.

@Luke: Okay, I will tell you: you missed the point. First, Assignment Zero is only the first beta run for NewAssignment.Net, for which our budget is closer to $50,000. The $450,000 is what's been raised so far, and it'll go to pay for all of the assignments NewAssignment takes on in the coming months and years (with another $1 million raised if Jay Rosen, the proprietor and founder, is lucky.) It's a non-profit, and everyone's working on below-market rates, including our generous Web designers. But even this is besides the point: I forwarded your comment on to Jay, and he points out that this isn't supposed to approximate a business model. The goal is experimentation and, ultimately, to realize a new practice in journalism.

I think your more substantive criticism is, Why involve the pros at all; that of course it's easy to polish up the contributions of the amateurs. And here, sir, I believe you overestimate the ability of any amateur to create good journalism. Rest assured we are sufficiently challenged in this project. If you look at the history of Linux, Torvalds and a select crew of "super-contributors," to use Jimmy Wales term for Wikipedia heavy lifters, conducted a lot of editing and filtering to develop Linux. My larger argument, that I'm still testing and thinking about, is that citizens/amateurs alone will not ultimately create professional product. It takes coordinators and editors and, really, someone or someones with veto power. But this critique too is valuable, and I think that I'm going to upgrade it to its own Post, complete with Jay's responses. We'll run it on AZ as well as CS.com, because I suspect you're not the only one asking these questions.

@Daren, et. al. As I said, I think Social Darwinism is a dangerous lens through which to view cultural, social, economic behavior. I think it was E.O. Wilson who pointed out that Darwin himself would have been horrified to see his theories applied to the social realm. I myself don't believe they make the leap very well.

More to the point, I don't think it's applicable to crowdsourcing. I actually think the phenomenon is a great leveler; it detaches ideas, content, cultural products in general from the race, creed, appearance, even identity of the creator. I've worked hard to attain a place on a national magazine, but my race (white), background (son of a professor, middle-class, college educated) and proclivities (highly extroverted, incapable of distinguishing between schmoozing and affability, etc.) have certainly played a role as well. Crowdsourcing eradicates these advantages (to my chagrin!).

But let me make sure I'm addressing your concerns: The "most viewed" videos on Youtube will always suffer--definitively, if you think about it--from information cascades (one person's knowledge being taken as wisdom by the next, and thus the next and the next ad infinitum). That will likely privilege majority taste, which given the digital divide and current Web demographics, white, protestant, etc. I agree this is a problem. However, I strongly dispute that the ultimate movement in this regard will be toward homogenization. I'd argue the opposite. The Internet has led to a flourishing of niche affinities and interests. Right now Digg and Youtube and Google all serve broad consensus, with all the distortion toward the mean that entails. But what about services like the Public Library of Science, in which the only commentators on a paper on, say, crystallography are going to be the small subset of scholars in that field? Even on Youtube we're seeing a movement toward the gathering into groups. Like Emo videos? Folk? Comedy? There's a group for you, which is far more likely to filter not just the "best" videos to the top, but those that are significant and meaningful for that subculture.

Sorta tossed off, all this, but those are my rough thoughts. I second Daren, let's keep this one poppin!

[alan]

I have been struggling to find a voice that might adequately describe my interest in the larger picture of crowdsourcing (CS) and in particular the individualized participatory process.

In both this blog and AZ there is, as of today, one element that has not been introduced to the conversation.
I have, in a couple of my comments, attempted to find the right words to bring to the surface just what I’m shooting for.

I would like to touch upon a central theme that runs through all individual, societal and cultural phenomena. I will focus here on CS.
Each of us, as our biographies unfold, impacts the world that surrounds us both near and far. Relationships, virtual or otherwise, bear many a mystery, but ultimately they bear within them the seeds of change.
I have studied Goethe for many years and have learned that the constellation and place of any such interaction has great meaning and springs out of our past deeds and sufferings.
Relationships and related activities offer much for our future, particularly if welcomed consciously. Our initial experience of such events might appear to contain little significance because only in retrospect, are we able to see the thread or meaning.

Our culture has moved through some very dramatic changes with the advent of recent technologies. Today’s technology, both for the individual and communities, has enabled a communications shift that touches upon very deep personal and cultural roots. Research tells us that whilst world boundaries are disappearing, in the USA we are becoming more isolated, communicating with far fewer people and qualitative interactions are on the decline.

What ever it is that sustains us as human beings (choose your own words, but please not chocolate or beer) is being subverted by a consensus trance. The great sleep induced by materialism, or having to feed the American Dream, the titles are many but the reality affects us all. This invisible mantle innocuously slipping over our shoulders is the antithesis of what phenomena such as CS might offer to re-connect us.

Economics and cultural forces, to name just two, provide some inputs for paradigm shifts but it is collective and individual destinies that manifest and result in meaningful change. I am not referring to pre-determination but rather a very organic, dynamic process.

In my childhood the family gathered around the mantle of the fireplace, which in turn became a gas fire then a radio and more recently a TV. We now gather, as individuals but with our fellow man and women connected by invisible threads all over the world, in front of “our” monitor.

How can this invisible force that connects us to each other and our culture be described?
All great world events can be traced back to particular individuals who through a biographical process and participation sparked a fire! The question is, how will CS fuel the fire
Alan.

[alan]

fire?

[LukePDQ]

Hey Jeff. Thanks for taking the time to reply. Your points are well taken and I appreciate your insights into the broader picture at AZ.

Actually, to correct one point; I think you mis-read what I was saying about pro's and am's.
You interpret me as suggesting...."Why involve the pros at all; that of course it's easy to polish up the contributions of the amateurs. And here, sir, I believe you overestimate the ability of any amateur to create good journalism..." Your words.

I am sorry if my words were misunderstood. My fault probably. Actually, If you re-read my words, I'm saying the opposite of what you inferred.
So my writing style is obviously not concise and clear, which makes one of your points for you. ;-)

For clarity, I certainly don't think that am's can turn out the quality of journalism that pro's produce. Not even close. So, no debate there.

My words "...how hard can it be to posit that pro's..." was meant to say that 'the experiment' of AZ to evaluate the qualitative outcome of pro's honing and polishing the quantity of am's grunt work - is predictable IMHO to produce fine journalism, which I believe was one of the tenets of the experiment.
Clearly, I'm not a journalist, so you were right to alert me to the sheer logistics involved for the pro journalists to physically deal with the tasks of sifting and honing the bulk material coming in - in order for the pro's to turn it into tight journalistic good copy.
Otherwise, I'm happy for you to post my comments onto AZ if it helps clarify the debate.
Your posts are coming in thick and fast.
Keep up the good work. It's getting lively in the comments contributions at last.