Crowdsourcing

Web Apps We'll Learn to Tolerate

Yesterday Read/Write Web, the Internet blog (is that redundant in the way "landscape gardener" is?) linked to one of the Assignment Zero interviews published by Wired.com. The interview was between JPG Magazine founder Derek Powazek and Ragnar Danneskjold, the founder of Subvert & Profit, a site that claims to represent the "crowdsourcing black market." Subvert & Profit pays people to vote for it's clients stories on social media sites like Digg.com and StumbleUpon.com. S&P, claims Danneskjold—a pseudonym swiped from Ayn Rand, the patron saint of sociopaths—"will operate a full-fledged marketplace for clandestine actions on the Internet." All together now: Ewwwwwwwwww.

And Subvert & Profit is not alone. A company called User/Submitted employs the same strategy, which is, in a few words, to stuff the ballot boxes on enormously influential sites like Digg, Reddit.com (owned by Wired.com) and Stumbleupon. Here's my somewhat contrarian reaction: Yawn—I hear they're gambling in Monte Carlo too. I find script kiddies and other species of black hat vermin as loathsome as the next guy, I'm about as surprised to see them as I am to see slugs on a mushroom. So long as there's been systems, there's been people who will exploit their weaknesses. In some cases, subversion serves the common good. In other cases it serves up splogs and spam.

This isn't to excuse Subvert & Profit. Rather I want to discourage the finger wagging and encourage social media sites in their attempts to build a more fool-proof voting systems that are less prone to being gamed. God knows that our government has had its challenges stamping out voter fraud. For most of the country's first century, gaming the system was simply how municipal officials (and perhaps a few presidents) got into office. The feds' attempts to ameliorate the situation has, and continues to be, pretty darn slipshod.

But then, that's government. We expect more from private industry, what with the free-market incentives and all that. If Digg proves itself to be one big link farm, I'm confident someone else will come up with a social media site with teflon protection from "the dark side of crowdsourcing." And it will work, at least for a while. As Assignment Zero interview notes, "let the arms race continue."

The Importance of Community

It was exceedingly difficult to sum up a complex, six-month project in the 2,000 words Wired.com allotted. As such, there were several themes left undeveloped. Due to the same space constraints, I was also unable to relate the various heroics that went into snatching (a qualified) victory from the jaws of defeat. It's easy enough to dispense with both of these tasks at the same time, as there's much overlap between the two.

Crowdsourcing projects are generally characterized as being the product of a few super-contributors and a mass of people who contribute some minor bits. I've heard this called the "dirty little secret of open source," the fact that most of the heavy lifting is done, not by the crowd per se, but by a few select individuals from within the crowd. I'd like to posit another rule: Any crowdsourcing project must install one go-to guy (or girl) who will thanklessly toil day and night to keep the project on the rails. At a magazine this person is called the Production Manager. On Assignment Zero he was called David Cohn.

It is no exaggeration to say that Assignment Zero would have never launched, must less reached completion, without David. Saddled with the totally inadequate title of "associate editor," in reality David did everything from customize Drupal for us, play Webmaster, manage the content on the site and play point person for a wide variety of volunteers and contributors. It's no accident that contributor after contributor emailed me to tell me how much they loved working with him. "The great thing about David isn't that he'll take on all the dirty jobs and work all night to get them done," Lauren Sandler said to me several weeks into the project. "It's that he never plays the martyr. He's all walk, no talk."  David is not motivated by laurels and glory, but he deserves both, in spades. (Full disclosure: David is my writing and research assistant on the crowdsourcing book. I'm lucky to have him.)

Another concept that by all rights should have been more fleshed out in the Wired.com piece was the importance of community. While I'd like to think this idea suffuses the piece, I could probably have been more explicit in noting its importance to making AZ productive. Lucky for us, our organizers, Tish Grier and Amanda Michel, understood this to a degree that the rest of us did not.

There was a crucial turning point when a rift opened up between the journalist types (myself included) on one side and Amanda and Tish on the other. They felt our volunteer editors had to play community manager, going out and soliciting contributors, keeping people engaged, holding a few hands. Us hard-bitten journos essentially snorted in disdain. Editors do not play cheerleader, and God knows they do not do outreach. We won the battle and, in doing so, contributed to losing the war. The plain fact is that  in the future, journalists will have to develop these skills if they want to succeed in a future in which their readers are also their writers.

The crowd does not contribute in a vacuum. They do so as part of a community of other contributors. I see this again and again in researching my book and, no surprise, it was true with Assignment Zero as well. Tish has written an excellent distillation of how this went down at Assignment Zero, and I'd suggest anyone serious about crowdsourcing and journalism experiments put it on their summer reading list.

"The Process of Elimination is Undervalued" — My Assignment Zero Evaluation on Wired.com

And so it comes to an end. After more than six months Assignment Zero passed its final milestone today when Wired.com published my decidedly ambivalent assessment of the project. It was, as I wrote in the piece, a "highly satisfying failure." A few of my readers had asked me to enumerate exactly what elements failed and what elements succeeded. I think the Wired.com piece performs this task adequately, with one exception, raised by Daren C. Brabham in the same comments section in the above link.

I neglected to point out the degree of mission creep Assignment Zero suffered. As Daren points out, to the extent  that exploring "crowdsourcing" was our mandate, that term came to become synonymous with everything Web 2.0. That's a terrible dilution of a process and methodology as rich, and precise, as crowdsourcing. As I noted in my June 28th post, the subject could more properly be called "cool forms of collaborative production on the Internet."

The sad fact is that being a buzzword, people tend to apply crowdsourcing to whatever new, nascent and exciting phenomenon they're attempting to define. As Daren suggests, a lot of work (and probably no small amount of bickering) will be needed to lasso that word and establish some quasi-permanent definition.

At any rate, I hope the Wired piece inspires more debate, on this blog and elsewhere.

Assignment Zero on Wired.com

Wired.com has published some of the best products from Assignment Zero today. I'm anxious to hear feedback if anyone wants to post here or send me trackbacks/links. More on my own reactions soon ...

Assignment Zero: The Interviews

Several weeks ago Assignment Zero executive editor Jay Rosen said that only about 28 percent of what we tried to do with Assignment Zero worked. (If you're unfamiliar with the AZ project, go here.) Rosen had been discussing the future of the interview with Washington Post media critic Howard Kurtz, and while 28 percent's a pretty stark assessment, it's probably accurate. Let's just say there's a lot we know now we didn't know when we launched the project. In a few weeks I'll be publishing an essay on Wired.com that will try to explore what  I've started to call "a wonderful failure."

But right now I want to celebrate everything that went right. About ten minutes ago I finished reading all 80 of the interviews conducted for the project. To properly appreciate this book-length body of work, I had to relinquish my expectation that they would hew to the putative subject, crowdsourcing. At first I was disappointed. Clearly Assignment Zero had suffered a case of mission creep. So I decided to revise my expectations. The general theme could be considered to be "emerging systems of collaborative production on the Internet." Mindset adjusted, I came away highly satisfied. Here's why:

With shockingly few exceptions, the interviews are compelling, thought-provoking and chock full of insights both philosophical and practical. The final package represents, to my knowledge, the most comprehensive and exhaustive knowledge base on the various ways the Internet has given rise to collaborative forms of production. I've been researching these issues for 18 months, and the collected information handily exceeds my own knowledge base—that itself is a testament to the wisdom of the crowds over the wisdom of the expert.

But allow me to gush in more detail: Most of the interviewers had obviously conducted a fair amount of research into their subject and prepared a list of appropriate questions to ask. AZ contributor Randy Burge, for instance, clearly drew deeply from the available literature on crowdsourced innovation before interviewing Innocentive co-founder Alpheus Bingham. The product is the finest distillation of the complex process by which Innocentive crowdsources problems in corporate R&D to its network of 120,000 scientists. In many cases, the interviewer was uniquely qualified to conduct the interview. James Surowiecki, for instance, was interviewed by one of his former editors.

This is the beauty of open organizational systems. People self-select, assigning themselves to tasks for which they are best-suited. Contrast this with the process by which an interviewer is assigned to interviewee in a closed system (a magazine or newspaper). A journalist is often chosen to conduct a Q&A with a subject based on his or her availability. That's a pretty poor qualification, though it's borne of simple necessity. The professional in this closed system (and I speak from personal experience), often lacks the time it takes to adequately acquaint oneself with the subject's work, ideas and experience. If the resulting product feels a little rote and indifferent, do you blame the journalist or the system?

But not only did the interviews betray a level of passion and specialization rarely found in the mainstream media, they were simply better reads. Magazines and newspapers tend to pasteurize such interviews to filter out any content that any reader anywhere might possibly deem offensive or obscure or simply irrelevant. The result is something that's leached of idiosyncrasy, complex ideas and the accidental poetry that arises from an animated conversation. The AZ interviews, lightly edited as they are, retain all these qualities (along with lots of typos and a few bits of asinine commentary.)

Over the next few months I plan on featuring several of the interviews on the blog as I'd like to directly engage many of the ideas—and their authors—on this blog. Until then ...

Photophiles Needed!

As I noted in my previous post, Assignment Zero will be producing feature stories on a reduced number of topics. In true crowdsourcing fashion, our contributors made this decision for us. If there's one similarity among all crowdsourcing models, it's that you do not control your contributors. That's the tacit contract in crowdsourcing: The crowdsourcer can't delegate work. It can only provide opportunities. The crowd decides the rest.

I can't exactly argue with the crowd's verdict, and in most cases I agree with it. We've had tremendous response to a handful of subjects, including the creation of Citizendium (we recently published the resulting article on Wired.com); and crowdsourcing in the novel, journalism and film. Clearly there's a thread to be teased out here. People get enthusiastic about culture, and how participatory technologies are changing the way it's produced and distributed. Likewise the topic of crowdsourcing in religion has drawn thoughtful, insightful reportage, and this too is understandable and to be applauded. I look forward to the results from all these projects.

But that said, I fear we're being forced to discard a few topics that I view as indispensable to any exploration of this phenomenon. Chief among these lacunae is the rise of the microstock industry. Stock photography has been irrevocably changed by the emergence of cheap, royalty-free photography, much of it shot by amateurs and sold through the so-called "microstock" agencies. Microstock undercuts its traditional stock photo competitors by well over 90 percent, and comprises, as far as I can tell, the most mature development of a crowdsourcing model in any industry. Anyone interested in how user-generated content is going to transform mass media needs to pay serious attention to the roiling waters in the photography world.

I know that a lot of photographers subscribe to my feed, or check this blog out regularly, and I'm hoping a few of you will agree with me and decide to contribute. We need people to interview key players involved with iStockPhoto, Fotolia, and ShutterStock, et. al. in an effort to determine how these companies have helped transform the world of stock photography. Who else should we interview? Are you ready to take some time to pitch in and increase our coverage of micostock houses? Let the editors of Assignment zero know here.

I know this subject inspires considerable passion (and divisiveness) in the photography community. This is your chance to transfer some of that passion into a researched, widely read and hopefully influential work of journalism.

Changes at Assignment Zero

As I noted yesterday, we're entering the final stretch at Assignment Zero. We've already learned some valuable lessons (which I'll explore in great detail in an upcoming piece for Wired.com) and we've responded mid-stream by modifying our ambitions somewhat. Instead of attempting to produce scores of magazine style features on how crowdsourcing has affected various fields and industries, we'll choose the half-dozen topics that have drawn the greatest amount of interest from our contributors, and explore the other topics through Q&A's with the key people involved. We're putting a lot of focus into what we're calling Interview Week. We've identified roughly 60 players in the world of crowdsourcing, and so far we've had an excellent sign-up rate from contributors interested in conducting interviews with them. There are still lots of openings however, so if you're interested go here to sign up.

Assignment Zero: What's it All For?

Assignment Zero, as my regular readers will probably know, is the NYU/Wired.com journalism experiment I'm working on. The idea is to use the crowd to research crowdsourcing. We're about halfway through the project, and couldn't be happier about our progress. Some 900 contributors and 30 professional jounalists have begun work on the roughly 200 assignment (go here for a full list). In the spirit of open source software, we decided to go public first and let our contributors help us refine our site, ideas and ultimate mission on the fly. Well, one point they're asking us to clarify is just what, exactly, is the purpose of all this work?

That's easily stated: It's to gather as much information as possible and write it up in a series of compelling, entertaining and edifying pieces. Written by our volunteer contributors, these stories will be published on NewAssignment.Net when Assignment Zero concludes (probably on June 1st, though like everything else on AZ, that's up for discussion). Over the next few days I will pour over all the assignments, published articles and survey data we've collected and tie it all together in a culminating essay that will run on Wired.com.

There's been some question about what form that Wired.com story will take. Here's what I can tell you: Not a whole lot like what we're used to reading. We're trying to create a new model of journalism here, and I'm determined that this spirit of experimentation run through my own bit of writing as well. First off, I will be relying heavily on our AZ contributors, and giving them due credit in the process. Not only will I be linking to the articles that run on NewAssignment, but I'll be linking to the original research and reporting on the AZ site as well. That's one of the reasons we've created home pages with their own URLs for each and every topic we're covering (Here's an example).

Of course, the ostensible point of my piece will be to draw some conclusions about how well Pro-Am journalism works and what we've learned about crowdsourcing. But just as important, my Wired.com piece will also show just how much research and reporting go into a feature story. In most cases, I do all that work myself. In the model we're all trying to pioneer however, there are hundreds of contributors filling that role. We believe that a large group of wildly diverse people can come together to perform the kind of in-depth investigation that was once the sole province of the professional. If we succeed—and we're cautiously hopeful we will—I'll be highlighting their contributions to show that our faith was well-placed. Thanks, and stay tuned.

The Cathedral and Assignment Zero, or what Eric Raymond Has to Teach Us About Crowdsourcing

It's been almost ten years since Eric Raymond presented his paper, "The Cathedral and the Bazaar" to the 1997 Linux Kongress. In that remarkable work Raymond compares two methods of software development. The "cathedral" approach requires a program to be "carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time." To contrast this he "anatomizes" the approach taken by Linus Torvalds in developing the Linux OS:

Linus Torvalds's style of development - release early and often, delegate everything you can, be open to the point of promiscuity - came as a surprise. No quiet, reverent cathedral-building here - rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches ... out of which a coherent and stable system could seemingly emerge only by a succession of miracles.

Raymond makes an astonishingly lucid and persuasive case for open source's raw, hurly-burly horsepower, which "not only did not fly apart in confusion but seemed to go from strength to strength at a speed barely imaginable to the cathedral builders."

The Cathedral and the Bazaar was required reading for any aspiring (or in my case, accidental) technology journalist back in the late '90s, and it's surely old hat for some of my readers. Yesterday I re-discovered it. I'm currently researching the third chapter of my book, in which I'll argue that crowdsourcing would never have emerged without the methodology, philosophy and digital architecture of open source software. To this end I re-read "Cathedral and the Bazaar," and was amazed by its prescience, and the degree to which it anticipated the challenges of crowdsourcing in general and Assignment Zero in particular.

In recent years the 10,000-odd words in Raymond's essay have been been boiled down to a single catchphrase: "Given enough eyeballs all bugs are shallow," which is to say, a diverse and large enough network can solve nearly any problem. It's the centerpiece of  Raymond's essay—what he calls "Linus' Law—and with good reason. It explains why a crowd can be wiser than the smartest individual within it in a pithy six words.

The problem is that like a lot of catchphrases, "All bugs are shallow" has entered the realm of the shibboleth—a way to flash one's Web 2.0 credentials without displaying any actual understanding of the underlying concept. I worry that the crowdsourcing community—which as far as I can tell does not contain many programmers—is largely unaware of all the other valuable lessons Raymond has to teach us. Raymond proposes 19 lessons in his paper. Most of them apply equally to all of us—Pros and Ams alike—at AZ. Below I highlight the most salient lessons, and how they apply to our little open source journalism project:

Every Good Work of Software Starts By Scratching a Developer's Personal Itch

Replace "Work of Software" with "Crowdsourcing Project" and you've got the reasoning behind Assignment Zero. Why did we decide to focus on crowdsourcing? Because we had an itch: We wanted to know more about  the phenomenon. But this lesson applies to our contributors as well. The greatest asset citizen journalists possess is passion. Find your itch on our Assignment Desk and scratch it.

Plan to Throw One Away. You Will Anyhow.

If you read yesterday's post by AZ Project Manager Steve Fox, you're already aware that Assignment Zero is—get this—a work in progress. But then, that's always been the plan.  (See "Release Early, Release Often below). We launched AZ knowing we'd probably have to scrap the first iteration to make way for the second iteration. "There's the way we want things," Steve told me last night. "And there's reality." How will we reconcile the two? "If people are dissatisfied, then the best way to make them satisfied is to bring them into the process of helping fixing it." Well put, Steve. And thanks for the segue to the next lesson:

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Whence springs "Given enough eyeballs, all bugs are shallow." This has implications for you, our users and contributors. Assignment Zero isn't our project — It's your project, and it's important to recognize that, well, with great power comes great responsibility. We don't need peanut gallery criticisms, we need constructive advice, hands-on help and most of all, a little bit of patience and the understanding that this is how open source projects develop, through trial and error and a great deal of collaboration from everyone concerned. For our part the implications are obvious: We should welcome participation from as wide—and crucially, as diverse—a group of contributors as possible, in order to leverage the infinite range of experience and expertise out there.

 Release early. Release often. This is what we learned this week. In Steve's words, we need to let everyone see each other's work, not hide it away until the day of the big release. In open source terms this means one doesn't try to write a perfect bit of code before letting ones fellow programmers start hacking away at it. For AZ, it means we should publish the results of our reporting and writing as we compile it. This allows our contributors to avoid conducting redundant research, but even more importantly, it allows for cross-fertilization of ideas.

Good Programmers Know What to Write. Great Ones Know What to Rewrite (And Reuse).

As is obvious now, I'm arguing that Assignment Zero has more in common with an open source project than it does a newspaper or even conventional news site. We're not writing poetry, we're writing mash-up journalism. This doesn't mean we have a license to plagiarize. On the contrary, as with open source software, proper attribution is even more important because it takes the place of more liquid currencies like, you know, actual money. This does mean all of us should be casting the broadest net possible and incorporating good ideas wherever we find them.

Too Much Collaboration Can Be a Bad Thing.

Okay, you won't find this one word-for-word in Raymond's essay. I extrapolated it from his discussion of what sociologists once called the "Delphi Effect" and what we now call the Wisdom of the Crowds. To wit,  in Raymond's words: "The average opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than that of a single randomly-chosen one of the observers." All well and good, but what's often misunderstood about the wisdom of the crowds is that it's the wisdom derives from a crowd of individuals thinking and working independently, not from a crowd thinking as a single unit, which leads to mobocracy. (See the jelly bean jar example in my post on the subject from a few days ago.)

What's this mean for AZ? Well, if all we wanted was the crowd to come to a consensus on crowdsourcing, we would have stopped after building our forums (we call it The Exchange.) But we have much more ambitious goals — We want you out there digging up research independently. Confer, by all means, but also be careful to come to your own conclusions. The wealth of the network lies in diversity and discourse, not from sermonizing amongst the choir. Besides, time on the forum is time you could be reporting!

It's been a long post, so congratulations to those of you who made it through. I hope you learned as much reading it as I did writing it. You'll learn much more by reading Raymond's original essay, which I love so much I'm going to link to it again.

How Do You Measure a Spare Cycle?

In the original Wired article about crowdsourcing I compared the model to SETI and other forms of distributed computing. Crowdsourcing essentially entails taking advantage of people's "spare cycles," just as SETI utilizes the spare cycles on people's computers. That trope stuck, and is probably the most quoted section from the article.

Well now we're starting to refine that idea. The most gratifying aspect of working on Assignment Zero (here's Wednesday's launch announcement on CS.com) has been the ability to report on crowdsourcing as a participant as well as a spectator. Journalists generally watch events from the bleachers. This is just like Paper Lion, but without bone-crunching details. Anyway, tonight Jay Rosen (AZ's executive editor) and I decided that we needed to establish a basic unit of exchange for the spare cycle.

In other words, what kind of time commitment can a reasonably motivated and enthusiastic volunteer make? We're speculating that it falls somewhere between 5 and 10 hours a week, but that's guesswork.You can bet we'll have a better idea in 12 weeks. You could expand on this hypothesis (and it truly is an untested theory) and attempt to determine a unit of exchange for the "super-contributor," which I believe is Jimmy Wales' term for those few hundred Wikipedians responsible for all the heavy lifting on Wikipedia. (Here's a link to AZ's assignment "Interview a Super-Contibutor," which includes an excellent post about the super-contributor phenomenon from our own Amanda Michel.) Maybe it's my positivity bias (otherwise known, wonderfully, as the Pollyanna Principle.), but I think we're really breaking some ground with AZ. The significance is that we're beginning to develop real methodology to a model that has—with some justice—been accused of being just another Web 2.0 buzzword.

I'll continue to post these nuggets from time to time. Hope they're helpful.