Crowdsourcing Medical Expertise Part Two: Reader Comments

Every once in a great while I find the comments to a post so thoughtful and intriguing I decide to "upstream" them into their own blog post, both to draw more attention to what I deem to be valuable points, and as an attempt to provoke more debate. A few days ago I received an email from a Louisiana-based writer and consultant named Louise Maynard. Having endured contradictory interpretations of her own MRI scans, she wondered whether their might not be a crowdsourcing application in, essentially, amateur radiology. In this case, the comments made for better reading than my original post.

The implications of crowdsourcing the act of diagnosis are troubling, problematic and also tantalizing. To some extent, medical expertise is already being crowdsourced, as the commenters pointed out. Nalts pointed out that the site CrazyMeds runs a very active forum in which people gather to discuss the ups and downs of the medicated life. As Nalts notes, he'd rather trust the consensus regarding a drug formed by 100 patients than a single doctor. (Ed's Note: It's also worth reading the witty and poignant personal essay on the front page.)

Jen McCabe Gorman notes that the site American Well poses an interesting case study in that it vets medical professionals and then allows them to offer their services online, which isn't far from what Maynard is proposing

I was also happy to see Alpheus Bingham—founder of InnoCentive and a veteran scientist (he holds a Stanford PhD in organic chemistry) in his own right—join the conversation. He aptly problematizes the prospect of crowdsourcing diagnosis: "Of course, it goes without saying that many 'portions' of the overall healthcare process (including research objectives) can and should be crowdsourced for a host of reasons that would improve outcome and quality," he writes. As most of the readers of this blog realize, this is an essential component of what companies like InnoCentive, NineSigma and YourEncore do—broadcast research problems to large groups of experts and non-experts alike in the hopes of unearthing novel solutions. However, "the specific issue of crowdsourcing diagnosis or treatment is far less clear."

On the one hand, the numbers would seem to favor such a project. Here's Alph:

An expert (say, right 95% of the time) is wrong 5% of the time. An amateur might be wrong 20% of the time, but the chance that two amateurs are both wrong is only 20%x20% or 4%. So two "informed" amateurs consistently reading a scan or collection of lab results has a pretty good record. Hmm, seems like a clear case FOR the crowd...

Ah, but the numbers can deceive:

Of course, as the number of semi-professionals or informed amateurs goes up, the chance of them ALL being wrong goes down, but the chance of getting mixed diagnoses goes up very fast. (73% say "benign" and 27% say "malignant.") What to do then? Majority rules? Supermajority required?

When is the "vote" compelling enough to stake your treatment on it? It would be well outside the scope of a blog comment to delve deeply into this. But let's briefly return to our two amateurs. When they agree there is a 96% chance of them being right. But how often do they agree? If they are looking at a cancerous scan, with an 80% individual accuracy rate, they agree on the cancer diagnosis only 64% of the time. They split opinions (a very confusing state of affairs since they are equally likely to get it right and now you don't know who to believe) 32% of the time (that's a lot) and they both get it wrong only 4% (as we said already). There may well be some sophisticated statistical analyses that would supplement such crowdsourcing approaches -- BUT -- 'experts' or 'crowds' or 'crowds of experts,' there will remain ambiguity when dealing with judgement calls. Our penchant for certainty is just not going to get fully satisfied.

This is, to some degree, the situation we're already in (albeit with very small crowds) when several professionals are asked to interpret medical data (such as an MRI scan): Ambiguity. That said, in the end Alph makes a strong case for the exploration of a crowdsourcing effort that would entice the semi-professionals to engage in such a project: "Total non-experts (the masses referred to in the post) do NOT help matters as their input is just noise -- obscuring a signal. But the crowd of semi-experts could well be, in my opinion, desirable, and we should investigate appropriate systems and knowledge aggregation tools for its exploitation."

Finally, Daniel Reda and Alexandra Carmichael, the co-founders of the very promising CureTogether.com, both posted useful comments. Here's Daniel: 

What would happen if you crowdsourced interpretation or even diagnosis? Well, the consensus interpretation of 100 amateurs on your MRI would probably not be at all helpful. What you'd want is a method to select the best interpretations and have them bubble up to the top. How do you select the best interpretations? One way is to keep historical data on how accurate those predictions were once more data became available. Ideally we'd gather data on doctors' performance as well. It's not about credentials - it's about accuracy. If the doctors don't want to participate, then their judgments will look progressively weaker compared with those of a supposed amateur who was proven to be correct 99% of the time on thousands of MRI interpretations.

"It's not about credientials—it's about accuracy." That's as concise a definition of crowdsourcing as it gets. Here's hoping we can keep this conversation going.

Comments

[Alan Booker]

"It's not about credentials—it's about accuracy." And what is accuracy?

That which emanates from a crowd, harnessed to a particular focus, might be the nearest thing to collective intuition.

As much as one might crunch numbers, fantastic insights Alph and I am certainly not talking about specialized skill sets or contemporary technical skills here.

The power of the crowd harkens back to a time whence one’s eye and knowing was turned as much inward as outward. Over the course of mankind’s march to knowledge, knowledge has become information and thereby that which was inherently known, the power of intuition, has been left to atrophy like any unused muscle.

As individuals we now turn to the many experts to show us the way, whether it’s regarding the process of childbirth, education, personal health or the many other areas in our lives where we are no longer able to trust our instinctive wisdom.

As an individual’s ability to “know,” perceptive insight, decreases, residues still exist and might more easily be released by a crowd.

I would suggest that a balance of reasoning processes and perceptive insight would be the best outcome when looking for a doctor! I would want my health professional to look me over, shake my hand and thereby already have a wealth of information about me!

Once my health outcomes are relegated to an MRI my trust has to be given over to a specialist or a crowd thereof!

Warm regards, Alan

[Canadian]

I see the Crowdsourcing aspect of this as being more part of a broader system, and also more interactive.

Assuming there is a collective of medical experts and semi-experts. I (or my doctor) take my MRI, or other collection of medical information, and submit it into this collective. Then two or three (not one) members would be assigned it at random for a diagnosis. My assumption is that the large pool of resources would result in fast turn around, and that for the majority of cases, the diagnosis would agree (common straight forward issues - do these exist or am I naive?)

Then, should the diagnosis not agree, within the design of the collectives support software, my case would be promoted to broader visibility and/or higher level experts. But it wouldn't be that the diagnosis would occur in a vacuum... thus be two conflicting diagnosis. I would hope the power of the power of the collective would create a dialogue on the case. This would serve two purposes 1) allow the community itself to come to a consensus (ie - I'm bit judging the merits of the diagnosis, the community does) and 2) allow the community to improve it's expertise and identify the experts of the experts.

I think the distinction I'm making here is from the many minds creating a few devices that many people can use (a cell phone) versus a potentially high volume of individual cases requiring a diagnosis.

I realize you are exploring a single 'difficult case' except I can't help but feel that for the vast majority, the true need is getting access to the experts in the first place (whether it is say Canada with extremely long wait times, or less developed nations with no experts at all).

So the value is a system that leverages a vast collection of experts to primarily expedite case reviews to the benefit of the patient, but also free's up time and identifies experts to collaborate (not compete) on diagnosing to cases that consensus was not immediately available. [and a bonus is the system inherently recognizes the expert experts]

[Melle Gloerich]

With thousands of amateurs joining in, the chance one of those amateurs hits 50 good predictions in a row gets bigger. It seems like this guy is a very good doctor, but in fact is just incredibly lucky. Or actually, it is really normal having a few of these 'lucky' amateurs when having thousands of not so lucky ones. When in a casino a roulette hits red 50 times straight after eachother, this roulette can't be trusted to hit red again, chances still are 50% (well, little bit less with 0 ofcourse)

I would trust the crowd, but only if it is made up of trusted people.

[Michel Bauwens]

Might be useful for finding crowdsourced medical applications: a tag monitoring health 2.0 developments: http://delicious.com/mbauwens/P2P-Healthcare

See also for the broader field of science: http://p2pfoundation.net/Category:Science

Michel

[Rachael]

Thanks for the information...I bookmarked your site, and I appreciate your time and effort to make your blog a success!

[Jerod Poore]

I can give you some more perspective from those of us doing the "Medical crowdsourcing."

First of all, we at Crazy Meds don't diagnose. We'll lend perspective as to the odds of a correct diagnosis and if a second, or third, or nth opinion is required, but that's about it as far as diagnosing goes. That takes the form of:

Someone writes they are depressed, anxious, irritable, have headaches and sometimes have phantoms smells. They have been previously diagnosed as depressed, and just got the dreaded bipolar diagnosis. Now they wonder if they have epilepsy. They write that their symptoms are depression, anxiety, irritability, headaches and sometimes phantom smells.

It's rarely that concise, but that's the gist of it.

A typical response would be...

"Depression, anxiety, irritability are symptoms of bipolar, usually bipolar 2. Phantom smells are sometimes an oddball symptom of bipolar disorder.

"Phantom smells are frequently a symptom of an epileptic aura, as are headaches and anxiety, but the depression and irritability usually happen just after or in between (post-ictal and inter-ictal) obvious seizures. Phantom smells are sometimes part of a migraine aura, but you'd know if you were having a migraine.

"Headaches are a side effect of most medications. They are also a result of stress, needing glasses or needing an upgrade to existing glasses, existing or new allergies and a host of other non-psychiatric and non-neurological conditions.

"The odds are the bipolar diagnosis is the correct one. Suck it up and deal with it through medication and therapy. Denial, anger, grief, etc. are why God invented support groups, which are elsewhere."

That's where a doctor is right, which happens most of the time. Sometimes doctors seem to come up with incorrect diagnoses. Doctors can be wrong, but mainly due to being limited to 30-minute intake sessions, 15-minute follow-ups once or twice a month, and patients being too frazzled to discuss the most important symptoms. Worst of all are the bipolar who concentrate on depression and not mania. Peers are in a better position to give them a virtual slap upside the head to deal with medicating mania.

Where we do have problems with far too many doctors are their selection of medications, their blind adherence to titration schedules, and their refusal to look up anything in the PDR regarding side effects, drug-drug interactions and the like. Too many decisions are made based on propaganda and borderline bribery from pharmaceutical representatives, as well as some older doctors not trusting newer medications which may be more appropriate and most younger doctors being ignorant of older medications which can be more effective for some people.

Don't get me started on the real differences between brand and generic medications, and how it can be downright dangerous to switch epileptics who started on brand name drugs to the generic versions.

The main reason we at Crazy Meds, and the citizen medical experts (as one pharmaceutical industry trade publication dubbed us) elsewhere are able to step in where the professionals have seemed to fail is that we have the luxuries of time and over-specialization. In addition to how much time each person has to research something, the very nature of crowdsourcing deals with the time factor. Bu we don't have to worry about knowing how to stick a tube down someone's throat, or how to do a C-section, or any of the numerous other things that any doctor, regardless of specialty, could do in case of an emergency. Instead we can list obscure off-label usages, first, second and third-line treatments, when it makes sense to plop down $100 to test your CYP450 2D6 alleles and a bunch of other neuropsychopharmacology trivia that is important only to the brain cooties outliers.

The key word is outliers. The majority of people are served just fine by the doctors they see, and the first or second treatment is the one that works. That's how it is with neurological / psychiatric conditions and I guess that's how it is with most everything else. Other than encouraging cyberchondria or engendering the nocebo effect, where someone would be convinced that nothing will ever work, most people shouldn't be delving deeply into obscure symptoms, conditions and side effects.

Getting a straight answer about side effects, chances of efficacy and time it takes for a med / treatment to work are what peer-to-peer medical sites should be about. Unfortunately far too many of them focus on complaining. They may as well be sponsored by Ye Olde Supplement Shoppe.

[medical]

just have a test of this blog.

[April]

Medical questions can be fairly hard to answer especially when everyone has a different opinion and answer but you can visitAsk A Nurse to have all of your questions answered! http://www.callcarenet.com/products-nursecare.asp

[health advocate]

Has anyone ever tried a medical call center, This is where you can ask a nurse questions about minor medical problems and conditions you may have which in turn could save you thousands in medical expenses.

[ask a nurse]

The healthcare system is falling apart, but as a health advocate I am assured a job as medical industries are in high demand always. Many poeple these days are utilizing
medical call center and telephone triage services to skirt high medical costs. A nurse line combined with certified nurse triage practicioners is a great alternative to going to the ER and being charged those astronimcal fees.

[Medical Billing Software]

It is very disturbing to know that our medical records are easily accessible to just about anyone if you know what you are doing. There needs to be some additional checks and balances to the healthcare system

[Increase Penis Size]

More & more people know that blog are good for every one where we get lots of information any topics !!!

[healthy cures]

You have made some really great points. It was a good choice to choose this comment to make a blog post.

Stay healthy with natural remedies