Can We Crowdsource Medical Expertise?

Earlier this morning I received an email from Christine Maynard, a writer and consultant from Natchez, Louisiana. I'm not sure if she heard me on the radio or came across crowdsourcing in some other way, but she raises an issue I've been wanting to dig into ever since publishing the original crowdsourcing article over two years ago: To what degree could the crowd lend its brainpower to the process of, say, diagnosing or suggesting treatments for diseases?

One is tempted to dismiss this notion out of hand. Let the crowd count birds, develop cell phone applications,  even create a new restaurant. But keep them away from my MRI scans. This is a pretty understandable reaction, and I've tended to view attempts to crowdsource the professions (law and medicine, namely) with considerable skepticism. But given the poor quality of much of the healthcare in the United States, to say nothing of the developing world, could the crowd really make so big a mess of things? Here's Maynard:

Greetings, Jeff.
Crowdsourcing would work for taking CT and MRI reading out of the hands of [radiologists] and into the hands of individuals who have become medical experts in pursuit of their own elusive diagnoses. Radiologists read scans in a couple of minutes, referring to any previous scans in hospital records which are pulled up as soon as social is entered, and simply note differences, if any. They have clues, from doctors "reason for request" which is their loophole to NOT closely examine a scan.

The technology is wonderful.  The vast majority of specialists refuse to do "exploratory" despite their patients desperate attempts to get relief from agony and, hopefully, live, because of the state of the art. "If there was a problem, we would have seen it," they say. Yet failure to find a problem, or five different interpretations, none hitting the mark, is the norm, resulting in no treatment or mistreatment.

The missing ingredient is better interpretation of diagnostics, which could be accomplished by crowdsourcing. Ideally the demographic without borders would be any reasonably intelligent person who has been on a medical merry go round with no help proffered, and who has resorted to examining and interpreting, for months, maybe years, their own scans.

I don't know if missing the mark exactly the norm, but it's certainly a frequent occurrence. (Full disclosure: Our son had had two MRIs by the age of six months. The readings, by separate radiologists, contradicted each other and neither was conclusive). Without going into the personal details of her story, Maynard has endured an extraordinary range of interpretations, as she says.

While I would never advocate going to the masses for radiology (this would, clearly, be ridiculous), this isn't quite what Maynard implies. Why not have an informed group of people—some might even be radiologists, or MDs in other fields—reviewing scans that were posted online. I fear that Maynard is right that overworked professionals have little time and little incentive to pour carefully over MRI and CT scans. If this particular crowd of semi-professionals—what Scott Page, the U. Michigan collective intelligence brillionaire I lean on in the crowdsourcing book, might call a "crowd of experts*"—was willing to act as a fact-check back-up resource, why not tap them?

I'm getting a lot of additional traffic today due to all those afore-posted radio gigs. So I put the question to you, and them: Would you trust the crowd to help analyze your medical data?

(Ed's Note: Written on the fly between 25 radio interviews (we had two cancellations) and one Red Sox/Yankees game. Forgive mistakes and correct me in comments.)

* Pre-correction correction: Scott actually calls this a "crowd of models," in his excellent book, The Difference, I believe. I've taken liberty with his terminology.

Comments

[nalts]

I'd trust opinions of 100 patients that tried a pharmaceutical drug more than I'd trust a physician (unless they had unique reasons to know about the drug). I think this concept is already happening with http://www.crazymeds.us/.

[Jeff]

In my own experience, I believe I have done this simply by seeking out and listening to, as well as sharing with others who have dealt with various medical conditions. I think having an organized means of doing this could be helpful to individuals and families. My greatest concern would be how to discern who to listen to. I turn to friends and people I trust in seeking advice on medical issues. I will not even go back to a doctor I don't feel confidence in, no matter their credentials.

[Alexandra Carmichael]

Excellent question, Jeff!

I would definitely trust the crowd for medical advice and interpretation. If the crowd is big enough, I would even venture to drop the requirement for "expert" credentials. Why not let everyone comment but have ratings or credibility indicators for commenters that either have expert credentials or have been rated highly as having provided a large number of useful/accurate comments on other people's medical tests?

Your question also reminds me of Melanie Swan's post/idea for InterpretMyXray.com (http://futurememes.blogspot.com/2007/10/interpretmyxraycom.html)

Congrats on your book! Can't wait to read it.
Alexandra Carmichael
Co-Founder, CureTogether.com

[-alph-]

As a former healthcare researcher and as founder of InnoCentive, I appreciate the question raised by Maynard and in this post. It's a meaty one. Of course, it goes without saying that many 'portions' of the overall healthcare process (including research objectives) can and should be crowdsourced for a host of reasons that would improve outcome and quality.

The specific issue of crowdsourcing diagnosis or treatment is far less clear, but a look at some actual calculations might be informative. An expert (say, right 95% of the time) is wrong 5% of the time. An amateur might be wrong 20% of the time, but the chance that two amateurs are both wrong is only 20%x20% or 4%. So two "informed" amateurs consistently reading a scan or collection of lab results has a pretty good record. Hmm, seems like a clear case FOR the crowd... Of course, as the number of semi-professionals or informed amateurs goes up, the chance of them ALL being wrong goes down, but the chance of getting mixed diagnoses goes up very fast. (73% say "benign" and 27% say "malignant.") What to do then? Majority rules? Supermajority required?

When is the "vote" compelling enough to stake your treatment on it? It would be well outside the scope of a blog comment to delve deeply into this. But let's briefly return to our two amateurs. When they agree there is a 96% chance of them being right. But how often do they agree? If they are looking at a cancerous scan, with an 80% individual accuracy rate, they agree on the cancer diagnosis only 64% of the time. They split opinions (a very confusing state of affairs since they are equally likely to get it right and now you don't know who to believe) 32% of the time (that's a lot) and they both get it wrong only 4% (as we said already). There may well be some sophisticated statistical analyses that would supplement such crowdsourcing approaches -- BUT -- 'experts' or 'crowds' or 'crowds of experts,' there will remain ambiguity when dealing with judgement calls. Our penchant for certainty is just not going to get fully satisfied.

Total non-experts (the masses referred to in the post) do NOT help matters as their input is just noise -- obscuring a signal. But the crowd of semi-experts could well be, in my opinion, desirable, and we should investigate appropriate systems and knowledge aggregation tools for its exploitation... but not to be treated over-simply. (caveat: even this little treatment is over simple as it has failed to consider independently the error rates for alpha errors and beta errors (often not the same, quantitatively or consequentially) or correlated errors, as well as other factors... but you get the basic idea.)

[Jen McCabe Gorman]

Alexandra, Jeff -

Great points both, and excellent post Jeff.

I've seen various medical bloggers post 'intriguing' mystery x-rays and other scans on their blogs and then ask readers to 'guess' the diagnosis (sometimes they also comment on the prognosis). Although I'm not a physician, nurse, PA, or other clinical professional, I do sometimes guess and post a reply in the comments. But these comments aren't in any way connected back to the person in the scan, or the clinical evaluation and/or treatment they received as a result.

This section of your post jumped off the screen: "I fear that Maynard is right that overworked professionals have little time and little incentive to pour carefully over MRI and CT scans."

I wonder if blog readers participate in crowd-sourcing findings on posted films because this is currently almost a form of clinical 'play' rather than the work this would morph into with a site dedicated to doing this?

However, as an interesting business model example, you could certainly look at a site/service like that of American Well, where radiologists could join (from all over the world), have licensure/credentials checked, and then be paid to read scans whenever they had time available. Remote radiology firms do this sort of thing now.

Thanks for the thought-provoking read!

Jen McCabe Gorman

Co-Founder, Nexthealth
Nexthealth.NL
www.healthmgmtrx.blogspot.com

[Daniel Reda]

Why do we go to the doctor or get a diagnostic test? It's to learn something about our bodies and then make a decision based on that result. All of us want to maximize the odds of health and minimize the odds of suffering.

When it comes to getting a diagnostic test, there are three variables - how good is the test itself, how good is the interpretation of the test, and how good is the doctor's diagnosis based on all the information available to them?

A poor test is one whose result is uninformative, one that equally supports multiple diagnoses. A good test is one that strongly supports a particular diagnosis while strongly refuting others. Some tests need no interpretation with regard to the result (e.g. a cholesterol level is just a number output by a machine). Others, like an MRI, require interpretation. An MRI is just a picture until an expert looks at it and determines what it indicates (e.g. tumor or all clear). This result is then forwarded to the doctor. In the case of an MRI, a good interpretation is one that faithfully and consistently reflects the best knowledge about what various features on an MRI scan mean.

The doctor's job is to incorporate hard numbers (e.g. cholesterol levels) and interpreted results (e.g. from MRI scans) into a model of your complete health profile and then make an overall diagnosis. Diagnoses are usually supported by more than a single test result (e.g. family history, other tests, an in-person visit, etc.) A good diagnosis is one that accurately assigns weights and probabilities to all the available information about your health, based on up-to-date medical knowledge, to create a reliable, most likely diagnosis, ideally with odds attached to alternative diagnoses supported by the same data.

Patients often question their diagnoses and seek second opinions, especially when faced with a major health decision. They are justified in doing so. Second (or third, fourth or fifth) opinions are too often contradictory. And there is a well documented (and shockingly high) incidence of incorrect diagnoses and other medical errors that cause unnecessary illness and even death.

When faced with contradictory diagnoses, you can no longer rely on the experts because they don't agree with each other. But now you have a new problem - which doctor is right? The challenge here is that we don't have statistics to tell us which doctor is historically most correct or which radiologist is most correct. In almost every other area of life, from baseball to buying a car or investing your money, there is a wealth of data to support our decision making. But when it comes to deciding which doctor to trust, we patients have almost nothing to go on.

So, what would happen if you crowdsourced interpretation or even diagnosis? Well, the consensus interpretation of 100 amateurs on your MRI would probably not be at all helpful. What you'd want is a method to select the best interpretations and have them bubble up to the top. How do you select the best interpretations? One way is to keep historical data on how accurate those predictions were once more data became available. Ideally we'd gather data on doctors' performance as well. It's not about credentials - it's about accuracy. If the doctors don't want to participate, then their judgments will look progressively weaker compared with those of a supposed amateur who was proven to be correct 99% of the time on thousands of MRI interpretations.

Personally, just like I have batting averages, crash test ratings and historical earnings, I'd like to see the data on whoever interprets my MRI and whoever makes my diagnosis, regardless of whether it's an amateur in a foreign country or the chief of staff at the world's "best" medical center.

Daniel Reda
Co-Founder, CureTogether.com

[Insurance Transcription India]

Great thing for share

[ngan hang]

Oh ! thank you for your advice .It is useful .

[kraloyun]

Great thing for share