Need an Expert? Try the Crowd

[zenmaster]

Generating models from large data sets—and determining which subsets of data to mine—is becoming increasingly automated. However, choosing what data to collect in the first place requires human intuition or experience, usually supplied by a domain expert. This paper describes a new approach to machine science which demonstrates for the first time that nondomain experts can collectively formulate features and provide values for those features such that they are predictive of some behavioral outcome of interest. This was accomplished by building a Web platform in which human groups interact to both respond to questions likely to help predict a behavioral outcome and pose new questions to their peers. This results in a dynamically growing online survey, but the result of this cooperative behavior also leads to models that can predict the user's outcomes based on their responses to the user-generated survey questions. Here, we describe two Web-based experiments that instantiate this approach: The first site led to models that can predict users' monthly electric energy consumption, and the other led to models that can predict users' body mass index. As exponential increases in content are often observed in successful online collaborative communities, the proposed methodology may, in the future, lead to similar exponential rises in discovery and insight into the causal factors of behavioral outcomes.

http://www.uvm.edu/~uvmpr/?Page=news&storyID=14100

[Parsons]

Isn't this how a social memory complex works? When you pose a question to it, only the individual consciousness responds with the exact correct answer (if there is one for the question posed).

---

And there are many reasons why this new approach might be helpful. In addition to forces that experts might simply not know about — “can we elicit unexpected predictors that an expert would not have come up with sitting in his office?” Hines asks — experts often have deeply held biases.

They certainly identified a huge advantage to "crowd sourcing" information compared to going to an expert. The inflated ego of an expert can make the "expert" think that THEY are correct all/most of the time so they are most likely correct about whatever issue they are giving their opinion.

[zenmaster]

I'm wondering if this approach could also somehow be useful in eliciting tacit knowledge.

[Sagittarius]

Merging machine and man, I like it.

[zenmaster]

Merging machine and man, I like it.

What do you mean by merging?

[Cyan]

Merging machine and man, I like it.

Am I the only one who sees this?

Besides the person i'm quoting ofcourse =)

[Sagittarius]

Merging machine and man, I like it.

What do you mean by merging?

Taking away the default judgement of machine like processes and turning it into something more human and less static. Creation can be likened to be machine like right? However it forever moves forward into new possibilities without getting stuck in a straight line so to say.

[Oceania]

well that explains everything. if it's run by a machine they're not gonna hear our cries of how the experiment went wrong.

[Cyan]

well that explains everything. if it's run by a machine they're not gonna hear our cries of how the experiment went wrong.

Run by does not = on command of. Your car is "run by machines" in that you move thanks to them but it happens of your volition.

[zenmaster]

Merging machine and man, I like it.

What do you mean by merging?

Taking away the default judgement of machine like processes and turning it into something more human and less static. Creation can be likened to be machine like right? However it forever moves forward into new possibilities without getting stuck in a straight line so to say.

Care to elaborate? Not sure what you mean by "default judgement"?

My understanding of the research is that they use the intuition of a pool of individuals (the crowd) to provide a pool of questions intended to arrive at the best answer. The question pool is then narrowed (again, by the crowd) to those questions which had the most relevance to arriving at the best answer. One could theoretically use this fool-proof approach for anything, even medical diagnosis.

[Patrick]

That's interesting.

[Sagittarius]

Merging machine and man, I like it.

What do you mean by merging?

Taking away the default judgement of machine like processes and turning it into something more human and less static. Creation can be likened to be machine like right? However it forever moves forward into new possibilities without getting stuck in a straight line so to say.

Care to elaborate? Not sure what you mean by "default judgement"?

My understanding of the research is that they use the intuition of a pool of individuals (the crowd) to provide a pool of questions intended to arrive at the best answer. The question pool is then narrowed (again, by the crowd) to those questions which had the most relevance to arriving at the best answer. One could theoretically use this fool-proof approach for anything, even medical diagnosis.

Yeh I think I went a bit of topic trying to come up with a reply, it just seams to me that technique is combining mechanic like processes with human intuition in a more truthful way then has been done before.

[zenmaster]

Yeh I think I went a bit of topic trying to come up with a reply, it just seams to me that technique is combining mechanic like processes with human intuition in a more truthful way then has been done before.

I think that historically it has been recognized by all serious researchers that people are biased and, in order to overcome this bias, a methodical approach must be employed. It's usually part of the early training for a number of professions. I personally think that same type of discipline can generally help society both ask and recognize better "questions" thus helping to re-evaluate needs and so to "optimize" our experience here.

[Sagittarius]

Yeh I think I went a bit of topic trying to come up with a reply, it just seams to me that technique is combining mechanic like processes with human intuition in a more truthful way then has been done before.

I think that historically it has been recognized by all serious researchers that people are biased and, in order to overcome this bias, a methodical approach must be employed. It's usually part of the early training for a number of professions. I personally think that same type of discipline can generally help society both ask and recognize better "questions" thus helping to re-evaluate needs and so to "optimize" our experience here.

Indeed, would you say it is getting rid of the bias out of the bias detection methods ?

[zenmaster]

Indeed, would you say it is getting rid of the bias out of the bias detection methods ?

The point is with this approach, the ultimate result, even if it was the result of heavily biased input, is still going to be practical and potentially superior to conventional wisdom or years of work from a specialized researcher. There is a crowd-sourced check with the responses which automatically filters out "bad" questions. So this approach makes the question of 'bias' irrelevant. However, one might gain new insight into how some process works which may uncover some long-standing conceptual bias.

Still.. garbage in, garbage out - you need people experienced with the subject matter to offer the most relevant questions. Would be interesting to see if an instruction manual could be created using a similar method. For example, if the person follows a set of "questions" they would be led to an "answer".

[Sagittarius]

Indeed, would you say it is getting rid of the bias out of the bias detection methods ?

The point is with this approach, the ultimate result, even if it was the result of heavily biased input, is still going to be practical and potentially superior to conventional wisdom or years of work from a specialized researcher. There is a crowd-sourced check with the responses which automatically filters out "bad" questions. So this approach makes the question of 'bias' irrelevant. However, one might gain new insight into how some process works which may uncover some long-standing conceptual bias.

Still.. garbage in, garbage out - you need people experienced with the subject matter to offer the most relevant questions. Would be interesting to see if an instruction manual could be created using a similar method. For example, if the person follows a set of "questions" they would be led to an "answer".

Haha yeh I see what you mean.