An Example of a card from Ash experiment
Image Source: Wikipedia
In a post in my other blog (written in Hebrew), I wrote that sometimes my opinion differed from the majority opinion. Sometimes it also differed, from leading experts opinions in the field or the topic. In those cases, I acted according to my opinion and not according to the Majority view.
Wisdom of the Crowd is a method used more frequently than before. The essence of Wisdom of the Crowd, is assembling many non-expert opinions as information supporting Decision making.
The classic usage of Wisdom of the Crowd is for estimation of quantified problem, e.g. a Taurus weight. However, usage of Wisdom of the Crowd could be misleading.
In this post I discuss two factors, which oppose usage of Wisdom of the Crowd, including few examples.
Factor 1: Conformity and Social Behaviour
The illustration in the heading of this post, depicts an easy task of distinguishing between lines length. It was included in Solomon Asch's Conformity Experiment. All but one of the participants were "confederates" (i.e. actors).
The "confederated" answers to some of the line similarity tasks were wrong and identical. 75% of the participants conformed with wrong "confederated" answers.
We should expect higher conformity in case of more difficult questions.
As far as Wisdom of the Crowd is concerned, expect identical answers in similar situations, i.e. Social Interactions between people answering the question.
Factor 1: Information Technology example
More than a decade ago, an Insurance Company's CIO presented in a local IT conference. The CIO was brave: He publicly admitted a colossal failure of selecting and implementing a strategic Software product.
He admitted a waste of hundred thousands of USDs and few years due to this mistake.
He was not the only one to commit this mistake: many others made the same mistake and get similar results. The only difference was that other CIOs did not admit their failure.
About half of Israeli Enterprises chose this software product. The CIO said: "We failed same as everybody in the Israeli IT".
A thought flashed in my mind: Should I scream not everybody? My clients, who followed my recommendations, did not fail. They did not chose that software product.
The Software Product was an excellent product for small systems, but not applicable to Large complicated CRUD systems with hundreds or thousands users.
By design, it was not Scalable.
The process of the product assimilation is a long process: studying, building a Pilot, experimenting and building simple applications. Only afterwards CRUD type applications were developed.
Usage of these CRUD applications was gradual i.e. in the beginning the application was implemented with limited number of users.
Only those who deployed CRUD applications with hundreds of users experienced the consequences of the Scalability limitations.
Those who failed used a kind of Wisdom of the Crowd. They asked for their colleagues opinion. They received positive feedback because their colleages' implementations were still in early stages therefore they chose this Strategic Product.
My approach was different: I read Analyst's Research Notes and discovered the inherent Scalability limitations. I also looked for feedback from USA Early Adopters.
As expected, all early Adopters failed in implementing large scale applications.
It was easy to recommend to consider other alternatives.
Factor 2: Unique Characteristics
The "confederated" answers to some of the line similarity tasks were wrong and identical. 75% of the participants conformed with wrong "confederated" answers.
We should expect higher conformity in case of more difficult questions.
As far as Wisdom of the Crowd is concerned, expect identical answers in similar situations, i.e. Social Interactions between people answering the question.
Factor 1: Information Technology example
More than a decade ago, an Insurance Company's CIO presented in a local IT conference. The CIO was brave: He publicly admitted a colossal failure of selecting and implementing a strategic Software product.
He admitted a waste of hundred thousands of USDs and few years due to this mistake.
He was not the only one to commit this mistake: many others made the same mistake and get similar results. The only difference was that other CIOs did not admit their failure.
About half of Israeli Enterprises chose this software product. The CIO said: "We failed same as everybody in the Israeli IT".
A thought flashed in my mind: Should I scream not everybody? My clients, who followed my recommendations, did not fail. They did not chose that software product.
The Software Product was an excellent product for small systems, but not applicable to Large complicated CRUD systems with hundreds or thousands users.
By design, it was not Scalable.
The process of the product assimilation is a long process: studying, building a Pilot, experimenting and building simple applications. Only afterwards CRUD type applications were developed.
Usage of these CRUD applications was gradual i.e. in the beginning the application was implemented with limited number of users.
Only those who deployed CRUD applications with hundreds of users experienced the consequences of the Scalability limitations.
Those who failed used a kind of Wisdom of the Crowd. They asked for their colleagues opinion. They received positive feedback because their colleages' implementations were still in early stages therefore they chose this Strategic Product.
My approach was different: I read Analyst's Research Notes and discovered the inherent Scalability limitations. I also looked for feedback from USA Early Adopters.
As expected, all early Adopters failed in implementing large scale applications.
It was easy to recommend to consider other alternatives.
Factor 2: Unique Characteristics
When a problem includes unique characteristics, many people will ignore them and try to solve it using the same approach they would use for standard similar problems.
Wisdom of the Crowd approach data, will support a wrong intuitive solution pointed by most people surveyed.
Wisdom of the Crowd approach data, will support a wrong intuitive solution pointed by most people surveyed.
Factor 2: The Monty Hall Dilemma
The Monty Hall problem in the Let's Make a Deal TV game is a classic example.
most of the votes, using Wisdom of the Crowd, will assign equal probabilities to a car behind the first and a car behind the third door. These votes support wrong decision.
Factor 2: Information Technology example
This example is also old one: more than a decade ago.
A client (An Enterprise) asked me to help him in choosing an Operating System, for migrating or developing his most crucial application.
I have to recommend UNIX or proprietary Operating System.
As in most cases, there were constraints.
The following constraints were applicable to this task:
1. Short Time for implementing the application -Completing the System development beyond the schedule was impossible.
2. The vendor which should provide the Server and Operating System was predefined.
3. I had to finish the task in two days.
 |
| The Monty Hall Problem. Source: Wikipedia |
A participant choses one door. Goats are behind two doors and a new car behind the third. He gets what he had chosen. However, before checking the door he had chosen, the game host opens another door discovering a goat behind it.
The participant can change is mind and pick the third door instead of the door which had been chosen.
Intuitively, people conclude, that the probability that the car is behind the original door chosen is 0.5. Many participants do not bother to change the door had been chosen. Wrong conclusion: It is a Bayes's theorem probability, therefore the probability of a car behind the third door is approximately 0.67.
most of the votes, using Wisdom of the Crowd, will assign equal probabilities to a car behind the first and a car behind the third door. These votes support wrong decision.
Factor 2: Information Technology example
This example is also old one: more than a decade ago.
A client (An Enterprise) asked me to help him in choosing an Operating System, for migrating or developing his most crucial application.
I have to recommend UNIX or proprietary Operating System.
As in most cases, there were constraints.
The following constraints were applicable to this task:
1. Short Time for implementing the application -Completing the System development beyond the schedule was impossible.
2. The vendor which should provide the Server and Operating System was predefined.
3. I had to finish the task in two days.
During this time frame I had to find data, read data, meet the local distributor's experts and the IT department people responsible for the application.
In addition, an ongoing process of choosing the Strategic Operating system was not completed. I was the leading advocate of UNIX. Others support proprietary Operating Systems.
Factor 2: Information Technology example Recommendation
When I discussed the issue with the vendor's experts and read the vendor's papers, I discovered that most (not to say all), users world wide facing the same decision selected UNIX.
There were hundreds of Enterprises which preferred UNIX. No evidence of enterprises choosing the proprietary Operating System was available.
It was easy for me to conform to the Wisdom of the Crowd and chose UNIX. Choosing UNIX, would also support the UNIX agenda, I advocated for the enterprise.
I recommended usage of the proprietary Operating System.
My paper included the evidence that most of the other Enterprises selected UNIX.
It also describe the uniqueness of my client's enterprise supporting selection of the proprietary Operating System.
Factor 2: Information Technology example unique characteristics
The unique characteristics were limited UNIX knowledge and experience, as well as a lot of knowledge and experience in the proprietary environment. Coupling the knowledge (propriety) and the lack of knowledge (UNIX) with the System importance and schedule constraint, will result in high Risk UNIX implementation.
The risk of deploying the application after schedule, in a relatively good scenario, and of failing to deploy it appropriately, in a less optimistic scenario, supported non-UNIX implementation recommendation.
Summary
It is easy to find other examples in Information Technology, as well as in other fields (as I did in my Hebrew post) in which usage of Wisdom of the Crowd could cause wrong decisions.
You can not count on the crowd, when there is strong Social Conformity.
If their are Unique characteristics, the crowd may ignore them and will recommend a wrong decision.
Even if you ask many people and few of them will perceive the uniqueness and recommend the right solution, the Wisdom of the Crowd data gatherer could reach a wrong conclusion, by preferring the majority opinion.
In addition, an ongoing process of choosing the Strategic Operating system was not completed. I was the leading advocate of UNIX. Others support proprietary Operating Systems.
Factor 2: Information Technology example Recommendation
When I discussed the issue with the vendor's experts and read the vendor's papers, I discovered that most (not to say all), users world wide facing the same decision selected UNIX.
There were hundreds of Enterprises which preferred UNIX. No evidence of enterprises choosing the proprietary Operating System was available.
It was easy for me to conform to the Wisdom of the Crowd and chose UNIX. Choosing UNIX, would also support the UNIX agenda, I advocated for the enterprise.
I recommended usage of the proprietary Operating System.
My paper included the evidence that most of the other Enterprises selected UNIX.
It also describe the uniqueness of my client's enterprise supporting selection of the proprietary Operating System.
Factor 2: Information Technology example unique characteristics
The unique characteristics were limited UNIX knowledge and experience, as well as a lot of knowledge and experience in the proprietary environment. Coupling the knowledge (propriety) and the lack of knowledge (UNIX) with the System importance and schedule constraint, will result in high Risk UNIX implementation.
The risk of deploying the application after schedule, in a relatively good scenario, and of failing to deploy it appropriately, in a less optimistic scenario, supported non-UNIX implementation recommendation.
Summary
It is easy to find other examples in Information Technology, as well as in other fields (as I did in my Hebrew post) in which usage of Wisdom of the Crowd could cause wrong decisions.
You can not count on the crowd, when there is strong Social Conformity.
If their are Unique characteristics, the crowd may ignore them and will recommend a wrong decision.
Even if you ask many people and few of them will perceive the uniqueness and recommend the right solution, the Wisdom of the Crowd data gatherer could reach a wrong conclusion, by preferring the majority opinion.
