BayesTheorem

Long introduction

Bayes theorem:

• notations: p(something) is the probability that something (called an event) occurred, "A|B" denotes the event "A knowing that B occurred", with A and B themselves events.
• p(A|B) = p(A/\B)/p(B): the probability that A happens, knowing that B happend
• with a little calculus, we also get p(A|B)p(B) = p(A/\B) = p(B|A)p(A)
• otherwise interpreted: if we know the probability that A has to occur, make a test or an experiment B, then we can compute a revized probability that A occurred. Bayes theorem makes a link between a priori and a posteriori probabilites.

Detailed example: think about a population of 1000 people, amongst which 1% has a certain type of cancer. Assume that some laboratory has invented a test such that:

• if you're sick, you have 80% of chance to have a positive reaction to the test
• if you're healthy, you have 10% of chances to have a positive reaction to the test

Question: if you have a positive reaction to the test, what are your chances to really suffer from this cancer ?

The answer lays in Bayes' theorem. Let call A the event "you are sick" and B the event "you have a positive reaction to the test". Then, we can compute that the wanted probability is roughtly 7.48%, very far from what one thinks when one has a positive test. (For more details: note that p(B) = 0,107 = p(B /\ A) + p(B /\ not A) = (99 + 8) / 1000 and apply Bayes' theorem. Equivalently, there is 990 healthy people, amongst which 99 will have a positive reaction, while 8 out of the 10 ill people will have also a positive reaction ; therefore you have 8 chances out of 107 (99 + 8) to be really sick).

Of course, the probability has increased, but only from 1% to 7,5%. And this is the case for every test we can develop. This is why we have to be careful in developing tests and experiments. A well-known problem concerning this is the notion of bias which is a very formal notion in statistics.

Reflexion on this basis: how to prove that CI has a positive effect ?

In order to convince your partners of the positive effects of CI, we have to think about tests, introducing the less bias possible. The main point is:

• if you are a partisan of CI, then you better have to measure the degree of failure of your project
• and conversely if you are an anti-CI

So, always search to invalidate your opinion by playing a proponent/opponent game. This will paradoxally enhance the good points as well as reinforce your position. Also, do this because you always tend to see only the points that are in favor of you opinion: on the same mode, a programmer should never test its own code, since he is already convinced that there is no bug.

The strength of a theory (or a program) is always evaluated by:

• the capacity it has to resist to experiments and tests (e.g. Newton's theory does not explains Mercury's orbital. Relativity does.)
• the capacity it has to predict events (e.g. the existence of aliens may explain everything, but does not predict anything)

Good arguments can also be:

• money spent on the project (a posteriori argument)
• a huge percentage of projects fail. Compare it with the percentage of projects that use CI methods.

Bibliography

• lesswrong.com
• overcomingbias.com
• Karl Popper work