Which p-values should be reported from mummichog? FET or gamma-adjusted? My understanding is that the FET are “raw” p-values, while gamma adjusted are “corrected” for the multiple permutations. Is one “more” correct?
This is a very good question. We have updated the tool tips and the result table - in the hope to clarify the things
- There are actually 4 types of p values computed for mummichog (available in the download table), including
- One-tailed Fisher Exact Test (very sensitive but based on Gaussian hypergeometric probability distribution)
- EASE score (a conservative adjustment to the Fisher exact probability that weights significance in favor of themes supported by more hits)
- Empirical p values based on non-parametric permutations - p value is the percentage of hits received better than original;
- Gamma p values - using Gamma distribution to model permutation results
In general, Fisher p is very sensitive but could give a lot false positives, while empirical p is robust but does not have a lot power (non-parametric), while gamma p tries to combine both worlds …
- The web interface shows the original p values as described above (not adjusted). The multi-testing adjusted p values are available in the download table. There are two considerations:
- One main interest is to identify the most “perturbed” pathways for hypothesis generation. We notice that in many times the adjust p values are all 1, making them uninformative for this purpose.
- Many peaks or compounds are shared among different pathways (i.e. pathways are correlated), violating the assumption of independent tests.
We suggest not to rely solely on statistical procedures, but combine this evidence with more biological context.