What are the main differences between cross validation and permutation tests

Both cross validation and permutations are used for performance evaluation of classification models.

  1. Permutation tests provide a binary (qualitative) answer to “is my model better than null (random) models?”. Here the baseline reference are null models - often generated using the same algorithm trained on permuted group labels, or permuted data.

  2. Cross validations provide quantitative answer to “how good is my model compared to other models?”. The baseline reference are other models - often generated using the different algorithms trained on the same data.

As you can see, permutation tests is the starting point to see if the model is of any use (i.e. capture any true signals). If so, you can further use CV to find out whether it is better than other models