GSEA is a statistical method that determines whether a predefined gene set (GO, KEGG, etc) demonstrates statistically significant difference between two groups. Taking as input a list of ranked genes and a gene set, it looks at whether the genes from the gene set are randomly distributed in the ranked list or significantly enriched in the top and bottom extremes of the ranked list.
In the following schema, the gene set A is significantly enriched, while gene set B represents a case where the genes are more randomly distributed.
In contrast to over-representation analysis (ORA) which is based on significant genes only, GSEA considers the complete ranked gene list, and is claimed to be able to detect subtle but coordinated changes. For more details, please refer to the original publication.