Number theory in 21 minutes

I'm a fan of riddles with mathematical or programming background ("one guard always tells the truth, one guard always lies, and the third guard stabs people who ask tricky questions"). Maybe you like those too. Anyway, this is my favorite, solvable by anyone who finished school:

p^2 - 1 is divisible by 24 if p is a prime number greater than 3.

Example:

5^2 - 1 = 1*24

7^2 - 1 = 2*24

11^2 - 1 = 5*24

Why is that?

Enjoy,

Philip