As long as your prime numbers fit in the L1 (or L2) cache of the processor, it's faster to test only prime numbers, because only roughly one in three odd numbers is a prime (one in three odd numbers is divisible by 3, and only one of the other two are most of the time a prime), and reading an one value out of an array and dividing by that is quicker than performing two divide operations on ALL modern processors.
If you have a HUGE number of primes, then it's better to continue from the largest that fits in the cache with every other number, since just incrementing a number is faster than reading from the memory outside of the processor.
--
Mats