The ternary operator is being used to compute the first index.
So (assuming index is of an appropriate integral type) the code is equivalent to
Code:
/* first expand the ternary operator */
if ((RANKS[square] - 1) < (FILES[square] - 1))
index = RANKS[square] - 1;
else
index = FILES[square] - 1;
/* now plug that into the rest of the if() statement */
if (GEN_SLIDING_ATTACKS[index][state6Bit] & CHARBITSET[attackbit])
/* whatever */
Note that I've done a literal interpretation of the code .... plenty of opportunities for making it simpler.
Whoever wrote that code was more interested in getting everything into one statement, than in clarity of code. IMHO, that has demonstrated their foolishness, no more.