The ternary operator is being used to compute the first index.
So (assuming index is of an appropriate integral type) the code is equivalent to
Note that I've done a literal interpretation of the code .... plenty of opportunities for making it simpler.
/* first expand the ternary operator */
if ((RANKS[square] - 1) < (FILES[square] - 1))
index = RANKS[square] - 1;
index = FILES[square] - 1;
/* now plug that into the rest of the if() statement */
if (GEN_SLIDING_ATTACKS[index][state6Bit] & CHARBITSET[attackbit])
/* whatever */
Whoever wrote that code was more interested in getting everything into one statement, than in clarity of code. IMHO, that has demonstrated their foolishness, no more.