I don't understand your algorithm.
But your innermost loop accesses elements that don't exist

for(l=(j+1) ; l<NUM && flag!=0; l++)
if(mat[i][j]!=mat[i][j+l])

btw: using l as a variable...