You should try to implement the algorithm I've already described:

Counting opening and closing brackets does not help one bit. ([)] has the right number of brackets, but they are not nested properly.For example:

- read one character at a time

- for each opening bracket, push the corresponding closing bracket on the Stack

- for each closing bracket:

if the Stack is empty => ERROR

if the popped value is not the same as current character => ERROR

- when no more input the Stack should be empty, or there were unmatched opening brackets.

You need to understand what the stack is for:

Step 1: Character is "(". Push ")" - the corresponding bracket we'll be expecting. Stack now contains ")".

Step 2: Character is "[". Push "]". Stack now contains ")]"

Step 3: Character is ")". Value popped from the stack is "]". There is a mismatch, hence the expression is incorrect. Return false.