It's a lot easier to just keep track of the *tangent vectors* in order to do billboards. I.e, instead of doing all of that matrix stuff for every billboard, you just rotate the local coordinate system by the current angles of the view vector. Then, you just specify the coordinates of a given billboard in terms of magnitudes of the tangent vectors.
Ultimately, this is MUCH faster than applying the matrix terms you have specified (assuming you have a lot of billboarded objects as you do).
Also, this is how most graphics apis actually rotate the scene...they simply rotate the coordinate system by using a matrix, but then the verticies are just specified in terms of the unit vectors...big speed boost.