> When is it ok to access class variables directly?
Probably never - but as with all things programming, exceptions to the rule are possible, and may even be preferred.
Your average 1-liner set/get is going to be implemented very efficiently anyway, so you should do this by default.
Plus if your set/get ever needs to become something else, the change to the internal implementation only has to be done in one place.