It is typical to break aligned box problems like this into 2d problems in each of the 3 axis. Cohen-sutherland deals with all the potential line crossings, and you test each of the 3 lines of the triangle. The test is is two phases, you may only need the first phase. The second phase deals with where to clip the line (the intersection of the line and the edge of the rectangle, if it occurs). Cohen-sutherland reduces the problem to determining what 'quadrant' the endpoints are in (including the interior), producing a bitmap of all the possible permutations, then providing a table (usually implemented in a switch) which determine the 'meaning' of a particular pair.
I forget the details, google reveals several discussions, but you'd skip the clipping step. Simply determining the relative position of p1 to p2, p1 to p3 to the square 'decides' that these two lines don't intersect the square, but the relative position of p2 to p3 indicates it does. If that's as far as you require, you're done (in 2D).
p1 | p2 |
p3 | |
You'd have to test the p2-p3 pair against the other axis, but (I forget exactly where I first read this, probably in one of Eberly's books), if the line doesn't cross in 2d, it doesn't cross in 3d. There's a proof on that somewhere, too - but now I'm reaching back over a decade to recall my source.
Is this part of a contact/collision test, or something else?
The general topic you're asking about approaches the tenets of collision/contact (a broad subject about object intersections), and I'd recommend on of the books in the Morgan-Kaufman series. David Eberly gives excellent treatments (he's a PhD in math, and a game developer), but also there's a book devoted to real time collision detection (words to that effect are it's title) from that publisher that go into exquisite detail.