Can't you just use a multimap instead?