Especifique la condición como parte de la cláusula ON:
SELECT m.id, u.first_name AS otherUser
FROM matches AS m
LEFT JOIN users AS u ON (u.id=m.user2ID and u.id = m.user1ID) or (u.id<>m.user2ID and u.id = m.user2ID)
WHERE m.user1ID=2 OR m.user2ID=2
Otra forma de hacer lo mismo:
SELECT m.id, u.first_name AS otherUser
FROM matches AS m
LEFT JOIN users AS u ON IF(u.id=m.user2ID,u.id = m.user1ID,u.id = m.user2ID)
WHERE m.user1ID=2 OR m.user2ID=2