Así es como incluí todos los modelos necesarios, esta consulta se traduce a 5 consultas sql ya que la precarga no se une (se ejecuta en una consulta separada).
Message.joins("LEFT JOIN messages AS m ON messages.id != m.id
AND m.conversation_id = messages.conversation_id
AND messages.created_at < m.created_at")
.where('m.id IS NULL')
.joins("INNER JOIN conversation_participants AS cp
ON cp.conversation_id = messages.conversation_id
AND cp.user_id = #{user_id}")
.order("cp.seen, cp.updated_at DESC")
.limit(5)
.includes(:sender)
.includes(conversation: [{conversation_participants: :user}])