¿Cómo obtener el número máximo de filas por grupo/partición en SQL Server?

Prueba esto:

SELECT a.*, CASE WHEN totalPayments>1 THEN 'NO' ELSE 'YES' END IsFirstPayment
  FROM(
                SELECT  p.payment_id,     
                                p.user_id,     
                                ROW_NUMBER() OVER (PARTITION BY p.user_id ORDER BY p.payment_date DESC) AS paymentNumber,
                                SUM(1) OVER (PARTITION BY p.user_id) AS totalPayments
                    FROM payment p 
            ) a
WHERE   paymentNumber = 1