Debe cambiar su código para usar decimales de la siguiente manera
SELECT
dst,
AVG(sp)
FROM
(
SELECT
dst,
cast(sp as decimal(5,2)) sp,
ROW_NUMBER() OVER (
PARTITION BY dst
ORDER BY sp ASC, id ASC) AS RowAsc,
ROW_NUMBER() OVER (
PARTITION BY dst
ORDER BY sp DESC, id DESC) AS RowDesc
FROM dbo.cars SOH
) x
WHERE
RowAsc IN (RowDesc, RowDesc - 1, RowDesc + 1)
GROUP BY dst
ORDER BY dst;
Actualmente, el comando AVG se ejecuta en un int, por lo que el resultado es un int