¿Qué tal:
select datepart(yyyy, [date]) as [year]
from shoptransfer
group by datepart(yyyy, [date])
O:
select count(*) as qty, datepart(yyyy, [date]) as [year]
from shoptransfer
group by datepart(yyyy, [date])
order by [year]
Esto se basa en el comando de OP:"Quiero agrupar por año parte de la columna de fecha (varchar)"