Esta consulta debería darte el resultado requerido.
SELECT category
,tran_type
,SUM(IF(month(date) = 1,ABS(amount),0)) as jan_total
,SUM(IF(month(date) = 2,ABS(amount),0)) as feb_total
,SUM(IF(month(date) = 3,ABS(amount),0)) as mar_total
,SUM(IF(month(date) = 4,ABS(amount),0)) as apr_total
,SUM(IF(month(date) = 5,ABS(amount),0)) as may_total
,SUM(IF(month(date) = 6,ABS(amount),0)) as jun_total
,SUM(IF(month(date) = 7,ABS(amount),0)) as jul_total
,SUM(IF(month(date) = 8,ABS(amount),0)) as aug_total
,SUM(IF(month(date) = 9,ABS(amount),0)) as sep_total
,SUM(IF(month(date) = 10,ABS(amount),0)) as okt_total
,SUM(IF(month(date) = 11,ABS(amount),0)) as nov_total
,SUM(IF(month(date) = 12,ABS(amount),0)) as dec_total
FROM transactions
WHERE YEAR(date) = '2011'
GROUP BY category, tran_type
No olvides filtrar por año si no quieres tener problemas.