¿Cómo puedo dejar de unir estos #temp (datos predeterminados) y otra tabla?

Es su cláusula where, lo que filtra algunos intervalos de tiempo. Debe establecer esa cláusula para la selección interna:

select Count(VisitingCount) as VisitingCount,[Time]      
from   
#temp as Alltimes
left outer join   
( SELECT Page,Date,[user],      
        dbo.fn_GetActivityLogArranger2(Date,'hour') as [Time]
        FROM scr_SecuristLog      
        where Date between '2009-04-30' and '2009-05-02'      
        and      
        [user] in      
        (       select USERNAME      
                from scr_CustomerAuthorities      
                where customerID=Convert(varchar,4)      
                and ID=Convert(varchar,43)      
        )      
) scr_SecuristLog      
on Alltimes.[Time] = scr_SecuristLog.[Time]
group by [Time] order by [Time] asc