Puedes hacer esto con AND y OR siempre que use suficientes paréntesis.
También asumo que activities.OrderBy puede ser nulo. Si ese no es el caso, puede eliminar la verificación nula:
SELECT activities.*,
activitytypes.orderby
FROM activities
LEFT OUTER JOIN activitytypes
ON activities.typeid = activitytypes.typeid
WHERE activities.userid = 86
AND activities.typeid NOT IN ( 5, 10, 11, 12, 19 )
AND ( ( activities.orderby = 1
AND activities.starttime >= '2013-08-26 04:00:00'
AND activities.endtime <= '2013-08-27 04:00:00' )
OR ( ( activities.orderby IS NULL
OR activities.orderby != 1 )
AND activities.activitydate = '2013-08-26' ) )
ORDER BY activitytypes.orderby,
activities.starttime
Alternativamente, si aún desea usar CASE , solo necesitas cerrar tu CASE declaración usando END , así:
SELECT activities.*,
activitytypes.orderby
FROM activities
LEFT OUTER JOIN activitytypes
ON activities.typeid = activitytypes.typeid
WHERE activities.userid = 86
AND activities.typeid NOT IN ( 5, 10, 11, 12, 19 )
AND (
CASE
WHEN activities.orderby = 1 THEN
activities.starttime >= '2013-08-26 04:00:00' AND activities.endtime <= '2013-08-27 04:00:00'
ELSE
activities.activitydate = '2013-08-26'
END
)
ORDER BY activitytypes.orderby,
activities.starttime
