mysql cte. usar with e insertarlo alerta de un error de sintaxis

Usa insert . . . select :

INSERT INTO flow_instances (custom_id_year, custom_id_counter)
WITH this_year AS (
      SELECT YEAR(CURDATE()) as this_year
     ),
     max_val AS (
      SELECT COALESCE(MAX(custom_id_counter), 0) as max_val
      FROM flow_instances AS max_val
      WHERE custom_id_year = YEAR(CURDATE())
     )
    SELECT ty.this_year, mv.max_val + 1
    FROM this_year ty CROSS JOIN
         max_val mv;

Debe hacer referencia a los CTE para poder utilizar los valores que definen.