When should I start training at Oblivion 4?

I can sorta train at O4, but mostly I train at either O2 or O3 now. That annoying defense buff that skelegons cast makes 'em slow for me