Why is the 2s orbital lower in energy than the 2p orbital when the electrons in 2s are usually farther from the nucleus?

My chemistry book explains that even though electrons in the $\mathrm{2p}$ orbital are closer to the nucleus on average, electrons from the $\mathrm{2s}$ orbital spend a very short time very close to the nucleus (penetration), so it has a lower energy. Why does this tiny amount of time spent close to the nucleus make such a big difference? It seems like it should be the average distance that matters, not the smallest distance achieved at any one point, in determining stability. What makes that momentary drop in energy so important that it is outweighs all the time spent farther away from the nucleus with a higher energy?