shemoneh esrei - What if, when the Kohanim washing their hands, less than ten remain?

Before Birkas Kohanim, the Levi'im wash the Kohanim's hands. In many communities, this occurs during the chazzan's repitition of Shemoneh Esrei, and in a room outside the main shul.

Normally, there need to be nine people paying attention to the chazzan's repetition. What if, during the hand-washing, less than ten people remain in the shul? Should the chazzan pause until they come back?