It's the Labels that are Confining!