The Benefit of Being Shallow