Google penalty – How long does it take to recover after fixed?

If Google penalizes your website, for keyword stuffing for example, how long can it take for your website to recover after you fix the issue?

Or does Google ban you forever? That wouldn't make much sense…