Too many affiliate links?

So I was wondering how many affiliate links you put on your website or article do avoid being penalized by Google. I currently don't put affiliate links (I just work with Amazon right now, besides AdSense) on every single article, but in some articles I have affiliate links in both text and pictures. How many is too many? Is there a rule?