According to my grandpa, back in the day those that didn't work would attend contests in which the prize was always bread. Jewish rye and sourdough were cream of the crop.
What Is The Origin Of The Word Breadwinner?