Past a certain savings threshold, there's no need to keep your emergency fund / home savings out of the stock market

Contentious statement I know but hear me out. Say you want to maintain a $20,000 emergency fund and a $100,000 reserve for a down payment on a home in a year or two. The conventional advice is to put this in a high yield savings account or some similar financial instrument. However, over a certain threshold of savings, this isn't necessary anymore. On average you will be losing out of potential gains from the market, and even if there was a catastrophic crash, provided you held a diversified por