Just wanted to thank you for the comment and clarify I didn't mean to criticize your choices - my background is very similar and it's just a realization I've had after stepping back and trying to think without those constraints/influences. I also know many people who lost houses or struggled, some which actually did cash-out refinances but then unwisely spent the money on unnecessary luxuries. Those seem easy to avoid. Some may be harder to avoid, like when someone has unforeseen costs such as medical related bills. But in cases such as ours, it seems we are well enough off to have cash on hand to eliminate the mortgage; the question just becomes whether that is the best use for the money. It certainly seems a bad idea to just keep the cash on hand. The index fund returns have been very good for long periods of time now, so seem like a good low-risk option, given that they are liquid and can be redirected to a mortgage payoff at any time.

Edit: having said that the difference is not that large (3-4% for the 30 year note, vs. 5-10% for the market return). Also, while I didn't pay off my mortgage, I probably won't put even more money where my mouth is and refinance in order to invest the cash-out into a fund.