Americans outside the market are losing faith in the economy