There is bipartisan agreement that government is a big part of the housing affordability problem. It is time to act.