what numbers count?

I've raised doubts before of the wisdom of letting schoolkids use calculators rather than learning how to do arithmetic tasks like long division, but now I'm willing to consider a radical change in the other direction -- dropping the requirement that all high school or college graduates need advanced mathematics like algebra and calculus.  A thoughtful opinion piece in the New York Times by a political science professor, Andrew Hacker, makes the case.

Hacker notes that failure in math classes correlates with failure to graduate from high school and later college. In effect, we are creating an insurmountable barrier for otherwise educable young people.

I want to hear more on the idea, for it sounds plausible, especially since Hacker still calls for numeracy, for schoolkids to learn how to handle finances and understand public policy issues expressed with statistics and other numbers. He argues that higher math just isn't necessary for ordinary functions and jobs of most citizens. I agree with the need to require numeracy, but wonder how much most of us need.