The cross-party Business Innovation and Skills Committee said most of those who would pay the tax would not benefit from the faster broadband service.
"We believe that a 50p levy placed on fixed telecommunication lines is an ill-directed charge," says the committee. "It will place a disproportionate cost on a majority who will not, or are unable to, reap the benefits of that charge."
Instead, the focus should be on providing basic broadband for all and allowing markets to deliver higher speeds, the committee argues.
However, the Government says the 50p levy – outlined in its Digital Britain report – remains "the best way to drive further investement".