A few good points brought up here by Chadstroma & Koya.
-The only way people are going to ride it is if it's better than the option of driving. Several factors could make a high speed rail more attractive than driving. a) Gas prices are too expensive b) Traffic is too bad c) parking is too expensive
I took an urban planning course last spring and learned that urban planners love congestion. You pretty much need congested chaos for projects like these to make sense.
Another issue is that usually infrastructure is playing catch-up to population growth. In the HSR case, and the Ybor City Trolley, the infrastructure will be there first.
Also, like Koya mentioned you are never going to see direct revenue from an infrastructure project. How much money is made off of an interstate?