In real world, Cathay Pacific is flying B773ER from HKG to JFK
but when I check in AS, the payload is restricted to 52%
does a pay load restriction means if the seat config is Y300C50
52% payload means Y150C25?
The Cathay Pacific aircraft have higher mtows which is not incorporated in AS yet. But C50Y300 should be ok for HKG-JFK-HKG, there is only a load restriction for the cargo hold.
You gotta calculate yourself.
Each passenger weight 90kg. Then you can see how much passenger you carry. The remaining would be cargo.
When I use the Aircraft type evaluation tools, B777300ER has a profit of -89,800 (-46%)
does it mean it is not wise to start such route?
depends on seating configuration and the price you used then evaluating the flight.
if you’re the only airline flying, no indirect competitors either and it’s a high demand route, you can probably charge 150% thus potentially making it lucrative
I’d trust the AS figures and assume its fairly foolish to start HKG-JFK with a 300ER, you’d be much better off perhaps with the 200LR
Your best option for a 777, based on the performance evaluation tool, is the 777-200ER, but the 787 kicks its butt too 
Use EcoPlus rather than standard seat. And then adjust the %, then you could see some profit.
And yes, 787 seems to be a better choice.
As far an I know cathey are flying their newer ac on this route.
The newer version of the 77w has a higher takeoff weight due to better performance of the engines. They are can fly further and use less gas than the earlier models of the 77w. This is why cathey can fly hkg-jfk, and i guarantee there is no pay load restriction for a fully sold flight including cargo.