Restrictions on the trade of goods seem to be less severe and more easily overturned than restrictions on labor mobility. We buy a lot of stuff from China and Mexico, but Chinese people and Mexicans have to jump through all kinds of hoops to come here, and in many cases they aren't allowed to. And that's well over and above the larger costs of transportation. Container shipping for goods is cheap, but if people come over that way something has gone horribly wrong. So transit costs are higher for labor mobility than for mobility of goods, and on top of that, governments add extra regulatory barriers.
I'm seeing this as an instance of the organized power of businesses to influence regulation, which the rest of us as poorly organized individuals don't have.