I have seen delays getting the links up on our systems. I have not investigated it since it is not an issue.
Have you tried adjusting any of the subnet manager configurations? I assume that you have the SM running on the switch? Perhaps there are sweep timings or caching that can be adjusted to improve the startup time.
Sean