Subnet manager is the control plane of the fabric. It assigns addresses (L2 LID address) and establishes routing tables.
It will assign a Layer 2 identified, not an IP address. If the port is active, it should have an address. IB managed and unmanaged switches have the same functionality except ability to run SM. In addition, managing a managed switch is easier.
If you can run SM on a server, the QDR switch you have must suffice.
Anand