The man page for perfquery states that only the data counters are divided by 4. It doesn't say anything about packets counters.
Note: In PortCounters, PortCountersExtended, PortXmitDataSL, and PortRcvDataSL, components that represent Data (e.g. PortXmitData and PortRcvData) indicate octets divided by 4 rather than just octets.
For RoCE adapters the counters do not count regular Ethernet frames, they only count Infiniband. However, the switch will count everything. So maybe that's the reason why port_rcv_data is smaller than the switch counters?