HI,
I haven't tested RS8264 switch connectivity with Spectrum. however simply try to change the port speed on the relevant port to 40G
run (for example on interface eth 1/1):
switch (config) # interface ethernet 1/1 speed 40000
and check if that works.
run
switch (config) # show interfaces ethernet status