How you recover after you get into this state? Does it recover itself?
"Lost" packets should leave some clues somewhere: do you see Rx/Tx errors or discards on hosts (with ifconfig for example) or on the switch ports you are using?
Other thing to check if you still have IP address assigned and ports are up on both ends when that happens: network manager is known to reset manually set IP config if it is enabled with DHCP on the interface - it would saddenly drop manually configured IP address.