Hi mahesh,
mlx4 is the low level driver for Mellanox ConnectX device family (4th generation adapters), first introduced in 2007:
RFC: "mlx4" drivers for Mellanox ConnectX HCAs [LWN.net]
While mthca, is the driver for the older generation: InfiniHost device family.