I figured out the problem -
Its mainly because after the sync from client -> server, the sent anchor was updated. When sync from server -> client happens a new record got added after the recently updated sent anchor - hence the next time the client to server sync happens, the same record is tried to be sent to server again and that causes a conflict.
The main reason for this is we were not using originator id for filterig out these records.