Found this issue last week and wanted to note it for future reference for myself (getting old and best way for me to remember as much as anyone:)).
So removed a VM from one of our datastore clusters. Now this machine had an affinity rule applied to it which i wasn’t aware of. Interestingly found i couldn’t remove the rule even wanted to from the datastore cluster.
Turns out the solution was slightly inelegant but it worked.
i) Moved said VM back into the datastore cluster
ii) In the vSphere client (or web) browse to the datastore cluster.
iii) Click the Manage tab and select Settings
iv) Under Configuration, click Rules
v) Pick out your offending rule you want deleted and click Remove and click Ok