Well, in this case Notes did what you asked it to.
If you “Remove” an invitee, they are removed from the meeting. It was not the “send notice to all invitees..” that got you, but the fact that you did the Remove action.
The correct way to do this would be to simple add the new invitee. It never hurts to let the Notes client send the notice to all invitees.