IMHO as I'm still new here, but I have noticed that some perfectly good(good as in neither deserving positive/negative) get hit for no reason. One thread like that came up today, it's here.
And then I've actually seen good and factual posts get negged (who knows why), but totally undeserving. Not just 2 or 3 or 4 negs, I'm talking very negative and extremely negative.
The AP system seems to be doing it's job, but I think,(again MHO ), the easiest way to limit abuse would be to have "AP Moderator" for lack of a better word. Yes some people will always come whining saying how they got negged for no reason etc, but one could put a limit on how many grievances you could bring, in say a month. Those who keep coming back when it appears that they dont have a case or whatever, the moderator could give them a warning.
As for people using their points unfairly, if a valid complaint is found, the abuser could be warned, and if it happens again, he/she starts losing one green per new abuse...