The anomaly isn't so much with the definition of infinity, or division
by zero, but with zero itself

Zero behaves differently from other numbers. The idea of zero is synonymous with absence, so I choose to view the set of all positive integers, and zero as opposite ideas rather than considering zero to be a distinct number like 1, 2 or 3. In set theory, the idea of a null (or empty) set, is a similar idea. I emphacise that it is the number zero, not the idea, that I choose to eliminate. Presence is the opposite idea of zero and in the realm of numbers, zero reflects the absence of numbers. In otherwords, something either exists, or it doesn't. If it exists, then it has a quality that we call number associated with it, and if it doesn't exist we call this absence, zero.
http://my.tbaytel.net/forslund/zero.html

We treat zero like any other integer when we put it in an equation (because it is so useful)
but it has unusual properties that lead to strange conclusions sometimes.