My Experiences....

Both Physical and IT Security Operate within the IT divison, reporting directly to the division head rather then within the infrastructure or applications branches.

Originally Physical (corporate) security was part of a different divison - corporate division but they were brought together because of the reasons Gandalf mentioned about having the two security teams together so they can use each others knowledge.

This works well for us, we are independent enough of the IT branches to be considered independent but we are close enough to have our fingers of the pulse of what is going on and what the IT areas are doing (I am in IT Security) which I think is important.

My organisation is small enough to not have an audit/compliance department which IMHO is a good thing because IT Security would logically go in that bucket which would keep us away from IT and make it harder for us to know what is going on.