Ok, so first off, a little background.

I help manage around 3000 Linux servers across 5 data centers worldwide. Some of the admins on my team have tried puppet before I joined on, but ran into scalability problems with the puppet master when reaching around 1000 servers. Since that time most of our change and deployments have been very manual, involving bash for loops and dsh. I've looked a little bit at chef, cfengine, and bcfg2. My questions are these.

What experience has everyone else had with any of these or other configuration management tools (pros/cons/cost etc...)? How well did they scale to thousands of servers? What recommendations do you seasoned professionals here on Antionline recommend or have come across in your various jobs?

I and others on our team have been toying with the idea of creating our own in-house config management tool, but if someone has already gotten something written that will do what is needed, I'd rather not re-invent the wheel.