Linux Config Management

Ok, so first off, a little background.

I help manage around 3000 Linux servers across 5 data centers worldwide. Some of the admins on my team have tried puppet before I joined on, but ran into scalability problems with the puppet master when reaching around 1000 servers. Since that time most of our change and deployments have been very manual, involving bash for loops and dsh. I've looked a little bit at chef, cfengine, and bcfg2. My questions are these.

What experience has everyone else had with any of these or other configuration management tools (pros/cons/cost etc...)? How well did they scale to thousands of servers? What recommendations do you seasoned professionals here on Antionline recommend or have come across in your various jobs?

I and others on our team have been toying with the idea of creating our own in-house config management tool, but if someone has already gotten something written that will do what is needed, I'd rather not re-invent the wheel.

Wow....so no one uses any kind of config management solutions for their servers?

Most of the stuff I have seen is geared more towards managing clients than servers.

Quote:

---

Wow....so no one uses any kind of config management solutions for their servers?

---

That does not follow, as the pertinent information you gave us is:

Quote:

---

I help manage around 3000 Linux servers across 5 data centers worldwide. Some of the admins on my team have tried puppet before I joined on, but ran into scalability problems with the puppet master when reaching around 1000 servers.

---

I doubt if anyone here (apart from yourself) handles anything like that number of servers. You guys have encountered scalability problems so I guess we would not feel confident in suggesting solutions that we only know to work in much smaller environments.

Have you considered a tiered or devolved deployment strategy? such as making each data centre responsible for locally applying policies devolved from the mothership?

You might try to make contact with IT people at NASA, government departments, and the military, and ask them what they use/suggest.

Otherwise Google is your friend.............I believe they have the most servers of anyone in the commercial sector...........you need to talk to your oppo at Google?

I suspect that where they have centralisation on your scale, it will be bespoke or in-house developed.

Thanks Nihil...

Quote:

---

I doubt if anyone here (apart from yourself) handles anything like that number of servers.

---

Wow so the way you responded there almost makes it sound like I came off as a bit of a braggart. That was by no means my intention in mentioning the size of our environment, I only wanted to make sure everyone had an understanding of the problems we were running into. :)

Quote:

---

You guys have encountered scalability problems so I guess we would not feel confident in suggesting solutions that we only know to work in much smaller environments.

---

Yeah I guess that makes, I wouldn't feel confident in making a suggestion like that either.

Quote:

---

Have you considered a tiered or devolved deployment strategy? such as making each data centre responsible for locally applying policies devolved from the mothership?

---

Yep this was the direction I was heading for creating an in-house solution, unfortunately we let one of our datacenters get a bit too large. Even to the point where our IPMI cards no longer work due to arp-cache being full :( So we might have "mother ships","father ships", and maybe even some little "child ships". I've also heard stories of people forcing puppet to behave in a similar fashion with crons and just remove puppet-master altogether.

Quote:

---

You might try to make contact with IT people at NASA, government departments, and the military, and ask them what they use/suggest.

Otherwise Google is your friend.............I believe they have the most servers of anyone in the commercial sector...........you need to talk to your oppo at Google?

---

Great, I wonder if NASA or the US government has an employee directory for all their IT grunts I can just start war dialing :D . Seriously though, if anyone on the list has any contact for someone like Nihil is suggesting, please shoot them my way(or ask them and then post back here on the thread). I would be greatly appreciate it.

I also would be interesting in anyone's thoughts or ideas for architecting such a scalable configuration management system. So if anyone's specialty is in designing scalable software please share.

I help menage around 400 servers and will be interested in what limitations have you run in with puppet as i currently use it quite a lot.

Hi ABS,

Quote:

---

Wow so the way you responded there almost makes it sound like I came off as a bit of a braggart. That was by no means my intention in mentioning the size of our environment, I only wanted to make sure everyone had an understanding of the problems we were running into.

---

On the contrary, I thought that you described your environment and questions very succinctly. That is why I suggested a reason for the lack of responses; otherwise I am sure you would have had several replies that you had already checked out. This is a pure scalability issue, right?

Quote:

---

unfortunately we let one of our datacenters get a bit too large.

---

Arrrghhhhhh! I just knew you were going to say that!!! It's always the way in my limited experience...........the business requirements and beancounters' budgets rule to begin with? Then we hit the tech limitations that we told them about, but they didn't listen.:rolleyes:

Now, please don't get me wrong, I'm no expert in this field, but I have encountered it so much in the applications development and deployment arena!!!:( OK, a different area, but the issues are pretty much the same.

You (quite rightly IMO) do not state your business.............not good idea on an open forum?.............and your size suggests you my be considered a competitor in the commercial sector?

That is why I suggested .gov type outfits. They don't have a commercial agenda..................they just waste (our) taxpayers' money as they like? :eek:

NASA might be a good start as I believe that they still lease space (no pun intended) and apps a la cloud computing?

If you use SUSE, or OpenSUSE, you have some options. Novell makes a bunch of stuff even though they were bought last month. But they still apparently have all their stuff. You can always try them though, they're good with Server management.