Results 1 to 8 of 8

Thread: Linux Config Management

  1. #1
    Junior Member
    Join Date
    Apr 2004
    Location
    United States
    Posts
    24

    Linux Config Management

    Ok, so first off, a little background.

    I help manage around 3000 Linux servers across 5 data centers worldwide. Some of the admins on my team have tried puppet before I joined on, but ran into scalability problems with the puppet master when reaching around 1000 servers. Since that time most of our change and deployments have been very manual, involving bash for loops and dsh. I've looked a little bit at chef, cfengine, and bcfg2. My questions are these.

    What experience has everyone else had with any of these or other configuration management tools (pros/cons/cost etc...)? How well did they scale to thousands of servers? What recommendations do you seasoned professionals here on Antionline recommend or have come across in your various jobs?

    I and others on our team have been toying with the idea of creating our own in-house config management tool, but if someone has already gotten something written that will do what is needed, I'd rather not re-invent the wheel.

  2. #2
    Junior Member
    Join Date
    Apr 2004
    Location
    United States
    Posts
    24
    Wow....so no one uses any kind of config management solutions for their servers?

  3. #3
    Gonzo District BOFH westin's Avatar
    Join Date
    Jan 2006
    Location
    SW MO
    Posts
    1,187
    Most of the stuff I have seen is geared more towards managing clients than servers.
    \"Those of us that had been up all night were in no mood for coffee and donuts, we wanted strong drink.\"

    -HST

  4. #4
    Senior Member nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,188
    Wow....so no one uses any kind of config management solutions for their servers?
    That does not follow, as the pertinent information you gave us is:

    I help manage around 3000 Linux servers across 5 data centers worldwide. Some of the admins on my team have tried puppet before I joined on, but ran into scalability problems with the puppet master when reaching around 1000 servers.
    I doubt if anyone here (apart from yourself) handles anything like that number of servers. You guys have encountered scalability problems so I guess we would not feel confident in suggesting solutions that we only know to work in much smaller environments.

    Have you considered a tiered or devolved deployment strategy? such as making each data centre responsible for locally applying policies devolved from the mothership?

    You might try to make contact with IT people at NASA, government departments, and the military, and ask them what they use/suggest.

    Otherwise Google is your friend.............I believe they have the most servers of anyone in the commercial sector...........you need to talk to your oppo at Google?

    I suspect that where they have centralisation on your scale, it will be bespoke or in-house developed.

  5. #5
    Junior Member
    Join Date
    Apr 2004
    Location
    United States
    Posts
    24
    Thanks Nihil...
    I doubt if anyone here (apart from yourself) handles anything like that number of servers.
    Wow so the way you responded there almost makes it sound like I came off as a bit of a braggart. That was by no means my intention in mentioning the size of our environment, I only wanted to make sure everyone had an understanding of the problems we were running into.

    You guys have encountered scalability problems so I guess we would not feel confident in suggesting solutions that we only know to work in much smaller environments.
    Yeah I guess that makes, I wouldn't feel confident in making a suggestion like that either.

    Have you considered a tiered or devolved deployment strategy? such as making each data centre responsible for locally applying policies devolved from the mothership?
    Yep this was the direction I was heading for creating an in-house solution, unfortunately we let one of our datacenters get a bit too large. Even to the point where our IPMI cards no longer work due to arp-cache being full So we might have "mother ships","father ships", and maybe even some little "child ships". I've also heard stories of people forcing puppet to behave in a similar fashion with crons and just remove puppet-master altogether.

    You might try to make contact with IT people at NASA, government departments, and the military, and ask them what they use/suggest.

    Otherwise Google is your friend.............I believe they have the most servers of anyone in the commercial sector...........you need to talk to your oppo at Google?
    Great, I wonder if NASA or the US government has an employee directory for all their IT grunts I can just start war dialing . Seriously though, if anyone on the list has any contact for someone like Nihil is suggesting, please shoot them my way(or ask them and then post back here on the thread). I would be greatly appreciate it.

    I also would be interesting in anyone's thoughts or ideas for architecting such a scalable configuration management system. So if anyone's specialty is in designing scalable software please share.

  6. #6
    Senior Member bAgZ's Avatar
    Join Date
    Jul 2001
    Posts
    206
    I help menage around 400 servers and will be interested in what limitations have you run in with puppet as i currently use it quite a lot.
    ----------------------------------------------------------------------------------------------------------
    "If I'd asked my customers what they wanted, they'd have said a faster horse." ~ Henry Ford

  7. #7
    Senior Member nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,188
    Hi ABS,

    Wow so the way you responded there almost makes it sound like I came off as a bit of a braggart. That was by no means my intention in mentioning the size of our environment, I only wanted to make sure everyone had an understanding of the problems we were running into.
    On the contrary, I thought that you described your environment and questions very succinctly. That is why I suggested a reason for the lack of responses; otherwise I am sure you would have had several replies that you had already checked out. This is a pure scalability issue, right?

    unfortunately we let one of our datacenters get a bit too large.
    Arrrghhhhhh! I just knew you were going to say that!!! It's always the way in my limited experience...........the business requirements and beancounters' budgets rule to begin with? Then we hit the tech limitations that we told them about, but they didn't listen.

    Now, please don't get me wrong, I'm no expert in this field, but I have encountered it so much in the applications development and deployment arena!!! OK, a different area, but the issues are pretty much the same.

    You (quite rightly IMO) do not state your business.............not good idea on an open forum?.............and your size suggests you my be considered a competitor in the commercial sector?

    That is why I suggested .gov type outfits. They don't have a commercial agenda..................they just waste (our) taxpayers' money as they like?

    NASA might be a good start as I believe that they still lease space (no pun intended) and apps a la cloud computing?

  8. #8
    Senior Member gore's Avatar
    Join Date
    Oct 2002
    Location
    Michigan
    Posts
    7,177
    If you use SUSE, or OpenSUSE, you have some options. Novell makes a bunch of stuff even though they were bought last month. But they still apparently have all their stuff. You can always try them though, they're good with Server management.

Similar Threads

  1. Network Management
    By FanacooL in forum Other Tutorials Forum
    Replies: 2
    Last Post: October 29th, 2004, 04:24 AM
  2. The history of the Mac line of Operating systems
    By gore in forum Operating Systems
    Replies: 3
    Last Post: March 7th, 2004, 08:02 AM
  3. Why linux is better than Windows?
    By MemorY in forum Miscellaneous Security Discussions
    Replies: 29
    Last Post: February 27th, 2004, 04:20 AM
  4. Evolution of a Linux User
    By gore in forum Tech Humor
    Replies: 9
    Last Post: October 1st, 2003, 08:53 PM
  5. Installing Linux
    By Rewandythal in forum Other Tutorials Forum
    Replies: 5
    Last Post: December 12th, 2001, 09:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •