Remote execution in itself is a big time saver at any scale.
For example, a particular incident happened at one of my previous organizations. We had to do a PoC on a lot of servers, where traffic was to be generated from a few hundred machines (something like 'bees with machine guns') and would allow us to benchmark one of internal components.
So, before we began working on it, I suggest the use of SaltStack because of its remote execution. They could easily start generating traffic from a few or all these servers and then get a good feel of a Flash Sale in Ecommerce.
Eventually, one of my colleagues was assigned this task and he used SaltStack. He liked the way SaltStack (on the entire cluster) was up and running in a few minutes, and also gave him flexibility to generate traffic, make config changes, etc. on the fly.
Improvements to My Organization
Currently, most of our configuration is in SaltStack, so scaling up when necessary with or without Salt Cloud would be real easy.
Traditionally, the team here expects the use of Golden AMIs for scaling up the infra, which, though useful, has its limitations:
- Security updates to the OS are the biggest concern.
- Non-standard configuration on one server would also cause some serious issues if its AMI is used by mistake in scaling up.
If, instead, we push configuration to new servers during scaling up, then we fix those issues.
And, I was also considering the fact SaltStack gives near flat-line performance (for both remote execution and pushing changes through states), whether the infra size is 10 servers or if it has grown beyond a few hundred. So, that is at least one area that we need not be worried about.
The configuration management is at least one aspect that would take care of itself (not considering redundancies, reporting, etc. required for SaltStack here at the moment).
Room for Improvement
Personally, I feel that SaltStack has many renderers, but the documentation was a bit lacking (in particular, for Py it was close to nothing) when I was studying it up a few months back.
Salt supports multiple renderers Py, PyObjects, etc. (https://docs.saltstack.com/en/latest/ref/renderers/index.html#multiple-renderers). These allow users to write states in JSON, Mako, MsgPack, etc. Py renderer allows us to write states in Pure Python.
I had many scenarios where SaltStack didn't have enough functionality at the time (it has been added in recent releases). For instance, I was trying to add an instance into ELB as the last step of orchestration. But, Salt didn't have anything to support it. So, instead I went ahead and wrote a small state in Py renderer.
There are also cases where Jinja + YML is not enough and to DRY up the states, one has to use either the Py or PyObjects renderer. I prefer Python, as you then don't have to look up the syntax of a particular renderer and a simple Python script would suffice. The catch here is that Salt expects output in a particular format and initializes its internal variables in a specific format, too.
I spent most of my time figuring out how to make this Python script work with SaltStack. Any such functionality that’s missing from SaltStack can be easily implemented using the Python (Py) renderer. So, if the documentation around renderers is improved, it will help anyone with a very specific use case.
Use of Solution
I have used it continuously for the last year, and sporadically for the last three years.
Sometimes, salt-minions do start consuming very high memory, but I've generally seen this to last just a few moments or at most a minute. On a production system, this might cause an impact on serious loads.
Additionally, if many of the servers in infra are down, and you bring all of them up simultaneously, it used to bring down salt-master. This happened until last year, when I was working at scale. Since then, I have switched from that job; it’s difficult to test this pain point now.
Customer Service and Technical Support
I haven't tried technical support.
I did not previously use a different solution.
Learning SaltStack did seem a bit daunting at the moment I was learning it. The concept of creating a top.sls with references to various states and their targets, then creating corresponding files in YML, took a day or two; beyond that, it was real easy.
Pricing, Setup Cost and Licensing
If someone is using it for an infra consisting of a 1000 servers or more, then support would be real useful. Others can go through the documentation and learn from forums or SO posts.
Other Solutions Considered
I tested Puppet and Chef, but could never get around to using them in production or at work.
Salt was more of a Swiss Army knife. And our work at the time was more focused on rapid manual changes.
Create valid states for all environments and keep the difference between these environments minimal. Use test cases as much as possible.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Sep 26 2016