ELK Elasticsearch Review

Good processing power, very scalable, and able to handle all data formats


What is our primary use case?

We try to detect malicious files by the logs. The logs are all centralized including all our PCs, our callers, our servers, Linux, windows, Polaris names. We scan everything. Then we have pre-defined specific use cases that allow us to identify if there is an attack on the machine or indirectly by the endpoint. On top of that, we can check with users as we're not directly dealing with the configuration, so we can follow up on the alerts we receive. On top of that, we have the systems in place that allow us to detect if certain inexcusable items are on the system, such as malicious files. We can do this because we also retrieve the log files of the identifiers.

What is most valuable?

The fact that you can dump any type of format in the database without any specific reformatting is fantastic. It makes it very flexible in collecting information and that saves us a lot of time because otherwise, we would really need to define specifically what we're looking for and reformat everything. With this solution, that's not necessary. We can directly, and in a really standard raw format, dump the data into the database. Only afterwards do we need to define what specifically we're looking for, however, at that point, it's not a big deal to actually add an additional log and to collect additional information. 

The solution is very scalable. 

There's lots of processing power. You can actually just add machines to get more performance if you need to. It's pretty flexible and very easy to add another log. It's not like 'oh, no, it's going to be so much extra data'. That's not a problem for the machine. It can handle it.

What needs improvement?

The solution has quite a steep learning curve. The usability and general user-friendliness could be improved. However, that is kind of typical with products that have a lot of flexibility, or a lot of capabilities. Sometimes having more choices makes things more complex. It makes it difficult to configure it, though. It's kind of a bitter pill that you have to swallow in the beginning and you really have to get through it. 

Once you begin to understand the concepts and how to actually look for data it's a very pleasant solution, but the learning curve is very steep in the beginning, to the point that they could improve it to make it a bit less intimidating to start. There needs to be a bit more intuition behind the architecture and the data search.

For how long have I used the solution?

This solution has been used for at least five years at the company.

What do I think about the stability of the solution?

It's very stable. The only thing that might happen is that sometimes when you do a search it will stress the machine a bit too much. If that happens, then it's a matter of, if you do it the wrong way, the machine gets stressed and then it slows down. However, it will not crash. It almost never crashes. You'll simply figure out that the machine is overwhelmed and take the stress off. 

The problem, occasionally, is that it may become unresponsive, but it isn't really unresponsive, it's just that the system is overloaded. That can only happen if you do your database search in the wrong way. That's why, especially when you have a lot of data and are really concentrating a lot of data on a few machines, you have to be careful of what you're doing. 

It's a very nice tool but you have to be a bit aware of how to deal with this, especially when you have a lot of data and you have limited processing capacity. If you have unlimited processing capacity you can do whatever you want with it. I personally can say that I've never seen a machine crash.

What do I think about the scalability of the solution?

The scalability of the product is good. It's our key system that generates alerts and does surveillance on a security level. This product is extensively used in our organization.

We have people of course, from the server team that makes sure that the logs get collected. And then we have the people that actually deal with the configuration of the ELK as well. That is a team of five or six people that we use now. Then, of course, we have all the teams that follow up on the alerts, and there, I would say, we have two or three different teams, which is between 10 and 20 people. That's just part of the people that work with the solution.

How are customer service and technical support?

I work on part of the team that deals with technical support issues. There's a good community around the solution. This is because the product is actually open-source. With a lot of typical issues, you can simply Google questions and you will find the answer. Of course, we do have a support contract with the company. I don't deal directly with that, however. We contact them directly if we really need to and we have maintenance contracts with them. Unfortunately, I can't really speak to how good or bad they are because I've never called them myself.

Which solution did I use previously and why did I switch?

Before we switched over to this, we used it in combination with an end product called QRadar, but both of them together were time-consuming. 

How was the initial setup?

It's easy to install the servers, that's not really the problem. The difficulty is afterward. Users need to understand how to explore the data.

The server setup is the easy part. Even, let's say, moving the log into the machine or into the database is no problem. However, then you have all this data and you will really struggle to understand the information. That is sometimes not always obvious at the outset. In order to do that in an effective way, it requires a little bit of manipulating.

To install the servers, a minimum installation takes me a day or more. It's for the most part usually pretty fast.

What about the implementation team?

I myself have already had quite a lot of experience with the product. Therefore, I can set it up myself.  Most customers or most IT departments will struggle to set it up due to the difficult learning curve in the beginning. 

I would definitely recommend most users or companies, at least for the beginning, to get help troubleshooting problems. It will help them understand a little bit more about the steep learning curve. It really makes things much easier, and much more effective. 

Which other solutions did I evaluate?

I have used different products myself due to the nature of my work. I'm a security consultant. I have been working with different customers who use different solutions, which means that I have used other things and can evaluate and compare them for clients.

I've worked with Splunk, for example. Splunk, for instance, on the level of data mining and inquiring, might be easier. It's a bit more intuitive. The downside of it is as soon as you start collecting a lot of data, it becomes extremely expensive to use Splunk. It's a very good product. However, typically, with the need to collect as many logs and as much data as possible, Splunk becomes expensive, and you can't put it in a budget easily. It's simply out of budget for many as soon as they start clicking. Also, the purpose of a security system is not the same.

With Splunk, some will not add additional logs because they don't often have the budget, especially when it immediately means that you're going to need to increase your costs enormously. That's not the purpose of a security system. For the system to be effective you must be able to have good surveillance and that means that you should not hesitate in adding your logs. Still, when the costs double, people hesitate and if they don't have the budget and cut the logs, things can get through. Fortunately, with ELK, you don't have that issue. With ELK you don't pay for gigabytes, or terabytes or the data that you use. That's the main advantage compared to Splunk. But Splunk, it has a less steep learning curve.

What other advice do I have?

I'm just using it as a customer

We tend to use the latest versions of the solution. We try to upgrade it on a regular basis.

I'd advise other companies considering implementing the solution to get a team in that knows the product and try to take advantage of their knowledge. It will help reduce the pain of the learning curve.

I'd rate the solution eight out of ten.

I would not give it a ten because of the steep learning curve. I know what the product is, but many do not, and for them it will be quite difficult to get started without becoming very frustrated in the process. 

Which deployment model are you using for this solution?

On-premises
**Disclosure: I am a real user, and this review is based on my own experience and opinions.
More ELK Elasticsearch reviews from users
Add a Comment
Guest