2018-07-22T09:26:00Z

What's the best way to trial log management tools?


We know it's important to conduct a trial and/or proof of concept as part of the buying process. 

Do you have any advice for our community about the best way to conduct a trial or PoC? How do you conduct a trial effectively? 

Are there any mistakes to avoid?

Guest
88 Answers

author avatar
User

At the risk of sounding flippant,  I personally believe that the best way to trial log management tools is best encapsulated in these three words, "clarity, clarity, clarity".  Clarity as in having an agreed upon internal understanding of exactly what problem(s) you and your team are trying to solve.  This is critically important because a fair amount of consumers inadvertently conflate Log Management with the components of a SIEM, i.e., Security Event Management, Security Information Management, etc.
If there is currently no one on staff that possesses the background and experience to develop and internally socialize a list of criteria for the trial / PoC, I would definitely leverage my interactions with the Vendors to develop my list of criteria.  If budget and internal strategies permits, I would recommend utilizing a 3rd party resource with demonstrated real-world experience and knowledge in this space to assist with providing the thought leadership and direction for the effort.
Some suggestions for decision / evaluation points that you many want to consider as part of your trial / PoC.
1.  Scope 
2.  A defined set of Use Cases
3.  Metrics     
a.  response time      
b.  retention requirements     
c.  other
4.  Log Sources      
a.  COTS - (diversity of log sources, Windows, Unix, Linux, Mainframe, Routers, Firewalls, etc)     
b.  In-house developed      
5.  Problem Set(s)     
a.  Regulatory Compliance     
b.  Security Monitoring / Analysis / Response     
c.  Security Audit Remediation     
6.   Key indicators for success / product selection

Some items to think about:
1.  If you perform a POC / Trial with vendor equipment, do you have a policy / plan that prevents the unintentional retention of internal log data on vendor's hardware, i.e., disks, etc?  ( you absolutely       want to ensure that critical / sensitive log data does not go out the door with the vendor! ) 

2018-08-13T22:42:11Z
author avatar
Real User

For trial, you need to define,

1. Use case

2. Budget

3. Resources

Use case is very important as withing same budget you may get multiple products.

Understand use case like,

What do we need to achieve?

a) Asset Logs

b) Network Logs

c) What kind of logs

d) Space Requirement

e) Vulnerability Scanning

f) Events alert Custom

g) 0 Day Protection

2018-08-16T08:34:31Z
author avatar
User

The rigor required to adequately test a SIEM tool can be overwhelming. 30 or 45 days may not be enough time, particularly if log management is just your part-time role. check the recommendations of others in your industry and lean toward those tools. you will have to spend some bucks to get something useable without tons of setup and configuration time. and it's not a set it and forget it situation; it requires lots of tuning, else you have wasted your money for a tool you don't use.

2018-08-13T21:07:16Z
author avatar
Vendor

Hi,

Choosing a Log Management solution depends strongly on its purpose.
What will you do with this solution?
If a mandatory feature is missing, you will have no other choice than to eliminate the product from your final choice.

This is why it is important to know which feature is mandatory, critical, important or nice to have.

There is one method in 4 steps:
Firstly you have to build a list of all the features you need to have or you would like to have.

You have to do this list for all aspects like (this list is not in a specific order):

Logs collection methods
Logs parsing capabilities
Logs format supported
Categorization
Inter-compatibility (very important if you have already a SIEM by example)
Product Support (vendor or open-source)
Chart types
Dashboards
Reports
Searches capability
Searches regex
Searches operator
Concurrent searches limitation
Cost
Evolution
Sizing
Data retention
Available Statistics information (like latest logs received, EPS, top values, bottom values)
Static correlation like lookup feature
High Availability
Load balancing
Performance
...
Etc
There is maybe other features or capabilities that are very important to you, do not forget them.

You have to do this exercise again and again with your team like a brainstorming.
I don't think you will think about every point in one shout.

This will permit you to build a spider chart with those features (I advise you to create 6 or 8 main categories and you put the above features in their related category to build the chart) that will be your reference to choose the product.
I recommend you to be very focused on the most important points for your business.
If a feature is not mandatory put a lower value this will permit you to increase your choices between different Log Management Solutions.

Thirdly, I will start to evaluate the different products in starting with the one with a free version like Splunk, ArcSight Loggers, etc...

In the last step, for each product, you will evaluate, build also a spider chart then compare it with your reference to know which solution will be the best for you.

To do the test, I recommend sending a bunch of all logs type you will collect to do not have a bad surprise.
Windows, Firewall, Proxy, AV, IDS, IAM, etc. . . And Custom logs, this last one is for me highly important.

This answer concerns only Log Management, I have not talked about SIEM.

Personally, we have chosen ArcSight Loggers just because we had already ArcSight SIEM. The inter-compatibility feature was so important that we have not tested Splunk which is for me the best current log management solution.

I hope I have successfully answered your question.
Please, do not hesitate to contact me if necessary.

2018-08-13T20:13:40Z
author avatar
User

Hello,

for my experience a good Log management POC task must include:

- POC Business Requirements and Drivers of customer

- POC Scope agreement

- POC Success Criteria agreement

o Use Cases to proof for primary interest of the customer

- POC Collecting environment acquisition and definition

o Customer IT Environment

o Technology in POC requirements for installation and configuration

- POC Archiving environment acquisition and definition

o Customer IT Environment

o Technology in POC requirements for installation and configuration

- POC Tasks plan

also:

- Make an evaluation score matrix for primary Log Management capability

o Log Collection capabilities – Using an Agent-based approach or Agentless approach, out-of-the-box log collection support for 3 rd party commercial IT

products

o Parsing & Normalization capabilities – Collected logs will be parsed and normalized to

o CIA guarantee capabilities - Information security's primary focus is the balanced protection of the confidentiality, integrity and availability of data (also known as the CIA triad) while maintaining a focus on efficient policy implementation, all without hampering organization productivity

- Avoid to proof capability or feature out of success criteria (wasted time)

2018-08-13T16:18:09Z
author avatar
Consultant

Mark is correct but there are things to look for. Do you have a set of requirements? Not all log managers collect the information, not all log managers are easy to navigate, and not all log managers provide the reports your are looking for. Check to see how much data it collects so you can plan storage. Does the log manager compress the data or does it dependent on a third party tool? Do you know what you are collecting, and why? Are the logs used for security, sox audits or something else? My advise, before testing, is to gather and review you're requirements and test against that. There are lots of free trials. In fact if there isn't one on the web, contact the vendor and they'll give you something to try out for 30 days.

2018-08-13T14:50:02Z
author avatar
User

A true SIEM product should be chosen in order to have real time correlation of the gathered logs (or even network flows with some of the vendors). You can start with the Gartner suggested visionaries and leaders, these are proven SIEM vendors that all have free trial of their full version product. In order to choose the best one to start, the most important thing is to clarify your requirements and to know what do you want to achieve with the Log Manager (SIEM). In this case you can choose the most appropriate one to start with a PoC, as most of them achieve the same, but have slightly different advantages like single management console, additional features included in the base product, etc.

2018-08-14T10:52:41Z
author avatar
User

Stick with the tried and true SIEM/Log Management Vendor that offers a free, online download and trial, easy to install and operate piece of software with proper documentation. A good example of that can be found here: https://www.snaresolutions.com/try-snare-free-for-45-days/

2018-08-13T13:47:09Z
Find out what your peers are saying about Splunk, LogRhythm, IBM and others in Log Management. Updated: October 2020.
442,141 professionals have used our research since 2012.