Tidal Automation Review

Robust calendaring enables us to set up very specific job run-times to even account for holidays

What is most valuable?

For us, the calendaring system is very robust. Some of the teams have very specific requests for when they need jobs to run. That's been really valuable, because a lot of times, when people run scripts, if they run on a holiday, they're going to fail. We've even started adding some European holidays and other times when scripts should not run because they're going to fail, because they try to connect to external exchanges that are closed on a holiday. For things like that, things you can't do in a lot of built-in scheduling tools, Tidal has been very helpful. A couple of times a month it probably saves us work and the necessity of logging in from home and checking to make sure everything's okay.

Especially in the newer versions of Tidal, the segmentation of user permissions enables us to give people operator permissions for their jobs, to rerun jobs, but view-only for other groups' jobs. We're able to keep people from hurting themselves or other groups accidentally. The permissioning is really good. We have 20 different root-level job groups that hold all of the jobs for each team underneath it, in our shared space. I can set it so that the database group only sees database jobs, if that's all they want to see, so it's not cluttered with everyone else's jobs. But if there are teams that need to see all the spaces, we can do that as well. We can let them see only certain servers or certain users to run jobs. You can edit it too so that people don't see too much or don't get confused and lost in this sea of the thousands of jobs that they could be seeing, when they only need to see their own. That's been nice to set up over time.

In the past year, in particular, the client has gotten tremendously better. If you asked me three years ago, I would have said that the client was the biggest problem with Tidal. The backend was always really solid, but the client was pretty bad for a while. In the past year, with the new company taking over and putting a lot of development effort into the clients, especially the web client, it has really made people a lot happier when having to use the client and work with it. In the past, they begrudgingly used the client, but now they're happy to use it, which is a big change.

Because we've been with Tidal for so long, I can't compare it to the way things were before Tidal. Back before Tidal, there was much less electronic trading.

But an example of how we benefit from it is that we have Tidal jobs that load all of the trading symbols into our database every morning before trading opens. That's mission-critical in terms of getting ready for the traders to start trading on a specific day. If they don't have that updated information through the database, they can't trade.

There's a lot of overnight, big-data processing that happens, things that need to run all night. That's launched through Tidal and monitored as well. It's pretty much a 24/7 operation in terms of uptime, and we've definitely used Tidal to meet that goal.

The solution has increased productivity by saving staff hours. We have an operations team that's here 24/7. We have a runbook that says, "Okay, if this job fails, do this." I'd say 80 to 90 percent of the time the operations team is able to resolve a problem by following runbook and steps without having to contact someone overnight or on the weekends. But Tidal does save the person who owns the Tidal job from having to do work in off-hours especially.

What I like about the new company is that if you ask for something, and they feel like it would be a valid improvement, they're willing to push it out, even if it's a few months out. They make sure to provide it at some point. It doesn't just get lost in the mix.

I work for a financial trading company: stocks and options. The use cases depend on each group that is using it. We have a compliance group, HR group, and a bunch of trading groups and technologists. It's used for a thousand different things depending on the group. It's all to support a financial trading firm, and the processes that happen before the market opens and after the market.

We have a pretty good mix of Linux and Windows boxes, a good 60 percent Linux and 40 percent Windows. We launch trading scripts to start processes up, to stop processes, and to pull in data from third-party vendors; we have FTP jobs that do that. We run an Oracle backend.

From talking to the Tidal people, we have a lot of agents connected to masters, compared to most other firms. But we're probably middle-of-the-road in terms of how many jobs run per day. We're only slightly over 100,000 jobs per day, throughout the whole space.

What needs improvement?

The biggest problem for us was the Transporter tool that works through the API. It's like a GUI into the API where you can transfer and compare jobs between two Tidal spaces. Up until the last few months, the Transporter tool that was offered was not really good at all. It was hard to take a job in development and promote it to production. There was no really good tool to do that. They offered a tool, but it wasn't that good.

But they just put out the Tidal Explorer tool, which is basically a replacement for the Transporter. That looks promising. I haven't really gotten to use it yet, but it seems to be a better system. That's what people have been requesting for a while now: an easy way to promote and review changes; something like a script repository-type of system, where you can promote something or pull it down, compare it, and then, if you like it, push it. If it doesn't work, you can back it out to previous revisions. It looks like it offers all those features, but I really haven't had a chance to dig into it. I set it up and it does look promising for the future. It's probably something that we're going to try to integrate into the day-to-day processing once it gets released. I don't think it has even been released as general-availability yet. It's still in beta. But once it gets to be production-ready, we would definitely love to use it. It's something that's been on our radar for a while now.

Tidal also had a cache database, which was a copy of the master database, that the web client used. They got rid of that in the latest version, and that is something we had been asking for, for a long time. The way it had been set up didn't really seem optimal.

It looks like they're trying to put forth a better tool for certain places that were lacking.

On another topic, we have to set up ways to send a job event that finds a job that completes abnormally. What we do is send it to an SNMP trap that gets aggregated into one space and we can see those errors. We try not to use Tidal for monitoring, as much as for job launching and tracking. We have a Nagios setup so that if something fails, the error can be sent to Nagios and checked there. If a job is a long-running job, like an eight-hour job, we don't want that job active in Tidal for the whole time and taking up a job slot. We'll kick the job off in Tidal and it will show that it has completed normally. Then we'll hand it off to another tool to monitor that the process is running for the specified amount of time. I don't know if Tidal wants to get into the business of monitoring long-running jobs, but that could be a feature for the future: a job launching and monitoring tool. Using Tidal for monitoring doesn't seem like a good fit, but if they could offer something that did that as an add-on or include it, it might be helpful.

Finally, the solution is a little tough to learn. Talking to people who are new to using the Tidal interface, it's difficult. But I don't have anything to compare that to. They have said it's not as difficult as Control-M or some of the larger scheduling systems that people have used. It's not as hard as that. Tidal has worked to prevent new users, especially, who aren't exactly sure what they're doing, from hurting themselves too much, which is good. They've put a lot of restrictions in place to prevent people from doing things that weren't intended. There is a learning curve, but I don't think it's steeper than any other new scheduling system. In the past, we've downloaded some other options and they had a learning curve too. If you've never used it, there's always a curve, with the terminology, etc. But I don't think it's any harder than any of the others.

New users of Tidal need at least a month of working with it a little bit each day. I give people a three-hour introductory course. Every quarter I provide an overview for new users of how things are set up. Luckily, in our company, a lot of these new users are joining groups that already use Tidal on a daily basis. If they have any questions after the initial course, they can talk to their team. Over time, the teams that use Tidal are resources for the new employees. That takes a little bit of training off of my plate. Within a few months people are confident and moving along. It takes a few hours to pick up but to be fully confident it would take a few months to really feel that you know what you're doing in the space.

For how long have I used the solution?

We've been using Tidal Workload Automation for 15 years.

What do I think about the stability of the solution?

I'm really confident in the stability.

Cisco owned the company for a few years and I felt that it was something of an afterthought for them; it wasn't really their business. They didn't really put the time and effort into it. It seemed like, for a couple of years, nothing was getting resolved and people were pretty unhappy. We ended up staying in a version that was years and years old, compared to what we should have been on because we were not confident in the solution that they were providing, to give us what we needed. 

In the past two or three years, since the new company took over, we have much more confidence. People are much happier with the direction that Tidal is going and the features that they're releasing.

What do I think about the scalability of the solution?

Our usage of Tidal goes up every year. That's not even from planning to increase usage. We have a few holdouts, people who still use Task Scheduler or cron, but over time they've all been folding into the Tidal space to have a better overview and a cross-platform way to see everything and rerun everything and be alerted. They've come to the conclusion that it's a better method, especially for overnight. We have an operations team that manages things overnight, so that if something fails in the middle of the night, that team can handle it, which they would not be able to do if it wasn't in Tidal, along with thousands of other jobs.

In terms of the number of jobs in Tidal, it's been increasing at between 10 and 20 percent a year. It's going up. It's definitely not going down. Initially, it was probably 50 percent a year because everyone was adopting it. Over the past five years, since it was already utilized by everyone, there has been a general 10 percent a year increase because of new jobs that need to be created and new processes that need to be started and stopped.

We're somewhere around 95 percent in terms of adoption of Tidal. There a few small groups that like to do their own thing and use open-source products, but those are groups that maybe only run Unix and that's it. They're happy with Jenkins or something open-source that only needs to run a few hundred jobs. It's only one platform, and it does what they need to do a little bit better than Tidal. But for groups that need an all-in-one solution, they've all gone to Tidal. If they need to do what Tidal offers, they're going through Tidal to do so. It's pretty accepted here.

How was the initial setup?

Tidal was here when I got here. It's been around for a while. But over the past 15 years, I've been the one who researches the new patches and service packs and revisions and I've done all the upgrades.

The upgrading process is straightforward for me, but I've been here so long that it's just something I know. It has gotten much better with the new company. We're on a Unix backend, so a lot of times, with a simple hotfix or service pack, you can just run a shell script and it replaces all the files. It does everything it needs to do. It places everything in the right location, and then all you have to do is start and stop the backend process and it picks up the new revision. That's been really good. In the past, it was a more manual process. In the past couple of years it's gotten much easier in terms of being able to do things with one script.

The releases have been good with very few bugs or installation problems. There were some in the past, a few years ago, where you would try to run something and it wouldn't take into account your environment and it would fail. You'd have to tweak some of the script. That was a lot of manual work. The upgrade scripts, recently, have worked pretty seamlessly.

What's my experience with pricing, setup cost, and licensing?

We have an enterprise contract, so if we want to add another agent, or if we want to add another master, we don't have any restrictions on those things. Other vendors don't have that flexibility. For me, as an admin, that makes it easy because I don't have to think about what a new master is going to cost or what a new agent is going to cost. If someone needs a new agent and they need to run a job from that agent, we just go ahead and do it. If we're in Dublin, Ireland and someone wants a new master because there's a group over there that wants to adopt Tidal, we can just say, "Sure, get a new license, create, and you're fine." For the license that we have, the flexibility is great.

I don't know if other people aren't happy with the licensing model because they have a non-enterprise license where they have to think about everything they change.

We're negotiating our new license now for April, which is when we have to renew. We've usually gone with a three-year license. The numbers that the new company has put forward haven't really changed significantly from our past renewal. People here are pretty happy with that. It's not like the new company came on and jacked the prices up exponentially. The new prices that we've received seem reasonable and comparable to the marketplace.

What other advice do I have?

Because our environment is older, it's a little tough to integrate some of the newer features that they're offering. That's because of the way we had to configure our environment for older versions that didn't have these newer features. In terms of how you delegate permissions, how you set up calendars, who you give permissions to, my advice would be to figure out how the permissioning structure works before you set up your environment, and stick with a standard. A lot of the time, we're having to go backwards to make things standardized. If we started over right now, I know how I would set up a Tidal environment. It's hard to do that after the fact, and after things have been set up differently in the past. So try to develop the best system for standards and then keep that.

We don't use any of the Tidal adapters that they offer, just because we're heavy on development here. A lot of the people here, in the past, felt that they could write their own wrapper scripts to do the same thing that the adapter jobs do. That's ingrained in our environment now. We don't even look at the adapters too often, just because we have an in-house solution to those.

The vendor is starting to offer tools such as Tidal Repository, but that's going to be an add-on cost. I'm still evaluating whether it's something that we want to try to get a price on and use. It would allow us to see if certain jobs are running longer than they usually run. We could also see if queue levels are hitting their limits often and what we could do about that. The Repository seems like it's going to be a tool to gives you the drill-down information, like seeing how calendars are configured and a lot of the information that you're trying to get at. It's more like an admin dashboard where you can drill down. Right now, I just go directly to the master to search the logs, or we have all the master logs sent to a repository. I can search them there. We're doing things from our side with other monitoring tools we have and log aggregation tools.

Which deployment model are you using for this solution?


Which version of this solution are you currently using?

**Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
More Tidal Automation reviews from users
...who work at a Financial Services Firm
...who compared it with Control-M
Learn what your peers think about Tidal Automation. Get advice and tips from experienced pros sharing their opinions. Updated: January 2021.
457,209 professionals have used our research since 2012.
Add a Comment