DBmarlin now integrates with ilert for on-call alerting

DBmarlin alert rules can now fire straight into ilert, so a database problem reaches whoever is on call, with escalation behind it if they don't pick up.

Plenty of the alert rules I come across are pointed at an email address or a Slack channel that nobody checks any more, so the alerts go out and nothing gets done about them. DBmarlin has always been able to tell you that a statement has slowed down or that a host is filling its disks, but getting that in front of a specific engineer on a Saturday evening has never been something we set out to do, and it isn't something we plan to build. That is the job ilert have spent years on, and we are happy for them to do it.

Who ilert are

ilert is an incident management, on-call alerting, and status page platform with a built-in AI SRE, made and hosted in Germany. They integrate with around 150 tools and DBmarlin is now one of them. They built the integration on their side, and it is live and documented today.

What it gives you

When a DBmarlin alert rule fires, ilert checks your on-call schedule, reaches whoever is on duty through whichever channel they have chosen, and moves on to the next person if nobody acknowledges within the window you have set. If the problem turns out to be larger than one engineer can handle, that same alert can be promoted into a declared incident with its own channel and incident commander, pushed out to a status page for customers, and written up as a postmortem once it is over.

An alert carries the rule name, the statistic that breached, its old and new values, the threshold and units, the time window, and the instance or host involved. That is enough for somebody squinting at a phone to work out how far past the threshold things have gone. The payload is driven by a template, so if there is something else your team wants to see on an alert you can add it, and it comes through as a custom detail in ilert.

Each alert also carries a link back into DBmarlin, pointed at the right instance over the right time window, so whoever picks it up lands on the problem itself. Worth saying from experience: the DBmarlin URL you configure has to be an address your responders can reach from their own machines. localhost only resolves on the DBmarlin host, which is easy to overlook until somebody tries it from a phone.

Host alert rules send an event when the condition ends, and ilert resolves the matching alert when it arrives. Database instance rules will do the same once you add a single eventType line to their template, which is worth doing early, before you have a few hundred alerts sitting open and no reliable way of telling which of them still matter.

Where Instana fits in

ilert also integrate with IBM Instana, which is useful for anyone running both. Instana will tell you that the application has slowed and the database is somewhere in the path. From there DBmarlin shows you which statement is responsible and what it was waiting on, along with anything that changed shortly before (more on that in why use DBmarlin alongside IBM Instana). ilert is what gets all of that to somebody while it still matters. We did not set out to build a three-way chain, but it fits together well enough to be worth pointing out. See DBmarlin for Instana for more on that side of things.

What you can alert on

Alert rules cover database instance statistics such as query time and execution counts, along with host statistics including CPU and disk utilisation. Disk arrived as an alert category in 6.7, so you can now be told about a filling filesystem in time to do something about it. Since 6.5 the rules have supported separate Info, Warning and Critical thresholds, and those come through to ilert intact, so one rule can warn you early and page somebody properly later on.

Setting it up

There is no agent and nothing to install, because the integration runs over the webhook support that is already in DBmarlin, and the data only travels one way, outbound from us. In outline, you create a DBmarlin alert source in ilert and copy the webhook URL it generates. Over in DBmarlin, under Settings and then Integrations, you will find Webhook listed under Alert notification, and you create a new one there using that URL. Paste in the two content templates, one covering instance rules and one covering host rules, and then either mark the webhook as default so that every alert rule uses it, or attach it to particular rules under Alert Rules.

ilert have documented the whole thing field by field, with both templates ready to copy.

Read the full setup guide on docs.ilert.com →

Try it

It works with any DBmarlin installation, including the free tier, so if ilert is already running somewhere in your organisation you could have database alerts reaching your on-call rota by this afternoon. There is more detail on the DBmarlin for ilert page, and anyone who has not used DBmarlin before can download it free and monitor one database instance for as long as they like.