Alerts and problems

In Event Monitoring, an alert is the rule you set, and a problem is what the rule finds. You set an alert to watch a measurement on your resources, such as memory in use. Whenever the alert’s condition is true on one of those resources, that is a problem. The problem shows on the Monitoring Dashboard, and the people in the alert’s alert groups are notified.

Alerts

An alert watches one measurement on one or more resources of a single service. It can watch nodes, load balancers, DBaaS instances, or the nodes of a Kubernetes cluster. However many resources it covers, it is one row on Manage Alerts.

You build an alert from a few choices:

• The service and the resources it watches.
• The trigger, the measurement it watches, such as Consumed Memory or Free Disk Space.
• An operator and a threshold, which together set the condition, such as greater than 80%.
• A severity, the label its problems carry.
• The alert groups it notifies.

Alerts belong to a project and a region, Delhi or Chennai, and Manage Alerts lists the alerts of the project and region selected in MyAccount. That includes the alerts you create from a resource’s own Alerts tab and the default alerts E2E adds. To set one up, see Create an alert.

Pausing an alert stops its notifications while its check goes on; see Edit, pause, or delete an alert.

Problems

A problem is one alert’s condition holding true on one resource. If an alert watches memory above 80% on three nodes and two of them are above 80%, there are two problems, one for each node. Two alerts that are both true on the same node make two problems as well. E2E’s built-in checks raise problems of their own; see E2E’s built-in checks.

The Monitoring Dashboard lists the open problems that began within the period you choose, with the resource and severity of each. To read it, see Read the monitoring dashboard.

Each problem notifies its alert groups once, when it starts. To find out whether a problem is still open, check the dashboard with a period that covers the time it began.

Severity

Each alert has one of six severity levels, from most to least serious: Disaster, High, Average, Warning, Information, and Not classified. A new alert starts at High.

Severity is a label. It sets how the alert’s problems are counted and shown. The operator and threshold alone decide when the alert fires. Use it to decide which problems to handle first:

• Problems by severity on the Monitoring Dashboard counts the open problems at each level, and selecting a level narrows the list to it.
• The Monitoring Alerts card on your MyAccount dashboard groups the levels into three. It shows Information and Warning as LOW, Average as MEDIUM, and High and Disaster as HIGH.

A Webcheck alert, which you create on a node’s Alerts tab, is always High; see Set up alerts on a node.

Alert groups

An alert group is a named list of MyAccount users that alerts notify. Every alert notifies at least one alert group, and one group can serve many alerts across all four services. You can make a group under Manage Alert Groups or from any alert form, including a resource’s Alerts tab. You change or delete it under Manage Alert Groups. Alert groups are separate from the user groups in IAM, which control what people can access.

A group holds at least one user, and its members are MyAccount users. For who can be added, see Who you can add.

Alert groups belong to a project and a region, like alerts, and an alert chooses its groups from its own project. See Manage alert groups.

How alerts reach people

When an alert fires, every member of each of its alert groups gets an email at the address they sign in to MyAccount with.

Notifications can also go out on WhatsApp, to the account owner’s phone number only. In General Settings, under WhatsApp Notification, tick Enable WhatsApp Notifications, turn on Monitoring Alerts, and click Save. This one setting covers the whole account, in both Delhi and Chennai. The account owner then gets a WhatsApp message for each alert whose groups include them. To have an alert reach the owner on WhatsApp, add the owner to one of its alert groups.

Other members of the groups get email only, and the email goes out to everyone whether WhatsApp is on or off. Any user allowed to change General Settings can turn Monitoring Alerts on or off, and the change always applies to the owner’s phone number. Choose WhatsApp notifications shows this setting with the account’s other defaults.

Default alerts and the default alert group

When monitoring is set up on a new machine, E2E adds default alerts to it. This happens for a node, a load balancer, a DBaaS instance’s main machine, and a Kubernetes node. The default alerts appear on Manage Alerts next to your own, and you can edit, pause, or delete them in the same way. Alert Condition shows what each one watches.

A node, a load balancer, or a Kubernetes node gets three default alerts, all at High. They are Free Disk Space less than 15%, Consumed Memory greater than 85%, and Processor Load greater than 85%. A DBaaS instance gets CPU Load Average greater than 85%, Free Memory greater than 85%, and Free Disk Space less than 15%, also at High.

Default alerts notify the project’s default alert group. E2E creates this group the first time it adds default alerts in a project and region, with the account owner as its only member. Its name starts with Default-, followed by the account and project numbers, and in Chennai it ends in -CHE. To send the default alerts to your team as well, add your teammates to this group.

E2E’s built-in checks

E2E also runs three checks on every running resource:

• It cannot be reached.
• It has been restarted.
• Its monitoring agent is not available.

When one of these is true, it shows as a problem on the Monitoring Dashboard, like the problems your alerts find. A machine whose agent cannot be reached also has blank graphs; to get it reporting again, see Troubleshooting node monitoring and encryption.