AI on-call
Let the AI investigate outages and problems by itself, run diagnostics on your servers and apply the repairs you allow, with every step recorded and every risky step left to you.
The on-call engineer is an AI that investigates problems by itself: it looks at the monitoring data, runs diagnostics on the server, applies a repair when it knows the cause and you allow it, checks the result and writes a conclusion. Every step is recorded.
On-call is off by default. It needs AI analysis to be set up first.
Turn it on
Open AI → On-call and turn On-call on (or use the switch on AI → Overview).
Choose what the AI may do on servers. The default, Diagnostics only, lets it investigate but leaves every repair to you.
If you want it to run repairs you approve, register a passkey and let your servers trust it.
Sandboxed diagnostics and approved repairs need agent 1.1.0 or newer on the server. Agents update themselves to the master's version; AI → On-call lists the servers where the AI can do less.
When it starts
| Trigger | When |
|---|---|
| Outage | An outage of a server or monitor is confirmed |
| Patrol | Every 60 minutes by default (AI → Settings → Patrol every). 3R Monitor checks first, without the AI, and only wakes the AI when it finds something to look at, so a quiet patrol costs nothing. |
| Vulnerability | An urgent vulnerability appears. The on-call engineer checks whether the software is running and reachable, prepares the upgrade command, and then decides whether to email you. |
| Your task | You give it a task |
| Check after approval | Operations you approved have run, and the AI checks the results |
Give it a task
On AI → Overview or AI → On-call, describe the task in the box, for example "find out why the shop is slow in the evenings", choose All servers or one server, and click Start (or press Ctrl / ⌘ + Enter; Enter starts a new line). What you are typing is kept while the page refreshes itself.
A task can run longer than an automatic investigation: up to 60 checks and 30 minutes.
The result of a task is shown in the panel; it is never emailed. When the AI has finished, you can continue the conversation.
Risk check: a task such as "check what risks this server has" covers patches, exposed ports, SSH, failed logins, accounts and sudo, scheduled tasks, processes and connections, the web server, certificates and system hardening. It ends with a risk check conclusion: each finding by severity, with the evidence, the fix and who does it (done by the AI, approve in the panel, or run it yourself).
What it can do on a server
Everything the AI wants to run is checked by the master, again before it is sent, and again on the server by the root helper. The levels, from safest to most dangerous:
| Level | Examples | What happens |
|---|---|---|
| Read-only | The fixed checks (service status and logs, failed services, system errors, kernel messages, disk space, directory sizes, large files, memory, processes, listening ports, connections, web server configuration test, a log file, available updates, open a URL, test a port, SSH settings, failed logins, firewall rules, accounts and sudo, scheduled tasks, certificates, AppArmor / SELinux, kernel security settings, automatic updates, containers and ports, open connections, DNS lookup, ping), and plain read-only commands such as ls, cat, grep -r, tail, ps, ss, journalctl or systemctl status | Runs right away. Commands that read file contents run as an unprivileged user, so files only root can read stay hidden. |
| Sandbox | Any other diagnostic: pipes, loops, awk, sort, find … -exec grep, a diagnostic script | Runs in the sandbox after the safety reviewer approves it. When the reviewer isn't sure, it waits for you. |
| Low risk | Start, restart or reload a normal service; clean old journal entries or the package cache; rotate logs; empty a huge log file | Runs after the reviewer approves it, or waits for you, depending on the setting. A service is restarted at most 3 times an hour, and automatic repairs are limited to 20 a day. Each server also limits them by itself: the same repair 3 times an hour, and 20 in 24 hours (ai_auto_per_day in its agent.conf). |
| Needs approval | Upgrade one package, stop or disable a normal service, end a process | Always waits for your Approve and run, signed with your passkey. The agent then runs exactly that operation, once, within an hour. |
| High risk | Reboot the server, install all updates, edit configuration, anything on SSH, network, firewall, VPN, Docker or 3R Monitor itself, any other command that changes the system | Never run by the system. It is listed for you to run yourself, with the AI's reason and the reviewer's opinion. |
| Malicious | rm -rf, mkfs, dd onto a disk, curl … | sh, fork bombs, reverse shells, reading or searching for passwords and keys, flushing the firewall, stopping SSH or the monitoring | Refused however it is written (with sudo, full paths, quotes, split options): never run, not even suggested. No approval changes that. |
Upgrading a package upgrades the packages that are really installed: vulnerabilities name the source package (for example perl or brotli), while the server has perl-base or libbrotli1. The package lists are refreshed first, an upgrade that changes nothing counts as failed, and the software list is read again right after, so a fixed vulnerability disappears within seconds.
The sandbox
Diagnostics at the Sandbox level run:
- as a throwaway unprivileged user, with a read-only filesystem and no privileges;
- for at most 120 seconds, with at most 256 MB of memory;
- with passwords and keys removed from their output.
The sandbox can only reach the server itself and the networks you allow for that server in Servers → Settings → Networks the AI sandbox may reach, within the local limit sandbox_networks_max in agent.conf (private networks by default). Everything else is blocked by the kernel; where the kernel can't filter, the sandbox has no network at all.
The reviewer alone isn't enough for everything: scripts may only use a fixed list of read-only tools. Anything else, and reading home directories or mail, needs an administrator's passkey approval.
The sandbox needs systemd 236 or newer and D-Bus on the server. Without it, only the fixed checks run.
What the AI may do on servers
Choose it in AI → Settings → What the AI may do on servers:
| Choice | Meaning |
|---|---|
| Off: nothing on servers | The AI only uses the monitoring data |
| Diagnostics only (default) | Logs, status, disk, processes, sandboxed scripts… repairs are left to you |
| Repairs need my approval | Diagnostics run by themselves; each repair waits for you |
| Low-risk repairs run automatically | After the reviewer approves, with hourly and daily limits. Upgrades, stopping services and ending processes always wait for you. |
Whatever you choose, high-risk operations are never run by the AI. Automatic repairs per day (20 by default) limits the automatic ones; above that, they wait for your approval.
Each server keeps its own count too, so that even a master that was taken over can't run more: a repair approved only by the reviewer runs at most 3 times an hour, and at most 20 such repairs run in 24 hours (ai_auto_per_day in agent.conf). Above that, the server refuses it, with "Limit on this server: …". Repairs you approve yourself aren't limited this way.
To exclude a server, tick Never let the AI run anything on this server (not even read-only diagnostics) in Servers → Settings → AI on-call. On the server itself, root can limit what the helper accepts, and the agent can't change it:
sudo 3rmonitor ai status # what the panel's AI may run on this server
sudo 3rmonitor ai off # nothing
sudo 3rmonitor ai read # read-only and sandboxed diagnostics only
sudo 3rmonitor ai on # also approved repairs (the default)Waiting for you
Operations that need you are listed under Waiting for you on AI → Overview, on AI → On-call and in the investigation itself. Each one shows what it would run, why, its level and the reviewer's opinion.
| Status | Your options |
|---|---|
| Waiting for your approval | Approve and run: you confirm with your passkey (fingerprint, face or PIN), and the server runs exactly this operation, once, within an hour. Reject. |
| For you to run | Copy command and run it yourself if you agree, then I did it. Or Ignore. |
A block the AI made can be removed with Undo.
When an investigation needs you, the AI can email you: the result, with the repairs that were done, problems that need you and operations waiting for your approval, goes into the combined alert email. It counts toward the AI's emails per day, so check Waiting for you in the panel too.
Investigations
AI → On-call → Investigations lists every investigation with its trigger, how many operations were done, are waiting or were refused, and its result:
| Result | Meaning |
|---|---|
| Working… | Still running. Stop ends it. |
| Standby: waiting for approval | The conclusion is written, and operations wait for your approval |
| Fixed · Improved | The problem is solved, or better |
| Needs you | Something only you can do |
| Nothing to do | Nothing was wrong, or nothing could be done |
| Failed · Stopped | The investigation failed or was stopped |
Open one to see the conclusion first, then What the AI did: every lookup, command and refusal, with its output.
Standby: when the AI has finished but operations still wait for your approval, it writes the conclusion right away. After you approve them and they have run, it comes back to the same investigation, checks the results, continues if needed and updates the conclusion (the earlier one stays in the timeline). Rejected or expired operations are only marked under the conclusion.
The AI remembers earlier investigations of the same server or problem from the last 30 days.
Continue a task
Under the conclusion of a task you gave the AI there is a Conversation. Ask a follow-up question, or tell the AI what to do next, and click Send (or press Ctrl / ⌘ + Enter). It answers with everything from that investigation, the checks it ran and their results, instead of starting again:
- it can look things up and check servers as in the task; repairs go through the same checks and approvals;
- the answer appears in the conversation, and the conclusion stays as it is;
- if you write while the AI is still working, it answers when it has finished what it is doing; at most 5 messages can wait for an answer.
You can continue for 24 hours after the AI's last answer; the hint under the box says until when. The conversation closes earlier, with "This conversation is closed: …", when:
| Reason shown | |
|---|---|
| it was stopped | You clicked Stop, or Stop all AI operations. Stop asks first, because the conversation can't be continued afterwards. |
| it reached the time, step or token limit | The task used all its checks, time or tokens |
| too many operations were refused | The reviewer or the rules refused too many operations |
| the model failed | The model returned errors |
| the conversation became too long for the model | Too much text for the model to read again |
| more than 24 hours have passed since the AI's last answer | |
| the master restarted while the AI was working | |
| it was made before conversations were possible | The task is from before 1.1.1 |
To go on after that, give the AI a new task. Investigations the AI started by itself (outages, patrols, vulnerabilities) can't be continued. Steps, time and tokens are limited; if the model fails, the default model is tried.
Blocking IPs
The on-call engineer can block one public IP by itself only with evidence in the access logs and the reviewer's approval: for 1 hour by default and 7 days at most, at most 10 an hour and 50 a day, and the block expires by itself. Any range of addresses needs a passkey approval.
These are never blocked, not even with approval: allowlisted and protected addresses, IPs that administrators signed in from, your servers and the master. Each block records its evidence and who approved it.
Stop everything
Stop all AI operations (on AI → Overview and AI → On-call) sets what the AI may do to Off, stops the running investigation, cancels queued and waiting operations, and closes every conversation. Turning On-call off stops new investigations.
Servers where the AI can do less
AI → On-call lists the servers where the AI can't do everything:
| Shown | Why | What to do |
|---|---|---|
| Limited: "Agent x.y.z: update it to 1.1.0 or newer…" | An older agent only runs the older read-only checks and low-risk repairs | Upgrade the master; agents follow automatically |
| Limited: "No sandbox on this server…" | No systemd 236 or newer, or no D-Bus | Only the fixed checks run |
| Nothing runs | The agent still runs as root, or its helper isn't running | Run sudo 3rmonitor agent secure on the server. See Troubleshooting. |
The Agent row in the System tab of a server's details page also shows how its agent runs. Repairs only run on servers that trust the master's signing key, and the repairs you approve only on servers that also trust your passkeys: see Passkeys.
How it is protected
- The agent doesn't run as root. Since 1.1.0 it runs as the unprivileged user
3rmonitor-agent. AI operations go through a small root helper,3rmonitor-helper, which fetches each operation from the master itself, checks it again with the same rules, checks the approval and its signature, and only then runs it. - Three checks: on the master, again before an operation is sent, and again by the helper on the server, which doesn't trust the level the master gave.
- Limits on the server itself: the helper counts the repairs approved only by the reviewer, and refuses more than 3 of the same repair an hour, or more than 20 in 24 hours.
- A reviewer: another AI, with its own context, checks every repair and every sandboxed diagnostic: necessity, side effects, files read or changed, network targets, data leaving the server, resources, and attempts to get around the sandbox or a refusal. Text from logs, web pages and command output is treated as data, never as instructions. Repeated refusals stop an investigation.
- Signed approvals: your approval is signed with your passkey over the exact operation, and the master signs every operation, so not even the master can approve, change or add anything alone. See Passkeys.
- Privacy: names, IPs and domains are sent to the model as codes.