Troubleshooting

Look for the exact message you see on this page (Ctrl + F, or ⌘ + F on a Mac): most entries start from what the panel, an email, the installer or a log says. The FAQ answers general questions. If nothing here helps, see Getting help.

First checksCopy Anchor Link

These commands answer most questions. Run them on the server concerned:

3rmonitor status                       # installation mode and service status
3rmonitor version                      # installed version
journalctl -u 3rmonitor-master -n 50   # last lines of the master log (on the master)
journalctl -u 3rmonitor-agent -n 50    # last lines of the agent log (on an agent server)
journalctl -u 3rmonitor-helper -n 50   # last lines of the agent's root helper
sudo 3rmonitor agent --test            # an agent reports once and shows the result

Where messages appear:

Many problems are already fixed in a newer version: check System → Software updates first.

InstallingCopy Anchor Link

The install command failsCopy Anchor Link

"Download failed: make sure the master is reachable, the token or enrollment key is correct, and the master was installed from the full release package" means one of those three:

• the server can't reach the panel address (test with curl -I), or the Panel public URL in Settings → General is an address the server can't reach (for example a LAN IP from another network);
• the token or enrollment key is no longer valid: copy the command again from Servers;
• the master has no agent packages. The Servers page then warns that "the dist directory on the master has no agent packages". Reinstall the master from the complete release package: sudo bash install.sh upgrade in the extracted package.

To see why, test the download yourself: curl -v "MASTER_URL/dl/SHA256SUMS?t=TOKEN". An answer of 401 means the token or key is wrong; 404, that the master has no agent packages.

Other messages from the installer (✗ stops or failed, ! is a warning; both are repeated in the summary at the end):

sudo says "command not found"Copy Anchor Link

On AlmaLinux, Rocky, RHEL and CentOS, sudo only searches /usr/sbin and /usr/bin. Before 1.1.1 the 3rmonitor command was only in /usr/local/bin, so sudo 3rmonitor … answered "sudo: 3rmonitor: command not found". Since 1.1.1 the command is also linked in /usr/bin: upgrade, and servers get the link by themselves. Until then, either:

• open a root shell with sudo -i and run the commands there without sudo, or
• write the full path: sudo /usr/local/bin/3rmonitor status.

The command lineCopy Anchor Link

A command says "Error"Copy Anchor Link

Most 3rmonitor commands need root. Without sudo some fail, and some give a misleading answer:

3rmonitor status doesn't list the helper: check it with systemctl status 3rmonitor-helper.

"approvers trust" doesn't trust the keysCopy Anchor Link

The panelCopy Anchor Link

The panel doesn't openCopy Anchor Link

1. Is the master running? sudo systemctl status 3rmonitor-master. If it isn't, the log (journalctl -u 3rmonitor-master -n 50) says why, for example a port already in use or a certificate file it can't read.
2. Is the port open? The installer opens it in ufw or firewalld, but cloud providers often have their own firewall (security groups) in front of the server. Open port 8899 (or your port) there too.
3. Is it listening only on 127.0.0.1? If listen in /etc/3rmonitor/master.conf is 127.0.0.1:8899, the panel is only reachable through the reverse proxy (your domain), not by IP.
4. Through a domain: check that Nginx or Apache is running, and that the domain's DNS points to the master.

A page says "This page could not be loaded"Copy Anchor Link

The browser couldn't get the page's data from the master: the connection dropped, the master restarted (for example during an update), or a reverse proxy in between failed. The short message in the corner shows the error. Click Try again. If it keeps happening, check that the master is running and look at its log; behind Nginx or Apache, also look at the proxy's error log.

I forgot the passwordCopy Anchor Link

On the master: sudo 3rmonitor user passwd USERNAME, then enter the new password. sudo 3rmonitor user list shows the usernames.

After more than 5 failed attempts for one user, or 10 from one IP, in 15 minutes, sign-in is locked: "Too many failed sign-in attempts. Try again in 15 minutes." Even the right password is refused meanwhile. Wait, or restart the master (sudo systemctl restart 3rmonitor-master), which clears the counter.

Behind a reverse proxy that isn't on the same machine, the local network or Cloudflare, every visitor seems to come from the proxy's IP, so a few failures lock everyone out. If the panel can only be reached through that proxy, set trust_proxy = true in master.conf (see Configuration files).

Other messages in the panelCopy Anchor Link

Signing in sends me back to the sign-in pageCopy Anchor Link

This happened in versions before 1.0.4 when the panel URL was https:// but the page was opened over http:// (for example by IP). Upgrade to the latest version. Also check that the browser accepts cookies for the panel's address.

The browser says "Not secure"Copy Anchor Link

The panel is served over http://. Set up HTTPS.

Agents and serversCopy Anchor Link

An agent does not reportCopy Anchor Link

Run on the agent server:

sudo 3rmonitor agent --test

It reports once and shows the result. Then:

In the agent log (journalctl -u 3rmonitor-agent), "Report failed (attempt N): …" gives the reason; it is written on the 1st failure and then every 20th. "connection refused" means nothing listens at that address and port, or a firewall rejects it; "i/o timeout" or "Client.Timeout exceeded", that a firewall silently drops the connection. "Connection to the master restored" shows it works again.

To see what the agent collects without sending anything: sudo 3rmonitor agent --print.

All servers show as offline at onceCopy Anchor Link

If 3 or more servers stop reporting at the same moment, you get one alert, "All N servers stopped reporting at the same time". The servers are almost certainly fine: the master isn't receiving reports. Check the master's network, its firewall (and the provider's), its reverse proxy and its certificate, and whether the panel address or DNS record changed.

Two servers show the same dataCopy Anchor Link

The panel shows one server whose data jumps between two machines, or "Reporting too often" in the log: two machines use the same token. That happens when a server is cloned (a VM template or snapshot) with /etc/3rmonitor/agent.conf. Register the copy as a server of its own (Servers → ＋ Add server), and on the copy run the bind command it gives you:

sudo 3rmonitor bind --master 'https://monitor.example.com' --token 'NEW_TOKEN'

To make templates, uninstall the agent first, or install it with the enrollment key when each machine starts (for example with cloud-init).

The server page says "The 3R Monitor helper is not running"Copy Anchor Link

The agent runs as an unprivileged user and does what needs root through 3rmonitor-helper. Without it, the agent keeps monitoring, but the AI, the blocklist and some data (processes behind ports, MySQL status) don't work there.

sudo systemctl restart 3rmonitor-helper
journalctl -u 3rmonitor-helper -n 50

If it doesn't start, or the page says "The agent still runs as root…" or "Could not switch the agent to an unprivileged user: …", run sudo 3rmonitor agent secure and read what it answers: for example "neither useradd nor adduser is available" (install the passwd or shadow-utils package), or "the helper did not start (…)". It needs systemd.

An agent stays on an old versionCopy Anchor Link

Agents update themselves to the master's version. If one doesn't:

• it can't reach the master (see An agent does not report);
• auto_update = false is set in its agent.conf;
• since 1.1.1, agents only install programs signed by 3R Soft. If System → Agent packages shows Not signed, the master wasn't installed or upgraded from the official release package: run sudo bash install.sh upgrade from the official package. The agent log then says "Automatic update failed: the master has no signature for the agent packages…" or "…are not signed for …".

You can also upgrade an agent by hand with the release package: sudo bash install.sh upgrade in its directory.

The traffic usage looks wrongCopy Anchor Link

• It starts on a recent day: counting began when the master was upgraded to 1.1.1 ("Counted since …").
• "The agent does not report … separately": you chose one interface in Servers → Settings → Traffic usage, but the agent is older than 1.1.1, or that interface no longer exists, so all interfaces are counted. Wait for the agent to update, or choose another interface.
• It differs from your provider's figure: see Traffic usage. Count only the interface your provider bills, choose how the quota counts (download, upload, both or the larger), and remember that 1 GB here is 1024³ bytes.
• A day is missing: the agent didn't report that day (the server or the agent was stopped). The traffic of that time is counted on the day it reports again, unless the server was restarted in between.

A server is in the wrong place on the mapCopy Anchor Link

Set its position in Servers → Settings → Map location. The master's own position is in Settings → Map. See Server positions.

MySQL / MariaDB shows "Cannot log in to the database"Copy Anchor Link

The agent can't sign in to read the database status. See MySQL credentials for the agent. Also:

• "mysql client not found; cannot read the server status": install the mysql or mariadb client on that server.
• On Debian and Ubuntu, /etc/mysql/debian.cnf is only used while the helper is running: see the helper is not running.
• "Redis requires a password; its status was not read": only information, Redis itself is still monitored as a service.

A service alert I don't understandCopy Anchor Link

Alerts and emailsCopy Anchor Link

Alert emails don't arriveCopy Anchor Link

1. Open System → Recent emails: each email shows whether it was sent, is being retried or was given up, and the error.
2. Send a test: Settings → Outgoing mail (SMTP) → Send a test email, or sudo 3rmonitor mail-test --to you@example.com on the master.
3. Common causes:
   • Gmail needs an app password, not the normal password.
   • Many cloud providers block outbound port 25. Use 465 (SSL) or 587 (STARTTLS).
   • The Encryption doesn't match the port: SSL with 465, STARTTLS with 587.
   • The server has Enable alerts turned off in its settings, or that kind of alert is turned off in Settings → Alert notifications → Send emails about.
   • The email went to the spam folder. Set a From address on a domain the SMTP server is allowed to send for.
   • System shows Alert recipients: Not set: without recipients nothing is sent, and only the master log says so.

The error of a failed email, in the test or in System → Recent emails:

Failed emails are retried up to 12 times, waiting longer each time (about 6 hours in all), then marked Gave up.

Alert emails arrive lateCopy Anchor Link

That is usually on purpose:

• Alerts are combined: the first one is sent after 1 minute, then at most one email every 5 minutes. Set Settings → Alert notifications → Combine alert emails (minutes) to 0 to send each alert right away.
• With AI analysis, an email about a new outage waits up to about two more minutes for the likely cause.
• Above At most this many emails per hour (10 by default), emails wait in the outbox. System shows how many are queued.

The AI found something but didn't email meCopy Anchor Link

The AI only emails what needs you, and at most AI → Settings → AI emails → Emails per day (5 by default). Each investigation shows on its timeline whether it emailed, and why not:

Tasks you give the AI are never emailed. System alerts (servers, services, monitors, certificates, disks, traffic quotas) don't depend on this. See Emails from the AI.

MonitorsCopy Anchor Link

A monitor says "down" but the site worksCopy Anchor Link

The check runs from the master, so what matters is what the master sees. Try from the master itself:

curl -sv -o /dev/null https://example.com/health

• HTTP 403, possibly blocked by a Cloudflare challenge: allow the master's IP in Cloudflare or in your WAF.
• HTTP 301/302 when you expected 200: redirects are followed by default; if you ticked Do not follow redirects, add the code to Expected status code (for example 200,301).
• "… was not found on the page": the keyword is case-sensitive, and only the first 2 MB of the response are read.
• Timed out: raise Timeout (s) if the endpoint is slow, or check that the master can reach it (firewall, IPv6, DNS).
• Certificate error: the certificate is expired or incomplete. For a self-signed certificate, tick Do not verify the certificate.

Click Test in the monitor's form to see the exact message. What the messages mean:

A Ping monitor is always downCopy Anchor Link

Ping uses ICMP, which many networks and cloud providers block. If ICMP is blocked between the master and the host, use a TCP port monitor instead (for example host:22 or host:443). The master also needs the ping command; it reports "The ping command is not available on this system" if it's missing.

Each check sends one packet, and any failure of ping is shown as "No response (packet loss)", also when the master isn't allowed to send ICMP. Test from the master with ping -c 1 HOST.

Traffic and the blocklistCopy Anchor Link

There are no traffic statisticsCopy Anchor Link

• The agent reads Apache and Nginx access logs in the usual locations. If yours are elsewhere, set access_logs in /etc/3rmonitor/agent.conf (for example access_logs = /data/logs/*.access.log) and restart the agent.
• Since 1.1.0 the agent runs as the unprivileged user 3rmonitor-agent, and the installer gives it read access to the logs with setfacl. If the acl package was missing, the agent log says "The agent (user 3rmonitor-agent) cannot read the access logs in …": install acl and run sudo 3rmonitor agent secure again. Do the same after setting access_logs to a new location.
• access_logs replaces the usual locations, it doesn't add to them: list every location you need.
• Rotated and compressed logs (access.log.1, access_log-20261020, .gz) and error logs are skipped on purpose.
• access_logs = off turns the statistics off.
• Countries need the IP-to-country database. The System page shows whether it is loaded and when it was last updated.
• Behind Cloudflare, all requests come from Cloudflare until the origin restores the real IP. See Cloudflare tips.

The blocklist doesn't blockCopy Anchor Link

When you add a rule, the panel refuses what would do harm:

I locked myself out with the blocklistCopy Anchor Link

From the server's console (your provider's web console, if SSH is blocked):

sudo 3rmonitor firewall off

The blocking rules are removed immediately. Fix the rules in Traffic → Blocklist / allowlist, then run sudo 3rmonitor firewall on. To avoid it in the future, set Blocklist only blocks these ports to 80,443.

Since 1.1.1, country rules and ranges larger than /24 never block the SSH port, so only a rule for your own IP or a small range around it can still lock you out of SSH.

A blocked country can still connect with SSHCopy Anchor Link

That is on purpose since 1.1.1: country rules and large ranges block every port except SSH, so that you can always get back in. To block SSH for an attacker, block the single IP or a small range (/24 or smaller). See So you don't lock yourself out.

VulnerabilitiesCopy Anchor Link

A server shows no vulnerabilities, or isn't checkedCopy Anchor Link

The Security page says why at the top:

Data sources and how matching works shows when each source was last updated. "Failed: …" there means the master can't reach it: it needs outbound HTTPS to api.osv.dev, services.nvd.nist.gov, www.cisa.gov, endoflife.date, cveawg.mitre.org, raw.githubusercontent.com and github.com. "NVD: HTTP 403" or "429" means NVD is limiting requests: add a free NVD API key in Settings → Vulnerability checks.

A vulnerability I fixed is still listedCopy Anchor Link

• The agent uploads a new software list when software changes, and everything is checked again every 6 hours. Click Scan now on the Security page to check right away.
• Upgrade the packages under "installed as …", not the source package name in the title: for example perl-base, not perl.
• "Nothing was upgraded: no newer version is available from the configured package sources…": the distribution hasn't published the fix yet, or the server's package sources are out of date. Tick Hide those with no fixed version yet to hide such vulnerabilities, or Ignore this one with a reason.
• If the service must be restarted to use the new version, the old one may still be running: restart it.

The AICopy Anchor Link

The AI doesn't run anything on a serverCopy Anchor Link

AI → On-call → Servers where the AI can do less, and the Agent row in the server's System tab, say why. In order:

The model test fails, or reports failCopy Anchor Link

The error appears in the Test message, on the report ("The report failed: …"), or in an investigation ("The model failed: …"):

An investigation stopped, or the AI only suggests repairsCopy Anchor Link

When the server itself refuses an operation, the operation shows why:

A repair was refused with "Limit on this server"Copy Anchor Link

Since 1.1.1 each server limits the repairs approved only by the reviewer AI, by itself:

• "Limit on this server: this repair already ran 3 times in the last hour": the same repair keeps coming back, so the cause is elsewhere. Look at the investigation, and fix the cause yourself.
• "Limit on this server: N automatic repairs in the last 24 hours (ai_auto_per_day in agent.conf)": raise ai_auto_per_day in that server's agent.conf if you really want more, or approve the repair yourself: repairs you approve aren't limited this way.

"This conversation is closed"Copy Anchor Link

A task's conversation can be continued for 24 hours after the AI's last answer. The message says why it closed (it was stopped, a limit was reached, the model failed…): give the AI a new task instead. See Continue a task.

PasskeysCopy Anchor Link

I can't add a passkey, or an approval failsCopy Anchor Link

• Passkeys need the panel on HTTPS (or localhost): the ＋ Add passkey button is disabled on an http:// panel. Open the panel at its https:// address. See HTTPS.
• "This browser does not support passkeys" or "cannot register passkeys": use a current Chrome, Edge, Safari or Firefox.
• "The passkey request was cancelled or timed out": try again and confirm within two minutes.
• A passkey only works at the host name it was created on. Open the panel at the same address.
• A new passkey stays Waiting for a signature until an administrator with a trusted passkey confirms it.
• "The approval was signed with a passkey this server does not trust": run sudo 3rmonitor approvers trust on that server.
• An approval is valid for one hour. After that the operation is marked Expired.
• "Approvals are signed with a passkey. Add one in Settings → Passkeys first": you have no passkey yet.
• "Sign the approval with your passkey (the request expired or the operation changed); try again": the request is valid for 5 minutes.
• "Sign with one of your own trusted passkeys": the passkey you used is still Waiting for a signature, or belongs to someone else.
• "The browser's answer does not match the request": the panel was opened at another address than its own. Use the panel's public address.
• "This passkey is already registered": that device already has a passkey for this panel.
• Settings → Passkeys shows "Not trusted yet: on the server run sudo 3rmonitor approvers trust…" after your first passkey: every server must trust it once, also servers installed before you had a passkey.
• If every passkey is lost, see Passkeys.

A server stays on "getting the newest record…"Copy Anchor Link

Since 1.1.1 servers pick up a new passkey record within seconds, and Settings → Passkeys refreshes by itself meanwhile. If a server stays there, its agent can't reach the master, or its helper isn't running: check systemctl status 3rmonitor-helper and journalctl -u 3rmonitor-helper -n 50 on that server.

UpdatesCopy Anchor Link

The update was downloaded but not installedCopy Anchor Link

Since 1.1.1, nothing is installed until you confirm. System → Software updates says "x.y.z is downloaded and its signature checked. It is installed only after you confirm.": click Install now… or Install. "This version has not been downloaded yet; download it first" means the download is missing or a newer version was published: click Update now again.

An update from the panel failedCopy Anchor Link

The System → Software updates card shows the error and the installer output. You can always upgrade by hand with sudo bash install.sh upgrade from the new package.

The master doesn't start after an upgradeCopy Anchor Link

Since 1.1.1, install.sh runs the master in a stricter systemd sandbox. On the few systems that don't support it (some LXC containers), the installer removes it by itself and says "This machine does not support the extra isolation for the master…". If the master still doesn't start, look at journalctl -u 3rmonitor-master -n 50, and remove the sandbox by hand:

sudo rm -r /etc/systemd/system/3rmonitor-master.service.d
sudo systemctl daemon-reload && sudo systemctl restart 3rmonitor-master

Getting helpCopy Anchor Link

If this page and the FAQ don't solve it:

1. Upgrade to the latest version (System → Software updates): the problem may already be fixed.
2. Email info@3rsoft.com.pa. In the panel, System → Help and support → Problems or bugs opens an email with your version in the subject.

To get an answer on the first email, include:

• what you did, what you expected, and what happened instead, with the exact message or a screenshot;

• the output of these commands on the server concerned (the master, or the agent server):

  Copy Text

  3rmonitor version
  3rmonitor status
  cat /etc/os-release
  journalctl -u 3rmonitor-master -n 100 --no-pager   # on the master
  journalctl -u 3rmonitor-agent -n 100 --no-pager    # on an agent server
  journalctl -u 3rmonitor-helper -n 100 --no-pager   # on an agent server
  sudo 3rmonitor agent --test                        # on an agent server

• for an installation problem, the complete output of the installer.

Don't send agent.conf, master.conf, tokens, enrollment keys, passwords or API keys: nobody at 3R Soft needs them. Replace them with … if they appear in what you copy.

Report security problems privately to security@3rsoft.com.pa, not by the normal address. See Reporting a security problem.