Skip to main content

Edge Agent Troubleshooting

Most problems fall into one of a few buckets: the agent can't reach Haltless, readings are being rejected, or a source device isn't responding. Start with the logs , they name the cause in almost every case.

Reading the logs​

When the agent runs under systemd, its logs go to the journal:

# Follow live
sudo journalctl -u haltless-agent -f

# Last 200 lines
sudo journalctl -u haltless-agent -n 200

# Since a point in time
sudo journalctl -u haltless-agent --since "1 hour ago"

If you run the agent in the foreground, the same messages print to the terminal. For more detail during diagnosis, set log_level: "DEBUG" in config.yaml and restart.

The key line to look for is the heartbeat: it reports the agent's connection status (active, or a reason it can't send) on a regular interval.

Connectivity and firewalls​

Symptom: log messages about connection errors or timeouts; the agent reports it is buffering locally.

The agent needs only outbound HTTPS to api.haltless.io on port 443. Nothing needs to connect into the agent.

  1. Confirm the host can reach Haltless , a 2xx here means the path is open: 
    curl -sS -o /dev/null -w "%{http_code}\n" https://api.haltless.io/health
  2. Check DNS resolves: nslookup api.haltless.io.
  3. Confirm your firewall allows outbound 443 to that host.
  4. If you route through a corporate proxy, make sure the agent's environment is configured to use it (for example, an HTTPS_PROXY value in the service environment).

While Haltless is unreachable, the agent keeps collecting and writes readings to its local buffer. It automatically resumes sending, oldest data first, once the connection is restored , so a brief outage is invisible downstream.

Buffering during outages​

The agent is designed to ride out network interruptions. When it can't send, it persists readings to a crash-safe on-disk buffer (buffer.db in buffer_dir) and drains them in order when connectivity returns.

Two limits keep the buffer bounded, and both mean data loss only in a sustained outage:

  • Size cap (max_buffer_size_mb, default 512) , once the buffer reaches this size while offline, the newest readings are dropped.
  • Retention (~24 hours) , buffered readings older than about a day are evicted automatically.

If you expect long outages, raise max_buffer_size_mb and make sure buffer_dir has enough free disk.

Disk space for the buffer​

Symptom: a growing buffer.db, or the agent warns it is dropping readings because the buffer is full.

A growing buffer almost always means the agent can't reach Haltless , fix connectivity first (see above) and the buffer will drain on its own.

  • Check the buffer size: ls -lh /var/lib/haltless-agent/buffer.db (adjust for your buffer_dir).
  • Make sure buffer_dir is on a volume with enough free space for your worst-case offline window.
  • Prefer fast local storage (an SSD) for buffer_dir so draining a large backlog is quick.
  • Keep buffer_dir on a normal data path such as /var/lib/haltless-agent. System directories are rejected at startup.

API-key and authentication errors​

Symptom: the log reports that readings were permanently rejected and points at the API key.

An authentication failure is treated as permanent , the agent will not endlessly retry a batch that was rejected because the key is wrong, so those readings are dropped and logged rather than buffered forever. To fix it:

  1. Confirm the api_key in config.yaml is correct and has not been revoked.
  2. Check for stray whitespace or a truncated value , the key must be pasted in full.
  3. If in doubt, create a fresh key and update the config. See API keys.
  4. Restart the agent after changing the key.

Subscription paused (payment required)​

Symptom: the heartbeat reports a payment-required status and the agent pauses collection.

This means the account's subscription is inactive. The agent intentionally stops collecting so it doesn't pile up data that can't be delivered. Restore the subscription in the Haltless billing area; the agent rechecks on its heartbeat and automatically resumes , flushing anything it had buffered , once the account is active again.

Rejected readings​

The ingestion endpoint accepts each reading individually and returns a per-reading result, so a single bad reading never blocks the rest of a batch. If you see rejections, check the reason , common ones are below. See the API error reference for the full list.

Machine not found. The machine_identifier on the reading doesn't match a registered machine. Identifiers are case-sensitive; CNC-001 and cnc-001 are different machines. Register the machine, or correct the identifier in the source.

Value must be finite. The value was NaN or infinite. For Modbus, check the scale and offset , a bad conversion can produce a non-finite result. Also check the source device for fault states that emit non-numeric values.

Missing unit. Every reading needs a unit. Make sure your register/node config sets unit, or that your CSV column / JSON key for the unit is populated.

Timestamp out of range. The reading's timestamp is too far in the past or future to accept. This is almost always a clock problem , see below.

Timestamp and clock issues​

Symptom: readings rejected for an out-of-range timestamp, or data appearing at the wrong time on the dashboard.

  • Timestamps are handled in UTC. For Modbus and OPC-UA, the agent stamps readings with the host's current time, so keep the host clock synchronized with NTP.
  • For CSV and JSON, the timestamp comes from your export. If it has no timezone offset, the agent assumes UTC and logs a one-time warning , make sure your exports really are UTC, or include an explicit offset.
  • If readings land in the future or far in the past, check both the source device clock and the agent host clock.

The agent replays buffered readings on reconnect; re-sending the same reading is safe and does not create duplicates.

Rate limiting​

Symptom: the log reports that a batch was rate-limited and buffered for retry.

If you send data faster than your plan allows, Haltless returns a rate-limit response. The agent buffers the batch and retries it later rather than dropping it. To reduce pressure, increase collection_interval_seconds, raise batch_size (up to 1000) so you make fewer, larger requests, or reduce the number of high-frequency sources. See Rate limiting for the limits.

Source device won't connect​

Modbus , no response.

  1. Confirm network reachability to the device on its port (default 502).
  2. Verify the host and port in the config.
  3. Confirm the device speaks Modbus TCP (not serial RTU only).
  4. If values look wrong, re-check address, register_type (holding vs input), and scale/offset. Remember count: 2 reads a big-endian 32-bit float.

OPC-UA , connection or node errors.

  1. Verify the endpoint_url uses the opc.tcp:// scheme and the host/port are reachable.
  2. If the server needs credentials, set both username and password; setting only one causes an anonymous connection.
  3. If a specific node fails, confirm the node_id exists on the server using an OPC-UA browser tool.

CSV / JSON , no readings.

  1. Confirm the watched directory exists and the agent's user can read it.
  2. Check that column names (col_*) or object keys (key_*) match your files exactly.
  3. Remember CSV files are read incrementally (only new rows each cycle) and JSON/JSONL files are processed once , drop each new export in as a new file.

Still stuck?​

Collect the recent logs with the API key and any passwords redacted, note the agent version and the exact error text, and share them with your Haltless support contact. The Getting started guides and the API reference cover the wider platform if the issue turns out to be downstream of the agent.