Logging and monitoring a live trading bot: what to record and what to watch

When a live system misbehaves, the logs are all you have. Written well, they let you rebuild exactly what the software believed at each step. Written badly, they tell you that something happened and nothing about why. This guide covers what we record, what we keep out, and how we know a bot has stopped.

What we log

• Every order that leaves the system: symbol, quantity, order type, intended price and a timestamp.
• Every fill that comes back: confirmed price, confirmed quantity, commission and the broker’s own execution ID.
• Every start and stop, with a session banner. The banner lists the markets the bot is active in and the ones it is skipping, with the reason.

What we never log

API keys, account credentials, or anything that could be used to rebuild a login. If a field holds a key or a token, it is replaced with a placeholder or dropped entirely. We do not mask it with asterisks. The safest value in a log is one that is simply not there.

Keep the audit log apart

Audit records go to their own file, separate from the operational log. The audit file has to survive a crash. The operational log can be as chatty as you like, and it is allowed to rotate away.

Log transitions, not only events

Most setups log “order placed” and “order filled”. What they miss is what the system believed in between. A line that says “position: 2 shares” is less useful than one that says “position changed from 2 to 1 after fill of order 47”. The second tells you which callback caused the change, and that is what you need when you rebuild the sequence.

Two bugs the logs caught

Both came up while we were testing our own software, one in a pre-market data refresh and one on a paper account.

An error code that was not an error

A pre-market refresh of historical data kept hitting Interactive Brokers error 162. Our first version treated it as a failure and aborted the daily cycle. In fact 162 means no data is available, which is exactly what you get on weekends and holidays. The log line was there, but the code reacted to it wrongly. Once we handled 162 as a normal result, the weekend runs stopped breaking. The lesson: log the error code and its plain-English meaning, not only the number.

A repeated callback that zeroed a position

On the paper account, the recorded position dropped to zero after only one lot had filled. Interactive Brokers can send the same order-status callback more than once, and its documentation says so. Our code read the “filled” field as the amount to subtract right now. It is a cumulative total. The first callback said filled: 1, and we subtracted 1. An identical second callback said filled: 1 again, and we subtracted 1 again.

In a live account that would have left a real lot at the broker, untracked and with no protective orders, and nothing in the logs would have said anything was wrong. What caught it was not an error. It was the routine position line we write on every status update, showing the quantity before and after. Seeing “qty: 2 to 1 to 0” on two callbacks with identical content gave it away.

The fix was one line: compare the new filled value with what we had already recorded for that order, and subtract only the difference. But the audit trail had to exist first. Had we logged only the final state, we would have seen “position closed” and assumed it was correct.

How we know a bot has stopped or is stuck

We use two layers.

1. Outside and inside checks. The bot writes a session start banner to a daily log the moment it connects, and a session end line when it disconnects cleanly. A missing end line means it did not shut down gracefully. Separately, UptimeRobot watches the server from outside. It does not care whether the app thinks it is alive, so a frozen process that no longer accepts connections is caught even when the app’s own logs look fine.
2. Silence from the broker. A broker sends informational messages constantly: routing notices, data-type confirmations and heartbeat-style pings. If they stop arriving, something is wrong even though no error has been raised. The bot keeps a “last heard from broker” timestamp and logs a warning when it goes stale.

One time zone in every log

The broker stamps messages in one time zone, the server runs in another, and a developer’s laptop is in a third. Pick one zone and convert to it at the moment you write the line. Our order timestamps use the exchange’s own zone, which for US markets is US Eastern. Without that rule you will eventually spend an afternoon chasing a timing bug that does not exist. The wider time rules, including holidays and half days, are in our guide to time, holidays and half days in a trading bot.

A short checklist

• Log every order, every fill and every start and stop.
• Record the broker’s own execution ID on every fill.
• Never write keys, tokens or credentials to a log, even masked.
• Keep the audit file separate from the operational log.
• Log the state before and after each change, not only the event.
• Log an error code together with its plain-English meaning.
• Write a start banner and a clean end line every session.
• Run an outside monitor that does not depend on the app’s own view of itself.
• Track when you last heard from the broker, and warn when it goes quiet.
• Put every timestamp in one time zone at the point of logging.

Where logging and monitoring sit in the whole system is shown in our reference architecture, and more of what separates a demo from a live system is in the edge cases of live trading systems. If you trade through Interactive Brokers, our Interactive Brokers page explains how a system handles the connection.

Where we fit

We build logging and monitoring into the software we write for clients, to each client’s own specification. This guide is about software engineering only. It does not give investment advice or recommend any trade. If you are planning a build, you can send us your brief.