On this page
Anyone running infrastructure on the public internet is being probed right now. Scanners try known weaknesses within hours of their disclosure, and stolen credentials are tested against every login page attackers can find. Most of that activity is noise, but some of it isn’t.
When something real does happen, you need an alert that fires while it is happening and enough evidence afterwards to work out exactly what was touched. Both depend on decisions you make long before the attack. What follows is what we have learned to get right while things are quiet, based on the activity we see across the industry today.
1.Know your retention before you need it
Every investigation starts with the same question, which is what still exists. That means what you can actually read today, not what you believe you log.
Retention varies widely between sources. One provider keeps sign-in records for months, while another keeps the record of who changed code or settings for only a few days unless you export it. Your own application logs might be kept for weeks and your load balancer’s for hours. If an attacker got in two weeks ago, some of the evidence may already be gone.
So write it down. For every log source that matters, know how long it is kept, where it is kept and how to extend it, and export the short-lived ones somewhere you control. It is far better to do this once and calmly than to discover the gaps in the middle of an incident.
2.Don’t throw away late logs
Logs don’t always arrive in order. A machine loses its network for a while, an agent restarts, or a batch job uploads yesterday’s records this morning. Many logging systems quietly reject anything that arrives too long after the fact, because doing so keeps them fast.
That is a trap. The logs that arrive late are often the interesting ones, because they come from a machine that was struggling or one that someone was tampering with. Check what your pipeline does with late data and make sure it is kept and searchable rather than discarded. If anything does get dropped, you want to hear about it the moment it happens.
3.Watch the watchers
The quietest failure in security is monitoring that has stopped working. A crashed web server is loud, but an alerting system that can no longer read its data is silent, and silence looks exactly like an all-clear.
Treat your monitoring as a production system with alerts of its own. Page someone when scheduled checks stop completing, when a data source that is never quiet goes quiet, or when the pipeline starts dropping records. It is worth asking directly how long it would take your team to notice if your alerts went blind right now. If the honest answer is that someone would spot it eventually, build the alert that turns eventually into minutes.
4.Hunt carefully
Threat hunting means searching your own history for signs of something you didn’t catch in real time, such as an address that appeared somewhere it shouldn’t or a token used from an unusual place. It is worth doing regularly and not only after an incident.
Hunt with care, though. Searches over large amounts of history are heavy, and the systems you search are often the same ones your alerts read from, so a single enormous query can slow everything else down at exactly the wrong moment. Keep each hunt narrow and time-boxed. Ask about a short window, step back through history in stages, filter inside the query instead of pulling everything back, and run one search at a time. A careful hunt usually comes back faster anyway.
5.See what actually runs
Logs tell you what applications chose to report, which is not always what is actually running. If something malicious is started inside a workload, the application’s own logs may never mention it.
Program-level visibility closes that gap. When you know which processes start, what they connect to and which files they touch, an unexpected program shows up as exactly that. Build privacy in from the start by recording what ran and where it connected rather than the contents of people’s data, and by being clear about who can see that record and for how long. Visibility that your customers can’t trust becomes a liability of its own.
6.Know your own tests
Security testing is healthy. Scans, penetration tests and red-team exercises find real problems, but they also look a great deal like attacks, and that creates a place to hide.
If anyone can run a test without telling anyone, a real attacker can shelter behind the assumption that it was probably a test. So make testing something you approve and record, including who is testing, from where, what is in scope and when. Anything that looks like an attack and isn’t on that list is then treated as an attack until proven otherwise.
7.Check the obvious first
Most real incidents don’t start with something clever. They start with a credential, whether that is a reused password, a token that leaked into a log or a repository, or an account that should have been closed months ago.
So when something looks wrong, start with the basics before the exotic. Ask which credentials could reach the affected system and whether any of them have been used from somewhere new, who signed in and from where, and whether that matches what you would expect. Look for tokens that never expire or that nobody can explain. These checks are quick and unglamorous, and they are very often where the answer is.
The short version
None of this needs a big budget, but it does need decisions made in advance.
- Know how long every important log is kept, and export the short-lived ones.
- Keep late logs, and never discard data for being old.
- Alert when your monitoring itself goes quiet or starts dropping data.
- Hunt in narrow, time-boxed steps so that a search can’t overload your alerts.
- See what actually runs inside your workloads, with privacy built in.
- Approve and record every security test.
- Check credentials, tokens and sign-ins first.
Attackers only need you to be unprepared once, and the time to get this right is while things are quiet.