You have been hacked. Now what?

To begin with, have a coffee and relax. Your server has already been hacked, which is a good sign - your server and your data are valuable enough for somebody to invest time and efforts into hacking it. Obviously something interesting keeps you busy and you are definitely not wasting your time.



When - I say "when" and not "if", because no matter how well you protect your server, sooner or later it will be hacked - so, when your system is hacked (and you know about it) you have to relax, accept the situation and follow your plan. You should have planned for this day exactly the same way as the UK officials plan for the day when Her Majesty passes away. Not the happiest event, but everybody should keep calm and know what to do.

NB: if you know about the hack, you are lucky. Really bad things happen if your system is compromised and you have not got a clue about it (detecting breaches requires a separate discussion).

Firstly, you recover from your backups - of course you have regular backups, well, you should either have backups or once and forever learn this sad lesson and start doing backups. If you think your system is too complicated to be backed up, consult somebody with more experience, absolutely everything can and ought to be backed up. If your architecture is planned wisely, recovering from backups takes 10-15 minutes. Do not delete the hacked server / instance, simply stop or isolate it, you might wish to analyse the hack and learn how to protect your system.

Cleaning malicious code from your system is too long and too boring, so recover from a clean copy of your product's codebase and do not forget to update the system to the last available version of all software, libraries and modules, which might be tricky because your server side applications/scripts must support the latest version of these libraries, modules, etc, which means you need to be constantly in "development" mode. You cannot just create some software once and live with it forever (true for both backend and frontend, by the way). Even in the most frequent case, a website on a CMS (content management system) - be ready to update your CMS regularly, sometimes it requires extra development work.

Secondly, you would probably never be able to figure out what type of hack you are dealing with. The attacker might have gained control over your system to inject malicious programs or the attacker might have stolen or damaged your data or both. The best thing you can do is keeping your codebase and data separated, in production and in backups, if your application server stores no data you can replace it with a clean version in minutes, avoiding loss of time on cleaning the infected instance.

Data damage or loss is much worse, inevitably you will have some data losses, due to bugs or attacks, but your users can survive a loss of a part of their chat messages or a mess in them. They forgive you short occasional downtimes, as they do forgive Facebook, Twitter and Skype. The real money industry demands more meticulousness, that is why banks still print all transactions and archive piles of paper files in order to provide a way to recover should electronic databases get corrupted. But if your are not a bank, proper database backups solve this problem for you.

Data leaks are your nightmare. Internet giants can afford huge fines or can appeal them in courts endlessly. Smaller companies will go bankrupt if users decide to legally sue them for damages caused by data leaks. Consider an insurance to provide a remedy to your users and consider data minimisation to mitigate the risks of such damages.

But the worst hack that ever strikes you is when the attacker gets full access to your hosting environment (e.g. steals your account password). Should it happen, all security measures go down the drain. Your encrypted data in the database becomes easily accessible, because keys and algorithms can be found on the application server hosted under the same account. In reality it does happen frequently due to human weakness: developers and sysadmins do not learn all passwords by heart, they put them down somewhere, on their laptops or stick-it notes or in notepads. So do you. Employees quit jobs and keep access credentials, contractors finish projects, nobody bothers to change passwords and keys. You paranoiacally control a comparatively small team but when your company eventually grows to the size of "big company - nobody's money", you can hardly know the names of all those who have you by the short and curlies.

I will not get into corporate culture and family spirit managerial mantras, despite them breaches and vulnerabilities became integral part of the reality long ago. You have to accept that this way or the other, sooner or later your system will be hacked, and you will need to struggle through somehow. Have your recovery plan at hand, try it (yes, I mean a complete simulation) and when it happens, have a coffee and relax.