How I got hacked in 10 hours on Google Cloud Platform
This is a long article, covering my entire journey in the Google Cloud Platform, choosing the right tool for my websites (mostly blogs), configuring the entire thing, and crashing in despair.
Trials and tribulations, sweat, tears, and the surprising defeat (for now…), I hope you’ll enjoy it.
Â
Free Credits!
About a year ago, I received a grant for the Google App Engine (for a specific project I had).
I didn’t end up using most of them, but it helped me experiment with App Engine (back then it was the only component offered), and get familiarized with it.
On The Web Summit, I received another big bunch of free credits, but this time, it was for the entire Google Cloud Platform, which by then grew to be quite a big contender to AWS.
I decided to try and move all my websites out of my shared hosting account (it felt like it was going too slow at times), and into the Google Cloud Platform.
It took me a while to understand what each service does, and what I needed to use.
I started with
Google App Engine
The Google App Engine is a fully managed PaaS (Platform as a Service), it scales automatically (similar to EC2, but more auto-managed), uses a custom way to store data (although one could use Cloud SQL instead), and is less than intuitive when it comes to deployment.
After understanding the idea of how the app.yaml should look like for a WordPress website, and finally deploying the website (App Engine is limited to 10,000 files per version, so keep it lean), I had trouble using the Developers Console to see what’s going on. I still don't understand why there's a separate Source Code section that's not linked to the Deployment process.
At the end of the day I found it too complicated for a WordPress website.
In General I don’t recommend App Engine for CMS projects (WordPress, Drupal), or MVC projects (Yii, Symfony).
For custom applications? It's AMAZING, but in my case, I recommend Compute Engine.
Â
Google Compute Engine
This is basically like Digital Ocean or AWS.
You create a VM instance, you have different sizes (CPU, RAM, Disk Space), and you have a Click to Deploy feature, that lets you automatically create LAMP/LEMP/MEAN stack instances in a few minutes.
This was definitely the way to go, because I have multiple websites, I could just export them all to a LAMP stack server, and enjoy the speed! Right?
Â
Well… There’s something I forgot about, EMAIL!
I needed to make sure all the emails I added for my various websites would work.
I decided to go with the easiest solution, which is to simply forward them using Postfix, and then see about installing a mail server.
Â
Postfix and Compute Engine don’t get along
I started looking into how to configure postfix, seems to be too good to be true, going to /etc/postfix/main.cf , I simply added:
1
2
virtual_alias_domains = kidsil.net ...
virtual_alias_maps = hash:/etc/postfix/virtual
Â
Then creating the /etc/postfix/virtual file, there I entered (replace (at) with @):
1
2
email(at)kidsil.net myemail(at)yahoo.com
...other addresses
Â
Then finally just had to execute these:
1
2
sudo postmap /etc/postfix/virtual
sudo service postfix restart
Â
Awesome, now I should’ve been getting all my emails without a problem! Right?
Â
Well, not exactly.
I couldn’t understand why, but I was just not getting any emails (tried a few test emails, nothing). I’ve looked into /var/log/mail.log and found out that it was unable to connect to any of the email servers it was forwarding to
1
….smtp.mail.yahoo.com[...]:25: Connection timed out
Interesting, I started digging deeper, could it be a specific Compute Engine problem? It was!
Turns out Google Compute engine blocks ALL outgoing traffic to port 25, and almost all outgoing traffic to 465, 587 (all except Google IPs).
This means, no email forwarding :(
Crushed, I figured maybe I could configure a simple mail server, receive my emails via POP3/IMAP and send them via Gmail/Yahoo (if connecting a POP3/IMAP address via either Yahoo or Gmail, they send emails using their servers by default anyways).
I chose Dovecot as the tool for the job, but the experiment was cut short because the configuration instructions looked like this (do you have 3 hours to spare for a mail server?), and the troubleshooting wasn’t a lot shorter either (so it’s a 50% chance of getting the first one it right without a need for help?). So I gave up on that idea.
Â
What other options do I have?
Â
Email Forwarding
Some domain registrars offer an Email Forwarding service, luckily, Namecheap offers it for free! The issue was that the DNS had to be pointing to Namecheap’s servers (i.e. only useful if I’m using their hosting). Long story short - I needed an email forwarding service. Fast.
Â
Mailgun to the rescue
Mailgun is actually mostly used for sending bulk emails to different newsletter lists or customers. It is free for 10,000 emails / month (received or sent I assume), so that’s more than enough for me.
The configuration is simple, just sign up, add a domain, and add the DNS records in your registrar (TXT, CNAME & MX).
I recommend using CloudFlare, their CDN is fast, free, and the control panel is very easy when you want to configure DNS records.
Â
Ok, Emails are done! Websites are flying! Now I can finally go to sleep knowing that the transition is complete!
The morning after
Fridays are usually gym days, during my workout I decided to check out how fast my website loads on my mobile, only to see the worst error known to man.
Error establishing a database connection
How is this possible? Everything was running fine, is it possible that MySQL just crashed?
I checked a few more websites hosted under the same instance, some showed the same message,
but one was working fine.
It can only mean one thing, my VM instance got hacked.
Needless to say it wasn’t easy to make the best of my workout after realizing this.
Â
When I came back home, I fired the Google Cloud console, to get into the instance via the online SSH window, only to get:
1
Could not connect, retrying (1/3).
Now this is weird, not being able to connect via the Developers Console? Someone had to gain substantial access.
I had another SSH key added, and connected through that, checked out /var/log/auth.log
and there it was:
1
Dec 5 09:03:24 … pam_unix(sshd:auth): session opened for user root
I was not on my computer at that time, I was not even home. Game, Set, and Match.
There was a HUGE number of access attempts for various users, various addresses, various methods.
I was also able to get the hacker’s IP address, located in China, *sigh*.
I can only assume the IP pool for VM instances is known as a Google IP pool, and therefore a MAJOR target to attacks.
I think that Google Compute Engine should be a lot more secured out of the box, obvious attacks should be completely blocked (just like port 25 is blocked for outgoing).
Now what?
Honestly, I don’t know.
10 hours after creating the simplest VM instance (click-to-deployed LAMP stack) with Compute Engine, I got hacked. Even if I do take more precautions next time, I still have a huge target sign on my back.
This is what I was scared of when going out of managed hosting, and into a VM environment:
The switch from being a Web Developer to being a System Administrator. I don’t want that.
I'll very likely work on App Engine for future developments, but I'll definitely think twice before working on none-managed VMs.