Abuse ticket

We take all reports of abuse seriously and make sure the issues are resolved. We also may adjust the required response time based on the severity of the type of activity. Phishing situations are pretty much the only situations where we provide such short notice as they are quite serious.

As a note to follow up, we do check these reports to ensure they are legitimate and are being received from your Linode. Only then do we open such tickets on your Linode account.

-Tim

@theckman:

As a note to follow up, we do check these reports to ensure they are legitimate and are being received from your Linode. Only then do we open such tickets on your Linode account.

"Received from your Linode" is somewhat ambiguous. This particular phishing email originated from 94.247.24.173 according to the email headers you sent me, and was then forwarded by our Linode. I'm sure that the person reporting it missed that detail, and maybe your support staff did too.

I'm not staff, but relaying bad mail and sending it out is just as bad.

I'm not trying to argue here, but it sounds like the email itself came from your Linode, regardless of whether that's where the message originated or not. The system you are responsible for was still used for malicious activity plain and simple. We reached out to you to let you know about said malicious activity, and because of the severity we gave you as much time as we felt comfortable given the activity that your Linode was taking part in.

In addition to that, such activity has the potential to negatively impact other customers (especially if some BOFH set up the filters on their mail servers to block entire /24s). It's in both of our best interests (yours and Linode's) to make sure this activity ceases to originate from our network in the quickest amount of time possible.

-Tim

deleted

Lots of people run their mailers on a Linode, if Linode's IPs get on a blocklist those people would be very unhappy. Plus lots of us hate spam with a passion we can't express in words.

The only thing Linode could have done instead was block outbound port 25 from your IP on their switches. That depends on them running ACL's on their switches which I'm not sure they do. Failing that taking the machine down in 4 hours is totally reasonable.

Linode did good. Well done Linode.

To put things a bit into perspective: We are running a cPanel server that handles mail among other things, and one of the services provided by cPanel by default is the ability to configure email forwarders. We try to avoid using the server for email, pointing our customers to other email services where possible, and the total volume of email in a 24-hour period is maybe 400 emails arriving at the server. We use various RBLs and Spamassassin to filter email, but obviously not everything is caught.

Not everyone uses email forwarding, but a few of our customers do. These email addresses are typically of the "contact@domain.com" kind, publicly accessible and so obvious targets for spammers. They are also the type of email addresses you will likely want to forward to your main address, or sometimes maybe to 2-3 different website admins.

From looking at the email logs, and based on the amount of phishing emails I receive on my Google Apps account (not Linode related, I hasten to add), I guess that a couple of this type of email must get forwarded every day. In other words, our system is "used for malicious activity" on a daily basis.

We are in the process of informing our customers that we can no longer allow email forwarding, since we clearly cannot run the risk of having our Linode shut down on this "one strike and you're out" basis. I'm wondering what others with similar setups do, though - or are we the only ones with this type of setup on our Linodes?

I can see the need for action within a limited timeframe in a case where a website is hacked, for example, and is used for sending out large volumes of spam, phishing or the like. In this case, though, we forwarded a total of three offensive emails received from another mail server. Just our bad luck that the admin of the mail server hosting our customers other email address got pissed off and forwarded one of them to spamcop.net. (But fortunately spamcop looks at more than just the single event, so we were in no risk of ending up on their RBL).

You forwarded mail to a mail server run by an admin who did not want to receive this particular mail from you.

It does not matter whether you were merely passing along the phishing message that originated elsewhere or whether the admin of this server is being overzealous, unreasonable, etc.

You can probably still do email forwarding, but only forward mail to addresses or servers that are willing to receive mail forwarded by you and are not going to report you to the spam police or your provider.

For example, Gmail understands the concept of mail forwarding and even has a page that explains how to properly set up your forwarding server for best results, so you would probably be OK to forward to gmail.com addresses.

Small, personally hosted mail servers are notorious for having admins that sit around all day watching their mail logs and submitting anything suspicious to every spam reporting service and ISP involved in the delivery of the message, so avoid them or at least have some sort of agreement with them that they will not try to get you in trouble when you forward phishing messages.

@dcraig Thanks for the info about Gmails policies. We do not setup the forwarders ourselves, our customers have that option, but we may be able to leave forwarders to Gmail in place a bit longer than the rest.

We run a web design business, and we use Linodes to host websites for some of our small customers. We have so far been quite happy with Linode as a provider, but our takeaway from this event is that Linode is not really a suitable platform for our purposes. We do our best, but we cannot guarantee that no offensive email will ever leave our IP address, nor will we always be able to respond to complaints within four hours. We are now aware that this exposes us to a significant risk of extended downtime, or even complete loss of our Linodes.

I have to agree – four hours is ridiculous before a shutdown, especially if the horse has already left the barn. If multiple messages are going out, sure. But a one-time phish has already done all of the damage that it's going to do.

I'm in the same position; I manage discussion lists for various topics and can't guarantee that a subscriber won't get a virus that spams their address book with a phishing message. We've had it happen twice in twenty years. I'll have to look for someplace else as well.

If you can't prevent your mail server from sending or relaying spam out, you might want to try a shady Russian host. Unlike Linode, they tend not to care when people abuse their services.

@gparent On the off chance that you are not simply trolling, I'd be very interested in understanding your email server setup.

If you know of a way to configure a real-world, useful mail server that can guarantee that not a single spam or phishing message is ever transmitted, I want to know how you do it :-)

> If you can't prevent your mail server from sending or relaying spam out, you might want to try a shady Russian host. Unlike Linode, they tend not to care when people abuse their services.

I wonder if I could just use emma. Most of us thought it was a phishing attack when Linode sent out the security alert in April through that service. ;)

I don't think that anyone's arguing that Linode doesn't care. We're arguing that a knee-jerk reaction like a server shutdown with 4-hour notification is overkill and draconian.

There's a fine line between customer satisfaction and trying to maintain the reputation of your network, and my 20+ years of *NIX experience tells me that this crosses it. There are better technical ways to handle this, and I'll look for a host that understands that.

@trisager:

@gparent On the off chance that you are not simply trolling, I'd be very interested in understanding your email server setup.

If you know of a way to configure a real-world, useful mail server that can guarantee that not a single spam or phishing message is ever transmitted, I want to know how you do it :-)

I'm not trolling. A lot of russian hosts really do not care about spam.

As for my experience, I can only speak about relaying it and how other providers seem to handle it because I do not let my users forward to emails they do not own. Have you used Gmail? To setup a relay there, you need to confirm that you are the owner of the receiving email address. Perhaps I'm not understanding your setup or your issue. Is it impossible for you to know where you're going to end up forwarding mail? Because that's the configuration weakness that spammers use to work so efficiently.

I don't think it's a knee-jerk reaction to avoid an entire net block to be banned from sending mail. In other situations, spamcop could've taken action and prevented dozens of servers from working correctly just because one person is sending/relaying bad mail.

I agree 4 hours is too short for a single abuse complaint like this, but it is true that email forwarding is seriously broken, not just for the reasons already discussed, but also because it creates backscatter: if you try to forward a spam message with a forged from address, and the next hop mail server rejects the message because it's spammy, your server sends a bounce message to the innocent person who had his address forged. Backscatter is extremely annoying and there are blacklists for servers which generate too much of it.

I commend you for phasing out forwarding. An alternative to forwarding which I've deployed is to hold the mail locally and tell users they can retrieve it using POP. A lot of webmail providers have the ability to pull email from a POP server. Once it's set up, it works just like forwarding from a user's perspective, though with a slight delay (typically 15-60 minutes for gmail).

It doesn't have to just be email forwarding. It can be anything that sends an email. If you have a bad PHP script, an old version of Joomla or WordPress, a comment area in your blog, a forum, etc. then you can be responsible for sending spam. I've even had people subscribe to discussion lists, then years later they forgot how to unsubscribe, didn't want to figure it out, and just complained that everything was spam.

What if someone registers on your wiki page, enters a bunch of spam, and then a watcher for that page gets an email with the spam in it?

What if one of your users gets their password hacked and sends a spam?

These aren't things that you can guarantee won't happen. They will. I manage a relay for 30,000+ accounts (elsewhere) with millions of messages passing through it daily. You can't stop spam, inbound or outbound. You can only try to be proactive with different filtering technologies and responsive when something fails. It will fail, and often in unique ways that you never envisioned.

In this case we're talking about something more serious – phishing. But even in the case of phishing we would never shut someone down unless they were actively sending out messages. A one-time message that was received and subsequently forwarded to another account would be investigated, but the host wouldn't be shut down within four hours. What's the point? Unless that system is hosting the site that the phishing attack sends people to, you're closing the door after the horse has already left the barn.

The best solution if you really need to act is to block ports 25/465/587 to/from that host. Remote systems will continue queuing email without killing other services that might be running on the same system. Outbound mail will also queue.

@trisager:

We are in the process of informing our customers that we can no longer allow email forwarding, since we clearly cannot run the risk of having our Linode shut down on this "one strike and you're out" basis. I'm wondering what others with similar setups do, though - or are we the only ones with this type of setup on our Linodes?

@trisager:

We run a web design business, and we use Linodes to host websites for some of our small customers. We have so far been quite happy with Linode as a provider, but our takeaway from this event is that Linode is not really a suitable platform for our purposes. We do our best, but we cannot guarantee that no offensive email will ever leave our IP address, nor will we always be able to respond to complaints within four hours. We are now aware that this exposes us to a significant risk of extended downtime, or even complete loss of our Linodes.

There isn't a problem with allowing email forwarding on your system. A point should be made that there should be strict controls in place for what can be forwarded and from where. We don't personally see a problem with forwarding email, as long as you whitelist the sending server(s) for example. Other measures can be put in place to ensure a random bot on the Internet can't use your Linode for malicious purposes.

I think there me some misunderstanding about how we operate when it comes to abuse complaints. If you send a malicious email from your Linode for example, we're not just going to obliterate your Linode. In addition to that, we're not worried about offensive email. We're worried about things that violate our terms of service and are malicious in nature (for the purposes of this conversation, let's consider unsolicited spam emails as malicious as well).

@mallorn:

I have to agree – four hours is ridiculous before a shutdown, especially if the horse has already left the barn. If multiple messages are going out, sure. But a one-time phish has already done all of the damage that it's going to do.

I'm in the same position; I manage discussion lists for various topics and can't guarantee that a subscriber won't get a virus that spams their address book with a phishing message. We've had it happen twice in twenty years. I'll have to look for someplace else as well.

Without some deep traffic inspection, which is something we do not do, it's hard to determine what content exists in the emails you are sending out of your system. I completely agree that single email has already done its damage. What seems to be missing here, is the understanding that the vector used is still available and can be used for further emails. Beyond that, we aren't able to reliably determine how your system used and what state it is currently in. So we don't know what else could happen with your Linode in its current state.

In short, phishing is an absolutely serious situation and as mentioned before, it's in everyone's best interest to get them ironed out ASAP. We're not going to delete your Linode, or ask you to leave, for a single complaint. In addition to that, making sure you have your system configured properly to allow emails from trusted sources you should be able to cut down on this.

Any reputable hosting provider, I'm sure, will have very similar policies when handling high-risk abuse situations. And lastly, shame on someone who would report such emails having obviously originated from a mailing list. I'd like to think most people on mailing lists are understanding enough to know it originated from a compromised system/account and that the list maintainers would take care of it.

-Tim

I understand your issue more now, but surely if you're sending millions of messages per day you can afford to have at least one person reply to something within 4 hours?

FWIW, "mail forwarding" causes a lot of problems. I know some smaller ISPs who have been blacklisted by AOL (for example) because that ISP forwarded (customer configured) mail to the AOL account; the AOL account owner saw spam and reported it. AOL saw the ISP server in the received headers, and so blocked the ISP server.

It was a semi-regular occurance at one ISP where a friend works.

This is a risk you chose to take by allowing users to forward their email elsewhere; your mail server is the only one that can be positively identified as handling the message.

@gparent:

I understand your issue more now, but surely if you're sending millions of messages per day you can afford to have at least one person reply to something within 4 hours?

Oh, absolutely. However, as previously mentioned, the amount of email leaving our server on a daily basis is on the order of hundreds, not millions.

Thanks to other posters for the inputs on the dangers of forwarding. We'll limit our exposure for now by eliminating that possibility, and by helping those of our users who need it to find alternative solutions.

The fact that remains, though, is that we were lucky to avoid our Linode being shut down - I am frequently away from my desk for a few hours. Uptime matters to us, and we now need to factor in the risk that Linode staff will shut us down based on single events like this one.

I'm sure that all reputable service providers take spam and phishing seriously, but knee-jerk reactions like this one is something I'm fairly confident are only common at the low end of the market.

> I'm sure that all reputable service providers take spam and phishing seriously, but knee-jerk reactions like this one is something I'm fairly confident are only common at the low end of the market.

You do realize that not only can the phishing hurt recipients of the messages, but your neighbors in the subnet as well? If the subnet is tainted by being a source of malicious traffic, how you you expect Linode to help me, or anyone else affected by your behavior after the fact? They can't. So, instead of your node going down for whatever time it takes you to respond after the four hours, I may have to deal with a situation where I cannot send email. This is not acceptable to me, a completely innocent party in this transaction.

I really can't see anything about the fact that the email originated somewhere else as mitigating the fact that your node delivered it. As you've said, you have taken steps to avoid this problem, which is appropriate. But claiming that preserving the reputation of all the other customers in your subnet is a "knee-jerk" reaction is probably not so well thought out. Your complaining here is irritating to Linode customers that are actually happy Linode has such strong policies in this area. You should probably stop.

@Yaakov:

You do realize that not only can the phishing hurt recipients of the messages, but your neighbors in the subnet as well? If the subnet is tainted by being a source of malicious traffic, how you you expect Linode to help me, or anyone else affected by your behavior after the fact? They can't. So, instead of your node going down for whatever time it takes you to respond after the four hours, I may have to deal with a situation where I cannot send email. This is not acceptable to me, a completely innocent party in this transaction.

Blocking SMTP access to/from the affected IP would also work, without shutting down the Linode and affecting other services that might be running on it.

@trisager:

Uptime matters to us, and we now need to factor in the risk that Linode staff will shut us down based on single events like this one.
Then you need to design that into your system architecture - there are many (MANY) other things that are also out of your control that could bring you offline - if that's unacceptable YOU need to figure out how to minimize that risk.

Blaming Linode for protecting everyone else is NOT the cause of your problem - deal with it.

> Blocking SMTP access to/from the affected IP would also work, without shutting down the Linode and affecting other services that might be running on it.

Linode sells unmanaged VPSs. Expecting them to firewall a particular 'node because it is misbehaving is really not reasonable. "Blocking SMTP access" requires filtering of a kind they don't otherwise do, and I, for one, don't want to pay them to be able to do it because some people don't know how to manage their 'node.

@Yaakov:

Your complaining here is irritating to Linode customers that are actually happy Linode has such strong policies in this area. You should probably stop.

Apologies for irritating you. I think the Linode response was excessive, but that point has been made by now.

Spamcop never listed our IP. Even if they had, that would have affected only our IP, and the listing would have automatically cleared within 24 hours, assuming the problem was dealt with. It would take a lot more than a single phishing email to endanger anyone elses Linode.

@trisager:

@Yaakov:

Your complaining here is irritating to Linode customers that are actually happy Linode has such strong policies in this area. You should probably stop.

Apologies for irritating you. I think the Linode response was excessive, but that point has been made by now.

Spamcop never listed our IP. Even if they had, that would have affected only our IP, and the listing would have automatically cleared within 24 hours, assuming the problem was dealt with. It would take a lot more than a single phishing email to endanger anyone elses Linode.

And you're the one to tell us what damage you think your Linode would do? For the most part, most of us are speaking first hand when an entire block gets nuked from sending email because one person messed up.

@trisager:

Spamcop never listed our IP. Even if they had, that would have affected only our IP, and the listing would have automatically cleared within 24 hours, assuming the problem was dealt with. It would take a lot more than a single phishing email to endanger anyone elses Linode.

I hope it doesn't come as a surprise that Spamcop isn't the only organization responsible for preventing spam on this planet. Netblocks can and will be blocked.

@KyleXY:

And you're the one to tell us what damage you think your Linode would do? For the most part, most of us are speaking first hand when an entire block gets nuked from sending email because one person messed up.

Look, I'm genuinely interested, not trying to prolong an argument here. Do you really have first hand experience of a situation where one or two spam/phishing emails caused an entire subnet to be blacklisted by a reputable RBL?

> Spamcop never listed our IP. Even if they had, that would have affected only our IP, and the listing would have automatically cleared within 24 hours, assuming the problem was dealt with. It would take a lot more than a single phishing email to endanger anyone elses Linode.

Unfortunately, this is both rhetorical and incorrect. If a phishing complaint comes in it is not at all clear how many messages will follow it. Linode doesn't watch your mail. The have no idea if is the first report for thousands. So, even if it was one message in this case, that can't be known to Linode.

Second, if you read the history of DNSBLs you will discover that even errors can lead to listing, and some lists use subnets. Why should we take the risk for your convenience?

The truth is, your complaining is annoying. The proper place to have this discussion is not in this forum but in email to Linode. Why should you make public denouncments about Linode's policies except to sully their reputation? I can't see any value other than being a nuisance to Linode because you don't like what they did. That is annoying. Full stop.

@gparent:

I hope it doesn't come as a surprise that Spamcop isn't the only organization responsible for preventing spam on this planet. Netblocks can and will be blocked.

I'm sure they can and will. We do it ourselves when we detect multiple intrusion attempts from the same subnet, for example. However, in this particular case, we are talking about a single event involving one phishing message. I don't see any realistic scenario where that could lead to a block of IP addresses being blacklisted. Do you?

@Yaakov:

Unfortunately, this is both rhetorical and incorrect. If a phishing complaint comes in it is not at all clear how many messages will follow it. Linode doesn't watch your mail. The have no idea if is the first report for thousands. So, even if it was one message in this case, that can't be known to Linode.
Granted.

@Yaakov:

The truth is, your complaining is annoying. The proper place to have this discussion is not in this forum but in email to Linode. Why should you make public denouncments about Linode's policies except to sully their reputation?
I have no interest in sullying Linode's reputation, but I do think it is fair to use the forum they provide to discuss what I see as an issue with their service. From your posts it is obvious that you agree with the treatment the abuse report received, and that's fine - we don't have to agree.

We run a low-volume mail service that we try to keep free from spam etc by using RBLs and spam filtering. Clearly this involves a danger to our servers that we were not aware of, and we will deal with that for now by removing the email forwarding service. We will also consider moving elsewhere if we think that reduces our exposure. If we find that everyone else deals with this type of issue the same way as Linode, we will probably stay where we are. Fair?

@Yaakov:

Unfortunately, this is both rhetorical and incorrect. If a phishing complaint comes in it is not at all clear how many messages will follow it. Linode doesn't watch your mail. The have no idea if is the first report for thousands. So, even if it was one message in this case, that can't be known to Linode.

I was really disappointed when I read this from the Linode rep. bro and netflows are very useful tools.

EDIT: It wouldn't work when scanning encrypted SMTP traffic, though.

@Yaakov:

The truth is, your complaining is annoying. The proper place to have this discussion is not in this forum but in email to Linode. Why should you make public denouncments about Linode's policies except to sully their reputation? I can't see any value other than being a nuisance to Linode because you don't like what they did. That is annoying. Full stop.

Personally, I'm glad that this was brought up. It's a big issue, and I want to make informed decisions for my business since this Linode policy isn't explicitly stated anywhere. It's nebulously covered in the ToS, so having hard numbers like four hours helps me understand what limitations I might run into here.

A good compromise is a port 25/465/587 block to/from the host. It won't send out any more spam/phishing messages, and the host won't lose its other services.

As an aside, I just received an email from a Spamcop admin that the user making the abuse report has had his/her reporting privileges suspended over this. Apparently they frown on users setting up email forwarders, forgetting about them, and then reporting the forwarding server for spamming :-)

@mallorn:

I was really disappointed when I read this from the Linode rep. bro and netflows are very useful tools.

I'm not sure if you're referencing Yaakov, but I am currently the only Linode employee active in this thread. Yaakov is a fellow customer, not a Linode employee.

-Tim

Edit: In addition to that, it still requires us to do some sort of deep inspection of your network traffic which is something we're not in the business of doing. Nor do we have the infrastructure in place to do this on customer systems.

> As an aside, I just received an email from a Spamcop admin that the user making the abuse report has had his/her reporting privileges suspended over this.

It is nice to hear of a DNSBL that has proactive and sensible policies. So much more often you hear of the messes they cause.

@theckman:

I'm not sure if you're referencing Yaakov, but I am currently the only Linode employee active in this thread. Yaakov is a fellow customer, not a Linode employee.

Hi Tim,

I was referencing your post when you said 'Without some deep traffic inspection, which is something we do not do, it's hard to determine what content exists in the emails you are sending out of your system'. Sorry about the confusion. I was too lazy to page back and get your name to attribute it properly.

I'm going to stop beating a dead horse here and want to say that I've generally been very happy with Linode. I'll continue recommending them and will use them myself when possible. I need to re-think the email side of things, though.

As a final follow up on this thread, let me briefly recap what actually led to the abuse ticket being opened:

• A user has an email address on our server, contact@somedomain.com. Our user forwards all email to this address to another email address that he owns, user@otherdomain.com

• We check arriving email against three RBLs and using spamassassin.

• A phishing email arrives for contact@somedomain.com from 94.247.24.173 which is not blocked or caught by our spam filtering. It is forwarded to user@otherdomain.com

• Our user sees the phishing email in his user@otherdomain.com inbox and forwards it to spamcop.net

• spamcop.net parses the email, and automatically sends an abuse report to Linode with our IP address in it

• Linode opens abuse ticket, stating likely compromise of our Linode. We are given four hours to repond or the Linode will be powered down.

While I understand that spamming and phishing from a Linode can lead to all sorts of bad consequences for other Linode customers, at no point did the above events endanger anyone else. Nor do I believe that this demonstrates that we are particularly inept system administrators, although we are now removing the email forwarding feature from our systems.

Given the facts of what actually happened, I continue to think that Linode's response was disproportionate. If an abuse ticket is opened at all in this type of case, a longer response time would have been appropriate.

I understand that it may not be cost effective for Linode to distinguish between different types of abuse reports, or to implement less dramatic measure of dealing with them, but that does mean that sending email from a Linode carries risks that we were not previously aware of.

Presumably this could have happened to many other Linode customers. I think it is a fair use of this forum to post about it here.

Linode support have subsequently told us that they will actually not simply power off the Linode after four hours, but try to reach us via phone first. This obviously improves matters quite a bit - I much prefer being waken up by a call from Linode support instead of an alert from our monitoring systems. Others may want to verify that they have current contact numbers in the Linode Manager :)

@trisager:

Given the facts of what actually happened, I continue to think that Linode's response was disproportionate.
Why do you keep complaining about this - Linode didn't actually do anything. They received a valid abuse complaint and warned you to deal with it or else they would handle it themselves. The fact that it was your own user who generated the complaint doesn't mean noone was hurt by it.

I have found this thread very informative. I have always been wary of email systems on the servers I administer. After reading all of this, I'm going to go back and review current techniques and see if I can reinforce my security.

Thanks, Jeff

So I think the moral of the story here is that until Linode does deep packet inspection on every customer (which I hope they never do), you're responsible for being proactive with the security of your mail server. > but that does mean that sending email from a Linode carries risks that we were not previously aware of. you weren't aware of. Most of us understand that sending abusive email can get our service temporarily suspended.