doc_strange (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
doc_strange) wrote2003-11-23 10:35 am
doc_strange) wrote2003-11-23 10:35 am
Email Whiteout: March 1, 2004?
Many clients of ours have been reporting a significant change in the rate of "spam" volume growth (in this case, unsolicited commercial email, mass-mailing worms, and the like) over the last two months. Some, who keep excellent records of filtered, accepted, and rejected mails, have been sharing their email growth numbers with us.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
Agreed, but slightly missing the point
(1) You write, "The law won't make much difference to this unauthenticated medium." Given that the machines have been hijacked, there isn't any authentication mechanism that can't be hijacked from the machine.
In other words, once a spammer has control of a hijacked machine, many of the client-based remedies will not work. If you charge for email delivery, the spammer will cheerfully use your email account and run up your bill. If you require authentication, the spammer will cheerfully bypass or hijack your authentication mechanism.
(2) My antispam filters are in decent shape. Pobox.com is my first-pass filter; anything that gets through is looked at by spam assassin. This weekend I received at least 250 spams -- 80% of my email is spam -- and about 5 got through. And I was very annoyed by those five, mind you; spamasassin can do better.
I think technology, including whitelists, will be a big help. Spamassassin is so accurate that at this point I have started to let the two-layer system dump all spam automatically. And I frankly don't understand why higher-level systems aren't dumping spam. Then again, since RCN can't even get So.Big worms out of their servers, apparently, I don't expect them to tackle a hard problem like email.
(3) I have said it before and I will say it again: we must attack the spammers' business models. For example: some spammers charge for click throughs. I'd cheerfully have an automated system do a click through for every piece of spam I get -- it wouldn't be hard, just wget and follow the href's. If implemented on a wide scale, auto-clickthrough would destroy the utility of both their payment schemes and their traffic-validation schemes.
Hmmm... what's that, about 30 minutes worth of Python coding? The hard part would be parsing the HTML...
Re: Agreed, but slightly missing the point
Re: Agreed, but slightly missing the point
Why not go after the advertisers who use illegal spammers?
(Anonymous) 2003-11-24 06:00 am (UTC)(link)Re: Why not go after the advertisers who use illegal spammers?
no subject
(no subject)
(no subject)
no subject
Or we could all run SAUCE.
Sure, people complain about false positives; then again, giving up email effectively trests all legitimate mail as false positives.
Also, unlike spamassassin and other content-based systems, SAUCE either rejects your mail or it doesn't - if it does, there's some definite problem with your system to fix. Spamassassin, by contrast, will always have an odd failure mode where a legitimate sender gets unlucky because they really are talking about fake Viagra prices or whatever.
SAUCE also (almost) never accepts mail at SMTP transaction time and then decides to bounce or blackhole it later, eliminating bounced bounces and legitimate senders left with no reason to believe their mail was not delivered.
(no subject)