Email Whiteout: March 1, 2004?
Nov. 23rd, 2003 10:35 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
doc_strangeMany clients of ours have been reporting a significant change in the rate of "spam" volume growth (in this case, unsolicited commercial email, mass-mailing worms, and the like) over the last two months. Some, who keep excellent records of filtered, accepted, and rejected mails, have been sharing their email growth numbers with us.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
Agreed, but slightly missing the point
Date: 2003-11-23 09:11 pm (UTC)(1) You write, "The law won't make much difference to this unauthenticated medium." Given that the machines have been hijacked, there isn't any authentication mechanism that can't be hijacked from the machine.
In other words, once a spammer has control of a hijacked machine, many of the client-based remedies will not work. If you charge for email delivery, the spammer will cheerfully use your email account and run up your bill. If you require authentication, the spammer will cheerfully bypass or hijack your authentication mechanism.
(2) My antispam filters are in decent shape. Pobox.com is my first-pass filter; anything that gets through is looked at by spam assassin. This weekend I received at least 250 spams -- 80% of my email is spam -- and about 5 got through. And I was very annoyed by those five, mind you; spamasassin can do better.
I think technology, including whitelists, will be a big help. Spamassassin is so accurate that at this point I have started to let the two-layer system dump all spam automatically. And I frankly don't understand why higher-level systems aren't dumping spam. Then again, since RCN can't even get So.Big worms out of their servers, apparently, I don't expect them to tackle a hard problem like email.
(3) I have said it before and I will say it again: we must attack the spammers' business models. For example: some spammers charge for click throughs. I'd cheerfully have an automated system do a click through for every piece of spam I get -- it wouldn't be hard, just wget and follow the href's. If implemented on a wide scale, auto-clickthrough would destroy the utility of both their payment schemes and their traffic-validation schemes.
Hmmm... what's that, about 30 minutes worth of Python coding? The hard part would be parsing the HTML...
Re: Agreed, but slightly missing the point
Date: 2003-11-24 07:37 am (UTC)I agree entirely. You're right, there. The change to the mechanism will have to be deeper than host authentication (deeper than IPv6, which many tout as a solution). Any such real solution, however, will kill the 'commons' that made SMTP email so nice (and so nasty): you can send a letter to anyone from anywhere, through any path, and it will work. That's charred ground, now, and only through the use of strongly authenticating gateways will we preserve something like real SMTP mail. Then we just have to lock out all unauthenticating senders... distribute keys... and... er... that's pretty expensive unless we make it an end-user solution of sorts (so the cost of fighting the spam is itself distributed among the beneficiaries). But that is doable.
I like your idea of crashing the click-through model, but that will take a lot more than a few participants (it will take thousands), and it has the negative effect that it won't fully respond to changes in the OTHER side of the equasion. For example, it used to be that anti-spammers thought it a good idea to publish thousands of bogus email addresses. Ironically, now, the spammers have massive capacity, and the double-bounces generated by the bogus-sender spams to those bogus recipients is more of a burden on servers than straight-up spam.
Re: Agreed, but slightly missing the point
Date: 2003-11-24 10:38 am (UTC)The random-email-address attacks were perhaps not properly thought through. I gave up generating bounce messages to spammers when my ISP sysadmin told me that these bounce in turn, which create more work for him.
The other hobby horse I ride on the antispam crusade is moving the fight to the analog domain. Flood the spammer with data that has to be sorted by analog means: phone calls, letters. The spammer then has to differentiate, manually, between real user data and fake data. This quickly reduces his profit margin towards zero. I saw a tool on the net that does this: the tool fills out spammer's forms with random but realistic data.
Remember the folks who played that Nigeria scammer for laughs? Imagine if there were a modified Eliza variant to that engaged each of these spammers... they'd have to spend days trying to sort through the real answers from he fake ones. If they received 100,000 realistic responses they'd never be able to find the real suckers.
The way to fight the RIAA is not to send them email; that's processed, sorted, and dumped in the digital realm. It's to call them on the telphone, politely, to complain about their policies. Ten thousand calls a day to them -- and their lawyers -- may not change their minds but will certainly get their attention.
Why not go after the advertisers who use illegal spammers?
Date: 2003-11-24 06:00 am (UTC)Re: Why not go after the advertisers who use illegal spammers?
Date: 2003-11-24 07:42 am (UTC)While, I'd love such legislation (carefully written, because there will be First Amendment issues in the U.S.,), I'm not sure it would have all that much positive effect over the long haul. The spammers would just all be "relocated" to places that don't have such laws, then. The cost to investigate such overseas operations' links to the jurisdictions that have these laws would outweigh the real value of the investigations, and law enforcement would be slow to aid (instead of investigating say, money-laundering or even more serious activities).
(no subject)
Date: 2003-12-01 05:57 am (UTC)(no subject)
Date: 2003-12-01 01:43 pm (UTC)(no subject)
Date: 2003-12-08 06:04 am (UTC)(no subject)
Date: 2004-03-04 05:47 pm (UTC)Or we could all run SAUCE.
Sure, people complain about false positives; then again, giving up email effectively trests all legitimate mail as false positives.
Also, unlike spamassassin and other content-based systems, SAUCE either rejects your mail or it doesn't - if it does, there's some definite problem with your system to fix. Spamassassin, by contrast, will always have an odd failure mode where a legitimate sender gets unlucky because they really are talking about fake Viagra prices or whatever.
SAUCE also (almost) never accepts mail at SMTP transaction time and then decides to bounce or blackhole it later, eliminating bounced bounces and legitimate senders left with no reason to believe their mail was not delivered.
(no subject)
Date: 2004-03-07 07:01 pm (UTC)I do think peer-review and bait/trap email addresses will become more and more useful. However, they're an example of how we're becoming buried under load to the point where we have to pool resources just to fight UCE. SAUCE is an example of going to a solution only steps away from whitelisting (and more resource intensive), showing a willingness to drop legit content because of bad packaging. That's a sign that SMTP mail as we know it is becoming less useful. And of course nothing even close to "everyone" is going to be using it. Sadly, I've had no fewer than 5 people talk to me about abandoning well-established email addresses just this week, because of UCE overload. It's not Armageddon, but it's certainly getting "expensive" in terms of time and materials to filter even at the 90% level. And that 10% is over 100 a day for some well-known users.