doc_strange (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
doc_strange) wrote2003-11-23 10:35 am
doc_strange) wrote2003-11-23 10:35 am
Email Whiteout: March 1, 2004?
Many clients of ours have been reporting a significant change in the rate of "spam" volume growth (in this case, unsolicited commercial email, mass-mailing worms, and the like) over the last two months. Some, who keep excellent records of filtered, accepted, and rejected mails, have been sharing their email growth numbers with us.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
From Jan 2003 through the present, the volume of spam email vs. desired commercial and personal mail was on the rise. From 40% of total corporate mail to 70%, on a gently steepening curve. Projecting the curve, the best formulaic match was geometric: The trend predicted a hyperbolic curve. Indeed, many individuals I know are already reporting over 90% spam (and are using filters as a result). But, the large corporate sites (and AOL's often-cited figures) do a better job as a significant sample set.
The curve projection also showed the "total mail volume" (all mail including spam) would be exceeded by the "spam" curve around April, 2004. Obviously, Y cannot be > X+Y where X and Y are positive integers, so that graphical projection is kind of nonsensical. However, it indicates two trends: the growth in spam and a relative falling off of normal email use. It also indicates something I'll get to, below.
I asked around, and indeed, most people I talked with said the spam volume or effects therefrom discouraged them from using email as much as they used to -- from "exposing" their email address or dealing with erroneous filtering or delivery delays caused by heavy-duty mail scrubbers (that are trying to keep out the spam). Having to worry about picking through filtered mail (for false-positives) becomes a chore when for every 10 good mails, you receive 100 that go into the filter trap. A daily, or twice-daily chore, to pick through for any 'good' ones.
But that's not all. Indeed, just two weeks ago, from all accounts, it looks like we hit the "knee" in the hyperbolic curve -- slightly ahead of the prediction. A revised curve prediction puts us around March 1 as the point where the curves come together.
March 1 is effectively a ballpark prediction of an email whiteout. That is, the volume of spam, if it continues along its present trend, will so exceed the volume of other, normal email correspondence that in effect all email will be "spam." Systems will spend almost all their processing, bandwidth, storage, time, and other resources processing these unsolicited tidbits and worm junk, that direct, personal correspondence will disappear under the flood.
It's already starting to happen.
An alternative interpretation is that, rather than a whiteout, the growth will lead to a dollar-out -- a point at which the cost of processing one direct, personal business correspondence via Internet-based email will exceed its business value. Internal email, inter-partner ("whitelisted") email, and other internal communications (corporate instant messaging (IM), voicemail, and ever-less-expensive cellphone costs) will probably continue, and indeed, increase in volume. Indeed, expect to see large corporations set up inter-corporate messaging solutions including private IM solutions. Many business partners already have private line or VPN inter-connects for the conduct of business transactions. That interpretation is also already being contemplated by some commentators.
Similarly, personal email will take so much time, energy, and passed-on filtering overhead delays or costs from ISPs, that email will start to fall from favor as a personal correspondence medium (and because Internet service providers that filter take on much of the burden, there will be a delay in effect, here). Good olde phonecalls, voicemails, SMS, blogs, web bulletin boards, and so on are likely to push into the gap (and SMS and blogging are already significantly on the rise, replacing personal mailing lists in many groups with whom I have talked).
So where does this leave us? A tragedy of the commons, in effect, will play out. "Unless someone passes a solid, anti-spam law," I hear you say. Ah, no. Over half of the UCE we see right now comes from compromised personal computers -- i.e., a crime has already been committed -- or from overseas, or it's advertising already-illegal items... or all three. The law won't make much difference to this unauthenticated medium.
No, the only way to win may be not to play, at least until the mechanism catches up with the threat model.
no subject
Or we could all run SAUCE.
Sure, people complain about false positives; then again, giving up email effectively trests all legitimate mail as false positives.
Also, unlike spamassassin and other content-based systems, SAUCE either rejects your mail or it doesn't - if it does, there's some definite problem with your system to fix. Spamassassin, by contrast, will always have an odd failure mode where a legitimate sender gets unlucky because they really are talking about fake Viagra prices or whatever.
SAUCE also (almost) never accepts mail at SMTP transaction time and then decides to bounce or blackhole it later, eliminating bounced bounces and legitimate senders left with no reason to believe their mail was not delivered.
no subject
I do think peer-review and bait/trap email addresses will become more and more useful. However, they're an example of how we're becoming buried under load to the point where we have to pool resources just to fight UCE. SAUCE is an example of going to a solution only steps away from whitelisting (and more resource intensive), showing a willingness to drop legit content because of bad packaging. That's a sign that SMTP mail as we know it is becoming less useful. And of course nothing even close to "everyone" is going to be using it. Sadly, I've had no fewer than 5 people talk to me about abandoning well-established email addresses just this week, because of UCE overload. It's not Armageddon, but it's certainly getting "expensive" in terms of time and materials to filter even at the 90% level. And that 10% is over 100 a day for some well-known users.