Wal-Mart's Data Obsession 581
g8oz writes "The New York Times covers Wal-Mart's obsession with collecting sales data.
Fun fact: 'Wal-Mart has 460 terabytes of data stored on Teradata mainframes, at
its Bentonville headquarters.
To put that in perspective, the Internet has less than half as much data, according to experts.'
That much information results in some interesting data-mining. Did you know hurricanes increase strawberry Pop Tarts sales 7-fold?"
I would have thought that the Internet had more. (Score:4, Interesting)
230 terabytes? Please (Score:5, Interesting)
Huh? (Score:2, Interesting)
Sunny Dubey
So, if Walmart put up a web interface... (Score:4, Interesting)
I think the expert they got their information from was full of baloney.
Heh, lets see if this "predicting" works (Score:5, Interesting)
Then one day, the managers were really excited, as we were going to have a computer order everything for us, from records of sales from before and it would "predict" what we would need. They said the extra stock on top of the aisles would be eliminated. We would be able to concentrate on customer service.
Well, the day came, and for a few months you could tell the computer was fighting with limited data. Some weeks would be rediculously overstocked on a few items, others, the leading sellers in the store would have empty shelves. When it finally settled down after a year, it was worse than before the computer.
The top of aisles were jammed to the ceiling with stock, there was never any room to put anything up there, and getting to the bottom for something you needed cost a lot of time. Plus, the backroom was packed with stock. You could hardly move around, and trying to find the last box of something buried underneath these huge piles was a task that killed your morale. During the slow months, one stocker for the whole store was enough for a night, now 3 were common to deal with all the stock.
Re:I would have thought that the Internet had more (Score:3, Interesting)
Google has 8E9 web pages and documents indexed. If the average document is 20 kB in length, then we have 160 TB of publicly available data on the internet, not including pictures and filesharing. The latter probably has a great deal of duplicate data anyway.
Re:I would have thought that the Internet had more (Score:4, Interesting)
That doesn't mean they know what to do with it... (Score:3, Interesting)
This makes me wonder... there must be some ideal point where a certain amount of data collected is worth the most money because you can act on that data. After that point, collecting additional data is increasingly more costly and counterproductive unless you invest in an infrastructure that lets you process more data. How does one figure out that ideal point? Just a thought.
Did you know... (Score:5, Interesting)
Activity of the cards is ACTUALLY monitored for discrepencies in buying habits to find abusive employees who buy things for their friends?
Did you also know Wal-Mart's employee name badges have RFID tags (and have had for many years) that allow Wal-Mart to track where an employee is at any given time?
Another interesting tidbit, did you know at Wal-Mart's Jewelery warehouses they actually WEIGH the amount of metal in your body when you enter a leave? (And I don't mean they ask you to put things in a dish and weigh the dish - they scan YOU)
Another interesting thing, Wal-Mart has a fallout facility in Oklahoma that has a near-real-time backup of each BIT of that 460 terabytes of data?
Wal-Mart could survive a direct nuclear blast and still keep on a truckin'.
And, of course, if you're in a Wal-Mart home office - ISD building - distribution center - et al... and dial 911 - BOOM - you get Wal-Mart's private security? Niiice, hope it's not a real emergency, you first have to explain it to them - then if they deem it neccessary THEY will call the REAL 911!
Re:2004 = 1984 + 20; (Score:3, Interesting)
Listen, if you really are that paranoid, pay in cash. Then there is no way for the evil Wal-mart overlords to find you and force you to buy more pop tarts.
460 TB is nothing we have 25X that (Score:4, Interesting)
Hurricanes and Pop-Tarts? Bah... (Score:4, Interesting)
Stuff like: women who buy from catalogs, eat "crunchy" peanut butter, own a cat and drive a minivan you are 87% more likely to react positively to prayer in schools as a "motivating issue."
I just made that up, but it's the sort of thing they find out. No tin-foil hats here - corporations and pollsters are shelling out millions of dollars for this stuff.
Here's a few google searches links to get you started:
Acxiom [google.com]
Seisint [google.com]
Re:Did you know... (Score:5, Interesting)
1) poor people shop there because it's cheaper than the other stores because wal*mart gets their stuff all from china and stong arms their suppliers to give them cheaper and cheaper products.
2) to keep up with walmarts demands, the companies have to outsource more and more to china and other cheap labor countries (or just move there entirely)
3) so more people lose their jobs, become poor and have to shop at wal*mart beacuse 1) it's cheaper than everything else around, and 2) all the other local businesses are now out of business because they can't compete with the special deals wal*mart gets for buying in such huge quantities...
(goto 1)
Re:Heh, lets see if this "predicting" works (Score:5, Interesting)
The Walmart shipping system is was very efficent, but it was designed to serve walmart, not the individual stores. We had an extremely finite space in which to store things, and an extremely finite shoe department, yet the thing shipped us INCREDIBLE ammounts of shoes. And you'e been to a walmart right? They were *EXTREMELY* ugly, horrible shoes.
One night I recall the system sent me *5* palettes of shoes (1-2 is normal) which took a herculean effort to find *somewhere*, *anywhere* to store them.
And that was the job, every night. Somehow put away the incredible ammount of shoes that come. Every night, re-arrange "the stacks", re-arrange "the steel" to fit shoes that nobody wanted, that nobody could stop from coming.
One morning the manager walks up to me and says "Good news, they've decided to keep you full time!" to which I replied "Oh no dont you dare".
Re:Huh? (Score:5, Interesting)
Understanding your method of assessing the data includes lumping data about vendors, data about shipping, inventory status (alone, a huge category), etc., 1.5 MB "per person" isn't huge. The error is in your model as most of the system contains data about things other than customers.
That said, you would be surprised what
The best thing a consumer can do to counteract this consumer surveillance is to toss junk into the system. Here are a few suggestions:
- borrow your mom's/mother-in-law's card and go on a shopping spree for frozen pizzas, candy corn, condoms and saran wrap.
- apply for new cards all the time. provide creative answers as to your address, occupation (animal disposal officer is one of my favorites - someone must be puzzled how many dead animals there are in my city from all the people with this occupation). BE SURE TO ONLY USE CASH with these cards so they don't get an identification anchor.
- spike the data with sustained purchases of one product for a period of time. this is especially fun at smaller retailers that use inventory management - keep buying them out of one product (preferably low cost and low shelf inventory so it is easier and cheaper to do). keep it up for 90 days. then stop buying it and go to another store.
The more you can junk up purchases (especially on anchored cards like friends, in-laws, etc. that have different buying habits), the less valuable the database is.
chaos in the mix (Score:3, Interesting)
If you're willing to break the law, you can even do worse harm. But I don't condone that.
Using legal methods to increase the entropy are the best way to fight the marketing databases.
Re:Please remind me (Score:5, Interesting)
Or you can be a privacy-advocate Slashdotter and hate that they want RFID tags in everything.
Or you can be a Republican or Libertarian Slashdotter and admire that Wal-Mart opposes government interference in business (you do NOT tell Wal-Mart how to operate).
Or you can be an apolitical Slashdotter and just agree that, for some products, it's the cheapest place to go.
I'm the socialist Slashdotter. I know it's not much better but if I need something that I know is at a big retailer I make the trip to Zeller's [hbc.com] first. SILE (Solution Involving Least Evil)
Re:"Nothing for you to see here. Please move along (Score:2, Interesting)
If anything, 460 TB seems like an understatement. Not to mention the claim that the Internet contains less than half of that. I alone have over a terrabyte of shit downloaded from the Internet. I seriously doubt there is only 229 more terrabytes to download.
Re:2004 = 1984 + 20; (Score:3, Interesting)
What they do with computers... (Score:4, Interesting)
The systems have the layout of every walmart store in them, and the stores respond to orders from the main office to move products around on the shelves. The systems will tell various stores to move products into different places, and anaylyze the results. If a store is making more money with XYZ sitting near the entrance, then the WOPR tells more stores the move that product into place, but still plays games against shoppers with a few more. It's basically an insanely well oiled statistical war against the shoppers to squeeze every last penny out of them. I hate to say it, but it doesn't work on me when I go there. But overall, it's creepy, and impressive at the same time.
PS- I had this evil idea. If anyone is into the hactivism role, embed a voice recorder IC into a telephone set that matches your local WalMart's phones. Get the code to get on the PA system, and setup your "rouge" telephone to bump onto the PA every 5 hours or so. Be sure to include sounds to make it sound like someone is picking up the phone, and hanging it up. It will drive them nuts. Some stores seem to use Lucent sets on the wall (MLX-xxx) which are most likely ISDN on the back. Other stores seem to have analog ports on a lucent system. Just remember to give me props. Feel free to announce all shoppers a winner of a contest where they get everything they can stuff into a cart for free. Or remind them about the $700,000 in taxes the minimum wage making people cost the community at every WalMart.
Comment removed (Score:5, Interesting)
Walmart does drop your income (Score:5, Interesting)
Re:Did you know... (Score:3, Interesting)
I read an article years ago by a fire chief, giving advice on fire safety for hotel guests. Among items like not taking a room above the seventh floor (the reach of a ladder truck), he said that if you smell smoke, you DON'T call the desk first. You dial an outside line, call the fire department, and THEN tell the desk.
rj
I've actually worked on this data before... (Score:5, Interesting)
IIRC, It seems like one of the strange correlations we found is that the two items most commonly purchased together were beer and baby diapers. Go figure...
Re:Walmart does drop your income (Score:3, Interesting)
They are pretty big. I wonder what will happen once they become too arrogant to behave rationaly.
Re:Correlation doesn't imply causation!!!!! (Score:3, Interesting)
If pop tart sales go up, head for high ground?
For some reason that statement reminded me of the theory (urban legend?) about Domino's being able to predict major events based on their pizza orders to the Pentagon and the White House.
Try Googling for some combination of "Domino's pizza pentagon desert.storm" without the quotes. Here's a sample [tafkac.org]: (emphasis mine)
Earlier this year we reported that Domino's Pizza claims it can predict when the government is about to undertake some sort of major activity based upon the increase in pizza deliveries to the Pentagon and the White House. Pizza orders increased substantially just prior to troop deployments to Grenada, Panama, and the Middle East.
According to The Washington Times of August 21, 1991, during the early hours of the abortive Kremlin coup in August, Domino's "Pizza Meter" registered 102 deliveries to the Pentagon, breaking the Gulf War record by one; the White House ordered 52 pizzas, breaking its Gulf War record by seven.
The CIA, by contrast, learned its OPSEC lesson: There were only two orders, and they were quickly cancelled.[9,10]
Re:"Nothing for you to see here. Please move along (Score:1, Interesting)
In my whole country I don't know of a single Wal-Mart...
Re:incredible! (Score:3, Interesting)
Using only a fraction of that data allows statisticians to prove that hurricanes cause an increase in the sales of certain goods, and by how much. Any schmuck can tell you non-perishables will sell more before a hurricane. Can he tell you how much?
Wal-Mart's predictions will be quantitative rather than qualitative, and they'll be able to make more money (at no-one's expense) as a result.
It's not incredibly complicated, either. Given the amount of data you'd need a something more sophisticated than just Excel to analyse it, but on small scale I could do the analysis with just a few basic Data Modelling notes from University and a PC
Re:So, if Walmart put up a web interface... (Score:2, Interesting)
Basically you can build queries, schedule them, and retrieve the data in certain typical format (Excel, text, CSV, etc). It was a tedious manual process because Wal-mart would not work with us to provide automated text feeds. Granted this was in 2002 so things might have changed since. They were also extremely strict about access (with good reason).
Re:Have another glass of cool-aid (Score:3, Interesting)
Brandybuck's Law states "the collective inteligence of an organization is inversely proportional to its size." There's a lot of reason for this, but it's a genuine observable phenomena. Just ask anyone who's been in the military.
If it's "a genuine observable phenomena [sic]" then surely there are scientific studies documenting those observations. Please point me to one because I'm currently under the impression that "Brandybuck's Law" is complete nonsense or just a funny observation from a frustrated corporate "human resource." (I can relate.)
If a law needs only one contradictory observation to prove it wrong, I offer the following:
I've always viewed Novell's products as technically superior to Microsoft's products. Novell, Inc. is also smaller than Microsoft, Inc. But Microsoft is a much smarter corporate player/criminal than Novell so they dominate their market. Novell tends to make stupid marketing and strategy decisions, as well, therefore the smaller-equals-smarter theory is disproven.