Why Big Data gets it Wrong

As a young man, I commanded an anti-aircraft gun platoon. We had one radar unit which tracked enemy aircraft and controlled two 40mm guns.

When I started, our radar and calculation vehicle was all analog with vacuum tubes and rotating spindles. It took hours to start up, but on the other hand kept the crew warm during winter. The radar tracked the aircraft, the analog computer calculated the distance from aircraft to each gun, calculated shell flight time and where the aircraft would be, and then instructed the guns where to turn and how high to elevate.

It was amazing that the whole contraption worked, but it did. Well, almost. When everything was set up, we would release a bright red ballon with a radar reflector, lock onto that and tell the computer to aim straight for the target. Then we looked through the barrel of the gun and had a little box to make small adjustments until the tracking was spot on. After dialing in the guns using the “cheat box”, we could hit our practice targets.

During my time in the air force, our analog computer was digitized. Now everything was carefully calculated by computer, so our beloved cheat box was not considered necessary. And we couldn’t hit a thing.

Too many modern systems are built on the assumption that careful calculation will provide a definite answer. But your input data is very rarely as correct as you think.

If you are using buzzword-compliant systems based on Big Data and Deep Learning, you’re making the mistake the designers of my updated gun system did. By all means let the system calculate and suggest, but let real people make the decisions and provide them with the equivalent of my “cheat box” so they can adjust the system to the real world. That’s the only way to build a system that will meet business needs in the real world.


I help people and organizations use appropriate information technology to achieve their goals. For more tips, sign up for the Technology That Fits newsletter at http://eepurl.com/0fpvT and follow @techthatfits.