Big Data – Old Problem, New Name?

During World War II, Germany sent coded communication message using their Enigma machine.

Hundreds of people in Bletchley Park, England spent every day intercepting the messages, cracking the code.

But there were just too many messages to process.

As soon as they started to look at one messages, another would arrive.

Then another. And another.

So they added more people, and more people, and more people.

But still the messages kept coming.

It seemed as though the more people they added, the more messages they could record.

And the more messages they could record, the more messages went unread.

It seemed as though the focus had shifted from breaking the code, to collecting the data.

They realised they were missing something.

So they took a step back.

Started to filter the endless flow of data.

Correlated it with data from other intelligence sources.

Focussed on the few messages that appeared, on the surface at least, to be the most important.

It changed the game.

With the noise turned down & prioritised, the talented analysts could focus on just the most important data.

For a long time we forgot about the ‘big data’ of the First World War.

After all, everything is relative.

Now we have the internet, virtually free storage, mobile devices, and the internet-of-things.

Once again we have more data than we know what to do with.

The team at Bletchley worked out how to handle it.