What is “big data”? It’s simply data that is too voluminous to be processed using standard tools. So, if Excel is your analysis tool of choice, when a file is so large that it chokes it, that’s big data.
Of course, you can adopt more sophisticated analysis tools, but, eventually, those will be overwhelmed as well—or the data will be too large for your computer. What do you do then?
There are two main challenges when working with big data—storing it and analyzing it. Both of these challenges can be met by upgrading to bigger and better hardware. But, this is expensive—and eventually you will reach the point where you have the biggest and best hardware available.
A cheaper, more flexible, approach is too link lots of cheap computers together and have then work on separate parts of the problem simultaneously. When they are finished, their work is collated to produce a final answer. Think of ants building their nest. No single ant could even conceive of such a structure, but, together, through massive parallelism, they work miracles. This is the idea behind distributed computing solutions to big data problems.
One approach to working with big data is to summarize or sample it until it is small enough to process with traditional data management tools. However, this results in a loss of detail and prevents us from using the data to its full effect. For example, if we wish to provide personalized products to our customers, we need to have access to each individual’s data—not treat them as a “market segment”.
If you would like to talk to us about how you can make better use of your big data, get in touch.
Technologies
Our expertise in big data analytics includes:
