Big Data Preprocessing Frameworks: Tools and Techniques

Abstract

Big Data Analysis blended with computational algorithms is a novel tendency in feature abstraction. This involves acquiring knowledge from reliable data sources, rapidity in processing information, and future prediction. Big Data analytics is dynamically evolving with variant features of velocity (analysis time has drastically decreased subsequently), volume (corpus size raise from Big Data to Bigger Data) and Vectors (consonance to dissonance). Organizations now focus on analyzing data that are getting accumulated and are interested in deploying analytics to withstand forthcoming challenges.

Big data involves a massive volume of data that are so large, and it is difficult to process using traditional database and software techniques. In the use of big data applications, a technical barrier is encountered when moving the data across various locations, which is very expensive, and it requires large main memory for holding data for computing. Big data includes transaction and interaction of datasets based on the size and complexity that exceed the regular technical capability in capturing, organizing and processing data in cloud environment. To encounter this problem data preprocessing is necessary. Various data mining and machine learning techniques are used for preprocessing. This paper provides an in-depth study of data preprocessing for big data analytics. It also contains generalized framework and tools used.

Keywords: Big data, data processing, data preprocessing, big data frameworks