Analyzing 3 TB Field Measurement Data Set

This arcicle describes the use of intelligent algorithms for analysingfield measurement data. The main focus is on describing the generalworkflow and practical issues when the amount of data is ''big''and typical data analysis methods for small data cannot be used. Whenthe amount of data is tens of terabytes, it is no longer fitting tocomputer memory. Data visualization is also challenging: visualizationtools can only render a small fraction of data to computer screenand visual inspecting of the whole dataset is not meaningful at all. Thedata is simply too big. Thus, new approaches to study data are neededwhere the data is processed automatically in calculation clusterswithout manual human work. The basic idea of data mining is to graduallyreduce the amount of data by various techniques, as long as the finaldata contains only information relevant to the research question and insuch a compact form that its viewing from the human point of viewis rational use of time.

This creative work has been published under the Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA 4.0) license which allows copying, and redistribution as well as adaptation of the original work provided appropriate credit is given and the same license is used as for the original work (the above link must be included). Any alterations to the original must also be mentioned.