Every day the world generates around 2.5 quintillion bytes of data (1 billion gigabytes). Actually, 90% of the world's data today was created in the last 2 years . That huge amount of data needs to be processed and analyzed as quickly as possible so it can provide meaningful information to improve our daily lives.
Effectively dealing with big data means sifting through it to find relevant information, model its elements, and transform it into useful information and knowledge. Using machine learning techniques for modelling, prediction, clustering, and knowledge discovery is the preferred solution. When the complexity or volume of the data to be treated per unit of time is beyond the capacity of human operators and experts, machine learning, as part of data mining, provides methods to automatically treat and extract the information from the data.
The amount of time machine learning systems take to handle these huge volumes of data is crucial. Therefore, big data initiatives should focus not only on the volume of data, but on the speed at which data is processed. Massive distributed computational components increasingly run large-scale machine learning and data analytics tasks. Modern distributed systems, like Spark*, and computational primitives, like Map Reduce*, have gained significant traction because they enable big data analytics . However, anomalous system behaviors and bottlenecks significantly affect the performance of these systems.
One of the factors influencing performance is the operating system (OS) that executes machine learning. The operating system’s handling of the hardware’s capabilities crucially impacts application performance. As the Linux community continues to redefine the boundaries of what is possible for cloud-based Linux distributions running on new silicon, both power and performance play an increasingly important role in the industry.
The Clear Linux* Project decided to use the latest Intel® Advanced Vector Extensions (Intel® AVX) technology. We are making it easier to develop applications that take advantage of new floating point and integer operations of newer servers, starting with Intel® architecture code named Haswell. Intel AVX 1.0 allowed programmers and compilers to do floating point math highly in parallel (SIMD), improving performance in compute-heavy workloads such as high performance computing and machine learning. Intel® Advanced Vector Extensions 2 (Intel® AVX2) introduced floating point math, more supported operations, and support for integer operations. However, as applications compiled for Intel AVX2 will not generally run on processors prior to Haswell, developers favor the older Intel AVX due to compatibility. In Clear Linux we developed a key enhancement to the dynamic loader of glibc. This enhancement lets developers ship two versions of a library: one compiled for Intel AVX2 and one compiled for systems without Intel AVX2. Thus, application performance is tailored for the underlying platform.
One of those libraries is OpenBLAS*. OpenBLAS is an open source implementation of BLAS, or Basic Linear Algebra Subprograms, which provides standard interfaces for linear algebra. Replacing the default linear algebra libraries with OpenBLAS increases speed, especially the computation of the dot function for matrix multiplication used in diverse scientific libraries.