A 2015 research by Gartner projected that there will be over 6.4 billion connected ‘things’ by the end of 2016. This represents a 22 percent increase from 2015. By 2020, this number will have risen to over 20.8 billion. For many companies, big data is the key to developing a deeper understanding of their core businesses. However, the prevailing challenge for many businesses lies in both data management, and how to derive insights from a vast pool of unstructured data. According to Michael Hoskins, Actian’s chief technology officer, many businesses lack the appropriate infrastructure and tools that will help them exploit the full potential of big data and reap the benefits it has to offer.
As business intelligence (BI) technologies continue to evolve, they exert immense pressure on traditional data management and analytics technologies. Hoskins notes that traditional tools are being strained because they cannot keep up with the speeds of data collection and neither are they able to manage the huge volumes of data collected. These technologies are also unable to “understand” the different varieties of data obtained by the business from multiple sources. In the second part of our discussion, we look at some of the effective strategies, and some of the new tools recommended for modernizing your BI implementation to meet new data and analytics demands.
Data Discovery Tools
One of the greatest barriers to the expansion of BI systems for meeting current analytics demands is the cost and complexity of implementing these extensions. Unfortunately, many businesses don’t have the luxury of huge budgets. This slows down the entire process of upgrading enterprise BI systems. They have resulted to the acquisition of data discovery tools in a bid to achieve autonomy in purchasing of BI technologies. According to David Stodder, the director of research for business intelligence at the Data Warehousing Institute (TDWI), data discovery tools both complement and compete with existing BI tools.
These new tools are tailored to simplify the process of deploying and customization, especially by the nontechnical users within an enterprise. These tools will capitalize on the significant gains made on improving features such as data visualization, in-memory analytics, dashboards, and search to equip non-technical BI system users with applications that are superior and more robust than those they worked with previously. The impact of adopting such tools is mostly seen in the advanced analytical flexibility they offer compared to standard BI platforms, which are a bit restrictive.
The Extended Data Warehouse (XDW)
BI and analytics experts, Claudia Imhoff and Colin White propose the extended data warehouse (XDW) as a significant improvement on existing enterprise data warehouse (EDW) in acknowledgement of the challenges facing the later. The experts both agree that indeed, the EDW is limited in its capacity to handle new data types, perform investigative or experimental analyses, and conduct in-application analytics. Of course the traditional EDW is still relevant and will continue to be in the foreseeable future because of its robustness in fueling BI analyses by providing high-quality data.
The proposed XDW contains three major components: the investigative computing platform, the data refinery, and the real-time analysis platform. The first component’s main task is the creation of new analytics and analyses models, and the exploration of new data. The data refinery receives all sorts of structured and unstructured data from multiple sources, such as e-commerce sites and social media, distills it into meaningful information, and transfers it into the data stores. The final components, the real-time analytics platform works within the operational environment to create and/or deploy applications that run in real-time. The application can perform complex tasks such as risk analysis and web traffic flow optimization, all in real-time.
Investing in BI Staff Training
Modernizing BI implementation is almost guaranteed to hit a brick wall if the business does not empower its personnel and staff to match the newer technologies and architectures. According to Gartner, traditional BI and Big data BI exhibit distinct characteristics in many aspects including “fundamental data structures and formats, programming languages and styles, processing environment, and data modeling approach”. Supporting big data BI projects such as Hadoop-MapReduce requires specialized and multifaceted skills.
It is almost impossible for one or two individuals to run and manage these projects. A team approach is preferred. Even when dealing with small and exploratory projects, a business stands to benefit most by covering all the skillsets required to run Hadoop-MapReduce big data. They include business domain knowledge, statistics and data mining technologies, knowledge of Hadoop computing cluster management, java programming, UDF design and functional programming, among others. Highly effective big data BI teams are those that are derived from multiple departments, according to Gartner.
Conclusion
It is evident that as big data BI technologies continue to grow and improve they pile pressure on traditional BI tools making the later unable to handle the immense volumes of data and their different varieties. Even if many businesses acknowledge these challenges, they may not be quick to expand their enterprise BI systems due to the challenges of expansion costs and complexities. Data discovery tools is the solution to bridging this gap. For businesses with human and capital resources to support BI expansion, the extended data warehouse (XDW) is ideal for expanding the current enterprise data warehouses. That, in combination with competent and specialized staff and guidance from an established technology consulting company will help businesses overcome the challenges posed by big data growth while reaping maximum benefits from newer BI technologies.