By Bala Cuddalore, January 17th, 2017
In the promise of becoming data driven decision making entities, more and more enterprises are building their next generation data platforms with improved data infrastructure to capture their ever growing data. With the growth of data, integrating data and building value out of data to fulfill increasing business needs is becoming exponentially complex for data engineering, BI and Analytics teams. It is not that these teams are not doing a good job of preparing data, surfacing intelligence and adding value. It is more a question of how timely and actionable their insights are to business. These insights must also come with a level of urgency and priority for the business leaders to leverage this knowledge in making effective decisions. Decisions that eventually result in growth, increased revenue, increased customer satisfaction and or possibly even prevent losses.
Presently, data engineering combined with the infrastructure team deliver data platforms that serves data from various source systems within the enterprise. The BI team with the help of data engineering integrates all data from within and outside the enterprise. Upon this data foundation, the BI team constructs metrics and KPI required for reporting to various stakeholders across the company. Having disseminated data, the analysts find trends and find out why things are happening the way they are. Finally the data scientists, prescribe and recommend what needs to be done to drive a certain business goal. Looking at a number of enterprises and how these teams are organizationally structured, they work mostly in silos and tackle individual problems or specific initiatives as opposed look at enterprise metrics as a whole.
With this being the basic construct of how data, BI, analytics and data sciences teams function, the expectation is that the data travels through the various teams within the organization and into the hands of the decision makers in a timely manner as actionable insights. This is where the true value of data is expected to be harnessed.
However, contrary to the above expectations, the value of data is somehow diminished by the time it reaches the decision makers and business leaders. The challenges can be summarized as follows:
What is Analytical Intelligence ? Coined by a renowned American psychologist, Robert Sternberg (Tri-archical Theory of Intelligence), it is the ability to analyze and evaluate ideas, solve problems and make decisions using analytical, creative & practical giftedness of individuals.
How does Analytical Intelligence apply to Data, BI, Analytics & Data Sciences ?
The parallel that can be draw to this is that, as much an individual has to apply various types of intelligence to be well rounded, we as data proponents must advocate use of all the tools and arsenal to better aid business leaders to be data driven and make fact based decisions.
Apart from the technologies that are available to source, transform, integrate and store data (small and big data), we must particularly deploy methodologies and processes that we use to make data more valuable to business.
The processes & methodologies being referred here are:
Business Intelligence (BI) - Creating BI from data by integrating and aggregating data points from several sources to provide insight on what is happening using a single source of truth.
Analytics - The observations made through BI must be subjected to further analysis on why it is happening. Using hypothesis and proving it right or wrong with analysis is a must to determine what is useful to business versus not.
Data Sciences - And finally, from data sciences, machine learning tool kits, create practical models to automate delivery of value added insights to business leaders to help drive decisions where it most matters.
A new breed of platforms called as Analytical Intelligence solutions, will bring BI, analytics and data sciences into play to leverage data to the fullest.
Analytical Intelligence platforms of the future will build BI from data by integrating multiple data sources within and outside an enterprise into a single holistic view. Single source of truth is the closest that is analogous to the holistic view term. Beyond the holistic view, the AI platform will allow organizations to define and configure metrics and or KPI based upon some agreement from all stakeholders and business leaders.
Using this holistic view, these AI platforms will break data into smaller snippets that can be packaged to deliver the maximum business value. This is still business intelligence and there is nothing new but extracting data nuggets with maximum value becomes critical. This cannot be done without analyzing data.
In order to distill the sea of data into data nuggets, AI platforms will run automated machine learning algorithms to perform analysis that is required to observe trends within the data and extract the essence. This is building the analytical portion of the solution. What are the trends being observed? What are the reasons?
These AI platforms will go on to tap into more complicated algorithms to reveal any associations, and causality for the trends that will then be captured along with the degree of importance and significance.
To determine the choice of the algorithm and increase probability of causality and predictability, the AI platforms of the future will bring all tools in the data sciences tool kit to the problem. Having applied the most relevant machine learning algorithms to identify patterns and increase probability of causation, the raw data will now be enriched with business value and be ready to be presented to the business leaders and decision makers to recommend and prescribe actions.
Finally presentation of all of this value added information into succinct messages requires nift packaging. The consumption of these nuggets of information will also highly depend on using an effective delivery platform. Relevancy and timeliness are critical to ensure that the right information is delivered to the right stakeholder with an enterprise based on role, business unit and or function.
Using Analytical Intelligence and combining the strengths of BI, Analytics and Data Sciences processes and methodologies, a new breed of Analytical Intelligence platforms will extract what is most important to business and provide value from data that is unprecedented. In doing so, the four challenges that were outlined above can be resolved.
To recap the challenges and how they would be resolved:
Since all the value delivered out of data is from a single source of truth, the entire enterprise will be looking at KPI and metrics where there will be no two interpretations of the data.
Timely delivery of succinct messages combined with business intelligence, analysis and data sciences built on top of pristine data will allow the decision maker to focus on what really matters and take necessary actions.
Finally with basic intelligence and analysis already taken care of, analysts can scale better with the understanding of where their efforts should be focussed and business leaders can drive their companies by pushing the right levers in the areas that are most impactful to their goals.
In closing, only with Analytical Intelligence can enterprises address the challenges we face today with extraordinary growth of data and address the inability of current day tools to discern actionable and timely insights. Analytical Intelligent systems are the future wherein data combined with BI, analysis and data science tool kits will be deployed to enable organizations to become completely data driven and make decisions that set them on a trajectory to greatness.
Find out more on how the Analytical Intelligence platform from nlightn Technologies can help your company stay on top of your metrics.
As always, thanks for reading & your comments!
-- Bala Cuddalore
By Bala Cuddalore, February 11th, 2017
Metrics are measurements of tasks performed by employees to produce, deliver products or render services in a given enterprise. It can also be measurements of actions taken by customers as they engage with any company, its website, their products or use their services. Metrics allows enterprises to track efficiency, performance, quality and even customer satisfaction.
Metrics can also be defined as measurements created when employees, customers, vendors perform their actions or tasks that are tied to the execution of business processes tied to the business model of the enterprise.
As an example, an enterprise manufacturing cars, will have to procure raw materials, possibly source and / or manufacture parts that are delivered to an assembly plant. The car is then assembled in a production line and then tested before being distributed to dealerships and showrooms. The enterprise then gets paid based on sales. This is their business model in a simplified sense and its employees perform various tasks to execute this business model.
To measure the efficiency of the tasks performed by its employees or to track how the company is performing, there is a need to measure. And this measurement is made possible through metrics.
Metrics are by products of data collected through automated systems as employees work on their tasks or when customers interact with systems to report on the actions they are working on or have completed.
Organizations define metrics based on what they want to measure. It is usually to track top line and growth related activities or events. But sometimes it has to do with bottom line related events that allows companies to save money. And more relevant than ever before Customer satisfaction has become an important metric.
Typically, top line growth is measured by the number of customers a company acquired during a time period or the amount of revenue they recognized.
Bottom line measurement is usually to do with much money or resources a given company saved while executing their business tasks such as manufacturing or operations. Some of the metrics include gross margins or net profit.
While these are more at the corporate level, individual teams can also measure their performance and efficiency for their individual department or function. For instance, a customer service team can measure how effectively they are handing customer complaints or phone calls. Or an operations team can measure how quickly they are able to turn around customer orders and are fulfilling them.
Similarly, a manufacturing company can have many metrics sprawling their entire organization. They can adopt Lean or Six Sigma methodologies to see how they can upkeep and improve quality of their products or minimize wastage.
Therefore depending on the line of business or industry, metrics are defined based on the need to track the performance, efficiency and or quality of company, its products or services and / or teams, business units within.
While a vision of a company is established to corral its employees towards its future, it is often necessary to track its performance on the road towards the vision.
And the road usually milestones that the company must cross towards its journey. These milestones tend to have metrics tied to them to ensure that they have hit the milestone successfully. Such metrics are measurements tied to corporate goals. They are loosely called Targets or OKR’s (Objective Key Results).
For instance, an Internet company can set a milestone to reach 1 billion user registrations by a certain year. And to track this milestone, User Registrations becomes a metric they would need to track.
Corporate metrics can also be tied to target goals such as Revenue or achieving a certain level of quality or even Customer satisfaction.
Corporate goals are set by the executive management of the company along with input from the Board. And the set of metrics that are defined to measure the progress made towards corporate goals become corporate metrics.
Now that does not mean that individual teams or departments cannot have their own set of metrics alongside corporate metrics. Those department level metrics can also co-exist. It would be wonderful to although tie department level metrics to corporate metrics or goals as it would allow employees within a specific department to see how their work impacts the company’s performance.
Once a set of metrics tied to corporate goals are set, these metrics are then tracked periodically to ensure that the company, its departments are on track to hit those milestones.
These metrics are reviewed by the management and the board in periodic meetings. Typically at the board level, it is done once every quarter in what is sometimes called as a QBR (Quarterly Business Review).
However executive management and business leaders who are key decision makers within an enterprise do look at their metrics more regularly so that they can make decisions that allow them to steer their company towards their goals and also avoid setbacks, hurdles. In the internet age, management looks at metrics at least every week if not daily. And sometimes even intra-day if an event warrants for that kind of scrutiny such as a big product launch.
Metric computation and governance begins with an enterprise coming together to define what metrics they want to define and standardizing them across the company.
This activity is usually done by business and more particularly by Finance or Operations. Once the business leaders agree on what their teams or business units or the organizations must measure and track, Technology takes over the gathering of data and the metric computation to prepare the metrics.
Once metrics are prepared in terms of data, Business Intelligence and/or Analytics teams produce dashboards, scorecards that render the metrics defined in a consumable format. These dashboards are provided access to across the enterprises.
For metrics to be shared and accepted across an enterprise, the stakeholders or keepers of these metrics must govern these metrics. Governance of these metrics includes standardization of metrics, their validation for accuracy and monitoring to ensure they are not compromised over time.
To see how your company can benefit from driving the right metrics towards your corporate goals ... sign up for a demo of nlightn - The Analytical intelligence Platform
Hope this provides an overview of what metrics are and their purpose. Thanks for reading & your comments!
-- Bala Cuddalore
By Bala Cuddalore, April 27th, 2020
Previously I had written about how three emerging technologies are going to converge causing a perfect storm in the world of data. In this blog, let us delve into the details of what that perfect storm would be and how that will change our thinking and computing paradigm for the future.
And for those who missed my previous writeup on Data Disruption from Convergence, the three emerging technologies are 5G Networks & 5G enabled devices, Industrial IOT and Edge Computing.
This article covers the emergence and convergence of these technologies causing a radical shift in the world of data, data pipelines, its analysis and its application. The underlying technologies that are currently in place will shift in an unprecedented manner changing how data is collected, processed, analyzed and how the results are pushed back to the user. This opens up a plethora of opportunities for startups and diversification for existing enterprises.
In the new world where in these three technologies have completely matured and converged, the data components can be broadly categorized into the following:
In the height of its maturity, industrial iOT would be embedded in every object, structure that surrounds us. We will see Data Emitters in every automobile, mobile device, building, office, home, appliance, machinery, computer, industrial hardware to name just a few.
At the very least these iOT devices would emit their identification, location, their surrounding, status along with possible user interactions and other diagnostic data. While each of these devices today might be capable of connecting to the Internet and transmitting their data, the first disruption is to take place here
The first disruption will be how these devices will connect to the Internet. They will transmit data not through our conventional wireless networks. The technology they are built on will hinder the amount of data that is needed to be transmitted causing bottlenecks, latency and failures. They would have to connect directly with their 5G carrier to transmit or be aggregated at its source and then transmitted through iOT hubs closest to their source called Data Aggregators.
The role of Data Aggregators will be critical to efficiently, economically aggregating and transmitting all of IOT data from a given location.
Data Aggregators will combine basic IOT data and generalize the common themes into either channels or packages. Data common to a location, device will be redundant between multiple broadcasts and hence can be channelized into different frequencies with the 5G spectrum or packaged by combining multiple like iOT packets into a string of packages to be transmitted together.
The Data Aggregators will be iOT hubs which will replace our current day Internet routers and will connect to the 5G networks directly from a given location.
In another scenario, our mobile devices such as phones, tablets and computers can be converted into iOT hubs by choice. These devices can also aggregate data from multiple iOT devices in the surrounding and connect to the 5G network to transmit.
And due to the sheer number of iOT device in a location, it will no longer be feasible to upload all of that data into the cloud for processing and dissemination. This is where Edge Computing will enable how and where data is efficiently processed.
The growth of Edge Computing will allow us to efficiently manage the amount of data that is required to be processed. New Data Processors on the Edge Clouds will harness the data and turn it into opportunities for businesses in real time.
As we know today, data centers that managed our Cloud infrastructure is widely distributed across a country or region. However, with the amount of data being transmitted and the need for near real time processing, feedback, it will be necessary for data centers to become more dense.
Meaning Edge Cloud computing will demand the need for cloud data centers to show up in every building that is capable of hosting the infrastructure and connectivity to the 5G network. This is where the second disruption will occur.
In an alternate scenario, our mobile computing device such as a tablet, phone or computer can also become a Data Processor capable of collecting the data from the iOT hubs and processing them instantaneously.
These Edge Computing Data Processors may very well also transmit to the large clouds in parallel or in sequence as they will only be able to hold a finite set of data and not capture data or persist data indefinitely.
These Edge Computing cloud grids will not only receive the data from the 5G networks and iOT hubs but will also process the data to analyze, predict and recommend outcomes, actions required. For this reason, the Edge Computing Grids will be sophisticated to run ML / AI models within micro seconds and beam back the results to the Data Receivers.
Data Receivers would primarily be receptacles that would receive prescriptions and recommendations from the Data Processors. All of the mobile devices can serve as a Data Receiver.
With the increasing power, speed and capacity of the mobile devices, they will be able to multi-task to perform the role of transmitters, aggregators and receptors or receivers.
Other receivers would include computers, television screens, screens on your refrigerators, windshields or dashboard panel on your cars or even screens on your treadmill.
These receivers will display messages, alerts, recommendations, routes, maps, vital statistics of your surroundings, environment and metrics from your body’s physiological functions. They can be descriptive, predictive and prescriptive messaging.
This seamless integration and functioning of emitters, aggregators, processors and receptors will not be possible without a supporting architecture.
We will dive into the architecture and the plumbing of data, data pipelines and the data processing prowess that is required to enable such applications in a future discussion.
The nature of the data sources are going to change considerably to warrant for a change in thinking of how to build data pipelines. To begin with, it will have certain characteristics of “Big Data” which includes the velocity, the volume and the variety.
However instead of having both large number of rows and columns, it will be a large number of rows in most cases. For example, all iOT devices will have the following:
Identification number, device location, device type, specific attributes passing the diagnostic or conditional or data that is constantly changing and therefore needing continuous transmission.
As data comes more frequently and at a high velocity, they have to be constantly processed continuously in micro batches. While data in the previous generation would be stored for future analysis, with this convergence, data would have to be both stored and analyzed in parallel. Lambda architectures would help with processing and storing.
While data is sent to cloud storage for future analysis, another parallel data pipeline should be capable of sending data to edge clouds for quicker / near real time analysis.
Analysis of data would have to occur in the micro batches received to conclude prescription and recommendations on what actions need to be taken based on the data received.
Algorithms residing on the edge clouds will run on this data, perform the analysis and provide prescriptive actions to be taken.
The result of data analysis would have to be packaged and transmitted back to the data receivers for users to take necessary action.
In the architecture illustrated above, the velocity, volume and variety of data created, transmitted, processed, analyzed and feedback returned will have to happen in near real time due to bandwidth available. To explain further, 5G will mandate that all other components work in near real time. In the lack of such sub-second responses, the end-user will be waiting with a loading sign for many minutes. Therefore the architecture of the applications that process, analyze and provide prescriptions, recommendations will have to happen in a split second.
Current cloud infrastructures as we know it today will be fragmented making it more distributed for process and analyzing data. This will give rise to data centers in every city and metro to reduce latency to transmit and also bring processing and analysis of data to closer proximity to where the data is generated.
Alternatively as mobile devices get more powerful, they can handle the volume, velocity, processing and analysis of data cutting down the entire need for the data to be transmitted. This explains why mobile manufacturers are building AI chips on to the phones.
Convergence applications will be leaps and bounds richer in content than current day applications as they will embed Augmented Reality (AR) to enhance user experience, intuitiveness and usability. The bandwidth combined with increased computing and processing power of adjacent technologies will lead to the rise of next generation applications.
Home appliance such as Refrigerators, automobiles emits diagnostic signals that helps prevent failures and notifies owner of preventive maintenance.
Electric circuit boxes, sprinkler systems, cooking ranges, bulbs emit usage, wattage to help home owners save energy and resources.
Installation and troubleshooting applications of home appliances would also become prevalent.
Industrial machinery have iOT devices that emit temperature, pressure measurements, report on gases, steam, toxic materials that would be hazardous and notify operations of potential risks, dangers or schedule preventive maintenance.
First responders such as EMR, Police and Firemen can largely benefit from receiving up to the second information on the surroundings, the situation and get recommendations on how to act, respond.
Weather, road and traffic conditions can help autonomous cars take decisions to prevent accidents and switch on lights, wipers apart from warn the passengers.
Thanks for reading & your comments!
-- Bala Cuddalore