Capstone case study

See how ML helps to prevent climate change and safeguard community health

This case study was done as part of IBM Applied Data Science Capstone Project via Coursera. The aim of this study was to enumerate suitable city waste management locations for the city of Kochi, Kerala — southern state of India. I chose Kochi city for this case study not only because it’s my native place but Kochi city was and is still continuing its struggle to efficiently manage city waste.

Kochi city — an overview

Kochi city (sometimes called Cochin city) is a port city, which has at least 600 years of trading history mainly with Chinese, European and Arab merchants and this transformed the city into the financial capital of Kerala. To name a few establishments, Cochin Stock Exchange — the only stock exchange of Kerala, international container transshipment terminal, ship-building industries, InfoPark — government operated IT park, Lulu mall — Asia’s largest shopping mall testifies the economic importance of Kochi city [1]. Kochi city is not only an ideal place for commerce but it also offers affordable housing options as well. Kochi is also a tourist spot — attracting tourists from all part of the world. So basically Kochi city is a limited edition all-in-one package.

The Waste Management Crisis

Kochi city generates about 380 tonnes of waste a day and 150 tonnes out of it are biodegradable and the remaining are plastic waste[2]. Presently, city waste is sent to Brahmapuram waste management facility to be treated. From the inception of this facility, complaints and concerns from public, environmentalists became headlines in the newspapers against the location of the plant and the way the site for the plant was fixed. Many of the residents were displaced by the government with compensation. River water pollution as the plant shares its borders with two fresh water rivers and destruction of plant life due to unscientific method implemented to select the location for the waste management plant. The current plant location has made the life of residents of Brahmapuram miserable — the nauseous stench from food waste and the nuisance houseflies and mosquitoes. Moreover, the incoming waste accumulated for processing is mixture of both biodegradable and plastic waste and the biodegradable materials decompose in the open air as the treatment processes in the plants are reported to be slow which causing another horror — climate change, as the bio-degradable materials are decomposing, thus emanating methane, a green house gas. Also, huge fire break-outs are becoming frequent and the toxic gas spreads throughout the city and disrupts the city life for days[3].

Due to this fact, city authorities have suspended waste collection from homes until a suitable solution comes up. Kerala State Pollution Control Board conducted a study on city waste management and reported that waste-to-energy is not a cost-effective solution for Kochi city’s waste management crisis [4] . Since, Kochi is growing, so is the waste generated per day. Brahmapuram facility is currently under overload as it has to accommodate medical waste in addition to commercial and residential waste, which further complicates the crisis. Even if an ideal solution is found out for managing city waste, the current location of the plant poses serious threat to the quality of ground water reserve as well as the two rivers that passes by. The current plant location is a disaster in the making and will affect the lives of Kochi residents and as well as Kochi’s economic health.

Case Study Roadmap

There have been several reports and concerns about the unscientific methodology implemented in finalizing the location of the current city waste management facility at Brahmapuram. [5]

One important evidence of fallacy in fixing the location of current plant is its nearness to InfoPark, Kochi’s own IT park which is only 5 km away from the plant. Another evidence is that the area is marshy and the risk of ground water penetration of chemical byproducts from waste decomposition. All these problems inspired me to look for an ideal plant location for setting up a new waste management plant — one that does not harm the nature or the society. Then came the next question, how?

The idea was to find the cleanest city in the world and simulate the locality/neighborhood of that city’s waste management facility in the city of choice. Here, the logic that I used was — if there exists a clean city then it is has a highly efficient waste management facilities and there will be strong and successful management practices behind it and those management practices will be governed by strong pro-environmental and socially-committed rules and policies. So, simulating the locality of such a waste management facility will be equivalent to simulating those highly successful rules and policies. Then, I went for the hunt and discovered that city of Calgary, Canada was one among the cleanest cities in the world and I chose it because city of Calgary consistently found a place in the top 10 cleanest cities in the world [6].

City of Calgary has two facilities for commercial waste management and I chose one of them, just by convenience sampling — the East Calgary facility. Then, it was time to represent the East Calgary facility somehow mathematically. The answer to that question was, Foursquare API. It provides location data for millions of venues across the globe. Using Foursquare API, venues within 5000 metres of the East Calgary facility was obtained and a vector of East Calgary facility was generated with the venue categories. Now, reference locality to be simulated was ready and it was time for obtaining list of neighborhoods in Ernakulam district, where Kochi city, the city of interest is located. Some of these neighborhoods will be selected as the best choices to consider for setting up a new waste management plant. Webscraping with BeautifulSoup was used to extract list of all neighborhoods of Ernakulam and Geopy was used to extract geolocation data of all these neighborhoods. Then just like the East Calgary facility, each neighborhoods of Ernakulam was represented as a vector of venue categories. Then there is the final preprocessing stage — ensure both East Calgary neighborhood and that of Ernakulam are represented via the same venue categories. In layman’s terms, I made sure that all neighborhoods were compared on the same grounds !

After the preprocessing, the East Calgary facility vector was merged into the dataframe containing the neighborhoods of Ernakulam — this is our clustering dataset.

The next step was to do K-Means clustering of neighborhoods and segment different neighborhoods out, but before that elbow curve analysis was performed to determine the best number of clusters. With the optimal number of clusters, K-Means was performed and the cluster to which our ideal neighborhood i.e that of East Calgary facility was determined and the neighborhoods of Ernakulam that belonged to the same cluster was found out. The initial part of neighborhood simulation was over. By taking into account, the drawbacks of present waste management facility at Brahmapuram, a second level of refinement was done on the list of neighborhoods of Ernakulam that was obtained after clustering.

Final Outcome

Webscraping and geolocation data capture resulted in 45 Ernakulam neighborhoods based on postal codes. The preprocessing stage, reduced the neighborhoods into 32. The clustering dataset was thus ready and elbow curve analysis indicated that optimal number of clusters was 4.

K-means clustering is depicted in the visualization below:

There were 16 neighborhoods of Ernakulam that belonged to the same cluster as that of East Calgary facility.

But, as a final step I decided to filter out the obtained Ernakulam neighborhoods by taking into consideration the an important drawback of the current facility at Brahmapuram — nearness to water bodies. This filtration process further reduced the number of choices to 7 (Palarivattom, Kaloor, Ernakulam North, Thammanam, Kadavanthra, Panampilly Nagar and Vyttila are the choices) and is depicted in the visualization below:

Conclusion

Kochi city is experiencing a major crisis in waste management for so long and this study presented with 7 suitable locations for setting up a new plant scientifically using machine learning. Now, the city managers can evaluate these recommendations and arrive at the best spot out of these, speeding up city administration and making decisions less error prone. Due to simulation of the neighborhood of one of the world’s cleanest city’s waste management facility, city managers can use the waste management policies and rules of city of Calgary with some tuning to fit Kochi city’s requirements. Therefore, all these decisions will make waste management process much more efficient and thus can prevent waste accumulation at facilities, thereby minimizing the amount of methane emissions due to it — a contribution towards sustainable development goals of United Nations (Goal 11) [7]. Serious public health concerns like epidemics such as malaria due to rain water stagnation, lung or skin disorders arising from unscientific location of the plant can be alleviated using this method. Geographical, political and community behavioral differences were not taken into account in this study and results may vary by considering these variables. Possible future extensions are implementation of the study with Density Based Spatial Clustering for Applications with Noise (DBSCAN), environmental hotspots or population density variable inclusion, usage of a different geolocation data vendor. Another advantage of this project is that it can be adapted for government and business purposes as well regardless of state or country.

References

1. Economy of Kochi
2. Fire break-outs becoming frequent at Brahmapuram
3. Waste management crisis of Kochi
4. Waste-to-energy not cost-effective at Brahmapuram
5. Public concerns about Brahmapuram plant
6. Top 10 cleanest cities in the world
7. Sustainable Development Goals of UN

Machine Learning for Scientific Waste Management was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Nature is Quiet
Its silence calms but also
Feeds and teaches us

In Response to Brett Christensen’s weekly prompt “Quiet”

A quick explanation of how important missing data is in Machine Learning and how to deal with it

Missing data as the name speaks for itself, means —

Either feature values or target values are missing from the dataset at hand

Classification of Missing Data based on the reason for missing

Missing data is basically classified into 3 —

1. Missing At Random (MAR) — because the reason why a random value is missing from a feature (predictor) or target can be understood from the exploratory data analysis itself. (Read about this in-depth here)
2. Missing Completely At Random (MCAR) — there is nothing fishy, the missing is absolutely random! (Read about this in-depth here)
3. Missing Not At Random (MNAR) — the most unfortunate situation for a machine learning enthusiast or student because the reason for missing here can be explained but not with the help of variables you have in your dataset. This situation demands a thorough investigation into the initial phase of the data analytics — data collection! (Read about this in-depth here)

Classification of Missing Data based on variables

Variables are basically two types, in machine learning terminology —

1. Features/Predictors and
2. Target

Thus, Missing Data can be found in both of these variables and hence can be classified as follows:

a. Missing data in feature variables or feature-missing

b. Missing data in target variable or target-missing — this kind of missing is also called class-missing when the target variable is discrete in nature, just like in a binary or multi-class classification problem. In the case of continuous target variable, the term target-missing can be used as default.

Consequences of Missing data and How to Deal with it

Missing Data puts forward 2 main issues —

1. Loss of information — this issue is associated with feature-missing and the best way to deal with it is to use imputation techniques. Some of the imputation techniques are: Mean, Median, Mode or constant value imputation.
2. Imbalanced dataset — this issue is only associated with class-missing and not with target-missing case. This is one of the fundamental problems in most of the classification problems. The solution to this problem is to use techniques like — Over-sampling or Under-sampling techniques. If there is target-missing issue, my recommendation would be to drop the datapoint corresponding to each target-missing case.

Drop or Impute?

The following strategies are my own and should be only given importance as such:

If missing values > 5% for feature-missing case — Drop the feature itself, else perform Imputation on the feature or features.

5% or 10% or any upper-limit — it’s up to you to decide.

The basic idea is too much imputation is alteration of the behaviour of the underlying distribution

Over-sampling or Under-sampling?

Before explaining over-sampling and under-sampling, one must know about majority class and minority class.

Majority Class and Minority Class

This terminology is strictly associated with classification problems as mentioned earlier. The majority class is the discrete variable value that dominates the target (or class) variable distribution and the class that has the least share is the Minority class. For example, 70 male participants in a survey of 100 random persons represent the Majority class and the 30 female participants represent the Minority class.

Over-sampling artificially introduces new or cloned datapoints corresponding to the minority class into the dataset and increases the dimension of your dataset, while converting the dataset from imbalanced to perfectly balanced one. This means, over-sampling is aimed at minority class.

Under-sampling is the opposite of over-sampling — it eliminates some of the majority class datapoints to ensure the dataset is perfectly balanced.

Some Over-sampling techniques —

1. Bootstrapping or resampling (sampling with replacement) — includes over and under-sampling
2. Synthetic Minority Over-sampling TEchnique (SMOTE) and
3. ADAptive SYNthetic sampling approach (ADASYN)

Some Under-sampling techniques —

1. Tomek-Links undersampling
2. Cluster Centroids

Hybrid technique — Oversampling + Undersampling

1. SMOTE followed by Tomek-Links

See a simple Kaggle notebook on Resampling technique for Imbalanced datasets — here

All the over-sampling and under-sampling technique APIs are available in sci-kit learn’s ‘Imbalanced learn’

References

1. Medium — https://medium.com/@danberdov/types-of-missing-data-902120fa4248
2. Medium — https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167
3. Kaggle — https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets
4. Sci-kit learn @Github — https://github.com/scikit-learn-contrib/imbalanced-learn

The Story of Missing Data was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Beautiful takes time
Patience is to sacrifice
Hope fuels to strive

Transform your perception with these 3 real life correlations.

What is correlation?

Correlation is a measure of association between two variables. You can find positive, negative and no correlations in the universe.

Correlation is something that data scientists deal with, in their day-to-day lives. Let’s dive in!

Correlation 1: Time spent in meetings vs your value at the workplace

If you spend the majority of your work time in meetings, then most probably you are very much valuable to your company. This may not be the case every time, but I’m speaking in a general sense. Meetings are meant to bring up ideas, strategies and decisions that will influence the future of your firm. So this is no place to have fun. So, if you are enlisted in almost all the meetings of your firm, you can be certain that your company trusts your communication skills and creative inputs — you are really valuable to your company.

Thus, we can say there is a positive correlation between the amount of time you spent in meetings and your value at the company.

Implications

You can analyze how valuable you are compared to another person in a different firm holding the same designation as you. You can analyze on a weekly basis, whether your meeting duration is rising or falling or remaining nearly the same — which means, is the company’s perception about you changing in any way; positive or negative? Also, you can also work out the opposite manner, if you know your value at your company, you can anticipate your meeting duration in the upcoming week or month. To go even further, you may even be able to predict the designation (value) of someone in your firm or a different one, given the time they spend in meetings.

Correlation 2: Boiling point of Water vs Time elapsed

Water is a chemical and boiling point is one of its property telling us at what temperature it starts to boil. There’s something known as colligative property — property of a solvent will be influenced by solute contained in it. The pure water (not talking about distilled one but the one with natural minerals in it) will possess a particular boiling point. While, as the water starts getting less and less pure i.e more impurities starts to populate the drinking water, the boiling point of water rises — a phenomenon known as Elevation of Boiling Point, one of the colligative property. If the boiling point of water from a river near your home is higher than what you measured yesterday, then you can start suspecting the river water quality.

Since drinking water should have a consistent quality, its chemical properties (boiling point, in this case) should show a constant boiling point or there should be no correlation of Boiling point of water with time.

Implications

A positive correlation between boiling point and time elapsed will be quite alarming as it is an indication of inclusion of impurities — at this point, you must investigate the reason. Maybe a local factory is dumping their untreated industrial waste into the river. You can find a locality or country that cares the most with their rivers or water bodies by comparing the correlation values between each country — the one with least correlation wins, as they put in place great management efforts to preserve the water quality for a long time.

Correlation 3: Wisdom vs Overestimation

Finally, we all have heard the adage “With Age Comes Wisdom” and there’s a study led by Frank Durgin, demonstrating that fact [1]. In that study, the participants were asked to estimate the slant of a hill and the study showed that older participants gave much better estimates than the younger ones — even though some of the older ones didn’t have any specific knowledge about hills. Thus, older ones are much better estimators than younger ones, as the study proved and they attributed it to experiential knowledge (or wisdom) of adults compared to the younger ones. Also, it should be noted that almost all the younger participants and some of the older ones (outliers) gave overestimates.

Therefore, we can say that wisdom is negatively correlated with overestimation i.e the more wisdom you have the less likely you are to make a wrong decision or estimation.

Implications

Negative correlation between wisdom level and degree of overestimation proves you will be more accurate in your decision when you are very old. But here’s the good news, depending on the accuracy you want (or the corresponding degree of overestimation), you can know the corresponding level of wisdom you need to acquire from the regression line. If wisdom is an attribute of the aged ones alone, then we would not see any young influential leaders across the globe. This indicates that young people too can gain the level of wisdom, the same as that acquired by the adults. What does this mean? To youngsters, this means you will make less wrong decisions than your peers and you will be successful within short time-frame relative to your peers because you are very good at judging things due to the high level of wisdom you acquired recently. Note that the time-frame reduction does not apply to the effort that goes in — in fact, it will be something humongous as your goal is to achieve the extra-ordinary, to be honest, so determination and passion should be your best friends. Remember that adage, “No pain, No gain”?

So, look at the regression line and find the level of wisdom you need to gain and start gaining it. But how?

Read as much as you can, Explore different domains of knowledge, Interpret your experiences, Discuss your ideas and Debate your views.

Basically, what you have witnessed is the power of a simple concept in Statistics. I will be soon exploring even more powerful ones in the most simple way possible. Stay Tuned !

P.S: Share this article if you found this helpful and let those in need be enlightened as well.

References

[1] Li, Z. and Durgin, F. H., ‘A Comparison of Two Theories of Perceived Distance on the Ground Plane: The Angular Expansion Hypothesis and the Intrinsic Bias Hypothesis’ (2012), i-Perception, pp. 368–383

3 Beautiful Real Life Correlations was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

An easy way to learn the terms in Machine Learning from a Philosophical point of view.

Through the Philosopher’s specs

Before diving into philosophy of Machine Learning, let’s bring in more clarity to some terms. Datapoints are the pieces of a whole — the ultimate essence from which all of it originated, machine learning experts usually call it, insight. More technically, the underlying distribution that creates these pieces of information. So every datapoint counts. As you are on a journey of learning you pick each of these stones (datapoint) along your way and get some understanding from it — you learn something from it. As you progress, you will see and pick many stones along the way and continue accumulating knowledge about it.

After you have traveled so far, you would have considerable amount of understanding accumulated about all those stones (or datapoints). Remember that speech from Steve Jobs at Stanford (click here to visit) —

“You cannot connect the dots by looking forward but only by looking backward” — Steve Jobs

Connecting the dots or the datapoints

Now what you do is follow Steve Jobs, connect those dots (your understandings) i.e you combine all the things you learned during your journey till now. What you are about to experience is something else.

When you analyze the information, you may find that all those stones have something in common — similar chemical composition, similar shape etc. And you may find that all these stones had a similar damage causing you to infer that those stones fell down at the same velocity and same height, which you may attribute to a truck that carried them all to its destination — do you feel like Sherlock Holmes now ? because anyone who analyses data is one. There are so many possibilities or so many explanations (models) as to why all these stones have something in common.

The idea is to find the best explanation (model) that is least surprising (Surprise Principle by Charles Sanders Pierce)

Fitting and Generalization

Connecting the story behind each stone (the understanding you gained from them) to generate a grand story or theory is called fitting a model in technical terms.

So, for instance, you coming up with a theory or a reason behind why your air conditioner is not cooling the room as usual, is actually you fitting a model behind the under-performance of your a/c. And this fitting happens as you try to fit all those evidences you got from inspecting the a/c, into a convincing theory, just like a jig-saw puzzle. In layman’s terms, you learned the reason behind the a/c breakdown and the interesting part is, there can be more than one explanation!

Mathematically, learning is fitting a line on your datapoints or evidences. And there can be more than one line possible!

The fitted line or model need not touch all the datapoints perfectly, because that would be overfitting. This means, there will be always some information left out from each datapoint and only such a model will have better generalization performance. So what is generalization performance?

Let’s play Sherlock again. Imagine you cracked a murder case and the culprit was the one who smoked an expensive limited edition cigarette. If you are assigned a second murder case and there are two suspects who smokes cigarettes — one smokes a regular and the other an expensive limited edition, you cannot conclude that the actual criminal is the latter even though all the other circumstances of the crime are similar to the first one you cracked. This is the moment where a data scientist would say your model or theory has poor generalization performance. That is, the model you fitted or learned from the first case cannot be applied to the second case. Firstly, the model for the first case is very much adapted to the first case (overfitted) i.e you have gone deeper into the details of each evidence so much so that the probability of another case having identical set of details is so small or is negligible. If you were to make a general enough model in the first case, you can also apply it to the second case and thus to great extend gives you convincing predictions about who should be the prime suspects.

What you need is a crime pattern theory that successfully explains the first crime and can also explain any future crimes to great extent. Such a theory or model is very general in nature and relatively has less overfitting than an overfitted model.

This philosophy works great if you apply this for your own learning. What you learn is that line.

The line is the physical representation of your knowledge.

This line is telling us that in order to get better understanding than bookworms or overfitters, one must only get the core ideas from each datapoint. You fit a line for physics when you learn physics chapters, so does for chemistry and biology. Thus, we can call them physics line, chemistry line and biology line.

Remember, each datapoint is a stepping stone for your line and every stepping stone is just part of the story and NOT the whole story.

If you focus on just few datapoints and fit a line, chances are, you will be highly biased as it would only contain part of the reality or knowledge that you are seeking. This is called underfitting.

In the murder investigation scene underfitting would mean, you collecting just one or two evidences and reporting the murder to be part of mob-lynching which was actually a one-man crime.

Note: The process of making a line less overfitting than before is known as regularization, in technical terms — in layman’s terms it is a process of making the line or model more general and hence can be called generalization.

Final Note

The ultimate information or theory you seek is spread across many datapoints and you must visit sufficient number of these to get a reasonably accurate understanding. For example, Andrew N G’s deep learning course is just one datapoint in the deep learning domain space. There is an actual truth or the complete concepts of deep learning out there, which is spread across all the different deep learning courses available, online and offline. If you spend too much time and effort into Andrew’s course, you will never reach the ultimate truth and you will be at a disadvantage. It’s necessary that you get the main concept out of Andrew’s course. There will be other teachers, with a different viewpoint and the part of story or dimension which Andrew oversees.

Seeing through one pair of eyes makes you blind to the actual picture, instead you must see it through many pairs of eyes.

P.S : Share this article if you found this informative so that people in need may find it.

To be honest, when I first entered into the world of AI and Data Science I saw myself using all those cool machine learning algorithms and creating solutions which had an impact on lives across the globe.

My thoughts were like, “ After I master all these algorithms, I’m gonna be ready to rock”. But as you might have guessed, reality hit me.

Trending AI Articles:

1. How to train a neural network to code by itself ?

2. Are you using the term ‘AI’ incorrectly?

3. Making a Simple Neural Network

4. Basics of Neural Network

It so happened that all my confidence stemmed from all those perfectly cleaned and balanced datasets, if you know what I mean, crying to be fed into one of those magical algorithms to produce mind-boggling results and filling me with pseudo-confidence.

The Realization

Reality knocked on my front door when I was assigned to an analytics project aimed at predicting the probability of success of a political candidate. It was at that moment I realized, I was hollow as a bubble — I had so many algorithms in my brain but nothing was helping — what’s the use of a gun without a bullet? I needed data to use any of the algorithms I knew. If I had any form of data, I could have done something but there was absolutely nothing!

I had two things at hand; collect data and do the prediction.

But What Data and How..?

Until I thought about it, collecting data felt easy. But again, yes you guessed it right, reality came after me — this time in the form of questions. What data should I collect? Which all questions or predictors will help me better predict the candidate’s success probability? How to define probability? What factors influences its value and the influence of social psychology? Things started going haywire. There were more questions being generated rather than answers.

I spend hours reading about my regional politics, journals of psychology and neuroscience —all these, not to generate survey questionnaire but decide what to include and what not to. One interesting thing I found is that humans retain the memory of negative action more than that of a positive action (Elizabeth A (2007))— something that politicians love to babble about and sometimes helps them win the elections. But we are not talking politics here…

The next thing was, how to ask the questions to the respondents — if they know we collect data for a specific party, chances are they will modify their response and we will lose some highly valuable predictors. We all know, prediction algorithms’ performance and reliability are not the sole responsibility of tuning process and that data quality has a very good part in it. So, the “how” part started creating problems after “what”.

People are just one side of the story

Soon, political publications, some news articles and magazines led me to the role of the opponent in the success of a candidate. This lead me to another realization — I have to do all those research and questionnaire preparation I did for people or voters for political opponents too. This means, there exist “what”s and “how”s for the opponents as well. All these lead me to create an idea for data collection framework — so that I don’t have to scratch my head again for any upcoming projects that could gift me with late night bedtimes if I don’t create one.

The Framework

Finally, the reason behind this post. Before going any further we need to make some things clear. From here on, I will be using three terms:

1. Organization — the one for which we are doing the analytics project
2. Customer— the ones who are at the receiving end, the users of solutions the organization creates
3. Enabler — data scientists working on the problem, acts as a bridge between organization and customer to deliver organization’s products and services

For any analytics project, there are two perspectives or elements — organizational and customer perspective.

Organizational Perspective

This perspective will help us to identify and define the “what”s and “how”s of the data collection associated with the organization. The first thing will be to identify the domain of the organization, which will filter out so many “what”s. The next thing will be to, understand the behavior of the organization — service industry or manufacturing? This will help you narrow down the variables. For example, time series of employee turnover, time series of scrap volume per day, time series of machine usage hours per day etc in a manufacturing sector will give you an idea about the behavior of that organization. All these behavioral data will guide you to the “what”s and “how”s to great extent and with the help of some research you will do just fine.

Thus, using organizational perspective is like the organization dictating you what it needs and will basically help shape your goal — but this is kind of narrow. So we move to the second perspective.

Customer Perspective

This is exactly like the Organizational Perspective, but with customers. Here we deal with the customer behavioral data — what is the gender?Age group?Number of previous associations with opponent firms? Financial soundness? Love to experiment with latest products?

For instance, number of companies for which the employee has worked for, number of disciplinary actions, quality of units machined belongs to the customer behavioral data of a manufacturing industry worker. Please don’t confuse the term customer with the usual term. By customer I mean the one who receives and uses the solution (advanced manufacturing units for machining intricate parts in an industrial setting) provided by the organization for overcoming a manufacturing challenge. Customer can be immediate or an end user.

Definition of customer varies with your objective.

Therefore, customer shapes the right solution or help you find the “what”s and “how”s. Again, this perspective is narrow too, so we need the best of both of above.

Hybrid Perspective — Best of Both

The best approach according to me, as this will help you see things in all the dimensions — you get to see the big picture. The choice of this perspective depends on the difficulty of acquiring organizational or customer perspective. If any of the perspective is difficult to attain, it’s always better to start with the customer perspective. According to me, customers can say so many things about the organization than the organization itself.

According to me, customers can say so many things about the organization than the organization itself.

Coming back to Political Success Prediction

As per the framework, the political environment in which the candidate of my interest competes, will be the organization and it can be basically classified into a service industry as politics help govern our state or country. The public image of the political party, the track record of the candidate of interest and that of the opponent or opponents will be some of the organization’s behavioral data and contributes to the organizational perspective. While, age group, unemployment rates, religious share in the population etc will be part of the customer’s behavioral data and contributes to the customer perspective. With these two combined, you have a hybrid perspective at hand and not only that you are much more confident than before that you have not missed anything knowingly.

The ultimate takeaway is that — there are only two sides in any analytics project; Organization and Customer and this Framework is general enough to apply to almost all the analytics projects.

Don’t forget to give us your 👏 !

A Simple Framework for data collection for any Analytics Project was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.