Success is usually understood as achieving something important. It is seen as an arrival, becoming successful, reaching your potential, making something of yourself, living a certain lifestyle, or not living a certain lifestyle. Implicit in all of this is the idea that success will somehow fulfill us.

But why do we seek success in the first place?

It seems to come from a feeling that something is missing. A sense that who we are right now is not quite enough. If we did feel like everything was good, we wouldn’t even define something called success and then spend our life chasing it.

This is very different from animals. Animals are driven by an instinct to survive, reproduce and, where possible, dominate. We sometimes describe an animal or a species as “successful”, but that is our label, not theirs. Nature itself has no concept of success because in the animal and plant kingdom, the “success” of one being is the death of another.

In the human world, many common defintions of success might not be too different from this. But it also has a relative and comparative basis.

For example, the Olympic 100m champion from 1900 would not even qualify today. The head of a village is respected, but is usually not considered as successful as the president of a country. Someone earning ₹10 lakh a year may feel successful until they meet someone earning ten times as much.

Our idea of success is constantly shifting because it depends on comparison.

And that raises an interesting question.

Has there ever been a point in your own life where success was finally enough?

You work hard in school. Then college begins. You get through college, and work begins. Then perhaps comes the desire for a better role, more money, a family, a house, recognition, influence, contribution… the list keeps changing.

The destination keeps moving.

Which means we never really succeed at “succeeding”.

That suggests that perhaps what we are chasing is not success itself, but the hope that success will remove a deeper feeling of incompleteness.

But where does that incompleteness come from?

Most of it is borrowed.

Long before we started making our own decisions, we had already absorbed countless ideas about what a successful life should look like. Parents, teachers, friends, society, pop culture, movies, TV, social media, etc. all contribute to the picture. Over time, we begin to mistake that picture for our own definition of success.

And once our definition of success is borrowed, it can never really be stable. It changes with the people around us, the culture we live in, and even the era we are born into.

So perhaps asking, “What is success?” is the wrong question.

Maybe the better question is, “Can I understand this feeling of incompleteness that makes me seek success whenever it arises?”

Can I see where it comes from? Can I see that much of it is conditioned rather than intrinsic? Can I see that many of the standards I live by were simply inherited?

Every time we clearly see one of these borrowed demands, it loses a little of its grip. Not because we replace it with a better definition of success, but because we no longer have to obey it. And as a result, we no longer have to be driven by the perennial incompleteness that causes so much strife.

Maybe that is where a different understanding of success begins.

Not in arriving somewhere. And not in becoming somebody. And not in behaving a certain way.

But in seeing, moment by moment, the borrowed ideas that create this endless chase, and responding to life with a little more clarity and a little more freedom each time they nag at us to chase them.

A viral video from an Anganwadi Centre in Maharashtra showed a woman kicking a young child at the entrance of the centre. Moments later, another woman emerged from inside, lightly tapped the woman who had kicked the child, and cheerfully picked up the crying child. Social media quickly erupted in outrage against Anganwadi Workers and the Anganwadi system, as it was widely assumed that the woman who kicked the child was an Anganwadi Worker.

Soon afterwards, reports from the investigation revealed that the woman was not an Anganwadi Worker but the child’s mother. Immediately, the outrage shifted from the Anganwadi system to the mother, and more broadly to Indian parenting. A second video then surfaced, showing members of the local community confronting the mother, publicly slapping her in front of the child, and asking the child whether she was also beaten at home. Again, this triggered outrage and debate. Some applauded the public reprimand, while others argued that humiliating the mother in front of her child was inappropriate and damaging.

What is striking in this case is how quickly we assigned blame, rushed towards taking some punitive actions, and moved from one villain to another without asking the really important question:

What produces such violence in our society on a regular basis?

The reason this question matters is that our answers determine the kind of solutions we pursue. If we believe violence is simply the result of certain people making bad behavioural choices, then our instinct will be to identify the culprit, punish them, tell them what good behaviour looks like, and move on. If, however, we see violence as the outward expression of a deeper understanding, then punishment cannot be the whole answer — it may be a necessary step but not a sufficient one. Indeed, if our understanding of violence changes, even our idea of punishment may change.

Therefore, in order to really address the root cause, we must also ask what understanding made such behaviour seem acceptable in the first place — not only to the mother in the video, but also to the millions of people who engage in similar behaviour every day, to those who publicly humiliated her believing justice had been served, and even to those of us who believe that teaching “good parenting behaviours” is enough to solve the problem.

By “understanding”, I do not mean information or awareness of what is right and wrong. I mean the fundamental way in which we relate to another being — whether we see them as an independent life with intrinsic value, or primarily in terms of what they can do for us.

A kick captured on camera shocks us because it is immediate, concrete, and undeniable. However, the quieter forms of violence building up pressure invisbly rarely provoke the same reaction because there is no dramatic footage. A parent who constantly compares a child with others, pressures them to fulfil their onw unfulfilled ambitions, uses them to cope with loneliness or boredom, vents life’s frustrations on them, or even lets them indulge in destructive habits for their own convenience does not become the subject of viral outrage. Yet these forms of violence are not exceptions; they are woven into the fabric of ordinary family life and often shape a child’s life far more deeply.

This is why discussions about violence so often remain trapped at the level of behaviour. We ask how to stop the kick, how to punish the offender, or what new rule should be introduced. These questions are important, especially when immediate harm must be prevented. But they are not sufficient. Behaviour is only the visible expression of something deeper. Unless we understand what gives rise to it, violence simply changes its form.

To understand that, it helps to distinguish four ways in which we use children and more broadly, other human beings, animals, and the planet as a whole to satisfy our own psychological conditioning. The violence that arises is not usually due to the need for cruelty for its own sake. Rather, it arises when we stop relating to another being as an independent life with its own intrinsic value, and instead begin relating to them in terms of what they can do for us. The child becomes a means to satisfy some psychological need of the parent. The same pattern extends beyond parenting to spouses, colleagues, employees, domestic workers, animals, and nature as a whole. While the outward expressions vary, they often stem from one of four underlying patterns:

These categories are not always mutually exclusive and they don’t necessarily manifest in visible, extreme ways either. However, a favourable set of internal and external circumstances may lead to an explosion that is visibly seen. Seen through this lens, the kick in the viral video is no longer an isolated act. It becomes one visible expression of a deeper pattern that can manifest in many different forms.

This also explains why our responses to violence are so often limited to behaviour change rather than transforming understanding; because behaviour is visible and changing it through punishment, incentives, measurement, surveillance, etc. does produce visible changes. Bringing about deeper understanding of one’s own psychology is far more difficult because it develops through dialogue with, education of, and reflection by individuals as well as institutions that prioritize understanding over behaviour change.

A government can outlaw corporal punishment, a company can mandate codes of conduct, and a school can encourage certain parenting practices, but these cannot change the understanding that gives rise to violent behaviour in the first place.

The same pattern can be seen almost everywhere. Companies discover that compliance training alone does not create ethical cultures. Schools find that discipline alone does not cultivate curiosity or responsibility. Criminal justice systems increasingly recognise that reducing reoffending requires rehabilitation alongside punishment.

A parent who no longer hits a child because it is socially or legally unacceptable may instead resort to emotional blackmail, conditional love, or constant comparison. A manager who can no longer yell at employees might resort to quietly excluding them from growth opportunities or resort to indirect jibes. A society that condemns overt discrimination may continue to practice subtle forms of racism or casteism. The underlying understanding has remained unchanged. Only its expression has become more socially acceptable or less detectable.

Every attempt to change behaviour without changing understanding merely teaches violence to become more sophisticated and invisible.

Seen through this lens, the public reaction to the mother is quite telling. Stopping the abuse and violence was absolutely necessary. And holding the mother accountable was also necessary. But was the public slapping really an attempt to transform the understanding of the mother that gave rise to the violent behaviour? Or was its primary purpose to make an example of her so that others would think twice before behaving similarly? Did it simply communicate a behavioural lesson to the mother and everyone else: “Don’t do this where others can see you.”

There is an irony here. Our response to the violence appears to have relied on the same principle that produced it: the belief that behaviour changes through fear, force, humiliation, and incentives. We condemned one act of domination with another.

The intention, of course, was entirely different i.e. to protect the child and deter future abuse. But if the mother’s understanding remained unchanged, have we reduced violence or merely changed where and how it will be expressed?

This is not an argument against accountability. Children must be protected and consequences must be established and enforced. This is necessary; but it is not sufficient. If we truly want to stop the exploitation of powerless and voiceless beings, then our investment cannot stop at teaching better behaviour.

No society can create a rulebook for every situation a human being will encounter. Behavioural rules will always remain incomplete because life is too varied. Therefore, the root, which is understanding, must be transformed. We must work to cultivate a deeper understanding of what it means to bring another life into the world, what it means to relate to another human being, and to recognise that no child, no spouse, no employee, no animal, and no part of the planet exists merely to complete what we have failed to understand about ourselves.

The tragedy is not merely that one child was kicked in front of a camera. The deeper tragedy is that millions of children and countless other vulnerable beings experience subtler expressions of the same misunderstanding every day, and they go unnoticed because there is no video to provoke our outrage.

If we wish to see fewer such incidents in the future, we must become at least as interested in cultivating understanding as we are in policing behaviour.

Most of us think of Holi as a time to play with colors, feast, dance, and make merry. But the story behind it is especially meaningful for those of us who live or work with children.

Over the last six months, I have been working with an organization that aims to significantly improve the quality of early childhood education for underserved communities around the world — Rocket Learning. And during this time, I have found myself constantly reflecting on what education really means and how we should serve the children of the world as individuals, organizations, and societies.

The story of Holi unexpectedly threw light on this question through its deep, timeless symbolism.

The Story

The center of the story is a small child — Prahlad. His father, Hiranyakashipu, is a powerful king who wants complete control. He wants comfort, security, and unquestioned authority. More importantly, he wants everyone around him to worship him and accept what he says as truth. But his son, who is about four or five years of age, refuses to accept what he clearly sees as false. For this, he is pressured, threatened, and punished. Yet, he does not bend. Then, the king’s sister, Holika, plots to kill the child in a fire by sitting in it with him on her lap. She supposedly had the ability to be unharmed by fire. However, in a miracle, Holika is burned, while the child survives. The Holika bonfire that begins the Holi celebrations remembers this moment.

Symbolism

• The name of the king, ‘Hiranya-Kashipu’, means golden (हिरण्य) bed (कशिपु). The symbolism is of a desirous person who wants unlimited comfort and security. His insistence on being worshipped illustrates the desire for unlimited, and unquestioned authority.
• The child, Prahlad, is symbolic of that entity within us (since the child is the progeny of the same king) that sees the greed and falsehood in our lives, albeit silently.
• The plot to kill the child is symbolic of our common experience in which we try to silence our inner child that speaks the truth when it begins to stand in the way of our unending desire for comfort, wealth, power, prestige, and security. It is also important to see how the king is actually fearful of a little child that he deems it important to kill him. All the power and wealth in the world were, in fact, shaken by the presence of a small child standing with the truth.

From a psychological perspective:

• Hiranyakashipu represents the ego that wants comfort, power, and unquestioned authority.
• Holika represents the forces that protect that ego when it feels threatened.
• Prahlad represents the quiet voice within us that recognizes truth even when it is inconvenient.
• And the plot to kill Prahlad symbolizes the deep sense of danger, discomfort, uneasiness, and restlessness that arises for the ego with even the slightest suggestion of its falsehood.

Relevance To Education

Those who have interacted with children have probably seen how they “Say The Darndest Things”. They often see things directly and clearly. To us, however, their questions and statements often feel uncomfortable, but we find ourselves unable to find any fault with them. This directness comes to them because they are not as conditioned as adults. And our discomfort comes from the many layers of societal conditioning we have accumulated since childhood that prevent us from questioning many things.

This is why, often, adults silence such questions — not out of intentional malice, but because they themselves feel uncomfortable and unsure how to respond. And unfortunately, this is the beginning of the end of a child’s curiosity and the beginning of the same type of conditioning that begets the silencing of the next generation.

Reflection

The ending of the story can feel like a fairy tale. In real life, truth does not automatically win. Children’s curiosity and clarity can easily be silenced if we are not careful.

In the end, it comes down to a choice we make every moment we spend with children: whether we will preserve the clarity and innocence that come from their relative freedom from societal conditioning and fixed beliefs. And doing this is not effortless for adults. It requires us to notice our own conditioning, to sit with the discomfort that children’s questions can sometimes create, and to resist the impulse to silence them simply because their honesty unsettles us.

Perhaps the real reminder of the Holi story for those of us who work with children is that their curiosity, honesty, and courage are precious, and our role is to protect these qualities rather than gradually extinguish them through our unconscious behaviors. And the only way we can do this, as highly conditioned adults, is by observing our own reactions, looking through the discomfort we feel, and allowing children to hold up a mirror to our lives with their unsullied clarity, just as Prahlad did for his father.

And perhaps it is not just the physically present children in our lives who can show us the mirror — our inner child is always ready to wake up, wake us up, and play Holi with us…given a chance.

The tragic loss of 11 lives and serious injuries to dozens more during the Indian Premier League (IPL) victory celebration in Bangalore has cast a shadow over what should have been a moment of joy. As investigations unfold and the details come to light, it’s a time not only to seek accountability, but also to pause and reflect, especially for us — the fans.

To understand the deeper implications of what happened, we must look beyond the event itself and ask: What has the IPL really become?

What Is the IPL, Really?

• The Indian Premier League is a global marketing juggernaut. It generates thousands of crores in revenue annually through ticket sales, advertising, and broadcasting rights.
• With smaller outfields and modified rules designed to favor high-scoring, fast-paced matches, every detail is engineered to maximize entertainment. It’s a spectacle crafted to keep viewers hooked — in stadiums, on screens, and across social media.
• But behind the glamour lies a business model that thrives on more than just cricket. A large share of advertisements blatantly promote addictive and harmful products such as gambling apps, tobacco, and alcohol. The franchise teams, though named after Indian cities, are private enterprises whose primary goal is profit.

Questions Worth Asking

Given this context, it’s worth asking ourselves — not to assign blame, but to seek awareness — what role we, the fans, play in shaping this system.

• Why are we so captivated by glitz, glamour, and spectacle over real sporting achievement — the kind that requires incredible discipline and sacrifice? For instance, India recently finished second at the Asian Athletics Championship in South Korea. One athlete even won a gold medal while competing in torn shoes. How many of us even knew that?

• Are we aware of what we’re endorsing when we stream matches, attend games, or promote IPL content? Many of the ads target vulnerable groups — particularly young people — with harmful and addictive products. Why does this not spark deeper concern or even protest?
• Do we spend more time watching cricket than playing it ourselves? Are we inspired by the athletes’ discipline and work ethic — or are we simply spectators of their lifestyle, endorsements, and celebrity?
• How much mental space does cricket occupy in our daily lives? Between live matches, social media reels, memes, and post-match analysis, do we ever stop to ask: what are we escaping from? Is it just about the sport — or is it a deeper craving for distraction, identity, and emotional stimulation?

A Call to Awareness

Cricket, like any sport, is an admirable pursuit. It can inspire, unite, and elevate. Certain forms of the game really test physical and mental limits of human performance.

But when it becomes a vehicle for distraction, manipulation, and unchecked commercialization — especially when public safety is compromised — it’s time to reassess our engagement. We don’t need to stop loving the game. But we do need to become more thoughtful fans.

Even passive support — simply watching, sharing, or cheering without awareness and understanding helps sustain a system that will not serve the public good. The IPL can thrive; but it can also evolve to have greater balance, ethics, and accountability if we, the fans, begin to ask for what is right for us.

Let’s engage with sport not just as spectators, but as conscious participants of society that determines the quality of our everyday lives.

The nature of work is changing — not just in form, but in what it demands of us.

Traditional jobs that offered stability in exchange for following instructions are slowly disappearing. In their place, a new kind of work is emerging — one that demands agency and adaptability.

Why This Shift?

Artificial Intelligence is becoming a new layer of abstraction over the world’s production systems. Knowledge worker tasks that once required years of training in a very specific technical skill can now be completed by non-specialists using AI tools.

This means more people can build and launch things. But it also means the mere possession of a skill is no longer enough; outcomes become the primary driver for everyone. Compensation will increasingly depend on the results one can deliver — not just on showing up or completing a task.

Just like salespeople today are paid for closing deals, future workers, especially in knowledge-based fields, will be paid for creating value that actually sells in the real world.

The New Baseline Skillset

To survive and thrive in this environment, individuals will need to:

• Reading reality accurately — Understanding customer and market needs, technology trends, global and local developments to develop insights.
• Creating relevant solutions — Turning insights into valuable offerings that are easy to use and deliver value to the end customer.
• Coordinating people and machines — Efficiently combining human effort and automated systems (software and robots).
• Communicating effectively — Persuading users, partners, and audiences to engage.

These aren’t nice-to-have skills anymore. They are becoming the minimum requirements for staying relevant in the economy.

Micro-entrepreneurship As The Default

The world is moving toward a model where everyone functions like a small business — accountable for what they produce and deliver.

This has always been the reality for farmers, vendors, and informal workers. Now it’s becoming true for software engineers, designers, and analysts as well.

The line between “stable jobs” and “entrepreneurship” is going to dissolve.
Overpaid, low-risk roles will likely become rare. And everyone will increasingly get paid for what works in the real work, not the work that they do.

Adaptability In The Face Of Uncertainty

This shift won’t be easy because the pace of change with AI seems to be orders of magnitude faster than other major shifts in the world’s economy that we have experienced in the past.

We can probably expect the following effects:

• Many people won’t be prepared — mentally, emotionally, or structurally — to take on that level of ownership. This is because their entire lives have been designed and conducted around the assumption of a stable job market. Their education, skills, investments, and mindsets won’t be aligned with this new reality. Adjusting will be difficult for most. And many won’t be able to adjust at all.
• This will cause increased economic stress and all the problems that come with it. Governments will have to come up with solutions to ease these stresses. However, given the novelty of the situation, we will likely see several failed models before we come to solutions that work.
• And yet, the change may also level old hierarchies.
  The distinction between “skilled” and “unskilled” may fade.
  Everyone — whether they write code or sell vegetables — will operate under the same pressure: produce something valuable, or go hungry.
• In this world, even traditional forms of work — farming, craftsmanship, local services — may feel less risky than volatile digital ventures. We may see people return to them, not out of nostalgia, but out of necessity.

The future of work won’t just test our skills. It will test our ability to see clearly, take responsibility, communicate, collaborate, and coordinate effectively, and adapt continuously.

The collective choices made by humanity are undeniably responsible for the multiple global crises that we face today. Whether it is climate change, mass extinction and biodiversity loss, desertification, the ever-looming threat of nuclear war, or even the obesity / diabetes epidemic — they have all been caused by the conscious and unconscious choices that humanity is making everyday. Many choices that are made at an individual level have big consequences collectively across the population. This is because the economic system and the leadership of democratic nations are both driven by the choice of the majority.

Moreover, there are many forces that influence individual choice that include personal and societal factors that are often outside the locus of an individual’s control. An individual’s beliefs and values are imparted by society’s various actors such as parents, teachers, colleagues, friends, books, social media, governments, etc. And in addition, there are a large set of base instincts that affect decision making through the creation of emotional reactions.

Given this, how do we ensure that humanity is making the best collective choices? The only way is to have enough educated people that are capable of making the right choices without falling prey to the various influences of one’s biology and society.

But what exactly is education?

Education is not merely something “useful”, it is an existential human need.
Education is often reduced to a means of economic productivity or societal stability, focusing on skills for earning a living or maintaining order. However, true education goes beyond this. It helps us understand who we are and our relationship with the world. This is a transformative process that unlocks the unique human capacities of empathy, compassion, kindness, meaning, and purpose.

Education enables true agency.
While animals can be trained to perform tasks or conditioned to respond to stimuli, they cannot be educated in the true sense of the word. Learned behaviors, including patterns of thinking, acquiring information, or practicing skills can only be classified as training, not education. Humans, on the other hand, have the ability to gain understanding through critical thinking, reflection, and imagination. And this could enable us to transcend any sort of conditioning and act with true awareness, understanding, and agency.

And in this way, education is the path to evolution of the human being — from conditioned slavery to freedom of choice.

And this freedom of choice is what will enable us to make choices that are not influenced by the same forces that are responsible for creating and perpetuating the major global crises of today — both external and internal forces.

Some questions to ask ourselves:

1. Am I really educated? Do I have true agency or am I still driven by base instincts and societal echo-chambers?
2. Is there evidence of true education in my life where I have risen above my conditioning to make choices for a higher purpose?
3. If I understand that education is the main crisis of our times, what is stopping me from trying to give access to true education to all human beings?

“Embarrassingly Good”

Large-language models (LLMs) have recently unleashed a rollercoaster of thoughts and emotions for technology enthusiasts like me — curiosity, pleasant surprise, reserved appreciation, jaw-dropping astonishment, utter disbelief, existential fear, resignation to humanity’s inevitable self-destruction, and now, I’m writing this article to feel like I’m in control by taking some or any action at all.

As many thousands of people have already recognized, LLMs with intuitive user-interfaces like ChatGPT, Bard, Dall-E, Stable Diffusion, etc. represent a leap in technology that seems more significant than anything else from the last decade. [1][2][3]

It is a technology that lets people use cutting-edge AI with nothing more than a web browser. No computing, programming, or language skills are required. It is not limited to any particular domain. It’s trained on the entire corpus of human knowledge. It produces terrifyingly accurate output. It can adapt to feedback. And many tools are practically free to use!

Is it perfect? No. But it is often pretty darn close and it is improving quickly. And so, it seems inevitable that it is going to disrupt any industry where value is generated through content generation (language, code, pictures, videos, etc.). This, so far, has been the domain of the “knowledge worker”. And LLMs seem to be embarrassingly good at this work.

The Future of Knowledge Workers

There are 1 billion knowledge workers today who make decisions based on information that they receive. They use their accumulated knowledge and experience to make these decisions. However, most of these workers are not making highly consequential decisions. Many are simply transforming information from one form into another. A significant portion of the work performed by most customer service executives, administrators, analysts, programmers, writers, illustrators, accountants, and lawyers can be categorized as information transformation. And all these jobs can be disrupted by current and future LLMs. In fact, it is already happening.

So, what does an average knowledge worker do in the future? Well, it is hard to say. We do not know for certain how many jobs will be replaced or how many new jobs will be created or how existing jobs will evolve into something else. Many scenarios could play out and it would be foolhardy to guess right away.

However, one clear thing is that if LLMs can do something at a level that is close to an average human, the LLM will replace the human. Or at least it will replace most humans engaged in that activity. The only ones left in such a job would be highly productive expert users of LLMs that can do the work of a multiple people from today.

The Human Element

Therefore, the skills that a human brings to the knowledge worker economy must be something beyond the capabilities of LLMs. However, it is difficult to say what kind of technical skills will fall outside the range of LLMs, given that all knowledge-based work uses language to produce and provide value.

And so, in my estimation, a safe bet is that the knowledge worker economy will value non-technical skills more than it does today. It may no longer be sufficient to possess technical skills to hold a job since LLMs will democratise a large fraction of technical ability.

Therefore, employers will look for people who excel at interpersonal relationships — something LLMs cannot develop (at least for now). Of course, customer-facing roles will continue to require interpersonal skills and the “human element”. But now, even traditionally insular technical teams will look for these skills.

This is because the teams with the highest motivation and inspiration will be the ones to come up with novel insights that can provide a competitive edge. And teams that collaborate the best will be the ones to survive in a highly competitive landscape.

Therefore, teams will look to hire great team players who can create a harmonious atmosphere. They will prefer people who can perform themselves but also inspire others to reach their potential. They will value the ability to communicate with different people across the organization and outside of it. They will also hire people who bring fresh perspectives and ideas from different domains. But most importantly, they will choose people who can empathise with a wide range of people in order to create products and experiences that are successful.

And this “human element” will be more important than it is today.

So, as I ponder my relevance in the future as a knowledge worker, I’m betting on becoming consistently curious, enthusiastic, energetic, personable, and most importantly, empathetic — something that I hope machines take at least a few more decades to master.

It’s one of the hottest jobs right now. And it comes with a cool-sounding title — Data Scientist! Sure, there is data involved, but is it really science? That’s a topic for another time.

However, we know that generally, the role of a data scientist is to use data to draw insights and build algorithms that can be applied to optimize a system. So what makes a good data scientist? And what about a great data scientist?

Good data scientists can do many things well. At a high level, they have knowledge in the fields of statistics, machine learning, optimization, and programming. Furthermore, they are also able to translate a business problem into one or more sub-problems that can be partially or fully solved by analyzing data. Of course, many other disciplines are useful to a data scientist, but that list is long. It includes many sub-fields of artificial intelligence, data engineering, mathematics, and software engineering.

However, excellence in these skills and fields still doesn’t make a GREAT data scientist. It only improves the efficacy and efficiency of a good data scientist. Great data scientists have other qualities too. The most important of them, in my opinion, are domain expertise and empathic communication.

Domain Expertise

Domain expertise is a critical requirement for a great data scientist since their role always involves solving a problem in some domain that is not data science. Improving a system within a particular domain that has a vast body of knowledge is generally not as easy as applying a canned machine learning algorithm on some data. At a minimum, it requires consideration and understanding of the following questions:

1. What does the business do? Where does its competitive advantage lie?
2. What are the aspects of the system that are difficult to model with conventional techniques such as first-principles models, rules, or heuristics?
3. What are the current models that are in use? And how can a data-based model add value?
4. Is there enough data to actually solve this problem? If yes, why hasn’t it been solved before? Is the reason anything to do with the lack of advanced statistical or artificial intelligence algorithm capability? Or is it due to a cultural, political, or logistical challenge?
5. What other types of data should be collected to solve this problem and how much would that data collection cost?
6. Is there enough value that can be realized by setting up an advanced data system?
7. …

This list is not exhaustive, but hopefully, you can see that answering these questions involves a deep understanding of the domain as well as a deep understanding of data-related technologies.

And often, building this level of understanding takes time if someone is new to a particular domain. And traditional domain experts often aren’t experts at data-related technologies. Therefore, finding this quality in your next data scientist candidate can be difficult. Of course, the way to work around this is to have data scientists work very closely with domain experts and build their own expertise in the domain. But this is only successful if the data scientist has the second quality that makes them great — empathic communication.

Empathic communication

Like most fields that involve some level of mathematics, the intricacies and details of data science are often grizzly to the uninitiated. Besides, when there is a large amount of data, our brains simply cannot comprehend all the different interactions that are going on in the system. To add to the confusion, the high non-linearity of some machine learning algorithms adds to the black-box like treatment being doled out to data. And sometimes, this does not sit well with customers who want to feel confident about what they are buying. Even within an organization, there can be pockets of doubting stakeholders who do not want to take on the burden of maintaining something that they don’t fully understand.

Therefore, a great data scientist has to excel in being able to listen to their internal and external customers’ anxieties, anticipate their needs, and be able to speak their language. This requires a great deal of empathy throughout the data science project. Some of the things that a great data scientist would be able to do are:

1. Communicate how existing systems can be improved using data-based modelling — both from a business perspective and domain perspective.
2. Explain the working of their tools and algorithms at a high-level. But also be able to dig deep when required.
3. Explain the repercussions of bringing in a new system into the business and how it affects existing domain processes, IT systems, and cultural practices.
4. Set up tools and processes to help non-data-scientists maintain new data systems.

Summary

The wide array of fields and tools that data scientists have to master are non-negotiable. However, the really great data scientists are domain experts and can communicate with empathy throughout the data science process.

TL;DR
Pandas provides a lot of flexibility to parse timestamp formats. But the flexibility comes at a performance cost. It is still a great solution when dealing with timestamps with a standard format. However, techniques like memoization and mapping with a pre-built lookup table can be orders of magnitude faster than Pandas when the timestamp formats are non-standard. Skip to the summary section for recommendations on methods to use under different conditions.

Much of the data that we generate today is in the form of time-series data. And analysis of this data often relies on representing the timestamps of the data in a structure that is amenable to time-based slicing and dicing. In standard Python and popular data analysis libraries such as Numpy and Pandas, there are dedicated data types to store time-based information. However, incoming timestamps are often strings with different formats. And parsing these strings into time-based data types is a time-consuming and sometimes tedious process.

In standard Python, a common way of parsing timestamp strings that have a known format is the time module’s strptime method (similar interface to C’s strptime).

However, since most data scientists have to do much more with a dataset than parse timestamp strings, powerful libraries like Pandas have become very popular. And in Pandas, the most common way of parsing timestamp strings is the to_datetime method. This method provides a lot of flexibility and it can even infer formats automatically. Therefore, many people use it almost blindly.

In this article, we’ll examine the performance and applicability of different timestamp parsing methods on different types of datasets. We’ll see when to blindly use Pandas and when to use something else.

Common methods

In this analysis, we’re going to compare six common ways of parsing a collection of timestamp strings.

1. time.strptime

For timestamp strings with a known format, Python’s time module provides this method to convert a string to a Python Datetime object.

Example:

2. Pandas.to_datetime without inferring

This method within the Pandas library can convert a collection of timestamp strings even without a pre-known format.

Example:

Note how the list of timestamp strings ts_str_list has timestamps in different formats. Pandas automatically infers out the format of each timestamp string before converting it.

3. Pandas.to_datetime with inferring

The same to_datetime method in Pandas has several optional arguments. One of these arguments is infer_datetime_format. By default, it is set to False. However, by setting it to True, the method infers the format of the first timestamp string in a collection, and then tries to use that format to parse the rest of the strings. If the inferred format doesn’t match any subsequent strings in the collection, the method falls back on the behaviour of infer_datetime_format = False.

The advantage of this method is that it saves a lot of time when parsing a collection of strings that have a consistent format.

Example:

4. Pandas.to_datetime with a specified format argument

Another argument accepted by the to_datetime method is format. Similar to time.strptime, this lets us explicitly define a format for parsing a collection of timestamp strings. As we will see later, the advantage of this method is that it is quite a bit faster than letting Pandas infer the datetime on its own. However, the pre-requisite is that the collection of timestamp strings has a consistent and pre-known format.

Example:

5. time.strptime with memoization

Memoization is a technique to store results of operations such that no operation has to be repeated. Using memos for the time.strptime method can ensure that in datasets that have duplicate timestamps, we don’t waste any time parsing the same string more than once. Of course, in datasets without any duplicates, this method will not have a benefit over the plain time.strptime method.

Example:

6. Pre-built lookup mapping

Another method to parse a long list of timestamps of a pre-known format and pre-known time-range would be to create a mapping of strings to Datetime objects. Then, we can use Python’s map method to obtain a list of Datetime objects that correspond to each timestamp string.

Example:

Experiments with unique timestamps

1. List of timestamps with a standard format

ISO-8601 is a widely accepted international standard for time-related information exchange. In addition to timestamps that follow the ISO-8601 standard, a few others are also a “standard” format as far as Pandas is concerned. This means that there is some set of timestamp formats that Pandas can parse very efficiently. An exhaustive list of these is not available (as far as I know) but in general, timestamp formats with all parts of the date and ones that start with the year seem to fall under this category.

So now, let us see how these methods perform when given timestamps of a known standard format. The formats of the timestamps are consistent throughout each dataset. We test the performance with datasets of different sizes given to the applicable methods.

The results show that Pandas.to_datetime significantly outperforms time.strptime in this instance. The pre-built lookup method also marginally outperforms the time.strptime method. However, it is still well short of the performance that Pandas delivers.

2. List of timestamps with a non-standard format

Now, if we run the same tests with datasets that have a non-standard timestamp format (e.g. 13–11–2000 04:50:32), we see some differences.

We notice here that Pandas.to_datetime with a specified format performs the best and a plain time.strptime loop comes in second place. The pre-built lookup method spends too much time building the map and therefore, its performance suffers. Pandas.to_datetime without the infer option also takes a long time because of the repeated format-inference of each timestamp string.

We also see some curious behaviour with the results of Pandas.to_datetime with infer. We see that it performs exceptionally well until it hits a dataset size of close to 20000. And then it performs the same way as Pandas.to_datetime without infer. What is going on here?!

This behaviour happens to be a side-effect of the dataset used in these experiments but it illustrates an important point. The dataset used in these experiments is a list of timestamps that starts with 12:00AM on January 1, 2000 and progresses consistently with an interval of 1 second. The format of the timestamp used is dd-mm-yyyy hh:MM:ss. Therefore, when Pandas tries to infer the first timestamp in the list, there is ambiguity about the format. This is because the timestamp string 01–01–2000 00:01:00 could be either in the format dd-mm-yyyy hh:MM:ss or mm-dd-yyyy hh:MM:ss!

So, when we have a dataset that starts with an ambiguous timestamp but has an unambiguous timestamp towards the end of the list, Pandas may realize that its inference is incorrect when it reaches the end of the list. It would then fall back to the behaviour of inferring the datetime format for each timestamp string individually. This would cause the performance of the operation to be similar to the case when infer_datetime_format = False.

Experiments on datasets with duplicates

The datasets used thus far have had no duplicates in them. However, in the real world, we are often dealing with datasets that have repeated timestamps or multiple datasets from the same time period. In the industrial intelligence domain (in which I currently work), it is not uncommon to process scores of datasets together from the same time-range and therefore, there is a lot of duplicated timestamp strings between all of them.

In the following experiments, we’ll see how our choice of timestamp parsing may change based on how much duplication we have within our dataset.

For the following experiments, all datasets contained 1 million timestamp strings. During a test, different numbers of duplicates were infused into each dataset while keeping the dataset size fixed. The lowest number of duplicates infused was 0 (all unique), and the highest number of duplicates infused was 100 (each timestamp in the dataset had 99 other copies).

Experiment with timestamp strings of a standard format (and consistent throughout dataset)

We see here that Pandas.to_datetime is an easy choice when dealing with a standard format. And as expected, memoization and pre-built lookup mapping improves as the number of duplicates in a dataset increases.

Experiment with timestamp strings of a non-standard format (and consistent throughout dataset)

But when the format of the timestamps is not standard and there are some duplicates in the dataset, memoization and pre-built lookup mapping both perform significantly better. In fact, I recently used the pre-built lookup mapping method to parse a large collection of timestamp strings and it saved me over 8 hours!

Summary

For data without too many duplicates:

1. Timestamps with a consistent known format
   Use Pandas and specify the format.
2. Timestamps with a consistent but unknown format
   Use Pandas with infer_datetime_format = True.
3. Timestamps without a consistent format
   Use Pandas with infer_datetime_format=False.

For data with a lot of duplicates:

For data with duplicates, the format of the timestamps matter. Therefore, here is a handy table to help you choose.

Timestamp parsing for time-series data analysis with Pandas and Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Earth-shattering events like a pandemic are rare chances to become a better version of yourself. If nothing else, you can learn humility by observing how dependent you are on everything and everyone around you.

You are at the mercy of nature.

Nature is less predictable than you can imagine. And you have less control over nature than you think. A more deadly virus could have wiped you out without giving you a chance.

You are at the mercy of your society.

An incompetent government or economy could have destroyed your society. Imagine if you couldn’t get access to food, water, or electricity during the lockdowns.

You are at the mercy of your community.

An inhumane employer or landlord could have made your life living hell. Undutiful or uncooperative employees could have sunk your business. And if your community gets infected, you will also bear the brunt.

You are at the mercy of your family.

Locked indoors with people you can’t get along with is constant stress and strife. Not having people to talk to and interact for weeks is like solitary confinement. On the flip-side, being able to spend all your time with people you love is a luxury.

You are at the mercy of your body.

A weak immune system is a liability. Bravado and carelessness can kill you or your family. And even if it doesn’t, it may have long-term consequences for the rest of your life.

You are at the mercy of your habits.

You have more addictions and more bad habits than you realize. The pandemic probably brought some of these to light. All of them were ugly revelations, but now, you have the opportunity to get rid of these ticking time-bombs forever.

You are at the mercy of your mind.

A mind that can’t be controlled wreaks havoc on your well-being during turbulent times. Develop equanimity, and the ability to observe your mind from the outside, rather than dance to its whims.

You are at the mercy of your values.

If you value pleasant experiences over the ones that can help you grow, there is much suffering during rough times. Do you see the lockdowns as a curse or an opportunity? That should help you identify your values.