• How much is the project cost?
• When will the project get finished?

The response to the above question will be quick if we are very certain about the work items that need to be completed.

But what if there are uncertainties in the scope of the project?

Estimating a Job or Task becomes a nightmare if there are uncertainties in the deliverables work items. And Project Sponsors always required the estimate to be accurate, which is sometimes difficult with uncertainties.

Estimated time is important even if there are uncertainties in your project. Let’s take a look at a real-life situation.

We all have Google maps used for navigating or driving to a location, if you see Google shows you an estimated time based on the uncertainties like road conditions, traffics, etc. No matter what’s the uncertainty we always get a single number from Google as an estimate which most of the time is accurate.

How to estimate these uncertainties? PERT is the answer.

PERT is used both for estimating and also to define the schedule of the project. In this article, we will be discussing only how to use PERT in estimating tasks.

What is PERT?

Back in 1950, US Navy faced the issues while developing the nation’s first, submarine-launched, ballistic missile — Polaris. These issues were probable and improbable project outcomes for task outcomes.

The program needed a new method to create a schedule for the Polaris, which would create a risk-adjusted schedule that considers uncertain project outcomes for the given task duration.

The Project Evaluation and Review Technique is founded to overcome these issues.

The PERT method is still widely used for project estimation by Project Managers/BA.

Lets understand

PERT wants you to come up with three estimates, considering the uncertainties and risks associated with the task.

These estimates are:

• Optimistic (O) — The estimate assuming that all the resources are available and understood, for the task to be able to deliver.
• Pessimistic (P) — The estimate assuming the worst-case scenario where all the uncertainty is considered, for eg: resource unavailability.
• Most Likely (M) — This will be estimated based on the experience or on the fact that if there is any Risk encountered, the Project team can manage it as they would have handled a similar risk in the past.

After gathering the estimates, the expected completion time for a task can be calculated as below:

Understanding with Example

Suppose, you are going to get your Mobile phone fixed which has its screen broken and you tell your wife that you’ll be back in an hour. This announcement is purely based on assumptions, not on experience. Now Let’s put PERT into action and see how much you are confident about that promise you made to your wife.

Optimistic:

• Reach the service center, following Google Map, and register the token for service desk: 10 min.
• Wait for your Token no. to be called: 30 minutes. (Yes, truly optimistic)
• Show the broken screen to the service engineer, understand the process and leave the phone for repair: 10 minutes.
• That’s 50 minutes, which is well within the hour. (Leaving room to visit a flower shop to buy some flowers for your wife, necessary sometimes).

Total estimate: 60 minutes

Most Likely:

• Reach the service center by following the Google map, there was a Roadblock in between so google has shown a long route: 20 min.
• Register for the token: 5 min. (Kiosk system was slow responding slow due to erratic network)
• Wait for your Token no. to be called: 40 minutes. (Less optimistic)
• Show the broken screen to the service engineer, understand the process and leave the phone for repair: 15 minutes. (Assuming engineer is taking time to check the phone and other issues in the phone if any)

Total estimate: 80 minutes

Pessimistic:

• Reach the service center by following the Google map: 30 minutes. (Traffic situations in consideration, considering if there is a flat tire situation or any other non-probable events)
• Register for the token: 15 min (Long waiting line and Kiosk is responding slow as well for each token to be raised).
• Wait for your Token no. to be called: 60 mins. (Those crowd on the front…)
• Show the broken screen to the service engineer, understand the process and leave the phone for repair: 15 mins

Total estimate: 120 minutes

Let’s what PERT say by when you will be back,

That means it would take approximately 80 mins for you to reach home. Now the question is How much confidence are you that you will reach in 80 mins (Approx.). For this, we will be using Standard deviation.

Standard Deviation (SD)

The standard deviation formula is,

Calculating Standard Deviation for the above problem,

That means it would take 80±10 mins you to reach home.

Conclusion

We all know estimation sometime gives us nightmare, and if there are uncertainties, that worsen the situation. I recommend the above estimate technique should be known to every individual. I have been using PERT since the beginning of PM Life.

The most important thing about PERT is, it’s easy to learn and will help you to make good and trustworthy estimates.

Originally published at Knoldus Blogs.

Introduction was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

With this vlog the idea is to make a common denominator of understanding so that the team can be succesful together.

Happy viewing!

Want us to help with building a well architected cloud native product. We are waiting to say hello :)

Let’s Mindmap: Cloud Native was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. Determine the need — There are multiple, multiple benefits of moving to the cloud. However, “I would like to move because he moved” is not a valid reason. Crucial reasons could be the need for elastic infrastructure, faster provisioning, cost-saving, focus on the core business than IT, etc. Once you are convinced about the need to move to the cloud, prepare a worksheet to evaluate what benefits could you actually achieve after the move. If the results are positive, you are a case study for making a planned move.
2. Understand the cloud ecosystem — There are multiple cloud providers offering competing specifications. Understand the pros and cons of each cloud vendor for your business. Some providers would have extensive APIs for development, some would have a better storage infrastructure, then there are networking options, pricing structures and service-level agreements (SLAs). Decide your need on the basis of your core business. If your business demands that you have to store a lot of data and that data is not referenced very often then you might choose a provider with better availability of data space for the same dollar.
   You would also want to make sure your management and monitoring tools can access the cloud-based applications and that cloud resources, in turn, can access data center services that remain behind the firewall. This would ensure that your costs of moving to the cloud are low and that they do not eclipse the advantages.
3. Chose your migration partner — This is a critical piece and one of the very important practices. With the evolution of the cloud, everybody is offering their piece of advice. When you chose a migration partner be sure to understand their prior experience in the cloud, their thoughts on their blogs, their presence in discussion forums and conferences. Do a reference check with their existing clients. Understand the companies that they partner with and the solutions that they offer. You might have heard that the best investment advisors are those who also have their money invested the right way. Likewise, the best cloud partners are those who themselves make heavy use of the cloud. If there is a cloud provider who does not have a serverless office, I would be worried.
4. Pick a pilot project wisely — Just like any other pilot project, the first application that you intend to move to the cloud should not be overly simple, which gives skeptics a chance to say that it was too easy, neither it should be very complex such that instead of understanding the significance of moving it to the cloud, you are worried about making it work.
   The application should not have a millisecond response time expectation. If the response time is that critical then perhaps this application is not a good candidate.
5. Involve the stakeholders — One of the missing pieces in cloud adoption is that the relevant stakeholders are not involved and they turn out to be the biggest barriers at a later stage. It is always prudent to involve the CIOs CFOs to show them the advantages and the cost-saving. It is important for the network and security experts to validate the cloud and voice their concerns opinions in the beginning rather than later.
6. Create a working sandbox — When you decide to move the application to the cloud, be sure to have a demo sandbox exposed to all the people who were using the earlier application. This allows the users to become comfortable with using the application on the cloud and point out any potential glitches. This also helps them feel that their feedback is being solicited and is important. If the users feel that the application is as good as or better than what they have been using, you have buy-in.
7. Have a plan B — Stuff happens. Sometimes even with the best of intentions, we do not achieve what we set out to. It is always important to have a well-laid-out plan B. This might include a decision not to go to the cloud at least for now. Make sure that all your data is available and can be ported back to the existing systems.

Once you take your first steps towards the cloud with above mentioned best practices, there is little chance of surprise on the way. If you are ready to take the first step, contact us.

An assessment can help an organization create a cloud migration plan to ensure a more successful migration. Explore more here if you are ready for migration>>

Best Practices for Moving to the Cloud was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

You would always get a ton of literature on Event Sourcing and CQRS. The key question is WHEN do you use it? Under what circumstances? Is your problem really in need of ES?

I would not go into the details of what Event Sourcing and CQRS is. The industry stalwarts have covered that in adequate detail. This post delves into battle-tested scenarios on where we should have used and otherwise ignored ES.

Reasons to use Event Sourcing

There are always tons of reasons to use event sourcing. Not necessarily all of them hold water. Some of them include

1. It is a natural extension of DDD — Well yes, it does find relevance along with the DDD patterns but then we could very well use all the concepts of DDD without event sourcing.
2. Helps in separating out write and read models — Hmm, do that with plain CQRS instead.
3. Allows building asynchronous and eventually consistent communication models — Well, you could do that with Actors and all the goodness of Akka like framework.
4. It helps you answer WHY — This one I would take hands down. If you really need to know the WHY you got to the current state rather than just knowing WHAT the current state is then yes, you would need ES.
5. Helps maintain an audit log — You could still maintain an audit log the old-fashioned way without ES but yes, it does make it easy because you have all the events that led to the current state. These audit logs can be used for analytics as well. There would possibly not be a debug problem and possibly no information would be lost.
6. Performance — Because events are immutable, you can use an append-only operation when you save them. Events are also simple, standalone objects. Both these factors can lead to better performance and scalability for the system than approaches that use complex relational storage models.
7. Deriving additional business value from the event history. By storing events, you have the ability to determine the state of the system at any previous point in time by querying the events associated with a domain object up to that point in time. This enables you to answer historical questions from the business about the system.
8. Replayability — You might discover a coding error that results in the system calculating an incorrect value or you want to gather additional statistics from the events which came in 6 months back until now. Rather than performing a manual adjustment on a stored item of data, you can replay the event stream so that the system calculates the value correctly and populates those additional data points based on the new version of the code.

Reasons not to choose Event Sourcing

Now let us look at some of the reasons to avoid ES.

1. Performance — Although event sourcing typically improves the performance of updates, you may need to consider the time it takes to load the domain object state by querying the event store for all of the events that relate to the state of an aggregate.
2. Versioning — You may find it necessary to change the definition of a particular event type or aggregate at some point in the future. When the event changes or the command changes, so does your way of handling them. Then you would have to deal with multiple versions of the event. One which came in historically and one which is current.
3. The development costs are high — Frameworks in this area tend to be heavyweight, overly prescriptive, and inflexible in terms of tech stacks. There is quite a bit of plumbing needed. You have to deal with classes for commands, command handlers, command validators, events, aggregates, AND THEN your projections, those model classes, their access classes, custom materialization code, and so on. The entire process needs quite a bit of scaffolding.
4. The UI needs to work heavily with eventual consistency — This is something that bit us real bad in the real world. First, the UI needs to be aware that any command which is sent through would have its own eventual consistency cycle time and it should be prepared for that. It should not expect that the user registration would have actually synchronously registered the user once the command went through. So, the UI needs to play along. It is said that for such scenarios the system should follow the usual CRUD which brings us to then building 2 kinds of systems. One which would follow the usual CRUD and the other which would use ES.
5. Projections are expensive as well — That first extra projection you add doubles the amount of code that touches your event stream. You would be writing more than one projection. So now you have N things processing this event stream instead of 1 thing. There’s no more DRY from this point forward.
6. An audit log has too much noise — most of it ends up being pure noise that actually needs to be filtered out. Both by end-users and by consuming sub-systems. A lot of information captured by the audit log would not be used 99% of the time.

So how do you decide?

OK, I believe by the time you get to this part, I should have managed to confuse you already. The question that I ask myself and I ask our clients to answer to make a final call is the following

Are you interested in the WHY of the current state? Does the history of collective data records add any value to you?

They would not have a straight answer to this and hence a story works.

Suppose you’re operating an e-commerce system with a CRUD based order module. At some point, you may ask yourself the question. How often does it happen that customers put items in their shopping cart, but remove them again before checking out? In the CRUD system, you ordinarily wouldn’t have the information required to answer the question. Unless it was explicitly known beforehand that this information would be relevant and logging of the removal event has been included. By contrast, in an ES system, you can always answer such questions about the past, even if didn’t know you were going to have that question because all history is available.

Some of our clients say, hmm that is interesting and we might need it in the future. For them, I would skip the ES journey. When they are interested in such events (which 90% of the time they would not be) then we would start writing some of these events in a actor, synchronously to a queue.

Or as Chris put it in his blog

I think you can generally answer it with some alone time, deep introspection, and two questions:

For which core problem is event sourcing the solution?

Is what you actually want just a plain old queue?

If you can’t answer the first question concretely Don’t. Those are not problems exclusively solved by event sourcing. A good ol’ fashion history table gets you 80% of the value of a ledger with essentially none of the cost.

Similarly, CQRS doesn’t require event sourcing. You can have all the power of different projections without putting the ledger at the heart of your system.

The latter question is to understand do those downstream systems care about those intermediate states, or will it just be noise that needs to be filtered out? If the end goal is just decoupled processes that communicate via something, event sourcing is not required. Put a queue between those two bad boys and start enjoying the good life.

How the hell did we get here?

Well these are battle scars from several projects where we had a chance to make a decision on the architecture. In some we did use ES and in others we did not. We might have used them where did not need to and the other way round as well.

We get too consumed by the latest toys in the industry and we forget about why we became software developers and architects in the first place. Software architecture best practices, enterprise architecture patterns, and formalized ways to describe systems are all tools that are useful to know of and might come in handy one day. But when designing systems, start simple and stay as simple as you can. Try to avoid the complexity that more complex architecture and formal tools inherently introduce.

This brings me back to something that I had learned almost 15 years back

The best architects are the ones who can keep the architecture simple without the need for needless complexity unless it is needed.

Talk to us for building beautiful systems with the right architecture.

Event Sourcing — Does it make sense for your business? was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

At the Edge

AI at the edge allows mission-critical and time-sensitive decisions to be made faster, more reliably and with greater security. Cloud-based AI suffers from latency. The delay as data moves to the cloud for processing and the results are transmitted back over the network to a local device. In many situations, latency can have serious consequences. For instance, when a security camera at an airport or a factory must recognise intruders and react immediately. An autonomous vehicle cannot wait even for a tenth of a second to activate emergency braking when the AI algorithm predicts an imminent collision. In these situations, AI must be located at the edge. This is where decisions can be made faster without relying on network connectivity and without moving massive amounts of data back and forth over a network.

As Gartner analyst Thomas Bittman points out his blog The Edge will Eat the Cloud:

“The agility of cloud computing is great — but it simply isn’t enough. Massive centralization, economies of scale, self-service and full automation get us most of the way there — but it doesn’t overcome physics — the weight of data, the speed of light. As people need to interact with their digitally-assisted realities in real-time, waiting on a data center miles (or many miles) away isn’t going to work,” writes Bittman. “Latency matters. I’m here, right now, and I’m gone in seconds. Put up the right appealing advertising before I look away, point out the store that I’ve been looking for as I drive, let me know that a colleague is heading my way, help my self-driving car avoid other cars through a busy intersection. And do it now.”

Moving AI to the edge has other strong reasons as well

• Across the plethora of industries, the edges are not usually connected to the cloud and they are not smart either. Every industry and every manufacturing unit might need to build smartness in the edge for its own needs. For example, a security camera in a chemical factory might need different set of intelligence tools than the one in manufacturing unit.
• AI will have more economic impact than AI in the cloud. The edge is where real money is being invested right now, and putting AI there even to get a small sliver of increased productivity will have a massive impact.
• While cloud computing offers unquestionable economies of scale, a distributed computing model is driven by the nature of the data itself. The volume of data will make it difficult or expensive to move due to bandwidth costs or availability. The velocity of data will catalyse more real-time applications that cannot be limited by network latency. And the variety of data will be governed by regulatory, privacy and security constraints.

Considerations

Processing Power

The edge needs more processing power. This will enable enterprises to run AI models at the edge, thereby bringing more intelligence to the edge.

Nowadays, a lot of edge devices have inbuilt compute power. A lot of IoT edge devices have GPU, TPU or VPU. For example, some of the high-end security cameras now feature GPU cards, which enables them to run AI-based image recognition models on the edge itself instead of sending all the HD video back to the cloud for processing. Moving the processing to the edge ensures better response times and reduced bandwidth usage.

Lifecycle on the edge

The heterogeneous nature of devices on the edge in an IoT world has its own set of challenges. Remote deployment of the models and monitoring the edge for performance is another big area that has immense potential. One has to have a robust mechanism to deploy and fine-tune the AI models remotely. It’s also critical to keep a close eye on the health of the hardware.

Continuous monitoring of the performance of these models is also a high ask. Managing the continuous deployment, debugging and fine-tuning of AI models on the edge is also an area in which few companies have made real advancements.

Security

Bringing the processing closer to the edge puts more pressure on having rock-solid security in and around the edge. Security at the edge has to be a multi-pronged strategy to ensure the safety of the hardware and software stack. You need to remain vigilant to detect rogue nodes entering the edge network. Once rogue nodes are detected, they need to be isolated and not be allowed to enter the edge network.

Learn

AI that learns at the edge is a paradigm-shifting technology that will finally empower AI to truly serve its purpose. This is by shifting intelligence to the compute edge where it is needed, at speeds, latency and costs that make it affordable for every device. This learning could be brought in by the cloud or better still the edge nodes learn themselves and pass on their learnings to the cloud to be propagated to other edges as well.

Synergies with Cloud

This is not to say that AI On the cloud is a thing of the past. Both edge and cloud would continue to co-exist.

There is a limit to the amount of computation power that can be put into a camera, sensor, or a smartphone. In addition, many of the devices at the edge of the network are not connected to a power source, which raises issues of battery life and heat dissipation. This gives a strong reason to train the model on the cloud and allow it to inference on the edge. Consider an AI-powered vehicle. AI at the edge powers decisions in real time such as braking, steering, and lane changes. At night, when the car is parked and connected to a Wi-Fi network, data is uploaded to the cloud to further train the algorithm. This is called call home data and this would allow the feedback loop to be completed to train the model better. Now consider that there are thousands of vehicles doing this in the night. The central server would take in all this data, work on it and send across a smarter algorithm back to the vehicles in a couple of days not if the next morning.

Conclusion

There will be a need for AI in the cloud, just as there will be more reasons to put AI at the edge. It is an AND depending on where the intelligence needs to live. In this vision of the future, intelligence at the edge will complement intelligence in the cloud, for better balance between the demands of centralised computing and localised decision making.

Opinion: Is your Business Ready for AI on the Edge? was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Components of a Learning Agent

The learning agent can be divided into four components

1. Learning Element — Responsible for making improvements.
2. Performance Element — Selecting and taking appropriate action. Takes the percepts from the sensor and passes on action to the actuator
3. Critic — Gives measures on how the agent is doing. The percepts received by themselves do not tell about the metrics of success. The critic puts the metrics in perspective. Learning element takes constant feedback from the critic to pass on better decision making logic to the performance element
4. Problem generator — This is like the chaos monkey. It would cause conditions which would lead to informative experiences. It can suggest exploratory actions which would be taken by the performance element and the knowledge of the results would be passed back to the Learning Element.

If we go through the sequence of steps, this is how the action might flow. The sensor gets an input from the environment (1). This inputs goes to the critic and the performance element (3). The performance element has to take an action on the basis of the input for which it consults with the Learning element to decide the best action (4). This action is passed to the actuator (5) which passes it to the environment (6).

At all times in the background or in parallel (T1), there is a process of learning which is running between the critic, learning element, problem generator and performance element which allows the problem generator to suggest alternative actions on the basis of what the learning element is learning. It allows the performance element to take actions and keep learning from that. This allows the learning cycle keeps going on.

Working together of components

Working of components together can be pretty involved depending on the use case. There are a few ways in which components can interact with each other

• Atomic — This is the simplest way. The components are not concerned about the internal state of other components. They treat them as black boxes with input and output. The output of the component B is just dependent on the input from component A. As an example, in our case the performance element might just get a knowledge from Learning element and base its action on that.
• Factored — This representation splits the state between variables and attributes. It tries to look into the state of the individual component. Many important areas of AI are based on factored representations, including constraint satisfaction algorithms, propositional logic, planning , Bayesian networks, and various machine learning algorithms.
• Structured — This is the most complex of the three. Each state has a series of object each of which might have attributes of its own and relationship to other objects. There would be many real life scenarios in which the variable and attribute associations would not occur in Factored representations. As an example for a drone to cross a tree an association might be present where the height of the drone and height of the tree might result in a true or false. As an instance, in this scenario, HeightOfDroneGreaterThanHeightOfTree might be true but if it is dependent on other factors like birds flying around the tree then such representations would not be possible in factored approach and would require a structured representation.

What kind of agents are you setting up for your AI practice. Knoldus would be excited to help you on the journey.

AI: Making Agents Learn for Better Business Decisions was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

We at Knoldus build a holistic software development capability for anyone who joins us as an intern. It does not matter if you have been working in the industry for 2 years or 10. When you undergo the internship we would give you a holistic software development immersion right from the Code Quality, Code conventions, Principles, practices, and patterns of software development further leading to Reactive Platforms and the ecosystem tailing into the stack that we embrace which is the Scala ecosystem and the Fast Data Platform.

The trigger for this post is the conversation with a top talent who joined us 3 months back. He was sad because he was not working on Big Data. When asked what did he mean by Big Data, the quick answer was Hadoop/Spark. When countered by the fact that he was learning Lagom and event sourcing which would allow him to build better solutions but he was not too convinced.

Now, there is nothing wrong with these technologies and in fact, they are what has made the ecosystem popular but these technologies are only a part, sometimes a very small part, of the product that would have any business value. They solve a particular piece of the puzzle and more often than not if you base your product “just” on these technologies you are bound to fail!

So where should we be headed if we are not talking about Big Data? The answer is to talk about Fast Data. Big Data as anomer/misnomer gets used in all kinds of scenarios. Talk to 10 CIOs and 9 would say that they struggle with Big Data. It is of no consequence whether one manages 1TB of data and the other is managing several hundred PB of data. I think where we should be headed is that with our solution/ product how do we make sure that the customers get the best experience. Customer Experience (CX) is going to be the king of the modern day applications. Just focusing on Spark/Hadoop/Flink and thinking that you can do Big Data is a fallacy.

Let us see how these set of so-called Big Data technologies fit into the grand scheme of things.

• If you are going to build a product which would include user interaction then you need a reactive front end to the product so that you can provide amazing customer experience.
• When hundreds and thousands of user requests come in, the product has to handle them without degrading performance. It has to be resilient.
• There are going to be transaction based processes like someone querying for something, adding an item, viewing their trades for the day. These could be handled by different micro services. These would have their individual life cycles and should be able to scale independently.
• You would like your system to be extensible and plan for any future business operations which are unforeseen at the moment. For this, you need to have event sourcing.
• You would want to separate out writes and reads to your system for making sure that the read and write SLAs are met and you are able to scale the read and write side separately.
• You would need to store your transaction data in the DB and for that, you would need either a SQL or NoSQL DB.
• Now some of your functionalities would also need analysis of data and come back with analyzed data. Now depending on the SLAs, this is where you would need Big Data frameworks to jump in.
• You would need to run some machine learning or deep learning algorithms for your product to stand out.

Of course, we are simplifying the scenario a lot but hopefully, you get the idea. Just being dependent upon a Big Data framework or hiring consultants who know a bit about Hadoop/Spark is not going to fly. You need an entire gamut of technologies that you need to work on. Right from

• Reactive UI
• Microservices framework
• Asynchronous Messaging System
• Big Data framework (there I said it!)
• Database
• Hosting strategy based on containers
• Monitoring and Telemetry
• Machine learning and AI

And believe me, this is a partial list.

And overlaying all of this is the Principles, Patterns, and Practices of effective software development. The main drivers of technology which are based on the principles of Reactive Manifesto would be

1. Scalable
2. Secure
3. Flexible
4. Realtime
5. Resilient
6. Everywhere
7. Intelligent
8. Agile

To sum it up here is one possible scheme of technologies that can fulfill the product vision.

As you would see Big Data frameworks are only a part of what you want to do. More than a drop in the ocean but still not big enough.

Hence, next time when someone comes and talks about Big Data and using Big Data framework to build the product, then do talk to them about all the other ancillaries and take what they say with a big bag of salt :)

Knoldus has implemented its Digital Transformation product KDP at two Fortune 50 organizations. The third implementation is underway.

Can we stop talking about Big Data now? was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Confluent, creators of the first streaming platform based on Apache KafkaTM, provides the most complete platform to build enterprise-scale streaming pipelines using Apache Kafka and simplify the development of stream processing applications. Via rapid adoption in the Fortune 500, Apache Kafka is quickly emerging as the leading streaming platform for real-time applications. Data-centric businesses can scale their production environments with confidence in its reliability, efficiency, and scalability for hundreds of use cases, thousands of applications, and trillions of messages.

Leveraging the power of Apache Kafka and Confluent Platform, Knoldus has already deployed several fast data solutions into production. One of the very first deployments of Knoldus on Kafka was in early 2012 for a real-time sentiment analysis platform that is being used by Coke, Target, P&G, T-Mobile to name a few. Since then, Kafka is being utilized as the heart of the system for Knoldus Fast Data Strategy.

Always-on streaming systems raise the bar for the product. In batch jobs, there are concerns about high availability, resilience, etc. However, a streaming job is expected to run for days, weeks, even years. Streaming systems must embody the Reactive principles of excellent responsiveness, scalability, resiliency, etc. That’s changing infrastructure for data-centric systems, in addition to the long-standing concerns about having sufficient capacity, security, etc. for using very large datasets.

Kafka ends up being the heart of such systems.

To build robust systems around Kafka, Knoldus fully acknowledges and utilizes the Lightbend Fast Data Platform. The general architecture of the products is as follows

The Fast Data Platform:

Our team is well equipped to qualify and deliver projects on the Confluent Platform and Apache Kafka through architectural guidance, team augmentation, training, and strong development support across multiple geographies.

“We’re excited to have Knoldus join our partnership program given their proven track record of implementing Apache Kafka-based solutions for their clients,” said Jabari Norton, vice president of business development at Confluent. “As both a Lightbend and Confluent partner, Knoldus is well-positioned to deliver stream-centric, real-time solutions to customers.”

About Confluent

Confluent, founded by the creators of open-source Apache Kafka™, provides the stream data platform that enables enterprises to maximize the value of data. Confluent Platform lets leaders in industries such as retail, logistics, manufacturing, financial services, technology, and media, move data from isolated systems into a real-time data pipeline where they can act on it immediately. Backed by Benchmark, Data Collective, Index Ventures, and LinkedIn, Confluent is based in Palo Alto. To learn more, please visit: www.confluent.io

About Knoldus

Knoldus specializes in building reactive products using the Scala ecosystem and data science platforms using Spark. Their strong background in Scala makes them attractive when you need deep Spark understanding. They work on a broad variety of big data initiatives like Knowledge graph systems (Ontology and semantic web), time-series & Real-time (IoT), Structured ( Relational & NoSQL), and cover varied domains including Pharma, Manufacturing, and Finance.

Knoldus is active in the Open source community with contributions to various libraries like Spark, Scala, Akka, Neo4j, Deeplearning4j, etc. Knoldus is known for its tech blogs and open source contributions in the area of big data and functional programming.

Some of the known direct clients of Knoldus include Huawei, Philips, Schlumberger, Elsevier, BoA, and quite a few niche startups. To learn more, visit www.knoldus.com

Leverage Knoldus strengths for your streaming product? Contact us today.

Knoldus Partners with Confluent to Power Real-Time Streams was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

All agents have the same skeletal structure. They get percepts as inputs from the sensors and the actions are performed through the actuators. Now the agent can either just act on a percept as a reflex for example if you throw a ball at me and I try to catch it (or duck from it given that I am bad at baseball) than that is a quick reaction to the percept. On the other hand if you throw a ball at me and tell me to arrange it by color or count the number different colors that you are throwing at me then I would have to maintain a state to do the counts correctly. So this would involve some state but is still ok. Now, if you want to trouble me further and you tell me to jump twice if you throw red ball at me and do a burpee if you throw a green ball at me apart from catching and counting then you got me for sure :)

This would involve a complex logic of me maintaining a mind table of what needs to be done on what percept and this is called condition-action rule. Now if all these percepts were to be indexed then this would become a significant data set.

Consider the automated taxi: the visual input from a single camera (eight cameras is typical) comes in at the rate of roughly 70 megabytes per second (30 frames per second, 1080 × 720 pixels with 24 bits of color information). This gives a lookup table with over 10 600,000,000,000 entries for an hour’s driving. Even the lookup table for chess — a tiny, well-behaved fragment of the real world — has (it turns out) at least 10 150 entries.

This can become a lot of information.

The key challenge for AI is to find out how to write programs that, to the extent possible, produce rational behavior from a smallish program rather than from a vast table.

So this brings us to 4 documented types of agent programs

Simple Reflex Agents

As the name suggests, these agents behave on reflex. Someone passes by a sensor on a motion detection light, the light goes on. It does not hold any state, any intelligence. Super cool! This agent has a list of condition rules. As soon it gets a percept (1) , it acts as per the condition rules (2) and then it responds back to the environment with the actuators (3).

In this scenario the agent does not claim to know more about the environment than what is presented as an event to the sensor.

Model Based Reflex Agents

These agents handle the scenario where the observability is incomplete. It would get events from the sensor but in order to determine the next step of action, it must consult its internal state.

The internal state of the agent is a combination of 2 things, transition model and sensor model.

Transition model is further divided into (i) How does the external environment change on the basis of agents action (ii) How does the external change over time. For example, a motion detector camera knows that when it asks the head to move 30 degrees right then it would see a different view of the environment in which it expects some objects. On the other hand given the changes in seasons, it would expect snow in winters and leaves in fall.

Sensor model is how the environment is represented in the percept of the agent. For example for this motion detector camera, if it has a big tree on the right then it knows that most of the motion that it needs to track would be concentrated on the left side of the area.

A combination of data from transition model and sensor model, further combined with the current event received from the sensor allows it to create a current state model.

The new percept coming from the environment (1) is matched with the results from the internal state i.e transition model + sensor model (2). This results in a new current state being generated (3). This current state is then checked with the condition-action rules to determine the next action (4).

Goal Based Agents

Many a times just getting to understand the current state based on the new percept and the internal state is not enough. The agent needs to determine whether it is getting closer to the goal or not. The goal for our motion detector camera could be to find the red car and send an alert as soon as it finds it. Now this would be very different from just an condition-action scenario. The condition-action might want the camera to turn left on the basis of internal state but the goal condition might want it to keep facing right for extra 10 seconds if it found a hint of a red car.

Notice that decision making of this kind is fundamentally different from the condition–action rules described earlier, in that it involves consideration of the future — both “What will happen if I do such-and-such?” and “Will that make me happy?” In the reflex agent designs, this information is not explicitly represented, because the built-in rules map directly from percepts to actions.

Goal based agents might be less efficient and costlier but they are more flexible.

Utility Based Agents

Goal based agents are not always high quality. The goal could be to capture the number of red cars but to achieve that goal, the agent might do constant movements from left to right, up and down but it would be costly, consume more power and probably reduce the life of the agent.

Utility takes the performance characteristics into account and refers to the quality of being useful.

There might be several ways for the camera to detect a red car but it tries to find the most performant, least costly way to do that. That is the utility angle to the equation. It must try to maximise the utility. Choosing the
utility-maximizing course of action is also a difficult task. A utility-based agent has to model and keep track of its environment, tasks that have involved a great deal of research on perception, representation, reasoning,
and learning.

Utility based agent uses a model of the world, along with a utility function that measures its preferences among states of the world. Then it chooses the action that leads to the best expected utility, where expected utility is computed by averaging over all possible outcome states, weighted by the probability of the outcome.

What kind of agents are you setting up for your AI practice. Knoldus would be excited to help you on the journey.

AI: Right Structure of Agents For your Business was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

There is a lot of interest in Machine Learning and AI. Ofcourse, a lot of it is still the level 1 of AI . This is when we are thinking about machines acting like humans. Everyone wants to jump on the bandwagon of AI. It is an amazing field and man organizations do not want to be left behind. That said, something which is ignored most of the time is the fuel, the data!

Teaching the machines required showing them a lot of “training data”. This could be in the form of videos, images, text, structured, unstructured, inhouse, locked in private spaces, public spaces, etc etc. Machines need to learn with a lot of data. They also need to learn with the Right Data.

In theory, the world is awash with data, the lifeblood of modern AI. A market-research firm, reckons the world generated 33 zettabytes of data in 2018, enough to fill seven trillion dvds.

Data issues are one of the most common sticking-points in any AI project.

Considerations of working with Data

Data-wrangling of various sorts takes up about 80% of the time consumed in a typical AI project. Machines, in order to perform accurately need a lot of training.

According to the economist, 80% of the time is spent in getting the right data. This is where most organizations falter. They spend too much time working on the model training and algorithms without realizing the data ownership and cleansing mechanisms.

These result in data anemic models where either the data is incorrect or inadequate . The results from these systems are usually questionable.

Two course data scientists (Engineers who take a couple of courses on data science) and proclaim mastery in the field, are almost always a victim to the pattern of ignoring the importance of data. The attention on detail for cleansing, labeling and combining of data cannot be ignored. Without that you are hunting for your car keys under the lamp post. Not because you lost them here but because there is more light here.

All said and done, internally, getting access to the right data is a huge challenge for many organizations. The data science and machine learning departments would quickly like to start building models with subset of data. Procurement and cleanliness of data come as an afterthought and that is where most ML/AI failures would happen.

Inadequate attention on building the pipelines. Once the data ownership and the cleansing etc are identified, the job is not complete. Many organizations would fail in building the pipelines to get the data from the source to the sink. Many AI/ML models would work on a streamflow of data. Consider adjusting the medicine dispensed to a patient in the hospital which depends on his vital statistics or detecting fraud in real time. These systems need robust data pipelines which can be high-availability and scale well.

Modern data pipelines include acquisition of data from various systems and combining that data to together to run analytics. A possible architecture would be the following

Data is collected from various systems, flowing into a lake with an orchestration layer. It is finally fed into a datawarehouse from where different modeling systems and AI systems can consume it further.

Data needs a lot of handholding. The training labels usually have to be applied by humans. Big tech firms often do the work internally. Companies that lack the required resources or expertise can take advantage of a growing outsourcing industry to do it for them. Mechanical turk from Amazon brings data labelling to a huge population of data entry operators who can label a lot of data like x-rays, image etc.

Further, the data itself contains traps. Machine-learning systems correlate inputs with outputs, but they do it blindly, with no understanding of broader context. That has led to a lot of bias being built into machines. A recent case which exploded in the media was the face recognition used in US. This was more often than not biased to skin tones other than white for identifying suspects. This is completely dependent on the kind of data being fed into the training set. It seemed that most of the colored data set fed in was for suspects which resulted in the algorithm behaving in unprecedented ways.

In 2017 Amazon abandoned a recruitment project designed to hunt through cvs to identify suitable candidates when the system was found to be favouring male applicants. The post mortem revealed a circular, self-reinforcing problem. The system had been trained on the cvs of previous successful applicants to the firm. But since the tech workforce is already mostly male, a system trained on historical data will latch onto maleness as a strong predictor of suitability.

Other Considerations

Privacy of data is another issue where a lot of legal and medical data is not readily available. This results in a lot of organizations generate synthetic data which can be close, in best case. In most cases it ends up being disjoint from the real data.

Another interesting one is drift in the data. Sometimes this drift is gradual. As in the case of shopping preferences for a region in the country where the population is aging and does not have too many kids. Other drift scenarios could be overnight. For example, booking a one way ticket on a credit card would trigger alerts but this was a norm in COVID situation. Similarly any prediction models which were based on forecasting the need for toilet paper or home gym equipment were thrown off with the surge in demand during the pandemic.

The world’s changeability means more training, which means providing the machines with yet more data, in a never-ending cycle of re-training.

AI is definitely not an install-and-forget system

Thinking AI? Think Data First was originally published in Knoldus - Technical Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.