Siddhartha Ahluwalia 00:00

So what has changed in the world in the last eight years is that even the common man is talking about artificial intelligence.

 

Manoj Agarwal 00:05

November 30th of 2022. That’s when for the first time I would say the world has realised that there’s something that has changed.

 

Siddhartha Ahluwalia 00:12

What I’m really afraid of is, if you see the rise of people like Osama bin Laden, they were engineers and they used technology to do attacks. No AI’s giving power to everyone, good people, bad people.

 

Manoj Agarwal 00:21

Vinod Khosla spoke about, ‘They’re going to be a billion developers’, you can ask the machine to write in Python or write it in JavaScript or TypeScript or whatever language just with one prompt. What do you look at it like day to day, How much time that you spent on the screen? How many ads do you see? How many news articles that you see? How do you know that these are not “Fake”?

 

Siddhartha Ahluwalia 00:42

It will be very hard to differentiate? What is the truth? What is not?

 

Manoj Agarwal 00:45

This probably is the biggest challenge!

Siddhartha Ahluwalia 00:48

Hi, this is Siddhartha Ahluwalia and welcome to The Neon Show. This episode’s guest is the co-founder of a business software company in the Bay Area. Before this venture, he was SVP engineering at the unicorn SaaS company Nutanix. But his journey actually started through growing up in a small village in Jharkhand. It’s my pleasure to welcome DevRevs Manoj Agarwal on the show. I would also like to thank our sponsors, Prime Venture Partners, for sponsoring The Neon Show. Hope you enjoy it!

Siddhartha Ahluwalia 01:25

You graduated from IIT in 1997′ went to the US in 2000. I think you joined Dheeraj Pandey’s Nutanix in 2009.

 

Manoj Agarwal 01:33

Yeah, I joined Dheeraj’s team in 2013, at the beginning of 2013.

 

Siddhartha Ahluwalia 01:37

The company was just four years old?

 

Manoj Agarwal 01:39

Yeah, but 50 developers at that point.

 

Siddhartha Ahluwalia 01:41

And you let the entire engineering team and you became SEP(Systems Engineering Plan) engineering?

 

Manoj Agarwal 01:44

Yeah. So initially I joined for a very small part of Nutanix engineering, but over time, quite a major chunk of data I highlighted, but many of the new initiatives, like the second product, and third product and fourth product that I got to work on from ground up.

 

Siddhartha Ahluwalia 02:05

Nutanix today is the $10 billion company traded on the US public market.

 

Manoj Agarwal 02:09

I mean, like in 2016, I remember just being at this IPO…the New York Stock Exchange. It was at Nasdaq, that’s where we did. And you don’t realise at that point, just the moment itself that when you are in that room, that history is getting created, and so on. Still I remember that, it can really give you goosebumps. But that was 2009, let’s say Company was founded in 2016, like in 7 years that it went public and 2020 when both Dheeraj and I left the company was at $1.6 billion in software revenue. I mean, quite remarkable. Journey.

 

Siddhartha Ahluwalia 02:55

Fantastic. And today, you and Dheeraj have built DevRev, right? and you’re talking directly to Salesforce…

 

Manoj Agarwal 03:04

I mean learning, I would say in building something, and then the pin points that you also see. So I’ll say that one of the things that we are super proud of at Nutanix was the very, very high net promoter score, NPS and NPS a lot of people that they confused with the CSAT (Customer Satisfaction Score), because customer satisfaction score that when you say 90% or 95% they are saying that on a score of let’s say one to 10, that they are giving you nine and above mark, right, that’s the way that you can see that maybe when you see 80%, now it’s like 80%, still considered very, very high on the CSAT side, so 80 scored that you got. When on Net Promoter Score when people read you at eight, it’s considered zero. So seven and eight is a score when customers give you is considered zero below that is minus negative score. And holy nine and 10 is considered plus one. So a score goes from minus 100 to plus 100. That’s the way it is normalised and despite the speed that we are growing every year that we consistently stayed above plus 90. And I mean, when you start to think about that Net Promoter Score, then what are the things that we did that were so different? That’s where it came down to? Okay, are there learnings from this company that we can bring to the worldwide companies because a lot of people don’t realise the NPS can be a great product for you, for your company, very unlikely you can start very small with any company that you can grow really big. So many of the accounts that we had very small initially, that they continue to grow in became million dollar and even 100 million dollars.

 

Siddhartha Ahluwalia 04:57

Okay, a single account giving you $100 million. DevRev, you started with a unique insight that you can marry customer support and product into one and build a new kind of a CRM.

 

Manoj Agarwal 05:08

Yeah, I’ll tell you that what happened just at Nutanix, itself, like $1.6 billion, that we were bringing revenue, we were spending close to $400 million just in r&d product, side $100 million in customer support, close to a billion dollar in sales and marketing, let’s say that’s the kind of spend that we are seeing. And of the $400 million that we were spending, there was always this question, and which, like, people do that every quarter, doing the QBR. And trying to figure it out, how much money that I’m spending on, let’s say, existing customers, how much that money that I’m spending on new feature sets that I’m building, how much of the work that I’m doing on the maintenance work, and so on. And this exercise, you bring the data onto the spreadsheet, and you go and try to categorise them, it’s like the moment you first of all, bring the data onto the spreadsheet become stale. And then after that, you start to do manual triaging of these and it’s just a complete mess. The number of people, amount of time and all of that that requires doing that is a very difficult problem. And I tried to solve this, just in the context of Nutanix. And we gave like two startups this money that can go and build, which can answer this question, here are the 10 questions that I want you to answer. Some of that was related to Okay, Which are the customers for which I’m doing the work? How much work is going there? How much new innovation, how much maintenance work, and all of that. Year of work that we did with these companies and not thinking concrete came out, from those exercises where at least it became very clear to us that it is a tough problem that we are talking about. It comes down to I mean, if the data model itself is not right, and data model, when you think about it, what are the most important thing, when you think about in the company—

 

Siddhartha Ahluwalia 07:11

For our audience, what is a data model? Let’s say you explain it to a child.

 

Manoj Agarwal 07:15

Yeah, so data model is like, information that you keep, it has to go and, like, connect to areas of concern that you have. So I’m just saying, for the companies, I’ll come and explain that. But let’s say for the companies, what are the most important things?

 

Siddhartha Ahluwalia 07:35

Customer data is most important thing—

 

Manoj Agarwal 07:37

Two things I would say they’re definitely customers. But there is also a product, the company because they have a product. And because they have customers, in the absence of either of them, you don’t have a company. Yeah. And then there are people inside the company they work. They work on what ? They work for the product, they work for the customers. So you start to see there is product, there is customers, and there is work associated with the product and customers. Now, when you think about a data model, now data model is like, when I define work itself, how do I really store this work? And how do you really connect this work to the product and to the customer? And if you could do that really correctly, then you have a beautiful system at that point. So you stop thinking about the current system that exists at that point. So let’s say work has very specific characteristics. Also, it starts and ends any work that you think about, and then it goes through a number of stages and a number of workflows that run on it. So if I’m doing software development, and most people use, let’s say, Atlassian, Jira. And that work goes from open state to let’s say, in development, stage two in Q&A, stage two, and resolve to closed state like that’s the typical way that you’ll see on the software development side. But when you do customer support, it’s also work. Just that we call it a different thing we call it customer tickets, we put it in let’s say ten days for Salesforce Service Cloud those systems and they are also working when it opens it gets assigned to an agent, agent is working on it. If there are product escalations, the product escalations come back. Eventually, Market is all waiting for the customer to say yes or no. And then you go and close, it turns out the salespeople also do work. And just that we call that work differently, we call them opportunity, they work on opportunity, and that also opens, goes to the pipeline, stays, gets to the strong upside to commit to, then finally closed, closed as one or loss, let’s say, so we thought about it hard. You said like, you know what, there is product, there are customers, there is work, and work has these characteristics. If you could go and represent that in a way with a customised customizability around Under work, then you can build a beautiful system of record. And then you can start to think about, okay, if I have to go and provide the customer support, then I can go and paint it, let’s say yellow, call it customer support related work, can paint it blue, call it development related work, and so on.

Siddhartha Ahluwalia 10:20

And today’s a very historic moment, right? Every product that we touch right, at least digitally, and some of it physically, also the cars that we travel in, The room that we sit in, a lot of it is governed by artificial intelligence. Even the product that you’re building, right at DevRev I believe the heart of it would be built on top of artificial intelligence machine learning, right. So, what has changed in the world in the last eight years, that today everybody, even the common man, is talking about artificial intelligence.

 

Manoj Agarwal 10:58

I mean, first of all, like when we started DevRev, we wanted to make sure that the AI is built in and why? What does it even need to represent the AI, it is DevRev.Ai after the fact it was part of the plan that when we started and when you think about what people do people are working and work itself there is machine and if machine has the knowledge can you let machine do the work?

 

Siddhartha Ahluwalia 11:30

Like a machine is intelligent right? Machine has data. And then what you are saying is can a machine think for itself?

 

Manoj Agarwal 11:37

Yeah, the thing is that when a machine has the data and if the machine is able to do the work, can you let the machine do it and when the machine doesn’t know how to do that work? Can it call the human to do the work? So in that sense, humans can act like a copilot to machines and vice-versa. Also, you can see that we are in human when they are doing the work and machine assist doing that work to the human—

 

Siddhartha Ahluwalia 12:02

What is a copilot? Like in basic terms, If you have to explain to a kid.

 

Manoj Agarwal 12:06

Just think like in the car itself. When you’re driving a car. There’s a copilot, let’s say or in a flight that you see there is a copilot who’s sitting who’s assisting you with certain things that whatever assistance that the people need. And we see that in the car also not a car like the truck when people are driving. Okay, take left or take right or look after other things. And so that in the human says that we see that copilot that exists.

Siddhartha Ahluwalia 12:34

If somebody’s carrying an expensive load in a machine the car truck, so that the person doesn’t sleep off or doesn’t commit an error. They have to prevent it.

 

Manoj Agarwal 12:44

I mean even that you see that the copilot when you’re driving the like pilot when they are driving this big aeroplane, how the machine is acting like a copilot also there, okay, I can land without really, me doing it the manoeuvring myself, the plane can land on itself. And that’s like a machine doing all that work directly. So like in the general sense, when you see the copilot right now, when the machine is given so much information? And if it has it and if it can determine that this particular question has been asked, with the information that I have, I’ll be able to go and answer that with a certain degree of let’s say, certainty, then I’ll let the machine answer that. And if I if machine determines that okay, no, I’m not certain then it should be able to go and ask some human to come and help answer.

 

Siddhartha Ahluwalia 13:38

Let’s say I’m an author, I’m writing a novel, right? So I have to give some input for the machine to write. It’s not that a machine can tap into my brain and start writing. We are away from it.

 

Manoj Agarwal 13:50

I mean, obviously, like when you’re writing a book, and I’m assuming that is a completely different idea that you have a machine that can’t think that way. machine can only act on things that already exists—

 

Siddhartha Ahluwalia 14:01

But can machines think also?

 

Manoj Agarwal 14:04

No machine has the data. And now we are talking about AGI(Artificial general intelligence) and all of that.

 

Siddhartha Ahluwalia 14:09

What is AGI ?

 

Manoj Agarwal 14:11

I can say that. First of all, AGI is beyond understanding. It’s almost like Okay, can I get to what human kind of thinking or human kind of intelligence AGI full form is Artificial General Intelligence, I know a lot of like OpenAI of the world that they are still working towards it. When it is achieved, then we’ll all know what it means and so on. I’m watching this TV series on Netflix and many of those concepts actually come to the surface there which tries to show that, if human kind of thinking or intelligence that comes in that how it can help a lot of human also especially people who are, let’s say, in their late,, 70s, 80s, 90s, that they need help, that AGI can help them, like through the use of robots, let’s say.

 

Siddhartha Ahluwalia 15:15

On their behalf because their neurons are dying or their cognitive abilities are dying.

 

Manoj Agarwal 15:20

Right. So how can it really go and help? Now, a lot of questions on the ethical part and this part and that part will come. I mean, there is going to be regulations, and law—

 

Siddhartha Ahluwalia 15:32

When everybody, even Elon Musk was one of the founders of OpenAI. And when they started open AI, the goal was to achieve AGI. And now everybody in the team is divided on whether we should achieve AGI or not?

 

Manoj Agarwal 15:45

I mean, it will happen. Any technologies, I mean, when you start to think about it. Now, the question is, what kind of regulations, what kind of safety that you go and put in? Even like the bots that you go and create, they’re all operating under certain rules. And rules basically, just can be okay, it can never harm humans—

 

Siddhartha Ahluwalia 16:12

These rules are set by humans to never harm humans—

 

Manoj Agarwal 16:14

Yes, I’m saying that it has to be protected, because there are always going to be elements in something or the other, or some element will say that, okay, no, it should go and harm a certain type of human, let’s say.

 

Siddhartha Ahluwalia 16:26

Well, I’ll give you a use case, right? And you can tell me, a bot is programmed to give the best outcome to the customer, right, and a human has designed the bot with certain set of rules. Now they’re building intelligent bots. And probably in some time, not very long, like six months, one year, there can be a bot that can say, hey, this rule designed by humans is not good enough. So my job is to give the best outcome to the customer. So I’ll build my own rules or ignore these rules. Because I have more data than the human who put these rules—

 

Manoj Agarwal 17:01

I will find out(Chuckles). I mean, at the end of the day, that they are going to operate under the way the rules that are defined—

 

Siddhartha Ahluwalia 17:08

But I think the rules are also changing very fast. Right?

 

Manoj Agarwal 17:12

This is where the regulations, I’m assuming that certain regulations, okay, here’s the foundational principle, like today for the human when they make mistakes, let’s say they do certain activities that are considered, let’s say, illegal by the law of the land, then there are repercussions of that, like there is a system that is in place, which has to deal with it. Now something similar that how do you really go and implement that for machines that will be really good to see. Its like how do you punish the machine?

 

Siddhartha Ahluwalia 17:43

Yeah, it’s almost like the movie Matrix. Right?

 

Manoj Agarwal 17:49

It will be quite fascinating—

Siddhartha Ahluwalia 17:51

And nobody imagined that this could accelerate so fast. At what is—

 

Manoj Agarwal 17:57

November 30th 2022. I mean, that’s when for the first time I would say the world has realised that there’s something that had changed when ChatGPT was released. And maybe for the first time that people also felt that actually, AI could do something.

 

Siddhartha Ahluwalia 18:15

And if you have to describe to a layman right. A student in India, why ChatGPT is so special and why it’s such a monumental in the history of humankind, why would it be?

 

Manoj Agarwal 18:27

First of all, like, when you think about the LLM (last language models)?

 

Siddhartha Ahluwalia 18:32

We can go back to what is the language model?

 

Manoj Agarwal 18:35

So people have heard about the NLP for the long term. NLP is Natural Language Processing, right. Natural Language Processing now, which is like, okay, the human the way that they speak, can you really process that? And go and answer certain things. So a large language model is supposed to, really get the input, and be able to go and answer in a way that the human can understand, in a language of like, let’s say, the human that they speak. And because it has so many, let’s say, parameters with which it is tuned. So you’re talking about 175 billion parameters, let’s say for the GPT 3.5. Or let’s say, you’re talking about those many parameters when you go and ask a question that how the model is going and spitting out an answer that humans can go and understand is what that the LLM ‘s are providing us today. ChatGPT was nothing but a UI representation or UI interface through which that human could go and interact with LLMs or GPT models. That’s what it provided a mechanism because prior to that okay, through the API’s and you know by writing the code and all that you could go and achieve that, but that didn’t really reach the masses. Because Masses don’t understand it. So they just want to go and interact in a way that they will interact with any machine or so. So chat GPU is nothing, just a UI interface. But underneath the models was something that they had for quite some time.

 

Siddhartha Ahluwalia 20:15

And now because it’s able to understand and speak, in these language models, now, the UI they gave was a chatbot. Now UI can be anything, UI can be somebody saying it to you, that’s what you know. At the end of the call, if you’re talking to anybody, you don’t know whether you’re talking to the machine or not.

 

Manoj Agarwal 20:37

Right. Because like, once you have the answer, then the answer is textual, visuals, or through speech, I mean, all those things, like text to speech that already exists. So you can easily can go and do that. Now. If you’re able to process the incoming text, or speech to text, then text to text, then text to speech, and it’s like you’re completing the entire cycle. It’s all there. I mean, then you start to go and apply that to bots. Even human bots, you can start to show that.

Siddhartha Ahluwalia 21:19

What I’m really afraid of is because, you know, if you see the rise of people like Osama Bin Laden, they were engineers, and they use a technology very well to do attacks now. Now AI is giving power to everyone, good people, bad people. When such power goes to wicked people who are fanatic by nature, they don’t consider themselves bad, they consider themselves highly purposeful.

 

Manoj Agarwal 21:52

I mean, one of the things that you see or start to see is leaders of the countries, let’s say, and if bot starts to make decisions, let’s say it starts to change the outcome of election results. I mean, right there that you see big risk—

 

Siddhartha Ahluwalia 22:12

I think the Indian Government got slightly scared, and they put in the media, there was a recent clip of Prime Minister Narendra Modi doing a Garba. Famously. And he later came on an interview said that the last Garba I did was in class eight.So this was completely a deep fake, which looked like original Prime Minister Modi—

 

Manoj Agarwal 22:33

Deep fake is, we all know that is real now. And what you see in media, what you see on videos anywhere on social media, and all of that can’t rely on any more. So I don’t know what kind of there is definitely going to be regulations and law of the land. And there’s going to be punishment, all kinds of things that are going to be put in place like when you’re caught, then there is going to be some ramification of that. Hopefully, with that, it reduces. But the bigger problem is when country to country, two different countries, …. they have different laws. And it’s not like that, okay, one country trying to go and change the outcome of an election in your country that you can go and punish them. How do you know which court, that you’re going to go and punish them—

Siddhartha Ahluwalia 22:39

They’ll go to International Court—

 

Manoj Agarwal 22:45

Right. So those are the kinds of risks that—

 

Siddhartha Ahluwalia 23:22

I think it was seven years ago, when Russia was accused of interfering with US elections to bring President Donald Trump in. Nobody knows the truth, what happened. But it was a very large acquisition. But—

 

Manoj Agarwal 23:40

But those things are real now, like it can just happen? I mean, what do you look at it like day to day, like how much time that you spend on? Let’s say, just on the screen? How many ads do you see? How many news articles that you see? How do you know that these are not fake?

 

Siddhartha Ahluwalia 23:55

Yeah. And these are just drops to change your mind.

 

Manoj Agarwal 23:59

You see it enough times it just starts to look real to you—

 

Siddhartha Ahluwalia 24:06

Earlier my fear was hackers used to send you fake links. For example, if you have a bank account, they will change the domain name and then it was still directed by intelligent humans. Some people fall prey to it. Now they can also start targeting intelligent humans to programme in such ways

 

Manoj Agarwal 24:26

Yes. And that’s what he’s doing. You’ll see it enough times that you start to believe that is the truth. Yeah. It’s quite scary.

 

Siddhartha Ahluwalia 24:41

It was called Phishing attacks. I think now Phishing attacks will become more intelligent. So they’ll not try to phish you on the first attempt. They will try to phish on the 100 attempt—

 

Manoj Agarwal 24:50

Thing is that they are not asking you anything else. You’re just seeing it and just changing your opinion. You don’t have to click on anything. Were you watching ads all the time? You just see it all around you.

 

Siddhartha Ahluwalia 25:07

I think it will be very hard to differentiate what is truth what is not. Right.

 

Manoj Agarwal 25:12

So this probably is the biggest challenge(Chuckles).

Siddhartha Ahluwalia 25:19

Do you have any fear that AI is progressing at a speed that the best of humans cannot? Not able to keep pace with it?

 

Manoj Agarwal 25:28

I mean, it’s a very interesting question. First of all, like such technology, first, basically, they move slowly, and then suddenly, that’s what happens. And it feels like it is moving very, very fast right now. Now to say that, there are a lot of good use cases, by the way, and there are a lot of bad use cases also will come come along, in at least in my head right now it is, if cognitive load that the human have, doing anything that they do a lot of busy work that they do. And if machine learning or AI, at least in the context of let’s say work that we do day to day at work, starts to take away, then we have more time to think, more time to do a lot more innovative work. So in that sense that it should help in a big way, for the people, in fact, even for the people where it takes away certain jobs completely, but then they can focus on something that is more meaningful to go into it. Maybe I was doing one thing that is now being done by the bot, but I can go and learn new skills, because there are also interfaces that are getting developed with which I can do a lot more in existing areas of work that I can start to do. So it’s going to be interesting, in my opinion, that biggest worry is that, from the people who are trying to do things that is going to be harmful for the people versus like, just the cognitive load that it is taking away from people, which is like, mostly positive—

Siddhartha Ahluwalia 27:08

Which are the jobs that you think will first go away? Because of AI. What kind of jobs are they?

 

Manoj Agarwal 27:15

I think that if you’re not producing something, from let’s say, yourself, if you’re not producing something yourself, and most of the time that , a question is being asked to you, and you have to go and look for that information, search for that information. Or you read about that information. That’s what you’re going and, and answering, those are the things that you will start to see that the bot can do, because they can go and search for the information, they have (Inaudible) information right there. And they can continuously learn about this information, like while humans will have to go and sleep and so much time, but machines they don’t need to go and sleep. So those are the jobs that I definitely see as the first one which can easily be displaced. But that just says that okay, because machines can do a better job.

 

Siddhartha Ahluwalia 28:13

Few use cases that I think of first are news anchors that consume news from 10 contents and summarise it. Second is as we discussed customer support agents that are on the other end, that their job is to find the solution which is sitting somewhere not to create a solution and relay it back to the customer.

 

Manoj Agarwal 28:31

Right. Yeah, in fact, you will see even there are a lot of people who just go and ask the question, some simple question, getting that answer tabulated. And then go and create the reports and so on. Many of those things also, as a program manager, you will see even like the jobs that what in the sales development rep, that they do, take the information and they just have to go and speak to the customers like, do a lot of dial, a lot of email, a lot of campaign and so on. Many of these things can’t be automated by machine. I was in Palo Alto, in my office next to it there is a cafe where I was seated. And there was a newspaper ad on the front page, it was saying that the city council is looking into the uses of AI to reduce the workforce. And I was like okay, if it is reaching the city level now, there is something real that is happening. They’re all trying to save cost now and they think that there is something in here that can help reduce the cost. So, I mean, it will get to that level, just a maximum but there’s going to be some other jobs that we get created. Then after that.

Siddhartha Ahluwalia 29:59

And what jobs do you think will be created first because of this revolution?

 

Manoj Agarwal 30:02

The very first thing is just the tons of regulations, that means that every company will require people that will …more regulators inside of ethical committee and compliance and I mean, tons of jobs that will just right there will get created inside the company itself, that you can see. Company will be doing a lot more work, I mean, maybe with the same resources will expect it to do a lot more work, maybe the cost will go down even more further because the cost of development is going down. So, accessibility to technology will just happen more and more. Vinod Khosla spoke about that, ‘They’re going to be a billion developers’. Billion developers.

 

Siddhartha Ahluwalia 30:51

The world’s population right now is I think 8 billion—

 

Manoj Agarwal 30:55

Yeah, so we were talking about what is happening with the access to technology now that people will be able to go and programme very easily. Now you don’t need a very, very specialised degree to be able to go and programme—

 

Siddhartha Ahluwalia 31:08

You don’t need to learn Python to programme—

 

Manoj Agarwal 31:10

Yeah, you can. Like you can ask the machine to write in Python, or write it in JavaScript, or TypeScript or whatever language like just with one prompt, that you can do that, right. Remember that this inversion of control that we talked about what is happening right now, inversion of control is previously for machine to be able to go and do the work, we had to go and write the language in which machine can understand it, which is like the programming languages that we have to go and write. That’s the only way that machine will do the work. Now it has reversed the role wherein machines have to learn the human languages to do the work. That’s what has happened. So I don’t have to go and even learn any of those things. As long as that I have the analytical skills, what question to ask or what I’m planning to go and build. That’s all it’s just instructions that I have to go and give. And the machine is supposed to understand it and create and do the job. Which language do they go and program internally?…. I don’t care, really. So that’s the inversion of control.

 

Siddhartha Ahluwalia 32:17

And thereby people like it’s in the news, I don’t know if that’s true or not that prompt engineering is in high demand.

 

Manoj Agarwal 32:23

I mean, those things, at least in my mind. Which is analytical skills. Okay, what more information that I can go and give the machine and can route them in the right direction? My thinking is that even that will become a bot. That will start to prompt based on the domain area that I am that it’s able to go and feed that information. So right now, obviously prompt engineering because then you can get the answer the way that you want it for the certain domain and all that you can also feed that domain expertise or the information by the bot. And I’m sure that there are companies that will get formed, which will just do that, which is just feeding the information is—Prompt Engineering companies. Yes. It is a bot that just does prompt engineering for you.

Siddhartha Ahluwalia 33:14

So we both are of the previous generation, right? But whatever gets defined in the new generation, for example, Social Media got started because of the Gen Z of today. How is Gen Z today using AI? Because that will shape up how AI gets used ultimately—

 

Manoj Agarwal 33:28

I mean, I can say that I have two daughters. They are both Gen Z. And what are they using right now? I can see the chatGPT definitely. It is getting used, that’s the way that they are getting introduced to AI—

 

Siddhartha Ahluwalia 33:45

For reducing their homework?

 

Manoj Agarwal 33:46

So it’s interesting. It’s mostly for asking questions right now, in most schools, and they don’t allow them to use so they don’t. But to go and ask a question, get an answer, and then go and figure it out. Like how to do it. Instead of going to Google to search for the information they are searching on the ChatGPT or asking the question, getting the more precise answer on ChatGPT. And even for learning things that they’re going more to this interface, newer interface, versus going to Google. Video definitely is playing a big role for them, which is like YouTube and all for searching the information and going and watching that. That is like it’s still harder to go and just displace right away. But I’m sure that we’ll go and see a lot more coming there.

Siddhartha Ahluwalia 34:40

And the other thing is that the AI models that people are training, right? How do we ensure that they don’t have any confirmation or harmful biases? Because we don’t know, right? The input data, or the data model that you talked about, where’s the source of those data models.

 

Manoj Agarwal 34:58

That’s the risk. So, especially when you go and look at any general purpose model that is out there, and what exactly are you using? If you’re using that for the entirety of the information to really get certain things done, then it’s probably going to be hard. So let’s say we think about it a lot. In the context of the companies, you have data, you have information, can you really restrict all your answers or any kind of things that you want to answer only to that set of information that you’re providing. So use that and then use the LLM, only to translate once you have the information in a language that the human can go and understand it, but don’t use the intelligence or the biases that are set with those LLMs today, so most enterprises that you will see that they will just do it that way, in many cases that you will also see that, especially for the enterprises, again, that they will go and train the LLM with the data set for their domains with their data, instead of just going and using the the general purpose for all things that they want to do.

 

Siddhartha Ahluwalia 36:11

And there’s a popular term called Reinforcement learning . What is that?

 

Manoj Agarwal 36:15

I mean, It just comes down to every time that you miss something, and a human comes and answers something, then how do you use that information for the next time around? When people ask the similar question?

 

Siddhartha Ahluwalia 36:30

Can you give a live example?

 

Manoj Agarwal 36:31

So let’s say a customer asked a question. And the machine goes and determines that okay, this answer, I don’t really have with certainty that I can go and answer. So at which point a human will come and answer. And when human answered that question, once it is clarified, let’s say somebody internally reviewed it and say that okay, this is the right answer, by the way, then at that point, the machine has to learn it, that next time when similar question comes, I can go and use that as an answer. So it’s like Reinforcement Learning that is happening in the process. And you can be the one interface that you see. The second one is that when the machine itself is answering, you’ll typically see that there is a thumbs up or thumbs down. That people put as part of that—The ChatGPT interface. Yeah. So with reach also, the machine is learning that okay, the answer that I’m giving Is that accepted answer or not accepted answer. So much that next time around can alter the answer.

Siddhartha Ahluwalia 37:37

Coming to Indian context, right, the first company that you built, Nutanix, I believe there were very few Indian engineers or Indian workforce that you hired, which was sitting there in India, and you and Dheeraj visited India seldom. And but here in DevRev, I have seen you visiting India more and more, what has changed in the last 15 years that that is—

 

Manoj Agarwal 37:59

First of all, Nutanix was founded in 2009. 2013 We came to India. So Bangalore office was set back then, one thing that you will see and I was talking to another founder today, the b2b culture is particularly in Bangalore is a lot more b2c that you see here, and less and less b2b, you go to Chennai, that you find a lot more b2b culture there. So for a company like Nutanix, to even come and establish their that time, we had to go and hire people, a lot of people from the college to really go and build what we built out of here. Obviously, like right now more than 1000 people that we have at Nutanix, just in Bangalore, office alone. There is a lot of learning that came, I would say, during that period, what we could go and build and how to go and build it and how you go and look at also people that were quite young, let’s say out of college, give them the internship, onboard them as new college grad provide them like all the way that they can be very successful inside the company, give them the environment with which that they can really produce similar to what like people, let’s say back in US also will go and produce that learning this time around that we didn’t want to wait for four years. So we debriefed on day one that we decided that okay, we just need to find out where outside the US where we’ll be the first office that we decided with Bangalore. And you also see like there are, at least on the enterprise b2b mindset. And there are Google came in like, especially with the Google Cloud that you see that they established here AWS came in, not amazon.com but AWS part that came in, in India during those period we are talking about post 2013 that has happened. So this time around just made it much easier. The other part that you asked me, like, we travel a lot more, I was travelling quite a bit during Nutanix days also at least once a quarter because we had a large presence here. But what do you see also the SaaS I mean, what has happened? The talent pool definitely is there. But also, lots of founders now, founders led companies that are out of India, like I don’t know, like, maybe right now. What close to 4000 SaaS companies just out of India alone right now. So that is quite interesting. I mean, you come here, you learn quite a bit also from these founders. But—

Siddhartha Ahluwalia 40:46

What is the noticeable change that you are seeing? Since you mentioned, you have been coming here every quarter, since 2013. And you are sitting at the Mecca of innovation, which is San Francisco—

 

Manoj Agarwal 40:56

First of all, like, in terms of the talent pool, and all that. I mean, it was always there. One thing that has changed is that we see differences definitely, like I’m looking at 2013 to 23. So just service mindset to product mindset. Like we were providing to the world a lot of services, back then to now we are providing a lot of products out of this place. So that’s a big change that has happened, because of which also what you see is a lot of talent, actually, they stay put in India now, as opposed to like in the past that everybody wanted to just move out to go and build products, somewhere else—

 

Siddhartha Ahluwalia 41:34

I believe you moved out in 2000, because there were not so many opportunities in India?

 

Manoj Agarwal 41:37

Not Yeah, so that’s the reason why you move out because there was not a product that is getting built. Every extension of the team or company that they were getting built here, they are doing maintenance work, mostly they are not building products, you will see that the one dot of the product is built somewhere else. It’s always like the enhancement is small enhancement, and all of that, which is well put in the maintenance category that was built which has reversed completely now. And every company like now you see that the extension is built but to extension is not to go and do the maintenance work, but to build the primary product. So that has changed. The second thing that I have noticed is previously, people wanted to become managers very fast. So I have three years, five years of experience. Now I want to become Manager, which was like the social pressure that the people had. And that also had changed dramatically. Now people want to stay technical. They want to grow in the technical ladder, they started to realise that also, there’s not much difference in terms of the Comp structure and all of that, in fact, you can grow a lot more just being technical. That’s like one change, or the second change that I’ve seen, which is like, quite interesting to watch.

Siddhartha Ahluwalia 42:51

And in terms of you know, DevRev right, you build DevRev.ai First, right? What are the difference between you seeing the companies that are not built AI? Firstly, now trying to integrate an AI company like you were built on AI on day zero.

 

Manoj Agarwal 43:09

I think the first of all, I would say I’ll normalise it because people probably thought that maybe AI is very, very hard. Or I need to be very specialised… PhD in AI. I mean, like you only if you hire the people with AI background and like PSD, Zeno, that’s the only way that you can be an AI company, I go back to just looking at AWS, like, even just when you go and look at each of the services that you go and consume today, out of AWS, back, then each of those services were multiple companies before and required a specialised skill set for you to be able to even go and use those services from those companies, you have to have the deep expertise. Not anymore, you just go and you consume the API. And you see that because of this, there are so many applications that are built by so many people around the world sitting in their dorm room that they are able to go and do that. And the question is can you really go and extend that to the also and which is what is happening, because of which you will see that just every application will have the built in in there. So I’m talking about just the orchestration layer itself, which let’s say if you pick(Inaudible), open source that anybody can go and use it. Then you start to look at okay, here are the embedding models. Here’s the caching layer, here is the vector database, here is the foundational model. These can all be consumed with the API. Now, it’s not that it requires you to have a massive amount of knowledge of the AI itself. So I think the normalisation is prior to let’s say, what two years ago I would say, that it was out of reach for the people. And today it’s all within reach for the people and the same people can go and in their companies can start to use or provide AI, as part of their platform, it’s not a differentiator anymore, It will not be, it will become just the necessity for you to stay relevant.

 

Manoj Agarwal 43:39

So you’re seeing every company that starts, every technical company that starts today will be AI first, or has to be AI— first

 

Manoj Agarwal 44:31

I mean, it just becomes anything that I’m doing with your product. And that requires me to go and write a bunch of things in your product, that AI should just take care of it. For me—

 

Siddhartha Ahluwalia 45:39

It’s just like 20-25 years ago, every software that we use to interact with most of them are command prompt based and it needs a command prompt specialist to interact, and then the world of UI came in. And that’s why you think, with a clean UI, the next thing is where I don’t need that kind of complex UI to navigate a software.

 

Manoj Agarwal 46:02

I mean, we’ll have to start to think about when we say they’re going to be billion developers, let’s say that’s what Mr. Vinod Khosla is saying. And that simply means that how do you really go and create a billion developers, the only way that can happen is when you make the interface so simple, right. And that’s what like every company who are coming out or whatever that they are building, they’ll have to make the interface so simple that anybody should be able to go and use it. It doesn’t require a specialised skill set, for me to be able to go into use, and the only way that you can give that kind of interface, if you go and build the AI part of your product, you have to just make it simple, I can come and ask question, and the systems will just do it for me, instead of me going and clicking at 20 different places to achieve that task, which is possible today. You just need to go and use these API’s and models that already exist.

Siddhartha Ahluwalia 47:00

So in a way it’s going back to where it all started. Initially, every software was based on a command prompt that I needed to have learned the language of command prompt. And then it shifted to User Interface with hundreds of buttons, which in the backend, was still a command prompt. And now I’m back to the language, which I understand. And I don’t need to use a button. I can try to go on a bank’s lesson, it might not be a website anymore, I can go to a bank interface, authenticate myself and let’s say transfer rupees 500 from Siddhartha To Manoj’s account, and I don’t need any application.

 

Manoj Agarwal 47:34

This is it. I mean, maximum that it will give you an interface saying that okay, is this what you meant? Just to reconfirm, you say yes, go.

 

Siddhartha Ahluwalia 47:46

Ultimately, it will lead to all crumbling down of all interfaces.

 

Manoj Agarwal 47:49

Yes. But then, underlying that will be the .AI.

 

Siddhartha Ahluwalia 47:56

But then it will, it was also a question or then why would a person need a mobile phone with hundreds of apps, because if I can get a single interface, that’s what everybody in the world was initially starting to build it right. Everybody wanted to build a Super App. WeChat became the Super App of China. But I think in the rest of the world, there was no super app. I mean—

 

Manoj Agarwal 48:15

Still, there are workflows, like for the kind of work that you’re doing. And companies are still going to go and specialise in that because in the end, whatever prompt or questions that you’re asking some workflow is still has to go and run in the background that you’re doing—

 

Siddhartha Ahluwalia 48:32

But they don’t care about if somebody wants to make the life of the user super easier—

 

Manoj Agarwal 48:37

You’re right, from the user perspective, they can just come and ask, and it has to leave, then transfer to the right app.

 

Siddhartha Ahluwalia 48:45

Yeah. Background which app is running user doesn’t care—

 

Manoj Agarwal 48:47

Yeah, probably that yeah, you’re right. That’s like, okay, I can come to Google, I can ask a question. I need to just figure it out, like, okay, which app and how to get that information—

 

Siddhartha Ahluwalia 48:48

If I order an Uber ride for me, I want to go from here to there.

 

Manoj Agarwal 48:52

And those will just happen much sooner.

 

Siddhartha Ahluwalia 48:58

Crumbling of, I think software and App interface—

 

Manoj Agarwal 49:06

Book the ticket for me from here to there. Yeah, I mean, those are pretty similar—

 

Siddhartha Ahluwalia 49:10

Why do I need to go to 10 websites, compare prices, then figure out if this date ticket is available or not.

 

Manoj Agarwal 49:17

That is still already happening. If you just think about it, like aggregator sites, that’s what they were trying to do is to give you the options and so on there,

 

Siddhartha Ahluwalia 49:27

But that’s what ultimately Open AI is doing to charge up at Google is giving you 10 options. Why do you want 10 options? I can give you the right answer—

 

Manoj Agarwal 49:36

The right answer itself. What you think is the right answer is still human will say no, no, I still have an opinion. Yeah. So give me some choices. And then I’ll go and decide. Because what do you think is the right answer? I may think that okay, maybe for 2%-3% of the cases that might not be right.

Siddhartha Ahluwalia 49:52

I think that is what is standard hallucination, right?

 

Manoj Agarwal 49:56

Yeah. I start to think like when you talk about hallucination, even human hallucinate, right?

 

Siddhartha Ahluwalia 50:04

What do you mean by Hallucination?

 

Manoj Agarwal 50:05

Hallucination is like, okay, you don’t know the answer. But you try to, like, relate it to something else and try to go and answer. And which simply means that okay, it’s not a precise answer. It’s not the answer that actually is the right answer for the question that has been asked, but you are hallucinating. Now. In the organisation, also what you will see, first of all, you ask the same question to 10 Different people in the organisation get different answers, you will not get the same answer. Now, do we say that? They’re all correct. Do you say that okay, only two answers are actually precisely the right answer. The rest of the answers are near about or maybe they are hallucinating also. And that’s what that you will see with the human but when humans do it we don’t take a note. We don’t say anything.

 

Siddhartha Ahluwalia 50:55

We internally think that they are confused.

 

Manoj Agarwal 50:59

So we might think that way. But when the machine does it, then we say that, okay, look, the machine is hallucinating. Now, if you provide, here is the set of documents or the information that I have. And I’m going to just feed you, you cannot really go out of the information that I have provided, if you don’t have the answer, just say that okay, I don’t have the answer. This is where the safe zone is. You only look at this document, nothing else. And then, so that’s where you start to remove the hallucination.

 

Siddhartha Ahluwalia 51:33

That was ultimately initially page 404. When a website couldn’t be found on a browser, it uses a ad not able to find the site. But now it gives the incorrect website. That’s I think hallucination.

 

Manoj Agarwal 51:43

Yes, you’re right. So here, what you will see and what, like every enterprise, the way that they’re going to go and adopt AI is like, Okay, here’s this, just, I want you to just play within this boundary, here is the information that I’m providing you. If you can get the answer, then answer it. If not, then just quietly say that, okay, I don’t know the answer. Let me get some human to come and answer. And that’s the boundary that will start to play there. And that’s the way that you start to remove the hallucination. Now, when we talk about the hallucination right now, it’s all in the context of this open thing that we ask a question. And the machine doesn’t want to say that. Okay, I don’t know the answer. It goes and gives you the answer. And because we are human, we can also understand we see that okay, I know that you are hallucinating right now, because this is not the right answer.

 

Siddhartha Ahluwalia 52:32

But to determine whether a hallucination is happening by a machine, you need an intelligent human on the other end, hence the machine to start to programme.

 

Manoj Agarwal 52:40

Right. So the adoption, particularly for the businesses, and all that, that’s why you see that they are why they are limiting that. And a lot of companies that are coming in, so try to go and solve for it.

 

Siddhartha Ahluwalia 52:46

That means that any enterprise or any business, when they are giving access to AI tools to their employees, they are very heavily guardrails.

 

Manoj Agarwal 53:03

They are guardrails, you can’t just go on, you can’t say that, okay, I can run the business on ChatGPT. You cannot! Just like you can’t run your business, just saying that go to Google and ask the question, and whatever answer that you get, you can go and run the business, right? You can’t. So previously, nobody was even allowed to go and get the answer, unless, like I am providing in the form of FAQ or KB’s, I have made it open and you go and determine whether even the link that you go and click it will take you to my website. And that’s the way you know that this information is right or wrong. And here, that’s why it will be guarded very heavily that you can only come to my chatbot and ask this question, and I will ensure that the answer that you get is the right answer.

Siddhartha Ahluwalia 53:50

I think the world hasn’t seen enough bad use cases of ChatGPT because if I’m searching on Google for a cure, they will not give me the answer. Google will redirect me to WebMD. But now ChatGPT will give me the answer of what medicine I should take in a fever. And if it’s given me the wrong medicine and didn’t know the history. Probably we’ll start seeing repercussions.

 

Manoj Agarwal 54:16

I’m sure like the domain specific things that will start to happen. So just wait for maybe GPT 5, GPT 4 is already quite intelligent. But the GPT 5 probably will take us to a different level. I’m quite hopeful, though. On the medicine side. We already see on the legal side that it’s making quite a bit of an impact already, or you can go and talk to the lawyers they are already talking about okay. There are so many jobs right now in the company that the machine is able to do a much better job versus the human in the law firm. So you’re seeing that the use cases are emerging which is quite safe also, with the information that is fed to the machine to only restrict yourself in that domain only.

 

Siddhartha Ahluwalia 55:05

Thank you so much Manoj. It was a wonderful conversation, and especially your insights on AI. I loved interacting.

 

Manoj Agarwal 55:12

Thank you for having me. A lot of questions are quite insightful. I also got quite a bit in this conversation. So thanks. Thanks for having me.