Description

This episode of the MLOps Weekly Podcast delves into the intricate journey from academic research in AI to its practical application in industry, highlighting the challenges of bridging theoretical concepts with real-world solutions. CEO and founder of Manot, Chinar Movsisyan, and Featureform CEO Simba Khadder discuss the evolution of MLOps practices, emphasizing the importance of collaboration across diverse teams to innovate and scale AI technologies. Through engaging stories and insights, listeners are offered a deep dive into the strategies for deploying, managing, and improving machine learning models, showcasing the critical role of MLOps in advancing AI's impact in various sectors.

Listen on Spotify

Transcript

[00:00:06.130] - Simba Khadder
Hey, everyone. Simba Khadder here, and you are listening to the MLOps Weekly Podcast. Today, I'm speaking with Chinar, who's the founder and CEO of Manot, an MLOps company based in San Francisco. She's been in the computer revision field for more than six years, from research labs to venture-back startups. She's led AI projects in different mission-critical applications such as healthcare, drones, satellites.

[00:00:27.420] - Simba Khadder
Manot solves one of the key problems of ML, aligning many different stakeholders on ML model failures from an automated feedback loop and insight management platform. Manot went for the Berkeley's SkyDeck Accelerator and their expanding business in several different verticals. Chinar, it's so great to have you on the podcast today.

[00:00:43.470] - Chinar Movsisyan
Yeah, thank you so much for having me.

[00:00:45.170] - Simba Khadder
Before we dive in some of the questions I have, I thought it'd be great to share a bit of your story. I'd love to learn more about your journey to Manot.

[00:00:54.110] - Chinar Movsisyan
Yeah, sure. I'm Chinar, and I'm from Armenia, originally. I have a technical background. Since childhood, I was passionate about mathematics, digits, and everything, and that's why I ended up in Applied Mathematics and Informatics, not faculty where girls can do their things. But yeah, I did that

[00:01:14.950] - Chinar Movsisyan
Then I started doing my master's in information technologies. I really like how mathematics is being used in different industries. Because of that, I started doing research in AI, specifically computer vision. It was five or six years ago when I started some real-world project in drones, like object detection for agriculture, automated spraying and everything. That was the first time I was trying to understand how we can use imaging and everything.

[00:01:43.440] - Chinar Movsisyan
Then after having this experience and everything I moved to France for my second master's. I started doing AI. It was in information technologies, computer science, and then systems, to understand how AI is being used in chip designs, how helpful it is, and everything. Then I started doing my PhD. It was machine learning integration into cardiovascular problems and identification.

[00:02:05.550] - Chinar Movsisyan
These all chapters have something related to AI, but different industries. I have always worked as an engineer, as a researcher, or tech lead in everything. I know the whole life cycle of this machine learning AI, computer vision development. I have faced every problem, every challenge in every step, like data collection, data preparation. How are we going to choose the model? How are we going to train it? Then deployment in the real world on the Raspberry Pi, Jetson, like these microchips and everything. Then model maintenance, how we make sure that these models are performing well.

[00:02:44.000] - Chinar Movsisyan
While working as an engineer, one of the problems that I have faced was how reliable is this model? I could have some 99% accuracy in the test set, but once it's deployed in production environment, after some months, your product manager, business owner, or product owner is coming, "Hey, Chinar, it's not working."

[00:02:59.840] - Chinar Movsisyan
What can I do with this information? Can you have some more actionable feedback about the model, or what's happening on the customer side where you are not happy? With this back and forth and everything, I started doing research in this field to understand what solutions are there. That was the reason I started doing Manot.

[00:03:21.490] - Simba Khadder
That's awesome. There's a ton on Pac Fair. I think where I want to start is, do you have any advice for someone who is... You were in academia for a while, then you jumped into the real world where you were literally like physical world projects where everything that can go wrong will go wrong. Do you have any advice for an academic, maybe someone that's listening, maybe much more of an academic background trying to move to industry, how they can be successful and best supply their skills?

[00:03:45.270] - Chinar Movsisyan
Wow, that's a really good question. I'll share this with you. I started doing the research, understanding how I can start a startup. Do I want to start a startup? What is the reason I want? To be really honest, I know my investors can listen to this podcast, but I didn't know what is pitch deck. When they told pitch deck, I was like, it's just a presentation. I have done this before several times and two different people. I didn't know anything about that.

[00:04:13.260] - Chinar Movsisyan
But what was the reason the passion and everything why I did that? I was trying to understand how this... Academia has a lot of good work, research and everything, but the question is how applicable those results in the real-world application.

[00:04:28.980] - Chinar Movsisyan
We have a lot of benchmarks on a lot of cool publicly available models or data sets, but those formulas cannot work in the real world, in the production environment. If you have that sense that there should be a link, there should be some adaptation, like smooth path, it will be much more easier to understand how to do that.

[00:04:48.970] - Chinar Movsisyan
The tip will be understanding the applications and also just accept that in academia, we do something. But in a startup and as we built something for more than hired, our goal is to build this for a lot of companies so they can use internally. We should make sure that it's working there. That sense should be there.

[00:05:12.310] - Simba Khadder
Almost like focusing on the end user and just like, that's all that really matters. It's almost like a means to an end, it sounds like, is what you're getting at. Getting that mode in from like, I need to build the best model ever to be every benchmark to, hey, this is what our users want to see. How do we get there?

[00:05:27.300] - Simba Khadder
We talked a lot about lifecycle and the lifecycle machine learning model. You gave a quick overview of it from data preparation, data cleaning, all the way to monitoring, evaluation, and debugging. I guess I'd maybe first love to have you unpack some of that model lifecycle and really focusing on where are the key challenges that people face, especially with computer vision models?

[00:05:50.900] - Chinar Movsisyan
That's a very good question. In general, in machine learning life cycle and then computer vision, which is the same life cycle with some specifications and everything, what is happening, let's say there is a task from an end user. They want a face detection model. What they do, they talk to a business owner within a company, and they try to explain, "We want a face detection. It's going to be deployed somewhere on a camera in front of shop or wherever." That's all. This kind of information.

[00:06:21.210] - Chinar Movsisyan
Then these business owners, they try to dive deep into some more details so they can write down the definition of done, these documentations and everything. Then they take that information, and they come to engineer saying, "Hey, we want a face detection model, some specifications, and that's all." Very limited information, maybe about features or attributes about those model specification.

[00:06:45.520] - Chinar Movsisyan
What's happening, engineers, what we do, we take that information, and we go, and we just look for face detection model and face data sets. Then what we do, we just get that information publicly available, we train that model, we look at the data and model itself, but the initial step is just to have an initial model.

[00:07:07.330] - Chinar Movsisyan
Then we try to get some data from real world where the model will be operating, and then we do that inference on that, see what's happening, and that's all. This is the model development and then testing on our side and then testing it on some limited amount of data from customer side.

[00:07:27.230] - Chinar Movsisyan
Having this pipeline, what we see that is missing, one of the questions that we should answer, and we should be really careful is how representative is the data and how representative is the test set, and what are the acceptance criteria, so these two stakeholders are happy with that.

[00:07:47.470] - Chinar Movsisyan
I'm happy with that as an engineer, so I have built, I know what they want me. Business owner is happy as he or she understands the problem that customer wants to be sold, and the customer is happy. But these three stakeholders, different stakeholders, have very different toolkits, very different background or expertise in AI or in general in the field. Because of that, these gaps bring a lot of challenges to the table, like how representative was the test set, how to make sure that it covers all potential scenarios that are happening in the real world and how to capture that.

[00:08:28.420] - Simba Khadder
Yeah, I guess, the three stakeholders you mentioned, the business owner, the end user, and the engineer, are there more typically or do you think those three are encapsulate? Who would you say the main stakeholders are in a mall life cycle?

[00:08:40.550] - Chinar Movsisyan
I would say these are three main groups of people, maybe business owners or product managers or just business analysts or director of ML, computer vision. End users are just normal people. They are just using AI. Yeah, the output of AI.

[00:08:56.380] - Simba Khadder
Today, I know you've seen the companies you've worked at. How do the business owners and engineers, what have you found? First, what's average? What do you typically see? Let's start there. What would you typically see of a business owner and the engineers and data scientists, et cetera? How does that collaboration look like? How are specs written? How is that done?

[00:09:15.600] - Chinar Movsisyan
I have spoken in my customer discovery. I have spoken to more than two hired business owners, mainly product managers and this kind of people, and then engineers, and yeah, they are not on the same. They say these business people, what they do whenever something is happening with the model or with some requirements from customer side, they just write down text. They just write it very normal English.

[00:09:43.860] - Chinar Movsisyan
Their feedback is whenever they are going back to engineer saying, "Hey, we have this piece of information. We should work on this." They are saying, "Can you please translate it to some work so we can work?" It's just a sentence saying that the model is this blah, blah. I cannot use that to fix it.

[00:10:03.710] - Chinar Movsisyan
It's just the Excel sheets, and it's just the documents, or I have seen cases where they have this mural diagramming and everything, which again, it's not working. Or the opposite side. Engineers, they are saying something about the model, let's say, performance, like metrics, F1 score or a lot of other metrics we know, and it doesn't make sense at all to these people.

[00:10:29.400] - Chinar Movsisyan
What does it mean? Well, false positive rate is this. What shall I do with that? How shall I communicate it to end users? With this, they're not on the same page. They're just using a lot of existing solutions they adapted to their needs.

[00:10:43.670] - Simba Khadder
Let's say right now someone's listening where either, let's say, an ML engineer or data scientist, and they're facing this problem, or the other side, the product manager facing this problem. What would you recommend? What does it look like when these two different stakeholders are aligned?

[00:11:00.580] - Chinar Movsisyan
That's the challenge that we are solving at Manot. Besides Manot, what I'm doing here, in order to be on the same page, in order to succeed the project and the whole life cycle and everything, I would say one of the most important thing is to understand the customer need, like how the model is going to be used, what are the main means of the customer, where the model will be deployed, and what are the criteria, and what business problem is being used.

[00:11:29.810] - Chinar Movsisyan
These questions are very critical. Also, with these questions, there should be data-driven way of collecting the answers, like where it is being used, some video from that environment, or some information about people or what's happening in that environment, like when it is, let's say, in the summer or in the spring, these all have effect on the decision, pre-development, preparation of this development. That's key.

[00:12:00.560] - Chinar Movsisyan
Then having this information, engineers, they should ask a lot of data-related questions. It's not about having a lot of data, but it's all about how representative is that data, how we capture that data, how informative is that. With this, the life will be easier to have a good model and then at least avoid further headaches.

[00:12:24.610] - Simba Khadder
Sounds like sticking to the end user, focusing there, and then just making sure that everything's almost like... Almost sounds like what you're getting at is PMs, business owners, think in one way, data scientists and managers think in a different way, but in the end, all that matters is the end user. Framing the questions, framing the problem statements in relation to the end user.

[00:12:44.740] - Simba Khadder
I know you've been working on Manot. I'm curious to maybe hear more about how that looks in the Manot environment. How would it look like if I had to keep your vision project, that feedback loop, how does Manot help with that?

[00:12:57.290] - Chinar Movsisyan
Yeah, what we have done with our solution platform that we are building, we have been building. It's a platform that is designed specifically for these two stakeholders, like product managers, like business people who don't have much technical expertise, skills, and developers. It's not a developer tool. It's a tool where these two people, they can log in, and they can see what's happening.

[00:13:18.790] - Chinar Movsisyan
The next step is, well, let's say there is a case of face detection model. What they do, they just link some information about their model performance on their collected data where they evaluate the model performance in general.

[00:13:34.070] - Chinar Movsisyan
After that, the platform proposes insights in the form of images so they can evaluate it further, helping them as a hint saying, "Hey, these are the other cases we see for face detection model, where the model is not going to work properly when faces, people are with curly hair, pink hair, this stuff, or the background is not so clear, or the input is blurry, these categories.

[00:14:01.210] - Chinar Movsisyan
One of the most important things we wanted to capture in this overall challenge of model evaluation, it doesn't require the model itself. There is no need to go just check the architecture and see, well, yeah, maybe seven billion parameters are trained. Maybe we need one more billion parameters to be trained, so the model will be perfect. No, we cannot say that.

[00:14:25.790] - Chinar Movsisyan
We say that it's more about data. It's more about seeing where are the gaps? Where are the blind spots of the model in a data-driven way? These product managers can take that data and can communicate with that data to engineer saying, "Hey, the model is not working well when there are faces, people with curly hair." We see that in the environment, in the customer environment, there are a lot of people like that, and we should address that issue.

[00:14:53.190] - Simba Khadder
That's great. For those listening who can't see, Chinar is rocking pink curly hair right now. She's describing her own experience of [inaudible 00:15:02] models. I'm almost trying to paraphrase what you're saying because it's almost like rather than communicating in English, like you mentioned earlier on, a lot of like, hey, it doesn't work in this situation, or someone said it's broken here. Being able to rather than send things in text, send things in data.

[00:15:20.050] - Simba Khadder
Here's almost like a packet of a directory of things that generally describe the problem statement. I have even some preliminary data showing you the data scientists that, hey, it seems like we're weak here. How can we fix this? Then you can take that work off.

[00:15:36.740] - Simba Khadder
It's almost like allowing the PM, giving them the tools to be... I'm sure every PM would love to be able to give a very technical, here's what we need to solve, but it's not exactly what most PMs are trained to do. It's not a good use of our time always to become that. But you're almost making it possible for someone to be able to speak data science to a data scientist and really get across, here's what I need you to do. Is that fair?

[00:16:00.160] - Chinar Movsisyan
Yeah, it is fair. Plus, this is from PM side to developers. We can imagine the same from engineers to data scientists, data engineers or computer vision engineers to product managers. What we do, we collect data based on their specifications that we have from customer side, PM side. We train that model, we test out it on the test set, and we see 99% accuracy on the test set or whatever.

[00:16:27.240] - Chinar Movsisyan
But with us, after that testing, we say, "Hey, just link to the platform. It will tell you more about your potential issues, just in case. You're fine. Your test set was perfect. But just beware, maybe there are cases that you have missed because test case is always limited. Like, criterias are limited and you can miss some cases."

[00:16:48.950] - Chinar Movsisyan
With that, computer vision engineer, when they deliver that model to the product manager, says like, "Hey, you had these requirements. I have built that. By the way, just FYI, these are the other cases where if the model will face this, the model is not going to work, but it's not my fault. I had your requirement, I did that, and the model is working perfectly. Just in case, know that these are the cases, and If you think that these are important to the business owner, to the customer, we can create more similar cases. We can create those scenarios and incorporate back to the model and fix it." This model evaluation and data curation is all linked, and they all are on the same page.

[00:17:33.880] - Simba Khadder
Yeah, that makes a ton of sense. We're ever given, here's our accuracy score, here's our F1. It's like, if I'm a PM and someone's like, "Here's my F1, I can't really do anything of that. It's like, "Can you make it better?" "Maybe." It's not really useful.

[00:17:47.670] - Simba Khadder
Going back to almost like academia versus real world, it abstracts away so much that it's almost useless other than it's not bad, but that's all you get. But it's not enough, especially nowadays with what users expect. So yeah, it's almost like being able to... It's not evaluation, but it's almost like explaining via data what's going on both ways. So that a PM can explain, here's what I'm seeing that's wrong in data, and then the data is at the phone, here's how it works, and how accurate it is.

[00:18:16.910] - Simba Khadder
Something I'm very passionate about and would love to get your take on is collaboration between data scientists, data engineers, and engineers themselves. In your experience, what were the best practices you found for collaborating with other data scientists and engineers?

[00:18:29.980] - Chinar Movsisyan
You mean how they collaborate, what toolkits they use, data scientists and data engineers?

[00:18:35.050] - Simba Khadder
We had the business side and the engineer side. I know the engineer side is a lot of different titles, but I'm curious to what you see there with regards to collaboration. How is this collaboration work there? Did you learn best practices in your own work or just in viewing customers now on what works for collaboration between data scientists and all the engineers, I guess?

[00:18:55.540] - Chinar Movsisyan
Yeah, it's evolving. What I have seen with these user interviews and customer discovery calls, just talking to engineers, talking to other industry experts, they have a lot of solutions in the market. You can see whenever you Google something, you have some problem, and you want to use them to solve it. One of the problems I see, first is having a trust towards these toolkits to see how beneficial are those tools, how compatible are those tools, and how they can integrate it into their stack and everything.

[00:19:29.620] - Chinar Movsisyan
As an engineer, and I'm maybe a little bit bad example because I'm not an early adopter. I don't like using tools and everything. I'm really comfortable with my tools and everything always. But what I see there just trying to understand if that tool is worth, or let's say platform or open source tool is worth, to spend time on it, and it will be compatible to their stack and everything, and everyone will be happy.

[00:19:57.570] - Chinar Movsisyan
Let's say I'm a computer vision engineer I need something so I can just use that and be able to work with labelers, like people who are doing data annotation labeling, this feedback thoughts are coming. They want something collaborative all in one, and they can just use it to communicate everything to collaborate with other engineers, other departments within companies, or also they use a lot of third parties, the example of data labeling. Usually, companies, they don't have internal data labeling. They just have teams outside of companies, and they use it.

[00:20:34.660] - Simba Khadder
You mentioned a little bit that you have your own tools that you really like to use. What's yours look like? Out of curiosity.

[00:20:40.290] - Chinar Movsisyan
I'm not coding anymore, unfortunately. But what I have I feel like I'm happy with Jupiter Notebook. I'm very, very comfortable with that and some Python libraries. Recently, I moved to Notion because of my co-founder. I was just using sheaths and everything. Slack is fine. Never was able to switch to Discord, but I'm not a good example.

[00:21:03.260] - Simba Khadder
It's fun for me to see all this is an explosion of tools. But a lot of data scientists I talk to stick to the basics, and I think it works well for many people. I think it's just figuring out what's your core problems and playing around the stuff, making sure you know what's going on. But I think you don't need to throw away everything every year, where if you were to listen to a lot of things, you might think you have to. I think that you can increment till you continue to grow.

[00:21:28.750] - Simba Khadder
I think this is a huge focus now with LLMs coming into the mix whenever generative AI tools where people are like, cool, well, do I throw away our MLOps platform and all our models or not? My take is it's more incremental.

[00:21:42.410] - Simba Khadder
Most of the problems that you mentioned are about data curation, understanding how models behave without having to look at the weights, which you could see the exact same thing for LLMs and other generative models. I'm curious to how you see things evolving, generative models. You mentioned things like face detection. Nowadays, I mean, even before generative AI and before LLMs, there was a ton of off-the-shelf models. What do you think the future is? Do you think that there will be a lot more custom models? Do you think a lot of focus will go to pre-trained models and fine-tuning?

[00:22:12.340] - Chinar Movsisyan
In fact, LLMs are powerful, yeah. But before LLMs, there are a lot of applications where right now, I don't see just an LLM model can be used for drones, for satellites, for those applications, use cases, those all are related to computer vision stuff, imagery and videos, and plus computational limitations. We cannot use LLM on any edges like Raspberry Pi or Jetsons or these microchips. That's an issue.

[00:22:45.760] - Chinar Movsisyan
These foundation models cannot be used there. Still, we are going to have small custom models, maybe distilled from some huge foundation models. But again, we are going to have data curation issue, like smart data, effective data that matters to be used to do some custom training and use it our own specific problems.

[00:23:07.720] - Chinar Movsisyan
Having said this, I do believe that these computer vision-related applications still need custom training, still need data curation, data pipeline, or this infrastructure of overall development, data preparation, curation, and then model development, and model maintenance, whole life cycle.

[00:23:28.550] - Chinar Movsisyan
In terms of LLMs, and in fact, the other day, two, three days ago, I was doing a customer discovery call with a company in finance. They were doing this LLMs custom training to different customers in finance. What she was telling me that because of the performance issues, they're having customers churn.

[00:23:48.570] - Chinar Movsisyan
Every finance company, they don't have the same data. They have very different customer pipeline. Their users are all like normal people, but their infrastructure is different in terms of what information they are gathering from their customers' users.

[00:24:05.900] - Chinar Movsisyan
She was telling that because of the hallucination or not good results in terms of output of the trained LLM, their customers are churning. This field as well, there should be some data-driven way, explainability or something to be able to diagnose and understand, well, if this model is perfect for this use case, for this customer, how we can adapt it to others. It's all about scalability. How scalable is that solution over there.

[00:24:37.470] - Chinar Movsisyan
This is two different directions, but on top of this, we can see this multimodel direction, which is growing so well. ChatGPT-4 has this DALL-E module which is working pretty well. I don't know from business, B2C is obvious. ChatGPT-4, multimodel solution is being used. I use that as well. But how businesses are going to utilize that? What is that real business need that is going to be sold by multimodel solutions?

[00:25:09.430] - Chinar Movsisyan
I don't know yet. I don't have anything there, but it's evolving really exponentially and in the next six months, we will see some results in multimodel with these foundation models.

[00:25:20.240] - Simba Khadder
What are you most excited for? Because we're in this weird in-between zone right now, where we're seeing this huge explosion of a new paradigm. It's not new, but it's almost like the models have finally gotten so good, like our transformers have gotten so good that it feels different to use them than it used to. I think it's just creating a whole different type of application that just didn't exist before. But I guess maybe going to the question, which is, what are you most excited for over the next year or two? Where do you think the world's going? What are you excited about?

[00:25:54.350] - Chinar Movsisyan
In general, I'm really excited about this fast-growing, evolving tech environment and everything. Every day, it's a challenge, and you should be really fast because it's growing so fast. Every day, they announce a new feature. With that new feature, you get a new opportunity in the market itself. You can use it. It's very exciting. It's really cool to be in the field and also be part of everything.

[00:26:22.500] - Chinar Movsisyan
Because of my expertise, because of maybe my interest, I'm really passionate about this computer vision and multimodel things. I do believe that it's going to be huge. We don't see that much computer vision in the commercial applications in terms of how many models are in real usage production. But I do see that the market is growing so fast, like robotics or these AV companies.

[00:26:52.110] - Chinar Movsisyan
Every industry, they try to use it to streamline their processes, manufacturing, or house management or construction or mining, oil and gas, and everywhere. With this massive growth, I do see that computer vision is going to be really helpful, useful, and because of that, because of the demand, it will grow.

[00:27:15.040] - Chinar Movsisyan
Most probably we will see some multimodel, real business needs that will require multimodel foundation models, integration and everything. But I feel like LLMs are not done, but the usage we see, finance is there, and it's a huge, huge market and LLMs are being used. And without LLMs, I don't know how they will survive without this solution.

[00:27:40.380] - Simba Khadder
I mean, two things you said that I really resonate with. One is the energy and just tech has just changed, especially in AI. It's completely different from what it was two years ago. It almost reminds me of when everyone was building mobile apps, and they were all silly and dumb and super small, like a flashlight app was a big deal, but it was this really high-energy time.

[00:28:02.480] - Simba Khadder
Then I think things... It just changed. It was just different. It was just a lot more complicated products, a lot more focused. There was almost like a SaaS wave I came through. I think the focus on startups changed and the energy around it changed. Now it feels like it's going back to that hacker-focused mode, which is fun. It's really exciting.

[00:28:21.550] - Simba Khadder
Then I think the other piece is just already it feels like everyone's thinking about how AI and this new explosion of models changes their business, and no one's slow on it. Different companies are moving at different speeds, getting things to production differently. But I don't think there's really companies who are betting against it. I think everyone realizes that this is what's next. How do we integrate this?

[00:28:46.610] - Simba Khadder
It's also been interesting to see how quickly that point of how are we going to do this goes into, we just have to do what we were doing before, but even better, which is data curation, data cleaning. Well, it's just data and monitoring. Traditional machine learning, the hardest problems, in my opinion, were always data and monitoring, and LLMs, it's the same thing. The hardest problems are data and monitoring.

[00:29:10.070] - Simba Khadder
Awesome. This has been great. We've really covered a lot of different topics. Thanks for hopping on and sharing all your insights with us and the listeners. This has been great. Thank you so much.

[00:29:18.280] - Chinar Movsisyan
Thank you so much for having me. Great talking to you.

‍

‍

‍

‍

‍