---
title: "Navigating the FDA's Total Product Lifecycle Framework for Generative AI Devices"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/exb6zeowj1"
content: auto-caption transcript, proper-noun corrected
---

# Navigating the FDA's Total Product Lifecycle Framework for Generative AI Devices

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/exb6zeowj1)

---

So welcome everybody for, our first webinar of twenty twenty five, navigating the FDA's total product life cycle TPLC framework for generative AI devices NAI devices, I think, overall. And not a huge difference, in many of the things that are gonna be talked about today. My name is Erez Kaminski. I'm the founder and CEO of Ketryx. With me, is Jen Dixon.

I'll let her introduce herself. Hi, everyone. I'm Jen. I am previously a quality and regulatory professional myself. I worked on software medical devices, and I have experience in quality management applications and AI in general.

We placed the webinar link for our next webinar in the chat below. Please go ahead and register for that next one. It'll also be focused around software, medical device, all that good stuff, and we're really excited to have you join us for the next one. So long question we always get, when we move into sort of this webinar phase is, you know, when are we gonna get the recording for the slides? Like, are we gonna get, them sent afterwards?

You will be get this be getting the slides at the end of this webinar. So sit back, relax, and join the show. We'll also be asking questions, or rather looking for your questions. We wanna stay engaged and get a sense of what you're looking to get answered, and we wanna tailor this webinar for your, for your needs. And lastly, because we're based, we apply ISO thirteen forty five.

We are always looking for feedback and trying to make these webinars a lot better for you. So there'll be a survey, a very short survey that'll be sent out after this webinar, well, where we'll be looking for your feedback. Of course, if you have any questions at any point throughout the webinar, please, use your Zoom controls, and submit your questions. We have folks, also on the call who are looking to answer those questions, and, of course, we'll try to take those questions as well, if they're, you know, questions that everyone wants to know the answers to. Alrighty.

So everyone should have access to a poll. So what we'd like to know today is what are your concerns about integrating GenAI into medical devices? So I'm gonna give everyone about a minute to answer this question. So there could be lots of things that are bringing you here today. We really wanna understand, you know, what are the things that, you know, keep you awake at night?

What brings cause for concern when it comes to generative AI? I know lots of people have been integrating generative AI into their workflow. Sometimes, you know, they've been integrating into their products. They've been keeping up with the news. So we just like to know your thoughts on, like, what are your concerns, particularly maybe in medical devices?

What are your concerns about integrating Gen AI into your products? And I'll just give it a When I was trying to do this for the first time moving from tech to, biotech and medtech, trying to figure out how I'm gonna integrate AI and Gen AI. At the time, it was more called NLP, and then this kind of token prediction type generation. I started to think, wow. This is gonna be really, really, really hard to do because the complexity of systems that at the time ten years ago almost was developed in med tech, was much lower than the complexity of these type of systems.

And these systems just have so many moving parts, so many softer items, that I thought, you know, it'd be almost impossible to do, to make a, a single validated version. But then trying to change that version, would would make it not worth it. Right? Like, updating it will become completely impossible. And so pretty interested to see how people responded.

And, you know, maybe we're ready to share the poll, Jen? Yes. Yes. So I'll go ahead and share the results for this poll. Talking about what you just said is, you know, you spend all this time building this perfect device, this perfect design history file, only to feel like you can never update it ever again.

So taking a look at the poll, hopefully, everyone can see the results. It seems like risk management, change management, as you just mentioned, Erez, definitely top of mind for some people. You know, you build this wonderful device and you document it and you find that you just can't maintain it very well, and you can't update the documentation. I see also traceability is a concern. So, tracing maybe your requirements, your risk management file, your testing, building that trace matrix can be quite the challenge for some people.

And, of course, risk management now risk management can be applied throughout the process, in some processes, and this can be very challenging as well. Any thoughts there, Erez, on you know, is this what you feel is, like, the general consensus when you're reaching out to other medical device manufacturers? Yeah. I think a lot of my partners and and our partners when we work with them talk about the need for a better way to do change management, the level of, complexity of their traceability or risk management and end to end risk management, something that comes up a lot. But this traceability issue where traceability matrix now have hundreds of thousands of lines, for some products.

And that's just gonna get worse and worse with this because even for unregulated products that do AI because of certain risks, I think that people already have hundreds of risk controls and thousands of risk controls being tested. Now if you take that to the kind of end of a true kind of medical AI system that does have, you know, can cause substantial harm to people, I think that that would cause an explosion of risk, and traceability items. And then you'll layer on top of that that that AI engine will actually be used as part of a hardware system that is automated, that performs more actions. I think that there's a real need for for something to change in the way we build these type of validated products. Hundred percent agree.

And that's why a total product life cycle, process is probably the best way to do this, and that's one of the things we wanna talk about today. You'll see that, GenAI is coming in to medical devices today, you know, know, not just in our workplaces. Dexcom, a major public company, they make lots of medical devices every year. They're integrating GenAI into things like insights for your glucose levels. You know, how can you change the way that you live day to day to improve your health outcomes?

Whether this is for a wellness device or it's an add on to an existing class one or class two device you're seeing GenAI creep into you know low risk high impact functionality. But what you're also seeing is potentially you know this might be setting a precedent for higher risk applications. Maybe the way that these companies are, you know, building out total product life cycle management platform, the way that they're doing it is going to set the precedent for how we submit our devices, how we get them cleared with the FDA. Erez, do you think that, Gen AI is really different than previous AIs that have come before? I think that it's not fundamentally different.

It's a it's a difference of scale. There is more uncertainty in these large language models because you don't really know what they were trained on. Right? They're trained on massive, amounts of data, like the entire Internet. And as a result, there's a lot of stuff in the Internet that we wouldn't wanna interact with the patient.

And I think that's one challenge. Like, you have even less and less visibility into your supply chain. I think another challenge, it it will be harder to validate them because they are gonna do many more things. So if, like, you know, you have some machine learning device like the classic, I think first device approved by FDA in the nineties is some kind of, multilinear forecasting model. It forecasts some, biomarker or some endpoint, and then it allows you to assess given other things, drugs or or some other intervention where that biomarker is gonna be.

I think that's a really, really different problem than, if you need to answer kind of questions about your medical conditions in general or or do things like that. And so the use case is much more narrow. And with generative AI, the use case is very, very broad. And as a result, I think it's gonna be pretty hard to validate them. I think there's gonna be a lot of validation compared to, like, gold standards, which is already done in many other fields of machine learning, and other fields of medicine, and also a lot of risk management.

I think, like, in generative AI models, risk management will weigh will play in an an oversized kind of portion of the work. For sure. And I think you've mentioned before that Jane AI is gonna be big. It's gonna be really big, particularly when it comes to making sure that we serve enough patients out there, and it expands our scalability. Right?

Yeah. It's it's it's funny because we were talking about it the other day as we're planning for this of, you know, there is a issue coming where the population is growing so fast compared to our ability to train providers, and to make things for for those providers. And so we need to find a way to give people in rural areas or not in rural areas just access, to more and more medicine. And I do think it's a great technology for that. We just need to make sure we're not telling them the wrong things.

And and we already know that even in kind of computer games or just chatbot applications, there is substantial harm being caused. There is, recently last year, I think, there's a a lawsuit going on about a young boy who killed himself. It's very, very sad and tragic for the family. And now you're thinking what happens when that is, that recommendation is coming from a medical device, not even just a game or a chatbot, and and what would be the result of that? Because at at the end of the day, in some ways, these generative AI models, right, they're just really, really good liars, and they make you feel like they said the right thing, and it's very hard to understand if they did or didn't.

And we need to make sure that people are doing that in the right way using all the modern techniques like rag and and and so on, to make sure that you kind of understand what's going on. And that's part of the risk management. Like, are you actually referencing, the right kind of retrieval, when you generate? For sure. And in the medical device world, you know, we apply something a little bit different than the gaming world.

We apply a total product life cycle approach. And especially if you're working in software, you should be really familiar with ISO, IEC sixty two three zero four. Is that right? And I think, Erez, you have been thinking about this standard a lot, over the past couple of years now, and you have a really firm mental model on how it should be applied in a total product life cycle approach. Yeah.

And I think that it it started from, the time I moved into biotech thinking of how to do this without knowing really the terms. And then I met a a gentleman who now works with us called Paul Jones who, led this kind of work at FDA for about twenty five years. And, me and him together, I think sat and thought, what what does this this really mean, and how would this be applied widely to AI systems and other automated systems? So if we go to the next slide, I think we could just, start talking about it. And so here is an example of what kind of what, a product life cycle would look like.

One of the points I always like to highlight is the supply chain. This is very different than how it is gonna look, for, like, a a hardware device because there are so much many so so much more components, that are used as part of software. Right? Most software system today that are web connected, apps, cloud systems, AI systems have thousands of supply chain items, what's called dependencies or soup in six two three or four that are used here. And what we're gonna do today is I'm just gonna give a really brief overview of the different steps here and phases and then tie them a little bit to the best we can to IC six two three or four and the new, kind of TPLC guidance.

So if you think of just the process of, product design, right, you decide to launch a project, you set a bunch of requirements, use case requirements, you start thinking how are you gonna fulfill them, You know, that's IEC six two three zero four section five. Right? Like, understand the requirements. And, of course, this is part of kind of broader standards like, ISO, eight two three zero four and six zero six zero one that deal with PIM systems and all the different parts. But, you know, six sixty three four is a pretty good place if you're building a a software only medical device.

And then, you know, for AI, we'll talk about model description, how you develop them. Right? So the requirements and system requirements and use case requirements eventually translate into specifications. That's a model, for example, or data specifications. You need to verify and validate them, not just the software now, but the data and the model.

So there's many, many more steps here. Then you need to show some kind of equivalency or go through a full clinical trial, get into the supply chain aspect, bring all this information in from external suppliers, into the specific variants, which are the the purple line here. We had a webinar last month that talks in more detail about these things. And then we have kind of the manufacturing services and ops. So manufacturing CICD pipelines have audits as as much we need to launch the product.

And eventually, you launch particular variants of the product. Right? It has to do with different regions, or or different things of that nature. And then you basically get into this post market surveillance step. Of course, I I slightly clicked earlier.

I forgot the risk management that's here, part of this, product design. And then at the end here, you have this need now that the product's in the market to understand what people are complaining about, how to modify it. Another form of modification will come from the fact that maybe you'll discover cybersecurity dependencies, in that, in those kind of, in that supply chain. Right? You have a cybersecurity risk, vulnerability.

You now need to understand that, analyze it, maybe change, and then make a new version. And And that's section six of sixty two three or four. And in this kind of guidance, it's talked about device performance and monitoring. So a lot of work to do around that. And then I think yeah.

And I I've loved the way that you mapped it to sixty three or four. Sixty three zero four, I think, is, you know, a standard that people love to hate, but it really does provide a firm structure around how you can possibly validate software in all the different ways that people build software, whether you're doing agile or waterfall. And I really love this diagram as it steps through all the different processes. And that loop back is a reminder that this is a total product life cycle approach. There is feedback for us to improve the product over time.

And this this whole product life cycle approach is something the FDA has always talked about. They talked about it recently in their executive summary. So if we just hop on to the next slide here. So they've always advocated for a TPLC approach, and there's no reason why Gen AI doesn't work for it as well. So they're thinking of, you know, your TPLC, the way that you create software might be what you need to show evidence that this generative AI product that you're developing is safe and effective.

You can cite those procedures. You can cite your practices and say that this is how we are planning to not just create a safe device, but maintain it over time. With the, total product life cycle approach, there's a lot of steps that are very specific to Gen AI if you wanna step forward another slide. So AI, Gen AI, there's a lot of different steps here, and I'm gonna step through each of them and talk about each step. So planning and design, obviously, the first one, you should probably know what your project goals are, the scope of your product, your requirements, your algorithms.

You should know what are the possible ethics of gathering certain information. Set that up front. You don't wanna think about ethics definitely, you know, two years, three years down the line. You wanna think about that straight away. Next step.

And then if you hop on into data management and collection, if you just click one more time. So gather your data is going to be very important. Data is cake. So you need to organize your data. You need to quality control your data.

What you wanna avoid is a situation where, you might have developed something in r and d and you know you've gathered these online data sets and then you sort of like renamed everything and you've lost control over or lost sight of like where did all this data come from and when did we use it? Did we use it during training or do we use it during tuning? Let's avoid using it also for validation. Otherwise, we're gonna run into overfitting issues. Next.

Of course, you need to develop and refine your AI model over time. You will likely do internal training so informal testing training that happens before the VNB cycle. So this is your developers checking to see have we even started building something that is effective before you hand it over to your quality engineers to make sure it actually is meeting all the requirements of the FDA. So that's hyperparameter tuning, optimization. When the FDA is thinking about total product bicycle approach and you say that you have an AI device, these are the questions they're gonna ask.

They're gonna expect these sorts of buckets of work to be happening when it comes to AI specific technologies. Next. And, of course, the big one, the one I think we all worry about all the time, which is verification validation. This is where we end up spending a lot of time, a lot of money, a lot of, expertise to assess whether the models we've created are actually accurate. So sometimes they're not super reliable, and we need to really prove that they are.

So you need to have firm validation strategies for how you're going to do that. Next. And our we're actually gonna go through model deployment and operation and maintenance because I think we have a question here around this. So, obviously, when you're deploying the model, you should know upfront whether you're going to be changing the model over time. So is this adapting in production?

Maybe it's a personalized algorithm that's working here, so it may need to learn over time. You wanna build that as part of your design, and part of your design should include, you know, what are some of, like, the the trigger points? What are the alarms for whether this model is potentially falling out of conformance? No longer in specification. So as you're deploying that, you wanna keep that in mind.

You also wanna make sure that, you know, it if your model is susceptible to performance issues, so maybe it hallucinates more because of poor performance or maybe there's not enough, memory or whatever it's running on. You need to consider that in deployment because it will impact the performance. So with operation and maintenance, the next step, we also wanna track real time performance. So we have a question here. So not all deployment options allow easy access to field data.

That is true. Sometimes, you know, we have requirements saying that we can't retain some of the results here. Right? We can't there has to be a memory. We don't want to, you know, use it for retraining.

There's no logs. Further, there may be contrary risks from a data privacy and security perspective to collect field data. Right. And it could be also an an embedded device that you just don't have access to this information. Right?

It just does things at the edge. Exactly. And, you know, how will the FDA FDA navigate post market requirements since obtaining additional training data may be limited? The FDA is very aware of this constraint, and they know that they don't wanna step into the realm of, you know, telling individuals, patients that that data has to be sent back to the manufacturer. They don't wanna take that away from you.

You wanna they want you to feel like you own your data. So the FDA has been pretty, nonprescriptive about what needs to happen here. Obviously, they want you to show evidence that you're mitigating risk as far as possible, which may include, you know, sending some data back to the manufacturer for retraining. But really this is up to you to make that case not necessarily, they're they're not gonna start saying hey you can expect this from patients. So I'd say the FDA is navigating it very cautiously.

For now, they are not, you know, stating a very hard requirement that you have to do this, but they might come to the assessment based on reviewing or submission that you should. So, as always with regulatory, the answer is it depends. And I think it'll evolve also over time as how the agency responds to that and helps people. But I do think there's also a lot of companies who are thinking, you know, in order to really do effective post market at scale, I do wanna wait to log this information and deal with it and monitor it, understand what I have in the market and also what kind of interaction the stuff I have in the market has with patients. Exactly.

Exactly. And that's also a part of it. It's not just a requirement of safety and effectiveness. It's also a commercial desire because the more information you have, the better your competitive advantage will get because everybody is gonna use and is using the same type of foundation model. So it's always about the data.

For sure. And what I would love to see is those really large manufacturers really set the standard of what that looks like, really set the best practices and what that post market real time feedback will look like. And for our last step with real time feedback so we want to continuously assess the effectiveness of our model you know through KPIs. You should have a firm sense of what these are and typically these might get included in your submission. So compare against some sort of baseline maybe you have comparative testing as part of your design history file as well.

Alrighty so today we're going to focus on just two of these. These two if you'll next slide. Yes this is sort of where I think Gen AI becomes most challenging to implement it particularly if you're a quality and regulatory professional so data collection and management might be something you aren't used to doing a lot of like maybe just a little bit but not as much as you would for an AI model. And lastly, verification validation. So let's dig into a little bit more.

Next slide. Alrighty. So if your design developers, developers are coming to you and saying, hey. We're gonna build an AI model. First questions you need to ask are, well, how are you acquiring the data?

Where is it coming from? And, you know, are you changing the data before you feed it to the model? What format or protocol might be used? In this case, you know, with generative AI, it could be prompts, it could be, you know long strings of data it could be JSON. You really want to know that upfront as you are collecting your data.

Where are you storing it? Is s3 the right place to store it? Should it be somewhere else? Do you have the rights to store it in that location? What prevents someone from accidentally coming in and adding or changing things?

Because these are not questions you will want to ask you know years down the line after you've used this data. Traceability. So is there a clear version history of this imaging data or this prompt data? You know, maybe you changed it a little bit. Maybe you, you know, preprocessed it and it's part of that model design.

We wanna be very sure. Right? From from raw data to what you're feeding the model, how has it been changed? How has it been cleaned? And are you using it for training, for tuning?

So not V and V yet, but rather, you know, still in that early informal testing design development process, or are you using for V and V? Because one of the first questions the FDA is gonna ask you is, have you reused your data at all? Have you made sure that the AI model has never seen this data before? Right. That's like the number one, kind of false step thing that happens that I've seen happen again and again and again.

There's people have in their test set, information and data from their training set. I think another thing just about data traceability is, like, does this data actually trace to the claims you made about the device and its population of intended use? Right? That's a big, big challenge given where most devices, you know, usually are deployed to, like, coastal regions, San Francisco, Boston, New York. And then, like, does that really represent the patient population you're trying to serve and that you've made the claims against in the label?

Exactly. Exactly. Which is one of the reasons why the FDA is really highlighting data traceability, you know, making it public in your five ten k summary or whatever, submission pathway you're going through. They're seeing data and information about the AI model is empowering. Lastly, about labeling and annotation.

So likely, if you're using, you know, a medical, you're using this for a medical application, you're probably gonna have labelers for the data that goes in. So those labelers, whether it be a clinician, an MRI technician, whoever you're trying to replicate in terms of intelligence, that person, their competencies will shape the way your model will perform. So they are absolutely essential. Understanding their competencies is essential in understanding how effective your model will be. So they'll want to take a look at that as well.

And how yeah. And I think how the data is annotated. I remember, I wanna say it was, like, eight years ago or or nine years ago, there's a real big problem in ImageNet, which is a big dataset for, images that is used all around the world to train a lot of models. And people found that, in images at ImageNet, there are categories, because it's open sourced, that would vary kind of, racist and very, wrong as well, completely just wrong. Like, people injected, kind of poisoned the data, injected wrong information.

And and there was a lot of efforts kind of from the open source community to clean that up. And some of those things that I was aware of at the time and I remember learning, I'm very, very happy or not making it into a medical device. Exactly. And you wanna be able to show up to the FDA and say, like, we are a hundred percent sure that none of that has ever happened with our data, and this will be a safe and effective device. And I think with foundation models that read the whole Internet, you'll never really, really know.

But what you will know is that you've done the best you can and have created safeguards along the way. Exactly. Verification validation. And that's how we're going to address those foundational models, right, with our with our testing. So one of the things the FDA will wanna know is how have you evaluated this model?

Can you tie it to the intended use as you were saying, Erez? You know, can we show evidence that this is really meeting the intended use and, you know, you've taken out as much as possible from any potential risks from the foundational model. As you hand this model to your clinicians, are they able to interpret the results? Are patients able to interpret the results? What we'll obviously, if, you know, a clinician finds the output of a Genii prompt is, like, completely hallucinatory, they're not gonna rely on it.

They're not gonna wanna use it. And it's really important that these models are not distracting. They're actually empowering, and they're they're gonna elevate the amount of care that and and care quality that you're going to receive. I think we have a question from, our q and a section. So you, you said for medical use of GenAI, if the model is validated only for one or a few specific use cases, what would be the effect of risk control measures that can be used to prevent the model from being used for other use cases once it's released into the market?

So it sounds like what you're talking about is how how do you make sure that you know you validated this and protected against potential off label use or, protect potential foreseeable misuse. This is something that you wanna definitely embed into your risk management file and have somebody addressing, that potential use. And it's really important that you build risk control measures around that. Now, of course, this is very contextual about, you know, what is your specific use case. Case.

I would at least say that, you know, if you're gonna have Gen AI, you should try to implement quality control metrics. So some sort of way to, you know, sort of like stop gaps or breaks in a way, safety breaks that would assess whether a Gen AI is maybe used for the wrong purpose and you can flag that to the end user you know label it for them say like hey maybe you might be using this for the wrong purpose that it's not been validated for please be careful or maybe just it's not recommended that you use this. Those are just some potential ideas of how you do it. Of course, as always, it depends, how you wanna embed it into your risk management file. Applying ISO fourteen nine seventy one, of course, is the way to do this.

And then I actually see one question here, that is we do wanna take right now is does FDA accept pretrained models, example, ResNet based on, ImageNet dataset? Of course. The FDA has accepted many kind of devices that leverage existing data sets and existing, pre trained models, what's called today foundation models. And, you know, it's part of devices. You can't really develop, most AI systems without having, kind of a lot of access to pre trained models.

Exactly. And, you know, they're not about adding an undue burden, to the way that you design developer devices. This might actually make it go faster or better, right, to use this foundational model, this pretrained model. And they cite that a lot in the premarket submission guidance. Alright.

Coming back go ahead. I think one of the things to say about this about the verification validation and all the traceability of that is, I think it would depend a lot on the particular case. I think, for example, for this question about foreseeable misuse and off label use, you know, sometimes you will need to put in controls. Let's say that if the AI feels that you're asking it a question that's very far from what it's good doing, it should say that and say, I don't know how to answer that. That's the wrong use.

I think there should be warnings in the user interface and things of that nature. And, overall, this is gonna be an evolving field that is gonna, depend on on the product and the company and the scale in which they operate and the level of scrutiny in which they operate under and, the risk of the product, of course. And I could tell you that even low risk or non risk kind of use cases, consumer, applications like Anthropic and OpenAI, if you listen to a bunch of their talks and know some people work there, you realize that they have, massive traceability databases for risks with hundreds and hundreds and hundreds of tests, possibly thousands now. They're trying to address these different questions and making sure that they're not doing something, that's totally off, including, human testing kind of, UAT for, for for prerelease candidates. I love all this talk about risk.

Risk is super important to how we develop medical device products, and we'll we'll just touch upon that in a second. Just wrapping up this verification and validation piece over here. One thing that the FDA has mentioned is that, you know, you wanna do potentially both standalone validation of your model alone as well as validation of the human device team. And this is really key. This is where you're gonna show, hey.

Our device, when you give it to a human, the combined performance is better, and that's something you can definitely showcase. This is your benefit that you're introducing with your technology. And, of course, you wanna document all of this. You never wanna miss out on the opportunity to document your V and V. Every single detail counts, and it might be something, that will come up more in the submission process.

So make sure that you have a strong quality management system that makes sure that, you know, you're meeting all the requirements around it, and, of course, you're connecting it to risk management. So you should see risk management in your V and V cycle and a consideration as well. And, Erez, I know you love to talk about risk management. I think of large part because you've been contributing to risk management, particularly in the ML application. Isn't that right?

Yeah. I, thanks for mentioning that, Jen. I was actually working on a, on a technical report that, supersedes this CR three four nine seven one the FDA mentioned in their guidance called TIR three four nine seven one applications of ISO one four nine seven one to machine learning in artificial intelligence, applications. And I think it's just very challenging. Right?

It's mostly that you're doing the same things. There are just many more risks that we're not as aware of similar to the first time an engineer walks into a medical device or pharmaceutical environment or any kind of high risk environment. They try to understand what it you know, how to do all this risk management. Now there's this next step. There's many, many risks that we are not used to.

There are not great tools to, manage that risk. And those risks will involve substantial amount of testing and human review, and all kinds of trials to make sure that we are managing them in a way that's appropriate. So, that's kind of why we wrote that technical report. I'd say that it's mostly a guide to many different ideas and a way to think about it. I think, the previous one from twenty twenty two that FDA mentioned, c r three four nine seven one is also a a great one for that.

And it's just trying to be a database of people to start for for professionals in the industry to start reading and understand, okay, here's the type of risks that people think of with these devices, and how do we now create procedures, tools, systems, and and teams to address those things systematically? Because they're just gonna come up over and over and over again. Yes. And I love how much information there is now about how to actually produce medical devices with ML and AI. I remember just a couple of years ago, there was not nearly this much information or even concern from the FDA.

And, you know, if you look back in time, you know, twenty nineteen is when we had one of the first guidance has come out about AI and ML for Sam Ds. And a lot of that was very aspirational. You know? It was like, yes. You know, you should be doing all the things you were doing before, but just consider some specific things.

And now we're all the way, you know, to twenty twenty five. Just two weeks ago, the FDA has put out the most prescriptive, descriptive way to actually submit these AI ML devices. And we've really come a long way. We're getting to the point now where, you know, there shouldn't be any questions anymore about what you should be doing and how you need to ensure safety and effectiveness of this device. I mean, the FDA has done a really fantastic job, engaging the community, bringing experts into the FDA as well as consulting with industry to understand what are the best practices out there, what can we tell our medical device manufacturers, whether you're a start up or a large company, what you need to do to get this out here.

So it's been really fantastic reading through all these guidances. Of course, PCP was a big one. If you're a regulatory affairs professional or you're a designer, and you haven't read the January guidance yet, we'll have a lot more information I think actually on the next slide about this but it really is a fantastic guidance because it really tells you what you need to submit and which section of your submission it needs to go in. So a couple of highlights are, you know, surprise surprise, implementing a robust QMS can be great evidence to show that you are, you know, building a device that is safe and effective. So implementing that Love how they they always position that, Jen.

It's a it's a great idea to do it. It is, like, not a recommendation. It is in the statute, but, they always recommend it again in the guidance. Yes. Yes.

And, you know, you might be submitting a five ten k, and it might not be required to submit maybe your QMS documents on, like, a PMA. But go ahead and do it anyway if it's what you need to really show that evidence in that case that you have, you know, a safe and effective process to update and maintain this model. Documenting your data management and your model performance in your submissions. So, you know, we just talked about how data management is super important. Everything from the raw data to how you preprocess it, those are all things you wanna trace and keep track of, and that's something you'll want to submit into your submission, obviously, in the software section.

And, of course, you wanna have a clear understanding of model performance, whether it's, you know, version one of the model, version five, your production equivalent model. They wanna see all of that. One thing they make extremely clear is that, you know, if you are using AI in your device, you should disclose it and disclose it in a very public way. Previously, manufacturers didn't necessarily have to do that, you know, if it wasn't very, like, important to or subject to the the submission, you know, it might not be the most important thing to disclose on the public summary, but now you do. And this is good because it means that we all know where the AI is.

It's not a surprise, and we can keep that in mind as we use the products and we see it performing within the field. One somewhat surprising, thing that they put in this guidance is, you know, requiring more user interface details. So we've always had to submit user interface details and labeling requirements to in our submissions, but they try to really highlight it as a form of context gathering. So they understand that AI has applications in many different ways and one of the best ways for them to assess whether it's you know put in a good way or it's you know aligning really well the intended use is to get, you know, your UI described. And a picture is worth a thousand words.

They really strongly recommend graphical representations of this. So if you have a really great UI for AI, they wanna see it. Lastly I think, Erez you might have talked about this earlier but cyber threats. There are AI cyber threats. We haven't seen a lot of them or heard much about it but when you have the intersection of a very adversarial field like cyber security with AI, which is a very unknown field, crazy things can happen.

Right? Yeah. And I think there's all kinds of ways people pack the AI system to, to just tell you stuff that they shouldn't tell you. Right? Like, that's one issue here, when you're retraining.

That's, I think, a big concern, or all kinds of ways to try and hack, to get into the database behind the divide the AI or make it do all kinds of, wrong things. So we're just in a kind of a modern generation of, you know, same way with infrastructure. When infrastructure came up and became really, really widespread and just robust, people found new ways to hack that. And now there's gonna be new hack ways to hack these things. So the FDA is definitely asking about that.

And, of course, you know, they wanna see how you're gonna maintain it over time. If you have pieces to be, make sure you implement it. Make sure you have post market surveillance planned out. You know, one of the key takeaways here I think in the executive summary for GenAI is that if you have GenAI it may be, you know, a major point in your submission. So if you have an existing product and you're gonna add GenAI to it, I think a lot of people are considering this right now, it may require resubmission and that's something you should definitely assess.

And of course a risk based total product life cycle approach is a good way to support a case of safety and effectiveness. Yeah. And I think I just wanna say before we go on is that, there's also a blog posted by you today, Jen, I think about the new guidance, how it you know, what it means for folks, a much more kind of detailed overview in some ways than this webinar. So I welcome folks to go read it. Reach out to us, if you wanna talk more, and and we're gonna do a whole series just on that draft in just a few weeks, to talk even more about this.

Yes. I'm very excited to see what the latest submissions are gonna look like and very excited to engage on that. Lastly, as we wrap up Gen AI, so one of the key takeaways from this executive summary is that, they're very well aware. Hallucinations are a major issue when it comes to generative AI, and it's gonna introduce a lot of uncertainty into your device largely because you're introducing foundational models. Right?

So you don't technically have a clear history of how this model was developed, but you're gonna use it for this new application for the benefit of patients. So these foundational models, might warrant more focus from our regulators. And this all ties into, you know, your quality management system and how you balance that supply chain uncertainty. So you this unknown training data, this unknown design history file, you wanna balance that out with, SEWP and OTS validation. So effectively, you're just going to make sure that you are testing it more if there is more uncertainty introduced into your supply chain.

So you want a very comprehensive VNB strategy that addresses all these potential risks such as hallucinations. Of course, model drift is also something you need to consider. These are all things that, you know, you should be embedding into your verification and validation process and into your v model. And I think, Erez, you have a little bit of a demonstration for us about walking us through the v model and how you can potentially do this for generative AI. Yeah.

And so we're just gonna see how this is done. And, thanks, James, for that wonderful overview. I think it's funny that a month ago, we were sitting and had a webinar talking about, kind of AI TPLC. Then two weeks later, this guidance come up. We wanted to do this webinar about GenAI in general, and it it merged immediately with the previous, webinar kind of, very synergistically.

So that's pretty fun. And so thank you for that wonderful overview about GenAI, GenAI use in medicine, the new guidance kind of touching some points, the TPLC. And, again, our last webinar is a much deeper dive into the TPLC, and and it's gonna be the beginning of many webinars, about different aspects of the TPLC as well as material where, as far as I know, the world is a kind of first and only total product life cycle, management platform. We allow people to then operate across different systems in their total product life cycle and execute that and generate evidence automatically as a result of work, with a lot of automation and a lot of AI, and a lot of other wonderful tools. And, again, depends on the level of risk you wanna go.

You can turn those things on or off. You can have different grades of risk. You don't have to use our AI. You can we can custom build your own AI to your company and things of that nature. So today, I wanna talk a little bit about, how the TPLC looks, and then I wanna do a demonstration and show you how to do that in the next ten minutes.

So, kind of a fast demo about how to actually do this in practice and how we've helped companies from Fortune five hundreds to start ups, do this and develop, and release software kind of on a monthly, weekly, daily basis. And and we're pretty excited about it. We just hit twenty million patients, kind of are impacted by products we we work on or our partners work on, and we help them deliver to market. And, seeing this combination of the TPLC approach that we've been talking about for many, many years and trying to promote with the PCCP, I think finally, a medical device manufacturer and also, kind of people developing GxP software have the toolset they need to go out and work as if they work in a tech company and do this really, really fast and effective, but still with the level of safety and risk management and validation that, we expect in our medical devices. So going back to this diagram, this diagram, is kind of now we talk about it all the time.

How do you start thinking of the total product life cycle and doing different sections of it? Right? There are many ways to do kind of product design. Frankly, there's, in an average kind of machine learning tool chain, you probably have ten to twenty products even in a small company to do just this part. There's then many other tools to do the other part, the other parts.

And what we wanna do is sit across those tools and help you generate the evidence in the middle. And so if we think of just kind of the pre and post market of of the AI life cycle. Right? So just not really, just generative AI, but AI overall, but also generative AI. You could see this diagram on the left.

So you have some AI application. It's ready. It makes predictions, and we monitor it. And then, you know, as it makes predictions, it generates new data. Right?

So we have now new data because it's making more and more predictions. We then wanna train that data. We wanna split that into a test set and a training set. We then use the training set to train a new model every software, every day, every week, every month, every quarter. Whatever works for your company and your particular application in deployment mode.

We do know that a lot of companies wanna move faster and faster. And I think kind of two weeks is is becoming the standard for leaders in medical devices now. And then we wanna start, testing it and seeing if it works well. So we test the the dataset. We see if that works, and then we, if that alleyway, to to see if that kind of if it's passed our metrics, which are really the PCCP metrics that people talk about often.

And then, after that evaluation, we need to go and ask, is this model really ready for the next step? And if it does pass those metrics, we replace it, right in production, and then we go through this over and over and over again. And there's all kinds of evidence that folks expect you to have. For example, pre market evidence for transparency. They wanna see, you know, do you know what your training datasets are?

Do you understand your model architecture? They then want to see a robust, performance and kind of metrics. How are you gonna measure this? Right? And those performance metrics, by the way, are the same metrics that you can use in your PCCP.

Right? Those critical quality attributes of how you evaluate if PCCP went well. And then you would, during post market, after you launch the device, they wanna see evidence of how you continuously evaluate it works and that you don't have a drift and that you understand what's going on and how patients are using it. And then, of course, you wanna have strategies for managing the regional biases. This is something that really kind of pains me because it is well known now that a model that works in one hospital doesn't necessarily work in another hospital for many different reasons, population, types of devices used, the way, physicians and other providers practice in that hospital, in the same way that it's known that, a model that's used to do something in a drug manufacturing facility or medical device facility works for one line, but doesn't necessarily work in the same way for another line making the same exact product.

It's because they're so tied to what's going on and and the specifics of their training, that the FDA wants to see that you understand that as well, that when you're serving patients in in Kenya or you're, serving patients in in Japan or you're serving patients in Europe or you're serving patients in Chicago, you understand that kind of challenge. And if we kind of break down that life cycle a little bit further for generative device devices, you'll see that in this case, instead of just training set training data and training your own, you have kind of, human who human led data generated, reinforcement learning with human feedback data. You have a foundation model that you're consuming as part of your app. You still kind of wanna, fine tune that foundation model with your data, evaluate if it works well or not. And that's really in many ways a v model.

Right? I have some system requirements I I wanna meet, whether that's my patient population or my performance or the things I'm trying to have it do. I then have a data set that has certain specifications, again, tied to the system requirements, the population, what I'm trying to do with it, the disease area. Then I have a specific model I've created that has a certain performance characteristics, and then I train the model, of course, verify that the model is correct. And, again, in machine learning, it's confusing because machine learning, they use the term validation for this thing.

In medical devices, it would be called verification, but, it's all basically the same. You test and you see that they're, meeting certain, standard. It's not like design validation. That's not the meaning of model validation where you're testing it in the intended use environment. You you also wanna verify the data is correct, of course, and then you wanna validate the entire system in the intended use environment conforms and can consistently meet your use cases.

And then you wanna do all of that again and again and again very, very, very fast. So you deploy it. You have an application, online prediction, and around and around we go. For references and most kind of AI startups, I'd say this is something that's done, probably every night, if not every week, or or every week. In in many large companies like, I don't know, Netflix, this is something that's done every hour, for thousands of applications.

All the time, they keep updating. CICD. We have a whole webinar about that from last year. That's great. And now let's talk about the demonstration we're gonna give in the demo.

So what we're gonna do today is we're gonna see how this is actually done in practice, for both Fortune five hundred companies and start ups and growth stage companies, by just making a device today together. So what we're gonna do is we're gonna have this, llama model. This is a meta model, I believe, a generative AI model that people use. And that kind of foundation model, we're gonna fine tune it. We're not gonna do everything, but we're gonna do kind of, I'd say, the QMS part of things.

And we're gonna incorporate that within another, device. So we're gonna show how that's part of some health insights platform. And the health insights platform, the system of systems, is where you have the system requirements, the risk test cases, and test execution for the top of the v model. And then there'll be specific things like that risk, test cases, test executions, design inputs for the bottom of the v model, this particular kind of you can think about it as a software item or a subsystem overall. You're then gonna be allowed to version the model independently.

I know there was a question earlier about how do you kind of update. That's what we're gonna show here. How do you version this independently? Keep moving, but then keep having a a system, that's updated and maybe deployed to different regions at different times. There'll be different types of, engineering controls applied to each, model.

Engineering controls are ways to enforce compliance and assure quality, through kind of, either physical or cyber physical ways. You know, in a pharmaceutical factor, it'd be maybe the size of the tube that, you know, takes some drug product from one place to another, and that limits the flow rate, the combination of the that and the, you know, kind of the pump. In our case, in in kind of computer land, that would be all kinds of validated controls that prevent you from doing things. One of my favorite ones that we do is we can connect into your CICD pipeline. People love this, especially at scale companies, and prevent you from releasing software that does not have all of its risk controls mitigated or verified.

And I think that's, pretty exciting. So let's, jump in and see how this is done. So we're gonna go into Ketryx. You can try this as well at app dot Ketryx dot com. Maybe someone can pop a link into the chat just if folks wanna try this.

And in in here, you can see that at Ketryx, you have projects. You can create different projects through the screen, try and do different things, create, connect to Jira, select the v model, connect to other systems. Many other systems now can be connected to Ketryx. And what we're gonna start with is going into the subsystem. So just as a reminder, right, there's the subsystem, this medical llama model, and then there's the top level product where the intended use is the actual product system, this health insights platform that is consuming the subsystem.

And you see the producing v one point one of this system is in fact a milestone that is required to release version two of this particular system. So let's go into this, and let's talk a little bit about what we see here. We see on the left all the different modules of the metrics from SBOM to risk management, risk control, traceability, graphing, all items. In the all items screens, we can see per sixty three or four all the configuration items of the system. We can just go into one of them like a m seven, a I m seven, and we see that we're now going into Jira, the system of work where people actually work.

In Jira, Ketryx provides you with traceability, means of approval. There's a part eleven compliant pallet here, and full audit history, including a very nice kind of traceability widget that can tell you what is traced to what and how things are connected. So this particular, requirement is, a child of multiple other requirements. It's actually a risk control as well for risk. We can see that here.

It is fulfilled by certain specifications and tested by others. Some of them are, again, in other systems or in Ketryx. As you make modifications to this item, that all is tracked in Ketryx. We can see here the entire version history and then Alcoa plus plus kind of audit trail. We can change version, update versions, lock versions, and then we can see all the different modifications that ever happened, all the different items that are generated based on this, so on and so forth.

We can then kind of look at many, many different items all at once. So if we look at the all items screen, we can see here that we have kind of tons of different items, some of them coming from Git. We're gonna explore that in in a in in a moment. Some of them integrate with, with Jira, come from Jira like we just saw. And we can ask all kind of questions, like, show me the difference between, version one and version two.

Show me all new items, all removed items, all the same items. We kind of see that there's not a lot of changes between version one and version one point one. That's because we're just trying to show kind of how this works. Show me all the risk controls. Show me all items that are missing tests.

Well, all of them have tests. These are all life cycle items. Right? This is all from IEC six two three zero four? Yeah.

Exactly. And it could be accommodating of your particular quality management system and life cycle process, but you see you have here requirements, software items, risks, test cases, and so on and so forth. And then we can go into the traceability screen to see how the traceability works. You can see here kind of that, we have certain design inputs, in this case, like a on AIM seven, transparency and auditability. That leads to many different specifications or design outputs.

For example, we're using a base model called Lima three eight eight b, eight billion parameters, and we can see and get all the different aspects of that. And now we can test and make sure that this is the right one. It does what we want. We can also use this to store information so we know exactly, where it's referenced, and how it's all connected, right, to make this very uniform and cohesive, and make it easy for people to understand what's going on in an audit and move very, very, very fast. I think this is super exciting.

This is the only way to really make massive, AI systems, complex surgical robots, and then release them at the frequency that, patients, providers, and our industry, expect. And it works, which I think is a very, very, exciting thing to see over kind of the five years we've been building the system. You can see here kind of the we can go from here to see the different gaps in traceability. In this case, we don't have any traceability gaps. But if we had traceability gaps, even with a hundred thousand rows, it would still tell you what the gaps are.

It would allow you to modify things, understand how things are connected. For example, you know, what design inputs have to do with the dataset specification to continue to explore this. We also see that we have a lot of not just manual tests, but automated tests that come from Git or come from other kind of part of my CICD, so I can run my tests. As they run, they create evidence of working, so you can run them every day as you release and keep working. We can see here the different risks we have.

In this case, we only wanted to make one risk to make it really simple. So we have one risk. That risk is about model hallucinations leading to inaccurate and harmful insights. We see that the it's an unacceptable risk because of the way we configured our, risk management matrix, which all could be configured to a particular organization's needs. We then have different risk control measures.

Some of them are our requirements, like we see here. Others are specifications, like we see here. Fine tuning the model is a risk control measure here. And then with that, the risk does become acceptable, so we're able to release. And if the risk is unacceptable, an engineer control is we can require you to have a benefit risk analysis, which is exactly like one four nine seven one requires.

We can then see here a list of all the risk control measures across my system. Again, it's it's relatively short list, because this is a a demo project. And then a really cool control, and someone asked about CICD, is you can actually integrate the testing of this into whatever CICD tool you use, Jenkins, GitHub Actions, GitLab, CircleCI, and many others. You can read more about that in our documentations, under API. Maybe someone can share in the chat our documentations.

So we go through all of this, and then we wanna understand how to release a project. So let's do that in kind of the last minute we have. If you go into the health insights platform, you'll see that it is configured slightly differently. It's more of a top level project that starts with user needs, system requirements, subsystem requirements. So we have really only one user need here, a h a one one o.

We have another system requirement that is actually a risk control. That's why it doesn't have a user need and all the downstream effects of that. We can also see that it is a graph if we want. We can see all the different requirements. So we can see all the different requirements from the system, like one we just talked about and how it interacts with the subsystem and the different aspects of the subsystem that we just saw in this, kind of traceability matrix.

And then we can go and release. So we see here we have one version being actively monitored from a post market surveillance standpoint. I can see the version two, what needs to happen. I see that, basically, there's a bunch of milestones I need to work on, all kinds of controls and items that need to be done, a release checklist that helps me understand what I need to do next, and how people in my company can operate while they're operating in so many systems across the TPLC. And at the end of the day, I see that what I'm missing is to have kind of an approval of the underlying project, that next project, that llama project.

So if I go back there and I just approve that project, let's go here. Let's jump into this version one point one. I see it's ready to be approved. I kind of reduced a lot of controls here, so the milestones are not ready just to make it easier to demo. I approve this.

Pardon eleven compliant biometric approval, and soon, confetti. Confetti pops. We did a release. It's very exciting and wonderful. And now we can go back into this project and see that that's been updated.

As we can see, this has changed. It's now ready to be approved, because this underlying product version has been approved. We approve this, and then we can release our our our system. Now all of these can block your release if you want. We're just trying to do a demo to make it a little easier, and then approve this version.

Before I approve it, I just wanna show you a hundred percent of my design controls are done. A hundred percent of my test executions have been run. I can see that I didn't modify anything, because again, it's a demo version. Change vulnerabilities, change impact analysis, if I had one. Risk controls that are being tested and passed.

All kind of controls we can do here. I approve this, and we are good to go. We just released a complex medical device systems. We showed that we went from Jira to Git back. And by the way, we generated all the needed documentation as part of that.

That. For example, from the p c from this TPLC guidance, our model card, our bias analysis report, and many other things like a testing report, our problem report, our system specification. And I'll just show this just as an example of of how this could look. This is the typical Ketryx configuration. You're welcome to use it.

Others can use, their templates, and that's kind of the whole thing. Thank you everybody for the time today.