---
title: "Understanding the FDA's New Draft Guidance on AI-Enabled Devices"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/wka2tljlft"
content: auto-caption transcript, proper-noun corrected
---

# Understanding the FDA's New Draft Guidance on AI-Enabled Devices

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/wka2tljlft)

---

Thank you everyone for joining us today. Welcome to today's webinar. We're here to understand a little bit more about FDA's new draft guidance on AI enabled medical devices. Just before we get started, some housekeeping, especially if you just joined just a couple seconds ago. This webinar will be recorded, and you will have some slides and the recording sent out afterwards.

If any questions come up, we have, a q and a functionality within Zoom. So we'll go ahead and try to answer your questions through that q and a functionality, as well as some folks, in our sort of backroom can also answer some questions as well. So feel free to put them into the webinar anytime. And after this webinar, we'll be sending out a feedback survey. Here at Ketryx, we really champion continuous improvement.

So, we really wanna hear your feedback so we can make it better for you. Right? So this this is a resource for you and the medical device community as well as the wider community. So we really like to be the best out there. Alrighty.

So quick introduction. Hi. My name is Jen. I've been talking a lot so far. A little bit about me.

So I am a current client operations manager here at Ketryx. So I joined the company about a year ago. I've been learning a lot about platform, all the different use cases we can use it for, and how we can better a lot of our clients or improve our clients' software device, software medical device life cycle practices. Sorry. That's a bit of a mouthful.

How did I get here? Well, I actually used to be in quality and regulatory for a medical device company called Omnitcient Neurotechnology. And while I was there, I was really frustrated with the tooling that was available out there. So we were building class two medical devices. And I also have previous experience in the neurological visualization field.

And, yeah, that's what brought me to Catchix. I looked at the platform. I loved it, and I thought this is gonna really change the way we develop devices and make it go so much faster. I'm also a professional licensed engineer, for those of you, in Canada who can maybe recognize that. And joining me today is Eric Henry.

Eric, I'm so excited to hear about what your thoughts are on this latest guidance. Eric, you come joining us today with over twenty five years of software quality experience. So you've been there at Medtronic. You've been there at GE, Phillips, and now you're here to or you've been helping a lot of people, work with the FDA, at K and Spalding, and working on litigation and acting as an expert witness. Yep.

Thank you. Good to be here. I'm really excited to get your thoughts on the latest guidance, because because I always think that you always bring a very unique perspective on things because, you know, those interactions with the FDA, they can sometimes be, you know, once every couple years for some people, but for you, that's every day. Right? So you can really see a cross section of what's going on in the industry.

That's right. That's right. It's, there's a lot there's a lot of moving parts going right now. Absolutely. Yes.

I know we'll get into that straight away, very soon. So I'm just gonna quickly share the results of the poll that was put into the Zoom, menu. So as you can see, you know, that first question, how do you think the FDA's recent changes will impact AI medical device approvals? It looks like a lot of folks think it's gonna really slow down approvals. What do you think about that, Eric?

Do you somewhat agree, disagree? Yeah. I'm I'm of two minds about this. I I think it it could possibly slow down approvals, because of some of the staffing changes. Although I was encouraged to see that a lot of the folks in the digital health space, both on the policy side and the inspection side and the review side, that that lost their jobs a couple of weekends ago have now gotten them back.

So we we do have some of the expertise back in the house now, but I'm also not sure how the culture is gonna change, if any, in some of those spaces. So if the culture is more tightly focused on the regulations and guidance, then I agree there could be some slowdowns here, if there's more layoffs and and issues that could cause that to happen. If, however, the culture changes more towards less regulation of AI, which we see in other parts of the executive branch right now, that may actually speed things up, and they may actually be encouraging some of these teams to exercise more enforcement discretion. So I can't say anybody with any answer on this is wrong right now. Jury is still very much out.

Yes. And I really appreciate your insights on that. I mean, it was really great to hear that some of the important people, in the FDA came back, and they're still, you know, they still wanna do good work, and really happy and thankful for all the hard work they do to protect us. The second poll, I think, is a little bit more tactical and less so, you know, in the broader picture. So what are the challenges, what challenges do you expect in bringing AI enabled products to market?

It seems like folks are thinking about all sorts of different challenges. So documentation being a big one, validation. I know there's been a lot of discussion about validation, and, of course, navigating the evolving regulatory requirements. And, of course, that ties into what you were talking about, Eric, about, you know, some things might require more enforcement discretion or rather less, and that can definitely change what the regulatory requirements are. Agree.

Alrighty. So let's get us started. So I think you touched upon this. There's a lot of we're kind of, like, at a crossroads at this point where, you know, we are here today, and we have some sort of framework about how to build AI devices, and we're kind of bringing them to the market now. And with the latest changes, you know, we could go potentially one of two ways.

Right? So we could see more AI in the future. You know, we have widespread AI adoption amongst different use cases. But there's also reality out there where things slow down. I think a lot of people are maybe a little bit unsure, feel like maybe the FDA might not approve so many innovative models.

So it could be possible that particularly in some fields like a high risk medical device field, you might see less AI adoption. I don't like you, I don't have the answer. It's very much an it depends situation, but, I think we really are at a very interesting time in medical device development. Alrighty. So for today's agenda, we'll just go through the AI, draft guidance.

We'll discuss a little bit about, you know, what is draft versus final mean. And my goal here today is for everyone on this webinar to come away with a little bit more tactical practical steps that they can apply in whatever context they're working in. So I assume if you come to this webinar, you wanted to learn a little bit more about AI, wanted to learn a little bit more about how to submit potentially an AI device to the FDA. And I'm hoping that at the end of this webinar, you have a couple more things that you can do and feel a bit more confident approaching this, and feel very confident talking about the guidance the next time you're in a conversation about it. So AI regulation or or guidances around, AI use in medical devices have been coming out for the past couple years.

Right? So in twenty nineteen, twenty twenty one, we had some very high level level sort of aspirational frameworks of how we would like to have AI built into our products. And every year or so, we've had a little bit more information, a little bit more grounded knowledge, practical steps that we can do to build it into our medical device platforms. And we are here now. In January twenty twenty five, not just in the medical device industry, but also in the pharma industry, the FDA released a bunch of new guidance documents around how to use AI in health care functions.

Eric, so this is a draft guidance now. Do you know when you think it'll be final? I think there's I think Vegas should get involved in that, answer. I I as as some of you may know, there is a there is an executive order out, that was that that president Trump, released not too long ago, on deregulation intent to deregulate and and to lower the regulatory burden in the executive branch. And that that executive order requires a ten for one exchange of any new regulations, to old to existing.

And so now that that definition is very broad. In section five of the executive order, that's defined as memos, administrative notes, guidances, regulations, and so on. But in order for them to make this rule final, they'll have to get rid of ten other somethings. So I don't know. You know, if I was a betting man a week ago, I would have said probably not released final in the next four years.

Now I would say it's still predominantly my my predominant view now is still it won't be final in that period, but it could be. So I I I but I I would I would caution people not to think because it's not final, it doesn't represent the FDA's current thinking. You may still see some of the language come out in things like inspections and deficiency letters. That's right. So, you know, even though it's not final, I guess, you know, you've seen some letters and things that actually cite guidance documents that haven't been finalized yet, and that's, you know, going to push your organization, your practices into a certain direction.

Right. Alrighty. Well, guidances are really, really fantastic, because they help us understand what are the sort of different things, ways we can interpret the regulatory requirements and, you know, align with FDA's latest thinking. And they help us manage some of the hardest problems when it comes to AI, which is change management. So AI is naturally a very iterative process.

And if you're planning to go into the world of AI and medical devices, devices, you're going to be dealing with these headwinds. So these are all things that you will need to tackle if you're going to be an AI medical device manufacturer. So AI has been around for a while. And with that, there's many, many additional concerns. You know, you have the classic ones, like regulatory compliance.

That's what we're discussing today. But also clinical validation, ethical concerns, and market adoption. Right? So people need to feel confident with your device, and the FDA knows that. Right?

They know that just because you put AI in a device doesn't mean that clinicians or the end user or patient really wants to use it straight away. They wanna feel confident and have that transparency. And that's why the guidance talks about transparency so much. So with that, let's dive into this guidance a little bit. What I love about, this guidance specifically so it talks about all the different software sections that you need to submit AI specific information to, and that, in a way, acts as a form of gate.

Like, it's a it's a milestone that you know that you need to provide this amount of information at this point in the product's life cycle. And that sets you up for, you know, what needs to happen before as well as what needs to happen afterwards when you monitor and postmark surveillance. So folks on the call who have submitted, using the latest ESTAR submission format, this should look very familiar to you. These are the main sections of an ESTAR submission, especially if you're doing, like, a five ten k. And we're gonna dive into, a few of these where the guidance document talks about what specific AI information needs to be submitted to these sections.

So let's get started with the big first one, which is device description. So for those of you unfamiliar with the device description, it's a very critical foundational section of your submission. It kinda tells the regulators, what are the key details they need to know in order to adequately, evaluate your device. Isn't that right, Eric? This is kinda like the setup, the foundation.

It is. And and I'll tell you, FDA sort of front end loads a lot of the information they wanna see into that device description. If you look at the premarket software guidance, you see that, and then you look at this one as well. A lot of the things they wanna see you put in, submit to them on software, are stuffed into that one that one section. So I can imagine, you know, it needs a lot of attention.

It needs to meet a lot of requirements. Right? Mhmm. Absolutely. And, are there some common pitfalls with the device description?

Like, what do you see go wrong with that section? Yeah. It the the issues that I've seen in deficiency letters coming from the FDA, in device description for software in particular, and, obviously, nobody submitted using this draft guidance yet, although there is some AIML language in the premarket software guidance that was finalized last year or the year before. But the the the lack of of clearly defined the intended patient population tends to become an issue. The details around the training data, where did it come from, how is it cleaned, how is it representative of the intended population, so on tends to be a problem.

Companies often don't put in here some of the required elements around transparency, how they're doing that, how they're ensuring transparency. The the one big a couple of big issues that that come up with the FDA is if you're using an AI system, it's obvious you're automating what was some manual process at some point. What is the degree of automation versus manual work that would engage a health care professional? And then finally, I think one of the big issues that I see is, if there's any external interfaces between this system and other systems, they often don't see the level of detail there that they think is important. For sure.

For sure. And I think that the part about how much of it is automated versus manual, that workflow, sort of, positioning, I think that's discussed here a bit in the guidance. Right? So describe specifically what is manual versus automatic. What I like about this guidance, what it says about AI is that, first of all, it says you have to put AI in your device description if it incorporates AI, which may seem obvious, but, you know, it doesn't necessarily say that, so some people don't do it.

But now that it's here, we gotta put it in there. And and I and I will say, Jen, you know, one of the things that with all due respect to our friends in the UK at MHRA who have who have created a classification of device called AI as a medical device, AI is never a medical device. AI is a component of a medical device, a software item if you wanna use the IEC standard language, of a medical device within a larger system. And the FDA is very clear in this guidance that you should, in the device description, lay out not just the AI functionality, but but the entire device, and you just should describe the entire device. You're absolutely right.

For sure. One thing to keep in mind, if you're, you know, building this device yourself and you're trying to work on your device description, chances are you're gonna be iterating on it a lot. You're gonna be revising it a lot. So make sure that you keep track of all your changes, and it really fulfills all those requirements as you change the device description. Alrighty.

Next up, labeling, which is perhaps my favorite section of any, submission. You can tell I'm a regulatory person. So labeling's always been very important. The user interface is talked about a lot in this guidance. And the user interface being considered labeling, that's not new.

Right, Eric? No. That's not new at all. That's been around since the nineties. And I think what's happening over here is that FDA is further reinforcing that, you know, the user interface, particularly for software.

Right? That is your labeling. That's honestly maybe the best way for you to disclose residual risk is directly in the user interface as the person is using the product. And that's where you're gonna see a lot of, you know, warnings, how you should be using the device. So it makes sense to me that the the FDA has really got a very strong opinion about how much you should disclose in your labeling section.

And and and they also sort of more so than they have in previous guidances, they're clear on why they think that's important. And they they are very clear that this labeling helps to explain and interpret the output of the AI system to its users and to patients. And that is the primary function of of the the enhanced labeling provisions within the guidance. And in fact, all of appendix d in the guidance is is exclusively dedicated to this this topic. For sure.

For sure. And I think labeling really contextualizes what this device is all about. I think when folks come to me, medical device manufacturers come to me and they say, hey. So, like, we wanna build an AI device that does this, this, and this. I'm like, well, show me, like, what is it what's the interface?

You know? What are you telling the what you want the user to do? Yep. Alrighty. On to software, which is probably the bulk of the submission that involves, AI.

So data management is a very big portion of this guidance. So it seems like a very straightforward ask. Keep track of your data. Keep track of how you're annotating it. Keep track of how you're visioning it.

Have audit trails of, you know, where have you assigned the data to, you know, testing versus training versus validation, so proper, you know, V and V. The guidance even makes a distinction about V and V. Right? It talks about how, you know, in the past, maybe in a data science realm, validation might be used actually more in a prototyping sense. Validation data should be absolutely kept separate from that sort of initial iteration loop of prototyping an AI model.

And then focused on this in the guidance in the draft guidance because, historically, this has just been a recurring theme in submissions pretty much since day one of FDA, reviewing AI reviewing AI submissions is people not adequately segregating training and and validation data. And so have you seen companies really struggle with that? Like, really struggle with proving that they segregated their data or they have managed their data very well as they built their model? I've seen it kind of all over the place, and and, frankly, I think, companies will either completely skip this topic or they overthink it, in the submissions. In a five ten k submission, the FDA is just looking for some confirmation that you've done this as opposed to providing, you know, the entire data structure and how the data set is is segregated and and and all the details of that.

But so I I would say you've gotta you've gotta come up with a you gotta have a good balance. Give them confidence you've done the segregation. Give them a sense of how you've ensured that, you know, at least from the standpoint of of how you have representative data, that that representative sort of cross section and the the proportion of data within certain categories is similar or the same between training and testing data. And I think if you can do that effectively, you've made your point, you know, kind of then you can just move on into execution. For sure.

And maybe one of the best ways to drive this is using a risk based approach. Right? If you have a model that's very sensitive to data, maybe explain a bit more or you confirm that a bit more. Less so, maybe a bit less. That's right.

That's right. In PMA submission, you're obviously gonna have to get a lot more verbose and be a lot more descriptive. Exactly. Alrighty. Model description and development.

So, you know, nothing crazy here. They wanna see a description of the model and how you developed it. It's really important that when you're doing this, you have a nice design history for maybe not just the entire device itself, but maybe just the model. So you might have a different team working on it, and it's really important that that team is documenting how they're, changing the design, how they're optimizing, that sort of thing. Eric, do you expect a lot of variation here?

Like, if you were looking at a submission and you looked at the model description, do you expect there to be a lot of different types of models out there, or do you feel like most of them are pretty static at this point? They're not really continuously learning? Like, what are you seeing out there? Yeah. I mean, right now I mean, obviously, if you were to if you were to create a a a pie chart of what FDA has cleared so far and and the the the different slices of the pie were, locked versus adaptive, and then maybe within the adaptive even going into, like, a generative model, locked would be one hundred percent of the entire pie.

There have been zero, adaptive or or even further than that generative AI models that have been cleared so far by the FDA. So they're very locked, very static at this point, and a lot, you know, a lot easier to describe, than if you were to have something that was continuously learning and and making changes unattended in the field. And that's why, you know, going back to our earlier point, we're fine we're kind of at a crossroads right now. You know, that could be very different within a couple of years. It could.

And and and it it could definitely. And I know that, you know, there's a couple of ways that can happen. Right? The FDA could help sort of rearchitect or adjust existing regulatory frameworks to make, to answer some of the change management big change management questions, which is some in some ways, the big question for adaptive algorithms. Right?

Which is how do you ensure that something that changes without going through a design change process is still safe and effective in the field after that change. It was cleared as such, but based on evidence. How do you get that evidence in place? And so the FDA can either make changes to the regulatory framework, or they can find ways for companies to meet those regulatory requirements inside an adaptive architecture that we don't currently have in place. And both of those things may be true or either of them, but one of those two things has to happen in order for make that next step.

It's almost as if your PCCP kinda needs to be not just a document, but, you know, embedded into the device requirements and design and, like, it's part of the functionality in there. Yeah. And, you know, the the whole PCCP question is a big question in and of itself. Right? Should you or shouldn't you?

Sometimes the juice is not worth the squeeze, in doing a predetermined change control plan. And the predetermined change control plan guidance as it currently is written is heavily, heavily weighted towards locked algorithms and makes it very difficult to execute an adaptive architecture, but FDA has been clear that they're open to these discussions in the context of maybe doing a Q sub discussion around PCCP. So, you know, even though that that you're kinda discouraged to do that in the PCCP guidance, it might be still haven't worth having a discussion if you're interested in doing it. For sure. For sure.

Eric, we have one question here from the audience. So, it's actually directed to you. So, Eric, can you put in here how you classify the locked versus adaptive? So I think we want a little bit more information on how you what you mean by that. Yeah.

So it's it's a good question. And I was actually speaking to cybersecurity conference last week where some of these people were in the AI world and some work. So when I use those terms, I got a lot of weird eyeballs. A locked algorithm is one that where, once it is trained on a a dataset, and you release that system into the field, there are no further changes to the algorithm based on maybe new data that could come in. In order to make those changes in a locked algorithm, you have to really actually bring that system back to the mother ship, make the changes, go through the design controls and design change process, and then rerelease it the same way we did in nineteen ninety five.

Only you don't use this gets this time probably. The a an adaptive algorithm is one that quite likely can make changes very rapidly, maybe even in an unattended fashion in the wild, in the field, based on new data that it gathers in its real world operating environment. And so the reason that the FDA has such issues with this right now is that when I submit a device that is locked and the FDA has cleared that device, it is based on the requirements, the architecture, the design, the algorithm description, the verification that was done, the validation of the entire device in that context. I don't have that evidence for the changes that have been made if it if it works unattended in the field, if it makes those changes unattended in the field without human intervention. So that's where the that's where the conundrum, get get gets us to.

And of course, if you're in a generative AI space like a ChatGPT or Copilot or a Gemini, you're in the place there then where, it's learning all the time from the Internet, from whatever large language model data set that it has, and in some cases, like those three I mentioned, it's the entire Internet all the time every day. And so it's constantly making changes, and the FDA would just find that frankly unacceptable. Yes. Because you have to demonstrate control over those changes all the time. That's correct.

That's correct. Alrighty. Let's, skip on ahead to the next space. So we talked about risk, a little bit earlier. Risk management, of course, is a very big part of your submission.

Would you say that so risk management has been center stage for a really long time. Do you think AI risk management is really that much different? Are usability and clinical workflow risks, like, more of a concern now? I I there are some new risks that come with, with the AI system. And in the draft guidance, they really focus on understandability and interpretability, of the AI recommendation as a primary risk.

So that so I I think, yes, there are some new risks. They're really primarily in that user usability space. For sure. For sure. And we're gonna hopefully see a couple of those, in the demonstration ahead.

We have a couple of q and a questions. We'll actually have a little bit of time closer to the end of the webinar, to answer some of those live. So I just wanna make sure that folks who are on the call right now are able to get to the remaining sections. So onto cybersecurity, which you mentioned earlier, Eric. Cybersecurity and AI are interestingly well intertwined.

The guidance talks about some very new AI risks just because, you know, AI risks are relatively, have only existed for a couple years, and it talks about how different cybersecurity vulnerabilities kinda exist with AI itself. Right? So they have they talk about data poisoning, model stealing, performance drift. You know, you might not think of that as a you know, someone's trying to hack you kind of situation, but, certainly, when, you know, there's any sort of breach of confidentiality, integrity, and availability of the data or a thing you're trying to access, that's definitely a cybersecurity risk. So do you think that organizations need to change their cybersecurity life cycle practices for AI specifically?

Not at all. I I think that, I think what it is is taking the tools that we have available to us now, like threat models and vulnerability assessments, software bills of material, which, by the way, are a statutory requirement for medical device companies now after December of twenty twenty two, and be able and use those those elements, enhance them with these different types of, threats and vulnerability types. And I would say to your point earlier about things like, performance drift and even overfitting, as nuts it's seeming as as appearing to be a a weird fit in a cybersecurity section of the draft guidance. We have to keep in mind that cybersecurity addresses both intentional and unintentional issues with confidentiality, integrity, and availability. And so that's why those are those are sitting there.

For sure. I've really enjoyed all the latest guidances that have come out in the past couple years around cybersecurity. I think it's really gonna do wonders for improving, you know, all those, those situations that are rising in hospitals of people getting into networks. We're really all part of a greater system, and it's important that we do our part. And I would say too that people need to understand the relationship between cybersecurity and risk management.

Right? We just talked about risk assessment, risk management. Cybersecurity, the threat models, they're an input to that. So in your threat model, always identify which of those threats and the vulnerabilities that those threats exploit are safety related, and then link those to your safety risk management. It's it's hugely important.

And and in fact, you could even think of a threat model like an FMEA on steroids. It's just that it's that one of those links, one of those inputs to the risk management file that you need to take very carefully. We get so many questions of how to do that from a practical sense. Sense. Like, you have so many risk matrices you need to manage, and they all need to relate to each other.

It's really helpful if you're able to break it down into you know, by different teams and different sort of inputs and outputs and really map that process so that, you know, your threat modeling, your vulnerability management, they all help each other make, the device safer. Alrighty. And last couple of sections, verification validation. The devil's in the details when it comes to this. You know, you might have a very different v model of or or idea of how you wanna do design put verification validation.

At Ketryx, you know, we like to use automation as much as possible, and I think automation will be absolutely essential when it comes to, continuous validation of an AI model. So there's many things you can do because, you know, it it's an AI model. You don't necessarily need to build it up on a bench and test it every single time. There's a lot of opportunity to just automate your model and just make sure that it has boundaries and constraints. So every time it changes, you have maybe a base set of data that you run against it, and you make sure that it's not spewing out something that is completely, not conforming.

Right? So that's why verification and validation, you know, you should expect to see a mixture of automated as well as manual tests. And, of course, your strategy here is absolutely essential to your submission. How you, verify and validate your model will be under intense scrutiny, by the FDA. Is that correct?

It is. And in fact, Jim, I'm a huge fan of the smoke test, sanity test type automated concept that you're talking about. And in fact, I've used that, I a an automated or even manual smoke test or sanity test that goes with every change in addition to the the the test you run specific to that change and what that change impacts, as a defense during inspections where the FDA felt that maybe we weren't covering all the functionality we should have or we can't prove to them why we should only test the things we did. With a good sanity or smoke test that covers sort of a wide range of functionality at a shallow level, you sort of you get rid of that that question, and you show that the the device is still functioning with the change. I think that's a really, really good, suggestion and recommendation.

I think, you know, a smoke test, most people do it. They feel confident with that. And you can always run it anytime you want. You don't necessarily have to run it, you know, version to version anytime you change the product. Sometimes you just wanna check.

That's right. We actually have a a question from, the audience. Can reports from services like weights and biases be used to track model design and performance to support submission documentation? Yes. They absolutely can.

In fact, how you're handling bias really needs and this is another one of those risk there's another yet another one of those risk analysis thingies, that you you need to do to be able to show that you're that you're mitigating some of these risks. Right? So when you when you're looking at managing bias within the system, you absolutely have to to treat that almost like a risk analysis. Right? First, identify what biases are intended and unintended, because inevitably, no data set is perfect.

You will likely have to build intended bias into the model, in order to accommodate some weaknesses in the data. And if and when you do that, you need to identify that, but then split out those biases that are unintended and that you don't want, and treat it like you would any risk in a risk analysis. Identify what the mitigation is, clearly use that identified mitigation as an input to the design, and then and then track it through the verification process like you would anything in a DFMEA or, or a risk analysis. Exactly. I love that.

Introducing biases of, like, form of the design. Yep. Alrighty. Also, in verification and validation, something you might wanna consider is device performance monitoring. So I think the FDA hasn't been overly, opinionated about, you know, making sure that you monitor your devices in production in a way that's something you can do with something like a SaaS platform or a SAMD that's sort of live.

They wanna see, you know, a description of how you plan to monitor the device performance, and it obviously very contextual to what sort of technology platform, you're using. Eric, I know the EU has requirements for post market clinical follow-up. Do you think some of these strategies are good enough? Or, like, do you need to build in design features into your actual product in order to do this performance monitoring to the FDA's, requirements? Yeah.

The problem is FDA's requirements, you need to use air quotes when you when you discuss because their requirements at this point are, yes, have some. And we don't have a lot of detailed criteria around what they wanna see. They're very adamant and becoming more so over time that they want post market monitoring. The the guy that just left as a director of Digital Health Center of Excellence, Troy Tazbaz, he and and, Bob Caleb, the former FDA commissioner, very heavily focused on the need for good post market monitoring to make sure that the the algorithm has cleared, continues to perform within those cleared parameters. So they definitely want it, but there's not any consensus on how that's to be done, what you're tracking, how you would report an issue, how people might become aware of that, even at at a regulator level or at an industry level or a public level.

So, I think you do need to build those those tools in, build them into the device itself to to allow those metrics to be captured and reported, have some monitoring capabilities within the company if you have something that's widely distributed that you can monitor, and clearly communicate within the submission how you're gonna be doing that. I think more and more, the FDA is gonna become less and less tolerant, for you to just rely on things like complaint management as your means for post market surveillance. Absolutely. And it's almost a, you know, a better way to lead a submission and say, like, hey. So, like, we have built this.

We understand somewhat how it performs, and we're gonna really pay attention to how it performs in the field, and we found every possible way to mitigate this. And that's how we know it's safe and effective. That's right. Alrighty. Just wrapping up here before we sort of dive into some of the tactical elements.

There's also a public submission summary, something to keep in mind if you're going to be preparing them. How can you best protect your IP when it comes to these five ten k's, PMAs, because, you know, knowing some of it's already public? Yeah. I I don't know that I can answer that question with a single good answer in terms of how you protect your IP. Right?

It's it's my my answer is gonna be a really lame one, which is make sure that your your general counsel's office in your company or your outside legal counsel, if you don't have a a good internal legal counsel that understands AI systems, looks at what you're disclosing. And and this is both in terms of the something like this, a public submission summary, if I can shift to cybersecurity for a second, when in your coordinated vulnerability disclosure program, you're also going to need to identify who you inform, what you tell them, and when. All of that, you know, has the potential to disclose patient privacy, patient records, and and patient information when it shouldn't. It has the potential to expose intellectual property, and other things that you may not and sometimes just, is sort of a little over your skis in terms of giving out too much information too early, to others that that are outside your organization. So my real piece of advice here is get a lawyer involved, make sure that you're communicating well, and and do it, and that would be my strategy.

Heard. I like that. Get a lawyer involved. I always think that, you know, you can have an internal opinion about something, but certainly getting a second opinion and a lawyer involved is always the right way to do it, especially considering, you know, this is a high risk scenario. You don't want this rejected.

You don't wanna, like, leak IP into a place that you don't want it to go. Alrighty. Last thing. I'm just gonna briefly touch upon model cards are one thing that they describe in the guidance. If you worked in the data science industry before that, you know, kinda separate from medical device, you might have heard it as well to find a way to describe what a model is doing, what are the inputs, outputs.

Certainly, if you are developing models, you may wanna include up to updating your model card as part of your definition of done if you're using an agile process. This is something where you don't wanna wait till the very end to write this model card. Obviously, you can use things like ChachiPT to summarize parts of the design history file. That's something we can actually do within, our product. But certainly, you want to keep stay agile with this and make sure that, you know, you're not accidentally submitting a very old version of a model card.

Yeah. And there's actually a really great appendix in the guidance that gives you a good example of this. And and by the way, if you're used to seeing some of the early nutrition labels and model cards that have been published by organizations like HealthAI partnership out of Duke or the Coalition for HealthAI that was started by MITRE and Mayo. This is a lot more detailed than those. There's a lot more here than either of those, and those are primarily designed for the health care delivery organizations, and for their use.

But it's been leveraged in the medical device space as well. This goes quite a few steps beyond that level of detail. Yes. When I saw that, I was like, wow. I wish I had this, like, three years ago.

This is exactly what I wanted to build. That's right. That's right. Alrighty. So one thing we we just wanted to reiterate.

So we we talked about the guidance, and the guidance talks a lot about, you know, here are all the things that need to be in there. When it comes to day to day, you want to be iterating on your design file. You don't wanna wait till the very end to put it together. And one thing we certainly champion at, Ketryx is, you know, being agile, you know, shifting left, making sure that you're not just writing a device description, your model card, your V and V, all at the very end. We wanna move that as much as possible to when the work happens.

If you're, you know, used to software development, you're used to probably working in an agile, maybe a scrum or a sprint. And all the practices you maybe saw earlier involve, you know, incorporating risk management as part of your sprints. So have a an update to the risk management file as you build out your model. So one thing I'd like to demonstrate is some of the things that were highlighted on the slides earlier, and how you can do that practically and tactically. So starting off with the device description, that high trafficked area that probably gets iterated on a lot.

So if you imagine actually, let me take a step back. Let me just drop into Ketryx over here. We talked about the model description being a section in your software description. So, you know, what you might wanna do is split up your teams where you might have a model team that works on something like medical, l l a m a, llama, whatever model you wanna call it. It might be a bespoke, you know, model that you're branding yourself.

If that's leveraged in the, you know, wider device system, as you said, Eric, you know, an AI in itself is not a medical device. It's once you, you know, put it in a clinical context, you wrap wrap a UI around it, that's when it starts, becoming something that's clinically relevant. You might still need to, from a tactical standpoint, break it up so that certain people are working in one project and another project is managed by maybe your systems engineers who incorporate that work. Jumping over to the device description, so here we have a device description that, I wrote it for Ketryx. If you imagine our product as a medical device, we build our product to sort of medical device quality.

But, though, you won't see anything like an intended use here that has, you know, clinical outcomes. So this device description actually lives in Jira because what we wanna do is potentially bring it into a sprint and update it if we find that it's necessary. So it has its own life cycle. You can manage it. Right now, it's classified as an IFU because we actually expect it to maybe show up in the, instructions for use, or maybe in a submission later down the line.

And what you can see is that, you know, this can be pulled into a sprint into your workflow. And every time there's been a change, you have an audit trail of how that change goes because, Jira today is not necessarily Part eleven compliant out of the box, and we found a way to integrate with your existing workplace in Jira, to generate the audit trail for you. So why is this useful? Like, what what is the you know, we could obviously stick this into a doc file, and just, you know, iterate on it there. One thing that's super useful about bringing it into Jira is, you know, obviously, you can use it in sprints, but also because we can deploy AI agents to monitor this regularly.

So if I'm a quality and regulatory person, I don't wanna spend every single Friday afternoon checking whether the device description has changed and making sure that it has met all of those requirements. So what I can actually do is I can deploy an AI agent, and I think I mentioned this previously in a slide, you know, using AI agents and ChatGPT and all those, like, generative AI functionalities. What I can actually do is run an AI agent maybe every Monday, to check that this device description is meeting all those original results. So if I go ahead here, I can actually say, yes. Please check that anytime someone changes this or once a week, this gets revised.

Please make sure that it still has a statement that AI is used in the device and has device inputs and outputs. And what that means is that on a Saturday morning sorry. On Monday morning, I can come in and I can get a report on whether things have changed or not. So I'll get a little buzz on my phone where I'll get an email about what happened and what they thought, and I can actually get a readout of, you know, do we still meet those requirements? I think, Eric, you mentioned that there's a lot of different guidance documents, that drive what goes in your device description, and it can be very tedious to update and make sure you're still meeting all those requirements as you update along.

Yep. True. And in particular, the premarket software guidance, that was, finalized, the the updated in two thousand twenty three as well as the, the cybersecurity guidance, and the, and this AI draft guidance as well. So, there are a lot piled in there. For sure.

And as you can see, this AI agent has actually highlighted that it does, in fact, include AI in the device description, but it's missing this part about inputs and outputs. So that's something that I'll probably have to address on a Tuesday. Alrighty. Now I think a big portion of the guidance obviously talks about verification validation, and we talked about automated tests being something that you can leverage and smoke testing as well. So one thing that Ketryx can do is, you we can have a master traceability matrix, which combines both automated tests as well as, manual tests, and you can trace all the way up to maybe your model training, your use cases.

Here, you can see that there was an automated test pulled in. One thing we didn't talk too much about was, you know, how do you actually manage all of your data? Where do you put it? Are you putting it in doc files? I don't think so.

Right? Eric, where have you heard people are managing their their data these days? Oh, well, the most advanced tool out there, of course, is Microsoft Excel. But, you know, you know, it's it's it's all over the place, really. And and by the way, there are people that are that are actually haven't gotten themselves out of the horrible Excel habit.

But, you know, every kind of SQL or structured database, structure that you can think of is is being used right now. It's it's really all over the place. For sure. And, also, I can imagine different teams wanna use different tools as well. Like, you might have more than one data science team that uses, GitHub or they wanna use, Excel.

We support the ability to integrate into GitHub, so you can actually, drop your raw data into GitHub, and you can actually cite that in your traceability matrix. You can actually trace it all the way up to maybe your, prototyping stage, or you can cite it specifically in your test cases. So you can see it's linked out into GitHub. Yeah. And and and, really, I mean, any configuration management tool, will do.

And I think that people should should the the thing that I think people don't often do is treat their data with the same configuration management discipline that they do with the executable software. Last minute thing or, you know, they're really focused on the design. Right. Risk management. So make sure that when you are developing your device, consider the AI risks.

We talked about that a lot. You know, if I have a risk over here that's talking about hallucinations, now some people I think maybe we had a discussion earlier about hallucinations might be a feature, might be a bug depending on how you see it. Either way, false or misleading, outputs from a generative AI model is a risky thing, especially if it cannot be detected. So you will want to build in risk control measures that will control that risk. So minimize it as far as possible.

So here you can see that I've linked, instructions for use as not quite a risk control, but related to this specific risk and how we are disclosing that residual risk to the end user. And what's nice about once it becomes, you know, an item that you track and you iterate on is that you can also verify that item, as part of your release process. So I can actually build out a separate traceability matrix that's just focused on labeling, which I know is something I've done a lot. I can't tell you how many labeling reviews I've been on where I had to scroll through all of the FDA requirements and health Canada requirements for labeling and check, you know, you know, have you got your name in place of the business? You don't want someone to accidentally remove that.

And it's really nice to have it all tidied and, you know, verifying your labeling is just very straightforward. I I will use this opportunity, however, to preach my sermon about ISO fourteen nine seventy one's hierarchy of risk controls, where labeling is the third and least preferred method for, for risk control measures, and you should really look to design or protective measures as your primaries, where at all possible. Exactly. Exactly. Alrighty.

And at the very end of your release, you likely wanna generate a bunch of documents that align super well with your submission package. So if you, you know, you've seen all the documents that need to be attached in a five ten k or a PMA, what we really preach and we really, think is the right thing to do is to always be submission ready. So always be ready to submit whether you want, you know, a revised risk management report. You can actually build in, as I've mentioned earlier, your model card into your definition done. You can get that finalized and approved by your head of data science.

You might have a bias analysis report maybe specific to your technology if you think that is particularly vulnerable to bias, and you can, of course, have labeling review as a final deliverable as well. We have other webinars which cover document generation, but I strongly recommend you take a look at them. We we have a very robust way to take all the information that you have in Jira as well as in Ketryx and generate them to documents that basically put you in a position where you can just attach directly from your downloads folder. Alrighty. I think I've shown you probably a little just enough about Ketryx.

I wanna save some time at the end because I know that we have a couple more questions that we wanna get answered. But before we do that, just want to remind everyone, we actually have another webinar coming up that I think we'll touch on document generation. But, you know, it's it's about being audit ready. So how do you, pardon me, building an audit ready, secure SDLC. I'm sorry.

I've got a bit of a mouthful of words today, and that's with Gabriel and David. David is a wonderful director of client operations. He worked on so many, he's worked with so many clients to build their SDLC. And, Gabriel, you might have seen before, excellent at at demonstrating our product. And hopefully, everyone here can join us as well.

Before we jump into sort of, like, the final q and a, I just wanna make sure I thank you, Eric, for joining me. Thank you so much for taking the time out of your very busy day. If folks wanna reach you, this is how they get in touch with you. Right? So they can reach you through LinkedIn.

They can reach through you, reach out to you through case law dot com. Mhmm. And, yeah, just anything in particular you feel like you would like to see more of from folks or anything in particular you feel like you're a specialist at? Well, gosh. I mean, I'm I'm it's been I've been around a long time.

Thirty five plus years in industry, and as you said, about twenty five of those in medical devices. So, you know, if if you've got a, any issue in terms of of compliance or FDA interactions or, technology questions around the in the medical device space or even the health care space more broadly, do quite a bit. And, of course, I will give a little bit of a plug for our FDA and life sciences practice at King and Spalding. The firm's about fourteen hundred attorneys altogether, about forty of them dedicated just to FDA and life sciences work, and about two hundred and thirty more broadly in health care. So, from a legal services point of view, and then they have people like me as well as, some ex FDAs and some ex, Harvard medical professors that function as advisers similar to me, but in different more specialized areas.

So, feel free to reach out. Wonderful. I love, like, how diverse you have in terms of, like, a team. Right? So you don't just have you.

You have, like, the medical, authorities as well. It seems like you can always connect with someone who has the answer to the question you're looking for no matter how specific or circumstantial it is. That's right. And the attorneys put us on in on staff for that reason. Right?

We all come with with, you know, a lot of gray in whatever hair we have left, and, and and and we're there to sort of just put the benefit of our years decades of experience. At thirty five plus years, I'm actually the junior guy in the, in the on the non attorney advisory team. But they know they can come to us, not have to pay extra for it, and just get the interest they need right away. So happy to be there for that. Awesome.

Well, thank you so much for joining me today, Eric. If you'll hang on for just a couple of minutes, we'll answer a couple of these live questions. Folks who are also on the call, maybe need a run to their next meeting, thank you so much for joining me.